SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The proof is completed in four steps: first, we prove that the Picard process closely tracks the flow (Flow), and then we bound the distance between the Picard process and the interpolation (Int). By using martingale convergence arguments, employing our assumption on bias Assumption 4, and Gronwall inequality, we conclude the proof of the WAPT property. Lastly, we show that dissipativity on average ensures a uniform bound on the second moments of the iterates. The convergence is then implied by invoking [23, Theorem 3].
In summary, according to Theorem 1, the convergence of the stochastic approximation scheme in (SAA) can be reduced to its continuous-time counterpart in (Sys-SDE) when the assumptions stated in Section 3.3 are satisfied.
Our Theorem 1 can be combined with existing results in the finite particle regime to establish the overall convergence towards the desired mean-field limit. To see this, let  $M_t^{i,N}$  be  $N$  independent processes, synchronously coupled with (Sys-SDE), each starting from  $X_0^i$  and following
$$
d M _ {t} ^ {i, N} = b \left(M _ {t} ^ {i, N}, \mu_ {t}\right) d t + \sigma \left(M _ {t} ^ {i, N}, \mu_ {t}\right) d W _ {t} ^ {i, N},
$$
where  $\mu_t$  is the mean-field solution. A common phenomenon in the study of interacting particle systems, known as uniform propagation of chaos [8, 22, 27, 47], implies the existence of a constant  $C$  such that for every  $N$ :
$$
\sup  _ {t \geq 0} \frac {1}{N} \sum_ {i = 1} ^ {N} \mathcal {W} _ {2} ^ {2} \left(X _ {t} ^ {i, N}, M _ {t} ^ {i, N}\right) \leq \frac {C}{N}, \tag {12}
$$
where  $X_{t}^{i,N}$  represents the particles following the continuous-time dynamics in (Sys-SDE). Letting  $\mu_{\infty}$  represent the limit of the mean-field equation, a straightforward application of the triangle inequality argument, which we defer to Appendix B.2, yields:
$$
\lim  _ {k \rightarrow \infty} \frac {1}{N} \sum_ {i = 1} ^ {N} \mathcal {W} _ {2} ^ {2} \left(x _ {k} ^ {i}, \mu_ {\infty}\right) \leq \frac {C}{N} \rightarrow 0 \quad \text {a s} \quad N \rightarrow \infty . \tag {13}
$$
In other words, as the number of particles approaches infinity, the law of the empirical distribution of the particles following the discrete-time algorithm (SAA) also converges to the mean-field solution.
$\S$  A note on the literature. The rich body of literature on McKean-Vlasov SDEs and interacting particle systems offers considerable insights on the convergence of Euler-Maruyama and Milstein type numerical schemes to their limiting mean-field equations, e.g., [2, 28, 41]. However, it is noteworthy that our study diverges in several key respects: Firstly, our work emphasizes on generic stochastic and biased drift oracles. This contrasts with the deterministic and unbiased drift oracles considered in the aforementioned studies, making our algorithmic approach broader in scope. Secondly, while those studies present strong finite-time error bounds, our convergence results focus on providing asymptotic guarantees. Lastly, we incorporate different underlying assumptions. For instance, we need global Lipschitz drifts to ensure globally integrable flows, while one-sided Lipschitz drifts are allowed in the works by [2, 28, 41]. However, our growth condition in Assumption 2 requires control on average, whereas the works mentioned assume stronger pointwise controls.
In the light of these distinctions, we believe that our work complements this body of literature.
The goal of this section is to demonstrate the wide-ranging applicability of our framework across diverse domains such as machine learning, game theory, and physics, which were previously analyzed in isolation. In each of these applications, we demonstrate that standard assumptions in their respective domains meet the necessary conditions to invoke Theorem 1. All proofs are deferred to Appendix C.
4.1 Two-Layer Neural Networks and Mean-Field Langevin
Our first application is providing a rigorous guarantee for the training dynamics of wide two-layer neural networks as alluded to in Section 2. To begin, let us quickly recall the notations therein: The noisy SGD iterates in  $(\mathrm{SGD}_{\sigma})$  for wide two-layer neural networks can be viewed as approximations of the mean-field dynamics (6) through the discrete-time system (SAA), whose drift term  $b(\cdot ,\cdot)$  is defined in (1). The corresponding continuous-time and finite-particle dynamics is (5), which has been extensively studied in recent years; see [11, 12, 36-38, 46, 51] and references therein. Under the standard assumptions as in [11, 12, 37], a simple application of Theorem 1 then yields the convergence of  $(\mathrm{SGD}_{\sigma})$ :
Corollary 1. Let  $\kappa(\cdot)$  denote the activation function. Assume that (1)  $\kappa$  and  $\kappa'$  are Lipschitz and bounded, (2) the data has bounded support, and (3)  $|a\kappa'(a)|$  is bounded. Then the discrete-time scheme  $(\mathrm{SGD}_{\sigma})$  converges in  $\mathcal{W}_2$  to the same limit sets as the continuous-time (5).
The mean-field dynamics for two-layer neural networks converges to a stationary point in Wasserstein space due to its gradient flow structure [12]. In addition, it is known that under fairly mild assumptions, the resulting limit becomes a unique global risk minimizer [12]. By employing uniform propagation of chaos for neural networks [46], it is observed that the limit sets of the continuous-time  $N$ -particle dynamics exhibit similar generalization error to the global minimizer. Remarkably, Corollary 1 validates that noisy SGD and other discretizations following template (SAA) eventually converge to this limit set, providing justification for the observed generalization behavior in neural networks.
$\S$  Comparison to prior work. Our result above cannot be directly compared to the existing analysis conducted on the discrete-time scheme  $(\mathrm{SGD}_{\sigma})$ , namely [37], due to the assumption A1 therein, which excludes step-size rules of the form  $\gamma_{k+1} \sim k^{-\beta}$  where  $\beta \in (1/2, 1]$ . Conversely, the bounds provided in [37] for a constant step-size are non-asymptotic (albeit doubly-exponential), resulting in stronger conclusions compared to our asymptotic results. Therefore, these two analyses complement each other and contribute to a more comprehensive understanding of the training dynamics exhibited by neural networks.
4.2 Stein Variational Gradient Descent
Sampling from a distribution  $\pi \propto e^{-V}$  is a crucial task in various machine learning applications. An effective method that has demonstrated practical success in this regard is the Stein variational gradient descent (SVGD) [29, 30]. Intuitively, this method emulates the steepest descent for the KL divergence in continuous-time with a continuous probability measure. In practice, the algorithm is implemented as an interacting particle system as follows (see [33] for a derivation). Let  $K:\mathbb{R}^d\times \mathbb{R}^d\to \mathbb{R}$  be a positive definite kernel. The SVGD algorithm updates the set of  $N$  particles as
$$
x _ {k + 1} ^ {i} = x _ {k} ^ {i} - \frac {\gamma_ {k + 1}}{N} \sum_ {j = 1} ^ {N} \left(K \left(x _ {k} ^ {i}, x _ {k} ^ {j}\right) \nabla V \left(x _ {k} ^ {j}\right) - \nabla_ {2} K \left(x _ {k} ^ {i}, x _ {k} ^ {j}\right)\right), \quad i = 1, \dots , N, \quad (\operatorname {S V G D} _ {k})
$$
where  $\nabla_2K$  is the gradient of  $K$  with respect to its second input. The corresponding continuous-time dynamics is then:
$$
\frac {d}{d t} X _ {t} ^ {i} = \frac {1}{N} \sum_ {j} \nabla_ {2} K (X _ {t} ^ {i}, X _ {t} ^ {j}) - \frac {1}{N} \sum_ {j} K (X _ {t} ^ {i}, X _ {t} ^ {j}) \nabla V (X _ {t} ^ {j}), \quad i = 1, \dots , N, \quad (\operatorname {S V G D} _ {t})
$$
which is a special case of (Sys-SDE) with  $b(x, \mu) = (\nabla_2 K * \mu)(x) - (K * (\mu \nabla V))(x)$  and  $\sigma \equiv 0$  (no diffusion), where  $*$  denotes the convolution operator. We now prove:
Corollary 2. Suppose that (1)  $\nabla V(x)$  is Lipschitz, (2)  $V$  is dissipative, $^3$  (3) for some  $C > 0$ ,  $|\nabla V(x)| \leq C(1 + |x|^2)$ , (4)  $\|K\|_{\infty}, \|\nabla_2 K\|_{\infty}, \|\nabla^2 K\|_{\infty} < \infty$ , (5)  $|\nabla_2 K(x, y)| \leq \eta / |x - y|$ , and (6)  $K(x, y) \leq \eta / |x - y|^2$  for some  $\eta > 0$ . Then the iterates in  $(\mathrm{SVGD}_k)$  converge in  $\mathcal{W}_2$  to the same limit sets as the continuous-time process  $(\mathrm{SVGD}_t)$ .
3That is,  $\exists m > 0$ $m^{\prime}\in \mathbb{R}$  such that  $\langle x,\nabla V(x)\rangle \geq m|x|^2 -m'$
Similar to neural networks training discussed in Section 4.1, the mean-field SVGD also exhibits a gradient flow structure, and in this case, the target distribution  $\pi$  serves as the only limit set for the mean-field SVGD [29]. Combining Corollary 2 with a uniform propagation of chaos argument for SVGD then confirms that the iterates of  $(\mathrm{SVGD}_k)$ , or any discretization of  $(\mathrm{SVGD}_t)$  that satisfies Assumption 4, effectively converge to a distribution that closely approximates  $\pi$  in terms of  $\mathcal{W}_2$ . This outcome underscores the effectiveness of these algorithms in sampling.
$\S$  Comparison to prior work. While there has been extensive theoretical work on the convergence of SVGD, most of it focuses on either the population limit (i.e., when  $N \to \infty$  in  $(\mathrm{SVGD}_k)$ ) or the vanishing step-size (which directly examines the properties of  $(\mathrm{SVGD}_t)$ ) [10, 14, 25, 29, 43, 45]. However, the convergence behavior of the discrete iterates  $(\mathrm{SVGD}_k)$  remains a challenging task with limited success. To the best of our knowledge, the only existing work in this direction is [44], but its result is not directly comparable to ours. Although our assumptions on the bounded derivatives of  $V$  and  $W$  are similar to those in [44], the difference lies in the requirement imposed on the target distribution  $e^{-V}$ . Specifically, Shi and Mackey [44] assume a T1-inequality on  $e^{-V}$  [50], whereas our work requires  $V$  to be dissipative. Notably, our approach holds a significant advantage over [44] in terms of simplicity: The result of [44] is only applicable to a highly specific and fairly complicated step-size rule (see [44, Cor 2]), whereas our sole requirement on  $\gamma_{k+1}$  is the standard Assumption 5. However, it should be noted that the bounds in [44] are non-asymptotic (albeit doubly-exponential in  $N$  and exponential in  $k$ ), while we can only handle asymptotic convergence.
Remark. It is worth mentioning that Corollary 2 can be straightforwardly extended to the Stochastic Particle-Optimization Sampling algorithm of Zhang et al. [53], which is identical to SVGD but includes a constant diffusion term  $\sigma >0$ . Since the analysis remains the same, we omit the details.
4.3 Two-Player Zero-sum Continuous Games
Min-max learning appears in several important machine learning tasks such as Generative Adversarial Networks [18] and adversarial training [35]. These learning problems can be formulated as a continuous zero-sum game between a min-player and a max-player. The min-player selects strategies from the set  $\mathcal{X} \subset \mathbb{R}^d$ , while the max-player selects strategies from the set  $\mathcal{Y} \subset \mathbb{R}^d$ , with the goal of finding a saddle point of a function  $K(x, y)$ :
$$
\min  _ {x \in \mathcal {X}} \max  _ {y \in \mathcal {Y}} K (x, y). \tag {14}
$$
However, solving (14) becomes challenging and sometimes impossible when  $K$  is non-convex in  $x$  and non-concave in  $y$ , as a solution to (14) may not even exist. To address this issue, mixed Nash equilibriums (MNEs) are introduced, where the pure strategies are replaced by probability distributions over the sets of strategies, which exist under mild assumptions on  $K$  [13, 17].
Specifically, an MNE is represented by a pair of measures  $(\mu^{\star}, \nu^{\star}) \in \mathcal{P}_2(\mathcal{X}) \times \mathcal{P}_2(\mathcal{Y})$ , forming a saddle point of the functional  $E(\mu, \nu) \coloneqq \iint K(x, y) \mu(dx) \nu(dy)$ . The quest for efficient solutions to the MNE problem in machine learning has led to the development of particle-based methods that offer approximate solutions [13, 20, 34]. However, the existing studies have primarily focused on analyzing the continuous-time dynamics of these methods. Below, we shift our focus to the more practical setting of the discrete-time system of interacting particles in these work:
$$
\left\{ \begin{array}{l} x _ {k + 1} ^ {i} = x _ {k} ^ {i} - \gamma_ {k + 1} \frac {1}{N} \sum_ {j} \nabla_ {x} K \left(x _ {k} ^ {i}, y _ {k} ^ {j}\right) + \sqrt {2 \tau \gamma_ {k + 1}} \xi_ {k + 1} ^ {i} \\ y _ {k + 1} ^ {i} = y _ {k} ^ {i} + \alpha \gamma_ {k + 1} \frac {1}{N} \sum_ {j} \nabla_ {y} K \left(x _ {k} ^ {j}, y _ {k} ^ {i}\right) + \sqrt {2 \alpha \tau \gamma_ {k + 1}} \zeta_ {k + 1} ^ {i} \end{array} , \right. \tag {GDA_k}
$$
where  $\xi_{k + 1}^i$  and  $\zeta_{k + 1}^{i}$  are independent standard Gaussians,  $\tau >0$  is a hyperparameter chosen by the user, and  $\alpha$  is the scale difference between the two players [34]. We will additionally consider the optimistic version of  $(\mathrm{GDA}_k)$  in [20], which has shown empirical benefits over  $(\mathrm{GDA}_k)$ :
$$
\left\{x _ {k + 1} ^ {i} = x _ {k} ^ {i} - \gamma_ {k + 1} \frac {1}{N} \sum_ {j} \left(2 \nabla_ {x} K \left(x _ {k} ^ {i}, y _ {k} ^ {j}\right) - \nabla_ {x} K \left(x _ {k - 1} ^ {i}, y _ {k - 1} ^ {j}\right)\right) + \sqrt {2 \tau \gamma_ {k + 1}} \xi_ {k + 1} ^ {i} \right. \tag {OGDA}
$$
$$
\left\{y _ {k + 1} ^ {i} = y _ {k} ^ {i} + \alpha \gamma_ {k + 1} \frac {1}{N} \sum_ {j} \left(2 \nabla_ {y} K (x _ {k} ^ {j}, y _ {k} ^ {i}) - \nabla_ {y} K (x _ {k - 1} ^ {j}, y _ {k - 1} ^ {i})\right) + \sqrt {2 \alpha \tau \gamma_ {k + 1}} \zeta_ {k + 1} ^ {i}. \right.
$$
At first glance, it may seem that  $(\mathrm{GDA}_k)$  and  $(\mathrm{OGDA}_k)$  are distinct algorithms that cannot be analyzed together. However, our subsequent corollary reveals that these systems actually converge to the same
continuous-time dynamics:
$$
\left\{ \begin{array}{l} d X _ {t} ^ {i} = - \frac {1}{N} \sum_ {j} \nabla_ {x} K \left(X _ {t} ^ {i}, Y _ {t} ^ {j}\right) d t + \sqrt {2 \tau} d W _ {t} ^ {i} \\ d Y _ {t} ^ {i} = \frac {\alpha}{N} \sum_ {j} \nabla_ {y} K \left(X _ {t} ^ {j}, Y _ {t} ^ {i}\right) d t + \sqrt {2 \alpha \tau} d B _ {t} ^ {i} \end{array} , \right. \tag {GDA_t}
$$
where  $W_{t}^{i}$  and  $B_{t}^{j}$  are collections of i.i.d. Brownian motions. The key enabling factor for this unified treatment is to allow for a non-zero bias in the algorithmic template (SAA).
To facilitate the analysis, we introduce a pairing of players' particles by setting  $Q_{t}^{i} \coloneqq (X_{t}^{i},Y_{t}^{i}) \in \mathbb{R}^{2d}$  and define the drift  $b: \mathbb{R}^{2d} \times \mathcal{P}_2(\mathbb{R}^{2d}) \to \mathbb{R}^{2d}$  as
$$
b (q, \mu) := \int \left( \begin{array}{c} - \nabla_ {x} K (q _ {1}, q _ {2} ^ {\prime}) \\ \alpha \nabla_ {y} K (q _ {1} ^ {\prime}, q _ {2}) \end{array} \right) \mu (d q ^ {\prime}), \quad q = (q _ {1}, q _ {2}),
$$
and the diffusion  $\sigma (q,\mu)\coloneqq \sqrt{2\tau}\mathrm{diag}(I_{d\times d},\sqrt{\alpha} I_{d\times d})$  . By doing so, we can then cast  $(\mathrm{GDA}_t)$  in template of (Sys-SDE).
Corollary 3. Assume that (1)  $\nabla_{x}K$  and  $\nabla_{y}K$  are Lipschitz, and (2)  $\nabla_{x}K$  and  $-\nabla_{y}K$  are dissipative, or (2') the domains  $\mathcal{X}$  and  $\mathcal{Y}$  are bounded. Then the algorithms  $(\mathrm{GDA}_k)$  and  $(\mathrm{OGDA}_k)$  converge in  $\mathcal{W}_2$  to the same limits of  $(\mathrm{GDA}_t)$ .
We note that the assumptions in (3) are mild and are even required for analyzing the continuous-time dynamics; see e.g., [34]. Consequently, our Corollary 3 provides a rigorous foundation for the consideration of continuous-time dynamics in existing studies such as [13, 20, 34].
Remark. The schemes  $(\mathrm{GDA}_k)$  and  $(\mathrm{OGDA}_k)$  above rely on simultaneous updates, i.e., from  $(x_k, y_k)$ , one obtains  $(x_{k+1}, y_{k+1})$ . However, empirical evidence suggests that alternating updates i.e., following  $(x_k, y_k) \to (x_{k+1}, y_k) \to (x_{k+1}, y_{k+1})$ , often preforms better. Our framework allows for this flexibility, as it is easy to cast the alternating  $(\mathrm{GDA}_k)$  and  $(\mathrm{OGDA}_k)$  as stochastic approximation schemes satisfying Assumption 4, see [24, Proposition 4] for an example of this argument.
4.4 Kinetic Equations
In this section, we study the kinetic equations defined on the space of probability measures. Initially emerging from the physics community, these equations have recently gained attention in the machine learning community due to their connection to Wasserstein gradient flows [15, 32, 39, 40, 48, 49, 52]. Here, we denote  $\rho$  as a probability density, and we examine three distinct "energy" functionals: an internal energy  $\mathcal{U}$ , a potential energy  $\mathcal{V}$ , and an interaction energy  $\mathcal{W}$ . These functionals are defined as follows:
$$
\mathcal {U} (\rho) = \int U (\rho (x)) d x, \quad \mathcal {V} (\rho) = \int V (x) \rho (d x), \quad \mathcal {W} (\rho) = \frac {1}{2} \iint W (x - y) \rho (d x) \rho (d y). \tag {15}
$$
The most interesting scenario is when  $U(s) = s \log s$  represents the entropy, in which case the Wasserstein gradient flow with finite particles becomes [5, 7]:
$$
d X _ {t} ^ {i} = - \nabla V \left(X _ {t} ^ {i}\right) d t - \frac {1}{N} \sum_ {j = 1} ^ {N} \nabla W \left(X _ {t} ^ {i} - X _ {t} ^ {j}\right) d t + \sqrt {2} d W _ {t} ^ {i}, \quad i = 1, \dots , N. \tag {Kin-SDE}
$$
By setting  $b(x,\mu) \coloneqq -\nabla V(x) - (\nabla W*\mu)(x)$  and  $\sigma (x,\mu)\equiv \sqrt{2}$ , we again see that (Kin-SDE) is a special case of (Sys-SDE).
In the physics community, the equation is commonly simulated using a system of interacting particles through the proximal point method [6, 48]:
$$
x _ {k + 1} ^ {i} = x _ {k} ^ {i} - \gamma_ {k + 1} \Big (\nabla V (x _ {k + 1} ^ {i}) + \frac {1}{N} \sum_ {j = 1} ^ {N} \nabla W (x _ {k + 1} ^ {i} - x _ {k} ^ {j}) \Big) + \sqrt {2 \gamma_ {k + 1}} \xi_ {k + 1} ^ {i}. \qquad \mathrm {(K i n - P r o x)}
$$
Note that the right-hand side of (Kin-Prox) involves the next iterates  $x_{k+1}^i$  so that, as opposed to the simple Euler discretization, it is an implicit rule. The key factor that enables the application of our framework to these implicit schemes is the observation that (Kin-Prox) can be formulated as a stochastic approximation scheme in (SAA) by incorporating a non-zero bias term; see Appendix C. Our next result provides a rigorous guarantee for these methods:
Corollary 4. Assume that (1)  $V$  is dissipative and  $\nabla V$  is  $L$ -Lipschitz, (2)  $W$  is symmetric and  $\nabla W$  is  $L$ -Lipschitz, (3) There exists some  $M_W \geq 0$  such that for all  $x, y \in \mathbb{R}^d$ ,  $\langle \nabla W(x) - \nabla W(y), x - y \rangle \geq -M_W$ . Then, the iterates (Kin-Prox) converge in  $\mathcal{W}_2$  to the same limit sets as (Kin-SDE).
We remark that the convergence of (Kin-Prox) under these standard assumptions is known [15, 48]. However, we emphasize that the aforementioned results only apply to deterministic updates, while our proof is robust enough to accommodate updates with finite-variance noise. This flexibility sets our approach apart, allowing for a broader range of practical applications.
In conclusion, our work has successfully bridged the gap between continuous- and discrete-time schemes by establishing the convergence of discrete-time algorithms to their continuous-time counterparts, drawing inspiration from dynamical system theory. This achievement offers a flexible framework that can accommodate practical schemes and unify tasks from various domains, including machine learning, game theory, and physics. By introducing a comprehensive framework for analyzing stochastic approximation schemes, providing convergence proofs, and presenting easily verifiable conditions at the finite particle level, our contributions enhance the understanding and application of stochastic approximation schemes for simulating interacting particle systems.
In our future works, we will explore the exciting possibilities offered by the dynamical system theory of [4] to derive convergence rates through  $\lambda$ -pseudotrajectories. This avenue of research will allow us to establish exponential convergence for dynamics using smaller step-sizes after a "burn-in" time, thereby sharpening our understanding for the long-term behavior of these practical schemes.
Additionally, we aim to relax the Lipschitzness assumption on the drift and expand the scope of guaranteed algorithms, such as the Ensemble Kalman Sampler [16]. We also aim to explore further applications in natural sciences, including stochastic mean-field FitzHugh-Nagumo models and networks of Hodgkin-Huxley neurons [1]. These models are important because they provide a mathematical description of neuronal dynamics, contribute to our understanding of neurological disorders, and inform the development of brain-computer interfaces. By leveraging our novel framework, we hope to offer rigorous guarantees for the algorithms employed in these domains while also designing new, efficient approaches that can support computational neuroscience research.
[1] Javier Baladron, Diego Fasoli, Olivier Faugeras, and Jonathan Touboul. Mean Field description of and propagation of chaos in recurrent multipopulation networks of Hodgkin-Huxley and Fitzhugh-Nagumo neurons, October 2011.
[2] Jianhai Bao, Christoph Reisinger, Panpan Ren, and Wolfgang Stockinger. First-order convergence of milstein schemes for McKean-vlasov equations and interacting particle systems. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 477 (2245), January 2021. doi: 10.1098/rspa.2020.0258. URL https://doi.org/10.1098/rspa.2020.0258.
[3] Michel Benaim. Dynamics of stochastic approximation algorithms. In *Seminaire de probabilités XXXIII*, pages 1–68. Springer, 2006.
[4] Michel Benaim and Morris W Hirsch. Asymptotic pseudotrajectories and chain recurrent flows, with applications. Journal of Dynamics and Differential Equations, 8(1):141-176, 1996.
[5] Dario Benedetto, Emanuelle Caglioti, José A Carrillo, and Mario Pulvirenti. A non-maxwellian steady distribution for one-dimensional granular media. Journal of statistical physics, 91: 979-990, 1998.
[6] Lev M Bregman. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR computational mathematics and mathematical physics, 7(3):200-217, 1967.