SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kernel-based machine learning methods use in general an RKHS as the hypothesis space, and learning often reduces to a search or optimization problem over this function space. For this reason, it is
<sup>3</sup>It is customary in the kinetic theory literature to switch to such a subsequence. However, for some results that are about to follow, it is important that no further switch to a subsequence happens, hence we need to be more explicit in these cases.
important to investigate the approximation properties of a given kernel and its associated RKHS as well as to ensure that the learning problem over an RKHS (which is in general an infinite-dimensional object) can be tackled with finite computations.
The next result asserts that, under a uniformity condition, the approximation power of the finite-input kernels  $k_{M}$  is inherited by the mean field limit kernel.
Proposition 3.1. For  $M \in \mathbb{N}_{+}$ , let  $\mathcal{F}_M$  be the set of symmetric functions that are continuous w.r.t.  $(\vec{x},\vec{x}^{\prime}) \mapsto d_{\mathrm{KR}}(\hat{\mu}[\vec{x}],\hat{\mu}[\vec{x}^{\prime}])$ . Let  $\mathcal{F} \subseteq C^{0}(\mathcal{P}(X),\mathbb{R})$  such that for all  $f \in \mathcal{F}$  and  $\epsilon > 0$  there exist  $B \in \mathbb{R}_{\geq 0}$  and sequences  $f_M \in \mathcal{F}_M$ ,  $\hat{f}_M \in H_M$ ,  $M \in \mathbb{N}_{+}$ , such that 1)  $f_M \xrightarrow{\mathcal{P}_1} f$  2)  $\|f_M - \hat{f}_M\|_{\infty} \leq \epsilon$  for all  $M \in \mathbb{N}_{+}$  3)  $\|\hat{f}_M\|_M \leq B$  for all  $M \in \mathbb{N}_{+}$ . Then for all  $f \in \mathcal{F}$  and  $\epsilon > 0$ , there exists  $\hat{f} \in H_k$  with  $\|f - \hat{f}\|_{\infty} \leq \epsilon$ .
Intuitively, the set  $\mathcal{F}$  consists of all continuous functions on  $\mathcal{P}(X)$  that arise as a mean field limit of functions which can be uniformly approximated by uniformly norm-bounded RKHS functions. The result then states (to use a somewhat imprecise terminology) that the RKHS  $H_{k}$  is dense in  $\mathcal{F}$ . We can interpret this as an appropriate mean field variant of the universality property of kernels: a kernel on a compact metric space is called universal if its associated RKHS is dense w.r.t. the supremum norm in the space of continuous functions, and many common kernels are universal, cf. e.g. [20, Section 4.6]. In our setting, ideally universality of the finite-input kernels  $k_{M}$  is inherited by the mean field limit kernel  $k$ . However, since the mean field limit can be interpreted as a form of smoothing limit, some uniformity requirements should be expected. Proposition 3.1 provides exactly such a condition.
Remark 3.2. In Proposition 3.1, the set  $\mathcal{F}$  is a subvectorspace of  $C^0 (\mathcal{P}(X),\mathbb{R})$ . Furthermore, if the  $\mathcal{P}_1$ -convergence in the definition of  $\mathcal{F}$  is uniform, then  $\mathcal{F}$  is closed.
Since  $k_{M}$  and  $k$  are kernels, we have the usual representer theorem for their corresponding RKHSs, cf. e.g. [27]. A natural question is then whether we have mean field convergence of the minimizers and their representation. This is clarified by the next result.
Theorem 3.3. Let \(N \in \mathbb{N}_{+}\), \(\mu_{1},\ldots ,\mu_{N} \in \mathcal{P}(X)\) and for \(n = 1,\dots ,N\) let \(\vec{x}_n^{[M]} \in X^M\), \(M \in \mathbb{N}_{+}\), such that \(\hat{\mu} [\vec{x}_n^{[M]}\] \(\xrightarrow{d_{\mathrm{KR}}}\mu_n\) for \(M \to \infty\). Let \(L:\mathbb{R}^N\to \mathbb{R}_{\geq 0}\) be continuous and strictly convex and \(\lambda >0\). For each \(M \in \mathbb{N}_{+}\) consider the problem
$$
\min  _ {f \in H _ {M}} L \left(f \left(\vec {x} _ {1} ^ {[ M ]}\right), \dots , f \left(\vec {x} _ {N} ^ {[ M ]}\right)\right) + \lambda \| f \| _ {M}, \tag {5}
$$
as well as the problem
$$
\min  _ {f \in H _ {k}} L \left(f \left(\mu_ {1}\right), \dots , f \left(\mu_ {N}\right)\right) + \lambda \| f \| _ {k}. \tag {6}
$$
Then for each  $M \in \mathbb{N}_{+}$  problem (5) has a unique solution  $f_{M}^{*}$ , which is of the form  $f_{M}^{*} = \sum_{n=1}^{N} \alpha_{n}^{[M]} k_{M}(\cdot, \vec{x}_{n}^{[M]}) \in H_{M}$ , with  $\alpha_{1}^{[M]}, \ldots, \alpha_{N}^{[M]} \in \mathbb{R}$ , and problem (6) has a unique solution  $f^{*}$ , which is of the form  $f^{*} = \sum_{n=1}^{N} \alpha_{n} k(\cdot, \mu_{n}) \in H_{k}$ , with  $\alpha_{1}, \ldots, \alpha_{N} \in \mathbb{R}$ . Furthermore, there exists a subsequence  $(f_{M_{\ell}}^{*})_{\ell}$  such that  $f_{M_{\ell}}^{*} \xrightarrow{\mathcal{P}_{1}} f^{*}$  and
$$
L \left(f _ {M _ {\ell}} ^ {*} \left(\vec {x} _ {1} ^ {[ M _ {\ell} ]}\right), \dots , f _ {M _ {\ell}} ^ {*} \left(\vec {x} _ {N} ^ {[ M _ {\ell} ]}\right)\right) + \lambda \| f _ {M _ {\ell}} ^ {*} \| _ {M _ {\ell}} \rightarrow L \left(f ^ {*} \left(\mu_ {1}\right), \dots , f ^ {*} \left(\mu_ {N}\right)\right) + \lambda \| f ^ {*} \| _ {k}. \tag {7}
$$
for  $\ell \to \infty$
The main point of this result is the convergence of the minimizers, which we will establish using a  $\Gamma$ -convergence argument. This approach seems to have been introduced by [28, 18, 29] originally in the context of multi-agent systems.
Remark 3.4. An inspection of the proof reveals that in Theorem 3.3 we can replace the term  $\lambda \| \cdot \|_M$  and  $\lambda \| \cdot \|_k$  by  $\Omega(\| \cdot \|_M)$  and  $\Omega(\| \cdot \|_k)$ , where  $\Omega: \mathbb{R}_{\geq 0} \to \mathbb{R}_{\geq 0}$  is a nonnegative, strictly increasing and continuous function.
4 Support Vector Machines with mean field limit kernels
We now turn to the mean field limit of kernels in the context of statistical learning theory, focusing on SVMs. We first briefly recall the standard setup of statistical learning theory, and formulate an appropriate mean field limit thereof. We then investigate empirical and infinite-sample solutions of SVMs and their mean field limits, as well as the convergence of the corresponding risks.
Statistical learning theory setup We now introduce the standard setup of statistical learning theory, following mostly [20, Chapters 2 and 5]. Let  $\mathcal{X} \neq \emptyset$  (associated with some  $\sigma$ -algebra) and  $\emptyset \neq Y \subseteq \mathbb{R}$  closed (associated with the corresponding Borel  $\sigma$ -algebra). A loss function is in this setting a measurable function  $\ell: \mathcal{X} \times Y \times \mathbb{R} \to \mathbb{R}_{\geq 0}$ . Let  $P$  be a probability distribution on  $\mathcal{X} \times Y$  and  $f: \mathcal{X} \to \mathbb{R}$  a measurable function, then the risk of  $f$  w.r.t.  $P$  and loss function  $\ell$  is defined by
$$
\mathcal {R} _ {\ell , P} (f) = \int_ {\mathcal {X} \times Y} \ell (x, y, f (x)) \mathrm {d} P.
$$
Note that this is always well-defined since  $(x,y)\mapsto \ell (x,y,f(x))$  is a measurable and nonnegative function. For a set  $H\subseteq \mathbb{R}^{\mathcal{X}}$  of measurable functions we also define the minimal risk over  $H$  by
$$
\mathcal {R} _ {\ell , P} ^ {H *} = \inf  _ {f \in H} \mathcal {R} _ {\ell , P} (f).
$$
If  $H$  is a normed vector space, we additionally define the regularized risk of  $f \in H$  and the minimal regularized risk over  $H$  by
$$
\mathcal {R} _ {\ell , P, \lambda} (f) = \mathcal {R} _ {\ell , P} (f) + \lambda \| f \| _ {H} ^ {2}, \qquad \mathcal {R} _ {\ell , P, \lambda} ^ {H *} = \inf  _ {f \in H} \mathcal {R} _ {\ell , P, \lambda} (f),
$$
where  $\lambda \in \mathbb{R}_{>0}$  is the regularization parameter. A data set of size  $N\in \mathbb{N}_+$  is a tuple  $D_N = ((x_1,y_1),\ldots ,(x_N,y_N))\in (\mathcal{X}\times Y)^N$  and for a function  $f:\mathcal{X}\to \mathbb{R}$  we define its empirical risk by
$$
\mathcal {R} _ {\ell , D _ {N}} (f) = \frac {1}{N} \sum_ {n = 1} ^ {N} \ell \left(x _ {n}, y _ {n}, f \left(x _ {n}\right)\right).
$$
If  $H$  is a normed vector space and  $f \in H$ , we define additionally the regularized empirical risk and the minimal regularized empirical risk over  $H$  by
$$
\mathcal {R} _ {\ell , D _ {N}, \lambda} (f) = \mathcal {R} _ {\ell , D _ {N}} (f) + \lambda \| f \| _ {H} ^ {2}, \qquad \mathcal {R} _ {\ell , D _ {N}, \lambda} ^ {H *} = \inf  _ {f \in H} \mathcal {R} _ {\ell , D _ {N}, \lambda} (f),
$$
where  $\lambda \in \mathbb{R}_{>0}$  is again the regularization parameter. Note that the notation for the empirical risks is consistent with the risk w.r.t. a probability distribution  $P$ , if we identify a data set  $D_N$  by the corresponding empirical distribution  $\frac{1}{N}\sum_{n=1}^{N}\delta_{(x_n,y_n)}$ .
In the following,  $H$  will be a RKHS and a minimizer (assuming existence and uniqueness) of  $\mathcal{R}_{\ell ,P,\lambda}^{H*}$  will be called an infinite-sample support vector machine (SVM). Similarly,  $\mathcal{R}_{\ell ,D_N,\lambda}^{H*}$  will be called the empirical solution of the SVM w.r.t. the data set  $D_N$ . Note that this is the common terminology in statistical learning theory, cf. [20], and corresponds to (empirical) risk minimization with Tikhonov regularization.
Statistical learning theory setup in the mean field limit Let now  $\emptyset \neq Y \subseteq \mathbb{R}$  be compact and  $\ell_{M}: X^{M} \times Y \times \mathbb{R} \to \mathbb{R}_{\geq 0}$ ,  $M \in \mathbb{N}$ , such that 1)  $\ell_{M}(\sigma \vec{x}, y, t) = \ell_{M}(\vec{x}, y, t)$  for all  $\vec{x} \in X^{M}$ ,  $\sigma \in S_{M}$ ,  $y \in Y$ ,  $t \in \mathbb{R}$ ; 2) there exists  $C_{\ell} \in \mathbb{R}_{\geq 0}$  and a nondecreasing function  $b: \mathbb{R}_{\geq 0} \to \mathbb{R}_{\geq 0}$  with  $|\ell_{M}(\vec{x}, y, t)| \leq C_{\ell} + b(|t|)$  for all  $M \in \mathbb{N}$  and  $\vec{x} \in X^{M}$ ,  $y \in Y$ ,  $t \in \mathbb{R}$ ; 3) there exists  $L_{\ell} \in \mathbb{R}_{\geq 0}$  with
$$
\left| \ell_ {M} \left(\vec {x} _ {1}, y _ {1}, t _ {1}\right) - \ell_ {M} \left(\vec {x} _ {2}, y _ {2}, t _ {2}\right) \right| \leq L _ {\ell} \left(d _ {\mathrm {K R}} \left(\hat {\mu} [ \vec {x} _ {1} ], \hat {\mu} [ \vec {x} _ {2} ]\right) + \left| y _ {1} - y _ {2} \right| + \left| t _ {1} - t _ {2} \right|\right)
$$
for all  $\vec{x}_1, x_2 \in X^M$ ,  $y_1, y_1' \in Y$ ,  $t_1, t_2 \in \mathbb{R}$ . In particular, all  $\ell_M$  are measurable (assuming the Borel  $\sigma$ -algebra on  $X^M$ ) and hence are loss functions on  $X^M \times Y$ . Proposition 2.1 ensures the existence of a subsequence  $(\ell_{M_m})_m$  and an  $L_\ell$ -Lipschitz continuous function  $\ell: \mathcal{P}(X) \times Y \times \mathbb{R} \to \mathbb{R}$  with
$$
\lim  _ {M \rightarrow \infty} \sup  _ {\substack {\vec {x} \in X ^ {M _ {m}}\\y \in Y, t \in K}} | \ell_ {M _ {m}} (\vec {x}, y, t) - \ell (\hat {\mu} [ \vec {x} ], y, t) | = 0 \tag{8}
$$
for all compact  $K \subseteq \mathbb{R}$ , and we write again  $\ell_{M_m} \xrightarrow{\mathcal{P}_1} \ell$ . For readability, from now on we switch to this subsequence. Furthermore, we also get from Proposition 2.1 that there exists some  $C_L \in \mathbb{R}_{\geq 0}$  such that  $|\ell(\mu, y, t)| \leq C_L + b(|t|)$  for all  $\mu \in \mathcal{P}(X), y \in Y, t \in \mathbb{R}$ .
Remark 4.1. Note that, for Proposition 2.1 to apply, it is enough to assume in item 2) above the existence of a function  $b: \mathbb{R} \to \mathbb{R}_{\geq 0}$  with  $|\ell_M(\vec{x},y,t)| \leq C_\ell + b(|t|)$ . However, we chose the slightly stronger condition that  $b$  is nondecreasing, since then  $\ell_M$  is a Nemitskii loss according to [20, Definition 2.16]. Since the function with constant value  $C_\ell$  is actually  $P_M$ -integrable, this means that  $\ell_M$  is even a  $P_M$ -integrable Nemitskii loss according to [20]. A similar remark then applies to  $\ell$ .
Lemma 4.2. The function  $\ell$  is nonnegative. Furthermore, if all  $\ell_M$  are convex loss functions [20, Definition 2.12], i.e., if for all  $M\in \mathbb{N}_{+}$ ,  $\vec{x}\in X^{M}$ ,  $y\in Y,t_{1},t_{2}\in \mathbb{R}$  and  $\lambda \in (0,1)$  we have
$$
\ell_ {M} (\vec {x}, y, \lambda t _ {1} + (1 - \lambda) t _ {2}) \leq \lambda \ell_ {M} (\vec {x}, y, t _ {1}) + (1 - \lambda) \ell_ {M} (\vec {x}, y, t _ {2}), \tag {9}
$$
then so is  $\ell$
Empirical SVM solutions Given data sets  $D_N^{[M]} = \left((\vec{x}_1^{[M]},y_1^{[M]}),\ldots ,(\vec{x}_N^{[M]},y_N^{[M]})\right)$  for all  $M\in \mathbb{N}_{+}$  with  $\vec{x}_n^{[M]}\in X^M$ ,  $y_{n}^{[M]}\in Y$ , and  $D_N = ((\mu_1,y_1),\dots,(\mu_N,y_N))$  with  $\mu_n\in \mathcal{P}(X)$  and  $y_{n}\in Y$ , we write  $D_N^{[M]}\xrightarrow{\mathcal{P}_1}D_N$  if  $\hat{\mu} [\vec{x}_n^{[M]}] \xrightarrow{d_{\mathrm{KR}}}\mu_n$  and  $y_{n}^{[M]}\to y_{n}$  (where  $M\to \infty$ ) for all  $n = 1,\ldots ,N$ . We can interpret this as mean field convergence of the data sets.
Furthermore, consider the empirical risk of hypothesis  $f_{M} \in H_{M}$  (and  $f \in H_{k}$ ) on data set  $D_N^{[M]}$  (and  $D_N$ )
$$
\mathcal {R} _ {\ell_ {M}, D _ {N} ^ {[ M ]}} (f _ {M}) = \frac {1}{N} \sum_ {n = 1} ^ {N} \ell_ {M} (\vec {x} _ {n} ^ {[ M ]}, y _ {n} ^ {[ M ]}, f _ {M} (\vec {x} _ {n} ^ {[ M ]})), \qquad \mathcal {R} _ {\ell , D _ {N}} (f) = \frac {1}{N} \sum_ {n = 1} ^ {N} \ell (\mu_ {n}, y _ {n}, f (\mu_ {n})),
$$
and the corresponding regularized risk
$$
\mathcal {R} _ {\ell_ {M}, D _ {N} ^ {[ M ]}, \lambda} (f _ {M}) = \frac {1}{N} \sum_ {n = 1} ^ {N} \ell_ {M} (\vec {x} _ {n} ^ {[ M ]}, y _ {n} ^ {[ M ]}, f _ {M} (\vec {x} _ {n} ^ {[ M ]})) + \lambda \| f _ {M} \| _ {M} ^ {2}
$$
$$
\mathcal {R} _ {\ell , D _ {N}, \lambda} (f) = \frac {1}{N} \sum_ {n = 1} ^ {N} \ell (\mu_ {n}, y _ {n}, f (\mu_ {n})) + \lambda \| f \| _ {k} ^ {2},
$$
where  $\lambda \in \mathbb{R}_{>0}$  is the regularization parameter.
Proposition 4.3. Let  $\lambda >0$ , assume that all  $\ell_{M}$  are convex and let  $D_N^{[M]}$ ,  $D_N$  be finite data sets with  $D_N^{[M]}\xrightarrow{\mathcal{P}_1}D_N$ . Then for all  $M\in \mathbb{N}_{+}$ ,  $H_{M}\ni f_{M}\mapsto \mathcal{R}_{\ell_{M},D_{N}^{[M]},\lambda}(f_{M})$  has a unique minimizer  $f_{M,\lambda}^{*}\in H_{M}$  and  $H_{k}\ni f\mapsto \mathcal{R}_{\ell ,D_{N},\lambda}(f)$  has a unique minimizer  $f_{\lambda}^{*}\in H_{k}$ . Furthermore, for all  $M\in \mathbb{N}_{+}$  there exist  $\alpha_{n}^{[M]}\in \mathbb{R}$ ,  $n = 1,\ldots ,N$ , such that  $f_{M,\lambda}^{*} = \sum_{n = 1}^{N}\alpha_{n}^{[M]}k_{M}(\cdot ,\vec{x}_{n}^{[M]})$ , and there exist  $\alpha_{1},\dots ,\alpha_{N}\in \mathbb{R}$  such that  $f_{\lambda}^{*} = \sum_{n = 1}^{N}\alpha_{n}k(\cdot ,\mu_{n})$ . Finally, there exists a subsequence  $(f_{M_m,\lambda}^*)_m$  such that  $f_{M_m,\lambda}^{*}\xrightarrow{\mathcal{P}_1}f_\lambda^*$  and  $\mathcal{R}_{\ell_{M_m},D_N^{[M_m]},\lambda}(f_{M_m,\lambda}^*)\to \mathcal{R}_{\ell ,D_N,\lambda}(f_\lambda^*)$  for  $m\to \infty$ .
Convergence of distributions and infinite-sample SVMs in the mean field limit We now turn to the question of mean field limits of distributions and the associated learning problems and SVM solutions. Let  $(P^{[M]})_M$  be a sequence of distributions, where  $P^{[M]}$  is a probability distribution on  $X^M\times Y$ , and let  $P$  be a probability distribution on  $\mathcal{P}(X)\times Y$ . We say that  $P^{[M]}$  converges in mean field to  $P$  and write  $P^{[M]}\xrightarrow{\mathcal{P}_1}P$ , if for all continuous (w.r.t. the product topology on  $\mathcal{P}(X)\times Y$ ) and bounded  $^4 f$  we have
$$
\int_ {X ^ {M} \times Y} f (\hat {\mu} [ \vec {x} ], y) \mathrm {d} P ^ {[ M ]} (\vec {x}, y) \rightarrow \int_ {\mathcal {P} (X) \times Y} f (\mu , y) \mathrm {d} P (\mu , y). \tag {10}
$$
This convergence notion of probability distributions (on different input spaces) appears to be not standard, but it is a natural concept in the present context. Essentially, it is weak (also called narrow) convergence of probability distributions adapted to our setting.
Consider now data sets  $D_N^{[M]}$ ,  $D_N$ , with  $D_N^{[M]} \xrightarrow{\mathcal{P}_1} D_N$ , then we also have convergence in mean field of the datasets, interpreted as empirical distributions: let  $f \in C^0(\mathcal{P}(X) \times Y, \mathbb{R})$  be bounded, then
$$
\int_ {X ^ {M} \times Y} f (\hat {\mu} [ \vec {x} ], y) \mathrm {d} D _ {N} ^ {[ M ]} (\vec {x}, y) = \frac {1}{N} \sum_ {n = 1} ^ {N} f (\hat {\mu} [ \vec {x} _ {n} ^ {[ M ]} ], y _ {n} ^ {[ M ]})
$$
$$
\xrightarrow {M \to \infty} \frac {1}{N} \sum_ {n = 1} ^ {N} f (\mu_ {n}, y _ {n}) = \int_ {\mathcal {P} (X) \times Y} f (\mu , y) \mathrm {d} D _ {N} (\mu , y).
$$
4Of course, since  $Y$  is compact, all continuous  $f$  are bounded in our present setting.
This shows that the mean field convergence of probability distributions as defined here is a direct generalization of the natural notion of mean field convergence of data sets.
Finally, consider the risk of hypothesis  $f_{M} \in H_{M}$  and  $f \in H_{k}$  w.r.t. the distribution  $P^{[M]}$  and  $P$ , respectively,
$$
\mathcal {R} _ {\ell_ {M}, P ^ {[ M ]}} (f _ {M}) = \int_ {X ^ {M} \times Y} \ell_ {M} (\vec {x}, y, f _ {M} (\vec {x})) \mathrm {d} P ^ {[ M ]} (\vec {x}, y)
$$
$$
\mathcal {R} _ {\ell , P} (f) = \int_ {\mathcal {P} (X) \times Y} \ell (\mu , y, f (\mu)) \mathrm {d} P (\mu , y),
$$
as well as the minimal risks
$$
\mathcal {R} _ {\ell_ {M}, P ^ {[ M ]}} ^ {H _ {M} *} = \inf  _ {f _ {M} \in H _ {M}} \mathcal {R} _ {\ell_ {M}, P ^ {[ M ]}} (f _ {M}) \quad \mathcal {R} _ {\ell , P} ^ {H _ {k} *} = \inf  _ {f \in H _ {k}} \mathcal {R} _ {\ell , P} (f).
$$
Our first result ensures that mean field convergence of distributions  $P^{[M]}$ , loss functions  $\ell_{M}$  and data sets  $D_N^{[M]}$  ensures the convergence of the corresponding risks of the empirical SVM solutions.
Lemma 4.4. Consider the situation and notation of Proposition 4.3 and assume that  $P^{[M]} \xrightarrow{\mathcal{P}_1} P$ . We then have  $\mathcal{R}_{\ell_{M_m}, P^{[M_m]}}(f_{M_m, \lambda}^*) \to \mathcal{R}_{\ell, P}(f_\lambda^*)$  for  $m \to \infty$ .
Next, we investigate the mean field convergence of infinite-sample SVM solutions and their associated risks. Define for  $\lambda \in \mathbb{R}_{\geq 0}$  (and all  $M \in \mathbb{N}_+$ ) the regularized risk of  $f_M \in H_M$  and  $f \in H_k$ , respectively, by
$$
\mathcal {R} _ {\ell_ {M}, P ^ {[ M ]}, \lambda} (f _ {M}) = \mathcal {R} _ {\ell_ {M}, P ^ {[ M ]}} (f _ {M}) + \lambda \| f _ {M} \| _ {M} ^ {2}, \qquad \mathcal {R} _ {\ell , P, \lambda} (f) = \mathcal {R} _ {\ell , P} (f) + \lambda \| f \| _ {k} ^ {2},
$$
and the corresponding minimal risks by
$$
\mathcal {R} _ {\ell_ {M}, P ^ {[ M ]}, \lambda} ^ {H _ {M} *} = \inf  _ {f _ {M} \in H _ {M}} \mathcal {R} _ {\ell_ {M}, P ^ {[ M ]}, \lambda} (f _ {M}), \qquad \mathcal {R} _ {\ell , P, \lambda} ^ {H _ {k} *} = \inf  _ {f \in H _ {k}} \mathcal {R} _ {\ell , P, \lambda} (f).
$$
Proposition 4.5. Let  $\lambda >0$ , assume that all  $\ell_{M}$  are convex loss functions and let  $P^{[M]}$  and  $P$  be probability distributions on  $X^{M} \times Y$  and  $\mathcal{P}(X) \times Y$ , respectively, with  $P^{[M]} \xrightarrow{\mathcal{P}_1} P$ . Then for all  $M \in \mathbb{N}_{+}$ ,  $H_{M} \ni f_{M} \mapsto \mathcal{R}_{\ell_{M},P^{[M]},\lambda}(f_{M})$  has a unique minimizer  $f_{M,\lambda}^{*} \in H_{M}$  and  $H_{k} \ni f \mapsto \mathcal{R}_{\ell,P,\lambda}(f)$  has a unique minimizer  $f_{\lambda}^{*} \in H_{k}$ . Furthermore, there exists a subsequence  $(f_{M_m,\lambda}^*)_m$  such that  $f_{M_m,\lambda}^{*} \xrightarrow{\mathcal{P}_1} f_{\lambda}^{*}$  and  $\mathcal{R}_{\ell_{M_m},P^{[M_m]},\lambda}(f_{M_m,\lambda}^*) \to \mathcal{R}_{\ell,P,\lambda}(f_{\lambda}^*)$  for  $m \to \infty$ . In particular,  $\mathcal{R}_{\ell_{M_m},P^{[M_m]},\lambda}^{H_{M_m}*} \to \mathcal{R}_{\ell,P,\lambda}^{H_k*}$ .
Finally, we would like to show that  $\mathcal{R}_{\ell_M,P[M]}^{H_M*} \to \mathcal{R}_{\ell,P}^{H_k*}$  for  $P^{[M]} \xrightarrow{\mathcal{P}_1} P$ . Up to a subsequence, this is established under Assumption 4.6. Define the approximation error functions, cf. [20, Definition 5.14], by
$$
A _ {2} ^ {[ M ]} (\lambda) = \inf  _ {f \in H _ {M}} \mathcal {R} _ {\ell_ {M}, P ^ {[ M ]}, \lambda} (f) - \mathcal {R} _ {\ell_ {M}, P ^ {[ M ]}} ^ {H _ {M} *} \qquad A _ {2} (\lambda) = \inf  _ {f \in H _ {k}} \mathcal {R} _ {\ell , P, \lambda} (f) - \mathcal {R} _ {\ell , P} ^ {H _ {k} *},
$$
where  $M \in \mathbb{N}_{+}$  and  $\lambda \in \mathbb{R}_{\geq 0}$ . Note that (for all  $M \in \mathbb{N}_{+}$ )  $A_{2}^{[M]}, A_{2}: \mathbb{R}_{\geq 0} \to \mathbb{R}_{\geq 0}$  are increasing, concave and continuous, and  $A_{2}^{[M]}, A_{2}(0) = 0$ , cf. [20, Lemma 5.15]. We need essentially equicontinuity of  $(A_{2}^{[M]})_{M}$  in  $0$ , which is formalized in the following assumption.
Assumption 4.6. For all  $\epsilon >0$  there exists  $\lambda_{\epsilon} > 0$  such that for all  $0 < \lambda \leq \lambda_{\epsilon}$  and  $M\in \mathbb{N}_{+}$  we have  $A_2^{[M]}(\lambda)\leq \epsilon$
Proposition 4.7. Assume that all  $\ell_M$  are convex loss functions, let  $P^{[M]}$  and  $P$  be probability distributions on  $X^M\times Y$  and  $\mathcal{P}(X)\times Y$ , respectively, with  $P^{[M]}\xrightarrow{\mathcal{P}_1}P$ . If Assumption 4.6 holds, there exists a strictly increasing sequence  $(M_m)_m$  with  $\mathcal{R}_{\ell_{M_m},P^{[M_m]}}^{H_{M_m}*}\to \mathcal{R}_{\ell ,P}^{H_k*}$  for  $m\to \infty$ .
<sup>5</sup>Note that Proposition 4.3 is actually a corollary of this result. However, since the former result is independent of the notion of mean field convergence of probability distributions, we stated and proved it separately.
We investigated the mean field limit of kernels and their RKHSs, as well as the mean field limit of statistical learning problems solved with SVMs. In particular, we managed to complete the basic theory of mean field kernels as started in [22]. Additionally, we investigated their approximation capabilities by providing a first approximation result and a variant of the representer theorem for mean field kernels. Finally, we introduced a corresponding mean field limit of statistical learning problems and provided convergence results for SVMs using mean field kernels. In contrast to other settings involving a large number of variables, for example, infinite-width neural networks, here we considered the case of an increasing number of inputs. This work opens many directions for future investigation. For example, it would be interesting to remove or weaken Assumption 4.6 for a result like Proposition 4.7. Another relevant direction is to find approximation results that are stronger than Proposition 3.1. Finally, it would be interesting to investigate whether statistical guarantees, like consistency or learning rates, for the finite-input learning problems can be transferred to the mean field level.
We would like to thank Noel Brindise, Pierre-François Massiani and Alexander von Rohr for very detailed and helpful comments on the manuscript, and the anonymous reviewers for their detailed and helpful comments. This work is funded in part under the Excellence Strategy of the Federal Government and the Länder (G:(DE-82)EXS-SF-SFDdM035), which the authors gratefully acknowledge. The authors further thank the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) for the financial support through 320021702/GRK2326, 333849990/IRTG-2379, B04, B05, and B06 of 442047500/SFB1481, HE5386/18-1,19-2,22-1,23-1,25-1.