On kernel-based statistical learning theory in the mean field limit
Authors: Christian Fiedler
Michael Herty
Sebastian Trimpe
Venue: NeurIPS 2023 poster (2023)
Keywords: Reproducing Kernel Hilbert Spaces
Kernel Methods
Mean Field Limit
Interacting Particle Systems
Support Vector Machines
Statistical Learning Theory
TL;DR: Mean field convergence results for kernel-based statistical learning methods (convergence of kernels, empirical and infinite-sample risks, and of SVM solutions), in particular, for machine learning in the context of large multiagent systems.

Abstract: In many applications of machine learning, a large number of variables are considered. Motivated by machine learning of interacting particle systems, we consider the situation when the number of input variables goes to infinity. First, we continue the recent investigation of the mean field limit of kernels and their reproducing kernel Hilbert spaces, completing the existing theory. Next, we provide results relevant for approximation with such kernels in the mean field limit, including a representer theorem. Finally, we use these kernels in the context of statistical learning in the mean field limit, focusing on Support Vector Machines. In particular, we show mean field convergence of empirical and infinite-sample solutions as well as the convergence of the corresponding risks. On the one hand, our results establish rigorous mean field limits in the context of kernel methods, providing new theoretical tools and insights for large-scale problems. On the other hand, our setting corresponds to a new form of limit of learning problems, which seems to have not been investigated yet in the statistical learning theory literature.

On kernel-based statistical learning in the mean field limit
Christian Fiedler'
Michael Herty?
Sebastian Trimpe
Instittuee for Data Science in Mechanical Engineering (DSME)
RWTH Aachen University, Aachen, Germany
{fiedler, trimpe)@dsme. www.-wawhhhee.n de
Institute for Geometry and Practical Mathematics (IGPM)
RWTH Aachen University, Aachen, Germany hertySigpm. wwth-aachen..de
Abstract
In many applications of machine learning, a large number of variables are consid- ered. Motivated by machine learning of interacting particle systems, consider the situation when the number of input variables goes to infinity. First, we continue the recent investigation of the mean field limit of kernels and their reproducing kernel Hilbert spaces, completing the existing theory. Next, we provide results relevant for approximation with such kernels in the mean field limit, including a representer theorem. Finally, we use these kernels in the context of statistical learning the mean field limit, focusing on Support Vector Machines. In particu- lar, we show mean field convergence of empirical and infinite-sampii solutions as well as the convergence of the corresponding risks. On the one hand, our results establish rigorous mean field limits in the context of kernel methods, providing new theoretical tools and insights for large-scale problems. On the other hand, our setting corresponds to new form of limit of learning problems, which seems to have not been investigated yet in the statistical learning theory literature.
Introduction
Models with many variables play important role in many fields of mathematical and physical sciences. In this context, going to the limit of infinitely many variables is an important analysis and modeling approach. Interacting particle systems are classic example; these are usually modeled as dynamical systems describing the temporal evolution of many interacting objects. In physics, such systems were first investigated in the context of gas dynamics, cf. [1]. Since even small volumes of gases typically contain an enormous number of molecules, a microscopic modeling approach quickly becomes infeasible and one considers the evolution of densities instead [2]. In the past decades, interacting particle systems arising from many different domains have been considered, for example, animal movement (inter alia swarms of birds, schools of fish, colonies of microorganisms) [3, 4], social and political dynamics [5, 6], crowd modeling and control (pedestrian movement, gathering at large events like football games or concerts) [7-9], swarms of robots [10-12] or vehicular traffic (in particular, traffic jams) [13]. There is now vast literature on such applications, and we refer to the surveys [14-16] as starting points. A prototypical example of such a system is given by à;- M Σe 6(2, (2, - 21), for 1,..., M, where M E N4 particles or agents modelled by their state x, E Rd l, M, evolving according to some interaction rule à : Rdd x d + TR. Typical questions then concern the long-term behavior of such systems, in particular, emergent phenomena like consensus alignment [17]. While first-principles modeling
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

has been very successful for interacting particle systems in physical domains, using this approach to model the interaction rules in complex domains like social and opinion dynamics, pedestrian and animal movement or vehicular traffic, can be problematic. Therefore, learning interaction rules from data has been recently intensively investigated, for example, in the pioneering works [18, 19]. The data consists typically of (sampled) trajectories of the particle states, potentially with measurement noise, and the goal is to learn a good approximation of the interaction rule ø.
Our work is motivated by related problem. Frequently, the state of such a complex multiagent system can be easily measured or estimated, e-g., by video recordings or image snapshots for bird swarms or schools of fish, and microscopy recordings for microorganism colonies; aerial imaging for human crowds (e.g.. via quadcopters); and polling and social media analysis for opinion dynamics. However, some interesting features of the whole system might be more difficult to measure. For example, how a swarm of birds or a school of fish will react to an external stimulus (like an approaching predator), given the current state of the population. Such a reaction could be change of density or spread of the population, or change in mean velocity. Another example given by features of society in opinion dynamics (average happiness, aggression potential, susceptibility to adversarial interventions), given the current "opinion state". Measuring such features can be difficult, for example, due to a required intervention. Formally, such feature is functional FM (R2)) - I of the current state of the system, and since the state is often easy to measure, it would be useful to have an explicit mapping from state to feature of interest. However, since first principles modeling is unlikely to be successful in the domains considered here, is promising to learn such a mapping from data. We can formalize
this as standard supervised learning task: The data set consists of D/! (( VI)..................... (TN, UN)). where F. E (Rd)M are snapshot measurements of the particle states (corresponding to the input of the functional) and yn E IR is the value of the functional of interest, potentially with mcasurement noise, at snapshot state Z,e Let us assume n = l..... N. where E,,..., EN E R are noise variables. This is now a regression problem that could be solved, for example, using a Support Vector Machine (SVM) [20]. Note that for this we need a kernel kas= (RR)) x (Rd) at + R on (R.)).
additive noise model. i.c., Vn. = Fx + en for
Similarly to classical physical examples like gas dynamics, the case of a large number of particles also relevant in modern complex interacting particle systems. Since this poses computational and modeling challenges, it can be advantageous to go also here to a kinetic level and model the evolution of the particle distribution instead of every individual particle. It is well-established how to derive kinetic partial differential equation from ordinary differential equations systems on the particle level, for example, using the Boltzmann equation or via a mean field limit, cf. [17] for an overview in the context of multi-agent systems. Formally, instead of trajectories of particle states of the form [0, 3 t + Z(t)= E ) āt, we then have trajectories of probability measures [0,T] 3 l + v(t) E )( This immediately raises the question of whether the learning setup outlined above also allows a corresponding kinetic limit. More precisely, let K C IRd be compact and assume that all particles remain confined to this compactum, i.c., xz (t) E K for all i = I,..., M and all E [0, T] under the microscopic dynamics.' If the underlying dynamics have a mean field limit, then it is reasonable to assume that the finite-input functionals FAs KM - R converge also in mean field to some F: : P(K) -> for M - oc, see Section 2 for a precise definition of this notion. In turn, we can now formulate a corresponding learning problem on the mean field level: A data set is then given by D~ = ((M1, 9) ),-
yN)), where μnn E P(K) are snapshots of the particle state distribution over time and y,, E IR are again potentially noisy measurements of the functional. Assuming an additive noise model, this corresponds to yn, = F(p.) + 6n, n = I....,N, N. If we want to use an SVM on the kinetic level, we need a kernel k: P(K) x P(K) + IR on probability distributions. There are several options available for this, see e.g. [21]. However, assuming that all ingredients of the learning problem arise as a mean field limit, this naturally leads to the question of whether a mean field limit of kernels exists, and what this means for the relation of the learning problems on the finite-input and kinetic level. In [22], this reasoning has motivated the introduction and investigation of the mean field limit of kernels. In the present work, we extend the theory of these kernels and investigate them in the context of statistical learning theory. In particular, since in practice one would use the mean field kernels on microscopic data with large, but finite M, we need convergence results of the various objects appearing in statistical learning with kemels. Exactly such results are provided in Section 4.
'This means the dynamics on the level of individual particles.
2

---

Finally, we would like to stress that the technical developments here are independent of the motivation outlined above, in that they apply to mean field limits of functions and kernels that do not necessarily arise form the dynamics of interacting particle systems.
Contributions Our contributions cover three closely related aspects. 1) We extend and complete the theory of mean field limit kemels and their RKHSs (Section 2). In Theorem 2.3, we precisely describe the relationship between the RKHS of the finite-input kemels and the RKHS of the mean field kernel, completing the results from [22]. In particular, this allows us to interpret the latter RKHS as the mean field limit of the former RKHSs. Furthermore, in Lemma 2.4 and 2.5, we provide inequalities for the corresponding RKHS norms, which are necessary for --covvegeence arguments. 2) We provide results relevant for approximation with mean field limit kernels (Section 3). With Proposition 3.1, we give first result on the approximation power of mean field limit kernels, and in Theorem 3.3 we can also provide a representer theorem for these kernels. For its proof, we use a -cconvereence argument, which is the best of our knowledge the first time this technique has been used in the context of kernel methods. 3) We investigate the mean field limit of kernels in the context of statistical learning theory (Section 4). We first establish appropriate mean field limit setup for statistical learning problems, based on a slightly stronger mean field limit existence result than available so far, cf. Proposition 1. To the best of our knowledge, this is new form of a limit for learning problems. In this setup, we then provide existence, uniqueness, and representer theorems for empirical and (using an apparently new notion of mean field convergence of probability distributions) infinite-sample solutions of SVMs, cf. Proposition 4.3 and 4.5, Finally, under a uniformity assumption, we can also establish convergence of the minimal risks in Proposition 4.7.
Our developments are relevant from two different perspectives: on the one hand, they constitute theoretical proof-of-concep that the mean field limit can be "pulled through" the (kernel-based) statistical learning theory setup. In particular, this demonstrates that rigorous theoretical results can be transferred through the mean field limit, similar to works in the context of control of interacting particle systems, see e.g. [23]. On the other hand, our setup appears to be a new variant of a large- number-of-variable limit in the context of machine learning, complementing established settings like infinite-width neural networks [24].
Due to space constraints, all proof's and some additional technical results have been placed in the supplementary material.
Kernels and their RKHSs in the mean field limit
Setup and preliminaries Let (X, dx) be compact metric space and denote by P(X) the set of Borel probability measures on X. We endow P(X) with the topology of weak convergence of probability measures. Recall that for M E P(X), we say that ann + f weakly if for all bounded and continuous f x - IR (since X is compact, this is equivalent to I continuous) we have lim fx ((r)d,, + fx (())du(a)). The topology of weak convergence can be metrized by the Kantorowich-Rubinsee metric dxR, defined by
dku( (2) = sup { (())d((1 - (2) o X + R is I-Lipschitz
Note that since X is compact and hence separable, the -antorowichRRubinttein metric is equal to the 1- Wasserstein metric here. Furthermore, P(X) is compact in this topology. For M E N_ and o E X denote the i-th component of by xi, and define the empirical measure for 7 by [ x Σ M 8z where 6e denotes the Dirac measure centered at x E X. The empirical measures are dense in P(X) metric. Additionally, define deR : P(X)² P(X)² + Ezo by dZ ((H1, (12, (H2,)4)) = dRR (M1, R2) + dkR (4)) and note that (P(X)² dh is a compact metric space. Moreover, denote the set of permutations on (I,...,A0 M) by SM. and for a tuple F E X and permutation a E Sy define OF = (Lo(I)s....) (o(M)) Finally, we recall some well-known definitions and results from the theory of reproducing kernel Hilbert spaces, following [20, Chapter 4]. For an arbitrary set X F 9 and a Hilbert space (H, (., +))) of functions on X, we say that map k: x > R. is a reproducing kernel for H if 1) k(.,z) E H for all x E X: 2) for all x E X' and f E H we have f(x) = (J,
w.r.t. the Heittooowwic-RRuiisteeee
Note that if reproducing kernel exists, is unique. such a Hilbert space has a reproducing kernel, we call H reproducing kernel Hilbert space (RKHS) and A its (reproducing) kernel. It is well- that reproducing kernel is
3

---

symmetric and positive semidefinite, and that every symmetric and positive semidefinite function has a unique RKHS for which it the reproducing kernel. For brevity, if k is symmetric and positive semidefinite, or equivalently, if it is the reproducing kernel of an RKHS. we call k simply kernel, and denote by (Hs, (: +)x) its unique associated RKHS. Define also H span span(k((; (k(,,)) x E } then for f = >n= (=, F.)) E H and g = >m= pm Fm) E H we have (f.9)k = EN EM Oi Bn.Bm Tn). and Hg is dense in Hk-
The mean field limit of functions and kernels Given fM : X - IR, M E E. and P(X) - R. we say that fas converges in mean field to and that f is the (or a) mean field limit of saa. if lim x [T))(T) f(a(()))))))))) = O. In this case, write fM Pi f. Let now (Y, dy) be another metric space and J'M X Ad x Y + R. M E N+, andand/ f P(() x Y - IR, then we say that feld to f and that f is the (or a) mean field limit of fM. if for all compact
fas converges in mean field K C Y we have
lim 700 ZEX M yeK
sup [fM( v) - f(Al)) 0.
and also write fas Pi f. The following existence results for mean field limits is slightly more general than what is available in the literature, and it is essentially a direct generalization of [25, Theorem 2.1], in the form of [26, Lemma 1.2].
Proposition 2.1. Let (X, dx) be compact metric space and (Z,dz) a metric space that has countable basis (Un), such that U, is compact for all n E N. Let J'M : X Z + R, M N be a sequence of functions fulfilling the following conditions: 1) (Symmetry in āf For all M E N E XM, 2 E and permutations a E SM. we have fas (ož, 2) = fM (Ø. 2): 2) (Uniform boundedness) There exists B Ez0 and function b Z - Rz0 such that VM N+, e x 2 2 [fa (z, 2)1 - +B)+b(z);;) (Uniform Lipschitz continuity) There exists some Lg E Ry0 such that for all M E Nr. F1,52 E XMM, 22 E Z we have [fM(01, 21) Ly (dKR (M(())) (. (Z])) + dz(21, z2)).
S [ #2) S
Then there exists subsequence (fMr)e and a continuous function f P(X) x Z + R such that SAs, 3y f for E + cc. Furthermore, / is Ly-Lisschitz continuous and there exists BF E R20 such that for all p P(X), = Z we have [f I BF b(z).
We now turn to the mean field limit of kernels as introduced in [22]: Given kMs XMM x X.M + IR and k P(X)= x P(X) + I., we say that kM converges in mean field to and that A is the (or a) mean field limit of KM. if
lim M M00000 Fx
sup (T, ( K((E) O.
In this case we write kM Pi k.
For convenience, we recall [22, Theorem 2.1], which ensures the existence of a mean field limit of a sequence of kernels.
Proposition 2.2. Let kM XMM x XM + be a sequence of kernels fulfilling the following
conditions. 1) (Symmetry in F7 For all M E N,, म, E XM and permutations E Sy we
have kM (oūT x² ki (Z, ); 2) (Uniform boundedness) There exists Cg E R22 such that VM e
Nr, x" E XM Ika (z, F") S Cg: 3) (Uniform Lipschitz continuity) There exists some Ta E Rso such that for all M E N. 11, 52 E X MM we have M (7,, F1)
KM (72, 7%)
<
LEdKR KR [1]]TT], ([Z)), ((T2).A(EE))
Then there exists a subsequence (KMe); and a continuous kernel k: P(X) x P(X) + R such that KM, Pl, k and l is also bounded by Ck.
Let kas: XM x X + R be given sequence of kernels fulfilling the conditions of Proposition 2.2. Then there exists a subsequence (kM,), converging in mean field to a kernel k : P(X) x P(X) + RR
As is well-known, cf. [26, Remark 1. 1.1.3,, this condition is actually implied by the next condition. However. usual in the kinetic theory literature, we kept this condition for emphasis.

---

MFL of KM
kM
> k
M 00
MFL of Ear
H
H
M-
Figure 1:
The kernel & arises as the mean field limit (MFL) of the kernels kas (Proposition 2.2). Every uniformly norm-bounded sequence fM E HM. M N+, has an MFL in Hg, and every function f E H3 arises as such an MFL (Theorem 2.3). Based on [22, Figure 1].
From now on, we only consider this subsequence and denote it again by (kM ar, i.., kas Pt, Unless noted otherwise, every time we need further subsequence, we will make this explicit.
The RKHS of the mean field limit kernel Denote by HM == HKY the (unique) RKHS corre- sponding to kernel kM and denote by H₃ the unique RKHS of k. For basic properties of these objects as well classes of suitable kernels we refer [22].
We clarify the relation between y and Hş in the next result.
Theorem 2.3. 1) For every f H₁ there exists sequence fM E Has, M N+, such that fM Pi 2) Let fM HM be sequence such that there exists B E Rz0 with lIfxrlla S B for all M E N.. Then there exists subsequence (JM,)) and E Hg with f's, Ps and || S B.
In other words, on the one hand, every RKHS function from Hgarises as a mean field limit of RKHS functions from HM. On the other hand, every uniformly norm-bounded sequence of RKHS functions (fM)at has mean field limit Hg.
Note that the preceding result is considerably stronger than the corresponding results in [22]: In contrast to [22, Theorem 4.4] do not need to go to another subsequence in the first item, and we ensure that the mean field limit f is contained in H2 (and norm-bounded by the same uniform bound), which was missing from Corollary 4.3 in the same reference.
The relation between the kernels kM and their RKHSs HM. and the mean field limit kernel & and its RKHS H₂ is illustrated as commutative diagram in Figure 1. In order to arrive at the mean field RKHS Hx. on the one hand, we consider the mean field limit & of the kM and then form the corresponding RKHS Hç. This is essentially the content of Proposition 2.2. On the other hand, can first go from the kernel kM to the associated unique RKHS HA (for each M E N+). Theorem 2.3 then says that Hg can be interpreted in H₁ arises as a mean field limit of a sequence of functions from the HM, and every uniformly norm-bounded sequence of such functions has mean field limit that is in HІ.
mean field limit of the RKHSs HM, since every function
Next, we state two technical results that will play an important role the following developments, and which might be of independent interest. They describe lim inf and lim sup inequalities required for I c-convergence arguments used later on.
Lemma 2.4. Let fM E HM, M N4. and f E H3 such that fMr Pig f. then
IIf la S lim Ilfarllas. M-+00
(3)
Lemma 2.5. Let f Ha. Then there exist fas HaS. M N+, such that lim supzexM (T) f(A[=)) = 0, and
lim lim |lfM $ [| M
Approximation with kernels in the mean field limit
Kernel-basee machine learning methods use in general an RKHS the hypothesis space, and learning often reduces to a search or optimization problem over this function space. For this reason, it is
'It is customary in the kinetic theory literature to switch to such a subsequence. However, for some results that are about to follow, it is important that no further switch to subsequence happens, hence we need to be more explicit in these cases.
5

---

important to investigate the approximation properties of given kernel and its associated RKHS as well as to ensure that the learning problem over an RKHS (which is in general an -imeni-aiienii object) can be tackled with finite computations.
The next result asserts that, under uniformity condition, the approximation power of the finite-input kernels kM is inherited by the mean field limit kernel.
Proposition 3.1. For M N., let FM be the set of symmetric functions that are continuous w.r.t. (z, σ²) + (gR( p[z")). Let F S (P(((X) such that for all f E F and e > 0
there exist E IR20 and sequences JM E FM, S' E HM. M E N4, such that 1) JM 2 J 2)
Il.M - fm oe 5 é for all M E N 3) [lffMllM S B for all M E N4. Then for all f E F and > 0, there exists f E H3 with ||s f1 <.
Intuitively, the set F consists of all continuous functioos on P(X) that arise as mean field limit of functions which can be uniformly approximated by uniformly norm-bounded RKHS functions. The somewhat imprecise terminology) that the RKHS H₃ is dense in F. We
result then states (to can interpret this as an appropriate mean field variant of the universality property of kernels: a kernel compact metric space is called universal if its associated RKHS is dense w.r.t. the supremum norm the space of continuous functions, and many common kernels are universal, cf. e.g. [20, Section 4.6). In our setting, ideally universality of the inite-input kernels kM is inherited by the mean field limit kernel k. However, since the mean field limit can be interpreted as a form of smoothing limit, some uniformity requirements should be expected. Proposition 3.1 provides exactly such condition.
Remark 3.2. In Proposition 3.1, the set J is subvectorspace of C (P((),, Furthermore, if the Pi-convergence in the definition of F is uniform, then F is closed.
Since kas and k are kernels, we have the usual representer theorem for their corresponding RKHSs, ef. e.g. [27]. A natural question is then whether we have mean field convergence of the minimizers and their representation. This is clarified by the next result.
Theorem 3.3. Let N E N4, M1.... NN E P(X) and for n = I,..., N let Z! E X M E N+. such that ji [i dus, Pa for M -> co. Let : RRN + Ryo be continuous and strictly convex and x O. For each M N+ consider the problem
min L(f(E|M) JeHx
f(ZN)) MM) + Allfll,,
(5)
as well as the problem
min I(f(m))........ ((,N))) + flkk- FEHt
(6)
Then for each M E N+ problem (5) has a unique solution fir. which is of the form fir [M] E IR, and problem (6) has unique solution f*. which of the form f =I (n)( E Hg, with o1...., ON R. Furthermore, there exists a subsequence (fM,)e such that fir. Pi, f and
En an kM xlM)) Has, with afM)
[M] !...,,a!!
L(fM, (o) ( ..... fhr fi (abadit)) Allff, + (ff" f" All lls-
(7)
for é + 00.
The main point of this result is the convergence of the minimizers, which we will establish using a T T-convergence argument. This approach seems to have been introduced by [28, 18, 29] originally in the context of multi-agent systems.
Remark 3.4. An inspection of the proof reveals that in Theorem 3.3 and ^|| ||x by 2[[| () and 2(1 [|s), where Ω: Rzo > R2zo is a nonnegative, strictly increasing and continuous function.
replace the term X|| /|
4 Support Vector Machines with mean field limit kernels
We now turn to the mean field limit of kernels in the context of statistical learning theory, focusing on SVMs. We first briefly recall the standard setup of statistical learning theory, and formulate an appropriate mean field limit thereof. We then investigate empirical and infinite-sample solutions of SVMs and their mean field limits, as well as the convergence of the corresponding risks.

---

Statistical learning theory setup We now introduce the standard setup of statistical learning theory, following mostly [20, Chapters 2 and 5]. Let X + (associated with some o-algebra) and 0 * Y S IR closed (associated with the corresponding Borel o-algebra). A loss function is in this setting measurable function X x Ih + R2o- Let P probability distribution on x Y and f X' > IR a measurable function, then the risk of f w.t. P and loss function E is defined by
Rer(s)) I (e, U..VJ((2 f())dd.
Note that this is always well-defined since (z, y) + [(z, function. For a set H C IR* of measurable functions we also define the minimal risk over H by
H y. f(x) is a measurable and nonnegative
R inf Re,r(s)-
JEH
If H is normed vector space, we additionally define the regularized risk of f E H and the minimal regularized risk over by
Re,,,a() Re) + AIIS
R.P.A
inf Re,,,((), FEH
where À E Ry0 is the regularization parameter. A data set of size N E N, is tuple DN (1)),)),--. (IN,WN)) E (X x Y) and for a function f : + R we define its empirical risk by
N
RIDD (aDN()= (1)=
1 N
Σ
Elain Yaa I (xn)).
If H is normed vector space and f E H, we define additionally the regularized empirical risk and the minimal regularized empirical risk over H by
Re,Dx. = (t. + X|
R = inf Re,Dx.a()), JEH
where À E Ryo is again the regularization parameter. Note that the notation for the empirical risks consistent with the risk w.r.t. a probability distribution P, if we identify a data set Dy by the corresponding empirical distribution 1
N Σ ((.,..n))
In the following, И will be a RKHS and a minimizer (assuming existence and uniqueness) of P will be called an infinite-sample support vector machine (SVM). Similarly, R.H+ Dx. will be called the empirical solution of the SVM w.r.t. the data set Dx Note that this is the common terminology in statistical learning theory, cf. [20], and corresponds to (empirical) risk minization with Tikhonov regularization.
Statistical learning theory setup in the mean field limit Let now + Y C R be compact and x T + R2 M E N. such that 1) Em (ar(T, y, t) las fas y. 1) for all o E XM, SM, y E Y. E R; 2) there exists Ce Rzo and nondecreasing function b R2o - Rzo
Las : M x
with [CM y, & Ct + b((t) for all M N and & X y Y,t E I;; 3) there exists L. Rgo with
(T (Ti, v1, - (Mr 92, 12)1 S Ld (A(Z)), (. ((b) - |y yo + 14 -22))
YY,4, E R. In particular, all ê are measurable (assuming the Borel o-algebra X and hence are loss functions on XM Y. Proposition 2.1 ensures the existence of a subsequence (f Dm and an Læ-Lipschitz continuous function E: P(X) x Y x IR + R with
for all F1, E XM, ,3, ,1,
lim sup M +000 am DEYHEKK
1e.... (Z, v. [(i () = 0
(8)
for all compact K C R, and we write again ÊM... Ps l. For readability, from now on we switch to this subsequence. Furthermore, we also get from Proposition 2.1 that there exists some Ct= E Rzo such that [e(u, y, 1)] S Cr. + b((1)) for all μ E P(X),y E Y R.
Remark 4.1. Note that, for Proposition 2.1 to apply. it is enough to assume in item 2) above the existence of a function slightly stronger condition that b is nondecreasing. since then EM is Nemilskii loss according to [20, Definition 2.16). Since the function with constant value C2 is actually Par-integrable, this means that ls is even a "M-integrable Nemitskii loss according to [20]. A similar remark then applies to L.
R - R20 with (er y. 1) S C. b((l). However, we chose the
7

---

Lemma 4.2. The function é is nonnegative. Furthermore, if all l are convex loss functions [20, Definition 2.12], i.e. if for all M E N., 전 XM . ,t,,, R and À E (0, 1) we have (M Mr
X)(2) XCM 1. + (Z,
(9)
then so is l.
Given data sets D = (T) vi (M]] )..... ( ) for all M e
Empirical SVM solutions
N₂ with M E XMM yn E Y, and DN = ((11 v1). (NN yN)) with Uin E P(X) and yn E Y, we write D! Mi Dy ((zh! des, pn and yM + yn (where M - xc) for all ,, N. We can interpret this as mean field convergence of the data sets.
Furthermore, consider the empirical risk of hypothesis fas HM (and H2) on data set D [M] (and D
N Σ
N Σ
R EM D (fM)
(
[M Las (o!
REDD(() =
E(uas Yas f(p))
N R=1
N
n=
and the corresponding regularized risk
N
R DM. (fM) =
1
Ls (a!M .8h [M] ff
(x , + All 133
N
#=1
N
Re.Dx
1
Σ
elpns Vn. f(()) 21
N
where À E lEso is the regularization parameter. Proposition 4.3. Let
assume that all ÊNs convex and let DJM DN be finite data sets with D Mi DN. Then for all M E N4, HM JM + Ress D , (fMr has unique minimizer fMA E HM and H₁ 3 f RE,DN.A(f) has unique minimizer f Hy. Furthermore, for all M E N4 there exist [M] E R.
N, such that fūxA Σn=1 [M] ( [M (M1) and
there exist a1...., ON E R such that fx = En= ank(ss Finally, there exists subsequence (fMm.))m such that fism.. Pa, fx and Rix... pDMm) a (fitm.)) - xe,,x. for m 00.
Convergence of distributions and infinite-sample SVMs in the field limit We now turn to the question of mean field limits of distributions and the associated learning problems and SVM solutions. Let (P(M)) as be a sequence distributions, where P[M] is probability distribution on XM x Y, and let P be a probability distribution on P(X) x Y. We say that PlM) converges in mean if for all continuous (w.r.t. the product topology on P(X) x Y)
field to P and write P[M] Pi, and bounded f we have
J f([ v)d
- f(u y)d v).
(10)
This convergence notion of probability distributions (on different input spaces) appears to be not standard, but is natural concept in the present context. Essentially, it is weak (also called narrow) convergence of probability distributions adapated to our setting.
Consider now data sets D (MM, Dv. with DMM Pi, DN, then we also have convergence in mean field of the datasets, interpreted empirical distributions: let (P(((X) Y, IR) be bounded, then
N
J f(AlE),))dD (ε = f(A[ELM)) M
Σ n=1
N
N M
f(un vn.) = - f(u, ((p,ydDN (H.v).
N
'Of course, since Y is compact, all continuous f are bounded in present setting.

---

This shows that the mean field convergence of probability distributions as defined here is a direct generalization of the natural notion of mean field convergence data sets.
Finally, consider the risk of hypothesis fM HM and f Hy w.r.t. the distribution PIM] and P. respectively,
pim] PiM (fx) [ XM (x U. fm (2))
(f) Spcceer eu,y. (.).di)))
as well as the minimal risks
Rare = inf EMr, P(ae
Res fareHlr
R inf Re, SEHl
Our first result ensures that mean field convergence of distributions P(M], loss functions 4 and data [M] sets D ensures the convergence of the corresponding risks of the empirical SVM solutions.
Lemma 4.4. Consider the situation and notation of Proposition 4.3 and assume that PIM] Pl P. We then have Mi,, P(Mmm) (M) (f)((f)) for co.
Next, we investigate the mean field convergence of infinite- sample SVM solutions and their associated risks. Define for À Rz0 (and all M E N4) the regularized risk fu E Has and f E Ha. respectively, by
Re, PIM a (fM) ReM plan (far) Al
RePx((() Re,((f) + Alffll2.
and the corresponding minimal risks by
RHM EM PIM)),
inf farEHar
((MM))
RHe = inf Re,,,)))- JEH,
Proposition 4.5. Let À > O, assume that all fM are convex loss functions and let PIM] and P be probability distributions on XM x Y and P(X) x Y, respectively, with P(M] Pt, P. Then for all M N+, HN 3 fas H3 3 Re,P.A(f) has a unique minimizer fā Hg. Furthermore, there exists subsequence (fMm.)) such that fism Pi fx particular, R Mm
R Mi RiMi (fM) has unique minimizer fMA E HM and
and Res P(Mm) (fMMm)) (REP,x(f))) for -> co. In
Reste P
RH H&*
P.P.
Finally, would like show that R. established under Assumption 4.6. Define the approximation error functions [20, Definition 5.14]. by
P - RH for PIM] Pi P. Up to a subsequence, this is
M (2)= (2)
inf REs. ((0, - -RHarr fEHM
P(M)
A2(A) = inf (e) - RH. fEH.
-R
where M E N., and E R2o- Note that (for all M E N4) A A2 R20 - R20 are increas- ing, concave and continuous, and A!M A2(0) 0, cf. [20, Lemma 5.15]. We need essentially cquicontinuity of (A(M)) As in 0, which is formalized in the following assumption.
Assumption 4.6. For all 6 > 0 there exists A₂ such that for all < X A ) and M E N, we
have A AM (A) e.
Proposition 4.7. Assume that all ls are convex loss functions, let PIM] and P be probability distributions on M Y and P(X) Y. respectively, with plM] Pi, P. If Assumption 4.6 holds, there exists strictly increasing sequence (Mm).. with R Hasm
Esim .PMM.) + R for m + co.
Ha*
NNtt that Proposition 4.3 is actually corollary of this result. However, since the former result is independent of the notion of mean field convergence of probability distributions, stated and proved separately.

---

5 Conclusion
We investigated the mean field limit of kernels and their RKHSs, as well as the mean field limit of statistical learning problems solved with SVMs. In particular, we managed to complete the basic theory of mean field kernels as started in [22]. Additionally, we investigated their approximation capabilities by providing a first approximation result and variant of the representer theorem for mean field kernels. Finally, we introduced corresponding mean field limit of statistical learning problems and provided convergence results for SVMs using mean field kemelss. In contrast to other settings involving large number variables, for example, infinite-widdt neural networks, here we considered the case of an increasing number of inputs. This work opens many directions for future investigation. For example, it would be interesting to remove or weaken Assumption 4.6 for a result like Proposition 4.7. Another relevant direction is to find approximation results that are stronger than Proposition 3.1. Finally, it would be interesting to investigate whether statistical guarantees, like consistency or learning rates, for the finite-input learning problems can be transferred to the mean field level.
Acknowledgments and Disclosure of Funding
We would like thank Noel Brindise, Pierre-François Massiani and Alexander von Rohr for very detailed and helpful comments on the manuscript, and the anonymous reviewers for their detailed and helpful comments. This work is funded in part under the Excellence Strategy of the Federal Government and the Länder (G:(DE-82)EXS- (EDE-82)XXS-SF-AAEE SEDEEEXSSF-FFDDEEEEE SF-DDD500,5), which the authors gratefully acknowl- edge. The authors further thank the Deutsche Forschungsgemeinschatt (DFG, German Research Foundation) for the financial support through B05, and B06 of 442047500/SBB 1881, 1F55386/18-1111122222111.
23849999//ITTG-2377, B04,
References
Carlo Cereignani. Rarefied gas dynamics: from basic concepts actual calculations, volume 21 of Cambridge Texts in Applied Mathematics. Cambridge university press, 2000.
[2] Carlo Cereignani, Reinhard Ilnner, and Mario Pulvirenti. 106 of Applied Mathematical Sciences. Springer Science & Business Media, 1994.
The mathematical theory of dilute gases, volume
[3] Michele Ballerini, Nicola Cabibbo, Raphael Candelier, Andrea Cavagna, Evaristo Cishani, Irene Giardina, Vivien Lecomte, Alberto Orlandi, Giorgio Parisi, Andrea Procaccini. et al. Interaction ruling animal collective behavior depends on topological rather than metric distance: Evidence from a field study. Proceedings of the national academy of sciences, 103 2008.
[4] Yael Katz, Kolbjorn Tunstrøm, Christos C Toannou, Cristián Huepe, and Tain D Couzin. Inferring the structure and dynamics of interactions in schooling fish. Proceedings of the National Academy of Sciences, (88(46):18700-18222 201 1.
[5] Giuseppe Toscani, Kinetic models opinion formation. Communications in mathematical sciences, 4(3): 481 196., 2006.
Claudio Castellano, Santo Fortunato, and Vittorio Loreto. Statistical physies of social dynamics. Reviews of modern physics, (112) 2009.
[7] John RG Dyer, Anders Johansson, Dirk Helbing, lain D Couzin, and Jens Krause. Leadership, consensus decision making and collective behaviour in humans, Philosophical Transactions of the Royal Society Biological Sciences, (064(118):781-76)) 2009.
[8] Emiliano Cristiani, Benedetto Piccoli, and Andrea Tosin. Multiscale modeling of pedestrian dynamics, volume 12 of Modeling, Simulation and Applications. Springer, 2014.
[9] Giacomo Albi, Mattia Bongini, Emiliano Cristiani, and Dante Kalise. Invisible control of self-organizing agents leaving unknown environments. SIAM Journal on Applied Mathematics, 76(4):1683 1710, 2016.
[10] Andrés A Peters, Richard H Middleton, and Oliver Mason. Leader tracking in homogencous vehicle platoons with broadcast delays. Automatica, 50(1):64-1355 2014.
[11] Kwang- Kyo Oh. Myoung-CDi Chul Park, and Hyo-Sung Ahn. A survey of multi-agent formation control. Automatica, 53:424 440, 2015.
10

---

[12] Young- Pil Choi, Dante Kalise, Jan Peszek, and Andrés A Peters. A collisionless singular Cucker-Smale model with decentralized formation control. SIAM Journal on Applied Dynamical Systems, 18(4):1954- 1981. 2019.
[13] Andrea Tosin and Mattia Zanella. Kinetie-controlled hydrodynamies for traffic models with driver-assist vehicles. Multiscale Modeling & Simulation, 17(2):16-749, 2019.
[14] Giovanni Naldi, Lorenzo Pareschi, and Giuseppe Toscani. Mathematical modeling collective behavior in socio-economic and life sciences. Springer Science & Business Media, 2010.
[15] Tamás Viesek and Anna Zafeiris. Collective motion. Physics reports, 517(3-4):71-140 2012.
[16] Xiaoqian Gong, Michael Herty, Benedetto Piccoli, and Giuseppe Visconti. Crowd dynamics: Modeling and control of multiagent systems. Annual Review of Control, Robotics, and Autonomous Systems, 2022.
[17] José A Carrillo, Massimo Fornasier, Giuseppe Toscani, and Francesco Vecil. Particle, kinetic, and hydrodynamic models of swarming. Mathematical modeling of collective behavior in socio-economic and life sciences, pages 297-336, 2010.
[18] Mattia Bongini, Massimo Fornasier, Markus Hansen, and Mauro Maggioni. Inferring interaction rules from observations evolutive systems i: The variational approach. Mathematical Models and Methods in Applied Sciences, 27(05):909-95) 2017.
[19] Fei Lu, Ming Zhong, Sui Tang, and Mauro Maggioni. Nonparametric inference of interaction laws in systems of agents from trajectory data. Proceedings of the National Academy of Sciences, 11((2)) 14424- 14433, 2019.
[20] Ingo Steinwart and Andreas Christmann. Support vector machines. Springer Science & Business Media, 2008.
[21] Andreas Christmann and Ingo Steinwart. Universal kernels on non-standard input spaces. Advances in neural information processing systems, 23, 2010.
[22] Christian Fiedler, Michael Herty, Michael Rom, Chiara Segala, and Sebastian Trimpe. Reproducing kernel Hlilbert spaces in the mean field limit. Kinetic and Related Models, 16(6):850-870, 2023. ISSN 1937-5093.
[23] Michael Herty and Mattia Zanella. Performance bounds for the mean-field limit of constrained dynamics. Discrete & Continuous Dynamical Systems, 37(4))2023, 2017.
[24] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in neural information processing systems, 32, 2019.
[25] Pierre Cardaliaguet. Notes on mean field games. Technical report, 2010.
[26] René Carmona and François Delarue. Probabilistic theory of mean field games with applications 1-11. Springer, 2018.
[27] Bernhard Schölkopf, Ralf Herbrich, and Alex Smola. A generalized representer theorem. In International conference computational learning theory, pages 416-426. Springer, 2001.
[28] Massimo Fornasier and Francesco Solombrino. Mcan-field optimal control. ESAIM: Control, Optimisation and Calculus of Variations, 20(4): 121-1152. 2014.
[29] Massimo Fornasier, Stefano Lisini, Carlo Orrieri, and Giuseppe Savaré. Mean-field optimal control as gamma-limit of finite agent controls. European Journal of Applied Mathematics, 30(6): 153-11886 2019.
[30] Serge Lang, Real and functional analysis, volume 142. Springer Science & Business Media, 2012.
[31] Mare Atteia. Hilbertian kernels and f functions. Elsevier, 1992.
[32] Gianni Dal Maso. introduction to T-convergence, volume Progress in Nonlinear Differential Equations and Their Applications. Springer Science & Business Media, 2012.
[33] Mattia Bongini. Sparse optimal control of multiagent systems. PhD thesis. Technische Universität München. 2016.

---

Supplementary Material
A
Proofs
In this section of the supplementary material, we provide detailed proofs for all results in the main text.
A.1 Proofs for Section 2
We start with Proposition 2.1, whose proof is based on [26, Lemma 1.2].
Proof. of Proposition 2./ For M N4 define the McShane extension FM P(X) x Z + R by
FM( = inf (z, + L #EXM
Observe that Far is well-defined (i.e., R-valued) since [- 2) and L. r (A--), ()) are bounded for every E Z (since JM and dRR (Ā[-), u1) are continuous and ((X) is compact, hence bounded).
Step 1 Fas extends sas, i.e., for all M N4. o X and z E Z we have Fir (A] =) J z).
To show this, let / E XMM M and 2 E Z be arbitrary and observe that by definition
inf JM Fas (A (1, X'Ex
(i(("") ((T) S fas (7, 2) + I j (1).) = San (z)
If (Mr())))) < (T, z), then there exists some J' E X M such that
far 2) + LYdKR (i(T), [[] < fM,
but this means that
I.ydx((A(F") Maa < Sas (ā, (x' S (f, Sas (T', 2)),
contradicting the ~Liischhit continuity of fMS.
Step All FMr are Ly-continuous: Let M E N4. H4 PXX) and 2 E = be arbitrary. Since XM is compact and fM((,2) 2) and [d (1). 1, 2, are continuous, the infimum in
the definition of FMr is actually attained. Let F2 E X.M such that FMr 22) fā 22) ddRR (i(T2), p2), then we have
FM 21)
S [M 21) + yd (f((T2), 11)
= fM 21) + , (i((2)) H2) - L L, (A 12) + L (f(Z2), mI)
S fas (#2, 22) + L (il (f(T2) u2) + L,ddz (21, 22) L (22,, 112)
+ T.jdRR (f1 : /1)
S Fas (12, 22) + L (21, 22) L 1 (i1 U2) + L d (H1,
+ L (p) [22] M2)
Fas (12, 22) + I. (dxR u2) + dz(21, 22));
where we used the definition of FM in the first inequality, the Lipschitz continuity of fM (w.rt. the second argument) for the second inequality, and then the fact that Z attains the infimum in the definition of Fas (H2,22) and the triangle inequality for dRR- Interchanging the roles of μ1, &1 and 42, 22 then establishes the claim.
Step 3 There exists BF E R20 such that for all M E N+. /1 E P(X) and E Z we have [Far (u, =)] & Br + h()): Let Dp(x) be the diameter of P(X) (which finite since P(X) is compact), then for all M E N4 and of E XM, 2 E Z, μ E P(X) we have
-(B, LB Dr(x) + b(2)) S fa, Lj (p(E), S B₁ + L, + b(2), therefore (M 2)1 S By Dp(x) b(z), showing the claim with BF By + Ly Dx(x).
Step 4 Summarizing, (Far) M is sequence of L -Lpsschtt continuous and hence equicontinuous functions such that for all f E P(X) and z E Z, the set (Fas (I, 2) M E N+) relatively compact (since it is bounded subset of RR). We can now use variant of the Arzela-Ascoli theorem, cf. [30,
12

---

Corollary III.3.3]. From the assumption on Z, we can find sequence (Vn),, of open subsets of Z such that all V, are compact. V, S 1 and we have Un Vn Z. Then (F v, ar is a sequence of functions that fulfills the conditions of the Arzela-Ascoli theorem (since P(X) x Kn, is compact), so there exists a subsequence ar( vv,)) that converges uniformly to a continuous function on
P(X) x Vn Denote the diagonal subsequence of all these subsequences by (FM,)e. then there exists continuous f P(X) x Z + IR such that (FMe)e converges uniformly on compact subsets to f. Since P(X) is compact, this means that for all compact C
lim sup PEP(X) *EK
[FM (H, f(u, 2)1 = 0.
This also implies that for all j E P(X) and & E Z we have (f(1, 2)1 - BF + b(2).
Furthermore, f is also Ly-Lisschitz continuous: Let μa; E P(X). 2; Z, = 1, and > 0 be
arbitrary. Let K S Z be compact with 21, 22 E and choose l N+ such that
sup
(FM 2) - f(u, - 2
μEP(X)
*EK
We then have
[f(11 f(u2, 22)1 S (f(11, FM, (H1,21)) 21)1 + |FMe (M1, 21) (M2, 22) + |M (12, 22) f(22, 22)l
s Ly (dgR u2) + dz (21, =2)) +
and since e was arbitrary, the claim follows.
Step 5 For & E N4 and I E MM, 2 E Z we have
If (Z, 2) - f([[E) 2) = |FM, (A(E), 2) fATE), 2)1
since FM, extends fM, and hence
sup 1,,, [(( =)1 + O.
Mc
=EK
Next, we provide the proofs for the r-lim inf and P-lim sup results.
Proof. of Lemma 2.4 Assume the statement not true, i.e., |l > lim in Ilfarllar. This means that there exists subsequence Mt and C E Rzu such that [f > lime fM, ll,, = C. Note that this implies that || >
Let 62 > 0 and a > 1, 3 E (0,1) be arbitrary. From Theorem B.1, there exists (77, a) E P(X) R such that
D(ū, ā. f)k) + 2 llflI
and w.L.ogg. can assume that c 0 is small enough that ā, 0. The latter implies that ( W(J, a. k) > 0, so defining
Ea = a-1
-=---(pp;11) E(
Eg = (1/3 11)-11,,,, kk
we get Ca. Cg > O. For each =1, 1,...,, N. choose H.Ml X such that (M| day un for M co.
Choose now L1 E N such that for all 2 L2 we get
[ ã. fM,) E(J7, f.f)l S Ea [M a. kM.) - W(Z, ã, k)) S Eg. ((X (MMi) ã. klMal) > O. We then get (cf. also the proof of Theorem 2.3) and W(X
a, < (E(M)M)), a. fM.) W k)) BW(FFM)) aa MMMe).
13

---

so altogether
E11, a f) W k.
aE(x SMJ) BW(XIM oi kebael)
Using Theorem B.1 again leads to ac (E( ā. fas) SW(X ( ā. MlMe)
D(XIMi) ā. fM k!Ma)) S 11ffeellxs
Finally, let L2 such that for all l > L2 we have [lf llas, 5C+ For & 2 L1, L2 we then get
C < 1 I DiJT, a. @.J.K f.k) +
E(ū, +
W(F.. ā.k)
aE(X) fMi))
+ 61
3W(XIMe), ã. klMal)
B [lMMellMe +
s B B f2 + t1. E1.
C+ +
Since é1, 62 > and a > 1, E (0, were arbitrary, this implies that
C < Iffll SC
contradiction.
Proof. of Lemma 2.5 Let Step 1 For each n E N choose
H3 be arbitrary and choose (cn)n S R2 with 6n >
fRD
Σ ai
(n) k(-,Pe
(n)) H
Em1
where af (n)
(a) at Tn
E IR and
(n)
P.....
(n)
with E P(X),
|l f 1
En
3vCk
and || l| < |l.
To see that such a sequence of functions exists, choose some sequence
(fn) E H with fn Ein The aLn) k(., m(")), ), where a(n) E R, pi") e P(X), with fn Iills f (exists since Hee is dense in H₂). Define now for E N
H. span (ka ((())) Im I,...,,,, n, f= 1..,.... I.n)
and fn. = Pa.f. where Pa. is the orthogonal projection onto Hn. Then H, S H llfallk flk S lllll and || - fullk I Ilf falls + (since f.. = PA.. f is the orthogonal projection of ff onto Hn and Fn. H,), hence fn 11-12 f. We now choose (fh")), )n subsequence of (fr)..
In choose xlme) E XM with ((at(n.)) duy (n) for M - co. Me
Next, for all E N and L = 1,.... Furthermore, for all n N choose Mn E such that for all M 2 M, and f I,..., . Ln we have
2 2h Σiy 1a (m) (10 2 (1 (u(i)))
dxR (Al 1.14") (m)) S min
On
3
1 + La Eve
and
a. ar
sup
Ik ( (#, - {((EE),(FT) $ min
3 (I=
Ln mm
6n
(a(")))) 2 '2(1 Etz
tn
2
la
10
(1)
(n)
14

---

W.L.og. we can assume that (M,),, n is strictly increasing. For M E 1, let n(M) be the largest integer such that Mn(M)) I M and define
fM
o (n(M))
Rc,,, م
Lacae)
(m(dd))
kad 2-, ((((M))))) H
fM
ag
Step 2 We now show that f'r Pi f. For this, let e > 0 be arbitrary and ne N such that fn S 6.
Let now M 2 Mn, (note that this implies that (MM) 2 ne and hence <n(ad) S 6n) and of E X.M then we have [f(((=) ( = (F(A)) (nMM)(((T)) + (A(M))((E))) fAr (A())
For (A( - faz (z))
=11
=111
We continue with
== [[(A(=)) - Su(a (A(T))
= K(f Fa(ar), R(- ((
I IlS SacAr) |allk( A[T))
Ilf -AFCM) IA A[T
1 (n(M) VC
3/Ck
where we first used the reproducing property of k. then Cauchy-Schwarz, again the reproducing property k; and finally the choice fa(M) and the boundedness of k. Next,
II Fa(M)) (A(=)) - PM (A(ET)
(n( (n(M)) k(-, K4 (n(M))
Lucar) E 1=1
a2 (n(M))
kc,, A (1
(n(M)))
k((, f| (1(=((())))))) (n(M))))
|k(- Me
(m(M)))
dxR (pt (((=)((()))))) 11 (m(()e)))
La
Cn(M)
3
where used the triangle inequality, the Lipschitz continuity of k. and then the choice of the sequence (Mn)..
Finally,
III
Eic (fi()1) fas (z))
Ta( Σ
(m(Ar)) le(,, fa, (il (1
(m(ad))
kas (s, s
ai
ai
Em
bitan
|k((, Al - kM x (0(M))))))) (x((A1))
a
(n(M)
3
15

---

where the triangle inequality has been used in the first step and then again the choice of the sequence (Mn)n. Altogether,
[f(A()) - fM S II III
Cn(M)
(n(M)
Cn(M)
3
3
S 6,
establishing fM Ps f.
Step 3 We now show lim sumsups-so If S llk Let > 0 be arbitrary and n, N such that én S and let M 2 Mn. We have
Li(
1lfM 13: =
Σ
oe (n(M))
(a(M)) KM KM (MM JMM
(M(()))))) AN(M)))))
f.
(n(ar)) (n(M)) k(KE (n(ar)) M/6 ( + RR + |Ra Op
f,t==1
l1 + R1 + R2
R1 RR2
with remainder terms
En(ar)
Lucar)
R =
(n(M)) (n(M))
y Km (1
1 An(M))))
(n(M) (n(M) (((M).)
ay k(jlaT " (A
0,0==1
2,0==1
kn(()r)
Ra
Σ
di
(n(M)) (n(M)) (1()))))))))))) (1(0)))))))
(m(d)))) (n(M)))
ae
( (((()()))) my (a(M))
We now bound these terms, so that
R1
(n(M) (n(M))
oid kws ( Fs
(
Σ
a
(n(M)) (n(M)) k( "1 1 (o ay
f,0=11
=,===
Fnn(ar)
<
|a
(n(MM))
ag
(o(MM))) (3/e s/(((M))0)) - M( FM
(
2,0==1
(((A))
2
2
and
Lu(M))
bu( Σ
Ra
O2
(n(M()) (m(M))
az
((( Al
(m(a))) (x(A))
og
og
k( (a(a)))
1
=1
Ln(M))
Σ
|a
(u(M)))
( 1. |1
(o(a)))
k( (u(A)))
(((M)))
=
Entan
< Li
la
(u((A)))1a ((())))) ( (dka (A(
1. (((M))) + dxx dxx( (Bl J,
1,1=1
2
< (n(A))
2
Altogether,
< 111fll 1M
+ |R1| + R2
2 (M)
2
(M)
< |l +
+ 2
2
< ll + +22
16

---

so II Σ |f + c for all M 2 Ma., and since > O was arbitrary, we finally get lim iimsupa-ss Il M $ &l
Finally, we can now provide the proof for the central Theorem 2.3.
Proof. Theorem 2.3 The first statement is part of Lemma 2.5. Let us turn to the second statement: The existence of the subsequence (fare) ) and the continuous function f P(X) R with fM, 2t f was shown in [22, Corollary 4.3]. so we only have to ensure that f E Hx with lifill S B. For this, use the characterization of RKHS functions from Theorem B.1. In particular, we will utilize the notation introduced there.
Step 1 Let (17, E P(X) x TR We show that if W(J., a, k) O, then E(TT, a. ā. 0.
Assume that W(Z, = O. B = O, then fM = 0 and fas, 25 f implies that f = 0, so the claim is clear in this case. Assume now B >> 0, let > 0 be arbitary and for = ,,...,N, NN choose sequences xiMl
X such that x!Ma daR, Yo for M + co. For convenience, define XIM] = ) Choose now Ee E such that for all M 2 Mr, we get ((X a, k3r) S
(M)
c/B. This is possible since kM Pi ls together with the continuity of k& and k as well as x!M] day Nh for M + x: and all n I,..., N implies that W(I k.) W(i7, ã. k) = 0. Let now L 2 &e arbitrary and observe that sas E Has implies N(MM, (Ja,,as)) < oo according to Theorem
B.I, so in particular D(XIM), ā. fMe km,)) < oc.
If WVX(Ma) ō. kM.)) = O, then we get that E(XMd), C. fMe) = 0 1 6 since
D(X ãi JM kM.)) < oc, which implies by definition that E(XlMll , (M.) = 0.
If I(X(XMM)) ā. kM.)) > O, then we have
E(X ã. fMe) D(PMM ā. fM, kM.) S N(fM, IIfMellm S B,
W(X(M), C. kM.)
which implies
E(XIMi) (M,.) S BW(XIMM C. (., S
Since fas, Pi f together with the continuity of fas and f as well as In dug Pn. implies that E(X(M)) ã. fMMe) + E17, ā, f), we get that E a, f) S and since >> 0 was arbitrary we arrive (fT, a. f) S. O.
Assume now that E o, f) < 0. This implies that there exist > and Eg E N such that for all 2 la we have {(X(Me, ã. fM,) < -5 < O, since {XFMM)) ã. fM,) + E o, f). Let E 2 la then we get that E(X(Ma) fM, 8 and we have W( -ā, kM,) W(XM)) a. kM.)) > 0.
We can then continue with
8
8(X1151), - - fML
W(X1aal,, ā. kase)
W W(XIM)l - kase)
S D(21 I kose))
S SN(MMr: kNN.)
11fmeelme, S B.
which implies that W(X(M)) -ā, )M,) W(XlM)) a. kM.))
5/B. But since
W(X ā. kM.)) + W( ã, k), this implies that W(I,, a, k) 2 8/B > 0, a contradiction. Alto- gether, E(JT, a. O.
Step 2 Let (ū, E P(X)N x N... (FW(T,a)) O and E a. 0, then E(ī, o, f)
W k, S B.
17

---

To show this, let a > and / (0, 1) be arbitrary. Define
ca E(i1, f)
eg (1/3-1)((Z,,)) a.
and observe that Ca, cg > 0. Furthermore, for all nt = 1.... choose a sequence s!M E X.M such
that s/ah]] duy, pn for M - oo, and define X = (I|| x Choose le E N+ such that
for all l le we have
|E(X(M)) fM.) E(Z,, ā, f)l S ca
W a. kM.) W(I,, ā. k)] S EB
and W(XiM)), ã, kM,) > O. Such an le exists because RM Ps k together with the continuity of ka
and &t as well as the convergence of a!M yn imply that W(X ã, kase) + W(JZ, k), and
fas, Pt, f together with the continuity of far and f imply that E ã. fM,) + E f).
Let now 2 be arbitrary. By definition of En we get ate S (a: 1) ã. f), which in leads to
ca Ca aca (a 1) o,
- 1)e, + (a- 1) f)
(a ()(E(((, - Ea)
< (a - 1)E(X Me), ã. fMI),
where we used in the last inequality that 1 > 0 and by choice of l, we have E(T, ã. f) s E(X(Me) ā. fas,) fa- We then continue with
f, S E(XMM)) ā. fMc) + Ecc
S {(( ã. fM.) + (a 1) 1) SMc)
ax(XXMe a, f(.))
Next, by definition of 63 and choice of le we find that W(FlM) ā. )M,) S W(I, k) + W( a, k) + (1/3 1)W(1, (1/B3)W(T, ā,k).
hence
W(T, I XW(X(Ma) ā. kwa)) Combining these results, we get that for all 2 E dt f) W(I, a. k) < a 8(XIM), ã. fMA) < 3
N(fss, kM)) [lfas, Il S a B. BBB.
3 W(XIM)) ā, k.k
Since a >> and B E (0, 1) were arbitrary, this shows that ā,
S B. W(J a,)
Step Let a)) P(X)N x N be arbitrary. If W( a. k) 0. then get from Step 1 that
E( a. = 0 2 B. Assume now W(Z, a, k) > 0. If E a. f) 0, then again E a. f) = 0 S
B. If a, 0, then Step 2 ensures that
E ã f)
= a, < B. W(Z, o k)
Finally, if E a. f) then again
[ D(j, a. f.k) < B.
W(n,
Altogether, we get that ā k) < B < co, hence f E H4 and
a. f,kk < Since (ī7, σ) was arbitrary, maximization leads to || =N(ff k) B.
D
18

---

A.2 Proofs for Section
In this section we provide the proofs for the results relating approximation with kernels in the mean field limit.
Proof. of Proposition 3.1 Let f E F and e 0 be arbitrary. Let Rz0 and fM E FM fMr E HM, M E 14, such that fM - f. IlfM - full s ğ and [ffMllM I B for all M E N4 (exist by definition of F). Theorem 2.3 ensures that there exists subsequence (fMi)e and f E Ha
with llfll I B such that fas, Pt, f for - oc. Choose now L1 N+ such that for all & 2 L1 we have
sup Ifad - f(A[))) -
Mc
sup ZEX Mr
[ s(i(a)) S 5"
Let now μ E P(X) be arbitrary and choose sequence Fm E X ds with [[ dug n. Finally, let
L2 E N4, such that for all l L2 we have
If(x) f(i(@M))l 5
(f()) f(i(FM.))) S
(such L2 exists due to the continuity of f and f).
We now have for f 2 max L23 that
[f()) f(4) $ 1f(x) f((EEM,)1 + - f,r, (Fss.) fas, (ZM,))
+ (fM, (Zme) - f(M(EM (f([(EM,) f(u)]
5
é c
+ 3 3 5
=
Since was arbitrary, the result follows.
Proof. of Remark 3.2 We first show that F subvectorspace. Let f.g E F and A R2 be arbitrary. W.L.og. we can assume * 0. Choose sequences fas.9M 9M E Fas fas IMs E HAS- M E N., and constants By, By E R2o from the definition of F for f, 2/x1- and g, 2. respectively. XM M be arbitrary, then Let M E N4, F
(F + g(Ø) (Af(Al=)) ((((E)) & (A) - ff f(((=)) + - g(A(F\)
together with fM P4, f, gM Pi, shows that AfM + 9M P4 Af + Next, we have for all M N4 that
[[AfM + gM) - (AfM lox (AllifM - farllo + l1a gmrloo $ 111
+ E
2|
2
Finally,
| + IMIlla S | + ll S [N[By + Bg
establishing that (Afas + .MM)s as is uniformly norm-bounded. Altogether, we have that Xʃ + E F.
We now turn to the second claim. Let (f(n)),. ) C F such that f(n) f for some e (P(X), and for all > 0 there exist ((n) E Fas, f E HM, (Pм)M C Rg and B(x) E Rzo with PM > 0. (l) JM lloc I and ((())) M I B(n) for all n, M E N4, and
S (n) (z)
((()(1 S PM
sup MEXM
19

---

for all n, M E N+. We now show that / F. For this, let > 0 be arbitrary and choose ((x) E FM JM E Ha, (pm))a S R2o and ((*) E R20 as above with É = 1'' Let N E N+ be such that [lf(m) - f( I for all m, 2 N (such an N exists since (f(m)), converges in (P()), R) and hence a Cauchy sequence). Furthermore, let M, E 2 be such that for all M 2 M, we have PM S $. Define now fM fM and fM ((M) for M =1,11... I,..., Mp 1, and fM M
(
fax = ((N) for M 2 Mp.
Step 1 Let M z M, and E XM be arbitrary. We have
(m) f(((a) = 1 (N++)) (3) -(((E)
S I N (F) | (alaaa))(((((()) f((Z))
(N(N
PM + [1 - A1
and since the right hand side (which is independent of 3) converges 0 for M + 00, get JM P4, S.
Step 2 For M 1,,... Mp we get
1l - fallo = -1. - ((M) 1loe SESE.
Let now M Mn and # XM be arbitrary. We have
(z) fā) = I 1 (3) - ((N)(()) (x))
I ((M++N) (z) - -((N+M (A(T)) + | (A(T)) ((N)
+15000 (i(z)) (1) 10 (2) (z) (3)
(M+NN) N) (F') - (2 + || - 1|0
< sup
FEEM
+ sup ( - / (x)) 11 11510 - -110 11m0
#EXM
S PM + 4 PM +ē
_ =,
4
and since J E XM was arbitrary, we get llfM - fualls Se.
Step 3 For M I,....,,4 Mp - 1 we get by construction that llfarllas- = (1 I| M S B((), and for
M 2 Mo find MI 1 IM B(N), Altogether, get for M E N+ that (,M,-1),(()).
ll S max (a)
Combining the three steps establishes that f E F.
Finally, here is the proof of the represnter theorem in the mean field limit.
Proof. of Theorem 3.3 The existence and uniqueness of JM and f follows from the well-known representer theorem (applied to all ki and k).
We now turn to the convergence of the minimizers. For all M N+ we have
l|lfArll S L(Sā ) .......,. (T + AlIlIl S L(O.....O), ie., II MI $ L(..... ((O......)) (.0....,))//,, Define
.0)A.
LM HM + R20, f L(f(at L: H - Rzo, f [(f(1),......... f(NN)) + A|ls[k,
[M]
f(xl)) )( AIffIlM
and let fas Hм with fM Pi f for some f E Hy. The continuity of fM.
and L as well as z day Un for M + oo and all N, imply then that
20

---

M- L(MM (z) ( ( L., Combining this with Lemma 2.4 leads to
((f) I lim (f)
M-+00
Let now f E Hg be arbitrary and let fM E HM be the sequence from Lemma 2.5. Using the same arguments as above find that
sup (fMs) & llf M-000
We have shown that LM _ L and hence Proposition B.3 ensures that there exists subsequence (fM,)e such that fis, P4, f and LM. (fr,) + ((f)).
A.3 Proofs for Section
Proof. of Lemma 4.2 That is nonnegative is clear from the proof of Proposition 2.1. Let now
all EM be convex and let μ E P(X), y Y, R and (0, be arbitrary, and define
I [minft1, f2), max(t1, t2)]. Furthermore, let ZM X with EM c > 0 be arbitrary. Choose now M so large that
dus r for M oo and
|e(,, y, 111 + (1 2)12) [((TT) M4 +1 -)12))
sup
(M((,, MA
#exM VEEYE
This is possible due to the continuity of e, as well as tM P1. l. We then have
E(u, y. Mr + (1 A)t2) s (((Ta) Nt + +1 (122 6
X1 +1 (1- A)62) +
s MM + (1 (FM M.
3
$ 8(( (1
+ +1+11-
$ Ae(p. y. 11) + +1 (1- A)((H, v, +
and since 0 was arbitrary, this establishes
f(u, y, At1 + (1 A)t2)= S xe(m, + + A,)((,,
i.e. convexity é.
Proof. of Proposition 4.3 From Lemma 4.2 we get that L is nonnegative and convex. The existence, uniqueness and the representation formulas follow then from the standard representer theorem, cf. e.g., [20, Theorem 5.5].
Furthermore, for all M E N+ have
1 S N Σ las (
MI fiM (=lA1))))) + AlI.fr,
< R M
< NCr,
hence Ilf NC.
Let f H& and (JM)M. JM E HM. such that fM Pr of LM, l, together with fM Pi, f and the boundedness of (vh
f. From [M]] Pi, DN and the continuity
IM
M E N+, 11, 1,..., N) Y
and ((MM)) M E NN, - 1,...., N) we find that
N
N
Σ
las (T(M w. [M] JM (f(M))
Elpa Uns ((n)).
M N
N
m=1
==1
21

---

Combining this with Lemma 2.4 and Lemma 2.5 then establishes that R R D A the remaining claims follow from Proposition B.3 and the uniqueness of the minimizers.
En Dr and
Proof. of Lemma 4.4 Let E > be arbitrary. Recall from the proof of Proposition 4.3 that for all M E N we have M S / NC., and hence for all X.M we have
(fM) I [IfMalll|/EM
NC
VC..
VV A
similar argument applies to f* E Hg. so we can find compact set K C with
(fār, x M E = l..... NN U ((5) n= 1..... N) S K.
=
Choose now m. E N such that for all m 2 me we have
sup
CM (T. y, f M (ō)) Lm (3. y. (A( -
xEX Mm
3
WEY
sup
(M, e(([0). $
xexim WEYbEK
3
Vemiax E(AT)) v. (1
JAckyer Elp y, fI()))d(H
S 3
Such a m, exists since JiMm.. Pi f and all las,,. are uniformly Lipschitz continuous (first inequal- ity). Lm P4 l and Y and K are compact (second incquality), and p[M] Ptg P as well as that (p, 9) + E(H, U. f(()) is continuous and bounded (third inequality). We now have
ReMm plamml (Jim.N) - (1))
|S XY
Caa (Z. U, I M L.m.. (H. fX M (Z.
J |xMmxx XY
LM (7. v. (((
(F,
[ ((plT) y.
(Z,
(Σ | B. (I(4))d(4x))
XMMxX XY
< /
1MM (Z. U, MM (z)) - L. (7. y, (X (F.
x 3m XY
1eM., (F, N. (X((E)) [ v. ((E)))aP.MM......
XXXX
+ 3
S, and since c > was arbitrary, the claim follows.
Proof. of Proposition 4.5 Observe that all kM are bounded measurable kernels, ReM, (fM) < co for all E HM, l' is a convex, p|M -ittegrraa Nemitskii loss (cf. Remark 4. and hence [20, Lemma 5.1, Theorem 5.2] guarantee the existence and uniqueness of fi A completely analogous argument shows the existence and uniqueness of fī-
We now show that PIMLA I Re,P... For the I-lim inf ineeuulitty, let fm H f Hy be
arbitrary with fM P4 f. and let > O. Choose Mr E N4 so large that for all M 2 M,
|[((LE), [((A)E)... ((IE)) - / f())) 8) S 2
22

---

(this is possible since (1, f(j,, v. f(1)) is bounded and continuous and p[M] P4 P) and
(M( v. JM - [(il[] (((+)) -
for all & E XMM y E Y (this is possible due to the same argument used in the proof of Lemma 4.4). For M 2 M. we then find
/ (()))d(()))) + AIlIII
Re)pa())=
/ U.
// E(AlE) ((llT))PPM (E.,)) / elu, f(u))d
| (.. - E(p [[] v. (((E)))aP v) + A115112
$ / lM (Z. (Z. (#. v) + lim inf f +c,
M
where we used Lemma 2.4 in the last inequality.
For the l- sup-inequality. let f Hg be arbitrary and let (fM) aM be the recovery sequence from Lemma 2.5. The desired inequality then follows by repeating the arguments from above.
Finally, using exactly the same argument as in the proof of Proposition 4.3 shows that || IaM S
V NC1, X so we can apply Proposition B.3 and the result follows.
Proof. of Proposition 4.7 Let (en)n R with cm Y O. We construct strictly increasing sequence (M,), such that
HM EM PIMM
RRPp" S en
for all n E N+.
We start with n = 1: Since A2( = 0 and Ag is continuous in O, cf. [20, Lemma 5.15], there exists X; R>o such that A₂(A) S fl for all 0 < < S X. From Assumption 4.6 we get X E R> such that for all M E N+ we have M (A) < for all 0 < X < Xy. Define now \ min X and observe that a > O. Proposition 4.5 ensures the existence of a strictly increasing sequence (MM') ) S N4 with
H M
R aa
H P.P.A.
P
1,
for oc. Choose my N4 such that for all m mi have
R
H *
P
H&+
<
61
m
3
We now set M1 M and get that
H
R Har
Rag plasz]
Hu
R
Mms
R
Mai
R
p
planfi
Rr,PAA
Hye
A1
+
Hix
RE.
H4*
(M
A
(At)+
A2(A1)
El.
We can now repeat the argument from above inductively: Suppose we have constructed our sub- sequence up n E N4, i.c. Mi...., Mn. Choose X' Rss0 such that A²(A) < -
fall for
23

---

all 0 < A < X (exists due to continuity), and X" E Rso such that for all M E N+ we have (A) (A) I tat1 for all À S X" (using Assumption 4.6). Define now Xn+1 min X"). and observe that An+1 > O. Proposition 4.5 ensures the existence of strictly increasing sequence (M such that
R wi
Min (1) Pladl 1,A4+1
He* P.P.
for m + oc. Choose m such that for all m 2 Mn+1 we have
R
H *((++)**
-R thg
6n+1
(x 1yi++1,, plarge 12)) 1,111
Define Mn+1 max + 1, Ammtt)) ), then we get
H (m+1)* #ma-1
R HM * EMe 1 pllatll
He+ R2
<
R
R
Mon+1
pl pl
M Ant1
Ant1)
11-
(n+1)
Mmn+1
M
H in+1)* R
Mintl
Hux P.ntt
+
Ant1
+
R P.
Hux
Hux
M(x+t)
(2n=1) + enti + A2(Ant1)
< A2
S Cn+1-
The resulting sequence (M,),, fulfills then
R HM. Exin P(M,1] + R He*
for n + 00.
B Additional technical results
In this section we state and prove two technical results that play an important role the proofs of the main results.
B.1 A characterization of RKHS functions
Here we recall the following characterization of RKHS functions from [31, Section 1.4). Let X' + be arbitrary. For k X' x X + R symmetric and positive semidefinite and some f E R as well as N N., N á E N define
E(T,0,1) = Σ and (xx)
==1
W(Z,d,)) aajz((z),zı),
where we might omit some arguments if they are clear. Furthermore, define
(( W(Z.
D &. (. =
0
24

---

and
sup (F,G)EX XB N
D f.K).
NEN,
We collect now some simple facts that will be used repeatedly.
Let EEN GERN N N+, be arbitrary, and define
N
Snk(;,an) E H
1. By construction, W(ž, 0z, k) E Rgo (recall that &s is positive semidefinite).
2. Since f E kt its RKHS norm has an explicit form and we find
N
a,a,k(az, = ((T,ā,))
This also implies that f if and only if W( k) =
3. If II 0, then
E(Z, D(E,ā,f,k)
W(7, a k)
a.f(aa)
EEy= a,))k(z,
EN= a;azk(Tj,
VEN= a,ahhl,, 24) W(ī, W(z,ā,k). a(Z,ā,k)
We can now state the characterization result.
Theorem B.1. Let k X' x + be kernel and f R Then f E H2 if and only k) < co. If f e Hx, then | = k).
For convenience, we provide a full self-contained proof of this result.
Proof. Step First, we show that for f E Hę, we have || = NN(f Nff, k) < Ifll Let E N4 and (Ø, @) XN RN be arbitrary. Observe that
E(F,ū, Σ an (#n)
Σ an (f: le(,, xn)))
N
(f.
Σ
an 1e xm))k
n=l
$ ||
Σ ank(( xmn)
= W1,
W = I a. C,, 0, then ΣN -= ank() = O hence E a. f)
(f. )= = 0 and by definition D(ø, a. f. f,k) = 0 < [/
25

---

If W2, k) > O, we can rearrange to get
( ā. f) D(ē, ā, f. & llflla W(z, ā. k)
Since (Ø,, α)) was arbitrary, we find that Wi ã. f. - |lflk
() 2 |f Let e > 0 and choose ft = Σx= k, rn) E H such that |f - fellk < c. If W(E flfelll 0, then fc 0HH and hence E(Ø, a. f) (f.feh (f.0H.) = 0. By definition, this then shows
f) =0 llfella 2 IlAll E.
Before we continue, note that for all f1. f2 E H3s we have
an -(f((zn)-(((z)
=
N
Σ an - -f2, kc,,, Ini)k
(fi f2,
an ki In)k
I f1 f2llx|1fella-
Assume now that W(Z a, > O. then we get
f)
D(E,a,f,k)
W(T,
f()
felfellllfell
W(F k,
E(),
ellSAlle W(T,
W(T,a,K)
a(T,a,
llfella -E
2 llfllA 2c
Altogether, by definition of ((,,)) we get that k(f,k) > f( 2 llfll 2e.
Since e > 0 was arbitrary, we find that ((f,)(fk))) Z ||
Step 2 Let f E R& be arbitrary. We show that if N k)) 00, then
l H
N
N
k (()(( + Σ anf(En)
n=1
is a well-defined, linear and continuous (w.r.t || 11x) map.
To establish the well well-posedness, let (Z, E N x RNN and (J.3) E X RM such that
N (n = Σ
M
Bm ) E H
n=l
m=1
This implies that
N
M
Σ
ank +
(- Bm )..)) ym) = OH
m=1
26

---

and hence W((, (c. = N ank(;,an) + (- = O. Assume now that
N
m
Σ n=1
On (n)( * Σ (Im))
m==
then get that
N
Σ (n)( +
(-3m)) =
m=1
which by definition implies that D((E, (ā, -3), f,k) = oo and therefore N(f,k) contradiction.
The linearity then clear. Finally, to show the continuity, let H 9 fo = En Onpili,, Inn) be ON). then
arbitrary and set o (x) IN), ā = (an
ley(fo)) Σ anf
-E(T,
S ()W(, a, = =N((f,
Since (f., is finite and independent of fo. and E is linear map, this shows the continuity of E
Step 3 Let f R² such that k)) cc. Since according to Step 2 f is linear and continuous map on H and the latter is dense in H₁, there exists unique linear and continuous extension Ey H2 + R of E.. Furthermore, from the Riesz Representation Theorem there exists unique He with E fx.. For all X we then get
f(x) <f,b(;)>))
(k(-,)), Air
()(k(,
()(((,2))
= f(x),
hence f - E Hg.
B.2
--convergence argument
We use repeatedly the concept of cconvergence, see for example [32]. For convenience, in this section we summarize the well-known and standard main argument, roughly following [33, Chapter 5J. Definition B.2. Let FM: HM + R U (20) and F: He - IR (oo}. We say that FM -ccovergges to F and write FM F F, if
1. For all sequences (fM) M, I'M E HM, with JM Ps f for some / Hg. have F(f) lim inf (fM))
2. For all f E H₁ there exists sequence (JM) as with J'm E Hi such that s'r Pi f and F(f) 2 lim limsup FM (fM))
M
The sequence in the second item is commonly called a recovery sequence (for f).
Proposition B.3. Let FMs - F and f& E argmin Ha (f) for all M N (in particular, all the minima are attained). If there exists B Rx0 such that llfi & B for all M N, then there exists a subsequence (e and f Hy such that fia, Pi f". Furthermore, FM (fr,,) F(F)).
27

---

Proof. From Theorem 2.3 we get the existence of (fM,), ) and f E Hj, and that fi. 3 f Let
H3 be arbitrary and let (fM) be recovery sequence for f. We then have
F(f) 2 lim FM
M
sup FM, (fare)
Mr
lim FM, (fMe)
lim inf FM, (fMe)
Me
F(S))
where we used the lim sup-inequality of I'-oovergennce in the first step, standard properties of lim sup and lim inf in the second and third step, the fact that fM, is minimizer of Fas, in the fourth step, and finally the lim inf inequaltyy of -convergeecce. Since f E H2 was arbitrary, this shows that f" minimizer of F.
Furthermore, let (fw) Mr be recovery sequence for f", then
F(f*) 2 lim sup F (fM)
lim FM, (f))
> limsup FM, (fis;);
where we used the limsuupineeuallty in the first step, elementary property of lim sup in the second step, and finally that faz, is a minimizer of Fas Since fire Fi fr, the lim inf-inequality of T T-convergence implies that
F(J)) S lim Fas (fMe))
so we find that
lim sup FM, (fM.) S F(F*) & lim FM, (frs.)
establishing that FM, (fM,) F(S*).
28


Reviews:
[{"review_id": "QKh1H8eE8Q", "reviewer": "Reviewer_m8Hw", "summary": "In this work, the authors considered learning problems where the input is a interacting-particles / multi-agent system and  studied the mean-field limit of the kernels, RKHS functions and SVM solutions as the number of particles tends to infinity. The authors showed that, essentially, taking the infinite-particle limit is exchangeable with 1) obtaining the RKHS associated with a kernel, 2) obtaining the solution of a variational problem over the RKHS, and 3) obtaining the \"SVM solution\" within the RKHS.", "questions": "I don't fully understand why the type of functions considered in Section 4 should be called SVM solutions - there is no support vector involved after all, and instead they seem closer to the solutions to ridge regression / ERM with Tikhonov regularization.\n\nThe loss function $l_M$ defined on Line 265, Page 7 is a bit non-standard: it also takes the input vector $\\vec{x}$ as an argument. Since it complicates things somewhat, it could be helpful to explain in what scenarios this term is relevant.", "limitations": "N/A", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This work continues the exploration by Fiedler et al. (2023) into the mean-field limit of RKHS when we consider interacting-particles systems with a growing number of particles. Compared to Fiedler et al. (2023), the current work extends the theory by 1) tightening the theoretical result on the commutative relation between taking the infinite-particle limit and obtaining the RKHS (no longer requiring taking another subsequence & controlling the RKHS norm of the limiting function, as reported by the authors on Page 4), and 2) expanding the discussion into the mean-field limit of minimization problems in the RKHS. The proofs of these results appear to involve rigorous techniques from functional analysis and gamma convergence, though I was unable to fully verify their correctness.", "weaknesses": "While the infinite-particle limit of the learning problem leads us to interesting theoretical investigations, its practical relevance remains to be seen. Overall, the analysis is carried out at a rather general and abstract level, which is perfectly fine for establishing the theory but makes the motivation harder to interpret by the reader. It is therefore not clear to me whether the work could be of great interest to the NeurIPS audience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your careful and detailed review. Below we address your concerns, answer the posed questions, and provide additional comments and remarks.\n\n**Weaknesses**\n>While the infinite-particle limit of the learning problem leads us to interesting theoretical investigations, its practical relevance remains to be seen. Overall, the analysis is carried out at a rather general and abstract level, which is perfectly fine for establishing the theory but makes the motivation harder to interpret by the reader. It is therefore not clear to me whether the work could be of great interest to the NeurIPS audience.\n\nWe fully agree that the present work focuses on theoretical aspects, which might pose a challenge to make the motivation and practical relevance clear. However, we decided to use this approach in order to avoid overloading the submission and keep it focused on the pertinent questions. Machine learning for interacting particle / multiagent systems has become a thriving field in the last years (cf. [21] and [R1]), and the challenges of very large-scale systems (corresponding to our $M\\rightarrow\\infty$ limit) in this context have been barely tackled so far. We are confident that the theoretical foundations we provide will be helpful in this area. In fact, we are currently working on empirical investigations of mean field limit kernels in the context of interacting particle systems where the computational benefit of treating the mean field case will become apparent. However, including these aspects and the necessary background (from interacting particle systems, kinetic equations, numerical methods for kinetic PDEs) is the beyond the scope of this submission, and we think it would actually deteriorate the quality of presention.\nFurthermore, as briefly mentioned in the Introduction, large-scale limits have become very popular in machine learning in different contexts, and we think that our theoretical results are an interesting contribution to this body of work, providing a different perspective.\n\n**Questions**\n>I don't fully understand why the type of functions considered in Section 4 should be called SVM solutions - there is no support vector involved after all, and instead they seem closer to the solutions to ridge regression / ERM with Tikhonov regularization.\n\nWe agree that in the rather general setting that we consider, the solutions $f^\\ast_{\\mathcal{D}_N,\\lambda}$, $f^\\ast_{P,\\lambda}$ etc., do not necessarily correspond to \"classic\" SVMs. However, we use the terminology from [26], and  decided to follow their convention, cf. the text immediately after (5.8) and before Theorem 5.3 in this reference. This terminology has also been used in other NeurIPS submissions in the past, e.g., [R2]. However, to avoid confusion, we will add some qualifications to it, i.e., that our infinite-sample / empirical SVM solutions correspond to (Tikhonov) regularized ERM solutions over RKHSs.\n\n>The loss function $\\ell_M$ defined on Line 265, Page 7 is a bit non-standard: it also takes the input vector $\\vec x$ as an argument. Since it complicates things somewhat, it could be helpful to explain in what scenarios this term is relevant.\n\nWe are following the rather general setup of [26, Chp 2]. A common special case is a supervised loss function as defined in [26, Def. 2.7], where the loss function does not depend on the input. In this special case, the setup in Section 4 simplifies considerably, e.g., we do not need to consider the mean field limit of loss functions anymore. However, there are practically relevant loss functions that depend on the inputs. A common example is given by unsupervised loss functions as defined in [26, Def. 2.8], which encompass for instance loss functions used for density level detection, cf. [26, Example 2.9]. Finally, in some applications loss functions with all three arguments appear. For example, in classification problems the cost of misclassification might depend on the input (i.e., there are some instances where misclassification is more costly than in others). By treating the general case (as formalized in [26, Def. 2.1]), all of these cases are handled by our results. We hence decided to use this general form here.\n\n**Strengths**\nFinally, we would like to point out that apart from 1) and 2) mentioned in the review, this submission contains two other substantial contributions.\n* The limiting function alluded to in 1) is actually contained in the mean field limit kernel RKHS (and as a byproduct, we can even control the corresponding RKHS norm). While intuitively clear, showing this is highly nontrivial and has not been established in [15], which allows us to \"close the commutative diagram\" (Fig 1).\n* In Section 4, we provide an appropriate mean field limit setup of the standard statistical learning theory setting, which allows us to formulate and prove the various mean field limit results using mean field limit kernels. To the best of our knowledge, this has not been done before, and required some additional technical contributions (e.g., Prop 2.1 and Def. (10)). While we only provided an initial investigation of this setup, we expect that interesting follow up work can be conducted here (actually, some of it is already in progress).\n\n[R1] Lu, F., Maggioni, M., & Tang, S. (2021). Learning interaction kernels in heterogeneous systems of agents from multiple trajectories. _The Journal of Machine Learning Research_, _22_(1), 1518-1584.\n[R2] Christmann, Andreas, and Ingo Steinwart. \"How SVMs can estimate quantiles and the median.\" _Advances in neural information processing systems_ 20 (2007)."}, {"review_id": "55hk8TINgp", "reviewer": "Reviewer_SuzB", "summary": "This work derives mean-field limits of kernels and their associated Reproducing Kernel Hilbert Spaces. In particular, the authors quantify the relationship between the finite-input RKHS and the RKHS of the mean-field kernel, derive a Representer Theorem for mean-field kernels, and provide asymptotic convergence guarantees for mean-field kernel SVMs.", "questions": "See above. Overall, the results are quite interesting and a bit of work on the motivation and overall presentation would greatly improve the paper. I would be happy to increase my score if the authors are able to address said concerns.", "limitations": "Limitations are adequately discussed. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The results are interesting and well-presented. The authors present a complete treatment of the mean-field limit for kernel methods. The proofs are clear and well-written.", "weaknesses": "1. While the results are interesting on its own, I am a little skeptical about the applicability of these results in existing ML problems. I think this work would greatly benefit from a more detailed discussion on the motivation and applications. On a related note, I also encourage the authors to present concrete examples of mean-field kernels for ease of exposition. \n\n2. The complete absence of rates of convergence to the mean-field limit (even for specific kernel classes) is somewhat unsatisfactory. I would appreciate it if the authors could comment on that. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review and insightful questions.\n\n**Weaknesses / Questions**\n>1. While the results are interesting on its own, I am a little skeptical about the applicability of these results in existing ML problems. I think this work would greatly benefit from a more detailed discussion on the motivation and applications. On a related note, I also encourage the authors to present concrete examples of mean-field kernels for ease of exposition.\n\nWe agree that the motivation and the discussion of practical aspects are rather terse, but we wanted to properly present the theory. In the general response above, we have detailed a concrete learning scenario that motivated the present work, on which we want to apply the theory developed here. We will incorporate this into the Introduction, to make the motivation and application scenario clearer.\n\nRegarding examples of mean field kernels: Several general classes of these objects have been described in [15]. Moreover, recently a concrete analytical form of a mean field kernel has been derived which is used for ongoing numerical investigations. However, we believe that introducing and investigating such a concrete instance would not fit the scope of the present work. Actually, we would argue that the abstraction here is a major strength of the work, since all of our results apply to all mean field kernels as introduced in [15], just as the developments in Section 4 are valid for all sequences of loss functions compatible with Proposition 2.1\n\n>2. The complete absence of rates of convergence to the mean-field limit (even for specific kernel classes) is somewhat unsatisfactory. I would appreciate it if the authors could comment on that.\n\nThis is an excellent point. In this work, we focus entirely on convergence per se, and not investigating rates of convergence in the mean field limit. On the one hand, convergence itself is already an interesting and challenging problem, and also helpful from a practical perspective (justifying working on a mesoscopic level). On the other hand, having rates w.r.t. $M$ would be interesting for practitioners (having a rule of thumb when $M$ is large enough to confidently rely on a kinetic approximation) and also in connection with learning rates (e.g., considering the simultaneous limit $M,N\\rightarrow\\infty$). However, getting such rates is non-trivial and to the best of our knowledge usually not readily available even in classic applications of kinetic theory. We therefore considered this (very interesting) question beyond the scope of the present work. Ensuring rates w.r.t. $M$ is actually subject to ongoing work, and we conjecture that it requires additional regularity conditions on the problem (which would also be contrary to our goal here of being very general).\n"}, {"review_id": "ovHUXiGkf9", "reviewer": "Reviewer_NtYB", "summary": "This paper develops mathematically rigorous construction of the mean field limit of kernels of probability measures, which is obtained as a limit of a sequence of kernels with increasing input dimensions, and its application to SVMs.\nThis is motivated by the analysis of interacting particle systems, when many observations of particles are available and we want to consider a function governing the dynamic as a function of a probability measure rather than a finite-dimensional vector.\n\nSpecifically, this paper first confirms the validity of such kernels by extending the existing results: the existence of the mean field limit of functions and kernels, and relations between a sequence of kernels and the mean-field limit of kernels.\nThen, they prove the representer theorem (a mean-field limit of a sequence of functions uniformly approximated by each corresponding RKHS functions can also be approximated by the mean-field limit of RKHS functions) and convergence of the minimizers.\nFinally, the results are applied to an L2-regularized (with $\\lambda \\geq 0$) loss minimization problem over RKHS.\nThey show the existence and uniqueness of the solution (when $\\lambda > 0$) and convergence of the loss value.", "questions": "- Is it possible to add any experiment to the paper?\n\n- Could you tell me more about the difference between Prop. 2.1 and the previous work, and whether such improvement is necessary for the rest part?", "limitations": "In order to avoid confusion with optimization and generalization theory of mean-field neural networks, the authors are encouraged to change the title.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Problem settings are well-motivated\n\nThe purpose of this paper is to develop a foundation of the theory of kernels with probability measures as input in order to view the state of an interacting particle system not as observations of a finite number of particles, but as a function of a probability measure that represents the distribution of the particles. The problem of learning a function with infinite dimensional input may have many other applications, and the potential impact in this area is of a certain significance. \n\n### Rigorous theory for infinite-dimensional input functions\n\nThis paper investigates the formulation and thus the basic properties of RKHS for the limit of a sequence of kernels of increasing dimension. However, it should be noted that the main focus is basically on how to define the limit, and not on how to use it specifically. It can be said that this research has a certain novelty if there is no similar discussion.", "weaknesses": "### Misleading title\n\nIt looks like a paper about generalization error analysis for the community of mean-field neural network optimization. Therefore, I strongly recommend that the authors change the title to characterize their contents more precisely.\n\n### Missing discussion on whether the existing analysis on learning infinite-dimensional input functions can be applicable.\n\nThe final part of this paper is essentially a problem of learning functions with infinite-dimensional inputs, but the discussion on the previous works / applications is missed. The authors should make a remark on whether such existing theories are applicable and hopefully conduct some experiments to verify their theory. An example of previous work I think is:\n\nF. Ferraty, A. Mas, and P. Vieu. Nonparametric regression on functional data: inference and practical aspects. Australian & New Zealand Journal of Statistics, 49(3):267–286, 2007.\n\n### The authors say that the first part of the paper is improvement from an under review paper. However, I cannot know the difference and whether such improvement is necessary for the rest part.\n\nThis paper consists of the three part. (i) Definitions of functions and kernels in the mean-field limit, (ii) approximation ability and convergence of the minimizers, and (iii) application to SVMs based on (ii).\nThe authors argue that (i) is improvement from an under review paper. However, I cannot see the difference and whether such improvement is necessary for deriving (ii) and (iii). If not, this paper looks like an assortment of different results. Also from the perspective of preventing double submission, I think the authors should clarify the difference between their results in (i) and whether such an improvement is necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your very detailed and careful review. Below we address your concerns and answer the posed questions.\n\n**Weaknesses**\n>Misleading title [...]\n\nThank you for pointing out this risk of confusion, which we weren't aware of. To avoid any possible confusion, we suggest to change the title to \"On kernel-based statistical learning in the mean field limit\". We would argue that the term \"mean field limit\" (MFL) should remain in the title, since this is the accepted term of the considered setting (cf. [7]).\n\n>Missing discussion on whether the existing analysis on learning infinite-dimensional input functions can be applicable.\n\nWhile functional data analysis (FDA), as considered in Ferraty et al, also deals with infinite-dimensional data, the crucial difference is that the infinite-dimensional data here arises as a particular limit (the MFL) of finite-dimensional data, and we are interested in this limit (and its consequences for kernels / kernel-based methods). Furthermore, the typical data in FDA is rather different ($L^2$ functions, often on $[0,1]$). The situation is similar with kernels for infinite-dimensional inputs: Of course, after going to the limit, we are in this realm (as mentioned in the Introduction, cf. [12]), but this work is about the limits themselves (definition, existence, properties) and the repercussions/chances for kernel methods/theory.\nWe will add a corresponding discussion to the Introduction to make this aspect clearer.\n\n>The authors say that the first part of the paper is improvement from an under review paper. \n\nThe paper [15] has been accepted and published in the meanwhile (doi: 10.3934/krm.2023010), we will update the bibliography accordingly.\n\n>This paper consists of the three part. [...]\n\nThe main contributions in part (i) (Section 2) are \n1. The MFL function in Theorem 2.2 (existence was already established in [15, Corollary 4.3]) is contained in the MFL kernel RKHS, and shares the same RKHS norm bound.\n2. Lemma 2.4, 2.5, necessary for the $\\Gamma$-convergence arguments used in Sections 3 and 4. These results are novel (in particular, not contained in [15]), and the corresponding proofs are technically demanding and use much more sophisticated arguments than those utilized in [15]. Furthermore, these results are necessary for most of the proofs of the results in parts (ii), (iii) (i.e., Sections 3 and 4).\n\nAdditional contributions in part (i) are\n1. Proposition 2.1, which is necessary for the statistical learning theory setup in Section 4, cf. the Questions paragraph below.\n2. Avoid going to a subsequence in Theorem 2.1. On the one hand, this is necessary for the $\\Gamma$-convergence arguments used later on, on the other hand, this makes the commutative diagram (Fig 1) and its interpretation nicer.\n\nWe would like to emphasize that this submission is building on [15], but nothing claimed as a contribution of this submission is contained already in [15].\n\n**Questions**\n>Is it possible to add any experiment to the paper?\n\nWe decided not to do so in order to keep the manuscript focused on the theoretical aspects (which take up already a significant amount of space). However, we are working on numerical experiments and empirical evaluations of the concepts introduced here. In our opinion, including such experiments (and the required background on interacting particle systems, kinetic equations, Monte Carlo methods, and numerical methods for kinetic PDEs) would severely degrade the quality of the presentation and make the submission less readable.\n\n>Could you tell me more about the difference between Prop. 2.1 and the previous work, and whether such improvement is necessary for the rest part?\n\nProposition 2.1 is a generalization of a well-known result (cf. [6, Lemma 1.2]). More precisely, Prop 2.1 allows sequences of functions with a non-compact input domain, and existing results like [6, Lemma 1.2] only work for compact input domains.\nThis extension is necessary because in general the third argument of the loss functions in Section 4 cannot be restricted to a compact set. Typically, a loss function $\\ell_M$ is used as $\\ell_M(\\vec x_M, y, f_M(\\vec x_M))$ for some input $\\vec x_M$, output $y$, and hypothesis $f_M\\in H_M$ (using the notation from the submission). However, in the regularized optimization approaches over $H_M$ as considered here, like the minimization problem\n$$\n\\min_{f_M\\in H_M} \\mathcal{R}_{\\ell_M,D_N^{[M]}}(f_M) + \\|f_M\\|_M^2,\n$$\nall $f_M\\in H_M$ are allowed. This means that in $\\ell_M(\\vec x_M, y, f_M(\\vec x_M))$, any $f_M\\in H_M$ could appear, and since even for a fixed $\\vec x_M\\in X^M$ we hence cannot restrict the range of $f_M(\\vec x_M)$ a priori, we cannot restrict the third argument of the loss functions to a compact subset of the real numbers. Note that clipping techniques frequently employed in kernel-based statistical learning theory (cf. [26, Def. 2.22]) unfortunately do not work here. Since we need the MFL of the loss functions $\\ell_M$ for our setup, and since existing MFL results like [6, Lemma 1.2] consider only functions with compact input sets, we needed Prop 2.1.\n"}, {"review_id": "CzY6mFqZJX", "reviewer": "Reviewer_sYSX", "summary": "This paper studies the theory of RKHS consisting of the functions over the space of probability distributions. Complementing the related study [15], the work develops the particle-based approximation theory to RKHS and develops the support vector machines fed distributions as inputs. Moreover, the statistical consistency is also proven.", "questions": "Readers might be interested in the particle complexities as $M\\to\\infty$ because the large number of particles can affect the computational complexity. How large is $M$ needed to achieve a given accuracy in a certain sense?", "limitations": "It is somewhat unclear how many applications can exist. It would be nice if some applications were mentioned.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper organizes well the theory of RKHS defined over the space of probability distributions. The results seem technically correct. There are potentially many applications in machine learning. Hence, this work can be of interest to the machine learning community.", "weaknesses": "- While I recognize many potential applications, it would be nice if the authors could provide concrete examples and datasets to emphasize the importance of the paper.\n\n- The usefulness and computational complexity of the learning setups studied in the paper are somewhat unclear because of the lack of specific examples and experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful review. We now address your concerns and answer the posed questions.\n\n**Weaknesses**\n>While I recognize many potential applications, it would be nice if the authors could provide concrete examples and datasets to emphasize the importance of the paper.\n\nWe have decided to focus on the theory in order to avoid overloading the submission (presenting the theory properly with all relevant details already takes up a lot of space). Furthermore, we would like to stress that beyond concrete applications, the theory itself appears to be interesting and relevant for the machine learning community.\n* Kernels and RKHSs are important and well-studied, but the type of kernel limits we consider (motivated by concrete applications) is new\n* To the best of our knowledge, the limit of statistical learning problems as introduced and investigated in Section 4 is new (even the notion of such a limit setup)\n* We connect kinetic theory (MFL, $\\Gamma$-convergence) with the theory of reproducing kernels\nFinally, regarding concrete examples and datasets, see the paragraph below.\n\n>The usefulness and computational complexity of the learning setups studied in the paper are somewhat unclear because of the lack of specific examples and experiments.\n\nThe focus of this submission is on the kernels / RKHSs / statistical learning problems / SVMs in the mean field limit, investigating and discussing computational complexity and practical issues is unfortunately beyond the scope of this submission. This approach is similar to past NeurIPS papers on related topics, for example [12]. We agree that these are relevant issues, but in our opinion, they should be carefully treated in a separate work.\n\n**Questions**\n>Readers might be interested in the particle complexities as $M\\rightarrow\\infty$ because the large number of particles can affect the computational complexity. How large is $M$ needed to achieve a given accuracy in a certain sense?\n\nThis is an excellent and important question. Regarding the computational complexity, when working with the limit objects directly (e.g., an SVM with a MFL kernel), a large $M$ is not a problem -- after all, this is one of the main motivations of this approach. This is completely analogous to the situation of kinetic PDEs, where only the state dimension (corresponding to our dimension $X$), but not the number of particles (here $M$) plays a role.\n\nRegarding the question of accuracy, we indeed do not provide rates of convergence in the mean field limit. To the best of our knowledge, such rates are readily available even in classic applications of kinetic theory, and hence our setup inherits this feature. Borrowing statistical terminology, our results are about consistency, not rates. However, we argue that already the convergence per se (i.e., consistency) is interesting, relevant, and helpful (since it justifies a kinetic theory approach for machine learning methods in this setting and might lead to more efficient numerical methods applicable on the equivalent mesoscopic level). Furthermore, note that having rates w.r.t. $M$ might not be very consequential in practice, since the $M$ is determined by the problem (e.g., the size of the population of interaction particles), and cannot be chosen by the learner (in contrast to the data set size $N$, which often can be increased by simply collecting more data). Nevertheless, providing rates for the mean field limit convergence is a very interesting question that should be investigated in future work.\n\n>It is somewhat unclear how many applications can exist. It would be nice if some applications were mentioned.\n\nDue to space constraints, and our focus on theoretical questions, we have not been able to discuss applications in detail. However, we are motivated by concrete learning problems, as we outlined in the introduction. Since the question of applications/learning scenarios was raised by several reviewers, we have elaborated on this in the general author's answer above."}, {"review_id": "sb18eMB8HE", "reviewer": "Reviewer_Vo6s", "summary": "The authors explore the mean field limit of kernels and their reproducing kernel Hilbert spaces, providing novel theoretical tools and insights for tackling large-scale problems. Furthermore, they employ these kernels in statistical learning, with a particular focus on Support Vector Machines. This new form of limit for learning problems has not been investigated before in the statistical learning theory literature, making the authors' research a valuable contribution to the field.", "questions": "1. What is the main improvement of this paper compared to the reference [15]?\n\n2. What is the relationship and difference of this study compared to neural tangent kernel? What is the advantage of mean field kernels?\n\n3. Any concrete examples for mean field kernels?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors provide a comprehensive investigation of the mean field limit of kernels and their reproducing kernel Hilbert spaces.\n\n2. The research extends the basic theory of mean field kernels, providing novel approximation results and a variant of the representer theorem for mean field kernels.\n\n3. The authors introduce a new form of limit for learning problems and provide convergence results for Support Vector Machines using mean field kernels.", "weaknesses": "1. The limitations or drawbacks of the research are not explicitly stated.\n\n2. Section 4 only provides a consistency result instead of a generalization error bound.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review and insightful questions. Below we address your concerns and answer the posed questions.\n\n**Weaknesses**\n>Section 4 only provides a consistency result instead of a generalization error bound.\n\nIndeed, we only consider convergence per se in Section 4 and do not provide rates or generalization bounds. These latter aspects are very interesting and relevant, and subject ongoing investigations. However, we would argue that establishing consistency is already interesting and challenging, which motivated us to focus on this in the present work.\n\n**Questions**\n>What is the main improvement of this paper compared to the reference [15]?\n\nReference [15] does not consider mean field limits of kernels in the context of statistical learning theory, so the focus of this work and [15] is rather different. The contributions of this work, in particular in comparison with [15], are as follows\n1. Extending and completing the theory of mean field limits of kernels and RKHS\n* Theorem 2.3.2: We show that the limiting function (existence has been established in [15, Corollary 4.3]) is actually in the RKHS of the mean field limit kernel, and shares the same RKHS norm bound. On the one hand, this result is necessary for the proofs in Sections 3 and 4. On the other hand, it completes the commutative diagram in Figure 1, which was already suggested in [15], but an important link was missing. Note that the proof of this result  requires considerably more sophisticated tools than those used in [15].\n* $\\Gamma$-convergence inequalities from Lemma 2.4, 2.5: These results are necessary for the proofs in Sections 3 and 4, and make the theory of mean field limits of kernels much easier to handle. The idea to use $\\Gamma$-convergence in the present situation and establish the necessary inequalities is a contribution of this work. Interestingly, it seems that this is also the first time that $\\Gamma$-convergence arguments have been used in the context of reproducing kernels.\n* Theorem 2.3.1: In contrast to the results in [15], we do not need to use a further subsequence. On the one hand, this is necessary for the $\\Gamma$-convergence results, on the other hand, it completes the commutative diagram from Figure 1.\n2. Approximation and mean field limits of kernels\n* Proposition 3.1, Remark 3.2: The approximation capabilities of mean field limit kernels have not been investigated in [15]. \n* Theorem 3.3: In many cases, numerical implementations of kernel methods rely on various forms of the representer theorem, and such a result in the mean field limit is completely new. Furthermore, it seems that this is also the first representer theorem that considers the situation of limits of kernels.\n3. Statistical learning theory and mean field limit kernels\n* Statistical learning theory setup in the mean field limit: The formalization of such a mean field limit, including the existence result in Proposition 2.1 and the notion of mean field convergence of probability distributions, is new (and nontrivial).  Furthermore, it appears that this is also the first instance of a limit of statistical learning problems that has been investigated.\n* Convergence results for SVMs and their risks (Section 4): All of these results (and their setting) are new. Note that we also need new and more sophisticated techniques compared to [15] to prove these results.\n\n>What is the relationship and difference of this study compared to neural tangent kernel? What is the advantage of mean field kernels?\n\nThis is an excellent question. Mean field limit kernels and the neural tangent kernel (NTK) are very different objects, though both arise through a large scale limit (i.e., the number $M$ of entities going to infinity). Some of the technical differences are\n* Mean field limit kernels arise as the limit of kernels, whereas the NTK is a way to connect neural networks to kernel methods\n* The overall input space stays the same for the NTK (the number of hidden units goes to infinity), whereas the input space changes for mean field limit kernels\n* The convergence notions are different (NTK usually involves convergence in probability, we consider mean field convergence results)\nIt is difficult to say whether the NTK or mean field limit kernels are \"better\" , since these objects are very different. Using mean field limit kernels in the context of neural network theory could be interesting, but we have not investigated this question. The main motivation for mean field limit kernels (and their practical strength) is their usage in the context of kinetic theory, as outlined in the introduction (see also the general response above for a more detailed description of our motivation). \n\n>Any concrete examples for mean field kernels?\n\nIn this work, we focus on the general theory, which we consider an advantage - all of our results are immediately applicable to any sequence of appropriate kernels with a mean field limit. Large concrete classes of mean field limit kernels (more precisely, kernel sequences that have a mean field limit kernel) have been in introduced in [15], in particular,\n* Kernels from the well-known pull-back construction\n* Double-sum kernels\nSince this work focuses on the theory of mean field limit kernels in statistical learning (which required extending the general theory of these kernels, cf. Section 2, and dealing with approximation questions, cf. Section 3), we decided to not include this aspect here.\n\nActually, we recently derived an analytical form of a mean field kernel, which we use for numerical experiments in the context of interacting particle systems. However, we found that including this (and the relevant background from kinetic theory) would deteriorate the quality of the submission (since it dilutes the focus)."}]

Author rebuttal:
We thank the reviewers for their careful, detailed reviews and their interesting and insightful questions. We have answered each reviewer separately, and hope to have addressed all of their concerns and questions.

In the general part, we would like to address an aspect that was raised by more than one reviewer, namely the motivation and application scenario of the present work. 

First of all, we would like to emphasize that the focus of this submission is on theoretical investigations, and in our opinion, the theory itself (beyond specific applications) is interesting and relevant for the machine learning community. Nevertheless, we were motivated by concrete learning problems (as outlined in the Introduction of the submission), which we consider relevant and interesting, and for which the developments presented here will be helpful in our opinion: 

Consider e.g. complex interacting particle systems (IPS) / multiagent systems (MAS), that are difficult to model using first principles. Important examples include (see for instance [22])
* Animal populations (swarms of birds, schools of fish, colonies of microorganisms)
* Human crowds (pedestrian movement, gathering at large events like football games or concerts)
* Vehicular traffic (in particular, traffic jams)
* Markets (where the agents can be human or legal entities)
* Opinion dynamics (inside a society), like in [27]

Frequently the state of such a system/population can be easily measured, e.g., by
* Video recordings or image snapshots of bird swarms/schools of fish, microscopy recordings of microorganism colonies
* Aerial imaging of human crowds (e.g., via quadcopters)
* Polling and social media analysis for opinion dynamics

However, some interesting features of the whole system might be more difficult to measure. For example, 
* How a swarm of birds or a school of fish will react to an external stimulus (like an approaching predator), given the current state of the population, e.g., whether there will be a change in the density/spread of the population or change in mean velocity.
* Features of a society in opinion dynamics (average happiness, aggression potential, susceptibility to adversarial interventions), given the current "opinion state"
Measuring such features can be difficult, for example, due to a required intervention. Formally, such a feature is a functional of the current state of the system, and since the state is often easy to measure, it would be useful to have an explicit mapping from state to feature of interest. However, since first principles modeling is unlikely to be successful in the domains considered here, it is promising to learn such a mapping from data (see the Introduction for a formalization). Since this is a standard supervised learning problem (state snapshots as input, noisy measurement of the feature of interest at the state snapshot), it is immediately amenable to kernel methods.

Many of the systems considered above can consist of a large number of agents (thousands of birds in a swarm, millions of microorganisms in a colony), and working on the microscopic or individual level becomes impractical. As done in kinetic theory, it might be beneficial to work on the mesoscopic level, considering only the density of particles/agents on state space. For example, when investigating large swarms of birds, the current density of agents over the state space can be easily measured using commodity imaging equipment, whereas tracking individual birds would require high-resolution imaging and working with very high-dimensional data. Formally, going from the microscopic to the mesoscopic level corresponds to letting the number of particles/agents $M$ tend to infinity, and this can be made rigorous using for example the mean field limit. Based on results and experiences from kinetic theory (e.g., [7]), it is reasonable to assume that our feature of interest (a functional on the state space) has a mean field limit as well (see the Introduction and (1)). Returning to the learning problem, we now have a data set on the mesoscopic level, so the input data (state snapshots) are now  probability distributions over the state space. Using kernel methods, we now lift kernels to  probability distributions over kernel spaces, cf. [12]. Now, all methods and results on the mesoscopic level should be connected to the microsopic level through the mean field limit, and this leads exactly to the problems investigated in this submission.

In practice, we would apply the methods on the mesoscopic level to data originating from the microscopic level (though with large $M$). The various convergence results in Section 4 then ensure that the learning outcome on the mesoscopic level (here, the learned functional of the state) is an approximation to the result on the microscopic level for large $M$. This mimics the situation in kinetic theory, where kinetic PDEs (and associated numerical methods) are used for finite, but large $M$ systems. 

We hope that we could make this motivation clearer, and we will adapt the Introduction accordingly.


Meta-review: This paper gives a theoretically rigorous construction of an RKHS on the space of probability measures. They provided a convergence guarantee for an infinite limit of the number of particles approximating each probability measure. This yields a complementary result on existing work.   
The work is theoretically rigorous and the reviewers are overall positive on this paper. I recommend acceptance of this paper.  

BTW, I encourage the authors to include more concrete examples of real world applications of this approach in the final version.
Decision: Accept (poster)