Title: On kernel-based statistical learning theory in the mean field limit
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reproducing Kernel Hilbert Spaces

Kernel Methods

Mean Field Limit

Interacting Particle Systems

Support Vector Machines

Statistical Learning Theory
Abstract: In many applications of machine learning, a large number of variables are considered. Motivated by machine learning of interacting particle systems, we consider the situation when the number of input variables goes to infinity. First, we continue the recent investigation of the mean field limit of kernels and their reproducing kernel Hilbert spaces, completing the existing theory. Next, we provide results relevant for approximation with such kernels in the mean field limit, including a representer theorem. Finally, we use these kernels in the context of statistical learning in the mean field limit, focusing on Support Vector Machines. In particular, we show mean field convergence of empirical and infinite-sample solutions as well as the convergence of the corresponding risks. On the one hand, our results establish rigorous mean field limits in the context of kernel methods, providing new theoretical tools and insights for large-scale problems. On the other hand, our setting corresponds to a new form of limit of learning problems, which seems to have not been investigated yet in the statistical learning theory literature.

On kernel-based statistical learning in the mean field limit

Abstract

In many applications of machine learning, a large number of variables are considered. Motivated by machine learning of interacting particle systems, we consider the situation when the number of input variables goes to infinity. First, we continue the recent investigation of the mean field limit of kernels and their reproducing kernel Hilbert spaces, completing the existing theory. Next, we provide results relevant for approximation with such kernels in the mean field limit, including a representer theorem. Finally, we use these kernels in the context of statistical learning in the mean field limit, focusing on Support Vector Machines. In particular, we show mean field convergence of empirical and infinite-sample solutions as well as the convergence of the corresponding risks. On the one hand, our results establish rigorous mean field limits in the context of kernel methods, providing new theoretical tools and insights for large-scale problems. On the other hand, our setting corresponds to a new form of limit of learning problems, which seems to have not been investigated yet in the statistical learning theory literature.

1 Introduction

Models with many variables play an important role in many fields of mathematical and physical sciences. In this context, going to the limit of infinitely many variables is an important analysis and modeling approach. Interacting particle systems are a classic example; these are usually modeled as dynamical systems describing the temporal evolution of many interacting objects. In physics, such systems were first investigated in the context of gas dynamics, cf. [1]. Since even small volumes of gases typically contain an enormous number of molecules, a microscopic modeling approach quickly becomes infeasible and one considers the evolution of densities instead [2]. In the past decades, interacting particle systems arising from many different domains have been considered, for example, animal movement (inter alia swarms of birds, schools of fish, colonies of microorganisms) [3, 4], social and political dynamics [5, 6], crowd modeling and control (pedestrian movement, gathering at large events like football games or concerts) [7-9], swarms of robots [10-12] or vehicular traffic (in particular, traffic jams) [13]. There is now a vast literature on such applications, and we refer to the surveys [14-16] as starting points. A prototypical example of such a system is given by ˙ x i = 1 M ∑ M j =1 ϕ ( x i , x j )( x j -x i ) , for i = 1 , . . . , M , where M ∈ N + particles or agents are modelled by their state x i ∈ R d , i = 1 , . . . , M , evolving according to some interaction rule ϕ : R d × R d → R . Typical questions then concern the long-term behavior of such systems, in particular, emergent phenomena like consensus or alignment [17]. While first-principles modeling

has been very successful for interacting particle systems in physical domains, using this approach to model the interaction rules in complex domains like social and opinion dynamics, pedestrian and animal movement or vehicular traffic, can be problematic. Therefore, learning interaction rules from data has been recently intensively investigated, for example, in the pioneering works [18, 19]. The data consists typically of (sampled) trajectories of the particle states, potentially with measurement noise, and the goal is to learn a good approximation of the interaction rule ϕ .

Our work is motivated by a related problem. Frequently, the state of such a complex multiagent system can be easily measured or estimated, e.g., by video recordings or image snapshots for bird swarms or schools of fish, and microscopy recordings for microorganism colonies; aerial imaging for human crowds (e.g., via quadcopters); and polling and social media analysis for opinion dynamics. However, some interesting features of the whole system might be more difficult to measure. For example, how a swarm of birds or a school of fish will react to an external stimulus (like an approaching predator), given the current state of the population. Such a reaction could be a change of density or spread of the population, or a change in mean velocity. Another example is given by features of a society in opinion dynamics (average happiness, aggression potential, susceptibility to adversarial interventions), given the current "opinion state". Measuring such features can be difficult, for example, due to a required intervention. Formally, such a feature is a functional F M : ( R d ) M → R of the current state of the system, and since the state is often easy to measure, it would be useful to have an explicit mapping from state to feature of interest. However, since first principles modeling is unlikely to be successful in the domains considered here, it is promising to learn such a mapping from data. We can formalize this as a standard supervised learning task: The data set consists of D [ M ] N = (( ⃗ x 1 , y 1 ) , . . . , ( ⃗ x N , y N )) , where ⃗ x n ∈ ( R d ) M are snapshot measurements of the particle states (corresponding to the input of the functional) and y n ∈ R is the value of the functional of interest, potentially with measurement noise, at snapshot state ⃗ x n . Let us assume an additive noise model, i.e., y n = F M ( ⃗ x n ) + ϵ n for n = 1 , . . . , N , where ϵ 1 , . . . , ϵ N ∈ R are noise variables. This is now a regression problem that could be solved, for example, using a Support Vector Machine (SVM) [20]. Note that for this we need a kernel k M : ( R d ) M × ( R d ) M → R on ( R d ) M .

Similarly to classical physical examples like gas dynamics, the case of a large number of particles is also relevant in modern complex interacting particle systems. Since this poses computational and modeling challenges, it can be advantageous to go also here to a kinetic level and model the evolution of the particle distribution instead of every individual particle. It is well-established how to derive a kinetic partial differential equation from ordinary differential equations systems on the particle level, for example, using the Boltzmann equation or via a mean field limit, cf. [17] for an overview in the context of multi-agent systems. Formally, instead of trajectories of particle states of the form [0 , T ] ∋ t ↦→ ⃗ x ( t ) ∈ ( R d ) M , we then have trajectories of probability measures [0 , T ] ∋ t ↦→ µ ( t ) ∈ P ( R d ) . This immediately raises the question of whether the learning setup outlined above also allows a corresponding kinetic limit. More precisely, let K ⊆ R d be compact and assume that all particles remain confined to this compactum, i.e., x i ( t ) ∈ K for all i = 1 , . . . , M and all t ∈ [0 , T ] under the microscopic dynamics. 1 If the underlying dynamics have a mean field limit, then it is reasonable to assume that the finite-input functionals F M : K M → R converge also in mean field to some F : P ( K ) → R for M →∞ , see Section 2 for a precise definition of this notion. In turn, we can now formulate a corresponding learning problem on the mean field level: A data set is then given by D N = (( µ 1 , y 1 ) , . . . , ( µ N , y N )) , where µ n ∈ P ( K ) are snapshots of the particle state distribution over time and y n ∈ R are again potentially noisy measurements of the functional. Assuming an additive noise model, this corresponds to y n = F ( µ n ) + ϵ n , n = 1 , . . . , N . If we want to use an SVM on the kinetic level, we need a kernel k : P ( K ) ×P ( K ) → R on probability distributions. There are several options available for this, see e.g. [21]. However, assuming that all ingredients of the learning problem arise as a mean field limit, this naturally leads to the question of whether a mean field limit of kernels exists, and what this means for the relation of the learning problems on the finite-input and kinetic level. In [22], this reasoning has motivated the introduction and investigation of the mean field limit of kernels. In the present work, we extend the theory of these kernels and investigate them in the context of statistical learning theory. In particular, since in practice one would use the mean field kernels on microscopic data with large, but finite M , we need convergence results of the various objects appearing in statistical learning with kernels. Exactly such results are provided in Section 4.

Finally, we would like to stress that the technical developments here are independent of the motivation outlined above, in that they apply to mean field limits of functions and kernels that do not necessarily arise form the dynamics of interacting particle systems.

Contributions Our contributions cover three closely related aspects. 1) We extend and complete the theory of mean field limit kernels and their RKHSs (Section 2). In Theorem 2.3, we precisely describe the relationship between the RKHS of the finite-input kernels and the RKHS of the mean field kernel, completing the results from [22]. In particular, this allows us to interpret the latter RKHS as the mean field limit of the former RKHSs. Furthermore, in Lemma 2.4 and 2.5, we provide inequalities for the corresponding RKHS norms, which are necessary for Γ -convergence arguments. 2) We provide results relevant for approximation with mean field limit kernels (Section 3). With Proposition 3.1, we give a first result on the approximation power of mean field limit kernels, and in Theorem 3.3 we can also provide a representer theorem for these kernels. For its proof, we use a Γ -convergence argument, which is to the best of our knowledge the first time this technique has been used in the context of kernel methods. 3) We investigate the mean field limit of kernels in the context of statistical learning theory (Section 4). We first establish an appropriate mean field limit setup for statistical learning problems, based on a slightly stronger mean field limit existence result than available so far, cf. Proposition 2.1. To the best of our knowledge, this is a new form of a limit for learning problems. In this setup, we then provide existence, uniqueness, and representer theorems for empirical and (using an apparently new notion of mean field convergence of probability distributions) infinite-sample solutions of SVMs, cf. Proposition 4.3 and 4.5. Finally, under a uniformity assumption, we can also establish convergence of the minimal risks in Proposition 4.7.

Our developments are relevant from two different perspectives: on the one hand, they constitute a theoretical proof-of-concept that the mean field limit can be 'pulled through' the (kernel-based) statistical learning theory setup. In particular, this demonstrates that rigorous theoretical results can be transferred through the mean field limit, similar to works in the context of control of interacting particle systems, see e.g. [23]. On the other hand, our setup appears to be a new variant of a largenumber-of-variables limit in the context of machine learning, complementing established settings like infinite-width neural networks [24].

Due to space constraints, all proofs and some additional technical results have been placed in the supplementary material.

2 Kernels and their RKHSs in the mean field limit

Setup and preliminaries Let ( X,d X ) be a compact metric space and denote by P ( X ) the set of Borel probability measures on X . We endow P ( X ) with the topology of weak convergence of probability measures. Recall that for µ n , µ ∈ P ( X ) , we say that µ n → µ weakly if for all bounded and continuous f : X → R (since X is compact, this is equivalent to f continuous) we have lim n →∞ ∫ X ϕ ( x )d µ n ( x ) → ∫ X ϕ ( x )d µ ( x ) . The topology of weak convergence can be metrized by the Kantorowich-Rubinstein metric d KR, defined by

Note that since X is compact and hence separable, the Kantorowich-Rubinstein metric is equal to the 1Wasserstein metric here. Furthermore, P ( X ) is compact in this topology. For M ∈ N + and ⃗ x ∈ X M , denote the i -th component of ⃗ x by x i , and define the empirical measure for ⃗ x by ˆ µ [ ⃗ x ] = 1 M ∑ M i =1 δ x i , where δ x denotes the Dirac measure centered at x ∈ X . The empirical measures are dense in P ( X ) w.r.t. the Kantorowich-Rubinstein metric. Additionally, define d 2 KR : P ( X ) 2 × P ( X ) 2 → R ≥ 0 by d 2 KR (( µ 1 , µ ′ 1 ) , ( µ 2 , µ ′ 2 )) = d KR ( µ 1 , µ 2 ) + d KR ( µ ′ 1 , µ ′ 2 ) , and note that ( P ( X ) 2 , d 2 KR ) is a compact metric space. Moreover, denote the set of permutations on { 1 , . . . , M } by S M , and for a tuple ⃗ x ∈ X M and permutation σ ∈ S M define σ⃗ x = ( x σ (1) , . . . , x σ ( M ) ) . Finally, we recall some well-known definitions and results from the theory of reproducing kernel Hilbert spaces, following [20, Chapter 4]. For an arbitrary set X ̸ = ∅ and a Hilbert space ( H, ⟨· , ·⟩ H ) of functions on X , we say that a map k : X × X → R is a reproducing kernel for H if 1) k ( · , x ) ∈ H for all x ∈ X ; 2) for all x ∈ X and f ∈ H we have f ( x ) = ⟨ f, k ( · , x ) ⟩ H . Note that if a reproducing kernel exists, it is unique. If such a Hilbert space has a reproducing kernel, we call H a reproducing kernel Hilbert space (RKHS) and k its (reproducing) kernel. It is well-known that a reproducing kernel is

symmetric and positive semidefinite, and that every symmetric and positive semidefinite function has a unique RKHS for which it is the reproducing kernel. For brevity, if k is symmetric and positive semidefinite, or equivalently, if it is the reproducing kernel of an RKHS, we call k simply a kernel, and denote by ( H k , ⟨· , ·⟩ k ) its unique associated RKHS. Define also H pre k = span { k ( · , x ) | x ∈ X} , then for f = ∑ N n =1 α n k ( · , x n ) ∈ H pre k and g = ∑ M m =1 β m k ( · , y m ) ∈ H pre k we have ⟨ f, g ⟩ k = ∑ N n =1 ∑ M m =1 α n β m k ( y m , x n ) , and H pre k is dense in H k .

The mean field limit of functions and kernels Given f M : X M → R , M ∈ N + , and f : P ( X ) → R , we say that f M converges in mean field to f and that f is the (or a) mean field limit of f M , if lim M →∞ sup ⃗ x ∈ X M | f M ( ⃗ x ) -f (ˆ µ [ ⃗ x ]) | = 0 . In this case, we write f M P 1 -→ f . Let now ( Y, d Y ) be another metric space and f M : X M × Y → R , M ∈ N + , and f : P ( X ) × Y → R , then we say that f M converges in mean field to f and that f is the (or a) mean field limit of f M , if for all compact K ⊆ Y we have

and also write f M P 1 -→ f . The following existence results for mean field limits is slightly more general than what is available in the literature, and it is essentially a direct generalization of [25, Theorem 2.1], in the form of [26, Lemma 1.2].

Proposition 2.1. Let ( X,d X ) be a compact metric space and ( Z, d Z ) a metric space that has a countable basis ( U n ) n such that ¯ U n is compact for all n ∈ N . Let f M : X M × Z → R , M ∈ N + , be a sequence of functions fulfilling the following conditions: 1) (Symmetry in ⃗ x ) 2 For all M ∈ N + , ⃗ x ∈ X M , z ∈ Z and permutations σ ∈ S M , we have f M ( σ⃗ x, z ) = f M ( ⃗ x, z ) ; 2) (Uniform boundedness) There exists B f ∈ R ≥ 0 and a function b : Z → R ≥ 0 such that ∀ M ∈ N + , ⃗ x ∈ X M , z ∈ z : | f M ( ⃗ x, z ) | ≤ B f + b ( z ) ; 3) (Uniform Lipschitz continuity) There exists some L f ∈ R > 0 such that for all M ∈ N + , ⃗ x 1 , ⃗ x 2 ∈ X M , z 1 , z 2 ∈ Z we have | f M ( ⃗ x 1 , z 1 ) -f M ( ⃗ x 2 , z 2 ) | ≤ L f ( d KR (ˆ µ [ ⃗ x 1 ] , ˆ µ [ ⃗ x 2 ]) + d Z ( z 1 , z 2 )) .

Then there exists a subsequence ( f M ℓ ) ℓ and a continuous function f : P ( X ) × Z → R such that f M ℓ P 1 -→ f for ℓ →∞ . Furthermore, f is L f -Lipschitz continuous and there exists B F ∈ R ≥ 0 such that for all µ ∈ P ( X ) , z ∈ Z we have | f ( µ, z ) | ≤ B F + b ( z ) .

We now turn to the mean field limit of kernels as introduced in [22]: Given k M : X M × X M → R and k : P ( X ) ×P ( X ) → R , we say that k M converges in mean field to k and that k is the (or a) mean field limit of k M , if

In this case we write k M P 1 -→ k .

For convenience, we recall [22, Theorem 2.1], which ensures the existence of a mean field limit of a sequence of kernels.

Proposition 2.2. Let k M : X M × X M → R be a sequence of kernels fulfilling the following conditions. 1) (Symmetry in ⃗ x ) For all M ∈ N + , ⃗ x, ⃗ x ′ ∈ X M and permutations σ ∈ S M we have k M ( σ⃗ x, ⃗ x ′ ) = k M ( ⃗ x, ⃗ x ′ ) ; 2) (Uniform boundedness) There exists C k ∈ R ≥ 0 such that ∀ M ∈ N + , ⃗ x, ⃗ x ′ ∈ X M : | k M ( ⃗ x, ⃗ x ′ ) | ≤ C k ; 3) (Uniform Lipschitz continuity) There exists some L k ∈ R > 0 such that for all M ∈ N + , ⃗ x 1 , ⃗ x ′ 1 , ⃗ x 2 , ⃗ x ′ 2 ∈ X M we have | k M ( ⃗ x 1 , ⃗ x ′ 1 ) -k M ( ⃗ x 2 , ⃗ x ′ 2 ) | ≤ L k d 2 KR [(ˆ µ [ ⃗ x 1 ] , ˆ µ [ ⃗ x ′ 1 ]) , (ˆ µ [ ⃗ x 2 ] , ˆ µ [ ⃗ x ′ 2 ])] .

Then there exists a subsequence ( k M ℓ ) ℓ and a continuous kernel k : P ( X ) ×P ( X ) → R such that k M ℓ P 1 -→ k , and k is also bounded by C k .

Let k M : X M × X M → R be a given sequence of kernels fulfilling the conditions of Proposition 2.2. Then there exists a subsequence ( k M ℓ ) ℓ converging in mean field to a kernel k : P ( X ) ×P ( X ) → R .

From now on, we only consider this subsequence and denote it again by ( k M ) M , i.e., k M P 1 -→ k . Unless noted otherwise, every time we need a further subsequence, we will make this explicit. 3

The RKHS of the mean field limit kernel Denote by H M := H k M the (unique) RKHS corresponding to kernel k M and denote by H k the unique RKHS of k . For basic properties of these objects as well as classes of suitable kernels we refer to [22].

We clarify the relation between H M and H k in the next result.

Theorem 2.3. 1) For every f ∈ H k , there exists a sequence f M ∈ H M , M ∈ N + , such that f M P 1 -→ f . 2) Let f M ∈ H M be sequence such that there exists B ∈ R ≥ 0 with ∥ f M ∥ M ≤ B for all M ∈ N + . Then there exists a subsequence ( f M ℓ ) ℓ and f ∈ H k with f M ℓ P 1 -→ f and ∥ f ∥ k ≤ B .

In other words, on the one hand, every RKHS function from H k arises as a mean field limit of RKHS functions from H M .On the other hand, every uniformly norm-bounded sequence of RKHS functions ( f M ) M has a mean field limit in H k .

Note that the preceding result is considerably stronger than the corresponding results in [22]: In contrast to [22, Theorem 4.4] we do not need to go to another subsequence in the first item, and we ensure that the mean field limit f is contained in H k (and norm-bounded by the same uniform bound), which was missing from Corollary 4.3 in the same reference.

The relation between the kernels k M and their RKHSs H M , and the mean field limit kernel k and its RKHS H k is illustrated as a commutative diagram in Figure 1. In order to arrive at the mean field RKHS H k , on the one hand, we consider the mean field limit k of the k M , and then form the corresponding RKHS H k . This is essentially the content of Proposition 2.2. On the other hand, we can first go from the kernel k M to the associated unique RKHS H M (for each M ∈ N + ). Theorem 2.3 then says that H k can be interpreted as a mean field limit of the RKHSs H M , since every function in H k arises as a mean field limit of a sequence of functions from the H M , and every uniformly norm-bounded sequence of such functions has a mean field limit that is in H k .

Next, we state two technical results that will play an important role in the following developments, and which might be of independent interest. They describe lim inf and lim sup inequalities required for Γ -convergence arguments used later on.

Lemma 2.5. Let f ∈ H k . Then there exist f M ∈ H M , M ∈ N + , such that lim M →∞ sup ⃗ x ∈ X M | f M ( ⃗ x ) -f (ˆ µ [ ⃗ x ]) | = 0 , and

3 Approximation with kernels in the mean field limit

Kernel-based machine learning methods use in general an RKHS as the hypothesis space, and learning often reduces to a search or optimization problem over this function space. For this reason, it is

important to investigate the approximation properties of a given kernel and its associated RKHS as well as to ensure that the learning problem over an RKHS (which is in general an infinite-dimensional object) can be tackled with finite computations.

The next result asserts that, under a uniformity condition, the approximation power of the finite-input kernels k M is inherited by the mean field limit kernel.

Proposition 3.1. For M ∈ N + , let F M be the set of symmetric functions that are continuous w.r.t. ( ⃗ x, ⃗ x ′ ) ↦→ d KR (ˆ µ [ ⃗ x ] , ˆ µ [ ⃗ x ′ ]) . Let F ⊆ C 0 ( P ( X ) , R ) such that for all f ∈ F and ϵ > 0 there exist B ∈ R ≥ 0 and sequences f M ∈ F M , ˆ f M ∈ H M , M ∈ N + , such that 1) f M P 1 -→ f 2) ∥ f M -ˆ f M ∥ ∞ ≤ ϵ for all M ∈ N + 3) ∥ ˆ f M ∥ M ≤ B for all M ∈ N + . Then for all f ∈ F and ϵ > 0 , there exists ˆ f ∈ H k with ∥ f -ˆ f ∥ ∞ ≤ ϵ .

Intuitively, the set F consists of all continuous functions on P ( X ) that arise as a mean field limit of functions which can be uniformly approximated by uniformly norm-bounded RKHS functions. The result then states (to use a somewhat imprecise terminology) that the RKHS H k is dense in F . We can interpret this as an appropriate mean field variant of the universality property of kernels: a kernel on a compact metric space is called universal if its associated RKHS is dense w.r.t. the supremum norm in the space of continuous functions, and many common kernels are universal, cf. e.g. [20, Section 4.6]. In our setting, ideally universality of the finite-input kernels k M is inherited by the mean field limit kernel k . However, since the mean field limit can be interpreted as a form of smoothing limit, some uniformity requirements should be expected. Proposition 3.1 provides exactly such a condition.

Remark 3.2. In Proposition 3.1, the set F is a subvectorspace of C 0 ( P ( X ) , R ) . Furthermore, if the P 1 -convergence in the definition of F is uniform, then F is closed.

Since k M and k are kernels, we have the usual representer theorem for their corresponding RKHSs, cf. e.g. [27]. A natural question is then whether we have mean field convergence of the minimizers and their representation. This is clarified by the next result.

Theorem 3.3. Let N ∈ N + , µ 1 , . . . , µ N ∈ P ( X ) and for n = 1 , . . . , N let ⃗ x [ M ] n ∈ X M , M ∈ N + , such that ˆ µ [ ⃗ x [ M ] n ] d KR -→ µ n for M →∞ . Let L : R N → R ≥ 0 be continuous and strictly convex and λ > 0 . For each M ∈ N + consider the problem

Then for each M ∈ N + problem (5) has a unique solution f ∗ M , which is of the form f ∗ M = ∑ N n =1 α [ M ] n k M ( · , ⃗ x [ M ] n ) ∈ H M , with α [ M ] 1 , . . . , α [ M ] N ∈ R , and problem (6) has a unique solution f ∗ , which is of the form f ∗ = ∑ N n =1 α n k ( · , µ n ) ∈ H k , with α 1 , . . . , α N ∈ R . Furthermore, there exists a subsequence ( f ∗ M ℓ ) ℓ such that f ∗ M ℓ P 1 -→ f ∗ and

The main point of this result is the convergence of the minimizers, which we will establish using a Γ -convergence argument. This approach seems to have been introduced by [28, 18, 29] originally in the context of multi-agent systems.

Remark 3.4. An inspection of the proof reveals that in Theorem 3.3 we can replace the term λ ∥ · ∥ M and λ ∥ · ∥ k by Ω( ∥ · ∥ M ) and Ω( ∥ · ∥ k ) , where Ω : R ≥ 0 → R ≥ 0 is a nonnegative, strictly increasing and continuous function.

4 Support Vector Machines with mean field limit kernels

We now turn to the mean field limit of kernels in the context of statistical learning theory, focusing on SVMs. We first briefly recall the standard setup of statistical learning theory, and formulate an appropriate mean field limit thereof. We then investigate empirical and infinite-sample solutions of SVMs and their mean field limits, as well as the convergence of the corresponding risks.

as well as the problem

Statistical learning theory setup We now introduce the standard setup of statistical learning theory, following mostly [20, Chapters 2 and 5]. Let X ̸ = ∅ (associated with some σ -algebra) and ∅ ̸ = Y ⊆ R closed (associated with the corresponding Borel σ -algebra). A loss function is in this setting a measurable function ℓ : X × Y × R → R ≥ 0 . Let P be a probability distribution on X × Y and f : X → R a measurable function, then the risk of f w.r.t. P and loss function ℓ is defined by

Note that this is always well-defined since ( x, y ) ↦→ ℓ ( x, y, f ( x )) is a measurable and nonnegative function. For a set H ⊆ R X of measurable functions we also define the minimal risk over H by

If H is a normed vector space, we additionally define the regularized risk of f ∈ H and the minimal regularized risk over H by

where λ ∈ R > 0 is the regularization parameter . A data set of size N ∈ N + is a tuple D N = (( x 1 , y 1 ) , . . . , ( x N , y N )) ∈ ( X × Y ) N and for a function f : X → R we define its empirical risk by

If H is a normed vector space and f ∈ H , we define additionally the regularized empirical risk and the minimal regularized empirical risk over H by

where λ ∈ R > 0 is again the regularization parameter. Note that the notation for the empirical risks is consistent with the risk w.r.t. a probability distribution P , if we identify a data set D N by the corresponding empirical distribution 1 N ∑ N n =1 δ ( x n ,y n ) .

In the following, H will be a RKHS and a minimizer (assuming existence and uniqueness) of R H ∗ ℓ,P,λ will be called an infinite-sample support vector machine (SVM) . Similarly, R H ∗ ℓ,D N ,λ will be called the empirical solution of the SVM w.r.t. the data set D N . Note that this is the common terminology in statistical learning theory, cf. [20], and corresponds to (empirical) risk minization with Tikhonov regularization.

Statistical learning theory setup in the mean field limit Let now ∅ ̸ = Y ⊆ R be compact and ℓ M : X M × Y × R → R ≥ 0 , M ∈ N , such that 1) ℓ M ( σ⃗ x, y, t ) = ℓ M ( ⃗ x, y, t ) for all ⃗ x ∈ X M , σ ∈ S M , y ∈ Y , t ∈ R ; 2) there exists C ℓ ∈ R ≥ 0 and a nondecreasing function b : R ≥ 0 → R ≥ 0 with | ℓ M ( ⃗ x, y, t ) | ≤ C ℓ + b ( | t | ) for all M ∈ N and ⃗ x ∈ X M , y ∈ Y, t ∈ R ; 3) there exists L ℓ ∈ R ≥ 0 with

for all ⃗ x 1 , x 2 ∈ X M , y 1 , y ′ 1 ∈ Y, t 1 , t 2 ∈ R . In particular, all ℓ M are measurable (assuming the Borel σ -algebra on X M ) and hence are loss functions on X M × Y . Proposition 2.1 ensures the existence of a subsequence ( ℓ M m ) m and an L ℓ -Lipschitz continuous function ℓ : P ( X ) × Y × R → R with

for all compact K ⊆ R , and we write again ℓ M m P 1 -→ ℓ . For readability, from now on we switch to this subsequence. Furthermore, we also get from Proposition 2.1 that there exists some C L ∈ R ≥ 0 such that | ℓ ( µ, y, t ) | ≤ C L + b ( | t | ) for all µ ∈ P ( X ) , y ∈ Y, t ∈ R .

Remark 4.1. Note that, for Proposition 2.1 to apply, it is enough to assume in item 2) above the existence of a function b : R → R ≥ 0 with | ℓ M ( ⃗ x, y, t ) | ≤ C ℓ + b ( | t | ) . However, we chose the slightly stronger condition that b is nondecreasing, since then ℓ M is a Nemitskii loss according to [20, Definition 2.16]. Since the function with constant value C ℓ is actually P M -integrable, this means that ℓ M is even a P M -integrable Nemitskii loss according to [20]. A similar remark then applies to ℓ .

Lemma 4.2. The function ℓ is nonnegative. Furthermore, if all ℓ M are convex loss functions [20, Definition 2.12], i.e., if for all M ∈ N + , ⃗ x ∈ X M , y ∈ Y, t 1 , t 2 ∈ R and λ ∈ (0 , 1) we have

then so is ℓ .

Empirical SVM solutions Given data sets D [ M ] N = ( ( ⃗ x [ M ] 1 , y [ M ] 1 ) , . . . , ( ⃗ x [ M ] N , y [ M ] N ) ) for all M ∈ N + with ⃗ x [ M ] n ∈ X M , y [ M ] n ∈ Y , and D N = (( µ 1 , y 1 ) , . . . , ( µ N , y N )) with µ n ∈ P ( X ) and y n ∈ Y , we write D [ M ] N P 1 -→ D N if ˆ µ [ ⃗ x [ M ] n ] d KR -→ µ n and y [ M ] n → y n (where M → ∞ ) for all n = 1 , . . . , N . We can interpret this as mean field convergence of the data sets.

Furthermore, consider the empirical risk of hypothesis f M ∈ H M (and f ∈ H k ) on data set D [ M ] N (and D N )

and the corresponding regularized risk

where λ ∈ R > 0 is the regularization parameter.

Proposition 4.3. Let λ > 0 , assume that all ℓ M are convex and let D [ M ] N , D N be finite data sets with D [ M ] N P 1 -→ D N . Then for all M ∈ N + , H M ∋ f M ↦→R ℓ M ,D [ M ] N ,λ ( f M ) has a unique minimizer f ∗ M,λ ∈ H M and H k ∋ f ↦→ R ℓ,D N ,λ ( f ) has a unique minimizer f ∗ λ ∈ H k . Furthermore, for all M ∈ N + there exist α [ M ] n ∈ R , n = 1 , . . . , N , such that f ∗ M,λ = ∑ N n =1 α [ M ] n k M ( · , ⃗ x [ M ] n ) , and there exist α 1 , . . . , α N ∈ R such that f ∗ λ = ∑ N n =1 α n k ( · , µ n ) . Finally, there exists a subsequence ( f ∗ M m ,λ ) m such that f ∗ M m ,λ P 1 -→ f ∗ λ and R ℓ Mm ,D [ Mm ] N ,λ ( f ∗ M m ,λ ) →R ℓ,D N ,λ ( f ∗ λ ) for m →∞ .

Convergence of distributions and infinite-sample SVMs in the mean field limit We now turn to the question of mean field limits of distributions and the associated learning problems and SVM solutions. Let ( P [ M ] ) M be a sequence of distributions, where P [ M ] is a probability distribution on X M × Y , and let P be a probability distribution on P ( X ) × Y . We say that P [ M ] converges in mean field to P and write P [ M ] P 1 -→ P , if for all continuous (w.r.t. the product topology on P ( X ) × Y ) and bounded 4 f we have

This convergence notion of probability distributions (on different input spaces) appears to be not standard, but it is a natural concept in the present context. Essentially, it is weak (also called narrow) convergence of probability distributions adapated to our setting.

Consider now data sets D [ M ] N , D N , with D [ M ] N P 1 -→ D N , then we also have convergence in mean field of the datasets, interpreted as empirical distributions: let f ∈ C 0 ( P ( X ) × Y, R ) be bounded, then

This shows that the mean field convergence of probability distributions as defined here is a direct generalization of the natural notion of mean field convergence of data sets.

Finally, consider the risk of hypothesis f M ∈ H M and f ∈ H k w.r.t. the distribution P [ M ] and P , respectively,

as well as the minimal risks

Our first result ensures that mean field convergence of distributions P [ M ] , loss functions ℓ M and data sets D [ M ] N ensures the convergence of the corresponding risks of the empirical SVM solutions.

Lemma 4.4. Consider the situation and notation of Proposition 4.3 and assume that P [ M ] P 1 -→ P . We then have R ℓ Mm ,P [ Mm ] ( f ∗ M m ,λ ) →R ℓ,P ( f ∗ λ ) for m →∞ .

Next, we investigate the mean field convergence of infinite-sample SVM solutions and their associated risks. Define for λ ∈ R ≥ 0 (and all M ∈ N + ) the regularized risk of f M ∈ H M and f ∈ H k , respectively, by

and the corresponding minimal risks by

Proposition 4.5. 5 Let λ > 0 , assume that all ℓ M are convex loss functions and let P [ M ] and P be probability distributions on X M × Y and P ( X ) × Y , respectively, with P [ M ] P 1 -→ P . Then for all M ∈ N + , H M ∋ f M ↦→ R ℓ M ,P [ M ] ,λ ( f M ) has a unique minimizer f ∗ M,λ ∈ H M and H k ∋ f ↦→R ℓ,P,λ ( f ) has a unique minimizer f ∗ λ ∈ H k . Furthermore, there exists a subsequence ( f ∗ M m ,λ ) m such that f ∗ M m ,λ P 1 -→ f ∗ λ and R ℓ Mm ,P [ Mm ] ,λ ( f ∗ M m ,λ ) → R ℓ,P,λ ( f ∗ λ ) for m → ∞ . In particular, R H Mm ∗ ℓ Mm ,P [ Mm ] ,λ →R H k ∗ ℓ,P,λ .

Finally, we would like to show that R H M ∗ ℓ M ,P [ M ] →R H k ∗ ℓ,P for P [ M ] P 1 -→ P . Up to a subsequence, this is established under Assumption 4.6. Define the approximation error functions , cf. [20, Definition 5.14], by

where M ∈ N + and λ ∈ R ≥ 0 . Note that (for all M ∈ N + ) A [ M ] 2 , A 2 : R ≥ 0 → R ≥ 0 are increasing, concave and continuous, and A [ M ] 2 , A 2 (0) = 0 , cf. [20, Lemma 5.15]. We need essentially equicontinuity of ( A [ M ] 2 ) M in 0, which is formalized in the following assumption.

Assumption 4.6. For all ϵ > 0 there exists λ ϵ > 0 such that for all 0 < λ ≤ λ ϵ and M ∈ N + we have A [ M ] 2 ( λ ) ≤ ϵ .

Proposition 4.7. Assume that all ℓ M are convex loss functions, let P [ M ] and P be probability distributions on X M × Y and P ( X ) × Y , respectively, with P [ M ] P 1 -→ P . If Assumption 4.6 holds, there exists a strictly increasing sequence ( M m ) m with R H Mm ∗ ℓ Mm ,P [ Mm ] →R H k ∗ ℓ,P for m →∞ .

5 Conclusion

We investigated the mean field limit of kernels and their RKHSs, as well as the mean field limit of statistical learning problems solved with SVMs. In particular, we managed to complete the basic theory of mean field kernels as started in [22]. Additionally, we investigated their approximation capabilities by providing a first approximation result and a variant of the representer theorem for mean field kernels. Finally, we introduced a corresponding mean field limit of statistical learning problems and provided convergence results for SVMs using mean field kernels. In contrast to other settings involving a large number of variables, for example, infinite-width neural networks, here we considered the case of an increasing number of inputs. This work opens many directions for future investigation. For example, it would be interesting to remove or weaken Assumption 4.6 for a result like Proposition 4.7. Another relevant direction is to find approximation results that are stronger than Proposition 3.1. Finally, it would be interesting to investigate whether statistical guarantees, like consistency or learning rates, for the finite-input learning problems can be transferred to the mean field level.

Acknowledgments and Disclosure of Funding

We would like to thank Noel Brindise, Pierre-François Massiani and Alexander von Rohr for very detailed and helpful comments on the manuscript, and the anonymous reviewers for their detailed and helpful comments. This work is funded in part under the Excellence Strategy of the Federal Government and the Länder (G:(DE-82)EXS-SF-SFDdM035), which the authors gratefully acknowledge. The authors further thank the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) for the financial support through 320021702/GRK2326, 333849990/IRTG-2379, B04, B05, and B06 of 442047500/SFB1481, HE5386/18-1,19-2,22-1,23-1,25-1.

References

Carlo Cercignani. Rarefied gas dynamics: from basic concepts to actual calculations , volume 21 of Cambridge Texts in Applied Mathematics . Cambridge university press, 2000.

Carlo Cercignani, Reinhard Illner, and Mario Pulvirenti. The mathematical theory of dilute gases , volume 106 of Applied Mathematical Sciences . Springer Science & Business Media, 1994.

Michele Ballerini, Nicola Cabibbo, Raphael Candelier, Andrea Cavagna, Evaristo Cisbani, Irene Giardina, Vivien Lecomte, Alberto Orlandi, Giorgio Parisi, Andrea Procaccini, et al. Interaction ruling animal collective behavior depends on topological rather than metric distance: Evidence from a field study. Proceedings of the national academy of sciences , 105(4):1232-1237, 2008.

Yael Katz, Kolbjørn Tunstrøm, Christos C Ioannou, Cristián Huepe, and Iain D Couzin. Inferring the structure and dynamics of interactions in schooling fish. Proceedings of the National Academy of Sciences , 108(46):18720-18725, 2011.

Giuseppe Toscani. Kinetic models of opinion formation. Communications in mathematical sciences , 4(3): 481-496, 2006.

Claudio Castellano, Santo Fortunato, and Vittorio Loreto. Statistical physics of social dynamics. Reviews of modern physics , 81(2):591, 2009.

John RG Dyer, Anders Johansson, Dirk Helbing, Iain D Couzin, and Jens Krause. Leadership, consensus decision making and collective behaviour in humans. Philosophical Transactions of the Royal Society B: Biological Sciences , 364(1518):781-789, 2009.

Emiliano Cristiani, Benedetto Piccoli, and Andrea Tosin. Multiscale modeling of pedestrian dynamics , volume 12 of Modeling, Simulation and Applications . Springer, 2014.

Giacomo Albi, Mattia Bongini, Emiliano Cristiani, and Dante Kalise. Invisible control of self-organizing agents leaving unknown environments. SIAM Journal on Applied Mathematics , 76(4):1683-1710, 2016.

Andrés A Peters, Richard H Middleton, and Oliver Mason. Leader tracking in homogeneous vehicle platoons with broadcast delays. Automatica , 50(1):64-74, 2014.

Kwang-Kyo Oh, Myoung-Chul Park, and Hyo-Sung Ahn. A survey of multi-agent formation control. Automatica , 53:424-440, 2015.

Young-Pil Choi, Dante Kalise, Jan Peszek, and Andrés A Peters. A collisionless singular Cucker-Smale model with decentralized formation control. SIAM Journal on Applied Dynamical Systems , 18(4):19541981, 2019.

Andrea Tosin and Mattia Zanella. Kinetic-controlled hydrodynamics for traffic models with driver-assist vehicles. Multiscale Modeling & Simulation , 17(2):716-749, 2019.

Giovanni Naldi, Lorenzo Pareschi, and Giuseppe Toscani. Mathematical modeling of collective behavior in socio-economic and life sciences . Springer Science & Business Media, 2010.

Tamás Vicsek and Anna Zafeiris. Collective motion. Physics reports , 517(3-4):71-140, 2012.

Xiaoqian Gong, Michael Herty, Benedetto Piccoli, and Giuseppe Visconti. Crowd dynamics: Modeling and control of multiagent systems. Annual Review of Control, Robotics, and Autonomous Systems , 6, 2022.

José A Carrillo, Massimo Fornasier, Giuseppe Toscani, and Francesco Vecil. Particle, kinetic, and hydrodynamic models of swarming. Mathematical modeling of collective behavior in socio-economic and life sciences , pages 297-336, 2010.

Mattia Bongini, Massimo Fornasier, Markus Hansen, and Mauro Maggioni. Inferring interaction rules from observations of evolutive systems i: The variational approach. Mathematical Models and Methods in Applied Sciences , 27(05):909-951, 2017.

Fei Lu, Ming Zhong, Sui Tang, and Mauro Maggioni. Nonparametric inference of interaction laws in systems of agents from trajectory data. Proceedings of the National Academy of Sciences , 116(29): 14424-14433, 2019.

Ingo Steinwart and Andreas Christmann. Support vector machines . Springer Science & Business Media, 2008.

Andreas Christmann and Ingo Steinwart. Universal kernels on non-standard input spaces. Advances in neural information processing systems , 23, 2010.

Christian Fiedler, Michael Herty, Michael Rom, Chiara Segala, and Sebastian Trimpe. Reproducing kernel H|ilbert spaces in the mean field limit. Kinetic and Related Models , 16(6):850-870, 2023. ISSN 1937-5093.

Michael Herty and Mattia Zanella. Performance bounds for the mean-field limit of constrained dynamics. Discrete & Continuous Dynamical Systems , 37(4):2023, 2017.

Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in neural information processing systems , 32, 2019.

Pierre Cardaliaguet. Notes on mean field games. Technical report, 2010.

René Carmona and François Delarue. Probabilistic theory of mean field games with applications I-II . Springer, 2018.

Bernhard Schölkopf, Ralf Herbrich, and Alex J Smola. A generalized representer theorem. In International conference on computational learning theory , pages 416-426. Springer, 2001.

Massimo Fornasier and Francesco Solombrino. Mean-field optimal control. ESAIM: Control, Optimisation and Calculus of Variations , 20(4):1123-1152, 2014.

Massimo Fornasier, Stefano Lisini, Carlo Orrieri, and Giuseppe Savaré. Mean-field optimal control as gamma-limit of finite agent controls. European Journal of Applied Mathematics , 30(6):1153-1186, 2019.

Serge Lang. Real and functional analysis , volume 142. Springer Science & Business Media, 2012.

Marc Atteia. Hilbertian kernels and spline functions . Elsevier, 1992.

Gianni Dal Maso. An introduction to Γ -convergence , volume 8 of Progress in Nonlinear Differential Equations and Their Applications . Springer Science & Business Media, 2012.

Mattia Bongini. Sparse optimal control of multiagent systems . PhD thesis, Technische Universität München, 2016.

Supplementary Material

A Proofs

In this section of the supplementary material, we provide detailed proofs for all results in the main text.

A.1 Proofs for Section 2

We start with Proposition 2.1, whose proof is based on [26, Lemma 1.2].

Proof. of Proposition 2.1 For M ∈ N + define the McShane extension F M : P ( X ) × Z → R by

Observe that F M is well-defined (i.e., R -valued) since f M ( · , z ) and L f d KR (ˆ µ [ · ] , µ ) are bounded for every z ∈ Z (since f M and d KR (ˆ µ [ · ] , µ ) are continuous and P ( X ) is compact, hence bounded).

Step 1 F M extends f M , i.e., for all M ∈ N + , ⃗ x ∈ X M and z ∈ Z we have F M (ˆ µ [ ⃗ x ] , z ) = f M ( ⃗ x, z ) . To show this, let ⃗ x ∈ X M and z ∈ Z be arbitrary and observe that by definition

If F M (ˆ µ [ ⃗ x ] , z ) < f M ( ⃗ x, z ) , then there exists some ⃗ x ′ ∈ X M such that

but this means that

contradicting the L f -Lipschitz continuity of f M .

Step 2 All F M are L f -continuous: Let M ∈ N + , µ i ∈ P ( X ) and z i ∈ Z , i = 1 , 2 , be arbitrary. Since X M is compact and f M ( · , z ) and L f d KR (ˆ µ [ · ] , µ i ) , i = 1 , 2 , are continuous, the infimum in the definition of F M is actually attained. Let ⃗ x 2 ∈ X M such that F M ( µ 2 , z 2 ) = f M ( ⃗ x 2 , z 2 ) + L f d KR (ˆ µ [ ⃗ x 2 ] , µ 2 ) , then we have

where we used the definition of F M in the first inequality, the Lipschitz continuity of f M (w.r.t. the second argument) for the second inequality, and then the fact that ⃗ x 2 attains the infimum in the definition of F M ( µ 2 , z 2 ) and the triangle inequality for d KR. Interchanging the roles of µ 1 , z 1 and µ 2 , z 2 then establishes the claim.

Step 3 There exists B F ∈ R ≥ 0 such that for all M ∈ N + , µ ∈ P ( X ) and z ∈ Z we have | F M ( µ, z ) | ≤ B F + h ( z ) : Let D P ( X ) be the diameter of P ( X ) (which is finite since P ( X ) is compact), then for all M ∈ N + and ⃗ x ∈ X M , z ∈ Z , µ ∈ P ( X ) we have

therefore | F M ( µ, z ) | ≤ B f + L f D P ( X ) + b ( z ) , showing the claim with B F = B f + L f D P ( X ) .

Step 4 Summarizing, ( F M ) M is a sequence of L f -Lipschitz continuous and hence equicontinuous functions such that for all µ ∈ P ( X ) and z ∈ Z , the set { F M ( µ, z ) | M ∈ N + } is relatively compact (since it is a bounded subset of R ). We can now use a variant of the Arzela-Ascoli theorem, cf. [30,

Corollary III.3.3]. From the assumption on Z , we can find a sequence ( V n ) n of open subsets of Z such that all ¯ V n are compact, ¯ V n ⊆ V n +1 and we have ⋃ n V n = Z . Then ( F M | ¯ V n ) M is a sequence of functions that fulfills the conditions of the Arzela-Ascoli theorem (since P ( X ) × K n so there exists a subsequence ( F ( n ) | ¯ V ) ℓ that converges uniformly to a continuous function on

P ( X ) × V n . Denote the diagonal subsequence of all these subsequences by ( F M ℓ ) ℓ , then there exists a continuous f : P ( X ) × Z → R such that ( F M ℓ ) ℓ converges uniformly on compact subsets to f . Since P ( X ) is compact, this means that for all compact K ⊆ Z

is compact), M ℓ n ¯

This also implies that for all µ ∈ P ( X ) and z ∈ Z we have | f ( µ, z ) | ≤ B F + b ( z ) .

Furthermore, f is also L f -Lipschitz continuous: Let µ i ∈ P ( X ) , z i ∈ Z , i = 1 , 2 , and ϵ > 0 be arbitrary. Let K ⊆ Z be compact with z 1 , z 2 ∈ K and choose ℓ ∈ N + such that

We then have

and since ϵ > 0 was arbitrary, the claim follows.

Step 5 For ℓ ∈ N + and ⃗ x ∈ X M ℓ , z ∈ Z we have

since F M ℓ extends f M ℓ , and hence

Next, we provide the proofs for the Γ -lim inf and Γ -lim sup results.

Proof. of Lemma 2.4 Assume the statement is not true, i.e., ∥ f ∥ k > lim inf M →∞ ∥ f M ∥ M . This means that there exists a subsequence M ℓ and C ∈ R ≥ 0 such that ∥ f ∥ k > lim ℓ ∥ f M ℓ ∥ M ℓ = C . Note that this implies that ∥ f ∥ k > 0 .

Let ϵ 1 , ϵ 2 > 0 and α > 1 , β ∈ (0 , 1) be arbitrary. From Theorem B.1, there exists ( ⃗ µ, ⃗ α ) ∈ P ( X ) N × R N such that

and w.l.o.g. we can assume that ϵ 1 > 0 is small enough so that D ( ⃗ µ, ⃗ α, f, k ) > 0 . The latter implies that E ( ⃗ µ, ⃗ α, f ) , W ( ⃗ µ, ⃗ α, k ) > 0 , so defining

we get ϵ α , ϵ β > 0 . For each n = 1 , . . . , N , choose ⃗ x [ M ] n ∈ X M such that ⃗ x [ M ] n d KR -→ µ n for M →∞ . Choose now L 1 ∈ N such that for all ℓ ≥ L 1 we get

(cf. also the proof of Theorem 2.3) and W ( ⃗ X [ M ℓ ] , ⃗ α, k [ M ℓ ] ) > 0 . We then get

so altogether a contradiction.

Proof. of Lemma 2.5 Let f ∈ H k be arbitrary and choose ( ϵ n ) n ⊆ R > 0 with ϵ n ↘ 0 . Step 1 For each n ∈ N choose

where α ( n ) 1 , . . . , α ( n ) L n ∈ R and µ ( n ) 1 , . . . , µ ( n ) L n ∈ P ( X ) , with

and ∥ f pre n ∥ k ≤ ∥ f ∥ k . To see that such a sequence of functions exists, choose some sequence ( ¯ f n ) n ∈ H pre k with ¯ f n = ∑ ¯ L n ℓ =1 ¯ α ( n ) ℓ k ( · , ¯ µ ( n ) ℓ ) , where ¯ α ( n ) ℓ ∈ R , ¯ µ ( n ) ℓ ∈ P ( X ) , with ¯ f n ∥·∥ k -→ f (exists since H pre k is dense in H k ). Define now for n ∈ N

and ˆ f n = P ¯ H n f , where P ¯ H n is the orthogonal projection onto ¯ H n . Then ¯ H n ⊆ H pre k , ∥ ˆ f n ∥ k = ∥ P ¯ H n f ∥ k ≤ ∥ f ∥ k and ∥ f -ˆ f n ∥ k ≤ ∥ f -¯ f n ∥ k → 0 (since ˆ f n = P ¯ H n f is the orthogonal projection of f onto ¯ H n and ¯ f n ∈ ¯ H n ), hence ˆ f n ∥·∥ k -→ f . We can now choose ( f pre n ) n as a subsequence of ( ˆ f n ) n .

Next, for all n ∈ N and ℓ = 1 , . . . , L n choose ⃗ x ( n,ℓ ) M ∈ X M with ˆ µ [ ⃗ x ( n,ℓ ) M ] d KR -→ µ ( n ) ℓ for M →∞ . Furthermore, for all n ∈ N choose M n ∈ N such that for all M ≥ M n and ℓ = 1 , . . . , L n we have

and

Using Theorem B.1 again leads to

Finally, let L 2 such that for all ℓ ≥ L 2 we have ∥ f M ℓ ∥ M ℓ ≤ C + ϵ 2 . For ℓ ≥ L 1 , L 2 we then get

Since ϵ 1 , ϵ 2 > 0 and α > 1 , β ∈ (0 , 1) were arbitrary, this implies that

W.l.o.g. we can assume that ( M n ) n is strictly increasing. For M ∈ N , let n ( M ) be the largest integer such that M n ( M ) ≤ M and define

Step 2 We now show that f M P 1 -→ f . For this, let ϵ > 0 be arbitrary and n ϵ ∈ N such that ϵ n ≤ ϵ . Let now M ≥ M n ϵ (note that this implies that n ( M ) ≥ n ϵ and hence ϵ n ( M ) ≤ ϵ n ) and ⃗ x ∈ X M , then we have

We continue with

where we first used the reproducing property of k , then Cauchy-Schwarz, again the reproducing property of k , and finally the choice f n ( M ) and the boundedness of k . Next,

where we used the triangle inequality, the Lipschitz continuity of k , and then the choice of the sequence ( M n ) n .

Finally,

where the triangle inequality has been used in the first step and then again the choice of the sequence ( M n ) n .

Altogether, establishing f M P 1 -→ f .

Step 3 We now show lim sup M →∞ ∥ f M ∥ M ≤ ∥ f ∥ k . Let ϵ > 0 be arbitrary and n ϵ ∈ N such that ϵ n ≤ ϵ and let M ≥ M n ϵ . We have

with remainder terms

We now bound these terms, so that

and

Altogether,

so ∥ f M ∥ M ≤ ∥ f ∥ k + ϵ for all M ≥ M n ϵ , and since ϵ > 0 was arbitrary, we finally get lim sup M →∞ ∥ f M ∥ M ≤ ∥ f ∥ k .

Finally, we can now provide the proof for the central Theorem 2.3.

Proof. of Theorem 2.3 The first statement is part of Lemma 2.5. Let us turn to the second statement: The existence of the subsequence ( f M ℓ ) ℓ and the continuous function f : P ( X ) → R with f M ℓ P 1 -→ f was shown in [22, Corollary 4.3], so we only have to ensure that f ∈ H k with ∥ f ∥ k ≤ B . For this, we use the characterization of RKHS functions from Theorem B.1. In particular, we will utilize the notation introduced there.

Assume that W ( ⃗ µ, ⃗ α, k ) = 0 . If B = 0 , then f M ≡ 0 and f M ℓ P 1 -→ f implies that f ≡ 0 , so the claim is clear in this case. Assume now B > 0 , let ϵ > 0 be arbitary and for n = 1 , . . . , N , choose sequences ⃗ x [ M ] n ∈ X M such that ⃗ x [ M ] n d KR -→ µ n for M → ∞ . For convenience, define ⃗ X [ M ] = ( ⃗ x [ M ] 1 · · · ⃗ x [ M ] N ) . Choose now ℓ ϵ ∈ N such that for all M ≥ M ℓ ϵ we get W ( ⃗ X [ M ] , ⃗ α, k M ) ≤ ϵ/B . This is possible since k M P 1 -→ k together with the continuity of k M and k as well as ⃗ x [ M ] n d KR -→ µ n for M →∞ and all n = 1 , . . . , N implies that W ( ⃗ X [ M ] , ⃗ α, k M ) →W ( ⃗ µ, ⃗ α, k ) = 0 . Let now ℓ ≥ ℓ ϵ be arbitrary and observe that f M ∈ H M implies N ( f M , k M ) < ∞ according to Theorem B.1, so in particular D ( ⃗ X [ M ℓ ] , ⃗ α, f M ℓ , k M ℓ ) < ∞ .

If W ( ⃗ X [ M ℓ ] , ⃗ α, k M ℓ ) = 0 , then we get that E ( ⃗ X [ M ℓ ] , ⃗ α, f M ℓ ) = 0 ≤ ϵ since D ( ⃗ X [ M ℓ ] , ⃗ α, f M ℓ , k M ℓ ) < ∞ , which implies by definition that E ( ⃗ X [ M ℓ ] , ⃗ α, f M ℓ ) = 0 .

If W ( ⃗ X [ M ℓ ] , ⃗ α, k M ℓ ) > 0 , then we have

which implies

Since f M ℓ P 1 -→ f together with the continuity of f M and f as well as ⃗ x [ M ] n d KR -→ µ n implies that E ( ⃗ X [ M ℓ ] , ⃗ α, f M ℓ ) →E ( ⃗ µ, ⃗ α, f ) , we get that E ( ⃗ µ, ⃗ α, f ) ≤ ϵ , and since ϵ > 0 was arbitrary we arrive at E ( ⃗ µ, ⃗ α, f ) ≤ 0 .

Assume now that E ( ⃗ µ, ⃗ α, f ) < 0 . This implies that there exist δ > 0 and ℓ δ ∈ N such that for all ℓ ≥ ℓ δ we have E ( ⃗ X [ M ℓ ] , ⃗ α, f M ℓ ) ≤ -δ < 0 , since E ( ⃗ X [ M ℓ ] , ⃗ α, f M ℓ ) →E ( ⃗ µ, ⃗ α, f ) . Let ℓ ≥ ℓ δ , then we get that E ( ⃗ X [ M ℓ ] , -⃗ α, f M ℓ ) ≥ δ > 0 and we have W ( ⃗ X [ M ℓ ] , -⃗ α, k M ℓ ) = W ( ⃗ X [ M ℓ ] , ⃗ α, k M ℓ ) > 0 . We can then continue with

which implies that W ( ⃗ X [ M ℓ ] , -⃗ α, k M ℓ ) = W ( ⃗ X [ M ℓ ] , ⃗ α, k M ℓ ) ≥ δ/B . But since W ( ⃗ X [ M ℓ ] , ⃗ α, k M ℓ ) →W ( ⃗ µ, ⃗ α, k ) , this implies that W ( ⃗ µ, ⃗ α, k ) ≥ δ/B > 0 , a contradiction. Altogether, E ( ⃗ µ, ⃗ α, f ) = 0 .

Step 2 Let ( ⃗ µ, ⃗ α ) ∈ P ( X ) N × R N . If W ( ⃗ µ, ⃗ α, k ) > 0 and E ( ⃗ µ, ⃗ α, f ) > 0 , then

To show this, let α > 1 and β ∈ (0 , 1) be arbitrary. Define

and observe that ϵ α , ϵ β > 0 . Furthermore, for all n = 1 , . . . , N choose a sequence ⃗ x [ M ] n ∈ X M such that ⃗ x [ M ] n d KR -→ µ n for M →∞ , and define ⃗ X [ M ] = ( ⃗ x [ M ] 1 · · · ⃗ x [ M ] N ) . Choose ℓ ϵ ∈ N + such that for all ℓ ≥ ℓ ϵ we have

and W ( ⃗ X [ M ℓ ] , ⃗ α, k M ℓ ) > 0 . Such an ℓ ϵ exists because k M P 1 -→ k together with the continuity of k M and k as well as the convergence of ⃗ x [ M ] n to µ n imply that W ( ⃗ X [ M ℓ ] , ⃗ α, k M ℓ ) →W ( ⃗ µ, ⃗ α, k ) , and f M ℓ P 1 -→ f together with the continuity of f M and f imply that E ( ⃗ X [ M ℓ ] , ⃗ α, f M ℓ ) →E ( ⃗ µ, ⃗ α, f ) .

Let now ℓ ≥ ℓ ϵ be arbitrary. By definition of ϵ α we get αϵ α ≤ ( α -1) E ( ⃗ µ, ⃗ α, f ) , which in turn leads to

where we used in the last inequality that α -1 > 0 and by choice of ℓ ϵ we have E ( ⃗ µ, ⃗ α, f ) ≤ E ( ⃗ X [ M ℓ ] , ⃗ α, f M ℓ ) + ϵ α . We can then continue with

Next, by definition of ϵ β and choice of ℓ ϵ we find that

hence

Combining these results, we get that for all ℓ ≥ ℓ ϵ

Since α > 1 and β ∈ (0 , 1) were arbitrary, this shows that

Step 3 Let ( ⃗ µ, ⃗ α ) ∈ P ( X ) N × R N be arbitrary. If W ( ⃗ µ, ⃗ α, k ) = 0 , then we get from Step 1 that E ( ⃗ µ, ⃗ α, f ) = 0 ≤ B . Assume now W ( ⃗ µ, ⃗ α, k ) > 0 . If E ( ⃗ µ, ⃗ α, f ) = 0 , then again E ( ⃗ µ, ⃗ α, f ) = 0 ≤ B . If E ( ⃗ µ, ⃗ α, f ) > 0 , then Step 2 ensures that

Finally, if E ( ⃗ µ, ⃗ α, f ) < 0 , then again

Altogether, we get that D ( ⃗ µ, ⃗ α, f, k ) ≤ B . Since ( ⃗ µ, ⃗ α ) was arbitrary, maximization leads to N ( f, k ) ≤ B < ∞ , hence f ∈ H k and ∥ f ∥ k = N ( f, k ) ≤ B .

A.2 Proofs for Section 3

In this section we provide the proofs for the results relating to approximation with kernels in the mean field limit.

Proof. of Proposition 3.1 Let f ∈ F and ϵ > 0 be arbitrary. Let B ∈ R ≥ 0 and f M ∈ F M , ˆ f M ∈ H M , M ∈ N + , such that f M P 1 -→ f , ∥ f M -ˆ f M ∥ ≤ ϵ 5 and ∥ ˆ f M ∥ M ≤ B for all M ∈ N + (exist by definition of F ). Theorem 2.3 ensures that there exists a subsequence ( f M ℓ ) ℓ and ˆ f ∈ H k with ∥ ˆ f ∥ k ≤ B such that ˆ f M ℓ P 1 -→ ˆ f for ℓ →∞ . Choose now L 1 ∈ N + such that for all ℓ ≥ L 1 we have

Let now µ ∈ P ( X ) be arbitrary and choose a sequence ⃗ x M ∈ X M with ˆ µ [ ⃗ x M ] d KR -→ µ . Finally, let L 2 ∈ N + such that for all ℓ ≥ L 2 we have

(such an L 2 exists due to the continuity of f and ˆ f ).

We now have for ℓ ≥ max { L 1 , L 2 } that

Since µ was arbitrary, the result follows.

Proof. of Remark 3.2 We first show that F is a subvectorspace. Let f, g ∈ F and λ ∈ R , ϵ > 0 be arbitrary. W.l.o.g. we can assume λ = 0 . Choose sequences f M , g M ∈ F M , ˆ f M , ˆ g M ∈ H M , M ∈ N + , and constants B f , B g ∈ R ≥ 0 from the definition of F for f , ϵ 2 | λ | , and g , ϵ 2 , respectively.

Let M ∈ N + , ⃗ x ∈ X M be arbitrary, then

together with

f

P

1

P

1

M

-→

f

,

g

M

-→

g

Next, we have for all M ∈ N + that

Finally,

.

establishing that ( λ ˆ f M +ˆ g M ) M is uniformly norm-bounded. Altogether, we have that λf + g ∈ F We now turn to the second claim. Let ( f ( n ) ) n ⊆ F such that f ( n ) → f for some f ∈ C 0 ( P ( X ) , R ) and for all ¯ ϵ > 0 there exist f ( n ) M ∈ F M , ˆ f ( n ) M ∈ H M , ( ρ M ) M ⊆ R ≥ 0 and B ( n ) ∈ R ≥ 0 with ρ M ↘ 0 , ∥ f ( n ) M -ˆ f ( n ) M ∥ ∞ ≤ ¯ ϵ and ∥ ˆ f ( n ) M ∥ M ≤ B ( n ) for all n, M ∈ N + , and

shows that

λf

+

g

P

1

-→

λf

+

g

.

̸

M

M

for all n, M ∈ N + . We now show that f ∈ F . For this, let ϵ > 0 be arbitrary and choose f ( n ) M ∈ F M , ˆ f ( n ) M ∈ H M , ( ρ M ) M ⊆ R ≥ 0 and B ( n ) ∈ R ≥ 0 as above with ¯ ϵ = ϵ 4 . Let N ∈ N + be such that ∥ f ( m ) -f ( n ) ∥ ∞ ≤ ϵ 4 for all m,n ≥ N (such an N exists since ( f ( n ) ) n converges in C 0 ( P ( X ) , R ) and hence is a Cauchy sequence). Furthermore, let M ρ ∈ N + be such that for all M ≥ M ρ we have ρ M ≤ ϵ 4 . Define now f M = f ( M ) M and ˆ f M = ˆ f ( M ) M for M = 1 , . . . , M ρ -1 , and f M = f ( M + N ) M , ˆ f M = ˆ f ( N ) M for M ≥ M ρ .

Step 1 Let M ≥ M ρ and ⃗ x ∈ X M be arbitrary. We have

and since the right hand side (which is independent of ⃗ x ) converges to 0 for M → ∞ , we get f M P 1 -→ f .

Step 2 For M = 1 , . . . , M ρ we get

Let now M ≥ M ρ and ⃗ x ∈ X M be arbitrary. We have

and since ⃗ x ∈ X M was arbitrary, we get ∥ f M -ˆ f M ∥ ∞ ≤ ϵ .

Step 3 For M = 1 , . . . , M ρ -1 we get by construction that ∥ ˆ f M ∥ M = ∥ ˆ f ( M ) M ∥ M ≤ B ( M ) , and for M ≥ M ρ we find ∥ ˆ f M ∥ M = ∥ ˆ f ( N ) M ∥ M ≤ B ( N ) . Altogether, we get for M ∈ N + that

Combining the three steps establishes that f ∈ F .

Finally, here is the proof of the represnter theorem in the mean field limit.

Proof. of Theorem 3.3 The existence and uniqueness of f M and f follows from the well-known representer theorem (applied to all k M and k ).

We now turn to the convergence of the minimizers. For all M ∈ N + we have

and let f M ∈ H M with f M P 1 -→ f for some f ∈ H k . The continuity of f M , f and L as well as ⃗ x [ M ] n d KR -→ µ n for M → ∞ and all n = 1 , . . . , N , imply then that

lim M →∞ L ( f M ( ⃗ x [ M ] 1 ) , . . . , f M ( ⃗ x [ M ] N )) = L ( f ( µ 1 ) , . . . , f ( µ N )) . Combining this with Lemma 2.4 leads to L ( f ) ≤ lim inf M →∞ L M ( f ) .

Let now f ∈ H k be arbitrary and let f M ∈ H M be the sequence from Lemma 2.5. Using the same arguments as above we find that

We have shown that L M Γ -→ L and hence Proposition B.3 ensures that there exists a subsequence ( f ∗ M ℓ ) ℓ such that f ∗ M ℓ P 1 -→ f ∗ and L M ℓ ( f ∗ M ℓ ) →L ( f ∗ ) .

A.3 Proofs for Section 4

Proof. of Lemma 4.2 That ℓ is nonnegative is clear from the proof of Proposition 2.1. Let now all ℓ M be convex and let µ ∈ P ( X ) , y ∈ Y, t 1 , t 2 ∈ R and λ ∈ (0 , 1) be arbitrary, and define I = [min { t 1 , t 2 } , max { t 1 , t 2 } ] . Furthermore, let ⃗ x M ∈ X M with ⃗ x M d KR -→ µ for M → ∞ and ϵ > 0 be arbitrary. Choose now M so large that

This is possible due to the continuity of ℓ , as well as ℓ M P 1 -→ ℓ . We then have

and since ϵ > 0 was arbitrary, this establishes

ℓ

(

µ, y, λt i.e., convexity of ℓ .

Proof. of Proposition 4.3 From Lemma 4.2 we get that ℓ is nonnegative and convex. The existence, uniqueness and the representation formulas follow then from the standard representer theorem, cf. e.g., [20, Theorem 5.5].

Furthermore, for all M ∈ N + we have

Let f ∈ H k and ( f M ) M , f M ∈ H M , such that f M P 1 -→ f . From D [ M ] N P 1 -→ D N and the continuity of ℓ M , ℓ , together with ℓ M P 1 -→ ℓ and the boundedness of { y [ M ] n | M ∈ N + , n = 1 , . . . , N } ⊆ Y and { f M ( ⃗ x [ M ] n ) | M ∈ N + , N = 1 , . . . , N } we find that

1

+(1

-

λ

)

t

2

)

λℓ

(

µ, y, t

1

) + (1

-

λ

)

ℓ

(

µ, y, t

2

)

,

Combining this with Lemma 2.4 and Lemma 2.5 then establishes that R ℓ M ,D [ M ] N ,λ Γ -→ R ℓ,D N ,λ and the remaining claims follow from Proposition B.3 and the uniqueness of the minimizers.

Proof. of Lemma 4.4 Let ϵ > 0 be arbitrary. Recall from the proof of Proposition 4.3 that for all M ∈ N + we have ∥ f ∗ M,λ ∥ M ≤ √ NC ℓ λ , and hence for all ⃗ x ∈ X M we have

A similar argument applies to f ∗ λ ∈ H k , so we can find a compact set K ⊆ R with

Choose now m ϵ ∈ N + such that for all m ≥ m ϵ we have

Such a m ϵ exists since f ∗ M m ,λ P 1 -→ f ∗ λ and all ℓ M m are uniformly Lipschitz continuous (first inequality), ℓ M m P 1 -→ ℓ and Y and K are compact (second inequality), and P [ M ] P 1 -→ P as well as that ( µ, y ) ↦→ ℓ ( µ, y, f ∗ λ ( µ )) is continuous and bounded (third inequality). We now have

and since ϵ > 0 was arbitrary, the claim follows.

Proof. of Proposition 4.5 Observe that all k M are bounded measurable kernels, R ℓ M ,P [ M ] ( f M ) < ∞ for all f ∈ H M , ℓ M is a convex, P [ M ] -integrable Nemitskii loss (cf. Remark 4.1) and hence [20, Lemma 5.1, Theorem 5.2] guarantee the existence and uniqueness of f ∗ M,λ . A completely analogous argument shows the existence and uniqueness of f ∗ λ .

We now show that R ℓ M ,P [ M ] ,λ Γ -→ R ℓ,P,λ . For the Γ -lim inf -inequality, let f M ∈ H M , f ∈ H k be arbitrary with f M P 1 -→ f , and let ϵ > 0 . Choose M ϵ ∈ N + so large that for all M ≥ M ϵ

(this is possible since ( µ, y ) ↦→ ℓ ( µ, y, f ( µ )) is bounded and continuous and P [ M ] P 1 -→ P ) and

for all ⃗ x ∈ X M , y ∈ Y (this is possible due to the same argument used in the proof of Lemma 4.4). For M ≥ M ϵ we then find

where we used Lemma 2.4 in the last inequality.

For the Γ -lim sup -inequality, let f ∈ H k be arbitrary and let ( f M ) M be the recovery sequence from Lemma 2.5. The desired inequality then follows by repeating the arguments from above.

Finally, using exactly the same argument as in the proof of Proposition 4.3 shows that ∥ f ∗ M,λ ∥ M ≤ √ NC ℓ λ , so we can apply Proposition B.3 and the result follows.

Proof. of Proposition 4.7 Let ( ϵ n ) n ⊆ R > 0 with ϵ m ↘ 0 . We construct a strictly increasing sequence ( M n ) n such that

for all n ∈ N + .

We start with n = 1 : Since A 2 (0) = 0 and A 2 is continuous in 0, cf. [20, Lemma 5.15], there exists λ ′ 1 ∈ R > 0 such that A 2 ( λ ) ≤ ϵ 1 3 for all 0 < λ ≤ λ ′ 1 . From Assumption 4.6 we get λ ′′ 1 ∈ R > 0 such that for all M ∈ N + we have A [ M ] 2 ( λ ) ≤ ϵ 1 3 for all 0 < λ ≤ λ ′′ 1 . Define now λ 1 = min { λ ′ 1 , λ ′′ 1 } , and observe that λ 1 > 0 . Proposition 4.5 ensures the existence of a strictly increasing sequence ( M (1) m ) m ⊆ N + with

for m →∞ . Choose m 1 ∈ N + such that for all m ≥ m 1 we have

We now set M 1 = M (1) m 1 and get that

We can now repeat the argument from above inductively: Suppose we have constructed our subsequence up to n ∈ N + , i.e., M 1 , . . . , M n . Choose λ ′ ∈ R > 0 such that A 2 ( λ ) ≤ ϵ n +1 3 for

all 0 < λ ≤ λ ′ (exists due to continuity), and λ ′′ ∈ R > 0 such that for all M ∈ N + we have A [ M ] 2 ( λ ) ≤ ϵ n +1 3 for all 0 < λ ≤ λ ′′ (using Assumption 4.6). Define now λ n +1 = min { λ ′ , λ ′′ } , and observe that λ n +1 > 0 . Proposition 4.5 ensures the existence of a strictly increasing sequence ( M ( n +1) m ) m such that

for m →∞ . Choose m n +1 such that for all m ≥ m n +1 we have

Define now M n +1 = max { M n +1 , M ( n +1) m n +1 } , then we get

The resulting sequence ( M n ) n fulfills then

for n →∞ .

B Additional technical results

In this section we state and prove two technical results that play an important role in the proofs of the main results.

B.1 A characterization of RKHS functions

Here we recall the following characterization of RKHS functions from [31, Section I.4]. Let X ̸ = ∅ be arbitrary. For k : X × X → R symmetric and positive semidefinite and some f ∈ R X as well as N ∈ N + , ⃗ x ∈ X N , ⃗ α ∈ R N define

where we might omit some arguments if they are clear. Furthermore, define

̸

̸

̸

and

We collect now some simple facts that will be used repeatedly.

Let ⃗ x ∈ X N , ⃗ α ∈ R N , N ∈ N + , be arbitrary, and define

By construction, W ( ⃗ x, ⃗ α, k ) ∈ R ≥ 0 (recall that k is positive semidefinite).

Since f ∈ H pre k , its RKHS norm has an explicit form and we find

This also implies that f ≡ 0 if and only if W ( ⃗ x, ⃗ α, k ) = 0 .

If W ( ⃗ x, ⃗ α, k ) > 0 , then

We can now state the characterization result.

Theorem B.1. Let k : X × X → R be a kernel and f ∈ R X . Then f ∈ H k if and only if N ( f, k ) < ∞ . If f ∈ H k , then ∥ f ∥ k = N ( f, k ) .

For convenience, we provide a full self-contained proof of this result.

Proof. Step 1 First, we show that for f ∈ H k , we have ∥ f ∥ k = N ( f, k ) .

N ( f, k ) ≤ ∥ f ∥ k : Let N ∈ N + and ( ⃗ x, ⃗ α ) ∈ X N × R N be arbitrary. Observe that

If W ( ⃗ x, ⃗ α, k ) = ∥ ∑ N n =1 α n k ( · , x n ) ∥ k = 0 , then ∑ N n =1 α n k ( · , x n ) = 0 H k , hence E ( ⃗ x, ⃗ α, f ) = ⟨ f, 0 H k ⟩ k = 0 and by definition D ( ⃗ x, ⃗ α, f, k ) = 0 ≤ ∥ f ∥ k .

If W ( ⃗ x, ⃗ α, k ) > 0 , we can rearrange to get

Since ( ⃗ x, ⃗ α ) was arbitrary, we find that N ( ⃗ x, ⃗ α, f, k ) ≤ ∥ f ∥ k .

N ( f, k ) ≥ ∥ f ∥ k : Let ϵ > 0 and choose f ϵ = ∑ N n =1 α n k ( · , x n ) ∈ H pre k such that ∥ f -f ϵ ∥ k < ϵ . If W ( ⃗ x, ⃗ α, k ) = ∥ f ϵ ∥ k = 0 , then f ϵ = 0 H k and hence E ( ⃗ x, ⃗ α, f ) = ⟨ f, f ϵ ⟩ k = ⟨ f, 0 H k ⟩ k = 0 . By definition, this then shows

Before we continue, note that for all f 1 , f 2 ∈ H k we have

Assume now that W ( ⃗ x, ⃗ α, k ) > 0 , then we get

Altogether, by definition of N ( f, k ) , we get that

Since ϵ > 0 was arbitrary, we find that N ( f, k ) ≥ ∥ f ∥ k .

Step 2 Let f ∈ R X be arbitrary. We show that if N ( f, k ) < ∞ , then

is a well-defined, linear and continuous (w.r.t. ∥ · ∥ k ) map.

To establish the well-posedness , let ( ⃗ x, ⃗ α ) ∈ X N × R N and ( ⃗ y, ⃗ β ) ∈ X M × R M such that

This implies that

and hence W (( ⃗ x, ⃗ y ) , ( ⃗ α, -⃗ β ) , k ) = ∥ ∑ N n =1 α n k ( · , x n ) + ∑ M m =1 ( -β m ) k ( · , y m ) ∥ k = 0 . Assume now that

̸

then we get that

̸

which by definition implies that D (( ⃗ x, ⃗ y ) , ( ⃗ α, -⃗ β ) , f, k ) = ∞ and therefore N ( f, k ) = ∞ , a contradiction.

The linearity is then clear. Finally, to show the continuity , let H pre k ∋ f 0 = ∑ N n =1 α n k ( · , x n ) be arbitrary and set ⃗ x = ( x 1 · · · x N ) , ⃗ α = ( α 1 · · · α N ) , then

Since N ( f, k ) is finite and independent of f 0 , and ℓ f is a linear map, this shows the continuity of ℓ f .

Step 3 Let f ∈ R X such that N ( f, k ) < ∞ . Since according to Step 2 ℓ f is a linear and continuous map on H pre k and the latter is dense in H k , there exists a unique linear and continuous extension ¯ ℓ f : H k → R of ℓ f . Furthermore, from the Riesz Representation Theorem there exists a unique ˆ f ∈ H k with ¯ ℓ f = ⟨· , ˆ f ⟩ k . For all x ∈ X we then get

hence f = ˆ f ∈ H k .

B.2 A Γ -convergence argument

Weuse repeatedly the concept of Γ -convergence, see for example [32]. For convenience, in this section we summarize the well-known and standard main argument, roughly following [33, Chapter 5].

Definition B.2. Let F M : H M → R ∪{∞} and F : H k → R ∪{∞} . We say that F M Γ -converges to F and write F M Γ -→ F , if

For all sequences ( f M ) M , f M ∈ H M , with f M P 1 -→ f for some f ∈ H k , we have F ( f ) ≤ lim inf M F M ( f M ) .

For all f ∈ H k there exists a sequence ( f M ) M with f M ∈ H M such that f M P 1 -→ f and F ( f ) ≥ lim sup F M ( f M ) .

The sequence in the second item is commonly called a recovery sequence (for f ).

Proposition B.3. Let F M Γ -→ F and f ∗ M ∈ argmin f ∈ H M F M ( f ) for all M ∈ N (in particular, all the minima are attained). If there exists B ∈ R ≥ 0 such that ∥ f ∗ M ∥ M ≤ B for all M ∈ N , then there exists a subsequence ( f ∗ M ℓ ) ℓ and f ∗ ∈ H k such that f ∗ M ℓ P 1 -→ f ∗ . Furthermore, F M ℓ ( f ∗ M ℓ ) → F ( f ∗ ) .

Proof. From Theorem 2.3 we get the existence of ( f ∗ M ℓ ) ℓ and f ∗ ∈ H k , and that f ∗ M ℓ P 1 -→ f ∗ . Let f ∈ H k be arbitrary and let ( f M ) M be a recovery sequence for f . We then have

where we used the lim sup -inequality of Γ -convergence in the first step, standard properties of lim sup and lim inf in the second and third step, the fact that f ∗ M ℓ is a minimizer of F M ℓ in the fourth step, and finally the lim inf -inequality of Γ -convergence. Since f ∈ H k was arbitrary, this shows that f ∗ is a minimizer of F .

Furthermore, let ( f M ) M be a recovery sequence for f ∗ , then

where we used the lim sup -inequality in the first step, an elementary property of lim sup in the second step, and finally that f ∗ M ℓ is a minimizer of F M ℓ . Since f ∗ M ℓ P 1 -→ f ∗ , the lim inf -inequality of Γ -convergence implies that so we find that

establishing that F M ℓ ( f ∗ M ℓ ) → F ( f ∗ )

Peer reviews:
comments: SUMMARY:
The authors explore the mean field limit of kernels and their reproducing kernel Hilbert spaces, providing novel theoretical tools and insights for tackling large-scale problems. Furthermore, they employ these kernels in statistical learning, with a particular focus on Support Vector Machines. This new form of limit for learning problems has not been investigated before in the statistical learning theory literature, making the authors' research a valuable contribution to the field.

STRENGTHS:
1. The authors provide a comprehensive investigation of the mean field limit of kernels and their reproducing kernel Hilbert spaces.

2. The research extends the basic theory of mean field kernels, providing novel approximation results and a variant of the representer theorem for mean field kernels.

3. The authors introduce a new form of limit for learning problems and provide convergence results for Support Vector Machines using mean field kernels.

WEAKNESSES:
1. The limitations or drawbacks of the research are not explicitly stated.

2. Section 4 only provides a consistency result instead of a generalization error bound.

QUESTIONS:
1. What is the main improvement of this paper compared to the reference [15]?

2. What is the relationship and difference of this study compared to neural tangent kernel? What is the advantage of mean field kernels?

3. Any concrete examples for mean field kernels?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This paper studies the theory of RKHS consisting of the functions over the space of probability distributions. Complementing the related study [15], the work develops the particle-based approximation theory to RKHS and develops the support vector machines fed distributions as inputs. Moreover, the statistical consistency is also proven.

STRENGTHS:
The paper organizes well the theory of RKHS defined over the space of probability distributions. The results seem technically correct. There are potentially many applications in machine learning. Hence, this work can be of interest to the machine learning community.

WEAKNESSES:
- While I recognize many potential applications, it would be nice if the authors could provide concrete examples and datasets to emphasize the importance of the paper.

- The usefulness and computational complexity of the learning setups studied in the paper are somewhat unclear because of the lack of specific examples and experiments.

QUESTIONS:
Readers might be interested in the particle complexities as $M\to\infty$ because the large number of particles can affect the computational complexity. How large is $M$ needed to achieve a given accuracy in a certain sense?

LIMITATIONS:
It is somewhat unclear how many applications can exist. It would be nice if some applications were mentioned.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops mathematically rigorous construction of the mean field limit of kernels of probability measures, which is obtained as a limit of a sequence of kernels with increasing input dimensions, and its application to SVMs.\nThis is motivated by the analysis of interacting particle systems, when many observations of particles are available and we want to consider a function governing the dynamic as a function of a probability measure rather than a finite-dimensional vector.\n\nSpecifically, this paper first confirms the validity of such kernels by extending the existing results: the existence of the mean field limit of functions and kernels, and relations between a sequence of kernels and the mean-field limit of kernels.\nThen, they prove the representer theorem (a mean-field limit of a sequence of functions uniformly approximated by each corresponding RKHS functions can also be approximated by the mean-field limit of RKHS functions) and convergence of the minimizers.\nFinally, the results are applied to an L2-regularized (with $\\lambda \\geq 0$) loss minimization problem over RKHS.\nThey show the existence and uniqueness of the solution (when $\\lambda > 0$) and convergence of the loss value.\n\nSTRENGTHS:\n### Problem settings are well-motivated\n\nThe purpose of this paper is to develop a foundation of the theory of kernels with probability measures as input in order to view the state of an interacting particle system not as observations of a finite number of particles, but as a function of a probability measure that represents the distribution of the particles. The problem of learning a function with infinite dimensional input may have many other applications, and the potential impact in this area is of a certain significance. \n\n### Rigorous theory for infinite-dimensional input functions\n\nThis paper investigates the formulation and thus the basic properties of RKHS for the limit of a sequence of kernels of increasing dimension. However, it should be noted that the main focus is basically on how to define the limit, and not on how to use it specifically. It can be said that this research has a certain novelty if there is no similar discussion.\n\nWEAKNESSES:\n### Misleading title\n\nIt looks like a paper about generalization error analysis for the community of mean-field neural network optimization. Therefore, I strongly recommend that the authors change the title to characterize their contents more precisely.\n\n### Missing discussion on whether the existing analysis on learning infinite-dimensional input functions can be applicable.\n\nThe final part of this paper is essentially a problem of learning functions with infinite-dimensional inputs, but the discussion on the previous works / applications is missed. The authors should make a remark on whether such existing theories are applicable and hopefully conduct some experiments to verify their theory. An example of previous work I think is:\n\nF. Ferraty, A. Mas, and P. Vieu. Nonparametric regression on functional data: inference and practical aspects. Australian & New Zealand Journal of Statistics, 49(3):267–286, 2007.\n\n### The authors say that the first part of the paper is improvement from an under review paper. However, I cannot know the difference and whether such improvement is necessary for the rest part.\n\nThis paper consists of the three part. (i) Definitions of functions and kernels in the mean-field limit, (ii) approximation ability and convergence of the minimizers, and (iii) application to SVMs based on (ii).\nThe authors argue that (i) is improvement from an under review paper. However, I cannot see the difference and whether such improvement is necessary for deriving (ii) and (iii). If not, this paper looks like an assortment of different results. Also from the perspective of preventing double submission, I think the authors should clarify the difference between their results in (i) and whether such an improvement is necessary.\n\nQUESTIONS:\n- Is it possible to add any experiment to the paper?\n\n- Could you tell me more about the difference between Prop. 2.1 and the previous work, and whether such improvement is necessary for the rest part?\n\nLIMITATIONS:\nIn order to avoid confusion with optimization and generalization theory of mean-field neural networks, the authors are encouraged to change the title.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

comments: SUMMARY:
This work derives mean-field limits of kernels and their associated Reproducing Kernel Hilbert Spaces. In particular, the authors quantify the relationship between the finite-input RKHS and the RKHS of the mean-field kernel, derive a Representer Theorem for mean-field kernels, and provide asymptotic convergence guarantees for mean-field kernel SVMs.

STRENGTHS:
The results are interesting and well-presented. The authors present a complete treatment of the mean-field limit for kernel methods. The proofs are clear and well-written.

WEAKNESSES:
1. While the results are interesting on its own, I am a little skeptical about the applicability of these results in existing ML problems. I think this work would greatly benefit from a more detailed discussion on the motivation and applications. On a related note, I also encourage the authors to present concrete examples of mean-field kernels for ease of exposition. 

2. The complete absence of rates of convergence to the mean-field limit (even for specific kernel classes) is somewhat unsatisfactory. I would appreciate it if the authors could comment on that.

QUESTIONS:
See above. Overall, the results are quite interesting and a bit of work on the motivation and overall presentation would greatly improve the paper. I would be happy to increase my score if the authors are able to address said concerns.

LIMITATIONS:
Limitations are adequately discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors considered learning problems where the input is a interacting-particles / multi-agent system and  studied the mean-field limit of the kernels, RKHS functions and SVM solutions as the number of particles tends to infinity. The authors showed that, essentially, taking the infinite-particle limit is exchangeable with 1) obtaining the RKHS associated with a kernel, 2) obtaining the solution of a variational problem over the RKHS, and 3) obtaining the \"SVM solution\" within the RKHS.\n\nSTRENGTHS:\nThis work continues the exploration by Fiedler et al. (2023) into the mean-field limit of RKHS when we consider interacting-particles systems with a growing number of particles. Compared to Fiedler et al. (2023), the current work extends the theory by 1) tightening the theoretical result on the commutative relation between taking the infinite-particle limit and obtaining the RKHS (no longer requiring taking another subsequence & controlling the RKHS norm of the limiting function, as reported by the authors on Page 4), and 2) expanding the discussion into the mean-field limit of minimization problems in the RKHS. The proofs of these results appear to involve rigorous techniques from functional analysis and gamma convergence, though I was unable to fully verify their correctness.\n\nWEAKNESSES:\nWhile the infinite-particle limit of the learning problem leads us to interesting theoretical investigations, its practical relevance remains to be seen. Overall, the analysis is carried out at a rather general and abstract level, which is perfectly fine for establishing the theory but makes the motivation harder to interpret by the reader. It is therefore not clear to me whether the work could be of great interest to the NeurIPS audience.\n\nQUESTIONS:\nI don't fully understand why the type of functions considered in Section 4 should be called SVM solutions - there is no support vector involved after all, and instead they seem closer to the solutions to ridge regression / ERM with Tikhonov regularization.\n\nThe loss function $l_M$ defined on Line 265, Page 7 is a bit non-standard: it also takes the input vector $\\vec{x}$ as an argument. Since it complicates things somewhat, it could be helpful to explain in what scenarios this term is relevant.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}