Generalization of Scaled Deep ResNets in the Mean-Field Regime
Authors: Yihang Chen
Fanghui Liu
Yiping Lu
Grigorios Chrysos
Volkan Cevher
Venue: ICLR 2024 spotlight (2024)
Keywords: ResNet
mean field
generalization
Rademacher complexity
Primary area: learning theory

Abstract: Despite the widespread empirical success of ResNet, the generalization properties of deep ResNet are rarely explored beyond the lazy training regime. In this work, we investigate scaled ResNet in the limit of infinitely deep and wide neural networks, of which the gradient flow is described by a partial differential equation in the large-neural network limit, i.e., the mean-field regime. To derive the generalization bounds under this setting, our analysis necessitates a shift from the conventional time-invariant Gram matrix employed in the lazy training regime to a time-variant, distribution-dependent version. To this end, we provide a global lower bound on the minimum eigenvalue of the Gram matrix under the mean-field regime. Besides, for the traceability of the dynamic of Kullback-Leibler (KL) divergence, we establish the linear convergence of the empirical error and estimate the upper bound of the KL divergence over parameters distribution. Finally, we build the uniform convergence for generalization bound via Rademacher complexity. Our results offer new insights into the generalization ability of deep ResNet beyond the lazy training regime and contribute to advancing the understanding of the fundamental properties of deep neural networks.

Published as a conference paper at ICLR 2024
GENERALIZATION OF SCALED DEEP RESNETS IN THE MEAN-FIELD REGIME
Yihang Chen
Fanghui Liu University of Warwick fanghui Hiuuwarwickkaen
EPFL
yihang. chen@epf1.ch
Yiping Lu New York University yplu@nyu. edu
Grigorios G. Chrysos University of Wisconsin-Madison
Volkan Cevher
EPFL volkan, cevhergepfl.ch
chrysos@wisc. edu
ABSTRACT
Despite the widespread empirical success of ResNet, the generalization properties of deep ResNet are rarely explored beyond the lazy training regime. In this work, we investigate scaled ResNet in the limit of infinitely deep and wide neural networks, of which the gradient flow is described by partial differential equation in the large-neural network limit, i.e., the mean-field regime. To derive the generalization bounds under this setting, analysis necessitates shift from the conventional time-invariant Gram matrix employed in the lazy training regime to time-variant, distribution-dependent version. To this end, we provide global lower bound on the minimum eigenvalue of the Gram matrix under the mean-field regime. Besides, for the traceability of the dynamic the linear convergence of the empirical eтror and estimate the upper bound of the KL divergence over parameters distribution. Finally, we build the uniform convergence for generalization bound via Rademacher complexity. Our results offer new insights into the generalization ability of deep ResNet beyond the lazy training regime and contribute to advancing the understanding of the fundamental properties of deep neural networks.
Kullback-Leiblee (KL) divergence, we establish
INTRODUCTION
Deep neural networks (DNNs) have achieved great success empirically, a notable illustration of which is ResNet (He et al. 2016), groundbreaking network architecture with skip connections. One typical way to theoretically understand ResNet (e.g., optimization, generalization), is based on the neural tangent kernel (NTK) tool (Jacot et al. 2018). Concretely, under proper assumptions, the training dynamics of ResNet can be described by a fixed kernel function (NTK). Hence, the global convergence and generalization guarantees can be given via NTK and the benefits of residual connection can be further demonstrated by spectral properties of NTK (Hayou et al., 2019 Huang al. 2020 Hayou et al. | Tirer al. (2022). However, the NTK analysis requires the parameters of ResNet to not move much during training (which is called lazy training or kernel regime (Chizat
al. 2019) Woodworth et al. 2020; Barzilai et al. 2022)). Accordingly, the NTK analysis fails to describe the true non- lineariyy of ResNet. Beyond the NTK analysis thus received great attention in the deep learning theory community.
One typical approach is, mean-field based analysis, which allows for unrestricted movement of the parameters DNNs during training, (Wei al. 2019 Woodworth al., 2020 Ghorbani et al. 2020 Yang & Hu [021] AAkiymma & Suzuki,2022 Ba et al., 2022; Mahankali et al.) [023] Chen et al.
2022 Mei et al. 2018; Rotskoff Vanden-Eijnden 2018 | Nguyen| [2019 (iiignnano & Spiliopoulos 2020bb. The training dynamics can be formulated an optimization problem over the weight space of probability measures by studying suitable scaling limits. For deep ResNet, taking the limit of infinite
depth is naturally connected to continuous neural ordinary differential equation (ODE) (Sonoda &
Muratal 2019. Weinan 2017]] Lu et al., 2018 Haber & Ruthotto [017 Chen et al. 2018), which makes the optimization guarantees of deep ResNets feasible under the mean-field regime (Lu et al.)

---

Published as a conference paper at ICLR 2024
2020 Ma et al. 2020; Ding et al., 2021 2022 Barboni et al. 2022). The obtained optimization results indicate that infinitely deep and wide ResNets can easily fit the training data with random labels, i... global convergence. While previous works have obtained promising optimization results for deep ResNets, there is a notable absence of generalization analysis, which is essential for theoretically understanding why deep ResNet can generalize well beyond the lazy training regime. Accordingly, this naturally raises the following question:
Can we build a generalization analysis of trained Deep ResNets in the mean-field setting?
We answer this question by providing generalization analysis framework of scaled deep ResNet in the mean-field regime, where the word scaled denotes a scaling factor on deep ResNet. To this end, consider infinite width and infinite depth ResNet parameterized by two measures: over the feature encoder and I over the output layer, respectively. By proving the condition number of the optimization dynamics of deep ResNets parameterized by т and v is lower bounded, we obtain the global linear convergence guarantees, aiming to derive the Kullback-Lein such two measures between initialization and after training. Based on the KL divergence results, we, therefore, build the uniform convergence result for generalization via Rademacher complexity under the mean-field regime, obtaining the convergence rate at (1/ vn) given n training data.
(KL) divergence of
Our contributions are summarized as below:
The paper provides the minimum eigenvalue estimation (lower bound) of the Gram matrix of the gradients for deep ResNet parameterized by the ResNet encoder's parameters and MLP predictor's parameters.
The paper proves that the KL divergence of feature encoder T and output layer can be bounded by a constant (depending only on network architecture parameters) during the training.
This paper builds the connection between the Rademacher complexity result and KL divergence, and then derives the convergence rate (lI/vni) for generalization.
Our theoretical analysis provides an in-depth understanding of the global convergence under minimal assumptions, sheds light on the KL divergence of network measures before and after training, and builds the generalization guarantees under the mean-field regime, matching classical results in the lazy training regime (Allen- Zhu et al. 2019 9 door to generalization analysis for feature learning and look forward to which adaptive features can be learned from the data under the mean-fiel -ield regime.
Du et al.) 2019b). We expect that our analysis opens the
2 RELATED WORK
In this section, we briefly introduce the large width/depth ResNets an ODE formulation, NTK analysis, and mean-field analysis for ResNets.
2.1 INFINITE-WIDHH INFINITE-DEPTI RESNET, ODE
The limiting model of deep and wide ResNets can be categorized into three class, by either taking the
width or depth infinity: a) the infinite depth limit to the ODE/SDE model (Sonoda & Murata 2019
Weinan 2017: Lu et al. [2018] CCen et al. 2018 Haber & Ruthotto 2017: Marion et al.) 2023); 3): b)
infinite width limit, (Hayou al. 2021] Hayou & Yang) 2023 Frei et al. 2019); c) infinite depth
width ResNets, mean-fiele ODE framework under the infinite depth width limit (Li et al. 2021 Lu
al. 2020 Ding et al. 202T 2022 Barboni et al. 2022)
In this work, we are particularly interested in mean-field ODE formulation. The deep ResNets modeling by mean-field ODE formulation stems from (Lu et al. 2020), which every residual block is regarded as a particle and the target is changed to optimize over the empirical distribution of particles. [Sander et al. (2022) discusses the rationale behind such equivalence between discrete dynamics and continuous ODE for ResNet under certain cases. The global convergence is built under a modified cost function (Ding et al.) 2021). I). and further improved by removing the regularization term the cost function (Ding et al. 2022). However, the analysis in (Ding al. (002)) requires more technical assumptions about the limiting distribution. [arboon et al. (2022) show a local linear convergence by parameterizing the network with RKHS. However, the radius of the ball containing
2

---

Published as a conference paper at ICLR 2024
parameters relies the N-universality and is difficult to estimate. Our work requires minimal assumptions under a proper scaling of the network parameters and the design of architecture. and hence foresters optimization and generalization analyses. There is a concurrent works (Marion et al. 2023) that studies the implicit regularization of ResNets converging to ODEs, but the employed technique is different from ours and the generalization analysis in their work is missing.
2.2 NTK ANALYSIS FOR DEEP RESNET
Jacot et al. (2018) demonstrate that the training process of wide neural networks under gradient flow can be effectively described by the Neural Tangent Kernel (NTK) as the network's 's width (denoted as M') tends towards infinity under the NTK scaling (Du al. 2019b). During the training, the NTK
remains unchanged and thus the theoretical analyses of neural networks can be transformed to those of kernel methods. In this case, the optimization and generalization properties of neural networks can be well controlled by the minimum eigenvalue of NTK (Cao & Gu 2019 Nguyen et al. [202] I). Regarding ResNets. the architecture we are interested in, the NTK analysis also valid (Tirer et al. 2022 Huang et al. 2020; Belfer et al., (02T). as well as an gggorthhm-depnndee
bound (Frei et al. 2019) in the lazy training regime. Compared with kernelized analysis of the wide neural network, we do not rely the convergence to fixed kernel as the width approaches infinity to perform the convergence and generalization analysis. Instead, under the mean field regime, the so-called kernel falls into time-varying, measure-dependent version. We also remark that, for ResNet with an infinite depth but constant width, the global convergence is given by Cont et al. (2022) beyond the lazy training regime by studying the evolution of gradient norm. To knowledge, this is the first work that analyzes the (varying) kernel eigenvalue of width/ ResNet beyond the NTK regime in terms of optimization and generalization.
2.3 MEAN-FIELD ANALYSIS
Under suitable scaling limits (Mei et al., 2018 Rotskoff & Vanden-Eijnden, 2018 Sirignano & Spiliopoulos] 2020a), as the number of neurons goes to infinity, i.e. M - oc, neural networks work in the mean-field limit. In this setting, the training dynamies of neural networks can be formulated as an optimization problem over the distribution of neurons. A notable benefit of the mean-field approach is that, after deriving a formula for the gradient flow, conventional PDE methods can be utilized to characterize convergence behavior, which enables both nonlinear feature learning and
global convergence (Araújo al. 2019 Fang et al. 2019 Nguyen 2019
Du
2019a; Chatterji
et al. 2021 Chizat & Bach 2018 Mei et al. 2018 Wojtowytsch
Lu et al
2020 Sirignano &
Spiliopoulos 2021 2020b et a.. 2020 Jabir et 2021).
In the case of the two-layer neural network. Chizat & Bach 2018): Mei et al. (2018); Wojtowytsch (2020): Chen et al. (2020); BBarboni et al. justiiu the mean-field approach and demonstrate the convergence of the gradient flow process, achieving the zero loss. For the wide shallow neural network, Chen et al. (2022) proves the linear convergence of the training loss by virtue of the Gram matrix. In the multi-layer case, [u et al. (2020); Ding al. (222 2022) translate the training process of ResNet to a gradient flow partial differential equation (PDE) and showed that with depth and width depending algebraically the accuracy and confidence levels, first-order optimization methods can be guaranteed to find global minimizers that fit the training data.
In terms of the generalization of trained neural network under the mean-field regime, current results are limited to two-layer neural networks. For example, Chen et al. (2000) provides a generalized NTK framework for two-layer neural networks, which exhibits "kernel-like" behavior. Chizat & Bach] (2020) demonstrate that the limits of the gradient flow of two-layer neural networks can be characterized max-margin classifier in a certain non-Hilbertian space. Our work, instead, focuses on deep ResNets in the mean-fiee field regime and derives the generalization analysis framework.
3 FROM DISCRETE TO CONTINUOUS RESNET
In this section, present the problem setting of our deep ResNets for binary classification under the infinite depth and width limit, which allows for parameter evolution of ResNets. Besides, several mild assumptions are introduced for our proof.
3

---

Published as a conference paper at ICLR 2024
3.1 PROBLEM SETTING
For an integer L, we use the shorthand [1] (1, Let X C Rd be a compact metric space and Y' _ R. We assume that the training set D,, = (1)),,))) 1= drawn from an unknown distribution on X x V, and fX is the marginal distribution of μ over X. The goal of our supervised learning task is to find a hypothesis (i.e., a ResNet used in this work) f + such that f(;; +) parameterized by θ is a good approximation of the label y corresponding to a new sample X. this paper, we consider a binary classification task, denoted by minimizing the expected risk, let (f. v) Hyf < 0}.
min 1 := (( to 80-1 (z O),
Note that the loss is non-convex and non-smooth, and thus difficult for optimization. One standard way in practice for training is using surrogate loss for empirical risk minimization (ERM), normally convex and smooth (or at least continuous), e.g., the hinge loss, the cross-entropy loss. Interestingly, the squared loss, originally used for regression, can be also applied for classification with good statistical properties terms of robustness and calibration error, as systematically discussed in Hui & Belkin (2020) Hu et al. (2022). Therefore, we employ the squared loss in ERM for training.
let 2(f v) [
min I(0) = Σ (f(x;; O), Ea-D, 1=1
θ
where Dn is the empirical measure of px over {m:) and note that y is function of æ.
We call the probability measure p p2 if p has the finite second moment, and p E C(P2; [0, 1)) O p2 Vs E [0, 1]. For Pl, P2 E p2, the 2-Wasserstein distance is denoted by W2(P1, P2); and for
P2 E C(P2; [0, Il), we define (2 P2) ;= supse(o.1] (2(pi,
3.2 RESNETS IN THE INFINITE DEPTII AND WIDTH LIMIT
The continuous formulation of ResNets a recent approach that uses differential equations to model the behavior of the ResNet. This formulation has the advantage of enabling continuous analysis of
the ODE, which can make the analysis of ResNets easier (Lu et al. 2020 Ding et al.) 202T 2022]
Barboni et al., 2022). We firstly consider the following ResNet (He et al. 2016 of depth L can be
formulated as 20 (x)
= x E Rd. and
(x+ (x) +
a
Σ
(=() E ERR, LE L [L[-]] 1],
MT
(2)
fax 02.
h h(zz, wk) ER,
where æ Rd is the input data, 3 R4 are the scaling factors. 1 10,m m E 12 4-1MM
==
is the parameters of the ResNet encoder a Rd - Rd (cativationn functions are implicitly included into σ). and f2k {w> e & K is the parameters of the predictor h Ed - R. We introduce a trainable MLP parametrized by w in the end, which is different from the fixed linear predictor in Lu et al. 2000 Ding al. 2022 2022). We make the assumptions on the choices of activation function and predictor h later in Assumption 3.3 Different scaling of B leads to different
vM, B vK K corresponds to the standard scaling in the NTK regime (Du et al., 2019b), while setting a B corresponds to the classical mean field
training schemes. Note that setting
analysis (Mei et al. 2018 Rotskoff & Handen-Eijndens 2018 Lu et al.) 2020 Ding et al. 2022). We will keep a, 3 as a hyperparameter in our theoretical analysis and determine the choice of a, 8 in future discussions. Besides, the scaling 1 1/L is necessary to derive the neural ODE limit (Marion let al.] (0222), which has been supported by the empirical observations from Bachlechner et al. (202T
Marion et al. (2023).
We then introduce the infinitely deep and wide ResNet which is known as the mean fiend limit of deep ResNet (Lu et al. 2020 Ma al. 2020 Ding al. 2022
4

---

Published as a conference paper at ICLR 2024
Infinite Depth To be specific, re- re-parametrize the indices | E [L] in Eq. (2) with = [0.1].
We view & in Eq. (2) as a function in s that satisfies a coupled ODE, with 17L being the stepsize.
Accordingly, we write (s) 0m (L)L) f and @ ( {0.m (s) m= The continuous limit of Eq. (2 by taking + co is
dz
a
(z (m))
(z((x,)) s), O)derr)d,,)))) z(x,0) æ,
(3)
ds
M
792m1
where the discrete probability (n s) is defined as (M, 37 E. de )m)(s) (0) (0). Accordingly, empirical risk in Eq. ① can written as
i(sk, am) ELnD, ((fax R (()).
(4)
Infinite Width The mean-field limit is obtained by considering a ResNet of infinite width, i.e. M + co. Denoting the limiting density (M,)ss) by v(0,s) E C(P2; [0, 1J), Eq. (3) can be written
(z(w, 8) ds
ax [ (z((x, 8), OO,), 8 SE (0.1)]
2(x, 0) x.
(5)
Juh
We denote the solution of Eq. (5) as Z, (a, 8). Besides, we also take the infinite width limit in the final layer, i.c. K - oc, and then the limiting density of w is T(w). The whole network can be written as
fr,( = B- / Rd+
n(Zu( 1), (w)
(6)
and the empirical loss in Eq. T can be defined as:
((T, EAND, ((T,(()),)(m)))
3.2.1 PARAMETER EVOLUTION
In the discrete ResNet mally small learning rate, the updating process can be characterized by the particle gradient flow, see Definition 2.2 Chizat & Bach (2018):
consider minimizing the empirical loss i(S2k, Oz.as) with an infinitesi-
dΩ (1) KV i(S (t), (t),
(8)
dt
d0 (0,.M(( (1) L. (t) LL.M
dt
where t is the rescaled pseudo-time, which amounts to assigning & mass to each particle, and is convenient to take the many-particle limit.
In the continuous ResNet (5. we use the gradient flow in the Wasserstein metric to characterize the evolution of T, (Chizat Bach 2018). The evolution of the final layer distribution T(w) can be characterized
Ør (w,t) = Vw (Tlo ,ī((, dr (w.t). (w))))) t20,
(10)
Øt
where
8L v)
(w) Ea-D, (fr,(()) - -(x)) h (ZZ,(z, w)]
(11)
8T
In addition, the evolution of the ResNet layer v(0, s) can be characterized as
Øt dv (0, s, Vo (M(B,s sĪ(T,V)
(0.s..)). 020.
(12)
8v
5

---

Published as a conference paper at ICLR 2024
From the results in Lu et al. 2020); Ding et al. 2002 2021 as follows:
compute the functional derivative
8L(T. v)
az, (z. () 6 (z, s)
(0, 8) = E [B (Sr v(x))
(0,s)
(13)
5v
(Z, ov
BgND, p [B
(x)) y())) P. (z, o (Z 8) 0)),
(14)
where Pv RR, parameterized by 2z , M, the solution to the following adjoint ODE, with initial condition dependent on
dpp
(x. 8) - Pu (2r,,) / V=o 8). (),
(15)
ds
Py (x,1)= 1)= J&e. 12)
1),
(16)
For the linear ODE (15), we can directly obtain the explicit formula, p, (a,, 8) = p, (a,, 1)q, (2, 8). where (2, 8) is the exponentially scaling matrix defined in Eq. (17 The correctness of solution can be verified by taking the gradient w.r.t. s at both sides.
au 8) exp (a (af.
/
(17)
3.3 ASSUMPTIONS
In the following, we use the upper subscript for ResNet ODE layer E [0. 1]. and the lower subscript
for training time t [0, +.)). For example, Tw(w) == 7( t), and v (0) v00 s, t). First,
assume the boundedness and second moment of the dataset by Assumption 3.1
Assumption 3.1 (Assumptions data). We assume that for I, F xj pxx, the following holds
with probability 1,
| = 1, (y( < 1, (æi, 2z)) S Cmax < 1, Vi,j E [n].
Remark:
The assumption, i.e., 2; Zj being not parallel, is attainable and standard in the analysis of neural networks (Du et al. 2019b; Zhu al. 2022).
Second, we adopt the standard Gaussian initialization for distribution 7 and v.
Assumption 3.2 (Assumption initialization). The initial distribution To, Vn is standard Gaussian: (To, (w, 0, s) exp
|| 1|011
Vs
[0, 1].
Next, we adopt the following assumption on activation o, terms of formulation and smoothness. The widely used activation functions, such as Sigmoid, Tanh, satisfy this assumption.
Assumption 3.3 (Assumptions on activation o,, h). Let O = (u, w b) E RR where u. wr Rh 6 I. i.e. k, 2d + 1; (a, w, b) RT, where Rha a. b P. iee. k, +
For any z E Rv, we assume
o(2,0)
h(z,w)= aoo(w =
T z 2+b).
To To:Re R-
(18)
In addition, we have the following assumption on [oo(z)] S C₁ max 1), [x)()
<
C1, (z)) S C₁, and let (a(o)) be the i-th Hermite coefficient of og-
Based on our description of the evolution of deep ResNets and standard assumptions, we are ready to present our main results on optimization and generalization in the following section.
MAIN RESULTS
In this section, we derive a quantitative estimation of the convergence rate of optimizing the ResNet. Our main results are three-fold: a) minimum eigenvalue estimation of the Gram matrix during the training dynamics which controls the training speed; b) quantitative estimation of KL divergence between the weight destruction of trained ResNet with initialization; c) Rademacher complexity generalization guarantees for the trained ResNet.

---

Published as a conference paper at ICLR 2024
4.1 GRAM MATRIX AND MINIMUM EIGENVALUE
The training dynamics is governed by the Gram matrix of the coordinate tangent vectors to the functional derivatives. In this section, bound the minimum eigenvalue of the Gram matrix of the gradients through the whole training dynamics, which controls the convergence of gradient flow.
In the lazy training regime (Jacot al. 2018), the Gram matrix converges pointwisely to the NTK as the width approaches infinity. Hence one only needs to bound the Gram matrix's 's minimum eigenvalue at initialization, and the global convergence rate under gradient descent is bounded by the minimum eigenvalue of NTK. In our setting under the mean-field regime, we also need similar Gram matrix/matrices aid our proof on the training dynamics, but we do not rely on their convergence to the NTK when the width approaches infinity. Instead, we consider the Gram matrix of the limiting mean-field model Eq. (6
For the ResNet parameter distribution v, we define one Gram matrix Gi,T, E Rax by
G. GT S( (T,, Gi(T,V, o)
(19)
where the row vector of J1 is defined as (T, s)),
isisn,
where the dependence on on the right side of the equality is from the initial condition p. (x, 1). We also define the Gram matrix for the MLP parameter distribution т, G2(T, E Ra by
Gz(T,v) E E Ja(,
(20)
where the row vector of J2 is defined as
(2,,ww)), V (W (2,; 1),
ssisn.
We characterize the training dynamics of the neural networks by the following theorem (the proof deferred to Appendix C.I), which demonstrates the relationship between the gradient flow of the loss and those of functional derivatives.
Theorem 4.1. The training dynamics of L(TT,, 14) can be written as:
dT( ------1 dl
-
oī, vz)
du)(O,s) SSrn Vw
oT Us) 87
V
0,5) s)
(w)
dr((w).
gm
du
From the definition of functional derivatives slrre (0,8) and sT((ee))) (w)) we immediately obtain
87c
Proposition 4.2 an extension of Theorem 4.1 which demonstrates that the training dynamics can be controlled by the corresponding Gram matrices.
Proposition 4.2. Let b = (Sr.,, (a1) - y(m1), ... (r. y(x,)), using the Gram matrix
defined in Eq. 19 and 20 the training dynamics of L(Te, vn) be written
dī v1)
g² b, (a² Gr(,,)) G2(Te,
dt
n²
Our analysis mainly relies on the minimum eigenvalue of the Gram matrix, which is commonly used in the analysis of overparameterized neural network (Arora et al. 2019 Chen et al.) 2020). The minimum eigenvalue of the Gram matrix controls the convergence rate of the gradient descent.
We remark that the Gram matrix G GT(T,, Pt) is always positive semi-definite for any t 2 0, and Gi (70, pD) Onxn- Therefore, we only need bound the minimum eigenvalue of G2(T,, mt). First, we present such result under initialization, i.e., the lower bound of Amin \min (G2(To, vo)) by the following lemma. The proof deferred Appendix C.2
Lemma 4.3. Under Assumption 3.7
3.2 3.3 there exist constant A A(d), only depending on is lower bounded by
the dimension d, such that Xeuin G To. Vol
A Amin (GT,, vn)) 2 Amin Amin(G2( (G2(To, vo)) 2 A(d).
7

---

Published as a conference paper at ICLR 2024
Remark: Using the stability of the ODE model, derive the KL divergence by virtue of the structure of the ResNet, build the lower bound of Amiu Amin(G2) (G2), and prove the global convergence. In fact, results, c.g., global convergence, and KL divergence can also depend on G1 by taking (t) = a² Xmin ( (t) + Amin ( in Lemma C.5 Due to Xmin [G (t)) 2 0 for any i, we only use Amin G2) 2 Mdd) in Lemma 4.3 for proof simplicity. Our model degenerates to two-layer neural network if the residual part is removed (can be regarded as an identity mapping).
Second, for T, V different from initialization 70, Po. first prove in the finite time Lmax, have the minimum eigenvalue is lower bounded Amin (G2 14) 2 Ao/2. 2. In the next, we choose proper scaling of a, P, such that maax oc, so that we obtain a global guarantee.
The proof is deferred to Appendix C.2
Lemma 4.4. There exists Tiax: such that, for v E C(P²; 00, 1J) and T E p2 satisfying max(WW( vo), W2(T, Tmax we have Amin (nin(G)(T, 2 2
A(d) Remark: The radius is defined as Tmax = min Vd. inCald,a] where A(d) is defined in Lemma 4.3 and Ca (d, a:) is some constant depending on and a, used for the uniform estimation of G2((T, v) around its initialization, refer Lemma C.2 We detail this in Appendix C.3
Definition 4.5. Define
mmax stp s.t.vt E to), max (M2(U,, Wa( ro)) S Fmax).
where Tmax is defined in Lemma 4.4
4.2 KL DIVERGENCE BETWEEN TRAINED NETWORK AND INITIALIZATION
Based on our previous results on the minimum eigenvalue of the Gram matrix, we are ready to prove the global convergence of the empirical loss over the weight distributions and of ResNets, and well control the KL divergence of them before and after training. The proofs in this subsection are deferred to Appendix C.4
We first present the gradient flow of the KI. divergence of the parameter distribution.
Lemma 4.6. The dynamics of the KL. divergence (T.(T: (T<(veluo)) through training can be characterize by
(TT.(TelTT)... = dt Jer 8Tc ((okl(/l())))
(va (KL(TTl
(vu (T(T)) ôT
dre(w),
dKL(v)))o)) / J
(KI(vi (oo (i((T,
Vo
dvi(0.s).
dt
dv
öv
Since the evolution of Tr, M4 is continuous, define the notation tmax > 0 in the following, such that the minimum cigenvalue of G2(Ft, v4) can be lower bounded for < fmax. In our later proof, will demonstrate that fmax
be achieved under proper a and
Combining Lemma 4.4 and Definition 4.5
we immediately obtain that Amin v)) 2
Auin (Gu(T, v)) 2 Ao/2 for << fuaxx.
By choosing certain a, B, we could prove tmax = 0o, which leads to the bound KL (T))) and (T(()))o)) uniformly for all > 0. (c.f. Theorem 4.7
Theorem 4.7. Assume the PDE (10) has solution Ti E P2, and the PDE (2) has solution E
C(P2, [0, 1)). Under Assumption 3.7 3.2 3.3
for some constant OKL. dependent on d, a, taking
8 (KCk(dd,))
=
Armax
the following results hold for all o0):
22 2x
Ckl(d,
(dI(d,
[(To))) se L((o, KL(FellT))
KL(velpoo) - A2
A252
We also derive a lower bound for the KL divergence. In Lemma C.10 we have that the average movement of the KI. divergence is in the same order as the change in output layers.
8

---

Published as a conference paper at ICLR 2024
4.3 RADEMACHER COMPLEXITY BOUND
After our previous estimates on the minimum eigenvalue of the Gram matrix as well as the KL divergence, we ready to build the generalization bound for such trained mean-field ResNets. The proofs in this subsection are deferred to Appendix C.55
Before we the proof, we introduce some basic notations of Rademacher complexity. Let Dx = {= be the training dataset, and ". , be an i.i.d. family of Rademacher variables taking values =1 with equal probability. For any function set H. the global Rademacher complexity is defined as Rr (?C) = E suPдEи 1 >i=1 m=h();)).
Let F = (c) = 3 2 1). [))( be the function class of infinite wide infinite depth ResNet defined in Section 3 We consider the following function class of infinite wide infinite depth ResNets whose KL divergence to the initial distribution is upper bounded by some 0: (r)(() = {Sr. E E (L(T]]>)) S r, (L)ppp))) <r.. The Rademacher complexity of FKL is given by the following lemma.
Lemma 4.8. Under Assumption
3.3 if S ro = 0(1 /n), the Rademacher complexity of FKL(r)
can be bounded by R,, (FKL(()))
S 8/r/n, where & hides the constant dependence on d, a.
Now we consider the generalization error of the 0-1 classification problem.
Theorem 4.9 (Generalization). Assume Ty E C(P²; [0. 1)) and Vy E p2 be the ground truth distributions, such that, y(x) FEwnr,h(2 hZ,,, (,, (xx 1),ww), Under the Assumption 3.1 an2 3.3 we set B > S2( n.). For any 8 > O, with probability at least 6, the following bound holds:
Ex-ax (S- (x) y(x)) 11/V1 6vloog(2/))/2
where S hides the constant dependence on
Remark: Our results of O(1/vn) matches the standard generalization error in the NTK regime (Du et al. 2019b). However, in contrast setting vM, B K in Eq. (2) as the NTK regime, we directly analyze the ResNet the limiting infinite width depth model in Eq. (6). and select proper choice of 3 independent of the width. We also validate theoretical results by some numerical experiments in Appendix C.6
CONCLUSION
In this paper, we build the generalization bound for trained deep results beyond the NTK regime under mild assumptions. Our results demonstrate that the KL divergence between the distribution of parameters after training and initialization of infinitely width and deep ResNet can be controlled via lower bounding the eigenvalue of the Gram matrix during training. Under some stronger data assumptions, e.g., k-sparse parity problem (Suzuki et al. 2023), we may ensure that the limiting distribution of deep ResNet moves far away from its initialization in terms of KL divergence, which cannot be derived under the current setting. We leave as the future work.
ACKNOWLEDGEMENT
This work was carried out in the EPFL LIONS group. This work was supported by Hasler Foundation Program: Hasler Responsible AI (project number 21043), the Army Research Office and was accomplished under Grant Number W911NF-24-1-000 and Swiss National Science Foundation (SNSF) under grant number 200001_205011. 205011. Corresponding authors: Fanghui Liu and Yihang Chen.
REFERENCES
Shunta Akiyama and Taiji Suzuki. Excess risk of two-layer relu neural networks teacher-student settings and its superiority kernel methods. arXiv preprint arXiv:2205. 14818. 2022.
Z. Allen-Zire Zhu, Y. Li. and Z. Song. A convergence theory for deep learning via over prramttreettiiiiiionn... In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 242-252. PMLR, 09-11 Jun 2019.

---

Published as a conference paper at ICLR 2024
D. Araújo, R. Oliveira, and D. Yukimura. A mean-field limit for certain deep neural networks. arXiv/1966001933, 2019.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference Machine Learning, pp. 322-332. PMLR, 2019.
Jimmy Ba, Murat Erdogdu, Taiji Suzuki, Zhichao Wang, Denny Wu, and Greg Yang. High- dimensional asymptoties of feature learning: How one gradient step improves the representation. Advances in Neural Information Processing Systems, 35:37932-37946, 2022.
Thomas Bachlechner. Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. Rezero is all you need: Fast convergence at large depth. In Uncertainty in Artificial Intelligence, pp. 1352-1361. PMLR, 2021.
Raphaäl Barboni, Gabriel Peyré, and Trançois-Xavier Vialard. On global convergence of resnets: From finite to infinite width using linear parameterization. Advances in Neural Information Processing Systems, 15:16385-66397, 2022.
Daniel Barzilai, Amnon Geifman, Meirav Galun, and Ronen Basri. A kernel perspective of skip connections in convolutional networks. arXiv preprint arXiv:2211.148111 2022.
Yuval Belfer, Amnon Geifman, Meirav Galun, and Ronen Basri. Spectral analysis of the neural tangent kernel for deep residual networks. arXiv preprint arXiv:2104.03093, 2021.
Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. Advances in Neural Information Processing Systems, volume 32, 2019.
N. Chatterji, P. Long, and P. Bartlett. When does gradient descent with logistic loss interpolate using deep networks with smoothed relu activations? arXiv/2102.04998, 2021.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances neural information processing systems, 31, 2018.
Zhengdao Chen, Eric Vanden-Bijnden, and Joan Bruna. On feature learning in neural networks with global convergence guarantees. arXiv preprint arXiv:2204. 10782, 2022.
Zixiang Chen, Yuan Cao, Quanquan Gu, and Tong Zhang. A generalized neural tangent kernel analysis for two-layer neural networks. Advances in Neural Information Processing Systems, 33: 13363-13373, 2020.
L. Chizat and Bach. On the global convergence of gradient descent for over per-parameterized models using optimal transport. In Advances in Neural Information Processing Systems, volume 31, 2018.
Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference Learning Theory, pp. 1305-1338. PMLR, 2020.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019.
Rama Cont, Alain Rossier, and Ren Yuan Xu. Convergence and implicit regularization properties of gradient descent for deep residual networks. arXiv preprint urXiv:2204.07261, 2022.
Zhiyan Ding, Shi Chen, Qin Li, and Stephen Wright. On the global convergence of gradient descent for multi-layer resnets in mean-field regime. urXiv preprint arXiv:2110.02926, 2021.
Zhiyan Ding, Shi Chen, Qin Li, and Stephen J Wright. Overparameterization of deep resnet: Zero loss and mean-field analysis. J. Mach. Learn. Res., 23:48-1, 2022.
Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time, i. Communications on Pure and Applied Mathematics, 28(1): 1975.
10

---

Published as a conference paper at ICLR 2024
S. Du, J. Lee, H. Li, L. Wang, and X. Zhai. Gradient descent finds global minima of deep neural networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1675-1685, 09-15 Jun 2019a.
S. Du, X. Zhai, B. Póczos, and A. Singh, Gradient descent provably optimizes wer-parameterized neural networks. In International Conference on Learning Representations, 2019b.
W. E. C. Ma. and L. Wu. Machine learning from a continuous viewpoint, Science China Mathematics, 63 10:2233-2266 Sep 2020.
C. Fang, Y. Gu, W. Zhang, and T. Zhang. Convex formulation of overparameterized deep neural networks. arXiv/911//0626, 2019.
Frei, Y. Cao, and Q Gu. Algorithm-dependent generalization bounds for overparameterized deep residual networks. In NeurlPS, 2019.
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. When do neural networks outperform kernel methods? Advances Neural Information Processing Systems, 33: 14820-14830, 2020.
Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse problems, 34 (1):014004, 2017.
Soufiane Hayou and Greg Yang. Width and depth limits commute in residual networks. In Interna- lional Conference on Machine Learning, 2023.
Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. Exact convergence rates of the neural tangent kernel in the large depth limit, arXiv preprint arXiv: 13654, 2019.
Soufiane Hayou, Eugenio Clerico, Bobby He, George Deligiannidis, Arnaud Doucet, and Judith Rousseau. Stable resnet. In International Conference on Artificial Intelligence and Statistics, pp. 1324-1332. PMLR, 2021.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Identity mappings in deep residual net- works. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pp. 630-645. Springer, 2016.
Tianyang Hu, Jun Wang, Wenjia Wang, and Zhenguo Li. Understanding square loss in training overparametrized neural network classifiers. Advances in Neural Information Processing Systems, 2022.
Kaixuan Huang, Yuqing Wang, Molei Tao, and Tuo Zhao. Why do deep residual networks generalize better than deep feedforward networks? neural tangent kernel perspective. In Advances in Neural Information Processing Systems, volume 33, pp. 2698 2708, 2020.
Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss vs cross- entropy in classification tasks. arXiv preprint arXiv:2006.07332,, 2020.
J. Jabir, D. Šižka, and t.. Szpruch. Mean-field neural odes via relaxed optimal control. arxiv/19/2.055455 2021.
A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization neural networks. In Advances in Neural Information Processing Systems, volume 31, 2018
Mufan Li, Mihai Nica, and Dan Roy. The future is log- Gag-saas::::: ResNets and their infinite-depth- and-width limit at initialization. In Advances in Neural Information Processing Systems, pp. 7852-7864 2021.
Y. Lu, C. Ma, Y. Lu, J. Lu, and L. Ying. A mean field analysis of deep ResNet and beyond: Towards provably optimization via overparameterization from depth. In Proceedings of the 37th International Conference Machine Learning, volume 119, pp. 6426-6436, 13-18 Jul 2020.
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In International Conference on Machine Learning, pp. 3276-3285. PMLR, 2018

---

Published as a conference paper at ICLR 2024
Chao Ma. Lei Wu, et al. Machine learning from continuous viewpoint. Science China Mathematics, 63(11):2233-222 2266, 2020.
Arvind Mahankali, Jeff Z Haochen, Kefan Dong, Margalit Glasgow, and Tengyu Ma. Beyond ntk with vanilla gradient descent: A mean-field analysis of neural networks with polynomial width, samples, and time. arXiv preprint arXiv:2306. 102230666616,,.. 2023.
Pierre Marion, Adeline Fermanian, Gérard Biau, and lean-Philipppe Vert. Scaling resnets in the large-depth regime. arXiv preprint (tti:22066069229 2022.
Pierre Marion, Yu-Han Wu, Michael E. Sander, and Gérard Biau. Implicit regularization of deep residual networks towards neural odes, 2023.
S. Mei, A. Montanari, and P. M. Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 1 15(33):E7665-E767) 2018.
P. M. Nguyen. Mean field limit of the learning dynamics of multilayer neural networks. arXiw/1902.02880. 2019.
Quynh Nguyen, Marco Mondelli, and Guido Montufar. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep relu networks. In International Conference on Machine Learning, pp. 8119-8129. PMLR, 2021.
Quynh N Nguyen and Marco Mondelli. Global convergence of deep networks with one wide layer followed by pyramidal topology. Advances in Neural Information Processing Systems, 33: 11961-11972, 2020.
Felix Otto and Cédric Villani. Generalization of an inequality by talagrand and links with the logarithmic sobolev inequality. Journal of Functional Analysis, 173(2):361-40) 2000.
Michael Poli, Stefano Massaroli, Atsushi Yamashita, Hajime Asama, Jinkyoo Park, and Stefano Ermon. Torchdyn: implicit models and neural numerical methods pytorch. In Neural Information Processing Systems, Workshop on Physical Reasoning and Inductive Biases for the Real World, volume 2, 2021.
Yury Polyanskiy and Yihong Wu. Wasserstein continuity of entropy and outer bounds for interference channels. IEEE Transactions Information T'heory, 62(7):3992 2016.
Grant Rotskoff and Eric Vanden-Fijnden. Parameters interacting particles: long time convergence and asymptotic error scaling of neural networks. Advances in neural information processing systems, 31, 2018
Michael Sander, Pierre Ablin, and Gabriel Peyré. Do residual neural networks discretize neural ordinary differential equations? In Advances in Neural Information Processing Systems, pp. 36520-36532, 2022.
J. Sirignano and K. Spiliopoulos. Mean field analysis of neural networks: A central limit theorem. Stochastic Processes and their Applications, 13003):1820-185) 2020a.
Sirignano and K. Spiliopoulos. Mean field analysis of neural networks: law of large numbers. SIAM Journal on Applied Mathematics, 80(2):725-75) 752, 2020b.
J. Sirignano and K. Spiliopoulos. Mean field analysis of deep neural networks. Mathematics of Operations Research, 2021. doi: 10.1287/moo..2200.111
Sho Sonoda and Noboru Murata. Transport analysis of infinitely deep neural network. The Journal of Machine Learning Research, 20(1):31-82, 2019.
Taiji Suzuki, Denny Wu, Kazusato Oko, and Atsushi Nitanda. Feature learning via mean-field langevin dynamics: classifying sparse parities and beyond. In Advances in Neural Information Processing Systems, 2023.
Tom Tirer, Joan Bruma, and Raja Giryes. Kernel-based smoothness analysis of residual networks, In Mathematical and Scientific Machine Learning, pp. 921-954. PMLR, 2022.
12

---

Published as a conference paper at ICLR 2024
Colin Wei, Jason D Lee, Qiang Liu, and Tengyu Ma. Regularization matters: Generalization and optimization of neural nets vs their induced kernel. Advances in Neural Information Processing Systems, 32, 2019.
Ee Weinan. A proposal machine learning via dynamical systems. Communications in Mathematics and Statistics, 1(5):1-11, 2017.
S. Wojtowytsch. On the convergence of gradient descent training for two-layer relu-networks in the mean field regime. arXiv/2005/13530, 2020.
Blake Woodworth, Suriya Gunasckar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pp. 3635-3673. PMLR, 2020.
Greg Yang and Edward Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In International Conference on Machine Learning, pp. 11727-11737. PMLR, 2021.
Zhenyu Zhu, Fanghui Liu, Grigorios Chrysos, and Volkan Cevher. Generalization properties of nas under activation and skip connection search. Advances in Neural Information Processing Systems, 35:23551-23565 2022.
13

---

Published as a conference paper at ICLR 2024
A OVERVIEW OF APPENDIX
We give a brief overview of the appendix here.
prove some lemmas that will be useful. In Appendix [8.2 we provide the estimation of the activation function a. In Appendix B.3 we provide the prior estimation of Zv.Pu.
Appendix B In Appendix B.1
Appendix In Appendix C.1 we prove the gradient flow of dž and dKL In Appendix C.2, we dt bound the minimal eigenvalue at initialization. Appendix [.3] we bound the perturbation of minimal eigenvalue. In Appendix CC.4. we bound the KI. divergence in finite time, and choose we bound the KI.
scaling parameters to prove the main results in Theorem 4.77 In Appendix C.5) divergence and provided the generalization bound.
Appendix C.6 We provide the experimental verification.
B USEFUL ESTIMATIONS
B.1 USEFUL LEMMAS
Lemma B.1 (2-Wasserttein continuity for functions of quadratie growth, Proposition in Polyanskiy Wu 2016)). Let u.v be two probability measures on RRd with finite second moments, and ler 9 a R be Cj function obeying
[V S <allu/la + c2 Vu E Rdd
for some constants ei and 02 O. Then
E Ewnag(w) Ew-rg(w) $ (c10 (2)((())))
where 0² max m || / EE | }
Lemma B.2 (Corollary 2.1 Ottoo& & Villani 2000 The probability measure vo( exp(- |00|
satisfies following Talagrand inequality (in short T ((t)) ()) for any P2 (R*,)
(V, vo) S (vloo)
Lemma B.3 (Donsker- Varadhan representation (Donsker & Varadhan 1975)). Let p., X be probability measures on a measurable space (X, 2). For any bounded, Y-measurable functions & X + R:
f kdp s KL(uIAA) + log / exp(O)dX.
B.2 ESTIMATION OF Lemma B.4 (Boundedness of (22 0)). Under Assumption 3.3 for E Rd, O IRk we have
|0 0)]1 S Co (1|=||/ 1)(10] 1),
(21)
Vz,0 +1)
(22)
(2. 0| ()(l 1).
(23)
Ago 0)]1) +](1
(24)
Ve(Voor(20)) 0)] Cr(100]2 +)[[==]|a 1),
(25)
Vo 0) I (|0)) + 1) +1,
(26)
where A is the Laplace operator. Let a (o,) (Vzr) = Vey (Vor) = Vo,
(ABo); Ago, (Veo- -0or = (,,
Proof of Lemma B.4 We prove the relations directly in the following:
(0 0)ll = | + b)1l < C1 C1|/u||2/w0 b) < (1 1) + +1), (27)
V (2,
||ull|/wo b) |lll| Glllo|12 S (1 +
(28)
14

---

Published as a conference paper at ICLR 2024
We can write Voo(2z0) Raxk by
oo (www') TouTz 2 +
j=i,
0,
j
(2))
(29)
1z2g-d0/ +b
d d+1sis24 2d
uoo(w 1o:w==+0 +
1d.
Therefore,
d
(2,
Σ
[o w T 2 +) u x b))² (1 + ||=11]]
1=1
$
1=
s 3
+ + - 2d II=1) + 10112))
Therefore,
[2 vV2CC + | |=1)(1 + 110112).
(30)
ior [d]
Az (2, |u(11 11=13000 Tx 2 bbbl s - n=(1==112 + +1).
(31)
Therefore,
(2)0(2,0)1l < (1 +1)
Σ 1=1
uf S <1 -110)22 1).
(32)
By Eq. 29
we obtain
Vz. (). = =u,o0w' 2 *2+ b) + u,w"" T + b)oo(w 2 +). 1sisd.
Hence,
oo(wT= T x + b) (w + 6)o((2 +
j=i,
j j S d,
(Vo(Voo(2, 0));
u1zydd (0b()) + (yo(()))' 1xt6 un (db(v) + (yo)(y))') T240
d+l 2d, 2d
By Assumption 3.3 we have
(2) 0 = (oo(w z + b) +
2 2 b))²
+ ( (vo((t)))) ) ||=13)
< [0 + + 11 +0) + 1
Hence,
Ve(Voorz..))0)
(z) 0)))
(10 + 1)2 4||u||3 + 1)2 (4d 12 1)²
(33)
For the last part, by Eq. 31
15

---

Published as a conference paper at ICLR 2024
(1 |==1)0 (w T = +bb b)
j=i
i
V2
wzz--a 11++1=1)))"
d+1 2d
u, + || (w 2+
j 2d+1
Therefore,
[2 S c1 (1(1)112))))
E 1+ 11
S SVdc + 1) و (100 22 11 = 3V + 1) +1 (34)
The last inequality is from that, for x > 0
1
3 I,
x3
x
3
3 - 23 +
x² +1
x
2
2
2
2
2
then, we have
23
(1+ (1+))=
)) + =1 x2 x
S
+ 2
(1
< 3(1 + x²).
From Eq. 27 Eq. 28 Eq. 30 Eq. (3) Eq.
(3). and Eq.
34 taking C₁ 4v/C1, the proof is
finished. We defer the definition of Co later.
Lemma B.5 (Stability of o (z, 0)). Under Assumption
3.3
for x E Rd 0 R have
| 0) ( 0) < C. (1101 + 1)]|= 22|12
(35)
V o 21, 0) V 0)] S C. [|00]²+])]zz z2 l2
(36)
1 z, 01) Vzo 02) S C. (|0]]12+ ||0212 + 1)] 02112
(37)
Voo 01) Veo 02) F 1 C. ( 1) 0211
(38)
Voo (21, 0) Vor 0) r I C. ([1011 + 1)]lz 22|12
(39)
Proof of Lemma
B.S By the value theorem, we have there exists E [0.1]
| 0) (z 12 S V (z) =1), 0)|F=|I11 z2l12
< C² (110113 1)1| - 22l12
Denote by 0 = (u, w. b), we have
V20 0) [20 0)1| < uuw (d6 (w T = + b) - -6ww0 22 + b))lrr
< C2 (110) _ - 1) -1 - #2ll2
and
[Vz0(2201) - V20 02)11 S w dow = + b1) - Fm clw = + b2)llF
<Caluuww - mmtm |F C² (1 uz) w22
T + (ui u2 uui
(w) w2) T ||
<1Ca + |02lla 1) 02112
In the next, we have
(Voo 01) (z, 0,,2))
Tu(w] + b1) ool ooww] bz)
j=i,
0.
u/23-a0t (w) x + b,) uي (w) bz)
d+1 2d, j j= 11
(w + b1) 2206( z + bz)
16

---

Published as a conference paper at ICLR 2024
and then,
C || W2 22122
j=i,
0,
i d.
(Vo0(2,01))g - (2)o(220))) (V
C [u 2z-2
d+1=is 2d,
C
[1;
j= +1
Therefore,
(oo (z S 22dC 1). 110 02112.
Similarly, we have
(200 0)); - (22 0))is colwT b) ou(w? 22 +
j=i,
0,
j _ d d+
d f(w 21 + b1) u.2 o 22 12+10
u + b) u. u 22 + b)
j j= 2d +1,
and then
C2 |2|||||21 - 22112
j=i,
0,
j d+1 2d,
(Vo0(22,01)lg (2)))(
C u/ --12)_d- 27-dl
C2
u
ui| 22 22/12/120112
j +1.
Therefore,
Vo (21, (2o 0)ll
vack +)]] 22ll2
taking C² = v 2dC, the proof is finished.
Combined with the estimation in the proofs Lemma B.4 and Lemma B.5 we let
C. = 6d C1
(40)
B.3 PRIOR ESTIMATION OF ODE
The Lemma B.6 Lemma B.7 establishes the boundedness and stability of Z, and py with respect to V.
Lemma B.6 (Boundedness and Stability of Zy). Suppose that Assumption 3.3 holds and that a is in the support of X. Suppose that v1, 1/2 E C(P2, j0, 11) and Zuy Zu, the corresponding unique solutions in Eq. (5)n the following two bounds are satisfied for all E [0, 1]:
|| Zun (xr, 8)1 I Cz 11111112:000
and
1z (x. Zuz (x, l)1 Cz (1 ((): W2(P1, where Cz(llu 1 |122 a) is a constant depending only on ||n ll || and a, and for
E C(P2; [0, 1)), we denote by ||vl|3 = supse[o,1] (o) ||0| co.
Proof of Lemma B.6
We firstly demonstrate that Eq. (5) has a unique C₁ solution and then prove the boundedness of Zμ under different probability measures.
By Lemma
B.5 we have
1 Rt
(0((z), 0) - ((22,0))dda (0,
< Co |2| - -zel12 (1111 + 1)dn (0. s)
j
Co 22 22112 113 + 1),
(41)
17

---

Published as a conference paper at ICLR 2024
which implies that fn+ (2 0) (0, s) is locally Lipschitz. Combining this with the a-priori estimate, the ODE theory implies that Eq. (5) has unique C1 solution.
In the next, we aim prove the boundedness of Zu. For any [0, 11, by Eq. 21 Lemma B.4 we have
M/ (2, ()0,
- S/c |0 1)1 - Co (1 11
(110 + 1)dv1 (0, 8).
To prove the boundedness of Zn using Eq.
(
and Lemma B.4 we have
dl Zuy (ar s)113 ds
mn (x. 8)
22 (z. 8)
ds
(x, 5)12 + Zy, (x) .)]l2) / (1012 + (0,
s 2aCo (1]
S $40Go(112g (a. s)1 / g*
(11013 1)dd (0,
By Grönwall's inequality, and Zu (x. 0) = we have
1z (ax, s)l(2 s exp 2aCa (% / 101B (0. +1)) (1121112 1)
S eepp(aCoo(1e1l)) + 1=)
By Assumption
3.1
||a||2 S 1, we thus have a priori estimate of Zn Let (1(llBBBB)))) + 1)), have I| (z, s)l(2 (1vall at.
2 eexp(2cCo (1| Te
In the next, to estimate the difference under different measures V1 and v2, define
5(x, Zun (Ix, Zuz (2,
and we can easily obtain
d| d1) s)]l2 ds
2a 6(x,s), s), / oZ (a, s), (), /
JRk
= 2a (Ō(x, 8), (A) + (B))
(42)
where, by Eq. (41). we have
[AAA]l2 = 1| / (a Zun (xr, 8), 0) or (Zvg (x, 8), d)) (C + 1),
and
(B) = / o olZn, (z) dv (0.
((Nn,( s). 0)
R
RR
By Lemma
B.I and VVo (z.
s Co (I (z, s). 0)112 + 1)(1 + 1), we can bound
(B) by
[BB]]]2 s Co (1 (
(|0112 + 1) W2
Co (1) (z. +1 (\ max b/ 2 |0 + 1) Wa(1f,15)
C. (C2(
+ [ 1l + 1 + 1),)
Plugging the estimate of (A) and (B) into Eq. 42
we have
d (1)00((,))))) 3)
2aCo (1)((z, 8) |1 113 + 1) - 118(x, s) s)ll
ds
a) + 1)
1 12 + 122 + +) W2(p1,
< (1 x) .) W (2),,12)
&. + 1 + ((1u 1)2.
Since 8(2, 0) = by Grönwall's inequality, have, Vs E [0, 1]. [16 s)]l2 < (exp(aC.) 1): (v |1n 12 + 1 + (11g a) +1) M2(11, which concludes the proof.
18

---

Published as a conference paper at ICLR 2024
Lemma B.7 (Boundedness and Stability of Py). Suppose that Assumption 3.3 holds and that in the support of X. Suppose that V/, 12 E C(P2, [0. 1]) and Pv, (x, 8), Pa, 8) are defined in Eq. 15.. Then the following three bounds are satisfied for all s E [0,
Il (2, 8) l1 S Cp(1 11 1=12:a),
(43)
and
Il (2t 5) - Paz (z, s)ll2 I Cp 1111 112 [TT (2(v1.12):
(44)
where Cp Cp(llen 1B:: 1| |=| (x) is a constant depending only on || 30 u1 llee |1 and a, for p2, we denote by || E 1 c0.
Proof of Lemma B.7
For any s E [0. 1). by Eq.
115 and the estimation of V.0 in Eq. 22 we have
d Pr, (x, s)12
dp,, (x, 8)
Put (x)
ds
ds
s 22 Iu (2. s)12 | V Zen ())
22Cool12 [IpT, (x, 8) 12 (110112 (0,8)
It follows from the estimation of V= in Eq. 22
Pp,, (z, 1)112
Tzh(Zz, (x) 1), w))
SCo - -11 + 1).
RKr
Therefore, by the Grönwall's inequality
(11 1) cxp @Coo / / (11112 (0,))
Pm (z. s)1 s C.
I 11 (1
In the next, we deal with Eq. 444 define
(2( = pu, (2, Puz (x,
we have (taking ==]]
111 2dr
|162(2, 1)l12
Vzh(Zen (x, 1) V (Zr (2, 1), w)d((w)
< CC + 1) Zzn (x. 1) Zoz (x, 1)1l
CC[[| |122 12, [[]3:a) (2(
The following ODE is satisfied by 62( 8) by Rq. 15
06, (x, 8)
/ g/
5, (z, w.s)
(Zu, (x, s), d), (0. s) + pr,( (z,s)
Os
with
(2, == / (a, 0)dee 0)d / (2)
RR
Furthermore, we also split D Dus,og (1) as
1/1 (2,, O)d 8) -
-/c A (z))
Dures (2, (2, s)(lF <
(V (2) (z,,
(x, dv (0.
(B)
19

---

Published as a conference paper at ICLR 2024
Clearly, (B) can be estimated by
(B) Cz( 112ll:: Co (11e +1) (2)
To estimate (A), denote क E II(1f, uS) such that 020,00))=;;10 110 - 0211 = WV by
Lemma B.3
we then have
(A) S Eo, 02)m 1 (I, 8), - V=0 (2.
S C. / (1 1112 + ||2113 2 + 1 /B os) 0211
CC. /311pul1 + 122 +1) (21
Combining the estimate of (A) and (B), we have
Den (z. F s C( (121)biaa) (2(
Accordingly, we are ready to estimate 82(
d||62 (x, s)113
ds
/ Veo s), 0)d,,(((0 + a. (r) 8) D oz (e. (z 8)
2 -a.8 60 (0) 8)
Bt
1+ / || (Zu, (x, 8) 0) 0)ll (0. + Il () D (0,
< 2a
[162(1, ))
2kk
< ax 1)1(
Co (1+ |1 12)) + 22 1121 (=(B:aa) W2(21,
< 1l 11 || [1: (1)6((0r, 3B U2)))
By the Grönwall's inequality, [152 8)122 S Cp(l1 11 /12 11T1B: proof is finished.
|| at) Wa(u1, v2), and the
L
C MAIN RESULTS
C.I GRADIENT FLOW
Proof of Theorem 4.7 To prove Theorem
4.1 need estimate
V(Tr. - [[ (to) = B.nDD [[ (() (x) - y(x))² - (Fri (2) y(0)))
2
-EwNDa(Prun (Frr (x) - y(x)) (f
Pronee, (x)) + olIFr.m (n (x) Fromee (x) 1),
by (a + +c o²² = 2ac + o((e)), where o(.) denotes the higher order of the error term. Then, we
estimate (x) Sro. (zc).
F (20) - -ingin (2) = 3. f N(Zu, (x. 1), (w) h h(Zun, (xx 1), w))rra (w)
L(Znn (x. 1), (dr,(w) draa(w)) + / ypt. (M(Zm, (x, 1), h(Zu, Zun, (x, 1), w (w)
=B-
Rkb
(A)
(B)
20

---

Published as a conference paper at ICLR 2024
We estimate Zu, (x, s) in the following, in which assume 0; (1(,,))) s), 02 ()o(.,s) in the expectation. Similar to the derivation in Lu et al. (2020): Ding et al. (2022). we have
d(Zu, Zun (x, = E (a Zm, (a) s), 07) o (un (x. 8), 0i,))
ds
=E (a (o (x,, 0) - o oZro (x, 8), 01)) + +E (o mn (x), 8), 0i) o Zr.o (x, s),
=E 02 Zeeo
(2, 8) 01) Zn (x, 8) 2n (x, b)) + 000 Zuo (x, 8), 0 (0; 0ig) + o(l - tol)
-E 0200( Zuzo
(). 0% Zv, (z,s) Zun [ s)) + 000 Zuno (x. 5), 0 (0; 0% + o(lt tol)
-E 0200 Zeno
(2) 0% Zv, (I, Zuro (I)
(T., Mla) dut
-E 0 Zui, (x, 3) 0% V
(0% 5)(t - to) + o(lt - tol)
We therefore have, by the definition of q, in Eq. 17
(Ze- 2
=-fo qnio (ce, -/0mnoa
( (0, 6) (1 - lo)ds + o(Ie - (ol)
B E aano (x,, 8), 0 Vo
duto
and then we have (Z Zuro (ax, 1)l12 O(I tal). Using this fact and by the evolution of Tı in Eq. (1O). we estimate (A),
[ h(Zung (a, 1), (d) (w))
(A)
+ [ (h (M(Zv,(a) (xx, 1), - h h(Zu,, (s., 1), w)) dr (w)
BAr
=- - MZZing (a,, V) oi(TTo Un! (w) - to)dTio (w) + a(lt - tol)
8TL
We can also estimate (B), in which h is hidden in the definition of in Eq. 17
(B)
Pir. (z, 1) (Zu, Zung (x, 1) (w) + o(l - tol)
f Put (1) get
(0i, 8i(Tior Uho) (0i,:)) 8) (t (t- (t-) dreo(w) + +oltt - tol)
E EaVer (x, 01,) V
dito
-Smn xRh x [0.1]
(Dur (2x, a Looo (x, 0)-
8T Mea) (0, 8) du (0, (t - to) + oflt - -tol)-
V
8ero
Combine the estimation of (A) and (B), we have
[(TU, vi) - [(TEO (1)) = EEND, 3 ((FFro (x) ((A) + (B))
= -ynnD 3 (imo (a)) y(1r) f Jut- dre (w)d (0, s) to)
8T.(Tio: utod (w)
Z (x. 1)
87t0
8L(Fin Uso) (0, + +oll - -tol)
+ Pung (x) NeVoo(Zun s),
6y
11-00 (ī(T,D) v)
(ī(T.V) (1)
((m)) (1 (4-4) (0) + o(1 (0));
= - (0, (0
(0,s) 8)
+ Vu
5v
8x
from the definition of functional gradient in Eq.
(11) and Eq. 14. In all, the theorem is proved. n
21

---

Published as a conference paper at ICLR 2024
Proof of Proposition 4.2
We expand the function derivative:
J
si(T,, vz)
Ve
(0, s) dr((w)du((Es)
2hr xatu x(0.1
dv:
a²
Σ (frre (xt) y ( y(0,))
n²
1,==
Pe. (a Vo Zy, (as, 8),
a. 2²
b, (1),
n²
and similarly,
J/L R
(w)) (i(
|2
8² b drllw)
Vo
ne
G2(Te,
In all, the lemma is proved.
Proof of Lemma 4.6
We use the expansion of the gradient flow:
(KI(TellTo)
(KL(Te]lTO)) drp
{ SKI(TallTo)
[[(WV (T(W)) 5t((es dw 87
dw
V.
df
BAF
8Ts
dt
JRC
85e
(o (TKK((rlTT)))
( (D(T) Uz)
Te(w)
dw.
RKr
ÔT
8Ts
Similarly, we have dKL(vf
(1()
f
JKL(ve [|22)
duz
de
dt
dt
[ Jpm
si(Te, v1)
Vo
(2(()Vo
(0,
do
de:
(KL(v)ll??))
Vo of((), (0,s)
(0)
Vo
d0.
JR
dug
8y
Therefore, the proof is completed.
C.2 MINIMUM EIGENVALUE AT INITIALIZATION
Proof of Lemma 4.3 In the proof, similar to Assumption 3.3 we assume 0 = (u, w. b) 122++1
(a, w. b) E Rd+2 u, w E Rd a, b E R. initialization, we notice that (0,)) xc are standard Gaussian. Since the distribution of u, is symmetric, and independent from other parts of 0, respectively, we have dZp,, (z.
exp 11012 and To(w) exp ||س||
2
Tooww" (x, s) + b)dvo (u, w, b,
ds
Rhv
{ gdd
T Tddvlu)
/ Rtuudd oo(w" Zoo (x. 8) + b)deo (w, b, 0, Vs E [0, 1),
dp T
(xr, 8) = -a. PT 1 8) I Veo(Zun (x, 8), 0)dun
JEL.
ds
= p (x, [ uw oo(w"" Zun (x, s) + b)d w. b) 0.
Rkv
Pvo (xx, 1) 1)= Jght
V. Zeo (xc, 1), (d)((w)
awo'(w" Zm 1) + b)To(a, w. w) 0.
22

---

Published as a conference paper at ICLR 2024
From the first two equations, we have
Bu (a. 8) xz Vs
Poo (x) = Pen (x.
By the definition of G2(To, 10), we have
G2(To, σo) N1. WV w)
-ia..,))--Noon/0((((( (w, 6)), ao (X (w, d((X, (w, b)))
()) (X, (w, b)), ( (w, b)))
( (XX ()) 2 [(c. 1) b))
Let a = (x, 1), by Assumption 3.1
the cosine similarity of Ø; and æj is no larger than (1 + Cmax)/2
Then we bound Amin (G
Amiu (G 2 Amin
(x
= Amin
Σ [o((o1) Oi (xXT)
Nguuyen & Mondelli 2020 Lemma D.3)
2 log((2n)
2 14r (01) (Of-r
takinn r 2
1 Cinux
2 (41(01)²
min iE(n]
| 2r (n 1)
max (Tt,, m3)11*) (fj
[Gershgorin circle theorem]
1 Cmax
2 (
1 - (n 1)
log(2n)
2 (
1 (n 1)
r
(0)
1- (n- 1) exp(p ((o-ooo(()))
(01
where the last inequality holds by the fact that (1 Ing )" is an increasing function of r.
C.3 PERTURBATION OF MINIMUM EIGENVALUE
In this section, we analyze the minimum eigenvalue of the Gram matrix.
Lemma C.1. The perturbation of G2(T, v) can be upper bounded in the following, for any i, E [m].
Gz(T, G2(7o. vo)le, S Ca(1I 11 d, (W) + W2(u,oo)).
where G2 is defined in Section and To, satisfies Assumption 3.2
Proof of Lemma C./ We deal with G2(T, in an element- way. Let (w, w)) ~ n. be the optimal coupling Wz(T, To), the difference can estimated by
(G((T.) Go(To.Pooles
<E Whhh (Z, (2,, 1),
Vwh(Zz, (2t, - 1) - Vah (xt,, (T))
W (Z,, 12. w)))
< E V hww.h(Zii (Z,, (V.
(V.h( (26, w) - Van / Zm ( 1). wi Z., (x), wo)
23

---

Published as a conference paper at ICLR 2024
We then estimate (A) and (B) separately. The term (A) involves
V (2, 1),w) Vih( (2,, wo)ll
< (z,
Fu (z. 1), (o) (/.
Vwh(Z (x. w).lo)
<C. (z, 1)1l2 + 1) || - wollz Co (1lwol12 1) (z, 1) Zoo (z: 1)
SC.
- ((C2(1)u11):)))) + 1) wolla + (1lwol1 1 + 1) 1 a M2( vo)):
where we use Lemmas B.5 and B.6 in our proof. Besides, the term (B) involves
(Z,, S C. (l) (z. 1) + 1) ([| 112 + 1) S - (C2 + 1) (1w||2 + 1).
Therefore, by B||wo|11 = 3k: = =3(d 2).
(A) + (B) - C(Ipl a () (
- wollz + (1wol1 1)W2 vv)) + llwlll2 + 2)
<C([) |10 wolla + (|lسس + ()))
+ قااا +4)
|| a) [E wa (1 + || 12 + +4 (2((,no))
|1
| 1 d. [ 3 - woll - (E(1) 12 + ww + 4)²) + Ma(u,
<C([ 1 11
11 d, (M) ro) + (2 vo)),
since E | woll (W2((T, +o)) by the definition of optimal coupling. Since | = d -
2. 1| 2d + I. we can drop dependence of C' on || || and replace them by d. In all. the lemma is proved. Specifically, we could set
CI id, a)
: + (C) C2( 2d (11 +
Lemma C.2. If (0 E Pd, Wa(u, S Vd. and W2(T, S Vd, we have Vi,j E [n].
G2(T, C2(TO.)0)l: S Co(d,a) (Wa(ui, un) W2 To))
Proof of Lemma C.2 For any E [0, 1]. let (0", () ~ गेe be the optimal coupling of W2("",,8
|
== S ( 0 - 0,1 + ||0)1(2 2w3 vn) 2(2d + 1) S 6d +
where the last inequality holds, since (2 v8) S W2 vu), As E 1].
We also let (w, wa) क. be the optimal coupling of W2
| B|| 12 s 2 wall3 + (1 = 2r To) + 2(d + 2) s 4(d 1). Therefore, ||0||000 Gd 2, || |||| S 2.
By Lemma replacing || ||/113 with their upper bound w.r.t. d in the definition of
||> Ca(ITTIB,
d, a.), there exist Ca(d, a) satisfying Lemma C.2
For ease of description, we restate Lemma 4.4 with more details here.
Lemma C.3. vf E C(P2; (0. E E p?, Wa(u, vn) I r and Wz(T, TD) I r, we have
Amin (G2(T,
with Tmax a) min {va, AnCa a)
A
where A is defined in Lemma
4.3 and Co(d, a) is defined in Lemma C.2
Proof of Lemma
C.3
By Lemma
C.2 let
min Vd,
A
4nCold,
24

---

Published as a conference paper at ICLR 2024
By Lemma 4.3 we have Vi,j [nd].
A
(2((T - G2 volli. S Ca(d;a) a) (M2(e, W2(T, -
2n
By the standard matrix perturbation bounds, we have
Amin (2(T, Amia (G2 vu)) ||G2(T, - G2((T, vo)ll2
2 min G2(To. vn) - - nI| - G2(To, 40)
A
C.4 ESTIMATION OF KL DIVERGENCE.
Inspired by Lemma C,3
propose the following definition:
Definition C.4. Define
tmax == sop s.t.Vt [0, fol, max Po),, We(rs, ro) & Tmax):
where max is defined Lemma 4.4
By Lemma C.2 and Lemma
for < (max, we have max 1Bor [|=el O(d), and C.3
Amin (G2( 14) 2
We first prove the linear convergence of empirical loss under finite time. Lemma C.5. Assume the PDE (10)
has solution Tt E p22 and the PDE 12 has solution E for all E (max), we have
C(P2; 1]). Under Assumption 3.7 3.2
3.3
e²A V(Tr, S e (I(To, vo), (T(Tellm) S (d,(d,a)
KL(U)|lu0) V (d.(d.
(45)
A2
A2
Proof of Lemma C.5
Please see Lemma C.6
Lemma C.7 and Lemma C.8
Lemma C.6. Assume Tt, 14 is the solution to PDE
10 and
12
we have for t < tmax
Ü( ve) Se LL(T,,,
AA
where A is defined in Lemmal 4.3
Proof of Lemma C.6
By Lemma
C.3 for t < traxx Amin (G(Ti, ([]T) 2 2
aL TT)
82
3²A
382
b.
(a²
GI G2(Tu <
L(Te, S
[(To, vo).
01 n²
2n
2n
Therefore, have
M(T, v.) Se ((To,
22A 22A
Lemma C.7. Assume the PDE 12 has solution De E C(P2; 1)), and the PDE (1O) has solution
E p22 Under Assumption 3.3
3.1
3.2 then for all f. E [0, (max)) ) the following results hold,
KL < A Ck)(d,
vs [
By Gaussian initialization of uß. log vf(0) 1013 + C. and we thus have log f O. Combining this with Lemma an6 and Eq. (44, have
Proof of Lemma C.7
OKL(s)Ilm Vo
Ouf
OKT((p
-EEDD, 3 (fr.,, (x) y(x))
Øt
a of ghr
Volp 3) s)o Zn, (x, s), 0)) (Vo (Velogyū (0)
25

---

Published as a conference paper at ICLR 2024
Define
(z, 0) = a (c, (VoorZz,, (Zu, (x, s).0) Doo (m, (2,,
By the definition of (x, 0), and integration by parts, have
@KL(ville)) B BRNDD ( Boxr
0t
We can obtain the gradient of J w.r.t. 0. VaJi 0) a -.. (2) "
(2, (VV(e,s))VVooooo((
(2,,
Therefore, by Lemma B.4 and the estimate Z (a, s)ll - C2(11=113 a) from Lemma B.6 we
have
(Z (2, 0)1 S Co (1101 + 1) (a)
(Z, 0) Co (100 - aC) +1).
Therefore, we can estimate (z) 0),
VoJi (2, 0)ll2 = 2C. (Cz( ) + 1)(100122 + 1)llpm, (z, 8][(2 + C, l1: (110122 1)
s 20, (C2()
s 11 (101|22 11).
By Lemma B.1 and Lemma B.2
Bug Ji (xx 0) - Egg (z, 0o) z C( 3. || 11 Wa v6)
< & || 0)) (L(l))))))
For é E [0, (max), by Definition C.4 we have | Ta 12 = Od).. We have
[f 0) Eng Jz S C(d, ( (L())1)g)
Since Pag (a. 8) O,
Egg (x. 0) = 0.
Therefore,
(KL (16))
3 EDNDA (frmm (x) - y(x))) ( (xx
Øt
= B EnnD, nn (Sr, (2)) - (x)) Euy (Ji (xc 0) - jz 0))
3B. C(d, a) KL KL
D. (Srrey v(x))
S B a) KL(uill ([ 112))22
B. a)
KI(V/l(6) (((,,
where the last inequality holds owing to the Jesen's inequality. By the relation d2 dæ/Vi.
a ( (2V (2VRELUEllVt) s C(d, VLv,,
We have for t E [0, tmax).
((), < 22. Vo),
26

---

Published as a conference paper at ICLR 2024
Hence,
2v (I(())l)))) s c(d,a) S.' Vilvoo The )dto _
4C(d, a)
AB
Since ru( is standard normal distribution, we have
Soan (2t) 3 a
--aa
S XR u To ~'ooww² Zuo (x,, 1) + ( w, 6) 0,
and [1(x))] S (Assumption 3.1 we have [(TO, uo) S 1. Therefore, we obtain CKI(d,
KL(vi|p1)) 5 A
where CKL. is constant dependent only on d, a.
Lemma C.8. Assume the PDE
T2) has solution Ta P?? and the PDE (10) has solution Ta P?? then for all t E [0, (max), the following results hold:
Under Assumption 3.3 3.1 3.2
1 A2 CkI(d,
(T((Il(TO))
Proof of Lemma C.8
By Gaussian initialization of 70. log (w(
| 1
+ C. and have
V
(KI(T) 07.
V log + w. Therefore, by Lemma 4.6 Eq. 14 we have
(TI(TT|)
/
=- B.
(ExnDa (f VLz (xx 1),w)) (V log Te + ()).
R/r
Define
(0) EEDDD ( 1),
we have
@KL(T) -
/ Rt-
Tr (V log + w)dw = -a / Tr/ūr Vo d.
0L
We also define
((,,w) = (Z w - AZ,, 1).
By the definition of Ie(xww).
OKL(Tel(To) EEEDD I(fT. (x) Eww,))\nrT. w)).
Øt
Similar to the estimate Jf, have the estimation of as
E, Le,, w) - Er, to( w) S
and have, by setting (a, w. b), have
Erg Jo w) = E,, (-vn (wh((, w + Awh(w,
Eic.w.s)-- 'T)TTTTTTTTT + b) al ( Ww x + b))w abo'(w" 6))
+ ((a.m..)~-V aow x + b) + ao (w + b)) 0.
Therefore, we obtain the KI. divergence in similar fashion.
(L(Te])7o) & CKL(d, a)
A
27

---

Published as a conference paper at ICLR 2024
Lemma C.9 (Lower bound the KL divergence). For any T, E P?² v,vl E C(P2, (0,1), 1). if
T',1/ v/ satisfy the Talagrand inequality T fref Lemma B.2 We have the lower bound for the KL
divergence of T, and v, v/, such that for constant Clow [ 1 11 11:
V VKI(TITT") + (ofrr)/KKKKOO// (L(()))') Eww h b (x, 1), w) Ew'nr h 1Z 1),w")
Clow 11 E 1 113
Proof of Lemma
C.9
We have the following estimaation
Bwwr / (z, 1), w) lEw' // [Z (x. 1), w),
(B 10 (Z (2x, 1), w) Ew''T (Z (x, 1), (.'w)
(A)
+ ((Zy 1), w)
Bewn (Z 1), w')),
(B)
By Lemma B.4
we have
W (2; 1), (,, S C. -0) Zaz, 1) + 1) ([ww12 + 1),
and by Lemma B.
Lemma B.6
B.6 and Lemma B.2 we have
(A) SCa (1) (z, 1)ll2 1/22222-102 1) mmx || || 1 W2(T, (')
<C. (Cz(
+ max
(
<2C (Cz (1:1 a)) + 1) m| || 12 KL(T|TT).
Besides, by Lemma
B.5 and Lemma B.2 we have
(B) S B Co (1 1) (1)
[., (ar, 1)1l2) s (1
W W2 22)
s 211 11 (KL(v|/))
We let Clow be
Clow (1 | || 12= || aa) =mi =2in (Cz( (1:: a) + 1) max (1| 11 |1-1 2(11 1) Cz(llen 1b [12)
Therefore, we have
V KL((T|TT) KL(U|I'') B h hZ 1), - Ewln h ZZ (x, 1,,
Clow (1 | 1u a))
Since || | || |1 (d). we have that the average movement of the KL divergence is on the same order as the change in output value.
12) has solution E PP2, and the PDE (TO) has solution T. E p2 Under Assumption 3.3 3.1 3.2 then for all t E [0, tmax). We have the lower bound for the KL divergence of and vt. have for constant Crow (d; a), such that
Lemma C.10. Assume the PDE
Eusr (Zu, (xr 1),
/ (TT(TT]) (KT((ulloo) Z
Clow a)
Proof of Lemma C.10 By the definition of Tmax < Vd. and the proof of Lemma C.2 we
have ||
[ lo Od). and we can directly obtain ||Tol d + 2, ||ol| = 2d + 1,
and E (Zen (o (x. 1), wa)
Besides, Gaussian initialization satisfies T(2) condition in
Lemma B.2
We have, by Lemma C.9
(LL( + (KL(v))u)) 2 E 11( M(,,, (xx, 1),
Clow (d: a) where Clow (d: a:) is a constant depending on d, a derived from Crow (1| || || || a). n
28

---

Published as a conference paper at ICLR 2024
1/Cxx(dd.a) have mma OC.
Lemma C.11. Under the Assumptions in Lemma
C.7 let 3
Afmax
Proof of Lemma
C.I. Otherwise, have the following inequality, for Vi mmaxi
W2(1 v0) s 2XLL(()))))) s VCLldd,)), Vs E [0, 1].
2
AB
Therefore,
We(vi, & AB VC
According to the definition of max, we have W2(U1, vo) S Tmax. Let
B 4VCKI,(d,
Armux
we have W2(vs, S 2m E [0, fmax), which contradict to the definition of fmax Defini-
tion C.4
Proof of Theorem 4.7
Combine the results of Lemma C.11
Lemma C.8 and Lemma
C.7 we prove
the theorem.
C.5 RADEMACHER COMPLEXITY
Proof of Lemma 4.8 Let y be a parameter whose value will be determined later in the proof. Let Ti, 1 S i S n be the ii.d. Rademacher random variables,
8 7
R2 (FLL (r))
E,
sup = vo)
E,
n (Z 1),
8
E,
sup (ull)()))
log ETo exp
h Ze Zz 1),
7
1=1
8
r + B.y log Brg exp
sup v:KL(vllpo)Sr
m/ch( Zu 1),w)
7
5
8_ 7
log E,, E., exp
1 sup
Σ Ahh(Zu 1), w)
n1 p:K((v))))
where the first inequality is followed by the Donsker- Varadhan representation of KL-divergence in Lemma B.3 The second inequality follows from the increasing function log(-), exp(-), and
sup, F, (r. S Fr sup, fxx y). for general variable v, y and function f. The third inequality
follows from log(-)'s ) convexity.
By Assumption
3.3 where w = (a, w.,),,), b), we have
11(11,)) - ((22 C ||21 -22l2 al|w]]22
We further estimate
n
Σ
nh( Zu [ 1), ) Σ
nh (2,, w)
1=1
<Cnn a || ||
[,, 1) 1
Zun (2:;; 1)ll2
<Cin a w la Cz(11u113 |1 11: a) W2(v, vo)
<Cin a a|[w||2 Cz((I//( (/ 11: 2//
given KT((vleo)) & r. Further, have | 113
2d + 1, and have || < 6d 2.
29

---

Published as a conference paper at ICLR 2024
In the following, Ca := 201 C2(6d 22 a;) to denote the constant.
(FL - 8 + log Ry Byy exp > (p (2 1),w) y. Cavi allerll2 ))
B
r+ log Bro exp (7 Cavi -euv-a/lllla E, exp
Σ n 1ih(Zzn (z (x 1)
22
r log Bro exp y. Cavr CavFralleolla +
Σ
h po (ac,, 1),
2n
(r+] log Ero exp (27 Cavr al 20
1 log Erto exp
h2 (Zop (2is, w)
n
where the first inequality follows from the previous bound; and the second inequality follows from the
tail bound: EE,, exp(_.=1 a;m)) I exp( 2i af); and the last inequality follows from the Cauchy ineqaulity.
Still,
we set the decomposition w = (a, (e, E Rd+22 we have Zuo (2;, w) =
|ooo(w Zuo (z 1) + 6)1 s |a|C1. We have
(r)
log Ba exp (29299)
<
(r+
r
log E (). exp (27 CayF alww|12) 2
We remark that
log E,-A(o.1)) exp(Ct)) log(1 2C) s 2C
2C
log (o)( exp(Ct) exp(C2 /2).
where the first inequality holds for C S 1/4. Therefore, setting 7 vnr/C1, we have
2 22aa09
log Ba- exp
- log(1 -2r) S r,
n
log Bea,u (a...))aN00. exp (2 Cavr al allw|lz) 1
log exp(2
s - log(1 - 4nr 2nr =nr (Ca/C1)
where the inequality holds iff < and nr (Ca/Cr) We set ro min C//CaJ, we have for Vr S ro.
R(Fxl(F)) < 8/7(r 2nr +r) 3 Vr/n-2(CC +
where S hides constant.
Theorem C.12 (Rademacher complexity). For any 5 0. with probability at least 5, the following bound holds V. E Fri.
log(2/8)/2n 6 Exwoy lo- (f- v())) < 4R, 6y
+ v ED, (Fr y(x))²
Proof of Theorem
C.12 We introduce the additional loss function
((f.v) = max mii (1 2yyf. 1), 0}.
By definition, have Z is 2-Lipschitz in the first argument, and (o-1(f.v) f(f v) If v/.
30

---

Published as a conference paper at ICLR 2024
for any f E I and {=1}. Using the standard properties of Rademacher complexity, have that with probability at least 8, for all f E FKL(r)
log(2/8)
Bxx {(. S ED, ((S.)) + (FK(((()))) 6V
2n
Therefore, we have
1og(2/6)
Epx fo-1(f))))) Exx EDD v)² 4R, (FkL((r) + 6V
2n
Lemma C.13. Let Ty C(P3; [0, 1J) and My E pd be the ground truth distributions, such that, y(æ) == Eurnr h( Zu (z, 1), w)
Then, we have the bound for the KI. divergence, for Tx: u satisfying [.(TT, vx) = O, - 3 3x (T + x2
Proof of Lemma C.13 We assume that (T.) v is the solution to the following minimization prob- lem:
(r,, = arg min D(r, + A(KL(T||TO) + KL(v(lu)))
7M
Consider the mixture distribution F, i be defined
(f,D) = 3
(To- 1/) + (T)
and we have
[ (3-1
I(F,D) = EnwD
B. h h(Zro w), + BBBurrh (2, w) - y(x)
B
2 0 3 y y/ 0,
= EEwD.
and by the definition of T v we obtain
i(r. 2Z) A(KT((7)1lm) KT(())Po)) & ((F + A(KT(FIlm)) + KT(())))))
This leads
Ī(T v.) < + KL(Ti|170) + KL (v 1100) I mL + (L(
Taking À 0, we have 7 Tu. and v + v+. Accordingly, have
T( v.) = (L(T.IITo) KL KL() + (L(())))
We explicitly compute the KI. divergence such that
(L()) x = / (2-1 (( + Ty(W)
- 1) 2
du (Tyloo)
Brolw)
and similarly, we have
KL(D|Pp)) s 28-22(uul11)))
Finally, conclude the proof.
31

---

Published as a conference paper at ICLR 2024
Proof of Theorem 4.9 By Theorem C.12 for r > 0, for any 8 0. with probability at least 6,
the following bound holds VIT E (r):
Exgpp lo (0-1 v(x)) - 4R, (FKL( + By log((//0)) VED, (frw(z)) y(z:))2
By Lemma C.13 and the definition of ro in Lemma 4.8 we set such that
( +
( x((uullo)) I ru.
i.e.
B (2(Ty + (mgllle)) = O(vT).
0((T))
and
fr... E (x + +x (o))))))).
Therefore, we apply the Rademacher complexity bound in Lemma 4.8 and we have
2 FTy (o) (v [))) O(1/Vn).
(F
(2(y)1lo)))) 2 3V
n
where 3 cancels out. By Lemma C.13 we have [[Te, v.) 0. Finally,
Ex-ux lo- 1 (frx()), (11 VT) + 6//oo(2/0)/22
C.6 EXPERIMENTS
We validate our findings on the toy dataset "Two Spirals", where the data dimension 2. We use neural ODE model (Poli et al., (02T) to approximate the infinite depth ResNets, where we take the discretization L = 10. The neural ODE model and the output layer are both parametrized by two-layer network with the tanh activation function, and the hidden dimension is M = K 20. The parameters
ResNet encoder and the output layer are jointly trained by Adam optimizer with an initial learning rate 0.01. We perform full-batch training for 1,000 steps the training dataset of size Reriin and test the resulting model on the test dataset of size noret = 1024 by the 0-1 classification loss. We experiments over 3 seeds and report the mean. We fit the results (after logarithm) by ordinary least squares, and obtain the slope is 1.022 with p-value 10-5, as shown in Figure [ That means, the obtained rate (1/nn), which faster than our derived ((L/Vn) rate. We hope could use some localized schemes, e.g., local Rademacher complexity, to close the gap.
10
1e.. tent arme
-5
-10
10a
10
Figure 1: Left: "Two Spirals" datasets. Right: La-1 test error v.s. the training dataset size Perain
(blue), OLS fitted line (red) which is close to the (11/n) rate with p-value 10
32


Reviews:
[{"review_id": "lztQgDwo8i", "reviewer": "Reviewer_JCPH", "summary": "The authors study the convergence of the gradient flow for infinitely wide and deep ResNets under a mean-field regime utilizing the technique developed by [Lu et al. (2020)] and [Ding et al. (2021, 2022)]. Specifically, they show the global convergence based on the estimation of the smallest eigenvalue of (varying) NTK matrix. Moreover, the generalization analysis using Rademacher complexity for the trained network is also provided.", "questions": "It would be nice if the authors could prove that feature learning certainly appears in the proposed setup.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This work studies the mean-field regime of the neural network, which has attracted attention due to the presence of feature learning. The optimization analysis with the convergence rate is quite challenging especially when not using the entropy regularization. This work nicely combines the mean-field analysis of infinitely deep and wide ResNet ([Lu et al. (2020)], [Ding et al. (2021, 2022)]) and the estimation of the smallest eigenvalue of NTK matrix, and derives the convergence rate. I think this work is interesting and suggests a new way of analyzing the deep ResNets under a mean-field regime.\n\n- The paper is well organized. The presentation is very clear and easy to follow.", "weaknesses": "- In fact, the mathematical method developed is quite interesting. However, I am not completely convinced whether the optimization dynamics considered in this paper indicates feature learning. The convergence analysis is based on the positivity of $G_2$ instead of $G_1$. Here $G_2$ is the Gram matrix corresponding to the last layer. Therefore, I suspect that the convergence property is fully relying on the optimization of the last layer. That is, it may not be possible to distinguish between the proposed dynamics and the optimization of the last layer only. Essentially, feature learning is caused by training the input layer rather than the output.\n\n- The theory of the paper does not seem to work when the last layer is frozen in contrast to many work for two-layer mean-field neural networks. This concern is related to the one above.\n\n- Missing reference. The following work is also closely relevant to the paper. \n\nZ. Chen, E. Vanden-Eijnden, and J. Bruna. On feature learning in neural networks with global convergence guarantees.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yiBP38qSEk", "reviewer": "Reviewer_x2zh", "summary": "The paper studies the optimization and generalization properties of infinitely wide and deep residual networks in the mean-field limit, where the weights are described by a probability density. The network is trained by gradient flow over the empirical risk. The paper shows two main results: the convergence to the global minimum of the empirical risk, and a generalization bound for the trained network. The convergence result is obtained by lower-bounding the smallest eigenvalue of the Gram matrix of the gradients. The generalization bound is obtained by bounding the KL divergence between the initial weight distribution and the weight distribution after training, then studying the Rademacher complexity of the family of models in a ball around initialization for the KL divergence.", "questions": "In your model, the output $y$ is a deterministic function of the input $x$. Could this assumption be relaxed (eg, adding noise to $y$)?\n\nAlso see weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper provides a very interesting take on the important question of jointly understanding the training dynamics and generalization properties of deep neural networks. The final result is strong since it shows a bound on the population risk of trained deep networks under reasonably mild assumptions. The paper seems technically solid, although I have not read the proofs in full details.\n\nFurthermore, the paper is easy to read and well-presented. The paper presents all the results in a unified setting, which is enjoyable. I also liked that the three main contributions are well-separated throughout the paper (in the explanations, the mathematical statements and the proofs). This allows the community to easily build on proof techniques presented in the paper.\n\nTo my knowledge, the idea of bounding the Rademacher complexity through a bound on the KL divergence wrt the initialization is novel.\n\nAll in all, this makes this paper a very interesting contribution for the community.", "weaknesses": "I have two main questions for the authors. [Update on Nov 22: the authors have answered to all the questions, except the first one, see thread at the top of the page].\n1. The proof of global convergence relies on lower-bounding the smallest eigenvalue of the Gram matrix of the gradients of the two-layer neural network in the end of the architecture. There is no guarantee on the Gram matrix of the gradients of the infinite-depth ResNet part of the architecture. **For this reason, I am wondering if the proof would still hold if the ResNet part of the architecture was fixed to its initial value or even altogether removed.** If this is the case, I think this should be stated more clearly, and in my opinion questions a bit the narrative of the paper which gives an important focus on the ResNet part of the architecture, whereas I do not fully understand at this point the role it plays in the proof.\n2. The generalization bound is proven in the case where the radius $r$ of the KL ball decreases in $O(1/n)$. The paper proves that, with the appropriate scaling of $\\beta$, the trained network remains in this ball, and thus the generalization bound holds for the trained network. This means that the distribution of the weights is required to be increasingly close to the distribution at initialization as the sample size increases. **For this reason, I am wondering if the regime where the generalization bound holds can really be though of as feature learning.** While this is not a problem in itself, it questions in my opinion the statement in the conclusion of the paper that “this is the first paper to build a generalization bound for deep neural networks in the so-called feature learning regime”. In my opinion, either more caution in the phrasing or more explanations would be highly welcome.\n\nI would be willing to raise my score if the remarks are answered by the authors, and the narrative of the paper changed accordingly if appropriate.\n\nA less important remark regards Theorem 4.11. The rate of convergence of the empirical risk in Lemma 4.7 actually depends on $n$ through $\\beta$ (which appears both in the exponential and implicitly in the loss at initialization, since the output of the network at initialization is proportional to $\\beta$). I don’t think this dependence is taken into account by the authors in Theorem 4.11, or at least it is not stated since the authors state that the empirical risk decays as $O(e^{-t})$. While I believe that the stated theorem is correct, the result could probably be sharpened (since the rate of convergence is actually $O(e^{-nt})$, although the magnitude of the loss at initialization also depends on $n$ and would have to be taken into account). In any case, a proof would be beneficial.\n\n**Literature review**\n\nThere are several papers which show global convergence of deep residual networks and which are not cited. In particular, the beginning of Section 4 suggests that this paper is the first one to prove a rate of convergence with mild assumptions. There have actually been other papers proposing this, see e.g. https://arxiv.org/abs/1910.02934, https://arxiv.org/abs/2204.07261, https://arxiv.org/abs/2309.01213. Please rephrase accordingly the beginning of Section 4. Also, I believe Barboni et al. do not “assume that the model parameters are close to the initialization”, they prove it to be the case in their setting, which is a very different statement.\n\n**Minor remarks that did not influence the score**\n\n1. page 2, second paragraph: $\\tau$ et $\\nu$ should be swapped.\n2. First paragraph of Section 2.1 is not very clear.\n3. page 3: “nerual” -> neural.\n4. page 3: in your ResNet model, you assume a scaling in $1/L$. While this is necessary to obtain an ODE limit and usually assumed in the corresponding literature, it departs from the actual practice. I think it would be interesting to mention this. See https://arxiv.org/abs/2105.12245, https://arxiv.org/abs/2206.06929 for a related discussion. \n5. page 4: the sentence “We introduce a trainable MLP …” is not very clear. What do you mean by “zero initialization of the network’s prediction”?\n6. page 5, equations (11), (14), (15): $D$ should be $D_n$.\n7. pages 8 and 9: who is $\\Lambda$? Should it be $\\Lambda(d)$?\n8. page 9, after Theorem 4.10: $r_0$ is defined but was already defined in the Theorem.\n9. There are duplicated entries in the references, eg Chizat and Bach, and (probably) Ding, Chen, Li and Wright. Please double-check this.\n10. Lemma B.2: $N$ should be $n$? This typo is present in several places throughout the proof.\n11. Lemma B.7: I think the infinite norm is undefined. It is possible to infer what it means from context, but please define it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MVJjorzMei", "reviewer": "Reviewer_8ogw", "summary": "The manuscript studies convergence rate and generalization in infinitely deep and wide ResNets within the mean-field scaling. Utilizing this limit, they recast the neural network's dynamics as a dynamics over weight probabilities per layer. They obtain the analog of the lowest NTK eigenvalue at initialization, and show that it remains of that scale up to some range (r) in Wasserstein-2 distance. This establishes a lower bound on the convergence rate in this r-region. They then turn to bound the KL divergence in this r-region (which they can also characterize by a maximal time t_0 for which the Wasserstein-2 distance remains below r). Finally, they characterize the generalization gap using a Rademacher complexity bound where they use the fact that at init the complexity is zero and afterwards it can be bounded by the KL divergence. All in all, they claim to obtain that within the r-region the generalization gap goes as 1/\\sqrt{n}.", "questions": "1. Can the authors clarify what their results mean in practical terms? Fixing hyper-parameters how do they expect r to scale t_max and n? How do they expect the training error to decrease with r?\n\n2. Can the authors provide some supporting experiments showing how tight are their bounds?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper studies convergence rate and generalization in a challenging setting, a deep ResNet with no assumption on the data-set measure. \n\nIt is a technically challenging work which requires skill and effort.", "weaknesses": "Regarding weaknesses, I found several of those.  \n\n1. Perhaps this is a necessity for using such learning-theory tools, but I found the use of r-regions somewhat misleading. Fixing the r-region, it seems to be that, in one way or another, one is limiting the training time. Thus qualitatively, one is limiting the number of data points as training for a very short time on infinitely many data points is not much different than training for such a time on a finite large number of data points. If so what is the meaning of taking large n here if we don't give the network enough time to learn these points?\n\n2. Similarly confusing for me was the claim that $R_n(F) = \\beta \\sqrt{r/n}$ when \\beta scales larger than $\\sqrt{n}$. If so it seems that $R_n(F)=O(n^0)$ and not $1/\\sqrt{n}$. \n\n3. The authors' results do not reflect the complexity of the task of the dataset measure. While this can also be viewed as a positive point, I find it hard to believe that one can provide tight or even order-of-magnitude estimates on actual neural networks (in the regime where they generalize well) by putting so little data in. While this is not a criticism of the specific work under consideration, it does weaken its broader appeal in my mind.  \n\n4. With relation to point number 3, some numerical experiments (to the very least on toy data sets) showing how accurate these bounds are would be useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)