Title: Analyzing Generalization of Neural Networks through Loss Path Kernels
Venue: NeurIPS 2023
Accepted: yes
Keywords: generalization

deep learning theory

neural tangent kernel

neural architecture search
Abstract: Deep neural networks have been increasingly used in real-world applications, making it critical to ensure their ability to adapt to new, unseen data. In this paper, we study the generalization capability of neural networks trained with (stochastic) gradient flow. We establish a new connection between the loss dynamics of gradient flow and general kernel machines by proposing a new kernel, called loss path kernel. This kernel measures the similarity between two data points by evaluating the agreement between loss gradients along the path determined by the gradient flow. Based on this connection, we derive a new generalization upper bound that applies to general neural network architectures. This new bound is tight and strongly correlated with the true generalization error. We apply our results to guide the design of neural architecture search (NAS) and demonstrate favorable performance compared with state-of-the-art NAS algorithms through numerical experiments.

Analyzing Generalization of Neural Networks through Loss Path Kernels

Abstract

Deep neural networks have been increasingly used in real-world applications, making it critical to ensure their ability to adapt to new, unseen data. In this paper, we study the generalization capability of neural networks trained with (stochastic) gradient flow. We establish a new connection between the loss dynamics of gradient flow and general kernel machines by proposing a new kernel, called loss path kernel. This kernel measures the similarity between two data points by evaluating the agreement between loss gradients along the path determined by the gradient flow. Based on this connection, we derive a new generalization upper bound that applies to general neural network architectures. This new bound is tight and strongly correlated with the true generalization error. We apply our results to guide the design of neural architecture search (NAS) and demonstrate favorable performance compared with state-of-the-art NAS algorithms through numerical experiments.

1 Introduction

Deep learning models have been increasingly used in applications with significant societal impact. Therefore, it is crucial to ensure that these models perform well not only on the training data but also on the new and unseen data. Classical learning theory attributes the generalization ability of machine learning (ML) models to the small complexity of the hypothesis class [56]. However, modern ML models, such as deep neural networks (NNs), can have billions of parameters yet still exhibit strong generalization abilities [26, 9]. This is because various elements of the learning algorithms, including optimization methods, training data, and neural architectures, can all influence the inductive bias, which in turn shapes the generalization abilities of neural networks [29, 40]. While the overall hypothesis class may be large, the 'effective domain' of this class, which ultimately determines the model's generalization abilities, is often much smaller [54, 24, 50, 15]. Hence, it is vital to develop algorithmic generalization bounds to capture this effective domain of the hypothesis class.

There has been significant work investigating the generalization of neural networks in their infinitewidth regime through kernel methods [28, 3, 32, 2, 11]. They showed that an infinite-width NN

trained by gradient flow and squared loss is equivalent to a kernel regression with neural tangent kernel (NTK) [28, 3]. Moreover, Arora et al. [2], Cao & Gu [11] further characterized the generalization behaviors of such ultra-wide NNs by deriving data-dependent and NTK-based generalization bounds. However, they only considered ultra-wide fully connected NNs with a square (or logistic) loss function. In practice, NNs are usually not ultra-wide and have more complex architectures such as CNNs. Hence, it is crucial to establish generalization bounds that hold in a more general setting.

In this paper, we analyze the generalization capability of NNs trained using (stochastic) gradient flow across a wide range of NN architectures. Our key technical contribution is to establish a new connection between the loss dynamics of (stochastic) gradient flow and general kernel machines with a special kernel that we named the loss path kernel . This new kernel calculates the similarity between two data points by integrating the inner product of the loss gradient evaluated at these points, along the path determined by the gradient flow. Based on this connection, we develop a novel generalization bound by analyzing the complexity of kernel machines induced by various training sets. Our generalization bound is tight and can be applied to a broad class of NN architectures, not restricted to ultra-wide NNs (see Table 1 for a comparison with existing results derived from NTK theory). Numerical experiments demonstrate that our bound maintains a strong correlation with the true generalization error of NNs trained with gradient descent (GD) (see Figure 1 and 2 in Sec. 6 & 7). Given this observation, we use our generalization bound to guide the design of neural architecture search (NAS) and demonstrate through numerical experiments that our approach can achieve a favorable performance compared with state-of-the-art training-free and minimum-training NAS algorithms [37, 13, 39]. In summary, our contributions are:

In Sec. 4.1 and 5, we show for the first time that the loss of NNs trained by (stochastic) gradient flow is equivalent to a general kernel machine. This result enables us to investigate the generalization capability of NNs from the perspective of kernel theory.

In Sec. 4.2 and 5, we derive tight generalization bounds for NNs based on the aforementioned equivalence. Our result is very general as it holds for any continuously differentiable NN architectures including finite-width and infinite-width NNs. Experiments demonstrate that our bounds are tight (4690 × tighter than existing norm-based bounds and 55 × tighter than existing NTK-based bounds as shown in Appendix A.2) and highly correlated with the true generalization error.

In Sec. 6 and 7, we apply our theory to study special cases including infinite-width NNs, stable algorithms, and norm-constrained NNs. We apply our bound to guide the design of NAS. Numerical experiments demonstrate that our approach achieves a favorable performance compared with stateof-the-art NAS algorithms on NAS-Bench-201 benchmark [19].

2 Related Work

Generalization theory in deep learning. Generalization is a crucial aspect of deep learning theory and various techniques have been proposed to study it. For example, Bartlett et al. [7] derived tight bounds for the VC dimension of NNs with ReLU activation functions. There is also a line of work that measures the capacity of NNs based on different norms, margins [5, 41, 6, 44], and sharpness-based measures [42, 43, 1] to explain the generalization behaviors of NNs. Additionally, there are theories studying the generalization of NNs from PAC-Bayes [35, 22] and information-theoretical approach

[48, 58]. For example, Dziugaite & Roy [22] numerically evaluated and optimized the PAC-Bayes bound of stochastic NN and obtained a non-vacuous generalization bound. In contrast, we study the generalization of NNs by building a new connection between NNs and kernel machines. We refer readers to Valle-Pérez & Louis [55] for a more comprehensive review of the generalization theory of NNs.

Neural tangent kernel (NTK). NTK was first introduced in Jacot et al. [28], where the authors demonstrated that a fully-trained, infinite-width NN follows kernel gradient descent in the function space with respect to the NTK. Under gradient flow and squared loss, the fully-trained infinite-width NN is equivalent to kernel regression with the NTK [28, 3]. Chen et al. [14] further established the equivalence between infinite-width NNs and regularized kernel machines. Arora et al. [2] studied the generalization capacity of ultra-wide, two-layer NNs trained by GD and square loss, while Cao & Gu [11] examined the generalization of deep, ultra-wide NNs trained by stochastic gradient descent (SGD) and logistic loss. Both studies derived generalization bounds of converged NNs based on NTK. Besides, Huang et al. [27] studied the convergence and generalization of PAC-Bayesian learning for deep, ultra-wide NNs. Later, Domingos [17] showed that every model trained by gradient flow is a 'kernel machine' with the weights and bias as functions of input data, which however can be much more complex than a typical kernel machine and our general kernel machine in Definition 2. Chen et al. [14] showed that every NN trained by gradient flow is a general kernel machine but their kernel is valid only in very limited cases - when the loss gradient of output is a constant. Otherwise, the kernel is not symmetric and not valid. In this paper, we consider an equivalence between the loss of NNs and general kernel machines, which resolves the previous asymmetric problem of the kernel function and also makes the generalization analysis of multi-outputs easier.

Neural Architecture Search (NAS). NAS aims to automate the discovery of top-performance neural networks to reduce human efforts. However, most existing NAS algorithms require heavy training of a supernet or intensive architecture evaluations, suffering from heavy resource consumption [47, 34, 18, 33]. Thus, it is crucial to develop training-free or minimum-training NAS algorithms to reduce the computational cost and select the best architecture at the same time [37, 13, 39]. Since our generalization bound has a strong correlation with the true generalization error, we apply it to design a new minimum-training NAS algorithm. We demonstrate in Table 2 that with a simple random search algorithm, our approach can achieve a favorable performance compared with state-of-the-art training-free and minimum-training NAS algorithms.

3 Kernel Machine and Loss Path Kernel

In this section, we define notation, provide a brief overview of kernel methods, and introduce the main concept of interest-the loss path kernel.

3.1 Preliminaries

Consider a supervised learning problem where the task is to predict an output variable in Y ⊆ R k using a vector of input variables in X ⊆ R d . Let Z ≜ X × Y . We denote the training set by S ≜ { z i } n i =1 with z i ≜ ( x i , y i ) ∈ Z . We assume each point is drawn i.i.d. from an underlying distribution µ . Let X = [ x 1 , · · · , x n ] T ∈ R n × d , Y = [ y 1 , · · · , y n ] T ∈ R n × k , and Z = [ X , Y ] ∈ R n × ( d + k ) .

We express a neural network in a general form f ( w , x ) : R p × R d → R k where w ∈ R p represents its parameters and x ∈ R d is an input variable. The goal of a learning algorithm is to find a set of parameters that minimizes a population risk L µ ( w ) = E z ∼ µ [ ℓ ( w , z )] where ℓ ( w , z ) ≜ ℓ ( f ( w , x ) , y ) is a loss function. Throughout this paper, we assume that ℓ ( w , z ) ∈ [0 , 1] and is continuously differentiable. In practice, the underlying distribution µ is unknown so the learning algorithm minimizes an empirical risk on the training set S instead: L S ( w ) = 1 n ∑ n i =1 ℓ ( w , z i ) . The generalization gap is defined as L µ ( w ) -L S ( w ) . The loss gradient with respect to the parameters w is ∇ w ℓ ( w , z ) = ∇ w f ( w , x ) ⊤ ∇ f ℓ ( f ( w , x ) , y ) ∈ R p × 1 . Our analysis only requires that ℓ ( w , z ) is continuously differentiable w.r.t. w and z so f can be either a fully connected neural network or a convolutional network or a residual network.

3.2 Kernel Method

Kernel methods [16, 51, 53] search for linear relations in high-dimensional feature space by using kernel functions. Rather than computing the coordinates in the feature space explicitly, they only need to calculate the inner product between data pairs, making it computationally easier.

A kernel is a function K : X × X → R , such that for all x , x ′ ∈ X , K ( x , x ′ ) = ⟨ Φ( x ) , Φ( x ′ ) ⟩ , where Φ : X → F is a mapping from X to an (inner product) feature space F .

Proposition 1 (Shawe-Taylor et al. [53]) . A function K : X × X → R , which is either continuous or has a finite domain, is a kernel function if and only if it is a symmetric function and, for any finite subset of X , x 1 , · · · , x n ∈ X , the matrix K ( X , X ) is positive semi-definite, where K ( X , X ) is a n × n matrix whose ( i, j ) -th entry is K ( x i , x j ) .

Definition 1 (Kernel machine) . Let H be the reproducing kernel Hilbert space (RKHS) corresponding to a kernel K ( x , x ′ ) = ⟨ Φ( x ) , Φ( x ′ ) ⟩ . A kernel machine g : X → R is a linear function in H such that its weight vector β can be expressed as a linear combination of the training points, i.e. g ( x ) = ⟨ β , Φ( x ) ⟩ + b = ∑ n i =1 a i K ( x i , x ) + b , where β = ∑ n i =1 a i Φ( x i ) and b is a constant. The RKHS norm of g is ∥ g ∥ H = ∥ ∑ n i =1 a i Φ( x i ) ∥ = √∑ i,j a i a j K ( x i , x j ) .

Next, we introduce general kernel machine, which generalizes the concept of kernel machine.

′

Definition 2 (General kernel machine) . A general kernel machine g : X → R with a kernel K ( x , x ) is g ( x ) = ∑ n i =1 a i K ( x i , x ) + h ( x ) , where h : X → R is a function of x . When h ( x ) is a constant, g ( x ) reduces to a kernel machine in Definition 1.

3.3 Neural Tangent Kernel and Loss Path Kernel

Neural tangent kernel (NTK) has been introduced by Jacot et al. [28] to establish an equivalence between infinite-width NNs and kernel regression. After then, there is a growing line of work applying NTKtheory to study properties of over-parameterized NNs, such as optimization convergence [21, 20] and generalization capability [2, 11]. The neural tangent kernel [28] associated with a NN f ( w , x ) at w is defined as ˆ Θ( w ; x , x ′ ) = ∇ w f ( w , x ) ∇ w f ( w , x ′ ) ⊤ ∈ R k × k . Under certain conditions, such as infinite width limit and NTK parameterization, the NTK converges to a deterministic limit kernel Θ( x , x ′ ) · I k that remains constant during training: ˆ Θ( w ; x , x ′ ) → Θ( x , x ′ ) · I k , where Θ( x , x ′ ) : R d × R d → R is a scalar kernel and I k is a k × k identity matrix. Next, we introduce the main concepts of interest in this paper: the loss tangent kernel and the loss path kernel . They are central to characterizing the generalization behaviors of NNs trained by (stochastic) gradient flow.

Definition 3 (Loss Tangent Kernel (LTK) ¯ K ) . The loss tangent kernel associated with the loss function ℓ ( w , z ) is defined as ¯ K ( w ; z , z ′ ) = ⟨∇ w ℓ ( w , z ) , ∇ w ℓ ( w , z ′ ) ⟩ ∈ R .

The LTK ¯ K has a natural connection with the NTK ˆ Θ by applying the chain rule:

Next, we introduce the loss path kernel, which integrates the LTK along a given path of the parameters. Later, we will characterize this path via the gradient flow dynamics.

Definition 4 (Loss Path Kernel (LPK) K T ) . Suppose the weights follow a continuous path w ( t ) : [0 , T ] → R p in their domain with a starting point w (0) = w 0 , where T is a predetermined constant. This path is determined by the training set S and the training time T . We define the loss path kernel associated with the loss function ℓ ( w , z ) along the path as K T ( z , z ′ ; S ) ≜ ∫ T 0 ¯ K ( w ( t ); z , z ′ ) d t .

In Appendix B, we show LTK is Riemann integrable so the integral in the above definition is welldefined. Intuitively, the LTK ¯ K ( w ; z , z ′ ) measures the similarity between data points z and z ′ by comparing their loss gradients when evaluated using a fixed neural network parameter w . The LPK K T ( z , z ′ ; S ) measures the overall similarity during the entire training time.

3.4 Rademacher Complexity

Rademacher complexity [52] measures the complexity of a hypothesis class. It takes into account the data distribution and is a central concept in statistical learning theory. Next, we recall its definition and a generalization upper bound via Rademacher complexity.

Definition 5 (Empirical Rademacher complexity ˆ R S ( G ) ) . Let F = { f : X → R k } be a hypothesis class. We denote G as the set of loss functions associated with each function in F , defined as G = { g : ( x , y ) → ℓ ( f ( x ) , y ) , f ∈ F} . The empirical Rademacher complexity of G with respect to a sample set S is defined as: ˆ R S ( G ) = 1 n E σ [ sup g ∈G ∑ n i =1 σ i g ( z i ) ] , where σ = ( σ 1 , . . . , σ n ) and σ i are independent uniform random variables taking values in { +1 , -1 } .

Theorem 1 (Theorem 3.3 in Mohri et al. [38]) . Let G be a family of functions mapping from Z to [0 , 1] . Then for any δ ∈ (0 , 1) , with probability at least 1 -δ over the draw of an i.i.d. sample set S = { z 1 , . . . , z n } , the following holds for all g ∈ G : E z [ g ( z )] -1 n ∑ n i =1 g ( z i ) ≤ 2 ˆ R S ( G )+3 √ log(2 /δ ) 2 n .

4 Gradient Flow

In this section, we establish a new connection between the loss dynamics of gradient flow and a general kernel machine equipped with the LPK. Using this result, we introduce a new generalization bound by analyzing the complexity of the collection of kernel machines induced by all possible training sets. Our analysis applies to a wide range of neural network architectures, as long as they are continuously differentiable. Our numerical experiments validate the tightness of our bound and its strong correlation with the true generalization error.

4.1 Loss Dynamics of Gradient Flow and Its Equivalence with General Kernel Machine

Consider the gradient flow dynamics (gradient descent with infinitesimal step size):

The above ODE is well-defined for a wide variety of conditions, e.g. local Lipschitz-continuity of the gradient or semi-convexity of the loss function [49, 23]. Next, we establish its connection with the general kernel machine (KM) in the following theorem.

Theorem 2 (Equivalence with general KM.) . Suppose w ( T ) = w T is a solution of (1) at time T with initialization w (0) = w 0 . Then for any z ∈ Z ,

where K T is defined in Definition 4.

The above theorem demonstrates that the loss of the NN at a certain fixed time is a general kernel machine. Herein, K T is the LPK and we prove in Appendix C.1 that it is a valid kernel. Unlike previous NTK works that establish the equivalence between infinite-width NNs and kernel machines, our equivalence is much more general and holds for any NN that is continuously differentiable. Based on this equivalence, we characterize the generalization of NNs from the perspective of kernels. Note that K T is a function of S and this property enables us to establish a data-dependent generalization bound shortly.

4.2 Generalization Bounds

We introduce the main result in this section: a generalization bound for NNs whose weights follow gradient flow in (1) at time T . We derive this bound by analyzing the Rademacher complexity of the function class of kernel machines induced by different training sets with constrained RKHS norms. Recall that each training set yields a distinct LPK. We define the collection of all such LPKs by

where B > 0 is some constant, S ′ = { z ′ 1 , . . . , z ′ n } , µ ⊗ n is the joint distribution of n i.i.d. samples drawn from µ , and supp ( µ ⊗ n ) is the support set of µ ⊗ n . Recall that S = { z 1 , . . . , z n } is the training

set. Note the set in (2) includes the case of S ′ = S if 1 n 2 ∑ i,j K T ( z i , z j ; S ) ≤ B 2 . Then we introduce a class of general kernel machines, corresponding to all different kernels in K T .

Note that g ( z ) ∈ G T corresponds to ℓ ( w T , z ) trained from one possible dataset S ′ ∈ supp ( µ ⊗ n ) . Next, we compute the Rademacher complexity of G T and use it to obtain a generalization bound.

Theorem 3. ˆ R S ( G T ) ≤ min { U 1 , U 2 } . Here

̸

where G S T = { g ( Z ) = ( g ( z 1 ) , . . . , g ( z n )) : g ∈ G T } , N ( G S T , ϵ, ∥∥ 1 ) is the covering number of G S T with the ℓ 1 -norm and

The term U 1 is composed by two components sup K ( · , · ; S ′ ) ∈K T Tr ( K ( Z , Z ; S ′ )) and ∆( z i , z j ) . The first component, according to the definition of LPK, quantifies the maximum magnitude of the loss gradient in K T evaluated with the set S throughout the training trajectory. The second component assesses the range of variation of LPK within the set K T . The term U 2 is obtained from analyzing the covering number of G T . It shows that if the variation of the loss dynamics of gradient flow with different training data is small, then the complexity of G T will also be small. The norm constraint 1 n 2 ∑ i,j K T ( z ′ i , z ′ j ; S ′ ) ≤ B 2 balances a tradeoff between the tightness of the bound and the expressiveness of the set G T (the number of datasets covered). Combining these two bounds with Theorem 1, we obtain the following generalization bound.

Corollary 1 (Generalization bound for NN) . Fix B > 0 . Let ˆ R gf S ( G T ) = min ( U 1 , U 2 ) where U 1 and U 2 are defined in Theorem 3. For any δ ∈ (0 , 1) , with probability at least 1 -δ over the draw of an i.i.d. sample set S = { z i } n i =1 , if 1 n 2 ∑ i,j K T ( z i , z j ; S ) ≤ B 2 , the following holds for ℓ ( w T , z ) that trained from S ,

where w T = A T ( S ) is the output from the gradient flow (1) at time T by using S as input.

Our result owns many compelling properties.

First, our bound holds in a general setting as it does not hedge on a special NN architecture. In contrast, existing works [2, 11] only consider fully connected NNs and require NN to be ultra-wide.

Our bound depends on the data distribution through the quantities in U 1 and U 2 . This property not only significantly tightens our bound but can also help explain some empirical observations of NNs. For example, different from classical generalization theory, e.g. VC dimension, our complexity bounds depend on the labels directly, which helps explain the random label phenomenon [59] as shown in Figure 3 in Sec. 7.

Our experiments in Sec. 7 (Figure 2) demonstrate the tightness of the generalization bound. Intuitively, our bound is tight because (1) instead of considering the entire hypothesis class, we focus on the subset of interest characterized by running gradient flow from a starting point w 0 ; (2) we get the bound from an equivalence between NNs and general kernel machines, whose generalization bounds are tighter. Finally, we compare our generalization bound with two existing NTK-based bounds in Table 1.

5 Stochastic Gradient Flow

In the previous section, we derived a generalization bound for NNs trained from full-batch gradient flow. Here we extend our analysis to stochastic gradient flow and derive a corresponding generalization bound. To start with, we recall the dynamics of stochastic gradient flow (SGD with infinitesimal step size). Let S t ⊆ { 1 , . . . , n } be the indices of batch data used in time interval [ t, t +1] and |S t | = m be the batch size. We establish a new connection between the loss dynamics of stochastic gradient flow and a general kernel machine. Then we investigate the complexity of the collection of such kernel machines that can be induced by various training sets.

Theorem 4. Suppose w ( T ) = w T is a solution of stochastic gradient flow at time T ∈ N with initialization w (0) = w 0 . Then for any z ∈ Z ,

where K t,t +1 ( z , z i ; S ) = ∫ t +1 t ¯ K ( w ( t ); z , z i ) d t with ¯ K defined in Definition 3.

The above theorem shows that the loss of the NN in stochastic gradient flow dynamics can be characterized by a sum of general kernel machines. In particular, when we use the full batch at each time interval (i.e., m = n ), the above result recovers Theorem 2. To study its generalization behavior, we introduce the class of kernel machines induced by different training sets S ′ ∈ supp ( µ ⊗ n ) with constrained RKHS norms. Specifically, given B t > 0 for t = 0 , · · · , T -1 , we define

Note this set includes the kernel induced by the training set S if it satisfies the constraints. Then ℓ ( w T , z ) trained from all feasible S ′ ∈ supp ( µ ⊗ n ) form a function class

Next, we upper bound the Rademacher complexity of the function class G T . This bound can naturally translate into a generalization bound by equipping with Theorem 1.

Theorem 5. The Rademacher complexity of G T defined in (3) has an upper bound:

̸

We assumed that mini-batches indices S t are chosen before training. However, our analysis can be extended to accommodate random mini-batch selections of any sampling strategy by enumerating all potential S t in K T .

6 Case Study & Use Case

In the previous sections, we derived generalization bounds for NNs trained with (stochastic) gradient flow. These bounds may initially appear complex due to their dependence on the training process. Here we show that these bounds can be significantly simplified by applying them to infinite-width NNs (and stable algorithms in Appendix E.2, norm-constraint NNs in Appendix E.3). Moreover, we demonstrate that our generalization bounds maintain a high correlation with the true generalization error. As a result, we use them to guide the design of NAS, and our experimental results demonstrate that this approach has a favorable performance compared with state-of-the-art algorithms.

6.1 Infinite-width NN

In this subsection, we consider a special case of infinite-width NNs trained by gradient flow and derive pre-computed generalization bounds. We focus on gradient flow to simplify the presentations but our results can be directly extended to stochastic gradient flow. For an infinite-width NN, under certain conditions, the neural tangent kernel keeps unchanged during training: ˆ Θ( w t ; x , x ′ ) → Θ( x , x ′ ) · I k . Consider a ρ -Lipschitz loss function, i.e. ∥∇ f ℓ ( w , z ) ∥ ≤ ρ . The Rademacher complexity in Theorem 3 can be bounded by ˆ R S ( G T ) ≤ U ∞ , where

In this infinite-width regime, our bound has no dependence on the initialization w 0 since the NTK converges to a deterministic limit and has no dependence on the parameters. That means the bound holds for all possible w 0 of infinite-width NNs trained by gradient flow from initialization. Compared with the bound ˜ O ( L · √ Y ⊤ ( Θ ) -1 Y n ) in [11], U ∞ has several advantages: (1) it has no dependence on the number of layers L ; (2) it holds for NNs with multiple outputs.

6.2 Correlation Analysis and NAS

As a practical application, we apply our generalization bounds to guide the design of NAS. We first introduce a quantity U sgd simplified from the bound in Theorem 5, defined as

U sgd can be computed along with training a NN via SGD on a training set S . Combining it with the training loss, we define the following quantity as an estimate of the population loss:

We analyze the correlation between Gene ( w , S ) and the true generalization error by randomly sampling 100 NN architectures from NAS-Bench-201 [19]. For each, we compute both Gene ( w , S ) and the true generalization error. Since solving the gradient flow ODE is computationally infeasible for the large NNs in NAS-Bench-201, we apply a trapezoidal rule to approximate the integration in LPK K t,t +1 . This approximation enables us to compute U sgd efficiently. Figure 1 demonstrates the correlation between Gene ( w , S ) and the test error. The left figures plot the test error at epoch 1 or 2 against Gene ( w , S ) of the respective epochs, showing a strong positive correlation between them. The right figures plot the test error at convergence against Gene ( w , S ) at epoch 1 or 2, which also demonstrate a positive correlation. The outlier is caused by some architecture with large loss gradients. This experiment shows that Gene ( w , S ) at the initial training stage can predict the performance of NNs at convergence. Based

Test error at epoch 1

0.98

Kendall's tau = 0.75

Testerroratconvergence

0.9

1.0

Kendall's tau = 0.17

0.96

0.8

0.94

0.7

0.92

0.6

0.90

0.88

0.4

0.86

0.3

Gene(w,S) at epoch 1

10

15

20

25

OE

35

Gene(w, S) at epoch 1

10

15

20

25

OE

35

1.00

Kendall's tau = 0.66

1.0

Kendall's tau = 0.29

Test error at convergence

0.9

Test error at epoch

0.95

0.8

0.90

0.7

0.6

0.85

S′0

0.4

0.80

0.3

Gene(w, S) at epoch 2

10

15

20

25

30

35

Gene(w, S) at epoch 2

10

15

20

SE

on this observation, we use Gene ( w , S ) as a metric in NAS for selecting architectures at the initial training stage (see Table 2). This approach significantly reduces computational costs compared with training-based NAS algorithms [47, 34, 18, 33].

(a)Logistic loss for training samples

(b)Logisticlossandbound

(c)Rademachercomplexitybound

0.8

NN GF

NN train loss

0.10

Generalization gap

0.7

NN GD

0.8

NN test loss

0.6

KM

......

NN test error

0.08

R′(GT),50 s

0.5

0.6

Ly(w) Bound, 100 S'

0.06

Rg(Gr), 100 s

0.4

0.3

0.4

0.04

0.2

0.2

0.02

0.1

0.0

0.0

0.00

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

t

t

t

7 Numerical Experiments

We conduct comprehensive numerical experiments to demonstrate our generalization bounds. We observe that our complexity bounds are tight with respect to the generalization gap and can capture how noisy label influences the generalization behaviors of NNs. Moreover, we apply Gene ( w , S ) in (5) to NAS and demonstrate favorable performance compared with state-of-the-art algorithms.

(I) Generalization bound in Corollary 1. In Figure 2 (more detailed in Figure A.4), we use a logistic loss to train a two-layer NN with 100 hidden nodes for binary classification on MNIST 1 and 7 [31] by full-batch gradient flow and compute its generalization bound. Due to the computational cost of solving the gradient flow ODE and computing the kernel, we only train and calculate the bound on n = 1000 training samples. The bound would be tighter with more training samples. The NN is initialized using the NTK parameterization [28]. We use the Softplus activation function, defined as Softplus ( x ) = 1 β ln(1 + e βx ) . This function is continuously differentiable and serves as a smooth approximation to the ReLU activation function. In our experiments, we set β = 10 . To train the NN via gradient flow, we solve the gradient flow ODE given by (1) to decide the NN parameters. For the equivalent general KM, we compute the LTK using the NN parameters and integrate it to get the LPK K T . These ODEs are computed with torchdiffeq [12]. To estimate the generalization bound, we train the NN on (20, 50, 100) independently sampled training sets S ′ to estimate the K T and G T , and the supremum in the bound U 1 is estimated by taking the maximum over the finite set K T . For U 2 , we compute an upper bound of it by setting ϵ as the largest ℓ 1 distance between any two g ( Z ) ∈ G S T and N ( G S T , ϵ, ∥∥ 1 ) = 1 because in this case any g ( Z ) ∈ G S T will satisfy as an ϵ -cover. We run each experiment five times and plot the mean and standard deviation. The numerical experiments demonstrate that our complexity bound is tight and can capture the generalization gap well. As the number of S ′ increases, our bound converges to the true supremum value. To estimate the true supremum value, we apply the extreme value theory in Appendix A.3 and show the gap between the finite maximum and supremum is small, validating using a finite maximum as an estimate for our bound.

We train the NN using GD with a finite learning rate η = 10 to compare with the NN trained by gradient flow. The training time t = η × training steps. In Figure 2 (a) and Figure A.4 (a)(b), we observe that the loss of the NN trained by GD and that trained by gradient flow are consistently close throughout the entire training process. Consequently, while we established upper bounds for NNs trained by gradient flow, these results can serve as an (approximate) upper bound for NNs trained by GD with a finite learning rate.

Notably, for the NN in this experiment, the VC dimension [7] is 55957 . 3 , the norm-based bound in [6] is 140 . 7 at T = 1000 , and the NTK-based bound for an ultra-wide NN in [11] is 1.44, which are all vacuous (larger than 1), while our bound is tight (0.03 at T = 1000 ). See a detailed comparison in Appendix A.2. We also conduct an experiment of three-layer NN (3072-100-100-1) trained on binary CIFAR-10 (cat and dog) [30] in Figure A.5, where there is a larger generalization gap.

(II) Generalization bound with label noise. The settings are similar to Experiment (I) except we corrupt the labels with different portions of noise and calculate the bound after training NN until T = 20000 . We estimate the bound with 20 training sets S ′ . The results in Figure 3 show that our bound has a strong correlation with the generalization gap and increases with the portion of label noise. Unlike classical generalization theory, e.g. VC dimension, our generalization bound can help explain the random label phenomenon [59].

(III) Neural architecture search (NAS). We apply Gene ( w , S ) in Eq. (5) to guide the design of NAS. The results are shown in Table 2. We use a simple random search (RS) with Gene ( w , S ) , where 100 architectures are sampled from the search space for evaluation, and the architecture with smallest Gene ( w , S ) is selected. U sgd is estimated with a batch of data of size 600. Build upon Sec. 6.2, we apply Gene ( w , S ) 1 and Gene ( w , S ) 2 (Gene ( w , S ) after training 1 and 2 epochs) to select NN architectures at the initial training stage in order to reduce computational costs. We compare our method with state-of-the-art training-free/minimum-training NAS algorithms [13, 39]. We run the experiments four times with different random seeds and report the mean and standard deviation. We reproduce the results in Chen et al. [13] using their released code and directly adopt the results reported in Mok et al. [39] as they did not release the code. The results show our approach of RS + Gene ( w , S ) can achieve favorable performance compared with state-of-the-art training-free/minimum-training NAS algorithms.

8 Conclusion and Future Work

In this paper, we establish a new connection between the loss dynamics of (stochastic) gradient flow and a general kernel machine. Building upon this result, we introduce generalization bounds for NNs trained from (stochastic) gradient flow. Our bounds hold for any continuously differentiable NN architectures (both finite-width and ultra-wide) and are generally tighter than existing bounds. Moreover, for infinite-width NNs, we obtain a pre-computed generalization bound for the whole training process. Finally, we apply our results to NAS and demonstrate favorable performance compared with state-of-the-art NAS algorithms.

There are several directions for future research. First, evaluating our generation bounds relies on the loss gradient, which may contain private and sensitive information. One potential fix would be accessing such information in a differentially private manner and it would be interesting to investigate how this 'noisy' observation of gradient information influences our generalization bounds. Second, it is worth exploring how other optimization algorithms and different model architectures influence the generalization bounds. Finally, our bounds provide worst-case guarantees to the generalization of NNs and it would be interesting to extend our results to obtain expected bounds for further sharpening the results.

References

Arora, S., Ge, R., Neyshabur, B., and Zhang, Y. Stronger generalization bounds for deep nets via a compression approach. arXiv preprint arXiv:1802.05296 , 2018.

Arora, S., Du, S., Hu, W., Li, Z., and Wang, R. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning , pp. 322-332. PMLR, 2019.

Arora, S., Du, S. S., Hu, W., Li, Z., Salakhutdinov, R. R., and Wang, R. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems , 32, 2019.

Awasthi, P., Frank, N., and Mohri, M. On the rademacher complexity of linear hypothesis sets. arXiv preprint arXiv:2007.11045 , 2020.

Bartlett, P. L. and Mendelson, S. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research , 3(Nov):463-482, 2002.

Bartlett, P. L., Foster, D. J., and Telgarsky, M. J. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems , 30, 2017.

Bartlett, P. L., Harvey, N., Liaw, C., and Mehrabian, A. Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks. The Journal of Machine Learning Research , 20(1):2285-2301, 2019.

Bassily, R., Feldman, V., Guzmán, C., and Talwar, K. Stability of stochastic gradient descent on nonsmooth convex losses. Advances in Neural Information Processing Systems , 33:4381-4391, 2020.

Belkin, M., Hsu, D., Ma, S., and Mandal, S. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences , 116(32):15849-15854, 2019.

Boucheron, S., Lugosi, G., and Massart, P. Concentration inequalities: A nonasymptotic theory of independence . Oxford university press, 2013.

Cao, Y. and Gu, Q. Generalization bounds of stochastic gradient descent for wide and deep neural networks. Advances in neural information processing systems , 32, 2019.

Chen, R. T. Q. torchdiffeq, 2018. URL https://github.com/rtqichen/torchdiffeq .

Chen, W., Gong, X., and Wang, Z. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. arXiv preprint arXiv:2102.11535 , 2021.

Chen, Y., Huang, W., Nguyen, L., and Weng, T.-W. On the equivalence between neural network and support vector machine. Advances in Neural Information Processing Systems , 34, 2021.

Chizat, L. and Bach, F. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory , pp. 1305-1338. PMLR, 2020.

Cristianini, N., Shawe-Taylor, J., et al. An introduction to support vector machines and other kernel-based learning methods . Cambridge university press, 2000.

Domingos, P. Every model learned by gradient descent is approximately a kernel machine. arXiv preprint arXiv:2012.00152 , 2020.

Dong, X. and Yang, Y. Searching for a robust neural architecture in four gpu hours. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 1761-1770, 2019.

Dong, X. and Yang, Y. Nas-bench-201: Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.00326 , 2020.

Du, S., Lee, J., Li, H., Wang, L., and Zhai, X. Gradient descent finds global minima of deep neural networks. In International conference on machine learning , pp. 1675-1685. PMLR, 2019.

Du, S. S., Zhai, X., Poczos, B., and Singh, A. Gradient descent provably optimizes overparameterized neural networks. arXiv preprint arXiv:1810.02054 , 2018.

Dziugaite, G. K. and Roy, D. M. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008 , 2017.

Grant, C. P. Theory of ordinary differential equations. Brigham Young University , 2014.

Gunasekar, S., Lee, J. D., Soudry, D., and Srebro, N. Implicit bias of gradient descent on linear convolutional networks. Advances in Neural Information Processing Systems , 31, 2018.

Hardt, M., Recht, B., and Singer, Y. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning , pp. 1225-1234. PMLR, 2016.

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 770-778, 2016.

Huang, W., Liu, C., Chen, Y., Da Xu, R. Y., Zhang, M., and Weng, T.-W. Analyzing deep pac-bayesian learning with neural tangent kernel: Convergence, analytic generalization bound, and efficient hyperparameter selection. Transactions on Machine Learning Research , 2023.

Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On largebatch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836 , 2016.

Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.

LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Lee, J., Xiao, L., Schoenholz, S., Bahri, Y., Novak, R., Sohl-Dickstein, J., and Pennington, J. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems , 32, 2019.

Li, L. and Talwalkar, A. Random search and reproducibility for neural architecture search. In Uncertainty in artificial intelligence , pp. 367-377. PMLR, 2020.

Liu, H., Simonyan, K., and Yang, Y. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 , 2018.

Maurer, A. A note on the pac bayesian theorem. arXiv preprint cs/0411099 , 2004.

McDiarmid, C. et al. On the method of bounded differences. Surveys in combinatorics , 141(1): 148-188, 1989.

Mellor, J., Turner, J., Storkey, A., and Crowley, E. J. Neural architecture search without training. In International Conference on Machine Learning , pp. 7588-7598. PMLR, 2021.

Mohri, M., Rostamizadeh, A., and Talwalkar, A. Foundations of machine learning . MIT press, 2018.

Mok, J., Na, B., Kim, J.-H., Han, D., and Yoon, S. Demystifying the neural tangent kernel from a practical perspective: Can it be trusted for neural architecture search without training? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 11861-11870, 2022.

Neyshabur, B., Tomioka, R., and Srebro, N. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6614 , 2014.

Neyshabur, B., Tomioka, R., and Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory , pp. 1376-1401. PMLR, 2015.

Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. Exploring generalization in deep learning. Advances in neural information processing systems , 30, 2017.

Neyshabur, B., Bhojanapalli, S., and Srebro, N. A pac-bayesian approach to spectrallynormalized margin bounds for neural networks. arXiv preprint arXiv:1707.09564 , 2017.

Neyshabur, B., Li, Z., Bhojanapalli, S., LeCun, Y., and Srebro, N. The role of overparametrization in generalization of neural networks. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=BygfghAcYX .

Novak, R., Xiao, L., Hron, J., Lee, J., Alemi, A. A., Sohl-Dickstein, J., and Schoenholz, S. S. Neural tangents: Fast and easy infinite neural networks in python. In International Conference on Learning Representations , 2020. URL https://github.com/google/neural-tangents .

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Pham, H., Guan, M., Zoph, B., Le, Q., and Dean, J. Efficient neural architecture search via parameters sharing. In International conference on machine learning , pp. 4095-4104. PMLR, 2018.

Russo, D. and Zou, J. Controlling bias in adaptive data analysis using information theory. In Artificial Intelligence and Statistics , pp. 1232-1240. PMLR, 2016.

Santambrogio, F. { Euclidean, metric, and Wasserstein } gradient flows: an overview. Bulletin of Mathematical Sciences , 7:87-154, 2017.

Savarese, P., Evron, I., Soudry, D., and Srebro, N. How do infinite width bounded norm networks look in function space? In Conference on Learning Theory , pp. 2667-2690. PMLR, 2019.

Schölkopf, B., Smola, A. J., Bach, F., et al. Learning with kernels: support vector machines, regularization, optimization, and beyond . MIT press, 2002.

Shalev-Shwartz, S. and Ben-David, S. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

Shawe-Taylor, J., Cristianini, N., et al. Kernel methods for pattern analysis . Cambridge university press, 2004.

Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., and Srebro, N. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research , 19(1):2822-2878, 2018.

Valle-Pérez, G. and Louis, A. A. Generalization bounds for deep learning. arXiv preprint arXiv:2012.04115 , 2020.

Vapnik, V. and Chervonenkis, A. Y. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and its Applications , 16(2):264, 1971.

Weng, T.-W., Zhang, H., Chen, P.-Y., Yi, J., Su, D., Gao, Y., Hsieh, C.-J., and Daniel, L. Evaluating the robustness of neural networks: An extreme value theory approach. In International Conference on Learning Representations , 2018. URL https://openreview.net/forum? id=BkUHlMZ0b .

Xu, A. and Raginsky, M. Information-theoretic analysis of generalization capability of learning algorithms. Advances in Neural Information Processing Systems , 30, 2017.

Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021.

Appendices

A Additional Experiments

(a) Logistic loss for training samples

(b) Difference of logistic loss

0.8

NN GF

0.0015

NN GF -KM

0.7

--- NN GD

-- NN GD - KM

0.6

KM

0.0010

0.5

0.4

0.0005

0.3

0.0000

0.2

0.1

-0.0005

0.0

0

200

400

600

800

1000

0

200

400

600

800

1000

t

t

(d) Rademacher complexity bound

1.4

Generalization gap

1.2

0.5

U,50 S'

1.0

0.4

U1,100 S'

NN train loss

U2,20 S'

0.8

.......

NN test loss

0.3

U2,50 S'

NN test error

U2,100 S'

0.6

Bound with U1,100 S'

Bound with U2,100 S'

0.2

0.4

0.1

0.2

0.0

0.0

0

200

400

600

800

1000

0

200

400

600

800

1000

t

t

Experiments are implemented with PyTorch [46] on 24G A5000 and 32G V100 GPUs.

A.1 Computation Cost of Experiments

In Experiment (I), estimating the bound with 20 S ′ (solving 20 gradient flow ODE) costs 500s and training NN costs 0.29s. The GPU memory required by estimating the bound is 2406MB and training NN requires 1044MB.

For Experiment (III), we report the averaged computational cost (GPU hours) of our approach for one architecture and the computational cost of training one NN architecture to convergence in

(a)Logistic loss for training samples

(b)Difference of logisticloss

0.8

0.00125

NN GF - KM

---NN GD -KM

0.7

0.00100

0.6

0.00075

0.5

0.00050

0.4

0.00025

0.3

NN GF

0.00000

NN GD

0.2

.

KM

-0.00025

0

500

1000

1500

2000

2500

3000

0

500

1000

1500

2000

2500

3000

t

t

(d)Rademacher complexity bound

0.70

Generalizationgap

0.175

Rg(GT)

0.65

0.150

0.60

0.125

NN train loss

KM train loss

0.100

0.55

NN test loss

NNtesterror

0.075

0.50

0.050

0.45

0.025

0.000

0

500

1000

1500

2000

2500

3000

0

500

1000

1500

2000

2500

3000

t

t

A.2 Comparison with Existing Generalization Bounds

To make a stronger comparison with existing work, we have conducted two additional experiments below to show that our bound is much tighter than prior work on both finite-width NNs and infinitewidth NNs.

(1) For the finite-width NN, we compare with previous uniform convergence bounds (VC dimension [7] and norm-based bounds [6]) as the NTK-based bounds [2, 11] are limited to infinite or ultra-wide NNs. Note our bounds are also uniform convergence bounds. We follow the same setting in our Experiment I, and calculate the bound for two-layer NN with 100 hidden nodes (finite-width) at training time T=1000 and sample size n=1000. Here L is the number of layers, p is the number of parameters, W i is the weight matrix for the i -th layer, and m is the largest width of NN including the input dimension. Constants in big O are ignored in the calculation. The results in Table 4 show that our bound is much tighter than previous VC dimensions and norm-based bounds.

Finally, we remark that existing works have observed that VC dimension and norm-based bounds are mostly vacuous (see e.g., Figure 5 of [44] and Figure 4 in [1]) while our bound is non-vacuous as shown above and in Figure 2 and Figure A.5.

(2) For the infinite-width NN, we compare with the NTK-based bound in Cao & Gu [11] in a similar setting as Experiment I - two-layer NN and binary MNIST (1 vs. 7) with n=1000. Note that their bound only holds for an ultra-wide NN. We compute the infinite-width NTK using Neural Tangents [45] and calculate their bound accordingly. For our bound, we train a two-layer NN with 1000 hidden nodes (to approximate an ultra-wide NN) and compute our bound at T = 1000 (almost convergence). Constants in big O are ignored in the calculation. The results in Table 5 show that our bound is much tighter than previous NTK-based bounds.

A.3 Estimate Supremum with Extreme Value Theory

In the all above experiments, we estimate the supremum in the Rademacher bound in Theorem 3 with the maximum over a finite set. To get the true supremum value of the bound, we apply the extreme value theory to estimate it, similar to estimating the local Lipschitz of NN in [57]. We show the finite maximum is close to the true supremum value, validating using a finite maximum as an estimate for our bound. We first state the following result from extreme value theory and then explain how we can apply it to estimate our bounds.

Theorem 6 (Fisher-Tippett-Gnedenko theorem) . Let X 1 , X 2 , · · · , X m be a sequence of independent and identically-distributed random variables with cumulative distribution function F . Suppose that there exist two sequences of real numbers a m > 0 and b m ∈ R such that the following limits converge to a non-degenerate distribution function:

Then the limit distribution G belongs to either the Gumbel class (Type I), the Fréchet class (Type II), or the Reverse Weibull class (Type III):

where a w ∈ R , b w > 0 and c w > 0 are the location, scale, and shape parameters, respectively.

Our bounds are finite as long as ℓ ( w , z ) is L ℓ -Lipschitz (will explain in detail later). Thus we are particularly interested in the reverse Weibull class, as its CDF has a finite right end-point (denoted as a w ). The right end-point reveals the upper limit of the distribution, known as the extreme value . In our case, the extreme value is exactly the supermum we want to estimate.

To compute U 2 , we compute an upper bound of it by setting ϵ as the largest ℓ 1 distance between any two g ( Z ) ∈ G S T and N ( G S T , ϵ, ∥∥ 1 ) = 1 because in this case any g ( Z ) ∈ G S T will satisfy as an ϵ cover.

Denote the right hand side as U ∗ 2 . Since each K ( z i , z j ; S ′ ) ≤ L ℓ T 2 , U ∗ 2 ≤ 2 L ℓ T 2 .

Note Z is fixed. Then for all g 1 , g 2 ∈ G T , ∥ g 1 ( Z ) -g 2 ( Z ) ∥ 1 are i.i.d. random variables, because g ∈ G T are trained from i.i.d. S ′ . Consider a finite set G N g ⊂ G T with size N g = ∣ ∣ G N g ∣ ∣ . As N g →∞ ,

Then we can apply extreme value theory to estimate U ∗ 2 . We generate N b batch of G N g , compute their ˆ U 2 and store them in a set. Then with these ˆ U 2 's, we perform a maximum likelihood estimation of reverse Weibull distribution parameters, and the location estimate a w is used as an estimate of the U ∗ 2 . To validate that reverse Weibull distribution is a good fit for the empirical distribution of the ˆ U 2 's, we conduct Kolmogorov-Smirnov goodness-of-fit test (a.k.a. K-S test) to calculate the K-S test statistics D and corresponding p-values. The null hypothesis is that ˆ U 2 's follow a reverse Weibull distribution.

We follow the same setting as Experiment I and want to estimate U ∗ 2 at T = 1000 . Figure A.6 shows a result of estimating U ∗ 2 with N b = 50 and N g = 1000 . The estimated U ∗ 2 = 0 . 06 is quite close to the finite maximum, validating using a finite maximum as an estimate for our bound.

(aw,bw,Cw)=(0.06,0.02,6.23),pval = 0.88,ks = 0.08

fittedpdf

100

80

60

40

20

0

0.030

0.035

0.040

0.045

0.050

U2

For U 1 , where

̸

Tr ( K ( Z , Z ; S ′ )) and K ( z i , z j ; S ′ ) for i, j ∈ [ n ] are random variables that only depends on S ′ . For different S ′ , S ′′ ∈ supp ( µ ⊗ n ) , Tr ( K ( Z , Z ; S ′ )) and Tr ( K ( Z , Z ; S ′′ )) are i.i.d. random variables. Similarly for K ( z i , z j ; S ′ ) , i, j ∈ [ n ] . We assume the finite maximum of each random variable follows a reverse Weibull distribution and estimate their supremum then sum them together to get

(a) Estimating

sup

K

(

·

,

·

;

S

′

)

∈K

Tr

(

K

(

Z

,

Z

;

S

′

))

.

(b) Estimating

sup

K

(

·

,

·

;

S

′

)

∈K

K

(

z

1

,

z

2

;

S

′

(aw, bw, Cw)=(11614.70, 1190.22, 20.24), pval = 0.72, ks = 0.10

(aw, bw, Cw)=(-1.13, 0.04, 5.22), pval = 0.97, ks = 0.07

0.006

fittedpdf

50

fittedpdf

0.005

40

0.004

30

0.003

20

0.002

10

0.001

0.000

10200

10300

10400

10500

10600

10700

10800

1.19

1.18

1.17

1.16

1.15

1.14

maxTr(K(Z,Z; S))

maxK(z1,Z2:S)

Kf-, : S )

KI-, • : 5′1

T

T

)

.

U 1 . Figure A.7 shows estimating sup K ( · , · ; S ′ ) ∈K T Tr ( K ( Z , Z ; S ′ )) and sup K ( · , · ; S ′ ) ∈K T K ( z 1 , z 2 ; S ′ ) . The estimated extreme values are close to the finite maximum, validating using a finite maximum as an estimate for our bound. Due to the computational cost of estimating n 2 extreme values, we leave estimating U 1 as future work.

B Integrability of Loss Tangent Kernel

By the assumption that the loss is continuously differentiable, the loss gradient ∇ w ℓ ( w ( t ) , z ) is continuous w.r.t. w ( t ) . Together with w ( t ) is continuous w.r.t. t , ∇ w ℓ ( w ( t ) , z ) is continuous w.r.t. t . After the inner product, the LTK ¯ K ( w ( t ); z , z ′ ) is still continuous w.r.t. t . By the continuity of LTK on the compact set [0 , T ] , LTK is bounded and Riemann integrable on [0 , T ] . Therefore, the integral in LPK is well-defined.

For the full-batch gradient flow (1) we considered in this paper, w ( t ) is differentiable therefore continuous w.r.t. t by the gradient flow equation. Hence, the integral in LPK is well-defined.

For stochastic gradient flow, the continuity of the path w ( t ) can be argued as follows. For each time interval [ t, t +1] , we assume that the same batch of data is used. Within this time interval, w ( t ) is continuous w.r.t. t . For the next time interval [ t +1 , t +2] , even if a different data batch is used, the gradient flow ODE initializes the w ( t +1) with the solution from the end point of the previous interval, [ t, t +1] . This ensures the continuity of w ( t ) across distinct time intervals. In short, the continuity of w ( t ) within each time interval, combined with the initialization of the ODE, will ensure the continuity of w ( t ) in the entire time interval [0 , T ] .

C Complete Proofs for Gradient Flow

C.1 Proof of Theorem 2

Theorem 2. Suppose w ( T ) = w T is a solution of (1) at time T with initialization w (0) = w 0 . Then for any z ∈ Z ,

where K T is defined in Definition 4.

Proof. Consider the gradient flow:

For any differentiable loss function ℓ ( w , z ) , by chain rule,

Plug in the gradient flow expression of d w t dt in (6) into (7). We have

Integrate both sides from 0 to T over the path w ( t ) taken by the parameters during the gradient flow,

Thus we have

Below, we prove that K T ( z , z ′ ; S ) : Z × Z → R is a valid kernel. We can show K T ( z , z ′ ; S ) is a valid kernel by proving K T ( z , z ′ ; S ) is continuous and the kernel matrix K T ( Z , Z ; S ) ∈ R n × n is positive semi-definite (PSD) for any finite subset of Z , z 1 , · · · , z n ∈ Z [53]. By Definition 4 of the loss path kernel,

Since ℓ ( w t , z ) is continuously differentiable for w t and z , ∇ w ℓ ( w t , z ) is continuous for z . After the inner product and integration, it is still a continuous function. Thus K T ( z , z ′ ; S ) is a continuous function for z .

Denote Φ t ( z ) = ∇ w ℓ ( w t , z ) , then by Definition 4 of the loss path kernel,

For ∀ u ∈ R n and ∀ { z i } n i =1 ⊆ Z ,

Thus the matrix K T ( Z , Z ; S ) is PSD and K T ( z , z ′ ; S ) is therefore a valid kernel.

C.2 Proof of Theorem 3

Theorem 3. ˆ R S ( G T ) ≤ min { U 1 , U 2 } . Here

̸

where G S T = { g ( Z ) = ( g ( z 1 ) , . . . , g ( z n )) : g ∈ G T } , N ( G S T , ϵ, ∥∥ 1 ) is the covering number of G S T with the ℓ 1 -norm and

C.2.1 Proof of U 1

Proof. Recall the definition of G T ,

where

Suppose K ( z , z ′ ; S ′ ) = ⟨ Φ S ′ ( z ) , Φ S ′ ( z ′ ) ⟩ . Define

We first show G T ⊆ G U T . For ∀ g ( z ) ∈ G T ,

We know by definition of G T , ∥ β S ′ ∥ 2 = 1 n 2 ∑ i,j K ( z ′ i , z ′ j ; S ′ ) ≤ B 2 . Thus g ( z ) ∈ G U T . Since ∀ g ( z ) ∈ G T , g ( z ) ∈ G U T , G T ⊆ G U T . But G U T is strictly larger than G T because β S ′ is a fixed vector for a fixed K ( · , · ; S ′ ) while β in G U T is a vector of any direction.

Then by the property of Rademacher complexity,

The second term in the square root is

̸

̸

̸

̸

̸

̸

̸

where we define ∆( z i , z j ) = 1 2 [ sup K ( · , · ; S ′ ) ∈K T K ( z i , z j ; S ′ ) -inf K ( · , · ; S ′ ) ∈K T K ( z i , z j ; S ′ ) ]

̸

C.2.2 Proof of U 2

Proof. To simplify the notation, denote K S ′ ( · , · ) = K T ( · , · ; S ′ ) . Denote g K S ′ ( z ) = ∑ n i =1 -1 n K S ′ ( z , z ′ i ) + ℓ ( w 0 , z ) for a fixed K S ′ , which is a singleton hypothesis class. Then G T is a union set of such function classes with different kernels,

Then we can rewrite G S T = { g K S ′ ( Z ) = ( g K S ′ ( z 1 ) , . . . , g K S ′ ( z n )) : K S ′ ∈ K T } . Suppose G ϵ T ⊆ G S T is a minimal ϵ -cover of G S T with ∥∥ 1 . Denote ˜ g K S ′ ( Z ) = ( ˜ g K S ′ ( z 1 ) , . . . , ˜ g K S ′ ( z n )) ∈ G ϵ T as the closest element to g K S ′ ( Z ) ∈ G S T and ˜ g K S ′ ( z ) = ∑ n i =1 -1 n ˜ K S ′ ( z , z ′ i ) + ℓ ( w 0 , z ) = g ˜ K S ′ ( z ) with ˜ K S ′ ∈ K T . Denote the set of all ˜ K S ′ as K ϵ T . Since one ˜ g K S ′ corresponds to one ˜ K S ′ , |K ϵ T | = |G ϵ T | = N ( G S T , ϵ, ∥∥ 1 ) .

Based on the above, we can write the Rademacher complexity of G T as

For any λ > 0 , by Jensen's inequality,

̸

̸

Utilizing the definition of ϵ -covering, the quantity on the exponent in Eq. (8) is

(by the definition of the ϵ -covering)

Substitute this inequality into Eq. (8). We have

where ( i ) is because we only use the g ˜ K S ′ instead of g K S ′ , which corresponds to ˜ K S ′ ∈ K ϵ T instead of K S ′ ∈ K T , so it is equivalent to take the maximum over ˜ K S ′ ∈ K ϵ T . Denote ξ ( σ 1 , . . . , σ n ) = 1 n ∑ n i =1 σ i g ˜ K S ′ ( z i ) . Note g ˜ K S ′ ( z i ) ∈ [0 , 1] for i ∈ [ n ] . Then for all i ∈ [ n ] ,

Thus ξ ( σ 1 , · · · , σ n ) satisfies the bounded difference property. Let c i = 2 n . By [36, 10], ξ is a sub-Gaussian variable and satisfies

Since g ˜ K S ′ is a singleton function, E [ ξ ] = 1 n E σ [ ∑ n i =1 σ i g ˜ K S ′ ( z i ) ] = 0 . Thus E [ e λξ ] ≤ e λ 2 2 n .

Take this into Eq. (9), in total, we have

Take the logarithm on both sides of the equation,

By taking λ = √ 2 n ln |K ϵ T | , we get

Take the infimum over ϵ > 0 and note |K ϵ T | = |G ϵ T | = N ( G S T , ϵ, ∥∥ 1 ) . We get

C.3 A lower bound of ˆ R S ( G U T )

Here we give a lower bound of ˆ R S ( G U T ) . Similar lower bounds for a linear model were proved in [4, 6] without the supremum. Our lower bound match the trace term in the upper bound U 1 , which shows the upper bound U 1 is relatively tight.

Theorem 7.

Proof. Recall

Divide λ on both sides,

The Rademacher complexity of G U is

C.4 Proof of Corollary 1

Corollary 1. Fix B > 0 . Let ˆ R gf S ( G T ) = min ( U 1 , U 2 ) where U 1 and U 2 are defined in Theorem 3. For any δ ∈ (0 , 1) , with probability at least 1 -δ over the draw of an i.i.d. sample set S = { z i } n i =1 , if 1 n 2 ∑ i,j K T ( z i , z j ; S ) ≤ B 2 , the following holds for ℓ ( w T , z ) that trained from S ,

where w T = A T ( S ) is the output from the gradient flow (1) at time T by using S as input.

Proof. Apply Theorem 1 to G T and S , for all g ∈ G T

Since g ( z ) ∈ G T corresponds to ℓ ( A T ( S ′ ) , z ) for all S ′ ∈ supp ( µ ⊗ n ) that 1 n 2 ∑ i,j K T ( z ′ i , z ′ j ; S ′ ) ≤ B 2 , it holds for all these feasible S ′ ∈ supp ( µ ⊗ n ) that

From Theorem 3, we have ˆ R S ( G T ) ≤ min( U 1 , U 2 ) . Define ˆ R gf S ( G T ) = min ( U 1 , U 2 ) and take into above inequality,

Since above holds for all S ′ ∈ supp ( µ ⊗ n ) that 1 n 2 ∑ i,j K T ( z ′ i , z ′ j ; S ′ ) ≤ B 2 , and S is also in supp ( µ ⊗ n ) , then if 1 n 2 ∑ i,j K T ( z i , z j ; S ) ≤ B 2 , we have

D Complete Proofs for Stochastic Gradient Flow

D.1 Proof of Theorem 4

Theorem 4. Suppose w ( T ) = w T is a solution of stochastic gradient flow at time T ∈ N with initialization w (0) = w 0 . Then for any z ∈ Z ,

where K t,t +1 ( z , z i ; S ) = ∫ t +1 t ¯ K ( w ( t ); z , z i ) d t with ¯ K defined in Definition 3.

Proof. For each time interval [ t, t +1] and data batch S t , Stochastic gradient flow can be treated as full-batch gradient flow. Applying Theorem 2 for each [ t, t +1] and data batch S t , we have

where

For time T ∈ N ,

D.2 Proof of Theorem 5

Theorem 5. The Rademacher complexity of G T defined in (3) has an upper bound:

̸

where ∆ t ( z i , z j ) = 1 2 [ sup K ( · , · ; S ′ ) ∈K T K t,t +1 ( z i , z j ; S ′ ) -inf K ( · , · ; S ′ ) ∈K T K t,t +1 ( z i , z j ; S ′ ) ] .

Proof. Recall,

where

For t = 0 , 1 , · · · , T -1 , let

Then we have

Since the set on the RHS involves combinations of kernels induced from distinct training set S ′ , it is a strictly larger set than the LHS. Apply Theorem 3 bound U 1 for each G t on S ,

̸

where ∆ t ( z i , z j ) = 1 2 [ sup K ( · , · ; S ′ ) ∈K T K t,t +1 ( z i , z j ; S ′ ) -inf K ( · , · ; S ′ ) ∈K T K t,t +1 ( z i , z j ; S ′ ) ] .

By the monotonicity and linear combination of Rademacher complexity [38] and take in (11),

̸

E Complete Proofs for Case Study

E.1 Generalization bounds for Infinite-width NNs

E.1.1 Proof of Eq. (4)

Proof. We bound U 1 in Theorem 3 for an infinite-width NN. For an infinite-width NN, the NTK keeps unchanged during training:

Then for our loss path kernel, for any z , z ′ ∈ Z and any K ( · , · ; S ′ ) ∈ K T ,

Consider a ρ -Lipschitz loss function, i.e. ∥∇ f ℓ ( w t , z ) ∥ ≤ ρ , e.g. ρ = 1 for hinge loss and logistic loss, ρ = √ 2 for cross-entropy loss with one-hot labels. Then we have

Thus

Then by this inequality and Eq. (12),

Since Θ( x i , x i ) ≥ 0 for i ∈ [ n ] , K ( z i , z i ; S ′ ) ≤ ρ 2 T · Θ( x i , x i ) and Tr ( K ( Z , Z ); S ′ ) ≤ Tr ( ρ 2 T · Θ( X , X )) .

Take these terms into U 1 ,

̸

̸

E.2 Generalization bound for stable algorithms

Let S ′ and S ′ 1 be two datasets that only differ in one data point. We make the following stability assumption for gradient decent.

Assumption 1 (uniform stability of GD.) . Assume ∥ A t ( S ′ ) -A t ( S ′ 1 ) ∥ = ct n for some constant c > 0 . Assume ℓ ( w , z ) is L ℓ -Lipschitz and β ℓ -smooth for any z ∈ Z . Then ∥∇ w ℓ ( A t ( S ′ ) , z ) -∇ w ℓ ( A t ( S ′ 1 ) , z ) ∥ ≤ β ℓ ∥ A t ( S ′ ) -A t ( S ′ 1 ) ∥ = cβ ℓ t n .

This kind of stability results of GD and SGD are proved in [8, 25]. Under these stability and smoothness assumptions, we can bound the deviation of g ( z i ) from its expectation with high probability and further bound the complexity based on Theorem 3.

Theorem 8. For any δ ∈ (0 , 1) , let G δ T ⊂ G T be a 1 -δ subset of G T , i.e. ∣ ∣ G δ T ∣ ∣ = (1 -δ ) |G T | . Under Assumption 1, we have at least one of such G δ T ,

This bound will naturally translate into a generalization bound by equipping with Theorem 1. This bound has a convergence rate of ˜ O (1 / √ n ) . It shows that the complexity of NN trained by GD has a polynomial dependence on L ℓ , β ℓ , and training time T .

Proof. Let G δ, S T = { g ( Z ) = ( g ( z 1 ) , . . . , g ( z n )) : g ∈ G δ T } . Then similar bound as U 2 in Theorem 3 holds for G δ T ,

We consider the covering of G δ T and upper bound the right hand side. Without loss of generality, suppose S ′ and S ′ 1 differ in the first sample. That is S ′ = { z ′ 1 , . . . , z ′ n } and S ′ 1 = { ˆ z ′ 1 , . . . , z ′ n } . For any fixed i ∈ [ n ] and g K S ′ 1 , g K S ′ ∈ G T ,

The first two terms are

The last two terms are

In total,

Similarly g K S ′ ( z i ) -g K S ′ 1 ( z i ) ≤ 1 n ( 2 L 2 ℓ T + cL ℓ β ℓ T 2 ) . Thus ∣ ∣ ∣ g K S ′ ( z i ) -g K S ′ 1 ( z i ) ∣ ∣ ∣ ≤ 1 n ( 2 L 2 ℓ T + cL ℓ β ℓ T 2 ) . Then by McDiarmid's inequality, for any δ ∈ (0 , 1) , with probability at least 1 -δ ,

Then, by a union bound, with probability at least 1 -δ , for all i ∈ [ n ] ,

This means that with probability at least 1 -δ , for all i ∈ [ n ] ,

Use the g K S ′ in this range to construct the G δ T . Then for any g 1 ( Z ) , g 2 ( Z ) ∈ G δ, S T ,

Take ϵ as this value, then N ( G δ, S T , ϵ, ∥∥ 1 ) = 1 . Take this into Eq. (14),

E.3 Norm-constrained neural network

For simplicity, we consider the one-dimensional output in this subsection, i.e. k = 1 , and assume the loss function is ρ -lipschitz for the model output, ∥ ℓ (ˆ y , y ) -ℓ (ˆ y ′ , y ) ∥ ≤ ρ (ˆ y -ˆ y ′ ) for every ˆ y , ˆ y ′ ∈ R . Consider one-layer NNs:

Proposition 2. For the function class of one-layer NN defined above,

For this one-layer NN, we do not need a norm constraint. For L -layer NNs with norm constraints:

where W h ∈ R d h × d h -1 for h ∈ [ L ] with d L = 1 , d 0 = d . σ is the element-wise activation function and is 1-lipschitz with σ (0) = 0 . With these norm constraints of the parameters during the training, we can further bound the trace term in U 1 as follows.

Theorem 9. For the function class of L -layer NN defined above,

This bound shows that this quantity has a linear relation with T . With a finer constraint of ∥ ∥ W i t ∥ ∥ during the training, for example, ∥ ∥ W i t ∥ ∥ ≤ B i,t ′ , t ∈ [ t ′ , t ′ +1] , we can get tighter bound. Although similar to previous norm-based bounds that have a polynomial dependence with the norms of the parameters, our bound has a clear dependence on training time T , which is not achievable from previous approaches. But note these bounds can be very loose since they are worse-case bounds.

E.3.1 Proof of Proposition 2

Proof.

Thus

Since this holds for any K S ′ ∈ K T ,

E.3.2 Proof of Theorem 9

Proof. Denote

where

Then for any h ∈ [ L ] , we can compute

where we have for the first term,

The second term can be bounded as

Thus in total,

Since the tangent kernel ⟨∇ w f ( w , x ) , ∇ w f ( w , x ′ ) ⟩ = ∑ L h =1 〈 ∂f ( w , x ) ∂W h , ∂f ( w , x ′ ) ∂W h 〉 , we obtain an upper bound for the tangent kernel,

Thus

Thus,

Since this holds for any K S ′ ∈ K T ,

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThanks to authors for their submission. Their paper tackles generalization of neural network from neural tangent perspective applicable on gradient flow. The novelty of the paper stems from the new tangent kernel $\\overline{K}(w,z,z')$ proposed. It is defined as an inner product of loss gradients w.r.t. weights and evaluated at two data points $z$ and $z'$. This leads to Loss Path Kernel defined as a path integral w.r.t. $t$ of $\\overline{K}(w(t),z,z')$. After introduction and recollection of NTK theory the definitions are laid out. The paper continues with proving the gradient flow loss is general kernel machine followed up by Rademacher complexity bound derivation. The later section 5 extends these results to SGD. Finally results are used to design neural architecture search (NAS) and demonstrate favorable performance compared with state-of-the-art NAS algorithms through numerical experiments.\n\nPaper follows an interesting idea of generalising tangent kernels to path integrals. It is timely and potentially interesting contribution. Being theoretical work, however, it lacks the clarity in presentation, rigor in definitions and theorems and this strongly undermines its validity. The amendments are recommended int he bellow.\n\nSTRENGTHS:\n+ Well introduced paper with well stated contributions in Table1 presenting paper's contributions to NTK theory.\n\n+ Paper strives for applicability and results are used to design of neural architecture search (NAS) supported by (limited) numerical experiments.\n\nWEAKNESSES:\n- $\\textbf{[Definition of the Loss Path Kernel]}$ For instance the line 108:The Theorem 2 \"... shows that the loss of the NN at a certain fixed time ($T$) is a general kernel machine ...\". But is the limit (when $T \\rightarrow \\infty$ and/or $dt \\rightarrow 0$) also in RKHS? Under what conditions? The existence/convergence of the integral is not commented neither in the body nor in the appendices. I would urge authors to detail this out very thoroughly as assumptions needed for a definition to work would possibly limit the applicability of the results. For instance the Lipschitz gradients may be needed, because loss may be Lipschitz locally, but its norm may be growing with $t$, e.g., in case of overfitting, and the integral (LPK) may still diverge as $T \\rightarrow \\infty$. \n\n- I try to provide alternative view: In case of the gradient flow over whole training data set considered by paper, i.e., gradient descent with infinitesimal steps, as opposed to stochastic (mini-batch) gradient descent, the weight updates are locally diffeomorphisms (locally invertible smooth maps - that is what paper implicitly assumes by infinitesimal steps). Then the kernel (LTK) $\\overline{K}(z,z')$ (Definition 3) can be seen as a pull-back inner product used for instance in [Principles of riemannian geometry in neural networks, Hauser, Michael and Ray, Asok, Advances in neural information processing systems, vol=30,2017] and the crucial Loss Path Kernel (LPK) $K_T(z,z')$ (Definition 4) is defined as a length of the parameterised curve $f \\odot w(t)$ w.r.t. this inner product $\\textbf{if it exists}$! It does not in general unless inner products along the path \"behave nicely\".\n\n- Following up on the above, assumptions required by Corollary 1 should be listed thoroughly. Line 203 reads  ...\"assuming $\\ell({w},z)$ is $L_{\\ell}^2$-Lipschitz ... \". Is it needed? If so, is it enough for Definitions to be well defined (see above).\n\n- $\\textbf{[Corollary 1 (Generalization bound for NN)]}$ the lines 203-208 are very vague, using terms \"usually, often etc.\" which degrade validity of the Corollary 1 and thus the main result of the paper. I strongly advise authors to avoid such terms. Especially in connection to the most important result of the paper. Try to clarify what is assumed, what is already existing result (please cite sources, if so) and what is hypothesis. For instance: l: 203-205: \"... $\\textbf{assuming}$ the $\\ell(w, z)$ is $L_{\\ell}$-Lipschitz. The second component is $\\textbf{often}$ not the dominant term since GD is a stable algorithm when the training data are modified slightly [24, 8]. It $\\textbf{usually}$ decreases with $n$ since GD becomes more stable when $n$ grows. ...\"\n\n- $\\textbf{[ $1/n^2\\sum_{I,j}K_T(z'_i,z'_j;S′)≤B^2$ ,see l 192]}$ This condition limits the datasets $S'$ LTK and LPK can be considered over. How does this limit the applicability/generality of results? \n\n- $\\textbf{[SGD, Section 5]}$ This section generalizes the previous results (bounds) to stochastic gradient descent. The corresponding gradient flow remains utterly deterministic, however. As opposed to previous works, where SGD is related to Langevin dynamics, [Stochastic modified equations and adaptive stochastic gradient algorithms, Li, Qianxiao, Tai, Cheng, Weinan, E, International Conference on Machine Learning, 2017], where path integrals involve higher terms due to quadratic variation induced by stochasticity. The paper neglects this. To improve the impact of the paper extended argument why deterministic flow (ODE) applies and what are assumptions needed.\n\nQUESTIONS:\nQ0: Figure 2c, Fig A4,d) and same for Fig. A5, d) Generalization gap is close to zero from the beginning and increasing later on (overfitting). It suggests the scale is large and thus Rademarcher upper bound is really vacuous compared to a real generalization gap (blue line), despite it improved on previous bounds as reported. Could authors rescale, redo experiments so Figure is more convincing? This is to support the strong claim of the paper mentioned on lines l43 and l55 again and l92 again” …” strong correlation (of the hereby derived upper bound) with generalization error” … and …”highly correlated” … \n\nQ1: Could authors elaborate in detail on assumptions needed for Loss Path Kernel (LPK) $K_T(z,z')$ to be defined for arbitrary $T$? Or is $T$ limited (early stopping assumed)? See section Weaknesses for details.\n\nQ2: The same as Q1 just for stochastic gradient descent versions in Section 5. Also see Weaknesses [SGD, Section 5].\n\nQ3: Condition $\\frac{1}{n^2}\\sum_{i,j} K_T(z'_i, z'_j ; S') \\leq B^2$ on the line $192$ and later similarly for SGD kernels (definition on line 242): This condition limits the datasets $S'$ LTK and LPK can be considered over. How does this limit the applicability/generality of results? \n\nQ4: $\\textbf{Gradient flow vs. Realistic NNs}$ How does the bound deteriorate when practical scenarios, e.g., finite and not infinitesimal learning rate, noise etc., are considered?\n\nLIMITATIONS:\nI find missing assumptions and not stated limitations to be the main weakness of the otherwise very interesting paper. See leading questions: Q1, Q2, Q3 and eventually Q4: $\\textbf{Gradient flow vs. Realistic NNs}$", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new complexity measure for neural networks based on tracking the changes of the weight vector of the entire model.  The work is rooted in theory by linking the proposed measure through the Neural Tangent Kernel framework to Rademacher complexity resulting in a new, meaningfully tight bounds on generalisation.\n\nSTRENGTHS:\nExplanation of the success of overparametrised neural network models using statistical learning theory is still very important problem we haven't solved, and this works seems like a significant development in that space.  \n\nThe proposed complexity measure is linked theoretically to a generalisation bound.\n\nThis looks like a significant amount work, extending and generalising existing theory in the Neural Tangent Kernel framework.\n\nEmpirical results, though shown on small examples, show that the derived generalisation bound is meaningfully tight (i.e. non-vacuous).\n\nFor the most part (with some exceptions mentioned below) the paper is very well written, and guides the reader well through very complicated subject matter.\n\nWEAKNESSES:\nThe maths are very dense...and the 32 page supplement, while admirable, is way too much to get through in the time given for this reviewing period.  I didn't get through the maths, so I couldn't verify it.  It almost feels like this should be a journal rather than a conference paper.\n\nI did get lost a little bit in the maths of the main part of the paper.  It should be possible to follow this paper at high level, taking the proofs of theorems at face value, and for the most part authors do a great job of guiding the reader....but still, there are some things (see questions below) that I found confusing.\n\nThe proposed method is computationally very expensive, and so not that easy to use in practice...but approximations and computational improvements might come in the future, and the theory (assuming the proofs (which I couldn't verify) are solid) is sound and of great interest.\n\nQUESTIONS:\nAs already stated, I didn't follow all the maths, but I wanted to verify my high level understanding of the work.  It seems to me, that the fundamental principle here is that while the hypothesis space of the network is immensely complex (hence very high VC dimension), that during training there is only a small subset of hypotheses \"visited\"...and for the purpose of generalisation guarantees, what matters is the set of hypotheses \"considered\" (when training), not the set of hypotheses available.  And the proposed loss path kernel measure is a measure of the complexity of the hypothesis space \"visited\".  Is this a fair high level summary?\n\nI am completely lost at what $\\mu^{\\otimes n}$ is - is it supposed to be obvious from the context?  And it all gets so complicated by Theorem 5...that I can't really tell from the math how tight the bound is.  Or, since this will always depend on the training path, the tightness of the bound can only be assessed empirically?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new generalization bound based on dynamic NTK, called loss path kernel in the paper. The loss path kernel is a kernel based on the integration of a loss tangent kernel (NTK with loss function) so that the generalization bound can be determined by the training dataset and training trajectory of parameters. Theoretically, the paper provides the loss path kernel generalization bound guarantee and beats their bound over previous work. On the other hand, empirically, the authors verify their theorem with numerical experiments and plug their loss path kernel into NAS and get good results.\n\nSTRENGTHS:\n- The paper studies a very important question of the generalization gap of general neural networks. The definition of the loss path kernel is intuitive and straightforward. From my perspective, the new generalization bound based on the loss path kernel has its value. It considers the training trajectory in the bound rather than calculating the whole function hypothesis class so that the generalization bound can be tighter.\n\n- The paper provides numerical experiments to support their theorem. The numerical simulation and implementation in NAS show that their analysis can be used in practice and their theorem is not vacuous.\n\nWEAKNESSES:\n- The empirical part of writing is not clear enough. I cannot get the message between Theorem 3 and Figure 2 (c). Also, Figure 1 looks strange. The paper does not provide a convincing explanation for the outlier. The implementation of the NAS part is missing, e.g., the search space. The best architecture was not reported. \n\n- The NAS part can be stronger. The paper only considers the random sampling of 100 architectures. However, in practice, we do not use random sampling in NAS anymore because it is not efficient. There are many dynamic sampling ways in NAS, e.g., [1]. The paper may improve its results by using these methods.  \n\n[1] Guo, Zichao, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. \"Single path one-shot neural architecture search with uniform sampling.\" In ECCV 2020.\n\nQUESTIONS:\n- Why do we need Definition 2? It seems that the analysis did not use the general kernel machine. \n- The loss path kernel needs to calculate integration, in GD or SGD. I wonder how to calculate them in practice, e.g., NAS, by math formulation or sampling. \n- The paper only considers gradient flow. Can we extend the results to normal gradient descent with a constant learning rate?\n\nLIMITATIONS:\nIt seems that the analysis does not hold for the ReLU network as it is not continuously differentiable. \n\n======\n\nChange score from 6 to 7 on Aug 10.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission provides data-and-architecture-dependent Rademacher complexity generalization bounds for neural networks. Unlike previous work, the approach takes the evolution of the neural tangent kernel under gradient flow into account. First gradient flow under an evolving kernel is expressed as learning with a general kernel machine on a kernel named Loss Path Kernel. Second, a data-dependent complexity measure of the general kernel machine is computed that is finally used to compute the generalization bound.\n\nSTRENGTHS:\nThe approach is novel as far as I know, although the literature on this topic is vast and it is hard to assess novelty. While computing the bound requires access to the true data distribution and training the network, a heuristic approximation of it from the training data and a few epochs of training is shown to correlate with performance in the experiments. Background, relevant work, and the proposed method are brilliantly presented. I did not notice any glaring issues in the math although checking the long proofs in appendix in detail is not possible given the review workload. Altogether I'm leaning towards acceptance.\n\nWEAKNESSES:\nThe major weaknesses of this work are the limited intuition it provides, an issue with the presentation of the experiments, and possibly high computation required to estimate the bound. I will elaborate on these points below. The second and the third point are critical and I ask the authors to improve them in the revision. \n\nW1. The results abstract away the details of training into a black-box evolving NTK and, although this improves the domain of applicability of the result, the obtained result does not give any insights about the role depth, width, or other choices of architecture on generalization. If theoretical connections between these choices and the evolution of NTK are known in the literature, this paper can be improved by theoretically studying the role of different architectural choices on the loss path kernel and its complexity.\n\nW2. The main motivation for the bound is that it is not limited to infinite-width or single output neural networks. A brief subsection 6.1 right before the experiments focuses on infinite-width neural networks to avoid the dependency on training. The subsection and its placement in the paper confused me about the following experiments. It seems like the experiments afterwards consider a finite-width neural network. If this is the case, I do not understand why this subsection is placed right before them and in the same section.\n\nW3. The experiments allude to intractability of a basic procedure to estimate the bound and computational speedups from approximation. The revision should report the computation or provide measures of complexity or at least clarify in the paper that estimating the bound is not as expensive as training the network. Otherwise one could perform architecture search by trying one or a few train/validation splits.\n\nMinor comment (did not influence score): \n\ni. If the Lipschitz constant is local, it can be influenced by the data or the trajectory of optimization in ways that are not captured in the analysis. This comment did not influence the score as it applies to previous work (Arora et al) as well and I believe fixing them is outside the scope of this work.\n\nii. Practical neural networks are typically trained with relatively large learning rates and especially the early evolution of NTK under large learning rate and gradient flow are known to be very different. This limits the significance of a study on gradient flow.\n\n------------\nPost-rebuttal: Raised the score to 7 since the revision will address W2 and W3.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nSee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}