Title: Learning Trajectories are Generalization Indicators
Venue: NeurIPS 2023
Accepted: yes
Keywords: Generalization

Learning Trajectory
Abstract: This paper explores the connection between learning trajectories of Deep Neural Networks (DNNs) and their generalization capabilities when optimized using (stochastic) gradient descent algorithms. 
Instead of concentrating solely on the generalization error of the DNN post-training, we present a novel perspective for analyzing generalization error by investigating the contribution of each update step to the change in generalization error. This perspective enable a more direct comprehension of how the learning trajectory influences generalization error. Building upon this analysis, we propose a new generalization bound that incorporates more extensive trajectory information.
Our proposed generalization bound depends on the complexity of learning trajectory and the ratio between the bias and diversity of training set. Experimental observations reveal that our method effectively captures the generalization error throughout the training process. Furthermore, our approach can also track changes in generalization error when adjustments are made to learning rates and label noise levels. These results demonstrate that learning trajectory information is a valuable indicator of a model's generalization capabilities.

Learning Trajectories are Generalization Indicators

Abstract

This paper explores the connection between learning trajectories of Deep Neural Networks (DNNs) and their generalization capabilities when optimized using (stochastic) gradient descent algorithms. Instead of concentrating solely on the generalization error of the DNN post-training, we present a novel perspective for analyzing generalization error by investigating the contribution of each update step to the change in generalization error. This perspective enable a more direct comprehension of how the learning trajectory influences generalization error. Building upon this analysis, we propose a new generalization bound that incorporates more extensive trajectory information. Our proposed generalization bound depends on the complexity of learning trajectory and the ratio between the bias and diversity of training set. Experimental observations reveal that our method effectively captures the generalization error throughout the training process. Furthermore, our approach can also track changes in generalization error when adjustments are made to learning rates and label noise levels. These results demonstrate that learning trajectory information is a valuable indicator of a model's generalization capabilities.

1 Introduction

The generalizability of a Deep Neural Network (DNN) is a crucial research topic in the field of machine learning. Deep neural networks are commonly trained with a limited number of training samples while being tested on unseen samples. Depite the commonly used independent and identically distributed (i.i.d.) assumption between the training and testing sets, there often exists a varying degree of discrepancy between them in real-world applications. Generalization theories study the generalization of DNNs by modeling the gap between the empirical risk [37] and the popular risk [37]. Classical uniform convergence based methods [20] adopt the complexity of the function space to analyze this generalization error. These theories discover that more complex function space results in a larger generalization error [38]. However, they are not well applicable for DNNs [33, 22]. In deep learning, the double descent phenomenon [6] exists, which tells that larger complexity of function space may lead to smaller generalization error. This violates the aforementioned property in uniform convergence methods and imposes demands in studying the generalization of DNNs.

Although the function space of DNNs is vast, not all functions within that space can be discovered by learning algorithms. Therefore, some representative works bound the generalization of DNNs based

on the properties of the learning algorithm, e . g . , stability of algorithm [11], information-theoretic analysis [40]. These works rely on the relation between the input ( i . e . , training data) and output (weights of the model after training) of the learning algorithm to infer the generalization ability of the learned model. Here, the relation refers to how the change of one sample in the training data impacts the final weights of model in the stability of algorithms while referring to the mutual information between the weights and the training data in the information-theoretic analysis. Although some works [24, 11] leverage some information from training process to understand the properties of learning algorithm, there is limited trajectory information conveyed.

The purpose of this article is to enhance our theoretical comprehension of the relation between learning trajectory and generalization. While some recent experiments [9, 13, 29] have shown a strong correlation between the information contained in learning trajectory and generalization, the theoretical understanding behind this is still underexplored. By investigating the contribution of each update step to the change in generalization error, we give a new generalization bound with rich trajectory related information. Our work can serve as a starting point to understand those experimental discoveries.

1.1 Our Contribution

Our contributions can be summarized below:

We demonstrate that learning trajectory information serves as a valuable indicator of generalization abilities. With this motivation, we present a novel perspective for analyzing generalization error by investigating the contribution of each update step to the change in generalization error.

Utilizing the aforementioned modeling technique, we introduce a novel generalization bound for deep neural networks (DNNs). Our proposed bound provides a greater depth of trajectory-related insights than existing methods.

Our method effectively captures the generalization error throughout the training process. And the assumption corresponding to this method is also confirmed by experiments. Furthermore, our approach can also track changes in generalization error when adjustments are made to learning rates and label noise levels.

2 Related Work

Generalization Theories Existing works on studying the generalization of DNNs can be divided into three categories: the methods based on the complexity of function space, the methods based on learning algorithms, and the methods based on PAC Bayes. The first category considers the generalization of DNNs from the perspective of the complexity of the function space. Many methods for measuring the complexity of the function space have been proposed, e . g . , VC dimension [39], Rademacher Complexity [4] and covering number [33]. These works fail in being applied to DNN models since the complexity of the function space of a DNN model is too large to deliver a trivial result [41]. This thus motivates recent works to rethink the generalization of DNNs based on the accessible information in different learning algorithms such as stability of algorithm [11], informationtheoretic analysis [40]. Among them, the stability of algorithm [7] measures how one sample change of training data impacts the model weights finally learned, and the information theory [30, 31, 40] based generalization bounds rely on the mutual information of the input (training data) and output (weights after training) of the learning algorithm. Another line is PAC Bayes [19] based method, which bounds the expectation of the error rates of a classifier chosen from a posterior distribution in terms of the KL divergence from a given prior distribution. Our research modifies the conventional Rademacher Complexity to calculate the complexity of the space explored by a learning algorithm, which in turn helps derive the generalization bound. Our approach resembles the first category, as we also rely on the complexity of the function space. However, our method differs as we focus on the function space explored by the learning trajectory, rather than the entire function space. The novelty of our technique lies in addressing the issue of dependence between training data and the function space explored by the learning trajectory, a dependence that is not permitted by the original Rademacher Complexity Theory.

Generalization Analysis for SGD The optimization plays an nonnegligible role in the success of DNN. Therefore, there are many prior works studying the generalization of DNNs by exploring property of SGD, which could be summarized into two categories: stability of SGD and informationtheoretic analysis. The most popular way of the former category is to analyze the stability of the weights updating. Hardt et al. [11] is the first work to analyze the stability of SGD with the requirements of smooth and Lipschitz assumptions. Its follow-up works try to discard the smooth [5], or Lipschitz [25] assumptions towards getting a more general bound. Information-theoretic methods leverage the chain rule of KL-divergence to calculate the mutual information between the learned model weights and the data. This kind of works is mainly applied for Stochastic Gradient Langevin Dynamics(SGLD), i . e . , SGD with noise injected in each step of parameters updating [28]. Negrea et al. [23], Haghifam et al. [10] improve the results using data-dependent priors. Neu et al. [24] construct an auxiliary iterative noisy process to adapt this method to the SGD scenario. In contrast to these studies, our approach utilizes more information related to learning trajectories. A more detailed comparison can be found in Table 2 and Appendix B.

3 Generalization Bound

Let us consider a supervised learning problem with a instance space Z and a parameter space W . The loss function can be defined as f : W×Z → R + . We denote the distribution of the instance space Z as µ . The n i.i.d samples draw from µ are denoted as S = { z 1 , ..., z n } ∼ µ n . Given parameters w , the empirical risk and popular risk are denoted as F S ( w ) ≜ 1 n ∑ n i f ( w , z i ) , and F µ ( w ) ≜ E z ∼ µ [ f ( w , z )] respectively. Our work studies the generalization error of the learned model, i . e . F µ ( w ) -F S ( w ) . For an optimizaiton process, the learning trajectory is represented as a function J : N → W . We use J t to denote the weights of model after t times updating, where J t = J ( t ) . The learning algorithm is defined as A : µ n × R → J , where the second input R denotes all randomness in the algorithm A , including the randomness in initialization, batch sampling et al . . We simply use A ( S ) to represent a random choice for the second input term. Given two functions U, V , ∫ t U ( t )d V ( t ) ≜ ∑ t U ( t )( V ( t +1) -V ( t )) and we use ∥ · ∥ to denote L 2 norm. If S is a set, then | S | denotes the number of elements in S . E t denotes taking the expectiation conditioned on { J i | i ≤ t } .

Let mini-batch B be a random subset sampled from dataset S , and we have | B | = b . The averaged function value of mini-batch B is denoted as F B ( w ) ≜ 1 b ∑ z ∈ B f ( w , z ) . The parameters updated with gradient descent can be formulated as:

where η t is the learning rate for the t -th update. The parameter updating with stochastic gradient descent is:

Let ϵ ( w ) ≜ ∇ F S ( w ) - ∇ F B ( w ) be the gradient noise in mini-batch updating, where w is the weights of a DNN. Then we can transform Equation (2) into:

The covariance of the gradients over the entire dataset S can be calculated as:

Therefore, the covariance of the gradient noise ϵ ( w ) is:

Since for any w we have E ( ϵ ( w )) = 0 , we can represent ϵ ( w ) as C ( w ) 1 2 ϵ ′ , where ϵ ′ is a random distribution whose mean is zero and covariance matrix is an identity matrix. Here, ϵ ′ can be any distributions , including Guassian distribution [12] and S α S distribution [35].

The primary objective of our work is to suggest a new generalization bound that incorporates more comprehensive trajectory-related information. The key aspects of this information are: 1) It should be adaptive and change according to different learning trajectories. 2) It should not rely on the extra information from data distribution µ except from the training data S .

3.1 Investigating generalization alone learning trajectory

As annotated before, the learning trajectory is represented by a function J : N →W , which defines the relationship between the model weights and the training timesteps t . J t denotes the model weights after t times updating. Note that J depends on S , because it comes from the equation J = A ( S ) . We simply use f ( J t ) : Z → R + to represent the function after t -times update. Our goal is to analyze the generalization error, i.e., F µ ( J T ) -F S ( J T ) , where T represents the total training steps.

We reformulate the function corresponding to the finally obtained model as:

Therefore, the generalization error can be rewritten as:

In this form, we divide the generalization error into two parts. ( i ) is the generalization error before the training. ( ii ) t is the generalization error caused by t -step update.

Typically, there is independence between J 0 and the data S . Therefore, we have E ( i ) = 0 . Combining with this, we have:

Analyzing the generalization error after training can be transformed into analyzing the increase of generalization error for each update. This is a straighforward and quite different way to extract the information from learning trajectory compared with previous work. Here, we list two techniques that most used by previous works to extract the information from learning trajectory.

(T1). This method leverages the chaining rule of mutual informaton to calculate a upper bound of the mutual information between J T and the training data S , i . e . I ( S ; J T ) ≤ I ( S ; J t ≤ T ) ≤ ∑ T t =0 I ( S ; J t | J i < t ) . I ( S ; J T ) is the value of concerning for their theory.

(T2). This method assumes we have another data S ′ , which is obtained by replacing one sample in data S with another sample drawing from distribution µ . J ′ is the learning trajectory trained from data S ′ with same randomness value as J . Denote ∆ k ≜ ∥ J k -J ′ k ∥ and assume ∆ 0 = 0 . Then, the value of concerning is ∆ T . The upper bound of ∆ T is calculate by iterately apply the formular ∆ k ≤ c k -1 ∆ k -1 + e k -1 .

(T1) is commonly utilized in analyzing Stochastic Gradient Langevin Dynamics(SGLD) [18, 2, 28], while (T2) is frequently employed in stability-based works for analyzing SGD [11, 15, 5]. Our method offers several benefits, including: 1) We directly focus on the change in generalization error , rather than intermediate values such as ∆ k and I ( S ; J t | J i < t ) , 2) The generalization error is equivalent to the sum of ( ii ) t , while (T1) and (T2) takes the upper bound value of I ( S ; J T ) and ∆ T , and 3) From this perspective, We can extract more in-depth trajectory-related information. For (T1), the computation of I ( S ; J t | J i < t ) primarily involves the information of ∇ F µ ( J t ) , which is inaccessible to us (Detail in Appendix D and Neu et al. [24]). (T2) faces the challenge that only the upper bounds of c k and e k can be calculated. The upper bounds remain unchanged across various learning trajectories. Consequently, both (T1) and (T2) have difficulty conveying meaningful trajectory information.

3.2 A New Generalization Bound

In this section, we introduce the generalization bound based on our aforementioned modeling. Let us start with the definition of commonly used assumptions.

Definition 3.1. The function f is L -Lipschitz, if for all w 1 , w 2 ∈ W and for all z ∈ Z , wherein we have ∥ f ( w 1 , z ) -f ( w 2 , z ) ∥ ≤ L ∥ w 1 -w 2 ∥ .

Definition 3.2. The function f is β -smooth, if for all w 1 , w 2 ∈ W and for all z ∈ Z , wherein we have ∥∇ f ( w 1 , z ) -∇ f ( w 2 , z ) ∥ ≤ β ∥ w 1 -w 2 ∥ .

Definition 3.3. The function f is convex, if for all w 1 , w 2 ∈ W and for all z ∈ Z , wherein we have f ( w 1 , z ) ≥ f ( w 2 , z ) + ( w 1 -w 2 ) T ∇ f ( w 2 , z ) .

Here, L -lipschitz assumption implies that the ∥∇ f ( w , z ) ∥ ≤ L holds. β -smooth assumption indicates the largest eignvalue of ∇ 2 f ( w , z ) is smaller than β . The convexity indicates the smallest eigenvalue of ∇ 2 f ( w , z ) are positive. These assumptions tell us the constraints of gradients and Hessian matrices of the training data and the unseen samples in the test set. Since the values of gradients and Hessian matrices in the training set are accessible, the key role of these assumptions is to deliver knowledge about the unseen samples in the test set.

In the following, we introduce a new generalization bound. We give the assumption required by our new generalization bound in the following.

Assumption 3.4. There is a value γ , so that for all w ∈ { J t | t ∈ N } , we have ∥∇ F µ ( w ) ∥ ≤ γ ∥∇ F S ( w ) ∥ .

Remark 3.5 . Assumption 3.4 gives a restriction with the norm of popular gradient ∇ F µ ( w ) . This assumption is easily satisfied when n is a large number, because we have lim n →∞ ∥∇ F S ( w ) ∥ =

∥∇ F µ ( w ) ∥ . When the n is not large enough, the assumption will hold before SGD enter the neighbourhood of convergent point. Under the case that SGD enters the neighbourhood of convergent point, we give a relaxed assumption and its corresponding generalization bound in Appendix B. According to paper [42], this case will ununsually happen in real situation. Section 4 gives experiments to explore the assumption.

Theorem 3.6. Under Assumption 3.4, given S ∼ µ n , let J = A ( S ) , where A denoted the SGD or GD algorithm training with T steps, we have:

where V ( w ) = ∥∇ F S ( w ) ∥ E U ⊂ S ∥ | U | n ∇ F U ( w ) -n -| U | n ∇ F S/U ( w ) ∥ , V m = max t V ( J t ) , γ ′ = max { 1 , max U ⊂ S ; t | U |∥∇ F U ( J t ) ∥ n ∥∇ F S ( J t ) ∥ } γ and η m ≜ max t η t .

Remark 3.7 . Our generalization bound mainly relies on the information from gradients. V ( w ) is related to the variance of the gradient. When the variance of the gradients across different samples in the training set S is large, then the value of V ( w ) is small, and vice versa. Note that we have | U | < n due to U ⊂ S . Our bound will became trival if E U ⊂ S ∥ | U | n ∇ F U ( w ) -n -| U | n ∇ F S/U ( w ) ∥ = 0 . This rarely happens in real case, because it requires that for all U ⊂ S , we have | U |∇ F U ( w ) = ( n -| U | ) ∇ F S/U ( w ) . We also give a relaxed assumption version of this theorem in Appendix B. The generalization bound provides a clear insight into how the reduction of training loss leads to a increase in generalization error.

Proof Sketch The proof of this theorem is placed in Appendix A. Here, we give the sketch for this proof.

Step 1 Beginning with Equation (8), we decomposite the F µ ( J T ) -F S ( J T ) into a linear part ( gen lin ( J T ) ) and nonlinear part( gen nl ( J T ) ). We have gen lin ( J T ) = ∑ T t =1 ( ii ) lin t , where ( ii ) lin t ≜ ( J t -J t -1 ) T ( ∇ F µ ( J t -1 ) -∇ F S ( J t -1 )) . The nonlinear part is gen nl ( J T ) = F µ ( J T ) -F S ( J T ) -gen lin ( J T ) . We takle these two parts differently. Here, we focus on analyzing gen lin ( J T ) because it dominates under small learning rate. Detail discussion of gen nl ( J T ) is given in Appendix (Propositon A.1 and Subsection C.3)

Step 2 We construct the addictive linear space L J | S ≜ { ∑ T -1 t =0 w t T ∇ f ( J t ) | ∥ w t ∥ ≤ ∆ t } , where ∆ t ≜ ∥ η t ∇ F S ( J t ) ∥ . Then E [gen lin ( J T )] ≤ 2 γ ′ V m E R S ( L J | S ) , where R S ( L J | S ) ≜ E σ sup h ∈L J | S ( 1 n ∑ n i =1 σ i h ( z i )) .

Step 3 Finally, we compute the upper bound of R S ( L J | S ) , which follows same techniques used in Radermacher Complexity theory. By combining this with Proposition A.1, we establish the theorem.

Technical Novety Directly applying the Rademacher complexity to calculate the generalization error bound fails because the large complexity of neural network's function space leads to trival bound[41]. In this work, we want to calculate the complexity of the function space that can be explored during the training process. However, there are two challenges here. First , the trajectory of neural network is a "line", instead of a function space that can be calculated the complexity. To solve this problem, we indroduce the addictive linear space L J | S . This space contains the local information of learning trajectory, and can serve as the pseudo function space. Second , the function space L J | S has a dependent on the sample set S , while the theory of Rademacher complexity requires that the function space is independent with training samples. To decouple this dependence, we adapt the Rademacher complexity and we obtain that E [gen lin ( J T )] ≤ 2 γ ′ V m E R S ( L J | S ) . Here, γ ′ is indroduced to decouple the dependent fact mentioned above.

Next, in order to draw a clearer comparison with the stability-based method, we present the following corollary. This corollary employs the β -smooth assumption to bound gen nl ( J T ) and leverages a similar learning rate setting to that found in stability based works.

Corollary 3.8. If function f ( · ) is β -smooth, under Assumption 3.4 given S ∼ µ n , let J = A ( S ) , η t = c β ( t +1) , M 2 2 = max t E t -1 ( ∥∇ F S ( J t ) + ϵ ( J t ) ∥ 2 ) and M 4 4 = max t E t -1 ( ∥∇ F S ( J t ) + ϵ ( J t ) ∥ 4 ) , where A denoted the SGD or GD algorithm training with T steps, we have:

3.3 Further Analysis

3.3.1 Interpreting the Generalization Bounds

We rewrite the obtained generalization bound here:

The "Bias of Training Set" refers to the disparity between the characteristics of the training set and those of the broader population. To measure this difference, we use the distance between the norm of the popular gradient and that of the training set gradient, as specified in Assumption 3.4. The "Diversity of Training Set" can be understood as the variation among the samples in the training set, which in turn affects the quality of the training data. The ratio Bias of Training Set Diversity of Training Set gives us the property of information conveyed by the training set. It is important to consider the properties of the training set, as the data may not contribute equally to the generalization[36]. The detail version of the equation can be found in Theorem 3.6.

3.3.2 Asymptotic Analysis

We will first analyze the dependent of n for V . The V is calculated as V ( w ) = ∥∇ F S ( w ) ∥ E U ⊂ S ∥ | U | n ∇ F U ( w ) -n -| U | n ∇ F S/U ( w ) ∥ . Obviously, the gradient of individual sample is unrelated to the sample size n . And | U | ∼ n . Therefore, V = O (1) . Similarly, we have E ∫ t dF S ( J t ) √ n √ 1 + Tr(Σ( J t )) ∥∇ F S ( J t ) ∥ 2 2 = O ( 1 √ n ) . As for the O ( η m ) term in Theorem 3.6, we have lim n →∞ O ( η m ) = 0 according to Proposition A.1. We simply assume that O ( η m ) = O ( 1 n c ) . Therefore, our bound has O ( 1 n min { 0 . 5 ,c } ) .

3.3.3 Comparison with Stability-based methods

We first compare our method with the stability-based methods in terms of the trajectory information. In Table 1, we present a summary of stability-based methods, while other methods are outlined in Appendix D. We focus on generalization bounds from previous works that eliminate terms dependent on extra information about data distribution µ , apart from the training data S , using assumptions such as smoothness or Lipschitz continuity. Analyzing Table 1 reveals that most prior works primarily depend on the learning rate η and the total number of training steps T . This suggests that we can achieve the same bound by using an identical learning rate schedule and total training steps, which does not align with our practical experience. Our proposed generalization bound considers the evolution of function values, gradient covariance, and gradient norms throughout the training process. As a result, our bounds encompass more comprehensive information about the learning trajectory.

Next, we give a detail comparison with Hardt et al. [11] in Table 2. The concept of uniform stability is commonly used to evaluate the ability of SGD in generalizaton, by assessing its stability when a single training sample is altered. Our primary point of comparison is with Hardt et al. [11], as their work is considered the most representative in terms of analyzing the stability of SGD. We find that First , the assumption of Uniform Stability requires the gradient norm of all input samples for all weights being bounded by L , whereas our assumption only limits the expectation of the gradients

for the weights during the learning trajectory. Secondly , Uniform Stability uses an epoch structure to model the stochastic gradient descent, whereas our approach regards each stochastic gradient descent as full batch gradient descent with added stochastic noise. The epoch structure complicates the modelling process because it requires a consideration of sampling. As a result, in Hardt et al. [11], the author only considers the setting with batch size 1. Thirdly , the bound of Uniform Stability only uses hyperparameters setting such as learning rate and number of training step. In contrast, our bound contains more trajectory-related information, such as the gradient norm and covariance. Finally , the Uniform Stability provides the generalization bound based on the stability of the algorithm, while our approach leverages the complexity of the learning trajectory. In summary , there are some notable differences between our approach and Uniform Stability, such as the assumptions made, the modelling process, the type of information used in the bound, and the perspectives.

4 Experiments

4.1 Tightness of Our Bounds

Table 3: Numeric comparison with stability-based work on toy examples. The reason for the value of Zhang et al. [43] is large is because that our and Hardt et al. [11] has dependent on L 2 β , while Zhang et al. [43] depends on L 2 . L and β are usually large numbers.

In a toy dataset setting, we compare our generalization bound with stability-based methods.

Reasons for toy examples 1) Some values in the bounds are hard to be calculated. Calculating β (under the β -smooth assumption) and L (under the L -Lipschitz assumption) in stability-based work, as well as the values of V and γ in our proposed bound, are challenging. 2) Stability-based methods require a batch size of 1. The training is hard for batch size of 1 with learning rate setting η t = 1 t in complex datasets.

Constuction of the toy examples In the following, we discuss the construction of the toy dataset used to compare the tightness of the generalization bounds. The training data is X tr = { x i } n i =1 . All the data x i is sampled from Guassian distribution N (0 , I d ) . Sampling ˜ w ∼ N (0 , I d ) ,the ground truth is generated by y i = 1 if ˜ w T x i > 0 else 0 . The weights for learning is denoted as w . The predict ˜ y is calculated as ˜ y i = w T x i . The loss for a simple data point is l i = ∥ ∥ y i -w T x i ∥ ∥ 2 . The training loss is L = ∑ n i =1 l i . The test data is X te = { x ′ i } , where x ′ i = ˜ x ′ i and ˜ x ′ i ∼ N (0 , I d ) . We use 100 samples for training and 1,000 samples for evaluation. The model is trained using SGD for 200 epochs.

We evaluate the tightness of our bound by comparing our results with those in Hardt et al. [11] and Zhang et al. [43] from the original paper. We set the learning rate as η t = 1 βt . Our reasons for comparing with these two papers are : 1) Hardt et al. [11] is a representative study, 2) Both papers have theorems using a learning rate setting of η t = O ( 1 t ) , which aligns with Corollary 3.8 in our paper, and 3) They do not assume convexity. The generalization bounds we compare include Corollary 3.8 from our paper, Theorem 3.12 from Hardt et al. [11], and Theorem 5 from Zhang et al. [43].

Our results are given in Table 3. Our bound is tighter under this setting.

4.2 Capturing the trend of generalization error

In this section, 1) we conduct the deep learning experiment to verify Assumption 3.4 and 2) Verify whether our proposed generalization bound can capture the changes of generalization error. In this experiment, we mainly consider the term C ( J t ) ≜ -2 ∫ t i =0 dF S ( J i ) √ n √ 1 + Tr(Σ( J i )) ∥∇ F S ( J i ) ∥ 2 2 . We omit the term γ ′ and V m , because all the trajectory related information that we want to explore is stored in C ( J t ) . Capturing the trend of generalization error is regarded as an important problem in Nagarajan [21]. Unless further specified, we use the default setting of the experiments on CIFAR-10 dataset [14]

Cifar10

Cifar100

SVHN

7.5

4

4

5.0

2

2

2.5

0

25

50

75

100

0

25

50

75

100

0

25

50

75

100

S

5

S

2

4

2

o

1

2

1

0

0

0

0

25

50

75

100

0

25

50

75

100

0

25

50

75

100

Epoch

Epoch

Epoch

2.5

Fs(t)

1.0

dc(Jt)/dFs(t)

2.0

Fs(Jt)

0.6

0.8

1.5

Fs(Jt)+C(Jt)

0.4

0.6

1.0

0.2

0.4

0.5

0.0

Fs(Jt）-Fs(Jt)

c(Jt)

0.2

0.0

0.0

0

50

100

150

200

0

50

100

150

200

0

50

100

150

200

Epoch

Epoch

Epoch

with the VGG13 [34] network. The experimental details for each figure can be found in Appendix C.2.

Our observations are:

Assumption 3.4 is valid when SGD is not exhibiting extreme overfitting.

The term of C ( J t ) can depict how the generalization error varies along the training process. And it can also track the changes in generalization error when adjustments are made to learnling rates and label noise levels

Exploring the assumption 3.4 for different dataset during the training process To explore the Assumption 3.4, we define γ t ≜ ∥∇ F µ ( J t ) ∥ ∥∇ F S ( J t ) ∥ and ˜ γ t ≜ ∥∇ F S ′ ( J t ) ∥ ∥∇ F S ( J t ) ∥ , where S ′ is another data set i.i.d sampled from distribution µ . Because S ′ is independent with S , we have ˜ γ t ≈ γ t . We found that ˜ γ t is stable around 1 during the early stage of training(Figure 1). When the training loss is reaching a relative small value, ˜ γ t increases as we continue training. This phenomenon remain consistant aross the Cifar10, Cifar100 and SVHN datasets. The γ in Assumption 3.4 can be assigned as γ = max t ˜ γ t . We can always find such γ if the optimizer is not extreme overfitting. Under the extremely overfitting case, we can use the relaxed theorem in Appendix B to bound the generalization error.

The bound capturing the trend of generalization error during training process The generalization error and the C ( J t ) both changes as the training continues. Therefore, we want to verify whether they correlate with each other during the training process. Here, we use the term ∇ F S ′ ( J t ) -∇ F S ( J t ) to approximate the generalization error. We find that ∇ F S ′ ( J t ) -∇ F S ( J t ) has similar trend with C ( J t ) (Figure 2 Center ). What's more, we also find that the curve of ∇ F S ( J t ) + C ( J t ) exhibits a comparable pattern with the curve F S ′ ( J t ) (Figure 2 Left ). To explore whether C ( J t ) reveals influence of the change of F S ( J t ) to the generalization error, we plot d C ( J t ) dF S ( J t ) (Figure 2 Right ) during

1.4

C(JT)

C(JT)

1.2

Fs(J)-Fs()

0.6

Fs(J)-Fs()

1.0

0.5

0.8

0.6

0.4

0.4

0.00

0.05

0.10

0.15

0.20

0.25

0.30

102

Noise

Learning Rate

the training process. d C ( J t ) dF S ( J t ) increases slowly during the early stage of training, but surge rapidly afterward. This discovery is aligned with our intuition about the overfitting.

The complexity of learning trajectory correlates with the generalization error In Figure 3, we carry out experiments under various settings. Each data point in the figure represents the average of three repeated experiments. The results demonstrate that both the generalization error and C ( J t ) increase as the level of label noise is raised (Figure 3 Left ). The another experiments measure C ( J t ) and generalization error for different learning rate and discover that C ( J t ) can capture the trend generalization error. The reasons behind a larger learning rate resulting in a smaller generalization error have been explored in Li et al. [17], Barrett and Dherin [3]. Additionally, Appendix E discusses why a larger learning rate can lead to a smaller C ( J t ) .

5 Limitation

The assumption of small learning rate is required by our method. But this assumption is also common use in previous works. For example, Hardt et al. [11], Zhang et al. [43], Zhou et al. [44] explicitly requires that the learning rate should be small and is decayed with a rate of O ( 1 t ) . Some methods have no explict requirements about this but show that large learning rate pushes the generalization bounds to a trivial point. For example, the generalization bounds in works [5, 16] have a term ∑ T t =1 η 2 t that is not decayed as the data size n increases. The value of this term is unignorable when the learning rate is large. The small learning assumption widens the gap between theory and practice. Eliminating this assumption is crucial for future work.

6 Conclusion

In this study, we investigate the relation between learning trajectories and generalization capabilities of Deep Neural Networks (DNNs) from a unique standpoint. We show that learning trajectories can serve as reliable predictors for DNNs' generalization performance. To understand the relation between learning trajectory and generalization error, we analyze how each update step impacts the generalization error. Based on this, we propose a novel generalization bound that encompasses extensive information related to the learning trajectory. The conducted experiments validate our newly proposed assumption. Experimental findings reveal that our method effectively captures the generalization error throughout the training process. Furthermore, our approach can also track changes in generalization error when adjustments are made to learning rates and the level of label noises.

References

K. Ahn, J. Zhang, and S. Sra. Understanding the unstable convergence of gradient descent. In International Conference on Machine Learning , pages 247-257. PMLR, 2022.

A. Banerjee, T. Chen, X. Li, and Y. Zhou. Stability based generalization bounds for exponential family langevin dynamics. arXiv preprint arXiv:2201.03064 , 2022.

D. G. Barrett and B. Dherin. Implicit gradient regularization. arXiv preprint arXiv:2009.11162 , 2020.

P. L. Bartlett and S. Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research , 3(Nov):463-482, 2002.

R. Bassily, V. Feldman, C. Guzmán, and K. Talwar. Stability of stochastic gradient descent on nonsmooth convex losses. Advances in Neural Information Processing Systems , 33:4381-4391, 2020.

M. Belkin, D. Hsu, S. Ma, and S. Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences , 116 (32):15849-15854, 2019.

O. Bousquet and A. Elisseeff. Stability and generalization. The Journal of Machine Learning Research , 2:499-526, 2002.

N. Chandramoorthy, A. Loukas, K. Gatmiry, and S. Jegelka. On the generalization of learning algorithms that do not converge. arXiv preprint arXiv:2208.07951 , 2022.

J. M. Cohen, S. Kaur, Y. Li, J. Z. Kolter, and A. Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. arXiv preprint arXiv:2103.00065 , 2021.

M. Haghifam, J. Negrea, A. Khisti, D. M. Roy, and G. K. Dziugaite. Sharpened generalization bounds based on conditional mutual information and an application to noisy, iterative algorithms. Advances in Neural Information Processing Systems , 33:9925-9935, 2020.

M. Hardt, B. Recht, and Y. Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning , pages 1225-1234. PMLR, 2016.

S. Jastrzebski, Z. Kenton, D. Arpit, N. Ballas, A. Fischer, Y. Bengio, and A. Storkey. Three factors influencing minima in sgd. arXiv preprint arXiv:1711.04623 , 2017.

S. Jastrzebski, D. Arpit, O. Astrand, G. B. Kerg, H. Wang, C. Xiong, R. Socher, K. Cho, and K. J. Geras. Catastrophic fisher explosion: Early phase fisher matrix impacts generalization. In International Conference on Machine Learning , pages 4772-4784. PMLR, 2021.

A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Y. Lei. Stability and generalization of stochastic optimization with nonconvex and nonsmooth problems. arXiv preprint arXiv:2206.07082 , 2022.

Y. Lei and Y. Ying. Fine-grained analysis of stability and generalization for stochastic gradient descent. In International Conference on Machine Learning , pages 5809-5819. PMLR, 2020.

Y. Li, C. Wei, and T. Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. Advances in Neural Information Processing Systems , 32, 2019.

X. Luo, B. Luo, and J. Li. Generalization bounds for gradient methods via discrete and continuous prior. Advances in Neural Information Processing Systems , 35:10600-10614, 2022.

D. A. McAllester. Pac-bayesian model averaging. In Proceedings of the twelfth annual conference on Computational learning theory , pages 164-170, 1999.

M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of machine learning . MIT press, 2018.

V. Nagarajan. Explaining generalization in deep learning: progress and fundamental limits. arXiv preprint arXiv:2110.08922 , 2021.

V. Nagarajan and J. Z. Kolter. Uniform convergence may be unable to explain generalization in deep learning. Advances in Neural Information Processing Systems , 32, 2019.

J. Negrea, M. Haghifam, G. K. Dziugaite, A. Khisti, and D. M. Roy. Information-theoretic generalization bounds for sgld via data-dependent estimates. Advances in Neural Information Processing Systems , 32, 2019.

G. Neu, G. K. Dziugaite, M. Haghifam, and D. M. Roy. Information-theoretic generalization bounds for stochastic gradient descent. In Conference on Learning Theory , pages 3526-3545. PMLR, 2021.

K. E. Nikolakakis, F. Haddadpour, A. Karbasi, and D. S. Kalogerias. Beyond lipschitz: Sharp generalization and excess risk bounds for full-batch gd. arXiv preprint arXiv:2204.12446 , 2022.

B. Oksendal. Stochastic differential equations: an introduction with applications . Springer Science & Business Media, 2013.

S. Park, U. Simsekli, and M. A. Erdogdu. Generalization bounds for stochastic gradient descent via localized ϵ -covers. Advances in Neural Information Processing Systems , 35:2790-2802, 2022.

A. Pensia, V. Jog, and P.-L. Loh. Generalization error bounds for noisy, iterative algorithms. In 2018 IEEE International Symposium on Information Theory (ISIT) , pages 546-550. IEEE, 2018.

Y. Ren, S. Guo, and D. J. Sutherland. Better supervisory signals by observing learning paths. arXiv preprint arXiv:2203.02485 , 2022.

D. Russo and J. Zou. Controlling bias in adaptive data analysis using information theory. In Artificial Intelligence and Statistics , pages 1232-1240. PMLR, 2016.

D. Russo and J. Zou. How much does your data exploration overfit? controlling bias via information usage. IEEE Transactions on Information Theory , 66(1):302-323, 2019.

L. Sagun, U. Evci, V. U. Guney, Y. Dauphin, and L. Bottou. Empirical analysis of the hessian of over-parametrized neural networks. arXiv preprint arXiv:1706.04454 , 2017.

S. Shalev-Shwartz and S. Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

U. Simsekli, L. Sagun, and M. Gurbuzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning , pages 5827-5837. PMLR, 2019.

B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, and A. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems , 35:19523-19536, 2022.

V. Vapnik. Principles of risk minimization for learning theory. Advances in neural information processing systems , 4, 1991.

V. Vapnik. The nature of statistical learning theory . Springer science & business media, 1999.

V. N. Vapnik and A. Y. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. In Measures of complexity , pages 11-30. Springer, 2015.

A. Xu and M. Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. Advances in Neural Information Processing Systems , 30, 2017.

C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021.

J. Zhang, H. Li, S. Sra, and A. Jadbabaie. Neural network weights do not converge to stationary points: An invariant measure perspective. In International Conference on Machine Learning , pages 26330-26346. PMLR, 2022.

Y. Zhang, W. Zhang, S. Bald, V. Pingali, C. Chen, and M. Goswami. Stability of sgd: Tightness analysis and improved bounds. In Uncertainty in Artificial Intelligence , pages 2364-2373. PMLR, 2022.

Y. Zhou, Y. Liang, and H. Zhang. Understanding generalization error of sgd in nonconvex optimization. Machine Learning , 111(1):345-375, 2022.

Z. Zhu, J. Wu, B. Yu, L. Wu, and J. Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. arXiv preprint arXiv:1803.00195 , 2018.

A Proof of Theorem 3.6

We rewrite the Equation (7) and Equation (8):

and

Using Taylor expansion for the function f ( · ) , we have:

Therefore, we can define ( ii ) lin t as:

The ( ii ) t can be decomposed as ( ii ) t = ( ii ) lin t +( ii ) nl t , where ( ii ) nl t ≜ ( ii ) t -( ii ) lin t . Then Equation 13 can be decomposited as:

Proposition A.1. For the gradient descent or the stochastic gradient descent algorithm, we have:

If T = O ( 1 η m ) , we have:

where η m ≜ max t η t , and we have:

Remark A.2 . We give an experimental exploration of the gen nl ( J T ) in Appendix C.3. We discover that if the optimizer doesn't enter the EoS (Edge of Stability) regime [9], we have gen nl ( J T ) ≈ 0 .

Proof. Analyzing of gen lin ( J T )

Combining with Formula (3), we have

Analyzing of gen nl ( J T )

Here, we denote M ≜ max t ∥∇ F S ( J t ) ∥ . According to the definition of gen nl ( J T ) .

Because all the elements of training set S are sampled from distribution µ , we have lim n →∞ ∇ F S ( w ) = F µ ( w ) , Therefore:

What's more, we also have:

Because F µ ( J T ) -F S ( J T ) = ∑ T t =1 ( ii ) lin t + ∑ T t =1 ( ii ) nl t , we have:

According to the Equation (17), we analyze the generalization error of F J | S ≜ { ∑ T -1 t =0 w t T ∇ f ( J t ) | w t = δ t ∇ f ( J t ) ∥∇ f ( J t ) ∥ } as a proxy for analyzing generalization error of the function trained using SGD or GD algorithm. The value of gen lin ( J T ) is equal to the generalization error of F J | S . To analyze F J | S , we construct an addictive linear space as L J | S ≜ { ∑ T -1 t =0 w t T ∇ f ( J t ) | ∥ w t ∥ ≤ δ t } , where δ t ≜ ∥ η t ∇ F S ( J t ) ∥ . Here, we use J | S to emphasize that J depends on S .

Under Assumption 3.4 (that is introduced in the main paper), we can have the following lemma.

Lemma A.3. Under Assumption 3.4, given S ∼ µ n , let J = A ( S ) , we have:

where V ( w ) = ∥∇ F S ( w ) ∥ E U ⊂ S ∥ | U | n ∇ F U ( w ) -| S |-| U | n ∇ F S/U ( w ) ∥ , V m = max t V ( J t ) and γ ′ = max { 1 , max U ⊂ S ; t | U |∥∇ F U ( J t ) ∥ n ∥∇ F S ( J t ) ∥ } γ .

Proof. For a function h , we define that h µ ≜ E z ∼ µ [ g ( z )] and h S = 1 n ∑ z i ∈ S h ( z i ) . Given a function space, the maximum generalization error of the space can be defined as: Φ( S, H ) ≜ sup h ∈ H ( h µ -h S )

where S ′ is another i.i.d sample set drawn from µ n and σ denotes the Rademacher variable. The σ i in z σ i i denotes z σ i i that belongs to S or S ′ . if σ i = -1 z σ i i ∈ S , otherwise, z σ i i ∈ S ′ .

where S + ≜ { z i | σ i = +1 } and S -≜ { z i | σ i = -1 } , and g S ( w ) ≜ | S |∇ F S ( w ) .

where S ′ + is a subset of S ′ with | S ′ + | = | S + | . Defining k ≜ γ ′ V m , we have:

Therefore, combining Equation (27) and (30), we have E [gen lin ( J T )] ≤ 2 γ ′ V m E R S ( L J | S ) .

Lemma A.4. Given J = A ( S ) , the formula R S ( L J | S ) can be upper bounded with:

Proof. Let us start with the calculation of R S ( w T ∇ f ( J t )) :

̸

where ▲ represents using the relation that for i, j satisfying i = j , we have E σ i σ j = 0 .

̸

Because w i is independent of w j if i = j , we have:

The covariance of gradient noise can be calculated as:

Taking Equation (33) and δ t ≜ ∥ η t ∇ F S ( J t ) ∥ into Equation (34), we have :

When η t is small, δ t ≈ -E ϵ ( J t + 1 -J t ) T ∇ F S ( J t ) ∥∇ F S ( J t ) ∥ ≈ -E ϵ F S ( J t + 1 ) -F S ( J t ) ∥∇ F S ( J t ) ∥ holds, therefore we have:

Theorem A.5. Under Assumption 3.4, given S ∼ µ n , let J = A ( S ) , where A denotes the SGD or GD algorithm training with T steps, we have:

where V ( w ) = ∥∇ F S ( w ) ∥ E U ⊂ S ∥ | U | n ∇ F U ( w ) -| S |-| U | n ∇ F S/U ( w ) ∥ , V m = max t V ( J t ) and γ ′ = max { 1 , max U ⊂ S ; t | U |∥∇ F U ( J t ) ∥ n ∥∇ F S ( J t ) ∥ } γ .

Proof. We rewrite Equation 2 of the update of SGD with batchsize b here:

where we simplify the ϵ ( w t ) as ϵ t , then we can expand the function at f ( J T ) as:

Note that when the learning rate is small, we have f ( J T ) ≈ f lin ( J T ) .

The difference between the distributional value and the empirical value of the linear function can be calculated as:

where ▲ using the equation that E [ ϵ T t ( ∇ F µ ( J t ) -∇ F S ( J t ))] = 0 , according to Equation 20.

Because of E [ F µ ( J T ) -F S ( J T )] = E [ F lin µ ( J T ) + O ( η m ) -F lin S ( J T ) -O ( η m )] = E [ F lin µ ( J T ) -F lin S ( J T )] + O ( η m ) (from Proposition A.1), by applying Lemma A.3 and Lemma A.4, the theorm is proved.

Corollary A.6. If function f ( · ) is β -smooth, under Assumption 3.4 given S ∼ µ n , let J = A ( S ) , η t = c β ( t +1) , M 2 2 = max t E t -1 ( ∥∇ F S ( J t ) + ϵ ( J t ) ∥ 2 ) and M 4 4 = max t E t -1 ( ∥∇ F S ( J t ) + ϵ ( J t ) ∥ 4 ) , where A denoted the SGD or GD algorithm training with T steps, we have:

M

2

β

+2

c

.

where V ( w ) = ∥∇ F S ( w ) ∥ E U ⊂ S ∥ | U | n ∇ F U ( w ) -| S |-| U | n ∇ F S/U ( w ) ∥ , V m = max t V ( J t ) and γ ′ = max { 1 , max U ⊂ S ; t | U |∥∇ F U ( J t ) ∥ n ∥∇ F S ( J t ) ∥ } γ .

Proof. If f ( · ) is β -smooth, we have:

2

2

Combining the two equations, we obtain:

The generalization error can be divided into three parts:

The term' ( A ) ' is caused by using F S ( J t + 1 ) -F S ( J t ) to replace F lin S ( J t + 1 ) -F lin S ( J t ) . The term " ( B ) " is induced by gen nl ( J T ) . Then, we want to give a upper bound of ( A ) using M 4 4 :

where ( ⋆ ) is due to the Equation 44, ( ⋆⋆ ) is due to the update rule of J t and ( ⋆ ⋆ ⋆ ) is que to Hölder's inequality. In the following, we use M 2 2 to give a upper bound for ( B ) :

Taking the upper bound value of "(A)" and "(B)" into Equation 45, we obtain the result.

B Relaxed Assumption and Corresponding Bound

Assumption B.1. There is a value γ , T 0 and ζ , so that for all w ∈ { J t | t ∈ N ∧ t < T 0 } , we have ∥∇ F µ ( w ) ∥ ≤ γ ∥∇ F S ( w ) ∥ and for all w ∈ { J t | t ∈ N ∧ t ≥ T 0 } , we have ∥∇ F µ ( w ) ∥ ≤ γ ∥∇ F S ( w ) ∥ + ζ .

Theorem B.2. Under Assumption B.1, given S ∼ µ n , let J = A ( S ) , where A denotes the SGD or GD algorithm training with T steps, we have:

Proof. Most of the proofs in this part are the same as those in Appendix A, except for Equation 30. The Equation 30 is replaced by:

Remark B.3 . Compared of Theorem 3.6, we have a extra term ∑ T t = T 0 η t ∥∇ F S ( J t ) ∥ ζ here. Since the unrelaxed assumption ∥∇ F µ ( w ) ∥ ≤ γ ∥∇ F S ( w ) ∥ is not satisfied only when ∥∇ F S ( w ) ∥ is relative small, the term ∑ T t = T 0 η t ∥∇ F S ( J t ) ∥ ζ is small value.

C Experiments

C.1 Calculation of C ( J )

To reduce the calculation, we construct a randomly sampled subset S sp = { z sp 1 , ..., z sp n } ⊂ S .

Denote the weights after t -epoch training as X t . We can roughly calculated C ( J t )

C.2 Experimental Details

Here, we give a detail setting of the experiment for each figure.

Figure 1 The learning rate is fixed to 0.05 during all the training process. The batch size is 256. All experiments is trained with 100 epoch. The test accuracy for CIFAR-10, CIFAR-100, and SVHN are 87.64%, 55.08%, and 92.80%, respectively.

Figure 2 The initial learning rate is set to 0.05 with the batch size of 1024. We use the Cosine Annealing LR Schedule to adjust the learning rate during training.

Figure 3 Each point is an average of three repeated experiments. We stop training when the training loss is small than 0.2.

C.3 Experimental exploration of gen nl ( J T )

In this section, our aim is to investigate the conditions under which gen nl ( J T ) ≈ 0 . Since directly calculating the difference | R µ ( J T ) -R S ( J T ) | is challenging, we concentrate on the upper bound value | R µ ( J T ) | + | R S ( J T ) | .

We conduct the experiment using cifar10-5k dataset and fc-tanh network, following the setting of paper [9]. Cifar10-5k[9] is a subset of cifar10 dataset. Building upon the work of [1], we compute the Relative Progress Ratio (RP) and Test Relative Progress Ratio (TRP) throughout the training process. We initially consider the case of gradient descent. The definitions of RP and TRP for gradient descent are as follows:

Therefore, we have:

Following the same way, we have:

Combining Equation (52) and Equation (53), we have:

Therefore, if we have for all t , RP( J t ) ≈ -1 and TRP( J t ) ≈ -1 , then | gen nl ( J T ) | ≈ 0 .

From Figure 4 we find that in stable regime, where the sharpness is below the 2 η , we have TRP ≈ RP ≈ -1 . Under small learning rate, the gradient descent doesn't enter the regime of edge of stability and we have TRP ≈ RP ≈ -1 during whole training process and gen nl ( J T ) ≈ 0 .

Next, we consider the case of Stochastic Gradient Descent (SGD). Due to the stochastic estimation of the gradient, we need to rely on some approximations. Let X i t represent the weights after the t -epoch and i -th iteration of training. We assume a constant learning rate η for SGD. The gradient is approximated as follows:

train loss

train loss

1

0

0

2000

4000

0009

8000

10000

0

0

2000

4000

6000

8000

10000

RP

RP

F0

0

-1

-1

2

0

2000

4000

6000

8000

10000

-2

0

2000

4000

6000

8000

10000

TRP

TRP

0

0

-1

0

2000

4000

6000

8000

10000

-2

0

2000

4000

6000

8000

10000

sharpness

sharpness

200

60

40

100

20

0

2000

4000

6000

8000

10000

D

0

2000

4000

6000

8000

10000

iteration

iteration

n=0.01

n=0.05

and we appximate ∇ F S ′ ( X i t ) as:

Therefore, we have:

train loss

train loss

train loss

1

1

1

0

0

2000

4000

6000

8000

10000

0

0

2000

4000

6000

8000

10000

0

0

2000

4000

6000

8000

10000

RP

RP

RP

1

Fo

Fo

0

1

1

1

-2

0

2000

4000

6000

8000

10000

2

0

2000

4000

6000

8000

10000

2

0

2000

4000

6000

8000

10000

TRP

TRP

TRP

-0

0

0

-1

-1

0

2000

4000

6000

8000

10000

2

0

2000

4000

6000

8000

10000

2

2000

4000

6000

8000

10000

sharpness

sharpness

sharpness

200

A

100

100

100

50

50

2000

4000

Epoch

6000

8000

10000

0

2000

4000

Epoch

6000

8000

10000

0

2000

4000

Epoch

6000

8000

10000

nef =0.01

nef =0.05

Nef = 0.1

We calculated the effect learning rate for SGD as η ef ≜ n B η . Figure 5 shows that the conclusions of SGD are similar as GD, except that the conditions of entering EoS are different.

D Other Related Work

This part compares the works that is not listed in Table 2. Table 4 gives other trajectory based generalization bounds. [25] is a stability based work designed mainly for generalization of gradient descent. It removes the Lipschitz assumption, and replaced by the term ∑ T t =1 η t 1 n ∑ n i =1 ∥∇ f ( J t , z i ) ∥ 2 in the generalization bounds. This helps enrich the trajectory information in the bounds. The limitation of this work is that it can only apply to the gradient descent and it is hard to extend to the stochastic gradient descent. Neu et al. [24] adapt the information-theretical generalization bound to the stochastic gradient descent. The Theorem 1 in Neu et al. [24] contains rich information about the learning trajectory, but most is about ∇ F µ ( w ) , which is unavailable for us. Therefore, we mainly consider the result of Corollary 2 in Neu et al. [24], which removes the term ∇ F µ ( w ) by the assumption listed in Table 4. For this Collorary, the remained information within trajectory is merely the √ Tη 2 . Althouth Neu et al. [24] dosen't require the assumption of small learning rate, the bound contains the dimension of model, which is large for deep neural network. Compared with these work, our proposed method has advantage in that it can both reveal rich information about learning trajectory and applied to stochastic gradient descent.

Chandramoorthy et al. [8] analyzes the generalization behavior based on statistical algorithmic stability. The proposed generalization bound can be applied into algorithms that don't converge. Let S ( i ) be the dataset obtained by replace z i in S with another sample z ′ i draw from distribution µ .The generalization bound relies on the stability measure m ≜ sup { 1 T ∑ T -1 t =0 f ( J t | S, z ) -1 T ∑ T -1 t =0 f ( J t | S ( i ) , z ) | z ∈ Z , i ∈ [ n ] } . We don't directly compare with this method because the calculation of m relies on S ( i ) which contains sample outside of S . Therefore, we treat this result as intermediate results. More assumption is needed to remove this dependence of the information about the unseen samples, i . e ., the samples outside set S .

E Effect of Learning Rate and Stochastic Noise

In this part, we want to analyze how learning rate and the stochastic noise jointly affect our proposed generalization bound. Specifically, we denote p t ( w ) as the distribution of the J t during the training with multiple training steps. Following the work [12], we consider the SDE function as an approximation, which is shown as below:

The SDE can be regarded as the continuous counterpart of Equation(3) when sets the distribution of noise term ϵ ′ in Equation(3) as Gaussian distribution. The influence of the noise ϵ on p t ( w ) is shown in the following theorem.

Theorem E.1. When the updating of the weight w follows Equation (59), the covariance matrix C is a hessian matrix of a function with a scalar output, then we have:

Remark E.2 . Previous studies [45, 32, 12] tell that the covariance matrix C is proximately equal to the hessian matrix of the loss function with respect to the parameters of DNN. Thus, the above condition that the covariance matrix C is a hessian matrix of a function with scalar output is easy to be satisfied. Formula (60) contains three parts. The item F S ( w ) p t ( w ) enlarge the probability of parameters being located in the parameter space with low F S ( w ) . ∇ Tr ( C ( w )) and C ( w ) ∇ w log( p t ( w )) ususally

contradict with each other. ∇ Tr ( C ( w )) enlarge the probability of parameters being located in the parameter space with low Tr ( C ( w )) value, while C ( w ) ∇ w log( p t ( w )) serves as a damping factor to prevent the probability from concentrating on a small space. Therefore, setting larger learning rate gives stronger force for the weight to the area with lower Tr ( C ( w )) values. According to Equation 5, we also have a lower Σ( w ) . As a result, large learning rate causes a small lower bound in Theorem 3.6

Proof. Based on the condition described above, we can infer that C ( w ) = ∇∇ G ( w ) , where G is a function with a scalar output.

We first prove that ∇· C ( w ) = ∇ Tr( C ( w )) as below:

So far, we can infer that ∇· C = ∇ Tr( C ) . According to Fokker-Planck equation( [26]), we have:

Therefore, the theorem is proven.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel generalization bound that incorporates trajectory information, aimed at providing deeper insights than existing methods on generalization at different points during training. The key idea is to analyze the increase in generalization error at each point in training by linearizing the network. Experimental results confirm the effectiveness of the proposed approach in capturing generalization error throughout the training process, even with adjustments to learning rates and label noise levels.\n\nSTRENGTHS:\n**Originality**\nAs far as I am aware, this is the first generalization bound of its kind; I'm not aware of any other that uses the approach used in this paper. A major strength of the paper is that the derivation of the bound and bound itself are simple to understand, but still seem relatively powerful.\n\n**Quality**\nThe theory in this paper appears solid. Experiments are also conducted well; testing on VGG13 trained on CIFAR-10 is a good choice. The authors experimentally validate the theoretical assumptions, which also is a major plus. I also view the fact that the bounds predict empirical generalization performance in Figure 3 as a major strength.\n\n**Clarity**\nGenerally speaking, the paper is understandable.\n\n**Significance**\nThe paper seems to take a relatively different approach than other works deriving generalization bounds for neural networks. Thus, I view the main significance of the paper as providing a new set of theoretical techniques. This paper is likely to be of significance to the field of developing practical generalization bounds for neural networks.\n\nWEAKNESSES:\nIn my mind, the main drawback of the paper is that it's hard to establish the significance of the proven results without comparing the numerical generalization bounds implied by Theorem 3.6 with those proved by prior work. Table 2 is helpful, but it would also be good to establish the relative tightness of these bounds numerically if possible. Also, as the authors point out, the small learning rate assumption is restrictive, but also in line with some prior work.\n\nAlso, here are several issues with the writing throughout the paper. For instance, \n- \"Even though, the function space of DNN is large,\" in the introduction\n- \"studying the generalization of DNNs by exploring property of SGD,\" in related work\n- \" the value of concerning\" in section 3\n- \"restricttion\" in section 3\n- \"popular gradient\" in section 3\n\nQUESTIONS:\nIs there some way to at least estimate $\\gamma'$ and $\\mathbb{V}_m$?\n\nRelatedly, how does the tightness of the bounds produced by this method compare to other practical generalization bounds?\n\nLIMITATIONS:\nLimitations are adequately addressed; the authors devote an entire section of the paper to limitations. No negative potential societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the generalization of a general function class under the (S)GD algorithm with minimal assumptions.\nSpecifically, it gives a generalization (upper) bound based on several training trajectory characters during training: variance of the gradients, gradient norm,  training loss values and the learning rate. \nNumerical experiments are conducted to verify the assumption and the generalization bound.\n\nSTRENGTHS:\n* The paper is written clearly and the ideas have been presented in the proper order.\n\n* This paper studies the generalization for a very general class function that not only includes for neural network models, which can provide more insights in other non neural network based scenarios.\n\nWEAKNESSES:\nThe major weakness is that this paper does not go into detail about studying the asymptotic order of the generalization bound in Theorem 3.6. \nTo show the superiority of this new proposed method in this paper, it is very important to provide (at least) the asymptotic analysis on the generalization bound with respect to the sample size $n$.\n\nFor example, in which case does the model benign overfitting (generalization gap goes to zero as $n \\xrightarrow{} \\infty$)?\nCan this new generalization bound cover (or outperform) previous bounds in some specific settings, e.g., linear regression, kernel regression, or overparameterization neural networks?\n\nIn the current version of this paper, even though it provides a general framework for generalization analysis, I cannot see the potentiality of this generalization bound.\n\n---\n\nAnother weakness is for the Assumption 3.4. \nIn fact, this assumption is trivially hold for finite set $\\mathrm{w}$ because we can always take the supremum $\\sup_{w \\in \\mathrm{w}} \\|\\nabla F_{\\mu}(w)\\|$ and the infimum $\\inf_{w \\in \\mathrm{w}} \\|\\nabla F_{S}(w)\\|$ and then get $\\gamma$.\nHowever, once the set $\\mathrm{w}$ contains a point that is a stationary point of the empirical risk, i.e., $\\|\\nabla F_{S}(w^*)\\| = 0$ for some $w^* \\in \\mathrm{w}$, then Assumption 3.4 implies that $w^*$ is also a stationary point of the population risk $F_{\\mu}(w)$, which is a very restrictive property as the data distribution matters. \nIt is also shown in the Figure 1 that as the training loss goes to zero (epoch increases), the $\\tilde{\\gamma}_t$ may diverge.\n\nThat is to say, the relaxed Assumption A.7 is more reasonable.\n\nQUESTIONS:\nFor Theorem 3.6, the optimal generalization bound one can get is $\\mathcal{O}(\\eta_m)$, where $\\eta_m = \\max_t \\eta_t$. \nThis means that to get a non-vacuous bound, $\\eta_m$ should decay to zero with respect to the sample size $n$, but the sample size dependent learning rate is not common in the practical setting, so why there is such a term?\nCompared with the uniform-stability based generalization bound $\\mathcal{O}\\(\\frac{\\sum_{t=1}^T \\eta_t}{n}\\)$ in *Hardt eta al.*,  we can see that as long as $T = o(n)$, constant step-size SGD is provably to having a non-vacuous bound, so what is the gap here?\n\nLIMITATIONS:\nThis paper proposes a framework on the generalization analysis for a wide class of function, but more works are still needed to be done to show the superiority of this new method.\n\nAs mentioned in the weakness part, I think this paper can benefit from the following two aspects:\n* more asymptotic analysis on the generalization bound in common machine learning settings, such as linear regression, kernel regression and overparmeterized neural networks.\n* since the bound in this paper does not require that $f$ is a neural network model, I think it would be more convincing if the authors could add some examples to show the generalization of other non neural network models, such as gradient boosting.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper present a new generation bound with moderate realistic assumptions that incorporate new information from gradients and trajectory of learning.

STRENGTHS:
1 - the paper is well written and easy to follow.
2 - the paper does a great job comparing their results with previous literature on this topic. 
3 - their experiments shows a promising result.

WEAKNESSES:
1- experiments are limited. This paper can definitely benefit from more experiments, however due to page limit, it is too much to ask.

QUESTIONS:
see above.

LIMITATIONS:
The paper address their limitation in their paper well.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposed a novel generalisation error bound which takes the learning trajectory of neural networks into consideration.\nInstead of focusing on the post-trained neural networks, the proposed bound bases on the parameter updates during the learning of the neural networks.\nThe core proof of the bound relies on the assumption 3.4, as well as decomposing the difference between the updates on the true generative distribution and the training instances, i.e. $F\\_{\\mu}(\\mathbf{J}\\_{T}) - F\\_{S}(\\mathbf{J}\\_{T})$, into a linear part $\\mbox{gen}^{lin}(\\mathbf{J}\\_{T})$ and non-linear part $\\mbox{gen}^{nl}(\\mathbf{J}\\_{T})$.\nWith reasonable assumptions, the authors bound the non-linear part with $\\mathcal{O}(\\eta\\_{m})$.\nRegarding the upper bound of the linear part, it relies on the other assumptions introduced in Section 3.2.\n\nSTRENGTHS:\n1. The main result, i.e. a generalisation bound that takes the learning trajectory information, is novel and interesting.\nPractical exploration also demonstrates that the learning trajectory information helps neural networks to obtain better generalisation performance.\nFor example, knowledge distillation [1] can be considered a method to update the original labels to incorporate the predictions from the teacher model.\nThere also exists works that directly modify the labels with learning trajectory information, e.g. [2], and they have shown that the learning trajectory information are indeed helpful for generalisation.\nSo, a theoretical work that can bound the generalisation errors with learning trajectory information is interesting to the community.\n\n2. The proof is well sketched, thus relatively straightforward to follow: not only the sketch from line 227 to line 238, also the formal proof in Appendix A.1.\n\n3. The authors have also provided empirical evidence to show that the core assumption 3.4 hold, as well as how the generalisation error varies during the learning of neural networks.\n\n4. The overall structure of the paper is clear, and the structure is also well-organised (excluding Section 1, my reasons are given below).\n\n[1] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. \"Distilling the knowledge in a neural network.\" arXiv preprint arXiv:1503.02531 (2015).\n[2] Ren, Yi, Shangmin Guo, and Danica J. Sutherland. \"Better supervisory signals by observing learning paths.\" arXiv preprint arXiv:2203.02485 (2022).\n\nWEAKNESSES:\n### Major\n\n1. **Displaying the main result at the beginning**:\nThis is not a concern about the technical side of this work, but rather a comprehensibility issue.\nWithout reading Section 3, the meanings of the notations in Equation 1 are mysterious, and I cannot interpret it.\nFrom my perspective, the Section 1.1 does hinder the comprehensibility of this work.\n\n2. **Lack of necessary details of the experiments in Section 4**:\nAfter reading the supplementary materials, there are still some necessary details of the experiments in Section 4 I didn't find.\nFor example, in line 284, the authors specify $S'$ to be another data set.\nHowever, since the data distribution of CIFAR is unknown, I can only assume that both $S$ and $S'$ are subsets of CIFAR, whereas I'm not certain about that.\nTo fully reproduce these experiments, more details are necessary.\n\n3. **Experiment in line 271 on Gaussian data**:\nThe authors omit $\\gamma'$ due to the lack of the true data distribution in line 271.\nThis can be relatively easily solved by sampling training instances from a Gaussian distribution.\n\n### Minor\n\n1. The first $w$ in line 148 is not bold.\n\n2. The font size of all figures is too small.\n(I understand that the authors may want to save some space, but the texts are really too small to read.)\n\nQUESTIONS:\nMy questions are pretty much the same as the weakness I listed above.\n\nThe only suggestion I want to raise is to move Section 1.1 after Section 3.2.\n\nLIMITATIONS:\nYes, the authors have explicitly and clearly pointed out the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the connection between the learning trajectories of DNNs and their generalization when optimized using SGD. Its main contribution is that it provides a good perspective for generalization error analysis by studying the contribution of the learning trajectory. Based on their analysis of the learning trajectory, a new generalization bound is provided for DNNs.\n\nSTRENGTHS:\nOn the whole, the theoretical analysis perspective and ideas of this paper have a certain value for the theoretical research work of deep neural networks. The main strengths are:\n1. It provides a perspective from the learning trajectory for the theoretical analysis of deep neural networks.\n2. The proposed generalization error bound track changes in adjustments of learning rate and noise level.\n\nWEAKNESSES:\n1.\tIt is unknown how tight the generalization error bounds given in the paper are. Furthermore, the impact of commonly used learning rate schedulers (e.g., exponential decay schedulers) on generalization error bounds has not been adequately analyzed.\n2.\tThe description of the insight of the theorem is not detailed enough. For example, the insight that Theorem 3.6 can bring to the reader is not presented in detail. In addition, what insights can Corollary 3.8 bring to readers?\n3.\tThe description of the experiment in the paper is not detailed enough. For example, the neural network models used for Cifar-100 and SVHN are not indicated.\n4.\tThe experiments done were not extensive enough. I'm curious to know whether the proposed theorem is also applicable to other commonly used neural network structures like ResNet, not just VGG.\n\nQUESTIONS:\n1.\tCan the authors give an analysis and description of how tight the proposed generalization error bound is? Under what conditions does the generalization error inequality in Theorem 3.6 take the equal sign? Besides, how does the learning rate scheduler affect generalization error?\n2.\tEquation (11) in Corollary 3.8 is confusing. In this formula, both the integral of the variable t and the sum of t are included, which makes it unclear to the reader that the author regards the learning trajectory as a continuous variable (maybe the reason is that the learning rate is assumed to be sufficiently small?), or treat the trajectory as a discrete variable?\n3.\tCan the author describe in detail the insights Theorem 3.6 and Corollary 3.8 bring to the reader? What insights can these theorems bring to researchers about the optimization of neural network algorithms?\n4.\tCan the authors describe their experiments more specifically? For example, a more specific description of the neural network model used for different datasets.\n5.\tWhy didn't the author show the test accuracy of the trained neural network model on different datasets? Can the test accuracy be put in an appendix to show that the experimentally obtained accuracy is reasonable or acceptable?\n6.\tIs the proposed theorem (generalization error bound) also suitable for NLP tasks? The supporting material provided by the author contains the relevant code of WikiText2, but the relevant results do not seem to be shown in the paper. Can the authors show experimental results on the WikiText2 dataset?\n7.\tAuthors need to correct the full text. In some sentences, there is a missing space between the content of the paper and the index of the reference, such as \"experiments[9, 13] have\" in line 40 of the paper. Besides, Equation 38 is incomplete.\n8.\tWhy \\Delta FS (Jt)- \\Delta FS(Jt) (line 293 of the paper) can be used to approximate the generalization error? It is different from the generalization error in formula (10). What is the author's reason for using it instead of generalization error in Equation (10) in the experiments?\n9.\tThe symbols of L2 norm in the paper are not unified. Some formulas do have punctuation.\n\nLIMITATIONS:\nThe authors describe their limitation as their method requiring small learning rates. How to eliminate this assumption is a direction worth studying in the future.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}