Title: Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regression
Venue: NeurIPS 2023
Accepted: yes
Keywords: data-algorithm dependent generalization analysis

overparameterized linear regression
Abstract: One of the major open problems in machine learning is to characterize generalization in the overparameterized regime, where most traditional generalization bounds become inconsistent even for overparameterized linear regression. In many scenarios, this failure can be attributed to obscuring the crucial interplay between the training algorithm and the underlying data distribution. This paper demonstrate that the generalization behavior of overparameterized model should be analyzed in a both data-relevant and algorithm-relevant manner. To make a formal characterization, We introduce a notion called data-algorithm compatibility, which considers the generalization behavior of the entire data-dependent training trajectory, instead of traditional last-iterate analysis.  We validate our claim by studying the setting of solving overparameterized linear regression with gradient descent. Specifically, we perform a data-dependent trajectory analysis and derive a sufficient condition for compatibility in such a setting. Our theoretical results demonstrate that if we take early stopping iterates into consideration, generalization can hold with significantly weaker restrictions on the problem instance than the previous last-iterate analysis.

Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regression

Abstract

One of the major open problems in machine learning is to characterize generalization in the overparameterized regime, where most traditional generalization bounds become inconsistent even for overparameterized linear regression [46]. In many scenarios, this failure can be attributed to obscuring the crucial interplay between the training algorithm and the underlying data distribution. This paper demonstrate that the generalization behavior of overparameterized model should be analyzed in a both data-relevant and algorithm-relevant manner. To make a formal characterization, We introduce a notion called data-algorithm compatibility, which considers the generalization behavior of the entire data-dependent training trajectory, instead of traditional last-iterate analysis. We validate our claim by studying the setting of solving overparameterized linear regression with gradient descent. Specifically, we perform a data-dependent trajectory analysis and derive a sufficient condition for compatibility in such a setting. Our theoretical results demonstrate that if we take early stopping iterates into consideration, generalization can hold with significantly weaker restrictions on the problem instance than the previous last-iterate analysis.

1 Introduction

Although deep neural networks achieve great success in practice [13, 14, 63], their remarkable generalization ability is still among the essential mysteries in the deep learning community. One of the most intriguing features of deep neural networks is overparameterization, which confers a level of tractability to the training problem, but leaves traditional generalization theories failing to work. In generalization analysis, both the training algorithm and the data distribution play essential roles [23, 28]. For instance, a line of work [46, 74] highlights the role of the algorithm by showing that the algorithm-irrelevant uniform convergence bounds can become inconsistent in deep learning regimes. Another line of work [8, 68] on benign overfitting emphasizes the role of data distribution via profound analysis of specific overparameterized models.

Despite the significant role of data and algorithm in generalization analysis, existing theories usually focus on either the data factor ( e.g. , uniform convergence [46] and last iterate analysis [8, 68]) or the algorithm factor ( e.g. , stability-based bounds [24]). Combining both data and algorithm factor into

(a) Linear Regression

(b) Corrupted MNIST

105

101

30

OE

ExcessRisk

TestError

104

100

25

Training Error

25

Risk

103

Loss

Error

101

Error

20

20

Excess

102

Training

Training

102

Test

15

15

101

10-3

10

10

100

10-1,

10-4

5

102

101

103

Epochs

105

107

105

0

50

100

Epochs

150

200

250

300

0

generalization analysis can help derive tighter generalization bounds and explain the generalization ability of overparameterized models observed in practice. In this sense, a natural question arises:

How to incorporate both data factor and algorithm factor into generalization analysis?

To gain insight into the interplay between data and algorithms, we provide motivating examples of a synthetic overparameterized linear regression task and a classification task on the corrupted MNIST dataset in figure 1. In both scenarios, the final iterate with less algorithmic information, which may include the algorithm type ( e.g. , GD or SGD), hyperparameters ( e.g. , learning rate, number of epochs), generalizes much worse than the early stopping solutions (see the Blue Line). In the linear regression case, the generalization error of the final iterate can be more than × 100 larger than that of the early stopping solution. In the MNIST case, the final iterate on the SGD trajectory has 19.9% test error, much higher than the 2.88% test error of the best iterate on the GD trajectory. Therefore, the almost ubiquitous strategy of early stopping is a key ingredient in generalization analysis for overparameterized models, whose benefits have been demonstrated both theoretically and empirically [1, 2, 27, 34, 41, 72]. By focusing on the entire optimization trajectory and performing data-dependent trajectory analysis, both data information and the dynamics of the training algorithm can be exploited to yield consistent generalization bounds.

When we take the algorithm into consideration and analyze the data-dependent training trajectory, generalization occurs if the minimum excess risk of the iterates on the training trajectory converges to zero, as the sample size tends to infinity. This accords with the real practice of training deep neural networks, where one can pick up the best parameter on the training trajectory, by calculating its loss on a validation dataset. We dub this notion of generalization as as data-algorithm-compatibility , which is formally defined in Section 3.2.

The significance of compatibility comes in three folds. Firstly, it incorporates both data and algorithm factors into generalization analysis, and is suitable for the overparameterization regime (see Definition 3.1). Secondly, it serves as a minimal condition for generalization, without which one cannot expect to find a consistent solution via standard learning procedures. Consequently, compatibility holds with only mild assumptions and applies to a wide range of problem instances (see Theorem 4.1). Thirdly, it captures the algorithmic significance of early stopping in generalization. By exploiting the algorithm information along the entire trajectory, we arrive at better generalization bounds than the last-iterate analysis (see Table 1 and 2 for examples).

To theoretically validate compatibility, we study it under overparameterized linear regression setting. Analysis of the overparameterized linear regression is a reasonable starting point to study more complex models like deep neural networks [17, 57], since many phenomena of the high dimensional non-linear model are also observed in the linear regime ( e.g. , Figure 1). Furthermore, the neural

tangent kernel (NTK) framework [4, 26] demonstrates that very wide neural networks trained using gradient descent with appropriate random initialization can be approximated by kernel regression in a reproducing kernel Hilbert space, which rigorously establishes a close relationship between overparameterized linear regression and deep neural network training.

Specifically, we investigate solving overparameterized linear regression using gradient descent with constant step size, and prove that under some mild regularity conditions, gradient descent is compatible with overparameterized linear regression if the effective dimensions of the feature covariance matrix are asymptotically bounded by the sample size. Compared with the last-iterate analysis [8], the main theorems in this paper require significantly weaker assumptions, which demonstrates the benefits of data-relevant and algorithm-relevant generalization analysis.

We summarize our contributions as follows:

We formalize the notion of data-algorithm-compatibility, which highlights the interaction between data and algorithm and serves as a minimal condition for generalization.

We derive a sufficient condition for compatibility in solving overparameterized linear regression with gradient descent. Our theory shows that generalization of early-stopping iterates requires much weaker restrictions in the considered setting.

Technically, we derive time-variant generalization bounds for overparameterized linear regression via data-dependent trajectory analysis. Empirically, we conduct the various experiments to verify the the theoretical results and demonstrate the benefits of early stopping.

2 Related Works

Data-Dependent Techniques mainly focus on the data distribution condition for generalization. One of the most popular data-dependent methods is uniform convergence [7, 29, 74, 75]. However, recent works [46, 47] point out that uniform convergence may not be powerful enough to explain generalization, because it may only yield inconsistent bound in even linear regression cases. Another line of works investigates benign overfitting, which mainly studies generalization of overfitting solutions [8, 21, 35, 68, 70, 76, 77].

Algorithm-Dependent Techniques measure the role of the algorithmic information in generalization. Aline of works derives generalization bounds via algorithm stability [9, 12, 18, 19, 24, 31, 32, 45, 67]. A parallel line of works analyzes the implicit bias of algorithmic information [11, 25, 39, 40, 59, 64], which are mainly based on analyzing a specific data distribution ( e.g. , linear separable).

Other Generalization Techniques. Besides the techniques above, there are many other approaches. For example, PAC-Bayes theory performs well empirically and theoretically [16, 42, 43, 48, 49, 58, 61] and can yield non-vacuous bounds in deep learning regimes [50, 54]. Furthermore, there are other promising techniques including information theory [6, 56, 71] and compression-based bounds [3].

Early Stopping has the potential to improve generalization for various machine learning problems [5, 30, 33, 38, 53, 62, 69, 74]. A line of works studies the rate of early stopping in linear regression and kernel regression with different algorithms, e.g. , gradient descent [72], stochastic gradient descent [15, 37, 51, 55, 66], gradient flow [2], conjugate gradient [10] and spectral algorithms [22, 36]. Beyond linear models, early-stopping is also effective for training deep neural networks [27, 34]. Another line of research focuses on the signal for early stopping [20, 52].

3 Preliminaries

In this section, we formally define compatibility between the data distribution and the training algorithm, starting from the basic notations.

3.1 Notations

Data Distribution. Let D denote the population distribution and z ∼ D denote a data point sampled from distribution D . Usually, z contains a feature and its corresponding response. Besides, we denote the dataset with n samples as Z ≜ { z i } i ∈ [ n ] , where z i ∼ D are i.i.d. sampled from distribution D .

Loss and Excess Risk. Let ℓ ( θ ; z ) denote the loss on sample z with parameter θ ∈ R p . The corresponding population loss is defined as L ( θ ; D ) ≜ E z ∼D ℓ ( θ ; z ) . When the context is clear, we omit the dependency on D and denote the population loss by L ( θ ) . Our goal is to find the optimal parameter θ ∗ which minimizes the population loss, i.e., L ( θ ∗ ) = min θ L ( θ ) . Measuring how a parameter θ approaches θ ∗ relies on a term excess risk R ( θ ) , defined as R ( θ ) ≜ L ( θ ) -L ( θ ∗ ) .

Algorithm. Let A ( · ) denote a iterative algorithm that takes training data Z as input and outputs a sequence of parameters { θ ( t ) n } t ≥ 0 , where t is the iteration number. The algorithm can be either deterministic or stochastic, e.g. , variants of (S)GD.

3.2 Definitions of Compatibility

Based on the above notations, we introduce the notion of compatibility between data distribution and algorithm in Definition 3.1. Informally, compatibility measures whether a consistent excess risk can be reached along the training trajectory. Note that we omit the role of the loss function in the definition, although the algorithm depends on the loss function.

Definition 3.1 ( Compatibility ) . Given a loss function ℓ ( · ) with corresponding excess risk R ( · ) , a data distribution D is compatible with an algorithm A if there exists nonempty subsets T n of N , such that sup t ∈ T n R ( θ ( t ) n ) converges to zero in probability as sample size n tends to infinity, where { θ ( t ) n } t ≥ 0 denotes the output of algorithm A , and the randomness comes from the sampling of training data Z from distribution D and the execution of algorithm A . That is to say, ( D , A ) is compatible if there exists nonempty sets T n , such that

We call { T n } n> 0 the compatibility region of ( D , A ) . The distribution D is allowed to change with n . In this case, D should be understood as a sequence of distributions {D n } n ≥ 1 . We also allow the dimension of model parameter θ to be infinity or to grow with n . We omit this dependency on n when the context is clear.

Compatibility serves as a minimal condition for generalization, since if a data distribution is incompatible with the algorithm, one cannot expect to reach a small excess risk even if we allow for arbitrary early stopping. However, we remark that considering only the minimal excess risk is insufficient for a practical purpose, as one cannot exactly find the t that minimizes R ( θ ( t ) n ) due to the noise in the validation set. Therefore, it is meaningful to consider a region of time t on which the excess risk is consistent as in Definition 3.1. The larger the region is, the more robust the algorithm will be to the noise in its execution.

Comparisons with Other Notions. Compared to classic definitions of learnability, e.g. , PAC learning, the definition of compatibility is data-specific and algorithm-specific, and is thus a more fine-grained notion. Compared to the concept of benign proposed in [8], which studies whether the excess risk at t = ∞ converges to zero in probability as the sample size goes to infinity, compatibility only requires that there exists a time to derive a consistent excess risk. We will show later in Section 4.2 that in the overpamameterized linear regression setting, there exist cases such that the problem instance is compatible but not benign.

4 Analysis of Overparameterized Linear Regression with Gradient Descent

To validate the meaningfulness of compatibility, we study it in the overparameterized linear regression regime. We first introduce the data distribution, loss, and training algorithm, and then present the main theorem, which provides a sufficient condition for compatibility in this setting.

4.1 Preliminaries for Overparameterized Linear Regression

Notations. Let O,o, Ω , ω denote asymptotic notations, with their usual meaning. For example, the argument a n = O ( b n ) means that there exists a large enough constant C , such that a n ≤ Cb n . We use ≲ with the same meaning as the asymptotic notation O . Besides, let ∥ x ∥ denote the ℓ 2 norm for vector x , and ∥ A ∥ denote the operator norm for matrix A . We allow the vector to belong to

a countably infinite-dimensional Hilbert space H , and with a slight abuse of notation, we use R ∞ interchangeably with H . In this case, x ⊤ z denotes inner product and xz ⊤ denotes tensor product for x, z ∈ H . A random variable X is called σ -subgaussian if E [ e λX ] ≤ e λ 2 σ 2 / 2 for any λ .

Data Distribution. Let ( x , y ) ∈ R p × R denote the feature vector and the response, following a joint distribution D . Let Σ ≜ E [ xx ⊤ ] denote the feature covariance matrix, whose eigenvalue decomposition is Σ = V Λ V ⊤ = ∑ i> 0 λ i v i v ⊤ i with decreasing eigenvalues λ 1 ≥ λ 2 ≥ · · · . We make the following assumptions on the distribution of the feature vector.

Assumption 1 (Assumptions on feature distribution) . We assume that

E [ x ] = 0 .

λ 1 > 0 , ∑ i> 0 λ i < C for some absolute constant C .

Let ˜ x = Λ -1 2 V ⊤ x . The random vector ˜ x has independent σ x -subgaussian entries.

Loss and Excess Risk. We choose square loss as the loss function ℓ , i.e. ℓ ( θ , ( x , y )) = 1 / 2( y -x ⊤ θ ) 2 . The corresponding population loss is denoted by L ( θ ) = E ℓ ( θ , ( x , y )) and the optimal parameter is denoted by θ ∗ ≜ argmin θ ∈ R p L ( θ ) . We assume that ∥ θ ∗ ∥ < C for some absolute constant C . If there are multiple such minimizers, we choose an arbitrary one and fix it thereafter. We focus on the excess risk of parameter θ , defined as

Let ε = y -x ⊤ θ ∗ denote the noise in data point ( x , y ) . The following assumptions involve the conditional distribution of the noise.

Assumption 2 (Assumptions on noise distribution) . We assume that

The conditional noise ε | x has zero mean.

The conditional noise ε | x is σ y -subgaussian.

Note that both Assumption 1 and Assumption 2 are commonly considered in the related literatures [8, 68, 76].

Training Set. Given a training set { ( x i , y i ) } 1 ≤ i ≤ n with n pairs independently sampled from the population distribution D , we define X ≜ ( x 1 , · · · , x n ) ⊤ ∈ R n × p as the feature matrix, Y ≜ ( y 1 , · · · , y n ) ⊤ ∈ R n as the corresponding noise vector, and ε ≜ Y -Xθ ∗ as the residual vector. Let the singular value decomposition (SVD) of X be X = U ˜ Λ 1 2 W ⊤ , with ˜ Λ = diag { µ 1 · · · , µ n } ∈ R n × n , µ 1 ≥ · · · ≥ µ n .

We consider the overparameterized regime where the feature dimension is larger than the sample size, namely, p > n . In this regime, we assume that rank ( X ) = n almost surely as in Bartlett et al. [8]. This assumption is equivalent to the invertibility of XX ⊤ .

Assumption 3 (Linear independent training set) . For any n < p , we assume that the features in the training set { x 1 , x 2 , · · · , x n } is linearly independent almost surely.

Algorithm. Given the dataset ( X , Y ) , define the empirical loss function as ˆ L ( θ ) ≜ 1 2 n ∥ Xθ -Y ∥ 2 . We choose full-batch gradient descent on the empirical risk with a constant learning rate λ as the algorithm A in the previous template. In this case, the update rule for the optimization trajectory { θ t } t ≥ 0 is formulated as

Without loss of generality, we consider zero initialization θ 0 = 0 in this paper. In this case, for a sufficiently small learning rate λ , θ t converges to the min-norm interpolator ˆ θ = X ⊤ ( XX ⊤ ) -1 Y as t goes to infinity, which was well studied previously [8]. This paper takes one step further and discuss the excess risk along the entire training trajectory { R ( θ t ) } t ≥ 0 .

Effective Rank and Effective Dimensions. We define the effective ranks of the feature matrix Σ as r ( Σ ) ≜ ∑ i> 0 λ i λ 1 and R k ( Σ ) ≜ ( ∑ i>k λ i ) 2 ∑ i>k λ 2 i . Our results depend on two notions of effective dimension of the feature covariance Σ , defined as

where c 0 , c 1 are constants independent of the dimension p , sample size n , and time t 1 . We omit the dependency of k 0 , k 1 on c 0 , c 1 , n, Σ when the context is clear.

4.2 Main Theorem for Overparameterized Linear Regression with Gradient Descent

Next, we present the main result of this section, which provides a clean condition for compatibility between gradient descent and overparameterized linear regression.

Theorem 4.1. Consider the overparameterized linear regression setting defined in section 4.1. Let Assumption 1,2 and 3 hold. Assume the learning rate satisfies λ = O ( 1 Tr( Σ ) ) .

If the covariance satisfies k 0 = o ( n ) , R k 0 ( Σ ) = ω ( n ) , r ( Σ ) = o ( n ) , it is compatible in the region T n = ( ω ( 1 λ ) , ∞ ) .

If the covariance satisfies k 0 = O ( n ) , k 1 = o ( n ) , r ( Σ ) = o ( n ) , it is compatible in the region T n = ( ω ( 1 λ ) , o ( n λ )) .

If the covariance does not change with n , and satisfies k 0 = O ( n ) and p = ∞ , it is compatible in the region T n = ( ω ( 1 λ ) , o ( n λ )) .

The proof of Theorem 4.1 is given in Appendix A and sketched in Section 5. The theorem shows that gradient descent is compatible with overparameterized linear regression under some mild regularity conditions on the learning rate, effective rank and effective dimensions. The condition on the learning rate is natural for optimizing a smooth objective. We conjecture that the condition k 0 = O ( n ) can not be removed in general cases, since the effective dimension k 0 characterizes the concentration of the singular values of the data matrix X and plays a crucial role in the excess risk of the gradient descent dynamics.

Comparison with Benign Overfitting. The paper [8] studies overparameterized linear regression and gives the condition for min-norm interpolator to generalize. They prove that the feature covariance Σ is benign if and only if

As discussed in Section 3.2, benign problem instance also satisfies compatibility, since benign overfitting requires a stronger condition on k 0 and an additional assumption on R k 0 ( Σ ) . The following example shows that this inclusion relationship is strict.

Example 4.1. Under the same assumption as in Theorem 4.1, if the spectrum of Σ satisfies

for some α > 1 , we derive that k 0 = Θ( n ) . Therefore, this problem instance satisfies compatibility, but does not satisfy benign overfitting.

Example 4.1 shows the existence of a case where the early stopping solution can generalize but interpolating solution fails. Therefore, Theorem 4.1 can characterize generalization for a much wider range of problem instances.

5 Proof Sketch and Techniques

5.1 A Time Variant Bound

We further introduce an additional type of effective dimension besides k 0 , k 1 , which is time variant and is utilized to track the optimization dynamics.

Definition 5.1 (Effective Dimensions) . Given a feature covariance matrix Σ , define the effective dimension k 2 as

where c 2 is a constant independent of the dimension p , sample size n , and time t . The term c ( t, n ) is a function to be discussed later. When the context is clear, we omit its dependencies on c 2 , c ( t, n ) , n, Σ and only denote it by k 2 .

Based on the effective rank and effective dimensions defined above, we provide a time-variant bound in Theorem 5.1 for overparameterized linear regression, which further leads to the argument in Theorem 4.1. Compared to the existing bound [8], Theorem 5.1 focuses on investigating the role of training epoch t in the excess risk, to give a refined bias-variance decomposition.

Theorem 5.1 (Time Variant Bound) . Suppose Assumption 1, 2 and 3 hold. Fix a function c ( t, n ) . Given δ ∈ (0 , 1) , assume that k 0 ≤ n c , log 1 δ ≤ n c , 0 < λ ≤ 1 c ∑ i> 0 λ i for a large enough constant c . Then with probability at least 1 -δ , we have for any t ∈ N ,

where

We provide a high-level intuition behind Theorem 5.1. We decompose R ( θ t ) into the bias term and the variance term. The variance term is then split into the leading part and tailing part based on the sprctrum of the feature covariance Σ . The eigenvalues in the tailing part will cause the variance term in the excess risk of the min-norm interpolating solution to be Ω(1) for fast decaying spectrum, as is the case in [8]. However, since the convergence in the tailing eigenspace is slower compared with the leading eigenspace, a proper early stopping strategy will prevent the overfitting in the tailing eigenspace and meanwhile avoid underfitting in the leading eigenspace.

The c ( t, n ) Principle. It is worth emphasizing that our bound holds for arbitrary positive function c ( t, n ) . Therefore, one can fine-tune the generalization bound by choosing a proper c ( t, n ) . In the subsequent sections, we show how to derive consistent risk bounds for different time t , based on different choices of c ( t, n ) . We present the case of choosing a constant c ( t, n ) in the next section. We leave the case of choosing a varying c ( t, n ) to Appendix 5.3.

5.2 Varying t , Constant c ( t, n )

Theorem 5.1 provides an excess risk upper bound uniformly for t ∈ N . However, it is still non-trivial to derive Theorem 4.1, where the remaining question is to decide the term c ( t, n ) . The following corollary shows the generalization bound when setting c ( t, n ) to a constant.

Corollary 5.1. Let Assumption 1, 2 and 3 hold. Fix a constant c ( t, n ) . Suppose k 0 = O ( n ) , k 1 = o ( n ) , r ( Σ ) = o ( n ) , λ = O ( 1 ∑ i> 0 λ i ) . Then there exists a sequence of positive constants { δ n } n ≥ 0 which converge to 0, such that with probability at least 1 -δ n , the excess risk is consistent for t ∈ ( ω ( 1 λ ) , o ( n λ )) , i.e.

Furthermore, for any positive constant δ , with probability at least 1 -δ , the minimal excess risk on the training trajectory can be bounded as

Lemma 5.1 below shows that k 1 = o ( n ) always holds for fixed distribution. Therefore, combining Corollary 5.1 and the following Lemma 5.1 completes the proof of Theorem 4.1.

Lemma 5.1. For any fixed (i.e. independent of sample size n ) feature covariance Σ satisfying assumption 1, we have k 1 ( n ) = o ( n ) .

Next we apply the bound in Corollary 5.1 to several data distributions. These distributions are widely discussed in [8, 76]. We also derive the existing excess risk bounds, which focus on the min-norm interpolator [8] and one-pass SGD iterate [76], of these distributions and compare them with our theorem. The results are summarized in Table 1, which shows that the bound in Corollary 5.1 outperforms previous results for a general class of distributions.

Example 5.1. Under the same conditions as Theorem 5.1, let Σ denote the feature covariance matrix. We show the following examples:

(Inverse Polynominal). If the spectrum of Σ satisfies λ k = 1 k α for some α > 1 , we derive that k 0 = Θ( n ) , k 1 = Θ ( n 1 α ) . Therefore, min t V ( θ t ) = O ( n 1 -α α ) and min t R ( θ t ) = O ( n -min { α -1 α , 1 2 } ) .

(Inverse Log-Polynominal). If the spectrum of Σ satisfies λ k = 1 k log β ( k +1) for some β > 1 , we derive that k 0 = Θ ( n log n ) , k 1 = Θ ( n log β n ) . Therefore, min t V ( θ t ) = O ( 1 log β n ) and min t R ( θ t ) = O ( 1 log β n ) .

(Constant). If the spectrum of Σ satisfies λ k = 1 n 1+ ε , 1 ≤ k ≤ n 1+ ε , for some ε > 0 , we derive that k 0 = 0 , k 1 = 0 . Therefore, min t V ( θ t ) = O ( 1 n ) and min t R ( θ t ) = O ( 1 √ n ) .

(Piecewise Constant). If the spectrum of Σ satisfies λ k = { 1 s 1 ≤ k ≤ s, 1 d -s s +1 ≤ k ≤ d, where s = n r , d = n q , 0 < r ≤ 1 , q ≥ 1 . We derive that k 0 = n r , k 1 = n r . Therefore, min t V ( θ t ) = O ( n r -1 ) and min t R ( θ t ) = O ( n -min { 1 -r, 1 2 } ) .

5.3 Varying t , Varying c ( t, n )

Although setting c ( t, n ) to a constant as in Corollary 5.1 suffices to prove Theorem 4.1, in this section we show that the choice of c ( t, n ) can be much more flexible. Specifically, we provide a concrete example and demonstrate that by setting c ( t, n ) to a non-constant, Theorem 5.1 can indeed produce larger compatibility regions.

Example 5.2. Under the same conditions as Theorem 5.1, let Σ denote the feature covariance matrix. If the spectrum of Σ satisfies λ k = 1 k α for some α > 1 , we set c ( t, n ) = Θ ( n α +1 -2 ατ 2 α +1 ) for a given α +1 2 α ≤ τ ≤ 3 α +1 2 α +2 . Then for t = Θ( n τ ) , we derive that V ( θ t ) = O ( n 2 ατ -3 α +2 τ -1 2 α +1 ) .

Example 5.2 shows that by choosing c ( t, n ) as a non-constant, we exploit the full power of Theorem 5.1, and extend the region to t = Θ ( n 3 α +1 2 α +2 ) = ω ( n ) . In this example, Theorem 5.1 outperforms all O ( t n ) -type bounds, which become vacuous when t = ω ( n ) .

6 Experiments

In this section, we provide numerical studies of overparameterized linear regression problems. We consider overparameterized linear regression instances with input dimension p = 1000 , sample size n = 100 . The features are sampled from Gaussian distribution with different covariances. The empirical results (a.) demonstrate the benefits of trajectory analysis underlying the definition of compatibility, since the optimal excess risk along the algorithm trajectory is significantly lower than that of the min-norm interpolator (b.) validate the statements in Corollary 5.1 , since the optimal excess risk is lower when the eigenvalues of feature covariance decay faster. We refer to Appendix C for detailed setups, additional results and discussions.

Observation One: Early stopping solution along the training trajectory generalizes significantly better than the min-norm interpolator. We calculate the excess risk of optimal early stopping solutions and min-norm interpolators from 1000 independent trials and list the results in Table 2. The results illustrate that the early stopping solution on the algorithm trajectory enjoys much better generalization properties. This observation corroborates the importance of data-dependent training trajectory in generalization analysis.

Observation Two: The faster covariance spectrum decays, the lower optimal excess risk is. Table 2 also illustrates a positive correlation between the decaying rate of λ i and the generalization performance of the early stopping solution. This accords with Theorem 5.1, showing that the excess risk is better for a smaller effective dimension k 1 , where small k 1 indicates a faster-decaying eigenvalue λ i . We additionally note that such a phenomenon also illustrates the difference between min-norm and early stopping solutions in linear regression, since Bartlett et al. [8] demonstrate that the min-norm solution is not consistent when the eigenvalues decay too fast. By comparison, early stopping solutions do not suffer from this restriction.

7 Conclusion

In this paper, we investigate how to characterize and analyze generalization in a data-dependent and algorithm-dependent manner. We formalize the notion of data-algorithm compatibility and study it under the regime of overparameterized linear regression with gradient descent. Our theoretical and empirical results demonstrate that one can ease the assumptions and broaden the scope of generalization by fully exploiting the data information and the algorithm information. Despite linear cases in this paper, compatibility can be a much more general concept. Therefore, we believe this paper will motivate more work on data-dependent trajectory analysis.

References

Madhu S. Advani, Andrew M. Saxe, and Haim Sompolinsky. High-dimensional dynamics of generalization error in neural networks. Neural Networks , 132:428-446, 2020. doi: 10.1016/j. neunet.2020.08.022. URL https://doi.org/10.1016/j.neunet.2020.08.022 .

Alnur Ali, J. Zico Kolter, and Ryan J. Tibshirani. A continuous-time view of early stopping for least squares regression. In Kamalika Chaudhuri and Masashi Sugiyama, editors, The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan , volume 89 of Proceedings of Machine Learning Research , pages 1370-1378. PMLR, 2019. URL http://proceedings.mlr.press/v89/ali19a.html .

Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research , pages 254-263. PMLR, 2018. URL http://proceedings.mlr.press/ v80/arora18b.html .

Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 322-332. PMLR, 2019. URL http: //proceedings.mlr.press/v97/arora19a.html .

Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. CoRR , abs/2106.15853, 2021. URL https://arxiv.org/abs/2106.15853 .

Pradeep Kr. Banerjee and Guido Montúfar. Information complexity and generalization bounds. CoRR , abs/2105.01747, 2021. URL https://arxiv.org/abs/2105.01747 .

Peter L. Bartlett, Dylan J. Foster, and Matus Telgarsky. Spectrally-normalized margin bounds for neural networks. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 6240-6249, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/ b22b257ad0519d4500539da3c8bcf4dd-Abstract.html .

Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. CoRR , abs/1906.11300, 2019. URL http://arxiv.org/abs/1906.11300 .

Raef Bassily, Vitaly Feldman, Cristóbal Guzmán, and Kunal Talwar. Stability of stochastic gradient descent on nonsmooth convex losses. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurips. cc/paper/2020/hash/2e2c4bf7ceaa4712a72dd5ee136dc9a8-Abstract.html .

Gilles Blanchard and Nicole Krämer. Convergence rates of kernel conjugate gradient for random design regression. Analysis and Applications , 14(06):763-794, 2016.

Olivier Bousquet and André Elisseeff. Stability and generalization. J. Mach. Learn. Res. , 2: 499-526, 2002. URL http://jmlr.org/papers/v2/bousquet02a.html .

Olivier Bousquet, Yegor Klochkov, and Nikita Zhivotovskiy. Sharper bounds for uniformly stable algorithms. In Jacob D. Abernethy and Shivani Agarwal, editors, Conference on Learning Theory, COLT 2020, 9-12 July 2020, Virtual Event [Graz, Austria] , volume 125 of Proceedings of Machine Learning Research , pages 610-626. PMLR, 2020. URL http://proceedings. mlr.press/v125/bousquet20b.html .

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 612, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html .

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) , pages 41714186. Association for Computational Linguistics, 2019. doi: 10.18653/v1/n19-1423. URL https://doi.org/10.18653/v1/n19-1423 .

Aymeric Dieuleveut and Francis Bach. Nonparametric stochastic approximation with large step-sizes. The Annals of Statistics , 44(4):1363-1399, 2016.

Gintare Karolina Dziugaite and Daniel M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In Gal Elidan, Kristian Kersting, and Alexander T. Ihler, editors, Proceedings of the Thirty-Third Conference on Uncertainty in Artificial Intelligence, UAI 2017, Sydney, Australia, August 11-15, 2017 . AUAI Press, 2017. URL http://auai.org/uai2017/proceedings/papers/173. pdf .

Melikasadat Emami, Mojtaba Sahraee-Ardakan, Parthe Pandit, Sundeep Rangan, and Alyson K. Fletcher. Generalization error of generalized linear models in high dimensions. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 2892-2901. PMLR, 2020. URL http://proceedings.mlr.press/v119/emami20a.html .

Vitaly Feldman and Jan Vondrák. Generalization bounds for uniformly stable algorithms. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 9770-9780, 2018. URL https://proceedings.neurips. cc/paper/2018/hash/05a624166c8eb8273b8464e8d9cb5bd9-Abstract.html .

Vitaly Feldman and Jan Vondrák. High probability generalization bounds for uniformly stable algorithms with nearly optimal rate. In Alina Beygelzimer and Daniel Hsu, editors, Conference on Learning Theory, COLT 2019, 25-28 June 2019, Phoenix, AZ, USA , volume 99 of Proceedings of Machine Learning Research , pages 1270-1279. PMLR, 2019. URL http://proceedings.mlr.press/v99/feldman19a.html .

Mahsa Forouzesh and Patrick Thiran. Disparity between batches as a signal for early stopping. In Nuria Oliver, Fernando Pérez-Cruz, Stefan Kramer, Jesse Read, and José Antonio Lozano, editors, Machine Learning and Knowledge Discovery in Databases. Research Track - European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part II , volume 12976 of Lecture Notes in Computer Science , pages 217-232. Springer, 2021. doi: 10. 1007/978-3-030-86520-7\_14. URL https://doi.org/10.1007/978-3-030-86520-7_ 14 .

Spencer Frei, Niladri S. Chatterji, and Peter L. Bartlett. Benign overfitting without linearity: Neural network classifiers trained by gradient descent for noisy linear data. CoRR , abs/2202.05928, 2022. URL https://arxiv.org/abs/2202.05928 .

L. Lo Gerfo, Lorenzo Rosasco, Francesca Odone, Ernesto De Vito, and Alessandro Verri. Spectral algorithms for supervised learning. Neural Comput. , 20(7):1873-1897, 2008. doi: 10.1162/neco.2008.05-07-517. URL https://doi.org/10.1162/neco.2008.05-07-517 .

Sebastian Goldt, Madhu Advani, Andrew M. Saxe, Florent Krzakala, and Lenka Zdeborová. Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 6979-6989, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ cab070d53bd0d200746fb852a922064a-Abstract.html .

Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In Maria-Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016 , volume 48 of JMLR Workshop and Conference Proceedings , pages 12251234. JMLR.org, 2016. URL http://proceedings.mlr.press/v48/hardt16.html .

Wei Hu, Lechao Xiao, Ben Adlam, and Jeffrey Pennington. The surprising simplicity of the earlytime learning dynamics of neural networks. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurips. cc/paper/2020/hash/c6dfc6b7c601ac2978357b7a81e2d7ae-Abstract.html .

Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 8580-8589, 2018. URL https://proceedings.neurips.cc/paper/2018/hash/ 5a4be1fa34e62bb8a6ec6b91d2462f5a-Abstract.html .

Ziwei Ji, Justin D. Li, and Matus Telgarsky. Early-stopped neural networks are consistent. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 1805-1817, 2021. URL https://proceedings.neurips.cc/paper/2021/ hash/0e1ebad68af7f0ae4830b7ac92bc3c6f-Abstract.html .

Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum?id=SJgIPJBFvH .

Vladimir Koltchinskii and Dmitriy Panchenko. Rademacher processes and bounding the risk of function learning. In High dimensional probability II , pages 443-457. Springer, 2000.

Ilja Kuzborskij and Csaba Szepesvári. Nonparametric regression with shallow overparameterized neural networks trained by GD with early stopping. In Mikhail Belkin and Samory Kpotufe, editors, Conference on Learning Theory, COLT 2021, 15-19 August 2021, Boulder, Colorado, USA , volume 134 of Proceedings of Machine Learning Research , pages 2853-2890. PMLR, 2021. URL http://proceedings.mlr.press/v134/kuzborskij21a.html .

Yunwen Lei and Yiming Ying. Fine-grained analysis of stability and generalization for stochastic gradient descent. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 5809-5819. PMLR, 2020. URL http://proceedings.mlr.press/v119/ lei20c.html .

Jian Li, Xuanyuan Luo, and Mingda Qiao. On generalization error bounds of noisy gradient methods for non-convex learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https: //openreview.net/forum?id=SkxxtgHKPS .

Jiangyuan Li, Thanh V. Nguyen, Chinmay Hegde, and Ka Wai Wong. Implicit sparse regularization: The impact of depth and early stopping. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 28298-28309, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ ee188463935a061dee6df8bf449cb882-Abstract.html .

Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In Silvia Chiappa and Roberto Calandra, editors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy] , volume 108 of Proceedings of Machine Learning Research , pages 4313-4324. PMLR, 2020. URL http://proceedings.mlr.press/v108/li20j.html .

Zhu Li, Weijie Su, and Dino Sejdinovic. Benign overfitting and noisy features. CoRR , abs/2008.02901, 2020. URL https://arxiv.org/abs/2008.02901 .

Junhong Lin and Volkan Cevher. Optimal rates for spectral-regularized algorithms with leastsquares regression over hilbert spaces. CoRR , abs/1801.06720, 2018. URL http://arxiv. org/abs/1801.06720 .

Junhong Lin and Lorenzo Rosasco. Optimal rates for multi-pass stochastic gradient methods. J. Mach. Learn. Res. , 18:97:1-97:47, 2017. URL http://jmlr.org/papers/v18/17-176. html .

Chaoyue Liu, Amirhesam Abedsoltan, and Mikhail Belkin. On emergence of clean-priority learning in early stopped neural networks. CoRR , abs/2306.02533, 2023. doi: 10.48550/arXiv. 2306.02533. URL https://doi.org/10.48550/arXiv.2306.02533 .

Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview. net/forum?id=SJeLIgBKPS .

Kaifeng Lyu, Zhiyuan Li, Runzhe Wang, and Sanjeev Arora. Gradient descent on twolayer nets: Margin maximization and simplicity bias. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 12978-12991, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ 6c351da15b5e8a743a21ee96a86e25df-Abstract.html .

Neil Mallinar, James B. Simon, Amirhesam Abedsoltan, Parthe Pandit, Misha Belkin, and Preetum Nakkiran. Benign, tempered, or catastrophic: Toward a refined taxonomy of overfitting. In NeurIPS , 2022. URL http://papers.nips.cc/paper_files/paper/2022/ hash/08342dc6ab69f23167b4123086ad4d38-Abstract-Conference.html .

David A. McAllester. Simplified pac-bayesian margin bounds. In Bernhard Schölkopf and Manfred K. Warmuth, editors, Computational Learning Theory and Kernel Machines, 16th Annual Conference on Computational Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003, Proceedings , volume 2777 of Lecture Notes in Computer Science , pages 203-215. Springer, 2003. doi: 10.1007/978-3-540-45167-9\_16. URL https://doi.org/10.1007/978-3-540-45167-9_16 .

David A. McAllester. A pac-bayesian tutorial with A dropout bound. CoRR , abs/1307.2118, 2013. URL http://arxiv.org/abs/1307.2118 .

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning . Adaptive computation and machine learning. MIT Press, 2012. ISBN 978-0-262-01825-8. URL http://mitpress.mit.edu/books/foundations-machine-learning-0 .

Wenlong Mou, Liwei Wang, Xiyu Zhai, and Kai Zheng. Generalization bounds of SGLD for non-convex learning: Two theoretical viewpoints. In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet, editors, Conference On Learning Theory, COLT 2018, Stockholm, Sweden, 6-9 July 2018 , volume 75 of Proceedings of Machine Learning Research , pages 605-638. PMLR, 2018. URL http://proceedings.mlr.press/v75/mou18a.html .

Vaishnavh Nagarajan and J. Zico Kolter. Uniform convergence may be unable to explain generalization in deep learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 11611-11622, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ 05e97c207235d63ceb1db43c60db7bbb-Abstract.html .

Jeffrey Negrea, Gintare Karolina Dziugaite, and Daniel Roy. In defense of uniform convergence: Generalization via derandomization with an application to interpolating predictors. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 7263-7272. PMLR, 2020. URL http://proceedings.mlr.press/v119/negrea20a.html .

Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally-normalized margin bounds for neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018. URL https://openreview.net/ forum?id=Skz_WfbCZ .

Emilio Parrado-Hernández, Amiran Ambroladze, John Shawe-Taylor, and Shiliang Sun. Pacbayes bounds with data dependent priors. J. Mach. Learn. Res. , 13:3507-3531, 2012. URL http://dl.acm.org/citation.cfm?id=2503353 .

María Pérez-Ortiz, Omar Rivasplata, John Shawe-Taylor, and Csaba Szepesvári. Tighter risk certificates for neural networks. J. Mach. Learn. Res. , 22:227:1-227:40, 2021. URL http://jmlr.org/papers/v22/20-879.html .

Loucas Pillaud-Vivien, Alessandro Rudi, and Francis R. Bach. Statistical optimality of stochastic gradient descent on hard learning problems through multiple passes. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 8125-8135, 2018. URL https://proceedings.neurips.cc/paper/2018/ hash/10ff0b5e85e5b85cc3095d431d8c08b4-Abstract.html .

Lutz Prechelt. Early stopping - but when? In Grégoire Montavon, Genevieve B. Orr, and Klaus-Robert Müller, editors, Neural Networks: Tricks of the Trade - Second Edition , volume 7700 of Lecture Notes in Computer Science , pages 53-67. Springer, 2012. doi: 10.1007/ 978-3-642-35289-8\_5. URL https://doi.org/10.1007/978-3-642-35289-8_5 .

Garvesh Raskutti, Martin J. Wainwright, and Bin Yu. Early stopping and non-parametric regression: an optimal data-dependent stopping rule. J. Mach. Learn. Res. , 15(1):335-366, 2014. doi: 10.5555/2627435.2627446. URL https://dl.acm.org/doi/10.5555/2627435. 2627446 .

Omar Rivasplata, Ilja Kuzborskij, Csaba Szepesvári, and John Shawe-Taylor. Pac-bayes analysis beyond the usual bounds. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, MariaFlorina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurips.cc/ paper/2020/hash/c3992e9a68c5ae12bd18488bc579b30d-Abstract.html .

Lorenzo Rosasco and Silvia Villa. Learning with incremental iterative regularization. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pages 1630-1638, 2015. URL https://proceedings.neurips.cc/paper/2015/hash/ 1587965fb4d4b5afe8428a4a024feb0d-Abstract.html .

Daniel Russo and James Zou. Controlling bias in adaptive data analysis using information theory. In Arthur Gretton and Christian C. Robert, editors, Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9-11, 2016 , volume 51 of JMLR Workshop and Conference Proceedings , pages 1232-1240. JMLR.org, 2016. URL http://proceedings.mlr.press/v51/russo16.html .

Mojtaba Sahraee-Ardakan, Melikasadat Emami, Parthe Pandit, Sundeep Rangan, and Alyson K. Fletcher. Kernel methods and multi-layer perceptrons learn linear models in high dimensions. CoRR , abs/2201.08082, 2022. URL https://arxiv.org/abs/2201.08082 .

Matthias W. Seeger. Pac-bayesian generalisation error bounds for gaussian process classification. J. Mach. Learn. Res. , 3:233-269, 2002. URL http://jmlr.org/papers/v3/seeger02a. html .

Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurips. cc/paper/2020/hash/6cfe0e6127fa25df2a0ef2ae1067d915-Abstract.html .

Vaishaal Shankar, Rebecca Roelofs, Horia Mania, Alex Fang, Benjamin Recht, and Ludwig Schmidt. Evaluating machine accuracy on imagenet. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 8634-8644. PMLR, 2020. URL http: //proceedings.mlr.press/v119/shankar20c.html .

John Shawe-Taylor and Robert C. Williamson. A PAC analysis of a bayesian estimator. In Yoav Freund and Robert E. Schapire, editors, Proceedings of the Tenth Annual Conference on Computational Learning Theory, COLT 1997, Nashville, Tennessee, USA, July 6-9, 1997 , pages 2-9. ACM, 1997. doi: 10.1145/267460.267466. URL https://doi.org/10.1145/267460. 267466 .

Ruoqi Shen, Liyao Gao, and Yi-An Ma. On optimal early stopping: Over-informative versus under-informative parametrization. CoRR , abs/2202.09885, 2022. URL https://arxiv.org/ abs/2202.09885 .

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy P. Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nat. , 550(7676):354-359, 2017. doi: 10.1038/nature24270. URL https://doi.org/10.1038/nature24270 .

Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, and Nathan Srebro. The implicit bias of gradient descent on separable data. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018. URL https://openreview.net/forum?id=r1q7n9gAb .

Pierre Stock and Moustapha Cissé. Convnets and imagenet beyond accuracy: Understanding mistakes and uncovering biases. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VI , volume 11210 of Lecture Notes in Computer Science , pages 504-519. Springer, 2018. doi: 10.1007/978-3-030-01231-1\_31. URL https://doi.org/10.1007/978-3-030-01231-1_31 .

Pierre Tarres and Yuan Yao. Online learning as stochastic approximation of regularization paths: Optimality and almost-sure convergence. IEEE Trans. Inf. Theory , 60(9):5716-5735, 2014. doi: 10.1109/TIT.2014.2332531. URL https://doi.org/10.1109/TIT.2014.2332531 .

Jiaye Teng, Jianhao Ma, and Yang Yuan. Towards understanding generalization via decomposing excess risk dynamics. CoRR , abs/2106.06153, 2021. URL https://arxiv.org/abs/2106. 06153 .

Alexander Tsigler and Peter L Bartlett. Benign overfitting in ridge regression. arXiv preprint arXiv:2009.14286 , 2020.

Tomas Vaskevicius, Varun Kanade, and Patrick Rebeschini. The statistical complexity of earlystopped mirror descent. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 024d2d699e6c1a82c9ba986386f4d824-Abstract.html .

Ke Wang and Christos Thrampoulidis. Benign overfitting in binary classification of gaussian mixtures. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, Toronto, ON, Canada, June 6-11, 2021 , pages 4030-4034. IEEE, 2021. doi: 10.1109/ICASSP39728.2021.9413946. URL https://doi.org/10.1109/ICASSP39728. 2021.9413946 .

Aolin Xu and Maxim Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 2524-2533, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/ ad71c82b22f4f65b9398f76d8be4c615-Abstract.html .

Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learning. Constructive Approximation , 26(2):289-315, 2007.

Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, and Sanghyuk Chun. Re-labeling imagenet: From single to multi-labels, from global to localized labels. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021 , pages 2340-2350. Computer Vision Foundation / IEEE, 2021. URL https://openaccess.thecvf.com/content/CVPR2021/html/Yun_Re-Labeling_ ImageNet_From_Single_to_Multi-Labels_From_Global_to_Localized_CVPR_ 2021_paper.html .

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Commun. ACM , 64(3):107-115, 2021. doi: 10.1145/3446776. URL https://doi.org/10.1145/3446776 .

Lijia Zhou, Danica J. Sutherland, and Nati Srebro. On uniform convergence and low-norm interpolation learning. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 4cc5400e63624c44fadeda99f57588a6-Abstract.html .

Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, and Sham M. Kakade. Benign overfitting of constant-stepsize SGD for linear regression. In Mikhail Belkin and Samory Kpotufe, editors, Conference on Learning Theory, COLT 2021, 15-19 August 2021, Boulder, Colorado, USA , volume 134 of Proceedings of Machine Learning Research , pages 4633-4635. PMLR, 2021. URL http://proceedings.mlr.press/v134/zou21a.html .

Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, and Sham M. Kakade. Risk bounds of multi-pass SGD for least squares in the interpolation regime. In NeurIPS , 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ 543924fdf260ba990f2ef84f940f3db2-Abstract-Conference.html .

Supplementary Materials

A Proofs for the Main Results

We first sketch the proof in section A.1 and give some preliminary lemmas A.2. The following sections A.3, A.4 and A.5 are devoted to the proof of Theorem 5.1. The proof of Theorem 4.1 is given in A.6.

A.1 Proof Sketch

We start with a standard bias-variance decomposition following Bartlett et al. [8], which derives that the time-variant excess risk R ( θ t ) can be bounded by a bias term and a variance term. We refer to Appendix A.3 for more details.

For the bias part, we first decompose it into an optimization error and an approximation error. For the optimization error, we use the spectrum analysis to bound it with O (1 /t ) where t denotes the time. For the approximation error, we bound it with O (1 / √ n )) where n denotes the sample size, inspired by Bartlett et al. [8]. We refer to Appendix A.4 for more details.

For the variance part, a key step is to bound the term ( I -λ n XX ⊤ ) t , where X is the feature matrix. The difficulty arises from the different scales of the eigenvalues of XX ⊤ , where the largest eigenvalue has order Θ( n ) while the smallest eigenvalue has order O (1) , according to Lemma 10 in Bartlett et al. [8]. To overcome this issue, we divide the matrix XX ⊤ based on whether its eigenvalues is larger than c ( t, n ) , which is a flexible term dependent on time t and sample size n . Therefore, we split the variance term based on eigenvalues of covariance matrix Σ (leading to the k 1 -related term) and based on the eigenvalues of XX ⊤ (leading to the k 2 -related term). We refer to Appendix A.5 for more details.

A.2 Preliminaries

The following result comes from Bartlett et al. [8], which bounds the eigenvalues of XX ⊤ .

Lemma A.1. (Lemma 10 in Bartlett et al. [8]) For any σ x , there exists a constant c, such that for any 0 ≤ k < n , with probability at least 1 -e -n c ,

This implies that as long as the step size λ is small than a threshold independent of sample size n , gradient descent is stable.

Corollary A.1. There exists a constant c , such that with probability at least 1 -e -n c , for any 0 ≤ λ ≤ 1 c ∑ i> 0 λ i we have

Proof. The right hand side of the inequality is obvious since λ > 0 . For the left hand side, we have to show that the eigenvalues of I -λ n X ⊤ X is non-negative. since X ⊤ X and XX ⊤ have the same non-zero eigenvalues, we know that with probability at least 1 -e -n c , the smallest eigenvalue of I -λ n X ⊤ X can be lower bounded by

where the second inequality uses lemma A.1, and the last inequality holds if λ ≤ 1 2 c ∑ i> 0 λ i .

A.3 Proof for the Bias-Variance Decomposition

Let X † denote the Moore-Penrose pseudoinverse of matrix X . The following lemma gives a closed form expression for θ t .

Lemma A.2. The dynamics of { θ t } t ≥ 0 satisfies

Proof. We prove the lemma using induction. The equality holds at t = 0 as both sides are θ 0 . Recall that θ t is updated as

Suppose that the dynamic holds up to the t -th step. Plug the expression for θ t into the above recursion and note that X ⊤ XX † = X ⊤ , we get

which finishes the proof.

Next we prove two identities which will be used in further proof.

Lemma A.3. The following two identities hold for any matrix X and non-negative integer t :

Proof. Note that X ⊤ XX † = X ⊤ , we can expand the left hand side of the first identity above using binomial theorem and eliminate the pseudo-inverse X † :

The second identity can be proved in a similar way:

We are now ready to prove the main result of this section.

Lemma A.4. The excess risk at the t -th epoch can be upper bounded as

where which characterizes bias term and variance term in the excess risk. Furthermore, there exists constant c such that with probability at least 1 -δ over the randomness of ε , we have

Proof. First note that XX ⊤ is invertible by Assumption 3. Express the excess risk as follows

Recall that θ 0 = 0 and Y = Xθ ∗ + ε and we can further simplify the formula for θ t in lemma A.2:

Plug it into the above expression for R ( θ t ) , we have

Applying lemma A.3, we obtain

which proves the first claim in the lemma. The second part of the theorem directly follows from lemma 18 in Bartlett et al. [8].

A.4 Proof for the Bias Upper Bound

The next lemma guarantees that the sample covariace matrix 1 X ⊤ X concentrates well around Σ .

n

Lemma A.5. (Lemma 35 in Bartlett et al. [8]) There exists constant c such that for any 0 < δ < 1 with probability as least 1 -δ ,

The following inequality will be useful in our proof to characterize the decaying rate of the bias term with t .

Lemma A.6. For any positive semidefinite matrix P which satisfies ∥ P ∥ ≤ 1 , we have

Proof. Assume without loss of generality that P is diagonal. Then it suffices to consider seperately each eigenvalue σ of P , and show that σ (1 -σ ) t ≤ 1 t .

In fact, by AM-GM inequality we have

which completes the proof.

Next we prove the main result of this section.

Lemma A.7. There exists constant c such that if 0 ≤ λ ≤ 1 c ∑ i> 0 λ i , then for any 0 < δ < 1 , with probability at least 1 -δ the following bound on the bias term holds for any t

Proof. The bias can be decomposed into the following two terms

For sufficiently small learning rate λ as given by corollary A.1, we know that with high probability

which together with lemma A.5 gives a high probability bound on the first term:

For the second term, invoke lemma A.6 with P = λ n X ⊤ X and we get

Putting these two bounds together gives the proof for the main theorem.

A.5 Proof for the Variance Upper Bound

Recall that X = U ˜ Λ 1 2 W ⊤ is the singular value decomposition of data matrix X , where U = ( u 1 , · · · , u n ) , W = ( w 1 , · · · , w n ) , ˜ Λ = diag { µ 1 , · · · , µ n } with µ 1 ≥ µ 2 ≥ · · · µ n . Recall that

for some constant c 0 , c 1 , c 2 and function c ( t, n ) .

We further define for some constant c 3 .

The next lemma shows that we can appropriately choose constants to ensure that k 3 ≤ k 2 holds with high probability, and in some specific cases we have k 2 ≤ k 1 .

Lemma A.8. For any function c ( t, n ) and constant c 2 , there exists constants c, c 3 , such that k 3 ≤ k 2 with probability at least 1 -e -n c . Furthermore, if c ( t, n ) is a positive constant function, for any c 1 , there exists c 2 such that k 2 ≤ k 1 .

Proof. According to lemma A.1, there exists a constant c , with probability at least 1 -e -n c we have

Therefore, we know that k 3 ≤ k 2 for c 3 = cc 2 .

By the definition of k 1 , we have

c +1 is a positive constant.

which implies that k 2 ≤ k 1 for c 2 = 1 c ( t,n ) , if c ( t, n )

Next we bound Tr[ C ] , which implies an upper bound on the variance term.

Theorem A.1. There exist constants c, c 0 , c 1 , c 2 such that if k 0 ≤ n c , then with probability at least 1 -e -n c , the trace of the variance matrix C has the following upper bound for any t :

Proof. We divide the eigenvalues of XX ⊤ into two groups based on whether they are greater than c 3 c ( t, n ) ∑ i> 0 λ i . The first group consists of µ 1 · · · µ k 3 , and the second group consists of µ k 3 +1 · · · µ n . For 1 ≤ j ≤ k 3 , we have

Therefore we have the following upper bound on [ I -( I -λ n XX ⊤ ) t ] 2 :

For positive semidefinite matrices P , Q , R which satisfies Q ⪯ R , it holds that Tr[ PQ ] ≤ Tr[ PR ] . It implies the following upperbound of Tr[ C ] :

Bounding 1 ⃝

Noticing X = U ˜ Λ 1 2 W ⊤ and Σ = ∑ i ≥ 1 λ i v i v ⊤ i , we express the first term as sums of eigenvector products,

Next we divide the above summation into 1 ≤ i ≤ k 1 and i > k 1 . For the first part, notice that

where ˜ x i is defined as ˜ x i = Xv i √ λ i = U ˜ Λ 1 2 W ⊤ v i √ λ i .

From the proof of lemma 11 in Bartlett et al. [8], we know that for any σ x , there exists a constant c 0 and c such that if k 0 ≤ n c , with probability at least 1 -e -n c the first part can be bounded as

Bounding 2 ⃝

Similar to the first step in bounding 1 ⃝ , we note that

U

˜

Λ

-

2

U

⊤

U

˜

Λ

1

2

W

⊤

Σ

W

˜

Λ

1

2

U

⊤

]

From Bernoulli's inequality and the definition of k 3 , for any k 3 +1 ≤ j ≤ n , we have

Hence,

which gives a bound for the first part.

For the second part we interchange the order of summation and have

for c large enough.

Putting 48 and 49 together, and noting that k 3 ≤ k 2 with high probability as given in lemma A.8, we know there exists a constant c that with probability at least 1 -e -n c ,

Putting things together

From the bounds for 1 ⃝ and 2 ⃝ given above, we know that there exists a constant c such that with probability at least 1 -e -n c , the trace of the variance matrix C has the following upper bound

Proof of theorem 5.1. Lemma A.4, A.7 and Theorem A.1 gives the complete proof. Note that the high probability events in the proof are independent of the epoch number t , and this implies that the theorem holds uniformly for all t ∈ N .

A.6 Proof of Compatibility Results

Corollary A.2 (Corollary 5.1 restated) . Let Assumption 1, 2 and 3 hold. Fix a constant c ( t, n ) . Suppose k 0 = O ( n ) , k 1 = o ( n ) , r ( Σ ) = o ( n ) , λ = O ( 1 ∑ i> 0 λ i ) . Then there exists a sequence of positive constants { δ n } n ≥ 0 which converge to 0, such that with probability at least 1 -δ n , the excess risk is consistent for t ∈ ( ω ( 1 λ ) , o ( n λ )) , i.e.

Furthermore, for any positive constant δ , with probability at least 1 -δ , the minimal excess risk on the training trajectory can be bounded as

Proof. According to Lemma A.7, with probability at least 1 -δ n 2 , the following inequality holds for all t :

If δ n is chosen such that log 1 δ n = o ( n ) , we have that with probability at least 1 -δ n 2 , we have for all t = ω ( 1 λ ) :

in the sample size n .

When c ( t, n ) is a constant, we have k 2 ≤ k 1 with high probability as given in lemma A.8. Therefore, according to Lemma A.4 and Theorem A.1, we know that if log 1 δ n = O ( n ) , with probability at least 1 -δ n 2 , the following bound holds for all t :

Since k 1 = o ( n ) , t = o ( n λ ) , we have k 1 n + λ 2 t 2 n 2 = o (1) . Therefore, there exists a mildly decaying sequence of δ n with log ( 1 δ n )( k 1 n + λ 2 t 2 n 2 ) = o (1) , i.e.,

To conclude, δ n can be chosen such that

and then with probability at least 1 -δ n , the excess risk is consistent for all t ∈ ( ω ( 1 λ ) , o ( n λ )) :

This completes the proof for the first claim. The second claim follows from Equation 55 and 57 by setting t = Θ ( √ n λ ) .

Lemma A.9 (Lemma 5.1 restated) . For any fixed (i.e. independent of sample size n ) feature covariance Σ satisfying assumption 1, we have k 1 ( n ) = o ( n ) .

Proof. Suppose there exists constant c , such that k 1 ( n ) ≥ cn . By definition of k 1 , we know that λ l ≥ c 1 ∑ i> 0 λ i n holds for 1 ≤ l ≤ k 1 ( n ) . Hence we have

summing up all l leads to a contradiction since ∑ i> 0 λ i < ∞ , which finishes the proof.

Theorem A.2 (Theorem 4.1 restated) . Consider the overparameterized linear regression setting defined in section 4.1. Let Assumption 1,2 and 3 hold. Assume the learning rate satisfies λ = O ( 1 Tr( Σ ) ) .

If the covariance satisfies k 0 = o ( n ) , R k 0 ( Σ ) = ω ( n ) , r ( Σ ) = o ( n ) , it is compatible with the region T n = ( ω ( 1 λ ) , ∞ ) .

If the covariance satisfies k 0 = O ( n ) , k 1 = o ( n ) , r ( Σ ) = o ( n ) , it is compatible with the region T n = ( ω ( 1 λ ) , o ( n λ )) .

If the covariance does not change with n , and satisfies k 0 = O ( n ) and p = ∞ , it is compatible with the region T n = ( ω ( 1 λ ) , o ( n λ )) .

Proof. For the first argument, notice that (a) the bias term can still be bounded when t = ω (1) and r (Σ) = o ( n ) , according to Lemma A.4; (b) the variance term can be bounded with t →∞ (that is, the variance loss would increase with time t ). Therefore, the first argument directly follows Theorem 4 in Bartlett et al. [8].

The second argument follows Corollary 5.1, and the third argument follows Corollary 5.1 and Lemma 5.1. Specifically, for any ε > 0 , there exists { δ n } n> 0 and N such that for any sample size n > N , we have

Let n →∞ shows that sup t ∈ T n R ( θ t ) converges to 0 in probability, which completes the proof for the second and the third claim.

B Comparisons and Discussions

In this section, we provide additional discussions and calculations for the main results, and compare it with previous works, including benign overfitting (Section B.1), stability-based bounds (Section B.2), uniform convergence (Section B.3), and early-stopping bounds (Section B.4).

B.1 Comparisons with Benign Overfitting

We summarize the results in Bartlett et al. [8], Zou et al. [76] and our results in Table 1, and provide a detailed comparison with them below.

Comparison to Bartlett et al. [8]. In this seminal work, the authors study the excess risk of the min-norm interpolator. As discussed before, gradient descent converges to the min-norm interpolator in the overparameterized linear regression setting. One of the main results in Bartlett et al. [8] is to provide a tight bound for the variance part in excess risk as

where ˆ θ = X ⊤ ( XX ⊤ ) -1 Y denotes the min-norm interpolator, and R k ( Σ ) = ( ∑ i>k λ i ) 2 / ( ∑ i>k λ 2 i ) denote another type of effective rank.

By introducing the time factor, Theorem 5.1 improves over Equation (63) in at least two aspects. Firstly, Theorem 5.1 guarantees the consistency of the gradient descent dynamics for a broad range of step number t , while Bartlett et al. [8] study the limiting behavior of the dynamics of t → ∞ . Secondly, Theorem 5.1 implies that the excess risk of early stopping gradient descent solution can be much better than the min-norm interpolator. Compared to the bound in Equation (63), the bound in Corollary 5.1 (a.) replaces k 0 with a much smaller quantity k 1 ; and (b.) drops the second term involving R k 0 ( Σ ) . Therefore, we can derive a consistent bound for an early stopping solution, even though the excess risk of limiting point (min-norm interpolator) can be Ω(1) .

Comparison to Zou et al. [76]. Zou et al. [76] study a different setting, which focuses on the one-pass stochastic gradient descent solution of linear regression. The authors prove a bound for the excess risk as

where ˜ θ t denotes the parameter obtained using stochastic gradient descent (SGD) with constant step size at epoch t . Similar to our bound, Equation 64 also uses the effective dimension k 1 to characterize the variance term. However, we emphasize that Zou et al. [76] derive the bound in a pretty different scenario from ours, which is one-pass SGD scenario. During the one-pass SGD training, one uses a fresh data point to perform stochastic gradient descent in each epoch, and therefore they set t = Θ( n ) by default. As a comparison, we apply the standard full-batch gradient descent, and thus the time can be more flexible. Besides, our results in Corollary 5.1 improve the bound in Equation (64) by dropping the second term. We refer to the third and fourth example in Example 5.1 for a numerical comparison of the bounds 2 .

B.2 Comparisons with Stability-Based Bounds

In this section, we show that Theorem 5.1 gives provably better upper bounds than the stabilitybased method. We cite a result from Teng et al. [67], which uses stability arguments to tackle overparameteried linear regression under similar assumptions.

Theorem B.1 (modified from Theorem 1 in Teng et al. [67]) . Under the overparameterized linear regression settings, assume that ∥ x ∥ ≤ 1 , | ε | ≤ V , w = θ ∗ , ⊤ x √ θ ∗ , ⊤ Σ θ ∗ is σ 2 w -subgaussian. Let B t = sup τ ∈ [ t ] ∥ θ t ∥ . the following inequality holds with probability at least 1 -δ :

Theorem B.1 applies the general stability-based results [19, 24] in the overparameterized linear regression setting, by replacing the bounded Lipschitz condition with the bounded domain condition. A fine-grained analysis [31] may remove the bounded Lipschitz condition, but it additionally requires

zero noise or decaying learning rate, which is different from our setting. We omit the excess risk decomposition technique adopted in Teng et al. [67] for presentation clarity.

Theorem B.1 can not directly yield the stability argument in Theorem 4.1, since obtaining a high probability bound of B t requires a delicate trajectory analysis and is a non-trivial task. Therefore, data-irrelevant methods such as stability-based bounds can not be directly applied to our setting. Even if one can replace B t in Equation 65 with its expectation that is easier to handle (this modification will require adding concentration-related terms, and make the bound in Equation 65 looser), we can still demonstrate that Theorem 5.1 is tighter than the corresponding stability-based analysis by providing a lower bound on E [ B 2 t ] , which will imply a lower bound on the righthand side of Equation 65.

Theorem B.2. Let Assumption 1, 2, 3 holds. Suppose λ = O ( 1 ∑ i> 0 λ i ) . Suppose the conditional variance of the noise ε | x is lower bounded by σ 2 ε . There exists constant c , such that with probability at least 1 -ne -n c , we have for t = o ( n ) ,

First we prove the following lemma, bounding the number of large µ i .

Lemma B.1. Suppose t = o ( n ) . Let l denote the number of µ i , such that µ i = Ω ( n t ) . Then with probability at least 1 -ne -n c , we have l = O ( t ) .

Proof. According to Lemma A.1, we know that with probability at least 1 -ne -n c , Equation 12 holds for all 0 ≤ k ≤ n -1 . Conditioned on this, we have

Since t = o ( n ) , we have l = O ( t ) as claimed.

We also need the result from Bartlett et al. [8], which gives a lowerbound of µ n .

Lemma B.2. (Lemma 10 in Bartlett et al. [8]) For any σ x , there exists a constant c, such that with probability at least 1 -e -n c we have,

We are now ready to prove Theorem B.2.

Proof. We begin with the calculation of ∥ θ t ∥ 2 . By Lemma A.2, the conditional unbiasedness of noise in Assumption 2 and the noise variance lower bound, we have

When µ i = o ( n t ) , we have

Plugging it into Equation 69 and then use Lemma B.1, B.2, we know that under the high probability event in Lemma B.1 and B.2,

Therefore, the stability-based bound, i.e., the right hand side of Equation 65, can be lower bounded in expectation as Ω ( λ 3 t 3 n 2 ∑ i>k 0 λ i ) . This implies that the stability-based bound is vacuous when t = Ω ( n 2 3 ( ∑ i>k 0 λ i ) -1 3 λ ) . Thus, stability-based methods will provably yield smaller compatibility

region than ( ω ( 1 λ ) , o ( n λ )) in Theorem 4.1 when ∑ i>k 0 λ i is not very small, as demonstrated in the examples below.

Example B.1. Let Assumption 1, 2, 3 holds. Assume without loss of generality that λ = Θ(1) . We have the following examples:

1. (Inverse Polynominal). If the spectrum of Σ satisfies

for some α > 1 , we derive that k 0 = Θ( n ) , ∑ i>k 0 λ i = Θ( 1 n α -1 ) . Therefore, the stability bound in Theorem B.1 is vacuous when

which is outperformed by the region in Theorem 5.1 when α < 2 .

(Inverse Log-Polynominal). If the spectrum of Σ satisfies

for some β > 1 , we derive that k 0 = Θ ( n log n ) , ∑ i>k 0 λ i = ˜ Θ(1) . Therefore, the stability bound in Theorem B.1 is vacuous when

which is outperformed by the region in Theorem 5.1.

(Constant). If the spectrum of Σ satisfies

for some ε > 0 , we derive that k 0 = 0 , ∑ i>k 0 λ i = 1 . Therefore, the stability bound in Theorem B.1 is vacuous when

which is outperformed by the region in Theorem 5.1.

(Piecewise Constant). If the spectrum of Σ satisfies

where s = n r , d = n q , 0 < r ≤ 1 , q ≥ 1 , we derive that k 0 = n r , ∑ i>k 0 λ i = 1 . Therefore, the stability bound in Theorem B.1 is vacuous when

which is outperformed by the region in Theorem 5.1.

B.3 Comparisons with Uniform Convergence Bounds

We first state a standard bound on the Rademacher complexity of linear models.

Theorem B.3 (Theorem in Mohri et al. [44]) . Let S ⊆ { x : ∥ x ∥ 2 ≤ r } be a sample of size n and let H = { x ↦→⟨ w,x ⟩ : ∥ w ∥ 2 ≤ Λ } . Then, the empirical Rademacher complexity of H can be bounded as follows:

Furthermore, Talagrand's Lemma (See Lemma 5.7 in Mohri et al. [44]) indicates that

where L = Θ(Λ) is the Lipschitz coefficient of the square loss function l in our setting. Therefore, the Rademacher generalization bound is vacuous when Λ = Ω( n 1 4 ) . By Theorem B.2, we know that E ∥ θ t ∥ 2 = Ω( n 1 2 ) when t = Ω ( n 3 4 λ ( ∑ i>k 0 λ i ) 1 2 ) . A similar comparison as in Example B.1 can

demonstrate that uniform stability arguments will provably yield smaller compatibility region than that in Theorem 5.1 for example distributions.

B.4 Comparison with Previous Works on Early Stopping

A line of works focuses on deriving the excess risk guarantee of linear regression or kernel regression with early stopping (stochastic) gradient descent. We refer to Section 2 for details. Here we compare our results with some most relevant works, including [37, 51, 72].

Comparison with Yao et al. [72]. Yao et al. [72] study kernel regression with early stopping gradient descent. Their approaches are different from ours in the following aspects.

Firstly, the assumptions used in the two approaches are different, due to different goals and techniques. Yao et al. [72] assume that the input feature and data noise have bounded norm (see Section 2.1 in Yao et al. [72]), while we require that the input feature is subgaussian with independent entries.

Furthermore, although Yao et al. [72] obtain a minimax bound in terms of the convergence rate, it is suboptimal in terms of compatibility region. Specifically, The results in our paper show a region like (0 , n ) while the techniques Yao et al. [72] can only lead to a region like (0 , √ n ) . See Proof of the Main Theorem in section 2 in Yao et al. [72] for details. Such differences come from different goals of the two approaches, where Yao et al. [72] focus on providing the optimal early-stopping time while we focus on providing a larger time region in which the loss is consistent.

Comparison with Lin and Rosasco [37]. Lin and Rosasco [37] study stochastic gradient descent with arbitrary batchsize, which is reduced to full batch gradient descent when setting the batchsize to sample size n . Their results are different from ours, since they require the boundness assumption, and focus more on the optimal early stopping time rather than the largest compatibility region, in the same spirit of Yao et al. [72]. Specifically, Lin and Rosasco [37] derive a region like (0 , n ζ +1 2 ζ + γ ) , where ζ and γ are problem dependent constants (See Theorem 1 in Lin and Rosasco [37] for details). The following examples demonstrate that this paper's results yield larger regions for a wide range of distribution classes.

Example B.2. (Inverse Polynominal). If the spectrum of Σ satisfies

for some α > 1 . For this distribution, we have ζ = 1 2 , γ = 1 α , and their region is (0 , n 3 α 2 α +1 ) , which is smaller than (0 , n 3 α +1 2 α +1 ) given in Example 5.2.

Example B.3. (Inverse Log-Polynominal). If the spectrum of Σ satisfies

for some β > 1 . For this distribution, we have ζ = 1 2 , γ = 1 , and their region is (0 , n 3 4 ) , which is smaller than (0 , n ) given Corollary 5.1.

B.5 Calculations in Example 5.1

We calculate the quantities r (Σ) , k 0 , k 1 , k 2 for the example distributions in 5.1. The results validate that k 1 is typically a much smaller quantity than k 0 .

Calculations for λ k = 1 k α , α > 1

. Define r k ( Σ ) = ∑ i>k λ i λ k +1 as in Bartlett et al. [8]. Since ∑ i>k 1 i α = Θ( 1 k α -1 ) , we have r k ( Σ ) = Θ ( 1 k α -1 1 k α ) = Θ( k ) . Hence, k 0 = Θ( n ) 3 , and the conditions of theorem 5.1 is satisfied. As ∑ i> 0 λ i < ∞ , By its definition we know that k 1 is the smallest l such that λ l +1 = O ( 1 n ) . Therefore, k 1 = Θ( n 1 α ) .

Calculations for λ k = 1 k log β ( k +1) , β > 1 .

∑ i>k 1 i log β ( i +1) = Θ( ∫ ∞ k 1 x log β x dx ) = Θ( 1 log β -1 k ) , which implies r k ( Σ ) = k log k . Solving k 0 log k 0 ≥ Θ( n ) , we have k 0 = Θ( n ) .

log n

By the definition of k 1 , we know that k 1 is the smallest l such that l log β ( l +1) ≥ Θ( n ) . Therefore, k 1 = Θ( n log β n ) .

3. Calculations for λ i = 1 n 1+ ε , 1 ≤ i ≤ n 1+ ε , ε > 0 . Since r 0 ( Σ ) = n 1+ ε , we have k 0 = 0 . By the definition of k 1 , we also have k 1 = 0 .

For

0

k < s

,

r

k

(

Σ

) = Θ(

Therefore, k 0 = s = n r .

1

1

s

1

1

-

d

s

Similarly, noting that λ k = 1 n r = ω ( n ) for 0 ≤ k < s and λ s = Θ( 1 n d ) = o ( n ) , we know that k 1 = s = n r .

B.6 Calculations for λ k = 1 /k α , α > 1 in Section 5.3

Set c ( t, n ) = 1 n β , where β > 0 will be chosen later. First we calculate k 2 under this choice of c ( t, n ) . Note that ∑ i>k 1 i α = Θ ( 1 k α -1 ) . Therefore, k 2 is the smallest k such that 1 k α -1 + n k α = O ( 1 n β ) . For the bound on V ( θ t ) to be consistent, we need k 2 = o ( n ) . Hence, 1 k α -1 = O ( n k α ) , which implies k 2 = n β +1 α .

Plugging the value of c ( t, n ) and k 2 into our bound, we have

which attains its minimum Θ( n 2 ατ -3 α +2 τ -1 2 α +1 ) at β = Θ ( 2 ατ -α -1 2 α +1 ) .

For V ( θ ) = O (1) , we need τ ≤ 3 α +1 2 α +2 . For β ≥ 0 , we need τ ≥ α +1 2 α . Putting them together gives the range of t in which the above calculation applies.

B.7 Discussions on D n

In this paper, the distribution D is regarded as a sequence of distribution {D n } which may dependent on sample size n . The phenomenon comes from overparameterization and asymptotic requirements. In the definition of compatibility, we require n →∞ . In this case, overparameterization requires that the dimension p (if finite) cannot be independent of n since n →∞ would break overparameterization. Therefore, the covariance Σ also has n -dependency, since Σ is closely related to p .

Several points are worth mentioning: (1) Similar arguments generally appear in related works, for example, in Bartlett et al. [8] when discussing the definition of benign covariance. (2) One can avoid such issues by considering an infinite dimensional feature space. This is why we discuss the special case p = ∞ in Theorem 4.1. (3) If p is a fixed finite constant that does not alter with n , the problem becomes underparameterized and thus trivial to get a consistent generalization bound via standard concentration inequalities.

C Additional Experiment Results

C.1 Details for Linear Regression Experiments

In this section, we provide the experiment details for linear regression experiments and present additional empirical results.

The linear regression experiment in Figure 1 follows the setting described in section 6. Although the final iterate does not interpolate the training data, the results suffice to demonstrate the gap between the early-stopping and final-iterate excess risk. The training plot for different covariances are given in Figure 2.

Next, we provide the experiment results of sample sizes n = 50 , n = 200 and feature dimensions p = 500 , p = 2000 . The settings are the same as described above, except for the sample size. The optimal excess risk and min-norm excess risk are provided in Table 3, 4, 6 and 5. The tables indicate that the two observations stated above hold for different sample size n .

) = Θ(

n

r

) =

o

(

n

)

, while

r

s

(

Σ

) =

= Θ(

n

q

) =

ω

(

n

)

.

(a)

λ

i

=

1

i

(b)

λ

i

=

1

i

2

(c)

λ

i

=

1

i

3

(d)

λ

i

=

1

i

log(

i

+1)

(e)

λ

i

=

1

i

log

2

(

i

+1)

(f)

λ

i

=

1

i

log

3

(

i

+1)

10?

10*

103

103

103

Excess Risk

Training Loss

10-2

10′

Training Loss

Excess Risk

102

Excess Risk

Training Loss

Excess Risk

10-4

LoSS

Risk

10°

Loss

Excess Risk

10°

Training Loss

10-1

Trainina

ExcessF

102

Training

101

:0t

10-14,

10

100

10-1#

10~1

10°1

10°

10-1.

ce-0t -

10°

109

103

102

Epochs

10

104

10§

10#

10°

106

10-1

109

10

102

Epochs

10

10#

104

10#

10

10-

10-

100

103

102

Epochs

103

104

105

10#

10

10-

103

101

10?

101

10?

101

Excess Risk

Excess Risk

Excess Risk

Training Loss

10°

Training Loss

Training Loss

Excess Risk

102

LoSS

Excess Risk

102

100

Loss

Excess Risk

10?

100

LoSS

10-1

Training

Training

Training

101

10

10

10-2

10-1

10-1

10°

10°

10°

10-3

10-1

10°

101

102

Epochs

101

104

105

10%

10′

104

101

109

101

c01

Epochs

103

10#

10§

106

10?

10-1

109

103

102

Epochs

103

10

10

10#

10?

10~2

C.2 Details for MNIST Experiments

In this section, we provide the experiment details and additional results in MNIST dataset.

The MNIST experiment details are described below. We create a noisy version of MNIST with label noise rate 20% , i.e. randomly perturbing the label with probability 20% for each training data, to simulate the label noise which is common in real datasets, e.g ImageNet [60, 65, 73]. We do not inject noise into the test data.

We choose a standard four layer convolutional neural network as the classifier. We use a vanilla SGD optimizer without momentum or weight decay. The initial learning rate is set to 0 . 5 . Learning rate is decayed by 0.98 every epoch. Each model is trained for 300 epochs. The training batch size is set to 1024, and the test batch size is set to 1000. We choose the standard cross entropy loss as the loss function.

We provide the plot for different levels of label noise in Figure 3. We present the corresponding test error of the best early stopping iterate and the final iterate in Table 7. Since the theoretical part of this paper focuses on GD, we also provide a corresponding plot of GD training in Figure 4 for completeness.

(a) 0% label noise

(b) 10% label noise

(c) 20% label noise

(d) 30% label noise

(e) 40% label noise

(f) 50% label noise

Test Error

Test Error

Test Error

50

Training Error

50

50

Training Error

50

50

Training Error

50

Training Error

Training Error

Test Error

TestError

40

TestError

20

20

10

10

Epochs

200

250

300

100

Epochs

0S1

200

250

100

Epochs

209

250

60

Test Error

Test Error

Test Error

Training Error

Training Error

Training Error

50

Training Error

TestError

TestError

Test Error

30

30

20

20

20

20

10

10

10

50

100

Epochs

150

200

250

00E

50

100

Epochs

150

200

250

300

50

100

Epochs

150

200

250

800

60

60

Test Error

50

Training Error

50

Training Error

Test Error

40

30

20

20

10

10

250

500

750

Epochs

1000

1250

1500

1750

2000

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper suggests a notion of compatibility, which is an algorithm-dependent and data-dependent term to bound the excess risk of overparameterized linear regression models. Using this notion, the authors come up with three regimes on the covariance matrix, and characterize the compatible region (that is the region of steps where we can train the model with GD and still converge in terms of excess risk). The authors show the connection between compatibility and benign overfitting and also compare to stability bound. Several examples are provided to demonstrate how the compatibility region depends on the covariance structure.\n\nSTRENGTHS:\nI find the paper to be well-written and easy to follow. The overall topic of generalization of overparameterized models (even in the linear case) is important. The authors make a considerable effort to introduce existing bounds.\n\nWEAKNESSES:\n1. I find the study of overparameterized linear model *together with* early stopping less motivating. In particular, early stopping for linear regression is equivalent to $\\ell_2$ regularization. In this case, we can already apply uniform convergence bound for generalization error. In this case, is the motivation mainly to get a better bound? If so, I would suggest the authors move the comparison to stability bound into the main text. This is in contrast to the overfitting case since if we are overfitting, there is no uniform convergence bound at all.\n\n2. One relevant work is missing, [1] (and its related citations) discussed cases when we have different kinds of overfitting. In particular, their theorem 3.1 (b) corresponds to your example 4.1. I would love to see how your results are related to theirs. \n\n[1] Benign, Tempered, or Catastrophic: A Taxonomy of Overfitting\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper suggests a new framework for evaluating the generalization error called compatibility. Compatibility takes into account both the data and algorithm and says the two are compatible if there is an early stopping region, as $n \\to \\infty$ that has zero excess risk. The paper then considers the linear regression framework and provides spectral conditions under which full batch constant step size gradient descent is compatible.\n\nSTRENGTHS:\nThe main strength of the paper is the generality of the data distributions considered. Specifically, most prior theory work assumes we have isotropic data or some fixed transform of isotropic data. However, this paper considers general anisotropic data. In this setup, they provide mild conditions under which we see that early stopping is beneficial for generalization. \n\nThe paper also provides clear evidence of the benefits of early stopping. Showing that the min norm solution is not necessarily the optimal solution for generalization. Further, the paper shows that this is true even in the large data limit when the min norm solution is benign.\n\nWEAKNESSES:\nThe are two main weaknesses (one major one minor)\n\n1) The paper is missing the comparison to Madhu S. Advani, Andrew M. Saxe, and Haim Sompolinsky. High-dimensional Dynamics of\nGeneralization Error in Neural Networks. Neural Networks, 2020 \n\nWhile the data assumptions in the above paper are more restrictive, that is isotropic data. The paper exactly characterizes the risk curve as a function of time and determines the optimal stopping time. Hence a proper discussion of this is crucial. \n\n\n3) While the paper is well written, the paper sweeps technical details under the rug that would be important to know. For example, the constants $c_0$ and $c_1$ are opaque and play an important role. See my questions for more instances.\n\nQUESTIONS:\nIn Assumption 1 subpoint 2, should it $\\lambda_p > 0$? Instead of $\\lambda_1 > 0$.\n\nAre the Cs on lines 169 and 173 supposed to be the same?\n\nWhy is $\\boldsymbol{Y}$ called the noise vector?\n\nIn Equations (4), (5), how do we know such an $\\ell$ exists?\n\nWhat is $c$ in the statement of Theorem 5.1 in $k_0 \\le n/c$?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the benign overfitting phenomenon in overparameterized linear regression problems under early stopping type gradient descent. \n\nSpecifically, this paper gives a time variant bound for the excess risk during the gradient descent training.\nBy comparing the optimal time variant bound with the previous bounds for the minimum-norm interpolant, this paper finds some data distribution examples that the optimal early stopping satisfies benign overfitting (compatibility) but the minimum-norm interplant does not. \nAmong all the examples, some of which outperform the previous early stopping type bounds in overparameterized linear regression problems.\n\nSTRENGTHS:\n* The organization and presentation of this paper is smooth and clear, and it provides a better understanding on the benign overfitting phenomenon for early stopping type gradient descent.\n* The theory in this paper is solid. Although it mainly follows from the previous two works *Bartlett et al.* and *Zou et al.*, the proposed notion 'effective dimensions' for the feature covariance matrix in this paper provides an alternative way to analyze the variance term in the Bias–variance decomposition.\n\nWEAKNESSES:\nThere is no major weakness for me.\n\n---\n\nA minior issue:\n\nFor the examples and the numerical experiments in this paper, it would be better if the authors could provide the patterns to show the interplay between the bias term $B(\\theta_t)$ and the variance term $V(\\theta_t)$ during training.\nEspecially when comparing with the bounds in previous works, it would be more intuitive to see why the compatibility happens but the benign overfitting does not, if the comparisons  of the evolutions of $B(\\theta_t)$ and $V(\\theta_t)$ are shown separately.\n\nQUESTIONS:\n* In table 2, for the distributions that $\\lambda_i = \\frac{i}{i \\log^2(i+1)}$ and $\\lambda_i = \\frac{i}{i \\log^3(i+1)}$, both cases are benign overfitting based on *Bartlett et al*, but the min-norm excess risk seems to be very large though, is this caused by the bias term or the variance term?\n\n* I am curious what the generalization bound for early stopping will be like if following the notations in *Theorem 4. [Bartlett et al.]* in terms of $r_0(\\Sigma), k^*, R_{k^*}(\\Sigma)$?\nI am trying to understand the difference between methods using the proposed effective dimension in this paper and using the effective ranks in *Bartlett et al.*, because understanding the differences may help to motivates other more refined controls for the generalization analysis.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a new mathematical framework for characterizing generalization of over-parameterized machine learning models. The new framework provides improved bounds from prior work, and improves on prior work by combining both data and algorithm information. To summarize their contributions: The paper formalizes a notion of compatibility, as a minimal condition for generalization based on the interaction between data and algorithm. They derive sufficient conditions for compatibility for over-parameterized linear regression with gradient descent, and then derive time-variant generalization bounds for the over-parameterized linear regression and show empirical results that verify the motivation of compatibility and demonstrate benefits of early stopping.\n\nCaveat: This paper falls outside my area of expertise to the level that in my available time I cannot vouch for the accuracy of the theorems provided in the paper.\n\nSTRENGTHS:\nTheoretical proofs are given, along with bounds comparisons, and empirical results. The paper seems well-written and rigorous in its assessment. The problem is well motivated and extremely well-referenced in its citations and descriptions of the current work in this area. The ability and degree to which over-parameterized models can generalize is one of the main mysteries of deep learning, and where traditional theory in generalization breaks down, so the paper is extremely relevant.\n\nWEAKNESSES:\nThis paper requires substantial knowledge of generalization theory to understand and is not accessible to a broader audience. For example, the introduction makes the assumption that terms like excess risk are understood by the reader, before they are later defined. While the theory is claimed to be general, thorough treatment is only given for linear regression, and derivations of sufficiency conditions and bounds for other problems are left as future work. While deep learning was given as a primary motivation to understand generalization for over-parameterized models, application to these models was not addressed in the paper. The experimental and conclusions sections were very minimal. I understand that the theory was the primary focus of the paper, but would have liked to see more practical demonstration and discussion of implications for the theoretical results that can be drawn from your investigations.\n\nQUESTIONS:\nAre there any roadblocks that would make application of this difficult for other models, such as non-linear regression problems including deep neural networks? Are there larger or broader implications of data-algorithm compatibility and your results that you can discuss?\n\nLIMITATIONS:\nSocietal impact does not seem relevant or needed to address for this paper. The paper acknowledges a limited scope to linear cases, but does not go into any additional weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-26"}