SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jing Xu*
xujing21@mails.tsinghua.edu.cn
Jiaye Teng*
tjy20@mails.tsinghua.edu.cn
Yang Yuan
Shanghai Qi Zhi Institute
yuanyang@tsinghua.edu.cn
andrewcyao@tsinghua.edu.cn
One of the major open problems in machine learning is to characterize generalization in the overparameterized regime, where most traditional generalization bounds become inconsistent even for overparameterized linear regression [46]. In many scenarios, this failure can be attributed to obscuring the crucial interplay between the training algorithm and the underlying data distribution. This paper demonstrates that the generalization behavior of overparameterized model should be analyzed in a both data-relevant and algorithm-relevant manner. To make a formal characterization, We introduce a notion called data-algorithm compatibility, which considers the generalization behavior of the entire data-dependent training trajectory, instead of traditional last-iterate analysis. We validate our claim by studying the setting of solving overparameterized linear regression with gradient descent. Specifically, we perform a data-dependent trajectory analysis and derive a sufficient condition for compatibility in such a setting. Our theoretical results demonstrate that if we take early stopping iterates into consideration, generalization can hold with significantly weaker restrictions on the problem instance than the previous last-iterate analysis.
Although deep neural networks achieve great success in practice [13, 14, 63], their remarkable generalization ability is still among the essential mysteries in the deep learning community. One of the most intriguing features of deep neural networks is overparameterization, which confers a level of tractability to the training problem, but leaves traditional generalization theories failing to work. In generalization analysis, both the training algorithm and the data distribution play essential roles [23, 28]. For instance, a line of work [46, 74] highlights the role of the algorithm by showing that the algorithm-irrelevant uniform convergence bounds can become inconsistent in deep learning regimes. Another line of work [8, 68] on benign overfitting emphasizes the role of data distribution via profound analysis of specific overparameterized models.
Despite the significant role of data and algorithm in generalization analysis, existing theories usually focus on either the data factor (e.g., uniform convergence [46] and last iterate analysis [8, 68]) or the algorithm factor (e.g., stability-based bounds [24]). Combining both data and algorithm factor into
(a) Linear Regression
(b) Corrupted MNIST
Figure 1: (a) The training plot for linear regression with spectrum  $\lambda_{i} = 1 / i^{2}$  using GD. Note that the axes are in the log scale. (b) The training plot of CNN on corrupted MNIST with  $20\%$  label noise using SGD. Both models successfully learn the useful features in the initial phase of training, but it takes a long time for them to fit the noise in the dataset. The observations demonstrate the power of data-dependent trajectory analysis, since the early stopping solutions on the trajectory generalize well but the final iterate fails to generalize. See Appendix C for details.
generalization analysis can help derive tighter generalization bounds and explain the generalization ability of overparameterized models observed in practice. In this sense, a natural question arises:
How to incorporate both data factor and algorithm factor into generalization analysis?
To gain insight into the interplay between data and algorithms, we provide motivating examples of a synthetic overparameterized linear regression task and a classification task on the corrupted MNIST dataset in figure 1. In both scenarios, the final iterate with less algorithmic information, which may include the algorithm type (e.g., GD or SGD), hyperparameters (e.g., learning rate, number of epochs), generalizes much worse than the early stopping solutions (see the Blue Line). In the linear regression case, the generalization error of the final iterate can be more than  $\times 100$  larger than that of the early stopping solution. In the MNIST case, the final iterate on the SGD trajectory has  $19.9\%$  test error, much higher than the  $2.88\%$  test error of the best iterate on the GD trajectory. Therefore, the almost ubiquitous strategy of early stopping is a key ingredient in generalization analysis for overparameterized models, whose benefits have been demonstrated both theoretically and empirically [1, 2, 27, 34, 41, 72]. By focusing on the entire optimization trajectory and performing data-dependent trajectory analysis, both data information and the dynamics of the training algorithm can be exploited to yield consistent generalization bounds.
When we take the algorithm into consideration and analyze the data-dependent training trajectory, generalization occurs if the minimum excess risk of the iterates on the training trajectory converges to zero, as the sample size tends to infinity. This accords with the real practice of training deep neural networks, where one can pick up the best parameter on the training trajectory, by calculating its loss on a validation dataset. We dub this notion of generalization as data-algorithm-compatibility, which is formally defined in Section 3.2.
The significance of compatibility comes in three folds. Firstly, it incorporates both data and algorithm factors into generalization analysis, and is suitable for the overparameterization regime (see Definition 3.1). Secondly, it serves as a minimal condition for generalization, without which one cannot expect to find a consistent solution via standard learning procedures. Consequently, compatibility holds with only mild assumptions and applies to a wide range of problem instances (see Theorem 4.1). Thirdly, it captures the algorithmic significance of early stopping in generalization. By exploiting the algorithm information along the entire trajectory, we arrive at better generalization bounds than the last-iterate analysis (see Table 1 and 2 for examples).
To theoretically validate compatibility, we study it under overparameterized linear regression setting. Analysis of the overparameterized linear regression is a reasonable starting point to study more complex models like deep neural networks [17, 57], since many phenomena of the high dimensional non-linear model are also observed in the linear regime (e.g., Figure 1). Furthermore, the neural
tangent kernel (NTK) framework [4, 26] demonstrates that very wide neural networks trained using gradient descent with appropriate random initialization can be approximated by kernel regression in a reproducing kernel Hilbert space, which rigorously establishes a close relationship between overparameterized linear regression and deep neural network training.
Specifically, we investigate solving overparameterized linear regression using gradient descent with constant step size, and prove that under some mild regularity conditions, gradient descent is compatible with overparameterized linear regression if the effective dimensions of the feature covariance matrix are asymptotically bounded by the sample size. Compared with the last-iterate analysis [8], the main theorems in this paper require significantly weaker assumptions, which demonstrates the benefits of data-relevant and algorithm-relevant generalization analysis.
- We formalize the notion of data-algorithm-compatibility, which highlights the interaction between data and algorithm and serves as a minimal condition for generalization.
- We derive a sufficient condition for compatibility in solving overparameterized linear regression with gradient descent. Our theory shows that generalization of early-stopping iterates requires much weaker restrictions in the considered setting.
- Technically, we derive time-variant generalization bounds for overparameterized linear regression via data-dependent trajectory analysis. Empirically, we conduct the various experiments to verify the theoretical results and demonstrate the benefits of early stopping.
Data-Dependent Techniques mainly focus on the data distribution condition for generalization. One of the most popular data-dependent methods is uniform convergence [7, 29, 74, 75]. However, recent works [46, 47] point out that uniform convergence may not be powerful enough to explain generalization, because it may only yield inconsistent bound in even linear regression cases. Another line of works investigates benign overfitting, which mainly studies generalization of overfitting solutions [8, 21, 35, 68, 70, 76, 77].
Algorithm-Dependent Techniques measure the role of the algorithmic information in generalization. A line of works derives generalization bounds via algorithm stability [9, 12, 18, 19, 24, 31, 32, 45, 67]. A parallel line of works analyzes the implicit bias of algorithmic information [11, 25, 39, 40, 59, 64], which are mainly based on analyzing a specific data distribution (e.g., linear separable).
Other Generalization Techniques. Besides the techniques above, there are many other approaches. For example, PAC-Bayes theory performs well empirically and theoretically [16, 42, 43, 48, 49, 58, 61] and can yield non-vacuous bounds in deep learning regimes [50, 54]. Furthermore, there are other promising techniques including information theory [6, 56, 71] and compression-based bounds [3].
Early Stopping has the potential to improve generalization for various machine learning problems [5, 30, 33, 38, 53, 62, 69, 74]. A line of works studies the rate of early stopping in linear regression and kernel regression with different algorithms, e.g., gradient descent [72], stochastic gradient descent [15, 37, 51, 55, 66], gradient flow [2], conjugate gradient [10] and spectral algorithms [22, 36]. Beyond linear models, early-stopping is also effective for training deep neural networks [27, 34]. Another line of research focuses on the signal for early stopping [20, 52].
In this section, we formally define compatibility between the data distribution and the training algorithm, starting from the basic notations.
Data Distribution. Let  $\mathcal{D}$  denote the population distribution and  $z\sim \mathcal{D}$  denote a data point sampled from distribution  $\mathcal{D}$ . Usually,  $z$  contains a feature and its corresponding response. Besides, we denote the dataset with  $n$  samples as  $Z\triangleq \{z_i\}_{i\in [n]}$ , where  $z_{i}\sim \mathcal{D}$  are i.i.d. sampled from distribution  $\mathcal{D}$ .
Loss and Excess Risk. Let  $\ell(\theta; z)$  denote the loss on sample  $z$  with parameter  $\theta \in \mathbb{R}^p$ . The corresponding population loss is defined as  $L(\theta; \mathcal{D}) \triangleq \mathbb{E}_{z \sim \mathcal{D}} \ell(\theta; z)$ . When the context is clear, we omit the dependency on  $\mathcal{D}$  and denote the population loss by  $L(\theta)$ . Our goal is to find the optimal parameter  $\theta^*$  which minimizes the population loss, i.e.,  $L(\theta^*) = \min_{\theta} L(\theta)$ . Measuring how a parameter  $\theta$  approaches  $\theta^*$  relies on a term excess risk  $R(\theta)$ , defined as  $R(\theta) \triangleq L(\theta) - L(\theta^*)$ .
Algorithm. Let  $\mathcal{A}(\cdot)$  denote a iterative algorithm that takes training data  $Z$  as input and outputs a sequence of parameters  $\{\pmb{\theta}_n^{(t)}\}_{t\geq 0}$ , where  $t$  is the iteration number. The algorithm can be either deterministic or stochastic, e.g., variants of (S)GD.
3.2 Definitions of Compatibility
Based on the above notations, we introduce the notion of compatibility between data distribution and algorithm in Definition 3.1. Informally, compatibility measures whether a consistent excess risk can be reached along the training trajectory. Note that we omit the role of the loss function in the definition, although the algorithm depends on the loss function.
Definition 3.1 (Compatibility). Given a loss function  $\ell(\cdot)$  with corresponding excess risk  $R(\cdot)$ , a data distribution  $\mathcal{D}$  is compatible with an algorithm  $\mathcal{A}$  if there exists nonempty subsets  $T_n$  of  $\mathbb{N}$ , such that  $\sup_{t \in T_n} R(\pmb{\theta}_n^{(t)})$  converges to zero in probability as sample size  $n$  tends to infinity, where  $\{\pmb{\theta}_n^{(t)}\}_{t \geq 0}$  denotes the output of algorithm  $\mathcal{A}$ , and the randomness comes from the sampling of training data  $\mathbf{Z}$  from distribution  $\mathcal{D}$  and the execution of algorithm  $\mathcal{A}$ . That is to say,  $(\mathcal{D}, \mathcal{A})$  is compatible if there exists nonempty sets  $T_n$ , such that
$$
\sup  _ {t \in T _ {n}} R \left(\boldsymbol {\theta} _ {n} ^ {(t)}\right) \stackrel {{P}} {{\to}} 0 \quad a s \quad n \rightarrow \infty . \tag {1}
$$
We call  $\{T_n\}_{n > 0}$  the compatibility region of  $(\mathcal{D},\mathcal{A})$ . The distribution  $\mathcal{D}$  is allowed to change with  $n$ . In this case,  $\mathcal{D}$  should be understood as a sequence of distributions  $\{\mathcal{D}_n\}_{n\geq 1}$ . We also allow the dimension of model parameter  $\theta$  to be infinity or to grow with  $n$ . We omit this dependency on  $n$  when the context is clear.
Compatibility serves as a minimal condition for generalization, since if a data distribution is incompatible with the algorithm, one cannot expect to reach a small excess risk even if we allow for arbitrary early stopping. However, we remark that considering only the minimal excess risk is insufficient for a practical purpose, as one cannot exactly find the  $t$  that minimizes  $R(\pmb{\theta}_n^{(t)})$  due to the noise in the validation set. Therefore, it is meaningful to consider a region of time  $t$  on which the excess risk is consistent as in Definition 3.1. The larger the region is, the more robust the algorithm will be to the noise in its execution.
Comparisons with Other Notions. Compared to classic definitions of learnability, e.g., PAC learning, the definition of compatibility is data-specific and algorithm-specific, and is thus a more fine-grained notion. Compared to the concept of benign proposed in [8], which studies whether the excess risk at  $t = \infty$  converges to zero in probability as the sample size goes to infinity, compatibility only requires that there exists a time to derive a consistent excess risk. We will show later in Section 4.2 that in the overpameterized linear regression setting, there exist cases such that the problem instance is compatible but not benign.
4 Analysis of Overparameterized Linear Regression with Gradient Descent
To validate the meaningfulness of compatibility, we study it in the overparameterized linear regression regime. We first introduce the data distribution, loss, and training algorithm, and then present the main theorem, which provides a sufficient condition for compatibility in this setting.
4.1 Preliminaries for Overparameterized Linear Regression
Notations. Let  $O, o, \Omega, \omega$  denote asymptotic notations, with their usual meaning. For example, the argument  $a_{n} = O(b_{n})$  means that there exists a large enough constant  $C$ , such that  $a_{n} \leq C b_{n}$ . We use  $\lesssim$  with the same meaning as the asymptotic notation  $O$ . Besides, let  $\| x \|$  denote the  $\ell_2$  norm for vector  $x$ , and  $\| A \|$  denote the operator norm for matrix  $A$ . We allow the vector to belong to
a countably infinite-dimensional Hilbert space  $\mathcal{H}$ , and with a slight abuse of notation, we use  $\mathbb{R}^{\infty}$  interchangeably with  $\mathcal{H}$ . In this case,  $x^{\top}z$  denotes inner product and  $xz^{\top}$  denotes tensor product for  $x,z\in \mathcal{H}$ . A random variable  $X$  is called  $\sigma$ -subgaussian if  $\mathbb{E}[e^{\lambda X}] \leq e^{\lambda^2\sigma^2 / 2}$  for any  $\lambda$ .
Data Distribution. Let  $(\boldsymbol{x},y)\in \mathbb{R}^p\times \mathbb{R}$  denote the feature vector and the response, following a joint distribution  $\mathcal{D}$ . Let  $\Sigma \triangleq \mathbb{E}[x x^{\top}]$  denote the feature covariance matrix, whose eigenvalue decomposition is  $\boldsymbol {\Sigma} = \boldsymbol {V}\boldsymbol {\Lambda}\boldsymbol {V}^{\top} = \sum_{i > 0}\lambda_{i}\boldsymbol{v}_{i}\boldsymbol{v}_{i}^{\top}$  with decreasing eigenvalues  $\lambda_1\geq \lambda_2\geq \dots$  We make the following assumptions on the distribution of the feature vector.
Assumption 1 (Assumptions on feature distribution). We assume that
1.  $\mathbb{E}[\pmb {x}] = 0$
2.  $\lambda_1 > 0, \sum_{i > 0} \lambda_i < C$  for some absolute constant  $C$ .
3. Let  $\tilde{\pmb{x}} = \pmb{\Lambda}^{-\frac{1}{2}}\pmb{V}^{\top}\pmb{x}$ . The random vector  $\tilde{\pmb{x}}$  has independent  $\sigma_{x}$ -subgaussian entries.
Loss and Excess Risk. We choose square loss as the loss function  $\ell$ , i.e.  $\ell(\pmb{\theta}, (\pmb{x}, y)) = 1/2(y - \pmb{x}^\top \pmb{\theta})^2$ . The corresponding population loss is denoted by  $L(\pmb{\theta}) = \mathbb{E}\ell(\pmb{\theta}, (\pmb{x}, y))$  and the optimal parameter is denoted by  $\pmb{\theta}^* \triangleq \operatorname*{argmin}_{\pmb{\theta} \in \mathbb{R}^p} L(\pmb{\theta})$ . We assume that  $\|\pmb{\theta}^*\| < C$  for some absolute constant  $C$ . If there are multiple such minimizers, we choose an arbitrary one and fix it thereafter. We focus on the excess risk of parameter  $\pmb{\theta}$ , defined as
$$
R (\boldsymbol {\theta}) = L (\boldsymbol {\theta}) - L \left(\boldsymbol {\theta} ^ {*}\right) = \frac {1}{2} \left(\boldsymbol {\theta} - \boldsymbol {\theta} ^ {*}\right) ^ {\top} \boldsymbol {\Sigma} \left(\boldsymbol {\theta} - \boldsymbol {\theta} ^ {*}\right). \tag {2}
$$
Let  $\varepsilon = y - \pmb{x}^{\top}\pmb{\theta}^{*}$  denote the noise in data point  $(x,y)$ . The following assumptions involve the conditional distribution of the noise.
Assumption 2 (Assumptions on noise distribution). We assume that
1. The conditional noise  $\varepsilon |x$  has zero mean.
2. The conditional noise  $\varepsilon |x$  is  $\sigma_y$ -subgaussian.
Note that both Assumption 1 and Assumption 2 are commonly considered in the related literatures [8, 68, 76].
Training Set. Given a training set  $\{(\pmb{x}_i, y_i)\}_{1 \leq i \leq n}$  with  $n$  pairs independently sampled from the population distribution  $\mathcal{D}$ , we define  $\pmb{X} \triangleq (\pmb{x}_1, \dots, \pmb{x}_n)^\top \in \mathbb{R}^{n \times p}$  as the feature matrix,  $\pmb{Y} \triangleq (y_1, \dots, y_n)^\top \in \mathbb{R}^n$  as the corresponding noise vector, and  $\varepsilon \triangleq \pmb{Y} - \pmb{X} \pmb{\theta}^*$  as the residual vector. Let the singular value decomposition (SVD) of  $\pmb{X}$  be  $\pmb{X} = \pmb{U} \tilde{\Lambda}^{\frac{1}{2}} \pmb{W}^\top$ , with  $\tilde{\Lambda} = \mathrm{diag}\{\mu_1, \dots, \mu_n\} \in \mathbb{R}^{n \times n}, \mu_1 \geq \dots \geq \mu_n$ .
We consider the overparameterized regime where the feature dimension is larger than the sample size, namely,  $p > n$ . In this regime, we assume that  $\mathrm{rank}(\mathbf{X}) = n$  almost surely as in Bartlett et al. [8]. This assumption is equivalent to the invertibility of  $XX^{\top}$ .
Assumption 3 (Linear independent training set). For any  $n < p$ , we assume that the features in the training set  $\{\pmb{x}_1, \pmb{x}_2, \dots, \pmb{x}_n\}$  is linearly independent almost surely.
Algorithm. Given the dataset  $(X, Y)$ , define the empirical loss function as  $\hat{L}(\theta) \triangleq \frac{1}{2n} \| X\theta - Y\|^2$ . We choose full-batch gradient descent on the empirical risk with a constant learning rate  $\lambda$  as the algorithm  $\mathcal{A}$  in the previous template. In this case, the update rule for the optimization trajectory  $\{\theta_t\}_{t \geq 0}$  is formulated as
$$
\boldsymbol {\theta} _ {t + 1} = \boldsymbol {\theta} _ {t} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} (\boldsymbol {X} \boldsymbol {\theta} _ {t} - \boldsymbol {Y}). \tag {3}
$$
Without loss of generality, we consider zero initialization  $\pmb{\theta}_0 = \mathbf{0}$  in this paper. In this case, for a sufficiently small learning rate  $\lambda$ ,  $\pmb{\theta}_t$  converges to the min-norm interpolator  $\hat{\pmb{\theta}} = \pmb{X}^{\top}(\pmb{X}\pmb{X}^{\top})^{-1}\pmb{Y}$  as  $t$  goes to infinity, which was well studied previously [8]. This paper takes one step further and discuss the excess risk along the entire training trajectory  $\{R(\pmb{\theta}_t)\}_{t\geq 0}$ .
Effective Rank and Effective Dimensions. We define the effective ranks of the feature matrix  $\pmb{\Sigma}$  as  $r(\pmb{\Sigma}) \triangleq \frac{\sum_{i > 0} \lambda_i}{\lambda_1}$  and  $R_k(\pmb{\Sigma}) \triangleq \frac{(\sum_{i > k} \lambda_i)^2}{\sum_{i > k} \lambda_i^2}$ . Our results depend on two notions of effective dimension of the feature covariance  $\pmb{\Sigma}$ , defined as
$$
k _ {0} \triangleq \min  \left\{l \geq 0: \lambda_ {l + 1} \leq \frac {c _ {0} \sum_ {i > l} \lambda_ {i}}{n} \right\}, \tag {4}
$$
$$
k _ {1} \triangleq \min  \left\{l \geq 0: \lambda_ {l + 1} \leq \frac {c _ {1} \sum_ {i > 0} \lambda_ {i}}{n} \right\}, \tag {5}
$$
where  $c_{0}, c_{1}$  are constants independent of the dimension  $p$ , sample size  $n$ , and time  $t^{1}$ . We omit the dependency of  $k_{0}, k_{1}$  on  $c_{0}, c_{1}, n, \Sigma$  when the context is clear.
4.2 Main Theorem for Overparameterized Linear Regression with Gradient Descent
Next, we present the main result of this section, which provides a clean condition for compatibility between gradient descent and overparameterized linear regression.
Theorem 4.1. Consider the overparameterized linear regression setting defined in section 4.1. Let Assumption 1,2 and 3 hold. Assume the learning rate satisfies  $\lambda = O\left(\frac{1}{\mathrm{Tr}(\Sigma)}\right)$ .
- If the covariance satisfies  $k_{0} = o(n), R_{k_{0}}(\pmb{\Sigma}) = \omega(n)$ ,  $r(\pmb{\Sigma}) = o(n)$ , it is compatible in the region  $T_{n} = \left( \omega \left( \frac{1}{\lambda} \right), \infty \right)$ .
- If the covariance satisfies  $k_{0} = O(n), k_{1} = o(n), r(\Sigma) = o(n)$ , it is compatible in the region  $T_{n} = \left(\omega\left(\frac{1}{\lambda}\right), o\left(\frac{n}{\lambda}\right)\right)$ .
- If the covariance does not change with  $n$ , and satisfies  $k_0 = O(n)$  and  $p = \infty$ , it is compatible in the region  $T_n = \left(\omega\left(\frac{1}{\lambda}\right), o\left(\frac{n}{\lambda}\right)\right)$ .
The proof of Theorem 4.1 is given in Appendix A and sketched in Section 5. The theorem shows that gradient descent is compatible with overparameterized linear regression under some mild regularity conditions on the learning rate, effective rank and effective dimensions. The condition on the learning rate is natural for optimizing a smooth objective. We conjecture that the condition  $k_{0} = O(n)$  can not be removed in general cases, since the effective dimension  $k_{0}$  characterizes the concentration of the singular values of the data matrix  $X$  and plays a crucial role in the excess risk of the gradient descent dynamics.
Comparison with Benign Overfitting. The paper [8] studies overparameterized linear regression and gives the condition for min-norm interpolator to generalize. They prove that the feature covariance  $\Sigma$  is benign if and only if
$$
k _ {0} = o (n), R _ {k _ {0}} (\boldsymbol {\Sigma}) = \omega (n), r (\boldsymbol {\Sigma}) = o (n) \tag {6}
$$
As discussed in Section 3.2, benign problem instance also satisfies compatibility, since benign overfitting requires a stronger condition on  $k_{0}$  and an additional assumption on  $R_{k_0}(\Sigma)$ . The following example shows that this inclusion relationship is strict.
Example 4.1. Under the same assumption as in Theorem 4.1, if the spectrum of  $\Sigma$  satisfies
$$
\lambda_ {k} = \frac {1}{k ^ {\alpha}}, \tag {7}
$$
for some  $\alpha > 1$ , we derive that  $k_0 = \Theta(n)$ . Therefore, this problem instance satisfies compatibility, but does not satisfy benign overfitting.
Example 4.1 shows the existence of a case where the early stopping solution can generalize but interpolating solution fails. Therefore, Theorem 4.1 can characterize generalization for a much wider range of problem instances.