Title: value: Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regression
Authors: value: Jing Xu
Jiaye Teng
Yang Yuan
Andrew C Yao
Venue: NeurIPS 2023
Keywords: value: data-algorithm dependent generalization analysis
overparameterized linear regression

Abstract: value: One of the major open problems in machine learning is to characterize generalization in the overparameterized regime, where most traditional generalization bounds become inconsistent even for overparameterized linear regression. In many scenarios, this failure can be attributed to obscuring the crucial interplay between the training algorithm and the underlying data distribution. This paper demonstrate that the generalization behavior of overparameterized model should be analyzed in a both data-relevant and algorithm-relevant manner. To make a formal characterization, We introduce a notion called data-algorithm compatibility, which considers the generalization behavior of the entire data-dependent training trajectory, instead of traditional last-iterate analysis.  We validate our claim by studying the setting of solving overparameterized linear regression with gradient descent. Specifically, we perform a data-dependent trajectory analysis and derive a sufficient condition for compatibility in such a setting. Our theoretical results demonstrate that if we take early stopping iterates into consideration, generalization can hold with significantly weaker restrictions on the problem instance than the previous last-iterate analysis.


Review discussion:
I find the paper to be well-written and easy to follow. The overall topic of generalization of overparameterized models (even in the linear case) is important. The authors make a considerable effort to introduce existing bounds.
1. I find the study of overparameterized linear model *together with* early stopping less motivating.
In particular, early stopping for linear regression is equivalent to $\ell_2$ regularization. In this case, we can already apply uniform convergence bound for generalization error. In this case, is the motivation mainly to get a better bound? If so, I would suggest the authors move the comparison to stability bound into the main text. This is in contrast to the overfitting case since if we are overfitting, there is no uniform convergence bound at all.
2. One relevant work is missing, [1] (and its related citations) discussed cases when we have different kinds of overfitting.
In particular, their theorem 3.1 (b) corresponds to your example 4.1. I would love to see how your results are related to theirs.
[1] Benign, Tempered, or Catastrophic: A Taxonomy of Overfitting
See weakness.
N/A
No ethics review needed.
We thank the reviewer for the insightful and meaningful comments which would greatly help us revise our manuscript.
The reviewer's major concern comes from whether we can cover the results from the relationship between early-stopping and ridge regression.
Below, we show why our results cannot be covered intuitively and defer the technical details later.
Firstly, in their relationship, the regularization parameter of ridge regression is set to $1/t$, where $t$ denotes the early-stopping time [1].
In linear regression, such $t$ usually relates to sample size $n$ (e.g., $t = \sqrt{n}$), and therefore, with small regularization, its parameter norm may still be large and generalization bound may fail.
Secondly, the information bottleneck principle tells us that minimal steps may provide better results. By relating the early-stopping and ridge regression, the information is partially lost.
Therefore our bound would be tighter.
We calculate the generalization performance of ridge regression under the regularization $1/t$ and show that it cannot cover our results.
We ensure to add the detailed discussion in the revision.
Below we do our best to address the reviewer's concerns in detail.
>Q1: Early stopping for linear regression is equivalent to $\ell_2$ regularization. In this case, we can already apply uniform convergence bound for generalization error.
A1: We thank the reviewer for the constructive comment.
Indeed, there is a close relationship between early stopping and ridge regression.
For example, [1] proves that early stopping excess risk at time $t$ can be upper bounded by ridge regression excess risk with **regularization coefficient $1/t$**.
However, ***our results cannot be covered since the results in [1] do not apply for large time $t$ (leading to small regularization $\lambda = 1/t$)***.
For example, consider the case that eigenvalue decays as $\frac{1}{i^{1.5}}$. if we take $t=\Theta\left(\sqrt{n}\right)$ and $\lambda=\Theta\left(\frac{1}{\sqrt{n}}\right)$, it can be shown that for the ridge regression solution,  a norm-based generalization bound will give an excess risk of  $\Omega(1)$, which only provides a vacuous generalization bound of the early stopping solution. As a comparison, our results give consistent generalization bound for this case.
Furthermore, this generalization bound in this approach cannot cover ours, even if we use a more fine-grained analysis rather than uniform convergence. Following Theorem 1 in [2], we can calculate that the variance term is $\Omega(1)$ when $t=\Omega\left(\sqrt{n}\right)$, and the bound is vacuous for the above example.
Due to space limit, we are unable to post the proof here. We will add the calculation details as soon as possible in the discussion period.
>Q2: One relevant work is missing... their theorem 3.1 (b) corresponds to your example 4.1. I would love to see how your results are related to theirs.
A2: We thank the reviewer for the recommendation. We ensure to add a discussion and cite the related papers [3], [4], [5], [6],[7],[8],[9],[10], [11] in the next revision.
As pointed out by the reviewer, Theorem 3.1(b) in [3] shows that for feature covariance $\lambda_i=1/i^\alpha$ with $\alpha>1$, the excess risk will converge to $(\alpha-1)\sigma^2>0$, and falls into the **tempered**  taxonomy as in [3]. As a comparison, we prove that with early stopping, the model can achieve $o(1)$ excess risk for these data distributions, and falls into the **benign** taxonomy as in [3]. In this sense, our results are consistent with and extend the result in [3].
Again, we would like to express our great gratitude to the reviewer for their thoughtful comments. We are eager to provide any additional clarification that may be needed to help the evaluation.
[1] Ali A, Kolter J Z, Tibshirani R J. A continuous-time view of early stopping for least squares regression[C]//The 22nd international conference on artificial intelligence and statistics. PMLR, 2019: 1370-1378.
[2] Tsigler A, Bartlett P L. Benign overfitting in ridge regression[J]. J. Mach. Learn. Res., 2023, 24: 123:1-123:76.
[3] Mallinar N, Simon J B, Abedsoltan A, et al. Benign, tempered, or catastrophic: A taxonomy of overfitting[J]. arXiv preprint arXiv:2207.06569, 2022.
[4] Carrell A, Mallinar N, Lucas J, et al. The calibration generalization gap[J]. arXiv preprint arXiv:2210.01964, 2022.
[5] Simon J B, Dickens M, Karkada D, et al. The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networks[J]. arXiv preprint arXiv:2110.03922, 2021.
[6] Liu C, Abedsoltan A, Belkin M. On Emergence of Clean-Priority Learning in Early Stopped Neural Networks[J]. arXiv preprint arXiv:2306.02533, 2023.
[7] Emami M, Sahraee-Ardakan M, Pandit P, et al. Generalization error of generalized linear models in high dimensions[C]//International Conference on Machine Learning. PMLR, 2020: 2892-2901.
[8] Beaglehole D, Belkin M, Pandit P. Kernel Ridgeless Regression is Inconsistent for Low Dimensions[J]. arXiv preprint arXiv:2205.13525, 2022.
[9] Sahraee-Ardakan M, Emami M, Pandit P, et al. Kernel methods and multi-layer perceptrons learn linear models in high dimensions[J]. arXiv preprint arXiv:2201.08082, 2022.
[10] Nakkiran P, Kaplun G, Bansal Y, et al. Deep double descent: Where bigger models and more data hurt[J]. Journal of Statistical Mechanics: Theory and Experiment, 2021, 2021(12): 124003.
[11] Belkin M. Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation[J]. Acta Numerica, 2021, 30: 203-248.
[12] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. Adaptive computation and machine learning. MIT Press, 2012.
## Uniform Convergence on Ridge Regression
We next show that ridge regression with regularization $1/t$ and uniform convergence cannot cover our results.
We consider a case with data covariance $\lambda_i=\frac{1}{i^{1.5}}$, and show that ridge regression would return a bound with compatibility region shorter than $(0, \sqrt{n})$, which cannot cover our results with compatibility region $(0,n)$.
The uniform convergence bound scales like
$O\left(\frac{||\theta_\lambda||}{\sqrt{n}}\right)$ (see, e.g., Theorem 5.10 in [12]), where $\theta_\lambda$ denote the ERM solution of ridge regression with regularization $\lambda$. We calculate the excess risk of $\theta_\lambda$ as follows. The key insight comes from the fact that the noise matters a lot in the parameter norm.
$$\mathbb{E}\left[||\theta_\lambda||^2\right]=\mathbb{E}\left[||X^\top (XX^\top+\lambda I)^{-1}Y||^2\right]=\mathbb{E}\left[||X^\top (XX^\top+\lambda I)^{-1}(X\theta^*+\varepsilon)||^2\right]
=\mathbb{E}\left[||X^\top (XX^\top+\lambda I)^{-1}(X\theta^*)||^2\right]+\mathbb{E}\left[||X^\top (XX^\top+\lambda I)^{-1}\varepsilon||^2\right]\ge \mathbb{E}\left[||X^\top (XX^\top+\lambda I)^{-1}\varepsilon||^2\right]
=\mathbb{E}\text{Tr}\left[X^\top (XX^\top+\lambda I)^{-1}\varepsilon\varepsilon^\top (XX^\top+\lambda I)^{-1}X\right]\ge \sigma_\varepsilon^2 \mathbb{E}\text{Tr}\left[XX^\top (XX^\top+\lambda I)^{-2} \right]
=\sigma_\varepsilon^2 \sum_{i=1}^n \frac{\mu_i}{(\mu_i+\lambda)^2},$$
where we assume the noise vector is conditionally unbiased and has a lower bounded variance, as in Theorem B.2.
We next assumes that  $t = 1/\lambda > \sqrt{n}$.
Then, according to Lemma A.1 and B.2 in our paper, it holds that $ \mu_n=\Theta(\frac{1}{\sqrt{n}})=\Omega(\lambda)$ for $\lambda_i=i^{\frac{-3}{2}}$, where $k_0$ denotes the effective rank in the paper.
Therefore, it holds that
$$\sigma_\varepsilon^2 \sum_{i=1}^n \frac{\mu_i}{(\mu_i+\lambda)^2}=\Omega\left(\sigma_\varepsilon^2 \sum_{i=1}^n \frac{1}{\mu_i}\right) = \Omega(\sqrt{n}),$$
which leads to a vacuous generalization bound in uniform convergence.
Therefore, the related compatibility region must be shorter than $(0, \sqrt{n})$.
However, in this case, our results show a compatibility region $(0,n)$, which is better.
## Benign Overfitting in Ridge Regression
We additionally show that even if we apply benign overfitting in ridge regression, the returned compatibility region still cannot cover our results.
To apply Theorem 1 in [2], we first calculate the effective rank $k^*$. Note that by plugging in $\lambda_i=\frac{1}{i^{1.5}}$ and $\lambda = 1/\sqrt{n}$,
r_k=\frac{1}{\lambda_{k+1}}\left(\lambda+\sum_{i>k} \lambda_i\right)=\Theta\left(\frac{k^{\frac{3}{2}}}{\sqrt{n}}+k\right),
which increase monotonically with $r_n = \Theta(n)$ and therefore
k^*=\min \lbrace\kappa: \rho_\kappa>b\rbrace=\min \lbrace\kappa: r_\kappa>b n\rbrace=\Theta(n)
According to Lemma 11 in [2], we have $\bar{k}=\Theta(n)$, $k=\min\lbrace k^*,\bar{k}\rbrace=\Theta(n)$ and the first term in the variance part $\frac{k}{n}=\Omega(1)$, which makes the bound vacuous for this setting.
Thanks for the clarification, my concerns have been addressed and I'm happy to bump up my score.
We thank the reviewer for the kind reply! We ensure to incorporate the discussions in the next revision, and remain eager to engage in further clarifications to assist the evaluation.
The main strength of the paper is the generality of the data distributions considered.
Specifically, most prior theory work assumes we have isotropic data or some fixed transform of isotropic data.
However, this paper considers general anisotropic data. In this setup, they provide mild conditions under which we see that early stopping is beneficial for generalization.
The paper also provides clear evidence of the benefits of early stopping. Showing that the min norm solution is not necessarily the optimal solution for generalization.
Further, the paper shows that this is true even in the large data limit when the min norm solution is benign.
The are two main weaknesses (one major one minor)
1) The paper is missing the comparison to Madhu S. Advani, Andrew M. Saxe, and Haim Sompolinsky. High-dimensional Dynamics of
Generalization Error in Neural Networks. Neural Networks, 2020
While the data assumptions in the above paper are more restrictive, that is isotropic data. The paper exactly characterizes the risk curve as a function of time and determines the optimal stopping time.
Hence a proper discussion of this is crucial.
3) While the paper is well written, the paper sweeps technical details under the rug that would be important to know.
For example, the constants $c_0$ and $c_1$ are opaque and play an important role. See my questions for more instances.
In Assumption 1 subpoint 2, should it $\lambda_p > 0$?
Instead of $\lambda_1 > 0$.
Are the Cs on lines 169 and 173 supposed to be the same?
Why is $\boldsymbol{Y}$ called the noise vector?
In Equations (4), (5), how do we know such an $\ell$ exists?
What is $c$ in the statement of Theorem 5.1 in $k_0 \le n/c$?
N/A
No ethics review needed.
We would like to express our sincere gratitude to the reviewer for their valuable feedback and insightful comments.
We apologize for not making everything clear and ensure to revise our manuscript according to the reviewer's suggestions.
Below we do our best to address the reviewer's concerns.
>Q1: The paper is missing the comparison to [1].
A1: We thank the reviewer for the recommendation. We ensure to add the discussion with [1] as well as its related papers [2], [3] in the revision.
As the reviewer points out, the major difference between our paper and [1] lies in the different assumptions on the data. [1] assumes that the input data, data noise, and teacher model weights all follow i.i.d. isotropic Gaussian distribution.
In particular, the data covariance in this setting is an identity matrix.
In comparison, our results only require bounded ground truth weights and subGaussian data and noise, which is milder.
>Q2: In Assumption 1 subpoint 2, should it $\lambda_p>0$ Instead of $\lambda_1>0$?
A2: We apologize for the confusion. Here we indeed assume that $\lambda_1 > 0$ to rule out the all-zero input cases. This holds trivially in practice but we still express it explicitly to make the paper complete. Our results hold even when $\lambda_p=0$, which is a degenerate case and can be transferred to another linear regression problem with lower dimensions.
>Q3: Are the Cs on lines 169 and 173 supposed to be the same?
A3: It can be treated as the same absolute constant that is large enough. The use of a unified $C$ is for notation simplicity.
For example, if we have $\sum_{i>0}\lambda_i< C_1, ||\theta^*||<C_2$, we can take $C$ as the maximum of $C_1$ and $C_2$. Such tricks are because we want to put more focus on the rate of dimension and sample size, as most relevant papers do.
>Q4: Why is $Y$ called the noise vector?
A4: We apologize for the typo. We will change it to the response vector in the revision.
>Q5: In Equations (4), (5), how do we know such an $l$ exists?
A5: Thanks for the good catch. We take $l=\infty$ if a finite $l$ does not exist. This would lead to a vacuous generalization bound. We will make it clearer in the next version.
>Q6: What is $c$ in the statement of Theorem 5.1 in $k0≤n/c$?
A6: $c$ is a large enough constant independent of time $t$, dimension $p$, sample size $n$.
Once again, we thank the reviewer for the time and feedback, which will undoubtedly help us improve the quality of our work. We remain committed to improving the clarity and rigor of our paper and welcome any further feedback.
[1] Advani M S, Saxe A M, Sompolinsky H. High-dimensional dynamics of generalization error in neural networks[J]. Neural Networks, 2020, 132: 428-446.
[2] Saxe A M, Bansal Y, Dapello J, et al. On the information bottleneck theory of deep learning[J]. Journal of Statistical Mechanics: Theory and Experiment, 2019, 2019(12): 124020.
[3] Goldt S, Advani M, Saxe A M, et al. Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup[J]. Advances in neural information processing systems, 2019, 32.
Thank you for the clarifications, but I am still confused about the cs.
The Cs on line 169 and 173 now make sense to me. I am still unsure about the ones in equations 4 and 5 and the one for Theorem 5.1. Clarification would be appreciated.
We apologize for any lack of clarity and would like to offer further clarifications below.
The constant $c, c_0, c_1$ are mainly served to make the statements more rigorously. One can replace the constants with $\mathcal{O}(\cdot)$ notations to make it more clear.
For example, the statements regarding $c$ in Theorem 5.1 is equivalent to $k_0 =\mathcal{O}(n), \log \frac{1}{\delta} =\mathcal{O}(n), \lambda =\mathcal{O} \left(\frac{1}{\sum_{i>0} \lambda_i}\right)$. In this equation, the first condition constrain the regularity of the problem class, the second condition lower bounds the error probability, and the third condition upper bounds the learning rate, which are all standard in related literatures. The constants $c_0,c_1$ can be interpreted similarly.
Constants  in our paper do not depend on input dimensions ($p$) or sample sizes ($n$) or input covariance ($\Sigma$). These constants may only be associated with other terms assumed to be of order O(1) in the paper, such as the subGaussian parameter ($\sigma_x$). This simplification aims to symplify our results and shift the focus towards generalization bounds concerning input dimensions and sample sizes. This technique is a common practice in the related literature [1, 2].
Therefore, one can select a suitably set of constants (without dependence on input dimensions or sample sizes) for the main Theorems to hold. Difference choices of constants may yield different generalization bounds, but these bounds all share the same rate with regard to sample size $n$, as established in Theorem 5.1 using the $\lesssim$ notation.
We again express our gratitude to the reviewer for the feedback. We will make a clear explanation on the constants in the next revision. We are eager to engage in any further discussions to help the evaluation. Thank you!
[1] Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. CoRR, abs/1906.11300, 2019.
[2] Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, and Sham M. Kakade. Benign overfitting of constant-stepsize SGD for linear regression.
* The organization and presentation of this paper is smooth and clear, and it provides a better understanding on the benign overfitting phenomenon for early stopping type gradient descent.
* The theory in this paper is solid.
Although it mainly follows from the previous two works *Bartlett et al.* and *Zou et al.*, the proposed notion 'effective dimensions' for the feature covariance matrix in this paper provides an alternative way to analyze the variance term in the Bias–variance decomposition.
There is no major weakness for me.
A minior issue:
For the examples and the numerical experiments in this paper, it would be better if the authors could provide the patterns to show the interplay between the bias term $B(\theta_t)$ and the variance term $V(\theta_t)$ during training.
Especially when comparing with the bounds in previous works, it would be more intuitive to see why the compatibility happens but the benign overfitting does not, if the comparisons  of the evolutions of $B(\theta_t)$ and $V(\theta_t)$ are shown separately.
* In table 2, for the distributions that $\lambda_i = \frac{i}{i \log^2(i+1)}$ and $\lambda_i = \frac{i}{i \log^3(i+1)}$, both cases are benign overfitting based on *Bartlett et al*, but the min-norm excess risk seems to be very large though, is this caused by the bias term or the variance term?
* I am curious what the generalization bound for early stopping will be like if following the notations in *Theorem 4. [Bartlett et al.]* in terms of $r_0(\Sigma), k^*, R_{k^*}(\Sigma)$?
I am trying to understand the difference between methods using the proposed effective dimension in this paper and using the effective ranks in *Bartlett et al.*, because understanding the differences may help to motivates other more refined controls for the generalization analysis.
NA
No ethics review needed.
We appreciate the reviewer's supportive and insightful comments.
We ensure to add the related discussion in the revision. Below, we do our best to address the reviewer's questions adequately.
>Q1: For the examples and the numerical experiments in this paper, it would be better if the authors could provide the patterns to show the interplay between the bias term $B(\theta_t)$ and the variance term $V(\theta_t)$ during training.
A1: Thanks for the constructive suggestion. In the experiment part, the bias term $B(\theta_t)$ decays rapidly, and the model achieves small excess risk in the initial phase of training (as in our Figure 1).
Meanwhile, the variance term $V(\theta_t)$ increases steadily in the training process, and it takes a long time for the variance term as well as the excess risk to converge to a plateau.
We also provide the following table to validate the above pattern theoretically.
The table shows different bounds on bias and variance in different methods.
We will add the discussion in the next revision.
| Reference | Setting | Bias | Variance |
|----------|----------|----------| ----------|
|This paper|gradient descent  | $\frac{1}{t}+\frac{1}{\sqrt{n}}$ |$\frac{k_1}{n}+\frac{t^2}{n^2}$|
| [1]  |  overfitting solution  | $\frac{k_0}{n}$    | $\frac{n}{R_{k_0}(\boldsymbol{\Sigma})}$   |
| [2] | one-pass stochastic gradient descent | $\frac{k_1}{n}$   | $ \frac{n \sum_{i>k_1} \lambda_i^2}{\left(\sum_{i>0} \lambda_i\right)^2}$    |
>Q2:  In table 2, for the distributions that $\lambda_i=\frac{1}{log^2⁡(i+1)}$ and $\lambda_i=\frac{1}{log^3(i+1)}$, both cases are benign overfitting based on Bartlett et al, but the min-norm excess risk seems to be very large though, is this caused by the bias term or the variance term?
A2: We thank the reviewer for the insightful question! Benign overfitting means that the excess risk goes to zero as the sample size $n$ goes to zero. For the practical cases, it may be numerically large for a fixed sample size $n$ (may be due to a slow convergence rate with respect to $n$).
As for the cause of the large excess risk, we believe that it comes from the variance term since the bias term is usually easy to learn in linear regression. We will add the discussion in the next version.
Although the exact number cannot imply benign overfitting, we can still compare the early-stopping regimes and the interpolation regimes. This is what the experiments want to show in this paper.
>Q3: I am trying to understand the difference between methods using the proposed effective dimension in this paper and using the effective ranks in [1].
A3: We thank the reviewer for the insightful question. The differences come from two folds: (a) our proposed effective dimensions is time-dependent ($k_2$) while those in [1] do not need to focus on the time; and (b) due to the time effect, we can split the data covariance with more fine-grained analysis.
Therefore, the term $k_0/n$ proposed in [1] can be improved to $k_1/n$ in our paper (where $k_1$ is provably smaller than $k_0$). We ensure to add the comparison in the revision.
Once again, we appreciate the reviewer's feedback and hope that our response address the concerns. We remain committed to improving the clarity and rigor of our paper and welcome any further feedback.
[1] Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. CoRR, abs/1906.11300, 2019.
[2] Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, and Sham M. Kakade. Benign overfitting of constant-stepsize SGD for linear regression.
I thank the authors for the rebuttal. Having read the rebuttal and other reviews I decided to keep my score.
We thank the reviewer for the supportive feedback. We will remain eager to engage in any further clarifications if needed.
Theoretical proofs are given, along with bounds comparisons, and empirical results. The paper seems well-written and rigorous in its assessment. The problem is well motivated and extremely well-referenced in its citations and descriptions of the current work in this area. The ability and degree to which over-parameterized models can generalize is one of the main mysteries of deep learning, and where traditional theory in generalization breaks down, so the paper is extremely relevant.
This paper requires substantial knowledge of generalization theory to understand and is not accessible to a broader audience.
For example, the introduction makes the assumption that terms like excess risk are understood by the reader, before they are later defined.
While the theory is claimed to be general, thorough treatment is only given for linear regression, and derivations of sufficiency conditions and bounds for other problems are left as future work.
While deep learning was given as a primary motivation to understand generalization for over-parameterized models, application to these models was not addressed in the paper. The experimental and conclusions sections were very minimal. I understand that the theory was the primary focus of the paper, but would have liked to see more practical demonstration and discussion of implications for the theoretical results that can be drawn from your investigations.
Are there any roadblocks that would make application of this difficult for other models, such as non-linear regression problems including deep neural networks? Are there larger or broader implications of data-algorithm compatibility and your results that you can discuss?
Societal impact does not seem relevant or needed to address for this paper. The paper acknowledges a limited scope to linear cases, but does not go into any additional weaknesses.
No ethics review needed.
We would like to express our sincere gratitude to the reviewer for the valuable feedback and supportive comments.
The reviewer's major concern comes from the linear example.
Indeed, we only derive a linear example to express the effectiveness of compatibility.
However, the idea behind is general which would bring some interesting insights into the generalization community.
Below we do our best to address the reviewer's concerns.
>Q1: Are there any roadblocks that would make application of this difficult for other models, such as non-linear regression problems including deep neural networks? Are there larger or broader implications of data-algorithm compatibility and your results that you can discuss?
A1: We thank the reviewer for the insightful and constructive comments. As the reviewer points out, the proposed data-algorithm compatibility is a general concept which can be deployed in deep learning, while the given example mainly focuses on the overparameterized linear regression.
Studying a linear example is widely adopted in generalization analysis to understand deep learning, since (a) existing theories on neural tangent kernel [1,2] and random feature model [3,4] (both are theoretical approximations of neural networks) draw a rigorous connection between them, and (b) their generalization performances has some inherent relationships and common patterns, as indicated by our experiments.
Due to technical limitations, we cannot directly generalize to neural networks now, but we believe that the concepts in this paper still provide some interesting insights for the generalization community.
>Q2: This paper requires substantial knowledge of generalization theory to understand and is not accessible to a broader audience. [...]  I understand that the theory was the primary focus of the paper, but would have liked to see more practical demonstration and discussion of implications for the theoretical results that can be drawn from your investigations.
A2: We thank the reviewer for the suggestions! We will reorganize the arguments and revise the paper accordingly to make it easier to understand. As an example, we will provide a definition of excess risk in the introduction, and add the discussion on the broader impact. Thank you!
Once again, we express our gratitude to the reviewer for the valuable feedback, which will undoubtedly contribute to enhancing the quality of our work. We remain committed to improving the clarity and rigor of our paper and welcome any further feedback.
[1] Jacot A, Gabriel F, Hongler C. Neural tangent kernel: Convergence and generalization in neural networks[J]. Advances in neural information processing systems, 2018, 31.
[2] Arora S, Du S S, Hu W, et al. On exact computation with an infinitely wide neural net[J]. Advances in neural information processing systems, 2019, 32.
[3] Rudi A, Rosasco L. Generalization properties of learning with random features[J]. Advances in neural information processing systems, 2017, 30.
[4] Mei S, Montanari A. The generalization error of random features regression: Precise asymptotics and the double descent curve[J]. Communications on Pure and Applied Mathematics, 2022, 75(4): 667-766.
I have read the rebuttal presented by the authors, and appreciate their clarifications. I look forward to seeing the reorganization and improvements in clarity in the next revision.
We again thank the reviewer for the precious time and valuable feedback, and make sure to revise the current manuscript to improve its clarity. We remain eager to engage in any further clarifications to help the evaluation.

Meta-review:
role: PC
data: comment: value: The submitted paper proposes a new mathematical framework called "compatibility" for analyzing the generalization error of over-parameterized machine learning models, specifically focusing on over-parameterized linear regression using gradient descent with early stopping. The reviewers generally acknowledge the significance of the paper's contributions. Namely, its theoretical contribution, clarity of presentation, and relevance towards the audience.
Nevertheless, while the theoretical insights are appreciated, some reviewers did express the desire for a more practical demonstration. That, together with a discussions of implications for real-world deep learning models, which were mentioned as one particular motivation, can significantly improve the paper.

Please consider reviewer's suggestions, including missing refs, and address those in the final revision.

Decision: Accept (poster)