Title: Improved Scaling Laws in Linear Regression via Data Reuse
Venue: NeurIPS 2025
Accepted: yes
Keywords: Scaling laws

linear regression

multi-pass SGD

data reuse
Abstract: Neural scaling laws suggest that the test error of large language models trained online decreases polynomially as the model size and data size increase. However, such scaling can be unsustainable when running out of new data. In this work, we show that data reuse can improve existing scaling laws in linear regression. Specifically, we derive sharp test error bounds on $M$-dimensional linear models trained by multi-pass *stochastic gradient descent* (multi-pass SGD) on $N$ data with sketched features. Assuming that the data covariance has a power-law spectrum of degree $a$, and that the true parameter follows a prior with an aligned power-law spectrum of degree $b-a$ (with $a > b > 1$), we show that multi-pass SGD achieves a test error of $\Theta(M^{1-b} + L^{(1-b)/a})$, where $L \lesssim N^{a/b}$ is the number of iterations. In the same setting, one-pass SGD only attains a test error of $\Theta(M^{1-b} + N^{(1-b)/a})$ (see, e.g., Lin et al., 2024). This suggests an improved scaling law via data reuse (i.e., choosing $L>N$) in data-constrained regimes. Numerical simulations are also provided to verify our theoretical findings.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors study the behavior of multi-pass SGD on task of linear regression with sketching. They \nassume that the inputs follow the Gaussian distribution $N(0, H)$ with the eigenvalues of $H$ following a power law with \nexponent $a$, the ground truth direction is $w^\\*$, and covariance matrix of the prior on $w^*$ share the eigenvectors \nwith $H$ and the eigenvalues follow a power law with exponent $b$. Under these assumptions, they show that the test \nloss of multi-pass SGD satisfies the scaling law $M^{1-b} + L^{(1-b)/a}$ with $M$ being the sketching dimension and \n$L \\lesssim N^{a/b}$ the number of steps, while the scaling law of single-pass SGD is $M^{1-b} + N^{(1-b)/a}$, where \n$N$ is the number of samples. Namely, multi-pass SGD is provably more sample-efficient than single-pass SGD (in this setting).\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths \n\n* This is a well-written paper and is easy to follow. The setting and results are clearly stated and described. \n* The analysis of multi-pass SGD is rather rare in this field, probably due to the additional dependence between iterations \n  introduced by the reuse of samples (most of the existing results about GD or online/single-pass SGD). It is nice to \n  have a result which proves that at least in certain scenarios, the reused samples behave similar to fresh samples and \n  multi-pass SGD can beat one-pass SGD in terms of sample complexity.\n* The authors also prove lower bounds on the errors, showing their bounds are not vacuous, and also verify their claims with \n  experiments. \n\n### Weaknesses\n\nMy main concern with this paper is that its contribution over [1-2] appears to be limited. \n\nWhen the prior of the ground truth is isotropic, we have $a = b$, and in this case, the results (cf. Corollary 3.3) in this \npaper do not give any improvement over the results in [1]. \n\nIn the regime where improvement can be obtained, the authors need to assume that the data distribution and the prior of \nthe ground truth share the same eigenvectors and the eigenvalues of the prior follow another power law, which is a rather \nstrong assumption (especially the first part). Moreover, the benefit of reusing samples in linear regression has been \nstudied in [2] and the results/observations are similar. To be fair, [2] does not analyze the dependence on the model \nsize, but that part is covered in [1].\n\n\n[1] Licong Lin, Jingfeng Wu, Sham M Kakade, Peter L Bartlett, and Jason D Lee. Scaling laws in linear regression: Compute, parameters, and data. 2024\n\n[2] Loucas Pillaud-Vivien, Alessandro Rudi, Francis Bach. Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passes. 2018\n\nQUESTIONS:\nSee the weakness section. In particular, I'd like to know the following. \n* Is it true that in the isotropic prior case, the bounds are essentially the same as the ones in Lin et al. (2024), or I am missing something here?\n* What will happen if, instead of assuming the data distribution and the prior share the same eigenvectors, we only assume \n  they are correlated?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies mulpi-pass SGD applied to linear regression. Under power-law spectrum assumptions of the features and the true model parameter, the authors prove scaling laws of the different components of the test error. The main contribution is extending the previous analysis to account for data reuse.\n\nSTRENGTHS AND WEAKNESSES:\n- Quality: the submission appears technically sound. The theoretical results are accompanied by experiments. However, some proofs are a bit hard to follow and I am not able to verify their correctness.\n- Clarity: the main paper is pretty clearly written. However, the proofs in the appendix sometimes are not that clear and detailed. E.g., from line 846-line 849, too many things are done at the same time. Also, nested lemmas (e.g., lemma D.2 and D.3 are stated in the proof of lemma D.1) makes the flow hard to follow. The clarity can be improved by providing a better high-level proof sketch of bounding the fluctuation error. (If I understand correctly, this is the key difference from the results in [1]).\n- Significance: The submission advances our understanding of scaling laws under data-reuse. While this is somewhat significant, the main limitation is that the analysis is limited to linear models. In addition, the Gaussian design matrix is a pretty strong assumption.\n- Originality: The submission extends previous results on scaling laws to multi-pass SGD. The proofs follow closely along the lines of [0].\n\nTypos:\n- Second line of equation after 521, missing a S?\n\n\n[0] Loucas Pillaud-Vivien, Alessandro Rudi, and Francis Bach. Statistical optimality of stochastic gradient descent on hard learning problems through multiple passes\n[1] Licong Lin, Jingfeng Wu, Sham M Kakade, Peter L Bartlett, and Jason D Lee. Scaling laws in linear regression: Compute, parameters, and data. arXiv preprint arXiv:2406.08466, 2024\n\nQUESTIONS:\n1. Could you provide a high-level overview of how the fluctuation error is bounded?\n2. Would the analysis/results extend to SGD with l2 regularization?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper derives test error bounds when training an M-dimensional linear model on N samples with sketched features using multi-pass SGD of $\\Theta(M^{1-b}+L^{(1-b)/a})$ for $L$ iterations, under the standard assumption on data covariance having a power-law spectrum of degree $a$, and the true parameter having a prior with an aligned power-law spectrum of degree $b-a$. This bound holds for $L\\leq N^{a/b}$ and improves upon the prior result of $\\Theta(M^{1-b}+N^{(1-b)/a})$, for one-pass SGD ($N<L$).\n\nSTRENGTHS AND WEAKNESSES:\nThis work makes an interesting and important contribution to the recent line of work on theory of scaling laws in simple settings. In particular, prior work considers one-pass SGD while this work considers multi-pass SGD, and shows that data reuse can be helpful to some extent in improving test performance.\n\nThe paper is very well-written overall. As someone who has not read other papers on provable scaling laws for simple settings, the paper contains sufficient background information, the authors thoroughly and clearly contrast their contributions, and (parts of) the proof technique with other related works, which was helpful, and discuss well the implications of their result.\n\nQUESTIONS:\nSome stylistic suggestions to further improve readability:\n\n- The notation paragraph can be moved to Section 2.\n- In Corollary 3.3, the part after ‘In contrast’ should be part of the text and not the result statement, since it only discusses a result from prior work.\n- In Fig. 1, the last part in the caption (values of M and N for the subfigures) can be included in the respective subcaptions.\n\nIn Section 5, the proof overview can be improved by including some more details. The headings in line 258 and 259 seem unnecessary and this part can be made more concise. In the paragraph discussing the main technical difficulty, currently, the details are helpful to contrast the technique with prior work, but some of the statements, such as the one about Lemma D.3, and the one about a ‘novel lower bound’, are not very informative. It would be helpful to include some mathematical expressions for these to make the proof overview easier to follow.\n\nLIMITATIONS:\nThe paper does not have a limitations section. The authors can include a paragraph to discuss some limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

comments: SUMMARY:
PAPER SUMMARY:
This paper provides a theoretical analysis of the effect of multiple SGD passes on scaling laws for linear regression.

Neural scaling laws successfully predict test accuracy as a function of model and data size. Empirically, they proved successful and in the past few years several theoretical explantations of this phenomenon have been proposed.
Recently, the theoretical work of Lin et al., (2024) characterized scaling laws of one-pass SGD for linear regression.
Empirically,  Muennighoff et al. (2023) showed that scaling laws hold for multi-pass SGD as if new data was used, granted that the number of passes is small.

This work explains the latter phenomenon by refining the analysis of Lin et al., (2024) to include multiple passes. In particular, this work retrieves the bounds of Lin et al., (2024) for one pass, and shows that multiple passes behave as if fresh data was used, as long as the number of passes is small enough.

Along with standard assumption in the analysis of GD for linear regression, this work's analysis assumes that the data covariance's and parameter's prior eigenspaces are aligned and both spectra follow a power law, as in Lin et al., (2024).

STRENGTHS AND WEAKNESSES:
Despite not being an expert in this field, I believe that this work makes progress towards explaining neural scaling laws.
Indeed, this work explains an important phenomenon already observed empirically, extending and strengthening the illuminating result from Lin et al., (2024).

QUESTIONS:
No.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the scaling of test error under SGD on squared loss (regression) for linear models acting on sketched inputs. \nThe paper considers the setting of Gaussian inputs with a power-law spectrum and a specified source condition quantifying the alignment along the target. The main result shows that upon the re-use of data, i.e., under multi-pass SGD, the number of samples can be replaced by the number of iterations in the scaling law (for a sufficient number of samples). The result thus shows that re-using data provably reduces sample complexity over online SGD, or equivalently, leads to improved scaling laws.\n\nSTRENGTHS AND WEAKNESSES:\n#  Strengths\n- Understanding scaling laws is a timely question of great interest in the community.\n- The paper is well-written with clear statements of the assumptions and theorems. The risk decomposition in (3) and the proof sketch improve the comprehensibility of the results and the proofs.\n- The results are supported by numerical simulations.\n\n# Weaknesses\n\n- The setting is limited and to some extent, appears artificial. In particular, the introduction of \"sketching matrix\" requires further motivation beyond it allowing the introduction of a \"model size\" parameter. Furthermore, for linear models, it is unclear if the sketching dimension \"M\" captures the model \"size\" since the capacity of the model does not grow once the sketching matrix becomes full-rank. The paper further assumes that $M << d$, which precludes the study of overparameterized regimes.\n- The analysis is of limited novelty and largely builds upon existing works, in particular the work of Pillaud-Vivien et al. 2018.\n- (minor) missing related work. Although for different reasons, another recent line of work also demonstrates improved sample complexity (and consequently, improved scaling laws by data reuse) in the feature learning regime for two-layer neural networks: \n\n  Dandi, Y., Troiani, E., Arnaboldi, L., Pesce, L., Zdeborova, L., &; Krzakala, F. The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents. In Forty-first International Conference on Machine Learning.\n\n  Lee, J. D., Oko, K., Suzuki, T., &amp; Wu, D. (2024). Neural network learns low-dimensional polynomials with sgd near the information-theoretic limit. Advances in Neural Information Processing Systems, 37, 58716-58756.\n\n  Arnaboldi, L., Dandi, Y., Krzakala, F., Pesce, L., &amp; Stephan, L. (2024). Repetita iuvant: Data repetition allows sgd to learn high-dimensional multi-index functions. arXiv preprint arXiv:2405.15459.\n\nQUESTIONS:\n* Could you provide brief motivations for the introduction of the sketching matrix and the assumption $M<<d$?\n* Lines 151-153 appear unclear without looking into Koltchinskii and Lounici (2017). Could you provide a short, self-contained summary of the source of the condition on the initial step-size?\n* The parameter b is unspecified in the abstract and only defined later in the source condition.\n* What happens when $a<b$?\n* What are the central challenges introduced by the introduction of the sketching matrix while bounding the fluctuation error (lines 274-276)?\n* What motivates the choice of the geometrically-decaying step-size schedule?\n\nLIMITATIONS:\nThe work is primarily theoretical and does not pose direct societal risks. The paper would benefit from a more detailed discussion of the limitations of their model listed under \"weaknesses\".", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-07"}