Title: Optimal Rates in Continual Linear Regression via Increasing Regularization
Venue: NeurIPS 2025
Accepted: yes
Keywords: continual learning

lifelong learning

last iterate

forgetting

task ordering

regularization
Abstract: We study realizable continual linear regression under random task orderings, a common setting for developing continual learning theory.
In this setup, the worst-case expected loss after $k$ learning iterations admits a lower bound of $\Omega(1/k)$. 
However, prior work using an unregularized scheme has only established an upper bound of $O(1/k^{1/4})$, leaving a significant gap.
Our paper proves that this gap can be narrowed, or even closed, using two frequently used regularization schemes:
(1) explicit isotropic $\ell_2$ regularization, and (2) implicit regularization via finite step budgets.
We show that these approaches, which are used in practice to mitigate forgetting, reduce to stochastic gradient descent (SGD) on carefully defined surrogate losses. 
Through this lens, we identify a fixed regularization strength that yields a near-optimal rate of $O(\log k / k)$.
Formalizing and analyzing a generalized variant of SGD for time-varying functions, we derive an increasing regularization strength schedule that provably achieves an optimal rate of $O(1/k)$.
This suggests that schedules that increase the regularization coefficient or decrease the number of steps per task are beneficial, at least in the worst case.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper derives two convergence rates for continual linear regression with regularization: for fixed regularization, they achieve $O(log k/k)$ convergence rate, whereas for scheduled regularization they achieve $O(1/k)$. As $O(1/k)$ is the optimal rate one can achieve for continual learning, they have achieved optimal convergence rate for continual learning for the first time. The proof is followed by first understanding the problem as Incremental Gradient Descent, and then applying the bound in [1].\n\n[1] Evron, Itay, et al. \"Better rates for random task orderings in continual linear models.\" arXiv preprint arXiv:2504.04579 (2025).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: A novel scheme that has optimal convergence rate is very valuable. The fact that continual learning can be understood as incremental gradient descent is also important I believe, as it enables continual learning to be studied by tools from SGD. They also consider many diverse settings, such as considering seen-task loss, different regularizations, etc.\n\nWeaknesses: One weakness I find is that there is no experimental results. I understand that this is a purely theoretical paper: of course showing that increasing regularization strength for isotropic regularization would have been ideal, but at least showing that the lower bounds / upper bounds make sense for certain continual linear regression problems could have been good.\n\nQUESTIONS:\nA general question I have is \"is joint realizability a reasonable assumption to study continual learning?\" I get that for continual learning, both learning from new tasks and remembering previous tasks are important. To me joint realizability feels like the framework doesn't have the notion of forgetting - because naively trying to do better for every given task naturally leads to better performance for all past tasks. Are there different setups in continual learning theory that can reflect such issues? Is your theory applicable in those cases too?\n\nLIMITATIONS:\nOne limitation is that the result is purely theoretical.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of realizable continual linear regression under random task orderings, a setting that is important for understanding and advancing continual learning. Previous work has established only suboptimal learning rates, achieving an upper bound of \\$O(1/k^{1/4})\\$ despite the existence of a lower bound of \\$\\Omega(1/k)\\$. The authors show that incorporating regularization—either explicit L2 regularization or implicit regularization via a finite number of gradient steps—significantly improves convergence. By reducing these schemes to stochastic gradient descent (SGD) on appropriately defined surrogate losses, they demonstrate that a fixed regularization strength leads to a near-optimal rate of \\$O(\\log k / k)\\$, while an increasing regularization schedule achieves the optimal rate of \\$O(1/k)\\$. These results close the theoretical gap and highlight the practical and theoretical importance of tuning regularization in continual learning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper tackles an important and fundamental problem in continual learning—the learning bounds of continual linear regression.\n\n2. The writing is clear, well-structured, and easy to follow.\n\n3. The theoretical results appear sound and provide tighter bounds compared to existing work.\n\n**Weaknesses:**\n\n1. The primary concern lies in the parameter setting of the algorithm. The data radius \\$R\\$ must be known in advance to determine the regularization coefficient. Is this assumption realistic in practical scenarios?\n\n2. In Figure 1, the loss and forgetting rate are combined into a single quantity. However, continual learning is typically evaluated using two separate metrics: average accuracy across all tasks and forgetting rate on previous tasks. Why do the authors merge these two metrics into one?\n\n3. Could the authors include synthetic experiments to further validate the effectiveness of the proposed method?\n\nQUESTIONS:\nSee Weaknesses above\n\nLIMITATIONS:\nSee Weaknesses above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies continual linear regression under random task orderings, wherein the learner faces a collection of randomly-ordered linear regression tasks and seeks to minimize the average error across them. A $\\log{k}/k$ last-iterate convergence rate is derived using a fixed regularizer, and the result is improved to a $1/k$ rate using increasing regularization; both results significantly improve the best-known rate of $1/k^{1/4}$. A lower bound of $1/k$ is also shown, demonstrating that the $1/k$ rate is optimal.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nThere are two common approaches to the problem studied: applying explicit regularization on each step (to penalize moving too far from the current iterate, thus \"forgetting\" much about previous tasks), and implicitly regularizing by applying only a limited amount of updates in each task. Section three shows that both of these options can be analyzed in terms of SGD on a particular surrogate losses, leading to a more unified analysis. Very clean.\n\n## Weaknesses\n\n- The lower bound is not particularly novel; it is standard but technically hasn't been explicitly stated in the literature (the paper is up front about this)\n\n- The writing and paper structure is not very polished. Generally the writing comes across as rushed.\n  -  e.g. Section 2 introduces the setting but only partially, with key assumptions (realizability and random task ordering) not being introduced until later, in Section 4.\n  - Section 4 introduces the 1/k lower bound, and then follows up with an algorithm which fails to achieve it, only to fix it with a separate algorithm afterwards. It seems redundant to present the sub-optimal algorithm if an optimal one is obtained at seemingly no compromise.\n  - Lemma 4.7 re-introduces all the assumptions in the lemma statement. This one lemma take up more than a third of a page\n\nQUESTIONS:\n- The comment before Lemma 4.9 was unclear to me; how can we simultaneously have a 1/k lower bound and non-isotropic regularization ensuring seen-task loss of zero? The elaboration of this comment in Section 6 did not resolve this question for me either.\n\n- I don't really understand the point of presenting both the fixed and increasing regularization scheme; the former seems to be strictly worse, so why not just present the optimal algorithm? It feels like this could have been included in the appendix rather than being featured prominently in the main text.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper proves new worst-case bounds on linear regression problems in a continual learning setting, specifically focusing on the regularized case.

I should note that this paper is far out of my field of expertise, and thus my assessment is just an educated guess. I hope the other reviewers are more qualified to review the paper.

STRENGTHS AND WEAKNESSES:
Strengths:
- Clarity: The paper is dense, but to me it seemed clearly written.
- Originality: Analyzing learning dynamics under regularization seems like an important topic to be covering, and if this has indeed not been covered in previous work then I think it is worth examining.

Weaknesses:
- Significance: I mostly work in more applied varieties of machine learning, and to me it was not very clear what the significance of this work would be. (Somewhat facetiously but also somewhat not,) we are approaching AGI and it's not clear that knowing this tighter bound will get us closer to that (although purchasing more GPUs likely will). It would be nice to have a little more grounding in what the practical implications of the work would be, and how they might eventually lead into new optimization algorithms, etc. That being said, this is not the variety of paper I typically read in the first place, so it might be that this is much more interesting to people for whom this is a central research interest.

QUESTIONS:
- As someone working on other varieties of ML, such as large language models, what is the significance of this work?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03