Memory-Statistics Tradeoff in Continual Learning with Structural Regularization
Authors: Haoran Li
Jingfeng Wu
Vladimir Braverman
Venue: ICLR 2026 Poster (2026)
Keywords: continual learning
deep learning theory
Primary area: learning theory
TL;DR: We theoretically study the trade-off between the memory complexity and the statistical performance of regularization-based continual learning.

Abstract: We study the statistical performance of a continual learning problem with two linear regression tasks in a well-specified random design setting. We consider a structural regularization algorithm that incorporates a generalized $\ell_2$-regularization tailored to the Hessian of the previous task for mitigating catastrophic forgetting. We establish upper and lower bounds on the joint excess risk for this algorithm. Our analysis reveals a fundamental trade-off between memory complexity and statistical efficiency, where memory complexity is measured by the number of vectors needed to define the structural regularization. Specifically, increasing the number of vectors in structural regularization leads to a worse memory complexity but an improved excess risk, and vice versa. Furthermore, our theory suggests that naive continual learning without regularization suffers from catastrophic forgetting, while structural regularization mitigates this issue. Notably, structural regularization achieves comparable performance to joint training with access to both tasks simultaneously. These results highlight the critical role of curvature-aware regularization for continual learning.

Published as a conference paper at ICLR 2026
MEMORY-STATISTICS TRADEOFF IN CONTINUAL LEARNING WITH STRUCTURAL REGULARIZATION
Haoran Li
Jingfeng Wu
School of Artificial Intelligence Shenzhen University Shenzhen, Guangdong, China 1ihr8szu. edu. cn
Simons Institute UC Berkeley Berkeley, CA 94720, USA
wwajf@berkeley.co edu
Vladimir Braverman
Department of Computer Science Johns Hopkins University
Baltimore, MD 21218, USA vovašcs. , juu. edu
ABSTRACT
We study the statistical performance of continual learning problem with two lin- ear regression tasks well-specified random design setting. We consider a struc- tural regularization algorithm that incorporates a generalized ey-regularization tai- lored to the Hessian of the previous task for mitigating catastrophic forgetting. We establish upper and lower bounds on the joint excess risk for this algorithm. Our analysis reveals a fundamental trade-off between memory complexity and statis- tical efficiency, where memory complexity is measured by the number of vectors needed to define the structural regularization. Specifically, increasing the number of vectors in structural regularization leads worse memory complexity but an improved excess risk, and vice versa. Furthermore, our theory suggests that naive continual learning without regularization suffers from catastrophic forget- ting, while structural regularization mitigates this issue. Notably, structural regu- larization achieves comparable performance to joint training with access to both tasks simultaneously. These results highlight the critical role of curvature-awwe aware regularization for continual learning.
INTRODUCTION
Continual learning (CL) is a machine learning setting where multiple distinct tasks are presented to learning agent sequentially. As new tasks are encountered, it is expected that the agent learns both the old and the new tasks as the number of tasks increases. However, due to limited long- term memory, the CL agent cannot retain all past information. This makes CL significantly more challenging than single-task learning, as it cannot perform joint training all available samples (Parisi et al., 2019). On the other hand, without using exceedingly large long-term memory, we view CL problem data from each task. Unfortunately, the final model obtained via such an online approach could suffer from catastrophic forgetting (McCloskey & Cohen 1989 Goodfellow et al. (013), where performance on earlier tasks degrades after adapting to new ones due to loss knowledge from previous tasks.
online multi-task problem and sequentially fit CL model with the
Many works in CL focus on developing heuristic algorithms that record information about previous tasks to reduce forgetting in later tasks. One effective category of methods mitigates catastrophic for- getting by applying structural regularization (Kirkpatrick et al. 2017 2017:Aljundi et al. 2018 Chaudhry
al.
2018 Kolouri et al. 2020 Li al. 2021). From a geometric perspective (Chaudhry et al.)
2018 g
Li ct al. 2021). structural regularization methods store a PSD importance matrix that estimates
Work done while H.I.. Rice University.

---

Published as a conference paper at ICLR 2026
the significance of model parameters for previous tasks. When learning tasks, quadratic reg- ularizer based on this importance matrix is applied to prevent significant deviation of important parameters from the previous ones (Kirkpatrick al. 2017 Aljundi et al. 2018).
While various effective importance matrices have been proposed, practitioners have observed that the full importance matrix can require prohibitively large memory, scaling O (d²) for neural network with d parameters. To address this, several approximation strategies have been applied reduce the memory cost, including diagonal approximations (Kirkpatrick et al. 2017, Aljundi et al. 2018), K-FAC (Ritter et al.) 2018), and sketching (Li et al. 2023). Empirical results suggest that more accurate approximations, which require higher memory costs, often lead better CL performance (Ritter et al. 2018: Li et al. 2021).
In contrast to these practical ad advancements, the theoretical understanding of CL algorithms remains in its early stages. Several theoretical works have analyzed CL and particularly regularization- based CL methods on linear regression (e.g.. |Evron et al. (002)); [i et al. (2023); ZZhoo et al. (2024)); however, these studies focus on specific regularizers with fixed memory costs, and none explicitly link memory usage to the statistical performance of the CL algorithm. Additionally, existing works often impose strong assumptions settings or scenarios with fixed input data or transforms.
input data, limiting their forgetting analysis to optimization
Contributions. In this paper, we theoretically study the memory-statistics trade-off in continual learning algorithms with structural regularization within the linear regression setting. Specifically, we consider two-task linear regression under covariate shift (see Definitions [ and ( in random de- sign. We study the generalized "e-regularized CL algorithm (GRCL), where user-defined quadratic regularizer is applied during the second task, controlling the extent to which the model deviates from the learned first task (see Our contributions
follows:
We provide sharp risk bounds for the GRCL algorithm in the one-hot random design set- ting. These bounds reveal a provable trade-off between CL performance and memory cost, governed by the regularization matrix: well-designed regularizer enhances CL perfor- mance at the cost of more memory usage.
We show that without regularization, catastrophic forgetting occurs when there is a signifi- cant difference in small subset of dominant features in the one-hot setting. Conversely, by selecting an appropriate regularizer with higher memory usage, the GRCL algorithm can prevent catastrophic forgetting and achieve the error rate of joint learning.
We extend the risk bounds without regularization to the Gaussian distribution setting as a technical advancement. We show that, in addition to differences in dominant features, catastrophic forgetting can also occur in the Gaussian design when there exists a slight difference in the tail features.
The paper is organized as follows. Section 2 covers the most related works to our paper; the others are deferred to Appendix[A. [A] Section 3 we set up the theoretical setting of the 2-task CL problem. We present our main results and messages under this setting with supporting numerical experiments in Section 4 additional details of numerical and practical experiments are deferred to Appendix [E] main results and messages into broader settings including Gaussian design, multi-
We extend task CL. and general neural networks in the NTK regime in Section
5 Finally, we conclude our work
in Section 6 All proofs are deferred to Appendices B
C and D
RELATED WORKS
Over the past decade, a long list of practical CL methods has been proposed to address the catas- trophic forgetting problem of neural networks. These CL methods can be roughly divided into four categories, utilizing regularization, replay, architecture expansion and projection respectively. One refer to Parisi et al. (2019); Wang et al. (2024) Appendix A for comprehensive overview.
perforo-pe relationship beyond regularization. The relationship between memory and CL performance has also been observed in other CL paradigms. In replay-based methods, the performance of ER (Chaudhry al. 2019) rises with the size of the episodic memory. Similarly, in
2

---

Published as a conference paper at ICLR 2026
projection- based approaches like GPM (Saha et al. 2021). the rank of the projection subspaces di- rectly influences CL accuracy. While theoretical analysis of the memory-performann relationship in these settings is exciting open direction, it is beyond the scope of this work.
CL theory. Several theoretical works have emerged recently on understanding forgetting in CL.. We discuss those immediately related to our paper; the others are deferred to Appendix
Evron et al. (2022) analyzes the ordinary continual learning in the setting with fixed input data and labels, limiting their results only the optimization aspects of CL. The work by Li al. (2023) considers the 62-regularized CL performance on the two-task fixed design setting; the work by Zhao al. (2024) also considers the fixed-design setting and evaluates the statistical performance with GRCL on the multi-task linear regression. In comparison, we consider the GRCL in the random de- sign, which the impact of input data randomness is taken into account. Moreover, all these works consider specific regularizers with fixed memory costs, while our work is the first to theoretically reveal the memory-stane statistics relation of CL.
PROBLEM SETUP
Linear regression under covariate shift. We set up 2-task CL setting with two linear regression problems our results can be extended CL. with more than two tasks without additional technical hurdles. We use E H and y E IR to denote an input data vector in Hilbert space (with a finite or a countably infinite dimension d S 20 ) and a label variable, respectively. Consider two data distributions D(1) and D In the problem of continual learning, we are first given pairs of data vectors and label variables independently sampled from the first task distribution, and then another n data that are drawn independently from the second distribution, which is denoted by
(1) Di D(Y, (z! M4
(2) (x. .y )z D(22)
For simplicity, the anfinite-dimensione matrix notation for the dataset: ( denotes the linear map from to R² corresponding to æ) ...., (4) E for tasks 1, 1,2. consider
CL problem under the covariate shift setting, defined by Definition I and
Definition (Covariance conditions). Assume that all entries and the trace of the data covariance matrices of both tasks are finite. Denote the data covariance matrices of the two tasks by
G = Ep: [x J,
H == Epray (xx
and denote their eigenvalues respectively by (#4)221 and (Ax)211 For convenience, assume that both G and H are strictly positive definite.
Definition 2 (Model conditions). For each model parameter w, define the population risks for the two tasks by
R1(w)= == Eav (v w x)²,, R2(w) == Epe (y - w xx)²
Assume that a shared optimal parameter exists for both tasks, i.e.,
arg min R, (w) n arg min R2(w) of
Denote w* as the minimal- norm solution that simultaneously minimizes both tasks, i.e., w" is the unique solution of the following program:
min
w ||/l
s.t. w E arg min (w) O aarg min R2(w).
The shared optimal parameter assumption is common in theoretical CL literature (Zhao al. 2024
Li al. 2023 Evron et al. 2022). allowing us to focus on the effect of covariate shift. This assump- tion is mild, since overparameterized neural network can solve multiple tasks together in practice, demonstrating the existence of shared optimal parameters. Furthermore. our theory ready to be generalized to allow different optimal parameters by standard application of the triangle inequality.
3

---

Published as a conference paper at ICLR 2026
Continual learning. The goal of CL is to learn model to minimize the joint population risk:
R(w) R1 (w) + R2(w).
Unlike multi-task learning problems, CL is under the constraint that the sampled datasets for the CL tasks are accessed in a sequential manner. Specifically, CL agent on two tasks has two learning phases where the agent draws samples first from D(1) and then from D(2), At the end of the second phase, the agent generates a model parameter that aims to achieve a small joint population risk. To achieve this goal, information from D()) needs to be transmitted to the learning phase of D(2) Therefore, the memory consolidation phase introduced between the learning phases (Kirkpatrick) Zenke
et al. 2017 Zenke et al. (017), in which the CI. agent retains only limited information from learning
D(1) and feeds it to the second learning phase.
Ordinary and & éz-regularized continual learning. Two basic CL algorithms that are well studied in the literature referred to by us ordinary continual learning (OCL.)Evron et al., 2022 Li et al., 2023] Zhao et al. 2024) and fz-regularized continual learning (f2-RCL) et al.) 2023 In the first learning phase, both OCL and 22-RCL perform ordinary least squares with the first dataset. In the memory consolidation phase, they transmit the obtained minimum norm estimator w for the first task. In the second learning phase, OCL finds the parameter (2(2) that fits the second dataset while minimizing the fz-distance to w(1)) The information consolidated and transmitted between the two learning phases is minimal, consisting of only one vector of dimension d (the estimator ( However, is known that this algorithm suffers from catastrophic forgetting (Evron al,
2022
Li et al. 2023). The parameters generated by this training procedure are specified by:
w
(x(1)
(())
(2) (1) (1()) ()) (2) (1) + (x) x(2))= - (y x w (D).
With slight sophistication, R2-RCL computes model parameter w(22 that fits the second dataset under an isotropic f2- f2-penalty from the previous parameter, with its intensity quantified by ?:
w 1 (X T ( -1)) y w((2 arg min n
(2) w0 + +//lww---- (2)
Compared OCL, the additional information saved and transmitted between the two learning phases is only scalar (the regularization parameter 7). This shows that 22-RCL is still low- memory-cost algorithm. However, proved in [i et al. (2023) that no choice of y can temper the catastrophic forgetting for certain two-task linear regression problems. Therefore, better algorithms involving more complicated memory consolidation need to be considered for better CL performance.
Generalized 6y-regularized continual learning. The primary CL algorithm in our analysis is the generalized fz-regularized continual learning (GRCL). The first learning phase of GRCI. is identical to the OCL. In the memory consolidation phase, GRCL transmits the obtained parameter w as well a semidefinite matrix Σ _ In the second learning phase, the algorithm finds the parameter that fits the second dataset under quadratic penalty of distance from the previous model parameter, quantified by the metric E. Specifically, GRCL outputs (2( such that:
(1) (x(1) x()) (1) (2) (2) arg min n
(1
(2) + w| w
(1)
(3)
The additional regularization matrix Σ usually low-rank and can be stored in the form Σ W W, where W E Rk Compared to OCI,, GRCL takes an additional memory of E, which can be expressed with & vectors of dimension d. The choice of Σ determines the balance between the memory cost and the statistical performance of the GRCL algorithm.
We note that GRCL covers several commonly studied CL algorithms as special cases. Specifically, GRCL becomes OCL when Σ - O; when Σ = 7I for 7 > 0, GRCL becomes f- (Li et al.) 2023). The EWC algorithm (Kirkpatrick et al. 2017) 7 is a special case GRCL when a diagonal matrix. We also note that GRCL has been studied in Zhao al. (0024) however, they only focus on the optimality of the algorithm where the size of the regularization Σ is unlimited, while we focus on the different choice of Σ that affects the balance of the memory-statisties trade-off.
4

---

Published as a conference paper at ICLR 2026
Evaluation metric. The output parameter w of the CL algorithms is evaluated by the joint excess risk, defined by the excess risk of the algorithm compared to the optimal performance:
( 2) - R(w min R(-).
We present set of assumptions that are used in our analysis. Assumption (Well-specified noise). Assume that for the distributions for both tasks t 1,2, the label variable is given by
4d x0) w" + +el0, e0 NN(0,
where e(t) is independent of (1) and the shared optimal parameter w" is defined in Definition 2
For conciseness, we assume that the two tasks have the same noise level o². This is not restrictive, and our results can be directly generalized to allow different noise levels well.
Assumption 2 (Commutable covariance matrices). Assume that the two data covariance matrices G and H are commutable, namely GH HG. Without loss of generality, we further assume that both G and H are diagonal matrices.
We note that the diagonality of covariance matrices G and H are only for convenience. Our results also hold for commutable covariance matrices since the CL problem is rotation invariant. These assumptions are commonly seen in the literature for the covariate shift scenario (Lei et al. 2021
Wu et al. 2022
Zhao al. 2024).
Notation. For two positive-value functions f(z) and g(x), we write f(x) S 9(x) or f(r) 2 g(x)
f(x) S cg(x) or f(x) 2 eg(x)) respectively for some absolute constant e; and we write f(x) m g(x) if f(x) S g(x) S f (x). For two matrices G and H, we denote G < H or G' > H G H
or G are PSD matrices respectively. Additional notations are deferred to Appendix
MAIN RESULTS
We investigate the performance of the CL algorithms the one-hot setting. The one-hot setting is type of random design setting (Hsu et al.) (0012) that is well-studied in previous literature (Zou et al.)
2021, where input vectors are sampled from the set of natural bases. Compared to the fixed design in Li et al. (202)), the random design considers the impact of the input distribution randomness.
Assumption (One-hot hot setting). Let (et) be the orthogonal bases for H.. Assume that (() e,) 14; and that (2) e;) = X, for i 1, where μ4: A, O. E, lr = 1 and E; =1
Risk bounds for joint learning. When all information in the first learning phase can be accessed in learning the second task, one can solve the two-task CL problem by joint learning, which the output is determined by
W joint
arg min X (o yf (3)
||
Since joint learning transmits more memory than any other CL algorithm, it is usually used as or an upper bound in evaluating the performance of CL algorithms in the literature (Farajtabar et al., 2020 Saha et al. 2) We expect that well-designed CL algorithm can match the performance of joint learning- To this end, the following proposition presents a risk bound for solving the two linear regression tasks with joint learning in the one-hot setting.
Proposition 1 (Joint learning bound). Suppose Assumptions
7 and 3 hold. Denote J, K. such that
[i: p4
11 and KK
{i: X 4). Then for woiint given by
EA RA(wooint) bias + variance,
where
22 bias (1-1))(11 (1) + Ad)wt? variance
(JUK + Σ (Mi Xi)
m
5

---

Published as a conference paper at ICLR 2026
We are partieularly interested in the CL problems that jointly learnable, i.e., having an o(1) excess risk with joint learning. From Proposition[] we can see that a necessary (and also sufficient) condition to be jointly learnable is that the "head" of the distributions (i.e., the number of large eigenvalues [J], [[] is o(n), and that the "tail" of the distributions satisfies the following conditions:
Σn =((/n), Σ =
(5)
iEKc
4.1 RISK BOUNDS FOR CONTINUAL LEARNING
Our main result, Theorem provides general risk bounds GRCL with any regularization matrix
Σ = O. In particular, GRCL reduces to OCL when Σ 0 and to R--RCL when Σ = for y >> 0.
By establishing matching upper and lower bounds for the joint excess risk of GRCL as a function of Σ, are able analyze the impact of regularization different CL algorithms.
Theorem 2. Suppose Assumptions
n J. and K i X 2 } Consider the PSD regularization matrix E diag(1) 1-1 where E is commutable with the data covariance matrices G, H. Then for the GRCL outpur 3). it holds that
and hold. Denote I, K. such that J = {i 2 11
EA(w bias + variance,
where the expectations of bias and vari ance with respect (1) and 2x satisfy
bias ((G HJ G) (I
H 22 + (I H) w wnww",
variance (² H)
+ (I - H)"): - Gj nGro)
(6)
+ + H H, Σx) Hx Hg + n + n = Hk=)).
Theorem 2 tightly characterizes the risk of GRCL in the one-hot setting. Specifically, the CL risk consists of bias term and variance term, and give matching upper and lower bounds on both parts. These bounds are functions of the CL task distribution parameters (G, H, n) and the regularization matrix Σ. In the following, we use this result analyze catastrophic forgetting in OCL and RCL and demonstrate how GRCL mitigates forgetting with arbitrary memory Σ.
Catastrophic forgetting of OCL and R2LRCL Compared GRCL, OCL retains minimal mem- ory carried over to the second learning phase. As a result, its statistical performance is expected to be inferior to joint learning. The following corollary quantifies this difference:
Corollary 3. Under the conditions of Theorem
2
for the OCL output 1 we have
E2(w EA EA(Wjoint)
o²
Σ
n
Σ 1.d.)
EE
iEInKo
As indicated by Corollary 3 for OCL to achieve an o(1) CL excess risk in a jointly learnable prob- lem, it is sufficient and necessary to have
Σ REK
MM+
+
Σ n4 Md, o(n).
(8)
iEIng=
The expression in 8 quantifies the statistical gap between OCL and joint learning. To minimize this gap, the intermediate eigenvalue space of H near 1/n must align with the small eigenvalue space of G. When ( is not satisfied, OCL experiences constant statistical underperformance due to catastrophic forgetting. Similarly, RgLRCL is also low-memory CL algorithm, carrying only an additional memory of the regularization parameter 7 compared to OCL. The following corollary helps us understand the performance of RCL.
Corollary 4. Under the conditions of Theorem
for the R-L outpur
we have
EA w (2) S EA (A( + + 4))l
o²
Σ
7
(9)
n
SEJUK
A +
+

---

Published as a conference paper at ICLR 2026
Corollary 4 shows that for f2-RCL to achieve o(1) risk in jointly learnable problem, it suffices that
y= o(],
Σ iEJUK
7 + o(n),
(10)
A + +Y
weaker of requirements than (8) as OCL is special case of f2-RCL when + O. However, there still exist CL tasks that are jointly solvable yet solvable by R2LLL. To demonstrate the failure of OCL and &--RL we show two examples of
holds for some < > is clear that (TO)
CL problem: the first is not solvable by OCL, and the second is further not solvable by 2-RRCI
Example 5. Under the conditions of Theorem
suppose o² and |20| 1. Then, for
G diag(14); and
If μ1 and A1 = 1/n, then the OCL excess risk satisfies EA =
Suppose thatfor' < fm, #4 =4=1 1 and A; = =, while for In2 Si fn, μi to and X 1. For all other 2 Gn, = > = 0. Then for any regularization parameter > 0, the
fg-RCL. excess risk satisfies ( 91))
Thus, both OCL and 22-RCL cannot match the performance of joint learning across all CL tasks.
GRCL avoids forgetting with full memory. To enhance CL performance, GRCL introduces the regularization matrix E to transfer information from the first task to the training phase of the second task. The following corollary shows that with sufficient memory and appropriate E, GRCL can achieve performance comparable to joint learning.
Under the same conditions as Theorem consider the regularization Σ diag(T)) with 7/2 = μe for μs 2 1/n and 7/ 0 otherwise. Then, for the GRCL output (6) have
Corollary 6.
EA(w (2) x EA (janm))
Corollary 6 demonstrates that by selecting regularization matrix of size [J] that captures all the top eigenvalues of greater than 1/n, GRCL can attain joint learning performance and eliminate catastrophic forgetting from OCL and RCRCL
4.2 SEMORY-STATISTICE OFA OF GRCL
In this part, we further investigate the constrained memory setting to analyze the trade-off between memory consumption and statistical performance GRCL. The memory cost of GRCL is measured by the number of distinct cigenvectors in the regularization matrix Σ. Our results demonstrate that the GRCL excess risk decreases the memory allocated regularization increases, and vice versa.
Catastrophic forgetting in low-memory settings. Our analysis in Section 4. reveals that both OCL and R-RCL exhibit catastrophic forgetting with limited memory. We extend this understanding to GRCL through the following example:
Example 7. Under the conditions of Theorem suppose o² = and || = 1. Then, for G diag(H:) and H (iag(>;)
For any positive constant << N, if = 1/(k and >; = 1/n for 1 1 k: then for any regularizer E of size k. the GRCL excess risk satisfies TED u (2) ((1).
This constructed scenario generalizes the OCL failure on Example GRCL: recall that in the previous example, there exists one eigenvalue mismatch on the dominant eigenspace (μ² \₁ 1/n). Since OCL lacks the memory to store this mismatched eigenvalue, catastrophic forgetting occurs. Similarly, in Example 7 the 1 dominant features in G coupled with intermediate H eigenvalues create an insurmountable memory bottleneck no k-dimensional regularization in GRCL can mitigate forgetting across & orthogonal directions.

---

Published as a conference paper at ICLR 2026
102
JI OCL
GRCL, k=1
Excess Risk
GRCL. kw@
Excess Rak
OCL
10-1
1000
2000
3000
4000
Sample Size
Memory
(a) Convergence of GRCL.
(b) Memory-statisties trade-off
Figure 1: Expected excess risk vs. sample size and bb) memory size k for generalized f-- regularized continual learning (GRCL), compared with joint learning (JI.) and ordinary continual learning (OCL). For each point in each curve, the y-axis represents the expected CL excess risk (in the logarithmic scale). The dimension of the task is = 200. The sample size is fixed at = 5000 in (b) The excess risk is computed by taking empirical average over 20 independent runs.
Tempered forgetting in moderate memory settings. In Corollary 6 and Example 7 we showed that GRCL can suffer catastrophic forgetting when memory is limited, and that GRCL performance matches joint learning with sufficient memory. The following example illustrates GRCL perfor- mance dynamics with intermediate memory.
Example 8. Under the same conditions in Theorem let G = diag(14) where μ4 = i~ with
> 1. Consider the top-k regularization Σ (iag((;))= 2o with 7i = p4 for I and 7 otherwise, where 1 I k I VT. Then, for the GRCL oulput (6). we have
BA(w S EA(W)aine)
(11)
In Example observe that the memory size l of GRCL increases, the ratio of GRCL's excess risk to the joint learning risk decreases at rate ga. approaching the joint learning performance at k = Vn. This example demonstrates that GRCL with an intermediate memory constraint can partially alleviate catastrophic forgetting.
Numerical Experiments. We conduct numerical experiments in Gaussian design to verify our theoretical results; additional experiments on practical CL datasets are deferred to Appendix E.2 Gaussian design is a more widely used random design setting in the statistical learning literature Hsu al. 2012, Bartlett et al. 2020).
Assumption (Gaussian design). Assume that (1) = 1/222(1) (1) and that x 122 where (1) ~ I).
We consider CL problem adapted from Wu et al. (222 and Li al. 2023 (see Appendix E. for details). As shown in Figure [(a)] OCL suffers from a constant excess risk due to catastrophic for- getting. contrast, GRCL's performance improves with memory size k: matching the convergence rate of joint learning at k = 5. To further verify the memory-statistics trade-off of GRCL, we vary the memory size l in Figure [[b) which shows that GRCL reduces excess risk with larger & and reaches joint learning performance with k < 15 for the given CL problem.
EXTENSIONS
In this section, we discuss extensions of our results and messages to some more general CL settings.
OCL in Gaussian design. From the numerical experiments, observe that our main results for the one-hot setting in Section 4 also empirically hold for the Gaussian design. However, our main theoretical results GRCL cannot be directly generalized to the Gaussian design due technical hurdles. As preliminary step, we provide the lower and upper bounds of the OCL excess risk in Appendix C
Theorems T8 and 19
8

---

Published as a conference paper at ICLR 2026
We would like highlight the different behaviors in Gaussian design from the one-hot setting: the following example shows the additional source of forgetting in Gaussian design.
Example 9. Under the same condition in Theorem 18 suppose that o² 1, 2 1. Then for (u and (Aa)
μn and A = 1/n, then the OCL excess risk and the the forgetting 92(1). satisfies B
14 i- log (i 1) and X; -
log log + 1) for some constants a. 3. > that then the OCL excess risk and the forgetting is 00log--a-1 n).
The first problem in Example Saalle@illuseee forgetting in the Gaussian case, which is also described the one-hot case Example However, the second problem in Example 9 shows that forgetting can also result from differences in the less significant features on the tail.
illlsstrates how single difference in essential features can result in
CL in the NTK regime. We study the extension of main messages in linear regression the NTK regime of neural networks. Consider a 2-task CL setting in which the goal is to learn a neural network model fw (x) specified by parameter w E Rd Lee et al. (211)) proved that under the NTK regime, neural networks evolve as a linear model:
(w) (0) (x) + (V. (z), - w
(12)
where f(0) denotes the model with w O as the initial weight. Under this setting, we consider convergence of GRCL with gradient descent (GD). We can reduce GRCL the NTK setting to the linear algorithm in (3) providing that give the following imputs x', y' to the linear model:
xx ()), = d)xx(2),, s/( y - (xx y²( = y (xx (13)
With the covariance matrices G = Epan (d(x (o) (zxx H == ED (o(x)))
(2) T de-
fined accordingly, which actually represents the Hessian of the model, have our main results:
Theorem 10 (Theorem 2 in the NTK regime). Suppose Assumption 5 holds. Denote G :=
Epoi (=(x()) (ozz()))((()) T H == Epo (o(x( (ox(())) Then for the GRCL output (2),
(2(w bias variance,
where the bias and variance satisfy (6) defined Theorem with G, defined as above.
Therefore, our main results in Theorem 2 well as its collollaries and messages still hold in the NTK regime and can be applied in general neural networks.
Multi-fask CL. We study the extension of our main messages the multi-task CL setting. For lasks f = 1, ... where 7' 2 3, the goal is to learn a model w to minimize the joint population risk R(w) Σ, (w). Corollary extends the message that with sufficient memory and appropriate regularization, GRCL can match the performance of joint training in multi-task CL. For details, please refer Appendix(Dx.
Corollary 11 (Corollary 6 in the multi-task setting). Suppose Assumptions 1",2", 2", 3" hold. Con- sider the regularization
(t) diag(>; (0) ( fort = I,...,T, with 7 (1) = μ (1-1) for pi (1-1) 2 1/n
(t) =0 otherwise. Then, for the GRCL output w!T), we have and 74
(T S (A(w))in))
(14)
Combining with the existing Example 7 where low-memory CL behaves poorly, we deliver the message that there is provable memory-statistics trade-off in multi-task CL.
Practical algorithms. We now examine how to generate the regularization matrix Σ from Corol- lary [ which achieves joint learning performance GRCL. Recall that Σ is computed after the first training phase when we have access to the sampled training data X (1), and the goal is memorize the top eigenvalues and their corresponding eigenvectors of G. In the one-hot setting, we can sim-
ply store all distinct a. (1) the eigenvectors and use their frequencies as unbiased estimation of

---

Published as a conference paper at ICLR 2026
eigenvalues. This approach has 0.999 probability of capturing the eigenvectors with eigenvalues H4 > 10/n. As a result, this makes GRCL a practical algorithm without requiring prior knowledge of the true covariance parameter. In the case of Gaussian distributions practical datasets, linear sketching methods such as CountSketch (Charikar et al. 2002) can be used to specify the memory utilized in GRCL,, where the GRCL algorithm corresponds to the Sketched Structural Regularization method proposed by Li et al. 22021)
6 CONCLUSIONS
We consider the generalized fz-regularized continual learning algorithm with two linear regression tasks in the random design setting. We derive lower and upper bounds for algorithm. We show a provable trade-off between the memory size and the statistical performance of the algorithm, which can be adjusted by the regularization matrix. We show that catastrophic forgetting occurs when no regularization is added and that well-designed structural regularization can fully mitigate this issue. We demonstrate the semory-statiiticcs trade-off of generalized f fz-regularized continual learning in random design with concrete examples and experiments.
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European confer- ence on computer vision (ECCV), pp. pp. 139-154, 2018.
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117749):30033-30077) 2020.
Mehdi Abbana Bennani, Thang Doan, and Masashi Sugiyama. Generalisation guarantees for con- tinual learning with orthogonal gradient descent. arXiv preprint arXiv:2006. /1942, 2020.
Moses Charikar, Kevin Chen, and Martin Farach-Colton. Finding frequent items in data streams. In International Colloquium on Automata, Languages, and Programming, pp. 693-703. Springer, 2002.
Arslan Chaudhry, Puncet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European conference on computer vision (ECCV), pp. $32-547, 2018
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny. Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning, arXiv preprint arXiv: 10486, 2019.
Thang Doan, Mehdi Abbana Bennani, Bogdan Mazoure, Guillaume Rabusseau, and Pierre Alquier. A theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In International Conference on Artificial Intelligence and Statisties, pp. 1072-1080. PMLR, 2021.
Itay Evron, Edward Moroshko, Rachel Ward, Nathan Srebro, and Daniel Soudry. How catastrophic can catastrophic forgetting be in linear regression? In Conference on Learning Theory, pp. 4028- 4079. PMLR, 2022.
Itay Evron, Edward Moroshko, Gon Buzaglo, Maroun Khriesh, Badea Marjieh, Nathan Srebro, and Daniel Soudry. Continual learning in linear classification on separable data. In International Conference on Machine Learning, pp. 9440-9484. PMLR, 2023.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for contin- ual learning. In International Conference on Artificial Intelligence and Statistics, pp. 3762-3773. PMLR, 2020.
Daniel Goldfarb and Paul Hand. Analysis of catastrophic forgetting for random orthogonal trans- formation tasks in the overparameterized regime. In International Conference on Artificial Intel- ligence and Statistics, pp. 2993. PMLR, 2023.
10

---

Published as a conference paper at ICLR 2026
Daniel Goldfarb, Itay Evron, Nir Weinberger, Daniel Soudry, and Paul Hand. The joint effect of task similarity and overparameterization on catastrophic forgetting an analytical model. arXiv preprint ar arXiv:2401.12617, 2024.
Ian Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empiri- cal investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprini arXiv: 111.62131,.6211, 2013.
Reinhard Heckel. Provable continual learning via sketched jacobian approximations. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 10448-10470. PMLR, 2022.
Daniel Hsu, Sham M Kakade, and Tong Zhang. Random design analysis of ridge regression. In Conference on learning theory, pp. 9-1. JMLR Workshop and Conference Proceedings, 2012.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, 31, 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-335 2017.
Soheil Kolouri, Nicholas Ketz, Andrea Soltoggio, and Praveen K Pilly. Sliced cramer synap- tic consolidation for preserving deeply learned representations. In International Conference on Learning Representations, 2020.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
Qi Lei, Wei Hu, and Jason Lee. Near-optimal linear regression under distribution shift. In Interna- tional Conference on Machine Learning, pp. 6164-6174. PMLR, 2021.
Haoran Li, Aditya Krishnan, Jingfeng Wu, Soheil Kolouri, Praveen K Pilly, and Vladimir Braver- man. Lifelong learning with sketched structural regularization. In Asian conference on machine learning, pp. 1000. PMLR, 2021.
Haoran Li, Jingfeng Wu, and Vladimir Braverman. Fixed design analysis of regularization-base continual learning. In Proceedings of The 2nd Conference on Lifelong Learning Agents, volume 232 of Proceedings of Machine Learning Research, pp. 513-533, PMLR, 22-25 Aug 2023.
Sen Lin, Peizhong Ju, Yingbin Liang, and Ness Shroff. Theory on forgetting and generalization of continual learning, arXiv preprint arXiv:2302.05836. 2023.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks, 113:54-71, 2019.
Liangzu Peng, Paris Giampouras, and René Vidal. The ideal continual learner: An agent that never forgets. In International Conference Machine Learning, pp. 77585-276101 PMLR, 2023.
Haoxuan Qu, Hossein Rahmani, Li Xu, Bryan Williams, and Jun Liu. Recent advances of continual learning in computer vision: An overview. arXiv preprint arXiv:2109, 1369, 2021.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. Advances in Neural Information Processing Systems, 31, 2018.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. arXiv preprint 103.:2103.09700) 2021.
11

---

Published as a conference paper at ICLR 2026
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pp- 4548-4557, PMLR, 2018.
Alexander Tsigler and Peter L Bartlett. Benign overfitting in ridge regression. Journal of Machine Learning Research, 24(123): 2023.
Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734. 2019.
Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027, 2010.
Liyun Wang. Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Jingfeng Wu, Difan Zou, Vladimir Braverman, Quanquan Gu, and Sham Kakade. The power and limitation of prerraining-finetninng for linear regression under covariate shift. Advances in Neural Information Processing Systems, 35:33041 1-30055,, 2022.
Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. Continual learning for large language models: A survey. arXiv preprint arXiv:240..01364, 2024.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pp. 3987-3995, PMLR, 2017.
Xuyang Zhao, Huiyuan Wang. Weiran Huang, and Wei Lin. A statistical theory of regularization- based continual learning. Forty-first International Conference on Machine Learning, 2024.
Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, Dean P Foster, and Sham Kakade.
The benefits of implicit regularization from sgd in least squares problems. Advances in neural information processing systems, 14:545655666 2021.
12

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORKS
CL practice. Over the past decade, continual learning has found applications in computer vision Quu et (021)) and, more recently, in large language models (Wu et al.) (024). A long list of practi- cal CL. methods has been proposed to address the catastrophic forgetting problem of neural networks. These CL. methods can be roughly divided into four categories: regularization- based methods (e-g.,
Kirkpatrick al. (0177). replay-based methods (e.g., (Chaudhry et al. (2019)), architecture-basee methods (e.g.. Saha et al. (2021)) Among methods, early structural regularizers including EWC (Kirkpatrick et al. 2017) and MAS (Aljundi et al. 2018) are widely used since they take only O (d) memory by ap- plying diagonal approximation to the importance matrix. More complex structural regularizers with higher memory costs proposed later for better CL performance, including Structured Laplacian Approximation (Ritter et al.] 20188) and Sketched Structural Regularization (Li et al.] 2021). Beyond structural regularization, replay-based methods reduce forgetting by retaining the memory of old data and replaying them in training new tasks, and projection-based methods reduce forgetting by projecting the gradient update into a memorized subspace. One can refer Parisi et al. (2019); Wang et al. (2024) for comprehensive overview.
methods (e.g.. Serra al. 2018 ) and projection-basse
the regularizationn
CL theory. In addition to the works by Zhao et al. (202)); Li et al. (2023); Evron et al. (2022)
introduced in Section 2 in this section we cover additional CL theory works the literature.
The theoretical effect of full Hessian regularization on CL has been mentioned in several papers to different extents, including Peng et al.) (2023) and [Evron et al. (2023). Peng et al. (2023) mainly focuses on the optimal solution space of general CL. problems and how different CL. paradigms (in- cluding regularization-basse
methods) approximately solve CL by approaching that solution space, and [vrron et al. (2023) mainly studies the continual linear classification scenario. In these two works, it is briefly mentioned and proved in optimization scenarios that a full Hessian regularization can fully mitigate forgetting when measured on the input dataset. Compared to these papers, we in- vestigate further by characterizing the is required to prevent forgetting, since the Hessian is usually too large store in the GPU memory.
trade-off as well as how much memory
The works by Goldfarb & Hand (202)) Goldfarb et al. 2024) consider CL settings in which the first task input dataset is considered random. However, in both works, the second dataset is dependent on the first dataset, whereas in our work, the two tasks are independent. As a result, their similarity metric transitionnddeeendent while ours is task-dependent. Similar to Evron et al. 2022 Goldfarb et al. 2024) also only considers the optimization eттor.
The works by [Hckel (2022); Lin et al. (2023) consider the statistical error in CL. However, they additionally assumed that the data covariance matrices are identity matrices for all tasks, while the optimal parameter for each task is different, limiting their generalization results to the underparam- eterized regime. This reflects the task-incremental CL setting (Van de Ven & Tolias (019), while we focus on the domain-incrementt CL setting where all tasks share the same optimal parameter, but the data covariance matrices change across different tasks. Therefore, our results are not directly comparable with theirs.
The rest of the papers not directly connected to ours. In particular, Doan et al. (2021) Bennani ct al. (2020) studies the forgetting and the generalization error of the Orthogonal Gradient Descent method (Farajtabar et al. (220) in the NTK regime (Jacot et al. 2018).
B PROOF OF ONE-HOT GRCL
Additional notations. For any index set IK, we denote its complement by KS For any index set K and any matrix M E Rx denote Mx as the matrix comprising the i-th columns that satisfy i K; additionally, for any matrix M E dd denote Mg as the submatrix of M comprising the i-th rows and columns that satisfy i E K. We note that all matrix inverses used in this paper are Moore-Penrose pseudoinverses.
13

---

Published as a conference paper at ICLR 2026
B.1 PRELIMINARIES
Computing (2)) We first compute w()), By definition, we have
((1) = (x(1) x x x00="
= =(xO) x -1 -x(1) x w + (xx) x(1)) (10)-1x1 -x(1) ..00)
Denote p(1) = I (x(1) x(1) -1 x1) ( Therefore
(2 -===== =-p(2) w (x x(1) - ee
x
Now noticing that
Ee(1) = 0, (1) = 1²
so the covariance of w1() w0
B:(w (ww(() ww)(w()) - ==) = p()) w'w" p(l)) + xx((0)) x(1))= - x(1) xx) (x) (())-
(15)
= =(()), w"w" p(1) + o² (x((1) x())
(16)
Computing (²).
Then we compute (²) By the first-order optimality condition, we have
1 x2) T (x))) (()) E(w(?) (w 0,
n
which implies
(2) = (x T (2) + n.)) -1 (x(2) () + ))
= (x122 (2) nz) (xx(2) w² + (2) (2) + D)
Then we have
(2) ~0 = (x(22) T (2) + n.) -1
(x(22) T (2) w + (2) T (2)) + Eww - (x x + +nE)w*
= (x (2) nz) (nE(w w²) + (2) ())
Similarly, notice that
E2 =0,
Ee (2) 12
so the covariance of (2) w" is
12w - (2 T
= (x(22 T (2) + n1)) n (( (1) w0 (w) - -w") .nE(x(2) x(2) +n
+ + (x (2) X (2) + =
x2)
T x (²) (x(2) T x²) nΣ -1
Denote p(() = (2) (2) + +nΣ) E. Therefore
E(w(2) w" ) (2) TT
=P(2) B(w(() w³ (w (1) w=) pp(2)
+ (x(2) x mz) -1 (2) (2). (x(2) x(2) + -1
(17)
14

---

Published as a conference paper at ICLR 2026
Risk decomposition. According to the risk definition and the assumption on the noise, we have
R2 (w) - - B/| (20 w - 1
-
71
1 = (w - w" "G(w ww w²) + 0²
1
xellx (1) ~ -
71
1
(G, (w w" ( w )²² 1T + 002.
(18)
Similarly, the risk for the second task is
R2(w) = HH, (w w )ww - w0) + 0²,
(19)
and the joint population risk R(w) = (G + H, (w w (w - wz) )²) + 20
(00)
B.2 PROOF OF THEOREM 2
To compute the joint population risk and the forgetting, we give several useful lemmas.
Lemma 12 (Bound on the head of (x(1) (t)
x(1) ), Suppose Assumptions
1 and hold. Denote
K such that K- = {i: X 2 }. Then, for n 2 2,
H K = [
(2) -1 1
12
H
2n
K
Similarly, denote J such that J
= {i: p4 2 }. Then, for n 2 2,
1 G = E X (1)
x ()))
12 G
4n
-1
The proof is the same for (x) (
x("))
Proof. We will give the proof for
For the upper bound, consider each index E K in diag (x (2) T
X (2)
-1
which is (X(2) T
x(2)) '
Notice that according to Assumption for every i E [d]. (x)
X follows the binomial
distribution B(n, As). Also notice that for n 2 2 and 1 1 j S [n/2].
j ( x x1x - ~1)= 1 j+1
1
n
1 d
x (1 - A1) >1)) nー
j n n-j j
j + 1
Xi
j+1 1 n
(1 (1)):)
1 n
x² -
(1-1)) 11) --
j n n-j
nd
j+1)
n
x X.) nーsー!
< nd
j+1
Since the zero values do not count in the expectation of Moore-Penrose pseudoinverse,
E (2)
X
n2
(1))))
5
3=1
[n/2]
<
2
Σ
1
(
(n)
X (1
(3-111) (1-))-
[n/
12
Σ
1
x -11-2))-- ---
nd:
j /+1)
[n/2]+1
12
Σ
())
x))((11
x((1>>)-*
-1)=
nd
1=2
12
nd
15

---

Published as a conference paper at ICLR 2026
For the lower bound, consider each index i E IK in diag
(xx²) (2)
xxxe1) -1
E ( (2) x(21)
- = x(21), 0) x ( T x ;: >0]
2 =(( x > [] x (x() x x > >0 -0-
P((X() T x 0) (x (2) x /P( (x x 1 > 00) -1
= =((x()) x(21), > 0) ((((e)) T x(1) 1-1
- P x(²) # > 0)
nd;
1
Andi
The second line is by Jensen's 's inequality, and the last line is because
(((x (2)
X (2)
0) = 1 (1 -1- A.)" 2 1 1/2
by noticing that for all S k. A d 2 1
Lemma 13 (Bound on the tail of (X(11 x(1)) ) Suppose Assumptions and 3 hold. Denote
J, K. such that J = {i p4 2 }. and K = {i:: d 2 }. Then
G 1 X (1)
X (1)
nnG=,
n Hg
=
E
X
nHg.
Proof. We will give the proof for
(x
(x
X
2 -1
The proof is the same for (x(") x
K
k
Recall that for every index i, (x (2)
X
(2)
follows the binomial distribution B(n, Az). Therefore,
E (2)
X
s E X
(2) i nds.
Also, for i E IK.
(x (2) (2 -1 2 E1 [EI[((X] (x) (2) x s 1]
15
= =1 - 213=
> 7
A
The last line is by X < for all i E K
Lemma 14 (Expectation on p(2)). Suppose Assumptions 1 and 3 hold. Then
Ep(1) = =(I - -)
Ep(?) (I H H)"
Proof. We will give the proof for ((2), For any index
EP!2) E1 [x x (2)) ; = = xx)"
The same holds for p(1).
Lemma 15. Suppose Assumptions and 3 hold. Then, for all index i that satisfies X 2 1/n, 7/ + and for > 2,
1
+ (1 (1- A1)") (1-A1)")
n²(), + 72 + (1 (1-
{E (2) + n
< 144
2 (² + )3
n2??
n2n?
16

---

Published as a conference paper at ICLR 2026
Proof. For the upper bound, notice that according to Assumption 3 for every E [d].
(x (2) ) ii follows the binomial distribution B(n, Az). Also notice that for n 2 2 and
1 و و - [n/2],
(
1+1 1+1 ndd i+1
n2 (1-1))----
53.
Therefore, we examine the following quantity:
E ( (2)
X (2)
+ nz) 2
2
(1 - X1)a (+).
n2??
Notice that
(3) x2( (1 - 21)
11)
(j + m7))2
j=1
[n/2]
(7) (1)x(1 - -2,)
S
(j + ny:)2
j=1
[n/2]
1 (f+1) 1)
(nm) n
(+1) ( (1-1))
Tud
(j + n7))
j=1
[1/2]+1
12
n1 x){((11 - x(1x)11))))) -7))-- (++n))33
nd: 2=2
Therefore,
[n/2]+1
(nd, +n7) (4) S 12
(y) (7) (1)x(1-A))) A1) )0)-0
i++ (7)x( A2(1 21)
my
11
2=
(1) x)) - 2.)m
< 12
j
+ nr
Use the same technique and we can get
(nd, + m7s)
Σ
n-1
1
(1) (7) (1(11)))*
12
Σ
n-
(1) x 11--1) (1-1,))
11
J+
+ n%/s
j=1
12
As result, (4) S 1=.2 +2 and
144
E (2) + n. 22 < 144
1 (2: + +76)2
+ n2??
(1 - 21)m)
For the lower bound, firstly, by Jensen's 's inequality.
x (2) nΣ)
2 2 (2) T x22) + +nZ), -2
(nds + m7x)
Secondly, by only selecting the j = 0 term in the sum,
n
n1) (7)x(11 21) G m 2
E E(X (2) i
1=
(1 A1)
n 22
Combine them and we have
E (2)
(2)
n
-2
2
1
1
" (1 (11-2))))
n² + 3z
17

---

Published as a conference paper at ICLR 2026
Lemma 16. Suppose Assumptions and and n 2 2,
Land 3 holdd Then, for all index i that satisfies X 2 1/n, 9 # 0
A - [ ( +2Z) -2 xx(22) x I 144 n(>; + 74)2
A;
48 n(A, + 7)2
Proof. For the upper bound, notice that according to Assumption for every E [d].
(x(2) x(²) j follows the binomial distribution B(n, X,). Also notice that for 2 2 and 1 S j $ [n/2],
( x0
23
1.. s+1 n (+11)x ( nd,
(1- 11) -1)=-
Therefore, we examine the following quantity:
B[(x (2) x²2 + +nZ) =xx x2)2) x = (+).
Notice that
())x(( (1 - -1:)
(+) < Σ
(j + m2:)
j=1
[n/2]
(1) (){((11- (11))=))" 21)
( ny:)) )2
j=1
[n/2]
1
ز(ز +
( )) { (1 21) -
nd:
(j + n3 )
i+1)
/=1
12
[n/2]+1
j2
( xx(1 9x(1-2)) -1,)
11)
nd
(j + my)
j=2
Therefore,
[n/2]+1
(j (i+m))² (c) - 21) 21)=0 + Σ
jny
(7) X2(1 x²x((1 25)
(nd: + +1y) (+) 12
nyn)
2=2
n-1 1
(7) - -2,)
< 12
j + ni
j=1
Use the same technique and we can get
(7) (((((1-)))
n-1 12
X²(1 xx(
(nd: + n7))
Σ
-1-)"
+ noti
1=1
==1
12nd.
As result,
= (2)) (2) nE) -2 x2) (2)
A n(); + )
is 144.
18

---

Published as a conference paper at ICLR 2026
For the lower bound, by applying Jensen's inequality on the positive values of (x T x(2) + n (2) (2)
(2) we have:
EI X (2) T x + + x2) X 1
> P((x X (2)) 11 > 0)² [E (2) x
+n 2 (x (2) T
X 11
P((x()
x > 0)([nX, + 2n77 + n² T (2)
x(21)==11
2 P(( x(3), > 0)² [nd;
mnd; + +2n7; + 12n 7
12m (A: + 22)2 (((x(*) x 13 0)²
A
A
48n +2
The last line is because
P((x1) x n >0) =1 11 (1- 21 2 1/2
by noticing that X, 2 1.
Lemma 17. Suppose Assumptions 7 [ann] 3 hold. Then, for all index i that satisfies 0 < A, S 1/n
anr n 2 2, Then
e(1 n S (
nd:
(2) T x² + =nE)= xx(2) T x(2 S
nd
(1 my)
Proof. Recall that for every index i, (x(2) T x(2), follows the binomial distribution B(n, Xz)- Therefore,
(2) +nE) -2 x2) (2) |u S
E(x (2)T T x(2)) 8
(1 + ny:) nd
(1 ny) )2
Also,
B((x (2) + =1)) -2 -n=)==-( (2) T x ji 2
1
Pl P((x(2) T x(2)) n =1] 1]
(1 + ny:)²
(1+ R11 - -1)
nd
e(1 ny:)²
The last line is by d S /
Now we can give the risk in the one-hot setting in Theorem
Proof of Theorem 2 Recall that Σ
(X (2) (2) + + n. According to (17) and 20
E3 (2) min R|
BB H, (x (2) w0) (w)() -w")T
= =GG H, P R (wn (u (1) - w (10
(1) - -w²) ) TT(())
o H. (x (3) n -2 2x()) T ((2)
= (G H. p (22)p(()) (1) u w p(1)pp(2 (2),
Σ
+ ( + H, Σ X T
22
X -1p(2),
G H, (x
(2) x (2) n1) -2 (2) (2) x (3))
(2)
= bias + variance.
19

---

Published as a conference paper at ICLR 2026
where the bias term satisfies
Ebias E H, PS pll) w w Tp(1pp())
~ + H1 - - 1)
w'w"1 T
and the variance term satisfies
Evariance =
BilaaanceeoBB(((A oOB( P (x() x -()))
+ (x x + -2 X x
(I H)", B(X(1) x(1)-
+G E(x +n - x2 T x(2))
~ E (I H)"); G nGze)
+ (G H. -Hz + Σx) Hx + n(Ize + xgo)) 2 Hg.))
We have finished the proof.
B.3 PROOF OF PROPOSITION
Proof of Proposition
Define the joint data and the joint label noise by
(e(1)) ((2)
X =
(2)
y =
y (2)
E 22n
E ::=
E
2n
then
Ee = 0,
Eee" T 0²1I.
By the definition of 4 we obtain
(X X) X y
joint
= (XXTx x) x Xw" + (x x) 'XTe,
which implies that
x) (XXx)-XXXXXX w + +(x xxTx)
x)= - XTe. e.
Wjoint 20"
Denote Prosnt I (x x X X. Therefore, it holds that
Ee (W - w0 ) w') wT) Pjoint w "T" Pyoonn + +o² -x
(21)
Moreover, the total risk can be reformulated to R(w) min R
R2 (w) min R₁ + (2(w) min R₂
= (G, (w - w' (w - w (T) + (H, (w w ww w²) )²
= (G + H, (w w (w - w", w²)T).
Bringing 21 into the above, obtain
Eg R(Wyosnt) min R = G + H, E. (W w0' "n)(( (W))) - =w")T)
= (G + H,P Pooin "n TT Pooinet + 0² (G + H, (XTx)
bias + variance.
For the bias part, notice that Pront = PiP2 in the one-hot S Therefore,
Ebias = E(G + H, Pjoine = "w" -Pooint)* +
((G + H)(I G)" H)", w wo=w"T
Ems + ++) -1) ???
20

---

Published as a conference paper at ICLR 2026
For the variance part, notice that (xT))= = (x()) x + (x x(2)); is for every index
where (X (x) follows the binomial distribution B(n, (s),, and (x x(2) follows the binomial distribution B(n, A₂). As such, consider a random variable 8,, which is the sum of n
independent trials, where each trial returns 1 iff at least one trial of the binomials (x T x(1)
and (X (2) X (2)) return 1. Then for every index (X)
T X)a satisfies
8, S (X ))x )z S 28c,
and 8; follows the binomial distribution B(n, μ4 + x mı),). Therefore, according to Lemma 12 for each index i that satisfies 14 + X 14>; N _ for constant >> 0,
1
1
E 1
E x) : S ExT -
12 14 11 uddi
2
n μ4 + x - -m
Also, according to Lemma 13 for each index i that satisfies 14 X p S - for constant C> 0,
n(ps + X p;d;)) - Es;" S <B(x ii S Esp" 2 n(Hs + x - mi\i..
2
Notice that for the index sets J {i 144 2 }. and K = fi X N }.. for i E JUK,
[4 X uadi 2 :; also, for E Jo n IKe p4 + X nز>; v 2.
Also notice that
n
0 S p= S (14 + 21) - 1 - A,).
Therefore,
1 1 5 x) -
1
144 X/ iEJUK;
n 14 + X;
m(u + Aτ) 1( x) 1s S 5n(u= + Ax), iEJ
As a result,
1 G T + ( + H
B [ x)
Now can bound the variance by
Evariance = 0². (G + H, E [(XTX)1)
~ 0². (G+ H, 1 G H)JU + n + H)r(E)
n
x² n
(JJUK)
+ n > (m4 + 21)9).
+n²
We have finished the proof.
B.4
PROOF OF COROLLARY 3
Proof of Corollary 3 Recall from Theorem 2 that
EA(w(2) = bias + variance,
where
bias = H)
w w
-1 (1) A:) + X;)u;??
H)n,, - - + +
variance ~ o2 + H)
G H. H + nHg))
VA 0²
Mi
+ 2
Σ
Midg + Σ (1-A) + Σ
(1 >x) a n1
iEK
NEJ
NEL.
K
+ n
x2 +
Aa(1 - )1
14 + n Σ xx - -611~xa)") >1)"m1
K
iEI
.EJ
21

---

Published as a conference paper at ICLR 2026
Also recall that the joint learning parameter Wjiint satisfy
E (oon))))) = biasjoint + variancejoint;
where
Σa (1--- 114 + Ae)ws?
biaS
variancejoint
(JUUK) K + n2
Σ
(mr + ):)
n
SEJonke
Notice that:
The bias matches the joint learning bias;
(IK] + n²2 EiEKo 27) is bounded by the joint learning variance;
.And that
0²
11(1 -1.) 14 n²
11(11-
(1-111))
n
NE
NEIe
1/2
1/2
A2 111 2))²m
Σ
2
P4
+
Σ
n2²
n²
iCJ
iEJ
1/2
1/2
K
x2
[J]
+
Σ
n² u
KEK
iE
is also bounded by the joint learning variance.
As result,
variance S variance joint
Σ
n
Σ
Midi +
(1-A)) +n² Σ (1
Xa)" 1
iEK
iEK*
iE]
SEJc
S variancejoint
14
+ n2
Σ
pGdE + n
Σ
mEd1 + [J] + 2
k
Aj
KEK
iEJnKc
iEJENKE
NEJ
< variance joint +
Σ
14
+ n
Σ
pide +
[J] 2m
Σ
14 +
2
Σ
X3
icK
iEJnKe
K
variance
Σ
M4
n²
Σ
p:di-
X
iEK
ieJnk
Thus we finish the proof.
B.5 PROOF OF COROLLARY 4
Proof of Corollary 4 Recall from Theorem 2 that
EA(w(?) (2)) = bias + variance,
where the bias satisfies
bias = ((G + (I "1 (I- wow
bia8joint
(+ H w w
S bia8 joint -12"(E++)) H) w' * T) (I, w w'w"
bia8joint
(7+ [++#))lw* 1)
22

---

Published as a conference paper at ICLR 2026
For the variance,
1 G + +G
x² variance 2 ())+
+ H HH + Ex) Hg n [k + nLgg) -
1 n
((G (IG+H)([I H)
Gj G + G n
Gj Ge)e ) + (H, H + nHgg
π
+HE Σ
Gj -1 +G (,, "
(Hg Σg) Hx + n [Ie nE Hy,)
n
variancejosa joint + ( - G
+ +nGr) + (G.
- HH + + -
x variance joint
Σ iEJUX
Hi
+7
14
n
We know from Corollary that ( learning variance. The tail terms are also bounded by the joint learning variance. Thus we finish the proof.
+ nGyz) ) is bounded by the joint
+
B.6 PROOF OF COROLLARY 6
Proof of Corollary 0 Recall that
EA(w(?), bias variance,
where bias and variance with respect (1) satisfy
bias ((G' + H)(I - G)" H) + (I H)"), w'w"))
= - (1++
+ (1 - (1-)i)") (M: + de)w;??
Σ H) 2 + (I - H))), _ + nG
variance S So + H)
+ (G + H, -H Ex) H k + n Ig. + nE Hx=))
n
For the bias, notice that it satisfies that the quantity - (111111)((1--- (1)), "(u + A:) is the joint learning bias. For the remaining part, when 7 = #4 for μ4 1/n and 74 0 otherwise, have
? E 11)" (A: 72 (Ma 2x)00²²
Σ
p4)
2
(ms A:)
(A: + 14)2
>
14) u!:!²
iEI
>
#4 21) + \e)ws,
VEl
Σ 14) (1 (1-1)) + Xw) S biassoine
<
The second last line is because the function x(1 x)" decreases when x 2 1/n. As a result, the whole bias is bounded by the joint learning bias.
23

---

Published as a conference paper at ICLR 2026
For the variance,
0² variance
H)" 1 G + n
(I[G (
+ (G+ + H, HHg- 2 Hz + n(Ike + + - HHx)))
Eg)
n
=(G HI H)",
G n.z'))
1
n
+ Σ (ps + Ac)
1
A n(7 + );)²
Σ
(mi + Xi)
1
75
+
n144
(% + )z)2
11j4i (76 + ),)2
n(7 + 1/n]2
iEJNK
VEINKo
A
77
+ X:
Σ iEJ=KK
(m: + Ai)
12p44
(7ss + )1)3
n(7++ + 11)3
Σ (m+ + X;)
11p4 (7s+ + )₂)2
n(7 1/n)²
KEJAnKe
We know from Corollary that o² (² + HJ Hy", G + nG}=)) is bounded by the joint learning variance. Thus the quantity of focus is
(+) E
(m: + Xx)
7
A
Σ VEInKs
(us + Aa)
7/
A
1i/ls (7s + x;)
n(% + Xx)3
1244 (1i
>x)2
n(7 + 1/n)²
JEJnK
2
A
7? 7/p4 (76 );)2
Σ (Mi + Xx)
Σ (us + Az) REIenKe
n1/4; (74 + x.)2
+
m(* + X;)3
n(>x + 1/n)3
vEJanK
When 1/2 = Mi for pr 2 1/n and ,/2 = 0 otherwise, the above quantity satisfies
ur
12 nlu + 2,)a
A
(+) S Σ (mi + Xx)
+ Σ
(us +
Az)
+ n(u + 2;)3
ni/4s + 2x)2
ni14 + >;)3
SEJNK
VEInK.
+
Σ
(MS + Ax)
1
+
Σ
(#: + 2a) nds
nd;
iEI'NX
iEJoKK
< ΣΣ
Σ
Σ
Σ
(mps);+ + nd?) S 0z
variance joint
n
LEJnk"
kElnke
KEJ+nK
kEJAKk
(22)
The last line holds because of Cauchy-Schwarc inequality. Thus we finish the proof.
B.7 PROOF OF COROLLARY g
Proof of Corvllary 8 Again, recall that
(2(w(2))- bias variance,
where bias and variance with respect x (1) satisfy
bias = ((G + - (E H) 2
+ (I -H)"), un
= -1- (A: + +3
+ (1 (1-111)) 21)x) )(ms + >e)w,?
1 - -1 + n.
variance 0² (((G + H) 2 Σ H + (I H)"),
n
+ (G + H. 1 H + Σx) Hx + nlIge + -2 Hxz))
24

---

Published as a conference paper at ICLR 2026
For the bias, notice that the quantity E, (1) - (,) + >x)w²² is the joint learning bias. For the remaining part, when 7/8 = 14; for S and 7/2 = otherwise, we have
m (A, + 14)2 (1s + Ac)w??
E - (1))))" (A, + 7122 (ms 2,)u²²
(1 -44
ish
Σ (1 H?:u?
isk
Σ (1 14 (1) + dr)w;
isk
11) 21) + Xa)u² < biasjoint-
The second last line is because the function x(1 - x)" decreases when æ 2 1/n. As a result, the whole bias is bounded by the joint learning bias. For the variance, similarly to the proof of Corollary 6 the quantity of our focus is
(+) = E (ps + Ac)
A n(n + 1:)2 +
+
Σ
(as + Ac)
7
11/4: (7s + x;)2
11/6 (76 + x₂)2
n(n + 1/n1)
SEJNK
iEJnKo
Σ (1u + 21)
11/4s (Te + 2,)2
73
mxx+
n(* + 1,)3
A
Σ VEI-NKe
(us + Xz)
(1s + X₂)2
73
+
n(x + 1/n)3
A
KEITK
When 1/3 = Mi for i < and 9:- = otherwise, the above quantity satisfies
2
A n(H + +x)22
{+) Σ (Mz + Ao)
Σ iERe,iSh
(As + +x))
+ n(w + x,)2
n(m + Xa)2
nn(4 + 1₂)2
iKK
Σ
(o: +
Ac)
1
+
E
(us + Ne) -nA+ +
Σ
(Ms + Ao)
ndi +
Σ
(wo + Ar) nd:
ndi
kiik
iEJnK"
KEJONK
SEJONKe
1+//2
Σ n
Σ iERSiik
Σ
(mpa) + mZ3)+ +
Σ +
T
n
iEK,cck
ieJnk,i>k
KEK*
iEJONK
1
(variance
+ k
variance)osme).
The last line holds because of Proposition I Jand Cauchy-Schwarz inequality; in particular, the fourth term is because that
1/2
1/2
In
Σ
x
KERs k
[J]]
1/2
1/2
I n
n n.
k2a
Σ X²
S
-varianCe
n
k
icKe
Thus we finish the proof.
B.8 PROOF OF EXAMPLES
Proof of Example 5 Recall that
EA w (2))
bias variance,
25

---

Published as a conference paper at ICLR 2026
where bias and variance with respect x: (1) satisfy
bias ~ ((G + H] -
+ (I-H)") w'w"
= - (1-
-((-M)) 1z)"
(A: 76) + (1 - Ai)" (wi + de)w??
variance S < EE [ H) -2 + (I H))), G + nGJo)
n
+ + H, n
Hk Ex) H + n(Ike +g H
1. By Corollary we have
EA(w ()) 2
0² Σ
M4 n Σ -ide
n
iEJnko
2. By Theorem we have
1
(3) 2 (HΣ*
(
G
(G,
-Hx ~xx
n
n
A
IEI Hy
n(7 + X1)2
+
+
14
n( 2,}3
iek
72
1
3 n2 nn y + 1/n)² =1(1),
(7 + 1)²
since
x²
211) when y 22 1112 and 3 n (n+1/n/ 211) when y 7 11/2
3 (x+1)²
Proof of Example
Note that Σ has rank at most k. By Theorem we have
EA w (2)) 2 (G,
(gg-
+ Σg)
Hg
n(Tg- + +xxx)) -
n
A
A n(7 + 11/n3
Σ H
Σ M4
+
n(7i Xx)3
YEK
KEKc
Σ
n²(7 + 1/n)² = 2(11,
k+1
1SiS
since h << is constant and there exists at least one such that y; = 0 for 1 Sisk
C PROOF OF GAUSSIAN OCL
In this section, we present the full analysis of OCL in Gaussian design as briefed in Section 5
C.I UPPER AND LOWER BOUNDS
We present the lower and upper bounds of the OCL excess risk in the following theorems.
Theorem 18 (Lower bound). There exist constants by. bz, 0 for which the following holds.
Denote index sets J. that are defined as follows. Let Ju = {i 24 > 4}. Let μt = max{u :
Tip (G) 2 b2n), and define J == Jy. Similarly, let KK, = [i X > }}, x* max : rwy (H) >
b2n), and define K == K... Then if S bin and [K] S for every n c, for the OCL output
[T. it holds that
E (( (2)) min R1 bias variance
26

---

Published as a conference paper at ICLR 2026
where
bias 2 ||(
( n²
G +
(
Hir
2 + Igo) $ $001
n²
0² n
n²
Hx* + IKc) (tr Hz=)²
(G
n² (tr Gy.
variance
G, H
Hio +
G3e
Hr.)
n²
The excess risk bound in Theorem 18 consists of both bias and variance errors. The bias error arises from the (incorrect) zero initialization, which deviates from the optimal parameter w². The variance error is attributed to the additive noise e(()
y (ax(1) w²)
We complement the lower bound in Theorem
18 with upper bound.
Theorem 19 (Upper bound). There exist constants b1, b2, bs > O such that the following holds. Suppose Assumptions 7 and f hold. Denote two index sets J, K. [d], which satisfy [J] S bin,
rro (G) 2 ban and
K 017, rg: (H) 2 b2n, where
tr(Gr.)
tr (
ry- ||Grella Ti: (
Hg 1|2
Then, for the OCL outpur T it holds that
E (w (
min R1] bias + variance,
where bias and vari ance satisfy that with probability at least byc-
(Ir Go )
bias
w
+ 1|w=
| l2,
n²
GT
0² tr( + n²
(r(),
tr Genge
variance S n
Ink]
+ n² (tr G3
(tr Hg.)²
n(tr(GroHHg Hus) + m Ggl|GKgHHlll H22)12)
(tr (1))
| 112
tr( tr(Hg.)2
(tr Hg-)²
tr {
n²
(r((H)))
G
G3e
H&
Hg:
)r(
n²
Theorem 19 provides an upper bound for the risk and forgetting in OCL under the Gaussian dis- tribution setting. can be shown that when G H, the upper bound reduces to the single-task linear regression bound in Bartlett et al. (2020). thus matching the lower bound. However, for the non-degenerate case, there exists at least H Hx gap in the fifth term concerning the variance error. The gaps between the upper and lower bounds are due to technical challenges in obtaining an accurate variance bound under covariate shift in the Gaussian distribution. We leave the task of tightening these bounds for future work.
C.2 PRELIMINARIES
For both task f = 1,2, for any index E [d] and index set K C [d] where S 00, we denote
(1) = x (1) (1)
Recall that according to 15) and 17 the second moments of the output used in bounding the risks specified by
E- (1) ww (w (1) - w²) = -p(1)_ w w p(1) + +x x()) -1 (())-"X") T (xxax)
x()
= =p(1) w " T p(1) + (1) A1)) -2 x(1).
(23)
E2 (2) -2) p(2) E (w(() w0 (w0 w²)T p(())
+ 0 (x T x (2) (2) T (1] (x(2) T x(2))
= p12) E. (w w (w w²) p(2) + 0² (2) (2)
-2)
xx(1)
(24)
27

---

Published as a conference paper at ICLR 2026
where p(1) I - -x T x(1) "x(())
x T (1) and p(2) = (x X - x(2) T x(2).
Therefore, the risk that we are bounding in Theorem
18 depends on terms including p(y), pla),
X A - (1) and x T A (2))
C.3 SPLIT THE COORDINATES
Motivated by the previous benign overfitiigg literature, including Bartlett et al. (2020) and Tsigler & Bartleu (2023). we split the coordinates of the covariance matrix of each task G and H into "head" part and a "tail" part. The head part represents a low-dimensional space with relatively large cigenvalues, and the tail part represents a high-dimmeniooall space with relatively small eigenvalues. The following lemma is an algebraic property in the same spirit of the one in Tsigler & Bartlett
2023 and is used in splitting the coordinates of (1) A -2 x() and (() A (2) (2).
Lemma 20. For any nonempty index set K C [d].
X A =(Ix + x (1)
(1)
X
(1)
-1
(1)
A (1) -1
K
K
X 2 T A -1 =(I X
(2)
22)
-1 x(2 (2)
T (2) -1
A
K
K
K"
(2) T
Proof. We will give the proof for X
A
Notice that
A X X
(2)
X 2
(2)
Age
X
(2) (2)
and according to the Sherman therman-Morrison Woodbury formula,
X (2) (2)
(2)
(2)
K
K
-x
(2) (2)
(I+ X
T (2)
(2) -1 (2)
)-
K
= =X K g
(2) (2)
[
A
(2)
X K
(2)
-1
(2)
4
(1)
K
[TK X (2)
(2)
[Ig
X (2)
(2)
2
1 X
(
1
( -1
go
= (Ix + (2)
A
(2)
x(( K
(2) -1
K
go
The proof goes the same for the first task.
The next lemma splits p()) and p(2) into submatrices with respect to the coordinate split in each task.
Lemma 21. Fort = 1,2, for any nonempty index set K,
p(()
Q0))
[I + X
X
-1
A0 -1
X
pio = -Q1
P!!
A (0)
Ig.
X x Kc
A
(1)
x01
kc
kc
Proof. Notice that
p(t)_ = 1 - -
X T
A-1
X
A- xx
K
(0)
A
(0)
A - k0
()
Ko
K"
Ik
(T)
A x
(t)
-x((1
(1) 4 (
Ig. - x00
A A-
g
Therefore, by Lemma 20
pff
Ig
(t) T K
A x'xx( C
Ig (Ix + xx(' ( T
Age
(t) (0)
x
X
(1)
K
4
k=
(0)
x( (
[Ik + x (
(8) -1 x
(1) -1
K
Ag
We have finished the proof.
28

---

Published as a conference paper at ICLR 2026
C.4 MATRIX CONCENTRATION INEQUALITIES
In the following lemmas, we will present several useful concentration inequalities on eigenvalues of
several matrices used in proof. The following lemma from Lemma 9 in Bartlett al. (2020) and we rewrite it in our notation.
Lemma (Bartlett et al. (0200).
There exists a constant 2 such that for any PSD matrix J,
with probability at least 1 2e "/c,
tr - Jn||] S pm (ZJZ) S fr1 (ZJZZ S [
Notice that one verify that this is identical to the original lemma in Bartlett al. 2020 since in their notation, As = Zz H2 2 and H₃ = (x:
Lemma 23. There exist a constant b such that with probability at least 1 - -2- e, if (G) 2 bn and Tge H 2 bn, then
(1) tr G In S A Grr Ins
tr Hg. In S A Hg In
Proof. We will give the proof for the second task regarding H and A (2), Recall that A (2) Notice that since rg. HH) =
tr
H
ge
ge lla 2 bn. Therefore,
Pn (
(2) k'
Mn Z 2
H k. Z 22)
ス tr H ge
en Hg-
1
2
ir H
b
Also,
A1 (1 (2) = μ1 2 Hg 2 (2)
c(tr Hz ni Hz
1 + !) Hus.
The proof remains the same for the first task regarding G and A
Lemma 24. There exist constants by, b2, c >> O such that for any nonempty index set K. that satisfies [K| =: l & bin, then with probability at least 1 2e -n/c
tr S pm
pk+1 (A 2) S (r-).
Proof. Notice that for KK that satisfies ry. since
bn. also holds for every that (g--(s) (1) (H) 2 bn,
((Hzx-10) tr(H - A
2 bml Hg Hg-ll
2 I'n H Hg--(().)
(A As result, br(H)-) S pm ( (1)) S p1 (A (1)) S tr tr(H))). Notice that A A (1) is
a PSD matrix with rank at most [K] = k. Therefore, p., (2) 2 (4n gc (1) 2 tr(H ) also,
(2)
there exists linear space L with rank h: such that for all L. x. A (2) U (2)
A
(2) M1 (A ge (1) ||e/ 2. and thus pk+1 (A (2)) S μn (1) H)-).
(2)
The next two are commonly used concentration inequalities for random matrices with standard Gaussian rows.
29

---

Published as a conference paper at ICLR 2026
Lemma 25. For = 1, there exist constants b, 0 such that for any nonempty index set K. that
satisfies [K] = k v bn, then with probability at least 2e n/c.
nlg k
(t) T
(1)
3 nly.
Proof. According to Theorem 5.39 in Vershynin (010), for some constants C1, C22 for every & 2 0, with probability at least 2 eexp(-c) (-cp (-), one has
(vn-c2v C2
KK
t)² Σ ( 1 s 0 2 S (Vn + 02 +
K t)².
Substituting with vn/ea ca and we get the result.
Lemma 26. Suppose Z is matrix with i...d. rows in the Hilbert space THI with standard Gaussian entries. Then for any PSD matrix J, with probability at least - 2e n/je
tr tr(ZZZ)) n nn-tr(J..
Proof. Denote VAVT as its singular value decomposition. Notice that each row 2 H of Z is standard Gaussian. As result,
11
(2Z2 = Σ LAV 2;
where |AV z: 12 are independent sub-exponential random variables with expectation tr(A) and sub-exponential norms bounded by 01 tr()). Also note that tr(A) = tr(J). Therefore, according to Bernstein's 's inequality,
P P(( tr(ZZZ J 2 ttr) s eexp(-can min(t?.t))
By substituting with constant we get result.
C.5 UPPER BOUNDS
Bounding XTA Ax
Lemma 27. There exist constants bi, 12, by 0 for which the following holds. Denote two index
sets J, K. which satisfy [J] < bin, rJe (G) 2 ben and [K]] S bin, Tg- (H) 2 byn. Then with
probability at least - bg/c
x (1) (1) G
A
H
n
Proof. We will give the proof for the second task. By Lemma 20
(2)
=(Ig
x
(2)
(2)
X
2
X
2
(2)
(2)
2) T (2) -
A
H 1/2
H g
(2)
12
+
4
(2) -1
Z
(2) -1
H 1/2
K
1/2
(2)
(2)
22)
22)
2
12
-1
H 1/2
(tr Hg²) )
3
K
H 1/2 (H²)
zk
(2)
A
(2
H k
1/2
(tr Hg
H 1/2 (z
(2) T
A z
-1) 12)
-2
H 1/2
(tr Hg. 1
(tr Hxz)² )2
n
(tr
Hg.)
n²
1
H
17
The proof goes the same for the first task.
30

---

Published as a conference paper at ICLR 2026
Lemma 28. There exist constants b1, by, by > 0 for which the following holds. Denote two index
sets J, K which satisfy [J] & b1n, 73. (G) 2 byn and [K] S bin, rg. (H) 2 byn. Then for any PSD
matrix J, with probability at least 1 bye n/c.
(J, X
(1) Kc
A
(J, Gr.),
Gy:)
(J, x
(2) (2)
(2)
(J, Hk))
kc
tr Hge}
Proof. We will give the proof for the second task.
(J.
x (2) T
(2)
= H
1/2
JH
1/2 K
1/2
2 g
A 2 -2
Kc
ge
JH 2 (2)
(tr Hg.
) Hg
1/2
Hz)
n1 (tr Hge
(J, Hg.)
The proof goes the same for the first task.
Bounding PGP. We first introduce this well-known lemma.
A
B
Lemma 29. For any PSD matrix
we have
B
C
A
B
A
0
T
C
2
0
C
A
-B
Proof. Note that the RHS minus the LHS is
By Schur's Lemma, this matrix is
B T
C
PSD) if and only if
A BC-1 B 0,
AAA B
B C
which holds since
= because of Schur's Lemma.
Lemma 30. There exist constants bi, b2 by > 0 for which the following holds. Denote two index sets J, K which satisfy [J] bin, Tye (G) byn and [K| S bin, rg= (H) byn. Then for any PSD
matrix J, with probability at least 1 bn/e n/c,
tr Gy=
(J, p()) GP S
J,
G + +G.)
n
tr Hge H + H
(J, 2 Hp!2) $ J
n
Proof. We will give the proof for the second task. Recall that according to Lemma
21
p(()
(0)))
[Ix +
10
plt) = -Q10T D(()
K
Ige
A
(1(
K
K'
As result,
Pl2 HxP( + 2 Hg. Q(2)
-P
( Hg Qg ((2)
H P(2
2(2) H2(2)
((2) H (22) (2) H&P
p! H 2(2) (2)
H (2))
p (2) (2)
& + 2 H 22) T
2
P
2 H Hg.Pll
p() Q
Hx (2)
Therefore, in order to bound (22) H2(2), we have four terms consider:
31

---

Published as a conference paper at ICLR 2026
(2) HxPK (2) We note that
p H HkPP -Ix + X (2)
A go
X (2)
H 1g + X (2) T
AKt (3)
-1
X K
-1
-H (H&
2
A (2) (2) -2 H 1/2
3 H 1/2 (Z
(2)
K 2 3 H 1/2
(2)
(tr H )2
H
n²
. 9 Hg.Q T,
: Note that
0 = xx 2 -1 (2)
X
H
2
A (2)-1 x
Kc
K
-x01 (2)
(2) -1 (2)
H Z
2
1 -1 (2)
Notice that according to Lemma 22
1 (2)
12 S (1 + + H2
S
< Hx tr( Hg.) + n|| Hg= 122)
a
(tr Hg.) )2
62 bp
Therefore,
(12) Hg Q(2)
xx
(2)
A
(2)
H 2 (2)
A x 2)
(tr H )2
x(2)
(2)
(2) -2 x!2)
n (tr H ke) )2
n2
K
p(2) ge
Hge
p(2), ge
Note that
p Rc P(2) = (Ix.
x(1) x(2) A -1 x k'
H ge I ge x(9)
A x(() x2)
2. (Kg. + x (2)
(2))
H
(2)
(2)
(2)
ge
ge
and that
(2) ge
(2)
H
(2) g=
(2)
x (2)
x2)
A
(2) Kg
H2
2)
2
x
ge
g=
36
(tr H gg
2
A (2) 2
x 2
K0
Therefore, for any PSD matrix J.
(J, P H P
(2) < (J. Hge)
(tr Hzz)² 2
(J, x K
T
1 X (2)
n
(J. Hge)
32

---

Published as a conference paper at ICLR 2026
H Q T
Note that
Q(2)
Hx
X (2)
A X (2) Hg X (2) T
A x 2
x K"
A
A
(2) -2)-11
X (Ix + x2
2
H [Ix + X (2)
A
X
-1
K3
x2) T
A (2) X 2
(tr Hge) 2
X
(2) T
Ag
(2) (
X H
X 2 T (2) x 3
n²
(tr Hg.)
X
22
X
n²²
K
(tr Hg. )
(2)
(2)
Kt
Kc
1
(2)
Kc
K
n
Therefore, for any PSD matrix J.
(J. Q02) Hx (2)
1 (J,
x T
x
71
1 tr
(2) g
1/2 J 12 2 (2)
tr
1/2 JH 1/2)
(J, Hg.)
Combine these four terms and we get the result.
Lemma 31. There exist constants b1, b2, by > 0 for which the following holds. Denote two index sets J, K which satisfy [J] S bin, rye (G) 2 byn and [K]] S byn, rg- (H) 2 byn. Then for any PSD matrix J, with probability at least 1 - bye
(J, plo) pl GD!R) JJ,Gxc+ + (IGHHE'l1+ n(tr(Gr Hx.) + n Gr Hx-ll2) (tr Hx.)²
(tr xxx)²
H Hise) )
n2
Proof. We will give the proof for the second task. Recall that according to Lemma
21
p (())
(IK + X
(1)
T
(t)
x
x(2)- -1
(t) T
A
p(t) =
ge
Kc
Q T
P(0
T
A
x
Ig. X (0) T
A
x
(t)
gc
As a result,
pl G (2)
+ ( Gg. Q(2)
-P QkQ(22 - Q Gg. Pk
p(u)
&
P (2)
Gz - -02) GxP(2 kc
P G G + ((2) Gk
Kc
G
(2)
2 GR. O(2) T
22
K
k
P + 22) T
Gy Q
Therefore, in order to bound GP(2), we have four terms to consider:
K Gv G))( We note that
pf2) GkP!!)
=
-(Ix + X
(2) K
A Kc
X
(2) -
Gx X T T
A
42) (2) -1
X
K
-H (H²² + +Z
(2)
(
1/2
GgH
1/2
(H
2(21
(2)
(2)
z
"H 1/2
Ag
x
3 |1 1/2 GyHH 1/3
g
1/212- H 1/2 H 3.
+
(
A
(2)
-2 H 1/2
g
3 |
||2 H
1/2
(z Age Z
(2)
T (2)
2
H
1/2
Hg:)
3 |G
1|2
H k
n²
33

---

Published as a conference paper at ICLR 2026
Gr. T
Note that
(2)
A
-1
X
(2)
Gg. X (2)
22) x (
K
X
(2)
A
2
2
G g.
H
1/2 (2) ke
2 (2)=
x (2)
Notice that according to Lemma
22
ge K.
(2) H
2/2
1/2 Gr. K
K
K
(2)
S
<
ge
1/2 Gg. H 1 + n H
ge
1/2
Gg. H 1/2 112)
l12)
<c( + n Gg. Hg 112).
Therefore,
@ H Q12)
X (2) k
A (2)
1
A (2)
1
x (2)
K'
K
(tr(Gge Hg + n Gk
122 (2)
A 2 x
(tr(G Hgo)
Gg. H g= (22)
H
n
pl2 G.. Note that
Pic..
PK Gz = (Ize
x(21 g=
A
Gz (Rg. - -x(2) T T
A
K=
22 (Gke
+x
(2)
A X (2)
Gke x (2)
x
ge
and that
x (2) (2)
X
(2)
H x x2
A (2)
X (2)
x
A A
H 1/2 G
H
1/2
A x
Kc
ge
Ke
A -2
1 (2)
3 (tr(Gg H ) + n Gx H
(2) x K
Therefore, for any PSD matrix J.
Gz. (2) (J, 2
S (J, Gxe) + (tr(Gxe + [[e GRo
(2) go n
A -
Kc
S (J, Gko) + (tr(Gx=Hx)) + n Gk Hg.
JJ, H
(tr Hg.)
Q(2) T
Gg Note that
2 Q2)
X
22) R
A -1
X
(2)T
- x
K
K
x 2 (2) - x
(Ig x(2)
T (2)
x
(2) -1
Gg(x xx²2 T
A
x(2))-1 -1
(2)
(2)
Ke
ke
G 2
(tr H 2
2)
(2)
x
(2)
H 1
(2)
(2)
-
x
(2)
K
n²
K
K0
(tr Hg.) 2
|G
112
(2)
(2) (2)
(2)
X (2)
n²
(tr Hg 2
3 Gx
1/2
(2)
A
(2)
X
(2)
ge
K2
| k
1|2
1
(2)
x (2)
n
Therefore, for any PSD matrix J.
JJQ02 G $ )
Gx
1|2
1
(J, x x
n 1
||| < |x
-
tr
2
H 1/ JH 1/2 Z
n
x H &
112 In (H
g. JH 1/2
1/2
Hx 112 (J, Hg.).
34

---

Published as a conference paper at ICLR 2026
Combine these four terms and we get the result.
Now we can prove our main theorem.
Proof of Theorem 19
According to 23 and
E. (w)())
min R1] (G, Ee (w w (w0 -ww
w0 )²)
(G, E w') ( -w) -w'(("
+ to (p))
x A A (
= (G, p(2) w w p(1)p(2)
+ +² (G, plv) ( A x (1)(2)) + (G. x T A 103" x(
bias + variance.
For the bias part, with probability least 1 b/c n/c
bias = (G, w"" w (())p(2))
I (pli) GP(L)) w"w"" "
tr Gyo
GJ + Gje, w w0 wT))
(tr G3
||000 110"1, + +1120112 GG
n²
For the variance part, with probability least bge-n/e.
variance = 0² ((p() Gp!2)) x0) 1 (1)) + (G, (2) (2)- x(2)))
n(tr(Grg-Ha) Hg-) + n G G.gHHg-ll 12))
(tr Hg= )²
5 Ggc+ +
|G 112 +
1 + HKs
(tr Hz
n2
- +
1 - + [lrGra]= Gr= G5:) + G.
n
n
Hx=
Hk.)
0²
((1GHHH
n((u GRe Hxe) + G//Gre GReGHHellll Hxo(le)
(Ir (IrHKo) Hke)² H H
112t n²
(tr Hz.)²
n²
n² G H +
n² tr G
G
G1 n(tr(Gr.HHg) + nl| GR. Hg.ll2) (tr Hg.)²
+ +JJn
(trGp))
(tr Hg.) )2
(tr Gy.)
8²
(tr Gy.)2
Gk lle
n
(tr Hx=)²
(tr Hg:))
n²
H HK,, Gj + (Lr Gr))
G3:
n2
n
(n (rooor))))
Gj, H
H gc +J]n + n²
(tr H go tr HR)
(tr Gr))
Tr(Goond
tn (r. 1 +
+ JJ
+
(tr H
Gtr
Gkt Hg-) + G G (tr H
(tr Gj=)² trr Hg:)
+ (1GHHg
+
(tr Hg-)²
n
H
Hge, Gj + (tr G} Gy:
n²
We have finished the proof.
C.6 LOWER BOUNDS
Lemma 32. There exist constants by by, c > which which the following holds. Denote two index sets IK which satisfy []] byn, rye (G= > byn and [K| < byn, rig- (H) 2 bgn. Then, in expectation
35

---

Published as a conference paper at ICLR 2026
to ( x(2) T (2) (2)) a diagonal matrix. particular, for every n. > e,
(EX(1) A X 1a
2
Pi
G
+ px)
(2 A 2
X
i3 n
Hg
+ A;) -2
n
Proof. Recall that x² = ( (2) Therefore,
(2) (2) 2 X (2) (2) (2) A 2 (2)
Consider the off-diagonal terms of ( A Z(2)
(EZ A ( (2) (
Z
[1 = e 2
A (2) z
(2)
(2) 2 (2)
z;
2j
(2) X12 (2) (2) ( 2) f(
=
Observe that -(-2,))
(2) -I(2!)) and that 2 follows the standard Gaussian distribution,
which is symmetric across O. Therefore (z
which kills all off-diagonal terms of
Z A (
For the diagonal terms, applying Lemma 20 on the single-element index set {i} and we get
(2) Z )2)
e, (2)
A Z (2)
= = (2)
A (2)-2
-,71 (2,1 (2) T
A (2) (2)
-12
(2)
A 2 21 +2
T
A
-1 (2)
(01))-1
2,
2
2,
(2)
A (2) (2)
(1 1
(2)
(2)
(2)
2
According to Cauchy- Cauchy-Schwarr Schwarz, we have
1
(2)
12
(2)
T
(2)
(2)
(2)
(2)
A 2 22
Recall that ||=
2
2 ~ n with probability
(2)
2 2exp( -nc) according to Lemma 25
Therefore with probability!
2 expp /n/c).
(2)
T
A
-1
(2)
(2)
2
(2)
ji 2
2
n(1
X,
(2)
2 2)
1
1
+ +1,) 2
n
1
2
Now we examine 2, A A 12) _2)
(2)T
T
2; Let L be the top (n k) dimension subspace of A (2) -1)
Then
with probability at least - (nxc(-n/c)..
A (2)
| (2)
Pn (A (2)
(n - k).
1k++1 (A (2)
n
(25)
Lr H
36

---

2 is increasing function, now we know that with probability at least 1 tr Hg= Published as a conference paper at ICLR 2026
Since ( + 2,)-² 6 6exp(-11 -nn/c).
(2 A (2)
+ xx:)
n n
Since (Z T A 2(2) ii 2 0, by taking expectation and multiply by X on each side we get the result.
Lemma 33. There exist constants bj, b2. for which the following holds. Denote two index sets J. IK which satisfy [J] < bin, vr-(G) 2 ban and [K] I bin, Ti- (H) 2 b2n. Then for any PSD diagonal matrix J, for every n c
(tr (tr
E(p()) jp(i)) x J.
Gj +Iy.)
n
Eppl Jp!2) x J.
n22 (
(Lr )² Hx² Ikc
Proof. Suppose J = diag(o:) We examine all diagonal elements of Jp(2).
(p Jp(2) H E 2 2.
As result, we only need to analyze the lower bound of p(2) According to Lemma 21
p
(1 + xr
(2)
(2)
(1 2;2 (2)
(2)
)
(2)
z;
Recall that in (55) have that with probability at least exxp(nn/c).
n
A
x tr Hg.
Therefore, with probability at least 4exp( -n/c).
nds y-2
(p(2) sz 2 2o. (1
tr Hy:
(tr H )2
Φi (1
n²
Since Jp(2)) 2 0, by taking expectation we get the result.
Theorem 34. There exist constants bı, b2, e >> 0 ffror which the following holds. Denote index sets IK that are defined as follows. Let Ju i: p4 2 p. Let μt max Tig (G) byn}, and define J Ju.- Similarly, let K2 i: X X X maxf> rgg (H) dyn), and define K. Kx.. Then if [] & byn and [K] < bin, for every
0² (
(tr Hx:)
n²
n²
Hgs )
E. RR (w (2)) min Rı
H Izo, Is (tr (trG
Gt. +
G. H
+ (tr Hg.)
n
n²
Proof. According
23 and 24
Eg (w - min R, = (G, Eg (:(
w0 ww) -w)
= (G, F w0 (w)l)) - w²) TT(2)) + o x(2) (2) 222)2x((11)
(G, p(2) wi ((())p(2))
+ + ((G, p() (1) 1 2x (1((2)) + (G (2) 2 -
bias variance.
37

---

Published as a conference paper at ICLR 2026
We give the bias lower bound in the expectation form. For the bias part.
Ebias GE p(2)p()) w " T p(2))(2))
= Ex (xx1)) p(2) GP((2) p(u) " w p(1))
E Exiop(i))
(tr Hx=)
H + Ige
p(1),
w www"
n²
(tr GT))
(tr Hge)
Gj + Is:
H Ige
n²
n2
Gg+)
) (tr
(tr Hg.)
Gj I5o)
H Iko)
w 22-
"
n²
n²
We also give the variance lower bound in the expectation form.
Evariance
o² ((EP(22)
Gp(2)
(X
A 2
(11) + (G, EX (2) A (x(31))
G
H (1r
+ Ige H K
Gj + Grr
n²
Gr
G, H n²
+
(tr Hy. )2 Hxc)
n²
n22
o²
(tr H
n²
Hxx Ige, Ij +
Gf. +
G, H
+
Hg
n²
(Ir Hg.))
We have finished the proof.
C.7 PROOF OF EXAMPLES
Proof of Example ting.
We examine I.. [R] - R[((")] to represent the joint risk and forget-
1. By Theorem 18 we have
variance 2 0² G. H H1
#1
=.
n
n
A
As a result, (1(w(()) R[ww") (1)
2. By Theorem
18 we have
σ² (G.
n²
variance 2
Hg.),
(tr Hx.) )2
where K (i: A > )3 and the choice X let K satisfies tr (x() H 1|2. By definition of H, K {i:12 }²} where i satisfies i* log it t n. Therefore,
σ² G
Eizn Mede (Eig 2i)
variance
Hg²)
Ei2 i log a-88 i
n.
(Eiiio i-1 log -8 -0))2
2 n 8- log log -a-B i
~ log 9 8-a-1
> log 9-a-1 n.
(log (log-B+1 jo)²
S+1
As a result, E (w) R, Ri(w") (ww))) = Ω (log8-a-1,) n).
38

---

Published as a conference paper at ICLR 2026
D PROOF OF EXTENSIONS
D.1 PROOF OF CL IN THE NTK REGIME
We set up 2-task CL setting in the NTK regime. Consider two data distributions D1, D2. For each task, n individual samples (a{ y/ >= (z) ,y )² are sampled independently from D,, D2- The goal is to learn neural network model fw (z) specified by parameter w E R Lee et al. 2019) proved that under the NTK regime, neural networks evolve as linear model:
())n
(z) f (x) (x)
V w (0) (x), (ww.w) w
the initial weight. This formulation implies that the feature map (bx=) = / (x) E Rd is constant over time.
where (-) denotes the model with w (0)
Under this setting, we consider the convergence of GRCL with gradient descent (GD), consistent with the setting of Section 3.2 in Bennani al. (2020) In this algorithm, we optimize the following training loss for tasks = 2;
L (2wx!)) Σ (Z) (1) - 1),
(26)
i=1
(2( (2)) = (f. (z) ( y, ) + |w w 1
(27)
where Σ is the regularization matrix, t is the GD iteration number, and (1),, w(() is the limit of w!) (1) w respectively. The update rule of GD for tasks = 1, that
w 7+ ( V un (w.."))
We have the following theorem connect GRCL in this NTK regime with linear model: Theorem 35 (Continual Learning in NTK as linear GRCL). In the NTK setting, with an infinites- imally small 1, GRCL outputs un such that:
(1) (db (((
( (x(1)) ((x(1) (1 ((00
(x()) - (w w (1)) || w 12.
w arg min
Proof. By putting 12) into 26 and 27 and computing the first-order optimality condition of (26 get the result.
This theorem essentially reduces the GRCL in the NTK setting to the linear GRCL algorithm in ( providing that we give the following inputs x', y the linear model:
= ((x(()) (1) x (2) = (())
(1) ( - (z0) y (2) = y - 100-((0 (x) (2)).
(28)
Assumption 5 (NTK data distribution condition). We assume that (1) (1)) (1))) ((2) satisfy
Assumptions
Note that in this case, the covariance matrices G = Epa (c((( (d(x H
Epcc (()(())))(())))) (2) are actually the Hessian matrices of the model. Now we have our main results in the NTK regime:
Theorem 36. Suppose Assumption holds. Then for the GRCL output (2) in the NTK regime, holds that
EA(w(2)) = bias variance,
where the bias and variance satisfy (6) defined in Theorem
Therefore, our main results in Theorem 2 and its collollaries and messages still hold the NTK regime and can be applied in general neural networks.
39

---

Published as a conference paper at ICLR 2026
D.2 PROOF OF MULTI-TASK CL
We set up a multi-task CL setting with linear regression problems. Forks = 1,..., where T > is constant, consider the data distributions (D(9), )-
For each task t, n samples (ax y (8) )==
are drawn independently from the task distribution. The goal is to learn a model w to minimize the
joint population risk R(w) = E, Rr(w), where R₂ R,(w) := Epce (y T a)²² is defined in
same way as Definition in the paper. We retain the shared-optimal assumption in Definition 2 For
the covariance condition in Definition[] [ we denote:
Definition 3 (Covariance conditions). diag(14 (2)).
(()) = Epit xx ] (iiag(A()), G( Σ H
One can verify that () = G and H is consistent with the original Definition Under this multi-task setting, consider the GRCL algorithm, in which the regularization matrix series ())) can adapt the number of task i grows:
w (1) = (1) - (
(2;
w (1) arg min ur π
1 x "wl2 + 1|1
(1 E
25
Assumption 6 (Multi-task data distribution condition). We assume that Assumptions I 2
3
hold
fort I....,T.
Proof of Corollary [7 14 already holds due to the original Corollary 6 Now we assume (14) holds for task t such that 3 I S T. In this case, according to 18). we have
We use induction to prove this corollary. Consider EA(w())). For 2,
Es(w(t-1)) = (G(t-1), (io,,),E(w(tter ((w - S (A(w)oin)).
Then for task 1, reall that p = (1) x(O) + (11 According to Eqs. (12) and (15),
Ed(w()) (G(t-1) + (()) E(w w (ww - w²)T)
= EE + (1) pd (1 - w
w²) P!()) (w) (1
+ + t + H EX (2) x + nz)1)) (1) x (1)).
For the first term, notice that according Lemma 13,
(,( ( ( (G(t-1),
H (((t-1) H(t)) 2 (I, H() s G(t-1).
Therefore,
E, (t (()), ( E(w - w²) -ww)(w((t1) -w²) Tp(() (()))
(GL--) ((w(t-1) w (w w0' T)
2 EA(W))int).
For the second term, following it is also bounded by variancejoint thus bounded by EA (. Adding them up and get EA(w(!)) S (E((Wooot))
By induction, we get the result.
Corollary [T extends the message in the paper that with sufficient memory and appropriate regular- ization, GRCL can match the performance of joint training in multi-task CL. Combining with the existing Example 7 where low-memory CL behaves poorly, we deliver the message that there is a provable memory-statisties trade-off in multi-task CI..
40

---

Published as a conference paper at ICLR 2026
E EXPERIMENTS
E NUMERICAL EXPERIMENT SETTINGS
We consider CL problem instance adapted from Wu et 2022) and Li al. the eigenvalues for G and H are 8 < 1, the CL problem P(k)
(003). Specifically, For given integer ko and small value
(1/2*- ]d
H)H specified as follows:
1 2 2
1
2F )...... (29)
G diag
H diag
2h-1 1o-T.
2h- 2k-2
w (1,2
0²=1,
We choose P(15) for our illustration. It is evident that G and H differ in their 1op-15 eigenvalues. For the GRCL algorithm, the regularization matrix Σ is specified by the rank-k approximation of the empirical covariance matrix of the first task (1) ( We then test this problem instance and compare the GRCL risk convergence with OCL and joint learning results.
E.2 NEURAL NETWORK EXPERIMENTS
In this section, further verify theoretical results with experiments practical CL datasets using neural networks, complementing the numerical experiments Section 4.2 We use Permuted MNIST and Rotated MNIST, two CL benchmark datasets widely used in the literature (Kirkpatrick) et al. 2017 Farajtabar et al. 2020). In both experiments, we consider two-task problem: the first consists standard MNIST digits, while the imput of the second task transformed from NMINST. In Permuted MNIST, a random shuffle of pixels is applied on the 28 x 28 images of the original MNIST hand-written digits to create the second task; in Rotated MNIST, instead of random shuffle, rotation of an unknown fixed angle is applied.
In analogy to the OCI and the GRCL algorithms that we theoretically analyzed in Section we ex- amine three algorithms: vanilla training without regularization, full regularization, and PCA-based low-rank regularization.
vanilla training, standard optimization performed with the Adam opti- mizer during the second task training. In the full regularization algorithm, the Hessian is computed at the end of the first task training and is used as the regularization matrix in training the second task. In the low-rank regularization algorithm, instead of the full Hessian, PCA with a predetermined rank is performed on the saved regularization matrix. We examine different network architectures for the two datasets due to their different properties. For Permuted MNIST, with 4 residual blocks, each consisting of 2 MLP layers with a hidden size of 40. The model input dimension is 196 (the MNIST figures are resized 14 14), and the output dimension 10. For Rotated MNIST, we use a CNN-based network with a LeNet5 structure.
MLP-based model
The experiment results are shown in Figure We make the following observation: In both settings, the vanilla algorithm suffers from significant drop in average accuracy due to catastrophic forget- ting. As the memory size increases, the regularized algorithm performance improves and achieves reasonable empirical result. This result corresponds to the memory-statistic tradeoff demonstrated our theory and numerical experiments.
41

---

Published as a conference paper at ICLR 2026
0.98
0.96
Average Accuracy
Accuracy
0.94
0.92
Vanila
0.7
Vanilla
Memory
Memory
0.88
Rank 50
Rank
0.6
Rank 500
Rank 50
Rank
Rank
10
15
20
25
30
35
40
5
10
15
20 Epochs
25
30
35
40
(a) Permuted MNIST
(b) Rotated MNIST
Figure Average accuracy across previously learned tasks on a Permuted MNIST and (b) Rotated MNIST after each epoch of training for the vanilla algorithm without regularization, the regularization-bant method with full Hessian, and with low-rank regularization. In both experi- ments, the Adam optimizer with learning of 10-1 The moving average parameter is = 0.25 and the regularization coefficient 10t for all algorithms.
42


Reviews:
[{"review_id": "bZ68KMjO3r", "reviewer": "Reviewer_YoRz", "summary": "The paper studies continual learning with two linear-regression tasks under random design. It analyzes three algorithms: OCL,l2-RCL and GRCL and establishes sharp upper/lower bounds on the joint excess risk of three lags. It shows that GRCL algorithm that uses a curvature-aware quadratic penalty (a PSD matrix) derived from the first task to mitigate catastrophic forgetting.", "questions": "Above", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper considers an interesting problem in theoretical continual learning community.\n2. The paper is well-written and well-structured.\n3. The results driven by the data covariance are novel and interesting.", "weaknesses": "1. Several related work about theoretical CL are not discussed, for example:\n\n[C1] Banayeeanzade et al., Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learning, TMLR 2025.\n\n[C2] Zheng et al., Towards Understanding Memory Buffer Based Continual Learning, 2025.\n\n[C3] Ding, et al. Understanding forgetting in continual learning with linear regression. 2024.\n\n[C4] Wen, et al. Information-theoretic generalization bounds of replay-based continual learning. 2025.\n\n[C5] Wan et al. Understanding the Forgetting of (Replay-based) Continual Learning via Feature Learning: Angle Matters. 2025.\n\n2. Two tasks in linear setting is simply and not enough to discuss the paper's findings. If it is only two tasks, what is difference between transfer learning vs continual learning? Based on equation 2, 3, it seems to be hard to extend the paper's results to $M$ tasks setting.\n\n3. How to understand the “tail” of the distributions in equation 5? \n\n4. Can the author decompose their results to show the separate loss evaluations on each tasks? Since the main results are based on joint learning.\n\n5. The results in section 4.2, why only give the example instead of general results. Since the paper investigates the impact of constrained memory while it still assumes that the data covariance ($\\mu_i$) is related to the constrained memory setting ($k$). Can the authors explain more?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ib84jU37zt", "reviewer": "Reviewer_RwWP", "summary": "This paper studies the trade-off between statistical performance and memory cost in Continual Learning (CL) with structural regularization methods. Focusing on a two-task linear regression model under a random design setting with covariate shift, the authors propose a Generalized ℓ₂-regularized Continual Learning (GRCL) algorithm and establish upper and lower bounds for its joint excess risk. The theoretical analysis reveals that Ordinary Continual Learning (OCL) without regularization and simple ℓ₂-regularized CL (ℓ₂-RCL) suffer from catastrophic forgetting in certain tasks, but a well-designed structural regularization (GRCL) with sufficient memory can mitigate forgetting and achieve statistical performance comparable to joint learning. The theoretical results are validated through numerical experiments and extended to neural networks in the Neural Tangent Kernel (NTK) regime.", "questions": "(1)\tCould you discuss the practical implications of the commutativity assumption (Assumption 2)? How might your conclusions change when this assumption is violated in real-world datasets?\n\n(2)\tWould you consider adding experiments on more challenging benchmarks (e.g., CIFAR-100, ImageNet subsets) to better demonstrate the practical relevance of your theory?\n\n(3)\tHave you compared GRCL with strong baselines from other CL paradigms (e.g., experience replay, expansion methods)? Such comparisons would help position your method in the broader CL landscape.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "First to provide rigorous theoretical formulation of the memory-performance trade-off in continual learning, with a unifying GRCL framework.\n\nQuality:\n\nSolid theoretical analysis with matching bounds and well-constructed examples. Clean extension to neural networks. Experiments properly validate theory.\n\nClarity: \n\nWell-structured and logically coherent. Core ideas are clearly communicated despite some heavy notation.\n\nSignificance: \n\nEstablishes valuable theoretical foundation for continual learning. Provides practical guidance for algorithm design and opens directions for future work.", "weaknesses": "(1)\tLimited Empirical Validation\n\nThe experimental validation, while supporting the theoretical findings, remains somewhat limited in scope and practical impact:\nNeural network experiments are conducted only on small-scale MNIST variants (Permuted/Rotated MNIST). To better demonstrate real-world relevance, experiments on more challenging benchmarks (e.g., CIFAR-10, Split CIFAR-100) would strengthen the claims.\nThe comparison focuses primarily on methods within the structural regularization family. Including strong baselines from other CL paradigm would provide a more comprehensive assessment of where GRCL stands in the current landscape.\n\n(2)\tStrong Theoretical Assumptions\n\nThe analysis relies on several strong assumptions that may limit practical applicability. For example, the commutativity assumption of covariance matrices (Assumption 2), while mathematically convenient, rarely holds in real-world datasets. The authors could discuss how violations of this assumption might affect their conclusions, or provide empirical studies showing the theory's robustness to approximate commutativity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gjv1vzUELU", "reviewer": "Reviewer_GXcu", "summary": "The paper studies continual learning (CL) for two linear regression tasks under random design and covariate shift. It analyzes a generalized $\\ell_2$-regularized CL algorithm (GRCL) that, during the second task, penalizes deviation from the first-task solution using a user-chosen positive semidefinite matrix $\\Sigma$. The work derives matching upper and lower bounds on the joint excess risk and shows a quantitative trade-off between memory and statistics: richer $\\Sigma$ (higher memory, measured by the number of stored vectors/eigen-components) yields lower excess risk. It proves that ordinary CL (OCL) and isotropic $\\ell_2$ regularization can suffer catastrophic forgetting, while an appropriately chosen $\\Sigma$ can match joint training performance. Extensions include lower/upper bounds for Gaussian design, a memory-risk curve with top-$k$ regularization, a discussion of NTK-regime neural networks, and small-scale experiments illustrating the theory.", "questions": "1. Strength of assumptions and scope.\n\n- Regarding the optimal parameter, can you add a proposition or appendix note quantifying the error when optimal parameters differ (the text hints at triangle-inequality arguments) and provide an explicit bound to show robustness to small misspecification?\n- Can you relax \"commuting $G, H$\" via a perturbation result? For example, if $||G H-H G||$ is small, how do constants in the bounds inflate? Even a qualitative theorem would help.\n\n2. From one-hot to Gaussian (and beyond).\n\n- Can you provide at least an upper bound for GRCL in Gaussian design under mild spectral decay (e.g., polynomial/log-corrected tails)? Even if looser, it would connect the top-k message to the more common model. I think you can use Theorem 4.2 of Zhao et al. (2024, ICML) to do this.\n- Can you include a short \"what breaks\" paragraph? For example, I was curious which steps in the one-hot proof fail for Gaussian, and what tools are perhaps needed?\n\n3.  Model selection for $k$ (or $\\Sigma$).\n\n- Can the authors propose a simple criterion?\n- The authors mention CountSketch can be used to specify the memory utilized in GRCL. Can the authors state a data-driven estimator for $\\Sigma$ with guarantees? (Formalize an estimation step and propagate its error into the risk bound).\n\n4. Positioning vs. prior theory.\n\n- Can the authors add a comparative lemma or table? For example, which settings (fixed vs random; isotropic vs structured $\\Sigma$; with/without matching lower bounds) each prior paper covers, and what this paper adds.\n\n5. Memory accounting and implementability.\n\n- Can the authors first define “memory” precisely, and then provide worked examples that show how to compute and compare storage costs for different choices of examples? For example, diagonal $\\Sigma$, block-diagonal by layer, K-FAC with factor sizes, and a sketch-based $\\Sigma$ showing CountSketch dimension versus error. I think this effort would close this gap and make the proposed trade-offs operational.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper makes a clear and useful move: it treats continual learning through a memory-statistics lens and asks how much structure one must store to avoid forgetting. This framing is fresh and concrete. The theory then backs it up with matched upper and lower bounds. The main theorem decomposes risk into bias and variance terms that depend on the regularization matrix. This gives readers a direct way to see why naive OCL or an isotropic ridge can fail, and why a structured $\\ell_2$ penalty helps. The \"top-$k$\" view of the matrix is especially appealing. It turns memory into a dial that one can set and shows that risk falls toward joint training as the rank grows. The examples are well chosen: they show catastrophic forgetting when the method ignores key directions, and they show how the proposed structure fixes that. The writing of the core results is also clear. Assumptions are stated, quantities are interpretable, and the logic builds in a steady way.", "weaknesses": "1. Strength of assumptions and scope: The shared optimal parameter and commuting covariances (diagonalizable in the same basis) simplify analysis but may be restrictive in practice.\n\n2. From one-hot to Gaussian (and beyond): Main sharp results are for one-hot design; Gaussian design coverage is partial (OCL bounds, phenomena via Example 9), while GRCL bounds are deferred.\n\n3.  Model selection for $k$ (or $\\Sigma$): The theory shows performance as a function of $k$, but there is no principled selection rule.\n\n\n4. Positioning vs. prior theory: The paper cites related work, but the novelty over Li et al. (2023) and Zhao et al. (2024) could be crisper, especially regarding random design + memory-explicit lower bounds.\n\n\n5. Memory accounting and implementability: Memory is \"number of vectors to store\" or \"rank of $\\Sigma$, but concrete byte/parameter costs are not spelled out, especially for neural nets where curvature is layer-structured.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)