Understanding the Dynamics of Forgetting and Generalization in Continual Learning via the Neural Tangent Kernel
Authors: Guodong Zheng
Peng Wang
Shengchao Hu
Quan Zheng
Li Shen
Venue: ICLR 2026 Poster (2026)
Keywords: Forgetting
Generalization
Continual Learning
Neural Tangent Kernel
Primary area: learning theory
TL;DR: We analyze the training-time dynamics of forgetting and generalization in standard CL within the Neural Tangent Kernel regime

Abstract: Continual learning (CL) enables models to acquire new tasks sequentially while retaining previously learned knowledge. 
However, most theoretical analyses focus on simplified, converged models or restrictive data distributions and therefore fail to capture how forgetting and generalization evolve during training in more general settings. 
Current theory faces two fundamental challenges: (i) analyses confined to the converged regime cannot characterize intermediate training dynamics; and (ii) establishing forgetting bounds requires two-sided bounds on the population risk for each task. 
To address these challenges, we analyze the training-time dynamics of forgetting and generalization in standard CL within the Neural Tangent Kernel (NTK) regime, showing that decreasing the loss’s Lipschitz constant and minimizing the cross-task kernel jointly reduce forgetting and improve generalization. 
Specifically, we (i) characterize intermediate training stages via kernel gradient flow and (ii) employ Rademacher complexity to derive both upper and lower bounds on population risk. 
Building on these insights, we propose \emph{OGD+}, which projects the current task’s gradient onto the orthogonal complement of the subspace spanned by gradients of the most recent task evaluated on all prior samples. 
We further introduce \emph{Orthogonal Penalized Gradient Descent} (OPGD), which augments OGD+ with gradient-norm penalization to jointly reduce forgetting and enhance generalization. 
Experiments on multiple benchmarks corroborate our theoretical predictions and demonstrate the effectiveness of OPGD, providing a principled pathway from theory to algorithm design in CL.

Published as a conference paper at ICLR 2026
UNDERSTANDING THE DYNAMICS OF FORGETTING AND GENERALIZATION IN CONTINUAL LEARNING VIA THE NEURAL TANGENT KERNEL
Guodong Zheng" 1: Peng Wang": Shengchao Hu', Quan Zheng', Li Shen 2:31
Huazhong University of Science and Technology
2 Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)
3 Shenzhen Campus of Sun Yat-sen University
4 Shanghai Jiao Tong University
zhenggd9 430 gma com, mathshenl aetthhenni@gmaiiin com
ABSTRACT
Continual learning (CL) enables models to acquire new tasks sequentially while retaining previously learned knowledge. However, most theoretical analyses focus simplified, converged models or restrictive data distributions and therefore fail to capture how forgetting and generalization evolve during training in more general settings. Current theory faces two fundamental challenges: (i) analyses confined to the converged regime cannot characterize intermediate training dynamics: and (ii) establishing forgetting bounds requires two-sided bounds the population risk for each task. To address these challenges, we analyze the training-time dy- namics of forgetting and generalization in standard CL. within the Neural Tangent Kernel (NTK) regime, showing that decreasing the loss's ' Lipschitz constant and minimizing the cross-task kernel jointly reduce forgetting and improve general- ization. Specifically, we (i) characterize intermediate training stages via kernel gradient flow and (ii) employ Rademacher complexity to derive both upper and lower bounds on population risk. Building on these insights, we propose OGD+, which projects the current task's gradient onto the orthogonal complement of the subspace spanned by gradients of the most recent task evaluated all prior sam- ples. We further introduce Orthogonal Penalized Gradient Descent (OPGD), which augments OGD+ with gradient-norm penalization to jointly reduce forgetting and enhance generalization. Experiments on multiple benchmarks corroborate our theoretical predictions and demonstrate the effectiveness of OPGD, providing principled pathway from theory to algorithm design in CL.
INTRODUCTION
Continual learning (CL) trains models on a sequence of tasks with the objective of maintaining strong performance across all of them. Unlike conventional training paradigms that operate on a fixed dataset single task, CL typically faces non-stationary data streams and complex task sequences. A major challenge in this setting is that models often experience substantial performance drop previously learned tasks when adapting to new ones. This phenomenon, known as catastrophic for- getting (McCloskey & Cohen 1989 McClelland et al. 1995). arises when learning new information interferes with or overwrites prior knowledge.
Despite the considerable empirical success of numerous approaches in CL (Chaudhry et al.) 2019a Farajtabar et al.) 2020: Dohare et al.) 2024), rigorous theoretical understanding remains limited. Recent studies on the theory of forgetting and generalization error in CL. have mainly focused on linear models and often assume restrictive data distributions, such as Gaussian distributions (Evron) al. 2022 Lin et al. 2023) Banayceanzade et al.) 2024 Li et al., (025b). While the linear regime provides explicit characterizations of lorgetting and generalization error, it is not suitable for more
"Equal contribution
1 "ooresponniig author

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of our results with Bennani al. 2020). Doan et al. (021) and Karakida Akaho (0022). We summarize whether each work provides theoretical bounds on forgetting and generalization error. and whether it analyzes training dynamics or includes ridge regularization.
METHOD
FORGETTING BOUND
GENERALIZATION ERROR
TRAINING DYNAMICS
RIDGE REOULARIZATION
BENNANI ET
2020
X
DOAN 2021
KARAKIDA AKAHO
2022
X
OURS
X
general models or the non-stationary data streams typical in CL. In contrast, the Neural Tangent
Kernel (NTK) regime (Jacot et al. 2018 Lee et al. 2019 Arora et al. 2019b) enables the analysis of
more general models without being restricted to special distributions. However, existing NTK-based
analyses (Bennani et al., 2020, Doan et al. 2021 Karakida & Akaho 2022) primarily focus on converged models and therefore cannot characterize the behavior at intermediate training stages.
The main challenges in the theoretical analysis of both forgetting and generalization error are twofold: (1) existing analytical approaches primarily focus the converged regime, either by obtaining explicit solutions from the loss function or by assuming asymptotic convergence as the number of iterations approaches infinity. However, the dynamics of forgetting and generalization error during training remain underexplored from a theoretical perspective. (2) bounding forgetting requires establishing both upper and lower bounds on the population loss for each task. In particular, while generalization bounds typically provide only an upper bound on the population loss, the forgetting metric involves averaging the discrepancy between the performance of the final model on previous tasks and that of the models trained directly on those tasks. Consequently, deriving two-sided bounds on the population loss in CL remains significant challenge.
In this work, we theoretically analyze forgetting and generalization in vanilla CL under the NTK regime. We derive bounds at intermediate stages of training, revealing two key insights: (1) both forgetting and generalization error increase with iterations when the Lipschitz constant of the loss with respect to predictions exceeds a threshold, while reducing it consistently mitigates these effects; and (2) decreasing the magnitude of the cross-tase task kernel similarly alleviates forgetting and improves generalization. To support this analysis, we characterize model dynamics via kernel gradient flow and define forgetting in terms of population loss, deriving upper and lower bounds using Rademacher complexity. These tools allow us to rigorously study the evolution of forgetting and generalization at intermediate training stages. Building on these theoretical findings, we propose OGD+, which projects the current task's gradient onto the orthogonal complement of the subspace spanned by gradients from last task evaluated on all previous samples, reducing forgetting and improving generalization compared to standard OGD. We further show that controlling the Lipschitz constant in OGD+ enhances these benefits. Inspired by this, we introduce Orthogonal Penalized Grudient Descent (OPGD), which integrates gradient-norm penalization into OGD+ to jointly reduce forgetting and improve generalization. Finally, we empirically validate our theoretical predictions and demonstrate the effecctivness of OGD+ and OPGD on Permuted MNIST, Rotated MNIST, and Split CIFAR-100, providing a principled pathway from theory to algorithm design in CL.
Our main contributions can be summarized as follows:
We derive bounds on forgetting and generalization error at intermediate stages of vanilla CL under the NTK regime, showing that both can be mitigated by reducing the Lipschitz constant or driving the cross-task kernel toward zero
We define forgetting and generalization error in terms of population loss, and provide upper and lower bounds via Rademacher complexity under the NTK regime. Furthermore, leverage kernel gradient flow to analyze model dynamics at intermediate training stages.
Building on these insights, we propose OGD+, which projects the current task's gradient onto directions orthogonal to the subspace formed by the last task's 's gradients on all previous samples. We further introduce Orthogonal Penalized Gradient Descent (OPGD), which adds gradient-norm penalization to OGD+ for tighter theoretical bounds.
We empirically validate our theoretical predictions and demonstrate the effectiveness of OGD+ and OPGD on Permuted MNIST, Rotated MNIST, and Split C00.
2

---

Published as a conference paper at ICLR 2026
Related Work. Due to space constraints, comprehensive literature review is provided in Ap- pendix B In particular, Table presents a detailed comparison between our theoretical results and prior studies (Bennani et al. 2020 Doan et al. 2021: Karakida & Akaho 2022).
2
PRELIMINARY
2.1 PROBLEM SETUP
We consider the standard CL setting with T sequential training tasks. For any positive integer n, denote [n] (1....,R.) For each task f [7]. let D, denote its data distribution, and let S, {X Y7 be the corresponding training dataset drawn i..d. from D,. Here, Xx = (a+ (#.,...,,²² E Rx is the feature matrix containing n, samples of d-dimensional feature vectors, and Y, E IR is the associated label vector. The model trained on task T is denoted by fr. parameterized by 67. Its state iteration t, is written as ftr and the final model after training is denoted by fü where training stops at iteration t, = t.. The model for task + 1 is initialized from final parameter of task т, that is, 0 0
n2).
Let (a(()) y) be loss function that quantifies the discrepancy between the model prediction f(x) and the true label y for a given sample yz. We now introduce two fundamental notions in learning theory: the population loss and the empirical loss.
The population loss with respect to the distribution D, defined as:
Lp. E (a,, )-D,, (et-.._[(((( [e(f(x)), V+)].
(1)
The empirical loss over a dataset S, { (xrt, w1 M i.idd. drawn from D, is defined as:
Ls, (f) = 1 Σ ((f(r))
(2)
As shown Lopez-Paz Ranzato (2017); Lin et al. (2023), forgetting and overall generalization
error can be defined in terms of the population loss as follows:
(1) Forgetting. This metric quantifies the degradation in performance on previously learned tasks after training the current task at iteration byp:
L., (
1 Fr T-
Σ
Lo. (T))
(2) Overall generalization error. This evaluates the model's generalization performance at iteration tr by averaging its population loss across all tasks:
T
G2 T
Σ (fT)
(4)
Notably, Doan et al. (2021) define the forgetting metric on discrete datasets, while the definitions in Lin al. (2023) are restricted to linear models. In contrast, formulation applies to arbitrary function classes. Specifically, we extend the widely adopted metrics of backward transfer and average accuracy, which are commonly used to characterize forgetting and generalization in CL, by redefining them in terms of population loss. This reformulation enables these metrics to reflect model performance the level the underlying data distribution. Moreover, our definitions do not rely on specific model assumptions and evaluate performance over the entire input space, enabling a more comprehensive theoretical analysis of Cl..
2.2 KERNEL REGIME FOR CONTINUAL LEARNING
We briefly review the basic concepts of the Neural Tangent Kernel (NTK) regime. Throughout this work, we assume that the model is trained using the mean squared error (MSE) loss. Accordingly, the empirical loss task can be written as:
3

---

Published as a conference paper at ICLR 2026
1 -((()) - y;)
L. (f) 2n,
(S)
Before introducing the NTK, we define the neural network kernel (NNK) at training time as: x²) = (x), Vo
(6)
Based on this, the gradient flow dynamics of f- with respect to the MSE loss can be written as:
d - [+(x) di
1 (x, (X,, -
NTK theory states that, in the shinnite-widthh limit, NNK K converges to time-invariant kernel Ky, known as the NTK (Jacot et al. 2018 Lee et al. 2019 Arora et al. 2019b). In particular, if task is initialized from the trained parameters of task 1, i.e. 04 0 then each kernel entry satisfies
(x x²) (V f2 (z), Vog (x = (Vo 1
(x), V f=-1(2²)
(8)
For any x E Rd and X (Z1..... . knd) T E R nxd we denote (x, X) = (K,(x, æ1),..... K, (n),, XX, = [K,
( (I Tyy]axx
(9)
Under the NTK regime, the training dynamics in Equation 7 reduce the kernel gradient flow
d K, _ (I, Yr),
di [+(x
(10)
The
where Kr remains fixed throughout the training process for task т. Hence, Equation (1O) defines an ordinary differential equation (ODE) in the time variable with an initial condition induced by 8 8 Furthermore, Equation (10) admits closed-form solution:
(x) 1 K, X+) (X,, x,) (f
(11)
where E,, := : 1-xe -L (X,, and f; denotes the predictor obtained after completing nr
the training of task 1. The derivation of Equation TT) follows directly from solving the linear ODE in Equation (TO) and provided Appendix D. For notational convenience, we define
(o) := (z X,, (X,,
(12)
where -f represents the residual between the ground- truth labels of task T and the predictions of the model trained on task T 1. By recursively applying Equation (II) across tasks, the model for task can be expressed as (x) = 2T f7(x) (z).
(X,)
3 THEORETICAL RESULT
In this section, we present upper bounds on forgetting and generalization error for vanilla CL under the NTK regime in Theorem I For clarity of presentation, 1 = lT for the final task T, i.c., the iteration index of task T coincides with its stopping iteration. Theorem 1. Consider sequence of ' tasks. For each task T E [T]. let D, denote the data distribution, and let S, = {X,, be the corresponding training dataset drawn i.dd from D,. Suppose the loss function e(..-) takes values in the interval [0,c] and is p-Lipschitz in the first argument. Then, with probability at least 1 the following bounds hold:
adopt the notational convention
(X,,
Ee,KgK((X Xx) Yx 1/2
Fey
2p
T
k=1
Tr Tr(Ka (x,,xx)))x) X, Ex, KA
K& (Xx, Xx) '8x 1/2
log (2g
+ 2p
6cg
Thy
2n,
k=1
T T-T
Σ
(T((X,,x))Ea, Kn -
|le
# (x,. X.)
2n
k=T+1
(13)
4

---

Published as a conference paper at ICLR 2026
T
Ger
T
Σ
Ex),
-Vxl
1
-lle k (x,,,XX,
nhe
n4
k=++1
(X,, X Eg, KA (X - + /
log(2/0)
+ 2p
n4r
2n,
(14)
To the best of our knowledge, Theorem provides the first upper bounds on both forgetting and generalization error at intermediate training stages vanilla CL. The proof given in Appendix E Furthermore, our bounds are explicitly dependent the number of training iterations, allowing us to
characterize the evolution of forgetting and generalization errors throughout the training process.
To facilitate the subsequent analysis of the upper bounds of forgetting and generalization errors, we denote puupper and Gupper as the respective upper bounds of Fr, and Gty in Theorem Based on Theorem we will provide insights on the following two aspects.
(1) Lipschitz constant. The Lipschitz. constant / characterizes the maximum rate of change of the loss with respect to the model's 's predictions. Formally, for any two predictions u, in the output space, [[(, y) E(v, r)1 I p llu - vll. In general, a smaller prediction-Lipschiie constant implies that the loss varies more smoothly with respect to the model outputs (i.c., flatter landscape in prediction space), which is often associated with improved generalization.
(2) Cross-task kernel. Under the NTK regime, define the cross-task kernel between any two tasks T < E [2 as (&(X,, Xx). In traditional machine learning, cross kernels characterize the similarity between
datasets (Akaho 2006 (Schölkopf et al. 1997). In the NTK setting, the cross-task kernel instead captures cross-task interactions by measuring the alignment between the model gradients with respect different task datasets. A larger norm of this matrix indicates stronger task interference, which in turn increases the risk of forgetting and generalization error.
To gain deeper insight into continual learning, we next analyze the effect of these two critical factors.
3.1 THE IMPACT OF THE LIPSCHITZ CONSTANT
The role of the Lipschitz constant has been extensively studied in non-CL settings, where smaller values are often linked to improved generalization performance (Bartlett et al., 2017 Miyato et al.) et al. 2022 Khromov & Singh| 2024). One widely used approach to approximately reduce the Lipschitz constant is to penalize the gradient norm (PGN) of the loss function (Zhao et al., 2022). Moreover, Gradient-norm Aware Minimization (GAM) further penalizes the gradient norm within a neighborhood of the parameters (Zhang et al. 2023), thereby promoting flatter solutions. Although reducing the Lipschitz constant has been both theoretically and empirically shown to improve performance in non-CL. scenarios, its effectiveness in CL remains largely unexplored.
Q1: Does the role of the Lipschitz constant in non-Cl. also hold in the context of CL.?
To address Q1, we examine how the Lipschitz constant affects the forgetting bound ppperr and the generalization error bound Gupper in Theorem In particular, we analyze the evolution of pupper by and Gupper with respect to the training iteration tr when the Lipschitz constant exceeds a certain threshold. The detailed proof of Lemma iss provided in Section G.I
Lemma 1. For any fixed tr, smaller Lipschitz constant p leads smaller values of both Gupper and Eupper Moreover, there exists constant p² > 0 such that, for all p > p", both Gupper and Edipper increase monotonically with respect to tr.
Lemma 7 indicates that reducing the Lipsehitz constant p consistently mitigates both forgetting and generalization error in CL. In contrast to non-Cl (L- settings, where reducing the Lipschitz constant 9 primarily improves generalization, in CL il also alleviates forgetting
Remark 1.
Remark 2. Lemma further implies that, once exceeds a threshold p², the upper bounds G and pupper increase monotonically with the number of training iterations LT. In the degenerate limit
Gupper
5

---

Published as a conference paper at ICLR 2026
of skipping updates (i.e., 0j, these quantities can be made trivially small, but at the cost of no adaptation to the new task an undesirable, pathological outcome. Therefore, it is crucial to design mechanisms that explicitly control reduce p.for each task to improve CL performance in practice.
As shown above, reducing the Lipschitz constant p is beneficial for mitigating forgetting and im- proving generalization in CL. To implement this practice, we adopt a penalized gradient-norm framework that approximately reduces p; further details are provided in Appendix G.2 In the CL setting, the training loss of PGN for any task E [2'] is given by
I PGN (0+) Ls, (0-) a-1lV 8. Ls, (0-)
(15)
where || lle denotes the Euclidean norm and at is the penalty coefficient.
In practice, we employ GAM rather than PGN, since GAM encourages flatter solutions. In the CL setting, the training loss of GAM for task T is defined as
LGAM (0-) Ls, (0+) + a,b, S.
Vo Ls, (6+)
(16)
B(((,.)))))
where the perturbation radius b, controls the neighborhood size, and B00, b,) denotes the open ball of radius b, centered 0, in Euclidean space. Importantly, GAM penalizes the neighborhood Lipschitz constant, thereby avoiding sharp minima and improving robustness. Furthermore, our experimental results in Table empirrilll demonstrate that GAM effectively mitigates forgetting and enhances generalization compared to vanilla CL (SGD), thereby validating our theoretical analysis.
3.2 THE IMPACT OF CROSS-TASK KERNEL
In this section, we examine the influence of the cross-task kernel between any two tasks T k [2. 27, i.e., K2 (x,XX,), on the forgetting bound Fir and the generalization error bound Grype in Theorem Each entry of (x Xx) ) is the inner product between the model gradients with respect to a sample from X, and a sample from X₃. Ideally, to eliminate the adverse effect of (x,X,,X,) pupper and Gupper all entries should be zero equivalently, the gradients with respect to different datasets should be mutually orthogonal. A natural and effective approach to enforce such orthogonality is Orthogonal Gradient Descent (OGD) (Farajtabar et al. 2020). In the following, analyze the behavior of the cross cross-task kernel to provide a theoretical explanation of how OGD mitigates forgetting and reduces generalization error.
We introduce OGD in the context of CL. For any [T'. define Ur.i Voft and E, span (vr | the subspace spanned by the parameter gradients of the converged model f evaluated on the inputs from task The core idea of OGD is to project the gradient of the current task onto the orthogonal complement of the subspaces spanned by all previous tasks, i.e., E, @ .... + E,-1- Let P(E.e...E,-.>> denote the projection operator onto the orthogonal complement of this space, which we write simply as P, for brevity. Under OGD, the gradient flow dynamics for task are given by
R, (x, (-) Ji (+()) = -1
(17)
The
where (,, (P f; P.Vo; 8z (x))). Thus, we obtain gradient flow analo-
gous the standard SGD gradient flow Equation TO). with the key difference lying in the form of the kernel. Additional details are provided in Appendix D.2
In the following, we demonstrate that OGD reduces the cross-task kernel between the datasets of two adjacent tasks to the zero matrix. The proof of Lemma i provided in Appendix G.3
Lemma 2. For any [2, under OGD we have (X,,,,, =0
Remark 3. In Lemma 2 we show that OGD can eliminate the cross-task kernel between two adjacent tasks, thereby yielding tighter bounds forgetting and generalization error compared to standard SGD. Moreover, we observe that the orthogonality constraints in standard OGD are unnecessarily strong. Specially, if the projector Pt is redefined onto the orthogonal complement of Eg-1, rather than E, @ + Ex-1, Lemma ] holds while avoiding overly restrictive constraints (see Appendix G.3 for details). Empirically, shown in Table 2 OGD achieves better performance than SGD).
G.3

---

Published as a conference paper at ICLR 2026
4 OGD+ AND OPGD ALGORITHMS
4.1 REFINED ORTHOGONAL. GRADIENT DESCENT (OGD+)
In Lemma we theoretically show that OGD exhibits less forgetting and better generalization than SGD in CL by proving that the cross-task kernel between two adjacent tasks is the zero matrix under OGD. Moreover, if this property could be extended so that the cross-task kernel between any pair of tasks were zero, CL performance should improve further. This naturally raises the following question:
Q2: How OGD be improved to eliminate the cross- cross-task kernel between arbitrary task pairs, thereby further reducing forgetting and enhancing generalization?
To address Q2, for any k E [2], we redefine the gradient subspace as E, := span Volk (x]") ) [k:, m E [ny]} and the projection operator as P PF; We refer this refined variant OGD as OGD+. Accordingly, the corresponding NTK under OGD+ can be reformulated as
(= = VP: fi-1(1), PoVo; f=-2(z')).
(18)
As shown in Lemma 3 OGD+ reduces the cross-task kernel between the datasets of any two tasks to the zero matrix. The detailed proof provided in Appendix G.4
Lemma 3. For any Remark 4. Lemma 1ommaaddenen demonstrates that OGD+ can eliminate the cross-task kernel between any pair of tasks, thereby yielding lower forgetting and better generalization compared to standard OGD. In particular, derive upper bounds both forgetting and generalization error for OGD and OGD+ and establish that both bounds for OGD+ are strictly lighter.
E (2,T), under OGD+ we have Rx, X1)=0
(see Appendix G.5
Comparison between OGD and OGD+. The key difference between OGD and OGD+ lies in how gradient information is stored and released. Specifically, OGD stores the gradients of the model after training on the current task using only the data from that task, and these gradients are retained indefinitely. In contrast, OGD+ stores the gradients of the model after training on the current task using all data from previous tasks, but releases them once training on the subsequent task completed. As shown in Remark OGD+ provides stronger theoretical guarantees than OGD due to its stricter enforcement of gradient orthogonality. Empirically, Table 2 shows that OGD+ forgets less and generalizes better than OGD on the two MNIST benchmarks, with pronounced improvements in both metrics. However, on Split 100, OGD+ slightly underperforms OGD on both metrics, which attribute to its excessive orthogonality. In particular, overly restrictive orthogonality reduces the feasible gradient subspace, thereby limiting the model's capacity to adequately fit the current task especillyy under large distribution shifts between tasks. We next explore strategies to mitigate the negative impact of excessive orthogonality in OGD+.
4.2 ORTHOGONAL PENALIZED GRADIENT DESCENT (OPGD)
In Section we theoretically demonstrated that OGD+ achieves lower forgetting and better generalization than OGD. However, while OGD+ enhances gradient orthogonality across tasks, it neglects inter-task performance and thus risks reducing plasticity in practice. A straightforward way to enhance inter-task performance is to reduce the Lipschitz constant of each task. Furthermore, in Section 3.1 we thcoretically established that reducing the Lipschitz constant consistently mitigates forgetting and improves generalization. These observations naturally motivate the following question:
Q3: Can reducing the Lipschitz constant in OGD+ further mitigate forgetting and enhance generalization compared to standard OGD+?
uupeer+ as the upper bounds of forgetting and generalization error for OGD+. We denote pupper+ and G To address Q3, analyze how these bounds vary the Lipschitz constant is reduced, and further examine their dependence on tr when the Lipschitz constant falls below a certain threshold. The formal results are stated in Lemma with proofs provided in Appendix G.6
7

---

Published as a conference paper at ICLR 2026
For any fixed tr, reducing the Lipschitz constant p leads strictly smaller values of Guppert and Forpert compared to their original values under OGD+ Moreover, there exists constant p' > such that, for all p p', Guppert decreases monotonically with respect to tr, while Emppert increases monotonically with respect to tr. Fiz
Lemma 4.
Remark 5. Reducing the Lipschitz constant in OGD+ yields tighter bounds on both forgetting and generalization error than standard OGD+. This indicates that incorporating mechanisms to reduce the Lipschitz constant within OGD+ can further mitigate forgetting while improving generalization.
Remark Lemma A implies that when the Lipschitz constant in OGD+ falls below a certain threshold, it helps avoid the degenerate phenomenon discussed in Remark 7 thereby benefiting generalization as training progresses. At the same time, longer training (larger Ir)) increases the Fupper grows with IT, consistent with the behavior of large
risk of catastrophic forgetting because IT Lipschitz constants noted in Remark This highlights a trade-off between mitigating forgetting and improving generalization. Importantly, this observation does not conflict with Remark S although extended training may increase forgetting. for any fixed iteration the bounds on both forgetting and generalization remain tighter when the Lipschitz constant is reduced.
OPGD algorithm: Leveraging Remark and
Algorithm 1: OPGD
Remark 6 we establish a principled pathway from theory to algorithm design: integrating OGD+, which enforces cross-task orthogonal- ity, with GAM, which reduces the Lipschitz con- stant. This unified approach, termed Orthogonal Penalized Gradient Dexcent (OPGD), jointly mit- igates forgetting and enhances generalization. As shown in Table 2 OPGD achieves substantial improvements over OGD+.
Input Task sequence 7i, To.... ... learning rate ry: balance coefficient o; perturbation radius b; small constant ç.
S <- Ø, MM + Ø, 0 + to: for Task ID T 1,2, do
repeat
// GAM
91 + VaLs, (0):
- VoL,, (0) IV,ks, (0lI+E)
VeLg, (0)
Next, present the details of OPGD. For the first task, we update the model parameters of fi by minimizing the GAM loss (Equation (T6)) which effectively reduces the Lipschitz constant of the loss and thereby enhances inter-task per- formance. The corresponding gradients are then stored. For each subsequent task т E [2, T). at each parameter update iteration we first minimize the GAM loss for fr. and then apply OGD+ to the resulting (penalized) gradient, ensuring that the gradient for task is orthogonal to the stored gradients from task 1. Finally, we release the stored gradients of task r - and replace them with the gradients of task r evaluated on samples from all previous tasks. The full procedure of OPGD is summarized in Algorithm T.
0' + + ालनिद
92 - b. Ls. (0)) Ls, (IT+
Farks, (0))
g -1 + ga:
// Orthegenal updates
9 - EvEs, (g);
0 ng:
until convergence;
S Ø; for (z. y) E St M and E [1.c] with Y4 1 do
vofr(z) - proj, (Vo (V)f+()))
S Sufuh
end Sample M, Sr. M+MUM
end
EXPERIMENT
In this section, we present extensive experiments to validate our theoretical findings and demonstrate the effectiveness of OGD+ and OPGD. Additional implementation details, further comparisons with baselines, and ablation studies are provided in Appendix C
Datasets. We evaluate our approach on three widely used CL benchmarks: Permuted MNIST (Kirk- patrick et al. 2017), Rotated MNIST (Farajtabar et al.) 2020), and Split CIFAR- 100 (Chaddhry et al. 2019a). Permuted MNIST and Rotated MNIST are variants of the original MNIST dataset, where each task is defined by a random pixel permutation or rotation, respectively. For both benchmarks, we construct 15 sequential tasks using different permutations or rotation angles. Split CIFAR- 100 is created by partitioning the 100 classes of CIFAR- 100 into 20 disjoint tasks, each containing 5 classes.
8

---

Published as a conference paper at ICLR 2026
Table Average accuracy (ACC) and backward transfer (BWT) over all tasks on different datasets. Higher ACC and BWT indicate better generalization and less forgetting. All results are reproduced by us and averaged over 5 runs. The best continual learning results are highlighted in bold.
Dataset
Permuted MNIST (15 tasks)
Rotated MNIST (1S tasks)
Split CTFAR-100 (20 tasks)
Mothod
ACC
BWT
ACC
ACC
BWT
SGD
70.29
1.50
25.33 1.57
0.43
28.09
52.08 1:
30.63 1.31
GAM
72.61 1.44
-22.47 1.57
0.44
-20,60
1.68
-22.63 1.60
OGD
82.17 = 0.64
-12.38
+ 0.06
71.52
2 0.69
-18.43 4: 0.76
63.91 + 1.02
-20.37 + 1.66
OGD+
56.22 1: 0.62
8.11 0.66
86.15 0.44
0.02 0.56
61.84
1: 2.51
23.47 1.48
OPGD
86.27 = 0.56
-7.73 0.61
89.15 0.22
-3.69 :: 0.27
68.17 + 0.71
-12.58 = 1.35
3GD
OPGD
-10
Average Accuracy (ACC)
Backward Transfer (BWIT)
-20
80
-40
-50
1.0000 20000 30000
50000 G0000 70000 80000
10000 20000
30000
50200 62000 Steps
40000
70020
Steps
Figure 1: Dynamics of average accuracy (ACC) and backward transfer (BWT) for different methods on Split CIFAR-100. Vertical dotted lines are used indicate the boundaries between different tasks. For each task, we record ACC and BWT at evenly spaced intervals, performing 40 evaluations per task by measuring performance every 100 training iterations.
Baselines. To align with our theoretical analysis, compare OGD+ and OPGD with three continual learning methods: vanilla CL with SGD, GAM (Zhang et al., 2023), and OGD (Farajtabar et al. 2020). Additional comparisons with other CL methods are provided in Appendix
Evaluation metrics. To align with our theoretical analysis, we adopt average accuracy (ACC) and backward transfer (BWT) as the evaluation metrics (Lope-Paz & Ranzato 2017). Formally, they are defined as
T ACC Σ AT
T-1 AT
BWT T
Asa
where Ar, denotes the accuracy of the model on task after completing training on task t, and T is the total number of tasks.
Performance. As shown in Table OPGD achieves significant improvements in both ACC and BWT over prior methods across all datasets, corroborating our theoretical claim that reducing the Lipschitz constant within OGD+ simultaneously mitigates forgetting and improves generalization. In particular, OPGD yields average relative gains of +4.59% in ACC and 73%.73% in BWT across three benchmarks. Furthermore, OGD+ consistently forgets less and generalizes better than OGD on Permuted MNIST and Rotated MNIST, with average relative gains of +4.27% in ACC and +33.82% in BWT across three benchmarks. However, Split CIFAR-100- substantially more complex than Permuted MNIST and Rotated MNIST-OGD+ underperforms OGD. We attribute this excessive orthogonality in OGD+, which reduces model plasticity and consequently degrades inter-task performance, as discussed in Section 4.1 similar observations have been reported by Zhao al. (2023); Yang al. (2023). Notably, OPGD mitigates this effect by reducing the Lipschitz constant within OGD+, thereby enhancing inter-task performance.
distribution is
Dynamics of forgetting and generalization. As shown in Figure] I the ACC of OPGD increases steadily with the number of iterations, indicating that longer training enhances generalization. This

---

Published as a conference paper at ICLR 2026
result is consistent with our theoretical analysis in Lemma 4 In contrast, the ACC of SGD does not consistently improve and even declines in the final tasks. This phenomenon aligns with Remark [ which suggests that without explicit control of the Lipschitz constant, prolonged training may accumulate instability and hinder generalization. Notably, incorporating GAM to reduce the Lipschitz constant helps SGD avoid this degradation, enabling more stable generalization. As shown in F the BWT of OPGD decreases within each task interval, indicating that additional iterations increase forgetting- again consistent with Lemma Taken together, these results highlight fundamental trade-off between forgetting and generalization: while longer training improves generalization, simultaneously exacerbates forgetting, in line with Remark 6 Despite this trade-off, OPGD consistently outperforms competing methods in terms of both ACC and BWT throughout training.
6 CONCLUSION
We derived upper bounds on forgetting and generalization error intermediate training stages in CL under the NTK regime. Our analysis shows that reducing the Lipschitz constant and enforcing gradient orthogonality both help mitigate forgetting and improve generalization. Building on these insights, we proposed OGD+ and OPGD, which refine gradient orthogonality and integrate gradient- norm penalization, respectively. Empirical results on standard benchmarks corroborate our theoretical predictions, providing principled pathway from theory to algorithm design in CL. We discuss limitations and our use of large language models in Appendix A
ACKNOWLEDGMENT
This work is supported by NSFC Grant (No. 62576364), Shenzhen Basic Research Project (Natural the Open Re-
Science Foundation) Basic Research Key Project (NO. search Fund from Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) (NO. GML-KF-24-233, CCF-DiDi GATA Collaborative Research Funds (NO. CCF-DiDi GAIA 202508).
REFERENCES
Shotaro Akaho. A kernel method for canonical correlation analysis. arXiv preprint cs/0609071, 2006.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pp. 139-154, 2018.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccr Caccia. Online continual learning with maximal interfered retrieval. Advances in neural information processing systems, 32, 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning, Advances in neural information processing systems, 32, 2019b.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International conference on machine learning, pp. 322-332. PMLR, 2019a.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in neural information processing systems, 32, 2019b.
Amin Banayecanzade, Mahdi Soltanolkotabi, and Mohammad Rostami. Theoretical insights into overparameterized models in multi-task and replay-based based continual learning. arXiv preprint arXiv:2408. 16939, 2024.
Peter I. Bartlett and Shahar Mendelson. Rademacher and gaussian complexitis: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482, 2002.
Peter L Bartlett, Dylan Foster, and Matus J Telgarsky. Spectrally normallzed margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
10

---

Published as a conference paper at ICLR 2026
Mehdi Abbana Bennani, Thang Doan, and Masashi Sugiyama. Generalisation guarantees for continual learning with orthogonal gradient descent. arXiv preprint arXiv:2006. 1942, 2020.
Blake Bordelon, Abdulkadir Canatar, and Cengiz. Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference Machine Learning, pp. 1024-1034. PMLR. 2020.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. Advances in neural information processing systems, 33: 15900-1599, 2020.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-GEM, International Conference on Learning Representations, 2019a.
URL
net/ forum? timpmmmii-Hitn id=Hkf2_sC557
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Kumar Dokania, Philip H. S. Torr, and Mare' Aurelio Ranzato. Continual learning with tiny episodic memories. CoRR, abs/1902. 2019b.
10486
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Generalization bounds for learning kemels. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 247-254, 2010.
Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 5138-5146, 2019.
Meng Ding, Kaiyi Ji, Di Wang, and Jinhui Xu. Understanding forgetting in continual learning with linear regression. In International Conference on Machine Learning, pp. 10978-11001. PMLR, 2024.
Thang Doan, Mehdi Abbana Bennani, Bogdan Mazoure, Guillaume Rabusseau, and Pierre Alquier. A theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In International Conference on Artificial Intelligence and Statistics, pp. 1072-1080. PMLR, 2021.
Shibhansh Dohare, Fernando Sermandez-Garriaa Qingfeng Lan, Parash Rahman, Rupam Mah- mood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026): 768-774, 2024.
Itay Evron, Edward Moroshko, Rachel Ward, Nathan Srebro, and Daniel Soudry. How catastrophic can catastrophic forgetting be in linear regression? In Conference Learning Theory, pp. 4079. PMLR, 2022.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International conference on artificial intelligence and statistics, pp. 3762-3773
PMLR, 2020.
Iordanis Fostiropoulos, Jiaye Zhu, and Laurent Itti. Batch model consolidation: A multi-task model consolidation framework. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3664-3676, 2023.
Reinhard Heckel. Provable continual learning via sketched jacobian approximations. In International Conference on Artificial Intelligence and Statisties, pp. 10448 10448-10470. PMLR, 2022.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Sham M Kakade, Karthik Sridharan, and Ambuj Tewari. On the complexity of linear prediction: Risk bounds, margin bounds, and regularization. Advances in neural information processing systems, 21, 2008
Ryo Karakida and Shotaro Akaho. Learning curves for continual learning in neural networks: Self- knowledge transfer and forgetting. In International Conference on Learning Representations, 2022.
URL URL hhtt:ss://ppnrreiieeeee htt::

---

Published as a conference paper at ICLR 2026
Grigory Khromov and Sidak Pal Singh. Some fundamental aspects about lipschitz continuity of neural networks. In The Tweifth International Conference on Learning Representations, 2024.
URL http://s//..paaeeeeeeeeesnsse/rr
net/ Eorum 2id=5 5 3WsW082Uh
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barnskaa-aaiiska,,, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
Haoran Li, Jingfeng Wu, and Vladimir Braverman. Fixed design analysis of regularization-bait based continual learning. In Conference on lifelong learning agents, pp. 513-533. PMLR, 2023.
Haoran Li, Jingfeng Wu, and Vladimir Braverman. Memory-statistiis tradeoff in continual learning with structural regularization. arXiv preprint arXiv:2504.04039) 2025a.
Hongbo Li, Sen Lin, Lingjie Duan, Yingbin Liang, and Ness Shroff. Theory on mixture-of-experts in continual learning. In The Thirteenth International Conference on Learning Representations, 2025b. URL pesttps://pperree
www.ewweeeeeeeeeeeeeeeeeeeeeeeee forum? [d=7ggKAbbspp
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pallern analysis and machine intelligence, 40(12):2935-2947) 2017.
Sen Lin, Li Yang. Deliang Fan, and Junshan Zhang. Trgp: Trust region gradient projection for
continual learning. In The Tenth International Conference on Learning Representations, 2022.
Sen Lin, Peizhong Ju, Yingbin Liang, and Ness Shroff. Theory on forgetting and generalization of continual learning. In International Conference on Machine Learning, pp. 21078-21 11000. PMLR, 2023.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
James L McClelland, Bruce L McNaughton, and Randall C O'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint /00..0590005555 2018.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint 1810.
2018
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuogiu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1rXiv:1606.04671 2016.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning, In International Conference on Learning Representations, 2021.
Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Kernel principal component analysis. In International conference artificial neural networks, pp. 583-588. Springer, 1997.
Zifeng Wang, Zheng Zhan, Yifan Gong, Geng Yuan, Wei Niu, Tong Jian, Bin Ren. Stratis Ioannidis, Yanzhi Wang, and Jennifer Dy. Sparel: Sparse continual learning on the edge. Advances in Neural Information Processing Systems, 35:20366-20388 2022.
12

---

Published as a conference paper at ICLR 2026
Enneng Yang. Li Shen, Zhenyi Wang. Shiwei Liu, Guibing Guo, and Xingwei Wang. Data aug- mented flatness-aware gradient projection for continual learning. In Proceedings of the IEEE/CVF international conference computer vision, pp- 5630-5639, 2023.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations, 2018.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pp. 3987-3995. PMLR, 2017.
Xingxuan Zhang, Renzhe Xu, Han Yu, Hao Zou, and Peng Cui. Gradient norm aware minimization seeks first-order flatness and improves generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20247-20257, 2023.
Xuyang Zhao, Huiyuan Wang, Weiran Huang. and Wei Lin. A statistical theory of regularization- based continual learning. In International Conference on Machine Learning, pp. 61021-61039. PMLR, 2024.
Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization deep learning. International conference on machine learning, pp. 26982- 26992. PMLR, 2022.
Zhen Zhao, Zhizhong Zhang, Xin Tan, Jun Liu, Yanyun Qu, Yuan Xie, and Lizhuang Ma. Rethinking gradient projection continual learning: Stability/plasticity feature space decoupling. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp.
2023.
Guodong Zheng, Peng Wang, Tao Sun, and Li Shen. Towards understanding memory buffer based continual learning. Neural Networks, pp. 108522, 2025.
13

---

Published as a conference paper at ICLR 2026
A ADDITIONAL STATEMENT
Limitations. Our theoretical analysis is developed under the NTK regime, which may not fully reflect the behavior of practical deep networks with fimite width or more complex architectures. In addition, our empirical evaluation is limited to classification benchmarks; extending both the theoretical framework and the proposed algorithms to other modalities and learning paradigms remains an important direction for future work.
The Use of Large Language Models. In this work, we exclusively employ large language models (LLMs) to refine the writing and presentation of our manuscript.
B RELATED WORKS
Empirical studies in CL. Continual learning has achieved substantial empirical progress, with existing approaches broadly categorized into three families: (1) Regularization- methods, which introduce explicit penalty terms restrict updates
parameters important for previous tasks (Kirkpatrick et al.) 2017: Zenke et al., 2017). or employ knowledge distillation by aligning the predictions of the current model (student) with those of the previous model (teacher) mitigate forgetting (Li & Hoiem 2017: Dhar et al. 2019; Fostiropoulos et al. 2023); (2) Replay-based
methods, which either store and replay data from past tasks during training on new tasks (Chaudhry) it al. 2019a; Riemer et al.) 2018: Buzzega et al. (020), or retain gradient information from prior tasks and enforce new updates be orthogonal to past gradients, thereby avoiding explicit data replay (Farajtabar et al. [2020 Saha et al., 2021 Lin et al.) (022); (3) Architecture- methods (Rusu) cet al. 2016;] Yoon et al., 2018 Wang et al. (022). which dynamically expand or adapt the network architecture to preserve knowledge from earlier tasks.
Theoretical analysis of CL. Recent works have begun to lay the theoretical foundations of CL by analyzing forgetting and generalization error under simplified settings. Several studies (Evron
eet al. 2022 Lin et al. 2023 [aaaayeaazzade et al., 2024 ZZeng et al.] 2025 Li et al.) 2025b)
investigate these phenomena within overparameterized linear models, typically assuming that datasets are drawn from Gaussian distributions. Ding et al. (2024) study forgetting in both overparameterized and underparameterized regimes, while relaxing the data assumptions to fourth-moment conditions. Recently, there has been growing interest in the theoretical amalysis of regularization based methods for CL. Heckel (022)) theoretically characterize how the performance of a model in a contrastive CL framework controlled by the training losses on previous tasks. [i et al. (2023) derive bounds on the average risk over two tasks for an lz-regularized CL algorithm. Zhao et al. (2024) provide a statistical analysis of regularization based CL on sequence of linear regression tasks and highlight how different regularization terms affect model performance. [L et al. (225a) establish upper and lower bounds on the joint excess risk for a generalized é fggrelulrd CL. algorithm. Another major line of theoretical work is based on the NTK regime (Bennani et al., 2020 Doan et al.) [021] Karakida & Akaho 2022). particular, |Bennani et al. (2020) established generalization error bounds for CL Via Rademacher complexity; Doan et al. (2021) analyzed forgetting by introducing the NTK overlap matrix as a timk-simyy metric; and Karakida & Akaho (2022), leveraging learning-curr analysis between two tasks (Bordelon et al. 2020). studied both forgetting and generalization error under the assumption that datasets from different tasks are drawn from the same distribution. Although these works provide valuable insights, most rely on simplified settings and converged models, thereby overlooking the evolution of models during training in realistic CL scenarios. In contrast, we theoretically characterize the evolution of both forgetting and generalization error in vanilla CL under the NTK regime, without requiring such restrictive assumptions.
The most relevant works to ours are Bennani et al. (2020); Doan et al. (2021). which theoretically analyze generalization error and forgetting separately ly under the NTK regime. However, our study differs in several key aspects: (1) Their analyses incorporate ridge regularization term in the loss function, effectively aligning with regularization based CL methods. By contrast, our framework makes no such assumption and corresponds to vanilla CL. thus serving as a clean theoretical baseline against which improved CL methods e.g., buffer-based and regularization-basee approaches- can be directly compared. (2) Their focus lies primarily on the performance of converged models, whereas we characterize the full evolution of forgetting and generalization error throughout the training dynamics. (3) Bennani et al.) (2020) study the generalization error of OGD in CL by leveraging the property that model outputs remain consistent between consecutive tasks, while Doan ctt al. (202I) analyze forgetting in OGD through the NTK overlap matrix. In contrast, we analyze both forgetting and generalization error in OGD through the structure of the cross-task kernel.
14

---

Published as a conference paper at ICLR 2026
C ADDITIONAL EXPERIMENTAL RESULTS
C.I EXPERIMENTAL SETTING
In this section, we provide additional details of our experimental setup, present extended comparisons with more baselines (Section C.3), and conduct an ablation study of OPGD (Section C.4). All experiments are conducted using the NVIDIA RTX 4090 with 24GB GPU memory, CUDA v11.8 and cuDNN v8.7.0 in PyTorch v2.4.1.
Architecture. For Permuted MNIST and Rotated MNIST, we adopt three-layer multilayer perceptron (MLP) with two hidden layers of 100 units each and final layer with 10 output logits. All layers except the last one use ReLU activation. For Split CIFAR-IO 1-00 we employ LeNet architecture for training. Table summarizes the hyperparameter settings used for OPGD.
Table 3: Hyperparameter Settings of OPGD
Hyperparameter
Permuted MNIST
Rotated MNIST
Split CIFAR-100
Task nums Network
15
15 MLP
20
MLP
LeNet
Epochs
50
Learning rate
10
10 10-3
10 10-3
Batch size
32
32
32
Hidden dimension
100
100
100
Balance coefficient
0.8
Perturbation radius b
0.02
0.2
0.05
Sampling size m
100
100
100
C.2 ADDITIONAL EXPERIMENTS
To verify that our methods also apply to other types of tasks, we conduct additional experiments on online continual learning. In online continual learning, data arrive sequentially in the form of a stream. Whenever small batch of data arrives, it is used for a single training iteration (Aljundi al. [2)9b]a). Following the standard online continual learning protocol, we use batch size of TO and train for one epoch. We perform experiments on Permuted MNIST, Rotated MNIST, and Split CIFAR-100. We adopt the same hyperparameter settings as those listed in Table
Table Average accuracy (ACC) and backward transfer (BWT) over tasks on online CL bench- marks. All results are reproduced by us and averaged over 5 runs. The best continual learning performance highlighted in bold.
Dataset
Permuted (15 tasks)
(15 tasks)
Split CIFAR-100 tasks)
Method
ACC
BWT
ACC
BWT
ACC
BWT
SGD
75.85 +: 0.78
12.24 +: 0.81
67.23 + 0.41
23.62 i: 0.40
31.90 + 1.57
16.21 :: 1.42
GAM
76.67 4: 1.0
-10.95 + 1.10
70.63 = 0.59
-14.86 1: 0.67
33.56 1.67
-14.29 = 1.65
0.50
-9.33
1.78
-7.92 1.66
01.35
37.05 1.77
1.08
OPGD
82.62++0.51
-4.00 + 0.49
87.14 + 0.22
2.75 + 0.25
39.88 1.49
-6.12. 1.45
To verify that our theoretical findings can benefit other continual learning algorithms, we integrate GAM or OPGD into both rehearsal based and regularization based methods on the Permuted MNIST dataset. Concretely, for rehearsal based baselines we adopt naive rehearsal protocol that randomly selects 100 samples from each past task. For regularization based baselines we use la regularizer so that the optimization problem for task takes the form
arg min {(O,) ||0 0 112
6-
The corresponding results are presented Table 5 The experimental results show that our theoretical messages also benefit other continual learning algorithms.
15

---

Published as a conference paper at ICLR 2026
Table Average accuracy (ACC) and backward transfer (BWT) over all tasks on Permuted MNIST. All results are reproduced by us and averaged over 5 runs. The best continual learning results are highlighted in bold.
Method
ACC
BWT
rehearsal
82.82 0.78
-12.60 0.36
rehearsal GAM
86.40 + 0.42
-7.64 0.44
rehearsal OPGD
88.23 #
-5.62 + 0.13
73.30 1,21
221.91
1.22
8 GAM
74.15 + 1.33
-19.19 1.43
2 OPGD
86.45 + 0.27
-7.42 0.33
C.3 COMPARISON WITH ADDITIONAL BASELINES
Additional baselines. We additionally compare against two types of CL methods. Regularization- based methods: EWC (Kirkpatrick et al.) 2017). MAS (Aljundi et al. 2018). SI (Zenke et al. (017)) Memory-based methods: ER (Chaudhry al. 2019b), A-GEM (Chaudhry al.J (019))). For fair comparison, we set the memory butfer size of ER and A A-GEM equal to that of OPGD.
Table 6: Average accuracy (ACC) and backward transfer (BWT) over all tasks on different datasets. Higher ACC and BWT indicate better generalization and less forgetting. All results are reproduced by us and averaged over 5 runs. The best continual learning results are highlighted in bold.
Dataset
Permuted MNIST
Rotated MNIST
Split CIFAR100
Method
ACC
DWT
ACC
BWT
ACC
BWT
SGD
70.29 +: 1,50
25.33
1.57
68.79 1 0.43
28.09 0.45
52.08 1 0.81
-3063
1.31
OGD
82.17 + 0.64
-12.38
0.06
77.32 + 0.09
18.43
63.91 + 1.62
-21.37
1.66
0.14
56
MAS
83.82 + 0.41
-5.486
0.36
79.50
0.16
1.60 0.48
66 1.41
-3.54
1.30
83.50 1: 0.22
-3.384
77.08
1: 0.37
113.78
0.48
67.45 1: 2.25
-8.77
2.38
GAM
72.61
1.44
-22-17 1.57
72.85
0.41
-00.00
61.70 1.68
-22.63
10:63-1-0 1.60
ER
83.35 1 0.91
-11.34 0.91
83.05 1: 0.34
0.39 12.75
66.03 0.34
-66.89
0.40
0.12
63
0.62
0.62
-33.47
OGD+GAM
83.73
-10.41 0.75
0.55
OPGD
86.27 + 0.56
-7.73
0.61
89.15 = 0.22
3.66 = 0.27
68.17 -: 0.71
-12.58 1.35
Discussion. As shown in Table OPGD achieves the highest ACC across all benchmarks, indi- cating that penalizing the gradient norm within OGD/OGD+ is an effective strategy for improving generalization. Moreover, although OGD+ underperforms OGD Split CIFAR-100, combining OGD+ with GAM outperforms combining OGD with GAM on this dataset. On the other hand, SI attains the highest BWT on Permuted MNIST, while MAS achieves the highest BWT on Rotated MNIST and Split CIFAR-100. As highlighted in Remark this pattern reflects an inherent trade-off in OPGD: while it substantially improves generalization during training, it may also increase the risk of forgetting as training progresses. Overall, our main contribution is to provide a theoretical framework for analyzing continual learning and to improve OGD from a theoretical perspective. Although OPGD may not always yield the best BWT, it consistently delivers substantial gains in ACC and exhibits large improvements over standard OGD, thereby validating the effectiveness of our theoretically motivated design.
C.4 ABLATION STUDY
In this section, we investigate effectiveness of each component in OPGD: balance coefficient (Table perturbation radius (Table 8). and per-task sampling size m (Table
Balance coefficient a. The balance coefficient a controls the strength of the GAM penalty. We perform a grid search over {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9} using a fixed seed. As shown in Table 7 OPGD is relatively insensitive a, with only modest changes across the range.
Perturbation radius b. The perturbation radius b controls the neighborhood size in which the GAM penalty is evaluated, thereby governing the magnitude of parameter perturbations. We conduct
16

---

Published as a conference paper at ICLR 2026
Table 7: ACC and BWT of OPGD with different balance coefficients a across datasets.
Permuted MNIST
Rotated MNIST
Split CHEARIOU
ACC
BWT
ACC
BWT
ACC
BWT
0.1
56.52
1.17
88.79
4.85
66.90
15.33
0.2
86. 41
-7.80
88.39
-1.70
67.70
-14.37
86.10
8.10
88.15
4.53
67.04
14.97
0.4
85.27
88.44
87.82
-4.76
67:23
114.51
0.5
86.48
7.35
87.47
4.56
68.03
13.84
86.55
87.02
0.7
84.76
9.30
86 86.15
5.58
67.98
14.21
0.8
85,40
8.62
85.79
5.78
68.18
14.03
86.41
85.76
-5.00
67.64
11477
Table 8: ACC and BWT OPGD with different perturbation radii across datasets.
Permuted MINST
Rotated MNIST
Split CIFARI00
ACC
BWT
ACC
BWT
ACC
BWT
0.02
86.41
7.47
88.47
5.40
68.48
13.82
0.05
86.01
-7.60
85.79
-1.85
69.31
- -11.3
8.17
-4.3
99.00
0.2
83.79
-5.96
89.09
-3.76
53.13
-9.
79.70
7.08
87.97
33.9
50.53
8.78
72.61
-6.59
87.72
-2.50
48.45
-8.39
2.0
70.36
8.05
77.18
2.90
47.45
9.12
grid search over {0.02, 0.05, 0.1, 0.2, 0.5, 1.0, 2.0} using a fixed seed. We observe that enlarging b is not always beneficial for OPGD, especially when b >> 0.2. Conceptually, OPGD aims to reduce the local Lipschitz constant; GAM does by penalizing the maximal gradient norm within ball of radius b, which serves proxy upper bound for the local Lipschitz constant. When b becomes too large, the neighborhood is no longer local, the proxy bound becomes loose- insse-ineding optimization and degrading fit thereby explaining the observed saturation or decline in performance at large b.
Table ACC and BWT of OPGD with different per-task sampling sizes m across datasets.
Permuted MNIST
Rotuted MNIST
Split CIFARI00
ACC
BWT
ACC
BWT
ACC
BWT
20
79.85
14.73
82.79
-11.05
66.00
-14.0
40
$3.80
10.46
85.21
-8.26
59.60
-2.08
83.51
10.76
87.12
-6.07
-18.10
85.57
8.44
88.33
4.80
65.21
14.79
100
85.53
8.32
80.09
-3.76
69.25
-11.10
140
87.17
-6.71
90.23
-2.29
68.56
-11.77
Per-task sampling size m. In OPGD, we randomly sample m examples from each task to store in the memory buffer. We perform grid search over (20, 40, 60, 80, 100, 120, 140) using fixed seed. As shown in Table both ACC and BWT exhibit a clear increasing trend as the buffer size grows. Due GPU memory constraints, we report results up m 140; nevertheless, the observed trend indicates that larger buffers would likely yield further gains for OPGD.
D KERNEL GRADIENT FLOW
D.1 KERNEL GRADIENT FLOW UNDER SGD
In this section, derive Equation III by solving the linear ODE in Equation (0). We start by evaluating Equation (T)) at the training dataset X,:
d di [(x))
KX - -Y))
(19)
The
Let g(1 = f((x)) Y,. Then, Equation
19) can be rewritten in the simplified form:
d 1 X-)g(t))))t),
dt g(t) nt
9 = fx(x) Y..
(20)
17

---

Published as a conference paper at ICLR 2026
Equation (20) is linear matrix ODE, where (X,,, X,) is time-invariann and real symmetric. Therefore, the theory of linear ODE guarantees a unique solution:
g(t) = exp (- Ky (X+, 9(0) exp n4 Xx), -F)).
(21)
Substituting g(t) = (X,) y into Equation (1) yields:
(X,) = + cxp ( (-
K, (X,, (x), -Y,)
(22)
Therefore, obtain the explicit form of (-) on the training dataset X,. Next, fix an arbitrary test point x E Rd, For this x, Equation 110) specializes to
(x, - (F){(() -F).
di
[+() - -- 74,
(23)
Integrating both sides of Equation (z) = f7-1 (z), we obtain
23 over the interval [0, and applying the initial condition
-xx))f((x)) -- 1 (=(x,x.) --10)=---KK / (F((+)) -Y)) ds.
(24)
nt
Substituting Equation (22 into the integral term of Equation
24
we obtain
S. ((:(x,)-Y))ds
Y2)ds S(" exp (- 1- (X), (1+(X,) - -Y,)ds
(25)
1//² exp (X,,,(X,,,,,))
-X,)))
The matrix integral in Equation 25 can be evaluated in closed form by applying the standard identity (valid for any constant matrix and scalar a > 0):
J exp(- - a-
-Iepp(-e (---------
(26)
provided that A is invertible.
Applying Equation 26) with A= 1= K, X2) and a 1/n, gives
/' exp (- 1- Xx) ",
exp (-- (-=_K(Xxxxx>))) (27)
Plugging Equation 27 into Equation
24 we obtain
f+() (x)
1
n, Kr Xx) (X,, cxp(- n. Kx(X,, (f2(X+) -Y,) (28)
Xx) xx) exp( n. - (x)(( -Y.). =-K,(I, K, Kr K,,
Recalling that (-)-) = (:- we arrive at the closed-form solution for any E Rd:
fz(z) = f=)((x) K, (x, Xr) xx)(( exp ( - K, K, (X,, )x), -1 -Y))
(29)
18

---

Published as a conference paper at ICLR 2026
For notational simplicity, define
I cxp (-11, (X,,,,(X,,x,,,
(30)
Thus, the solution be compactly expressed
- f K, (x, ( ( - Y?).
(31)
We complete the derivation of Equation II by solving the linear ODE in Equation (10.. Next, we introduce an important lemma that will be used in the subsequent proofs.
Lemma 5. For any E [T]. both e (X,,,,),, and ET,; are symmetric and positive definite.
Proof. Let Ak., > (A E [n,]) be the eigenvalues of K, (X,, Therefore, there exists an orthogonal matrix Q, such that
= X,),
(32)
where ,,,,, Ar, the eigenvalues of K, (X,,;;
- mr 1. -07
Σ
0
-
1
K, (X (x))))
The
k=0
1 Q (X,, Xx)
k!
Thr
(33)
diag frngf)r,1,......
Armn }tt)
k!
k=0
dr.
A Armn
=diog
kd!
k!
k=0
k=0
Ar,
Ava
diagie
For any 1, > ( we have 0 < K2 XX,, f is symmetric positive definite, and thus Er.:; is also symmetric positive definite with eigenvalues 1 exp(-Ax. (//n)) E (0, 1). Even if we relax the condition to Kr(:;)) being only positive semi-definite, the matrix exponential exp( ュ K, (Xr, X,) remains symmetric positive definite, since the exponential of any symmetric matrix with nonnegative eigenvalues yields strictly positive eigenvalues.
< 1. Consequently the matrix exponential exp(
D.2 KERNEL GRADIENT FL.OW UNDER OGD
In this section, we derive Equation [7 which characterizes the gradient flow dynamics under OGD. For any task 5 [T], the parameter 6, evolves according to the differrnniall equation
d0t dt
P, Vo ((0))
nt Σ (J+(2)) - Vot
(34)
where 2 denotes continuous time.
Based on Equation (34. the evolution of the network output satisfies
d {xxx) - (x)) di
do,
1
(f((4))
Pee
nht
(35)
1 R+
-
nrr
j=1
19

---

Published as a conference paper at ICLR 2026
Therefore, under the NTK regime, the kernel gradient flow takes the following form:
-- K, (a, (f((x,)
di f+(x)
(36)
where K-(x, = V (-) P. ( Therefore, the resulting kernel coin- cides with the one derived in Bennani et al. (2020).
E PROOF OF THEOREM
In this section, we provide the proof of Theorem I which establishes the upper bounds of forgetting F1 and generalization error Gr. We first introduce the notion of Rademacher complexity in Subsec- tion E. We then derive the upper bound of the generalization error in Subsection E.2 followed by the proof of the upper bound of forgetting in Subsection
E.3
E. GENERALIZATION AND RADEMACHER COMPLEXITY
There are several ways to quantify the complexity of function class F, one important and widely used measure the Rademacher complexity. Following the notation in Arooa et al. (2019a), we define the empirical Rademacher complexity as follows:
Definition 1. Given a sample set S, = {( y Hi the empirical Rademacher complexity of function class F defined as:
Rs, (F)
1
Ee
ctf(xt)
(37)
JEF
i=1
where (a)) is a vector of ii.dd random variables drawn from the Rademacher distribu- tion, i.e,, ~ Uni
Rademacher complexity provides data-dependent upper bound on the generalization error of learning algorithm (Bartlett & Mendelson 2002).
Theorem 2. Suppose the loss function -(..-) is bounded in [0,c] and is p-Lipschitz in the first argument. Then, with probability at least 8, for all F, holds that
log(2/0)
+ (
- 2pRs(F)
30 2n
(38)
Based on Theorem we state the following corollary:
Corollary 1. Suppose the loss function E(.,)) is bounded in [0,c] and is p-Lipschitz in the first argument. Then, with probability at least 1 for all F. holds that
log(2/8)
- - LD(f) < 2pRs(F) + 30
(39)
2n
Proof. L ((( F). where each gs takes values in [0. c). For any E F. define the population risk T.D(S) [Rp(g and the empirical risk I.s(J) = Es(gr]
From the standard Rademacher generalization bound, with probability at least - 6, it holds that
log(2/5)
Lo(J) S Ls(S) + (Rs(g) + 3eV
(40)
2n
To obtain the reverse direction, consider the shifted function class
g' (z) = -g((2) F)F,,
which also takes values in [0, c] and satisfies Rs' Rs( Applying the same bound to g' yields
Ts(s) Io)( < 2Rs(9) 3e/
log(2/8)
2n
20

---

Published as a conference paper at ICLR 2026
Finally, by the contraction lemma, since E(:, v) is p-Lipschitz in its first argument, have
Rs(9) (F).
Substituting this completes the proof.
Next, we provide an upper bound on the Rademacher complexity of a specific form function class, stated in Lemma
Lemma 6. Let (Ke : X x Xe + R be sequence of positive semi refinief kernels such that supzEx || Ke(x, x)ll < 00 for all E (T]. For each t E (T), let Hy be the reproducing kernel Hilbert space (RKHS) associated with Kr, equipped with inner product (1, : He- Given sequence of positive constants [Brl== we define the function class FT as
Fr ( J:X-RA Σ [()), K, Xt)an, Welll S Be m)
(41)
Then the empirical Rademacher complexity FT over S, satisfies
T B₁ (x,)k((,,,
Rs, (FT) S
(42)
Proof. For any kernel KG, there exists associated feature map $; X4 - Hr such that for all T1.22 E Nr, we have Ke(I1, 22) (fe(11), (x(x))),, In particular, the kernel vector Ka X,) (Kr x!).... ..., Ke K2 xx" and the coefficient vector given by at = a.)) Consequently, for any f E Fr, we have
defined as Ke(I,X,
f(x) Σ f((z)
K2 (x, X()aı
(x,
(43)
Rt (xt)) R.
Let we, = Em, (z Then the function f can be represented as:
T
f(x)
Σ (we, (x(x))
(44)
t=1
Moreover, the squared norm of w, in H, satisfies:
100 1222 =
Σ
afaf (x)), (,,), (x()))
1
=
Σ
afat Ki (x)
(45)
=a, KK(,
=I Il..-
21

---

Published as a conference paper at ICLR 2026
We define the function class Fr as follows:
{ f: x R
Fr
Σ (wr, (x)))))), I V [T]
(46)
By construction, we have Fi C FT. Consequently, the empirical Rademacher complexity of Fr over S. can be upper bounded by that of Fr, i.e.
Rs, (F)) S Rss (FT)
Eg
sup 1 &B, VEIT
(w): (x(x)))) Hil
n4
T
Ee
sup
E 2e (z mml
11.
1lwellite VIETT
1=1
Eel
sup
(wi,
(xt M
1 l1 SB:
T
Bt E |1|
P (2+ [1
T4.
i=1
Bi Ee
(47)
T
Be
TeleeeKK(at,..
nh
1.
T
B.
Belc? (xt,,
B
Σ
ke(x),af)
Thr
T B (IX(
n2+
E.2 BOUND ON THE GENERALIZATION ERROR Gtr
In order to derive an upper bound on the generalization error defined in Equation 4 we utilize the inequality provided in Equation (38). To proceed, will separately bound the empirical loss term Ls, (fT) and the Rademacher complexity term Rs,, (Fr).
Proof. (1) For the term Ls, (ft) for any T E [T], we have:
(s-
|1 (X+) Y+12
2n
T
[1);(X+) +
Σ (x(x))-YI1
2n,
(48)
km++1
- [xx, - Y, 1| - Σ (x)))
k=
Notably, the convention Σ T+= always holds, which is known as the empty sum convention. Therefore, when = T, Equation 48 remains valid.
22

---

Published as a conference paper at ICLR 2026
Next, we compute the term (X,, - Y+11² follows:
(x,) - Y,11 =1ff (X+) + f;(x,) 1+11 -111
=X,) - -211
= =I|KA (X,, )) Br, Kr X,) -1y, - 8:112
(49)
=I - (X,, X-)e )e K. (X,,,, x,) Kr (X X,, 1111
=I =||K((X,, x))e (X, K (X,: X,) -Ý=11.
In order to simplify the result in Equation 449.. we use the Taylor expansion of the exponential function, i.e., =b x X Therefore, we have:
[x)()++ - Y+12 = K
(x,,
1ly
1
Kr (X X₂)
-
K, (X x,) K, X,,,
k!
Th.
(50)
20
tr K= X
=|
Σ
k!
n2
=I|e K. (x)
Xx) 8-11
Using the Cauchy- Schwarz inequality bound:
En voll2 s m Em /vll?, obtain the following upper
T-T Σ
- - n2r
Ls. (f7)
Ka(KK, Xx) "Y +
7T k=T+1
(51)
(2) For the term Rss (Fr), we first consider a bound on the reproduced kernel Hilbert space (RKHS) norm of fr. Let (HK,, || ||Hx,) be the RKHS induced by the kernel K,. We define
a Er,; Kr (X X,)
(52)
Then, fz can be written as:
(+ = (x,
(53)
The RKHS norm of Jr is then given by:
1 a, (X,,
= Y. KX X,) ) 11 Er,,, K2 Kx
(54)
SY, EY,; Er,; Kr (X,, X,) -- = =B,
The final inequality in Equation (54 is easily verified by the following equation:
" - Y.T E (x,, -Ý,7 (X,, Er,r K [X,, X,) E K2 (X,, -
(X,, Er. K, (x. X,, X,
-Y,T (X,, Xx) Er,: Kr (X,, X,) - K,(,,, X,) Kr(X,, [x,, Kr
=Y. Kr(X,, (x) (x) (FF,,);
T)K-(Xx, (X,, Er, K~(X,, -
=-Y,K, (x)
X1e - - KX -
K, X,, Er, Kr (X,, X,) -Y,,
(55)
23

---

Published as a conference paper at ICLR 2026
Based on Lemma 5 since K- (X,,, X,), K2 -,,- -= K.,,,X,.))) and Er,,; all posi- tive semi-definite, It follows that Y- (X,, ET,;; K (X ET, K, (x)+, Y, Er,; ( ",, - 0.
(56)
Therefore, we verify that Equation 540 holds, and we obtain an upper bound for || 11 || which we denote by B,. We define the function class Fr follows:
K, (,, Ilf 11xx, S BB, m.) (57)
FT
f X
E (t)))
Based on Lemma 6 we obtain the following bound the empirical Rademacher complexity of FT over S,:
BR
Rs, (Fr)
n4
(58)
X
Ka (x
Rr
Based on Theorem 2
and by combining Equation
51
with Equation 588) we have:
Lp, (f))
s
T
KR (X Xx. Yx 12 lle (x
k=T+1
(x,
Dx.e; Ku (Xt: Xx) -1 1 12/2
log(2/8)
+ 2p
+ 301
2n,
(59)
By substituting the bound from Equation (59 into Equation 4 we obtain:
T
Gt
1
Σ
[Kg(X,,
Fx Ka(XA, Xx) 1
1 -
l|e my
K. (X, 18111
T
nr
T=1
k=7+1
(X,, Ex Kx Xx) 1/4 1/2
log(2/8)
+ 2p
+ 3e
2n,
k=1
(60)
E.3 BOUND ON FORGETTING Für
In this section, we derive an upper bound on the average forgetting, as presented in Equation
Proof. We decompose each term in Equation 33) as follows:
For any T E [T - -1], we have:
LD. (ft) (D. LD. Ls. (f)) (f+) Ls.
Ls. LD, (f=)
(a)
(b)
(c)
(61)
Next, we derive upper bounds for terms (a), (b), and (c), respectively.
For term (a), by applying Theorem
2 together with the bound in Equation 58 obtain:
(2g
L.,, (f+) Ls, (fF) S 2pRs, (Fr) + /
2n,
T [T, X Exe; KR Xx),
log(2/8)
< 2p
+ +30
n1
2n,
(62)
24

---

Published as a conference paper at ICLR 2026
For term (b), we begin by deriving explicit expressions for Ls, (ft) and Ls, (ff). follows:
T
1 -1 - Y=112 Σ
Ls, (1T) -
(63)
k=++1
Ls. (f+) 2n,
(X 111
(64)
Subtracting the two expressions, we obtain:
(5)) Ls. (J2)
T
ا111 Σ |((+))12 + (x+) -Y11
k=++1
2nt
(65)
T-T
Σ
Xx) Xx)
1 2n
lle KX
n4
For term (c), by applying Corollary
logether with the bound in Equation
58
we obtain:
(s. (D=(ft)
(29()
22pRs. (F+) 30/
2n,
(66)
(X,, E (x -VA-11111/2
l2g(2/8)
22p
+ 30
74
2n.
Then, have:
Xx-)x( Ex, Ka(Xx, Xx) -
Fir
2p
nh
Tr(Ke(XT, X Exg Ku(x - + Boy
log(2/0)
+ 2p
2n
k=1
+
T T-T
Σ
-x)
lle
+ Kx 11
ny
2n,
(67)
25

---

Published as a conference paper at ICLR 2026
F FORGETTING AND GENERALIZATION ERROR OF PGN
We first present the main result in Theorem 3 To help readers quickly understand the proof strategy,
we also provide proof sketch. The detailed proofs are given in Appendix F.I Appendix F.2
Appendix F.3 and Appendix F.4
Theorem 3. Consider sequence of TT tasks. For each task [T], let Dr denote the data distribution, and let S = (X,, Yr} be the corresponding training dataset drawn i.d. from Dr-- Suppose the loss function E(..-) lakes values in the interval [0,c] and is p-Lipschitz in the first argument. Then, with probability at least 1 - 8, the following bounds hold:
T-1
T
(X, (ERE 2
Kx(K,, Xn) -Yx]
PLT
2p
T
R=1
TT(Ka & (DR,
(EE,, -Ya 1/2
log(2/0)
+ 2p
+ 6c//
14.
2n.
k=1
(68)
T-T T
Σ
K Xx Ex Ka (X,, -Yall
Bl
k=T+1
S. KX
(r1(C) [K. (X, XX., x Y:11 }.
||e
2n7
Gt
1
T
Σ
Xx) ( Xx)) "V
T
nly
k=T+1
+
-
1
|le
My
(X,.x.))
(+())
(X,, 11
(69)
7
[Tr(Ka(X,, X (K 2 (Xx, -
log(2/6)
+ 2p
+ 3
n4.
2n,
where EPGN - exp
( K- 4,10) (3(X,, and staissf
(t) = = VIx..X))xXI X. 11x
ds.
proof sketch. Our proof consists four main parts.
(1) Gradient flow of PGN. We first compute the gradient of the PGN loss in Equation 15) and apply
the chain rule (x(()) Vo (x)
di to derive the kernel gradient flow of PGN:
d dt [1()) - K,(,,, n4
a+ (X, -Y-)
Thr
V(+(X.) ", - -Y;)
(70)
The detailed derivation is provided in Appendix F.I
(2) Solution of the kernel gradient flow for PGN. We adopt a similar approach to Appendix D.I to solve the ODE. We first derive the solution on the training set X,:
f(xx)) Y, + exp x2-
K, KX,,
4.(1) n-
K2(X,,, (F(((())))))(()))))))))))))))) -
(71)
We then obtain the solution an arbitrary point 2::
(xx) = (x))) K, X+)
PPNN (x,, Yz).
(72)
The detailed derivation is provided in Appendix
F2
26

---

Published as a conference paper at ICLR 2026
(3) Bound on the generalization error. We standard techniques from statistical learning theory to bound the generalization error of the regularization based method via Rademacher complex- ity (Kakade et al. 2008 Cortes et al. 2010). According to Theorem 2 we need to control the empirical loss I.s, (T)) and the Rademacher complexity of the function class R.s. (Fr).
The empirical loss Ls, (f1-) can be bounded using Equation
722 which yields
1.s, (17) S
T
Σ
[xx(X,,
Xx)
(Xx: Xx) xx) PPp Kr (X+
n4
k=
(73)
1
Kx(,,.X))>
(t)
y
74
We bound the Rademacher complexity through the RKHS norn of f; and Lemma 6
Tr (K,
X,)
k (X,, [Y,] 1/2
(ERNN)
Rs. (FT)
(74)
14
By combining Equations 73 (74) with Theorem 2 we obtain the desired upper bound on the generalization eтror Gt,- The detailed derivation is provided in Appendix F3
(4) Bound on forgetting. We decompose each term in the forgetting metric in Equation (3) as
LD, LD. (fr)) - Ls. (F7) (ff(
(s.)) (F) + LL. (f7).
(a)
(b)
(c)
(75)
for any [T'1]] 1].
Theorem and Corollary limply that terms (a) and (c) are controlled by the Rademacher complexi- ties R.s. (Fr and R. (F+) respectively. The bounds on Rss, (Fr) and Rss (F+) have already been
obtained in step (3). The second term (b) can be bounded as
Ls, (f7) Ls, (ft)
Σ (e(),))
12
1 11f;(x,) - Y-112 2n,
nt
T
[K PGN K2 (X Xx)
(76)
k=r+1
(+()
Kr
K (X V=12
|le
2n
Combining these bounds yields the desired upper bound on the average forgetting. The detailed derivation is provided in Appendix F.
F.I KERNEL GRADIENT FLOW OF PGN
In the CL setting, the training loss of PGN for any task [T] is given by
PGN
(0-) L.s, (0+) + at Vo, Ls, (0~) S.
(77)
Therefore, we have
PGN PGN (0-) Vo, Ls, a, V2, Ls. (0+) Vo, Ls, V (0,))1
VOL S
(78)
27

---

Published as a conference paper at ICLR 2026
Based on Equation (5) we have
Vo, (0,)
((x.)) nh
For any task T E (T], the parameter 8, evolves according to the differential equation
Vo Ls, (0*) d = Vo Lis, (0%) at Vig Ls,
dt
Vo oz Ls. (0+
B (FXx
- (C(X)))) ar Vog Ls, (0+)
n4
Vo (XX
Th
(79)
where 2 0 denotes continuous time.
Under the NTK linearization, we simplify
Vj, (0+) o.. (((-)) 1 ((X,)
1 [o, vo.
n2,
nT
Therefore, we have
d0; 1
Vo fx(x)) (F((,)) - Y7))
dt
n4
W (F((,) - -YG)
(80)
aT Vo: Vo: (ox
T4
(IH(X,)) Y-) (,((X,,
where we used [[Voy (F((x)))) = ( (X,,
Based on the chain rule, we have
d
d
dt f+(x) -V (z)
dt
Vos Vo Y+)
(81)
Vos
n4.
( Y.) XX
Therefore, under the NTK regime, the kernel gradient flow takes the following form:
dt fx()) - nhe
d
Or (x
(X,,, -Y=)
The ((H(X.) -Y, (X,,,,,,) - Y2)
(82)
where (2, = (Voz fin_ Vo: Fir((()
F.2 SOLUTION OF KERNEL GRADIENT FLOW FOR PGN
We follow the approach in Appendix D.I first derive the solution on the training set X,, then extend it to an arbitrary input x.
(1) Solution on the training set X,. Evaluating Equation 82) at X₂ gives
nt K-_K((X,.) d - xx nT V (f((X,) - YY)) K- X, X+) - Y,)
Kr
di (+(x+)
(83)
28

---

Published as a conference paper at ICLR 2026
Let g(t) (X, Y, and abbreviate K, (X+: Then Equation 83 matrix ODE
becomes the
d
1 K ((t) ar K3 g(t) n27
dt (t)
s(0) (X, Y,.
(84)
nt vg(1) K+g(t)
Since K, is real symmetric positive semidefinite and K, commutes with K2 the theory of linear time-varying ODEs with commuting coefficients yields the solution
+1(1) = Lo vg(s) K(g(s)
s(l) exp R2
Kr
2-(1) k? g(0); n+
ds.
(85)
Substituting g(t) (X,, Y, into Equation (85) yields
(x+) Y, eepp
- Kr
4.10 (f+(x,))
(86)
ni
nr
(2) Solution at an arbitrary point x. For any Rdd Equation 82 can be rewritten as
d
fx) Kr X+) g(t) ar Kr X,) K g(()
(87)
dt n4
n Vg(t) (t)
Notably, multiplying Kr Xx) K-² on both sides of Equation 84 gives K, X-) K, d dt g(t)
1
ar K, X,) K g()
K-
nr Vg(l) Ktg(t) dt fx()).
(88)
nt
Integrating Equation 88 over [0, t] yields
f=(x) - (x) (x, (g(() 9(0)).
(89)
Substituting Equation (85) into Equation (89
and using ff(x) = 1x) gives the closed form
f+x) (x) K, (x, X,) KF exp
n, Kr 41(1) K?) 1) ( Yz) f--1(xx) K, (x, X,) EPGN K- 8 (x, Yr):
n
(90)
where EPON = 1 exp 1 K, (.(() K Therefore, we obtain form for PGN that is
analogous to the SGD solution in Equation I1b. The only difference is that PGN uses EPGN whereas SGD uses F,,-- For notational convenience, define
fz())) = K, PGN K, (x)
EPON
(91)
Therefore, the predictor for task T can be written as fx( the notation, we adopt the notational convention ty. fr for the final task T, that is, the iteration index of task T coincides with its stopping iteration.
ΣE ET (x) ff(x). To simplify
F.3 BOUND ON THE GENERALIZATION ERROR
In this section we derive an upper bound on the generalization error. In particular, we bound the population loss Lp, (fT) using the Rademacher complexity of the hypothesis class, the empirical loss Ls. (ft),, and appropriate constants as shown Theorem 2 This approach follows standard statistical techniques for regularization based methods as in Kakade et al. (2008); Cortes et al. (2010). Specifically, the regularization term affects the bound through its influence on the Rademacher complexity of the function class.
29

---

Published as a conference paper at ICLR 2026
(I) For the term Ls, (f7) for any [T], we have:
1
Ls, (Sī)
2n, (x+) - -111
T
(92)
(X+)
+
Σ
2n,
kmr+1
Next, we compute the term [X,) - Y=112 as follows:
( - Y+12
-X,) (xx,) 11
- - -2,12
(X,,, D Kr (X r. -x)- -
(93)
0. (x-
=1 K-(X,, X-)e K- (-)
n4
Kr (X,,,,
(+) -x)-1"y- -1112
(+(())
+/f Kr (X+
Irr (Xx,)) Kr (x,; -
=I K|| (X,, X,
= =|le k.(,,X).) 9, (t:)
(X+ 1r11.
n²
Therefore, obtain the following upper bound:
Ls. 5
[K,, Xx) Xx)
nlr
(94)
t (X X.) (t(( (Xr.,,) Y=113.
+ lle
71
(2) For the term R(Fr), we first consider a bound on the reproduced kernel Hilbert space (RKHS)
norm ft. Let (Hk,, [|xx,) ) be the RKHS induced by the kernel Ky. We define
å = EPGN K2 (X,, (X,,
(95)
Then, f; can be written as:
(96)
The RKHS norm of fü is then given by:
|l a (X,,
(X,,
PGN K2 (X Xx) Ex) EPNN (X,, Brdt
(97)
= - - -
YÝ := B?
We define the function class Fr as follows:
11, &B, m)
Fj f R Σ f()) =
(98)
Based on Lemma 6 we obtain the following bound the empirical Rademacher complexity of Fr over S,:
BB
Rs. (Fr) <
(TT(Ka(X,,Xx)))
n
(99)
(X,,(XX,
EPGN ( K -x)
Rr
30

---

Published as a conference paper at ICLR 2026
Based on Theorem we have:
T
(f7)
Σ
(Xr, Xx) Xx) PON K Xx) 11
11
k=T+1
1
13 (x(X,,X,))
x+)
(1-) Kr (X+,)))) 8=112
+
my
74
T
[T(KK X,
(EE Kv (Xx, Xx) -x)-1
log l2g(2/6)
+ 2p
+ +3c
2n,
1=1
(100)
By substituting the bound from Equation
400 into Equation
we obtain:
GT
1
Σ
[K Xx) EE
PPNN KR (Xxx Xx) -x)
1=1
k=
+
||e
Kr
(x,,,,))* 8=11
(101)
11g
[Tr(Ka(X,, X.DD
EPONY 2 Xx) -V611
log(2/6)
+
2p
301
2n4
F.4 BOUND ON FORGETTING
We decompose each term in Equation
follows:
For any T TE [T - 1], we have:
(ft) Lo. =
- Lss (f))
Ls. - (f+
(1) (f+).
(a)
(b)
(102)
Next, we derive upper bounds for terms (a), (b), and (c), respectively.
For term (a), by applying Theorem together with the bound in Equation 99 we obtain:
(f.(fi) (s-
log(2/8)
<2pRs. + 3
2n,
(103)
T Tr(Ka(X,, X
(EPGN) K -18)1111
log(2/8)
22p
+ 3c
n,
2n,
k=1
For term (b), we begin by deriving explicit expressions for Ls. (ft) and Ls, (f+), follows:
T
Ls,(ft) (f+) < 1 - - 1=112 T T-T r
Σ
1x)((
(104)
14
n4 R=T41
Ls, (f:) 2n,
[x= 1-
(105)
31

---

Published as a conference paper at ICLR 2026
Subtracting the two expressions, obtain:
L.s. (17) Iss, (f?)
T
- T
(x+)(1 +
(X+) Y+11?
14
2n,
k=T+1
(106)
T
[ X
EPGN K E
Xxx Xx. xx) "
k=T+1
1-((+) [K, (x X, 1| Y=11
(X,,
+ 2n1
For term (c), by applying Corollary together with the bound in Equation 99 we obtain:
Ls, (ft) LD,(Ft)
1og(2/0)
2ppss (Fr) /
2n
(107)
(X,,
(FE. n1
(EPGN) (X
log(2/8)
22p
3e1
2nF
k=1
Then, we have:
(X,
EPPNN K2 (X,, Xx) 1Yx]1/2 R
PT
2p
(X, (EPGN ( (X,, Xx) 1Y,1//2
log(2/8)
+ 2p
+ bCy
n4.
2n,
(108)
+ T-T Σ
K (X(( ER
PON KR (X2, Xx) -
nl k=T-1
2-(G) (X.,,.) ) 8:
(X,,x,)
ur
2n0
G ADDITIONAL PROOFS
G.1 PROOF OF LEMMA
Proof. To better reflect practical training scenarios, consider a finite number of training iterations, denoted by ET and restrict tr to the interval [1, eman). We exclude the trivial case tr = O, as it corresponds to the stopping point of task T 1. Accordingly, we analyze the evolution of the upper bounds Gupper and tr over [1, mmax..
(I) We first analyze Gupper, Its derivative with respect to tr can be written as
dG
(IT() + (92((T)),
(109)
dtr
where the functions 91 and 92 are given by
T 2(T 2 T VTT
(t(t)) = Σ
(K((Nr, XX,
XT) -
ST (XT..X)))
n2
1=1
yT 2 Kr KT XT)
?
(110)
32

---

Published as a conference paper at ICLR 2026
and
Tr(Kr(X,,, x,))] 1/2
yf
I Kr ( (Xr,NT) mx
g2(tr)
1TT
[Yg (T
2x. Kr (Xr, Kr (Xr, - "
(111)
We first assume that [X, x,)) 1/2
* 0 for any E [T], and that Yy * O. This assumption mild, since if either term equals zero, the corresponding component can simply be omitted. From
the proof of Lemma we know that up (T)) is positive definite. Consequently, (2 holds for all tr E [1, TT
Moreover, both 91 and 92 are continuous with respect to br on the interval [1, tTTT). We therefore define
mi
min (1.
(2(x)
Mi = max 191 (x) = (
Let py = Mi. mi
Then, for any > Pg. we obtain
dG
1
T (0r(1))) pg2(tr))
dtr
7 (IT( + M1
92 (TT))
(112)
mi
7 (9l (tr) + Mi)
Therefore, for any Lipschitz constant > Pg. G.r is monotonically increasing with respect to IT-
Since
Tr( Xo) 0,
(113)
n4.
follows that a smaller Lipschitz constant directly results in smaller values G for any fixed
dr.
(II) Then, we analyze Expee similarly. Its derivative with respect to t can written as
dFupper
T-1 T-1 pfr(t)),
(114)
dtr
where the functions fi and fz are given by
fi(tr) = Σ 2(T T) y
2x (xr. ux
Kr X, (X,,
nT
(115)
'r (I
Ky ( (N Xx) -V
"T
and
T-1 Tr(Kr X/))
y
cr Ky KxT, Y oT
f2(1T) =
TyT
Y (I
27. m7
Kr (Xr. Ky (Ny, Xx) 1 1/2
(116)
We adopt the same assumption in Part (I). Hence, fa(tr) 0 for all tr [1, 4max Moreover, both f1 and f2 are continuous with respect to tr on the interval [1, tpus We therefore define
m2 = min f2(x) >
M2 = max
[x)]-
ze p
2E(1, moo))
33

---

Published as a conference paper at ICLR 2026
M2. Let Ps mi
Then, for any p Ps. obtain
dF de
1
T T-1 (fi(IT) pf2(tr))
dtp
T (fi(tT) + M2 f2(t))
1
m2
(117)
1
T (fi(IT) + M2)
M 0.
Therefore, for any Lipschitz constant > Ps. the bound bo is monotonically increasing with respect to IT. Let p² == max pat. It then follows that both Gty and ptr are monotonically increasing tr whenever p p
In addition, since
(1,, Ex.q Kx Xx)) (Yx)' 1/2
0,
(118)
11.
and
(X, X+)) Ek. Ka Xx)
-Yx 1/2
0,
(119)
follows that a smaller Lipschitz constant p directly leads to smaller values of E for any fixed lT.
G.2 BRIDGING PREDICTION- AND PARAMETER-LIPSCHITE CONSTANTS
We first introduce types of Lipschitz constants associated with the loss function. For any fixed label the prediction Lipschitz constant P measures the smoothness the loss function {(-.) with respect the model prediction. Specially, for any 21, 22 E Z,
(1(21,8) ((22 v)1 S Ps || - -22ll-
Similarly, the parameter Lipschitz constant pe characterizes the variation of the loss with respect to the model parameters. Specifically, for any 01, 02 E Q and any sample (x, y).
[e(So, (z). e(o, (x). S Po |00 02ll-
By the mean value theorem and the Cauchy-Schwarz inequality, there exists a point z' on the line segment between 2₁ and 22. and point 0' between 0, and 02. such that
((21 v) { v\1 s |V v)ll || -21l.
le(f, (z), ((Se, (x), U)r < [V (z), v)ll - 021l-
Therefore, the magnitudes of P and Pa are upper bounded by the supremum norms of the prediction gradient ,||l and the parameter gradient /Velll. respectively.
For any individual sample x, let its prediction be f(z). By the chain rule,
Vol(z,v) = (x)) TV.()( V=((z,v)
where (x) 0f(z)
3े denotes the Jacobian matrix of the model output with respect to parameters
at input x. By properties of singular values, the sample-level gradient norm satisfies the following two-sided bound:
miin (J(()))(())))))))))))))))))))))) [[, w)ll (Voe(z.)))l < [[,(()) [V((2,))).
34

---

Published as a conference paper at ICLR 2026
Consequently, in regions where the Jacobian is well conditioned, i.e., there exists constant such that mnin (1))( 2 m for all samples I, we obtain
|V S - [lVo
This inequality indicates that controlling the parameter gradient norm | the prediction gradient norm [ thereby locally regulating the prediction Lipschitz constant Pf-
effectively constrains
For simplicity of presentation, we do not distinguish between the prediction and parameter Lipschitz constants in the remainder of this paper and refer both collectively as the Lipschitz constant.
G.3 PROOF OF LEMMA AAMMADANND DISCUSSION
We first present the proof of Lemma as follows.
Proof. Consider any entry KA (x (.]__1,)]] xy))) with i E [mt-1]] and [ma]]:
x = (Pk Vo (1 1). P& 1
(120)
By definition, the subspace Ex-1 = span Voi_
fi ,
(o
P Since Vo; fi e
E applying the projection operator Ps yields Px V 0 Sh- (ot 0. Therefore, it follows that
=x( = (R =0, vi E [m j E [mx), E [mx-1]-
If we define the projector Ps Pot.. then Lemma 2 still holds, since Ve,, (=k-1) = remains satisfied. This ensures that the gradients of the current task are orthogonal only to those of the immediately preceding task.
Further discussion of Lemma 2
G.4 PROOF OF LEMMA 3
We present the proof of Lemma as follows.
Proof. Consider any entry (x 2f) with E [n=] and E [me]:
ko(x, x() = Vh Pa fi (at))
(121)
By definition, the subspace Ey span Vag fx (z)") 1 E [k], [m:]]: Since Vo (xt
E applying the projection operator Pk yields P P/ 0: - (x) = 0. Therefore, follows that
RR(o,, xf) Ke(at, = 0,, ei [n+), j [mx]. E [nz].
G.5 FORGETTING AND GENERALIZATION ERROR BOUNDS FOR OGD AND OGD+
In this section, we derive upper bounds on forgetting and generalization error for both OGD and OGD+. Define Ér,
exp (- n. (X,.,, and É., = exp 12+ K (X,,,,
Theorem 4 (OGD). Consider sequence of tasks. For each task E [T], let D, denote the data distribution, and let S, = {) be the corresponding training dataset drawn i.i.d. from D,, Suppose the loss function f(..:) takes values in the interval [0, c) and is p p-Lisschitz in the first argument. Then, with probability at least 8, the following bounds hold.
35

---

Published as a conference paper at ICLR 2026
T
(X, X Be.eg (Xx,
Pbr <
2p
T
n4
T=1
k=1
(X,, Eus (Xk: Xx)
1/2
log
+
2p
n4
2n-
k=
+
T T-r
[Kx(X,, Ex),,, (x)
Rr ||e
(X+ (x)
fly k=T+2
2n,
(122)
Gb S
1
T
Σ
|| X Ek,, (X,, -Vall
+ lle
1
(X xx
111111
nl
nt
N=T+2
Xx,))) Ex,,; (Xx, Xx) 1/11/22 3eV
(2g()
+ 2p
ny
2n,
km
(123)
Theorem 5 (OGD+), Consider a sequence of T tasks. For each task T E [T], let Dr denote the data distribution, and let S, - (X,,Y=) be the corresponding training dataset drawn i.i.d. from D,. Suppose the loss function ((..-) takes values in the interval [0, c] and is p- p-Lipschitz in the first argument. Then, with probability least 1 the following bounds hold.
T T-1
Tr(Ka X,)), Ex.1; (XR, Xx) (Y, 1/2
(2g()/))
Fir
2p
60
T
2n-
T=1
k=1
T
X
Fant RR (Xx, Xx) -1)) 1/2 1/2
1
# k. 81112
+ 2p
+
lle
nr
2n,
(124)
T (X,
En.; -1y
1
G S
2p
T
nt
1=1
kml
(125)
(29(2/8)
+ - lle Xx + 30V
11
2n,
Therefore, the bounds on both forgetting and generalization error for OGD in CL are tighter than those for standard SGD. Furthermore, OGD+ achieves even tighter bounds than OGD, providing stronger theoretical guarantees.
G.6 PROOF OF LEMMA 4
Based on the bounds established in Theorem we characterize the evolution of Ftr
pupper and Cupper
by
Proof. Similar the proof in Appendix G.1 and uppprr+ over the interval [1, tmpx).
analyze the evolution of the upper bounds Gupper+
(I) We first analyze Gupper+ Its derivative with respect to lT can be written as
dG +
1
T (o1(t)) pg2(tr)),
(126)
dtr
36

---

Published as a conference paper at ICLR 2026
where the functions 91 and 92 given by
2
Yx e by Rr(xxx (X (T(X
gi =
(127)
nf
and
Tr Tr(KT x,))]
Yr e
T Kr (Xr, YT
g2(1T):
Σ
Lx Kr (xr, Kr (XT. XT) - 1/2 "y
YT (I
(128)
Since KXrr is positive definite,
We adopt the same assumption in the proof of Lemma
26 Rr (Xr, (XT, Xr) is "T
it follows that 92 (4()) > 0 for all lT [1, Ampx). Moreover, since positive definite, it follows that 91 (fr) < 0 for all tr E [I, t7*).
Moreover, both 91 and 92 are continuous with respect br on the interval [1, (7 We therefore define
M1
max 91 (1(x) > O, #E 1
M2 max 92(x)) 0.
#F (1
Let P/ = Mj. Then, for any p Pyg we obtain M2
dG
1
T (9l(t)) pga(tz))
dtp
T (gi(tT)
1
M (2
Mi
M2
(129)
$ T (9t(t))+ +
5..
Therefore, for any Lipschitz constant p < P/g Cuppert is monotonically decreasing with respect to ly.
Since
(X,, (x), 2 0,
(130)
follows that smaller Lipschitz constant p directly results in smaller values of Gumpert for any fixed
tr.
(II) Then, we analyze pupper+ similarly. Its derivative with respect ty can be written as
uppert P dtr T T- fe(tr).
(131)
where the function fe is given by
f2(1T) (X,, X.))
(Xr. Yr cr Y
my
TTT
T (I
"
Sr. (Xr:))) K (Xr, (Nr, Xx.) - 18/11/2
(132)
Sr. (xX:.XXr)) is positive "T
We adopt the same assumption in the proof of Lemma Since
d
definite, it follows that f2((T) 0 for all lT E [1,1) which leads to puppert is monotonically increasing with respect to lr. Fer
Therefore,
der
In addition, since
(x, Xx) 2 0,
(133)
37

---

Published as a conference paper at ICLR 2026
and
(X,, (x Xx)) -
-Yx]
2 0,
(134)
14r
follows that smaller Lipschitz constant p directly leads to smaller values of Fupper+ for any fixed lr
38


Reviews:
[{"review_id": "dibWZT0pCl", "reviewer": "Reviewer_5JLu", "summary": "This paper addresses the theoretical properties of CL in the NTK regime in the intermediate training stage, and give two insights on the role of Lipschitz and cross-task kernel on CL performance.\nWith these insights, they give two refined versions of the OGD algorithm.\nThese two algorithms are verified on standard CL datasets.", "questions": "How do you think the two theoretical messages can benefit other CL algorithms?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is a good paper in both theoretical and practical aspects. \n\nOn the theoretical side, intermediate-stage CL performance analysis is novel to the theoretical CL literature. Also, the Lipschitz observation is novel to the CL field. The cross-task kernel result is valid and kind of expected.\n\nOn the practical side, with the two theoretical messages, the refined algorithms refined a well-used CL algorithm on these aspects and improved its performance on standard basic CL datasets.\n\nThe logic is clear. The mathematical tools used are standard.\n\nI am pushing this paper for acceptance.", "weaknesses": "I do not see major weaknesses in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "drvBcrbvTj", "reviewer": "Reviewer_Q8vv", "summary": "The paper addresses the challenges of continual learning (CL), specifically focusing on the dynamics of forgetting and generalization. Continual learning aims to enable models to learn tasks sequentially without forgetting previously acquired knowledge. The authors analyze these dynamics within the Neural Tangent Kernel (NTK) regime, addressing two main challenges: characterizing intermediate training stages and establishing forgetting bounds using population risk bounds. They introduce two novel algorithms, OGD+ and Orthogonal Penalized Gradient Descent (OPGD), both of which project gradients orthogonally to reduce forgetting and enhance generalization. Their empirical studies on benchmarks like Permuted MNIST, Rotated MNIST, and Split CIFAR-100 validate the theoretical predictions and demonstrate the effectiveness of their approaches.", "questions": "1. **Computation Complexity**: Is the projection step computationally heavy for complex NN? How to address this problem? \n\n2. **Expansion Beyond NTK**: How would the authors suggest overcoming the limitations of the NTK-based analysis when dealing with finite-width networks or other types of architecture beyond classification tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Theoretical Contribution**: The paper extends the NTK-based analysis to intermediate training dynamics, which is crucial for understanding real-world continual learning scenarios. By providing both upper and lower bounds on population risk using Rademacher complexity, the authors offer rigorous theoretical insights.\n    \n2. **Algorithm Design**: OGD+ and OPGD are innovative in their approach to mitigate forgetting and improve generalization. The use of gradient orthogonality and gradient-norm penalization represents a well-founded strategy for continual learning models.\n    \n3. **Empirical Validation**: The experiments conducted on various benchmarks, such as Permuted MNIST and Split CIFAR-100, confirm the theoretical discussions, demonstrating the practical efficacy of OPGD and OGD+ compared to standard methods.\n    \n4. **Clear Presentation**: The paper clearly outlines its contributions, setting a solid baseline through comparative analysis with other methods in Table 1. This provides context for the improvements made by the proposed methods.", "weaknesses": "1. **Incremental Algorithmic Novelty**:  The algorithms are mainly based on OGD and gradient norm penalty, which seems to be a little bit incremental. \n\n2. **Limited Applicability**: The theoretical framework is developed under the NTK regime, which might not encapsulate the full behavior of practical deep networks with finite width and more complex architectures. This limitation is acknowledged in the paper, suggesting a need for exploring applicability to diversified settings.\n    \n3. **Specific Task Benchmarks**: The empirical tests are confined to a specific set of benchmarks, potentially overlooking performance variations across other task types. While benchmarks like Permuted MNIST and Split CIFAR-100 are standard, testing across a broader spectrum could provide more insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n7qxMcVons", "reviewer": "Reviewer_dpae", "summary": "The paper analyzed continual learning in the Neural Tangent Kernel regime and derive intermediate-training bounds for both forgetting and generalization using kernel gradient flow and Rademacher complexity. They show that two quantities govern CL behavior during training: (i) the Lipschitz constant of the loss w.r.t. predictions—smaller is better for both forgetting and generalization, (ii) the cross-task kernel smaller norms reduce interference and forgetting. Building on these findings, they propose OGD+, which projects the current task’s gradient onto the orthogonal complement of a subspace formed by the previous task’s gradients evaluated on all earlier samples (eliminating cross-task kernels between any task pair), and OPGD, which augments OGD+ with gradient-norm penalization to lower the Lipschitz constant.", "questions": "Above", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is comprehensive, well written and well structured.", "weaknesses": "1. What means intermediate training?\n\n2. The continual-training protocol is unclear-for example, how does the transition from task $k$ to task $k+1$ work? After training $t$ steps on task $k$, how is gradient flow for task $k+1$ initialized and defined relative to the state from task $k ?$\n\n3. The assumption also requires a bounded loss function.\n\n4. The paper did not specifically provide the forgetting and generalization error of PGN and PGN, only lemma 2 and lemma 3 provided, which is not enough; it should give the formal results like Theorem 1.\n\n5. The comparison among related work is not sufficient. There are lots of theoretical CL works that consider the regularized setting, but the paper did not discuss.\n\n6. The proposed methods are common in empirical CL studies, while the theoretical CL findings are not enough in my view.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h6hcV65YYq", "reviewer": "Reviewer_iWjj", "summary": "This paper leverages the Neural Tangent Kernel (NTK) to derive upper bounds for both forgetting and generalization error at any step during training in vanilla CL tasks, while the bounds explicitly depending on the training steps. Two key conclusions are drawn: 1. Reducing the loss function’s Lipschitz constant (ρ) simultaneously decreases forgetting and generalization error. 2. Decreasing the magnitude of the cross-task kernel also mitigates forgetting and improves generalization. Based on these theoretical insights, the paper proposes two methods, OGD+ and OPGD, which project the current task’s gradient onto the orthogonal complement of the subspace spanned by\nthe gradients of recent tasks on all historical samples. Additionally, a gradient norm penalty (GAM) is introduced to jointly enforce “orthogonal constraints + Lipschitz reduction,” thereby suppressing forgetting while enhancing generalization. Experimental results demonstrate that the proposed methods significantly reduce errors. The writing is clear, the methodology sound, and the experimental outcomes show significant improvement", "questions": "1. Is there a plan to evaluate them on more realistic tasks, such as online visual data streams or natural language sequences?\n\n2. If the model deviates from the NTK assumption (e.g., CNNs), have different behaviors been observed in terms of forgetting and generalization mechanisms?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper models the error upper bound as an explicit function of the training step, allowing it to not only identify the main factors influencing the error but also discuss the effects of “training longer or shorter” on the final performance.\n\n2. The main theorem clearly provides upper bounds for both forgetting and generalization errors, explicitly highlighting their dependence on training steps. By deriving a closed-form solution through kernel gradient flow and incorporating Rademacher complexity for two-sided control of the overall risk, the paper establishes a solid theoretical foundation. It further proposes a corresponding improvement method (OPGD) and proves its superiority, forming a coherent and logically complete framework.\n\n3. The paper is organized around a clear framework: problem → challenges → theory → analysis → algorithm→ experiments → conclusion. The overall exposition is thorough, and the figures and tables effectively present the results in a clear and intuitive manner.", "weaknesses": "1. OPGD still requires storing a large amount of gradient data, leading to high computational and memory costs.  This may limit its applicability to large-scale models and tasks.\n\n2. Beyond the current experimental datasets, it would be valuable to introduce more complex and realistic scenarios to further validate the robustness and generalization ability of the proposed algorithm.\n\n3. Theoretical analysis occupies a large portion of the paper, while the main experimental results are relatively limited, making the demonstration of the method’s advantages less compelling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)