PAC-Bayes bounds for cumulative loss in Continual Learning
Authors: Lior Friedman
Ron Meir
Venue: ICLR 2026 Poster (2026)
Keywords: Continual Learning
PAC-Bayes
Generalization bounds
Lifelong Learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: Upper bounds on the loss accumulated during online learning (offline or online)

Abstract: In continual learning, knowledge must be preserved and re-used between tasks, requiring a balance between maintaining
good transfer to future tasks and minimizing forgetting of previously learned ones. As several practical algorithms have been
devised to address the continual learning setting, the natural question of providing reliable risk certificates has also been raised.
Although there are results for specific settings and algorithms on the behavior of memory stability, generally applicable upper bounds on learning plasticity are few and far between. 

In this work, we extend existing PAC-Bayes bounds for online learning and time-uniform offline learning to the continual learning
setting. We derive general upper bounds on the cumulative generalization loss applicable for any task distribution and learning
algorithm as well as oracle bounds for Gibbs posteriors and compare their effectiveness for several different
task distributions. We demonstrate empirically that our approach yields non-vacuous bounds for several continual learning
problems in vision, as well as tight oracle bounds on linear regression tasks. To the best of our knowledge, this is the first general upper bound on learning plasticity for continual learning.

Published as a conference paper at ICLR 2026
BOUNDS FOR CUMULATIVE LOSS IN CON-
TINUAL LEARNING
Lior Friedman
Ron Meir
Department of Electrical Engineering
Technion Institute of Technology
Haifa, Israel
lior f wwcaacccc.......co...ommu. www.teechhnn...o ac. i1, rmeirdee. tec.niin... ac.il
ABSTRACT
In continual learning, knowledge must be preserved and re-used between tasks. requiring balance between maintaining good transfer to future tasks and mini- mizing forgetting previously learned ones. several practical algorithms have been devised to address the continual learning setting, the natural question of pro- viding reliable risk certificates has also been raised. Although there are results for specific settings and algorithms on the behavior of memory stability, generally applicable upper bounds on learning plasticity are few and far between.
In this work, we extend existing PAC-Bayes bounds for online learning and time- uniform offline learning the continual learning setting. We derive general upper bounds on the cumulative generalization loss applicable for any task distribution and learning algorithm as well as oracle bounds for Gibbs posteriors and compare their effectiveness for several different task distributions. We demonstrate empir- ically that our approach yields non-vacuour bounds for several continual learning problems in vision, well as tight oracle bounds on linear regression tasks. To the best of our knowledge, this is the first general upper bound on learning plasticity for continual learning.
INTRODUCTION
Continual learning machine learning setting in which collections of examples, known as tasks, arrive sequentially. These tasks may be different skills and capabilities, represent changes in the data distribution over time, or encapsulate different contexts or environments. Since tasks change over time, continual learning is also referred to incremental learning lifelong learning. Due to the limited model capacity and the sequential nature of the continual learning setting, issues often arise in adapting the model to new tasks as they appear while also preserving its performance on previous tasks and thus avoiding Catastrophic Forgetting (Goodfellow et al.) 2015 Kirkpatrick et al. 2017). where performance on previous tasks degrades significantly the model adapts to new tasks. This dilemma is facet of the trade-off between learning plasticity and memory stability (Wang et al., 2024a), two key aspects the learner's behavior during the continual learning process.
There are several methods and algorithms aiming to effectively resolve this tradeoff in various con- tinual learning scenarios (see Wang al. (2024a)), such via regularization (Kirkpatrick et al. 2017], replay of data from previous tasks (Rebuthi et al., 2017) or other methods. Due to the in- herent trade-off between forgetting and forward transfer, metrics to evaluate continual learning al- gorithms differ. Common metrics include the average accuracy of the model for all previous tasks, memory stability measures such as forgetting, and learning plasticity measures such as intransigence Chaudhry et al. 2018), forward transfer and cumulative eπror incurred in each task.
While several empirical methods aimed to tackle the challenges of continual learning have been proposed in recent years, there are significantly fewer theoretical works aiming to analyze the prop- erties of continual learning problems and provide estimates or guarantees on overall performance. Some works (Evron al. 2022 Doan al. 2021: Benavides-Praze derstanding the behavior of forgetting, an interesting topic in and of itself, while other works (Lin
& Riddle, 2022) focus
et al. 2023
Li et al. 2024 Bennani & Sugiyama, 2020 Zhhao al. 2024 Levinstein et al. 2025)

---

Published as a conference paper at ICLR 2026
focus on average model loss for specific settings such as continual linear regression the Neural Tangent Kernel (NTK) regime (c.g. very wide neural networks). One promising direction to de-
rive more general performance guarantees is the PAC-Bayes framework (McAllester 1999; Catoni
2004
Alquier 2024). Existing upper bounds on meta-learning (Pentina & Lampert 2015 Amit &
Meir
2018 Balcan al. 2019 Chen et al. (023) and online learning (Haddouche & Gued] 2022)
offer guarantees for the loss on unseen tasks as well as the online setting that serves as a special
case of continual learning. A recent paper by Friedman & Meir (2025) derived general PAC-Bayes bounds on forgetting in the continual learning setting. To the best of our knowledge, however, there are no PAC-Bayes bounds for the cumulative loss in the continual learning setting. The cumulative loss and plasticity in general has received a surge of interest in recent years (Wang et al.
2024b
Dohare et al. 2024 Kumar et al. 2025).
In this work, we extend existing PAC- PAC-Bayes bounds for online learning (Haddouche & Guedj 2022) and time-uniform offline learning (Haddouche Guedj] 2023; Chugg et al. 2003) the continual learning setting, allowing us derive the first (to our knowledge) algorithm-agnostic risk certificates for learning plasticity via the cumulative loss, We derive bounds applicable for bounded and sub- Gaussian losses for offline and online continual learning. Equation 3] for instance, suggests that we continually re-train a model daily over a year, only a few dozen examples are required to provide effective high-probabillty risk certificates for the expected cumulative loss. We also analyze our bounds as oracle bounds and compare their effectiveness for several different task distributions to provide additional insights into the relationship between forward transfer. model complexity and task similarity. We demonstrate empirically that our approach yields non-vacuous bounds for several continual learning problems in vision, as well as tight oracle bounds on linear regression tasks.
BACKGROUND
2.1 PAC-BAYES BOUNDS
We provide here a brief, high-level overview of several key notions related to PAC-Bayes bounds. For more complete introduction including detailed proofs, see Alquier (2024). In the classi- cal supervised learning setting, the learner attempts to learn an unknown data distribution D via a training dataset of size m, S {=1,... 3 where 2, E Z. A standard assumption is that the training dataset is sampled i.i.d. from D. A learning algorithm assigns a probability to each hypoth- esis h in hypothesis class H (a deterministic algorithm selects a single hypothesis). A loss function e: Z R U (0} (e.g. classification error) is used to measure the performance of a learning algorithm. Since D is unknown, the expected risk L(,, D) A Ez-DE(h, 2) cannot be minimized directly, and so a generalization bound would upper bound the gap between and the empirical loss L(h, S) = 1 Em E(h, Aayyeeyes bounds provide an upper bound on this gap with high probability w.r.t. the empirical sample S. A classic example is Catoni's 's bound (Catoni 2004)
Theorem 2.1. (Catoni's 's single task bound) Assume Vh E H. E D, E(h,,) E [0. K]. Let P E M(H) be some data-free prior distribution over H. Then, for any X > O. for any 8 E (0, with probability at least 8 over the choice of S. uniformly for all posteriors Q E M(H).
L(Q. < S(Q 1 (xL(())))+
XK²
log 1/8
8m
X
Of note is that this upper bound applies uniformly over all posteriors, implying that we can optimize the r.h.s. w.r.t. the posterior Q, and that the bound strongly depends on given data-free prior. Recent work such as Pérez-Ortiz et al. (22T) showed that using part of the training set to learn the prior can lead to tighter bounds in practice. Notably, all of the training data may be used for finding the posterior Q While we will not go over the full proof structure for Catoni's bound, we note that there are three main components: (1) Markov's inequality (2) A change-of-
inequality e.g. (Donsker& & Varadhan 1975) (3) Hoeffding's 's inequality or extensions thereof. These components or some variation of them are common elements used for most PAC- bounds.
2.2 CONTINUAL LEARNING AND PAC-BAYES BOUNDS
In order to obtain upper bounds on the cumulative loss for continual learning. we must first define sufficient theoretical framework for describing the learning process in question. Figure F provides
2

---

Published as a conference paper at ICLR 2026
an overview of the continual learning process using the same symbolic terminology described be-
low. We follow a similar framework as Haddouche & Guedj (2022 2023), adapted to the continual
Average error:
((0,.D)) 2(0-2.) + 440,
2
22
S
S
Sr
Q2
Qr
Cumulative error-frou 2g)
2,0
Depiction of the continual learning process. data-free prior P adapted to task D via an empirical sample S 'DT, resulting in posterior Q1- The posterior's expected (test) loss C(Q P1) added to the cumulative loss. Q1 serves as the new prior for the next task D2 and so on until we reach final posterior Qr. The cumulative loss is the sum of errors for all tasks.
Figure I:
learning setting by changing each task from single example to a set of samples
Framework Consider a data space (Ia, U1) 2, where スς E X, De E V. We fix an integer sample size m 0. For our continual learning process, we consider finite sequence of tasks of length T, where each task E [7] is associated with an unknown data distribution D, We assume that for each task we are given an i.dd sample of size m from the corresponding data distribution, S, E D Our complete .2....,,,))
X' y defined as pairs or inputs and outputs, i.c. 2,
data sample is therefore composed of the sequence (S:)?= & assumptions regarding the relation between tasks or the length of the task sequence, and our results seeuuee are applicable for any T > 0. We denote our hypothesis space H. such that E H is a function h X' y and denote the space of probability distributions over H as M1(H).
We make no
We set a sequence of priors, starting with a data-free prior distribution P1 = P E M1(H). and (Pr)122 such that each P. (H) is -=-r-measurable, with (Fi)t20 being an adapted filtra- tion to (S,)==1- We denote (Q.)__ sequence of posterior distributions such that Q2 is absolutely continuous w.r.t. P.. Similarly to Haddouche & Guedj) (2022), we introduce the notion of stochas- tic kernels (Rivasplata al., (020) as data-dependent measures within the PAC-Bayes framework, allowing us to define bounds with data-dependent priors. We denote Σn the o-algebra on H.
Definition 1. (Stochastic Kernels) A stochastic kernel from S E 2 to H. is defined Q: TH x Σn + [0, 1] where (1) For any E En, the function S - ( B) is measurable. (2) For any S 2m the function B -> QSS, B) is a probability measure over H.
mapping
Stochastic kernels allow us to refer to a distribution dependent on dataset S, marking Qs = Q(S.-).
Definition 2. A sequence of stochastic kernels (P:)T is called an online predictive sequence if (1) For all & 2 1, S tm (....)) -) is F.- -measuauuee and (2) for all t 2 2, P.(S..) is absolutely continuous w.r.t. (S., We will also use the notation Pt: to denote such kernel.
This definition of online predictive sequences allows for the description of stochastic kernel (and posterior distribution for hypotheses) that changes as new tasks arrive sequentially, and can be used to describe a variety of continual learning algorithms.
Definition 3. A loss function é H. - + U [0) is function mapping a hypothesis h and data sample 2 to the set of nonnegative numbers. The expected loss for E is L(h, D) & Egepe(h, 2). The empirical loss of a hypothesis w.r.t. a sample S E 2 is [(h, S) A 1 , 2(h,
We assume for the sake of convenience that the loss function E(h, 2) is task-agnostic, but our results are applicable even if the loss function depends on the task identifier. We restate the main result Haddouche & Guedj] (022) for online learning using our terminology in Theorem A. Finally, we define the cumulative loss (CuL) and the average loss (AL). We note that this paper will focus on the cumulative loss, corresponding to notions such as forward transfer (FWT). intransigence and learning plasticity rather than the average loss (corresponding average accuracy). For more in- depth overview of these metrics, see Wang et al. (2024a). We note that both metrics refer to expected (test) errors.
3

---

Published as a conference paper at ICLR 2026
Definition 4. For given online predictive sequence (Qt), sequence of tasks Di...., Dr, and sample SIT = .S.....,ST) ~ Di x ... x DT. the cumulative loss (CuL) is defined as
CUL((QU) P.J
Bi [E [e 2 F.]]-
The Average loss (AL.) is defined
T AL(Qr) = Σ S. - Br PRIND,le(hr. 21)
As we can see from the definitions, the CuL is measured w.r.t the entire predictive sequence sequentially, and can be extended to new tasks by simply adding an additional element for task T + 1. The AL, the other hand, is measured retroactively for single posterior w.r.t. all tasks, must be re-calculated from scratch for each new task, and requires additional memory propor- tional to the number of tasks in order to estimate the expected loss for each task. We note that both terms differ from the meta-learning loss, that can be expressed (for unknown task DD+1) as MetaL(Qr) A EEs++~DD__. considers the loss on a future task rather than past or current performance. We note that this frame- work allows for the derivation risk certificates for both offline continual learning, where samples for each task are given as a batch, and online continual learning, where each sample is given sequen- tially and cannot be re-used.
P++1)]. Unlike CuL and AL, the meta-learning loss
2.3 OTHER RELATED WORK
Continual learning In recent years, there have been several prominent papers focused on under- standing forgetting and average error in the context of continual learning. [Evron al. (2022) as well as Lin et al. (2023) and Li et al. (2024) provide upper bounds and equations that describe ex- pected behavior for sets of noisy linear regression tasks. Factors such as the number of parameters vs the number of samples, task relationships and task order are all relevant parts of forgetting, Other works such as Bennani & Sugiyama (2020) and Doan et al. (202T) provide upper bounds on average and cumulative errors for the Stochastic Gradient Descent (SGD) algorithm as well as the Orthog- onal Gradient Descent (OGD) (Farajtabar et al. 2020) algorithm, for models that correspond the Neural Tangent Kernel (NTK) regime. Friedman & Meir (2025) provide general PAC-Bayes upper bounds on average error that are applicable regardless of optimization method, task structure and hypothesis class for bounded loss functions. Our bounds instead focus on the cumulative loss, measure of learning plasticity. Unlike results for the NTK regime, our bounds are algorithm and architecture agnostic.
Meta-learning There numerous similarities between continual and meta-learning, We note that in meta-learning data from previous tasks remains accessible, and tasks are usually assumed to be taken from single task-generating distribution. Pentina & Lampert (013)) well as Amit & Meir (2018) derive PAC-Bayes bounds for the meta-learning loss described above for bounded loss functions. Other relevant works consider the cumulative loss and overall regret in the meta- learning problem from convex optimization perspective; Balcan et al. (2019) show upper bounds for variants of the online SGD algorithm and follow- teeader methods under some assumptions on loss convexity and the parameter space. We note that follow-the seoowwtth--eeddee methods require access to data from previous task, making their application to continual learning problems difficult.
Continual Meta-learning Chen et al. (0033) consider the problem of continual meta-learning, where a meta- parametrr is updated based on past and current task data and task-specific parameters can only access data from the current task. For loss functions bounded in [0, 1] and several base- learner algorithms, they derive upper bound excess risk incurred in both the static and shifting environments. Interestingly, the meta-algorithm used does not require data from past tasks, allowing its application in continual learning, unlike follow-the-leader methods. In contrast to the above methods, our general bounds are applicable for both offline and online continual learning, regardless of optimization algorithm, model architecture and task environment.
4

---

Published as a conference paper at ICLR 2026
3 UPPER BOUNDS ON CUMULATIVE LOSS
Assumption 1. The loss function ((h, 2) is either: (1) upper bounded by constant K or (2) is K sub- Gaussian. Formally, Ah H, Z, E [0, K] or E.,D, [e*(((h,a)) EzwD, Ex,,, S kr²
We note that Assumption ! only applies to the loss used for the upper bound calculation. Some relevant examples include classification loss, cross entropy loss with non-zero minimal class prob- ability, and any light-tailed unbounded loss. First, we extend the result of Theorem A.1 to the continual learning setting (m > 1) and to sub-Gaussian losses.
Corollary 3.1. Under Assumption 1 for any À O, for any set of tasks, for any online predictive sequence of priors (Pt), for any sequence of stochastic kernels (Qi) with probability at least 1 - 8 over the draw of (Si...., .....,,r)r)))) Din x... DT,, we have that the following holds for the data- dependent measures (((S)..)) Qr(ua.))),
CuL((Q))E
1 Σ -
T SQ)
1 XT
Σ (L(Q((S.))II(SS..)
T
AK² log(1/8) + m XT
The full proof is in Appendix A and follows a similar outline to Theorem A. but adapted to the continual learning setting. In particular, unlike the online setting, if we have m >> T. the r.h.s. converges to the l.h.s. as m + oc, similarly to the standard PAC-Bayes bounds. This result is a straightforward extension of Theorem A.I and requires new technical tools.
Since the prior sequence may be detendependnnt, we can select P1 = P and Qo similarly to Haddouch & Guedj] (2022), giving a KL-divergence term that depends the change in the posterior between subsequent tasks. We note that while the posterior stochastic kernel Q--1 must be meesurabee and is therefore independent of S,, the measure Qi-1 (S S, in equation [ In practical terms, this means that for models with task-specific parameters (i.e. parameters that are used only for specific tasks), KL--dieegeence can be measured w.r.t. only shared parameters. This may serve to lower the overall KL-divergence, as task-specific parameters can be used specialize a more general model that varies little between tasks.
use
While Corollary 3.1 is a useful upper bound, equation haa terms in the r.h.s. that do not converge the l.b.s. as the number of tasks T increases, namely the term AK2 /m. In the context of continual learning, upper bounds that converge as the number of tasks increases are vastly preferable as they converge even if the number of samples per task fixed. In order to achieve such bounds, we must make some additional assumptions on our tasks. Specifically, we assume that the loss function is upper bounded by constant, and that the number of samples per task is not much smaller than the number tasks, i.c m >> VT.
Theorem 3.2. Under the same setup as Corollary 3.1 assuming E(h, 2) K) any predictive sequence of posteriors (Qt), for any 82 E (0. 1], with probability at least 1 - 5 over the draw of (S,...., ST) D x ... Dtp, the following holds for measures (,(S (S(Saa)
T Σ ((Q.(S).))))) Σ (L(QU((L:)IPP((S.....
T
1 (LL((Q
T
T
(2)
+ log { (1- (1-82)e / )
1
log(1/6a)/2mmT + dge2k]+
log (1g(1/6)
A
Specifically, we can set X
TVT 02 TVF K=
resulting in
T
K T
- CuI,(()))E
1 <
Σ ((Q.(S).). St)
Σ (I(QQ(())))))--((1iia))) [=1
T
TVT
K KVT
K(1 log(1/0))
22m
TVT
5

---

Published as a conference paper at ICLR 2026
where Qu P, meaning that long >> vT/2 the r.h.s. converges the l.h.s. T - This result gives us an upper bound that converges as more tasks are added, provided that we have sufficient samples per task. It also suggests good rule of thumb for effective continual learning the number of samples per task should exceed the square root of the number of total tasks.
careful analysis of task and posterior dependencies that is specific to online and continual learning, as well as bad-event analysis that is not commonly utilized in the context of PAC-Bayes bounds.
Proof of Theorem 3.2 appears in Appendix A This Theorem
4 ORACLE BOUNDS FOR SPECIFIC SETTINGS
provide general upper bounds that are applicable for a wide variety of tasks and continual learning al alggorthss, but their generality makes detailed analysis more difficult. In order to better demonstrate these results and compare them for different settings, we derive and compare oracle upper bounds on the Gibbs posterior.
Corollary 3.1 and Theorem 3.2
Definition 5. Recall the Gibbs posterior distribution and the expected Gibbs posterior distribution, for 1, x 0, for Q = Q = P. (h) x SA
(2(1) x e - 2(h, 12 (1))
The main advantage of the Gibbs posterior is that it removes the KL-divergence terms from the r.h.s. of equation (as can be seen, for example, in Lemma A.4). This property allows to derive upper bounds on the cumulative loss for the sequence of expected Gibbs posteriors. We note that while these bounds refer to the expected Gibbs posterior and discuss the limit where m, T - co, they originate from non-asymptotic bound appearing in Appendix A Equivalent bounds for empirical Gibbs posteriors can be derived by changing the final condition be taken w.r.t. the empirical loss on the training set.
Theorem 4.1. For any À > assuming (1) Q: = Q² the expected Gibbs posterior. (2) P1 P is a data-free measure over H. (3) > 1: P Q--1 Qi (4) H is compuct, bounded subset of gd, (5) Vt E [, the total expected loss Ei= C(h, D:) has strict global minimum at hist and is twice continuously differentiable w.r.t. h. Under Assumption 7 we have
m, +00 T CUL((QQNE S lim 1
lim
1
Σ ((h
T-+00 T
The full proof is provided in Appendix A and relies mainly on Laplace's method (Hwang 1980 Shun & McCullagh 1995), which is rarely used for PAC-Bayes bounds. We note that under different strict minimum assumptions we can derive another useful oracle bound,
Corollary 4.2. For any X O, assuming (1) Q. Q² is the expected Gibbs posterior (2) Pi P is a data-free measure over H (3) VI > 1: P, = Qi-1 = =Qi (4) H is a compact, bounded subset of TRdd. (5) Vt e [1, the expected loss L(,, D.-1) has strict global minimum at h and is we have
twice continuously differentiable w.r.t. h. Under Assumption !
1 CUL(IINNE T lim m-+00 T
1 -.P.D.)
Σ C(h.,, D.) 1=
(5)
In both cases, that the CuL can be upper bounded by the loss of predictor obtained from previous tasks. Using these oracle bounds, we can compare the cumulative loss for several different continual learning setups and contrast the effects of task similarity between them.
Assumption 2. (Lipchitz loss) For non-negative GH and metric d(-,.) (e.g. Wasserstein distance),
Vi,j, h H. [c(h, D.) L(h, D,)1 S D,)
This assumption implies that similarities in the task distribution are reflected in the loss function. Assumption 3. For each task Dr, the optimal (minimum) loss L; a achievable via a hypothesis of dimension d, Additionally, for any two tasks D;, Dj. if d dim(H) d, + dj. i.e. H. is over- parametrized, then there exists hi, E H C R such that L(h),) = minneH (C(h, D.) f(h, D2))-

---

Published as a conference paper at ICLR 2026
This assumption aims link model complexity and representation power, stating that given enough parameters, it is possible to find a hypothesis achieving minimal loss and that given more parameters, can find optimal hypotheses the average loss. Taking into account all of our assumptions,
S
S
S,
Sp-1
(a) Scenario 2
(b) Scenario 3
(c) Scemario 4
Figure 2: Depictions of scenarios 2, 3 and 4. In scenario 2 we alternate between two task distribu- tions. In scenario 3 take 7/2 tasks from distribution and another T from second one. In scenario 3, each new task is of distance at most from the previous one, and any two tasks are at distance at most 2Φ from each other.
can provide oracle bounds for several different scenarios in terms of task similarities and ordering.
De = D1, i.e. we have different samples from the same task distribution, we have 1. If vt,
lim T (LI
o -
2. If tasks alternate between distributions D, and D2, we have that if d 2 di + d2, we also obtain the optimal (minimal) result
CuL((Q)) + +0
(
lim 12-000
T
T
and otherwise have (via Theorem
lim 1 CuL((QU) < m=-00
min, [Lhh, D1) + ((h,D2))
1
1 T
Gxd((DD, + +0
< min (Ci, (5) + (Hd(D D2) (
3. If the first T/2 tasks are from D1 and the second half are from D2, we have
Ci+ci ( G + + OHdd(D,, D2)
lim m-+ce 7 [CIL((
d2 d1 + d2
< di d2
4. If tasks change gradually and do not differ significantly from one another, or more formally, if Vt, d(Dr, Do) < & and Vt < T, d(D,, De-1)
< Φ. we have
T
lim M0 T (UL((Q)) - T Σ ct rgn +
(
These results imply that (given limited capacity) task order significantly impact the cumulative error. This is not particularly surprising, as existing results on the expected loss and forgetting in continual learning also demonstrate that task order and relatedness are a major factor in overall error, but is not immediate to deduce that this should also be the case for the cumulative loss. These results also demonstrate that sufficiently over- over-parametrizer models can overcome issues such as task order and learn each task effectively.
5 EMPIRICAL DEMONSTRATION
In this section we demonstrate the utility of our bounds from Sections and 4 We study the efficacy of our bounds several simple (but non-trivial) environments with varied task similarity.

---

Published as a conference paper at ICLR 2026
Table 1: Average and final cumulative error percentage for vision tasks. Lower is better.
Domain
Method
CuL
Bound (equation B
Error @/
120
Bound @1 120
EWC
1.0 0.0
10.6 0.2
1.0 + 0.0
5.1 + 0.2 6.8 0.1
VI
15.5 0.1
17.9 0.1
4.7 0.1
Split-MNIST EWC
0.0
4.2 0.1
0.9
2.5 0.1 7.5 0.5
VI
17.6 0.4
19.4
5.1 0.3
Split-IIFARIO
EWC
34.4 0.2
47.9 0.5
3.97
39.7 7 1.4
VI
49.8 0.1
52.2 0.2
49.5 0.1
52.5 + 0.2
Table 2: Average and final cumulative error percentage and forgetting for Split-ImageNet on 3 random seeds. Lower is better, random model achieves 98% cumulative error.
Domain
Method
CuL
Bound (equation
Forgetting
Split-ImageNet
EWC
33.5 1 0.2
40.1 + 0.3
5.7 + 0.4
SGD
34.8 0.3
41.4
7.8 = 0.1
Replay
55.7 0.7
62.5
2.8 1 0.1
5.1 VISION-BASER TASKS
In order to examine the bounds of Corollary 3.1 and Theorem 3.2
we made use of a few well-
known computer vision tasks in the context of continual learning, namely: (1) Permuted-MNIST [Goodfellow et al. 2015), dommain-incremental problem (De Lange et al. (021) where a random
permutation is applied on each image for each task. (2) Split-MNIST (Zenke et al. 2017). a se-
quential set of binary classification tasks constructed from the MNIST (LeCun al. 1998) dataset. (3) Split-CIFARIA (Zenke et al.] (017). a sequential set of binary classification tasks constructed from the CIFAR-10 (Krizhevsky al. 2009) dataset. (4) Split-ImageNet,, a sequential set of binary classification tasks constructed from the ImageNet (Deng et al.
2009) dataset.
We used both variational inference (VT) (Hoffman al., 2013) and Elastic Weight Consolidation (EWC) (Kirkpatrick et al. (2017) in our experiments. For Split-ImageNet, we also examined SGD with and without experience replay. Detail on the prior and posterior distributions used is available in Appendix B We note that there is no significant computational or memory overhead required for bound calculation and no additional samples are required besides the training set.
We used convolutional neural networks (CNNs), with T' = 120 tasks in total for all domains. We approximated via a held-out test set for
measured the average cumulative error C(T) each task, and its upper bound based on equation 3 across 5 random seeds, reporting standard error and the average value. We also report the loss and upper bound for the final task. full detailing of hyper-parameters and the experimental setup is available in Appendix B
Tables 2 detail the error percentage (error out of 100%, similar to accuracy percentage) for both VI and F. WC C models on all tasks, as well as the cumulative error and average test forgetting (see definition of forgetting measure (FM) in Wang et al. (2024a)) as well as Appendix (B) for Split- ImageNet. The values reported are for both the average cumulative error and the error at the stopping point. Figure the average cumulative loss and the upper bound as a function of the number of tasks for the split-MNIST task. We note that the values reported in the first two columns of Table I correspond to the final values of Figure 3 and that the last two columns correspond to only the error and bound for the final task in the continual learning process. Looking over the results, we see that the bound is very tight for VI, and somewhat looser for EWC. This is to be expected, to an extent, as the VI algorithm aims to optimize the r.h.s. of equation 3 w.r.t. the posterior distributions, whereas for EWC the bound is not directly linked to the learning process. Across all settings, we see that the bound becomes increasingly tight as the number of tasks increases, owing to both the fact that several terms in equation decrease in proportion to the number of tasks, and to the tendency of the KL-divergence term to decrease during the continual learning process as we experience additional tasks, the posterior for the previous task Q. that serves as the prior for task D, becomes an increasingly better informed and predictive prior, resulting in a tighter upper bound. With the exception of the VI algorithm for the Aplit-CIFARIII task, all of our empirical upper bounds non-vacuous, with several of them being tight enough to provide a useful risk certificate, especially if we consider only later tasks. The upper bounds for average cumulative error tend to suffer for
8

---

Published as a conference paper at ICLR 2026
early tasks, possibly due to the randomness of the training process for the first few tasks. This is somewhat encouraging within the context continually retraining a complex model to handle new data, as the forward transfer would be weighted towards generalizing new tasks, and our bounds are consistently tighter (empirically) for later tasks in sequence.
Woard
(a) VI
(b) EWC
Figure 3: Average Cul. percentage and upper bound over time for split-MNISS Error bars represent standard error over random seeds. A random predictor would have 50% average loss.
5.2 ORACLE BOUNDS
While the oracle bounds 4 specific scenarios, verifying and comparing the bounds empirically may provide additional insights. To that end, considered equation 4 and equation 5 in several simple scenarios that correspond to the specific thcoretical scenarios discussed in Section We utilized linear regression tasks following similar setup as discussed in Lin et al. (2023) (a linear ground truth (Belkin et al. 2018 Evron
already useful, as be seen in the explicit oracle bounds for
et al., 2022) with the true weight vector for each task being subset of all features) while varying the linear regression, and simple 2 layer fully connected (FC) neural networks (with wide hidden layer to approximately adhere to the NTK regime (Jacot al. 2018 Bennani Sugiyamal 2020)). In order to approximate posterior sampling from the Gibbs posterior, we used the Stochastic Gradient Langevin Dynamic (SGLD) (Neal [001]] I Welling & Tch 201I) algorithm in our experiments. The full detail of task construction and model and training hyper-parameters is available in Appendix
model between over and muder-parametrian
alongside detailed numeric results.
280
105
20
(a) Task swap
(b) Task alternating
Figure 4: Average cumulative loss, oracle bound (equation 4 and average model error over time for under-parametrized linear regression. (a) task changes at f 100 (b) tasks alternate.
Figure 4 compares scenarios 2 and 3 for under- parametrized linear regression, with each task char- acterized by a different true weight vector. We can clearly see that task order matters significantly for both the cumulative loss and the average model loss. Specifically, we see that for cumulative loss, single swap is significantly better than alternating between two tasks. This result agrees with the obtained theoretical bound. As can be seen for average model loss from the expected behavior detailed in Lin et al. (2023) (that agrees with the empirical average model loss for most of the con- tinual learning process), sudden task swap results in sudden significant increase in average error that is slowly corrected, whereas alternating tasks quickly stabilize to a constant average error.
Figure 5 contrasts changing tasks corresponding to scenario 4 of Section 4 Each task corresponds to a generating
and under-parametrizer 2 layer neural networks
sequence of gradually
linear weight vector, and adjacent tasks have similar weight vectors (see Appendix B We can

---

Published as a conference paper at ICLR 2026
0350
Anre OL
940
020
6:4
115
(a)
(b) Over-parametrized
Figure 5: Cul., oracle bound (equation 4 and average model error over time for linear regression with gradually changing tasks. (a) Under-parametrized model (b) over-parametrized model.
that in the over-parannnn aver-parametrized regime, cumulative error remains near constant and the upper bound is nearly exact. In the under-pane parametrized regime, however, we gradual decrease the model stabilizes. In both cases, the average error tends to increase with the number of tasks (though is lower in the rree-parametrized setting), likely due the increasing distance between tasks resulting from the random walk process. We note that our bounds are several orders of magnitude tighter in this setting compared to NTK-based generalization bounds for SGD (Bennani & Sugiyama 2020
LIMITATIONS AND FUTURE WORK
In this work, we derived several upper bounds on the cumulative error for both general hypothesis classes and for the Gibbs posterior. Our bounds offer tight risk certificates when the number of samples per task is large as well as in several concrete scenarios for the oracle bounds. Our results assume that the loss is either bounded or is sub- -abssaussinn though extensions to heavy-tailed losses similarly to to/Hadoucche & Guedi can be relaxed to allow for finite number of global minima. We note that our derived oracle bounds are taken w.r.t. the expected Gibbs posterior. Equivalent bounds can be derived w.r.t. empirical Gibbs posterior by modifyyng our assumptions the global minimum to apply in expectation.
may be possible. The assumption of a strict global minimum
While our results are applicable for both offline and online continual learning, we acknowledge that for offline continual learning the cumulative error is often less relevant than the average error. A common assumption in this setting is an unbounded number of training samples per task, making learning plasticity irrelevant any task can be learned from scratch. We also note that in the online continual learning setting, if task boundaries are blurry or unknown, the number of samples per task bounds.
m and the number of tasks T must be approximated in order
Our PAC-Bayes bounds contain complexity terms (KL-divergence) that may be difficult to scale for large models with many parameters. While this is not an issue in our oracle bounds, it is concern for the general bounds such as equation 5 While there is some work in the context of PAC-Bayes bounds with other divergence measures (Bégin et al., 2016 Amit al. 2022 Kuzborskij et al. 2024). this can be a potential limiting factor in applying our results for large model classes, though codee--ompprssion bounds (Loth et al.) 2022) may serve as an avenue overcome this limit.
We tested and verified our theoretical results on two main algorithms for several simple computer vision benchmarks, yielding non-vacuous bounds on the cumulative test error. Tables 2 the risk certificate is not always tight. We note that in most cases, the VI bound is nearly tight whereas the upper bounds for deterministic methods tend to looser. This is somewhat unsur- prising as the VI training objective attempts to directly optimize the r.h.s. the upper bound.. As our main focus was measuring the efficacy of our upper bounds, we focused mostly on relatively small neural networks and classification problems. While encouraging, used vision datasets, and we only examined our bounds on VI methods, EWC and experience re- play algorithms. A more comprehensive empirical analysis of common continual learning methods, combined with applying some of the recent insights into obtaining tight risk certificates via PAC- Bayes bounds (Pérez-Ortiz al., (02T), may yield further insights into the practical application of PAC-Bayes bounds for cumulative loss in continual learning for larger models.
we can see in
preliminary experiments only
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
Proofs for all Theorems and Corollaries is available in Appendix All required assumptions appear in both the main text and the Appendix. Code for reproducing the experiments available as part of the supplementary material and a more detailed explanation of the experimental setting and hyper- parameters available in Appendix B All data sources are publicly available and code is available on mngithub
ACKNOWLEDGMENTS
This work was partially supported by grant 1693/22 from the Israel Science Foundation, and by the Skillman chair in biomedical sciences (RM)

---

Published as a conference paper at ICLR 2026
REFERENCES
Pierre Alquier. User-friendly introduction to pac-bayes bounds. Foundations and Trends in Ma- chine Learning, 1 17(2):17-300) 2024. ISSN 1935-8237. doi: 10.1561/22000000000
Ron Amit and Ron Meir. Meta-leamiig by adjusting priors based extended pac-bayes theory. In International Conference on Machine Learning, pp. 205-214. PMLR, 2018.
Ron Amit, Baruch Epstein, Shay Moran, and Ron Meir. Integral probability metrics pac-bayes bounds. Advances in Neural Information Processing Systems, 35:3123-3136, 2022.
Maria-Florina Balcan, Mikhail Khodak, and Ameet Talwalkar. Provable guarantees for gradient- based meta-learning. In International Conference on Machine Learning, pp. 424-433, PMLR,
2019.
Lue Bégin, Pascal Germain, François Laviolette, and Jean-Francis Roy. Pac-bayesian bounds based on the rényi divergence. In Artificial Intelligence and Statistics, pp. 435-444. PMLR, 2016.
Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to under- stand kernel learning. In International Conference on Machine Learning, pp. 541-549. PMLR. 2018.
Diana Benavides-Prade and Patricia Riddle. A theory for knowledge transfer in continual learning. In Conference on Lifelong Learning Agents, pp. 647-660. PMLR, 2022.
Mchdi Abbana Bennani and Masashi Sugiyama. Generalisation guarantees for continual learning with orthogonal gradient descent. In 4th Lifelong Machine Learning Workshop at ICML 2020, 2020.
Olivier Catoni. A PAC-Bayesian approach to adaptive classification. preprint 840, 2004.
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European conference on computer vision (ECCV). pp. 532-547, 547, 2018
Qi Chen, Changjian Shui, Ligong Han, and Mario Marchand. On the stability dilemma in continual meta-learning: Theory and algorithm. Advances in Neural Information Processing Systems, 36:27414 27468, 2023.
Ben Chugg, Hongjian Wang, and Aaditya Ramdas. A unified recipe for deriving (time-uniform) pac-hayes bounds. Journal of Machine Learning Research, 24(372) 2023.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. continual learning survey: Defying forgetting classification tasks. IEEE transactions on pattern analysis and machine intelligence, (4(7):3366-3385, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 Ieee, 2009.
Thang Doan, Mehdi Abbana Bennani, Bogdan Mazoure, Guillaume Rabusseau, and Pierre Alquier. A theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In International Conference Artificial Intelligence and Statistics, pp. 1072-1080. PMLR, 2021.
Shibhansh Dohare, Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A Rupam Mah- mood, and Richard Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026): 768-774. 2024.
MD Donsker and SRS Varadhan. Large deviations for markov processes and the asymptotic evalu- ation of certain markov process expectations for large times. In Probabilistic Methods in Differ- ential Equations, pp. 28. Springer, 1975.
Itay Evron, Edward Moroshko, Rachel Ward, Nathan Srebro, and Daniel Soudry. How catastrophic can catastrophic forgetting be in linear regression? In Conference on Learning Theory. pp. 4028- 4079. PMLR,, 2022.
12

---

Published as a conference paper at ICLR 2026
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for contin- ual learning. In International Conference on Artificial Intelligence and Statistics, pp. 3762-3773.
PMLR, 2020.
Lior Friedman and Ron Meir. Data-dependent and oracle bounds forgetting in continual learning. In Proceedings of The 4nd Conference on Lifelong Learning Agents, 2025.
lan J Goodfellow, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based based neural networks. stat, 1050:4, 2015.
Maxime Haddouche and Benjamin Guedj. Online pac- pac-bayes learning. Advances in Neural Informa- tion Processing Systems, 01125255-2773 2022.
Maxime Haddouche and Benjamin Guedj. Pac-bayes generalisation bounds for heavy-tailed losses through supermartingales. Transactions on Machine Learning Research, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational infer- ence. the Journal of machine Learning research, 14(1):10031337, 2013.
Chii-Ruey Hwang. Laplace's method revisited: weak convergence of probability measures. The Annals of Probability, pp. 1177-1182, 1980.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen-
cralization in neural networks. Advances in neural information processing systems, 31, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. (
// aarxiivoorg///aaa.vvaaaa.......a ats:/bba///44 6980
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Bareskkkaaaaiikaaa,.a et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):351133526 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of 2009.
features from tiny images.
Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity in continual learning via regenerative regularization. In Conference on Lifelong Learning Agents, pp. 410- 430. PMLR, 2025.
Iia Kuzborskij, Twang-Sung Jun, Yulian Wu, Kyoungseok Jang, and Francesco Orabona. Better- than-kl pac-bayes bounds. In The Thirty Seventh Annual Conference on Learning Theory, pp- 3325-3352. PMLR, 2024.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):278-23224 1998.
Ran Levinstein, Amit Attia, Matan Schliserman, Uri Sherman, Tomer Koren, Daniel Soudry, and Itay Evron. Optimal rates in continual linear regression via increasing regularization. arXiv preprint arXiv:2506.0651 06501, 2025.
Hongbo Li, Sen Lin, Lingjie Duan. Yingbin Liang, and Ness B Shroff. Theory on mixture-of-eeppers in continual learning. arXiv preprint arXiv:2406. 16437, 2024.
Sen Lin, Peizhong Ju, Yingbin Liang, and Ness B. Shroff. Theory forgetting and generalization of continual learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 21078-2 2188-11111 100. PMLR, 2023.
13

---

Published as a conference paper at ICLR 2026
Sanae Lotfi, Mare Finzi, Sanyam Kapoor, Andres Potapezynski, Micah Goldblum, and Andrew G Wilson. Pac- bac-b compression bounds so tight that they can explain generalization. Advances in Neural Information Processing Systems, 11:31149-314777 2022.
David A McAllester. Pac-bayesian model averaging. In Proceedings of the twelfth annual confer- ence on Computational learning theory, pp. 164-170, 1999.
Radford Neal. Mcme using hamiltonian dynamics. Handbook of Markov Chain Monte Carlo, Pp- 113-162, 2011.
Anastasia Pentina and Christoph H Lampert. Lifelong learning with non-iid tasks. Advances in Neural Information Processing Systems, 28, 2015.
María Pérez-Ortiz, Omar Rivasplata, John ShylerTaylor, and Csaba Szepesvári. Tighter risk certifi- cates for neural networks. Journal of Machine Learning Research, 22(227):1-40, 2021.
Sylvestre-ANve Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, pp. 2001-2010, 2017.
Omar Rivasplata, Ilja Kuzborskij. Csaba Szepesvári, and John Shawe-Taylon Pac-bayes analysis beyond the usual bounds. Advances Neural Information Processing Systems, 33:16833-168444 2020.
Zhenming Shun and Peter McCullagh. Laplace approximation of high dimensional integrals. Jour- nal of the Royal Statistical Society Series B: Statistical Methodology, 57(4):749-760, 1995.
Liyun Wang. Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2024a.
Maorong Wang. Nicolas Michel, Ling Xiao, and Toshihiko Yamasaki. Improving plasticity in online continual learning via collaborative learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23460 23460-23469, 2024b.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 68 681-688. Citeseer, 2011.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference machine learning, pp. 3987-3995, PMLR, 2017.
Xuyang Zhao, Huiyuan Wang, Weiran Huang, and Wei Lin. A statistical theory of regularization- based continual learning. In Forty-first International Conference on Machine Learning, 2024.
14

---

Published as a conference paper at ICLR 2026
A PROOFS
Theorem A.I. (Haddouche Guedjj (2022). Theorem 2.3) Assuming vh E H, E 2, 2,8(h,
[0, K for online learning (m = 1), for any distribution p over tasks (T), any X O and any online predictive sequence (Pt), for any sequence of stochastic kernels (Qt). we have with probability at the following, holding for the data-dependent least ó over the sample S (Z1,...;
measures Qi,s e,,
P((S..).
1 [KI(O,,s((P.s)))
E Beeey (ehh, 2 1-111 -
Σ
Bni-Qu, le 2))
1=1
XTK²
log(1/8)
Corollary A.2. Restatement of Corollary 3.1 Under Assumption for any À > O, for any set of tasks, for any online predictive sequence of priors (Pi), any sequence of stochastic kernels (Qt) with probability at least 1 over the draw of (Si...., ST) 'D x x DD, we have that the following holds for the data data-dependent measures P,
1
1
S1)
(L(Q((S::)II((5...)))
T
T
XT
1=1
AK
log(1/8)
+
XI
Proof. We begin by applying the main Theorem of Rivasplata et (2020) with Qo P ② ... @ Qr, and with
Pr, Q = Q
X
F(S S @ ST.K h @ ... 2hT) (.....T)))
Σ Ex, D. [K 21 Fr Σ E(he St)
T
Reorganizing terms, we have with probability at least
(over the draw of S S1 @ ... 8 Sr),
(e [e(He, Fx
(S.),
T
(6)
+ KL(QQ(S::.)IP((SS)))))
1 + x log Er
log(1/6)
X
where
Er EsE
c F(S,))
We then apply similar Lemma D.2 of Haddouche & Guedj (2022) (with tasks instead of single examples) that yields
ET
II E.. E.s..... E P
and applying Hoeffding's Lemma for bounded losses or the exponential moment bound for sub- Gaussian random variables we have
2kk Er S m/
Combined with equation
6 we have as follows:
T
T ((Q,()))),..)
E
(s..)) [e(he, Fu-
<
T
T
1
(I(Q((())))P(SL))
X
t=1
2
log(1/8)
m
X
15

---

Published as a conference paper at ICLR 2026
Picking X XI completes the proof.
Theorem A.3. Restatement of Theorem 3.2 Under the same setup as Corollary 3.1 assuming
f(h, 2) E [0, K. for any predictive sequence of posteriors (Qi). any d2 (0, 11, with probability
at least over the draw of (Si...., ST) ~ D x... X DT, the following holds for the data-
dependent measures P;(St:);; Qc(5):t)
1 T
1 Σ (L
T Σ ((Q,(S
1 Σ (L(Q(()))))P((S5.))
log { {1 (1 / +82e
+ x
log(11 1
Proof. Starting from equation 6 with
(11 [((h,
1
F(S S.8. 25.....Srr.. = 11 =
21)
E 1=1
(he, St)
T
for convenience, have
Qunn BZNDD [e(he, F-1] S ((..()
T
(L(Q((S
log Er + log(1/6)
A
Since each element of mI Et EØ= (15) is bounded in range [0, K/m7), we can apply Hoeffding's Lemma on each task; we first apply Markov's inequality: for 8, c 0
Pr(F(S= = .... ..................... ST, h= h1 @ @ 2 Pr(c 2 e²t) E ( Et= B (f(1))22)) Ft Σt= E(he,
Since we assume that the expected loss is F: this equals
II Es Se......,. Eny caM( (e nhb EP ((ha.ra)))
Since we assume data from each task is drawn iid we have
S
E
Es.. B, air ((h...))))
T TT
(AK/mIT)// 18
<
aA2g2 2 5m37
16

---

Published as a conference paper at ICLR 2026
Where the last inequality is due to applying Hoeffding's Lemma each element in the product. Minimizing 8, we have
Pm 2 c) S 22N2 2 82.
Pr(F(S= S1 @... @ ST, h = h1 @ @ hr)
Moving terms around, we get
10g(1/62) 2m7
We can then split Er = EsEn~qo [e [SS.)]] into two events with appropriate probabilities, with one event (the good event, with probability 1 12) fulfilling this inequality and the other (the bad event, with probability at most 62) violating it giving us upper limit F(S, h) S XK, giving
TT (1 -2)e /og(L//a)/2mm + Sge
Plugging in this inequality in equation 6 completes the proof.
Lemma A.4. Under the same setting as Theorem 3.1 assuming that
1. Qt = Q is the empirical Gibbs measure
2. P1 P is data-free measure
3. > 1: P. = 1=-1= =p-11
we have
T
T Ss...... Sr ((Qx-(())))),, + AK2
Es
ST T
(IO. & Es.....
t=1
t=1
Proof. Starting from Theorem 3.1 (in expectation), we begin by decomposing the KL-divergence under assumptions:
XT BB-@d log e --1
(8)
X7
2 6 AC(h.S.)
1 (())
1
log E 62 -1((,.)))
XT
Applying this equality to Theorem 3.1 we have
-AC(h,S)) AK²
Es, s, (s.)
Bs, s, log Bng
XT
m
t=1
Applying Jensen's inequality
log 1 e xk(h. XI
-(Q0
< -
for all E [1,T] completes the proof.
Theorem A.5. Restatement of Theorem
For any
0, assuming
1. Q: Q. is the expected Gibbs measure
2. Pj P a data-free measure over
3. 1I > P. = Q=-1= Q²-
4. H. is compact, bounded subset of dRd
17

---

Published as a conference paper at ICLR 2026
5. Vt E [], the total expected loss Ei=1 [(h, D;) has strict global minimum at hig and is twice continuously differentiable w.r.t. h.
we have
T
T 1( Dz)
1
lim -00
Ei D.) T
lim T-000
Proof. We begin with the change-of- mfeasure inequality; for any > and any measurable function f: H DMM + R. for any prior and posterior P. Q, for any sample S ~ DMM a.s.
log EA-pe A((,S) f(h, + X
KL(Q(S))1P)
Taking expectation S and using Jensen's 's inequality, we have
S E (4,,)) - X log ESND" BA-Pe A((h,) as well as
1 A RR..DMKI(((S1))
Choosing Q Qi 2 .... 8 Q p= P Q 8 .... @ Q
T
T
0hr)= = = S(S = S0...0..0ssh @ )
Σ E(e,, St)
yields
1 C.)... S Σ ES
X log EsB, hysBammen by P. P4
ex X (00,,, 900.)
T
KL(QNIQQ)))
(10)
Using Hoeffding's Lemma for bounded losses or the exponential moment bound for sub- Gabssinnn random variables, we get
XR² 1 mf
Σ 1.I
D.)[F-1)s T E Es.~D (.)
+ KL(Q
N
Picking X = XI' and decomposing the KL-divergence as before, we get
L(Q), D.)
( D.)
(11)
X
Σ
log En-rr e AL(h.D.)
m
XT
by unrolling the last term according to the definition of the Gibbs posterior, i.c.
E En -CChh, DC
E
Ennor e XLth,.D-_.)
Ea-pe XE ((h,P,)
Enape EE -C(..))
18

---

Published as a conference paper at ICLR 2026
get
L(Q2,
D.) -
RS (2(Q), [(Q)) D.)
AK
log EA -AC0,.-) AC(h.11)
(12)
m
XT
En - Ejjl L(h,
log
XT
BR Ai C(),
Suppose that 76 is compact, bounded subset of Rd. Assuming that L(h, Dj) is twice continuously differentiable w.r.t. h for all j. we can apply Laplace's 's method (Shun & McCullagh 1995) on both numerator and denominator. Let
M1a-1(h)
- i-
Σ ==1
Ch,
hi--1 - arg min M 1: (h) NEW
arg min KEH
c(,, D,)
then marking Mf,-1 det M 1 (1)+-1) the determinant of the Hessian matrix, have the Taylor expansion
T
E Ej=i ((h,Ds)
log
XT
En
d/2
2x
R
A(1-1)
log
XT
t=2
2t A(e-1)
d/2
111-1
2,)0
where R = 1+ 2X(1-1)MM,,
+ O((A(t 11) 1))-
Due to setting the same function to seek an optimum for in both the numerator and denominator (with differing reminder). Since most of the elements of both numerator and denominator are the same, we have (in the limit where + 0c)
EpPpe AΣ ((b,D,) log
1
log e AC(Ma-
Σ t=
C(ija-1.D)
XT
ENOPE
E D(N,
XT
T
combined with equation 12 we have (in the limit)
T
T
Σ IC(2 D.)
ES.-Dr
18(07 - ( D.)(FF-1l)
t=1
t=1
(13)
A
log B -AC(t,D)) AC(h,D)) +
1
Σ
T
f(hia-1: D.)
m
X7
7
1=2
Taking À / /m/T, we have
T
lim
Σ (CQ2) D.)
lim
1
C( Dz) + lim C(P.D))
m.T +oc T
T-000
T
T-000 7
Corollary A.6. Restatement of Corollary 4.2 For any X 0, assuming
1. Qi Q¹ is the expected Gibbs measure
19

---

Published as a conference paper at ICLR 2026
2. Pi =P is data-free measure over H
3. Vt 1: P: Q
4. H is a compact, bounded subset of TR...
5. VI E 7), the expected loss L(,, Di-1) has a strict global minimum at hi-1 and is twice continuously differentiable w.r.t. h.
we have
1 lim T P.I1--1
1 ((P.D.) + E (1)
Proof. Via unrolling the last term in equation II once we get
T
[(Q) 1.)
ES (((Q) -())) D:)
AK2
log Enpre En-Pe-AA(N.Di) AC(h,D1)
(14)
m
XT
E E e-ACA,,- BAN
log
X7
-XC(h,D.-1)-A((....))
Using similar arguments of Laplace's 's approximation, we get for X vm.
1 T lim = +00 T 1=1
D.)
1 _ Σ c(hi1, De) + T ((P.D.)
T 12
Corollary A.7. Under the same conditions as Theorem 4.1 if we also have that Vi, h E H, (e D,) [(h, D,)l s GHd(D,, D2)
for some non-negative GH and metric d(-,-), then
lim m.T 100
Σ F.)
GH
1 1-1 d(D;, D.) o
lim m,T ++00
Σ
min NEW
((,,D))
Σ ==1
where the 0 (+) term is DI),
Proof. Starting from equation
we decompose
C(ii--1, Di) = t-
Chi Dí) + t-
C(hi.- D,) -- Σ Chi Dj)
1
1
1
C(hin-1: D,) +
(C(hi.-),Di) -(hi D2))
1-1
Chi D,) +
((hi:-2,D)) C(bi
By the definition of his-11-
Σ (,)
- Σ
C(ii,-11, Di) (1))
NEW
20

---

Published as a conference paper at ICLR 2026
Iu
1-1 Dj)
min EET j=1
Σ
(h,
Dj)
+
Σ d(Dr, j=1
(15)
t
t-
Using equation 15 to upper bound each loss in equation 13 and taking the limit completes the proof.
21

---

Published as a conference paper at ICLR 2026
B APPENDIX EMPIRICAL SETTING AND HYPER-PARAMETIE
All experiments were run on local hardware with an NVIDIA GeForce 4090 GPU and an Intel i9 CPU, All results were run for random seeds and averages and standard error were reported in all tables. Anonymized code is available in the supplementary material and full code on github
B.1 PSEUDO-CODE FOR BOUND CALCULATION
Algorithm Continual learning training and bound calculation for VI
function CONTINUAL-LEARN(SS
Sr, P)
Choose algorithmic parameters A
Let Q1:0 & P(h)
each task t from to T do Update Q via
Qu arg min L(Qu,, + (aa(Q1:llQua--) Q1.
Update upper bound for (test) CuL. based on equation oo return Qu., upper bound
Algorithm 2 Continual learning training and bound calculation for deterministic methods
function DENTINUULLLEARN(S...
Sr, P)
Choose algorithmic syper-parameters 0, bound parameters Ai,...y Ar, initialize model weights w0
Let Q1.0 & (Noo, ra) for each task t from to T do
Update model weights via the algorithm Aa(wo....; W.
Let Q = (w,, o Update upper bound for (test) CuL based on equation or
return wr, upper bound
B.2 POSTERIOR CONSTRUCTION
For the variational inference (VT) (Hoffman et al. 2013) algorithm, both the prior and posterior distributions were multivariate Gaussian distributions model parameters.
For deterministic methods such as EWC, a posterior distribution constructed after training by adding Multivariate Gaussian noise, i.c. Qut = NWw, La), where we E Rd is the weight vector given output after task D,.
B.3 VISION DATASETS
For all vision tasks except Split-ImageNet, we used convolutional neural network consisting of convolution blocks each consisting of 64 two-dimensional convolutions, max max-poooligg and tanh acti- vations. The convolution blocks are then followed by fully connected layer and an additional tanh activation. Reported results also use a linear classification head for each task, but results without separate classification heads task were not significantly different. In all cases, training for task consisted single pass over all training examples, and expected error for a task is estimated via held-out test set.
For both MNIST-based tasks, we used two convolution blocks of 5 x convolutions and the linear layer contained 400 neurons. For the CIFAR10 10-besed task, we used three convolution blocks of 3 x convolutions and the linear layer contained 800 neurons. For Split- ImageNe,,,,Net, we used a pre-traing pre-trained ResNet- 18 (He et al. 2016) model.
For permuted-Min MNIST, we used a different pixel permutation per task, and each task involved 10- way classification. All 60000 training samples were used for training with batch size of 128. The
22

---

Published as a conference paper at ICLR 2026
learning rate static at 1e and the À parameter involved half of the labels (at random) chosen as positive and half as negative. This is a minor departure from the standard split-MNIST problem where 5 different binary classification tasks created and their loss is averaged, but the overall behavior similar. All other hyper-parameters permuted-MNIST For split-CIFAR10, task construction
1c For split-MNIST, each task
were set identically split-MNIST but on the CIFARIO dataset. All 50000 training samples were used for training with batch size of 256. The learning rate static 1e-3 and the parameter was For split-ImageNet, each task contained a disjoint subset of 50 classes, and the À parameter was set to le- The notion of forgetting measured and reported for ImageNe is the average test forgetting, defined as
the same as
5e 4
Σ
FM, DI.... D--1)
max {a(h; Dj) a(h,, D2)}. iE(LA-1]
where a(h, D.) is the accuracy on task l. This measure be written in terms of the loss function as
[h Di,..., D.-1)
max {llhe, Dj) E(h,, D,)}. El[1.-111
For VI, we used Markov Chain Monte Carlo (MCMC) estimation with 3 draws from the posterior, and the prior was a multivariate Gaussian with noise of 5e and posterior noise of 1e- For EWC, the o² parameter used for posterior construction was set as 1e and the regularization weight was set at žgwe 100 for all datasets except Split-ImageNet and λgwe = 40 for Split- ImageNet. For experience replay, a replay buffer of size 1000 was used. Training was done using the Adam optimizer (Kingma & Ba 2015) except for 1e-4 was used. Hyper-paramen were chosen via manual trial and error using a held-out validation set.
where SGD with weight decay of
bound
20-
60-
60
22
10
120
120
120
Tonk rumber
(a) VI
EWC
Figure 6: Average cumulative loss percentage and upper bound over time for PerMuted-MNIST. Error bars represent standard error over random seeds. A random predictor would have 50% average cumulative loss.
B.4 ORACLE BOUND EXPERIMENTS
Tasks were constructed as linear regression tasks of the form Y, X " + ft, where each element in X, follows standard Gaussian distribution N (0, 1), and e; (0, o²1a) with 0.3. Test data is drawn without noise Y, Xfuwf_ All elements in w follows standard Gaussian distribution N 1). Like in Lin et al. 2023). for the linear case the true weight vector for each task wy is partially zeroed out (we zero out 80% of the weights) to construct zero-filled features for different tasks. For non-linear experiments use all input dimensions.
For the task swap setting, we use the same weight wi until = 100, then swap to tasks with weight wi until T 200. For the alternating task setting, we altemate between wi and wy. For the
23

---

Published as a conference paper at ICLR 2026
Figure 7: Average cumulative loss percentage and upper bound over time for split-CIFAR10 and the EWC algorithm. Error bars represent standard error over 5 random seeds. A random predictor would have 50% average cumulative loss.
gradual change setting, we draw an initial normalized wi (lwi | = 1), and for any new task we update w1+1 (w, + <)/llug =l12, where ~ (0, 2
We SGLD (Apache 2.0) for 20 iterations each task, with initial temperature of 3e- and halving temperature after each epoch. The learning rate was constant at le- We generated 2048 training samples per task and 400 test samples. The training batch size for SGLD was fixed at 128. We used a total of 200 tasks for all settings. Loss is measured via the mean square error (MSE).
bou
2.0
Sag
2.0
3.3
3.0
6.5
03-
50
75
100
125
-50
175
300
0
25
100
125
150
3/5
200
725k number
(a) Swap
(b) Alternating
Figure 8: Average cumulative loss, oracle bound equation 4 and average model error over time for linear regression with Under-parametriied non-linear models. (a) Task swap (b) Alternating tasks. Cumulative loss and the oracle bound converge to similar values.
For linear experiments, we used W E R600 and 3000 For non-linear (deep) experiments, we used i: 10 using fully connected neural network with one hidden layer with ReLU activations. For the for the over-parametrized experiments, the hidden layer was of dimension 4000.
experiments, the hidden layer of dimension 100, and
C LLM USAGE
LLMs were used during the editing process paper for punctuation and checking for grammat- ical errors.
24

---

Published as a conference paper at ICLR 2026
Table Average and final cumulative error (MSE) percentage for oracle datasets. Lower is better.
Domain
Method
Cumulative error
Bound (cquation 4
Average error
Swap
Linear over parammerrrde
10.3 = 0.2
19.0 = 0.3
572.7 + 10.8
Swap
Deep under- pardmeraraet
0.11 0.01
0.15 = 0.01
2.72 - 0.47
Swap
Deep over-paramenn wver-parametrized
0.01 0.00
0.01 0.00
0.12 0.00
Alternating
Linear over- per-ppaaaetrizeed
280.1 = 5.4
520.8 = 10.3
398.3 8.1
Alternating
Deep under-pare padee-parametrizedd
0.11 0.01
0.16 0.01
2.02 0.21
Alternating
Deep over-parame eer-parametrized
0,01 1.00
0.01 + 0.00
0.12
Gradual
Linear wer-paramettriee
0.2
8.2 0.4
7.8
Gradual
Deep under-parametrizer
0.10 0.00
0.12 1 0.01
0.19
Gradual
Deep wer-parametrized
0.01 1: 0.00
0.01 1: 0.00
0.00
25


Reviews:
[{"review_id": "DYM0k71yNo", "reviewer": "Reviewer_a1bF", "summary": "The paper extends PAC-Bayes tools from online/time-uniform settings to derive upper bounds on the cumulative (forward) loss in continual learning. The authors give conditions under which the bounds converge as the number of tasks grows (notably when the per-task sample size scales sufficiently with the number of tasks), and they analyze several controlled scenarios (task repeats, alternation, gradual change) to interpret the oracle bounds. There is some empirical validations of the claims.", "questions": "1. In Corollary 3.1, the constant (K) (and the assumption of bounded loss) should be re-introduced. \n2. Why is the KL divergence measured only with respect to task-shared parameters (line 182)? Would it not be possible to include task-specific parameters by defining them as zero for other tasks?\n3. The empirical results on CIFAR10 show unusually high errors. Could the authors clarify the training procedure and whether models were trained to convergence?\n4. How is optimization error accounted for in the experiments? If models are not trained to convergence, how does this affect the interpretation of the bounds?\n5. Could the authors explain why variational inference yields tighter bounds than EWC or SGD, beyond the observation that it optimizes a PAC-Bayes objective?\n6. The oracle bounds assume strict global minima and compact hypothesis space. Are these assumptions essential, or can the results be extended to more realistic cases?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- The paper provides an original contribution by formulating PAC-Bayes bounds on the cumulative error in continual learning, addressing a problem that has received limited theoretical treatment so far.\n- The specialization of the results to different continual learning scenarios (repeated tasks, alternating tasks, gradual changes) is a useful aspect that makes the theoretical framework more interpretable.\n- The proof techniques might be of methodological interest.", "weaknesses": "* **Lack of narrative and intuition.**\n  The paper is presented as a sequence of results with limited discussion. The authors should provide more intuition after each main theorem, explain the meaning of the key terms, and discuss when the bounds are informative.\n\n* **Insufficient comparison with prior work.**\n  The relationship with existing results (e.g., Friedman & Meir; Haddouche & Guedj) is not clearly established. A more explicit contrast between this setting and previous PAC-Bayes formulations for continual or online learning is needed.\n\n* **Assumptions and applicability.**\n  Some assumptions (bounded or sub-Gaussian loss, compact hypothesis space, strict minima) are strong and not discussed in detail. The paper should clarify how these assumptions affect the applicability of the results to neural networks.\n\n* **Presentation and readability.**\n  The paper assumes substantial familiarity with PAC-Bayes theory. Proof ideas and key steps should be summarized in the main text. Figures and tables could be improved for clarity.\n\n* **Experimental section.**\n  The experiments are not clearly described. Some results (e.g., high CIFAR10 error) are unexplained. It is unclear whether models were trained to convergence, and how optimization error is handled in the evaluation of bounds. Also, the visualizations are not well annotated.\n\n* **Discussion of overparameterization.**\n  Although the text mentions that overparameterization affects the bounds, this is not explored in detail. A more systematic analysis or clearer interpretation would improve the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jC1wfA1pQ6", "reviewer": "Reviewer_cjXg", "summary": "The paper provides a PAC-Bayes bound on the cumulative loss (loss summed over each retraining of the model) for continual learning, which is an evaluation metric that measures the efficacy of learning algorithms over an entire predictive sequence. The author then demonstrates the efficacy of the proposed bounds on standard continual learning datasets.", "questions": "1. In Figure 1, the author used the notation $Q_{1:t}$, but it doesn't seem like this notation is used anywhere else in the analysis. Is this notation the same as $Qt$?\n2. Line 187-188, Can the author address why, in a continual learning context, the upper bounds that converge as the number of tasks increases are vastly preferable? This is an important motivation to motivate the rest of the analysis in the paper. \n3. Line 189-190, Is the assumption that the loss function is upper-bounded by a constant realistic? Most of the loss functions, such as MSE and cross-entropy, are not upper-bounded. The author should provide a few applicable loss functions here as an example. \n4.  In the case of equation 3, can the author explain which part of the bound applying a proper continual learning method reduces? Intuitively, does a buffer-based algorithm help reduce the first term, while a regularization-based algorithm reduces the KL term of the bounds?\n5. In the experiments, error percentage is used to evaluate the efficacy of different methods as a loss function, but it is generally not used to optimize a learning algorithm. Will this violate the assumptions made to prove the bounds?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is clearly written as an extension of previous papers, namely (Haddouche & Guedj, 2022). The writing and proofs are clear and seem technically correct. The formulation of the problem is original as the consideration of CuL is indeed new.", "weaknesses": "Minor:\n1. Although the presentation is clear, the writing can be more inviting to readers. E.g, some brief introduction on the setup of PAC-Bayes bounds in general can greatly help the readers with the required background. \n2. The related work section can include some newer papers, e.g [1,2], that propose both empirical methods, but also with the appropriate theoretical analysis on forgetting/knowledge transfer bounds. \n\nMajor:\n1. Some of the assumptions might be unrealistic.\n2. The bounds seem loose in scenarios where a proper continual learning technique was applied in the setting. \n\n[1]: Wu, Yichen, Long-Kai Huang, Renzhen Wang, Deyu Meng, and Ying Wei. \"Meta continual learning revisited: Implicitly enhancing online hessian approximation via variance reduction.\" In The Twelfth international conference on learning representations, vol. 2. 2024.             \n[2]: Yang, Haoming, Ali Hasan, and Vahid Tarokh. \"Parabolic Continual Learning.\" In International Conference on Artificial Intelligence and Statistics, pp. 2620-2628. PMLR, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pFZnfzShPf", "reviewer": "Reviewer_9vg7", "summary": "The paper extends existing PAC-Bayes bounds used for online and offline learning to address the setting of continual learning, where tasks arrive sequentially. The authors focus on deriving general upper bounds on cumulative generalization loss applicable to any task distribution and learning algorithm. The paper also addresses oracle bounds for Gibbs posteriors and compares these approaches across various task distributions, as well as testing their empirical validity on vision-based continual learning problems. This work represents the effort to establish upper bounds on learning plasticity within the domain of continual learning, given its unique challenge of balancing knowledge retention and forward transfer.", "questions": "As the authors have acknowledged several limitations in the main paper, and considering the questions raised above, I would like to know how the authors plan to address them.\n\nI would expect the authors to adequately resolve most of my concerns; otherwise, I would be reluctant to recommend acceptance, as the limitations appear to be quite significant.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. **Rigorous Theory**: The extension of PAC-Bayes bounds to continual learning, especially in providing non-vacuous bounds, is a theoretical contribution.\n    \n2. **Empirical Validation for the Theory**: The authors provide empirical evidence that supports their theoretical claims across various task distributions, including vision-based datasets. Experiments on linear regression tasks demonstrate the effectiveness of their bounds.\n\n3. **Good Presentation:** The paper is well written and easy to follow.", "weaknesses": "1. **Use of Cumulative Loss:** The Cumulative Loss is a typical measure in online/continual meta-learning. For continual learning, the bound should be analyzed with the average loss in Definition 4. In contrast, the whole paper focused on cumulative loss. \n\n2. **Novelty and Related Work Discussion:** The paper claims to be the ﬁrst general upper bound on learning plasticity for continual learning, while it missed a lot of theoretical work in continual learning [1, 2, 3], and its most related work is in online/continual meta-learning [4,5,6,7]. In particular, the discussion regarding [4, 5, 6, 7] should be in depth, where these methods also brought novel algorithms to reduce the cumulative loss as well as improve its upper bounds.\n\n3. **Theoretical Limitation:** The proposed bounds depend on $\\mathcal{G}_{\\mathcal{H}} d(\\mathcal{D}_1, \\mathcal{D}_2)$, which are hard to evaluate and the bound could be very loose. In addition, the results require that the number of samples per task should exceed the square root of the total number of tasks, which neglects the positive transfer among tasks.\n\n4. **Limited Experiments:** The experiments are somehow too simple.\n\n[1] Itay Evron, Edward Moroshko, Rachel Ward, Nathan Srebro, and Daniel Soudry. How catastrophic can catastrophic forgetting be in linear regression? In Conference on Learning Theory, pp. 4028–4079. PMLR, 2022.\n\n[2] Hongbo Li, Sen Lin, Lingjie Duan, Yingbin Liang, and Ness Shroff. Theory on mixture-of-experts in continual learning. In The Thirteenth International Conference on Learning Representations, 2025. \n\n[3] Sen Lin, Peizhong Ju, Yingbin Liang, and Ness Shroff. Theory on forgetting and generalization of continual learning. In International Conference on Machine Learning, pp. 21078–21100. PMLR, 2023.\n\n[4] Giulia Denevi, Carlo Ciliberto, Riccardo Grazzi, and Massimiliano Pontil. Learning-to-learn stochastic gradient descent with biased regularization. In International Conference on Machine Learning, pages 1566–1575. PMLR, 2019.\n\n[5] Qi Chen, Changjian Shui, Ligong Han, and Mario Marchand. On the stability-plasticity dilemma in continual meta-learning: Theory and algorithm. Advances in Neural Information Processing Systems, 36:27414–27468, 2023.\n\n[6] Maria-Florina Balcan, Mikhail Khodak, and Ameet Talwalkar. Provable guarantees for gradientbased meta-learning. In International Conference on Machine Learning, pages 424–433. PMLR, 2019.\n\n[7] Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar. Adaptive gradient-based metalearning methods. arXiv preprint arXiv:1906.02717, 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)