Temporal-Difference Variational Continual Learning
Authors: Luckeciano Carvalho Melo
Alessandro Abate
Yarin Gal
Venue: NeurIPS 2025 poster ()
Keywords: continual learning
online variational inference
temporal-difference learning
Primary area: probabilistic_methods

Abstract: Machine Learning models in real-world applications must continuously learn new tasks to adapt to shifts in the data-generating distribution. Yet, for Continual Learning (CL), models often struggle to balance learning new tasks (plasticity) with retaining previous knowledge (memory stability). Consequently, they are susceptible to Catastrophic Forgetting, which degrades performance and undermines the reliability of deployed systems. In the Bayesian CL literature, variational methods tackle this challenge by employing a learning objective that recursively updates the posterior distribution while constraining it to stay close to its previous estimate. Nonetheless, we argue that these methods may be ineffective due to compounding approximation errors over successive recursions. To mitigate this, we propose new learning objectives that integrate the regularization effects of multiple previous posterior estimations, preventing individual errors from dominating future posterior updates and compounding over time. We reveal insightful connections between these objectives and Temporal-Difference methods, a popular learning mechanism in Reinforcement Learning and Neuroscience. Experiments on challenging CL benchmarks show that our approach effectively mitigates Catastrophic Forgetting, outperforming strong Variational CL methods.

Temporal-Difference Variational Continual Learning
Luckeciano C. Melo" 1,2
Alessandro Abate
Yarin Gal
OATML, University of Oxford
OXCAV, University of Oxford
Abstract
Machine Learning models in real-world applications must continuously learn new tasks to adapt to shifts in the data-generating distribution. Yet, for Continual Learn- ing (CL), models often struggle balance learning tasks (plasticity) with retaining previous knowledge (memory stability). Consequently, they are suscepti- ble to Catastrophic Forgetting, which degrades performance and undermines the reliability of deployed systems. In the Bayesian CL literature, variational methods tackle this challenge by employing learning objective that recursively updates the posterior distribution while constraining it to stay close to its previous estimate. Nonetheless, we argue that these methods may be ineffective due to compounding approximation errors over successive recursions. To mitigate this, we propose new learning objectives that integrate the regularization effects multiple previous posterior estimations, preventing individual errors from dominating future posterior updates and compounding over time. We reveal insightful connections between these objectives and Temporal Diifference methods, a popular learning mechanism in Reinforcement Learning and Neuroscience. Experiments on challenging CL benchmarks show that our approach effectively mitigates Catastrophic Forgetting. outperforming strong Variational CL methods.
Cemporal-Differenee
1 Introduction
A fundamental aspect of robust Machine Learn- ing (ML) models is to learn from non-stationary sequential data. In this scenario, erties are necessary: first, models must learn from new incoming data - potentially from different task - with satisfactory asymptotic per- formance and sample complexity. This capabil- ity is called plasticity. Second, they must retain the knowledge from previously learned tasks, known as memory stability. When this does not happen, and the performance of previous tasks degrades, the model suffers from Catastrophic Forgetting 2 These two properties are the central core of Continual Learning (CI.) [3]4] being strongly relevant for ML systems suscep- tible to test-time distributional shifts.
PermutedMNIST-Hard
1.00
main prop-
TCCA-VCL
0.95
Accuracy
0.90
0.85
0.80
0.75
5
10
Number of Observed Tasks
Figure 1: Average accuracy across observed tasks in the PermutedMNIST-HHard benchmark. The TD-VCL approach, proposed in this work, leads to a substantial improvement against stan- dard VCL and non-variational approaches.
Given the critical importance of this topic, ex- tensive literature addresses the challenges of CL and, more
in traditional ML methods [ 2 6
Correspondence to: luckeiin.cavahhonbbbeesss..o
Denotes equal supervision.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

VCL
Temporal-Differenee VCL
xO Psa)
v10 Da.)
(1)
6 Paa)
0s(0)
26 (Pia)
A00 Das)
110 (ua))
(0)
pi0 (P))
pC0 PP.)
2
p(0
p(
Cradent Staps
Stelgg x/ (Dua))
An intuitive illustration of how TD-VCL functions in comparison to vanilla VCL. At each timestep f, new task dataset D, arrives. Both methods aim to learn variational parameters a (0) over a family of distributions Q that approximates the true posterior p(@ Di:e) via minimizing the KL divergence D (gu (0) lI p(0 Dia)). VCL optimization (left) is only constrained by the most recent posterior, which compounds approximation errors from previous estimations and potentially deviates far from the true posterior. TD-VCL (right) is regularized by sequence of past estimations, alleviating the impact of compounded errors.
Figure 2:
recently, for overparameterized models g In this work, we focus Bayesian CL methods, for two reasons. First, it provides a principled, self-consistent framework for learning in online or low-data regimes (9]. Second, Bayesian models express their uncertainty over predictions, which is crucial for safety-critical applications [10] and for enabling principled data selection
Particularly, investigate Variational Continual Learning (VCL) approaches [33 As detailed in Section 3 VCL identifies a recursive relationship between subsequent posterior distributions over tasks. A variational optimization objective then leverages this recursion, which regularizes the updated posterior to stay close to the very latest posterior approximation. Nevertheless, we argue single previous posterior estimate for building up the next optimization target may be ineffective, as the approximation error propagates to the next update and compounds after successive recursions. If particular estimation is especially poor, the error will be carried over the next step entirely, which can dramatically degrade model's performance.
that solely relying
In this work, we show that the same optimization objective can be represented as function of sequence of previous posterior estimates and task likelihoods. We thus propose a new Continual Learning objective, ntep KL VCL, that explicitly regularizes the posterior update considering several past posterior approximations. By considering multiple previous estimates, the objective dilutes individual errors, allows correct posterior approximates to exert corrective influence, and leverages a broader global context to the learning target, reducing the impact of compounding errors over time. Figure illustrates the underlying mechanism.
We further generalize this unbiased optimization target to a broader family of CL objectives, namely Teempoal-DDiffrrence VCL, which constructs the learning target by prioritizing the most recent approximated posteriors. We reveal link between the proposed objective and Temporal-Differenc (TD) methods, popular learning mechanism in Reinforcement Learning [14 and Neuroscience (15). Furthermore, we show that TD-VCL represents spectrum of learning objectives that range from vanilla VCL to n-Step KL VCL. Finally, we present experiments several challenging and popular CL benchmarks, demonstrating that they outperform standard VCL (as shown Figure DD. other VCL-based methods, and -arriiiaaional baselines, effectively alleviating Catastrophie Forgetting.
Related Work
Continual Learning has been studied throughout the past decades, both in Artificial Intelligence 3, 5. [66 and in Neuro- and Cognitive Sciences [1 2 More recently, the focus has shifted

---

towards overparameterized models, such as deep neural networks 1 18] Given their powerful predictive capabilities, recent literature approaches CL from wide range of perspectives. For instance, by regularizing the optimization objective to account for old tasks [9121]] by replaying an external memory composed by set of previous tasks 24] or by modifying the optimization procedure manipulating the estimated gradients 27 We refer to Wang et for an extensive review of recent approaches. Our proposed method is placed between regularization-bat and
replay-based methods.
Bayesian CL. In the Bayesian framework, prior methods exploit the recursive relationship between subsequent posteriors that emerge from the Bayes' rule in the CL setting (Section 3). Since Bayesian inference is often intractable, they fundamentally differ in the design of approximated inference. We highlight works that learn posteriors via Laplace approximation (29 [0]. sequential Bayesian Inference 31 [2], and Variational Inference (VI) in the latter category.
Our work and proposed method lies
Variational Inference for CL. Variational Continual Learning (VCL) [33 introduced the idea of online VI for the Continual Learning setting. It leverages the Bayesian recursion of posteriors to build an optimization target for the next step's posterior based on the current one. Similarly, our work also optimizes a target based on previous approximated posteriors. On the other hand, rather than relying on a single past posterior estimation, it bootstraps on several previous estimations to prevent compounded errors. Nguyen et al. [13] further incorporate an heuristic external replay buffer to prevent forgetting, requiring two-step optimization. contrast, our work only requires single-step optimization as the replay mechanism naturally emerges from the learning objective.
Other derivative works usually blend VCL with architectural and optimization improvements 35 46) or different posterior modeling assumptions [34] 41+44) We specifically highlight UCB which adapts the learning rate according the uncertainty of the Bayesian model, and UCL
33
38
which introduces a different implementation for the VCL objective by proposing the notion of node-wise uncertainty. While their contribution are orthogonal ours, we adopt UCB and UCL as comparison methods to further show that our proposed objective may also be combined with other variational methods and enhance their performance.
43.
3
Preliminaries
Problem Statement. In the Continual Learning setting, model learns from streaming of tasks, which forms a non-stationary data distribution throughout time. More formally, we consider a task distribution T and represent each task i ~ T as set of pairs {(x,, Dt) %., where N, is the dataset size. At every timestep f the model receives a batch data D, for training. We evaluate the model in held-out test sets, considering all previously observed tasks.
In the Bayesian framework for CL, we assume a prior distribution over parameters p(0), and the goal is to learn a posterior distribution p(0 Dirr) after observing T tasks. Crucially, given the sequential nature of tasks, we identify a recursive property of posteriors:
p(0 Du.T) x p(0)p(DDer Lid p(0)
II (DD 0) x p(0) Di:r-1)((Dr 0),
(1)
where we assume that tasks are i.i.d. Equation Ianrriionssooee shows that may update the posterior estimation online, given the likelihood of the subsequent task.
Variational Continual Learning. Despite the elegant recursion, computing the posterior p(0 Di.r) exactly is often intractable, especially for large parameter spaces. Hence, we we rely on an approximation. VCL achieves this by employing online variational inference [4). It assumes the existence of
variational parameters (0) whose goal is to approximate the posterior by minimizing the following KL divergence over space of variational approximations Q:
1
gr (0) = arg min 2x (q(0) Z, th (0)p(() 0)),
||
ge@
Forr notational simplicity, we use the index f to denote both tasks and timesteps. Note that neither the VCL framework our proposed methodology requires knowledge of task boundaries, argued in the Appendix N
3

---

where Z, represents a normalization constant. The objective in Equation i equivalent to maximizing
the variational lower bound of the online marginal likelihood:
CVvL (0) = Eg log p(P 0)] (xz( q1-1)))))
(3)
We can interpret the loss Equation through the lens of the stability-plastictty dilemma [4] The first term maximizes the likelihood of the new task (encouraging plasticity), whereas the KL term penalizes parametrizations that deviate too far from the previous posterior estimation, which supposedly contains the knowledge from past tasks (encouraging memory stability).
4 Temporal-Difference Variational Continual Learning
Maximizing the objective in Equation 3 is equivalent to the optimization in Equation 2 but its computation relies on two main approximations. First, computing the expected log-likelihood term analytically is not tractable, which requires a Monte-Carlo (MC) approximation. Second, the KL term relies on a previous posterior estimate, which may be biased from previous approximation errors. While updating the posterior to account for the next task, these biases deviate the learning target from the true objective. Crucially, as Equation solely relies on the very latest posterior estimation, the егхог compounds with successive recursive updates.
Alternatively, may represent the same objective estimations and alleviate the effect of the approximation error from any particular one. By considering several past estimates, the objective dilutes individual errors, allows correct posterior approximates to exert corrective influence, and leverages broader global context to the learning target, reducing the impact of compounding errors over time.
function of several previous posterior
4.1 Variational Continual Learning with n-Step KL Regularization
We start by presenting new objective that equivalent Equation while also meeting the aforementioned desiderata:
Proposition 4.1. The standard KL minimization objective in Variational Continual Learning (Equa-
tion 2 is equivalently represented the following objective, where n E No is hyperparameter:
(0) arg max E ( QEQ
(n i)
log (D.ss |0)]
Σ
2KL (g)()((0)) (0)--(0))) (4)
We present the proof of Proposition 4.1 in Appendix A We name Equation 4 as the n-Step KL regularization objective. It represents the same learning target Equation 2 as a sum of weighted likelihoods and KI. terms that consider different posterior estimations, which can be interpreted as "distributing" the role of regularization among them. For instance, if an estimate (t=; deviates too far from the true posterior, only affects 1/n of the KL regularization term. The hyperparameter assumes integer values up to & and defines how far in the past the learning target goes. If n is set to 1, we recover vanilla VCL.
An interesting insight comes from the likelihood term. contains the likelihood of different tasks, weighted by their recency. Hence, the idea of re-estimating old task likelihoods, commonly leveraged as a heuristic in CL methods, fundamentally emerges in the proposed objective. We may estimate these likelihood terms by replaying data from different tasks simultaneously, alleviating the violation of the i.i.d assumption that happens given the online, sequential nature of CL. (7).
From n-Step KL to Temporal-Different Targets
The learning objective in Equation 3 relies on several different posterior estimates, alleviating the compounding error problem. A caveat that estimates have the same weight the final objective. One may want to have more flexibility by giving different weights for them - for instance, amplifying the effect from the most recent estimate while drastically reducing the impact of previous ones. possible to accomplish that, shown in the following proposition:

---

Proposition 4.2. The standard KL mininization objective in VCL (Equation 2 is equivalently represented the following objective, with n E No. and X E [0, 1) hyperparameters:
X(1 - ) Σ 1=0
ge ((( arg max E Q
log (D 0)
1-
(11 A)
(2) (---1(0))).
(5)
X"
The proof is available in Appendix We call Equation the TD(A)- objective It augments the n-Step KL Regularization weight the regularization effect of different estimates in way that geometrically decays via the A term as far as it goes the past. Other A-related terms serve as normalization constants. Equation provides more granular level of target control.
Interestingly, this objective relates intrinsically to the X-returns for Temporal- Difference (TD) learning in falued-basett reinforcement learning [46]. More broadly, both objectives of Equations and S are compound updates that combine n-step
targets, as shown below. First, wc
formally define TD target in the CL context:
Definition 4.3. For a timestep i, the n-Step Temporal-Differene target for Variational Continual Learning is defined as, Yn E Ng,
(D. Eg (0)
Σ log
2xz(ge (O))
(6)
In Appendix C we reveal the connection between Equation 6 and the TD targets employed in Reinforcement Learning, justifying the adopted terminology. From this definition, it follows that:
Proposition 4.4. Vn No, S t, the objective in Equation 2 can be equivalently represented as:
go( arg max (n),,),
with (n) as in Definition 4.3
Furthermore, the objective in Equation 5 can also be represented
(0) = arg max
1-
Σ TD
(8)
1 A
TEQ
Discounted targets
The proof is in Appendix D Proposition 4.4 states that the TD(A)-VCL objective is a sum of discounted TD targets (up normalization constant), effectively representing A-returns. In parallel, one can show that the n-Step KI. Regularization objective, as a particular case, is simple average of n-Step TD targets. Fundamentally, the key idea behind these objectives is bootstrapping: they build learning target estimate based on other estimates. Ultimately, the "A-target" in Equation 5 provides flexibility for bootstrapping by allowing multiple previous estimates to influence the objective.
The TD-VCL objectives generalize a spectrum of Continual Learning algorithms. As a final remark, in Appendix E, show that, based on the choice of hyperparameters, the TD(A)-CCL objective forms family learning algorithms that span from Vanilla VCL to n-Step KL Regulariza- tion. Fundamentally, it mixes different targets of MC approximations for expected log- ligliiilidood and KI. regularization. This process is similar to how TD(A) and n-step TD mix MC updates and TD predictions in Reinforcement Learning, effectively providing mechanism to strike balance between the variance from MC estimations and the bias from bootstrapping (46
We refer both StStep KL Regularization and TD(A)-VCL TD-VCL objectives.

---

5 Experiments and Discussion
Our central hypothesis is that for Bayesian CL, leveraging multiple past posterior estimates mitigates the impact of compounded errors inherent to the VCL objective, thus alleviating the problem of Catastrophic Forgetting. We now provide an experimental setup for validation. Specifically, we evaluate this hypothesis by analyzing the questions highlighted in Section 5.1
Implementation. We use a Gaussian mean-field approximate posterior and assume Gaussian prior (O0, I)), and parameterize all distributions as deep networks. For all variational objectives, we compute the KL term analytically and employ Monte Carlo approximations for the expected log- likelihood terms, leveraging the reparametrization trick [47] for computing gradients. We employed sicelhhood-tempering [33 to prevent variational over-pruning 48]. Lastly, for test-time evaluation, we compute the posterior predictive distribution by marginalizing out the approximated posterior via Monte-Carle sampling. We provide further detail about architecture and training in Appendix G and our code
Comparison Methods. We compare TD-VCL and n-Step KL VCL against several methods. We first evaluate non-variational naive methods for CL: Online MLE naively applies maximum likelihood estimation in the current task data. It serves evaluate how challenging the benchmark is. Batch MLE applies maximum likelihood estimation considering a buffer of current and old task data. Next, we adopt the following variational methods for direct comparison in the Bayesian CL setting: VCL, introduced by Nguyen et al. [13], optimizes the objective in Equation 3 VCL CoreSet a VCL variant that incorporates a replay set to mitigate 43 is another variational method that implements adaptive regularization based on the notion of node-wise uncertainty. Finally, UCB [8 also optimizes the objective of Equation (3 but adapts the learning rate for each parameter based on their uncertainty. Particularly for UCL and UCB, we compare them with the proposed TD-UCL and TD-UCB, which incorporate the introduced objective into UCL and UCB, respectively.
lower bound for other methods, well way to
any residual forgetting [13.. UCL
Benchmarks. We evaluate five benchmarks for Continual Learning (CL). First, we introduce three new benchmarks: PermutedMNIST--Hare SplitMNISTTHardd and SplitNotMNIST-Haai
These are more challenging versions of traditional CL benchmarks with similar names. They are significantly harder due to two key restrictions. First, the amount of replay memory that any method can use is limited in both dataset size and the number of tasks. As empirically shown in Appendix I this creates a much more acute scenario of Catastrophic Forgetting. Second, they enforce the adoption of single-head classifiers. As also shown in Appendix H. this requires the model to account for the potential negative transfer learning among tasks, which makes MNIST/NotMNNSST based benchmarks non-trivial for current research. Next, we also evaluate on two other popular CL benchmarks: CIFAR100-10 and TinylmageNet-10. Both benchmarks are very challenging classification problems, particularly in our setting where no pre-trained representations are used. In Appendix we detail all benchmark tasks and specific constraints adopted for robust evaluation.
5.1 Experiments
We highlight and analyze the following questions to evaluate our hypothesis and proposed method:
Do the TD-VCL objectives effectively alleviate Catastrophic Forgetting in challenging CL benchmarks? Tables and present the results for all benchmarks. Each column presents the average accuracy across the past ł observed tasks, and we show the results starting from t = 2 as t t= 1 is simply single-task learning. For PermutedMNIST-HHard all methods present high accuracy for t = 2, suggesting that they could fit the data successfully. As the number of tasks increases, they start manifesting Catastrophic Forgetting at different levels. While Online and Batch MLE drastically suffer, variational approaches considerably retain old tasks' performance. The Core Set slightly helps VCL, and both n- Step KL and TD-VCL outperform them by a considerable margin, attaining approximately 90% average accuracy after all tasks. For completeness, Figure ] grapphically shows the results. We emphasize the discrepancy between variational approaches and naive baselines and highlight the performance boost by adopting VD- objectives.
For SplitMNIST-Hard we highlight that the TD-VCL objectives also surpass baselines in all configurations, but with a decrease in performance for t = 5, suggesting a more challenging setup for
*Our code is available at htttttts///gthubb http://....bouuuccccaaaaat//-A/

---

Table I: Quantitative comparison on the PermutedMNIIT--Hard SpliMMNIST-Hard, and SplitNotMNNISTHHard benchmarks. Each column presents the average accuracy across the past t observed tasks. Results are reported with two standard deviations across ten seeds. Top two results are in bold, while noticeably lower results are in gray. TD-VCL objective consistently outperforms standard VCL variants, especially when the number of observed tasks increase.
PermutedMNIST-FHam
t= 2
t= 3
t= 4
t=5
t= 6
t=
t= 8
9
t= 10
Online MLE
0.87a0.07
0.77.0.06
0.73a0.08
0.69
0.65a0.13
0.57
0.51
0.46a0.11
0.40a0.08
Batch MLE
0.95.0.01
0.93x0.01
0,88a0.04
0.83
0.77a0.10
0.71
0.57 0.64a0.12
0.5 20.06
VCL
0.9520.00
0.9420.01
0.9320.02
0.91
0.89a0.03
0.8620.03
0.83a0.04
0.80a0.06
0.7820.04
VCL CoreSet
0.9620.00
0.95+0.00
0.94+0.00
0.93s0.02
0.91
0.89=0.02
0.86+0.03
0.84
0.81 0.81-s +0.03
n-Step TD-VCL
0.95.n.01
0.94+0.00
0.94+0.00
0.9320.01
0.92a0.01
0.91=
0.90.a.02
0.89 0.88=n.02
TD(A)-CCI.
0.97
0.962n.00
0.95+0.00
0.94s0.01
0.93s0.01
0.92=0.01 0.91 0.90 0.89
SplitMNNITTHard
SplitNotMNNSTTHarrd
t= 2
3=3
t=
1= 5
1 ==
1=3
t=4
5=
Online MLE
0.86.0.02
0.61
0.75.0.44
0.57=0.06
0.72=0.02
0.6140.05
0.61a0.00
0.51 +0.04
Batch MIEE
0.95#0.04
0.65+0.04
0.82+n.04
0.59
0.71=0.0
0.65#0.03
a0.00 0.50=a.06
VCL
0.8720.02
0.6620.04
0.82+0.03
0.64s0.11
0.6920.04
0.63+0.03
0,60a0.00
0.50
VCL CoreSet
0.9320.04
0.68e0.07
0.8420.04
0.62:0.03
0.6920.04
0.65g0.02
0.60a0.01
0,5 10.07
n-Step TD-VCL
0.98=0.01
0.79+0.08
0.88+0.04
0.67#0.04
0.72=0.04
0.73
0.70=0.04 0.58+0.08
TD(A)-VLL
0.98.0.01
0.81.0.07 0.8920.03 0.6620.02
0.7420.02
0.7320.03 0.69.0.03 0.5820.09
Table 2: Quantitative comparison on the CIFARI00-10 and TinyImaagent--1 benchmarks. Each column presents the average accuracy across the past t observed tasks. Results are reported with two standard deviations across five seeds. TD-VCL variants consistently outperform the baselines in harder benchmarks with more complex architectures, such as Bayesian CNNs. The full table is available in the Appendix M
CIFAR100-10
TinyImageNet-10
t= 4
t=
t=8
t= 10
t=
t= 8
t= 10
Online MLE
0.57a0.06
0.56an.03
0.53 0.52a0.04
0.45.a.02
0.4420.01
0.45a0.02 0.44a0.03
Batch MLE
0.58.0.04 0.58
0.56.0.06 0.54
0.48aa.02 0.4820.02 0.50a0.02 0.51
VCL
0.63 0.60=n.02
0.61 40.05 0. 0.66.0.01
0.5 0.51+0.03
0.51 40.02 0.51
VCL. CoreSet
0. 0.63+0.03 0.63=no2 0.61 40.02
0.65
0.51
0.51+0.02 0.54 0.54+0.02
n-Step TD-VCL.
0.67 0.65=a.01 0.68.0.04 0.69.1.02
0.55.a.02
0.5420.02 0.56#a.02 0.5640.02
VCC
0.66 0.66=n.02 0.67 0.71
0.56xa.0z 0.55 0.56=a.02 0.5620.02
addressing Catastrophic Forgetting that opens a venue for future research. We discuss SPlitMNIST- is a harder benchmark, as
Hard results in more detail in Appendix K Next, SplitNot the letters come from a diverse set of font styles Furthermore, we purposely decided to employ a modest network architecture (as for previous benchmarks). Facing hard tasks with less expressive parametrizations will result in higher posterior approximation error. Our goal is to evaluate how the variational methods behave in this setting. Once again, n-step KL and TD-VCL surpassed the baselines after observing more than three tasks. The effect more pronounced after increasing the number of observed tasks. These objectives are the only ones whose resultant models achieved non-trivial average accuracy after observing all tasks.
Lastly, we analyze the results on CIFAR100 and TinyImageNet-10 in Table 2 These are considerably harder benchmarks, as the distribution of images and classes is much richer than the previous benchmarks. Furthermore, they necessarily require better architectures to attain non-trivial performance. Following previous work [8] 49 50] we adopt an AlexNet architecture [5I] This setup is ideal for evaluating how the learning objective functions at a larger scale with more complex, deep architectures such as (Bayesian) convolutional networks. Once again, TD-VCL objectives attain superior performance, particularly for later timesteps, where Catastrophic Forgetting is more pronounced in the baselines. This suggests that leveraging multiple posterior estimates for learning is better than only the latest one, even when the approximation error is high.
7

---

PermutedMNNiS--Hardd Per Task Performance
Task
Task
Taisk
Task
Task 5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
Accuracy
0.6
0.6
0.4
0.4
0.4
0.4
0.4
022
0.2
1 2
10
2
910
1 2 3 7 9.10
2
100
2
910
Number Observed lasks
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
Accuracy
Method
0.6
Online MLE
Batch
0.4
0.4
0.4
0.4
0.4
Corcšet
0.2
0.2
2
9.
3 5.6
1 2 6 910 Number Observed Tasks
2 6
1 2
1
023356673 910
Figure 3: Per-task performance (accuracy) over time in the PermutedMNIST-HHad benchmark. Each plot represents the accuracy one task (identified in the plot title) while the number of observed tasks increases. We highlight a stronger effect of Catastrophic Forgetting on earlier tasks for the baselines, while TD- DCL objectives are noticeably more robust this phenomenon.
How do the TD-VCL objectives affect per-task performance? While the previous question analyze the performance averaged across different tasks, we now investigate the accuracy of each task separately in the course of online learning. This setup is relevant since solely considering the averaged accuracy may hide a stronger Catastrophic Forgetting effect from earlier tasks by "compensating" with higher accuracy from later tasks. We show the results for PermutedMNIST-HHae in Figure [ (we defer additional per-task results for Appendix K. It presents a sequence plots, where each figure represents the accuracy of one task while the number of observed tasks increases. Naturally, the tasks that appear at later stages present fewer data points: for instance, "Task 10" has single data point as it does not have test data for earlier timesteps.
As observed, per-task performance explicitly shows a stronger effect of Catastrophic Forgetting for earlier tasks in the adopted baselines. We particularly highlight how non-variational approaches fail for them. this direction, TD-VCL objectives presented more robust performance against others. For instance, we highlight the results for Task 1. After observing all tasks, the proposed methods demonstrated accuracy of around 80% and 85%. The VCL baselines dropped to 50% and 60%, and MLE-based methods failed with only 20% of accuracy.
How does TD-VCL (and variants) perform against other Bayesian CL methods?
In this work, we focus on Continual Learning with Bayesian lens. As highlighted in Section [ it provides formal, uncertainty- framework crucial for safety-critical applications and data- efficient learning. Thus, we analyze the TD objective (Fquation 5) on other Bayesian CL methods. UCL and UCB are variational methods that optimize the objective in Equation 2 but propose new mechanisms for regularization and learning rate adaptation. Since these enhancements are orthogonal the objective, incorporate the proposed TD objective with these methods, resulting in TD-UCL and TD-UCB, respectively. We aim to show that the TD objectives for CL work across different base methods and promote performance boost on them.
Table 3 compares the base methods (VCL, UCL, and UCB) with their TD-enhanced counterparts (complete results in Appendix MM. While there is no dominant base method across the benchmarks, the TD counterparts consistently improve upon their respective base methods, especially at later timesteps. These results indicate that the TD objective is robust among different Bayesian CL algorithms and may be incorporated effectively into methods that rely on the variational objective in Equation 2
How do the TD-VCL objectives behave with the choice of the hyperparameters n, X, and the likelihood-temen teeeeermmingggg parameter S² The proposed learning objectives introduce two new

---

Table 3: Quantitative comparison between Bayesian CL methods and their TD-enhanced counterparts. The TD-enhanced methods incorporate the objective in Equation | in each base method. Although single base method consistently outperforms the others across all benchmarks, their TD-enhanced versions consistently achieve better performance, particularly at later timesteps. The full table is avaliable in Appendix M
PermufedMNIST--Hard
SplitMNIST-Hard
t=
t=6
t=
t= 10
t= 2
1=3
t=4
t= 5
VCL
0,93+002
0.89
0.83
0.78a0.04
0.87a0.02
0.66an.4
0.82a0.03
0.64a0.11
TD(I-VCL
0.95+0.00
0.93w0.01
0.91
0.89
0.98
a.79
0.88
0.67
UCL
0,94:0.00
0.89+0.02
0.83+006
0.73+0.12
0. 0.88
0.68+000
0.83
0.66+0.06
TD()-UCL
0.95=0.00
0.92+0.02
0.88=0.04
0.84
0.97
0.85
0.90+0.02
0.70
UCB
0..220.01
0.89
0.8620o2
Z0.02
T.85aa 0.79 12
O. T.7
TD())UCB
0.9320.00
0.9120.01
0.90
0.88
0.93s0.02
2.89 20.03 0.87 2.80
CIFAR100-10
TinyImageNet-Dl
4 4
t=
t=8
t= 10
t= 4
t=
t= 8
t= 10
VCL
0.6320.02
0,60ao.02
0.61
0.6620.01
0.5
a.5 0,5 a0.02 0.5
TD(A--CL
0.6620.04
0.6620.02
0.67 0.71.0.01
0.5610.02
0.5520.03
0.56s0oz 0.5622.06
UCL.
0.641..0
0.60....5
0.58-0022
0.622a..22
0.524a.03
0.5T.m.02
0.32.
0.50 0.500
TD(A)-UCI
0.64
0.70s0.02
0.66#0.03
0.6720.03
0.54.0.01
0.5420.01
0.55
0.56x0001
UCB
0.66x0002
0.6620.03
0.65e0:r
066000001
0,48a0.04
0,45s000
0,42+0.03
TD(A)-UCB 0.6620.01 0.67 0.68=0.01 0.70z0.01
0.52x0.01 0.51 0.50=a.03 0.47
hyperparameters: n (the number of considered previous posterior estimates in the learning target) and for TD(A)- VCL (which controls the level of influence for each past posterior estimate). Furthermore, it also inherits the 3 parameter from VCL. Hence, we evaluate the sensitivity of the proposed objectives concerning these hyperparameters, presenting results and detailed discussion in Appendix L We highlight three main findings. First, similarly to VCL, TD-VCL objectives are hyperparameter. Second, increasing n is beneficial up to a
sensitive to the lkelihhod-temperring certain point, from which it becomes detrimental, suggesting the existence of optimal range for leveraging posterior estimates. Lastly, TD-VCL objectives present robustness over the choice of X, with more pronounced effect when the number of observed tasks increases.
6 Closing Remarks
In this work, we presented a new family of variational objectives for Continual Learning, namely Temporal-Difference VCL. TD-VCL is unbiased proxy of the standard VCL objective but lever- ages several previous posterior estimates to alleviate the compounding error caused by recursive approximations. We showed that TD-VCL represents a spectrum of Continual Learning algorithms and is equivalent to a discounted sum of n-step Temporal-Difference targets. Lastly, we empirically presented that it helps address Catastrophic Forgetting, surpassing Bayesian CL baselines in several challenging benchmarks.
Limitations. Despite being theoretically principled and attaining superior performance, TD-VCL presents limitations. First, the hyperparameters n and X depend on the evaluated setting, which may require certain tuning. Second, the objectives rely on past posterior estimates, which may increase memory requirements. Still, we believe this is not a major limitation as TD-VCL suits well modern deep Bayesian architectures that target smaller parameter subspaces for posterior approximation
52 53 [2]
Future Work. While presenting connections with Temporal-Differenne methods, TD-VCL is not an RL algorithm. Further mathematical connections with Markov Decision/Reward Processes formalism left future work. Another interesting direction is to apply TD-VCL objectives for other problems that involve sequential variational inference, such as probabilistic meta-learmin [54.55).

---

Acknowledgments and Disclosure of Funding
The authors thank Panagiotis Tigas for insightful discussions on variational inference. Luckeciano Melo acknowledges funding from the Air Force Office of Scientific Research (AFOSR) European Office of Aerospace Research Development (EOARD) under grant number TA8665-21-1-7017. Yarin Gal is supported by Turing AI Fellowship financed by the UK government's Office for Artificial Intelligence, through UK Research and Innovation (grant reference EP/V030302/1 and delivered by the Alan Turing Institute.
References
[1] J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-basee neural networks. In International Conference on Learning Representations, pages 1-10, 2015.
[2] Michael McCloskey and Neal J. Cohen, Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24:109-165, 1989. URL https://api semanticscholar.
[3] Jeffrey C. Schlimmer and Douglas Fisher. A case study of incremental concept induction. In Proceedings of the Fifth AAAI National Conference on Artificial Intelligence, AAAI'66, page 496-501, AAAI Press, 1986.
[4] Wickliffe C. Abraham and Anthony Robins. Memory retention the synaptic stability versus plas- ticity dilemma. Trends Neurosciences, 28(2):73-78, 2005. ISSN 0166-2236. doi: https://oi. org/10. 101 6/j tins. tins.2004. 2004, 12.003. URL hhtttt:/tt///ttt http://www.eeeeeeeeeeeeeeeeeeeee 50166223604003704
[5] Richard S. Sutton and Steven D. Whitehead, Online learning with random representations. In Proceedings of the Tenth International Conference Interaational Conference Machine Learning, ICML'93, page 314-321, San Francisco, CA, USA, 1993. Morgan Kaufmann Publishers Inc. ISBN 1558603077.
[6] Robert French. Catastrophic forgetting connectionist networks. Trends in Cognitive Sciences. 3(4):128-135 135, 1999. ISSN 1364-6613. doi: http:://doi..rg///.... 1016// 36990666111991119944 URL com wttp:///....oaaaaaaaa///////////
https: : http///wwwcencceeee
[7] Raia Hadsell, Dushyant Rao, Andrei A. Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in Cognitive Sciences, 24(12):1028 1040, 2020. ISSN 334466613.
doi: https://dooi.or//0... 101.. 00.0.09.004. URL http::::/wwwwssceeceedeeeee com/science aat.cclepp11/1184661100888888880
[8] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In Jennifer Dy and Andreas Krause, editors. Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research. pages 4548 4557, PMLR, 10-15 Jul 2018. /tttss:t:////rtoeeeei wtw../eedde.n.......rrr////////o html
mlr. wwwps//80////eee serra18a
[9] Tom Rainforth, Adam Foster, Desi R. Ivanova, and Freddie Bickford Smith. Modern Bayesian Experimental Design. Statistical Science, 39(1):100 114, 2024. doi: 0111211/23-SS99911 URL https://doi. org/
10. 10-112223-SSSSS2
[10] Alex Kendall and Yarin Gal. What uncertainties do need in bayesian deep learning for computer vision? In Proceedings of the 31st International Conference on Neural Information Processing Syxtems, NIPS' 17, page 5580-5590, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
[11] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICMI. 17, page 1183-1192. JMLR.org, 2017.
[12] Luckeciano Melo, Panagiotis Tigas, Alessandro Abate, and Yarin Gal. Deep bayesian active learning for preference modeling large language models, 2024, http/httpp:////xw.orrr
International Conference on Learning Representations, 2018. URL http///axii..orr//////////// hwtp://ab///4..... 10023 id Ld=BkQqqQgRb
[13] Cuong Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learning. In wwtp:ewwwwwwieeeeoruuuu...
10

---

[14] Richard S. Sutton. Learning to predict by the methods of temporal differences. Mach. Leam.. 3(1):9-44 August 1988. ISSN 0885-6125. doi: (0.1023/A.1022633331449 /t// 1023/A 1022633531479
[15] Wolfram Schultz, Peter Dayan, and P. Read Montague, A neural substrate of amedictioonamddewaat and reward. Science, 1759 1997. doi: 1126/ccience.27.550666 doi/abs/10. 1ttp:///0...1////iieeeeee 275 5006..536.... 6306.1693
URL/httss:/www..c..eeo org/
[16] Mark B. Ring. Child: A first step towards continual learning. Mach. Learn, 28(1):77-004, jul 1997. ISSN 0885-6125. doi: 10. 1..022/A:107733172355 URI. hhttps://aii.rg/ 1003//A1100033172333777
[17] Timo Flesch, Andrew Saxe, and Christopher Summerfield. Continual task learning in natural and artificial agents. Trends in Neurosciences, 46(3):199-210, 2023. ISSN 0166-2236. doi: hitps:
Mlboi.org/10.1016f..iinssss 2022. 12.006. URL BBLLitttee///...eeccciieeett http://rrr..cmm/ccccccc///rtttta
p11/901662236222022600
[18] Tameem Adel, Han Zhao, and Richard E. Turner. Continual learning with adaptive weights (CLAW). In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL htt.httt////peeeiiieeeee net//ooruum.dddRRRl22222r
[19] James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, Demis Has- sabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114:3521 3526, 2016. URL org/ : 4704285
[20] Friedemann Zenke, Ben Poolc, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' page 3987-3995. JMLR.org. 2017.
[21] Arslan Chaudhry, Puncet K. Dokania, Thalaiyasingam Ajanthan, and Philip S. Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Vittorio Ferrari, Martial Hehert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision ECCV 2018, pages 556-572, Cham, 2018. Springer International Publishing. ISBN 978-3-030-01252-6.
[22] David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learn- ing. In I. Guyon, U. Von Luxhurg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL htttps://pooceedeee www.spssccppppppppeeeeeeeeee.e http:/ppppe.lllleeepp/p///////// 01752276222222111557772777766822
neurips...... pdf
[23] Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Ha, and Jonghyun Choi. Rainhow memory: Continual learning with memory of diverse samples. 2021 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8214-8223 8223, 2021. doi: 10.1 (t009//CVPR4447.221...00.
[24] Sylvestre-AIie Alvise Rebuffi, Alexander Kolesnikov, Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5533-5542, 2016. URL/ttta:: //a/ii semanticachollar org/ Corpus ID 206596226666000
[25] Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent pro- cessing in neural networks. Nature Machine Intelligence, 1:364 372, 2018. URL https://app semanticacholat org/ IorpusID: :59086422
[26] Khurram Javed and Martha White. Meta-learning representations for continual learning. Curran Associates Inc., Red Hook, NY, USA, 2019.
[27] Hao Liu and Huaping Liu. Continual learning with recursive gradient optimization. In International Conference on Learning Representations, 2022. hwallaatppp//ppppeeeeeeee//e///p
Zgm
[28] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. comprehensive survey of continual learning: Theory, method and application. IEEE transactions on pattern analysis and machine intelligence, PP. February 2024. ISSN 0162-8828. doi: 10. wt..1009pami2222433333333 2302.00487
URL htttt::/t//rrrat org/pdt.
[29] Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18 18, page 3752-3752, Red Hook, NY, USA, 2018. Curran Associates Inc.

---

[30] Jonathan Schwarz, Wojciech Czarnecki. Jelena Luketina, Agnieszka
Yee Whye Teh. Razvan Pascanu, and Raia Hadsell. Progress compress: A scalable framework for continual learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4528-44484 45337, PMLR, 10-15 Jul 2018. URL https://proceeedings mlr wwwpprresssvveee ww0/schwarz18a...mmm html
[31] Michalis K. Titsias, Jonathan Schwarz, Alexander G. de G. Matthews, Razvan Pascanu, and Yee Whye Tch. Functional regularisation for continual learning with gaussian processes. In International Conference on Learning Representattons, 2020. URL http:///ppppaaeeeaaeeeeee/to/onr
[32] Pingbo Pan, Siddharth Swaroop, Alexander Immer, Runa Eschenhagen, Richard Turner, and Mo- hammad Emtiyaz Khan. Continual deep learning by functional regularisation of memorable past. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713899
[33] Noel Loo, Siddharth Swaroop, and Richard E Tumner Generalized variational continual learning. In International Conference on Learning Representations, 2021. URL htt.htttee//ooeereviiee www.eerrrrreeneeeeoounn net/forrm?
id=_IM- AM-AfFhna9
[34] Tim J. Rudner, Freddie Bickford Smith, Qixuan Feng, Yee Whye Teh, and Yarin Gal. Continual learning via sequential function-space variational inference. In Kamalika Chaudhuri, Stefamie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors. Proceedings of the 39th International Conference Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages PMLR, 17-23 Jul 2022. URL http:tttt//poceediiigeee
mmr www.trreesssvte rttpe///rddreaaa.// html
[35] Noel Loo, Siddharth Swaroop, and Richard E Turner. Combining variational continual learning with fiLM layers. In 4th Lifelong Machine Learning Workshop at ICML 2020, 2020. /tLL:tttp:://peerrriii pet/forumm1dd-22EGAA
[36] Liu Guimeng, Guo Yang. Cheryl Wong Sze Yin, Ponnuthurai Nagartnam Suganathan, and Ramasamy Savitha. Unsupervised generative variational continual learning. In 2022 IEEE International Conference on Image Processing (ICIP), pages 0028-002,, 2022. doi: 10. 10.1100/CI1446556222222.99999999
[37] Hanna Tseran. Natural variational continual learning. 2018. URL https:://ap...emmttiiiccccc org/CorpusID: 155098533
[38] Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach. Uncertainty-guiie continual learning with bayesian neural networks. In International Conference on Learning Representations, 2020. URL http:ttt///opeeeeeiiee www.eneee//ttnnieeeeeeeeett Hiikruunwkdd
[39] Jeevan Thapa and Rui Li. Bayesian adaptation of network depth and width for continual learning. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2025.
[40] Djohan Bonnet, Kellian Cottart, Tifenn Hirtzlin, Tarcisius Januel, Thomas Dalgaty, Elisa Vianello, and Damien Querlioz. Bayesian continual learning and forgetting in neural networks, 2025. URL https
//arxiv.org// abgg/bb///04..... 13569
[41] Sayantan Auddy, Jakob Hollenstein, and Matteo Saveriano. Can expressive posterior approximations improve variational continual learning? Workshop Lifelong Learning for Long-tern Human- Robott Interaction of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.
[42] Yang Yang, Bo Chen, and Hongwei Liu. Memorized variational continual learning for dirichlet process mixtures. IEEF Access, 7:150851-10065 2019. doi: 10. 10.1E0//CCSSS 2019,,97772
[43] Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Tacsup Moon. Uncertainty based continual learning with adaptive regularization. Curran Associates Inc., Red Hook, NY, USA, 2019.
[44] Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task-agnostic continual learning using online variational bayes with fixed-point updates. Neural Computation, 33(1))3133 3177, 10 2021. ISSN 0899-7667. doi: 10. 1662/ecooaaaaae a__o______0...__1111 URL URL htttpt:ts////o 10440000..1////e0_...44000
[45] Zoubin Ghahramani and H. Attias. Online variational bayesian learning. In NewrlPS Workshop on Online Learning, NeurlPS, 2000.
[46] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
12

---

[47] Diederik P. Kingma and Max Welling, Auto-Encoding Variational Bayes. In 2nd International Conference Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Truck Proceedings, 2014.
[48] Brian Trippe and Richard Turner. Overpruning in variational bayesian neural networks, 2018
[49] Abhishek Kumar. Sunabha Chatterjee, and Piyush Rai. Bayesian structural adaptation for continual learning. In Marina Meila and Tong Zhang, editors, Proceedings the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5850- 5860. PMLR, 18 24 Jul 2021. URL httLhttps:// /ttpps://pooeeedingg maa.eddiinnm..........mmmmmmm..m mlr. press/ v139 wwm./ 2a. html
[50] Tatsuya Konishi, Mori Kurokawa. Chihiro Ono, Zixuan Ke, Gyuhak Kim, and Bing Liu. Parameter-level soft-masking for continual learning. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, JogR, 2023.
[51] Alex Krizhevsky. Learning multiple layers of features from tiny images. In Technical Report, University of Toronto, 2009. URL http::/www. cs toronto. wwwpenttt..udeeeeeeeeee wwwkkk/ieeaaiinn http://aaaannnneeteeeeeeeeeee
features 200eeeeees-------0 2009-TR, pdf
[52] Adam X. Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison, Bayesian low-rank adaptation for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL
https://openrrevovv
1m
[53] Vikranth Dwaracherla, Seyed Mohammad Asghari. Botao Hao, and Benjamin Van Roy. Efficient exploration for LLMs. In Forty-first International Conference Machine Learning, 2024. URL
[54] Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning, In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, 18, page 9537-9548, Red Hook, NY, USA, 2018. Curran Associates Inc.
[55] Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Scbastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for tttp:httts////peerreee International Conference on Learning Representations, 2020.
deep rl via meta- veea-le. In
id=Hk19J1BBy
[56] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Commun. ACM, 60(6)84-96 90, May 2017. ISSN 0001-0782, doi: 10.1145/3065366 URL
httpa://doi. /ttp::////oooo//1100 1145/3065386
[57] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representattions (ICLR), San Diego, CA, USA, 2015.
[58] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255 2009. doi: 10.1
200.
[59] Jasper Snock, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayaman Sundaram, Mostofa Patwary, Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2171-2180, Lille, France, 07-09 Jul 2015. PMLR. URL http:htttss::tteeeeeet wttp:ggp......pp...//smm mlr wrwss/v37 snoek15 html
13

---

A Derivation of the n-Step KL Regularization Objective
In this Section, we prove Proposition
Proposition 4.1. The standard KL minimization objective in Variational Continual Learning (Equa- tion is equivalently represented as the following objective, where n. E No is a hyperparameter:
a- Σ n
(0) arg max E.g. B QEQ
logp(D:-: 10]]
ØK ((:(() q-=-- (4)
n
Proof. Starting from Equation we can expand it as sum of equal terms and utilize the recursive property (Equation D to expand these terms:
1 Z.
(0) = arg min (kz(a(0)
||
-t- (0)p(() 0))
arg min
9 (KL(((0)
||
91 (0))(() 0)) 7
QEQ
arg min qeQ
@kI(α(0)
Z ft 9- (0)p((0 0))
1xx((((0) ||
7477-1 41 (0)pp (D. 0)p((D-11 0)) + |
2xr(1(0) ||
4t 00)
p(D.- 0))
I1:00 Zi_i
1=0
arg min (EQ
(xI(q:(0) (t)((0)) Eg
|log pDD; 0)]
(z(( (()) Ea (0) log (D, 0) + log p (Di-1 0) +...
(kz( (0)) E En Dogg(Dr-1) 0)]
1=
n-1
arg min
2 (q (0)(())))))))))) (0) (0 n log 0)
(n log Mallog(Dee.. (D
log ((Di-n+1) 0)]
n log (o (D=
arg max Eg)g(
Σ
Σ
2 (2: (())
9E@
n
n
14

---

B Derivation of the Temporal-Difference VCL Objective
Before proving Proposition
4.2 we start by presenting a well known result for the sum geometric
series:
Lemma B.1. The finite sum of a geometric series with terms, common ratio and initial term given by:
Xa a(1- X*) (1- X)
(10)
Proof. Let Sn = Eb== aa. Hence,
n- x a x Σ
8p
Ann
xa x= at
k=0
k=0
- ( -- A) = =a11= - x²)
(11)
a(1 X²)
- 8n
(1 A)
Now, we prove Proposition 4.2
Proposition 4.2. The standard KL minimization objective in VCL (Equation 2 is equivalently represented the following objective, with n E No, and X E [0, 1) hyperparameters:
n-1
X(1 Xn-1)
(0) = arg maxE
Σ
log 0)]
qEQ
1-
1=0
X A)
2KI 11 1
(5)
1- N
1=0
Proof. We can Lemma B. to expand the sum of KL terms:
15

---

(: arg min 2KL. (2()) || (t-1(0)p(D.) 0))
Zt
qEQ
1 An
arg min
(xL(g(0) || ( 0))
q
1 \"
1 - x
Z
1
1
arg min
9kL(a(0) || (0)((0) 0))
1 1- X"
7
+ (²
||
Z,Z,-1 4: (02(())( ()p(Pi-11 :0::0000001000::111111111000 10) +...
+ A -" (xz(0) ||
(0)
p(D 10))
IT Zr-t
=
1
arg min Q
2 xxI(q:(0) || (0)) Ea (0) log p Dr 0)]
1- Am
+ X x2xL(9(
|| (()) AE [og pl Dr 0) + logp(D 0)] +...
+ 2x=1 (0) A" A" Bo-1Bo-q(00 (0)
(ogp(D [0]]
1
arg min q
X x9 (1:---1(0)) E0
X log (0)
A- Am
1
1=0
X log (D) 0) + +....+ 1 log D D-n++1 [0)]
+
arg min 1EQ
1
n
Σ
X *K
|| (0)=-((0)) E0
\" log DDg 0)
1 Am
A(1 A"
log pl Dz_1 0) + + -- log pl ( Di-++1 00)
arg max E. Eg~gg (0)
X X
logp Dr_ 0)] -
Σ
n-
x ((1-))
2KT. (9) || ())-)
1 1 A
1- \"
qEQ
1m0
1=0
(12)
16

---

C The connection of TD Targets in TD-VCL and Reinforcement Learning
In the Section we formalize the concept of n-Step Temporal-Differeen for the Variational CL objective (Definition 4.3). In this Section, reveal the connections between this definition and the widely used Temporal-Dife Difference methods in Reinforcement Learning. Our aim is to clarify why Equation 6 indeed represents temporal-diiferene target, both in broad and strict senses.
broad sense, bootstrapping characterizes Temporal-Difiee Difference target: building a learning target estimate based on previous estimates. Crucially, the leveraged estimates are functions of different timesteps. TD-VCL objectives applies bootstrapping in the KL regularization term, by considering one more of posteriors estimates from previous timesteps.
In a strict sense, we can show that Equation deeply resembles TD targets in Reinforcement Learning. RL assumes the formalism of Markov Decision Process (MDP), defined by tuple M = (S, S.A.P P. R. Po. y, H), where S is state space, AA an action space, P transition dynamics. R: : S x A + [- Rmman Rmaar] is a bounded reward function, Po S - 00, o0) an initial state distribution,
S [0. 00) is
[0. 1] is discount factor, and the horizon.
The standard RI. objective is to find a policy that maximizes the cumulative reward:
H
TØ = arg max E
R($s4ks
(13)
T
with a, ~ T0(a st), 8t ~ P(s St-1, at a-11), and so Po(s), where To S A 00) is policy parameterized by 0. Hence, we can define the following learning target, which represents "value" function at each state s,:
Un (8t) = E
(1844,00004)) 8= 81), Vo, E.
(14)
k=
Naturally, it follows that z = arg max, (s), E S. Crucially, expand Equation follows:
14
H
Un(()) = Em
R(s4), 01+k) st)
k=U
EE az) + Σ "R(s s= st]
km
[E,,R((s,, ac) + (
E,,[R(s,, 0z) + *+R(s541, u (8142)),
Ex RR az) + x (st+n),, Vs E S,n < H.
(15)
Temporal-Differtt methods estimates learning target directly from Equation 15
(s) (D) Eg
t R(s,, ar)
3n Gz (S
vs, E E H.
(16)
Bootstrapped via past estimations
Estimated via MC Sampling
Now, we turn our attention back to our Variational Continual Learning setting. The standard VCL. objective is given by Equation! 2
17

---

1
gr( arg min ( || 7 0))
rE
We can similarly define a learning target as "value" function which we aim to maximize:
(t( -KL((((0) ||
Ze ( 0))
(0) logp(D, [0]] + log Z4
2kz(s:()) (()
1
(0)) logp(D; 0)] + log Ze
2kz(q:(0) Ze-1 (0-2(0)(DD,1110
(0)
logp(D; (D [)] + log Ze
+ 1g(0)) -
n-2
(0))
Σ log Dogp(Drr) (D. 0)]
Σ
log Zr-i
ul n 1),n No,ns S.
(17)
Similarly to the RL case, it follows that q: (0) arg max, gEO (o(() (t). Lastly, we assume the following estimation of the "value" function defined Equation 17:
(t)o)(t) (t) (0)
log 0)] + Σ
log Ze-s
+ (t ++11
km0
R-1
(0)
log 0)
-t- (
log Zr-
Bootstrapped via past posterior estimations
Estimated via MC Sampling
Constant wurt
(18)
We notice that Ze is constant with respect to 0, hence we can disregard it and still have the same learning target. Thus, we have:
(0) arg max (,(0) (t) gEQ
arg max E (0) q
log (D.i 0))
(k.(a((0) || (+)) +
log 7
1=0
arg max Eg (0)
log (D. 0)]
9KL(q((0) ()))
(19)
gEQ
1=0
TD)
Equation 19 is exactly n-Step Temporal-Differenee target in Definition 4.3 from Section The main differences from the CL recursion in Equation 17 and the RL one in Equation 15 are two-fold. First, the CL setup is not discounted (or, equivalently, assumes the discount factor RL recursion looks over future timesteps, while the CL one looks over past timesteps. Besides these two differences. both scenarios are strongly connected, Particularly, they share the same purpose for leveraging TD targets: to strike balance between MC estimation (which incurs variance) and bootstrapping (which incurs bias) while estimating the learning objective.
1). Second, the
18

---

D TD(A)-VCL is a discounted sum of n-Step TD targets
In Section 4 mention that the TD-VCL learning target is compound update that averages n-step semporal-differees targets, as per Proposition [444] which we prove below.
Proposition 4.4. Vn No, I the objective in Equation 2 can be equivalently represented
g:(0) arg max (D)
QEQ Furthermore, the objective in Equation 5 can also be represented
with (n) as in Definition 4.3 as:
À
n-1 Σ k=
TD 1))
%e (+) = arg max
(8)
qEQ
1 - X"
Discounted targets
Proof. We start by proving the equivalence between Equation and Equation 7
(0) = arg min 2 (z((a(0)
Zr. qt-
(0)((()
TE
arg min 2K (k)
q0- (
II
p(Do-e 0))
QEQ
-i=0 Ze-i
1=0
(20)
arg max Eg (0)
log Dogp(DD.1) 0)]
arg max (n).
Q
Now, we show that Equation 5 is discounted sum of n-Step targets:
1
4: (0) arg max. QEQ
Es n (0) ((D () [ q-)))())))
1 X
+ AE (0) (og 0) + logp(D,-1
0)) - AZkI(q || a: 2()) +
+
E Ea a~ (0)
logp(D,
0)] - x" xxx q: (0))
A
TD,(1) + XTD, + +-.. (D,(m)
arg max TE
1 X
1
TD, 1))
arg
1 A
VEQ
A=
Disconted sum of 'TD targets
(21)
In Equation if we set n 1, the n-Step TD target recovers the VCL objective. Furthermore, is worth highlighting that an n-Step TD target is not the same as n-Step KL Regularization. The latter leverages several previous posterior estimates, while the former only relies on a single estimate. Lastly, we can follow similar idea to prove that the n-Step KI. Regularization objective is simple average of n-step TD targets, by leveraging the expansion in Equation and identifying the sum of TD targets.
19

---

E TD-VCL: A spectrum of Continual Learning algorithms
In this Section, we describe how TD-VCL spans a spectrum of algorithms that mix different levels of Monte Carlo approximation for expected log-likelihood and KL regularization. Our goal is to show that by choosing specific hyperparameters for Equation 5 one may recover vanilla VCL in one extreme and n-Step KL regularization in the opposite.
Let us consider the TD-VCL objective in Equation 5
arg max Eg (g. (0) q
X 1
log Dogp(Di-11 0)]
1(- Σ == X-
9 ( || 9-s-1(0)).
1 A
2
Trivially, if we set X O, assuming 0 1, it recovers the Vanilla VCL objective, stated in Equation regardless of the choice n.
More interestingly, we investigate the learning target as + 1:
n
n- -1
lim X-1
())
log logp(Dr-i|00 -
[kI( (1
1 A
1 1-
1=0
1¹(1 - \n-1)
n-1
111 A)
(0)a(())
lim
log logp(D...1 [0)
lim
9xL(g((0) (1-
1- \
1- A
(I)
(II)
Let us develop (I) and (II) separately by applying the L'Hôpitalls rule. First, for (I):
{ X(1 2
(it-
i
(1-
\n
2i(n 8)2
lim x+1 Am
= lim X+1
-nan-1 (n 11)1-1
(22)
lim X-1
-ndn-1
n
Now, for (II):
(² - A)
i fi A) X2 1
lim
lim
(23)
1 xa
n
Applying Equations
and 23 TD-VCL objective, obtain:
nn i) log (Dg 0)] - Σ
arg max E.
Eg
2 kt (0) (1---((())))
1=
1=0
which is exactly the N-Step KI. Regularization objective.
20

---

F
Impact Statement
This work develops a novel learning objective for Bayesian Continual Learning. As such, we believe our work has a positive impact on fundamental research for Machine Learning for three reasons. First, we argue that advancing Continual Learning research is crucial for ensuring the long-term quality of ML models in production systems, as they are vulnerable to potential distributional shifts in the data generation distribution. We also argue that CL is crucial for developing safe autonomous learning agents, Catastrophic Forgetting may be dangerous challenge while interacting with the physical or digital world. Second, our particular focus on the Bayesian framework is relevant for designing uncertainty aware models, which, argued in Section is crucial for robust Machine Learning and general Al safety, Lastly, we provide a solid thcoretical connection between Variational Continual Learning methods and Camporal-Different methods, effectively bridging two seemingly distant disciplines into a unified family of algorithms. We believe this will inspire further research in the intersection of both areas.
G Implementation Details and Reproducibility
Operationalization. For all experiments, we use Gaussian mean-field approximate posterior and assume Gaussian prior A(0, ((00,²1) for the variational methods. We parameterize all distributions as deep networks. For all considered objectives, we compute the KL term analytically and employ the Monte Carlo approximations for the expected log- lilkiiihhood terms, leveraging the reparametrization trick [47] for computing gradients. Lastly, we employ Akelihood-tempering [33 to prevent variational over- over-pruning [88..
Model Architecture and Hyperpatameters. We adopt fully connected neural networks for PermutedMNNIST-Hacd SplitMNIST-Hard and SpliNNotMNIST-Hrrd We choose different depths and sizes depending on the benchmark, and we provide a full list of hyperparameters in Appendix H For CIFARIO0-1 and TinyImageNet-10, we implement Bayesian version of the AlexNet [66. a traditional convolutional neural network architecture, as in prior Bayesian CL literature [39]. Crucially, also following prior literature 33. we do not use pre-trained representations, as our goal is to evaluate how the proposed objectives perform in the CL setting, which also requires learning their own robust representations. Finally, for training, we adopt the Adam optimizer [57] and employ early stopping with a patience parameter of five epochs, which drastically reduces the number of epochs needed for each new task in comparison to previous work [3]
Hyperparamter Tuning Protocol. We conduct hyperparameter tuning for all methods in the paper, including the baselines (VCL, UCL, UCB). We follow random search for each evaluated benchmark. For fair comparison, we ensure that all methods use approximately the same compute of GPU day. We provide the search space for each method in our released code. For the proposed methods, we mainly tuned three hyperparameters: n (as in n-Step KL), À (as in TD-VCL), and B (the likelihood tempering parameter). We conducted grid search for each evaluated benchmark. with E {1, 2, 3, 5, 8, 10}. À E {0.0, 0.1, 0.5, 0.8, 0.9, 0.99}, and | E {1e- - 5, le 4, 1e 3, 5e le 2, 5e 1e 1.0).
Reproducibility. Reported results are averaged across ten different seeds for PermutedMNNIST-Hard SpliMMNIST-Hard, and SplitNotMMNIST-Hard and five seeds for CIFAR100-11 and TinylmageNet- 10. Error bars represent 95% confidence intervals, while tables show 2-sigma errors up to two decimal places. We execute all experiments using a single GPU RTX 4090. We provide our implementation code for the proposed methods (TD-VCL, TD-UCB, TD-UCL, and n-Step), as well as considered baselines (Batch MLE, Online MLE, VCL, VCL CoreSet, UCB, and UCL) in https:/ / /i thub. wom/luckooiano//DDooo.
21

---

H Hyperparameters
Table provides the shared hyperparameters used in each benchmark. Tables Ind provided the specific hyperparameters for the proposed methods and baselines, respectively.
PermMNISTT-aard
SpliMMNIST-Hard
SpliNNotMNIST-Har
CIFAR100-10
TinyImageNet-10
Size Max Epochs
256
100
100
100
100
100
NN Architecture
[100, 100)
[256, 256]
[150, 150. 150, 150]
AlexNet
AlexNet
Number Heads
Learning Rate
le-3
le-3
le-3
le:3
le-3
Table 4: Training hyperparameters. These are shared across all evaluated methods.
PermMNIST-Hard
SpliMMNIST-Hard
SpiiNotMNNIS-HHad
CIFAR100-10
TinyImageNet-10
5
4
5
n-Step KI.
Se-3
5e-2
5e-2
3e-5
8
4
3
10
TDQA-VCL
0.5
0.8
0.1
0.1
Ie-3
5e-2
le-3
le-5
lo-9
8
4
TDQA-UCL
0.5
0.8
0.1
le:3
Se:2
1e:3
le:5
1e-7
8
TD(A-UCB
0.5
0.8
0.1
0.8
0.1
le:3
5e-2
1e-3
le-5
Table 5: Hyperparameters for different methods across benchmarks.
PermMNIST-HHrr
SplitMNIST-Hard
SplitNotMNIST-Hard
CIFARI00-10
TinyImageNet-I0
5e-3
5e-3
5e4
le-5
1.0
10.0
0.5
1.0
10.0
8
0.001
1.0
0.001
0.001
1.0
UCL
7
1.0
1.0
0.005
0.1
0.5
0.5
0.5
0.5
Se-3
le-3
le-5
lc-4
1c-7
UCB
a
1.0
1.0
0.1
10.0
100.0
B
le-2
le-2
Se-2
5e-5
le-5
Table Hyperparameters for different methods across benchmarks.
22

---

I PermuteMMNIST-Hard, SplitMNNIST-Hrdd, and SplitNotMNIST-Hardd Introducing Higher Standards for MNISTNNO Continual Learning Benchmarks
Popular Continual Learning benchmarks, such as PermutedMNIST, SplitMNIST and SplitNoIMNIST,
[ 20 13 provide an effective experimental setup. These benchmarks offer tasks that, while conceptually simple in isolation, present a challenging task-streaming setup that highlights the phenomenon of Catastrophic Forgetting. This combination facilitates the study of Continual Learning methods through rapid iterations and modest deep architectures, making it ideal for academic settings. Nonetheless, argue that the "unrestricted" versions of these benchmarks addressed by simple baselines or do not reflect a challenging evaluation setup for Catastrophic Forgetting in current Bayesian CL research. This observation motivates our work to incorporate certain restrictions in the considered methods, resulting in a more challenging setup for Continual Learning while maintaining the benchmarks' original desiderata.
either trivially
PermutedMNIST: Replay Buffer Analysis
1.0
1.0
1.0
0.9
0.9
0.9
0.8
0.8
Accuracy
0.7
0.7
0.7
0.6
0.6
0.6
10. 50300
50200
80000
30300
10200
10000
10, 1030
0.5
5-5,0 1024
T- 10. 200
5.8 200
2. 200
0.4
T 10. 50
0.4
5.8
50
0.4
2
1 10, 8 6-10
5. 10
T B 10
Online MLT
Online MLE
Dnlnn MLE
0.3
0.3
0.3
3
4
6
9 Number Observed Tasks
5
Figure 4: A Replay Buffer analysis the PermutedMNIST. Each curve represents model re-trained on a buffer composed of "T"" previous tasks, "B" examples of each. Online MLE only considers the current task. Allowing "unlimited" access to previous task data trivializes the CL setting, and a simple MLE baseline is enough to attain strong results. Nevertheless, as we restrict the replay buffer in size and number of tasks, the benchmark becomes substantially more challenging and shows signs of Catastrophic Forgetting.
Restricting replay memory size imposes a new challenge for MNIST/NOtMNIST CL bench- marks. Figure presents MLE models trained on different levels previous tasks' data (besides the data from the current task) for the classic PermutedMNIST benchmark. Online MLE means no usage of data from previous tasks. On the flip side, we re-train the remaining models considering the data of T' previous tasks, with B examples of each. It shows that allowing access to all the old tasks is enough for an MLE model to maintain high accuracy even when presenting to only set as tiny as 200 examples. As we reduce the number of old tasks in the buffer, performance decreases, showing clear signs of Catastrophic Forgetting. For T 2, all models present an accuracy lower than 60% regardless of the volume of old task data. Therefore, in order to impose harder evaluation setup, we impose additional restrictions for re-training in prior tasks. For PermutedMNIST--Hark we restrict re-training to the two most recent past tasks, with 200 examples per task; for SplitMNISTT-ard and SpliNNotMNIST-Harr we allow only the most recent past task with 40 examples. As shown in Figure 4 MLE-based methods do not perform well in this setting. Crucially, these adopted replay buffers are very small in comparison with the training data the current task, which is more realistic than retaining the full data. Nonetheless, they strictly follow the core set sizes used in prior work 13.. ensuring that the adopted baselines (e.g., VCL CoreSet) work as proposed and promoting a fair comparison.
"Single-Head" Classifiers prevents the saturation of PermutedMNIST, SplitMNIST, and Split- NotMNIST. "Multi-Head" networks train different classifier for each task on top of a shared backbone. The goal is alleviate Catastrophic Forgetting by disregarding the effect of negative transfer among tasks. While this may be acceptable for harder datasets where multi-head architec-
23

---

ture is necessary to avoid trivial performance, current methods with multi-head classifiers already saturates the classic MNIST/NotMMNIST benchmarks, achieving accuracy above 99%. For empirical evidence, we evaluate the methods on SplitMNIST (which allows multi-head architecture, Figure 5) and SplitM (which restricts to a single-head classifier, Figure @ Appendix (K). In former, all baselines trivially attain high average accuracy; in the latter, all methods face much more challenging setup. Hence, PermutedMNNIST-Hard, SplitMNIST-Harr and SpliNNotMNIST-Harr enforces single-head architecture.
SplitMNIST: Per Task Performance
Task '0/1'
Task 22/3'
Task '4/5'
1.0
1.0
1.0
Accuracy
0.9
0.9
0.9
0.8
0.8
0.8
0.7
0.7
0.7
0.6
0.6
0.6
2
5 Number of Observed Tasks
3
5
Task '6/7'
Task '8/9'
Average
1.0
1.0
1.0
Methad
Accuracy
0.9
0.9
Onlin MLE
0.9
Batch MLE
0.8
VCL CoreSet
0.7
0.7
N-Stup TD-VCL
0.7
TDIM VOL
1
2
4
5
2
3
5
Number Observed Tasks
Figure 5: SplitMNIST results. The first five plots show results per task, and the last one is average across tasks. As a consequence of multi-heas networks simplifying the Continual Learning challenge, all methods attain high accuracy. In particular, variational methods accuracies ranging from 97% and 98%. In constrast, SplitMNiST-Hard Figure 6 provides considerably more challenging benchmark.
Lastly, we highlight that all evaluated methods including the proposed ones are subject to the adopted restrictions highlighted in this Section. Therefore, they are trained in the same data with the same parametrization, ensuring fair comparison setup.
24

---

Benchmarks Description
PermutedMMIIST-Hardd. This benchmark uses the MNIST dataset. Each task corresponds different permutation of the pixels in the MNIST data. Similarly to MNIST, PermutedMNIST is multi-class classification problem recognize the handwritten digit associated with the image. The benchmark runs 10 successive tasks, and each evaluation iteration considers the performance in all past tasks. For the "Hard" version, we restrict any method two ways, as described in Appendix first, replay buffers are restricted to the two most recent tasks, with a fixed set of 200 data points per tusk; second, we restrict the model architectures to single-head classifiers.
SplitMNISTTHard.
This benchmark also considers the MNIST dataset but in a binary classification setting. The model selects between two different digits. Five tasks from the MNIST dataset arrive in sequence: 0/1, 2/3, 4/5, 6/7, and 8/9, and evaluation considers the performance in all past tasks. For the "Hard" version, we apply the similar restrictions: replay buffers restricted to the most recent task, with fixed set of 40 data points. We also restrict the model architectures to single-head classifiers.
SplitNotMNNIST-aree This benchmark contains similar structure to SplitMNIST-Haiie but it leverages the notMNIST dataset. This more challenging task contains characters from diverse font styles, comprising 400,000 examples. The five tasks are A/F, B/G, C/H, D/I, and E/J. The "Hard" version applies the same restrictions as SplitMNISTTHard.
CIFAR100-10. This challenging benchmark contains 10 different tasks, each of them comprising 20 distinct classes from the CIFAR-IO0 dataset [5I]. Evaluation considers the performance in all previous tasks. The dataset contains 50,000 images (5,000 per task) for snining/validation and 10,000 images (1,000 per task) for evaluation. For this benchmark, we restrict the replay buffer to contain 200 data points per task.
Tiny Ieageeeeeet-WW 10. This challenging benchmark also contains 10 different tasks, each of them comprising 20 distinct classes from the ImageNet dataset [58). The dataset contains 100,000 images (10,000 per task) for training/validation and 10,000 images (1,000 per task) for evaluation. Particularly for Tiny ImageNet- 10, also adopt memory restriction: replay buffers are restricted to the three most recent tasks, with a fixed set of 200 data points per task.
25

---

K
Per Task Performance:
Additional Results
K.1
SplitMNIST-Hard
Figure presents the per-task performance for the SplitMNISTTHard results. As expected, the performance of all methods drops substantially in comparison to traditional SplitMNIST, as the CL becomes considerably harder. However, highlight that n-Step KL and TD- ID-VCL presented better results than VCL and VCL CoreSet, demonstrating again the effectiveness of the proposed learning objectives.
Interestingly; the average accuracy does not decrease monotonically, as one might typically expect due to Catastrophic Forgetting. Instead, it drops significantly after Task 3 and then rises again. This evidence indicates two potential dynamics of transfer learning: a negative transfer from Task while learning Task 3, and positive transfer from Task I while learning Task 4. For instance, the digit "O" from Task 1 is rounded, similar to the digits "5" and "6" in Tasks 3 and 4, respectively. Additionally, the digit "1" is composed of straight lines, much like the digits "4" and "7." We believe that the employed architecture, given its inherent and intended simplicity. relies on features of this nature. expressive architectures that better disentangle these features may potentially prevent the negative transfer. However, exploring this possibility is beyond our scope, as our focus is on studying the effects of Catastrophic Forgetting in Continual Learning.
Therefore,
SplitMNIST-Hard: Per Task Performance
Task '0/1'
Task '2/3'
Task '4/5'
1.0
1.0
1.0
Accuracy
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
1
2
3
4
5
2
2
3
5
Number Observed Tasks
Task '6/7'
Task '8/9'
Average
1.0
1.0
Accuracy
0.8
0.8
Online MLE
Mathod
0.95
0.6
0.6
Batch
0.4
0.4
VCL CorcSet N Stop TD VCL
TRIAIVEL
0.55
1
2
3
4
5
Number Observed Tasks
Figure 6: SplitMNNIST-Hard results. In this robust evaluation setting, tasks enforced to share a single classifier with restricted replay memory. Consequently, the effect Catastrophic Forgetting (and task negative transfer) is explicit. TD-VCL objectives present slightly better average accuracy across tasks in comparison with standard VCL variants.
K.2 SplitNotMNIST-Hard
In this section, we show per-task performance for SplitNotMNISST-Hard As highlighted in Section 5.1 NotMNIST is a considerably harder dataset than MNIST, and the choice simpler deep architectures naturally results in higher approximation errors. Our goal is to evaluate how the presented methods behave under this cireumstance.
Figure 7 presents the results. As expected, even learning the current task is challenging. This characteristic contrasts with MNIST-based benchmarks, where all models could at least fit the current task almost perfectly. MLE methods fit the current task slightly better since their objectives are not regularized by prior or previous posterior. However, this same reason caused them to suffer from Catastrophic Forgetting more drastically, as they tend to focus fitting the current task and disregard past ones. Overall, TD-VCL objectives maintained the best trade-off between plasticity and memory stability, aligning with the results in the other benchmarks.
26

---

SplitNotMNIST-Hrrd: Per Task Performance
Task A/F"
Task 'B/G'
Task 'C/H'
1.0
0.9
0.9
0.9
Aecurao 8
0.8
0.8
0.8
0.7
0.6
0.6
0.6
0.5
0.4
0.4
0.4
Number of Observed Tasks
Task 'D/I'
Task 'E/J'
Average
1.0
Accuraty rcurabo
0.9
0.9
0.8
0.8
0.7
0.7
0.7
Batch MLE
0.6
0.6
VCL
CoreSet
0.6
0.5
0.5 N-Stup
TDIAI VCL
0.5
5
3
Number of Observed Tasks
Figure 7: SpliNNotMNIST-Hare results. The first five plots show results per task, and the last one average across them. SpliNNotMNIST-Hare is considerably harder to fit with modest deep architectures, leading to a setup where posteriors induce high approximation errors. As a result, the standard VCL variants performs similarly non-variational approaches. TD-VCL surpasses all methods and shows more robustness to Catastrophic Forgetting under this high approximation error setting.
K.3 CIFAR100-10
Figure displays the per-task performance the CIFAR100-10 benchmark. Non-variational baselines consistently struggle with Catastrophic Forgetting, even in more recent tasks. VCL and VCL CoreSet also show a consistent drop in accuracy as the number of observed tasks increases, although this decline is less noticeable in some certain tasks. In contrast, the proposed TD-VCL objectives demonstrate a significant improvement the baselines and show little indication of Catastrophic Forgetting, despite the harder challenge posed by the CIFAR 100 dataset.
and occasionally followed by slight increase in accuracy for
Interestingly, variational methods, which experience less Catastrophic Forgetting, exhibit surprising effect in some tasks: their accuracy initially drops after observing a few consecutive tasks before subsequently increasing again. For example, in Task 3, this effect is evident across all variational methods. As a result, the average accuracy tends to rise as the total number observed tasks increases, which is also reported in prior work (see Figure 7a in Ahn et al. 43), and Table 2 in Thapa and Li (39) We hypothesize that the process of explicit posterior regularization, combined with training successive tasks, leads parameterization that learns features more generalizable across tasks, incurring positive transfer learning.
K.4 Tiny TinylmageNet-100
Lastly, Figure iillsstrtte the per-task performance in the TinyImageNet- 10 benchmark. As seen in previous scenarios, Online MLE consistently fails to achieve continual learning. Interestingly, VCL also encounters difficulties in this more challenging benchmark, showing per-task performance similar to Batch MLE. VCL CoreSet outperforms the standard VCL and achieves performance comparable to the TD-VCL objectives in tasks. Nevertheless, the TD-VCL objectives consistently demonstrate superior performance across all tasks, reinforcing the findings from the earlier benchmarks.
27

---

CIFAR100-10: Per Task Performance
Task
0.75
0.75
0.75
0.75
0.70
0.70
Accuracy
0.65
0.65
0.65
0.65
0.65
0.55
0.55
0.55
0.55
0.55
0.50
2
Task
Task
Task
Task
Task
0.75
0.75
0.75
0.75
0.75
accuracy
Method
0.65
0.65
0.65
0.65
0.65
Online MIT
Basch MLE
VIL
VEL CoreSet
0.55
0.55
0.55
0.55
0.55
Nstwa TDANCI
TDUX-VL
0.50
Figure 8: Per-task performance (accuracy) over time in the CIFAR100-10 benchmark. Each plot illustrates the accuracy of a specific task (as indicated in the title) the number of observed tasks increases. Non-variationa baselines consistently struggle with catastrophic forgetting, while VCL and VCL CoreSet show mild effect. However, the TD-VCL objectives demonstrate noticeable improvement over these methods, even the more challenging setup.
TinylmageNet-10: Task Performance
Task
Task
Task
Task
Task
0.65
0.65
0.85
0.65
0.65
Accuracy
0.55
0.55
0.55
0.55
0.55
0.45
0.45
0.45
0.40
3
4
7
919
0.40
2.3
4
5
910
0.40
1 5
910
0.40
2
7 910
0.40
2.3 8.84567799988 910
Number of Observed Tasks
Task
Task
Task
Task
Task
0.65
0.65
0.65
0.65
0.65
Accuracy
0.55
0.55
0.55
0.55
0.55
Method
0.45
0.45
0.45
TOOD-KL
0.40
3
4
6
9.10
0.40
4
0.40 Number of Observed Tasks
4 910
0.40
1
4
0.40
2 6 910
Figure 9: Per-task performance time in the TinyImageNet-10 benchmark.. In the most challenging benchmark presented in this work, we observe similar trends to the previous ones, where TD-VCL objectives show superior performance across tasks.
28

---

L Hyperparameters Robustness Analysis
In this Section, we present robustness studies in the PermutedMNIST-Haute benchmark with respect the relevant hyperparameters. Our goal is to evaluate how they affect the performance of the proposed methods.
L.1 n-Step KL Regularization
Figure presents the ablation study of the n-step KI. Regularization method in the PermutedMNNSST. Hard benchmark. We designed this study highlight the two most sensitive hyperparameters: n, the n-Step size, and 3, the ceelihood-temperiigg parameter.
Similarly VCL, this method is sensitive to the choice of B. Higher values will prevent the model from fitting new tasks, a manifestation of variational over weer-pruning. On the other hand, lower values will not retain knowledge properly, suffering from Catastrophic Forgetting. Mild values (0.001, 0.005, 0.01) balanced well this trade-off.
In terms of we observe benefits of up to steps. Beyond that, the effect saturates, even becoming slightly detrimental. This observation suggests the existence of an optimal range for n while leveraging past posterior estimates.
PermutedMNIST-Hard. N-Step TD-VCL Ablation
n-Step
n-Step =
n-Step
AccuraCy
le
0.001
0.5
0.5
1
2
3
4
5
6
8
9
10
1
2
4
6
7
10
2
3
5
7
8
10
Number of Observed Tasks
n-Step
n-Step =
n-Step = 10
1.0
1.0
1.0
Accuracy
0.9
0.9
0.9
0.8
0.8
0.8
0.7
0.7
0.7
0.6
0.6
0.6
0.5
0.5
0.5
Number of Observed Tasks
Figure 10: Hyperparameter Robustness Analysis for n-Step KL Regularization in PermuteMMNIST-HHree The plots show the effect the likelihood-eeppering parameter 3 for different For 3, too high values negatively affect fitting tasks, and too low values disregard the regularization previous posteriors, leading to Catastrophic Forgetting. For n, we observe benefits while increasing up 5, and the effect saturates.
L.2 TD(A)-VCL
Figure II shows the ablation study for TD-VCL. For this setup, considered fixed value of as our hyperparameter search suggested the same trends for n-Step KL Regularization and TD-VCL. Hence, we simplify the analysis consider only n and
TD-VCL presents mild sensitivity to the choice of The effect is more pronounced as the method observes more tasks, with slight preference for lower values for some choices of We believe that the choice of À will fundamentally depend on how most recent estimates are better and more informative than old ones. In the where they present similar approximation errors, the choice of À causes less impact, and, therefore, there is less difference between leveraging N-Step TD-VCL and TD(A)-VLL objectives.
29

---

PermutedMMIST-Hardd TD(X)-VCL Ablation
n-Step: 1
n-Step:
n-Step: 3
1.00
1.00
1.00
Accuracy
0.95
0.95
0.95
0.90
0.90
0.85
A 0.5
0.85
0.85
3-
0.80
1- 0.
0.80
0.80
A 0.99
0.75
0.75
0.75
Number of Observed Tasks
n-Step: 5
n-Step:
n-Step: 10
1.00
1.00
1.00
Accuracy
0.90
0.90
0.90
0.85
0.85
0.85
0.80
0.75
0.75
0.75
1
3
7
10
2
3 4
6
8
10
1
6
8
9
10
Number of Observed Tasks
Figure 11: Hyperparameter Robustness Analysis for TD())-VCL in Permuted.MNISST-Hard The plots show the effect of for different choices of n. The learning objective presents mild sensitivity to the choice of À in this benchmark, and the effect more pronounced as the number of observed tasks increases.
30

---

M Full Table Results
In this Appendix, we report the full version Tables and for the sake of completeness. Table 7 shows the results on (IFAR100-1T 10 and TinylmageNet-10 10, considering all timesteps from t = 2 10. Table & shows the results for all benchmarks, including SpliNNotMNIST- for the Bayesian CL methods and their TD-enhanced counterparts.
Table 7: Full table for quantitative comparison the CIFARI00-10 and Tiny Imagenet- benchmarks. Each column presents the average accuracy across the past observed tasks. Results are reported with two standard deviations across five seeds. TD-VCL DCL variants consistently outperform the baselines in harder benchmarks with more complex architectures, such as Bayesian CNNs.
CIFAR100-10
2
t= 3
t= 4
==
==
t=
10
Online MLE
0.5620.05
0.5620.06
0.57a0.06
0.5620.04
0.5620.03
0.5520.03
0.53g0.06
2..04
0.5220.04
Batch MLE
0.5720.03
0.58a0.04
0.58a0.04
0.59
0.58a0.05
0.58g0.06
0.5620006
0.54
0.54
VCL
0.64
0.63s0.03
0.63a0.02
0.60
0.60a0.02
0.60=0.03
0.61
0.65
0.66+0.01
VCL CoreSet
0.64s0.05
0.6520.03
0.63an.03
0.62a0.03
0.63a0.2
0.63
0.61
0.64
0.65
n-Step TD-VCL
0.67
0.6820.01
0.67
0.67
0.65a0.01
0.6620.01
0.6820.04
0.69
0.69
TD(A)-CL
0.66:0.02
0.67#0.02
0.66#0.04
0.66#0.01
0.66:0.02
0.66=0.01
0.67
0.69 0.71
Tiny 11ngenagenet-10
t= 2
t= 3
t= 4
5=
1
t=
[= 8
t= 9
[= 10
Online MLE
0.48a0.03
0.45a0.02
0.45a002
0.46s0.02
0.44a0.1
0.44
0.45a0.02 0.45a0.02 0.44
Batch MLE
0.50=0.02
0.47+0.02
0.48s0.02
0.49=0.02
0,48+0.02
0.48:0002
0,50+0.02 0.50+0.02 0.51
VCL
0.53+0.06
0.5l 0.50+0.02
0.52+0.02
0..0
0.49=0.01
0.51 0.51+000
0.51+0.02
VCL CoreSet
0.52+0.03
0.50+0.02
0.51 0.53+0.01
0.51 +0.02 0.52=0.01
0.54+0.22 0.55+0.02 0.54+0.02
n-Step TD-VCL
0.5620.02
0.5420.03
0.55a0.02
0.5520.02 0.5420.02
0.5420.01
0.5620.2
0.5620.01 0.5620.02
TD(A)-VLL
0.57
0.5520.02
0.56=0.02
0.5620.01
0.55.0.03
0.55=0.03
0.5620.02
0.57.0.02 0.56:0.02
31

---

Table 8: Full table for quantitative comparison between Bayesian CL methods and their TD- enhanced counterparts. The TD-enhanced methods incorporate the objective in Equation S in each base method. Although no single base method consistently outperforms the others across all benchmarks, their TD-enhanced versions consistently achieve better performance, particularly at later timesteps.
PermutedMNIST-HHrve
t =
t=3
t= 4
5=5
t=
t=
t=
t=
t= 10
VCL
0,9520.00
0,9420.01
0,93w0.02
0,9
0.8920.03
0.8620.05
0.83.0.44
0 0.80a0.06
0.7820.04
TD()-VCL
0.97=0.00
0.96:0.00
0.95.0.00
0.94:0.01
0.93+0.01
0.92x0.01
0.91
0.90
0.89=0.02
UCL
0.97
0.95.0.0f
0.94.0.01
0.92=0.02
0.39.
0.80.
0..3
0.7 8...09
0.73
TD(A-UCL
0.97.0.00
0.9720.00
0.95.0.00
0.94
0.92+0.02
0.90
0.88
0.85.0.09
0.84
UCB
0.93+0.01
0,93+0.01
0.92+0.01
0.89 0.90=0.01
0.87.0.02
0.86s0.02
0.85
0.83
TD(A)-UB
0.94
0.93s0.00
0.93a0.00
0.92=0.00 +.910
0.91 0.90=a.01
0.89+0.02 0.88=0.02
SplitMNIST-Hard
SpliNooMMNIST-Hard
1
3=3
1=4
1=5
t 2
t=3
t=4
5=
VCL
0,87a0.02
0.66:0.04
0.82+0.03
0.64=0.11
0.69#0.04
0.63s0.03
0.60a0.00
0.51=0.06
TD()-VCL
0.98
0.79#0.08
0.88
0.67=0.04
0.74+0.02
0.73+0.03
0.69#0.03
0.58=0.09
UCL
0.66.5.0
0.8 0.830.
0.6600.06
0.71
0..320.04
0.6
0.5220.04
TDAA-UUL
2.97
0.05
0.9020.02
0.70=0.04
0.7220.03
0.71
0.63.0.02
UCB
20.1
0.79
0.8 833000 0,77500.0
0.70zo.u8
0.6320.06
0.6 [ao.o
0.6 1 20.05
TD(A)-UCB
0.93a0.02 0.89 0.87 0.80aa.03
0.72.0.01
0.72.0.01 0.70.0.02 0.63
CIFAR100-10
t= 2
t 3
t= 4
5=
t=
t=
t= 8
t= 9
t= 10
VCL
0.64
0,63 0,63a0.02
0,60zo.2
0.60ao.0z
0.60au.us
0,61 20.05 0.65.0.02 0.6620.01
TD()--CIL
0.66a0.02
0.6720.02
0.6630.04
0.66:0.01
0.6620.02
0.66.0.01
0.67
0.69.0.02 0.71
UCT.
0.63
0.66.5.07 0.645o.05
0..2.5..4
0.60.a.n5
0.60
0.58
0.6 Ta.02
0.62
TD(A)-UC
0.6820.02
0.67
0.6420.01
0.70=0.04
0.7020.02
0.6820.03
0.6622.03
0.6520.06
0.67
UCB
0.65
0.65.0.02
0.6620.02
0.66=0.03
0.6620.03
0.66 0.66#0.01
0.65s0.01
0.64 0.66g0.01
TD(A)-UBB
0.64
0.65.0.02 0.6620.01 0.67
0.682a.01 0.67
0.6820.01 0.6820.02 0.70
Tiny Sinylmagenet-10
t ==
t= 4
t=
t=
1=7
t=
t
t= 10
VCL
0.53+0.06
0.50eo002
0.5 10.00
0.52=0.02
0.51
0.49=0.01
0.5
0,5 00002
0.5 1=0.02
TD()-VCL
0.57
0.55
0.5620.02
0.56.0001
0.55.0.03
0.55
0.56...0z
0.57 0,5620022
UCLL"
0..3..000
0.32.5.33
0.52aa.03
0..2.
03 Too.oz
0.5 0.50
0.55 0.o0
0.3
0. 0.50
TD(A)UCL
0.55
0.53
0.54
0.55
0.54
0.54
0.55+0.01
0.56=0.01
0.56
UCB
0.552+.06
0.00+
0.51
0..00
0. 064
0.
0242
TD(A)-UCB
0.54+0.04
0.54.0.01
0.52#a.01
0.52+a.02
0.51
0.50 0.50+a.03
0.49.0.02
0.47
32

---

N Does TD-VCL Assume Knowledge of Task Boundaries?
In this Section, we argue that the TD-VCL objective (and VCL objectives in general) does not require knowledge of task boundaries, and we provide theoretical and empirical evidence for that. The theoretical argument comes from the principle that the Bayesian framework self-consistent: given stream of data, the final posterior distribution should be the same regardless of how many Bayesian updates are executed.
Based on that, the key thing is to realize that the number of updates does not need to be equal to the number of tasks. Mathematically, suppose have stream of ' tasks (represented by (). At a particular update k. we may consider a Bayesian update that includes data from multiple (m) sequential tasks (e.g., from to ta+m):
D2 U Dj-
(24)
f=0
Crucially, this does not impose any assumptions boundaries. Rather, once decide where to start and end the data stream for the Bayesian update, there could be potentially many tasks included. Under the same assumptions stated in Section we have that:
(DD 0)
II 1=
PD, 0)
(25)
And, the recursive relationship (Equation 1 also follows:
p( D1.)) x p(0 D14-11 II P(Dx ()
(26)
1=0
Finally, following the same variational objective and ELBO derivation, arrive
c²(00= Esngu (0)
Σ log (Dx 0)
DxL (gx (0) || 11-z)(0)))
(27)
Therefore, the objective itself does not discriminate or require task boundaries, TD-VCL will estimate the likelihood terms for multiple terms simultaneously, which is something already done while replaying past tasks.
We highlight that most benchmarks - including the ones presented in this work isolate tasks, which makes convenient to consider one Bayesian up- date per task. To provide further practical evidence that the method does not require knowledge of boundaries, we present another benchmark called StreaninggerrmutteMNNSss Hard. This benchmark does not provide any boundary between tasks. From the full data stream of T tasks, we create sequential streams of data where boundaries are placed randomly and provide them to the methods for contin- ual learning. We execute an evaluation after the complete data stream, considering held-out splits composed of all tasks. We report the av- erage accuracy across them, equivalently to the t 10 column in Tables and Table shows empirical results over 10 seeds. We observe no negative impact in the VCL/TD-VCL methods In fact, some methods improved performance, because we are likely replaying the same task into different chunks, alleviating the catastrophic forgetting challenge. The proposed objectives still outperform all other methods.
Empirical Evidence.
Table 9: StreamingPermuiccNNNSS---H re- sults. We observe negative impact in the TD-VCL methods in comparison with PermuteMMNIST-Hard, suggesting that these meth- ods do not require knowledge of task boundaries.
StreaningerrmutddNNsSS-Haaddd
Method
( 10
Online MLE
0.54+0.09
Batch MILE
0.64++.09
VCI.
0)82+00.5
VCL CoreSet
0,85
N-Step TD-VCL
0.89+0.02
TD(A)-VL
0.89+0.02
in comparison with PermutedMNIST-HE
33

---

Further Questions
This Appendix presents additional clarification questions aimed at improving the understanding of the proposed method and experiments. These questions were raised during the peer-review process, and we refer to the OpenReview page for the full discussion.
0.1 What is the computational cost associated with TD-VCL?
We analyze the computational cost from three aspects: training, inference, and hyperparameter search.
Training Cost. The training cost arises from the computation of Equations 4 and 5 which depend on two components: (I) Monte Carlo estimation of the likelihood term, and (II) the KL regularization term. (I) corresponds to the standard cross-entropy loss for classification, averaged over samples of 0 from the variational distribution. In practice, we approximate this average with a single sample, the cost is equivalent to standard classification under the MLE objective. (II) is the KL regularization term, which can be computed in closed form. This computation is lightweight since it does not involve data or forward passes through the network. Overall, the training costs of VCL and TD-VCL are nearly identical, as the main bottleneck lies in (I), which is similar in both methods. Additionally, we employ early stopping, which reduces the number of training epochs and thereby lowers computational requirements compared to prior implementations.
Inference Cost. Bayesian inference is approximated by the posterior predictive distribution (0) [p(y" (,xx,11a) which we estimate via Monte Carlo sampling. The computational cost of this step depends on the number of parameter samples 0. In our experi- ments, we use a single sample to ensure computational fairness across all baselines. With this choice, inference reduces to a single forward pass through the network, identical to a standard classifier. Using more samples increase the cost proportionally but also improve predictive performance.
p(y" x D1e)
Hyperparameter Search Cost. The cost of hyperparameter search can be managed by constraining the computational budget. As descrihed in Appendix HH] we restrict all methods (including baselines) to budget of at most one GPU-day. Our method introduces two additional hyperparameters beyond show moderate to good robustness to these choices, suggesting that the search cost could be further reduced if necessary.
those of VCL (n and A). Results in Appendix [
0.2 What is the memory cost of TD-VCL?
We analyze memory usage in terms of maintaining the replay buffer and storing the previous posteriors.
Replay Buffer Cost. The buffer has memory complexity O(n), where is the n-Step hyperparameter. This cost is no greater than that of the core sets used in VCL CoreSet. In our benchmarks, the replay buffer is intentionally limited to most 200 data points from previously observed tasks, which is negligible compared to the 60,000 data points in the current task. As a result, the replay buffer does not represent major bottleneck.
Posterior Storage Cost. The storage of posteriors also has memory complexity O(n). For smaller networks (such as those used in the MNIST and NotMNIST benchmarks), the memory usage is comparable to that of the replay buffer: for example, storing 200 MNIST data points requires about 0.60 MB, while storing the posterior requires about 0.68 MB (assuming float32 precision). Naturally, the memory required for posteriors increases with larger and deeper networks.
0.3 Is maintaining previous posteriors a major bottleneck? Can we optimize this cost?
As noted in the Limitations (Section 6). TD-VCL may increase memory requirements. However, this is not necessarily a major limitation, and we also discuss strategies to reduce memory usage.
Memory Complexity is a function of n, which the user controls. The buffer size and number of posteriors are defined by n. If memory a bottleneck, one can control v to satisfy memory constraints. Crucially, is not always equal to the number of tasks. Our robustness analysis (Appendix L shows that performance increases monotonically with n up to a level where it may saturate. Therefore, any should be better than vanilla VCL, and, if performance is expected to saturate, one can also set
34

---

to be much lower than the number of tasks. The employed hyperparameters (Appendix) H suggest that we can usually assume a value of that is lower than the number of tasks.
Assume a memory-efficient variational family Q. Since memory may be a challenge for large Bayesian networks, there alternative architectures, such last-layer variational methods bayesian LoRA adapters [59 (2),, which approximates the posterior distribution in fixed number of parameters. These methods drastically reduce the required memory at the cost of expressiveness of the variational family.
Store previous posteriors in cheaper memory alternatives. Since TD-VCL does not previous posteriors for inference but only for computing the KL regularization, they do not need to occupy GPU memory. In fact, the regularization term could be computed asynchronously with an external computer) while the GPU is used to generate predictions and estimate the likelihood terms. While implementation is more involved, it allows the use of both CPU/GPU and avoids having previous posteriors in GPU memory, which is usually the bottleneck.
CPU (or even
Estimate TD objective with fewer posteriors but covering older timesteps. A corollary of Proposition 4.4 is that we can represent the learning target as any combination n-step TD targets. This means that we may store posteriors every steps, instead of every step. In this case, given T tasks, we only store T/m posteriors. Naturally, this leads to a different way of estimating the learning objective, but ensures that it is covering older tasks to prevent catastrophic forgetting.
Lastly, we highlight that there are realistic Continual Learning settings where storing posteriors is not a major bottleneck. For instance, when continually learning on embedded systems with access to a cloud storage. These embedded systems (mobile phones, wearables) usually present limited storage/GPU memory onboard. Some problems require on-the-fly model adaptation and, for privacy- related reasons, the data must be kept on the device for limited time. Nonetheless, we may upload model snapshots the cloud without problems (sometimes this upload is required to conduct quality evaluation or audits). A concrete example an on-device speech recognition model on a smartwatch adapting user's voice. We believe our TD-VCL objective is well-suited for this problem setting.
0.4 When should one use n-Step TD-VCL or TDQ)-VCL?
TD(A)-VCL VCL a generalization of n-Step TD-VCL. As we presented in Appendix E. TD(A)-VCL forms spectrum of CL algorithms, and we recover n-Step TD-VCL when À + 1. Therefore, the "choice" depends on X, which controls how much the learning objective should prioritize recent posteriors. If one believes that the most recent posterior retains the knowledge of previous tasks, then higher À should work better. Otherwise, one should use lower values as past estimates contain information that has not propagated over the recursive updates. In practice, it depends on the continual learning problem and the potential transfer/intefference among tasks. The recommendation is to start from TD(A)-VC and tune the Ả hyperparameter.
35

---

0.5
What is the impact of Early Stopping in the presented methods?
We perform an ablation study to evaluate the impact of Early Stopping, with results reported in Table We find that removing Early Stopping does not significantly impact the performance of TD-Ve TD-VCL, while VCL shows slight degradation. The methods most negatively affected are the MLE baselines. This result is expected, since these models lack any form of regularization, and training without Early Stopping leads to overfitting. Notably, experiments without Early Stopping required approximately to 10 times more training time. As discussed in the main paper, Early Stopping substantially reduces computational cost.
T0.
Table 10: Results without Early Stopping, We observe that TD-VCL maintains strong performance even without early stopping, while VCL shows slight degradation and MILE baselines suffer from overfitting. Training without early stopping also took 5-10x longer.
PermutedMNIST-HHrd
t=
t=
t
t=5
t=
t=
t= 8
t
10
Online MLE
0.75a0.04
0.58a0.07
0.55
0.44
0,42a0.05
0.36.0.04
0.3420.05
0.31
0.30a0.04
Batch MLE
0.9420.01
0.89a0.02
0.82a0.04
0.69a0.03
0,64a0.04
0.55
0.52a0.03
0.46.0.04 0.45a0.3
VCL.
0.96=0.01
0.94+0.01
0.92+0.01
0.89+0.04
0.86x0.04
0.84=0.04
0.80+0.09
0.78=0.11
0.71
VCI. CoreSet
0.96=0.00
0.95+0.01
0.94=0.01
0.92+0.02
0.90+a.02
0.88+0.02
0.85a0m2
0.83+0.03
0.79+0.09
N-Step TD-VCL
0,95=0.00
0.95#1.00
0.94=0.00
0.93en.00
0.92x0.01
0.92:0.01
0.89+0.01
0.88=0.02
0.85+0.04
TDCA-VCL
0.97=0.00
0.97:0.00
0.96:0.01
0.95=0.00
0.94:0.01
0.93:0.01
0.91=0001
0.91=0.02 0.89
36

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All the claims are accompanied with either theoretical development or experimental validation, which are found in Section 5.1 and appendices.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Refer Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
37

---

Justification:
Please refer to Sections 3 and and appendices A B
C D
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
38

---

Answer: [Yes]
Justification:
Please refer to the codebase shared as link in Section 5.1
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeeee //nips.com htte://alllllliiiee/e//eeeuaaass ssiniiiiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
This information is shared in Section 5.1 and Appendices G and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Experimental results reported with two standard deviations across ten five seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
39

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Refer to Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://pp....aeo..o.oo/ooo////aa
Answer: [Yes]
Justification: The authors reviewed the NeurIPS Code of Ethics and the research conforms to
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Refer Appendix F
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
40

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets properly referenced throughout the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
41

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All details of the code and newly introduced benchmarks are available in the paper and released codebase.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
42

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buttps://neure wwwparipppsc.coooeeeeeeeee for what should or should be described.
43


Reviews:
[{"review_id": "YIFBLgPR9i", "reviewer": "Reviewer_Fkvs", "summary": "The paper proposes Temporal-Difference Variational Continual Learning（TD-VCL）, a novel variational continual learning (CL) objective that mitigates catastrophic forgetting by integrating multiple past posterior estimates through *n*-step KL regularization and temporal-difference (TD) targets. This paper's presentation is good and clear, which makes it easy to understand and follow. The conclusion is aligned with its experiment results, and the author discussed their limitations in the last section.", "questions": "Please see Weakness", "limitations": "Please see Weakness", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength: \n1. The investigated problem: continue learning is important.\n\n2. The core idea of leveraging multiple past posteriors (via n-step KL and TD(λ)-VCL) to reduce compounding approximation errors is theoretically proved and innovative.\n\n3. The presentation is clean and clear. I really like the organization of figures 3 and tables 1,2,3.\n\n4. The appendix is detailed, making the experiments more convincing.\n\n5. The experiment results are good, which outperforms the baseline: MLE series and vanilla VCL and align with the authors' conclusion.\n\n## Weakness:\n1. I notice sometimes TD(λ)-VCL outperforms n-Step TD-VCL, but sometimes fall behind (table1, SplitMNIST-Hard, t=5; table1, SplitNotMNIST-Hard, t=4). So what do you like the reason?  When should we use n-Step TD-VCL and when to use TD(λ)-VCL?\n\n2. As all the experiments in this work is MLE, AlexNet architectures,  I'm very curious about the performance on transformer architecture models, as today's LLMs frequently include this architecture, discussing TD-VCL performance on transformers will bring much greater impact.\n\n3. The computation cost is also important. I'd appreciate it if this analysis is included.", "was_revised": true, "final_justification": "I appreciate the authors' detailed response, which has resolved the issues I raised. In light of these clarifications, I will increase my rating.", "rebuttal": "Thank you for your review! We appreciate that you found our work **theoretically principled and innovative**, our **presentation clean/clear**, the experiments **detailed and convincing**, with **good results**. We also appreciate that you found our quality/clarity **excellent**, and significance and originality **good**.\n\nWe aim to address your concerns below:\n\n**Q1** Does sometimes TD(λ)-VCL fall behind N-Step TD-VCL? When should we use one or the other?\n\n**A1** First, we would like to highlight that in both examples raised by the reviewer, given the overlap in confidence intervals, we cannot conclude with high statistical confidence that one method outperforms the other -- they are statistically on par. Nonetheless, the question remains valid and important.\n\nTD(λ)-VCL is a generalization of N-Step TD-VCL. As we presented in Appendix E, TD(λ)-VCL forms a spectrum of CL algorithms, and we recover N-Step TD-VCL when $\\lambda \\rightarrow 1$. Therefore, the \"choice\" depends on λ, which controls how much the learning objective should prioritize recent posteriors. If one believes that the most recent posterior retains the knowledge of previous tasks, then a higher λ should work better. Otherwise, one should use lower values as past estimates contain information that has not propagated over the recursive updates. In practice, it depends on the continual learning problem and the potential transfer/interference among tasks. The recommendation is to start from TD(λ)-VCL and tune the λ hyperparameter.\n\n**Q2** TD-VCL in transformer architectures\n\n**A2** The choice of architecture follows prior work in the area of Bayesian CL [1-4], so that we can establish a fair and reliable comparison with past claims from this literature. We believe this choice is solid as it reflects what is advisable to fit the tasks in the employed benchmarks. \n\nFurthermore, **our proposed objective is agnostic to the architectural inductive biases as long as it approximates a variational distribution**. In this sense, our method is readily applicable to **Bayesian transformers, which has already been shown to be effective for Variational Inference [5]**. Since we employ the same mechanism, we believe the effectiveness also extends to our problem setting.\n\n**Q3** Computational cost\n\n**A3** Thank you for your suggestion. We provide a detailed analysis below, which will be incorporated in a new appendix of our work.\n\nWe divide this analysis into three perspectives:\n\n\n\n1) Training Cost: The training cost is defined by how we compute Equations 4/5, which relies on two terms: MC estimation of the likelihood term (I) and the KL regularization terms (II). (I) is the standard cross-entropy loss for classification, but averaged across different $\\theta$ sampled from the variational distribution. **We approximate this average with a single sample**, so the cost is the same as for standard classification with MLE objective. (II) is the KL regularization term, which is computed in closed form. It is lightweight since it does not rely on the data/forward passes in the network. Comparatively, the costs of VCL and TD-VCL are approximately the same, as (I) (the bottleneck) is roughly equivalent. We also implement early stopping, which greatly reduces the number of training epochs, reducing computational resources considerably in comparison with the implementations of prior work.\n\n2) Inference Cost: The Bayesian inference involves approximating the posterior predictive distribution $p(y^* \\mid \\mathbf{x}^*, \\mathcal{D}\\_{1:t}) = \\mathbb{E}\\_{q\\_t(\\theta)} [p(y^\\* \\mid \\theta, \\mathbf{x}^\\*, \\mathcal{D}\\_{1:t})]$ via MC sampling, which again depends on the number of samples of $\\theta$. In our work, **we used a single sample** to be computationally fair across all baselines. Thus, our cost is simply a forward pass in the network, as in a standard classifier. More samples would incur additional computational cost, but it should also lead to higher performance.\n\n\n3) Hyperparameter search cost: This cost can be controlled by fixing the search compute. As described in Appendix G, we limit all methods (including baselines) to at most 1 GPU day budget. Our method incorporates 2 hyperparameters on top of VCL (n, λ), and results in Appendix L presents a moderate-to-good robustness to the choices, which suggests this cost could be cut down if required.\n\n\nReferences:\n\n[1] Nguyen et. al. Variational Continual Learning. ICLR, 2018.\n\n[2] Ahn et. al. Uncertainty-based Continual Learning with  Adaptive Regularization. NeurIPS, 2019.\n\n[3] Ebrahimi et. al. Uncertainty-Guided Continual Learning with Bayesian Neural Networks. ICLR, 2020.\n\n[4] Bayesian Adaptation of Network Depth and Width for Continual Learning. ICML, 2024.\n\n[5] Sankararaman et al. BayesFormer: Transformer with Uncertainty Estimation, 2022."}, {"review_id": "kmpB6aEgNP", "reviewer": "Reviewer_LVzR", "summary": "- This paper studies continual learning with known task boundaries on image classification\n- This paper looks at the Bayesian continual learning (particularly VCL), and reformulates the object so that it is a weighted combination of task likelihoods estimated through a small coreset, and task likelihoods estimated through Kl-divergences with past posteriors\n- The authors argue that this allows better performance than VCL since it uses multiple previous posterior estimates, rather than only the most recent one\n- The authors make connections to Temporal difference learning in reinforcement learning\n- There is a variation for VCL, UCB and UCL\n- Empirically the TD variant of the algorithm outperforms the vanilla variant in all scenarios", "questions": "- How does this compare to non-bayesian methods of continual learning? As the authors stated, generally this method can be adapted to any regularization based approach, so it would be interesting to see more methods\n- Is is possible to limit the number of posteriors which need to be stored? As stated in the weaknesses section, currently we need to store N posteriors. Is it possible to do a reweighting trick similar to what was done to derive the TD objective to limit the number of past posteriors which need to be stored?", "limitations": "See Weaknesses/Questions Section", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Well motivated and theoretically principled\n- Well written\n- Interesting connections to reinforcement learning which is valuable to the community\n- Consistent improvement over non-TD baselines\n\nWeaknesses\n- Comparisons are only done relatively to the non-TD variants (VCL, UCB, and UCL). While improvement is consistent, these methods are generally not competitive CL methods, and there is no comparison to newer higher performance methods\n- This method requires a large number of additional storage. Namely, it requires the use of a coreset to estimate the monte-carlo objective, and also requires storing the model posterior estimate for all past objectives. Compared to VCL, this is large, since it requires only storing the previous posterior estimate. Over large N this could be problematic (as storage space grows linearly with N)", "was_revised": true, "final_justification": "I will maintain the score of 4, as there are clear contributions of the paper, but the scope and potential impact of the paper seem more limited, namely due to the more limited evaluation of the method compared to stronger continual learning methods.", "rebuttal": "Thank you for your review! We appreciate that you found our work **theoretically principled and well motivated, well-written**, with **consistent results** and with **interesting connections to RL**. We also appreciate that you found our work to have good quality, clarity, and originality.\n\nWe aim to address your concerns below:\n\n**Q4** Why compare with only Bayesian CL baselines? How would this method compare with other families of CL methods?\n\n**A4** We agree that providing a more exhaustive set of CL baselines would provide a better perspective on the current CL research landscape. However, CL research spans several directions, adopting different assumptions and desiderata. We opted not to broaden the scope too much, as there are literally hundreds of different methods available. Otherwise, it would be really hard to control the experiments and perform fair comparisons. Alternatively, our work keeps baselines consistent in these terms, allowing us to make direct claims about the **impact of the proposed objective**, which is our object of study.\n\nAdditionally, **most methods explore orthogonal design choices** (e.g., architecture, memory, regularization). **Given the flexibility of our objective, it can be directly combined with different methods (and even objectives)**. To validate this property, in Table 3, we extend other methods (UCL, UCB) to employ the TD objective, presenting consistent improvements. **We believe most methods would also be complementary to our objective.**.\n\nLastly, as stated in the paper, our work has a particular goal of **advancing Bayesian methods for CL**. We highlight that the Bayesian framework follows a principled approach that **allows the development of epistemic uncertainty-aware models**, which is crucial for robust, safe Machine Learning. These models are naturally capable of performing Active Learning [1], Out-of-distribution Detection [2], and Risk-Sentitive Optimization [3]. They also provide a new layer of interpretability for predictions via uncertainty estimates. **These are capabilities that non-Bayesian methods do not provide** and are very relevant for online/continual learning under distributional shifts and beyond.\n\n\n**Q5** Memory Cost Analysis and Discussion\n\n**A5** We first analyze the memory cost associated with TD-VCL. The buffer size does present memory complexity of O($n$), where $n$ is the n-Step hyperparameter. This cost is lower than or equal to the core sets in VCL CoreSet. Nonetheless, the designed benchmarks purposely limit the size of the replay buffer to contain up to 200 data points for previously observed tasks, which is negligible given the full training data of the current task (60,000 data points). Thus, it does not configure a major bottleneck.\n\nThe posteriors' regularization also presents memory complexity of O($n$). For smaller networks (e.g., the ones used for MNIST and NotMNIST benchmarks), the memory usage is comparable: 200 MNIST data points use ~0.60 MB, while the posterior uses ~0.68 MB (assuming float32). Naturally, the posteriors use more memory as we leverage larger/deeper networks.\n\nAs described in the Limitations paragraph (Section 6), we do acknowledge that TD-VCL may increase the memory requirements. Nonetheless, we would like to highlight why this is not necessarily a major limitation and also present alternatives to optimize memory usage.\n\n1) **Memory Complexity is a function of $n$, which the user controls**. The buffer size and number of posteriors are defined by $n$. If memory is a bottleneck, one can control $n$ to satisfy memory constraints. **Crucially, $n$ is not always equal to the number of tasks.** Our robustness analysis (Appendix L) shows that performance increases monotonically with $n$ up to a level where it may saturate. Therefore, any $n > 1$ should be better than vanilla VCL, and, if performance is expected to saturate, one can also set $n$ to be much lower than the number of tasks. The employed hyperparameters (Appendix H) suggest that we can usually assume a value of $n$ that is lower than the number of tasks. \n\n2) **Assume a memory-efficient variational family $\\mathcal{Q}$**. Since memory may be a challenge for large Bayesian networks, there are some alternative architectures, such as last-layers variational methods or bayesian LoRA adapters [4, 5, 6], which approximates the posterior distribution in a fixed number of parameters, drastically reducing the required memory (at the cost of expressiveness of the variational family).\n\n\n3) **Store previous posteriors in cheaper memory alternatives**. Since TD-VCL does NOT use previous posteriors for inference but only for computing the KL regularization, they do not need to occupy GPU memory. In fact, the regularization term could be computed asynchronously on CPU (or even with an external computer) while the GPU is used to generate predictions and estimate the likelihood terms. While implementation is more involved, it allows the use of both CPU/GPU and avoids having previous posteriors in GPU memory, which is usually the bottleneck.\n\n4) **Estimate TD objective with fewer posteriors but covering older timesteps**. A corollary of Proposition 4.4 is that we can represent the learning target as **any** combination of n-step TD targets. This means that we may store posteriors at every $m$ steps, instead of every step. In this case, given $T$ tasks, we only store $T / m$ posteriors. Naturally, this leads to a different way of estimating the learning objective, but ensures that it is covering older tasks to prevent catastrophic forgetting.\n\n\nLastly, we highlight that there are realistic Continual Learning settings where storing posteriors is not a major bottleneck. For instance, when **continually learning on embedded systems with access to a cloud storage**. These embedded systems (mobile phones, wearables) usually present limited storage/GPU memory onboard. Some problems require on-the-fly model adaptation and, for privacy-related reasons, the data must be kept on the device for a limited time. Nonetheless, we may upload model snapshots to the cloud without problems (sometimes this upload is required to conduct quality evaluation/audits). A concrete example is an on-device speech recognition model on a smartwatch adapting to a user's voice. We believe our TD-VCL objective is well-suited for this problem setting.\n\n\n**Q6** Is it possible to limit the number of posteriors that need to be stored?\n\n**A6** Yes. As we stated in Q5, one could:\n- Control $n$ to use fewer posteriors and satisfy memory requirements (as we highlighted, $n$ is a hyperparameter and not necessarily equal to the number of previously observed tasks); and\n- Use fewer posteriors from a longer history of tasks by maintaining posteriors only after every $m$ timesteps.\n\n\nReferences\n\n[1] Gal et. al. Deep Bayesian Active Learning with Image Data. ICML, 2017.\n\n[2] Nguyen et. al. Out of Distribution Data Detection Using Dropout Bayesian Neural Networks. AAAI, 2022.\n\n[3] Depeweg et. al. Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning. ICML, 2018.\n\n[4] Snoek et. al. Scalable Bayesian Optimization Using Deep Neural Networks. ICML, 2015.\n\n[5] Melo et. al. Deep Bayesian Active Learning for Preference Modeling in Large Language Models. NeurIPS, 2024.\n\n[6] Yang et. al. Bayesian Low-rank Adaptation for Large Language Models. ICLR, 2024."}, {"review_id": "T0INPb9bKB", "reviewer": "Reviewer_aVHF", "summary": "This paper looks at integrating temporal-difference style approach for limiting compounding errors in continual learning. The motivation behind this work mainly follows the bayesian framework, where instead of just considered the parameters of just the previous timestep, one should consider a combination of n previous time steps. Part of the proposed approach was inspired by the concept of TD lambda return, which is used as the variational distribution q. The authors evaluated their approach using variants of MNIST, CIFAR100 and Tiny ImageNet benchmarks. The results show that their approach had more success in mitigating catastrophic forgetting.", "questions": "1. It seems that the main focus is on catastrophic forgetting. However, in continual learning, catastrophic interference is also a challenge. Can the authors share how their work can or will relate to catastrophic interference?\n\n2. Will the model also work for reinforcement learning problems?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "# Strength\n1. The writing is clear and the mathematical theorems used are easy to follow. \n2. Figure 2 is important to give a clear motivation of this paper.\n3.  I appreciate the authors providing the intuition in section C and D in the Appendix on the connection of TD targets in TD-VCL and RL. It is a creative approach of using bootstrapping targets.\n4. The authors also included implementation details and hyper parameters tuned.\n\n\n# Weakness\n1. The baseline models are not the most competitive baseline models. For example, one model that the authors should consider which is relevant to the Bayesian continual learning literatures is the elastic weight consolidation model (EWC) [1].\n\n2. The proposed approach requires knowledge of task information and task boundary. Are there ways to overcome this limitation? \n\n\n[1] Kirkpatrick, James, et al. \"Overcoming catastrophic forgetting in neural networks.\" Proceedings of the national academy of sciences 114.13 (2017): 3521-3526.", "was_revised": true, "final_justification": "The authors provided further clarification and additional empirical results which showed that their approach does indeed learn well without task knowledge.", "rebuttal": "Thank you for your review! We appreciate that you found our work **clear and easy to follow**, with good illustrations and implementation details, and **enjoyed the connections with RL methods and found our idea creative**. We are also happy that you understand our work as having **good quality, clarity, and significance**, and **excellent originality.**\n\nWe aim to address your concerns below:\n\n**Q7** Are the current Bayesian CL baselines competitive? \n\n**A7** We start by highlighting that our work has a particular goal of **advancing Bayesian methods for CL**. We highlight that the Bayesian framework follows a principled approach that **allows the development of epistemic uncertainty-aware models**, which is crucial for robust, safe Machine Learning. These models are naturally capable of performing Active Learning [1], Out-of-distribution Detection [2], and Risk-Sentitive Optimization [3]. They also provide a new layer of interpretability for predictions via uncertainty estimates. **These are capabilities that non-Bayesian methods do not provide** and are very relevant for online/continual learning under distributional shifts and beyond.\n\nBesides that, **most methods explore design choices that are orthogonal to ours** (e.g., architecture, memory, regularization). **Given the flexibility of our objective, it can be directly combined with different methods (and even objectives)**. To validate this property, in Table 3, we extend other methods (UCL, UCB) to employ the TD objective, presenting consistent improvements. **We believe most methods would also be complementary to our objective.**.\n\nGiven our scientific goal of validating the proposed learning objective under the Bayesian CL setting, we believe our baselines are representative and provide solid evidence to support our claim. And we also believe they are competitive in this particular setting. For instance, **EWC (suggested in the review) is consistently outperformed by VCL [4] (Figs. 2 and 5), UCL [5] (Figs. 3, 6, and 7), and UCB [6] (Table 2), across different benchmarks and implementations**. We believe this evidence is strong to support that our baselines outperform EWC and are, in general, strong enough for the Bayesian CL setting.\n\n**Q8** Does TD-VCL assume knowledge of task boundaries?\n\n**A8** Thank you for raising this concern. We plan to clarify it here and incorporate this clarification as part of our paper.\n\nWe argue that the **TD-VCL objective (and VCL objectives in general) does not require knowledge of task boundaries**, and we provide theoretical and empirical evidence for that.\n\n\nThe theoretical argument comes from the principle that the **Bayesian framework is self-consistent**: given a stream of data, the final posterior distribution should be the same regardless of how many Bayesian updates are executed.\n\nBased on that, the key thing is to realize that **the number of updates does not need to be equal to the number of tasks**. Mathematically, suppose we have a stream of $T$ tasks (represented by $t$). At a particular update $k$, we may consider a Bayesian update that includes data from multiple ($m$) sequential tasks (e.g., from $t_{a}$ to $t_{a + m}$): $\\mathcal{D}\\_k = \\bigcup\\_{j=0}^{m} \\mathcal{D}\\_{j}$. Crucially, this does not impose any assumptions on boundaries. Rather, once we decide where to start and end the data stream for the Bayesian update, there could be potentially many tasks included.\n\nUnder the same assumptions stated in Section 3, we have that: $p(\\mathcal{D}\\_k \\mid \\theta) = \\prod\\_{j=0}^{m} p(\\mathcal{D}\\_j \\mid \\theta)$. And, the recursive relationship (Equation 1) also follows: $p(\\theta \\mid \\mathcal{D}\\_{1:k}) \\propto p(\\theta \\mid \\mathcal{D}\\_{1:k-1}) \\prod\\_{j=0}^{m} p(\\mathcal{D}\\_{j} \\mid \\theta)$. Finally, following the same variational objective and ELBO derivation, we arrive at $\\mathcal{L}^k(\\theta) = \\mathbb{E}\\_{\\theta \\sim q\\_k(\\theta)}\\left[ \\sum\\_{j=0}^{m} \\log p(\\mathcal{D}\\_j \\mid \\theta) \\right] - \\mathcal{D}\\_{\\text{KL}}(q\\_k(\\theta) \\,\\|\\, q\\_{k-1}(\\theta))$, which is a tractable objective equivalent to Eq. 3 where the number of steps $k$ is isolated from the number of tasks $T$.\n\n**Therefore, the objective itself does not discriminate or require task boundaries**. TD-VCL will estimate the likelihood terms for multiple terms simultaneously, which is something already done while replaying past tasks. \n\nNow, we highlight that most **benchmarks** do isolate tasks, including the ones in this work, which makes it convenient to consider one Bayesian update per task. We also acknowledge that our writing in the Preliminaries section leads to concluding that the method requires task boundaries by conflating the meaning of the index $t$, and **we will clarify this point in the potential camera-ready version**.\n\nTo provide further practical evidence that the method does not require knowledge of boundaries, we present a new benchmark called **StreamingPermutedMNIST-Hard**.  **This benchmark does not provide any boundary between tasks.** From the full data stream of $T$ tasks, we create random chunks of sequential data and provide them to the methods for continual learning. We execute an evaluation after the complete data stream, considering held-out splits composed of all tasks. We report the average accuracy across them (equivalent to the t = 10 column in Tables 2 and 3 of the paper). The Table below shows the empirical results over 10 seeds. **We observe no negative impact in the VCL/TD-VCL methods in comparison with PermutedMNIST-Hard.** In fact, some methods improved performance, because we are likely replaying the same task into different chunks, alleviating the catastrophic forgetting challenge. The proposed objectives still outperform all other methods.\n\n**StreamingPermutedMNIST-Hard:**\n\n| Method          | t = 10         |\n|------------------|----------------|\n| Online MLE       | 0.54 ± 0.09     |\n| Batch MLE        | 0.64 ± 0.09     |\n| VCL              | 0.82 ± 0.05     |\n| VCL CoreSet      | 0.85 ± 0.04     |\n| N-Step TD-VCL    | 0.89 ± 0.02     |\n| TD(λ)-VCL        | 0.89 ± 0.02     |\n\n\n\n**Q9** Does TD-VCL assume knowledge of task information?\n\n**A9** Following the answer to the previous question, we argue that the TD-VCL objective does not require task information. We have a single replay buffer for all tasks, and the data is mixed. We also make sure that the newly introduced benchmarks (PermutedMNIST-Hard, SplitMNIST-Hard, SplitNotMNIST-Hard) are all built with single-head classification architectures.\n\n**Q10** How does TD-VCL relate to/address \"catastrophic interference\"?\n\n**A10** We found the formal definition of catastrophic interference and catastrophic forgetting to be equivalent, as in [7, 8]. We believe the question relates to **negative transfer between tasks**, i.e., learning a new task negatively correlates with learning another one.\n\nSuch task interference is often associated with **representational incompatibility**: the features learned to solve task A may not align with useful representations for task B. Therefore, feature reusing becomes misleading, which affects performance in negatively correlated tasks. We argue that this representational incompatibility is due to learning tasks in isolation: when solely learning task A, the optimization disregards task B. Alternatively, if we have a learning signal from both tasks simultaneously, then this issue is alleviated. \n\nIn this sense, we believe that the TD-VCL objective addresses such interference by allowing multi-task learning via a replay buffer that naturally emerges in its design. By replaying past tasks, the objective ensures that the learned representation does not completely disregard them, even if the buffer is tiny and limited. \n\nOn the other side, if this interference is caused by \"gradient interference\" [9], then it would require employing gradient projection methods (as in [9]), which is out of the scope of this work but can be readily integrated into the optimization procedure.\n\n\n**Q11** Is TD-VCL applicable to RL problems?\n\n**A11** While TD-VCL is not an RL algorithm (in the sense of synthesizing policies for maximizing rewards), it is readily applicable for RL algorithms with probabilistic formulation and that require online learning. For instance, it can be applied to probabilistic modeling of dynamics [10] or value functions [11], and RL formulations that leverage sequential variational inference, such as probabilistic meta-RL [12]. Bayesian RL is indeed very appealing, as the epistemic uncertainty estimates could be directly used for active exploration or conservative policy learning.\n\nReferences\n\n[1] Gal et. al. Deep Bayesian Active Learning with Image Data. ICML, 2017.\n\n[2] Nguyen et. al. Out of Distribution Data Detection Using Dropout Bayesian Neural Networks. AAAI, 2022.\n\n[3] Depeweg et. al. Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning. ICML, 2018.\n\n[4] Nguyen et. al. Variational Continual Learning. ICLR, 2018.\n\n[5] Ahn et. al. Uncertainty-based Continual Learning with\n Adaptive Regularization. NeurIPS, 2019.\n\n[6] Ebrahimi et. al. Uncertainty-Guided Continual Learning with Bayesian Neural Networks. ICLR, 2020.\n\n[7] McCloskey et. al. Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem. Psychology of Learning and Motivation, 1989.\n\n[8] French et. al. Catastrophic interference in connectionist networks: Can it be predicted, can it be prevented?. NeurIPS, 1993.\n\n[9]Yu et. al. Gradient Surgery for Multi-Task Learning. NeurIPS, 2020.\n\n[10] Chua et. al. Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models. NeurIPS, 2018.\n\n[11] Ghavamzadeh et. al. Bayesian Reinforcement Learning: A Survey, 2015.\n\n[12] Zintgraf et. al. VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning. ICLR, 2020."}, {"review_id": "vkjTj7D31x", "reviewer": "Reviewer_xMRf", "summary": "The authors propose an extension to the Variational Continual Learning (VCL) framework, which performs online variational inference by using the posterior from the previous task as the prior for the current one. To address the accumulation of approximation errors across tasks, they introduce new training objectives that combine likelihoods and regularization terms from both current and past tasks using weighted sums. Notably, they show that the original VCL objective is a special case of their formulation for a particular choice of weights.", "questions": "* Can the author include the upper bound on the average accuracy achieved by training on all tasks up to time t?\n* Early stopping is used in the experiments. Could the authors include an ablation study to assess its effect on performance and stability?\n*  The buffer size, which is critical for understanding the experimental setup, is only mentioned in the appendix. Could the authors move this information into the main paper?\n* Would it be possible to extend this method to task-agnostic settings, or are there theoretical or empirical reasons preventing this?\n\nI am open to increasing my overall score if the authors adequately address my concerns, particularly regarding the baseline comparisons, memory analysis, and assumptions about task boundaries.", "limitations": "* The proposed method is task-aware and requires explicit knowledge of task boundaries to adjust the loss (e.g., update the regularization term and buffer). Consequently, it is not applicable in task-agnostic or non-stationary scenarios where changes in data distribution occur without clear segmentation.\n* As with many Bayesian continual learning methods, the evaluation is limited to fully connected networks and AlexNet, which does not incorporate modern architectural features such as Batch Normalization. While this is not unique to this work, it reflects a broader limitation of Bayesian deep learning methods that deserves acknowledgment.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "***Strengths:***\n\n* The authors leverage the Bayesian framework to introduce a principled method that incorporates information from both current and past tasks, while addressing the accumulation of approximation errors inherent in online variational inference.\n\n* The experimental results show that the proposed method improves upon VCL (with/without CoreSet) across several datasets under the reported experimental settings.\n\n***Weaknesses:***\n\n*\tThe preliminaries lack important context. In particular, the authors do not explicitly state a key assumption of their method: that task boundaries are known. This assumption is critical for applying the approach and should be clearly stated in the introduction and discussed in the related work section.\nRelevant task-agnostic Bayesian approaches include:\n1.\tBonnet et al., Bayesian Continual Learning and Forgetting in Neural Networks, arXiv 2025\n2.\tZeno et al., Task-Agnostic Continual Learning Using Online Variational Bayes with Fixed-Point Updates, Neural Computation 2021\n\n* Since the method is task-aware, a stronger baseline for online maximum likelihood estimation (MLE) should be included. For instance, Mirzadeh et al. (2020) report over 80% average accuracy after 20 tasks, significantly higher than the ~40% reported in this paper after only 10 tasks, by using dropout and carefully tuned hyperparameters. A fair comparison would require similarly optimized training setups.\nReference: Mirzadeh et al., “Understanding the Role of Training Regimes in Continual Learning,” NeurIPS 2020.\n* As the method relies on storing past posteriors (i.e., the mean and variance of each weight), the authors should analyze the memory requirements of this representation and compare it with the buffer size used in each experiment (typically 40–200 examples). In addition, a comparison with rehearsal-based methods should be included—for example:\nBuzzega et al., “Dark Experience for General Continual Learning: A Strong, Simple Baseline,” NeurIPS 2020.", "was_revised": true, "final_justification": "The authors have fully addressed my main concerns.", "rebuttal": "Thank you for your review! We appreciate that you found our work **principled**, with **good quality and originality.**\n\nWe aim to address your concerns below:\n\n**Q12** Does TD-VCL assume knowledge of task boundaries?\n\n**A12** We kindly refer the reviewer to our answer **A8** (under reviewer **aVHF**'s review). In summary, we clarify that our method does NOT assume task boundaries and provide theoretical and empirical evidence.\n\n**Q13** Relevant task-agnostic Bayesian approaches (Bonnet et. al., Zeno et. al.)\n\n**A13** Thank you for raising this literature. We will discuss these works here and add such discussion to the Related Work section.\n\nThe work of Bonnet et. al. [1] (contemporaneous to ours) proposes a forgetting mechanism for older tasks. Similarly to our work, they start from the VCL objective (Eq. 5 in [1]). In contrast, they add a positive likelihood term to the loss in order to forget old tasks. Since our work focuses on **preventing** catastrophic forgetting, this mechanism would actually hinder performance. In summary, these methods concern different problem settings. \n\nThe work of Zeno et. al. [2] derives an analytical solution to the optimization problem via fixed-point matrix equations. Similarly to our work, they also start from the VCL objective (Eq. 6 in [2]). In contrast, we opted for a deep learning approach and optimized via stochastic gradient descent. We believe that a fixed-point matrix solution might not scale well with the number of parameters. In any case, the TD objectives may also be applied here. Our work proposes a new optimization objective, but does not impose how to solve it. \n\nWe also highlight that both methods are indeed task-agnostic due to the same reasons as our TD-VCL objective (described in **A8**).\n\n**Q14** Stronger MLE baseline from Mirzadeh et al [3].\n\n**A14** Thank you for highlighting this work. We analyzed this paper and experiments thoroughly, as the baseline results seemed surprisingly strong.\n\nWe carefully looked at their provided code [4] and executed it exactly as documented for PermutedMNIST. We found the following results for their method \"Stable SGD\":\n\n| t = 2   | t = 4   | t = 6   | t = 8   | t = 10  | t = 20  |\n|--------|--------|--------|--------|--------|--------|\n| 0.95±0.02 | 0.88±0.04 | 0.79±0.03 | 0.72±0.03 | 0.63±0.03 | 0.41±0.03 |\n\nOur first point is that **the code does not reproduce the results in the paper**, which claimed an 80% average after 20 tasks. With further investigation, we found that other researchers tried to reproduce the same results without success ([4], github issue 5). The authors provided responses regarding other benchmarks in the paper, but never for PermutedMNIST. Interestingly, one of the original reviewers of this paper also could not reproduce the results ([5], see Reviewer 3 \"Weaknesses\"), and the authors only responded by considering the other benchmarks, but not PermutedMNIST ([6], see \"R3-Reproducibility\"). Therefore, it is unclear if these results are really reproducible.\n\nWe also question their experimental design. It involves a strong search in very sensitive hyperparameters (learning rate and decay, batch size, dropout), using the test set as a validation objective *exhaustively*. This process may lead to overoptimizing the metric for that test set, which we believe is not a good practice for ML research. In our case, we adopted the analogous hyperparameters consistently with prior work to establish a fair comparison.\n\nLastly, if we consider this tuning procedure acceptable, we argue that all methods in our paper would benefit from it. Indeed, Mirzadeh et. al. themselves [3] show that this tuning procedure improves all baselines (Table 4 in [3]). Therefore, we do not expect this to change our conclusion that the TD-VCL objective helps address catastrophic forgetting.\n\n**Q15** Memory Cost Analysis\n\n**A15** We kindly refer the reviewer to our answer **A5** under reviewer **LVzR**, where we describe the memory requirements and discuss it.\n\n**Q16** Can the author include the upper bound on the average accuracy achieved by training on all tasks up to time t?\n\n**A16** We refer the reviewer to Figure 4 (left) in our paper, specifically for the curve where T = 10 and B = 60,000, which effectively means training in all past tasks with the full available data. The average accuracy is ~97%.\n\n**Q17** Early stopping is used in the experiments. Could the authors include an ablation study to assess its effect on performance and stability?\n\n**A17** Thank you for the suggestion. We report here the results without early stopping. We observe that the absence of Early Stopping does not affect the performance of TD-VCL. VCL seems to have a slight degradation in performance. The only ones that really struggle are the MLE baselines. This makes sense as they do not have any form of regularization, and the absence of early stopping leads to overfitting.\n\nMore interestingly, we highlight that running experiments without Early Stopping took around 5 to 10 times longer. As we stated in the paper, Early Stopping considerably reduces computation time.\n\n\n| Method Name     | t = 2       | t = 3       | t = 4       | t = 5       | t = 6       | t = 7       | t = 8       | t = 9       | t = 10       |\n|-----------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|--------------|\n| Online MLE      | 0.75 ± 0.04 | 0.58 ± 0.07 | 0.55 ± 0.07 | 0.44 ± 0.05 | 0.42 ± 0.05 | 0.36 ± 0.04 | 0.34 ± 0.05 | 0.31 ± 0.04 | 0.30 ± 0.04  |\n| Batch MLE       | 0.94 ± 0.01 | 0.89 ± 0.02 | 0.82 ± 0.04 | 0.69 ± 0.03 | 0.64 ± 0.04 | 0.55 ± 0.03 | 0.52 ± 0.03 | 0.46 ± 0.04 | 0.45 ± 0.03  |\n| VCL             | 0.96 ± 0.01 | 0.94 ± 0.01 | 0.92 ± 0.01 | 0.89 ± 0.04 | 0.86 ± 0.04 | 0.84 ± 0.04 | 0.80 ± 0.09 | 0.78 ± 0.11 | 0.71 ± 0.15  |\n| VCL CoreSet     | 0.96 ± 0.00 | 0.95 ± 0.01 | 0.94 ± 0.01 | 0.92 ± 0.02 | 0.90 ± 0.02 | 0.88 ± 0.02 | 0.85 ± 0.02 | 0.83 ± 0.03 | 0.79 ± 0.09  |\n| N-Step TD-VCL   | 0.95 ± 0.00 | 0.95 ± 0.00 | 0.94 ± 0.00 | 0.93 ± 0.00 | 0.92 ± 0.01 | 0.92 ± 0.01 | 0.89 ± 0.01 | 0.88 ± 0.02 | 0.85 ± 0.04  |\n| TD(λ)-VCL       | 0.97 ± 0.00 | 0.97 ± 0.00 | 0.96 ± 0.01 | 0.95 ± 0.00 | 0.94 ± 0.01 | 0.93 ± 0.01 | 0.91 ± 0.01 | 0.91 ± 0.02 | 0.89 ± 0.02  |\n\n\n**Q18**  The buffer size, which is critical for understanding the experimental setup, is only mentioned in the appendix. Could the authors move this information into the main paper?\n\n**A18** Thank you for the suggestion. We added this information in the Benchmarks paragraph of Section 5.\n\n**Q19** Would it be possible to extend this method to task-agnostic settings, or are there theoretical or empirical reasons preventing this? and\n\nLimitation: The proposed method is task-aware and requires explicit knowledge of task boundaries to adjust the loss (e.g., update the regularization term and buffer). Consequently, it is not applicable in task-agnostic or non-stationary scenarios where changes in data distribution occur without clear segmentation.\n\n**A19** We hope our answer **A8** under reviewer **aVHF**'s response clarifies these points.\n\n**Q20** Limitation: As with many Bayesian continual learning methods, the evaluation is limited to fully connected networks and AlexNet, which does not incorporate modern architectural features such as Batch Normalization. While this is not unique to this work, it reflects a broader limitation of Bayesian deep learning methods that deserves acknowledgment.\n\n**A20** Our choice of deep network architectures indeed follows prior work in the area [7, 8, 9]. Nonetheless, we highlight that stabilizing variational architectures is an active area of research. Indeed, there are works that bring Bayesian ResNets [10] (which incorporates batch norm) and Bayesian Transformers [11] -- perhaps not yet validated in CL settings. We also believe that validating building blocks like MLPs already unlocks applications in deeper networks via last-layer Bayesian methods [12] or Bayesian adapters [13]. In summary, we do agree that this area is still in development and presents limitations, but we also believe that the employed architectures are significant and provide solid evidence to support our claims related to the TD-VCL objectives.\n\nReferences\n\n[1] Bonnet et al., Bayesian Continual Learning and Forgetting in Neural Networks, 2025.\n\n[2] Zeno et al., Task-Agnostic Continual Learning Using Online Variational Bayes with Fixed-Point Updates. Neural Computation, 2021\n\n[3] Mirzadeh et al. Understanding the Role of Training Regimes in Continual Learning. NeurIPS, 2020.\n\n[4] Code for \"Understanding the Role of Training Regimes in Continual Learning\". Available in \"github: imirzadeh/stable-continual-learning\". Accessed in July 2025.\n\n[5] NeurIPS official review for \"Understanding the Role of Training Regimes in Continual Learning\". Available in \"proceedings.neurips.cc: paper_files/paper/2020/file/518a38cc9a0173d0b2dc088166981cf8-Review.html\". Accessed in July 2025.\n\n[6] NeurIPS Offical Author Feedback for \"Understanding the Role of Training Regimes in Continual Learning\". Available in \"proceedings.neurips.cc :paper_files/paper/2020/file/518a38cc9a0173d0b2dc088166981cf8-AuthorFeedback.pdf\". Accessed in July 2025.\n\n[7] Nguyen et. al. Variational Continual Learning. ICLR, 2018.\n\n[8] Ahn et. al. Uncertainty-based Continual Learning with\n Adaptive Regularization. NeurIPS, 2019.\n \n[9] Ebrahimi et. al. Uncertainty-Guided Continual Learning with Bayesian Neural Networks. ICLR, 2020.\n\n[10] Krishnan et. al. Efficient Priors for Scalable Variational Inference in Bayesian Deep Neural Networks. ICCVW, 2019.\n\n[11] Sankararaman et al. BayesFormer: Transformer with Uncertainty Estimation, 2022.\n\n[12] Melo et. al. Deep Bayesian Active Learning for Preference Modeling in Large Language Models. NeurIPS, 2024.\n\n[13] Yang et. al. Bayesian Low-rank Adaptation for Large Language Models. ICLR, 2024."}]

Author rebuttal:

Meta-review: This paper proposes a variational continual learning method inspired by temporal-difference updates. The work is theoretically sound, clearly presented, and demonstrates consistent improvements over VCL and related Bayesian baselines. Some concerns remain about limited evaluation, scalability, comparisons to non-Bayesian methods, and the potential experiments on other scaled settings. The rebuttal addressed key questions and strengthened confidence in the approach. Overall, the contribution is solid within Bayesian continual learning. The reviewers converged toward positive ratings. Taking the overall strengths and weaknesses into account, the AC recommends acceptance.
Decision: Accept (poster)