Forget Forgetting: Continual Learning in a World of Abundant Memory
Authors: Dongkyu Cho
Taesup Moon
Rumi Chunara
Kyunghyun Cho
Sungmin Cha
Venue: ICLR 2026 Poster (2026)
Keywords: continual learning
model merging
machine learning
large language models
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We study continual learning in a realistic setting where exemplar memory is abundant, while computation resources are limited.

Abstract: Continual learning (CL) has traditionally focused on minimizing exemplar memory, a constraint often misaligned with modern systems where GPU time, not storage, is the primary bottleneck. This paper challenges this paradigm by investigating a more realistic regime: one where memory is abundant enough to mitigate forgetting, but full retraining from scratch remains prohibitively expensive. In this practical "middle ground", we find that the core challenge shifts from stability to plasticity, as models become biased toward prior tasks and struggle to learn new ones. Conversely, improved stability allows simple replay baselines to outperform the state-of-the-art methods at a fraction of the GPU cost. To address this newly surfaced trade-off, we propose Weight Space Consolidation, a lightweight method that combines (1) rank-based parameter resets to restore plasticity with (2) weight averaging to enhance stability. Validated on both class-incremental learning with image classifiers and continual instruction tuning with large language models, our approach outperforms strong baselines while matching the low computational cost of replay, offering a scalable alternative to expensive full-retraining. These findings challenge long-standing CL assumptions and establish a new, cost-efficient baseline for real-world CL systems where exemplar memory is no longer the limiting factor.

Published as a conference paper at ICLR 2026
FORGET FORGETTING: CONTINUAL LEARNING IN A WORLD OF ABUNDANT MEMORY
Dongkyu Cho', Taesup Moon?, Rumi Chunara', Kyunghyun Cho',, Sungmin Cha'*
New York University 2 ECE/ / ASRI / IMNC/ / IPAI, Seoul National University dongkyu, cho @ nyu,edu, tsmoon @ snu.ac.kr, Trumi.chunane kyunghyun.cko sungmin.chat @ @nyu.edu
ABSTRACT
Continual learning (CL) has traditionally focused on minimizing exemplar memory, a constraint often misaligned with modern systems where GPU time, not storage, is the primary bottleneck. This paper challenges this paradigm by investigating a more realistic regime: one where memory is abundant enough to mitigate for- getting, but full retraining from scratch remains prohibitively expensive. In this practical "middle ground", we find that the core challenge shifts from stability plasticity, as models become biased toward prior tasks and struggle to learn new ones. Conversely, improved stability allows simple replay baselines outperform the state-of-the-art methods at a fraction of the GPU cost. To address this newly surfaced trade-off, we propose Weight Space Consolidation, lightweight method that combines (1) rank- based parameter resets to restore plasticity with (2) weight averaging to enhance stability. Validated on both class-incremental learning with image classifiers and continual instruction tuning with large language models, our approach outperforms strong baselines while matching the low computational cost of replay, offering scalable alternative to expensive full-retraining. These findings challenge long-standing CL assumptions and establish new, cost-efficient baseline for real-world CL systems where exemplar memory is no longer the limiting factor.
INTRODUCTION
As machine learning systems are increasingly considered to be deployed in dynamic, real-world environments, continual learning (CI.) has emerged critical paradigm for adapting to evolving
data streams without catastrophic forgetting (Wang et al. 2024a). A central challenge in this setting is
the plabitityyplastiiityy dilemma (Carpenter & Grossberg 1987 Mermillod al. 2013): models that
maintain high stability across prior tasks often fail to incorporate new knowledge (high stability, low plasticity), while those that remain highly plastic tend forget earlier information (high plasticity, low stability). Various CL scenarios have been explored -most notably in incremnacteettlll learning (class-IL) for image classification (Masana et al. [2022), ) and more recently in the continual learning of large language models (LLMs) (Wang et al. [222aa] 2023). Among CL approaches, exemplar-based methods-which store and replay representative samples from past tasks have become particularly popular due to their simplicity and effectiveness (Masana et al.) 2022 Zhou et al.) 2024: Wang ctt 2003). However, a notable trend in these methods is the use of highly constrained memory budgets. For instance, many class-IL. benchmarks assume only 20 exemplars per class rrouhhly 4% of the total training data-are are retained across tasks (Rebuffi et al., 2017, Zhou et al., 2024). Similarly, LLM-focused CL approaches often rely on restricted caches or memory-free mechanisms to sidestep the issue of long-term storage (Wang et al. (023). Yet the necessity and realism of such severe memory constraints remain questionable (Chavan et 2023 Yousuf Harun et al. 2023). and
practical solutions to address this gap are still underexplored.
In real-worke machine learning deployments, this assumption of severely limited exemplar memory is often misaligned with practical constraints (Prabhu et al. (203)). Modern storage solutions such as cloud-based object stores or local SSDs are both affordable and scalable. In contrast, GPU time, especially for training lcrge-scale foundation models (e.g., LLMs), constitutes a significant bottleneck. For example, an AWS instance with 100 GPUs can cost over $30 per hour, while storing 1TB
"Corresponding author.

---

Published as a conference paper at ICLR 2026
of data costs less than S25 per month (Amazon Web Services 2024). If the primary goal of CL is to enable efficient model adaptation to non-stationary data without full retraining from scratch (i.e. avoiding expensive joint training), then reducing GPU cost rasterather than storage usage-sharr be the main optimization objective.
Building on this observation, we revisit the CL setting under more realistic scenarios where exemplar memory constraints are relaxed. Our analysis reveals a critical trade-off across the memory spectrum: while traditional memory-constrainee setups are often unrealistic and full-data retraining (joint training) is prohibitively expensive, a practical "middle ground" of precisely in this realistic regime that we identify new challenge: while stability improves due to reduced forgetting, plasticity diminishes as the model becomes biased toward prior tasks. This highlights the urgent need for cost-efficient mechanisms that can restore plasticity without sacrificing stability, a gap our work aims to fill. We further investigate how this regime affects existing CL. methods across two domains: class-IL. and continual instruction tuning of LLMs. In class-IL, we observe that many wfte-of-the-arr methods incur significantly higher GPU training costs yet offer marginal improvements over naive replay baselines. In continual instruction tuning, common model merging strategies often suffer from limited plasticity or require storing a separate model per task, which limits scalability.
(i.e.,
sufficient) memory regime emerges.
These limitations across both domains point to the need for new approach that is cost-efficient yet effective under abundant exemplar memory regimes. Motivated by this, we propose Weight Space Consolidation, simple yet effective method that operates directly in the model's weight space. It combines (1) ranking-basee parameter resets, which periodically reset the dormant parameters (mea- sured via gradient-based signal accumulation) to their pretrained values restore plasticity, and (2) weight averaging, which maintains a running average of model weights during training to encourage convergence toward flatter, more stable optima. By modifying weights, our approach facilitates fast convergence without additional compute overhead in GPUs. Across elass-IL benchmarks and L.M continual instruction tuning, our method consistently matches or surpasses the performance of state-of-the-art methods while maintaining training costs comparable to naive replay. These results demonstrate that, when exemplar memory is no longer the bottleneck, cost-efficient CL is both achievable and practical. Our contributions are summarized follows:
We revisit continual learning under relaxed exemplar memory constraints and show that even naive replay can achieve strong performance while significantly lowering GPU costs.
We conduct an extensive analysis across the memory spectrum to reveal the cability-plasticity trade-off, demonstrating that in the "practical" abundant memory regime, restoring lost plasticity becomes as critical as preserving stability.
We propose lightweight and practical method, Weight Space Consolidation, which combines ranking-based parameter resets and weight averaging to address the stability-llastiity trade-off. We validate our approach across class-IL tasks (e.g., CIFAR-I00, ImageNet-100) and LLM contin- ual instruction tuning (TRACE; Wang et al. (0233) ), demonstrating consistent accuracy improve- ments and 3 4x cost reductions over state-of-the-ee methods.
2 RELATED WORKS
Continual learning. Continual learning (CL) has been actively studied in various scenarios and methodological categories. Among the three scenarios of CL (Van de Ven & Tolias 2019), class- incremental learning (class-IL) has been considered the most challenging and has been the most actively studied scenario (Masana et al. 2022). Generally, CL algorithms (including class-IL) can be categorized into regularization-basee approaches, which penalize changes important parameters for past tasks (Kirkpatrick et al. 2017 Aljundi et al., 2018 Cha et al., 2020 Kang et al.] 2022). rehearsal-basee approaches, which store and replay exemplars from past tasks (Rebuffi al. 2017
Cha et al.. 2023). and ixpansion-basee approaches, which expand the model's 's capacity to balance the trade-off between stability and plasticity (Yan et al. 2021 Wang et al. 2022). Additional approaches focus on addressing classifier bias toward recent tasks while using the exemplars (Wu et al.) 2019) Zhao et al. 2020). While exemplar-based methods have demonstrated state-of-the-art performance, they typically rely on strict memory constraints, often limiting memory size to a small percentage of the dataset (Rebuffi et al. 2017 Zhou et al. 2024). Recent studies challenge the necessity of these strict memory constraints, highlighting that the computational cost of maintaining and processing
memory-especcalll especially GPU usage- can far outweigh storage costs (Prabhu et al.) [2023] Chavan et al.,
2023
Harun et al. 2023). This shift in perspective opens the door to relaxing memory limits in order
2

---

Published as a conference paper at ICLR 2026
to reduce training costs, which is the focus of our work. Lastly, another line of work solely focuses on the loss of plasticity in CL (Dohare et al., (2024). where parameter resetting is commonly suggested as solution (Ash & Adams, 2020] Galashov et al. 2024 Wang et al., 2024b; Farias & Jozefikk 2024). In contrast, we show how both stability and plasticity are issues under realistic CL. scenarios.
Weight space operations. A growing body of work has explored directly manipulating model parameters in weight space across various domains, including domain generalization (Wortsman ct al.
2022 Cho al., 2025a). multi-task learning (Yu et al. 2024 Yang al.. (023), and continual learning Marout et al. 2025 Marczak |
et al. [025 Dziadzio et al., 2024). Most of these approaches operate as post hoc methods by merging the weights of pretrained models. For instance, TIES (Yaday itt al. ( proposes a selective merging strategy to mitigate interference between different tasks, while Iiharco al. (Iharco et al., 2022) demonstrate that simple arithmetic on task-specific weight deltas can edit models without further training. Building on these ideas, recent studies have extended weight-space operations to continual learning. Kozal et al. (Kozal al.. 2024) apply weight averaging techniques in a CL setting, and Marczak et al. (Marczak et al. 2025) introduce a selective merging approach tailored for continual adaptation. However, such methods typically require storing multiple full model checkpoints during training, fail in accumulating various task knowledge (Dziadzio et al. 2024), and more critically, may violate the sequential constraints of CL. In contrast, our method operates in weight space during training (Izmailov et al. 2018 Jang et al. (025). ) requiring neither multiple model copies nor post hoc merging. This enables cost-effective and online editing of the model's parameters while maintaining compatibility with the CL setup.
Positioning. We clarify the positioning of our work in Table []] Several recent papers have sought cost-effective methods for CL (Prabhu let al. 2023 Harun et al., 2023] Chavan et al.,
Table I: Comparison of continual learning papers across different criteria.
Piper
Alsscar
Neruyy
Fall
Coniiiinal Corpesaton
Lom of
Plegeriar
2023). but do not expand their study on varying exemplar memory scenarios (e.g., constrained/ abundant/ full). On the other hand, we enumer- ate our experiments across scenarios, illuminat- ing the effect of exemplar memory sizes on the model's stability and plasticity. While some works have already studied the loss of plasticity, they focus on extreme settings (i.e., full memory) where only plasticity is considered (Dohare al.) 2024]. neglecting the issue of forgetting (Galashov ctt al.] 20244) or failing to consider the computation costs (Wang et al.] (2024b). In contrast, we focus on realistic setting (i.e., sufficient memory) where both plasticity and stability considered.
MOTIVATION
Notation. We generally follow the setting of continual learning (CL) Masana et al. 2022 Zhou
et al. 2024). We consider sequence of T tasks, each associated with distribution P,. Let D, be the training dataset for task t, where D.- ~ Pt. The tasks are presented in order t = 1.... T. The model F (its parameters 0) does not retain explicit access to previous task datasets Dj for j < t, except via an exemplar memory buffer M of a capacity of K. Thus, at the training step of task I, the model updates its parameters 0 using the combined data D, U M1a-1 and the task-desig loss E(D, U Mza-1;6), where M14-1 includes selected exemplars from earlier tasks.
3.1 DEFINING THE SUFFICIENT EXEMPLAR MEMORY REGIME
Most prior works assume a strictly limited exemplar memory budget, such that < ΣΣ [ Under this constraint, the memory buffer M can retain only small subset of examples from each past dataset (Di, D.-1). For instance, common class-ll. benchmarks typically allocate only 20 exemplars per class, which corresponds to approximately 4% of the total training data (Rebuffi et al., 2017 Zhou et al. 2024). As a result, the buffer provides only a partial approximation of the true task distributions PP. P. leaving the model vulnerable to catastrophic forgetting.
By contrast, motivated by real-world scenarios where storage cost is relatively low but GPU cost is high, re-examine CL in a practical regime with sufficient memory enough to mitigate forgetting but where full retraining remains computationally expensive. Therefore, we pursue a practical "middle ground" of abundant-but-nn huandaam-but-ootessausvvve exemplar memory. We define the memory buffer
3

---

Published as a conference paper at ICLR 2026
M to be sufficient if it can retain enough samples to approximate the distribution of each previous task Pj for 1 S < t. Formally, we assume the total memory budget K satisfies K E (Dy where (0, 1] determines the fraction of past task data that can be stored. A larger value of implies that M contains a more representative subset of earlier examples, though not necessarily the entire datasets. Section 3.2 we investigate when the exemplar memory size becomes sufficient by experimenting over various memory settings. Under this sufficient exemplar memory setting, the mixture distribution Past of previously encountered tasks at the training step task t. becomes:
p(t) past T
Σ E
2/=1 Tj
where π, denotes the relative importance (e.g., proportional to the number of stored samples or task frequency) of each past task, and Pj represents the corresponding data distribution. In practice, we approximate Ry with its empirical counterpart îtj. which can estimated from the samples stored in the buffer. During training on task the aggregated past distribution Past is combined with the current task distribution Pa to form the hybrid training distribution:
plt) train
R AP: + (1 - A)P
(2)
past!
where x E [0, 1] is factor that balances the influence of the current and previously observed tasks.
In the next section, we analyze how this training distribution under sufficient exemplar memory influences the model's learning dynamics, leading to improved stability but degraded plasticity.
FOSTER
Replay
iCaRL
Memory = 20 (4%)
Memory = 200 (40%)
Memory = 300 (60%)
Memory = 400 (B0%)
65
x
*
77-
*
Accuracy
73
60-
72
74
55-
73
73
74
70
72
73
50-
69
Train Time (min.)
Figure Comparison of (y-axis) average class-incremental accuracy and (x-axis) training time under different exemplar memory sizes in class-incremental learning for 10-task using CIFAR-100. As memory increases, catastrophic forgetting is mitigated (i.e., increase in accuracy), ), but training time (i.e., computation cost) also grows proportionally. Note that the DER, FOSTER, and MEMO are expansion-bar based methods (shown with X mark): FOSTER doubles the model size, while DER and MEMO scale with the number of tasks. Compared to these costly methods, Replay and Ours demonstrate high accuracy with significantly lower cost, where our method offers the highest cost efficiency, closely approaching that of the cost lower-bound cost (i.e., Replay).
3.2 WHIT CHANGES UNDER A SUFFICIENT EXEMPLAR MEMORY REGIME?
Stability. Under sufficient exemplar memory, the distributions of previous tasks can be closely approximated. which effectively reduces forgetting that is, improves stability. Specifically, with large buffer M, the empirical distributions P, of past tasks approximate their true distributions Pj for j < This allows the empirical risk (1)((0) cooppted over the stored exemplars- closely approximate the ideal joint risk R1. (0) = Σ Ee- [e(0;z) =)], as if the model were trained jointly on all tasks. As a result, the learned parameters 0i.- remain close to the joint optimum 0j in parameter space, preserving performance previous tasks and mitigating catastrophic forgetting. For the complete derivation of this result, please refer to Appendix A.I.I
Experimentally, Figure shows that catastrophic forgetting is substantially reduced when exemplar memory is sufficient (e.g., [M| 2 200). Notably, the simplest baseline (i.e., Replay) outperforms more sophisticated methods while incurring significantly lower training cost (see Figure IT. As result, by the end of each task t-1, 1, the model serves both as a strong minimizer for previously learned tasks 1:t-1 and as reliable initialization for the upcoming task 1.
4

---

Published as a conference paper at ICLR 2026
Avg. Task Acc.
80
Train Loss
Continual Leaminng (wio. Reset!
2
Retraining
D3R
Sapley
0
100
200
300
400
500
1000
2000
3000
4000
Memory Size
Epochs
(a) Loss of plasticity
(b) Train Loss under full exemplar memory
Figure 2: Comparison of (a) average new-task accuracy under different exemplar memory sizes and (b) training loss under full memory in class-incremental learning for 10 tasks using CIFAR-100. As memory increases, the model's s ability to adapt to new tasks declines, resulting in reduced accuracy and slower convergence. Notably, in (b), resetting model weights before each task restores plasticity and facilitates training.
Plasticity. We find that under this condition, the challenge shifts from stability to plasticity. We conjecture that as exemplar memory becomes increasingly sufficient, the model's ability to learn new tasks (i.e., plasticity) gradually deteriorates. At task step 6, the model is trained on a hybrid pust (Eq. 2). where À E [0, 1] controls the emphasis on the new task.
distribution p(t)
train R XP:+ (1 Plll
As memory size increases, À decreases, causing the past-task distribution p(t)
pasi to dominate. When ((t=1), This similarity train
memory is sufficiently large, p(()) closely resembles p(t-1), and thus p(t) ~
traio.
results in high gradient alignment (Du et al., 2018), which we quantify via cosine similarity Pt
(o(" . //1a (aox) [ || (in Eq. 11), where St (ow) and 9; (puus) denote the mean gradients from the new
tasks and past tasks, respectively. When Pt 2 1, the expected gradient ÿr Age (new) (1 ))g(
(pust)
has a reduced norm, causing smaller updates (Yu et al. 2020):
||0(0) ((111) = vl@xel S nllg, (new)
which shrinks further as -> O, where 7 is the step size. This limits the model's capacity to adapt, as it tends to reuse previously learned parameters rather than learning new representations.
Such "parameter reuse" behavior has been observed when two sequential tasks are similar, leading to minimal drift across tasks (Lee et al. 2022 Dohare et al. 2024), which is hallmark of low plasticity. Our interpretation aligns with the stability plasticity dilemma (Mermillod et al. 2013] Zhang et al.) 2024), where retaining prior knowledge comes at the cost of adapting new information. It also corroborates observations from Rolnick et al. (2019). which showed that excessive exemplar memory hinder the learning of new tasks. Please refer Appendix A.1.2 for more detailed discussion.
stability-plastiiityy
Figure 2a compares the model's average accuracy on new tasks across various CL methods, which commonly used to measure the model's ability to acquire new knowledge (i.e., plasticity) (Liang) & Li 2023 Wang et al. 2024b). Here, we experimentally confirm that as memory increases, the model's 's plasticity generally degrades, resulting in lower average accuracy new tasks (also see Table 6). This aligns with Figure 2b] which depicts the model's 's train loss under full exemplar memory. We can observe here that the large memory interferes with convergence under the CL setting. Notably in Figure 2b we find that reinitializing the model parameters before each task (Dohare et al.) 2024 Farias & Jozefiak 2024) is simple solution restoring plasticity, as demonstrated by the low train loss of the model trained with reset and not continually (Orange) in Figure 2b
4 WEIGHT SPACE CONSOLIDATION
Based on insights from the previous section, we propose cost-efficient CL method-Waatt Weight Space Consolidation leverages weight space operations (e.g., selective resets and running averages) to reconcile the trade-offs of the sufficient exemplar memory regime: high stability and low plasticity.
4.1 RANKING-BASER PARAMETER RESET FOR IMPROVED PLASTICITY
The core idea behind our method is that sufficient exemplar memory enables the model to start from stable initialization (as discussed in Section 3) but remaining too close to this point can hinder plasticity. To address this, we introduce a ranking- based reset technique that selectively reinitializes less important parameters based on their estimated contribution to learning.
5

---

Published as a conference paper at ICLR 2026
Before training the ht task. the model parameters parameters-pptimizzed previous tasks 1:t-1-a are stored as peev (Line 6 in Algorithm l). Training on the tth task proceeds using the task loss (Line 9), and after Taarm warm-up epochs, we identify dormant parameters and gently reset them (Lines 10-11).
To rank parameter importance, we compute a moment-based metric Si for each parameter ele- ment I, using the first and second exponential moments of its stochastic gradients (mi, 61), in Adam (Kingma Ba (017):
S ( 0r.
This score favors parameters that consistently receive strong gradients, while penalizing those with low or noisy updates. Specifically, a large my indicates that the parameter has received gradients in a consistent direction, while large it implies that the parameter has experienced high gradient energy overall. By taking the product of the two, St becomes sensitive to both focused and sustained learning signals. Conversely, low S suggests that the parameter has received weak noisy gradients, indicating limited contribution to learning. We treat such parameters as dormant and reinitialize them to recover plasticity. This formulation provides simple yet effective heuristic for identifying underutilized parts of the network based on gradient dynamics during warm-up. In Section 5 we compare existing methods that use weight resets for plasticity recovery (see Table
In implementation, we retain only the top-Q% of the parameter element I (with Q-20 by default following Yadav et al. 2024)) ), and reset the rest by softly blending them with Peev (Lines 12-13):
01]]
0[] + a) Pprev (1,
0.5.
(5)
This gently pushes the model out of the previous solution basin to improve plasticity, while preserving parameters critical to prior tasks-thus maintaining stability.
Notably, this operation resembles model merging in the weight space, where two sets of parameters blended. However, unlike conventional model merging approaches that combine multiple trained models post hoc, our reset mechanism is applied during training, with the explicit goal of restoring plasticity. In this context, we treat the merged weights not as final model, but improved initialization point that facilitates adaptation to the new task without sacrificing stability.
4.2 WRIGHT AVERAGING FOR IMPROVED STABILITY
Using the reset model as a fresh starting point, we resume training 0 for the remaining epochs. From this point on, we accumulate a running weight average è (see Lines 14-16) following the stochastic weight averaging (SWA) (Izmailov et al., 2018), which is known to promote convergence to flatter optima. The running average is updated every j iterations after a warm-up up phase of Twarm steps:
Algorithm 1:
Weight Space Consoli- dation for cost-efficient CL.
1. Input: Model parameters 6, training data
Dis, memory buffer M, average interval j. warming epoch 7lwarm
2. Output: Trained model parameters 0
3.
for t <- to T do
4.
θ 0// Init. Averaged Model
if then
Ravg + +
6.
L Oprev +-
(6)
Navg +
7.
for i 1: Miser do
8.
Sample minibatch from (D,U M1+-1}
where Ravg = 1/s and is the current iteration index.
We find that this averaging is particularly effective
9.
Update 0 using {(b; 0) and SGD
10.
if (t: > 1 and i Twarm then
under sufficient exemplar memory settings, where data diversity introduces significant gradient variance.
11.
Ineact f-
FindDormantParams(( 0prec)
After the warm-up phase, the model often oscillates around multiple distinct low-loss regions due to this variance. By averaging weights across these regions, converges to a flatter and more robust minimum that consolidates knowledge across both past and current tasks (Izmailov et al. 2018 Cha et al. 2020).
12.
for E Treser do Reset weights using eq. (
13.
14.
if (t and Rwarn and 1%j=0) then
15.
navs + i/j
Importantly, our approach differs from traditional model merging methods in CL, which often combine independently trained fask-specific models to con- struct a final model (Mharco et al.) 2022). In contrast,
16.
(0 navy + 0)/ +
17.
0
method performs in-siu averaging during the training of single model 0, progressively updating

---

Published as a conference paper at ICLR 2026
@ as byproduct of the optimization trajectory. This eliminates the need to store and merge multiple per-task models, improving our method's 's cost-efficiecyy and scalability to longer task sequences.
At the end of training task t, replace the model parameters with the averaged weights O (see Line 17), which then serve as a stable initialization for the next task, preserving knowledge while enabling further adaptation. Please refer to Appendix A..2
(..2 for further implementation details.
Summary. Our method combines two simple yet effective weight-space operations to balance the stability- plasstiity trade-off in the sufficient exemplar memory regime: (1) ranking- bbaen resets recover plasticity by reinitializing dormant parameters, and (2) weight averaging enhances stability by converging to flat, robust minima. Both are directly motivated by our analysis in Section 3 and introduce negligible overhead, requiring no storage of per-task models or additional backward passes.
5
EXPERIMENT
5.1
EXPERIMENTAL. SETTINGS
We evaluate our method's performance and cost-efficiency under various exemplar memory settings.
Class-IL benchmarks. We use two standard class-incremental learning benchmarks (Masana et al.) 2022) via the PyCIL framework (Zhou et al.] 2023a): CIFAR-100, an image classification dataset with 100 classes split into 10 sequential tasks (10 classes each), and ImageNet- 100, a 100-class subset of ImageNet also split into 10 tasks of 10 classes each. We compare our method to seven
exemplar-bar based class-IL basclines: iCaRL (Rebuffi et al.) 2017). . BiC (Wu et al. 2019)) WA (Zhao
al. 2020), DER (an et al.] 2021), FOSTER (Wang et al. 22022) MEMO (Zhou et al. 2023b and
Replay, a naive baseline that finetunes using current data and stored exemplars.
LLM continual instruction tuning. We also evaluate TRACE (Wang al.
2023 continual
instruction tuning benchmark for LLMs across 8 domains. We compare our method against Replay and model-merging baselines, following recent findings that such methods be effectively applied
to CL (Roth et al. 2024 Dziadzio et al. 2024): Model Soups (Wortsman et al.) 2022). SLERP
(Jang
al. 2024). Mag (Marezak al. 2025), and Task Arithmetic (Ilharco
2022).
Architectures and protocol. For image classification, we use ResNet-32 (He et al. 2016) on CIFAR-
100 and R8 (He et al., 2016) ImageNet-100 (He et a 2016). For LLM experiments, we use an instruction-tumm averaged over five seeds and reported as average class-IL accuracy across all tasks after the final step (Masana et al. (2022). For TRACE, final scores are reported after completing all sequential tasks. We select hyperparameters following the realistic CL evaluation protocol proposed by (Cha Cho 2025). Further details on experimental settings are in Appendix A.5
version of LLaMA-3.2 (Grattatiori el al.) 2024). All class-IL results are
5.2 EXPERIMENTAL RESULTS
Class-IL results. We report class-incremental learning results in Table 2 and
and Figures 1 Note that
DER, FOSTER, and MEMO are expansion-baas based methods that increase model size over time during training. To ensure fair comparison, we follow the evaluation protocol of (Cha & Cho 2025) and reuse the best hyperparameters found on CIFAR- 100 when evaluating on ImageNet- 100. From these results, we make three key observations: First, as shown in Table under the conventional constrained memory setting (e.g., 4% memory), existing class-IL methods outperform Replay. However, as the memory size increases, the performance gap narrows substantially. With 20% memory, most methods perform similarly to Replay, suggesting diminishing returns of algorithmic complexity in the abundant exemplar memory regime. Second, when training cost (i.e., training time) taken into account, Figures and (see Appendix[A.A. A..4) show that expansion-basse methods become highly inefficient. For instance, while FOSTER maintains strong accuracy even under abundant exemplar memory, its training time is 4-5x higher than that of Replay. Third, our proposed method bethod-Weight Space mansolidation-ddmmnnnnaa
D
9
both strong performance and high efficiency. Table shows that consistently outperforms existing baselines (except for aopansion-based methods) under abundant exemplar memory (i.e. over 20% memory). Meanwhile, note that its training cost remains comparable to that of Replay, as shown in Figures [ and 9 Together, these results validate that method effectively mitigates the plasticity- stabilliy trade-off in class-IL using abundant exemplar memory with minimal GPU computation overhead.
7

---

Published as a conference paper at ICLR 2026
Table 2:
Average class-IL accuracy (%) on CIFAR-100 and ImageNet-100 with varying ceemplar-memory sizes. We report experimental results with varying memory sizes, ranging from 20 exemplars per class (a common setting in class to 400/600 exemplars per class (storing 80% of the previous dataset in CIFAR- 100 and nearly half in ImageNet-100), Bold highlights the best non-expansion method. We report the standard error across 5 runs.
Memory Size (the number exemplars per class) (oass) (Mlldasss
CIFARI00 100
ImageNet100
Method
20(4))
50(1050)
200()05))
400
200
400
(00
DER
63.95 63.95
70.13
7161111
75. 75.60
71.9620.
78.59ga.
79.6 79.61a
80.53
FOSTER
66.22+1.
67.67
73.53
77.28
70. 14..7
76.01
$0.9440.
82.792a.
MEMO
61.99110
70.58g10
73.71
75.39
66.35
77,89 77.89g0.4
50.05go42
81.11
Replay
18.6321.1
63. 63.78212
71.00gn.9
75.71
50. 30.5
73.79
28.4
81,08
iCaRL
49.9511.5
64.811
72.69108
75.1910s
50.322g0.
73.57
78. 18.45
80,8T
BIC
53.6540.9
64.74206
69.15ggo7
72.50
59.31
74.14go.
77.51 77.51
79. 79.292a.6
WA
61.32+1.
06. 06.19
71.42=1.2
73.83+14
61.44_1.1
75.85
78.79
80.21
Ours
52.16
66.892o.0
77.71 74.4920.
54.97
26.4
80.26
82.64.0.4
SLERP
Avg. (Online)
Avg. (Offline)
Task-Arith.
MagMax
Ours
Memory = 20%
Memory
40%
Memory 60%
Memory = 80%
0.40
0.46
0.42
0.44
Avg. Score
0.41
0.45
0.42
0.40
0.38
0.44
0.40
0.38
0.37
0.30
0.43
1000
2000
3000
1000
2000
3000
1000
2000
3000
1000
2000
3000
4000
VRAM Usage (min.)
Figure 3: Comparison of average score and relative VRAM usage measured as minutes under different exemplar memory sizes in LLM continual instruction tuning for 8-task using TRACE.
Continual instruction tuning results. We evaluate our method in a more practical setting of continual learning for foundation models, where exemplar memory is abundant. To this end, conduct continual instruction tuning on the TRACE benchmark, following its standard 8-stage setup across diverse domains. As strong baselines, have been actively explored for continual learning with foundation models (Roth et al. 2024 Dziadzio al. 2024). Among them, methods such as Task Arithmetic, MagMax, and Model Soup (i.e., Avg. (Offline)) require storing all T task-specific models or representations and merging them post hoc. In contrast, methods like Ours, SLERP, and Avg. Avg. (Online) operate in a single model trajectory and do not require saving all intermediate checkpoints. We denote [W| as the standard VRAM usage of a single model. To ensure a fair evaluation, we follow the protocol proposed by (Cha & Cho 2025): all hyperparameters are tuned only under the 20% memory setting, and the same configuration is reused across other memory sizes without additional tuning. Figure B shows that our method consistently outperforms the baselines when the exemplar memory exceeds 20% (Also see Figure TO). In particular, we observe that our method achieves 2-9% higher accuracy across all memory sizes compared to offline merging approaches like Task Arithmetic and MagMax, which require relative VRAM usage of [W| 7 for saving all previous task models in the GPU. Notably, as summarized in Appendix Table these offline merging methods struggle to integrate knowledge from diverse tasks (Dziadzio et al. 2024) and tend to underperform under abundant exemplar memory conditions. Conversely, our method avoids these issues by operating directly in weight space during training and achieves both high performance and GPU efficiency (relative VRAM usage: [W] 2). More experimental results or detailed numerical experimental results are introduced in Appendix A..4
compare with recent model merging approaches that
5.3 ABLATION STUDY
We conduct ablation studies to validate the effec- tiveness of two key components of our method. (1) ranking-bast -based parameter reset for plasticity recovery, and (2) weight averaging for stability.
Table 3: Ablation study CIFAR-100
Memory Size
Method
300
500
Replay
73:46
76. 76.90
reset
58.19
66.50
75.00
77.92
Contribution of each component. Table
sag.
48.73
$6.30
65:32
75.49
74.90
76.47
Ours
$2.00
59.69
66.51
67.23
73.57
76.11
77.42
78.25
summarizes the performance of our method when
8

---

Published as a conference paper at ICLR 2026
either reset averaging is ablated. The baseline Replay uses neither. We find that weight reset alone (w/o avg.) yields limited gains over Replay, while averaging without reset (w/o reset) improves stability but fails to fully adapt to new tasks. Only when both are combined do we observe substantial gains, highlighting that the two operations are complementary and that weight reset primarily serves to enable effective averaging.
Effectiveness of the importance metric. Table 4 com- pares alternative parameter importance metrics used for reset. Our moment- based metric (Eq. 4 performs on par with expensive Hessian-based scores, while being signif- icantly more efficient. Simple metrics like parameter drift also perform well, echoing prior work in pruning (Zhu & Cupta 2017 Liu et al. 2018). In contrast, using only the first or second moment leads to poor selection, confirming the necessity of combining both for robustness.
Table 4: Study of Parameter-Importance Metric on Average class-IL. accuracy (%) in CIFAR-100
Memory Size
Metric
Cost
200
500
Param. Drift
U
51.93
73,02
77.90
Fisher-based Hessian-based
52.35
73,49
77.57
52.81
73,63
78.16
First Moment
U
50.10
71.38
75.05
Second Moment
47.29
70.44
73.89
(oq. +
52.00
73.57
78.25
Reset strategies: how and when to reset. Finally, we investigate two design choices in the reset operation: the reset rule (how to reset) and the reset frequency (when to reset). As shown in Table []] our soft reset method (a weighted blend with prev) consistently outperforms random reinitialization and hard reversion. This advan- tage becomes more pronounced with larger memory. We also compare with existing methods (e.g., Shrink & Per- turb (Ash & Adams 2020) and Continual Backprop. (Do- hare et al. 20044), where our method outperforms competi- tors. Regarding frequency, performing reset only once after Wearm works well in most settings. However, in con- strained memory regimes, applying multiple resets yields further consisetntt with findings in sparse reinitial- ization (Frankle & Carbin 2019). We provide further analysis in Appendix A.4 on how tuning the reset frequency and retain rate Q affects performance.
Table 5: Study of reset strategies Av- erage class-IL accuracy (%) in CIFAR- 100
Memory Size
Strategy
20
200)
500
Method
Random
Revert
75.78
Shrink&Portrbl3
48.26
70.32
76.05
Continanl Backprup.
18
47.15
70.49
76.52
Ours (wlo Avg.)
48.73
70,81
76.47
Ours
$2.00
73.57
78.25
Reset Frequeacy
Every lier. Every Epoch Oned (Ours)
47.38
63.11
70.32
53.36
72.73
76.57
52,00
73.57
78.25
Continual Learning vs. Full Retraining. Prior work (Dohare et al.) (024) reports that with full exemplar memory, retraining from scratch (i.e.,
Table 6: Comparison of Full Retraining from
scratch and Continual Learning under a standard
CL setting. Average class-Il. accuracy (%) on CIFAR- is reported.
joint-training) can surpass continual learning (CL).
In Section 3.2] we revisit this observation theoret- ically and argue that, under full memory, per-task resets (reinitializing weights before each task) re- cover plasticity while preventing forgetting, yield- ing strong performance. However, in the more
Memory Size (f exemptan per citat)
Mishal
(45))
(00:200))
(409)
300 (60%)
(800))
Retmiiing
46.15
78.94
Continuall (Roplay)
48.63
75.71
52,16
67.35
74.49
75.97
17.71
realistic sufficient memory regime (e.g., 20-40%), retraining from scratch degrades markedly as forgetting is insufficiently addressed. In these settings, algorithms that leverage prior models (e.g., ours) offer clear advantages, as seen in Table Finally, while maximal replay might appear idcal, it is often impractical at scale (c.g.. LLMs). Our approach provides a cost-effective alternative that balances performance with efficiency
6 CONCLUDING REMARKS
This paper challenges the long-standing assumption in continual learning (CL) that exemplar memory is the primary bottleneck, arguing that in modern deployments, GPU cost is the true constraint. We investigate the consequences of having sufficient memory practical regime where forgetting is largely solved but full retraining from scratch remains prohibitively expensive- and demonstrate that the central challenge shifts from mitigating forgetting to overcoming loss of plasticity. To address this, we propose Weight Space Consolidation, lightweight method combining parameter resets and weight averaging to navigate this new Subility--lasttciis trade-off. Our method is empirically validated across image classification and LLM instruction tuning, where it outperforms strong baselines at fraction of the cost, establishing a scalable and cost-efficient alternative to full retraining. Ultimately, our findings call for crucial shift in focus for future CL research: from optimizing for unrealistic memory constraints toward designing computationally efficient algorithms for the real-world scenarios of today.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
We thank NYU HPC for their generous support and help in computational simulations and exper- iments. This work was supported by the Institute Information Communications Technology Planning & Evaluation (ITPP) with a grant funded by the Ministry of Science and ICT (MSIT) of the Republic of Korea in connection with the Global AT Frontier Lab International Collaborative Research. It is also supported in part by National Research Foundation of Korea (NRF) grant [No. A021RIA2C200788/ No. RS-2025-022636288, No. RS-202300265406) and the Institute of Infor- mation & communications Technology Planning & Evaluation (IITP) grants RRS-2021-I12211 12068, RS-2022-11220113, RS-2022-11220959, 153-022--12213431
ETHICS STATEMENT
The authors acknowledge and concur with the ICLR Code of Ethics, namely in its pursuit of (1) human well-being, (2) high standards of scientific excellence, (3) consideration for the societal impacts (i.e., harms) of AI, (4) honesty & trustworthiness, (5) fairness, (6) mutual respect for other researchers' works, (7) privacy, and (8) confidentiality.
REPRODUCIBILITY STATEMENT
For reproducibility, we provide the source code, experimental guidelines, and the scripts used in experiments. Please refer the README.md file in the supplementary materials how to reproduce our experiments. We also used a fixed seed setting, which is implemented in the source code. We also include notebook (.pynb) files reproduce the figures appearing our paper. Lastly, we thoroughly explain how our method and its experiments are
in Section 5. and Appendix A.5 implemented.
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference computer vision (ECCV), pp. 139-154, 2018.
Amazon Web Services. Amazon ec2 on-demand pricing. 2024. URL https:t//tss..... amazon. com/ Accessed: 2025-05-11.
Jordan T. Ash and Ryan P. Adams. On warm-starting neural network training, 2020. URL https
//arxiv. org.orgg/aaoo ats///bs//1 09477
Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic wttp:////.bb......a///
gradients, 2020. URL
Gerald Beer, RT Rockafellar, and Roger J-B Wets. A characterization of epi-convergence in terms of convergence of level sets. Proceedings of the American Mathematical Society, 11613):753-761 1992.
Daniel Brignac, Niels Lobo, and Abhijit Mahalanobis. Improving replay sample selection and storage for less forgetting in continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3540-3549, 2023.
Gail A Carpenter and Stephen Grossberg. Art 2: Self-organization of stable category recognition codes for analog input patterns. Applied Optics, 26(23):4919-4990, 1987.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 22405-22418 22418. Curran Asso- ciates, Inc., 2021. URL URthhttss://prooceddnss wwwp::eeeeeeddnn.n.ennennerreeee wtt.iiss..ppppppee htcpppapr__pe files/ (aw.le//pppe///// pptps//parr/////1llee bcb41 Coc21lccccccss.344aa1777772222a0
10

---

Published as a conference paper at ICLR 2026
Sungmin Cha and Kyunghyun Cho. Hyperparameters in continual learning: A reality check. Transac- tions on Machine Learning Research, 2025. ISSN 2835-8856. URL BtR.htttps//pperrevieew net/
foorum/id=n11RRXmbAZ
Sungmin Cha, Hsiang Hsu, Taebaek Hwang, Flavio P Calmon, and Taesup Moon. Cpr: classifie- projection regularization for continual learning. arXiv preprini arXiv:2006.07326, 2020.
Sungmin Cha, Sungjun Cho, Dasol Hwang, Sunwon Hong, Moontae Lee, and Taesup Moon. Rebal- ancing batch normalization for exemplar-based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20127-20138 2023.
Vivek Chavan, Paul Koch, Marian Schlüter, and Clemens Briese. Towards realistic evaluation of industrial continual learning scenarios with an emphasis energy consumption and computational footprint. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 11506-11518, 2023.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Long live the lottery: The existence of winning tickets in lifelong learning. In International Conference on Learning Representations, 2020.
Dong Kyu Cho, Jinseok Yang, Jun Seo, Seohui Bae, Dongwan Kang, Soyeon Park, Hyeokjun Choe, and Woohyung Lim. Sherpa: Leveraging neuron alignment for In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, volume 3, 2024.
fine-tuning.
Dong Kyu Cho, Inwoo Hwang, and Sanghack Lee. Peer pressure: Model-to-model regularization for single source domain generalization. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 15360-15370, 2025a.
Dongkyu Cho, Miao Zhang, and Rumi Chunara. Expert-guided clinical text augmentation via query-based model collaboration, 2025b. URL htt.:ttss//aaaii.rrraaa///////// 21530
Jack Chong, Manas Gupta, and Lihui Chen. Resource efficient neural networks using hessian based pruning. arXiv preprint arXiv:2306.0700 07030, 2023.
Shibhansh Dohare, Fernando Hamaadez-Garriaa Qingfeng Lan, Parash Rahman, Rupam Mah- mood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026): 768-774, 2024.
Yunshu Du Wojciech M Czarnecki, Siddhant M Jayakumar, Mehrdad Farajtabar, Razvan Pascanu, and Balaji Lakshminarayanan. Adapting auxiliary losses using gradient similarity, arXiv preprint arXiv: 2018.
Sebastian Dziadzio, Vishaal Udandarao, Karsten Roth, Ameya Prabhu, Zeynep Akata, Samuel Albanie, and Matthias Bethge. How to merge your multimodal models over time?, 2024. URL
Paul Escande, On the concentration of the minimizers of empirical risks. Journal of Machine Learning Research, 25(251):1-53, 2024.
Vivek F Farias and Adam D Jozefiak. Self-normalized resets for plasticity in continual learning. arXiv preprint arXiv:2410.2009g, 2024.
Massimo Fornasier, Lorenzo Pareschi, Hui Huang, and Philippe Sünnen. Consensus-based optimiza- tion on the sphere: Convergence to global minimizers and machine learning. Journal of Machine Learning Research, 22(237):1-55, 2021.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2019. URL httts: http://axxiv..og//bbbbb/b/sa wwwg/bs/////a/ 803. 03635
Alexandre Galashov, Michalis K. Titsias, András György. Clare Lyle, Razvan Pascanu, Yee Whye Teh, and Maneesh Sahani. Non-stationary learning of neural networks with automatic soft parameter reset, 2024. http:/::///aaxvvorr//a//aa/
11

---

Published as a conference paper at ICLR 2026
Siavash Golkar, Michael Kagan, and Kyunghyun Cho. Continual learning via neural pruning, 2019.
URL https://arxiv. org.orgg////..ooo /tt///b/119/0 1903.. 04476
Annette Rios Gonzales, Nicolas Spring, Tannon Kew, Marek Kostrzewa, Andreas Säuberli, Mathias Müller, and Sarah Ebling. A new dataset and efficient baselines for document-level text simplifica- tion in german. In Proceedings of the Third Workshop on New Fronliers in Summarization, pp.
152-161, 2021.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and Abhishek Kadian et al. The Ilama 3 herd of models, 2024. URL httphttsss//a...otrrrrrrr/
2407 21783
Md Yousuf Harun, Jhair Gallardo, Tyler L. Hayes, and Christopher Kanan. How efficient today's 's continual learning algorithms? Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 2431-2436, June 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Yebowen Hu, Tim Ganter, Hanich Deilamsalehy, Franck Demoncourt, Hassan Foroosh, and Fei Liu. Meetingbank: A benchmark dataset for meeting summarization. arXiv preprint arXiv:2305. /7529, 2023.
John Hutchinson. Second fundamental form for varifolds and the existence of surfaces minimising curvature. Indiana University Mathematics Journal, 35(1):45-71, 1986.
Dongseong Hwang. Fadam: Adam is natural gradient optimizer using diagonal empirical fisher information, 2024. URL / 12807
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint 0rXiv:2212.04008 2022.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Av- eraging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05400 2018.
Dong- Hwan Jang, Sangdoo Yun, and Dongyoon Han. Model stock: All need is just few fine-tuned models. In European Conference on Computer Vision, pp. 207-223. Springer, 2025.
Young Kyun Jang, Dat Huynh, Ashish Shah, Wen-Kai Chen, and Ser-Nam Lim. Spherical linear interpolation and text-anchoring for zero-shot composed image retrieval, 2024. URL https //arxiv.org/abss wteaabb//222..000 00571,
Haeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jaehong Yoon, Mark Hasegawa-Johnson, Sung Ju Hwang, and Chang D Yoo. Forget-free continual learning with winning subnetworks. In International Conference on Machine Learning, pp. 10734-10750. PMLR, 2022.
Simran Kaur, Jeremy Cohen, and Zachary Chase Lipton. On the maximum hessian eigenvalue and generalization. In Proceedings on, pp. 51-65. PMLR, 2023.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7482 7491, 2018
Grigory Khromov and Sidak Pal Singh. Some fundamental aspects about lipschitz continuity of neural networks. In The Twelfth International Conference on Learning Representations, 2024.
Chris Dongjoo Kim, Jinsco Jeong, and Gunhee Kim. Imbalanced continual learning with partitioning reservoir sampling. In European conference on computer vision, pp. 411-428. Springer, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL
https://arxivv http:///wxxiv.oo/ab////a/s/ www.//gg////aaoo. 14t.s/112.699900
12

---

Published as a conference paper at ICLR 2026
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Grabska-Barwinskee et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3522) 2017.
Jędrzej Kozal, Jan Wasilewski, Bartosz Krawezyk, and Michal Wožniak. Continual learning with weight interpolation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4187-4195, 2024.
Seanie Lee, Hae Beom Lee, Juho Lee, and Sung Ju Hwang. Sequential reptile: Inter-task gradient alignment for multilingual learning. arXiv preprint arXiv:2 10.02600, 2021.
Sebastian Lee, Stefano Sarao Mannelli, Claudia Clopath, Sebastian Goldt, and Andrew Saxe. Maslow's hammer for catastrophic forgetting: Node re-use vs node activation, 2022. URL
09029
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
Yan-Shuo Liang and Wu-Jun Li. Adaptive plasticity improvement for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 7816-7825, 2023.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint 1810.05200 2018.
Pan Lu,, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102. 04664, 2021.
Daniel Marezak, Bartlomiej Twardowski, Tomasz Trzciński, and Sebastian Cygert. Magmax: Lever- aging model merging for seamless continual learning. In European Conference on Computer Vision, pp. 379-395. Springer, 2025.
Imad Eddine Marouf, Subhankar Roy, Enzo Tartaglione, and Stéphane Lathuilière. Weighted ensemble models are strong continual learners. European Conference on Computer Vision, pp- 306-324. Springer, 2025.
Marc Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evaluation image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513-5533, 2022.
Gabriele Merlin, Vincenzo Lomonaco, Andrea Cossu, Antonio Carta, and Davide Bacciu. Practical recommendations for replay-based continual learning methods. In International Conference on Image Analysis and Processing, pp. 548-559. Springer, 2022.
Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The stability-plastiiie dilemma: Inves- tigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in Psychology, 4:504, 2013.
Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. Numglue: A suite of fundamental yet challenging mathematical reasoning tasks. arXiv preprint arXiv:2204.0500 2022.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation 18771
for neural network pruning, 2019. URL
13

---

Published as a conference paper at ICLR 2026
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019.
01703
Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puneet K Dokania, Philip HS Torr, Ser-Nam Lim, Bernard Ghanem, and Adel Bibi. Computationally budgeted continual learning: What does matter? In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp- -7987 2023.
Sylvestre- Alviss Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning, 2017. URL htlttp:://axxiw.orr/aaaaa/ 07725
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning, In Advances in Neural Information Processing Systems, volume 32, 2019.
Karsten Roth, Vishaal Udandarao, Sebastian Dziadzio, Ameya Prabhu, Mehdi Cherti, Oriol Vinyals, Olivier Hénaff, Samuel Albanie. Matthias Bethge, and Zeynep Akata. A practitioner's guide to continual multimodal pretraining. arXiv preprint arXiv:2408. 14471, 2024.
Agam Shah, Suvan Paturi, and Sudheer Chava. Trillion dollar words: A new financial dataset, task & market analysis. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 6664-6679, 2023.
Sara van de Geer and Martin J Wainwright. On concentration for (regularized) empirical risk minimization. Sankhya A, 79:159-200, 2017.
Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv: 2019.
Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for clas-iincremental learning. In European conference on computer vision, pp. 398-414. Springer, 2022.
Liyun Wang, Xingxing Zhang, Hang Su, and Jun Zhu, comprehensive survey of continual learning: theory, method and application. IEEE Transactions Puttern Analysis and Machine Intelligence, 2024a.
Maorong Wang. Nicolas Michel, Ling Xiao, and Toshihiko Yamasaki. Improving plasticity in online continual learning via collaborative learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23460-23469, 2024b.
Xiao Wang, Yuansen Zhang, Tianze Chen, Songyang Gao, Senjie Jin, Xianjun Yang, Zhiheng Xi, Rui Zheng, Yicheng Zou, Tao Gui, al. Trace: A comprehensive benchmark for continual learning in large language models. arXiv preprint arXiv:23 06766, 2023.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison. Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's ss transformers: State-of-the-aar natural language processing, 2020. URL htttss://axxiv.rr//bbb///// 03777
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Loper Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp. 23965-23999 PMLR, 2022.
Yue Wu. Yinpeng Chen, Lijuan Wang, Yuancheng Ye. Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 374-382, 2019.
14

---

Published as a conference paper at ICLR 2026
Prateck Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties- meesinggging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024.
Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3014-3023, 2021.
Enneng Yang, Zhenyi Wang. Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. arXiv preprint 1rX1v:2310.02575 2023.
Md Yousuf Harun, Jhair Gallardo, Tyler L. Hayes, and Christopher Kanan. How efficient are today's continual learning algorithms? arXiv e-prints, pp. arXiv-23000 2200, 2023.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning. 2024.
Shixing Yu, Zhewei Yao, Amir Gholami, Zhen Dong, Sehoon Kim, Michael W Mahoney, and Kurt Keutzer. Hessian-awant pruning and optimal neural implant, 2021. URL https:://axxiv.orgg abs/2101.0890
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 5824-5836, 2020.
Yipeng Zhang, Laurent Charlin, Richard Zemel, and Mengye Ren. Integrating present and past in unsupervised continual learning, 2024. URL
19132
Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimination and fairness in class incremental learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 13208-13217, 2020.
Chenye Zhao, Yingjie Li, and Cornelia Caragea. C-stance: A large dataset for chinese zero-shot stance detection. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 13369-13385 2023.
Da-Wes Wei Zhou, Fu- Yun Wang, Han-Jia Ye, and De-Chuan Zhan. Pycil: A python toolbox for class- incremental learning, 2023a.
Da-Wa Zhou, Wi- Wang, Han-Jia Ye, and De-Chuan Zhan. A model or 603 exemplars: To- wards memory-efficient slass--nnreeentta learning. In The Eleventh International Conference Learning Representations, 2023b.
Da-Wei Zhou, Qi- Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Class- incremental learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12)99851-9873 2024.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01877 2017.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 ABUNDANT EXEMPLAR MEMORY REGIME
Summary. In the abundant exemplar memory regime, the replay buffer becomes excellent proxy for the union of past tasks, which (1) mitigates forgetting (see Appendix [AI.I] but (2) reduces the effective learning signal for novel information (lask), causing high node-reuse and low plasticity (see Appendix ). Section emmprricllly quantified this trade-off across different replay ratios on CIFAR- 100 and ImageNet-
A.1.1 STABILITY: ABUNDANT EXEMPLAR MEMORY MITIGATES CATASTROPHIC FORGETTING
Forgetting (CNN)
Forgetting (CNN)
Forgetting (CNN)
DER TOSTER
Forgetting (CNN)
TOSTER
WA
Repby
33
3.
200
470
12
264
Memory Size
Memory Size
CIFAR-100
(h) ImageNet- 100
Figure 4: The impact of exemplar memory size on catastrophic forgetting. Increased memory drastically reduces the forgetting between tasks, while it persists.
Next, we discuss how a sufficiently large exemplar memory buffer M approximates previous tasks' distributions, thus reducing catastrophic forgetting.
Ideally, we could train jointly on tasks 1,-", ,, the obtained model parameter 0j. would minimize the risk:
(1) Σ [=,r,[((0:0]]
which minimizes forgetting by design. By joint-training on all task data up to task t, would find 0i., = arg (0)) with no conventional notion of forgetting between tasks.
Similarly, with large enough exemplar memory, the replayed data for previous tasks closely approximates their true distributions (P, Pj for all j < t). Therefore training Dj U M1a-11 (its distribution eq. 2 yields a risk
(9) AEp, [[(0:=)] (1 +1- Bp, [((0:2)]
(8)
where ity indicates the empirical importance (i.e., size) of the tasks. Here, we may bound /2, (0)
R1: (0) < e under standard assumptions (c.g., Lipschitz continuity in 0 (Khromov & Singh, 2024). use of representative samples during empirical risk minimization). with shrinking as the replay buffer size K increases, leading P3 2 Ps. Assuming R1: is p-strongly convex in neighbourhood of 0i (i.c., locally strong convex), this risk gap implics |0 fi S /2e/Ui (Fornasier et al. 2021 Escande 2024). Intuitively, if the two risk surfaces are proximate, their minimizers are also close in the parameter space (Beer et al. 1992 van de Geer & Wainwright 2017).
In this sense, forgetting of a previous task j arises when 07. drastically changes its predictions on the previous task distribution Py. But if or remains near 8% (which, by definition, does well on previous tasks by training jointly), must still perform well on task j. Hence, if we measure forgetting in the parameter space as
Ay-- = Ee-P, [L(0i..: 2) =()
the measure of how replay- based parameters after task perform task j compared to the parameters after task j. Here, Aj- remains small if 0r R or.-- Naturally, since 0f is reliable minimizer on
16

---

Published as a conference paper at ICLR 2026
all tasks, the small parameter drift ensures that the performance of the model trained under abundant exemplar memory does not degrade on earlier tasks ie.. forgetting is reduced as exemplar memory becomes larger (Merlin al. 2022 Brignac al. 2023).
Experimentally, Figure 4 validates that increasing the exemplar memory size can reduce forgetting (hence improving stability). Here, forgetting is measured as the average over all previously learned tasks of the drop from each task's 's best-ever accuracy to its accuracy after the final task (Zhou et al.) 2023ab. Formally:
Forgetting
Σ km
akas
OK.T).
T
ISST
where we denote ak.j and dg,T the accuracy task & immediately after learning task i, and the accuracy on task after learning the final task T, respectively.
A.1.2
PLASTICITY: ABUNDANT EXEMPLAR MEMORY DETERIORATES MODEL'S CAPACITY TO LEARN NEW TASKS
Plasticity Loss
Plasticity Loss
Avg. Task Acc.
Avg. New Task Acc.
90
8.0
80
70
DER
100
200
300
400
500
0
100
200
300
400
500
600
Memory Size
Memory Size
(a) Plasticity Loss CIFAR-100 100
(b) Plasticity Loss ImageNct-100
Figure 5: A comparison of plasticity loss (measured using the average of the new task accuracy) across different exemplar memory sizes in the 10 task scenario of CIFAR-100 and ImageNet-100.. As memory size increases, models lose their ability to learn new tasks.
We claim that while abundant exemplar memory improves stability, it inevitably suppresses plasticity
(Dohare et al. 2024). Let ((+-1) and (()) denote the network parameters right before and right after training on task t. Denote by (z. y) V ( (t)+))();))) 1);,,, x, y) the per examexl gradient when we begin task Under the hybrid distribution of equation 2 the expected update direction is
piti [g)(
Xg (new)
+ (1 A) gl (pad)
(10)
where D. (new) (patt)
.9t _ is the gradient mean over Pt and Past. respectively. Define the gradient alignment
(new)
(past)
Pt
E 1-
(11)
9t
[gg
pastt)
using the cosine similarity (Du et al. past task data at task step t stored in the replay memory) and p(t-1)
2018 Lee et al. 2021). When the buffer is abundant, p!!)
past (i.e.,
train (i.c., the mixed train data at task step 1) close, and by definition the distribution p(t) task is similar to the distribution
p(t-1) at task t 1; hence x 1. Consequently, the effective step taken during task
wain
l| [| 17 llō.ll S n[> - (1 -x)] 19, (new)] 17 [lg[ (now) ||.
(12)
(new) and (past) are almost colinear. In the limit À - 0 (i.e. the current data is overwhelmed by replay) the
where 77 is the learning rate. Here, Equation 12 shrinks monotonically with À because gt
St
update direction collapses onto the span of previous gradients (Yu et al. 2020 Kendall et al. 2018).
yielding
[| 1(--1)
0,
(13)
2
17

---

Published as a conference paper at ICLR 2026
which encourages the network to remain dormant (i.e., loss of plasticity (Dohare al. tasks.
2024) between
This analysis connects with the node-reuse phenomenon studied in (Lee et al. 2022). which claimed that when two sequential tasks are similar-as the case of pd drain and pit under the abundant exemplar memory regime-models tend to reuse their nodes (i.e., loss of plasticity), and also aligns with the model's underperformance in training under extreme replay (i.c., full data is provided as exemplar memory) as reported in (Rolnick et al. 2019
The results of Figure 5 depict how a model's plasticity, more specifically its ability to learn new tasks, diminishes as exemplar memory becomes abundant. Here, the plasticity measured using the model's 's average accuracy on new tasks. In Section 3.2] we also provided another analysis focused on the train loss, where observe that under abundant exemplar memory, convergence becomes difficult. In the sequel, we investigate this phenomenon more deeply.
A.1.3 DISCUSSION: NEW CHALLENGES UNDER ABUNDANT EXEMPLAR MEMORY
Using exemplar memory results in a hybrid training dataset that combines current task data D, U Mi:-1 with replayed samples from previous tasks.
This combined dataset introduces several side effects. First, as discussed in Section 3.2 stability increases, hence reducing forgetting (Appendix [A.I.I]. On the other hand, plasticity worsens (Appendix A.1.2). In this section, we investigate whether the abundant exemplar memory regime poses additional issues.
First, gradient interference may occur as the gradients computed on current task samples can conflict with those from past tasks, leading to partial cancellation of updates and impeding effective learning across tasks. We view that this potentially could lead to a new form of forgetting that distorts the learned features and ultimately interfering with the model's learning process. Second, the heterogeneous nature of the hybrid dataset increases the variance of the stochastic gradient estimates, resulting slower convergence. This heightened variance necessitates either more iterations or more sophisticated optimization techniques to minimize the loss reliably. Furthermore, imbalances in task representation can arise if the replay buffer unevenly captures the diverse distributions of past tasks.
Conventional continual learning methods typically address forgetting by assuming that new data is markedly different from prior data and focusing on preserving performance on previously learned tasks. However, in the abundant exemplar memory regime, where all tasks are presented simulta- neously within combined dataset, these methods fall short. They are not equipped to handle the multi-task learning dynamics and the associated balancing issues that emerge when the model must integrate and harmonize learning signals from a diverse set of tasks. We believe a more thorough investigation is required, and set this as a key objective of our future work.
A.2 METHOD (CONTINUED.)
In this section, we elaborate our method weight space consolidation, highlighting the role of each component. The proposed method is combination of two weight-space operations. (1) ranking-basee
parameter reset (Yadav et al. 2024) and (2) weight averaging (Wortsman et al. 2022
Ranking-based weight reset. The weight re- step selects and resets redundant parameters in adapting to the new task. The aim of this pro-
Ablation Study
mixed value of the previous task model pprev and the current model value using Equation (5). Accuracy
cedure is find the minimal set of parameters that are capable of adapting to the new th task,
and reinitializing the redundant parameters to the
Regloy we restt wo ory
220 Memory Size
Y00
which helps recover learned features. In this pro- that
cess, we use a simple metric Equation uses the parameter's 's first and second moments to measure its importance (Kingma & Ba 2017 Balles & Hennig 2020. Molchanov al. 2019
Figure 6: Ablation Study on CIFAR-100. We empirically find that resetting and averaging col- laboratively benefit each other.
18

---

Published as a conference paper at ICLR 2026
Hwang 2024). In Appendix A.3 also compare alternative measures (e.g., parameter drift) and discuss their limitations.
Using this metric, we retain the top-Q% parameters that have largely drifted during the current task and reset the dormant parameters using eq. ( The idea of resetting model weights is not novel in the continual learning literature, but most have focused on improving plasticity (Dohare et al. 2024). Conversely, we improve both plasticity and stability using a mixing technique (eq. (5) integrating previous and present tasks. In appendix A.4 we provide experimental results the effect of Q, the percentage of weight reset on model performance (see fig. T. Furthermore, we study alternative variations of our reset method, namely (1) different parameter importance metrics (see Table F. (2) different reset methods and reset frequencies (see Table We find that resetting more frequently (e.g., per epoch) displays significant gains in performance especially under limited memory settings. We discuss this observation on Appendix A.4
Weight Average. The weight-averaging step aggregates various learned signals into single model, allowing for faster convergence and improved generalization (Izmailov et al. 2018). The underlying idea is that the model weights can address catastrophic forgetting, functioning as substitute for replay memory. Using this, we aim to fill the gap between the full exemplar memory setting (i.e.. joint-training and the abundant exemplar memory setting. Another motive behind this approach is the new challenges (see section 331) rooted in training with a mixture of datasets D, U.M1a-1. which generally requires more training steps for convergence. In Appendix A..4 we empirically observe that the abundant exemplar memory setting complicates training, requiring more training epochs for convergence (see Figure 2b). The weight averaging technique is also an emerging practice in the continual learning literature (Marouf et al., 2025 Kozal et al. 2024). However, such works merge the model weights after training (i.e., offline merging (Dziadzio al. 2024)), which requires the storage of multiple model weights (proportional to the number of tasks). On the other hand, our method uses moving average model that is updated during training (i.e., online merging).
Why does Weight Space Consolidation work? Our starting point is the abundant memory regime, where abundant exemplars reduce forgetting but drive the optimizer toward sharp, over-specialized minima that harm plasticity. Weight Space Consolidation is designed to counteract this effect with two complementary weight-space operations. First, targeted re-initialization of dormant parameters restores unused capacity, enabling the network to escape locally saturated directions and learn new tasks. Second, in-situ weight averaging biases training toward flatter regions of the loss landscape, a mechanism that prior work has linked to improved generalization and reduced forgetting Izmailov et al. (2018); Cha et al. (2021) 2020); Cho et al. (2024). Our experiments further show that this averaging step induces sparsity in the effective parameter usage, and sparsity is known enhance plasticity by preventing the model from over-relying on a small set of critical parameters Golkar et al. (2019): Dohare al. (2024). Together, these effects provide an explanation of why our proposed method can simultaneously recover plasticity and maintain stability in the abundant-memorg regime.
A.3 MEASURING PARAMETER IMPORTANCE
In our work, we measure a parameter's contribution to learning new task by using a moment-based score (eq. (4).. However, there are several alternative approaches we could take. this section, we investigate the alternative measures that could be used to measure parameter's importance to learning à new task.
First, we can simply use the parameter drift to measure a model's contribution to the new task, formulated as:
(14) where H[T] and [ indicates the th parameter of the current model 0 and the previous task model 8prees respectively. However, a problem with this approach is when the previous model ppres should be stored. This is critical issue in cases we wish to reset multiple times.
A = [0[]] 0pren [1]]
more principled alternative weights each parameter by the empirical Fisher diagonal:
N
Ini 0) 0))²
F
(Don log p(ya
(15)
19

---

Published as a conference paper at ICLR 2026
which captures how strongly the log-likelihood reacts to perturbations of 0(]. This idea underpins many existing continual learning methods (e.g., EWC (Kirkpatrick et al. (017). 7, MAS (Aljundi et al.
2018)). However, Fisher scores usually difficult to compute real-time and lack scalability.
Another metric we can use is the Hessian estimate (Yu al. 2021 Chong et al. 2023). Second-order methods replace Fl by Hessian diagonal estimate, which efficiently obtain with Hutchinson's trick (Hutchinson) 1986):
h
(k) o Holl))
v(0) (-,,+1111
(16)
where each Hutchinson probe costs single Hessian vector product. The absolute curvature s; [fy] can then serve as our importance score. However, similar to the Fisher matrix, Hessians are exceptionally costly to compute, especially for large models. In Section 5 we investigate the efficacy of these metrics as a parameter importance measure.
We also think of differentiating between how parameter changes (1) between tasks (2) within task.
Inter-task Parameter Drift. parameter be measured
When transitioning from one task to the next, the change in the I-th
A Ameer [4]
[0,14 (--1
where 0,[4] denotes the (sh parameter after training on the current task and 0.-1 [ represents the tth parameter after training on the previous task t - 1. A large value indicates that the parameter is highly spee-ppe,,,,, while small value suggests robustness across tasks.
Alternatively, we can analyze how a parameter evolves during the training process of single task. Let g [4] denote the value of the 1th parameter at the 1th iteration during training. Then, the intra-task parameter drift is given by
Intra-task Parameter Drift.
(
mnraa [1] 10 ( 1) 0. [4]]
(i)
This measure captures the incremental updates of the parameter as the model optimizes its perfor- mance on the current task. By comparing both the inter-task and intra-task parameter changes, we gain more comprehensive understanding of the role each parameter plays in adapting to new tasks as well as the dynamics of learning within a single task. In our future work, we will seek a more reliable metric to express parameter's behavior in the weight space.
A.4 EXPERIMENTAL RESULTS AND ANALYSIS
In this section, we provide the full results of our experiments, namely (1) CIFAR- 100, (2) ImageNet- 100, and (3) TRACE.
TOSTER
Accuracy
Time Spent
Sepaay
Our
230
DER #oSTeh
Sapbyy
Memory Size
Memory Size
(a) Acc. graph CIFAR- 100
(b) Training time spent CIFAR- 100
Figure 7: Comparison of (a) average class-incremental accuracy and (b) training time under different exemplar memory sizes in class-incremental learning for 10-task using CIFAR-100, As memory increases, catastrophic forgetting is mitigated, but training time (iee, computational cost) also grows proportionally. Note that DER. FOSTER, and MEMO are expansion- based methods (shown with dashed lines).
20

---

Published as a conference paper at ICLR 2026
Table 7: Average class-IL accuracy (%) on CIFAR-100 with varying Seemplar-memonn sizes. We report experimental results with varying memory sizes, ranging from 20 exemplars per class (a common setting in class-IL) to 500 exemplars per class (storing the entire previous dataset in CIFAR- 100. Bold highlights the best non-expansion method. We report the standard error across 5 runs.
Menury Size (the number of exemplars per class class)(nnon (nno momory hidd dara)
Method
20pe%;
40g55))
80(119))
100
2000))0))
300
(00()05))
(000
DER
63 63.95
67. 07.27=1.5
70.13
70.98
74.64g11
75.06
75.60.0.a
75.92
POSTER
21
65.58_1.5
67.67
89.01
73.58
78. 7
27.28
78.07
MEMO
61.99
06.59=1.1
70.58
71.44208
75.20
75.50
75.83
Replay
48 63
56.80
63.78g1.
66.11
71
71.6020.
73. 73.29
26.71
77.0240a
iCaRI.
49.95
57.12-1.a
64.81+1.1
66.23=1.2
72.69+0.5
73.63
75.49
76.16407
BIC
63.65 63.6540s
61. 83.13.0.8
64.7440.5
65.59
19.15
71.22::0.5
72.50
72 8343
WA
61.
66.19
66.90
71.1241.2
72.15
73.83
71.09112
Ours
62.16
60.34
20.8
67 25.0
74. 74.19:0a
75.97
77. 77.71.ua
78. 78.16;a.
C-F0R- The results of the CIFAR- 100 class-ll. experiment are reported in Table 7 Here, we validate that our method indeed the strongest among non-expansion methods, and even surpasses expansion-basee methods under abundant exemplar memory (see Figure (7)). Specifically, we see that in cases where the exemplar memory size is larger than 16% of the full data, our weight space consolidation method outperforms all non non-expansion methods, and begins to match the costly expansion-ben based methods when the exemplar memory ratio exceeds 40%. The true strength of our method lies in its training cost (see Figure 7b), where our method's 's train time is at with the cheapest baseline (Replay), while taking one-fifth the time of the expansion-basee FOSTER (Wang
et al. 2022) and con--exaansion-aaee based BiC (Wu et al. 2019).
Time Spent (min.)
DSR
Kael
POSTER ER
Accuracy
NENO
Toplay
VA
Ous
750
Memory Size
Memory Size
(a) Acc. graph on ImageNet-100
(b) Training time spent on ImageNet-100
Figure 8: Comparison of (a) average class-incremental accuracy and (b) training time under different exemplar memory sizes in class-incremente learning for 10-task using ImageNet-100.
Table 8: Average class-IL accuracy (%) on ImageNet- 100 with varying exemplar-memon sizes. Bold highlights the best non-expansion method, We report the standard error across 5 runs.
Memory Size (the number of exemplars per class) / (una of momory fill dara)
Method
20(1.58)
50(45)
100gs%)
200(
3000
400
500(
60D;
DER
71 71.96
75.53_o.5
76.80
78.59
79. 79.12
79.61..o.
79.97 1o.8
80.53
FOSTER
70.14407
67.69_0.7
70.74405
76.01
79.032a5
81.87
62.79406
MEMO
66 60.35
71.12.0.6
74.26
77.89
78.7440.5
80.06
80.37
81.11 20.4
Replay
50 40.52.0.4
61.6420.6
68.49.05
73.79
76.9320.4
78.50
80.25
81.08
iCaRI.
50.3240.s
59.7620.9
67.2341.5
67.
73.57
76.8440.5
78.45220.
79.63
60.87
BIC
59 59.31407
65.98-a.r
70.63
74.14.08
76.224us
77.512077
78, 78.76
79.2940s
WA
61.44.1.
67.52 67.5220.
71.33+11
76.86
77.5040.
78.7920.4
79.63ga.
60.21
Ours
54.97+0.5
64.95
71.49
76.4320.5
78.71
80.26....
81. 81.56200.
82.6420.4
ImageNet-100. The experimental results in the ImageNet-100 100 benchmark display similar pattern. In Figure Sal we observe a gradual increase in average task accuracy as the exemplar memory size increases, which eventually saturates as it enters the abundant exemplar memory regime. The cost of training increases proportionally to the memory size, where methods like BiC or DER require substantially larger training time. On the other hand, method displays high accuracy while using roughly one-third half of the training time (see Figure 8b). For detailed results, please refer to Table
21

---

Published as a conference paper at ICLR 2026
DER
FOSTER
MEMO
Replay
iCaRL
Ours
Memory 20 (1.5%)
Memory 200 (16%)
Memory 400 (30%)
Memory 600 (46%)
79
Accuracy
60-
76-
75
78
00
600
1000
1500 1000
2000
Train Time (min.)
Figure 9: Comparison (y-axis) average class-incremental accuracy and (x-axis) training time under different exemplar memory sizes in class-incremental learning for 10-task using ImageNet-100. Note that the DER, FOSTER, and MEMO are expansion-based methods (shown with X mark).
SLERP ([W)*2)
Avg. (Offline) ([W) 7)
MagMax ([W) T) Replay ([WI
Ours [[W]
Avg. (Online) [[W]
Arithmetic ([W)
Avg. Score
0.40
0.35
0.30
80%
Memory Size
Figure 10: Comparison of average score on different exemplar memory sizes in LLM continual instruction tuning for 8-task using TRACE. The value inside the parentheses indicates the model weight complexity proportional to the number of models loaded in the VRAM.
TRACE. TRACE (Wang al. of LLMs across eight sequential domains, including science (Lu et al. 2022), policy (Shah et al.
2023 is continual instruction tuning benchmark for the evaluation
2023). meeting summarization (Hu al. (023), multilingual classification (Zhao al. 2023
Gonzales et al. 2021), code generation (Lu et al. 2021). and math reasoning (Mishra et al. 2022
Table 9 reports the LLM continual instruction tuning benchmark results in TRACE, Similar the results in the Class-IL benchmarks (c.g.. CIFAR-100, ImageNet-100), we observe a similar pattern where the accuracy grows accordance with the increase in memory size (see Figure [0]. An interesting observation is that offline merging (i.c., model merging methods that merge post-training) tends underperform, displaying lower scores while requiring the VRAM cost of storing multiple models. This aligns with the empirical results of Dziadzio et al. (Dziadzio et al.) (024), which showed that offline merging methods face difficulties in accumulating the multi-task knowledge. Compared to this, our method In addition, note that for TRACE, we use different method to measure the train cost, which is the relative VRAM usage. The relative VRAM usage is a scaled version of the training time, which measures the computation cost of the training based on the VRAM usage. This scaling required to distinguish from models that take a similar time to train, but use different numbers of GPUs. For instance, our method and Mag require similar time to train, but MagMax requires multiple (t) task vectors to loaded in the VRAM, and this is reflected in the relative VRAM usage measure. Considering this, visualize the results in Figure
Zero-Memory Setting To relate our study to exemplar- fremplar-free continual learning, we also evaluate all methods in a strict zero-memory configuration on CIFAR-100, where no exemplars are stored in the buffer. We report the results in Table TO This setting corresponds to the idealized regime suggested in some recent CL work, but lies outside our main assumption of exemplar-based CL with sufficient memory. Consistent with our analysis, removing exemplars leads to severe performance degradation across all algorithms: while most methods reach 94-74% average class-IL accuracy in the 40% memory regime (Table 7), their accuracy drops to 23-53% under cero-memory. DER and MEMO achieve the highest periormance (53.37% and 43.98%), and our method still improves over Replay (29.83% vs. 26.07%), but the gap the sufficient-ment memooy regime remains large. These
22

---

Published as a conference paper at ICLR 2026
Table General language understanding and reasoning scores on TRACE with varying exemplar- memory sizes. We report the standard error across 5 runs.
Method
Size
R 0
stoeed
Runtine (min.
C-STANCE
POMC
Acc.
MustingRank
ROUGE-L
Siuularity
1y150
Sclenos44
NumtH NunKil&/F.p
Aec.
Acc.
30Moutte
Ang.
0.35
anas
a.35
0.32,ars
2.39
0.31
Reploy
D.4
0:31anar
0.38ax
0.7anos
0.35
0.39
0.39ants
0.37asr
021an
0.56mm
0.37
0.2
5751
040
0.39
SLERP
0.6
0.4
47219
0.26ar16
0.33
0.43
0.40
0.5
0.45
0.38
0.37
a.37
a.38
0.41
0.2
374'5
0.33
0.37
0.40
0.4
1.42
0.35
0.55
0.28
0.43
0.44
0.41
(Online)
0.6
491'4
0.50
0.55
0.35asus
0.Maaus
0.52.
0.39
0.64
0.3
534'5
0.35
u.50
0.16asas
0,36
0.39
0:2
177'
0.29
0.Maars
0.5
0.35
Ang.
0.4
630'6
0.38at
0.3
0.40zams
04
a.37
0.38
(OFliine
066
479'3
0.44gn
Ma
0.36
0.38
0.S
0.36gams
0.45
03
535'4
0.50gm
0.35
0. 0.Wagne
0.49
0,37
0.41
0.2
574'6
an
a.3
a.2
0.52
g.
0.36
Tesk-Aith.
0.4
0.6
4276
474'9
a
a.20
0.38
0.39
0.50
0.59
0.2
0.32
042ant
1.
0.53_10
0,40an13
0.39
0.45
0.
52311
0.38
0.39
0.57
0.4gas
14
0.38
0.43
02
573'3
0.4
0,25 sace
0.26
0.3
0.3
0,48
L.39
0.37
MigMax
0.6
0.4
424'8
669'9
046
0.59
04l1aa
0455
0.37a
0.39
0.32.n0e
0.26
0.43
0.35
0.3
0.4
0.45.
0,40
0.40ga:3
0.39
0,42
0.
525'6
0.32
0,54
0.23
0:25
0.3
1.29
0,51_am
0.39
0.36
0.3
570'9
2.39
0.33
0.36
0.51
a.2
2.40
0.56_am
0.39
0.38
Ours
04
06
422'6
468'8
Ddlanss
0.57amm
0.48
0.38aree
0.35
0.55201e
0.522m1e
042aem
0.8aamm
0.38
0,30
0.S5
0.39
0.392nca
0.46
0.42
03
522'9
0.46ges
0.47
0.37gmr
0.55
0466gom
0.36g
0.53
0.40gees
0.45
results support our claim that exemplar-free CL is substantially more challenging and currently yields accuracy that is insufficient for many real-world deployments, whereas allocating a moderate exemplar buffer enables practical performance gains that our cost-efficient method is designed to exploit.
Weight Reset Experiment
80
Accuracy
70
With Avg.
65
60
Without Avg.
55
0
20
40
60
80
100
Keeping Top-Q% Parameters
Figure 11: Weight reset experiment on CIFAR-100. Most parameters (80%) are redundant in learning new tasks. Weight Avg. boosts sparsity, retaining performance with fewer (10%) parameters.
Sparsity analysis after reset. Next, we examine how much of the model contributes to learning new tasks. In Figure [T] we vary the retain rate Q in the reset step. We find that resetting up to 80% of pa- rameters yields minimal degradation in accuracy, suggesting that only small subset of weights are actively involved in continual learning. This aligns with findings in sparse training and pruning (Frankle & Carbin 2019 Chen al., 2020). We observe that weight averaging helps retain performance under extreme resets (e.g., Q-10%), sug- gesting that it encourages robust, sparse models that do not rely on small set of critical parameters.
Table 10: Average class-IL. accuracy (%) on CIFAR-100 under zero memory.
Memory Size
Method
DER
53.37
FOSTER
26.43
MEMO
43.98
Effect of Sampling techniques In this section, we study the effect of batch sampling in exemplar-based continual learning. Prior work has examined the role of exemplar selection and sampling, with early re- sults suggesting that sophisticated schemes such reservoir sampling may improve performance under severely constrained memory Kim et al. (2020), but more recent large-scale studies report that random
Replay
26.07
iCaRL
29.03
BiC
28.29
WA
42.10
Ours
29.83
23

---

Published as a conference paper at ICLR 2026
Table 11: Average class-IL accuracy (%) on CIFAR-I00 with varying semplar-memory sizes using reservoir sampling. Memory Size indicates the number of exemplars per class. The value inside the parentheses indicates the gap between the default random sampling and the reservoir sampling results. Bold highlights the best non-expansion method.
Memory Size Size of memory to full data)
Method
(0(4%)
(40(4
DER
(4022))
(4.83(+0.19)
FOSTER
(-0.25)
73.71(+0.18)
MEMO
62.36(+0.37)
(3.40(-0.31)
Replay
49.32(+0.69)
71.66(+0.06)
iCaRL
(0.34(+0.30)
72.89 (+00.00
BiC
53.86(+0.21)
69.27 (+0.12)
WA
61.68 (-0.6))
71.65 71.65 (++0233)
Ours
(2.49(+0.33)
74.63
Table 12: Average class-IL accuracy (%) on CIFAR- 100 with varying exemplar- sizes
longer task sequences (T=20). Memory Size indicates the number of exemplars per class. The value inside the parentheses indicates the gap between the default task sequence (T=10; 10 tasks) and the longer task sequence setting (T=20; 20 tasks).
Memory Size rrati (maio of memory to full data)
Method
20(4%)
200()0%)
DER
55.42(-8.53)
(0.91(-3.73)
FOSTER
55.94(-10.22 (1.80(-1.73)
MEMO
54.61(-7.39)
70.14(-3.57)
Replay
44.73 -3.90)
70.88 (-0.72)
iCaRL
45.28
69.24,
BiC
4
68.97
WA
52.36 -8.96)
70.35
Ours
49.77
76.65 (+0.22)
and advanced sampling strategies behave similarly in practice Masana et al. (202)). We expect this trend to persist in the abundant- regime considered our work. To verify this, we re-run all methods CIFAR-100 100 using reservoir sampling for exemplar selection, ensuring that the memory buffer fairly represents past tasks. As shown in Table [T] the differences with respect to our default random sampling (values in parentheses) are consistently small, and the gap shrinks further as memory size (i.e., number of exemplars per class in slass-IL) increases from 4% 40%. This indicates that, when sufficient memory is available, the choice of sampling strategy has only a marginal impact on overall performance.
An alternative way to improve plasticity is to enforce fixed ratio of current-task data within each mini- batch by concatenating current and replay examples at the batch level. While such design can indeed enhance plasticity, it directly reduces stability: aggressively prioritizing current data accelerates catastrophic forgetting of previous tasks and increases GPU cost due to more constrained batch construction. Thus, batch-level concatenation complementary but cannot replace our approach, which targets the prasticity-stabille trade-off through weight-space operations rather than through carefully engineered batches. Moreover, as highlighted by Dohhaee et al. (2024), loss of plasticity is a general phenomenon in deep continual learning; our Weight Space Consolidation provides robust way to restore plasticity that is largely independent of specific batch-design or sampling heuristics.
Effect of Longer Task Sequences To assess how our findings scale with the length of the continual learning stream, further evaluate all methods longer task sequence with T 20 tasks
24

---

Published as a conference paper at ICLR 2026
CIFAR-106 while keeping the total data and deemplar-memor budget fixed. As shown in Table average class-IL accuracy systematically degrades when moving from the default T = 10 setting to
T = 20, and this degradation is particularly pronounced in the low-memory regime (20 exemplars per class), indicating that longer task sequences amplify both forgetting and the loss of plasticity. Nonetheless, in the sufficient-memrry setting (200 exemplars per class, i.e., 40% memory), our method maintains strong performance and exhibits a comparatively smaller drop than many baselines, demonstrating that our Weight Space Consolidation remains effective even the task sequence becomes longer.
Additional Experiments. Lastly, we show that under abundant exemplar memory, convergence becomes difficult, similar to a multi-task learning setting. The results are illustrated in Figure 2 Specifically, we analyze the training loss of the model under two cases. (1) Continual Learning: train the model across 20 tasks sequentially, with abundant exemplar memory. (2) Full retraining (Joint Training): train individual models for each task using all known task data. For this experiment, we used the CIFAR-100 dataset divided into 20 subtasks. In Figure 2b we can see that under abundant exemplar memory, converging to the training data becomes difficult, especially for the continual learning model. While simple solution would be to extend the training epochs for convergence, it would collide with our aim for cost-efficiency. On the other hand, joint-trained models relatively converge better. This result aligns with our conjecture on plasticity (see Section 3.2), the results of (Dohare et al. 2024) that a model sequentially trained on different tasks (i.e., continual learning) suffers loss of plasticity (i.e., model's ability to learn new tasks), well as the results of (Rolnick (et al. 2019). which demonstrated that learning a new task becomes difficult under extreme (i.e., full ratio) replay memory settings. We believe a thorough investigation of this new phenomenon is required.
A.5 EXPERIMENTAL DETAILS
In this section, we report the experimental details of our experiments. all our class-IL experiments, we have used the PyCIL (Zhou et al. (023a) library, which allows easy replication. We followed the standard training hyperparameters of the PyCIL library, which are fixed across experiments. For the LLM continual instruction tuning experiments, we have used the TRACE (Wang et al.) 2023 library. We followed the default training hyperparameters of the TRACE library. Regarding unique hyperparameters, the average interval j was set 5, and the warming epoch nwarm was set as 25% of the total training epochs (default set as 70 under the PyCIL setting). j and Awwrrm were selected using grid search. The hyperparameters searched in the CIFAR-I00 benchmark were applied without modification to the ImageNet- 100 experiments. In the LLM experiments, the hyperparameters were selected in the 0.2 (20%) memory setting and applied the other settings. Note that in the TRACE setting, only one epoch is provided in the replay stage, hence, the average interval was set as 20 iterations, not epochs. Please refer to Section 4 a better understanding of each hyperparameter. Regarding the model architectures, we used ResNet-32 for the CIFAR- 100 and a ResNet- 18 for the ImageNet- 100 experiments, following standard settings in PyCII. For the TRACE experiments, we used a Llama-3.2-1 IB model. Lastly, regarding the computing resources, we used a single NVIDIA RTX 6000 GPU for all class-IL experiments. For the LLM experiments, we used three V100 GPUs. For our experiments, we used the 2.2.1 version of Pytorch (Paszke et al. 2019).
A.6 RELATED WORKS (CONTINUED.)
Weight Space Operations. Recent works have shown that manipulating model parameters directly with weight-space operations (c.g., model merging (Wortsman et al. (2022) can handle multi-task learning YYu et al.] 2024) and continual learning (Marouf et al.) 2025 Marezak et al.) (025) in more principled way. These techniques usually intervene post-training by merging the weights of different models e.g., (Yadav et al.) 2024) suggested selective merging algorithm that mitigates the interference between different models, while (Ilharco et al.) 2022) showed that arithmetic operations on the weight space can edit the model without training. Unlike these post-training interventions Cho let al. (2025b), our approach manipulates the model's weight space amidst training (Izmailov et al. 2018: Jang al. 2025) without storing multiple model parameters, aiming for cost-effective editing of the continual learner. Another relevant yet overlooked topic is the effect of weight space operations on model attributes e.g., loss landscape (Li et al. 2018. Kaur al. 2023) and plasticity (Dohare et al.)
25

---

Published as a conference paper at ICLR 2026
2024), that contribute to continual learning and generalization. This work empirically investigates various aspects of the model to study their effect on the model's ' ability to handle distribution shifts. In the continual learning literature, several works have adopted weight-space operations to obtain multi-task solutions without full retraining. For instance, (Kozal et al. 2024) has suggested the use of weight averaging techniques for continual learning, and (Marczak al. 2025) has extended the idea using task arithmetic. However, these approaches merge models post-training and require the storage of multiple model weights during training. On the other hand, our approach utilizes weight-space operations amidst training. without the redundant storage of multiple model weights. We view this as an important difference modern settings where the model size is exponentially growing.
A.7 LIMITATIONS
While method demonstrates strong performance and cost-efficiency under abundant exemplar memory, assumes access to a representative subset of past data, which may not always be feasible in privacy-sensitive streaming- -nly environments. Additionally, our analysis primarily focuses on class-incremental learning and continual instruction tuning with relatively clean task boundaries. Future work may explore how the proposed weight-space strategies generalize more complex settings such as task-agnostic CL, online CL, or continual reinforcement learning.
A.8 SOCIETAL IMPACTS
This work focuses on improving a general capability (e.g., continual learning) of machine learning models, and thus does not directly relate to mention and consider the computational cost of deploying models. We believe that the energy consumption issue of modern machine learning models is an important topic, and in that sense, work on cost-efficient learning algorithms can indirectly contribute to building a sustainable practice for the training and deployment of artificial intelligence.
negative societal impacts. However, do
A.9 ASSETS
In our work, many existing assets were used. For the implementation of the models and the learning algorithms, we have used the Pytorch (Paszke et al. 2019) (BSD-3 license) and Huggingface (Wolf)
al. (020)) libraries (varies each library. For instance, the core Transformers library uses Apache 2.0 license). All of the datasets we have used are public datasets. For instance: CIFAR-100 (MIT License), ImageNet- 100 (Custom, Non-commercial Research Only, has separate terms of use). Regarding the language datasets in TRACE: C-STANCE (CC BY-NC 4.0) ), FOMC BCC NC-SA 4.0), MeetingBank (CC BY-NC-SA 4.0), Py150 (MIT License), ScienceQA (MIT License), NumGLUE--cm (MIT License), NumGLUF-de (MIT License), and 20Minuten (CC BY-NC-SA 4.0).
26


Reviews:
[{"review_id": "oFL9LCo5eM", "reviewer": "Reviewer_pTBZ", "summary": "Brief Summary: The paper tackles the task of continual learning. The authors suggest gpu compute cost is the main constraint and find the problem is with plasticity where bottleneck is on struggling to learn new tasks. To address this, authors propose Weight Space Consolidation involving rank-based parameter resets and weight averaging. Experiments are conducted on both image-classificaiton tasks (cifar-100) and llm instruction tuning (trace) showing improvement at significantly reduced costs.", "questions": "Q1. Can plasticity be measured as a metric on an eval set? Currently, plasticity loss is provided in the graph.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Pros:\n\n1. The overall point about gpu compute constraints being more than storage is good. Exploration of such middle-ground strategies makes sense to me. The core idea of reseting certain parameters for plasticity also is interesting.\n\n2. In the high-sample regime, the proposed method almost always outperforms existing baselines by 2-3 points. \n\n3. The paper has nice ablation experiments such as comparing replay with reset (table 3), reset strategies (table 5). The plasticity loss experiment (Fig2) in particular suggest the issue with lower plasticity with increasing size of past examples. The additional experiments in the supplementary are appreciated.", "weaknesses": "Cons:\n\n1. My main concern is with the framing of the paper for continual learning and its potential application. The point about GPU cost being dominant with storage is fine, but the main problem is that the data itself might not be available in the first place. So storage was never the real issue, it is access to data. For a practical example, assume we have llama-3.2b instruct model but we don't know what data was used in the base to instruct training. Here, the authors are essentially assuming we have access to the underlying data, which is not the case. A few other points:\n\n(i) While storage cost is not a problem, the cost of downloading setting up s3 buckets high throughput speed for gpu access are all real costs. These need to be highlighted. \n\n(ii) If the authors are motivating it based on cost, some experiments on exact cost saved for practical reference should be provided.\n\nThis is not to say the original point about GPU vs storage cost isn't correct, but that storage is not the only factor, data access itself is a big one. The entire argument essentially dilutes the author's novelty.\n\n2. It seems a naive full-fine-tuning baseline is missing in Table 2 ? Difference between proposed method and full-fine-tuned on the same set would be very helpful. \n\n3. Authors primarily explore task-based paradigm only, but it seems the method might be more impactful in task-free settings [Ref1]. \n\n4. For LLM datasets, authors only consider TRACE dataset, so tough to know the generalizability.. Given that authors are exploring image-based and llm-based separately, it might be worth repeating experiments on multi-modal datasets as well such as on VisCOLL [Ref2]. \n\n5. (Minor) Some qualitative visualizations would be very helpful.\n\n6. (Minor) For Trace experiments, the training hyper-parameters are underspecified. What is the fine-tuning method for llama-3.2-1B? Some experiments using LoRA adaptors would also be interesting. \n\n---\n[Ref1]: Aljundi, Rahaf, Klaas Kelchtermans, and Tinne Tuytelaars. \"Task-free continual learning.\" In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11254-11263. 2019.\n\n[Ref2]: Jin, Xisen, Junyi Du, Arka Sadhu, Ram Nevatia, and Xiang Ren. \"Visually grounded continual learning of compositional phrases.\" arXiv preprint arXiv:2005.00785 (2020).\n\n\n---\n\nOverall Rating: 4/10\n\nThe main framing of gpu vs storage cost is not really novel. While the authors get 2-3 points improvement in high-sample regime, it is unclear how interesting that is. Authors miss key experiments such as full-fine-tuning baselines, only considering task-based not task-free, and only one trace dataset for llm experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mfRCOO570K", "reviewer": "Reviewer_JQyd", "summary": "The authors challenge the assumption of traditional CL. Instead, they argue that, instead of the storage, the GPU time is the main bottleneck. They investigated the scenario where memory is sufficient enough to mitigate forgetting, but full retraining from scratch remains the main challenge. As the authors have discovered that models become biased toward prior tasks and struggle to learn new tasks, they propose Weight Space Consolidation, a lightweight method that combines rank-based parameter resets to restore plasticity with weight averaging to enhance stability.", "questions": "See \"Weaknesses.\" What are the technical contributions (s) of this approach, if the A and B components were previously proposed and employed by existing works?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The paper investigates a new paradigm/scenario that challenges the previous assumptions. Such new thinking is always valuable.\n+ The mathematical formulations are rigorous and I discovered zero mistakes there.\n+ The new paradigm is not just an \"assumption,\" they have evidence (Section 3) to empirically demonstrate that the new assumption is valid.", "weaknesses": "- While the mathematical formulation is rigorous, the theoretical foundation on why this new approach would work is lacking.\n- The proposed approach reads like A (rank-based parameter resets) + B (weight averaging). A + B isn't always necessarily bad, but both A and B are the results of prior work, so what's the technical contribution here with this approach?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XX4wGIPU9p", "reviewer": "Reviewer_6HA7", "summary": "Memory-based methods in Continual Learning focus on learning a sequence of tasks with help of a buffer, which store samples from previous tasks and use them when learning the current tasks. Normally, this buffer is limited in the amount of samples they can store due to some constraints of the environment. In this paper, the authors challenge this constraint mainly because storage costs much less than training time. However, as we increase the samples we store, models could face a different challenge, and instead of having forgetting (too much plasticity) this shifts to a high stability and not learning the current task enough. Considering all this, this paper proposes Weight Space Consolidation, which combines a reset strategy to improve plasticity and weight averaging to enhance stability. Experiments on multiple modalities show good performance of the proposed method as they increase the memory budget. Also, multiple ablation experiments help understand why and how the method performs.", "questions": "- Why not compare against better memory-based methods, like for example:\n    - Buzzega, Pietro, et al. \"Dark experience for general continual learning: a strong, simple baseline.\" Advances in neural information processing systems 33 (2020): 15920-15930.\n        - It has been shown to scale well with the number of tasks.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is well motivated and written. The authors raise the challenge that, in current systems, memory is not the most important constraint, as GPU time is more costly. Presenting references and numbers, the paper encourages researchers not to focus on minimising memory size.\n    - However, there are scenarios where having a small memory is required, such as when resource constraints are imposed or when the privacy of previously learned data is an issue. These are not mentioned or explained in the paper.\n- Along with raising the challenge and presenting a new scenario, the authors analyse the limitations of naively increasing memory size and propose a new approach to increase plasticity and enhance stability in this new scenario.\n- The experiments and results clearly help to demonstrate what is discussed in the text. Experiments across multiple benchmarks and modalities provide robustness to the results, and ablation experiments help understand the method's limitations and provide further insights into how it works.", "weaknesses": "- A common approach to increase the plasticity of memory-based methods is to concatenate buffer samples with current-task samples at the batch level. This is different from what is shown in the paper, where the full batch is sampled from the intersection of the current data and the buffer. As shown in the paper, this last approach suffers from plasticity because it is least likely to sample data from the current task; however, by concatenating at the batch level, there is no plasticity problem.\n    - How does this sampling method affect the scenario, results and the proposed method? \n    - Concatenating at the batch level can be more costly (in terms of GPU time), but it may mean not keeping a copy of the model in memory, which can increase the batch size.\n- Figure 2b is unclear. The orange line makes it impossible to see the blue lines and compare them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZATdmURklF", "reviewer": "Reviewer_nQUU", "summary": "Traditionally, continual learning has been studied under the memory-restricted setting (i.e., completely exemplar-free or only limited buffer sizes allowed). Authors argue that this is unrealistic under modern standards: compute, and not memory-costs, are the main drivers for the high costs related to model training. Thus, authors study the setting where larger-than-common buffer sizes are used, and where instead the compute availability is the driving factor for comparison. They find that, as the memory buffer grows and thus allows storing more samples of prior tasks, the challenge in CL shift from forgetting to plasticity. Models are stable wrt old tasks, but not plastic enough to new tasks. To alleviate, they propose using stochastic weight averaging and resetting weights with relatively little gradient activity. They study their method, termed Weight Space Consolidation, under the class-incremental setting on CIFAR100, ImageNet-100, and text-based TRACE benchmark, where competitors are outperformed.", "questions": "I have several questions/ideas for improvement\n- You use a rank-based approach --> where are the ranks (i.e., matrix ranks) used here? Do you mean \"ranking based approach\"?\n- Equation 2 and 5 both use $\\alpha$ for different jobs\n- Equation 2: how is alpha annealed (l 232)\n- Equation 5: describe that the [l] are the weights to be reset\n- Figure 1: are the memory sizes per class or total? Additionally give percentage\n- L 460: Q versus q (in the top-q%)\n- Figure 3 and 405ff: how is VRAM usage measured? Do not modern frameworks pre-allocate as much GPU memory as possible (i.e., aiming at 100% GPU memory utilization)", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Strengths:\n+ novel take on memory buffers in CL\n+ simple and effective method to improve plasticity under performance-stable regimes\n+ image and text modalities tested\n+ easy-to-follow description of algorithm", "weaknesses": "- l 311 to 319: this is a central part of the proposed method, but no experimental results support this thesis\n- Paper claims less compute overhead from their method, but no tabular results indicating the overhead versus naive replay are given\n- if we can store samples of old tasks, then we could, in turn, learn longer task sequences. Paper should integrate longer task sequences (e.g., TinyImageNet split into 20 or 40 tasks)\n- Full re-training is often used as comparison, but missing from the tables\n- Its unclear how \"from scratch\" is interpreted. Is that an entirely fresh model that's trained on increasing unions of memory buffer? Then, no knowledge would be transferred by design\n- Hyperparameter tuning on ResNet32 and then using tuned params on ResNet18 seems strange (l 364). Please elaborate on this.\n- I do not agree with the definition of post-hoc merging versus in-training merging (l292ff). Post-hoc merging would be, in my understanding, merging after Training on a task, which is thus before a next task -- which is still \"during training\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)