Forget Less, Retain More: A Lightweight Regularizer for Rehearsal-Based Continual Learning
Authors: Lama Alssum
Hasan Abed Al Kader Hammoud
Motasem Alfarra
Juan C Leon Alcazar
Bernard Ghanem
Venue: Accepted by TMLR ()

Abstract: Deep neural networks suffer from catastrophic forgetting, where performance on previous tasks degrades after training on a new task. This issue arises due to the model’s tendency to overwrite previously acquired knowledge with new information. We present a novel approach to address this challenge, focusing on the intersection of memory-based methods and regularization approaches. We formulate a regularization strategy, termed Information Maximization (IM) regularizer, for memory-based continual learning methods, which is based exclusively on the expected label distribution, thus making it class-agnostic. As a consequence, IM regularizer can be directly integrated into various rehearsal-based continual learning methods, reducing forgetting and favoring faster convergence. Our empirical validation shows that, across datasets and regardless of the number of tasks, our proposed regularization strategy consistently improves baseline performance at the expense of a minimal computational overhead. The lightweight nature of IM ensures that it remains a practical and scalable solution, making it applicable to real-world continual learning scenarios where efficiency is paramount. Finally, we demonstrate the data-agnostic nature of our regularizer by applying it to video data, which presents additional challenges due to its temporal structure and higher memory requirements. Despite the significant domain gap, our experiments show that IM regularizer also improves the performance of video continual learning methods.

Published in Transactions on Machine Learning Research (01/2026)
Forget Less, Retain More: A Lightweight Regularizer for Rehearsal-Based Continual Learning
Lama Alssum" Hasan Abed Al Kader Hammoud
Motasem Alfarra Juan C Leon Alcazar
Bernard Ghanem
King Abdullah University of Science and Technology (KAUST)
"Corresponding author: lama. olssum, @kaustt. edu.sa
Reviewed on OpenReview: https: //
// opwneeveew.........mmee net/ pue//forue id- htw.://aaaaae///eeooooooooooooo0
Abstract
Deep neural networks suffer from catastrophic forgetting, where performance on previous lasks degrades after training on a new lask. This issue arises due to the model's tendency to overwrite previously acquired knowledge with new information. We present novel ap- proach to address this challenge, focusing the interscction of memory-basod methods and regularization approaches. We formulate regularization strategy, termed Informa- tion Maximization (IM) regularizer, for memory-based continual learning methods, which is based exclusively on the expected label distribution, thus making it class-agnostic. As consequence, IM regularizer can be directly integrated into varions rehearsal- based continnal learning methods, reducing lorgetting and favoring faster convergence. Our empirical vali- dation shows that, across datasets and regardless of the number of tasks, our proposed reg- ularization strategy consistently improves bascline performance at the expense of minimal computational overhead. The lightweight nature of IM ensures that it remains practical and scalable solution, making it applicable to real-world continual learning scenarios where efficiency is paramount. Finally, we demonstrate the data-agnostic nature of our regular- izer by applying it to video data, which presents additional challenges due to its temporal structure and higher memory requirements. Despite the significant domain gap, our exper- iments show that IM regularizor also improves the performance of video continual learning methods.
1 Introduction
Continual learning (CL) aims to develop models that can learn from evolving data distributions with minimal forgetting Rolnick et al. (2119). Due to the high computational and financial costs of training deep neural network models and growing concerns over privacy regulations, the applicability of CL in various real-world scenarios has become increasingly critical. For instance, video-sharing platforms such YouTube and TikTok receive millions of newly uploaded videos daily, each introducing new trends, visual concepts, and styles. In these dynamic environments, traditional training algorithms for deep learning models struggle maintain their performance due to the necessity of frequent retraining, which resouroe-intensive and impractical at scalc. CL can significantly enhance the offectiveness of modcls designed for such dynamic data streams by continually adapting previously trained models, rather than retraining them from scratch new data is made available.
In recent years, memory-based methods Rolnick et al. 2019 Buzzoga et al. 2020bb also known as rchcarsal methods, have emerged as the front-runners in CL, demonstrating better performance al mitigating forget ting compared to their regularization- based counterparts. This superior performance of rehearsal methods is attributed to the use of memory buffer, dedicated storage that retains subsct of training data from previously learned tasks. By having access subset of past samples, the model can estimate class prototypes effectively thus alleviate forgetting despite distribution shifts Rolnick et al. (2019); Rebuffi et. al.

---

Published in Transactions on Machine Learning Research (01/2026)
2017]. This ability to retain past information enables rehearsal-based approaches to maintain stability in long-term learning while adapting to new tasks.
The performance improvement of rehearsal methods over regularization methods comes at the cost of in- creased memory requirements and greater computational overhead. Experience replay methods retrain for several epochs on both the current data and memory buffer samples, effectively approximating a joint dis- tribution whenever new data becomes available. This continuous reprocessing of stored data samples not only increases computational demands but also increases training time, making it less scalable for large datasets. This computational penalty is further emphasized outside the image domain; for example, in video data, a single minute-long video recorded at 30 frames per second occupies as much memory as 1800 indi- vidual images. Despite consuming a significant storage, such video typicality represents a single instance of a class within the memory buffer, thus limiting the diversity of stored information and further challenging the efficiency of experience replay in temporal data.
In this paper, we introduce a class-agnostic regularization strategy for continual learning (CL), termed Infor- mation Maximization (IM), which operates over the distribution of network predictions Liang et al. (2020). IM is lightweight, efficient, and orthogonal to key design choices in CL algorithms, as it works across different rehearsal- methods, memory budgets, and number of tasks. IM jointly encourages prediction diver- sity and confidence, resulting in robust feature representations under distribution shifts, thereby improving generalization to past tasks while minimizing memory and computational overhead. Extensive empirical evaluation shows that IM emerges as an effective regularization technique, outperforming current regular- ization strategies tailored for the CL setting in both accuracy and retention of past knowledge. Moreover, TM is not specific to the image CL domain; we further validate its effectiveness by applying it to video continual learning setup, where it demonstrates similarly improved results in handling the challenges posed by temporal dependencies and increased data complexity.
Contributions. Our work makes two main contributions: (i) We provide a systematic evaluation of four major regularization techniques: Elastic Weight Consolidation (EWC), Synaptic Intelligence (SI), Entropy Minimization (EM), and Information Maximization (TM), applied to image continual learning. This eval- uation highlights the advantages of the proposed 1M regularizer, demonstrating its superiority in terms of both performance and overall reduction in catastrophic forgetting. Furthermore, we demonstrate that IM is orthogonal to critical design choices in CL (rehearsal-based method, memory size, and nunber of tasks), enabling straightforward integration while preserving computational efficiency. (ii) We extend our analysis beyond image-based settings by demonstrating the applicability of IM within the context of video continual learning. Given the additional complexity of temporal dependencies and larger data volumes in videos, our results show that TM maintains its effectiveness, achieving substantial gains over traditional memory- baselines while preserving computational efficiency.
2 Related Work
Image Continual Learning. In the field of image-based continual learning, numerous innovative ap- proaches have been proposed to address catastrophic forgetting. Memory-based methods, such as iCaRL, Rebufli et al. (2017). utilize incremental classifiers and representation learning to balance new and old knowledge, while GEM Lopez-Paz& & Ranzato (2117) and its more efficient variant A-GEM Chaudhry et al. (0199a) optimize gradient-bast basee episodic memory to mitigate catastrophic forgetting. Other approaches, including DER Buuzega et al. (2020b), enhance rehearsal by incorporating logit distillation, while CoPE
De Lange & Tuytelaars| (2021) leverages class prototypes to structure the latent space, and ER-ACE (aCciaa et al. (2021) modifies cross-entropy loss to address task imbalance. Recent work includes Refresh Learning Wang et al. (2024), which unifies rehearsal with selective unlearning to refresh model knowledge, and STAR Eskandar ct al. (2025), plug-and-play regularizer that leverages stability-inducing weight perturbations during rehearsal to mitigate forgetting. Regullriization-based methods aim to preserve past. knowledge by constraining weight updates, typically by identifying the importance of parameters, like Elastic Weight Con- solidation (EWC) Kirkpatrick et al. (2017) and Synaptic Intelligence (SI) Zenke et al. (2017). Architectural innovations also play crucial role continual learning, with L2P Wang al. (2022c) demonstrating the effectiveness of learnable prompts in guiding pre-trained models without relying on a rehearsal buffer, and
2

---

Published in Transactions on Machine Learning Research (01/2026)
DualPrompt Wang et al. (2022b) introducing two-level prompting mechanism for transformer-baseed ar- chitectures. These diverse approaches underscore the rapid advancements in continual learning, paving the way for more scalable and adaptable models in real-world applications.
Video Continual Learning. To mitigate catastrophic forgetting in video data, various strategies have been developed, which can be broadly categorized into regularization and memory-based techniques. While regularization methods apply constraints to preserve previous knowledge, memory-based approaches leverage data or representations from past tasks. When analyzing video continual learning, the importance of memory becomes even more pronounced due to the temporal complexity and higher dimensionality of vidco data. SMILE Alssum al. (2023) underscores this by proposing an efficient replay mechanism that stores single frame per video, emphasizing video diversity over temporal information. This approach addresses memory constraints effectively, showcasing the critical role of memory in video continual learning. vCLIMB Villa et al. (022) and PIVOT Villa et al. (2023) introduce novel benchmarks and methods focusing on class incremental learning and the use of prompting mechanisms,, respectively, pushing the boundaries of current methodologies. Utiliziig Wimning Subnetworks for efficient learning Kang et al. (2023), and creating
multi-modal datasets for egocentric activity recognition Xu et al. (2023) illustrate the expanding scope
of continual learning in video domains. Additionally; Continual Predictive Learning [Chen el. al. (2022)
and approaches to Video Object Segmentation significant advancements in handling non-stationary environments and long video sequences. Finally, efforts to learn new class representations while preserving old ones through time-channel importance maps Park et al. b further demonstrate the innovative and diverse strategies being developed for video continual learning.
continual learning task Nazemi et al. (2023) represent
Test-Time Adaptation. Test-Time Adaptation (TTA) aims to alleviate performance drop of protrained models at test time when exposed to domain shifts Sun et al. (2020); Alfarra et al. (2023). Earlier works augmented the training objective with a sel--sueervised loss function that is later leveraged at lest time combat domain shifts (Sun al. (2020); Liu al. (2021). More recent TTA methods optimize an unsupervised loss function at test-time on the received unlabeled data to improve performance under domain shifts Niu et al. (2022). This includes simple adjustments to the statistics of normalization layers Li et al. 2016 entropy minimization Wang et al. (2020), information maximization Liang et al. (2020), among others Boudiaf et al. (2022); Waag et al. (2022a). However, most TTA methods are proposed to combat covariate domain shifts at test time. In this work, we get inspiration from the source hypothesis adaptation method Liang et al.] (2020) to propose an effective regularizer for continual learning. We also analyze the effectiveness of other adaptation methods such as entropy minimization in mitigating catastrophic forgetting in continual learning.
This work aims to enhance continual learning performance by introducing a cost-effective regularizer that improves results even in memory-constrained scenarios. Such scenarios are particularly important when dealing with memory-intensive data, such as videos, or when sample storage is restricted due to privacy concerns. We investigate a class-indepente regularizer designed to facilitate the learning of generalizable features.
3 Methodology
In this section, we formalize the problem of class-incremennal learning in visual recognition tasks. We define the underlying framework and introduce the necessary notation to describe the incremental learning process. Additionally, we present the formulation of the proposed regularizer, Information Maximization (IM), along with the selected baseline regularizers: Elastic Weight Consolidation (EWC), Synaptic Intelligence (ST), and Entropy Minimization (EM).
We focus the offline continual learning problem for visual recognition tasks, where classifier So X + P())) (a DNN parameterized by 0) maps an input E X into the probability simplex P(Y). with y (1,2...,K). In continual learning, So is presented with sequence of T tasks
l..g. the network's output after Softmax.
3

---

Published in Transactions on Machine Learning Research (01/2026)
(I), (X) (2)...)..... Yr)) where X, C X and Y, C y Vi Villa et al (2022 Furthermore, we consider the class-incremental problem setup, where the labels presented in each individual task are mutu- ally exclusive (Y = = Vi + j). The main objective of the learner is to maximize its performance (e.g. classification accuracy) on all observed tasks. This objective is often hindered by the catastrophic forgetting problem: while learning lask i, So tends to lorget previously learned lasks < i, significantly dropping its performance for any x E Xev-
For our baseline, consider rehearsal-based continual learning methods where the learner is allowed to store up to N training examples from previous tasks into replay memory buffer M Chauddhyy et al. (2019b). Let M, denote the replay buffer at task t containing examples from the tasks < t. Rehearsal-based methods update the parameter set at task & in the following form:
0 = arg min BGo.)--X..,x
((
(1)
That is, for each batch sampled from the newly available data the thh task, the learner samples another batch from memory Mr and updates the model on the combined loss.
3.1 Regularizing Replay Methods with Information Maximization
Inspired by the work of Liang et al. (020) in the domain of test-time adaptation, we propose that in a continual learning setup, fe should output confident predictions that distinetly separate all previously seen classes. To achieve this, we propose regularizer that encourages the model to make confident predictions across all encountered classes without biasing its predictions towards recent task data. This means that for any given input, the model should assign a high probability to a single class. We can achieve this by maximizing information in the logits, as a result our approach helps reinforce discriminative representations for all learned classes, improving robustness against distribution shifts. Our proposed regularizer (RIM) takes the following form:
R1, = Lent (0, + (0,
(2)
K
K f log fo,
with Lent X.) = -E Σ (x)) S0
Laiv
k=1
where fo = Ex,xx, [z(()] and (x) is the ttth element in the vector fo(z). The IM regularizer is grounded in the principle of mutual information maximization between imputs X and predicted outputs Y Shannon -Y))).))
1948), formalized as Y) = H(Y) HY[)X). This decomposition reveals two complementary objectives: (I) minimizing conditional entropy (Y ensures confident predictions for each input (achieved by Lent), and (2) maximizing marginal entropy H(Y) ensures predictions are distributed across all output classes (achieved by Laiv). Without Laier the model could trivially minimize Lent by assigning high-confidence predictions to single class for all inputs, causing prediction collapse. Laiv prevents this by forcing the Together, these terms encourage confident, yet balanced predictions across all classes. Our regularized rehearsal-bacee method integrates this IM regularizer into the continual learning objective as follows:
model's average predictions lo span the entire label space +
njj (xc.))-(x..,)C((((())) + C(F + RIM Xt).
Our proposed regularizer has the following advantages: (i) It is orthogonal to the most critical design choices of contimual learning algorithms, it can operate regardless of the choice of fe, the replay-based method, the size of the memory buffer, and the number of tasks. (ii) Efficientt computation of Rias: where both Cent and Laiv depend exclusively on the output predictions of the model and can be computed in O(n). This aspect is essential when dealing with memory-intensive setups. For example, on video data, our regularizer
2Te subscripts "ent" stands for entropy and "div" stands for diversity
4

---

Published in Transactions on Machine Learning Research (01/2026)
estimates Cent and Laiv over clip predictions instead of per-frame estimates. (iii) Our formulation is agnostic to the type of data used in the continual learning problem. Without any modifications ions, our formulation can be applied to both image-based or hideo-based continual learning problems.
3.2 Baseline Regularizers
We compare our proposal against different regularizers to assess its effectiveness in mitigating forgetting and improving continual learning performance. We follow the formulation in Equation @ and study alterna- tives to (0, X,). In particular, we analyze different regularizers from the continual learning literature, namely Elastic Weight Consolidation Kirkpatrick et al. (2017) and Synaptic Intelligence Zenke et al. (2017). Furthermore, we explore Entropy Minimization Wang
(0200) from the test-time adaptation literature.
Elastic Weight Consolidation (EWC). Kirkpatrick et al. 2017 proposed to regularize the parameter update during continual learning prevent catastrophic lorgetting by constraining changes to important weights. The key idea behind EWC is to estimate the importance of each parameter for previously learned tasks and penalize deviations from their learned values. We analyze the effectiveness of combining EWC with rchearsal-based methods by replacing RIM in Equation
3 with REwc, dcfinod as:
(ow = Σ F. => 11-1)²
where F; denotes the i-th diagonal element of the Fisher information matrix F, which quantifies the impor- tance of the i-th parameter based on how sensitive the loss function to changes in that parameter. Here, 0i and 0i are the i-th parameters at the current task (we are optimizing for) and previous task, respectively, and À is a hyperparameter balancing the relative importance of old tasks with respect to the current task.
Synaptic Intelligence (SI). It is a biologically inspired regularizer from the continual learning literature. Ii follows a similar principle to EWC but determines weight. importance using a different approach. Instead of using the Fisher Information Matrix, SI tracks the contribution of each parameter during training by accumulating an importance measure based on changes loss. This adaptive tracking mechanism allows the model to selectively constrain updates to crucial parameters while remaining flexible for learning new tasks. We replace RIM in Equation with Rsi which takes the following form:
(o) = Σ
(AO -0 +
where A0% 0% -8.-8% 06-1 is the change in the k-th parameter between task t and task t 1, w is the per- parameter importance weight for parameter & at task i, h indexes the parameters, t indexes the tasks, and the damping parameter & avoids division by zero.
Entropy Minimization (EM). Following the self-supervised spirit of our proposed rogularization ap- proach, we include one self-supervised regularizer that encourages the model to produce more confident predictions. In particular, we follow Wang et al. (2020) and apply entropy minimization to regularize the output distribution, reducing the model's uncertainty when making predictions. Entropy minimization re- places RIM in Equation
3 with REM where:
REM(0,X,) -E Σ fo (x) (o (0,
(4)
Entropy minimization encourages the model to assign higher confidence to its predictions, effectively sup- pressing uncertain outputs. This can be beneficial a continual learning sctup, where distribution shifts can lead to increased uncertainty. This entropy minimization term forms one component of our proposed IM regularizer (Lent)-
5

---

Published in Transactions on Machine Learning Research (01/2026)
4 Experiments
In this section, procoed with the empirical assessment of our proposed approach to validate its effective- ness. For completeness, we first evaluate several rehearsal-based continual learning (CL) methods (ER, DER, and DER++) when paired with the regularizers LM, EWC, SI, and EM. We then extend the analysis by applying IM to more advanced rehearsal approaches (Refresh Learning and STAR), showing that its benefits generalize consistently across different methods and datasets.
Datasets. Following the image CL literature, we focus on two main datasets: Split-CIFAR100 Zenke et al. (2077) and Split-Tiny ImageNet. Le & Yang (2015). Split-CIFAR100 contains a total of 100 classes and 6000 images per class. It is divided into 10 tasks, each containing 10 classes. Split-Tiny ImageNet consists of 200 classes with 500 images per class, and is divided into 10 tasks of 20 classes each.
Evaluation Metrics. To evaluate the performance CL methods, we consider two metries: Average Accuracy, which is defined as the average performance across all lasks, and Forgetting Rate that measures the impact of the learned task on the performance of the previous tasks Chaudhry et al. (2019a). These metries provide complementary perspectives the effectiveness of cach approach in balancing stability and adaplability in continual learning setting.
Average Accuracy (ACC) quantifies the model's overall performance across all tasks it has en- countered. It is defined as:
T ACC
a5
(5)
T
where T is the total number of lasks and a, represents the accuracy of the model on the i-th lask after it has been trained on all T tasks. ACC provides comprehensive measure how well the model learns and retains knowledge across a full sequence of tasks.
Forgetting Rate (FR) measures the decrease in performance on past tasks after model has been trained ones. It directly measures catastrophic forgetting. FR is defined as:
T-1
FR-T-
FR
Σ
max max(a;; a.T), K&T
(6)
where aij is the accuracy on task immediately after training on task j, and a,r is the accuracy on task i after the final task has been learned. lower FR indicates better retention of previously learned knowledge, while higher FR points significant forgetting.
Implementation Details. We train a 18sNet18 He et al. (2016) model from scratch and summarize the training parameters the Appendix. Following our limited memory setting, we define a budget of 5, 10 and 20 samples per class the maximum allowed in M, at any moment. We integrate the different regularizers into the Mammoth CL framework Boschini et al. (2022 Buzzega et al. (2020a). To balance the loss terms, multiply the regularization term by A=0.5 and the cross-entropy loss with 1-X. All main experiments employ a fixed random seed to ensure fair comparison across methods. Further details on hyperparameter sensitivity, training paramoters, and statistical validation are provided in the Appendix.
Baselines. We consider four regularizers: EWC Kirkpatrat et al. (017)) SI Zenke el al. (201)) EM Wang et al. (2020), and IM Liang et al. (2020), applied on top of three memory- based continual learning msthods: ER Rolnick et al. (2019). DER Buzzega et al. (2020b). and DER++ Buzzoga et al. (202bb). For the IM regularizer, we further extend the evaluation include Refresh Learning Wang et al. (2024). implemented top of DER++, and STAR Eskandar et. al. (2025). implemented top of E.R.

---

Published in Transactions on Machine Learning Research (01/2026)
Split-CIFAR 100
Split-Tiny ImageNet
Accuracy (%)
Forgetting (%)
Accuracy (96
30
Forgetting (%)
40
60
60
30
20
40
20
40
500
1000
2000
1000
2000
2000
4000
1000
2000
Accuracy (%)
Forgetting (%)
Accuracy (%)
40
Forgetting (%)
60
40
20
40
500
1000
2000
500
1000
2000
1000
2000
4000
1000
2000
4000
Accuracy (%)
Forgetting (%)
Accuracy (%)
Forgetting (%)
60
60
DER+
40
30
40
20
40
20
500
1000
2000
1000
2000
2000
4000
1000
2000
4000
Memory Size
Memory Size
Memory Size
Memory Size
Baseline
Baseline+
Baseline+EM
Baseline+EWC
Baseline+SI
Figure 1: Results of Integrating Different Regularizers Split- and Split-Tiny ImageNet. This figure plots the average accuracy and forgetting rate of three baseline methods (ER, DER, and DER++) across various sizes of memory buffer, in combination with the analyzed regularizers (IM, EM, EW, and SI), and across two datasets (Split-CIFAR100 and Split- ImageNet). The results demonstrate that the proposed information maximization regularizer (TM) consistently outperforms other methods, achieving higher accuracy and lower forgetting rates on both datasets regardless of the memory setting.
4.1 Regularized Rehearsal Methods Results
Figure I summarizes the performance of rehearsal-baas methods ER, DER, and DER++ on Split- CIFAR100 (left columns) and Split-Ting ImageNet (right columns) for the selected memory sizes. We include the bascline performance (light blue) and outline the impact of incorporating regularization techniques on top of these rehearsal methods.
Our results demonstrate that introducing 1M on top of rehearsal-basee based methods consistently leads im- provements across all memory sizes. For instance, when IM is applied to ER on Split-CIFAR100, observe enhancement of 10-13% in performance across all memory sizes. The improvement is slightly lower for DER and DER++, ranging around 2-7% for DER and 1-2% for DER++. In contrast, other regularizers like EWC, SI, and EM generally do not improve the baseline methods, and can even degrade performance in some cases, as is the case for EWC on DER and DER++, where the model's accuracy drops by nearly half.
We can also observe in Figure (1) that forgetting is significantly reduced when the rehearsal methods are paired with IM. For ER (paired with IM) applied on Split-CIFAR100, the reduction in forgetting is around (22-25%) across all memory sizes. For DER and DER++ reduction compared to ER with about (9-13%) and (4-6%), respectively. On the other hand, EM and SI do not generally reduce forgetting compared to the original baselines. This is since EM aims at increasing the model's confidence in predicting samples from the current task. While this approach might accelerate the learning process over tasks, does not promote the retention of previously learned information.
Split-CIFAR110, our results show smaller
For EWC, we observe reduction in forgetting when paired with ER, but not with the remaining baselines. More detailed and comprehensive results be found in the Appendix.

---

Published in Transactions on Machine Learning Research (01/2026)
Split-CIFAR 100
Accuracy (%)
Forgetting (%)
60
40
40
20
20
0
0
500
1000
2000
500
1000
2000
Memory Size
Memory Size
Split-Tiny ImageNet
40
Accuracy (%)
30
Forgetting (%)
60
40
20
10
20
1000
2000
4000
1000
2000
4000
Memory Size
Memory Size
ER
DER
DER
Refresh (Refresh) IM
STAR
(ER)
(DER) IM
(DER++) IM
(STAR) IM
Figure 2: Results on Split-CIFAR100 and Split-Tiny ImageNet. This figure presents the average accuracy and forgetting rate of five rehearsal-based methods (ER, DER, DER++, Refresh Learning, and STAR) across different memory buffer sizes, both in their baseline form and when combined with Information Maximization (LM). The results show that integrating IM consistently enhances performance, leading to higher accuracy and reduced forgetting across all methods, datasets, and memory settings.
To further validate the effectiveness of IM, we expand our analysis to include two additional rehearsal based baselines: Refresh Learning and STAR. Figure (2) presents the performance gains when IM integrated into all five rehearsal methods (ER, DER, DER Refresh Learning, STAR) across both Split-CIFAR100 and Split-Tiny ImageNet.
On SpLit-CIFAR100, IM consistently improves the performance of Refresh Learning and STAR, but with varying impact. When paired with Refresh Learning, the improvements are more modest, about 1-2% accuracy, but forgelting reduced by (7 (7-9%) across all memory sizes. This indicates that while Refresh Learning already stabilizes training to some degree, IM provides an additional layer of retention without significantly altering the learning dynamics. For STAR, the effect is more significant, we observe accuracy gains of 2-6%, along with substantial reduction in forgetting (18-2) (18-24%).
On Split-Tiny ImageNet, IM continues to yield consistent improvements. For Refresh Learning, the accuracy gains are larger than on Split-CIFAR100, ranging from 2 8%, with forgetting reduced by (7 20%). STAR also benefits considerably, with accuracy improvements of 3-9% and forgetting reductions between (9 (5%1.%
Conclusion. These results reveal that Information Maximization (IM) serves as an effective regularization technique, consistently improving the performance of image continual learning baselines across various mem-
8

---

Published in Transactions on Machine Learning Research (01/2026)
Ablation Study on Compute Budget. This table presents the performance of ER and DER, with and without the Information Maximization (TM) regularizer, on SpFRt-CTFAR100 and Split-Tiny ImageNet datasets. For Split- the baselines are run with 10 epochs instead of the original 50, while for Split-Tiny ImageNet, the experiments are run with 50 epochs instead of the original 100. The results indicate that incorporating IM leads to improved convergence and consistently higher average accuracy.
Table 1:
Split-Cifarld
Split-Tiny ImageNet
Buffer Size
500
1000
2000
1000
2000
4000
ER
20.8
26.8
35.6
13.2
18.7
25.6
ER IM
28.8
35.7
40.1
21.5
26.6
31.8
DER
24.1
21.1
19.9
21.6
26.3
24.2
DER + IM
33.5
37.3
34.4
27.9
32.1
33.3
ory budgets. By encouraging confident yet balanced predictions, 1M enhances both accuracy and knowledge retention, thereby mitigating catastrophic forgetting.
4.2 Ablation Analysis
To explore the limitations of Information Maximization (IM) as a regularizer for continual learning methods, we conduct three ablation experiments aimed at understanding its performance under various conditions. First, we assess the impact of IM when the computational budget reduced to determine whether it improves the convergence of the baseline methods. Second, we evaluate if the improvement obtained by using IM diminishes with additional tasks. Specifically, we investigate the performance of our approach when the number of tasks exceeds 10, to understand how IM scales with an increasing number of tasks. Third, we examine the effect of applying IM exclusively to buffer samples, as opposed to both buffer and current task samples, to assess whether applying IM beyond current task samples is beneficial.
Computational Budget. In the experiments presented in Section (4.1). the computational budget was set following the default settings of Mammoth CL framework, i.e. 50 epochs per task for Split-CIFAR100 and 100 epochs per task for Split-Tiny ImageNet. These settings ensure sufficient training iterations for each task, allowing models to stabilize and learn meaningful representations. However, in practical scenar- ios, computational efficiency is a critical factor, as training deep learning models over multiple tasks can be pvolibitively expensive. Currently, there is a growing interest in exploring low-computational regimes Ghunaim et al. (2023) : Prabhu et al. (023)) for CL, as computing resources are far more expensive than storage Prabhu et al. (2023). This shift in focus reflects the necessity of developing continual learning ap- proaches that remain effective even with reduced training time. Therefore, investigate how our proposed regularizer, IM, performs under computational constraints, we ablation experiments with about half of the computational budget Split-Tiny ImageNet (i.e. half of the training epochs per task) and less than quarter of the computational budget on Spli--CIFAR100 compared to our original experimental setup.
The results in Table I show that, even with a lower computational budget of 10 and 50 epochs per task and Split-Tiny ImageNet, respectively, the proposed ER+IM and DER+IM methods outperform their counterparts without TM regularizer. For instance, on the Split-CIFARIN R100 dataset with a buffer size of 1000, ER+IM achieves an accuracy of 35.7%, significantly higher than ER at 26.8%. Similarly, DER+IM attains 37.3% accuracy, surpassing DER's 21.1% by large margin. These trends hold for different buffer sizes and datasets, highlighting the effectiveness of the proposed regularization in low-compute regimes.
on
Number of Tasks. In the experiments presented in Section 4.1 we used the conventional 10-tasks split for Split-CTFAR100 and Split-Tiny ImageNet, which commonly used in continual learning studies. However, as shown in Villa al. (2022): Prabhu et al. (2023), performance may vary when more tasks are introduced. More tasks can make the problem harder because the model has to remember more information
9

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: Ablation Study on Number of Tasks. This table presents the performance of ER and DER with and without Information Maximization (TM) regularizer, sequence of 20 tasks for both Split- CIFAR100 and Split-Tiny ImageNet datasets. Our evaluation shows that, despite the increased amount of tasks our proposal still outperform the ER and DER baselines in every single scenario.
Split-CIFAR100
Split- ImageNet
Buffer Size
500
1000
2000
1000
2000
4000
ER
16.6
25.8
34.7
9.1
14.5
22.1
ER + IM
23.4
31.3
38.9
18.3
23.4
29.1
DER
25.1
35.8
38.9
18.0
22.5
27.8
DER + IM
32.8
39.8
45.0
23.2
27.0
32.1
and avoid forgetting earlier tasks while learning new information. Consequently, we reran the experiments in Section (4.1), doubling the number of tasks from 10 to 20. This allows us to evaluate whether Infor- mation Maximization (IM) regularizer remains effective when the continual learning problem becomes more challenging due to having more tasks to learn. The results presented in Table (2) show that incorporating IM into ER and DER can significantly improve their performance on longer sequences of tasks. For ex- ample, ER+ IM shows improvement of (4-7%) and (6-8% SpIit-CIFAR100 and Split-Tiny ImageNet, respectively. On the other hand, DER+IM shows (4-7%) and (4-5%) improvement on Split-CTFAR100 and Split-Tiny ImageNet, respectively.
Rogularization Targets. For the experimental assossment Section 4.1 we apply the regularization loss to current lask samples only. This raises the question of how the proposed method would behave if the IM loss were applied exclusively to memory samples or to both memory and current task samples. For this reason, reran the experiments shown in Section (4.1p for both variants, and the results are summarized in Table (3). We find that applying the IM loss to the current task (CT) is superior to applying it to the memory/ / buffer samples only (BF) or to both buffer and current task samples (ALL). Notably, this trend remains consistent across various buffer sizes, datasets, and continual learning methods, highlighting the robustness of this strategy.
For example, with a buffer size of 500 on the Split-CTFAR100, ER+IM (CT) achieves an accuracy of 33.6%, which is significantly higher than ER+IM (ALL) and ER-IM (BF), which achieve 25.9% and 21.7%, re- spectively. Similarly, DER+IM (CT) consistently outperforms DER+IM (ALL) and DER+IM (BF) across various buffer sizes and datasets, reinforcing the advantage of applying IM solely to current task samples. For example, it achieves 48.7% accuracy on SpIit-CIFAR100 with a buffer size of 1000, while DER +IM (ALL) and DER +IM (BF) achieve 46.0% and 41.0%, respectively. These results indicate that applying IM loss to the current task samples more effective than applying exclusively to the memory /arrry//buuu /buffer samples only or to both buffer and current task samples (refer to Appendix for further analysis).
4.3 Generalization Video Continual Learning
To further validate the effectiveness of Information Maxinization (IM) as a cost-effective regularization technique for enhancing continual learning methods, extend our analysis to representative experiments in the video domain. Specifically, we experiment with the iCaRL approach as part of the popular vCLIMB
framework Rebulli et al. 2017 Villa et al. (2022) to assess IM's performance in this video CL context.
We evaluate the use of IM loss on two widely recognized datasets in the video domain: UCF-101, consisting of 101 classes split into 10 tasks, and ActivityNet, comprising 200 classes also divided into tasks. We set the memory size to 5, 10 and 20 samples per class and adopt the training hyperparameters from Villa et al. h022
10

---

Published in Transactions on Machine Learning Research (01/2026)
Table 3: Ablation Study Regularization Target Selection. In the main results, the Information Maximization (IM) regularizer is applied exclusively to the current task samples (CT). This table presents the results of applying the regularizer to the buffer samples only (BF) and to both current task samples and buffer samples simultaneously (ALL). The findings indicate that applying the regularizer to the current task samples consistently leads to superior performance compared to the other variants.
CIFAR00
Split- ImageNet
Buffer Size
500
1000
2000
1000
2000
4000
ER + IM (ALL)
25.9
34.6
42.7
15.0
20.3
27.8
ER + IM (CT)
33.6
39.8
46.2
21.6
27.0
32.1
ER + IM (BF)
21.7
28.9
38.8
12.7
17.7
24.2
DER IM (ALL)
33.5
46.0
53.5
27.7
32.7
35.1
DER IM (CT)
40.6
48.7
53.6
28.1
33.3
38.7
DER IM (BF)
27.3
41.0
50.2
21.2
27.7
34.6
UCF-101
ActivityNet
25
Accuracy (%)
80
Forgetting (%)
25
Accuracy (%)
55
20
50
20
70
15
45
15
10
40
500
1000
1500
2000
500
1000
1500
2000
1000
2000
3000
4000
1000
2000
3000
4000
Buffer Size
Buffer Size
Buffer
Buffer Size
iCaRL
iCaRL
Figure 3: Application of Information Maximization to Video Continual Learning. This figure illustrates the average accuracy and forgetting rates of the iCARL video continual learning variant, intro- duced by vCLIMB Villa et al. (002),, with and without our Information Maximization (LM) regularizer. The results demonstrate that incorporating the IM regularizer on top of iCARL leads to consistent improvements in average accuracy and reductions in the forgetting rate.
Upon applying iCARL to UCF-101 with IM, we achieve an improvement, 2-8% across all memory sizes. Similarly, on ActivityNet, the accuracy gain is between 4-8% with the incorporation of IM. These results further underscore the potential of IM valuable regularization technique for enhancing performance in the video domain and continual learning scenarios. Note that video data has additional temporal dimension compared to images, which requires more memory to store. Being able to improve continual learning performance with small memory buffer sizes, as shown in Figure is crucial for facilitating the development of memory-effciient approaches for video continual learning.
5 Conclusion
In conclusion, this paper explores the combined potential of memory-based methods and regularization techniques in the context of Continual Learning (CL), specifically within class incremental sctup. We introduce novel, class-agnostic regularization strategy for CL, which focuses on the distribution of the network's predictions. This strategy, termed Information Maximization (IM) regularization, facilitates the
11

---

Published in Transactions on Machine Learning Research (01/2026)
learning of enhanced feature representations across multiple distribution shifts, while simultancusly mini- mizing memory requirements and computational overhead. Our extensive empirical evaluation underscores the effectiveness of the proposed IM regularizer, demonstrating its superiority over existing regularization strategies designed for CL. Furthermore, the simplicity and versatility of our approach allow it to be applied across different input domains, as evidenced by its successful application in the video continual learning setup. Unlike traditional image-based settings, video CL presents additional challenges due to its tempo- ral structure and higher memory demands. Despite these complexities, our method demonstrates strong performance, reinforcing its applicability to real-world, resource-constrrinee scenarios.
Acknowledgment
This work supported by the KAUST Center of Excellence for Generative Al under award number 5940. The computational resources are provided by IBEX, which is managed by the Supercomputing Core Laboratory at KAUST.
References
Motasem Alfarra, Hani Itani, Alejandro Pardo, Shyma Alhuwaider, Merey Ramazanova, Juan C. Pérez, Zhipeng Cai, Matthias Müller, and Bernard Ghanem. Revisiting test time adaptation under online eval- uation, 2023.
Lama Alssum, Juan León Alcázar, Merey Ramazanova, Chen Zhao, and Bernard Ghanem. Just a glimpse: Rethinking temporal information for video continual learning. In CVPRW, pp. 2473-2482, 2023.
Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara. Class- incremental continual learning into the extended der-verse. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca. Bertinetto. Parameter-free online test-time adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8344 8353, 2022.
Pietro Buzzega, Malteo Boschini, Augelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. In H. Larochelle, M. Ranzato, R. Hadsell, M. F, Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 15920 15930. Curran Associates, Inc., 2020a.
Pictro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. NewrIPS, 33:15920 15930, 2020b.
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky, New insights on reducing abrupt representation change in online continual learning. arXiv preprint arXiv:2104.0555 2021.
Arslan Chaudhry, Marc' Aurclio Ranzato, Marcus Rohrbach, and Mohamed Elhosciny. Efficient lifelong learning with a-GEM. In ICLR, 2019a. URL
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10 10486, 2019b.
Geng Chen, Wendong Zhang, Han Lu,, Siyu Gao, Yunbo Wang, Mingsheng Long, and Xiaokang Yang. Continual predictive learning from vidcos. In CVPR, pp. 10728-10737, 2022.
Matthias De Lange and Tinne Tuytclaars. Continual prototype evolution: Learning online from non- stationary data streams. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8250 8259, 2021.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Masih Eskandar, Tooba Imtiaz, Davin Hill, Zifeng Wang, and Jennifer Dy. Star: Stability-incucing weight perturbation for continual learning. Xiv preprint or (rXim:2503.01500 2025.
Yasir Ghunaim, Adel Bibi, Kumail Alhamoud, Motasem Alfarra, Hasan Abed Al Kader Hammoud, Ameya Prabhu, Philip HS Torr, and Bernard Ghanem. Real-time evaluation in online continual learning: A new hope. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11888 11897, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEFE conference on compuler vision and paltern recognilion, pp. 770 770-778, 2016.
Haeyong Kang, Jaehong Yoon, Sung Ju Hwang, and Chang D Yoo. Continual learning: Forget-free winning subnetworks for video representations. arXiv preprint a11iv:2312.11973, 2023.
James Kirkpatrick, Rasvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka BaabbskaBarwiskka, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521 3526, 2017.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. arXiv preprint 1rXi::6603.044 2016.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pp. PMLR, 2020.
Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Dellot-Gurlet Taylor Mordan, and Alexandre Alahi. Tu When does self-supervised test-time training fail or thrive? Advances in Neural Information Processing Systems, 34:21808-21888 -188000000,0,,,, 2021.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. NewrIPS, 30, 2017.
Amir Nazemi, Zeyad Moustafa, and Paul Fieguth. Clvos23: A long video object segmentation dataset for continual learning. In CVPR, pp. 2495 2504, 2023.
Shnaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International conference on machine learning, pp. 16888-16905. PMLR, 2022.
Jaeyoo Park, Minsoo Kang, and Bohyung Han. Class-incremental learning for action recognition in videos. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 13698-13707, 2021.
Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puncet K Dokania, Philip HS Torr, Scr-Nam Lim, Bernard Ghanem, and Adel Bibi. Computationally budgeted continual learning: What does matter? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3698-3701 3707, 2023.
Sylvestre-Alviee Rebuffi, Alexander Kolesníkov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR, pp. 2001-2010, 2017.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
Claude E Shannon. A mathematical theory of communictionn The Bell system technical journal, 27(3): 379 423, 1948.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexci Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning, pp. 9229 9248..922..2 PMLR, 2020.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Andrés Villa, Kumail Alhamoud, Victor Escorcia, Fabian Caba, Juan León Alcázar, and Bernard Ghanem. velimb: A novel video class incremental learning benchmark. In CVPR, pp. 19035-19044, 2022.
Andrés Villa, Juan León Alcázar, Motasem Alfarra, Kumail Alhamoud, Julio Hurtado, Fabian Caba Heil- bron, Alvaro Soto, and Bernard Ghanem. Pivot: Prompting for video continual learning. In CVPR, pp- 24214-24223, 2023.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint 0rXiv22006.00990 2020.
Qin Wang, Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceed- ings of the IEEE/CVF Conference on Compuler Vision and Pullern Recognilion, pp. 7201-7211, 2022a.
Zhenyi Wang, Yan Li, Li Shen, and Heng Huang. A unified and general framework for continual learning.
arXiv preprint ar X0o:203. 13249, 2024.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European conference computer vision, pp. 631-648. Springer, 2022b.
Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, pp. 139 149, 2022c.
Linfeng Xu, Qingbo Wu, Lili Pan, Fanman Meng, Hongliang Li, Chiyuan Hc, Hanxin Wang, Shaoxu Cheng, and Yu Dai. Towards continual egocentric activity recognition: A multi-modal egocentric activity dalaset for continual learning. orXiv preprint arXiv:2301.10931 2023.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Contimual learning through synaptic intelligence. In TCML, pp. 3987-3995. PMLR, 2017.
14

---

Published in Transactions on Machine Learning Research (01/2026)
6 Appendix
6.1 Regularized Rehearsal Methods Results
Tables and 5 contain the numerical results for the average accuracy and forgetting rate metrics, respec- tively, on three baseline methods (ER, DER, and DER+++ in combination with the analyzed regularizers (IM, EM, EW, and SI), as well as for integrating IM into Refresh Learning and STAR. These results were summarized as plots in Figures [ and (2) the main paper. We observe that our proposed regularize (IM) consistently outperforms other methods across different memory settings. The trper-parametes we used for every baseline when combined with the analyzed regularizers (IM, EM, EW, and ST) are listed in Table and
Table 4: Average Accuracy on Spli-CIFFRR100 and Split-Tiny ImageNet. This table shows the average accuracy of three baseline methods (ER, DER, and DER+++ across various sizes memory buffer, in combination with the analyzed regularizers (IM, EM, EW, and ST), as well as for integrating TM into Refresh Learning and STAR.
Dataset
Split-CIFARIN
Split-Tiny ImageNet.
Buffer
500
1000
2000
1000
2000
4000
ER
21.6
29.6
36.3
13.3
18.0
25.2
ER (IM)
33.6
39.8
46.2
21.6
27.0
32.1
ER (EM)
20.8
28.8
38.3
12.2
16.6
24.1
ER (EWC)
24.2
31.4
37.4
11.4
15.4
21.3
ER (SI)
21.8
29.0
36.4
11.2
15.7
20.4
DER
33.6
42.6
51.9
23.3
31.4
35.7
DER (TM)
40.6
48.7
53.6
28.1
33.3
38.7
DER (EM)
32.3
42.6
49.3
24.4
30.9
35.5
DER (EWC)
20.5
19.5
25.0
10.9
15.2
17.6
DER (ST)
29.5
42.3
50.4
13.2
17.0
16.8
DER+
40.4
45.9
52.5
23.0
31.2
38.0
DER (IM)
41.6
47.3
54.3
27.5
33.3
38.5
DER (EM)
40.5
44.6
52.6
25.1
30.7
36.8
DER (EWC)
18.0
17.6
22.8
12.9
14.3
21.3
DER + (SI)
33.7
42.3
50.3
17.6
17.4
22.2
Refresh
40.8
48.0
53.6
18.5
21.8
33.6
Refresh (LM)
42.7
49.5
55.1
25.5
30.1
35.2
STAR
28.6
36.9
46.6
16.8
23.2
26.3
STAR (IM)
34.7
43.4
48.8
25.8
26.1
34.7
6.2 Generalization to Video Continual Learning
Tables and present the numerical results of average accuracy and forgetting rate, respectively, for iCaRL approach with and without IM on two video datasets (UCF101 and ActivityNet). Combining iCaRL with IM shows improvement in average accuracy and reduces forgetting rate across different memory settings.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Table 5: Forgetting Rate Split-CIFAR100 and Split-Tiny ImageNet. This table shows the forgetting rate of three baseline methods (ER, DER, and DER-+) across various sizes of memory bulfer, in combination with the analyzed regularizers (IM, EM, EW, and ST), as well the results for integrating IM into Refresh Learning and STAR.
Datasot
Aplit-CIFARION
Split-Tiny ImageNet
Buffer
500
1000
2000
1000
2000
4000
ER
73.8
64.5
54.8
71.8
65.9
57.5
ER (IM)
47.9
40.6
32.8
53.3
47.6
40.9
ER (EM)
75.5
66.9
55.5
72.9
68.3
59.3
ER (EWC)
65.2
55.3
47.4
62.0
57.5
50.4
ER (SI)
73.0
65.3
56.0
68.5
60.8
57.6
DER
58.8
46.5
32.4
55.9
43.0
35.7
DER (IM)
45.1
32.7
23.2
44.3
34.5
26.0
DER (EM)
60.0
46.9
38.2
53.7
44.8
38.4
DER (EWC)
67.7
72.0
64.8
67.4
61.3
58.8
DER (SI)
63.8
47.1
36.0
58.9
49.7
51.5
DER++
50.7
42.7
35.1
56.7
41.0
33.9
DER + (TM)
46.0
37.1
28.1
46.1
32.1
28.4
DER+ (EM)
49.6
43.1
32.5
48.8
40.5
32.9
DER (EWC)
68.4
70.7
63.0
64.7
64.4
53.8
DER
(SI)
58.3
46.7
36.1
57.4
52.7
49.8
Refresh
45.0
35.1
28.5
62.7
37.9
42.2
Refresh (IM)
36.4
25.9
21.8
44.6
37.9
35.4
STAR
62.6
55.1
42.2
65.7
50.9
54.1
STAR (IM)
40.8
31.4
23.7
50.4
42.0
40.4
6.3 Sensitivity Analysis: Regularization Strength (A)
We provide a detailed sensitivity analysis of the regularization strength (A) to demonstrate the robustness of our findings and address concerns regarding fair hyperparameter comparison. We show that 1M consistently outperforms bascline regularizers across range of scttings, confirming that our main conclusions are not artifacts of a specific hyperparameter choice. For the primary results presented in the paper, adopted fixed hyperparameter protocol, using a consistent regularization strength (A 0.5) for all regularizers (TM, EM, EWC, and SI). This protocol ensures a direct and fair test of robustness by evaluating all methods under identical experimental conditions. To validate that our findings are not artifacts of this fixed À choice, we conducted a sensitivity analysis across À E 0.2, 0.5, 0.8 on Split-CIFAR100 with a memory buffer of 500 samples. The results for Experience Replay (ER) and Dark Experience Replay (DER) are presented in Table 10 and Table [1] respectively. The analysis demonstrates that our proposed IM regularizer substantially outperforms all other regularizers, results confirm that our conclusions are robust to hyperparameter choices and are not an artifact of favoring IM through biased selection.
when each method uses its individually optimal À value. These
6.4 Regularization Target Selection
A critical design choice in the application the Information Maximization (IM) regularizer concerns the selection of the target samples to which the regularization is applied. Three natural candidates emerge: (i) Current Task (CT) samples only, (ii) Buffer (BF) samples only, and (iii) All (ALL) samples, encompassing both current task and buffer samples. While intuition might suggest that regularizing the rehearsal buffer
16

---

Published in Transactions on Machine Learning Research (01/2026)
Table Experiments Hyper-Parametern (Split-CIFAR100), Columns list the training hyper- parameters. Buf. buffer size, LR lcarning rate, Mom. momentum, WD weight docay, a distillation coelficient (used in DER/DER+++ //efreshh) B memory balancing coefficient. (used in DER++ //RR++///rrs)) ?, X, (specifie to STAR), r, J (specific to Refresh Learning). A dash (-) indicates that the parameter is not applicable to the method.
Baseline
Buf.
LR
Mom.
WD
8
7
J
ER
500
0.1
FR
1000
0.1
ER
2000
0.1
0
0
DER
500
0.03
0
0
0.3
DER
1000
0.03
0.3
DER
2000
0.03
0.3
+
500
0.03
0.3
0.5
+
1000
0.03
0.3
0.5
+
2000
0.03
0
0.3
0.5
Refresh
500
0.03
1e-4
0.3
0.5
le-5
1
Refresh
1000
0.03
1e-4
0.3
0.5
le-5
1
Refresh
2000
0.03
1c-4
0.3
0.5
le-5
1
STAR
500
0.1
0.05
0.05
1
STAR
1000
0.05
0.05
0.05
STAR
2000
0.05
0
0
0.05
0.05
1
should be most effective for mitigating catastrophic forgetting, our empirical findings reveal a counter- intuitive result: applying the IM regularizer exclusively current task samples yields substantially superior performance (Table 3).
Feature Drift Analysis. To understand why CT-only regularization outperforms BF-only regularization, we conducted a feature drift analysis. Feature drift is quantified as the mean L2 distance between class- specific feature representations (penultimate layer activations) immediately after learning task versus after all tasks are completed. We report the feature drift for the first two tasks (Task and Task 2).
The feature drift analysis in Table 12 reveals striking and counterintuitive finding: both IM-CT and IM- BF increase feature drift compared the baseline, yet only IM-CT yields improved performance. This observation shows that effective continual learning regularization does not necessarily minimize feature drift; instead, it can enable controlled feature evolution that maintains class discriminability.
The per-task breakdown in Table 13 shows that IM-CT consistently improves old task accuracy (T1-T9), while sacrificing recent task performance (T10: 92.0% - 77.2%). This stability-plasticity trade-off is char- acteristic of effective continual learning. In contrast, IM-BF maintains high performance on the recent task (T10: 93.2%) but fails to prevent forgetting on old tasks, effectively as IM-CT, indicating that the consistent improvement pattern of IM-CT reflects systematic feature refinement rather than random effects.
6.5 Experimental Methodology and Statistical Validation
Our experimental methodology involved two complementary approaches balance rigor with computational feasibility. First, we conducted preliminary experiments with multiple random seeds validate observed behaviors and identify consistent trends across different initializations. Second, given the extensive scope of
17

---

Published in Transactions on Machine Learning Research (01/2026)
Table Experiments Hyper-Parameters (Split-Tiny ImageNet). Columns list the training hyper-paramcters. Buf. buffer size, LR learning rate, Mom. momentum, WD weight docay, distillation coelficient (used in DER/DER D / memory balaneing coefficient (used in DER++/Refresh) 7, X, (specific to STAR), r, J (specific to Refresh Learning). A dash (-) indicates that the parameter is not applicable to the method.
Baseline
Buf.
LR
Mom.
WD
B
7
A
J
ER
500
0.03
FR
1000
0.03
ER
2000
0.03
0
DER
500
0.03
0
0.1
DER
1000
0.03
0.1
DER
2000
0.03
0.1
+
500
0.03
0.3
0.5
+
1000
0.03
0.3
0.5
+
2000
0.03
0
0.3
0.5
Refresh
500
0.03
1e-4
0.3
0.5
le-5
1
Refresh
1000
0.03
1e-4
0.3
0.5
le-5
1
Refresh
2000
0.03
1c-4
0.3
0.5
le-5
1
STAR
500
0.1
0.01
0.1
1
STAR
1000
0.1
0.05
0.1
STAR
2000
0.1
0
0
0.01
0.1
1
Table 8: Average Accuracy on UCF101 and ActivityNet. This table shows the average accuracy of iCaRL with and without IM across various sizes of memory buffer two datasets (UCF101 and ActivityNet). The results demonstrate that the integration of information maximization (IM) regularizer consistently outperforms iCaRL on both datasets regardless of the memory setting.
Dataset
UCF101
ActivityNet
Buller
505
1010
2020
1000
2000
4000
iCaRL
68.44
74.70
81.07
41.72
45.05
46.91
iCaRL (IM)
75.60
79.11
83.53
45.76
50.01
55.20
our evaluation across multiple methods, datasets, and hyperparameter configurations, we employed a fixed random seed across all experiments to ensure fair comparison and computational feasibility.
To demonstrate the robustness of findings, provide results with multiple seeds (3 independent, runs) for representative configuration: DER with and without IM across all memory sizes Split- CIFARIIO (Table 14
The results demonstrate consistent improvements across all memory buller sizes, with IM achieving gains of 7.53, 5.93, and 2.42 percentage points for buffer sizes of 500, 1000, and 2000, respectively. Importantly, the improvement margins substantially exceed the standard deviations, indicating that the observed gains are statistically reliable and not attributable random variation.
18

---

Published in Transactions on Machine Learning Research (01/2026)
Table 9: Forgetting Rate UCF101 and ActivityNet. This table shows the forgetting rate of iCaRL with and without IM across various sizes of memory bulfer on two datasets (UCF101 and ActivityNet). The results demonstrate that the integration of information maximization (IM) regularizer consistently achieves lower forgetting rates on both datasets regardless of the memory setting.
Dataset
UCF101
ActivityNet
Buffer
505
1010
2020
1000
2000
4000
iCaRL
25.55
20.37
14.31
21.29
19.46
18.55
iCaRL + IM
16.87
16.02
10.54
19.01
18.55
15.48
Table 10: Hyperparameter Sensitivity on ER (Memory=500). This table shows the performance (Average Accuracy) of the ER bascline combined with different regularizers (IM, EM, EWC, SI) across three different values for the regularization strength A. The best performance for each method is highlighted in bold.
Method
A=022
A=0.5
x=0.8
Best À
Best. Performance
ER (Baseline)
21.6
ER + IM
24.46
33.6
24.06
0.5
33.6
ER + EM
22.16
20.8
16.32
0.2
22.16
ER + EWC
23.74
24.2
26.73
0.8
26.73
ER + SI
21.07
21.8
19.72
0.5
21.8
Table 11: Hyperparameter Sensitivity on DER (Memory=500). This table shows the performance (Average Accuracy) the DER baseline combined with different regularizers (LM, EM, EWC, SI) across three different values for the regularization strength The best performance for each method highlighted
in bold.
Method
A=0.2
A=0.5
A=0.8
Best
Best Performance
DER (Baseline)
33.6
DER + IM
36.53
40.6
25.95
0.5
40.6
DER + EM
33.29
32.3
30.37
0.2
33.29
DER + EWC
19.09
20.5
13.89
20.5
DER SI
32.8
29.5
28.07
0.2
32.8
19

---

Published in Transactions on Machine Learning Research (01/2026)
Table 12: Feature Drift Comparison Across Regularization Targets. Feature drift is measured as mean L2 distance between class-specific feature representations after initial task learning versus after all tasks completed. Results are reported Split-CTFAR100 with Experience Replay baseline (buffer size 500).
Task
Method
Mcan Drift
Task 1
No IM (Baseline)
21.21
Task 1
IM-CT
23.80
Task 1
IM-BF
24.26
Task 2
No IM (Baseline)
9.74
Task 2
IM-CT
12.01
Task 2
IM-BF
11.20
Table 13: Per-Task Accuracy After All Tasks Learned (Split-CIFAR100, ER with buffer size
500). IM-CT substantially outperforms baseline on all old tasks (T1-T9) with expected trade-off on most recent task (T10).
Task
No IM
IM-CT
IM-BF
T1
8.5
18.8
9.3
T2
7.1
22.7
6.7
T3
11.4
34.8
12.6
T4
6.4
19.0
7.7
T5
18.6
35.0
21.0
T6
13.2
32.1
10.2
T7
15.2
29.0
16.7
T8
19.5
32.1
20.5
T9
23.9
41.4
26.6
T10
92.0
77.2
93.2
Mean (T1-T9)
13.8
29.4
14.6
Table 14: Multiple Runs Analysis: DER on Split-CIFARI00. Results report mean accuracy and standard deviation across independent runs. Improvement margins consistently exceed standard deviations, indicating statistical reliability.
Memory Size
DER (mean + std)
DER + IM (mean + std)
Improvement
500
33.57 1.42
41.10 0.55
+7.53
1000
43.35 + 0.68
49.28 + 0.52
+5.93
2000
51.46 + 0.52
53.89 + 0.71
+2.42
20


Reviews:
[{"review_id": "jGQmxs1wOJ", "reviewer": "Reviewer_Ebdg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a regularization-based method for continue learning named information maximization. Compared to classical rehearsal-based approaches, it add two additional objectives: 1) an entropy loss to encourage confident predictions; 2) a diverse loss to promote diverse predictions across the batch. The method is validated in CIFAR-100 and Tiny ImageNet, demonstrating improved accuracy and less forgetting over baseline methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "While most claims are accurate, there are some missing experiments that make the results unconvincing:\n* The paper does not detail the tuning process for the critical hyperparameters (e.g., the $\\lambda$ for EWC or SI) for these competing methods; the claim that IM is superior to other regularization strategies is not fully supported.\n* The authors find that applying the IM regularizer only to the current task (CT) samples yields the best performance (Table 3). This is counter-intuitive for a method designed to reduce forgetting of old tasks. One would logically expect the most effective regularization to be applied to the rehearsal buffer (BF) samples, which directly represent the old knowledge.", "audience_yn": "Yes", "audience_explanation": "The researchers in the community might find this method interesting as it is relatively light-weight compared to the traditional memory-based methods.", "requested_changes": "* **Fair hyperparameter search for baseline regularizers.** The current claim that IM is superior to EWC, SI, and EM is not convincingly supported. The performance of these baseline regularizers, particularly EWC, is anomalously poor when combined with rehearsal methods (e.g., dropping DER/DER++ accuracy by nearly half). The authors shall conduct and report a fair hyperparameter search for the competing regularizers (EWC, SI, EM).\n* **More comprehensive explain of the \"current task (CT) only\" result.** The ablation study in Table 3 finds that applying the IM regularizer exclusively to the current task samples (CT) yields better performance than applying it to the buffer (BF) or both (ALL). The authors shall add a new analysis section to investigate why CT-only regularization is the most effective strategy for preventing forgetting, which should be beyond reporting accuracy and explore the underlying dynamics. For instance, authors may analyze the feature representations to see if CT regularization prevents feature drift or collapse for old classes stored in the buffer.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fEJoWP5Q13", "reviewer": "Reviewer_6WE2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The topic of improving continual learning performance is important and broadly relevant. The proposed approach is clearly described (adding additional regularization terms in the training loss), and the simulation results indicate that it achieves higher accuracy and reduced forgetting.\n\nHowever, a key concern is the lack of justification for the proposed methodology. In particular, the introduction of the term $L_{\\text{div}}$ in Eq. (2) is not well motivated. In fact, $L_{\\text{div}}$ is the core part of the proposed approach. It is unclear why this term is beneficial, or why it appears with a positive sign rather than a negative one like $L_{\\text{ent}}$. As written, $L_{\\text{div}}$ seems ad hoc and its role is confusing.", "claims_and_evidence_yn": "Yes", "claims_explanation": "There are many simulation results to illustrate the performance of the proposed approach.", "audience_yn": "Yes", "audience_explanation": "This paper proposed a new approach to improve the performance of continual learning. Researchers in the field of continual learning will be interested.", "requested_changes": "1. In Eq. (2), the meanings of the subscripts \"ent\" and \"div\" are not explained. Please clarify what these terms denote.\n\n2. Based on the definition of $\\hat{f}\\_{\\theta}(x)$ as the expectation of $f\\_{\\theta}(x)$ over $x$, the expression should no longer depend on $x$. I recommend removing $(x)$ from the notation $\\hat{f}\\_{\\theta}(x)$.\n\n3. In the definition of $R\\_{\\text{EWC}}$, the roles of the indices $i$ and $t$ are unclear. Additionally, since $F$ is a matrix, it is ambiguous whether $F\\_i$ refers to a row, a column, or some other component. This should be specified.\n\n4. In the definition of $R\\_{\\text{SI}}$, the summation index is written as $\\sum\\_t^T$; should this instead be $\\sum\\_{t=1}^T$? Also, the quantity $\\omega\\_t^k$ is not defined.\n\n5. It appears that $L\\_{\\text{ent}}$ and $R\\_{\\text{EM}}$ are identical. If that is the case, the notation should be made consistent.\n\n6. All reported simulation results are given as single numbers. Are these results averaged over multiple runs? Reporting the variance or standard deviation would make the evaluation more convincing.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1O4m03cJFE", "reviewer": "Reviewer_dHxS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores continual learning by introducing a regularizer into rehearsal-based methods. Specifically, it proposes an information maximization regularizer applied to the model outputs to encourage learning new tasks while mitigating forgetting of previous ones. The regularizer comprises two components: entropy minimization and diversity maximization. Extensive experiments on both image- and video-based continual learning benchmarks demonstrate the effectiveness of the proposed approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The main contributions of this paper lie in its evaluation, which is supported by substantial experimental results.", "audience_yn": "No", "audience_explanation": "The proposed solution lacks novelty, and the results are neither surprising nor state-of-the-art.", "requested_changes": "1. This paper appears to be a special case of the prior work [1], sharing a similar evaluation protocol and writing style, but it does not provide a direct comparison with that work.\n2. The motivation behind $L_{div}$ is insufficiently explained. It seems to primarily serve as a form of prototype learning.\n3. The set of baselines is too limited; more recent and representative baselines should be included for a fair comparison [2][3].\n4. The three claimed contributions could be consolidated into a single, unified contribution for clarity.\n\nReferences:\n\n[1] A unified and general framework for continual learning. ICLR 2024.\n\n[2] Continual learning with tiny episodic memories. https://arxiv.org/abs/1902.10486.\n\n[3] Dark experience for general continual learning: a strong, simple baseline. NeurIPS 2020.", "broader_impact_concerns": "N.A.", "additional_comments": "See above.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)