Retrospective Feature Estimation for Continual Learning
Authors: Nghia D. Nguyen
Hieu Trung Nguyen
Ang Li
Hoang Pham
Viet Anh Nguyen
Khoa D Doan
Venue: Accepted by TMLR ()

Abstract: The intrinsic capability to continuously learn a changing data stream is a desideratum of deep neural networks (DNNs). However, current DNNs suffer from catastrophic forgetting, which interferes with remembering past knowledge. To mitigate this issue, existing Continual Learning (CL) approaches often retain exemplars for replay, regularize learning, or allocate dedicated capacity for new tasks. This paper investigates an unexplored direction for CL called Retrospective Feature Estimation (RFE). RFE learns to reverse feature changes by aligning the features from the current trained DNN backward to the feature space of the old task, where performing predictions is easier. This retrospective process utilizes a chain of small feature mapping networks called retrospector modules. Empirical experiments on several CL benchmarks, including CIFAR10, CIFAR100, and Tiny ImageNet, demonstrate the effectiveness and potential of this novel CL direction compared to existing representative CL methods, motivating further research into retrospective mechanisms as a principled alternative for mitigating catastrophic forgetting in CL. Code is available at: https://github.com/mail-research/retrospective-feature-estimation.

Published in Transactions on Machine Learning Research (01/2026)
Retrospective Feature Estimation for Continual Learning
Nghia D. Nguyen 1.2
20nghia wwwwinnaaniinnmmmmmmmmmmmmmmmvmm edu, un thnguuen Üse. cuhk. edu. hk ang@simular ai
Hieu Trung Nguyen
Ang Li
Hoang Pham
hoong. Saaagg.pham@uorwiiee ac. uk
Viet Anh Nguyen²
nguuyen Cssc. cubk edu. hk
Khoa D Doan
khou, dd wineanii edu
VinUni-
Umnn--linnis Nlinois Smart Health Center
University of Mllonoi Urbana-Chaen Champaign
The Chinese University of Hong Kong
Simular Research University of Warwick
Reviewed on OpenReview: https: // openreview. net/ forum? id= 9Wnh VME446
Abstract
The intrinsic capability to continuously learn a changing data stream a desideratum of deep neural networks (DNNs). However, current DNNs suffer from catastrophic forgetting, which interferes with remembering past knowledge. To mitigate this issue, existing Continual Learning (CL) approaches often retain exemplars for replay, regularize learning, allocate dedicated capacity for new tasks. This paper investigates called Retrospective Feature Estimation (RFE). RFE learns to reverse feature changes by aligning the features from the current trained DNN backward to the feature space of the old task, where performing predictions is easier. This retrospective process utilizes a chain of small fcature mapping networks called retrospector modules. Empirical experiments on several CL benchmarks, including CIFAR10, CIFAR100, and Tiny ImageNet, demonstrate the effectiveness and potential of this novel CL direction compared to existing representative CL methods, motivating further research into retrospective mechanisms a principled alternative for mitigating catastrophic forgetting in CL. Code is available at: https://
unexplored direction for CL
1 Introduction
Humann exhibit the innate ability to incrementally learn new concepts while consolidating acquired know]- odge into long-term memorics (Rasch & Born, 2007). More general Artificial Intelligence systems in real-world applications would require similar imitation to capture the dynamics of the changing data stream. These systems need to acquire knowledge incrementally without retraining, which is computationally expensive and exhibits a large memory footprint (Rebuffi et al., 2016). However, existing learning approaches cannot match human learning in this so-called Continual Learning (CL) problem due to catastrophic forgetting (MeCloskey & Cohen, 1989). These systems encounter difficulty in balancing the capability of incorporating new task knowledge while maintaining performance on learned tasks, or the plasticity-stabille dilemma.
Representative CL approaches in the literature usually involve the use memory buffer for rehearsal (Rat- cliff, 1990; Chaudhry et al., 2019a; Buzzega et. al., 2020; Caccia et al., 2022; Bhat et al., 2023; Arani et al.,

---

Published in Transactions on Machine Learning Research (01/2026)
2022: Prabhu et al., 2020), auxiliary loss term for learning regularization (Kirkpatrick et al., 2017; Ebrahimi et al., 2020; Zenke et al., 2017; Schwarz et al., 2018), or structural changes such as model pruning, growing or finding sub-networks (Rusu et al., 2016; Mallya & Lazebnik, 2018; Fernando et al., 2017; Yan et al., 2021; Serra et al., 2018; Wortsman et al., 2020; Kang et al., 2022). These methods share the common objective of discouraging the deviation of learned knowledge representation. Rehearsal-based methods allow the model to revisit past exemplars to reinforce previously learned representations. Alternatively, regularization- methods prevent changes in parameter spaces by formulating additional loss terms. However, both ap- proaches present shortcomings, including keeping rehearsal buffer of all tasks during the model's lifetime or infusing ad hoc inductive bias into the regularization process. Meanwhile, structure-based methods utilize the over- peee-parammttriiztioon property of the model by pruning, masking, adding parameters, or finding suitable Lub-neewwrkss to reduce new task interferences.
This paper studies a novel approach for CL named Retrospective Feature Estimation (RFE), where we al- low the model "forget" knowledge of old tasks but then "correct" such "catastrophic forgetting" during inference using sequence of lightweight feature mapping networks. These networks, called retrospector modules, help significantly reduce information loss in learned tasks by incrementally reversing changes in the feature space. Specifically, for cach new task, we add a small, simple, and incxpensive auxiliary unit that aligns the feature from the current task to the previous task. Our method differs from many network expansion methods, in which additional parameters are allocated to minimize changes to the old parameters. Instead, extra modules are used to iteratively recover past representations by propagating the current rep- resentation backward through series of mapping networks. With this mechanism, RFE allows the optimal learning of new task (plasticity) while separately mitigating catastrophic forgetting through retrospectors (stability). RFE does not require saving past data to achieve strong performance, but flexibly utilize past data to further improve capability. In addition, different from several CL approaches that heavily modily the training or network architecture, RFE imposes minimal changes to new task learning as modifications are mainly performed after the training has been completed using auxiliary modules. Hence, RFE can be easily integrated into existing CL pipelines.
Contributions. We propose a new paradigm for CL by soquentially correcting the current task's repre- sentation into the past task's representation using a chain of light retrospector modules:
We propose RFE, a novel approach to CL that separates catastrophic forgetting mitigation from new task learning via sequence of light retrospector modules. The proposed retrospector module, by compensating for information loss and reversing feature changes, can incrementally mitigate catastrophic forgetting.
To train the retrospector modules, rely only task t 1's feature extractor, and keeping past data is optional to improve performance. At inference time, for the task-incremental setting, we construct a chain of relrospector modules based on the provided task identity and forward the current features to correct the feature space. For the class incremental setting, RFE forms the final prediction from an average of predictions based on the reconstructed representations.
We empirically evaluate our approach on three popular continual learning benchmarks (CIFAR10. CIFAR100, and Tiny ImageNet) to demonstrate that our approach achieves comparable performance with the existing representative CL directions.
This paper unfolds as follows. Section 2 discusses the literature on CL problems, and Section 3 describes the proposed RFE method. Finally, Section 4 provides the empirical evidence for the effectiveness of our proposed solution.
2 Related Work
Catastrophic forgetting is a critical concern artificial intelligence and arguably among the most promi- nent questions to address for DNNs. This phenomenon presents significant challenges when deploying models in different applications. Continual learning addresses this issue by enabling agents to learn throughout their
2

---

Published in Transactions on Machine Learning Research (01/2026)
lifespans. This aspect has gained significant attention recently (Sun et al., 2022; Hu et al., 2021; Kirichenko et al., 2021; Balaji et al., 2020). Considering a model well-trained on past tasks, we risk overwriting its past knowledge by adapting it to new tasks. The problem of knowledge loss can be addressed using different methods, as explored in the literature (Yin et al., 2020; Farajtabar et al., 2020; Kirkpatrick et al.; 2017: Li & Hoiem, 2018; Chaudhry et al., 2019a; Bhat et al., 2023; Rusu et al., 2016; Yan el al., 2021; Buzzega et al., 2020; Caccia et al., 2022; Arani et al., 2022; Prabhu et al., 2020; Ebrahimi et al., 2020; Zenke et al., 2017; Schwarz et al., 2018; Mallya & Lazebnik, 2018; Fernando et al., 2017; Serra et al., 2018; Wortsman et al., 2020; Kang et al., 2022). These methods aim to mitigate knowledge loss and improve task perfor- mance through three main approaches: (1) Rehearsal- methods, which involve reminding the model of past knowledge by using selective exemplars; (2) Regularization-base
methods, which penalize changes in past lask knowledge through regularization techniques; (3) Parameter-isolation and Dynamic Archilecture methods, which allocate subnetworks or expand new subnetworks, respectively, for each task, minimizing task interference and cnabling the model to specialize for different tasks.
Rehearsal-based. Experience replay methods build and store a memory of the knowledge learned far (Rebuffi et al., 2016; Lopez-Paz & Ranzato, 2017; Shin et al., 2017: Riemer et al., 2018; Rios & Itti, 2019; Zhang et al., 2019; Chaudhry et al., 2019a; Buzzega et al., 2020; Caccia et al., 2022; Bhat ct al., 2023; Arani et al., 2022; Prabhu et al., 2020). As an example, Averaged Gradient Episodic Memory (A-GEM) (Chaudhry et al., 2019a) builds an episodic memory of parameter gradients, while DER (Buzzega et al., 2020) uses a reservoir sampling method to maintain episodic memory. These methods have shown strong performance in past studies, but they require a significant memory to store the examples.
Regularization--basll A popular early work using regularization is the elastic weight consolidation (EWC) method (Kirkpatrick et al., 2017). Other methods (Zenke et al., 2017; Aljundi et al., 2018; Van et al., 2022; Nguyen et al., 2018; Ahn et al., 2019; Ebrahimi et al., 2020) propose different criteria to measure the "importance" of parameters. A later study showed that many regularization- based methods are variations of Hessian optimization (Yin et al., 2020). These methods typically assume multiple optima in the updated loss landscape in the new data distribution. One can find a good optimum for both the new and old data distributions by constraining the deviation from the original model weights.
Parameter Isolation. Parameter isolation methods allocate different subsets of the parameters to cach lask (Rusu et al., 2016; Jerfel et al., 2019; Rao et al., 2019; Li et al., 2019; Serra el al., 2018; Kang et al., 2022). From the plaaiityyp perspective, these methods implement gating mechanisms that improve stability and control plasticity by activating different gates for each task. Masse et al. (2018) proposes a bio-inspired approach for a context-dependent gating that activates a non-overlapping subset of parameters for any specific task. Supermask in Superposition (Wortsman et al., 2020) is another parameter isolation method that starts with a randomly initialized, fixed base network and, for each task, finds a sub-network (supermask) such that the model achieves good performance.
Dynamic Architecture. Different from Parameter Tsolation, which allocates subnets for tasks in fixed main network, this approach dynamically expands the network structure. Yoon ct al. (2018) proposes a method that leverages the network structure trained on previous tasks to effectively learn new tasks, while dynamically expanding its capacity by adding or duplicating neurons as needed. Other methods (Xu & Zhu, 2018; Qin et al., 2021) reformulate CL problems into reinforcement learning (RL) problems and leverage RL methods to determine when to expand the architecture when learning new tasks. Yan et al. (2021) introduces two-stage learning method that first expands the previous frozen task feature representations by a new feature extractor, then re-trains the classifier with current and buffered data.
An orthogonal direction of CL is adapting a frozen pre-trained model to a sequence of tasks (for example, by learning new adapters), which share some similarities to our work but are not directly related. RFE assumes a more challenging scenario with training from scratch and evolving model parameters.
3 Proposed Framework
We consider the task--ncremental learning (TIL) and class-incremental learning (CIL) scenarios, where we sequentially observe set of tasks & (I,...,N) The neural network comprises a single task-agnostic
3

---

Published in Transactions on Machine Learning Research (01/2020)
Unavailable inference
Available inference
D
Input
E D
Feature extractor
f((-)
Te+1
YN-1
TN
fx(()
Approximated
Representation
(a)
(...
fi(z))
fx 2
fN-1(#)
fw
Task 1
Task N-2 representation
Task N-1 representation
Task representation
Classifier
wy(1
representation
Figure 1: At lask i, the feature extractor Ji and classifier head Wi are optimized on the dataset Dirain, During inference for a test sample from task i, we forward the input data x E Diest through the feature extractor and classifier head to obtain the logits. After learning all N tasks, the DNN loses performance on task due catastrophic forgetting. Therefore, the latent representation fx(2) is propagated through series of retrospector module FN,.... r+++1 to perform incremental latent. rectification and obtained approximated representations fN-1....,fe The logits can be obtained by passing the recovered representation to the respective classifier head.
feature extractor / and classifier with lask-specific heads. The archilecture of S is fixed; however, its parameter set O is gradually updated as new tasks arrive. At task i, the system receives the training dataset Dirrin sampled from the data distribution D, and learns the updated parameter sets 0, of the feature extractor f and W. To ease the discussion, the feature extractor and the classifier obtained after learning in task are denoted as ft and Wt, respectively. For an input-label pair (æ: y) sampled from D, the logits computed by is denoted as z. Thus, after learning on task t, we obtain the evolved feature extractor fa and the classifier Wt- We call the latent space created by the feature extractor trained with Dirain as the t-domain. Catastrophic forgetting occurs the feature extractor fe is updated into ft (t' < t), which causes the f'-domain to be overwritten by the t-domain. This domain shift degrades the model's performance over time.
To overcome catastrophic forgetting, we propose new CL paradigm: learning a retrospective feature esti- mation mechanism. This mechanism relies on lightweight retrospector module r, that learns to align the features from the (-domain to the (L - 1))dmainn Intuitively, this module "corrects" the feature change of a sample from the old task 1-1 due to the evolution of the feature extractor when learning the newer task t. These retrospector modules will establish chain of corrections for the features of any task's input, allowing the model to predict the past features better. Fig. provides a visualization of the inference process task-l sample, after learning N tasks.
Learning the mechanism is central to our proposed framework. In general, each retrospector module should small compared to the size the final model or the feature extractor f, and its learning process should resource efficient. The following sections present and describe our solution for learning this mechanism.
3.1 Learning the retrospector
As the training dataset Dirain of task arrives, we first update the feature extractor ft and the classifier W/- The primary goal herein is to find (fe, wy) that has a high classification performance for task f, and the secondary goal is lo choose Jr that can reduce the catastrophic forgelting on previous lasks. To combat catastrophic forgetting, we will first discuss the objective function for learning the lightweight retrospector module r, and the potential options for training data.
3.1.1 Feature Estimation Loss
The goal of r; is to reduce the discrepancy between task t's representation f((x) and the t t-I's representation (x=) for æ D,-1; i.c., re(f((x), = (z). A simple choice the lz error between f=-((x) and (2), Since the feature estimation loss will be reused multiple times in this paper, we define s to be function with parameter set 0. that encodes input x D into its respective features. More specifically,
4

---

Published in Transactions on Machine Learning Research (01/2026)
would serve placeholder for different functions in different training scenarios. We define the loss as:
(rF(O.is,D.ff = EnnD [1]s(2) f(z)
(1)
For retrospector training, at task i, we set s(x) (ft(),), m)) and aim to minimize the difference between and fi-1; therefore, the objective function becomes
D, = EEND [Irr(f((()),)) (
(2)
3.1.2 Training Data
Training the retrospector Tt to map the representation from the f-domain back to the (d-1)-domain requires the representations in both domains (Ji(z),, (x)( as training data. Ideally, the best choice would be to keep all the samples and the respective representations from task However, it is impractical to keep all æ Dirain due to efficiency, scalability, or privacy issucs. A practical approach is to only keep the previous task feature extractor fi-1 and use current task samples ~ Dirain to approximate the mapping of the representation space of the task back to that of the task t - 1. Another approach is to keep only subset P C Dirain of the previous tasks' samples and their respective representation. Nonetheless, the second approach heavily relics on the number samples [P] that could be saved. Therefore, we opt for the first approach to train the retrospector. Keeping additional past data optional and could be used to further improve performance. In this paper, we evaluate three strategics for training RFE.
to train the retrospector module.
Without past task's data (RFE). RFE can effectively recover fa-1(x) from fx) without relying previous task's data. By only keeping the previous task feature extractor fi-1 and using current task data a ~ Dirain as an approximation for Di-1. the retrospector modulc can learn to map from the t-domain to the (L - 1)-domain. It is common for continual learning methods to exhibit performance degradation over time. However, even without access to any past task data, RFE demonstrates comparable performance to several rehearsal- based methods.
With subset P of task 6- I's samples (RFE-P). In addition to keeping fe-1, to improve rectification performance, small subset. of task - 1's samples can also be saved together with their representation (x)) With the use of additional past samples, RFE can sustain classification performance even under long chain of retrospector modules. Therefore, RFE-P can be a good trade-off between performance and privacy since data are only stored with the maximum life cycle of 2 tasks (task t - and task 1) rather than indefinitely as buffer-based methods. However, is possible to generalize this approach to store data from corresponds to RFE-P, meaning only task - I's samples are stored.
the past lasks, where
With buffer B of all tasks' samples (RFE- B). Similar to many buffer-based methods, RFE can also make use of reservoir buller for subset of all tasks" samples to sustain performance under very long retrospector module chaining. Instead of only being used for training the new retrospector mod- ule, these data samples can also be used to tune the learned retrospector modules, ensuring a stable rec- tification chain. On the other hand, unlike RFE and RFE-P, RFE-B can not be used in scenarios where privacy is a major concern.
Table 1: At task t, different training data require stor- ing different components of the training process, which impose different trade-offs in terms of performance and privacy.
Keep ft
Keep
Keep
Variation
P S Dimin
B C _ Ba,,Dmm Drrai
RFE RFE- RFE-B
V
V
V
3.2 Retrospective Feature Estimation
The retrospective feature estimation mechanism relies on chain of task-specific retrospector modules (r.) that aims to correct the distortion of the feature space the extractor f learns new task.
5

---

Published in Transactions on Machine Learning Research (01/2026)
3.2.1 Past Feature Estimation
For an input a at task t - 1, its representation under the feature extractor St-1 is ft-1(x). One can heuristically define the (L 1)-domain as the representation space of the input under the feature extractor 1)-dmmain is brittle under extractor update: as the subsequent task t arrives,
fi-1- Unfortunately, the the feature extractor is updated to fa, and the corresponding features of the same input will be shifted to (x)) Likely, the I-domain and the (1 1)-dmmain do not coincide, and ft(x)) + (z=)
The retrospector module T aims to offset this representation shift. To do this, Y takes æ, and its t-domain reprosentation f((z) as input, and the module outputs the approximated features that satisfics
Tt (((f(()), fi-1(x),
With this formulation, we can effectively minimize the difference between the rectified features ry (f(())))) and the original features (x))
3.2.2 Retrospector's Architecture
Trainable module
Frozen module
Voctor,
f#())
The proposed retrospector module com- prises several trainable components: a weak auziliary feature extractor, soft gat- ings, and linear mappings. The size of the retrospector module increases lin- early with the number of tasks, similar to the classification head. However, since the retrospector module is lightweight, this is trivial compared to the size of the full model. Fig. visualizes the mod- ule. The architecture of the retrospector is described here, with further implemen- tation details included in the Appendix.
0
f(Y
Teb.r)
95
91
f=)(
ft(z))
rx(fe(m))
Cre
r=(f<(z). 2)
The retrospector module includes a weak auxiliary feature extractor h,, linear mappings af at h,, and soft gatings gf g.. The joint information from the projected representations from both ft and h, is used to compute the gating value for the rectified reprosentation.
Figure 2:
Auxiliary feature extractor ht- Due to catastrophic forgetting, the main feature extractor will gradually forget learned knowledge. Therefore, the auxiliary feature extractor will partially compensate for this loss of information. For our experiment, we chose a simple and naive design of an auxiliary feature extractor that has a low performance to demonstrate that the effectiveness of RFE based on retrospective feature estimation capability and not the auxiliary feature extractor's performance. The auxiliary feature extractor he processes the input data x to generate a simplified representation h₂(z). h4 is distilled from fi-1 to compress the knowledge of ft-1 into a more compact, low-capacity parameter-offiiient network. The weak auxiliary feature extractor is composed of only two 3x3 convolution layers and two max pooling layers. For simplicity and efficiency, instead of processing the full-size image, we use max-pooling to down-sample the input to 16x16 images before feeding it into ht. The auxiliary feature extractor is a very small network compared to the main model.
Linear mappings af af, bt To capture only relevant information and reduce noise from the main feature extractor and auxiliary feature extractor, both representations will be linearly projected down to a smaller- dimensional space of o f(x and at o (xx), in which an element-wis multiplication @ is applied to combine both representation information.
ax(2 f((z), = (of o f((x)) o (at o hz()))
(4)
with dim(af) = dim(a*) < dim(fi) (dim is the dimension of the output layer).
On the other hand, as by may have a representation of a different dimension compared to that of ft, another linear projection 6, used to map from ht's dimension to ft' dimension as
M((x)) by (z)
(5)
6

---

Published in Transactions on Machine Learning Research (01/2026)
Soft gatings gf .6. To combine the main feature extractor and auxiliary feature extractor representations fi(x) and he(x)), an element-wist wise gating value of (.) and gf (.) is computed from the encoded joint information (u(z,S((a) (a))). As ge st. af, af: by, hy are components of the relrospector module, we compute the rectified representation as a function of the input a and its current representation f((x)
re(f((z), = (ol o az (((a),)((z)) @ fa) + (ol o az f((x), (x)))) @ (x)
The galing mechanism is simply a linear layer followed by sigmoid activation.
Distinction from eepaork-expansion approach. It could be argued that one can, instead, separately train a weak feature extractor h, for each task, making it a network--cpansion CL approach. However, because he is a small and low-capacity network, this approach is ineffective; specifically, our experiments demonstrate that the task-incremental average accuracy across all tasks of this approach on CTFAR100 falls below 53%. Furthermore, for network expansion approaches, the dedicated parameters are allocated for new task learning, which fundamentally differs from RFE's objective to correct representation changes. The new task's knowledge is acquired by Ji and uy with high plasticity.
3.3 Training Procedure
Network training. Similar to conventional DNN training, the performance of the feature extractor ft and the classifier head Wt is measured by the standard multi-class cross-entropy loss:
Cc U Owii We St, Dirain) ((.a)))Drraa
|
- yo log(ze)
Σ
M,
(6)
where Mj is the number of classes of task t, z is the probability-valuch network output for the input x (hat depends on the feature extractor and the classifier w/ as we fe(x)). Since the retrospector will correct feature changes back to the original feature space, classifier units that are learned from past tasks are excluded (masked) and not updated to prevent mismatch due to gradient updates.
Furthermore, to reduce forgetting and enable more effective rectification, we regularize (or distill) task t - 1 representation knowledge by using the approximated previous task's 's representation from ft-1 (and additional saved data in P or B if available) to train the current feature extractor ft. Let s(x) = fr(x), then we can similarly use LFE in Eq. (1) with hyperparameter a in the training loss Lr:
CO, U Q Dirain US) LCE(O, (0, U J o fi, Dirain) + aCFF((,;) ft. Dirain fi-(
(7)
where S can be the empty set, the set P, or the buffer B corresponding RFE, RFE-P, or RFE-B, respectively.
Retrospector training. Training the retrospector module follows two main steps: train the weak auxiliary feature extractor h4 at task t - 1 and then the remaining components at task t. The weak feature extractor h is distilled from fe-1 as task / - training is completed using CFE(On,; (0, he, Dirain y ft-1) as in Eq. (1) with s(x) = he(z). After training, he parameters are frozen to prevent modifications. Similarly, after the training of task t is completed, we train the remaining components using (Fr \0,,; re, Dirain fc-1) as Eq. (1) with sxx) (x)) For the case of RFE-B, the representations fi()) of input E B are rectified to their corresponding domains i 21,2, 1) and learned retrospector's parameters 0 are also optimized. Details of RFE's training algorithm are provided in Algorithm 1.
RFE imposes minimal changes to the standard training process as the majority of the additional training happens after the main (standard) training ends. However, end-to-end training of all modules can also be used as demonstrated in Section B.3.
3.4 Inference Procedure
We now describe how stack multiple retrospector modules r, into chain for inference. As new task arrives, our model dynamically extends an additional retrospector module, forming a sequence of retrospector modules.
7

---

Published in Transactions on Machine Learning Research (01/2026)
Algorithm Training process at task {1,2,... N). let S be 0, P,B for RFE, RFE-B, RFE-P, respectively // main training starts
optimize Wi o ft with Cr(O,, U U Dirain US)
// main training ends
optimize 14-1 with LFF (on 1 Dirain St) freeze 0h1-1
if <>> 1 then
optimize with Lre(Or, Vn., Dirain fi-)
end if
Table 2: Task-Incremental Average Accuracy across all tasks after CL training. Oracle: the upper bound accuracy when jointly training on all tasks (i.e., multi-task learning). Finetuning: the lower bound accuracy when learning without CL techniques. [B] is the buffer of all past task samples. [P] the subset of task 1-1 training data. params (aainniinnIneeenne)e) is the number of parameters used during training (first value) and inference (second value) (lower is better), and accuracy is the average accuracy of all tasks (higher is better). For RFE, RFE-P, and RFE-B, parameters of all accumulated retrospectors are included for both training and inference.
Method
S-CIFAR10
S-CIFAR100
S-TinyImg
Exemplars
TIL
params
accuracy
params
accuracy
params
accuracy
Oracle
98.37a
86.57=
81.47
11.17/11.17
11.22/11.22
11.27/11.27
Finetuming
60.08 a.1a
24.90 9.58
13.67 0:37
AGEM ER
91.37 0.40
65.50= 0.28
38.73 1.23
93. 79
66.88
44.85
11.17/11.1
11.22/11.22
11.27/11.27
DER++
92.10+ 0.74
68.65+ 0.93
47 17.92+ 0.73
ER-ACE
[8| 500
93.60
67.97
47 47.88-
ER-MKD
22.35/ 1 11.17
93.75
12.44/11.22
70.63+ 0.80
22.54/11.22
51.89+ 0.24
TAMIL
22.68/ 22.68/11.. 11.51
94.56+ 0.09
22.77 11.60
75. 12+ 0.25
23.20/12.08
63.28 0.03
CLS-ER
33.52/ 33.52/11.17 /1.177
94.99 n.25
33.66/11.5 1..22
76.79= 0.17
33.81 33.81/11.22
50.28 1.03
RFE-P
PP 500
92.94 0.92
80.57 0.41
71.80 0.61
23.76/12.59
23.81/12.64
25.63/14 1.63/11.46
RFE-B
[ 500
91.35
80.69
69.91
AGEM
90.26+
69.91 + 0.02
45.58 1.16
94.91+ 0.64
72.17 0.42
53.98 1.08
11.17/11.17
11.22/11.22
11.27/11.27
DER++
93.35
72.90
57.17
ER-ACE
B 1000
941.33 0.30
72.36 0.68
56.96 0.51
ER-MKD
22.35/11.17
95.28 0.04
22.21/11.22
74.04= 0.43
22.54/11.27
57.55 0.53
TAMIL
22.68 /1.51
95. 11
22.77 22.77/11.60 11.60
77.94 0.95
23.20/12.00
68.81
CLS-ER
83.52/114 /11.17
96.02 + 0.16
33.66/11.22
79.82= 0.11
33.81/11.27
60.78 0.40
RFE-P
IP] 1000
92.92 0.92
80.64 0.59
72.65 0.56
23. 23.76/12.59
23.81/12.64
25.63/14.44
RFE-B
B 1000
90. 74 1.85
81.06 0.28
71.92 0.34
RFE
23. 23.76/12.50 12.59
91.15s
23.81 23.81/12.61
79.54= 0:27
25.63/11.46
69.66 0.17
Task- mnleemnnceee We consider a task-incremental learning setting where test sample æ; is coupled with task identity f; E f1,..., .N. To classify æ, recover fr, (z) by forwarding the current representation fx(2) through chain of N t, retrospector modules. We then pass this recovered latent variable through the classifier head WA, to make prodiction. The output ÿ; is computed as
9. Wi, (fr, (x))
(8)
8

---

Published in Transactions on Machine Learning Research (01/2026)
where (x) ((.t))))..((()()).)))))))))))))) with t, < <N, fN-
Class-nncremental. RFE relies on the task identity to reconstruct the appropriate sequence of retrospector modules for propagating the features to the original space. However, no identity is provided for the CL method in the caassinnrrmmetal learning setting. We instead provided a simple method for inference without task identity, which demonstrates the method's 's extension to class-incremental learning; however, more robust task-identity inference methods could also be incorporated.
We obtain the class- incremenalt probabilities by forming an average of the class probabilities over all do- mains. For each domain, irrelevant classifier units (not belonging to the task) are excluded (masked) before computing the softmax probability. More specifically, from the current task Nss domain, we extract the representation (x: and iteratively rectify the latent back to lask N N task N 2, ... lask I's domain. These representations (xx and fr, (=) are then forwarded through respective classifier Wt, li to compute the softmax probabilities. We then average the softmax probabilities of all domains.
4 Experiments
Our implementation is based partially on the Mammoth (Boschini et al., 2022; Buzzega et al., 2020), TAMiL (Bhat et al., 2023), and CL.S-ER (Arani et al., 2022) repositories.
4.1 Evaluation Protocol
Datasets. We select three standard contin- ual learning benchmarks for our experiments: Sequential CIFAR10 (S-CIFAR10), Sequen- tial CTFARIO0 (S- (S-CIFAR100), and Sequential Tiny ImageNet (S-Tinylmg). Specifically, we divide S-CIFAR10 into binary classification tasks, S-CIFAR100 into tasks with 20 classes cach, and S-TinyImg into 10 tasks with 20 classes each.
Table 3: Class-Tncremennal Average Accuracy across all
lasks after CL training. The settings are similar to Table 2.
Method Exemplars
S-CIFAR100
S-Tinylmg
Oracle
71.15+ 0.68
58.23+ 0.21
Finetuning
17.65
7.73+
AGEM ER
24.75=
9.30
28.60= 0.88
10.09 0.08
DER+
38.90+
13,50+ 0.34
ER-ACE
[3] = 500
40.18 0.80
17.361 0.08
Baselines. We evaluate RFE against strong rehcarsal-based CL methods, including ER (Chaudhry el al., 2019b), AGEM (Chaudhry et al., 2019a), DER++ (Buzzega et al., 2020), ER-ACE (Caccia et al., 2022), CLS-ER (Arani et al., 2022), TAMiL (Bhat et al., 2023), and ER-MKD Michel et al. (2024). We further pro- vide an upper and lower bound for all methods by joint training on all tasks' data and fine- tuming without catastrophic forgetting mitiga- tion. We employ ResNet18 (He et al., 2016) the feature extractor for all benchmarks. The classifier comprises fixed number of linear
ER-MKD
34.39 1.16
12.644 0.54
TAMiL
44.43= 1.04
20.48 0.56
CLS-ER
50.68 0.01
20.25 0.48
RFE-P
PP 500
45.53= 0.08
29.25 0.58
RFE-B
[B] 500
45.85 0.58
27.29 0.50
AGEM ER
26.66
9.73a
34.77= 0.64
13.47 + 0.98
DER++
45.92
20.14+ 0.80
ER-ACE
[B] 1000
46.88 = 0.86
22.96+ 0.44
ER-MKD
38.85= 0.90
16.09 1.18
TAMiL
50.37 = 1.00
28.21 0.78
CLS-ER
56. 16.188.. 0.20
27.45 0.68
heads for each task.
RFE-P
P 1000
45.85 0.54
30.58 0.28
RFF-B
B 1000
46.01 2
30.471
Additional results and further details on datasets, baselines, and hyperparameters are provided in the supplementary materials.
RFE
43.63 1.29
26.59+ 0.44
4.2 Results
Task-ineremental. Table shows the performance of RFE, RFE-P, RFE-P, and other CL methods on multiple sequential datasets, including S-CIFAR10, S-CIFAR100, and S-Tinylmg. From the table, we see that RFE-P and RFE-B achieve results comparable to the baselines S-CIFAR10. On S-CIFAR100 and
9

---

Published in Transactions on Machine Learning Research (01/2026)
S-Tinylmg, RFE, without any past data, is equivalent to outperforms all the baselines, inchuding strong rehearsal-based methods such as TAMiL and CLS-ER, indicating its ability to estimate feature representation retrospectively, Notably, given either P or B, RFE-P and RFE-B outperform the baselines.
Class-iccremental. Table 3 demonstrates the extension of RFE-P and RFE-B to class-incrementee settings. As the class-incremental probabilities are obtained through simple averaging, which may suffer from an overconfident classifier, we can achieve performance comparable other methods on S-CIFAR100. On S- Tinylmg, RFE-P and RFE-B demonstrate further optimized by integrating task identity prediction methods (i.e., OOD detection) Kim et al. (2022), which are left as potential improvements to avoid complicating the method.
improvement over other methods. The performance can be
Long chaining. Continual learning methods, including rehearsal-basee approaches, often experience per- formance degradation over long task sequences. In Fig. 3, we demonstrate that RFE-P and RFE-B exhibit less forgetting than several continual learning methods across the ten tasks of T-nnIIm.. Fig. 4, we demonst.rate the evolving average accuracies over 20 tasks of S-Tinylmg, in which RFE is more stable and consistently improves over other methods.
DER
75
CLS-ER
TAMiL
Train T2 65 69
71
78
69 72 79
Train T3
-63 64 68
80
71
71
75
79
69 71
72
81
Train T4 -4-50 61 64
68 67
76
73
68 68
75 76
Train T5
8D
77
67
82
Train T7
54 55
53
60
61
60
60
60
68
65
66 67
73
65
73
70
82
Train T8
54 54
56
60
52
57
55
62 75
60
59
62
67
57
63
61
69
68
65 65
68
71
63
70
67
73
Train T9
53
54 54
49
56
51
62 77
56
58
60
65
55
60
57
64
71
65
65
63
70
64
70
72
Train TO 75
RFE
75
RFE-P
75
RFE-B
Train T1
-72 71
74 77
74 77
Train T2
71
T3
72--7777
72
Train 48 67 71
76
71 69
75
73
70
77
76
Train TS
67 68 70
77
73 82
70
69
72
79
74
81
73 70
71
76
73
60
Train T6
65 64 66
74 71 79
69
66
77
72
79
77
71
68
75
71
78
76
Train
68
77
Train T9
63 63 67 66
71 69
67 65 8
69 76 73
80
71 68
71 67 74 68
80
Test TO
Jest
Test T2
Test
Test 14
Test 15
Rest T6
Test T7
Test
Test T9
Test TO
Test T2
Test T2
Test
Test T4
Jest 15
Rest T6
Test T7
Test T8
Test
1est TO
Test
Test T2
Test
T3 T4
Rest
Test 15
Kest T6
Test T7
Test 19
The TIL accuracy with 1000 exemplars vertical axis represents the task the model has been trained on. The horizontal axis represents the task identity. The value in the cell is the task's accuracy. RFE-P demonstrates forgetting rate comparable to or better than other methods without revisiting distant task samples. RFE-B performance is more stable for long chaining.
Figure 3:
10 tasks of S-Tinylmg (lighter color is better). The
Overhead. Further details on time and space overhead are the supplementary materials, Sections B. and B.2.
Other architectures. Further results on VIT (Dosovitskly al., 2020) are in the supplementary materials, Section B.5. RFE demonstrates strong performance across both architectures.
4.3
Retrospector Experiment
While the retrospector has several components, the core idea to efficiently combine f(x) and h(xx) to rectify the representation using soft gating and linear layers for dimension mapping. Therefore, the components are tightly integrated and cannot be easily decomposed for individual testing. We cannot remove h,(x)
10

---

Published in Transactions on Machine Learning Research (01/2026)
0.80
Average Accuracy
0.75
RFE-P
0.70
RFE-B
TAMiL
0.65
DER++
0.60
2
3
4
5
10
11
12
13
14
15
16
17
18
19
20
After
Task
Figure 4: The evolving TIL average accuracies of CL methods with 1000 exemplars on 20 tasks of S- Tinyyyng, RFE-P and RFF-B consistently improve over baseline.
Tusk 2 rectifiee to Task
Task metified to Task
Task rectified to Task
Tisk S rectified to Task
20
15
10
Original
Original
Original Rectilied
Original Drikted
Rectified
Rectrlle
Rectified
15
Drifted to Original Rectified to Original
Drifted to Original Rectifled to Driginal
Drifted to Original Rectified Original
Dritted to Original Rectified to Original
RMSE
0.97
0.57
1.21
Figure We employ PCA to visualize the rectified latent space after training on task and predicting task t'(t' of S-CIFAR100. By visualizing the original representation (f))))) the drifted representation (fi())), the rectified representation (x)), we demonstrate RFE effectiveness. The closer the rectified representation and the original representation, the better the performance. For Eq. (7), set 0 (no regularization) to clearly visualize the catastrophic forgetting and retrospector module performance.
we cannot recover fi-1 with alone due to catastrophic forgetting. However, can study the overall effectiveness of the retrospector by visualizing the representation of the "catastrophic forgetting" network and the reetified result by the retrospector. In Fig. 5, we utilize Principal Component Analysis (PCA) to visualize the latent space. The new representations of past data (red) after learning new tasks change significantly from the original representation (blue), which demonstrates catastrophic forgetting. With RFE, the rectified representations (green) align with the 'true' representations (blue), supporting the empirical offectiveness of our framework. The RMSE between representations is also computed. Additional ablation alternative retrospector designs in the supplementary materials, Section B.3.
5 Limitations
We have shown the potential and high utility of RFE's continual learning mechanism in this paper. Neverthe- less, RFE also has some limitations. Despite being lightweight, RFE still maintains additional parameters, i.c., the retrospector module, which incurs an additional overhcad as the number of tasks increases. In- ference cost. for a significantly long chain would be considerable, which can be improved with modified chaining methods such as skipping (i.e., building a retrospector every two tasks). RFE does not outperform thea-- methods on the S-CIFAR10 dataset, which may indicate potential instability in rectifying fine-grained details of the representation. Additionally, since RFE relies on the task identity to reconstruct the retrospector sequence, application to class-incremental learning requires either inferring task identity
11

---

Published in Transactions on Machine Learning Research (01/2026)
averaging predictions. The current approach might suffer from overconfident classifiers. Class-incremental learning is still an open research area, where more effective adaptations of RFE can be discovered.
6 Conclusion
This work proposes new CL paradigm. RFE tackles catastrophic forgetting through its novel Retrospective Feature Estimation mechanism that learns to align the newly learned representation of past data to their past representations. Unlike existing CL methods, RFE operate as data-free method while achieving comparable performance to rehearsal-based methods. Additional past data is optional and can be used to improve performance. Furthermore, RFE imposes minimal modification to task learning, as most of the training for rectification occurs after main task training.
References
Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon. Uncertainty-based continual learning with adaptive regularization. In Advances in Newral Information Processing Systems, pp. 4394-4404, 2019.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Timne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pp. 139-154, 2018.
Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general continual learning method based on complementary learning system. In International Conference on Leurning Representa- tions, 2022.
Yogesh Balaji, Mehrdad Farajtabar, Dong Yin, Alex Mott, and Ang Li. The effectiveness of memory replay in large scale continual learning, 2020. URL https://arxiv.. org/ http//bs//000..... 02418.
Prashant Shivaram Bhat, Bahram Zonooz, and Elahe Arani. Task-aware information routing from com- mon representation space in lifelong learning. In The Eleventh International Conference on Learning Representations, 2023.
Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara. Class- incremental continual learning into the extended der-verse. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advonces in Neural Information Processing Systems, volume 33, pp. 15920- 15930. Curran Associates, Inc., 2020.
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. In International Conference on Learning Representations, 2022.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Effiiient lifelong learning with A-GEM. In International Conference Learning Representations, 2019a.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, P Dokania, P Torr, and M Ranzato. Continual learning with tiny episodic memories. In Workshop on Multi- Task and Lifelong Reinforcement Learning, 2019b.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Golly, Jakob Uszkoreit, and Neil Houlsby, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference Learning Represendations, October 2020.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Sayna Ebrahimi, Mohamed Elhosciny, Trevor Darrell, and Marcus Rohrbach. Uncertainty-guidet continual learning with bayesian neural networks. In International Conference on Learning Represenlations, 2020.
URL
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference Artificial Intelligence and Statistics, pp. 3762-3773. PMLR,
2020.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks, 2017. URL http::///axxiioor/abbbba
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770 778, 2016.
Huiyi Hu, Ang Li, Daniele Calandriello, and Dilan Gorur. One pass imagenet. In NearIPS 2021 Workshop on ImageNet: Post, Present, and Future, 2021. URL http::///ppppoeeeee..eee//rrorrd
Ching-Y? Yi Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen. Compacting, Picking and Growing for Unforgetting Continual Learning. Advances in Neural Informa- tion Processing Systems, volume 32. Curran Associates, Inc., 2019.
Chassen Jerfel, Erin Grant, Thomas L. Griffiths, and Katherine A. Heller. Reconciling meta-learning and continual learning with online mixtures of tasks. In NewrIPS, 2019.
Haeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jaehong Yoon, Mark Hasegawa- Johnson, Sung Ju Hwang, and Chang D. Yoo. Forget-free Continual Learning with Winning Subnetworks. In Proceedings of the 39th International Conference on Machine Learning, pp. 10734-10750. PMLR, June 2022.
Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Ke, and Bing Liu. A thcoretical study on solving continual learning. In Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022.
Polina Kirichenko, Mehrdad Farajtabar, Dushyant Rao, Balaji Lakshminarayanan, Nir Levine, Ang Li, Huiyi Hu, Andrew Gordon Wilson and Razvan Pascanu. Task-agnostic continual learning with hybrid probabilistic models. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021. URL http:p////pppeeeeeeeeeeeeterrrrs
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaune Desjardins, Andrei A Rusu, Barsinska et al. Overcoming catastrophic
Kieran Milan, John Quan, Tiago Ramalho, Agnieszka forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521 3526, 2017.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to Grow: A Continual Strue- ture Learning Framework for Overcoming Catastrophic Forgetting. In Proceedings of the 36th International Conference on Machine Learning, pp. 3925 3934. PMLR, May 2019.
Zhizhong Li and Derek Hoiem. Learning without Forgetting. IEEE Transactions on Pallern Anolysis and Machine Intelligence, 40(12):2935-290) 2947, December 2018. ISSN 1939-3539. doi: 1..109/TPAMM.200 2773081.
David Lopez-Pas Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruning. 2018 IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2017.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks single network by iterative pruning. In Proceedings of the IEFE Conference on Computer Vision and Pattera Recognition, pp. 7773, 2018.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Nicolas Y Masse, Gregory D Grant, and David J Freedman. Alleviating catastrophic forgetting using context- dependent gating and synaptic stabilization. Proceedings of the National Academy of Sciences, 115(44): 10467-10475, 2018.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist. networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109 165. Elsevier, 1989.
Nicolas Michel, Maorong Wang, Ling Xiao, and Toshihiko Yamasaki. Rethinking momentum knowledge distillation in online continual learning. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of ICML'2, '24, pp. 35607-35622 Vienna, Austria, July 2024. JMLR.org.
Cuong V. Nguyen, Yingzhen Li, Thang Bui, and Richard Turner. Variational continual learning. International Conference on Learning Representations, 2018.
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In European Conference on Computer Vision, pp. 5400, 2020.
Qi Qin, Wenpeng Hu, Han Peng, Dongyan Zhao, and Bing Liu. Brs: Building network structures dynamically for continual learning. Advances in Newral Information Processing Systems, 34:20608 20620,, 2021.
Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascamu, Yee whye Teh, and Raia. Hadsell. Continual unsupervised representation learning. In Advances in Neural Information Processing Systems, pp. 7645 7655, 2019.
Björn Rasch and Jan Born, Maintaining memories by reactivation. Current Opinion in Neurobiology, 17(6): 698-703, 2007.
Roger Ratcliff. Connectionist models recognition memory: Constraints imposed by learning and forgetting functions. Psychology Review, (7(2):285-300 308, April 1990.
Sylvestre-Alvise Alvise Rebulli, Alexander I Kolesuikov, Georg Sperl, and Christoph H. Lampert. iCaRL: Incre- mental classifier and representation learning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5543, 2016.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and and Gerald Tesauro. Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference. In Interna- lional Conference on Learning Representations, September 2018.
Amanda Rios and Laurent Itti. Closed-loop memory GAN for continual learning. In Proceedings of the 28th International Joint Conference on Arlificial Intelligence, IJCAL11, pp. 3332 3338, Macao, China, August 2019. AAAI Press.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukeuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. ar preprint arXiv:1606. 04671, 2016.
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska- Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International Conference on Machine Learning, pp. 4528 4537. PMLR, 2018.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming Catastrophic Forgetting with Hard Attention to the Task. In Proceedings of the 35th International Conference Machine Learning, pp. 4548-45500 4557. PMLR, July 2018.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pp. 2990-2999, 2017.
Shengyang Sun, Daniele Calandriello, Huiyi Hu, Ang Li, and Michalis Titsias. Information-thcorcti online memory selection for continual learning. In International Conference on Learning Representations (ICLR), 2022.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Linh Ngo Van, Nam Le Hai, Hoang Pham, and Khoat Than, Auxiliary local variables for improving regu- larization/pris prior approach in continual learning. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 16 28. Springer, 2022.
Johannes Oswald, Christian Henning, Benjamin F. Grewe, and João Sacramento. Continual learning with hypernetworks. In International Conference Learning Representations, 2020. URL https://
spenreview.net/forre wtt.:e//oorromidddssgeeeeeee.ee
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in Superposition. In Advances in Neural Information Processing Systems, volume 33, pp. 15173-1514 Curran Associates, Inc., 2020.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. S. Bengio, H. Wallach, H. Larochelle, K. Grau- man, N. Cesa- Cesa-Bianchi, and R. Garnett (ods.), Advances in Neural Information Processing Systems, vol- une 31. Curran Associates, Inc., 2018.
Shipeng Yan, Jiangwei Xie, and Xurning He. Der: Dynamically expandable representation for class incre- mental learning. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 2014-3023, 2021.
Dong Yin, Mehrdad Farajtabar, Ang Li, Nir Levine, and Alex Mott. Optimization and generalization of regularization-bese continual learning; a loss approximation viewpoint. arXiv preprint arXiv:2006. 10974, 2020.
Jaehong Yoon, Eunho Yang, Jungtae Lee, and Sung Ju Hwang, Lifelong learning with dynamically expand- able networks. In Sirth International Conference on Learning Representations. ICLR, 2018.
Friodemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. International Conference on Machine Learning, pp. 3987-39979999 3995. PMTR, 2017.
Mengmi Zhang, Tao Wang, Joo Hwee Lim, Gabriel Kreiman, and Jiashi Feng. Variational prototype replays for continual learning. Xivi preprint arXiv:1905 190g..94777 2019.
15

---

Published in Transactions on Machine Learning Research (01/2026)
A
Detailed Experimental Setup
A.1 Baselines
We evaluate RFE, RFE-P, RFE-B, ER, AGEM, DER++, ER-ACE, ER-MKD, CLS-ER, and TAMiL. More specifically, ER, AGEM, DER++, and ER-ACE are common rehearsal baselines that utilize a simple buffer. ER-MKD, TAMiL, and CLS-ER utilize additional exponential moving average (EMA) backbones for dis- tillation. ER-MKD and TAMiL utilize 1 EMA backbone while CLS-ER utilizes 2 EMA backbones during training, which explains the high parameter usage in Table 2. TAMiL further uses task-specific auto-encoders during training and inference, which is also reflected in Table 2. RFE, RFE-P, RFE-B utilizes 1 additional backbone (training-only and retrospectors (training and inference).
Due to the usage of additional EMA backbones and hioloyy-inspiree design, TAMiL and CLS-ER are SOTA rehearsal-based methods. Prior works that compare with CLS-ER and TAMiL often use the non-EMA version, while we use the stronger EMA version for fair comparison against our methods.
For comparison, we provide RFE-B and rehearsal-based methods with a buffer B with a max capacity of 500 and 1,000 samples. For RFE-P, we also provide a set P consisting of task i --1 data with 500 and 1,000 samples for fair evaluation.
For RFE-B, ER, DER++, ER-ACE, TAMiL, and CLS-ER, we employ the rescrvoir sampling strategy remove the reliance on task boundaries as in the original implementation. On the other hand, RFE, RFE-P, RFE-B, AGEM, and TAMiL rely on the task boundary to learn the retrospector module, modify the buffer, or add new task-attention module, respectively. For ER-MKD, perform standard augmentation instead of multi-view augmentation. For TAMiL, we use the best-reported task-attention architecture. For CLS-ER, we perform inference using the stable backbone per the original formulation.
A.2 Datasets
To demonstrate the effectiveness of our mothod, we perform empirical evaluations on three standard con- tinual learning benchmarks: Sequential CIFARIO (S-CIFARI0), Sequential CIFAR100 (S-CIFARI00), and Sequential Tiny Tmage.Net (S-TinyImg). The datasets are split into 5, 5, and 10 tasks containing 2, 20, and 20 classes, respectively. The dataset of S-CIFAR10 and S-CIFAR100 each includes 60,000 32 x 32 images split into 50,000 training images and 10,000 test images, with each task occupying 10,000 training images and 2,000 testing images. The dataset S-TinyImg contains 110,000 64 x 64 images with 100,000 training images and 10,000 test images divided into ten tasks with 10,000 training images and 1,000 test images each. We augment the data using random horizontal flips and random image cropping for each training and buffered image.
A.3 Training
We employ ResNet18 (He et al., 2016) as the feature extractor for all methods and benchmarks. By default, the Resnet18's output dimension is dim(f) 512. For RFE's retrospector, we set dim(h) = dim(a dim(a") 128.
The training set of each task is divided into 90%-10% for training and validation. All methods are optimized by the Adam imizer available in Py Torch with learning rate of 5 x 10-1. As the validation loss plateaus for three epochs, we reduce the learning rate by 0.1 times. Each task is trained for 40 epochs. For RFE, we train hy and rt using the same formulation with Adam optimizer at a lcarning rate of 5 x 10-3 for 40 epochs.
A.4 Hyperparameter search
For all methods, experiments, and datasets, we perform a grid search over the following hyperparameters in Table using a validation set of 10% of the training data. Some hyperparameters are obtained directly from their original paper or implementation to narrow the search range. The final results are the average over 3 runs with the best hyperparameter, with different random seeds.
16

---

Published in Transactions on Machine Learning Research (01/2026)
For RFE, single value hyperparameete search is already sufficient in most cases.
Table 4: Hyperparameter search. See the original paper for each specific hyperparameter,
Hyperparameters Search range
Method
Implementation name
Joint, Finetuning, ER, AGEM, ER-ACE
a 0..2.
distill weight
+
B (0.5, 1.0)
replay weight
" E (0.5, 0.8)
plastic frequency
r, E (0.2, 0.5]
stable frequency
ap E (0.999)
plastic alpha
CLS-ER
0, (0.999)
stable alpha
X (0.2,0.5)
distill weight
(1.0}
replay weight
a (0.5, 13
replay weight distill weight
B (..2.
TAMIL
À (0.1)
pairwise weight frequency ema alpha
E {0.05]
E {0.999}
Aa E [2,4]
distill weight temperature
ER-MKD
T E {2
1 (0.99)
ema alpha
RFE, RFE-P, RFE-B
a E {1)
regularize weight
A.5 Retrospector module
Parameters. The total number of parameters for each retrospector module is 0.35 million, with 0.08 million parameters occupied by the auxiliary feature extractor.
Auxiliary feature extractor. We provide the architecture of the auxiliary feature extractor hy Table We chose simple design of two 3x3 convolution layers and two max-pooling layers. Depending on the use cases, robust feature extractor design be used improve performance and serve lower bound for the RFE. Nonetheless, to demonstrate that RFE depends the rectification capability and not only the auxiliary feature extractor's performance, we opt to use a low-performance design.
Table 5: Architecture of the auxiliary feature extractor by We use ReLU activation after each convolution.
Layer
Channel
Kernel
Stride
Padding
Output size
Input
3
16 16
Conv 1
64
3 x 3
2
8 x 8
MaxPool
2
4 x
Conv
128
3 3
2
2 x
MaxPool
2
17

---

Published in Transactions on Machine Learning Research (01/2026)
B Additional Experimental Results
B.1 Time complexity
We report the training and inference time of the class-incremental setting on the S-Tiny dataset in Table 6 to demonstrate the time overhead by the retrospector. For RFF, there is almost overhead in training compared to ER, while for RFE-P and RFE-B, the training time moderately increased. For inference, the classinncremental setting represents the worst-case scenario for the RFE method, where features are propagated through all 10 tasks. Nonetheless, the inference time only moderately increased compared to other methods.
Table 6: Training and inference time for all 10 tasks on the S-Tinylmg dataset with CIL setting.
ER
TAMIL
CLSER
RFE
RFE-P
RFE-B
Training (hours)
2.06
2.51
2.59
1.94
3.17
3.32
Testing (second)
7.36
7.42
7.71
11.54
B.2 Space complexity
Combining the buffer size and the parameters in Table 2 in the main paper reflects the memory footprint of each method. During training. both the images and the params are loaded the same float32 format.
Consider I-nnnylmg, for buffer-only methods (AGEM, ER, ER-ACE, DER++), save 1000 64 64 RGB images, which is approximately equivalent to 12.28 M params. RFE (no data) adds slightly more at 14.36 M params for the previous backbone and retrospectors. However, RFE's accuracy is 12.49%, higher than buffer-only methods (DER++).
Similarly, RFE-P and RFE-B additionally utilize saved data, while ER-MKD, TAMiL, and CLS-ER ad- ditionally utilize EMA backbone and/or auxiliary modules. Nonetheless, RFF-P outperforms TAMiL, by 3.84% in average accuracy.
B.3 Ablation on alternative designs
Table 7: TIL Average Accuracy across all tasks after CL training for RFE variants on S-CIFAR100.
Type
Average Accuracy
Total Retrospectors' Parameters
RFE
79.54
1.42
RFE end-to-end
79.18 + 0.44
M.P-Rosiduul
78.53z 0.82
2.66
MLP-Projection
52.78 6.29
2.66
Variants of retrospector
A naive design is to map fi directly back to fi-1 by directly using an MLP without using hr However, due to catastrophic forgetting, there no straightforward method to recover information loss of task f. without an external source (h+).
A simple but inefficient alternative of the retrospector is to concatenate both fe(x) and (x(x,, which are then forwarded through simple MLP. We use two-layer MLP with output dimension of 512 for both layers and ReLU activation. The retrospector can then be used to learn the projection from task t back to (- - (MLP-Projection)
r (f((())))) (LP([)((s)) hx()))
(9)
18

---

Published in Transactions on Machine Learning Research (01/2026)
learn the residual of such projection (MLP-Residual): r:()(()), (LP([ H
+ f(())
(10)
Nonetheless, this design results in very high parameter usage while only delivering similar or worse perfor- mance than the gating design, as demonstrated in Table 7. The setting is similar to Table in the main paper, using the S-CIFAR100 dataset.
End-to-end training of RFE
While RFE is designed to be mainly post-training to reduce interference with main network training, RFE can also function using end-to-end training as demonstrated in Table 7. In this case, post-training for and h441 is no longer necessary. The main training loss LT in Algorithm 1 becomes:
CO O D Dirain US) LCE( (O U w wi o fi, Derain) + aCFE(O,.in Drraiu US, ft-1)
(11)
B.4 Feature estimation over a long sequence
Drifted Representation RMSE
Rectified Representation RMSE
Train 701
0.00
0.00
Train T02
1.00
1.040.00
0.480.00
Train T03
1.20.88
0.000.0000
0.510.570.
Train
1.04
0.000.00
0.54..62
Train T06
-1.241.28
1.28
0.000
6. 0.566.677
0.00
Train T07
11.25 1.29
1.30
1.201
000
0.00
0.57 0...0.99000 0. 0.75 0.70
0.68 0.00
Train TOB
-1.271.30 1.29 1.
28.0.0
0.00
0.58.0700
73 0.75
0.72 0.740.0....000 140.560.000
Train T09
-1.291.33 1.371.351400 1.33
1.10
0.500.700 0. 74
7. 0.7000...0
Train 1.333..40 1.32 1.421.99144... 1.44..22 1.31 160.000
0.580.700
0.76..744
Test 701
Text 102
Test 10
Test 06
Test 07
Text r08
Test 09
Test 01
Test 02
Text 703
Test 00
Test 705
Test 07
Tast 00
Tost 09
Test 100
Test
Test
Test
Figure 6: RMSE of the drifted representation and rectified representation against the original representation (darker color is better). Similar Fig. 3, the vertical axis and horizontal axis represent the task the model has been trained on and the task identity, respectively. The retrospector is demonstrated to be effective in mapping from the drifted representation back to the original representation.
We exlend the experiment Fig. 5 to S-CIFAR100 10 tasks to dernonstrate the effectiveness of the retro- spector over longer sequences. Similarly, we a (no regularization) clearly visualize the catastrophic forgetting and retrospector module performance. Each vertical column demonstrates the increasing forgetting of the model after training. Nonetheless, with the retrospector, can effectively rectify the representation to reduce forgetting and maintain stable performance.
B.5 Generalization to other architectures
We repeat subset of the experiment for task- indeeencremttall learning with S-CIFAR100, but replacing the backbone from Resnet18 to ViT-S/16 (Dosovitskiy et al., 2020) pre-trained on ImageNet-21k and fine- tuned on ImageNet-1k. The vision transformer backbone is followed by a linear layer with ReLU activation to ensure compatible dimensions with the ResNet18 backbone. Table demonstrates the result of RFE using vision transformer backbone finc-tune tasks of S-CIFARI00, which shares high similarity with results using a convolution neural network backbone. Each task is trained for 30 epochs with learning rate of 10-5 Other training details kept the same in Section A.3.
19

---

Published in Transactions on Machine Learning Research (01/2026)
Table 8: TIL Average Accuracy across all tasks after CL training using VIT-S/16 backbone.
Method
S-CIFAR100
Exemplars
TIL
params
accuracy
Oracle
96.93 0.04
21.91/21.91
Finetuning
78.88 7.84
AGEM
92. 0.16
ER
91. 0.02
21.91/21.91
DER+
91.42: 0.33
[B] 500
ER-ACE
93. 23 1.26
ER-MKD
43.83/21.91
92.07 0.88
TAMiL
44.16/21.91
93.87 0.23
CLS-ER
65.74/21.91
94.28
RFE-P
[P] 500
95.58+ 0.12
45.19/23.28
RFE-B
[ 500
95.59 0.18
RFE
45.19/23.28
95.54+ 0.01
B.6 Comparison with non-rehearsal methods
RFE can be compared with existing works using Kim et al. (2022) and Bhat et al. (2023).
In Kim et al. (2022), the comparable setting is T-10T (similar to S-Tinylmg). We consider the following methods: LwF (Li & Hoiem, 2018), HAT, (Serra et al., 2018), Sup (Wortsman et al., 2020), and HyperNet (von Oswald et al., 2020). It should be noted that we use (RFE) or 1000 (RFE-P, RFE-B) samples with standard Resnet18, while methods in Kim et al. (2022) use 0 or 2000 samples with a larger Resnet that has double the channels. Consider the task-incremental learning setting in Table of [4], the best baselines (both rehearsal and non-rehearsal) accuracy are 68.4, and the SOTA are HAT+CSI (72.4) and Sup+ (74.1)- RFF (69.66) without data outperforms all baselines. RFF-P (72.65) and RFF-B (71.92) are in the same range of SOTA.
In Bhat et al. (2023), we consider PNN (Rusu et al., 2016), CPG (Hung et al., 2019), and PackNet (Mallya & Lazebnik, 2017). In Figure of Bhat et al. (2023), TAMiL is demonstrated to outperform all baseline methods in iask-incremental final average accuracy. In our experiments, RFE, RFE-B, and RFE-P outperform TAMiL in both CIFAR100 and Tiny ImageNet datasets for the task-incremental setting.
C Versatility of RFE Framework
In RFE, the tasks arrive, conventional fine-tuning or training on the new task happens with minimal CL's intervention. RFE only augments adds to this process with a separate training of the retrospective feature estimation mechanism. The attractiveness of this framework is two-fold. First, RFE allows the best adaptation on the new task to possibly achieve maximum plasticity, while the backward rectification mechanism mitigates catastrophic forgetting, Second, unlike previous CL approaches that heavily modify the sequential training process, RFE minimally changes the fine-tuning process, allowing the users to flexibly incorporate this framework into their existing machine learning pipelines.
Relationship to Memory Linking. RFE's process of mapping newly learned knowledge representation re- sembles the popular human mnemonic memory-linkingg technique, which establishes associations of fragments of information ion to enhance memory retention or recall. As the model learns new lask, the rotrospector module establishes mnemonic link from the new representation of the sample from the past task to its past task's 8 correct representation.
20


Reviews:
[{"review_id": "O6JfpS9JNh", "reviewer": "Reviewer_2PNf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel Continual Learning (CL) framework called **Retrospective Feature Estimation (RFE)**. Instead of constraining the main model to prevent forgetting during training (which limits plasticity), RFE allows the model to freely learn new tasks and addresses forgetting via a post-hoc mechanism.\n\n**Key contributions include:**\n1.  **Methodology:** Introduction of \"Retrospector\" modules that learn to map/align the feature space of the current model back to the feature space of previous tasks. This effectively decouples the stability-plasticity dilemma: the backbone handles plasticity, while the retrospectors handle stability.\n2.  **Privacy/Data Efficiency:** The method can function in a data-free (zero-shot) manner without storing old exemplars, addressing privacy concerns, though it can also leverage buffers (RFE-P/B) for improved performance.\n3.  **Empirical Results:** The method demonstrates competitive performance on standard benchmarks (CIFAR-10, CIFAR-100, Tiny ImageNet) compared to established rehearsal-based methods, particularly in maintaining accuracy over longer task sequences.\n\n**Strengths:**\n* **Novelty:** The idea of \"fixing\" features via a chain of auxiliary modules is an interesting departure from standard weight regularization or replay methods.\n* **Visualization:** The PCA visualizations (Fig. 5) provide convincing evidence that the retrospector effectively realigns drifted features.\n* **Privacy:** Being able to operate without a replay buffer is a significant advantage for privacy-sensitive applications.\n\n**Weaknesses:**\n* **Inference Latency:** The sequential chaining of retrospector modules during inference introduces a linear increase in latency, which may be prohibitive for long task sequences.\n* **Scalability:** While modules are lightweight, the parameter count still grows linearly with the number of tasks.\n* **Class-Incremental Learning (CIL) Limitations:** The strategy for CIL (averaging predictions across all domain rectifications) seems computationally expensive and heuristic compared to the elegance of the Task-Incremental setting.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors provide extensive experiments on standard CL benchmarks (S-CIFAR10, S-CIFAR100, S-TinyImg) and compare against strong baselines like DER++ and CLS-ER. The experimental setup is described in detail in the appendix. The ablation studies (e.g., comparing different retrospector designs like MLP-Projection vs. MLP-Residual) and PCA visualizations support the claim that the retrospector modules are indeed correcting feature drift rather than just acting as a generic ensemble.", "audience_yn": "Yes", "audience_explanation": "The paper addresses the fundamental problem of catastrophic forgetting in neural networks. The proposed perspective of \"retrospective feature alignment\" is conceptually distinct from the dominant replay/regularization paradigms. Researchers interested in Continual Learning, privacy-preserving ML, and dynamic network architectures would find this work relevant.", "requested_changes": "I lean towards accepting this paper as it presents a novel and effective idea. However, there are several concerns regarding the practical deployment and analysis that the authors should address to strengthen the submission.\n\n**Critical Adjustments:**\n\n1.  **Inference Latency Analysis:**\n    The paper briefly mentions overhead in Section 5 (Limitations), but it needs more quantitative data. In the Task-Incremental setting, inference for Task 1 after learning Task $N$ requires passing through $N-1$ retrospector modules.\n    * **Request:** Please provide a plot or table showing the *inference time per sample* as a function of the number of tasks (e.g., from Task 1 to Task 20 on Tiny ImageNet). This will help readers understand the trade-off between accuracy and latency.\n\n2.  **Clarification on CIL Strategy:**\n    The approach for Class-Incremental Learning (CIL) involves \"averaging predictions based on the constructed representations\" across all domains. This implies that for every test sample, the model must perform inference $N$ times (one for each historical domain).\n    * **Request:** Please explicitly discuss the computational cost of this strategy. Is it $N$ times slower than a standard classifier? The authors should acknowledge this computational bottleneck more clearly in the main text, as it significantly impacts the method's scalability compared to standard replay methods (e.g., ER/DER++) which use a fixed-size backbone.\n\n3.  **Memory Overhead Clarification:**\n    While \"lightweight,\" the retrospector modules add parameters.\n    * **Request:** In Table 2, please clarify if the \"params\" column includes the stored parameters of *all* retrospector modules accumulated up to that point. It is crucial to ensure the comparison is fair against dynamic architecture methods or fixed-capacity methods.\n\n4.  **Scalability to Large Foundation Models (LLMs):** \nThe current experiments are limited to relatively small backbones (ResNet-18 and ViT-S/16). Given the increasing importance of Continual Learning in Large Language Models (LLMs), it would be valuable if the authors could discuss the scalability of RFE to billion-parameter models. Specifically: Training Memory Cost: Algorithm 1 implies that training the retrospector $r_t$ requires access to the previous frozen feature extractor $f_{t-1}$ to compute the loss. For LLMs (e.g., Llama-3-8B), keeping a copy of the previous model in memory alongside the current model for distillation/alignment is computationally expensive and VRAM-intensive.High-Dimensional Features: LLMs operate on much larger hidden state dimensions (e.g., 4096 or larger) compared to ResNet-18 (512). Would the \"lightweight\" retrospector need to be significantly larger to effectively capture the transformation in such high-dimensional spaces? Inference Latency in Autoregressive Generation: For LLMs, inference involves token-by-token generation. If RFE requires passing features through a chain of retrospectors for every token generated, the latency overhead might be prohibitive for real-time applications. A brief discussion on these potential bottlenecks would help position the paper better within the context of modern foundation models.\n\n**Strengthening Suggestions (Non-critical):**\n\n1.  **Comparison with \"Expansion\" Methods:** Since RFE adds parameters over time, it shares similarities with dynamic architecture/expansion methods (e.g., PNN, DYTOX). Including a brief discussion or comparison with a modern expansion-based method would better position the work.\n2.  **Task ID Prediction:** The authors mention that OOD detection could be used for CIL. While not required, implementing a simple task-id predictor (even a baseline one) would make the CIL results much more convincing than the current \"average all\" heuristic.", "broader_impact_concerns": "### **Broader Impact Concerns**\n\nNo concerns.\n\nThe paper discusses a fundamental learning algorithm. The authors also highlight the privacy benefits of their method (reducing the need to store raw user data), which is a positive ethical implication.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mXBVuhkFsS", "reviewer": "Reviewer_68rz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary Of Contributions:\nThis paper introduces Retrospective Feature Estimation (RFE), a new continual learning paradigm designed to mitigate catastrophic forgetting by rectifying the drift in feature representation after training on a new task and/or class. Instead of regularizing the model or extending the network, this method tries to rectifies the feature representation back into the feature space previously learned for that specific task.\n\nStrengths:\n- The idea of rectifying latent representations retrospectively is interesting and quite innovative. Rather than preventing drift, RFE adds lightweight modules to reverse drift, providing a clear conceptual distinction from typical rehearsal, regularization, or parameter-isolation approaches.\n- RFE leaves standard training nearly untouched, avoiding complicating the primary learning objective of the network.\n- The authors benchmark RFE across multiple datasets, task lengths, and CL scenarios (new tasks and classes).\n- The PCA visualization clearly shows how retrospector modules recover past representations.\n\nWeaknesses:\n- Some aspects of the retrospector architecture are not extensively discussed. The authors might provide a more extensive discussion and/or ablation study on the several modules (auxiliary extractor, linear mappings, soft gatings) and their interplay.\n- Even if each retrospector module is lightweight, inference requires chaining potentially many modules. Although the authors acknowledge this limitation, the paper lacks quantitative analysis and discussion of its practical implications for real-world deployment.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors show that their approach matches or improves over state-of-the-art methods across several datasets, reduces forgetting in long sequences, and accurately reconstructs past representations. The empirical evidence is consistent with the claims.", "audience_yn": "Yes", "audience_explanation": "The paper addresses one of the most central and longstanding challenges in continual learning—catastrophic forgetting, with a novel approach that does not rely on buffering, regularization, or dynamic architectures.", "requested_changes": "- Provide a more intuitive explanation of the gating + auxiliary extractor mechanism. Specify the dimensionality choices and justify them.\n- The paper acknowledges inference-time cost but lacks detailed measurements. A table summarizing these aspects would help practitioners assess feasibility.\n- Add a short subsection contrasting RFE with progressive networks and other expansion-based approaches, to better emphasize the conceptual novelty.\n- While the technical novelty is appreciated, the paper would benefit from a clearer discussion of practical advantages. Why is RFE preferable to rehearsal-based methods or network expansion approaches in applied settings? Which applications would most benefit from this paradigm?", "broader_impact_concerns": "No concern.", "additional_comments": "- Discussing failure cases and include examples where RFE does not fully recover past representations or where chaining becomes unstable, to clarify the practical limits of the method.\n- Do you plan to share code and experiments?", "extra_scores": {}, "extra_text": {}}, {"review_id": "BPdLhco2IG", "reviewer": "Reviewer_1diB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new approach to continual learning (CL) using deep neural networks (DNNs) termed Retrospective Feature Estimation (RFE). The main idea is to align the features learned by the DNN on the current task with features learned for previous tasks.\n\nIn more detail, the classifier for each task $t$ is decomposed into two parts: a feature extractor $f_t$ (mapping each data point to a set of learned features) and a classifier $w_t$ (mapping features to the logits for classification). The training enforces that features learned by $f_t$ should \"encompass\" the previous feature $f_{t-1}$. This is done by introducing a \"retrospector module\" $r_t$ that aims to recover $f_{t-1}(x)$ from $(f_t(x), x)$ for each training example $x$.\n\nThe method is evaluated on three continual learning datasets: the \"sequential\" versions of CIFAR10, CIFAR100, and Tiny ImageNet. Three variants of RFE are compared to several baselines in both task-incremental and class-incremental setups. The new methods give improved or comparable (within 5% of error) performance (Figures 1 and 2). The authors conducted additional experiments that visualize the degradation of performance over a long sequence of tasks (Figures 3 and 4) and the learned features (via PCA, in Figure 5).", "claims_and_evidence_yn": "No", "claims_explanation": "Some aspects of the proposed method do not seem to be the most \"natural\" design choice, and they do not appear to be sufficiently motivated or justified in the manuscript. In addition, the PCA-based visualization in the experiments was a bit vague and unconvincing.\n\n**End-to-end training?** Having defined the feature estimation loss (Equation (2)) and the architecture of the retrospector module (Section 3.2.2), arguably the most natural approach would be minimize the total loss (a weighted sum of the cross entropy and feature estimation losses) by training $f_t$, $w_t$, and $r_t$ (which includes $h_t$) simultaneously. Instead, the authors proposed a three-step training procedure (Section 3.3). Thus, I believe that some motivation / justification is in order, and it would be nice to have an ablation study that investigates the end-to-end alternative.\n\n**The PCA-based visualization.** I found it a bit hard to understand Figure 5 and the discussion in Section 4.3. In the figures, the red points (the drifted representations) are much more concentrated around the origin compared to the blue ones (the original representations), while the green points (the rectified representations) are a lot closer to the blue ones. While this does suggest that the retrospector succeeds in matching the representations, I wasn't very convinced of the claim that the red-vs-blue comparison explains catastrophic forgetting---we could've obtained the red scatter from the blue one simply by scaling all features down by some factor, but this would keep all the information in the feature. Instead, using other metrics between feature spaces (e.g., the one of Wang et al. (NeurIPS'18)) would give a more convincing comparison.\n\n**Effectiveness of feature estimation.** It would be enlightening if you evaluate the accuracy of the feature estimation module (the RMSE values reported in Figure 5) over a long task sequence. Specifically, if we store all the previous feature extrators $f_1, f_2, \\ldots, f_{N-1}$ (for evaluation purpose only), for each $t \\in [N-1]$, we can evaluate the distance between the actual representation $f_t(x)$ and the estimate $\\hat f_t(x)$ (from Figure 1). It would be interesting to see how this error grows as we decrease $t$, which could say more about the effectiveness of the retrospector modules. (This could also be done by fixing $t = 1$ and increasing $N$, or having a 2D plot similar to Figure 3.)\n\n**Reference**\n\nLiwei Wang, Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun He, John Hopcroft. Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation. NeurIPS 2018.", "audience_yn": "Yes", "audience_explanation": "This work proposes a new approach to continual learning, which, to the best of my knowledge, is quite different from previous ones. The method is very intuitive, nicely motivated, and shown to be effective. If the authors could give a convincing response to the questions above, I believe that this work would be interesting to the continual learning community.", "requested_changes": "Summarizing the above, the submission would be stronger if it included:\n- Justification and/or motivation of the training procedure.\n- An ablation study of an \"end-to-end\" variant of the training procedure.\n- A more convincing interpretation on why Figure 5 \"explains\" catastrophic forgetting (or retraction of this claim).\n- Optionally, a plot of the feature estimation error over a long task sequence.\n\nOther comments/writing suggestions:\n- Page 2: \"the past task's representation\" $\\to$ \"the past tasks' representation\", if multiple past tasks are being referred to here.\n- Page 3: There is a typo in Figure 1: \"Available at infernece\" $\\to$ \"Available at inference\". Also, the legend for \"Available at inference\" is not very visible; consider making the outlines of the corresponding modules a bit thicker.\n- Page 4: It might be clearer if you specify the \"input\" of $r_t$ when the notation is first introduced. Currently, it is not clear that $r_t$ takes $x$ (in addition to $f_t(x)$) as an input until Section 3.1.1.\n- Page 6: The notations $\\hat z$ and $\\mathcal{L}_{\\mathrm{align}}$ do not seem to be defined in the main text.\n- Page 6: \"will be linear projected down to\" $\\to$ \"will be linearly projected down to\"?\n- Page 6: \"a smaller dimension space\" $\\to$ \"a smaller-dimensional space\" or \"a space of a lower dimension\"?\n- Page 6: Operations $\\circ$ and $\\odot$ should be defined when they first appear.\n- Page 7: The use of set operations (\"$\\cup$\" and \"$\\setminus$\") on parameters might not be conventional; adding footnotes or a sentence at the first use could make things clearer.\n- Page 7: The subscript \"T\" in the loss $\\mathcal{L}_T$ was a bit unclear---does \"T\" stand for \"Total\"?\n- Page 7: In Algorithm 1, the three for-loops over the epochs seem a bit redundant: the loop variables ($i$, $j$, and $k$) are not mentioned inside the loops. Removing those does not seem to affect readability, and you could save six lines of space.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)