Autoencoder-Based Hybrid Replay for Class-Incremental Learning
Authors: Milad Khademi Nori
IL MIN KIM
Guanghui Wang
Venue: ICML 2025 poster (2025)
Keywords: Autoencoders
Catastrophic Forgetting
Generative Modelling
Hybrid Replay
.
Primary area: general_machine_learning
TL;DR: A new scalable hybrid autoencoder architecture for class-incremental learning.

Abstract: In class-incremental learning (CIL), effective incremental learning strategies are essential to mitigate task confusion and catastrophic forgetting, especially as the number of tasks $t$ increases. Current exemplar replay strategies impose $\mathcal{O}(t)$ memory/compute complexities. We propose an autoencoder-based hybrid replay (AHR) strategy that leverages our new hybrid autoencoder (HAE) to function as a compressor to alleviate the requirement for large memory, achieving $\mathcal{O}(0.1 t)$ at the worst case with the computing complexity of $\mathcal{O}(t)$ while accomplishing state-of-the-art performance. The decoder later recovers the exemplar data stored in the latent space, rather than in raw format. Additionally, HAE is designed for both discriminative and generative modeling, enabling classification and replay capabilities, respectively. HAE adopts the charged particle system energy minimization equations and repulsive force algorithm for the incremental embedding and distribution of new class centroids in its latent space. Our results demonstrate that AHR consistently outperforms recent baselines across multiple benchmarks while operating with the same memory/compute budgets. The source code is included in the supplementary material and will be open-sourced upon publication.

Autoencoder-Basee Hybrid Replay for Class-Incremental Learning
Milad Khademi Nori
II-Min Kim
Guanghui Wang'
Abstract
Table I: Hybrid replay strategy versus baseline strategies.
In class-incremental learning (CIL., effective incremental learning strategies are essential to mitigate task confusion and catastrophic forgetting, especially as the number of tasks t increases. Current exemplar replay strategies impose (t) memory/compute complexities. We propose an autoencoder- based hybrid replay (AHR) strategy that leverages our new hybrid autoencoder (HAE) to function as a compressor to alleviate the requirement for large memory, achieving (0.1t) at the worst case with the com- puting complexity of (t)) while accomplishing state-of-the-art performance. The decoder later recovers the exemplar data stored in the latent space, rather than in raw format. Additionally, HAE is designed for both discriminative and generative modeling, enabling classification and replay capabilities, respectively. HAE adopts the charged particle system energy minimization equations and repulsive force algorithm for the incremental embedding and distribution of new class centroids in its latent space. Our results demonstrate that AHR consistently outperforms recent baselines across multiple benchmarks while operating with the same mem- ory/compute budgets. Implementation is available at aathhbo.wwmmiilcccccmmmmaaaaoooe hybrid-replay-cil
CIL Strategies
Memory
Compute
Performance
Generative Replay
O(cte)
Ott)
non-SOTA
Generative Classifier
(()
(ate)
non-SOTA
Exemplar Replay
O())
O(t)
SOTA
Hybrid Replay (ours)
(
O())
SOTA
ios are often considered: task-incremental learning (TIL) and class-incremental learning (CIL). CIL at the test phase requires jointly discriminating among all classes seen in all previous tasks without knowing task-IDs. Given task-IDs to the model during the test phase, CIL reduces to TIL van de Ven & Tolias 2019). The issue with CIL is that it not only suffers from catastrophic forgetting (CF) but also task con- fusion (TC), which happens when the model struggles to distinguish among different tasks observed far while still being able identify classes within a given task (Rebuffi
et al. 2017 Belouadah et al. 2021 Masana et al. 2020 Cormerais et al. 2021).
Various strategies have been proposed for incremental learn- ing such as regularization (Kirkpatrick et al. 2017: Zenke et al.
2017
Hoiem 2017b). bias-correction (Zeo
et al. 2021
replay (Shin al. 2017 Ven et al.,
2020
and generative classifier (Ven et al. 2021 Pang et al.
2005
Zajac et al. (023). However, in the context of CIL, only the latter two strategies have been proven effective in over- coming TC: replay and generative classifier (Masana et al.) 2020 Cormerais et al. 2021 Ven et al. 2021] Nevertheless, current realizations of the generative classifier strategy (Ven) ct al. 2021 Pang et al. 2005 Zajac al.) 2023) demand
1. Introduction
expanding architecture, leading to linear increase of memory (t) with respect to the number of learned tasks t. Furthermore, such expanding architectures fail to consoli- date features of different tasks within a single model.
Incremental learning addresses the challenge of learning from an upcoming stream of data with changing distribu- tion (Parisi et al. 2019 De Lange et al. 2021: Hadsell et al. 2020). To study incremental learning. two common scenar-
To develop a scalable incremental learning algorithm suit- able for CIL. we capitalize on replay strategies (Shin et al., 2017 Ven al. 2020). However, the current exemplar re- play strategies (Rebuffi et al. 2017) not scalable due to their reliance on large memory sizes. Specifically, the mem- ory requirements for exemplar replay increase linearly with the number of tasks, resulting in O(t) memory complexity
'Department of Computer Science, Toronto Metropolitan University, Toronto, Ontario, Canada "Electrical and Computer Engineering, Queen's University, Kingston, Ontario, Canada. Correspondence to: Milad Khademi Nori <miladkhademi- nori @ @mmaiimmmcoo......
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
(Hou et al. 2019 Wu et al. 2019 Hayes et al. 2020a).
To address this issue, generative replay strategies (Shin

---

Autoencoder-Banee Based Hybrid Replay for Clas--Incremental Learning
al. 2017
Ven et al. 2020), instead of storing the exact
Training Phase
data samples of the previous tasks, train generative model to generate the pseudo-data pertaining to the previous tasks for replay to the discriminative model, achieving (cte) memory complexity. Since the quality of the generated pseudo-data not satisfactory, these strategies also undergo significant CF unless a very cumbersome generative model trained which is inefficient. Also, generative replay diverts the problem of training discriminative model incrementally to training generative model incrementally which can be equally challenging, if not more Venn et al., 2020).
Φi w:)
ab (v:)
ae
(a)
Test Phase
The complexity of (t) for either memory or compute is in- dispensable. Because in principle every time an IL strategy learns task I, there have to be mechanisms at play watch for t 1 conditions imposed by prior tasks on the weights of the neural networks lest those knowledge
db( un:))
sng szin
(b)
overwritten.
Dataset
That requires either (l) memory or (t) compute com- plexity. In Table I either case can be seen: while generative replay achieves (cte) for memory, it nevertheless needs (t) for computation. Conversely, the generative classifier exactly the opposite: it achieves (cte) for computation but requires (t)) for memory accommodate the new tasks. Neither of these strategies is optimal. The performant strategy is exemplar replay which requires (tt) for both memory and compute.
Compression
T
Φi w:)
*b(1),)
T
And Decompression
Figure 1: (a) Usage of RFA for the latent space. (b) Adop- tion of Euclidean distance during test. (c) HAE for compres- sion and decompression of the dataset for replay.
We demonstrate that it is feasible to combine the strengths of both the exemplar (Rebuffi et al. 2017) and generative replay (Shin et al., 2017 Ven et al. 2020) in hybrid replay strategy and consistently achieve ofate-o-thheaa (SOTA) performance while significantly redueing the memory foot- print of the exemplar replay strategy from (tt) to O at the worst case, using new hybrid autoencoder based on hybrid replay. Our main contributions are as follows:
DD Ti
AHR
T2
Ti
DD
Figure 2: Task-based and task-free.
We propose novel autoencoder named hybrid autoen- coder (HAF): the term hybrid autoencoder' indicates that HAE is capable of both discriminative and gen- erative modeling (Goodfellow et al.] 2016), for classi- fication and replay, respectively. Furthermore, HAE employs the charged particle system energy minimiza- tion (CPSEM) equations and repulsive force algorithm (RFA) (Nazmitdinov et al.) 2017) for the incremental embedding and distribution of new classes in its latent space. HAE uses RFA (Nazmitdinov et al. 2017) in its latent space to repel samples of different classes away from each other such that in the test phase the Euclidean distance can be used to measure the distance of sample from centroids of different classes in the latent space to know to which class the test sample belongs to (see Figs. aaa and [b).
to the fact that AHR utilizes combination of exem- plar replay and generative replay. Specifically, AHR does not store the exemplars in the input space like exemplar replay which would require a large memory O(); it rather stores the data samples in the latent space after they are encoded (0.1f)). Hence, AHR has characteristics that leverage the advantages of both exemplar and generative replay. AHR can decode data samples when they are needed for replay with neg- ligible loss of fidelity because the decoder has been designed to memorize the training data as opposed to being designed generalize and produce novel data samples. Fig. (c) shows how the data generation for replay is performed. As result, AHR does not suffer from hazy pseudo-data during replay like other gen- erative replay strategies but has access to the original data. Table contrasts the memory/compute complex- ities of ours and three baseline strategies. A detailed
We propose a new strategy called autoencoder- based hybrid replay (AHR): the term hybrid replay' refers
2

---

Autoencoder- Based Hybrid Replay for Clas--Incrementa Learning
description of how we derived Table along with rel- evant discussions, can be found in Appendix A in the supplementary material.
gether in the latent space L2)
Læ z) = Ln(2) a:)
+ (z.
J K2
We provide comprehensive experiments to demonstrate the strong performance of AHR: we conduct our ex- periments across five benchmarks and ten baselines to showcase the effectiveness of AHR utilizing HAE and RFA while operating with the same memory and compute budgets.
ΣΣΣ
|جا
11 + A1122
-p:ll
(1)
where || 11 denotes the L norm, p is the ith task and jth class centroid embedding (CCE), and À is a hyperparameter. While the given loss function helps in clustering samples of the same class together, another crucial aspect is separating different classes. This separation, and in general, the incre- mental placement of CCEs in the latent space is addressed through CPSEM equations and RFA, where p 's are akin to charged particles.
We present our new strategy AHR the following section. In Section 3, we contextualize AHR in the incremental learn- ing literature. Section 4 details evaluation methodology, including the baselines and benchmarks used, as well as the results of our experiments. Finally, Section 5 concludes this paper and provides future works for CIL.
To model the energy dynamics within our system akin to charged particles, AHR employs the formulation based on Coulomb interaction energy. Consider fixed set of I x Σ, J: particles representing CCEs, with charges of at positions pz. The potential energy of this system is given by u =
2. Autoencoder-Based Hybrid Replay (AHR)
In this section, we present our strategy AHR in a task- based CIL system model for simplicity and comparability with most of the works in the literature (Parisi et al.) 2019
I.J.
De Lange et al.
2021 van de Ven & Tolias 2019: Masana et al. 2020 Zajac et al. 2023). Nevertheless, our approach
(2)
2
Pd lIIt p!l
AHR is not restricted to the task-based CIL setting and can operate within the task-free setting as well (see Fig. 2 (Ven) al. 202T Aljundi et al.) 2019b]. According to the task- based CIL system model, AHR visits series of distinct non-repeating tasks Ti, T2, ................................ Tr. During each task Ti,
Each particle also possesses kinetic energy K// = I m || [1?, where mi and uy represent the mass and speed of particle ij (CCE of task and class j), respectively. In our optimization framework, AHR aims to minimize the total energy = + K., where K. = Σx fmm |e]] This can be achieved through the calculus of variations, leveraging the Lagrangian:
2
dataset D; := (< =/(") ) J.k=i is presented where i, J. and l index task, class, and sample, bounded by I, Jr. and K' In the test phase, AHR must decide to which class a given input a belongs among all possible classes U'=1 Jr- Task-ID i is not given during the test phase of CIL, indicat- ing that AHR has to distinguish not only between classes that have been seen together in a given task but also between different tasks visited at different times.
J.
C=
Σ k=i
1.4.
1
ml
of
I. 11
Σ
(
2
I - p!ll
k=i.j
(3)
The equations of motion for the particles, derived via the Euler- Euler-Lagrange equation
HAE, AHR utilizes HAE consisting of an encoder and a decoder that can be formulated as follows: the encoder func-
tion ((x] R b TM maps the input data = E R to the low-dimensional latent representation 2/th IR. The decoder function (=x R + R" reconstructs the input data from the latent representation. Note that AHR intentionally refuses to use the most popular autoencoders, Variational Autoencoders (VAE) (Kingma & Welling 2013). since the goal here not generalization or generating new images. Indeed, the goal of AHR is precisely the oppo- site: to have the decoder deterministically memorize pairs of 3. ) Compared to traditional autoencoders, HAE not only minimizes the reconstruction error between the input and the reconstructed data L (x, 2) but also ensures that samples from the same class are clustered closely to-
d ac dt Ou!
Oc
(4)
Op!
enable us to determine the optimal positions of the particles, effectively minimizing the total energy of our system. The above equation helps HAE to efficiently distribute CCEs within the latent space.
AHR. Algorithm 1 outlines the steps of the AHR strategy in the context of task-based CIL.: Whenever a new task T3 arrives AHR invokes three main rou- tines of CCE PLACEMENT,,,, ORY URY_POPULATION (i) in CCE PLACEMENT, AHR de- termines the positions of the new CCEs for Dg and returns
and MEM-
3

---

Autoencoder-Bass Hybrid Replay for Clas--Incremental Learning
Algorithm 1 Autoencoder- Hybrid Replay
Algorithm 3 HAE Training
1: Input: Tasks (2,,2,.... Tr} with {Di, D2....,Drhh
1: Input: Φ(urg-1), (0(0e11), De. (Pi) (Mi
HAE model (Ø(wo), (vv))), memory Mu
2: Output: ((wi), 1b(n))
2: Output: (d(w)), (. {MM)) CCEs (P:)
3: D - Dr U 11
3: for tasks 1...., do
4: Copy ((w,-1), () as ((ws), (b(x))
4: P.
CCE__CCCCEEEE PLACEMEETTIO((
(1V_1). D.,
5: for epochs e I............................. do
(Pr) via RFA in Algorithm 2 solving Eq. 4
6: for minibatch b 1...., B do
5: ((w:), (vv)) HAE. TRAIN((((,;-), (0; 1), Dj.
7:
Minimize the HAE losses in Eq | and Distillation losses:
(M MM 1 (P:))-)) via 3
8:
L(D, (be (i ((w,, D)), oww, D))
6: M; = MEMORY POPULATION( (w.), (b;)), (ww:-1),
9:
-| D) ( D)Il
((;,-1), D., (Ma). 11 (Pr ()) via Algorithm 4
10:
+ ( Φ ( D)) (be dwwr. D)II
7: Delete Φ(w,, (b)
11: Optimize with an optimizer to obtain o (ww)) (b(1))
8: end for
12: end for
13:
end for
Algorithm 2 CCE Placement
I: Input: (Pi,P2... P2..... , Pe-1), Repulsive Constant Ç, Particle
ples. In practice, for each iteration of the SGD, besides 1/L fraction of the minibatch size that is provided by the new task Tr. - (e- fraction of minibatch size selected and instantly decoded from memory ( k= in a statistically representative fashion with respect the previous tasks/classes for training. These vectors require significantly less memory O x 1) than raw data and can be decoded at any time by
Mass m, Time Step At, Simulation Duration T
2: Output: Pe
3: Initialize positions p/=0 = (b(=z-1, D:) finitialize Pe)
4: for time step =
r do
5: for classes j = 1, 1.................... Jr do
6: for tasks i 1
8 do
7: for classes 1" L....,J. J/ do
8:
if j + L then
(-1).
9:
Compute displacement vector d/d pp-
P
(iii) AHR populates its memory Me. via MEM- ORY POPULATION in Algorithm 4 based on Herding as in (Rebuffi et al. 2017). Currently, there are two competi- tive approaches for sampling of exemplars in the literature Masanaa et al.) 2020), Herding and Naive Random, where the Herding approach on average demonstrates a slight im- provement over Naive Random sampling when applied to longer sequences of tasks (Masana et al. (022). AHR, in Algorithm 4, computes the losses L ( ure D), (P:) and ranks them ascendingly via RANK and then selects number of classes for both the new task é and previous ones.
10:
Compute repulsive force f/
12
11:
Accumulate repulsive force: F?- end if
F +
12:
13:
end for
14:
end for
Update CCE velocity: v/ = v/ F At
15:
16:
Update CCE position: p = pp- + 0--- At
17:
end for
18:
end for
Pa = / based on RFA outlined in Algorithm solv- ing Eq. [ @wheee&& É denotes the latest arrived task (throughout this paper, we use the notation l referring to the latest task index whereas i might refer to any task). Note the distinc- tion that, unlike the centriods in iCaRL (Rebulfi et al. 20177). CCES in AHR do not change over the course of learning.
Finally, at the test stage, it is determined to which task-class ij given data sample x belongs via computing the Eu- clidean distance as follows: argmin,,, [|o(w") x)) --''l 1| It is clear that in CIL, not only must the classes within a given task be discriminated, but the different tasks themselves must also be distinguished, which is why TC takes place on top of CF.
(ii) After the placement of the CCES Pj = (r1)/-1 HAE TRAIN is invoked by AHR (outlined in Algorithm 3) where the model (d( (bv;--)} is copied as (Φ(we). (bvE)) While { (b((-1)} is kept frozen, (d(w))) (b+))) is trained on the combined train- ing set featuring the new tasks and the decoded exemplars D + D, U (b (M for E number of epochs with the loss function in Eq. [and a distillation loss (serving as data regularization strategy) to obtain (Φ(w)), vb( (bz)),
3. Literature Review and AHR
Exemplar replay. The most popular exemplar replay strat- egy, iCaRL (Rebuffi et al. 2017). fuses exemplar replay and LwF (Li & Hoiem| 2017a). To mitigate the task task-recency bias, iCaRL separates the classifier from the representation learning (implicit ctarecoorec At inference, iCaRL classifies via the closest centroid. EEIL (Castro et al. 2018) introduces balanced training, where at the end of cach train- ing session an equal number of exemplars from all classes is used for a limited number of iterations. Similar to iCaRL, EEIL incorporates data regularization (distillation loss) into
Jt- 5t-1,Kk-1 stores the ex- where the memory M = (e fee-11
emplars of task 8- 1 and j.k contains exemplar (E - 1)jk coupled with its label. In AHR, the memory M stores embedded vectors in the latent space, not raw data sam-

---

Autoencoder- Hybrid Replay for Class-Incrementa Learning
Algorithm 4 Memory Population
replay in that it does not rely on synthetic data or pseudo- data; instead, it relies on the memorization of the original data. By encoding and decoding the exemplars into and out of the latent space, AHR mitigates the memory complexity of current exemplar replay strategies significantly and can be readily applied to the exemplar replay strategies. In the literature, hybrid replay has been studied in several works Zhou et al. 2022): (Tong et al. (022) utilizes closed-loop sreoding-decoding framework that stores only the means and covariances of features rather than the individual fea- tures themselves. The authors organize the latent space using Linear Discriminative Representation, which parti- tions the latent space into a series of linear subspaces, each corresponding to a distinct class of objects. (Hayes et al.) 2020b; Wang et al. 202T Nori et al. 2025) do not store raw exemplars but instead using compressed exemplars derived from mid-level CNN features (a strategy that is increasingly recognized as a promising direction in incremental learn- ing research). In (Hayes et al. 2020b; Wang et al. 202T) classification occurs after the decoding process, employing a cross-cntropp loss function. In contrast, our method per- forms classification directly within the latent space of the encoder, similar (Tong et al.] 2022). (Pellegrini al.) 2020 stores 'activations volumes' of intermediate layers rather than raw data.
1: Input: ((we), 15(ve), ((we-1), (w(-1)), De, (P.M {M.
2: Output: {M.
3: E + MMM/ x Ej=1 J3) (memory size divided by the number
of classes far)
4: D+ x De U (i (=l)
5: Calculate the losses (1 D), (n
6: for tasks i 1...., l do
7: for classes j = 1.................... Jr do
8: for samples k= = E.....,d. do
9:
Store RANK (o(w,, D), (P, (P.)( as ef*
10: end for
11: end for
12:
end for
exemplar replay. As discussed in the previous section, our strategy, AHR, also separates the representation learning and classification. Furthermore, AHR employs balanced training (from EEIL) and distillation loss via LwF (similar to iCaRL).
MIR (Aljundi et al. 2019a) trains on the exemplar memory by selecting exemplars that have a larger loss increase after each training step. GD (Lee et al., 2019) utilizes external data to distill knowledge from previous tasks. BiC (Wu et al. 2019) learns to rectify the bias in predictions by adding an additional layer while IL2M (Belouadah & Popescul 2019) introduces saved certainty statisties for predictions of classes from previous tasks (explicit bias-correction.. LUCIR (Hou
4. Experimental Results
Baselines. We consider five categories of baselines: vanilla strategies, generative classifier, generative replay, exemplar replay, and hybrid replay. Vanilla strategies include the naive strategies that serve the lower upper bound: Fine-Tuning (FT) does simple fine-tuning whenever a new task arrives, FT-E incorporates exemplar replay into fine- tuning, and Joint trains on all tasks seen so far (Masanal et al. allal122.00 2020). For generative classifiers, we include SLDA Hayes & Kanan 2020p, Gen-C (Ven et al., 202T), and PEC (Zajac al. 2023). For generative replay, DGR (Shin) et al. 2017). MeRGAN (Wu et al., 2018). and BI-R-SI Ven et al. 2020) are considered. For exemplar replay, the most strong baseline strategy, we include GD (Lee et al. 2019), GDumb (Prabhu et al., 2020). iCaRL (Rebuth et al.
eet al., 2019) replaces the standard softmax layer with a
cosine normalization layer. GDumb (Prabhu et al. 2020)
ensures balanced training set. ER (Chaudhry et al. analyzes the effectiveness of episodic memory.
2019
Generative replay reseuso-re This strategy gen- erates synthetic examples of previous tasks via a genera- tive model trained on previous tasks. DGR (Shin et al. 2017) generates synthetic samples via an unconditional GAN where an auxiliary classifier classifies synthetic sam- ples. MeRGAN (Wu al., 2018) improves DGR via a label- conditional GAN and replay alignment. DGM (Ostapenko
et al. 2019) combines the advantages of conditional GANs and synaptic plasticity using neural masking leveraging dy- namic network expansion mechanism to increase model capacity. Lifelong GAN (Zhai et al. 2019) extends im- age generation without CF from label-conditional to image- conditional GANs. Other strategies perform feature replay (Ven et al. 2020: Xiang al., 2019 Kemker & Kanan 2017) which needs fixed backbone network to provide good representations.
2017 EEIL (Castro et al. 2018). BiC (Wu et al., 2019)
LUCIR (Hou al., 2019), and 11.2M (Belouadah & Popescu
2019 Finally, for hybrid replay, we include i-CTRL (Tong
ct al. 2022). REMIND (Hayes al. 2020b), and REMIND+ Wang et al.) 2021). Note that since regularization and bias-correction strategies have been often applied to the aforementioned strategies supplementarily (and they are not performant for CIL on their own), we do not provide results for them independently.
Hybrid replay (AHR). Technically, our AHR strategy lies somewhere between exemplar replay and generative replay: AHR differs from exemplar replay in that it stores the sam- ples in the latent space. AHR also differs from generative
Benchmarks. The series of tasks for CIL constructed according to MMasana et al. 2020: Ven et al. 2021 Za
5

---

antoencoder-Basee Hybrid Replay for Class-Incremental Learning
The Impact Mermory Size for CIFAR-10
The Impact of Memory Size for CIFAR-100
The Impast of Memory Size for minilmageNet
30-
70-
60-
60-
70-
30-
30
40-
40
50-
UCC
30
FTF
112M
30-
AIR
20-
10
GDumb
Kart
Aith-up
AiB-oo
20-
32
255
32
64
10
Exemplers
Ecemp
Required Resources Achieve the Accuracy of 75% for CIFAR-10
Required Resources to Achieve the Accuracy 50% for CIFAR-100
Required Resources Achieve
the Accuracy of 45% for minimagellett
55.0
85.0
Compute Requueement (epocts)
52.5
825-
50.0
80.0
47.5
30-
45.0
77.5
42.5
40.0
35
150
(Number
Namory Excmplons)
Final Accuracy for Various Amounnt of Allocated Compute Time for CIFAR-10 (16 Esemplors(Cless)
Final Accuracy for Various Amount of Allocated Compute Time for CIFAR-100 (16 ExemplorsiClass)
Final for Various Amount Allocated Compute Time for minilmageftet 116 ExemplersCeess
50
50-
40
30
40
20-
30-
30
20-
10-
420
600
Vhll-Clock Time (mlnc)
Wall-Cloce Time (mins)
Wall-Clock Time (mins)
Figure 3: The impact of the memory size (first row). The required resources achieve a target performance (second row). The achieved performance for given compute time (third row).
jąc et al. 2023), where the popular image classification datasets are split up such that each task presents data per-
rehearsal. All the strategies always follow the fixed ex- emplar memory, not growing exemplar memory, implying that the number of exemplars per class decreases over time to keep the overall memory size constant (Masana et al., 2020 Rebufh et al. 2017). The learning rates, batch sizes, and strategy-dependes hyperparameters are detailed in Ap- pendix B in the supplementary material.
taining to a subset of classes, in a non-overlapping manner. For naming benchmarks, we follow (Masana et al. 2020). where dataset is divided into T tasks with C classes for each task. Hence, benchmark is named as DTTCC).
Accordingly, have (5/ST(5/22 (LeCun et al. 2010
Balanced SVHN(5/2 2) (Netzer et al. 2011), CIFAR- (5
Table 2 demonstrates that hybrid approaches on average out- perform alternative baselines on all five benchmarks (for a fixed compute, matched parameter count, and equal memory size for exemplar replay to ensure fairness in comparisons). Although hybrid approaches are given the same memory size, they can store more exemplars, depending on the com- pression rate given in Table 2 for each benchmark. The performance improvement of hybrid approaches, compared to exemplar rehearsal based strategies, can be attributed to exemplar diversity (availability of more exemplars). The effecctiveness of the availability of more exemplars, exem- plar diversity, is dell-documented in the literature (Masana
(Krizhevsky et al.) 2009), CIFAR- 100 (Krizhessky ct al. 2009), and minilmageNNn((0) (200)) 5) (Vinyals et al., 2016) benchmarks. Metrics. Performance is evaluated by the final test accuracy after training on the series of all tasks.
Network architectures. For MNIST benchmark, as in (Ven et al. )021), dense network with 2 hidden layers of 400 ReI.U units was used. We utilized its mirror for the decoder. For larger datasets, as suggested by (Masana et al., 2020
He et al. 2016). ResNet-32 is used. We employed 3 layers of CNNs for the decoder. Hyperparameters. We adopt Adam (Kingma & Bal 2014) as the optimizer. Exemplar
6

---

Autoencoder-Base Based Hybrid Replay for Class-Incremental Learning
Table 2: Empirical evaluation of AHR against suite of CIL baselines. Accuracies and the SEMs.
Benchmarks
BC
Reg
MNIST
BalancedSVHN
CIFAR-10 10
CIFAR-100
minilmageNet (20/5)
Tasks Conf.
(5/2)
(5/2)
(5/2)
(10/10)
#Total Exemplars
200
200
200
2000
2000
Vanilla Strategies (Lower Upper Bounds)
FT
N
N
19.93 #0.03
19.19
#0.04
18.72 =0.30
8.91
=0.12
4.32
=0.06
FT-E
T
N
92.17 10.16
87.13
20.37
72.17 20.84
48.47
20.83
39.02 20.74
Joint
N
98.48
#0.00
95.88 #0.04
92.37 #0.00
73.87 #0.10
73.45 20.15
Generative Classifier Strategies (Dynamically Expanding Architectures)
SLDA
N
N
87.30
+0.02
42.32
+0.04
38.34 +0.04
25.83 +0.01
19.03
#0.02
Gen-C
N
89.19
+0.05
51.92
49.38
29.69 40.62
22.57
PEC
90.81
+0.06
55.61 +0.21
52.41 40.33
37.53 #0.41
28.39
#0.36
Generative Replay Strategies (Rehearsal P
Pseudo- (iinddd-xempars))
DGR
N
88.50
20.43
28.17
=1.27
25.43 11.72
9.20
1122
6.59 11.13
McRGAN
I
89.83 20.37
33.49
#1.35
27.17 #1.84
11.39
11.23
7.82
1.05
BI-R-II
38.32 +1.43
37.48 +1.96
34.37
#1.20
29.71
#1.03
Exemplar Replay Strategies (Rehearsal of Exemplars)
GD
I
D
92.02 +0.17
89.11 #0.06
71.13 =0.72
49.01
+0.86
38.47
=0.62
GDumb
D
91.13 10.19
88.02 #0.47
72.79 +0.50
47.29
#0.71
39.64
20.70
iCaRL
I
D
93.06 10.33
89.63 10.61
73.29 10.73
10.62 49.38
43.51
20.68
EEIL
E
D
93.88 88 10.39
90.75 10.53
73.85 10.84
51.03
10.75
41.09
20.54
BIC
II
E
D
94.13 10.25
91.04 10.63
75.01 10.93
51.41
10.68
44.80
20.57
LUCIR
D
92.62 20.29
87.01
71.52 =0.71
47.08
20.94
36.95
70.79
IL2M
W
94.07 20.21
90.64 20.57
73.86 20.78
50.06 20.73
43.64
20.69
Hybrid Exemplar Generative
Strategy (Rehearsal of Decoded Exemplars)
C-TRL
D
94.11
20.27
91.07 #0.40
74.61 =0.61
51.74
#0.69
44.78
=0.68
REMIND
93.95
10.19
91.38
20.04
75.02 60.65
50.93
1:0.75
43.92
00.71
REMIND+
I
D
95.62 40.33
92.15
40.72
75.49
00.70
52.36
60077
45.02
#0.65
AHR
I
D
97.53 40.32
93.02
40.60
77.12 40.75
54.43
4-0.93
48.09
4-0.64
AHR Ablation Study (Impact of Compression and RFA)
AHR-lossy-mini
93.35 20.32
90.40 20.58
73.28 20.47
50.29
10.90
42.39
20.64
AHR-lossless-mini
93.77 76 20.26
90.88
20.50
73.68 #0.41
50.85
20.81
42.88
20.59
AHR-lossless
98.12 10.08
94.21 20.23
78.35
56.71
10.57
49.70
20.32
AHR-contrastive
95.12 #0.29
91.43
#0.54
74.87
#0.47
51.98
10.76
44.60
20.53
AHR-GMM
92.49 20.23
88.70 20.50
72.63 20.39
49.48
20.71
42.52
20.48
al. 2020p. Notably, our AHR approach outperforms other hybrid replay methods. This superior performance is due to AHR's more effective embedding in the latent space ing RFA, in contrast to i-CTRL, which relies on Linear Discriminative Representation. Additionally, AHR's archi- tecture offers an advantage by classifying directly in the latent space, whereas REMIND and REMIND+ perform classification only after the decoding process.
than the performance gain in the AHR-lossless setting. This observation supports key findings: (a) more exemplars, even decoded exemplars, significantly enhance performance (Masana et al. 2020); and (b) for mitigating CF, decoded exemplars are nearly as effective as perfect exemplars (Ven) et al. 2020). Fig. shows both the original and decoded images at different tasks (minilmageNet).
A Ablating the approaches for structuring the latent space. Hybrid approaches such REMIND and REMIND+ do not impose any explicit structure on the latent space. As result, our discussion here will not cover these methods, focusing instead on techniques that structure the latent space. Among these, i-CTRL employs Linear Discriminative Representa- tion, whereas AHR utilizes RFA. The comparative analysis presented the latter rows of Table also considers alter- native structuring methods, such contrastive loss (Chal et al. 2021) and the Gaussian Mixture Model (Ven et al. 2020 Our findings, as detailed in Table 2 indicate that RFA outperforms these alternatives. This is because RFA sy systematically embeds the class centroids of new classes
Ablating the impact of lossy compression. As shown in T 2 in the AHR-lossy-ming and AHR-lossless-min set- tings, AHR is given many exemplars as other exemplar rehearsal-basee baselines, with imperfect and perfect qual- ity. In the AHR-lossless setting, AHR given as many exemplars as AHR, but with perfect quality. The aim of these three settings is to investigate how much compres- sion by the encoder, and therefore the opportunity store more examples benefit AHR. In the AHR-lossy-nn mini and -HR-lossless-me mini settings, AHR performs on par with other exemplar rehearsal-based strategies. Interestingly, the performance loss these settings significantly greater

---

Autoencoder- Hybrid Replay for Class-Incremente Learning
Origital
Task
Tosk
Task
Task
Figure 4: Images produced by the decoder at different tasks (for the decoder size of 1.8M).
Table Performances for fixed memory (both the decoder and exemplars) and compute budgets.
Memory
Strategies
Benchmarks
Exemplars
Epachs
Wall-Clock Time
Performance
Decoder
Exemplar
LIFAR-100(10))
150 (latent)
1.4M
1.6M
50
462min
54.43 +0.93
AHR
minilmageNet(20/44)
190 (latent)
1.8M
40.54M
70
842min
48.09 20.64
CIFAR- 100(10/10)
20 (raw)
6M
60
473min
52.12 1:091
BIC
MinilmageNNet(20))
20 (raw)
42.34M
80
837min
45.23 #0.62
CIFAR- 10)
20 (raw)
6M
60
455min
50.81 40.74
11.2M
ninilmageNet(20555
20 (raw)
42.34M
80
861min
44.67 40.63
CIFAR- 100000100
20 (raw)
6M
60
478min
51.70 20.79
EEIL
minilmageNet(20/55
20 (raw)
42. 12.34M
80
859min
41.83 40.35
The Impatt Decoder Size (CIFAR-100)
The Impact Decoder Size (minllmagekke)
for three benchmarks: CIFAR-IO(5/2), CIFAR-100(10/00) and minilmageNe((205)) It explores the relationships be- tween (i) exemplar memory size and performance, (ii) ex- emplar memory size and compute time (epochs) for a target performance, and (iii) performance for a range of allocated compute times (wall-clock time), presented in the first, ond, and third rows, respectively. In the first row, observe that for small exemplar memory sizes, the performance gap between AHR and the baselines is more significant com- pared to larger memory sizes across all benchmarks. the second row, AHR proves to the least resource-consuming in terms of exemplar memory size and compute (epochs) needed meet target performance. In the third row, the performance is reported for various allocated wall-clock times.
K0
50
60
30
20
16 30
61
178
256
16
64 128 256
Number Examplars
Number of Exemplars
Figure 5: Performances for various decoder/memoryy sizes.
into the latent space with minimal amount of shift from their natural position and minimum changes to the weights of the neural network achieved by CPSEM. This level of system- atic embedding, necessary for ensuring the structuredness of the latent space, is not possible alternative approaches.
Decoder impact. Fig. 5 examines decoder sizes of [0.6, 0.8. 1, 1.2, 1.4] and [1, 1.4, 1.6, 1.8] million pa- rameters for CIFAR- MAAOR--00(1111000 and minilmageNer(20/55 benchmarks, respectively. is surprising how much mem- ory can be saved and, conversely, how much performance can be improved-witt sisting of just three layers of CNNs, which incurs minimal memory and compute overhead. Table 3 shows that the memory requirement of the decoder negligible compared to storing raw exemplars, and that the encoder-decoder ar- chitecture of AHR makes it possible to store one order of magnitude more exemplars in the latent space. Overall, Ta- ble demonstrates that AHR delivers superior results with the same memory/compute footprints.
Bias-correction and regularization. Table alsoo outlines the bias-correction and regularization methods used by dif- ferent strategies. For bias-correction, "N," "I," and "E" rep- resent none, implicit, and explicit, respectively. Implicit bias-correction, as seen in (Castro et al.) 2018), relies on data equalization without directly manipulating the weights, whereas explicit correction, as in (Belouadah Popescu 2019), involves direct weight adjustment. For regulariza- tion, "N," "D," and "W" denote none, data regularization, and weight regularization, respectively. Most exemplar rehearsal-based baselines, along with our AHR strategy, use implicit bias-correction via data equalization and data distillation for regularization.
relatively simple decoder con-
Resource-cossumptinn. Fig. assesses the performances
8

---

untoencoder-Baseee
Hybrid Replay for Class-Incremental Learning
5. Conclusion
sive study of class-incremental learning algorithms for visual tasks. Neural Networks, 135(1):38-54, 2021.
Exemplar replay strategies rely on storing raw data, which can be highly usually require orders of magnitude more memory than mod- els. Meanwhile, generative replay, training a (generative) model for generating the pseudo-data of past tasks, requires far less memory but tends to be less effective. We proposed AHR, a hybrid approach that combines the strengths of both methods, utilizing HAE with RFA for the incremen- tal embedding of new tasks the latent space. Instead of storing the raw data in the input space, AHR stores them in the latent space of HAE. Our experiments demonstrate the effectiveness of AHR.
especially since datasets
Buciluundefined, C., Caruana, R., and Niculescu-Mizil A. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Dis- covery and Data Mining, pp. 535-541, 2006.
Burda, Y., Grosse, R., and Salakhutdinov, Importance weighted autoencoders. In Proc. of International Con- ference on Learning Representations (ICLR), pp. 17-26, 2016.
Castro, F. M., Marin-Jimenez, M. J., Guil, N., Schmid, C., and Alahari, K. End-to-end incremental learning. In Proc. of the European Conference on Computer Vision (ECCV), pp. 233-248- 2018.
Acknowledgment
This research was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) through grants RGPIN-2021-04244 and RGP1N-2025-04778..
Cha, H., Lee, J., and Shin, J. Co2l: Contrastive continual learning. In Proceedings of the IEEE/CVF International conference on computer vision, pp.
2021.
Impact Statement
Chaudhry, A., Dokania, P. K., Ajanthan, T., and Torr, P. H. Riemannian walk for incremental learning: Understand- ing forgetting and intransigence. Proceedings of the European conference on computer vision (ECCV), pp. 532-547, 2018
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must specifically highlighted here.
Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajan- than, T., Dokania, P. K., Torr, P. H., and Ranzato, M. A. On tiny episodic memories in continual learning. arXiv: /90::/92////0 10486, 2019.
References
Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., and Tuytelaars, T. Memory aware synapses: Learning what (not) forget. In Proceedings the European conference on computer vision (ECCV), pp. 139-154, 2018
Cormerais, A. S., Masana, M., Van de Weijer, J., and Twar- dowski, B. On the importance of cross-task features for class-incremental learning. In Proc. of International Conference on Machine Learning (ICML) Workshops, pp. 3611-3620, 2021.
Aljundi, R., Caccia, I.., Belilovsky, E., Caccia, M., Lin, M., Charlin, L.,, and Tuytelaars, T. Online continual learning with maximally interfered retrieval. arXiv: 1908:/9000000 04742, 2019a.
De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A.. Slabaugh, G., and Tuytelaars, A contin- ual learning survey: Defying forgetting in classification tasks. IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI), 44(7):3366-3385 2021.
Aljundi, R., Kelchtermans, K., and Tuytelaars, T. Task-free continual learning. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11254-11266 1263,, 2019b.
Dhar, P., Singh, R. V., Peng. K.-C., Wu, Z., and Chellappa, R. Learning without memorizing. In Proc. of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 5138-5146, 2019.
Belouadah, E. and Popescu, A. 112m: Class incremental learning with dual memory. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 583-592, 2019.
Elsayed, M. and Mahmood, A. Addressing catastrophic forgetting and loss of plasticity in neural networks. In Proc. 12th International Conference on Learning Repre- sentations (ICLR, 2024), 2024.
Belouadah, E. and Popescu, A. Scail: Classifier weights scaling for class incremental learning. In Proc. of the JEEE/CVF Winter Conference on Applications of Com- puter Vision (WACV), pp. 1266-1275, 2020.
Farquhar, S. and Gal, Towards robust evaluations of continual learning. arXiv: 1805 1805 009333, 2018a.
Belouadah, E.,, Popescu, A., and Kanellos, I. comprehen-

---

Autoencoder-Basce Hybrid Replay for Clas--Incremental Learning
Farquhar, S. and Gal, Y. unifying bayesian view of contin- ual learning. In Proc. International Conference on Neural Information Processing Systems (NeurIPS) Workshop, pp. 3987-3995, 3995, 2018b.
Kao, T.-C., Jensen, K. T., Ven, G. M., Bernacchia, A., and Hennequin, G. Natural continual learning: success is journey, not (just) destination. In Proc. International Conference on Neural Information Processing Systems (NeurIPS), pp. 1266-1275, 2020.
Goodfellow, I., Bengio, Y., and Courville, A. Deep learning. MIT press, 2016.
Kemker, R. and Kanan, C. Fearnet: Brain-inspired model for incremental learning, arXiv:1711. 2017.
Hadsell, R., Rao, D., Rusu, A. A., and Pascanu, R. Embrac- ing change: Continual learning in deep neural networks. Trends in cognitive sciences, 22212):1028-1 1040, 2020.
Kim. T.-K., Stenger, B., Kittler, J., and Cipolla, R. Incremen- tal linear discriminant analysis using sufficient spanning sets and its applications. In Proc. of International Journal of Computer Vision (IJCV), pp. 216-232, 2011.
Hayes, T. L. and Kanan, C. Lifelong machine learning with deep streaming linear discriminant analysis. In Proc. of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR) Workshops, pp. 220-221, 2020.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv: 1412,6981 2014.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv: 1312,1111 2013.
Hayes, T. L., Kafle, K., Shrestha, R., Acharya, M., and Kanan, C. Remind your neural network to prevent catas- trophic forgetting. In Proc. of European Conference on Computer Vision (ECCV), pp. 466-483, 2020a.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Trabska-Barwingg A., et al. Overcoming catastrophic forgetting in neural networks. National Academy of Sci- ences, 11((13):5221-322 2017.
Hayes, T. L., Kafle, K., Shrestha, R., Acharya, M., and Kanan, C. Remind your neural network to prevent catas- trophic forgetting. In European conference on computer vision, pp. 466-483. Springer, 2020b.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers features from tiny images. 2009.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
LeCun, Y., Cortes, C., and Burges, C. Mnist handwritten digit database. 2010.
Lee, K., Lee, K., Shin, J., and Lee, H. Overcoming catas- trophic forgetting with unlabeled data in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 312-321, 2019.
Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253// 2015.
Hiratani, N. Disentangling and mitigating the impact of task similarity for continual learning. In Proc. International Conference on Neural Information Processing Systems 2024.
Li, S., Du, Y., Ven, G. M., and Mordatch, I. Energy-based models for continual learning. arXiv:2011.122 2020.
(NeurIPS), volume 34, pp.
Li, Z. and Hoiem, D. Learning without forgetting. IEEE Trans. on Puttern Analysis and Machine Intelligence (TPAMI), 40(1 (001)))923529920 2017a
Hou, S., Pan, X., Loy, C. C., Wang, Z., and Lin, D. Learn- ing unified classifier incrementally via rebalancing. In Proc. International Conference on Neural Information Processing Systems (NeurIPS) Workshop, pp. 831-839, 2019.
Li, Z. and Hoiem, D. Learning without forgetting. IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI), 40(12):2935 2947, 2017b.
Hsu, Y.-C., Liu, Y.-C., Ramasamy, A., and Kira, Z. Re- evaluating continual learning scenarios: A categorization and case for strong baselines. In Proc. International Conference on Neural Information Processing Systems (NeurIPS) Workshop, pp. 3987-3995, 2018
Lomonaco, V. and Maltoni, D. Core50: A new dataset and benchmark for continuous object recognition. In Proc. of International Conference on Robot Learning (ICRL), pp. 17-26, 2017.
Jung, H., Ju, J., Jung, M., and Kim, forgetting learning deep neural networks. arXiv preprint arXiv: Tciiv/600.000 160..00/2. 2016.
Maltoni, D. and Lomonaco, Continuous learning in single- Lingleincreementa-taa scenarios. Neural Networks, 116 (3):56773, 2019.
10

---

Satoencoder-Baseee Hybrid Replay for Clas--Incrementa Learning
Masana, M., Liu, X., Twardowski, B., Menta, M., Bag- danov, A. D., and van de Weijer, J. Class-incremental learning: Survey and performance evaluation image classification, arXiv:2010.15277. 2020.
Serra, J., Suris, D., Miron, M., and Karatzoglou, A. Over- coming catastrophic forgetting with hard attention the task. In Proc. of International Conference Machine Learning (ICML), pp.
2018.
Nazmitdinov, R., Puente, A., Cerkaski, M., and Pons, M. Self-organization of charged particles in circular geome- try. Physical Review E, 95(4):042603, 2017.
Shin, H., Lee, J. K., Kim, J., and Kim, J. Continual learn- ing with deep generative replay. In Proc. International Conference on Neural Information Processing Systems (NeurIPS) Workshop, pp. 2994-3003, 2017.
Netzer, Y., Wang. T., Coates, A., Bissacco, A., Wu, B., Ng, A. Y., et al. Reading digits in natural images with unsu- pervised feature learning. In Proc. Advances in Neural Information Processing Systems, volume 2011, pp. 7, 2011.
Tong, S., Dai, X., Wu, Z., Li, M., Yi, B., and Ma, Y. In- cremental learning of structured memory via closed-loop transcription. 2xXiv:2202.05411, 2022.
van de Ven, G. M. and Tolias. Three scenarios for continual learning. In Proc. International Conference on Neural In- formation Processing Systems Workshop, pp. 3611-3620, 2019.
Nguyen, V., Li, Y., Bui, T. D., and Tumee, R. E. Vari- ational continual learning. In Proc. International Con- ference Learning Representations (ICLR), pp. 3987- 3995, 2018.
Nori, M. K., Kim, I.M., and Wang. G. Federated class- incremental learning: A hybrid approach using latent exemplars and data-free techniques to address local and global forgetting. arXiv:2501. 2025.
Ven, G. M., Siegelmann, H. T., and Tolias, A. S. Brain- inspired replay for continual learning with artificial neural networks. Nature Communications, 1 11(1):1-14, 2020.
Ven, G. M., Li, Z., and Tolias, A. S, Class-incremental learn- ing with generative classifiers. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. pp. 3611-3620, 2021.
Ostapenko, O., Puscas, M., Klein, T., Jahnichen, P., and Nabi, M. Learning to remember: A synaptic plasticity driven framework for continual learning. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1/321-11329, 2019.
Vinyals, O., Blundell, C.,, Lillicrap. T., Wierstra, D., et al. Matching networks for one shot learning. Proc. Inter- national Conference on Neural Information Processing Systems, 29, 2016
Pang, S., Ozawa, S., and Kasabov, N. Incremental linear discriminant analysis for classification of data streams. IEEE Trans. on Systems, Man, and Cybernetics (TSMC), 35(5):905 914, 2005.
Wang, K., van de Weijer, J., and Herranz, L. Acae-remind for online continual learning with compressed feature replay. Puttern Recognition Letters, 150: 150:122-129, 2021.
Parisi, G. I., Kemker, R., Part, L., Kanan, C., and Wermter, S. Continual lifelong learning with neural networks: A review. Neural Networks, 11::5477,, 2019.
Wang. X., Geng. C., Wan, W., Li, S.-y,, and Chen, S. Forgetting, ignorance or myopia: Revisiting key chal- lenges in online continual learning. In Proc. International Conference on Neural Information Processing Systems (NeurIPS), volume 34, pp.
Pellegrini, L., Graffieti, G., Lomonaco, V., and Maltoni, D. Latent replay for real-time continual learning. In 2020 IEFE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 10203-10200 IEEE, 2020.
2024.
Wu, C., Herranz, L., Liu, X., Van De Weijer, J.. Raducanu, B., et al. Memory replay gans: Learning to generate new categories without forgetting. Proc. Advances in Neural Information Processing Systems, 31, 2018.
Prabhu, A., Torr, P. H., and Dokania, P. K. Gdumb: A simple approach that questions our progress in continual learning. In Proc. of the European Conference Computer Vision (ECCV), pp. 524-540. Springer, 2020.
Raghavan, S., He, J., and Zhu, F. Online class-incrent incremental learning for real-world food image classification. In Pro- ceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision, pp. 8 8195-8204, 2024.
Wu, Y., Chen, Y., Wang, L.., Ye, Y., Liu, Z., Guo, Y., and Fu, Large scale incremental learning. In Proc. of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 374-382, 2019.
Rebuffi, S.-A Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. In Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2001-2010, 2017.
Xiang, Y., Fu, Ji, P., and Huang, H. Incremental learning
using conditional adversarial networks. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6619-6628, 2019.

---

Antoencoder-Basel Hybrid Replay for Clas--Incremental Learning
Yu, D., Zhang, X., Chen, Y., Liu, A., Zhang, Y., Yu, P. S.,, and King, I. Recent advances of multimodal continual learning: A comprehensive survey, arXiv:2410.05352. 2024.
Zajac, M., Tuytelaars, T., and van de Ven, G. M. Prediction error-based classification for class-incremental learning. arXiv preprint a7806 2023.
Znke,, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence. In Proc. International Con- ference on Machine Learning (ICML), pp. 3987-3995, 2017.
Zeno, C., Golan, I., Hoffer, E.,, and Soudry, D. Task-agnostic continual learning using online variational bayes with fixed-point updates. Neural Computation, 33(11):3133) 3177, 2021.
Zhai, M., Chen, L., Tung, F., He, J., Nawhal, M., and Mori, G. Lifelong GAN: Continual learning for conditional image generation. In Proc. of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 2019.
Zhang, J., Zhang, J., Ghosh, S., D., Tasci, S., Heck, L., Zhang, H., and Kuo, C.-C. Class-incremental learning via deep model consolidation. In Proc. of the IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pp. 1131-1140, 2020.
Zhou, D.-.., Wang, Q.- Ye, H.-J., and Zhan, D.-C. A model 603 exemplars: Towards memory-efficient class- incremental learning,
2022.
Zhou, D.-W., Wang, Q.- Qi, Z.-H., Ye, H.-J., Zhan, D.- C., and Liu, Z. Class-incremental learning: A survey. IEEE Transactions Pattern Analysis and Machine In- telligence, 2024.
Zhuang, H., Chen, Y., Fang, D., He, R., Tong, K., Wei, H., Zeng, Z., and Chen, Gacl: Exemplar-free general- ized analytic continual learning. In Proc. International Conference Neural Information Processing Systems (NeurIPS), volume 34, pp. 29105-29205, 2024a.
Zhuang. H., Liu, Y., He, R., Tong, K., Zeng, Z., Chen, C., Wang, Y., and Chau, L.P.. F-oal: Forward-only online analytic learning with fast training and low memory foot- print in class incremental learning. In Proc. International Conference on Neural Information Processing Systems (NeurIPS), volume 34, pp.
2024b.
12

---

Autoencoder-Basee Hybrid Replay for Clas--Incremental Learning
A. Complexity Analysis
A.1. Generative Replay
This strategy (Shin et al. 2017) utilizes a generative model denoted by GEN() whose size is not growing with respect to the total number of tasks GEN() generates data of the past tasks (I....,Te 1} to be interleaved with the new task Te to be fed into discriminative model DIS() well as GEN() lest they are forgotten. Since neither the size of GEN() nor Dis() is growing with respect to the total number of tasks 7, the memory complexity can be said to be O(cte). The compute complexity, however, depends on the total amount of data to be fed to both GEN() and DIs() which is function of the number of tasks seen so far if they are not to be forgotten. Hence, the compute complexity is (t).
A.2. Generative Classifier
This strategy (Ven et al., 2021) trains brand new out-of-distribution detector for each new class (not task) denoted by OODxO. Hence, the number of the models grows as new classes arrive indicating that the memory complexity is Ott) Because this strategy trains a brand new model each time, it does not have to overwrite previous knowledge, and therefore, does not require replaying of the old data. As a result, the amount of data to be fed into each OODa() whenever each model trained does not grow with the number of classes so far because only the data of class is fed into OOD,0. Hence, the compute complexity (cte)..
A.3. Exemplar Replay
This strategy (Rebuffi et al. 2017) uses a memory denoted by MEM that stores dozens of exemplars per class so that each time new task arrives a representative minibatch of data consisting of all previous tasks is fed into the discriminative model denoted by DIs(). In this strategy, both the memory MEM has to grow in proportion to the number of tasks, and, the amount of data each time the discriminative model Dis() must consume has to increase in proportion to the number of tasks seen so far, therefore, memory and compute complexities are (t).
A.4. Hybrid Replay (Ours)
Although learning new tasks requires high-quality data, mitigating forgetting, as it has been reported (Ven et al. 2020), can be effectively accomplished with tolerably lossy data. Leveraging that, our hybrid replay strategy, a combination of generative and exemplar replay, proposes to use an autoencoder consisting of an encoder denoted by ENC() and a decoder denoted by DEC(). ENC() serves two goals: (i) it maps the input data to the latent space making it possible to use Euclidean distance for classification, and (ii) it compresses down the input data such that they can be stored in the memory MEM efficiently. Meanwhile, DEC() decompresses the data in MEM each time learning a new task. Since ENC() compresses down the input data, 10 times at the very least in our experiments, the memory complexity becomes (0.14), whereas the compute complexity is (tt. Note that the introduced overhead by adding DEC() is negligible as discussed in the experimental results section.
B. Hyperparameters and Implementation Details
We outline the hyperparameters for the 19 CIL strategies including vanilla and joint strategies serving as the lower and upper bounds for image classification on 5 benchmarks as specified in Table
C. Extended Literature Review
Task-based or task-free. Incremental learning literature features various learning scenarios that present their own unique challenges, and accordingly, diverse strategies have been developed (Parisi et al.] 2019: De Lange et al.] 2021 Yu et al.)
2024 Elsayed & Mahmood 2024). In the first place, there are two learning scenarios of fask-basee (van de Ven & Tolias
2019 Masana et al. (020) and task-free (Ven et al., 202T Aljundi et al., 2019b). In task-based, the model receives the data in the form of tasks: The task-based scenario is divided into two popular scenarios: task-incremental learning (TIL) and mass-incremmntal learning (CIL). Whereas in TIL the model receives the task-IDs during both training and inference, in CIL the tDsk-IDs are not given during inference and the model must infer them (Zhuang et al.) 2024a: Zhou et al. 2024). The task-free free scenario, meanwhile, totally abandons the notion of tasks altogether (Ven et al. 2021 A Aljundi et al. 2019b). AHR makes no prohibitive assumptions and can operate in all the above scenarios. Nevertheless, for comparability with the
13

---

Autoencoder-Bane Based Hybrid Replay for Class-Incremental Learning
Table 4: Number of latent exemplars for each benchmark for our AHR strategy.
Dataset
MNIST
SVHN
CIFAR-10
CIFAR-100
minilmageNet
Tasks Configuration
(5/2)
(5/2)
(5/2)
(10/10)
(20/5)
# Tasks
5
5
5
10
20
# Classes/Task
2
2
2
10
5
Classes
10
10
10
100
100
Model
Dense
ResNet32
ResNet32
ResNet32
ResNet32
Learning Rate
0.001
0.001
0.001
0.001
0.001
Momentum
0.9
0.9
0.9
0.9
0.9
# Epochs
40
50
50
50
70
Minibatch Size
128
128
128
256
256
Input Dimensions
28 x 28 x 1
32 x 32 x 3
32 x 32 x 3
32 x 32 x 3
84 x 84 x 3
Input Size
784
3072
3072
3072
21168
Latent Size
20
307
307
307
2117
Compression Ratio
40
10
R 10
2 10
= 10
# Raw Exemplars
200
200
200
2000
2000
# Latent Exemplars
8000
2000
2000
20000
20000
majority of works in IL, this paper examines the performance of AHR in the CIL setting.
Offline or online. Incremental learning can be either offline (Masana et al. 2020) or online Zajac et al., 2023 Wang et al.) 2024, Zhuang et al., 2024bb Raghavan et al. 2024), where in the offline learning scenario, the data of each task can be fed to the model multiple times before moving on to the next task. Conversely, in the online scenario, the model visits the data only once as they arrive and cannot iterate on them. In the literature, there are more works on the offline scenario (Parisi) et al. 2019; De Lange et al. 2021 Masana et al. 2020 Therefore, this paper examines the performance of AHR in the offline scenario.
Challenges and strategies. CIL faces many challenges such as CF, weight drift, activation drift, task-recency bias, and TC (Masana et al. 2020 Cormerais et al.) 202T). To tackle the weight drift and activation drift challenges of CIL (Kao et al., 2020), the most popular category of strategies, regularization, is often adopted (Zenke et al. 2017). Although regularization not alone effective in mitigating the TC challenge of CIL (Ven al. 2021p. , it has been proven productive in tandem with
other strategies like exemplar strategies (Rebuffi et al. 2017 Farquhar & Gal 200 8a
Hsu al. 2018 Castro al. 2018
Dhar et al. 2019 Serra al. 2018). Regularization strategies have two branches:
mitigates weight drift of the parameters optimized for the previous tasks by assigning an importance coefficient for each parameter in the network (assuming the independence of weights) after learning each task (Kirkpatrick et al.) 2017 Zenke et al., 2017 Nguyen (et al. 2018 Farquhar & Gal. 2018b Aljundi et al., 2018: Chaudhry et al. 2018). When learning new tasks, the importance coefficients help in minimizing weight drift. EWC (Kirkpatrick al., 2017) and (Zenke al., 2017) are popular weight
Weight regularization and data regularization. Weight regularization (Hiratani 2024
regularization strategies. EWC (Kirkpatrick al. 2017) relies on diagonal approximation of the Fisher matrix to weigh the contributions from different parameters. SI (Zenke et al. 2017) maintains and updates per-parameter importance measures in an online manner.
Data regularization is the second regularization strategy, aimed at preventing activation drift through knowledge distillation (Bueiluundefine et al. 2006. Hinton et al. 2015), originally designed to learn a more parameter-efficiee student network from a larger teacher network. Differs from weight regularization which imposes constraints on parameter updates, knowledge distillation focuses on ensuring consistency in the responses of the new and old models. This distinctive feature provides broader solution space which is why distillation has become the dominant strategy used in tandem with rehearsal strategy (Li & Hoiem 2017a Jung et al.) 20166) Dhar et al. 2019 Zhang et al. 2020 Lee et al. 2019). LwF (Li & Hoiem 2017a), a popular data regularization strategy, uses a distillation loss to keep predictions consistent with the ones from an old model. Our AHR strategy as discussed in the previous section incorporates LwF into exemplar replay to overcome activation drift.
Bias-correction, Its aim is to address the challenge of task task-recency bias, which refers to the tendency of incrementally
14

---

Autoencoder-Baan Based Hybrid Replay for Clas--Incremental Learning
learned networks be biased towards classes in the most recently learned task (Belouadah al. 202 Li al. 2020 Wu
et al. 2019; Maltoni & Lomonaco
2019 Lomonaco & Maltoni 2017 Zeno al., 2021 Belouadah & Popeseu 2020).
Labels trick (LT) (Dhar et al. 2019) a rehearsal-free bias-correcting algorithm that prevents negative bias past tasks. LT often improves the performance when added on top of other strategies (Wu et al. 2019). However, because AHR separates representation learning from classification similar to (Rebufh et al. 2017). which gives a dose of immuunit to task-recency bias. Because AHR samples the tasks/classes for its minibatch in a statistically representative manner during training inspired by (Castro et al. 2018), incorporuting an explicit bias-correction strutegy such as LT proved unnecessary.
Generative classifier. CIL strategies can be categorized into two types: discriminative- and generative-baser classification. Strategies count discriminative classifiers when eventually discriminator performs the classification. However, generative classifiers perform classification only and directly using generative modeling (Ven et al. 2021 Pang et al. 2005
Zajac
ct al. 2023). Gen-C (Ven et al.
2021) is a novel rehearsal-fri generative classifier strategy; the strategy does baseed
generative modeling (Li et al.
2020) via VAEs (Kingma & Welling, (013) and importance sampling (Burda et al. 2016).
SLDA (Hayes & Kanan 2020 (popular in data mining (Kim et al. 2011 Pang et al., (005)) is thought to another form
of generative classifier
Ven et al. 2021); 1); however, prevents representation learning.
15


Reviews:
[{"review_id": "upNy5XUA5U", "reviewer": "Reviewer_Ej9V", "summary": "The paper tackles the problem of class incremental learning, where the model sees a sequence of tasks of different classes, and needs to adapt to them sequentially while minimizing catastrophic forgetting and task confusion. During testing, the model does not have access to the task ID. \n\nTo solve this problem, the authors propose to train a novel autoencoder architecture that is used for both replay and classification. The contributions of the paper are as follows: \n* The authors propose a hybrid autoencoder, that is used both for computing the latent representation to store in memory and replay, and for classification. This autoencoder is coupled with a physics-inspired approach: charged particles system energy minimization, and repulsive force algorithm to incrementally add components to the latent space memory. The goal of this approach allows for the different classes to be away from each other in the latent space, allowing for a simple use of the Euclidean distance to the class centroids to classify samples during test. \n* These autoencoders are used in an approach that combines exemplar and generative replay ideas. As mentioned above, the approach stores exemplars from the latent space, reducing the memory footprint of the approach. When replay data is needed, the autoencoder is used to decode samples from the memory. Contrarily to generative replay, the decoder is trained for memorization and not to generate new samples, which hedges the approach against some of the drawbacks of generative replay. \n* The approach is tested on 5 benchmarks, and compared to 10 baselines, showing an improvement in most cases. The authors also conduct some ablation studies, mainly testing the importance of the use of the physics inspired approach to structure the latent space.\n\n## Update after rebuttal \n\nThe authors answered most of my questions and addressed most of my concerns. I also thank the authors for their reactivity during the rebuttal. With the added evaluations and additional experiments to the revised version, I am happy to increase my score to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are impressed by the reviewer's high-quality feedback, which reflects a deep understanding of the field and engagement with our work. We particularly appreciate the fairness of their critique. In response, we offer the following clarifications:\n\n### On the Choice of Not Changing Class Centroid Positions\n\nWe acknowledge the importance of considering more realistic evaluation scenarios where task data may not be perfectly *class-disjoint*, in which case the idea of fixing class centroids (successfully employed in recent works [1]) is **suboptimal**. However, our decision to employ the **standard, well-established CIL benchmarks** (which typically feature *disjoint* classes) was primarily driven by the necessity of ensuring **fair, direct, and easily interpretable comparisons** with the large body of existing literature evaluated under these protocols.\n\nWe commit to adding a dedicated discussion in the '**Limitations and Future Work**' section of our revised paper, where we state that while the proposed fixed-centroid approach demonstrates strong performance under the standard *disjoint* CIL setting, future work should investigate **adaptive centroid mechanisms** to optimize performance for *non-disjoint*, real-world data distributions [2,3,4]. We will also mention that in such realistic **non-disjoint** benchmarks, the decoder's degradation could significantly be **mitigated** because the overlap of classes across tasks could lead to the decoder **revisiting** older classes.\n\n### On the Choice of Benchmarks and Sequence Length\n\nWe acknowledge the **limitations of standard CIL benchmarks**, particularly their often *artificial nature* and relatively *short task sequences*. We thank the reviewer for providing references to **more realistic benchmarks** [2, 3, 4] and studies on the **impact of sequence length** [5]. We commit to incorporating a discussion of these limitations and citing these important works in the '**Limitations and Future Work**' section.\n\n### Incorporating Additional Evaluation Metrics\n\nWe will incorporate more **granular evaluation metrics** beyond final accuracy and agree that metrics such as **average forgetting**, **forward transfer**, **incremental confusion maps**, and potentially **accuracy curves over time** can provide better understanding of any potential biases (recency/primacy). We commit to including these evaluations in the **revised version** of our paper.\n\n### Additional Points and Clarifications\n\n*   **On Latent Space Dimension:** We clarify that the compression ratio varies depending on the dataset and the underlying network architecture used for feature extraction. The specific compression ratios and latent space sizes used in our experiments are summarized below:\n\n    | Dataset          | MNIST | SVHN | CIFAR-10 | CIFAR-100 | miniImageNet |\n    | :--------------- | :---: | :--: | :------: | :-------: | :----------: |\n    | Compression Ratio (%) | ≈ 40  | ≈ 10 |   ≈ 10   |    ≈ 10   |     ≈ 10     |\n    | Latent Space Size | 20  | 307 |   307  |    307   |     2117     |\n\n*   **On Figure Readability and Color Choices:** In the revised paper, we will ensure **the distinctness of color palettes**, potentially incorporating different line styles or markers where appropriate.\n\n*   **Inclusion of a Latent Replay Study:** We will include reference [6] and discuss how it compares with our proposed work.\n\n### Kind Request for Reassessment\n\nWe believe our proposed Autoencoder Hybrid Replay method remains highly effective, **even under realistic benchmark conditions**, due to its ability to store **a significantly larger and more diverse set of exemplars**. Extensive research supports that *exemplar diversity* effectively mitigates catastrophic forgetting. Additionally, we have devised solutions for handling non-class-disjoint benchmarks with adaptive centroids, which we could not include here due to space constraints but will gladly discuss during the discussion period.\n\nWe have strived to address the key concerns, regarding benchmarks and evaluation metrics, through clarification and planned revisions (expanded limitations/future work section, additional metrics). We hope these responses and commitments are satisfactory. If you feel that our clarifications and planned updates satisfactorily address your concerns, we kindly request you to consider increasing your score. This would greatly support the acceptance of our work and help us contribute meaningfully to the field. Regardless of your decision, we are sincerely grateful for your thoughtful feedback. Thank you.\n\n**References**\n\n[1] Combating Inter-Task Confusion and Catastrophic Forgetting, Moslem et al. 2025.\n\n[2] Meta-Album: Multi-domain Meta-Dataset, Ullah et al. 2022\n\n[3] A Large-scale Study of Representation Learning, Zhai et al. 2020\n\n[4] NEVIS'22: A Stream of 100 Tasks, Bornschein et al. 2023\n\n[5] Challenging Common Assumptions, Lesort et al., 2023\n\n[6] Continual Learning with Foundation Models, Ostapenko et al. 2022", "claims_and_evidence": "The paper main claim is to achieve better performance for class incremental setting while reduce the memory footprint and keeping a linear compute as in competitor methods. \n\nThe authors prove this claim empirically through extensive experiments, on several datasets, and comparing to a good number on baselines. While I have some comments on the choice of baselines and used benchmarks (see next sections), I think the evidence is clearly provided and relatively convincing.", "methods_and_evaluation_criteria": "Strengths: \n* The whole approach is relatively novel to the best of my knowledge.\n* In particle, the use of the physics inspired approach to structure the latent space is interesting and see=ms to add a significant effect. \n* The experiments are extensive. I particularly appreciated the ablatio studies. \n\nWeaknesses: \n* The authors made the choice of not changing the class centroid positions. While this works well for the tested benchmarks, I think it is mainly due to the fact that classes are perfectly disjoint across tasks in this artificial benchmarks. In real applications, this might not be the case, and allowing for a more flexible adaptation of the latent space can not only be more generalizable, but also have other effects (e.g. help reduce the degradation of the decoder).\n* Regarding the benchmarks, despite their wide use in the literature, they are highly artificial and have a very limited representation of real world applications. There are multiple attempts to propose alternative benchmarks for continual learning and related topics (e.g. meta-learning) that are more realistic. I will provide the references in the dedicated section below. They also consist of relatively short sequences. Recent works have indicated the impact of the sequence length on the model behavior (see reference below too). \n* For evaluation, the authors base their analysis on accuracy at the end of the sequence only. It would be interesting to have a more granular approach, with metrics such as forgetting of forward transfer, to test if the approach has a recency or primacy bias, if it increases knowledge accumulation, etc ...", "theoretical_claims": "The paper is empirical in nature. \nThe main theoretical contribution is in deriving the different objectives and algorithms that constitute the approach. Except of the first comments under Weaknesses in the previous section, I didn't detect any other issues.", "experimental_designs_or_analyses": "My main critique to the experimental design is the choice of benchmarks and metrics as explained above. \n\nAs mentioned above, I appreciated the ablation studies that highlight the importance of different components of the approach. \nThese results can be improved with additional tests.  For example, it seems from the paper introduction and claims that the latent space size is fixed to 2.5 or 10% of the input size. If this is not the case, this should be made clearer. If it is the case, it would be interesting to test the impact of the latent space dimension on the results. \n\nOn the form, the curves can be improved and made more readable. In particular, the choice of colors is not optimal. For example. the finetuning with replay exemplars baseline and the proposed approach (AHR-up) have the same color in figure 3.", "relation_to_broader_scientific_literature": "While the paper already include a detailed literature review, there are some missing references as mentioned above. Nevertheless, the paper is relatively well situated in the literature. In particular, the comparison to several generative and exemplar replay approaches and strategies is interesting. \n\nBenchmarks: \n* Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classification, Ullah et al. 2022\n* A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark, Zhai et al. 2020 \n* NEVIS'22: A Stream of 100 Tasks Sampled from 30 Years of Computer Vision Research, Bornschein et al. 2023\n\nOn the impact of sequence length\n* Challenging Common Assumptions about Catastrophic Forgetting, Lesort et al, 2023", "essential_references_not_discussed": "Regarding the idea of latent replay, and related baselines, I think the authors are missing an important reference: \n* Continual Learning with Foundation Models: An Empirical Study of Latent Replay, Ostapenko et al. 2022", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "All of my questions are related to my previous comments: \n1. Can the authors comment on the choice of not changing the class centroids? \n2. How would the method behave and how should it be modified to be applicable in more realistic scenarios and benchmarks?\n3. How does the method impact other continual learning metrics? \n4. How does the latent space dimension impact the results?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EazOLBcYM4", "reviewer": "Reviewer_obFb", "summary": "The paper proposes an Autoencoder-Based Hybrid Replay (AHR) strategy for class-incremental learning (CIL), addressing catastrophic forgetting (CF) and task confusion (TC) while reducing memory complexity. The core innovation is a Hybrid Autoencoder (HAE) that compresses exemplars into a latent space using a repulsive force algorithm (RFA) inspired by charged particle systems. This allows efficient storage and recovery of exemplars via a decoder designed for memorization. Extensive experiments validate the effectiveness of this method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thorough review and insightful comments. We are particularly grateful for the recognition of AHR's strengths. Below, we clarify and address issues pointed out by the reviewer:\n\n### On Applicability to Task-Free CIL\n\nWe'd like to clarify that **Figure 2** is intended solely as a *visualization* to illustrate the concept of a task within our **task-based CIL** framework. We do **not** claim that our current approach performs equally well in *task-free* settings. Nevertheless, due to the inherent **compression capabilities** of our proposed architecture, which facilitate the storage of larger and more diverse exemplars, it could potentially, with minor adjustments, be adapted for use in *task-free* settings. We will explicitly clarify this point in the revised paper and leave a detailed exploration of this possibility to **future work**.\n\n### On the Choice and Recency of Baselines\n\nOur selection aimed to include **representative and relevant methods** from the CIL literature, particularly focusing on **hybrid replay strategies** where latent or compressed representations are stored. To this end, we included methods like **i-CTRL** [1] (2022), **REMIND** [2] (2020), and its variant **REMIND+** [3] (2021), which align closely with the concept of AHR. However, we remain **open to incorporating additional recent baselines** if the reviewer can point to specific, highly relevant examples where a direct comparison would be particularly insightful. Furthermore, we believe AHR's core contribution, using an **efficient hybrid replay**, is a concept potentially **orthogonal and complementary** to other advancements in CIL backbone architectures. One could envision integrating AHR's lightweight decoder mechanism with the latent representations of other contemporary CIL models to potentially enhance their performance under strict memory constraints, leveraging our memory-efficient replay approach.\n\n### On Performance with ViT Architectures\n\nOur current experiments utilize standard backbone networks like ResNet32 commonly employed in the CIL literature. This choice was primarily driven by the goal of ensuring **fair and direct comparability** with a wide range of existing CIL methods, many of which report results using these architectures. While we believe AHR's core mechanism, efficient exemplar storage, is **conceptually compatible** with features extracted by ViTs, rigorously evaluating this combination was beyond the scope of the current study due to the aforementioned reasons of comparability and computational cost. We will note the exploration of AHR with ViT backbones as a valuable avenue in the '**Limitations and Future Work**' section.\n\n### On Performance with Imbalanced Data Distributions\n\nWhile our current experiments utilize standard balanced benchmarks for clear comparability, we believe AHR's core contribution, its highly **efficient hybrid replay mechanism**, is conceptually **orthogonal** to the challenge of class imbalance itself. Critically, AHR's ability to store **significantly larger and potentially more diverse sets of exemplars** within a fixed memory budget remains a powerful tool, regardless of the underlying class distribution. Since maintaining knowledge of past classes especially minority classes, in an imbalanced setting is vital, the enhanced replay capability offered by AHR is anticipated to be **advantageous** even under such conditions. However, we acknowledge that we have **not explicitly evaluated** AHR under imbalanced scenarios in the present work. Thoroughly investigating AHR's performance in such settings, potentially in combination with techniques specifically designed for imbalance, is indeed crucial. We recognize this and will explicitly identify the evaluation of AHR under **class imbalance** as an important area for **future investigation** in the '**Limitations and Future Work**'.\n\n### On Hyperparameter Sensitivity (λ)\n\nConcerning the weighting factor λ in Equation 1, which balances the reconstruction and classification losses, while detailed ablation studies were omitted from the main paper due to space constraints, we performed sensitivity analyses during development. The results for the **CIFAR-10 5/2 split benchmark**, demonstrate the impact of varying λ:\n\n| λ Value          |   0   |   20  |   40  |   60  |  **80** |  100  |  120  |\n| :--------------- | :---: | :---: | :---: | :---: | :-----: | :---: | :---: |\n| Avg. Accuracy (%) | 42.03| 63.93| 70.51| 73.70| **77.12** | 76.53| 72.26|\n\nWe commit to including a more comprehensive sensitivity analysis for λ in the **appendix** of the revised paper.\n\nThank you for reading our response.\n\n**References**\n\n[1] Incremental learning of structured memory via closed-loop transcription, Tong et al., 2022.\n\n[2] Remind your neural network to prevent catastrophic forgetting, Hayes et al., 2020.\n\n[3] Acae-remind for online continual learning with compressed feature replay, Wang et al., 2021.", "claims_and_evidence": "Most claims are generally supported by rigorous experiments and ablation studies. However, the paper claims that AHR is applicable to task-free CIL, but it is only briefly mentioned in Figure 2 and lacks relevant experimental proof.", "methods_and_evaluation_criteria": "Yes, the proposed method makes sense to the issue at hand.", "theoretical_claims": "Yes, its theoretical claims I have not found to be incorrect.", "experimental_designs_or_analyses": "Experiments are comprehensive but have some shortcomings:\n1.\tThe baselines compared are somewhat outdated, with most of the methods from around 2020 and only one from 2023. What is the relative performance of the AHR compared to the most recent methods?\n2.\tIn the five datasets, the architecture of the network uses the underlying dense network and ResNet32, how does the AHR perform for the models of the ViT architecture?\n3. Benchmarks use balanced class splits; performance on imbalanced data (common in real-world CIL) is untested.", "relation_to_broader_scientific_literature": "The work effectively builds on prior CIL strategies and addresses the shortcomings of previous approaches.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1.Innovative combination of physically-inspired RFA and autoencoder provides new ideas for CIL.\n2. Extensive experiments on different benchmarks have demonstrated the effectiveness of AHR.\n3.The proposed AHR practically focuses on memory/computation efficiency and achieves good results.\nWeaknesses:\n1.\tReferring to the questions posed in Claims And Evidence and Experimental Designs Or Analyses sections.\n2.\tThe effect of some hyperparameters on the performance is missing, such as λ in Eq. 1.", "other_comments_or_suggestions": "None", "questions_for_authors": "Please refer to the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}, {"review_id": "jDzzKJX9Oc", "reviewer": "Reviewer_rgDX", "summary": "In this paper, the authors proposed a method named hybrid autoencoder (HAE) and a strategy named autoencoder-based hybrid replay (AHR). For HAE, it is an autoencoder learnt with charged particle system energy minimization (CPSEM) equations and repulsive force algorithm (RFA). This autoencoder is used for both decoding stored embeddings for replay and encoding inputs for classification. The AHR is storing data embedding rather than raw data.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and feedback. We have carefully considered the points raised and we offer the following responses and clarifications:\n\n### On the Theoretical Clarity\n\n- **Regarding O(cte)**:\nThe notation \"O(cte)\" was intended as shorthand for \"constant time complexity\" (O(1)). We agree this was unclear and will explicitly replace \"O(cte)\" with the correct, widely recognized notation **\"O(1)\"** in the revised manuscript.\n\n- **Regarding O(0.1t)**:\nThe notation \"O(0.1t)\" was deliberately chosen to emphasize that our hybrid replay method uses approximately 10 times less memory compared to standard non-hybrid replay methods. Such notation, indicating proportional improvements in complexity, is commonly used in professional literature and textbooks to highlight practical differences in resource usage.\n\n## Incorporating Additional Evaluation Metrics\n\nWe commit to incorporating metrics such as **average forgetting**, **forward transfer**, **incremental confusion maps**, and **accuracy curves over time** to provide a better understanding of any biases (recency/primacy).\n\n### Clarification on Hybrid Method Novelty\n\nIndeed, hybrid replay strategies have been explored previously, and we cited relevant works like **i-CTRL** [1] (2022), **REMIND** [2] (2020), and its variant **REMIND+** [3] (2021) in our paper. To provide clearer context and comparison, we will include the following comparative analysis:\n\n|                  | Classification Approach                                                 | Quantization                                                                                              | Learning Scenario | Latent Space Representation                                | Architectural Simplicity                                            |\n|------------------|------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------|-------------------|------------------------------------------------------------|---------------------------------------------------------------------|\n| **AHR**           | Classification within the latent space of the encoder using Euclidean distance. | Easy integration possible, potentially improving performance.                                             | Offline           | Structured latent space (Lennard-Jones Potential).         | Minimalistic and clean design.                                      |\n| **Remind**       | Classification after decoding with cross-entropy loss.                 | Applies quantization to latent exemplars for compression.                                                 | Online            | Unstructured latent space (CNN-based).                     | Complex architecture, complicating implementation and scalability.  |\n| **Remind+**      | Classification after decoding with cross-entropy loss.                 | Uses quantization for feature compression.                                                                | Online            | Unstructured latent space (autoencoder-based).             | Complex architecture, hindering scalability.                        |\n| **i-CTRL**       | Classification within latent space using Euclidean distance.           | Easy integration possible, potentially improving performance.                                             | Offline           | Structured latent space (Linear Discriminative Representation). | Minimalistic and clean design.                                      |\n\n## Additional Points and Clarifications\n\n* **Relevant References:** We will discuss the relevant work by Skantze and Willemsen [4] to provide further context.\n\n*   **On Latent Space Dimension:** We clarify that the compression ratio varies depending on the dataset and the underlying network architecture used for feature extraction. The specific compression ratios are\n\n    | Dataset          | MNIST | SVHN | CIFAR-10 | CIFAR-100 | miniImageNet |\n    | :--------------- | :---: | :--: | :------: | :-------: | :----------: |\n    | Compression Ratio (%) | ≈ 40  | ≈ 10 |   ≈ 10   |    ≈ 10   |     ≈ 10     |\n\nThank you very much for your thorough review and feedback. We respectfully invite you to review the comments from the other reviewers as well, who have positively acknowledged key strengths and contributions of our work. We hope this broader context might offer additional perspectives on the significance and novelty of our contributions, leading to a more balanced and constructive reassessment. We greatly value your insights and remain committed to addressing your concerns comprehensively.\n\n**References**\n\n[1] Incremental learning of structured memory, Tong et al., 2022.\n\n[2] Remind your neural network to prevent forgetting, Hayes et al., 2020.\n\n[3] Acae-remind for online continual learning, Wang et al., 2021.\n\n[4] Collie: Continual learning of language grounding. Skantze et al., 2022.", "claims_and_evidence": "The theoretical part was not described clearly. The experiment part is relatively reasonable, but more metrics should be added.", "methods_and_evaluation_criteria": "The author used several baselines on different benchmark datasets with accuracy. They also compared memory used among methods.", "theoretical_claims": "This part makes people confusion, especially the mess notations. It is hard to find the definitions of them. For example, they stated that complexity was O(cte) but hard to find the meaning. They mentioned that their space complexity was O(0.1t). It is a very strange and unprofessional way to describe the complexity. O(0.1t) is O(t).", "experimental_designs_or_analyses": "The experiment design is relatively reasonable. As I mentioned above, more metrics should be considered like how performance dropping along with more tasks leant.", "relation_to_broader_scientific_literature": "Storing embedding rather than raw data is not new in continual learning. The idea of learning autoencoder to reconstruct for replay makes it better.", "essential_references_not_discussed": "There are some works that have the idea of storing embedding, like “Skantze G, Willemsen B. Collie: Continual learning of language grounding from language-image embeddings[J]. Journal of Artificial Intelligence Research, 2022, 74: 1201-1223.”", "other_strengths_and_weaknesses": "The good things are the idea of learning an autoencoder to reconstruct stored embedding and experiments included many baselines. However, the description of the method and notations are confusion. The experiment part should include more metrics.", "other_comments_or_suggestions": "See other parts.", "questions_for_authors": "What is O(cte)? Is there any misunderstanding of O(0.1t)? Why did you just said the embedding size is 1/10 of the raw data?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a hybrid AE that is coupled with charged particle system energy minimization and repulsive force algorithm. This algorithm has linear time complexity (O(0.1t) should be notated by O(t)). Complaints include (1) using outdated baselines to compare and (2) only using final accuracy metric instead of forgetting and forward transfer. The only reviewer who gave rejection score did not elaborate the reviews and was not very responsive to authors' rebuttal. Therefore, the AC recommends to weakly accept the submission to ICML 2025.
Decision: Accept (poster)