Semantic Aware Representation Learning for Lifelong Learning
Authors: Fahad Sarfraz
Elahe Arani
Bahram Zonooz
Venue: ICLR 2025 Poster (2025)
Keywords: lifelong learning
continual learning
sparsity
semantic relationships
sparse coding
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: SARL enhances lifelong learning by employing sparse coding and semantic similarities between objects across tasks, facilitating improved feature reuse and minimizing interference.

Abstract: The human brain excels at lifelong learning by not only encoding information in sparse activation codes but also leveraging rich semantic structures and relationships between newly encountered and previously learned objects. This ability to utilize semantic similarities is crucial for efficient learning and knowledge consolidation, yet is often underutilized in current continual learning approaches. To bridge this gap, we propose Semantic-Aware Representation Learning (SARL) which employs sparse activations and a principled approach to evaluate similarities between objects encountered across different tasks and subsequently uses them to guide representation learning. Using these relationships, SARL enhances the reusability of features and reduces interference between tasks. This approach empowers the model to adapt to new information while maintaining stability, significantly improving performance in complex incremental learning scenarios. Our analysis demonstrates that SARL achieves a superior balance between plasticity and stability by harnessing the underlying semantic structure.

Published as a conference paper at ICLR 2025
SEMANTIC-AWARE REPRESENTATION LEARNING FOR LIFELONG LEARNING
Fahad Sarfraz!2 Elahe Arani!A.& Bahram Zonooz
Dep. of Mathematics and Computer Science, Eindhoven University of Technology (TU/e), Netherlands 2 TomTom, Netherlands
Wayve Technologies Ltd, London, United Kingdom
(f. sarfraz, a.arnni, b. Z.zooozz0ttue ml
ABSTRACT
The human brain excels at lifelong learning by not only encoding information in sparse activation codes, but also leveraging rich semantic structures and relation- ships between newly encountered and previously learned objects. This ability utilize semantic similarities is crucial for efficient learning and knowledge consol- idation, yet it is often underutilized in current continual learning approaches. To bridge this gap, we propose Semantic-Aware Representation Learning (SARL), which employs sparse activations and principled approach to evaluate similar- ities between objects encountered across different tasks and subsequently uses them guide representation learning. Using these relationships, SARL enhances the reusability of features and reduces interference between tasks. This approach empowers the model to adapt to new information while maintaining stability, sig- nificantly improving performance in complex incremental learning scenarios. Our analysis demonstrates that SARL achieves superior balance between plasticity and stability by harnessing the underlying semantic structure.
INTRODUCTION
Deep neural networks (DNNs) have been designed for static in-distribution batch learning, whereby the model assumes that each batch of training data is representative of the underlying joint distri- bution. However, this assumption fails when processing sequential data continual learning (CL) scenarios where information is made available incrementally over time, and hence the training batch at given point only contains samples from the current task. The inadequacy of standard train- ing to handle such dynamic data distributions often results in catastrophic forgetting (McCloskey & Cohen 1989), erasing previously acquired knowledge as the model learns the new data. To ad- dress these challenges, CL methods aim to retain previous knowledge while learning new tasks, enabling models to accumulate knowledge across tasks (Parisi et al. 2019). The key challenge in CL is maintaining optimal balance between stability and plasticity (Mermillod et al.) 2013): the model needs enough plasticity to quickly adapt to new tasks, but also enough stability to preserve previously acquired knowledge. Existing approaches often trade off between high plasticity (quick adaptation) and high stability (knowledge retention), without effectively balancing the two. This trade-off highlights the difficulty in consolidating knowledge efficiently while learning sequentially.
Many CL approaches tend to emphasize either stability or plasticity, often overlooking the complex relationship between these two essential components. Stability-focused methods aim to preserve important weights to maintain previously acquired knowledge (Zenke et al.) 2017 Li et al. 2023)), while caasticity-orientee
approaches allow significant representation changes to adapt flexibly to new tasks (Caccia et al., 2022 Liang & Li 2023). This dichotomy often treats stability and plasticity as separate objectives rather than interconnected factors knowledge consolidation. We hypothesize that leveraging semantic relationships between objects can bridge this gap by enhancing knowledge transfer and integrating new information with existing knowledge. By exploiting these semantic structures, CL model can reuse and consolidate overlapping concepts, enabling it to learn new
"Equal advisory role.
Code available at /
git

---

Published as a conference paper at ICLR 2025
objects without severely disrupting previously learned representations. This approach mirrors the human cognitive ability to form associations between related concepts, facilitating retrieval, transfer, and consolidation of knowledge across tasks (Binder & Desai 2011: Saxena et al. 2022).
The human brain exemplifies the power of semantic relationships in organizing and retaining knowl- edge. By forming associations between related concepts, the brain facillittes the retrieval, transfer, and consolidation of information. When encountering a new object similar to a previously learned one, the brain retrieves relevant information using semantic relationships (Saxena al., 2022), , aid- ing the learning of new objects and linking new information to existing knowledge. Furthermore, semantic relationships support knowledge consolidation and retention by reusing common concepts and adapting them in a cohesive manner. Moreover, semantic relationships play an important role in organizing memory (Artuso al. 2022). necessitating that information be encoded in a way that captures these relationships. The brain employs sparse coding (Foldiak) (003), where information is represented by the strong activation of small subset of neurons. Notably, objects sharing concepts likely activate overlapping sets of neurons, efficiently encoding semantic relationships. We hypoth- esize that emulating these mechanisms can enable DNNs to better balance stability and plasticity by leveraging the rich semantic structure between objects.
To this end, Semantic- Aware Representation Learning (SARL) emulates sparse coding in the brain and employs a principled approach to capture the semantic relationships between objects and to use them to guide representation learning. SARL. leverages activation sparsity to emulate sparse coding and represents each object through semantically rich object prototypes (Snell et al.] (017) derived from the mean sparse activations of the object samples. These prototypes effectively capture the semantic structure of object relationships, allowing the model to reuse and consolidate knowl- edge efficiently. By capturing the inherent similarities between objects, SARL leverages semantic relationships to guide the alignment of new object prototypes with those of similar classes from pre- viously learned tasks, while promoting separation from dissimilar ones. This alignment encourages a structured representation of knowledge, facilitating knowledge transfer across tasks and reducing interference. Furthermore, SARI. incorporates prototype regularization to ensure model stability, mitigating the risk of forgetting previously learned information, and fostering effective consolida- tion of knowledge in cohesive manner. This mechanism enables the preservation of valuable representations while adapting to new tasks, ultimately enhancing the model's overall learning ca- pabilities.
Our empirical analysis on challenging inusemincremettll learning scenarios across various datasets demonstrates that L significantly enhances lifelong learning performance by leveraging seman- tie structure for representation learning. By effectively aligning representations based on semantic relationships, SARL facilitates efficient knowledge transfer and consolidation, enabling the model to adapt seamlessly to new tasks while retaining previously acquired knowledge. These capabilities are further supported by our analysis, which reveals that SARL achieves a better balance between model stability and plasticity, mitigates forgetting, and reduces task recency bias.
2 RELATED WORK
Approaches addressing catastrophic forgetting in CL can be broadly categorized into three groups Regularization-batt methods penalize changes in the model either in parameter space (Farajtabar
al., 2020; Ritter et al., 2018) or functional space (Benjamin et al. 2019; Li & Hoiem| (017).
These methods primarily focus on stability and generally fail in mnrementl learning settings where task information is unavailable. Dynamic architecture approaches (Rusu et al., (016), expand the network to allocate distinct parameters for each task. While reducing forgetting, these methods lead to model size scaling linearly with the number of tasks and often require task identity at test time, limiting their applicability. Rehearsal-bert -based approaches draw inspiration from the brain's 's experience replay for knowledge consolidation (Ölafsdóttir et al. 2018). They store subset of data samples from previous tasks in a memory buffer, which are interleaved with new tasks samples to approximate the joint distribution. Rehearsal-basee approaches have proven to be more general and effective for various continual learning scenarios (Farquhar & Gal 2018).
The baseline Experience Replay (ER) (Riemer et al. 2018) interleaves the training of new task sam- plcs with previous task samples in memory. Several approaches have since been proposed to pro- vide additional learning cues to the model from its previous state. Dark Experience Replay (DER++)
2

---

Published as a conference paper at ICLR 2025
Semantic Aware Metric
SAA
dng
Posilive Choups
dog: Mist cal, deer deer: hirt cal, dog
Model
pato
Thiesholding
Running Prototypes
Cop
Dog
Dog
Dovt
Objjact Prototypes
piano outo bird eat dog door
Stream
Data
Memory Buffer
aulb
plane
Simionity
D00
Figure I: SARL employs activation sparsity to emulate brain-like sparse coding, representing cach object with class prototype derived from the mean representations of object samples. The seman- tic relationships are utilized to encourage new object prototypes to align with the class prototypes of similar objects and diverge from dissimilar ones. Additionally, SARL ensures model stability through prototype regularization, mitigating forgetting and enabling effective consolidation in- formation in lifelong learning. Darker shades represent higher values.
Buuzzga ct al. 2020) additionally stores the output logits and applies consistency loss to the mem- ory samples. ER-ACE (Caccia et al. 2022) uses an asymmetric cross-entropy loss, considering only new task logits, the incoming samples to reduce representation drift. Gradient Coreset Replay (Twwai et al. 2022) selects and maintains a core-set based on gradient approximation.
Recently, there has been shift towards multi me---mmmoorreeee CL approaches, inspired by the com- plementary learning systems theory in the brain (McClelland et al. 1995). CLSSER (Arani et al., 2022) simulates the interaction between fast and slow learning by using two semantic memories that aggregate model weights at different rates with an exponential moving average. COºL. (Cha et al. 2021 initially learns representations with a modified SupCon loss and subsequently trains classi- hier using samples from the last task and buffer data. Deep Retrieval and Imagination (DRI) Wang al. 2022a) leverages an embedding network and generative model to retrieve and generate imag- inary data. SCoMMER (Sarfraz et al.) 2023) enforces sparse coding for efficient representation learning and uses multiple memories, using the previous state of the model to reduce similarity drift. While these multi-memory approaches effectively reduce forgetting, they incur additional compu- tational and memory costs, which can be prohibitive some real-world settings. Therefore, we believe that it is crucial improve CL performance in single-model setting, which can later be extended to a dual-memory framework. Please see extended related work in Appendix
METHODOL.OGY
We begin with an overview of the biological mechanisms that inspire our approach, followed by details on how we mimic these mechanisms in DNNs and an outline of our overall formulation.
3.1 BIOLOGICAL MOTIVATION
The human brain excels at adapting to dynamic environments by transferring knowledge from previ- ously learned concepts to new ones, mitigating interference, and efficiently consolidating informa- tion. A critical component of this lifelong learning ability is the brain's capacity to exploit seman- tic relationships between objects, which guides both information encoding and memory formation [inder & Desai 2011). By establishing associations between related concepts, the brain orga- nizes information into semantic networks in which interconnected concepts share common features, enabling efficient retrieval, transfer, and consolidation of knowledge (Saxena et al., 2022). When encountering new objects, the brain rapidly integrates them into these networks, leveraging previ-
3

---

Published as a conference paper at ICLR 2025
Dense
Sparse
plane
0:87
0.80
0.77
0:87
0.90
0.86
0:82
auto
0.64
0.89
0.82
bird
0.07
cat
0.84
0.88
0.81
0.83
0.91
0.94
0.85
deer
0.85
0.84
0.85
0.82
0.86
0.91
0.89
0.91
dug
0.77
0.81
0.83
0.88
0.84
0.81
0.84
0.80
0.01
0.82
0.90
0.94
0:85
0.87
0.79
0.82
frog
0.76
0.79
0.84
0.84
0.80
0.01
0.74
0.77
0.79
0.78
0.81
0.88
0.89
0.85
0.85
0.79
0.77
0.80
horse
0.80
0.82
0.80
0.83
0.85
0.84
0.74
0.88
0.80
0.84
0.81
0.86
0.85
0.91
0.87
0.79
0.81
0.85
ship
0.67
0.89
0.82
0.81
0.82
0.80
0.77
0.83
0.66
0.88
8.
0.83
0.82
82
0.79
0.77
0.81
0.87
truck
0.02
0.91
0.01
0.80
0.79
0.01
0.79
0.80
0.06
0.06
0.91
0.04
0.85
0.05
0.82
0.80
0.05
0.07
plane
auto
bind
deer
dog
freg
horse
ship
track
plane
INNo
hird
deer
dog
trag
horse
ship
truck
Figure 2: Similarity matrices were computed using object representations from the model trained on the joint CIFAR-10 dataset, with dense and sparse activations, respectively.
ously acquired knowledge to facilitate learning and reduce relearning efforts. This process is further supported by the brain's use of spurse coding (Foldiak 2003), where only small subset of neurons activated. Objects with conceptual similarities activate overlapping neuron sets, hence efficiently encoding semantic relationships which can be leveraged for retrieval and learning.
Inspired by these cognitive mechanisms, we propose that incorporating sparsely activated neurons, which allow for the extraction of semantic structures, and leveraging these relationships for learning in DNNs, can significantly enhance their lifelong learning capabilities.
3.2 SEMANTIC-AWARY REPRESENTATION LEARNING
Building on these cognitive insights, we introduce Semantic-Aware Representation Learning (SARL), which adopts principled approach to extract the semantic relationships between objects encountered across sequential tasks and subsequently utilizes them to guide representation learning in cohesive manner. SARL draws inspiration from the brain's 's sparse coding approach and employs activation sparsity to facilitate semantic information encoding representations. Each object rep- resented by an object prototype, capturing the average features of its instances, which are saved in memory at the end of each task. By evaluating the similarity between these stored prototypes and the average features of new objects, SARL identifies sets of positive (similar) and negative (dissimilar) objects from both previous and current tasks. Leveraging these relationships, SARL applies a novel semantic-awir aware metric learning approach, guiding the model to bring representations of new objects closer to similar prototypes while distancing them from dissimilar ones. This mechanism enables the model reuse knowledge from similar objects, enhancing learning efficiency and reducing interfer- ence. To further promote stability, SARL implements prototype-basee regularization, encouraging the activations previously learned objects to remain consistent with their established prototypes and thus preventing drift over time. By integrating new information in structured manner, SARL fosters effective knowledge consolidation and maintains balance between plasticity and stability in lifelong learning. The following sections introduce the different components of SARL.
3.3 SPARSE CODING
To mimic sparse coding in the brain, we employ activation sparsity using k-winners-take-al (k- WTA) (Maass 2000), where only the K most active neurons in each layer are allowed to propagate to the next layer. For convolutional layers, we apply k-WTA locally across the channels at each spatial location rather than globally across the entire input and all channels. This localized approach ensures that the most relevant features are selected at each position, promoting specialization of filters. As result, the network can focus on distinct patterns at different spatial locations, leading to more efficient feature extraction, better semantic encoding, and reduced interference in lifelong learning scenarios.
4

---

Published as a conference paper at ICLR 2025
3.3.1 OBJECT PROTOTYPES
Building on the sparse coding mechanism through k- K-WTA, we extend this principle to our formula- tion of object prototypes. These prototypes are central to capturing and incorporating the intrinsic similarities between objects to guide representation learning. By capturing shared concepts through overlapping sparse activations, k-WTA offers an efficient and intuitive method to encode such re- lationships. Each object is represented by its object prototype, defined as the mean of all instance representations, strategy commonly employed in metric learning (Kaya & Bilge (019)). Specif- ically, we utilize the fz-normalized activations of the penultimate layer as object representations, with the prototype for each class e computed as:
O₂
D U(u T(=== c)04 No
1
(1)
where df represents the normalized activation of sample z; with label yi. and Ne is the number of instances in class c. This provides an efficient means of forming object prototypes that guide subsequent learning. At the end of each task, we calculate and store the object prototypes of all the new object categories. Hence, while learning task 4, we have access to the object prototypes for all the object categories learned until task
3.3.2 SEMANTIC-AWARE METRIC LEARNING
Having developed an efficient method for representing objects using prototypes that capture the essence of each category, we can now evaluate the similarity between objects across tasks. We use cosine similarity, which measures the cosine of the angle between two class prototypes, providing a scale-invariant measure independent of magnitude. This approach is particularly advantageous when comparing object prototypes, as it focuses on directional alignment between vectors rather than their magnitudes, effectively capturing semantic similarity in high-dimensional spaces. The similarity between the object categories and and the corresponding object prototypes O, and O given by:
O, sim 00, O,) |O ||O,1I
This provides us with an efficient approach to evaluating the similarity between object categories encountered across different tasks, which is subsequently used to inform learning. Figure 2 shows that the proposed approach can capture semantic similarities, and sparse activations further enhance the model's ability distinguish between similar and dissimilar objects. Section El in Appendix shows that sparse activations enhances the capacity of the model to capture semantic similarities between objects within animal and vehicle clusters as well as between the two clusters.
Specifically, for each new object category introduced in task f, we calculate intermediate object pro- totypes O using Equation and evaluate the similarity between each new object and other objects in the combined set of both newly introduced and previously learned object categories. We employ a warm-up stage for each new task to allow the model to learn meaningful representations before computing the intermediate object prototypes. For each new object category c, we use the degree of similarity between object prototypes to identify a set of similar object categories S.:
s-=(i O 2 T. Ai
Vi E CO:1}
(3)
where т, is the similarity threshold ranging from 0 to 1, and Cht represents the set of all object categories observed up to task t.
metric learning loss in subsequent epochs following the warm-up stage. For each new object category c, we encourage the mean & fy-normalized representations of its object instances in the batch, denoted o., be closer the object prototypes of objects in the set i.diiiiitin similarity. Simultaneously, they are pushed to be distant from dissimilar objects. Formally, the loss is given by:
We employ semantic
Σ S. 110 - 0z|
LsM= Σ
oE Ct. ie S. 10, Ta
5

---

Published as a conference paper at ICLR 2025
where C' is the set of new object categories task t. The semantic metric learning loss ables the model to leverage similarities between objects and acquire semantic-awerd aware representations. This, in turn, facilitates effective knowledge sharing and reusing while mitigating interference. Such enhancements augment the model's capacity to consolidate knowledge more cohesively.
3.3.3 REGULARIZATION ON OBJECT PROTOTYPES
To enhance stability and preserve the semantic structure within the acquired representations, we introduce regularization loss the object prototypes. This regularization is applied to buffer samples, penalizing the divergence of mean representations of buffered samples in the batch from their originally learned object prototypes. Formally, in each training batch, we sample from the memory buffer, Mr. calculate the & éz-normalized mean representations for each object category, o². and minimize the mean squared error between of and their corresponding object prototypes, O.. stored the prototype memory, Mo:
COp= Σ
O.l?
(5)
CE CO:-11
where C0:t-1 is the of previously learned object categories. By imposing this regularization, we encourage the model to maintain the semantic integrity of learned object categories across tasks, minimizing deviations in the representations of buffered samples from their learned prototypes. This not only fosters stability in the model's knowledge but also mitigates the risk of forgetting important semantic relationships established in earlier learning phases.
3.3.4 FUNCTIONAL REGULARIZATION
Functional regularization plays crucial role in helping models retain previous knowledge and ef- fectively consolidate new information by directly regulating changes in the model's input-output behavior rather than just its parameters. When applied to new data, encourages the model adapt while remaining within the functional vicinity of its previous state, thereby facilitating the smooth integration of new information. On old data, functional regularization helps maintain consistency with prior learned functions, minimizing forgetting by directly aligning the current model's 's outputs with those from earlier tasks. This approach stabilizes the learning process and allows the model to evolve while preserving essential knowledge.
Specifically, before training on a new task, we pass its samples through the model and save the output logits, denoted z", in memory. This allows to enforce functional consistency during training the new task without the need to maintain the previous model state in memory. To further promote consistency in model behavior and maintain semantic relationships, along with the data samples in a memory buffer. During the replay of these samples, similar to Buzzega al. (2020), apply consistency regularization loss the model's output. At each training step with new task samples x and y" along with their precomputed output logits z² from previous model state, we sample x², y², and z6 from the memory buffer and apply the following functional consistency loss:
also save the output logits
CFR=O||F(a") 8) 2 3)F(2' 2012
where F()) is the model's output parameterized by weights 0. and 3 control the strength of functional consistency on buffer samples and new task samples, respectively. This combination of regularization on object prototypes and regularization of consistency improves the stability of the model and preserves the semantic structure learned in previous tasks.
3.4 OVERALL FORMULATION
SARL involves training a model (.; on a non static data stream, D containing sequence of
i..dd tasks D² D To employ rehearsal, it maintains small episodic memory, M, using reservoir sampling /iiteer 1985). Additionally, it maintains a memory of object prototypes, M.. During each training iteration of task t, the model is fed with the training batch (x', v²) D' and a random batch from episodic memory (x", P..y" 2²)²") ~ M,. Task is trained using standard cross-entropy loss. For each subsequent task, we employ warm-up stage where train the model

---

Published as a conference paper at ICLR 2025
Table I: Comparison analysis of single- and dual-model CL methods across various CL settings. The baseline taken from the repective methods. We ran SCoMMER for Seq-Tinylmagehi using the hyperparameter setting approach provided by the authors. We report the average accuracy and std of 3 different seeds.
Seq- CeQ-CIFARIO
Beq-CIFARI00
Seq- Te--Tinlmgg
Buffer
Method
Class-IL
Task-IL
Class-IL
Task-IL
Class-IL
Task-IL -IL
JOINT
92.20=au5
98.31 +0.12
70.62+a04
86. 19:44
19
82.04
SGD
19.62
61.02
17.58=004
40.46=1.9
7.92+0.6
18.31 +0,8
ER
44.79 1189
91.19 10.94
21.401022
61.36
8.49
38. 18.17.2.00
FDR
30.91 =274
91.01 22.74
22.02=000s
61.72:+1.00
8. 8.70+a.19
40.36=2..
DER++
64.88
91.92.10.0
29. 19.60.z. 1.114
62.49 1.0.8
10.96
40.87
ER-ACE
62.08 1.44
35.17 1.17
37.4410064
25.29
11.25 10.54
44.17.1..0
GCR
14
90.8
33.69
64.24
13.05
42. +1.01
200
CO
65.57=137
93.43
31.90=a38
55.02=0a
13.88
42.37
CLS-ER
66.19
93,9010.0
43.80
73,49 11.0
23,47 1..800
49.6010.2
SCOMMER
69.19 19.1910.
93.201..10
40.251.00
69.39
16.30
49.37
SARL
70.97
95.72+t.36
48.96
78.91
28.95
71.56+
ER
57.74
93.61 10.21
28.02
68.23 10.16
9,9910.29
48.64
FDR
28.71 =3,23
93.2 +3.29
29. 19.19
69. 39.76:t..1
10.54
49.88+0071
DER+
72.70=136
93.88
41,40
70.6 +0.11
19.38
51.91
ER-ACE
68.45 1.78
40.67 10.06
30.14
24.81 10.63
17.73 1..55
49.99.1.51
GCR
74.69
94, 4..33
45.91 +1.30
71.64
19.66=068
52.99
500
CO
74.26=077
39. +0.39
39.21
62.98
20. 12.022
53.04+0.0
CLS-ER
75.22=071
94. 9..53
51.40=100
78. 12
31.03
60.41
SCOMMER
74.97
94.3610.88
49,63
75.49 10.43
22.60
58. 18.46
SARL
75.64=036
95.95+004
55.30=061
81.23
32.56.1123
70.94=043
with the combination of cross-entropy loss, Lcg on the training batch and buffer samples and the regularization losses. At the end of the warm-up stage when the model has learned meaningful representations, we evaluate the intermediate object references for each object category, in the current task using Equation T and identify a of corresponding similar object categories, S. using Equation to employ the semantic-aware metric learning loss in the subsequent training on the task. The overall loss for training the model is given by:
== 0), v') CCE(F(>² 0). ASMCSM AoPLop + LFR
where AsM and XOP are the hyperparameters that control the weight of each loss.
At the end of each task, we evaluate the object prototypes of the object categories introduced in the task using Eqn[ Additionally, to mitigate the biases of the batch norm statistics towards the current task (Pham et al.) (002),, we employ a simple yet effective technique of passing the model through all buffer samples to shift the batch norm statistics towards the approximate joint distribution.
4 EMPIRICAL EVALUATION
Settings. To evaluate the effectiveness of SARL, we consider various CL scenarios that each present unique challenges for lifelong learning models. In the class-ine flass-incrmmentll learning (Class- II.) scenario, with each new task, the model is introduced to a new set of classes and must learn to distinguish not only among the current task's classes but also across previously encountered classes. This setting tests the model's ability to build generalizable representations, consolidate knowledge, and transfer that knowledge to efficiently distinguish all classes seen so far. We only train models for the CLass-IL setting; for completion, (Task-IL) setting. In this context, the model has access to task labels during inference, enabling the utilization of task labels selectively choose the subset of output logits.
also provide task aaskiincrementll learning
While Class-IL focuses on knowledge accumulation across distinct tasks, it does not fully capture the complexity of real-world scenarios, where tasks often lack clear boundaries, and classes can

---

Published as a conference paper at ICLR 2025
Table 2: Comparison analysis of single- and dual-model CL. methods GCII.CIFAR- 100 dataset.
Dist.
Buffer
JOINT
SGD
ER
DER++
CLS-ER
SCoMMER
SARL
200
16, 40
18. 18.84
25.06eas:
30.84
36.97
Uniform
58.36-102
12.67
500
28.2 20000
32,92a074
36.34
36.87
39.03
200
19.27
26.94-1m7
28.54.087
29.08-031
35.26
Longtail
56.94
22.88
500
20.30
25.82.088
28.63+004
35.20
35.47-1.30
During Training
Test Time
plone
0.60
0.92
0.81
0.91
0.76
89
0.66
0:91
0.58
0.74
0.92
0.90
0.89
0.07
0.82
0.82
0.09
0.71
auto
0.85
0.85
0.74
0.77
0.92
bird
0.92
0.66
0.82
0.93
0.82
87
0.75
0:89
0.63
92
0.74
0.95
0:94
0.92
0.90
0.86
0.63
0.71
cat
0.81
0.76
0.82
0.87
0.90
0. 85
0.80
0:84
0.70
0.
0.77
0.95
0.92
0.97
0.89
0.89
0.79
0.77
deer
0.93
0.89
0.90
dog
0.76
0.85
0.62
0.90
0.89
0:83
0.90
0.80
0.78
0.87
0.75
0:92
0.97
0.91
0.84
0.92
0.73
0.75
frog
0.92
horse
0.66
0.85
0.75
0.
0.80
0.82
0.90
0. 66
0.66
0.83
0.82
0.74
0.86
0.89
0.88
0.92
0.73
0.64
0.77
ship
0.91
0.60
0.89
0. 84
0.80
0. 92
0.66
0.53
0.80
0.59
0.83
0.79
0.79
0.73
0.77
6.
0.52
truck
0.8
0.63
0.78
0.53
0.27
0.64
0.52
plane
auto
bird
cat
deer
dog
frog
horse
ship
truck
plane
auto
bird
cat
deer
dog
frog
horse
ship
truck
Figure 3: Cosine similarities between the object prototypes computed at the end of each training
task on Seq-CIFARIO with 200 buffer size and the similarity structure of the final model on test set.
reappear with varying distributions. To address this, we also evaluate SARL in the generalized class- incremental learning (GCIL) setting. GCIL introduces a more dynamic and realistic environment where classes reoccur across tasks, and the number of classes and their sample sizes can vary. This scenario tests the model's 's ability to deal with class imbalance, sample efficiency, and the continual integration of knowledge from overlapping or reappearing classes.
Additionally, we consider online continual learning (Online CL), a highly challenging setting where the model is exposed to each data instance only once, requiring it to learn and adapt dynamically without revisiting previous data. Details of the experimental setup and datasets are provided in Appendix A
Results. We benchmarked SARL's performance against established rehearsal-based approaches across varying CL settings and dataset complexities. Table I demonstrates consistent performance improvements across all settings. Notably, under the low buffer regime, where the model has to retain knowledge while having access to very limited samples from previous tasks (approximately two samples per class for Seq-CIFAR-100 and one sample per class for Tegg--eee-m SARL effectively retains previously acquired knowledge, demonstrating its ability to learn new tasks while remaining the functional vicinity of earlier tasks despite the limited buffer size and consolidating knowledge as it learns new tasks. Interestingly, as the complexity of the dataset increases with the number of classes with high semantic similarity, the performance gains are more pronounced, sug- gesting that SARL is able to exploit the semantic structure and utilize it to consolidate knowledge.
Table 2 highlights the advantage of SARL the challenging GCIL setting, where the model must update its knowledge across multiple occurrences of the same object with varying sample sizes, while addressing class imbalance and task complexity. SARL's ability to update object prototypes when re-encountering a class ensures efficient use of new samples without penalizing for deviations from earlier suboptimal representations. SARL shows remarkable resilience to these challenges and provides additional credence to the benefits of semantic-aware representation learning.
These results are particularly impressive considering that SARL uses only a single model, while many sfate-of-the-ar methods rely on multiple models. For instance, CLS-ER uses two additional models, and SCoMMER incorporates one. Although these methods claim to use a single model for inference, the distinction between training and inference in lifelong learning is blurred, the model
8

---

Published as a conference paper at ICLR 2025
Table 3: Performance comparison across Seq- CANQ--CFRRIOOO and Seq-CIFAR in the online CL setting. Baseline results are taken from Wu et al. (2024). We report the mean and std of three runs.
Seq- CIQ-CIFARIO
Seq-CIFAR 100
Method
AAA
Ace
AAA
Acc
SGD
34.85
16. 26
11.632038
5.27=908
ER
55.53
43.83 23.83
23.19408
16.07
DER
45.85
29.87 1:295
13.35
6.12
DER+
64.22.10.0
52.29
19.88
11.79
CLSER
63.02+11.44
52,80+166
25.464005
17.88
OCM
66. +0.95
53.39 11000
22.54
14.40
ER-OBC
65.82.101
54.85
25.541025
17.21
On-EWC
+8.44
17.12
11.8 +0.81
5,88 +021
IS
37.33++023
17.39
12.32+022
5,20
La-MAML
42.98
33.43
12.55
11.78
VR-MCL 66.97
56.48
27.01
19,49
SARL
66.85
57.21 +0.27
31.66.1..
24.39
must continuously learn from the environment. This can make the overhead of maintaining multiple memories prohibitive weal-world applications, especially for resource-constrained systems like embedded devices. The ability of SARL achieve superior results with single model further underscores its practicality and efficiency for lifelong learning.
Additionally, while SARL is not explicitly designed for Online CL. adapted it to this setting to assess its versatility and broader applicability. By maintaining running feature sums and sample counts to dynamically update object prototypes, SARL effectively learns in an online manner. We introduce a brief warm-up phase during the first 10 iterations of each task to stabilize feature ac- cumulation before evaluating prototypes. Despite this adaptation being secondary focus, SARL-Online demonstrates strong performance. As shown in Table 3 it achieves notable improvements across all datasets, particularly on Seq.CIFA where it significantly boosts both AAA and Ace. These results further reinforce the effectiveness of semantic- aware representation learning, highlighting SARL's adaptability to different continual learning paradigms without requir- ing extensive modifications. Our ablation study in Appendix B.I shows that each component of SARL contributes to the performance gains.
our primary
ANALYSIS
Here, we delve deeper into what enables performance improvements in SARL. We compare with the baseline ER and the state-of-the-ar in single-model (DER++) and multiple-memory (SCoMMER).
5.1 SEMANTIC STRUCTURE
At the core of SARL is semantic-aware representation learning. We investigate whether SARL can learn and retain meaningful semantic structure during sequential task training. We evaluate the similarities between object prototypes calculated stored in memory during training that are evaluated the end of each task for new object categories and compared them with the similarity structure of the trained model on test data. Figure 3 shows that SARL not only retains the similarity structure but also updates it by consolidating new knowledge. Notably, the class prototypes from early tasks, where the model was trained to distinguish only between two classes, show strong similarities even with other classes, indicating that the initial representations were not yet fully optimized for all classes. However, as the model encounters more tasks, the semantic relationships across classes become more refined and meaningful. For instance, classes such as 'cat', 'dog' and 'horse' exhibit higher similarity values post-training, reflecting their semantic coherence, while unrelated classes such as "plane" and 'truck' maintain lower similarities. The semantic-an anwre metric learning loss, combined with object prototype regularization, guides training to enforce and maintain this semantic

---

Published as a conference paper at ICLR 2025
(a) Stability Plasticity Tradeoff
(b) Recency Bias
ER
Ttsk
80
DER- SCOMMER SARL
Toxt
70
Tosk
Probability
0.6
50
40
90
Task
20
Plasticity
Stability
Tradeoff
ER
DER++
SCoMMER
SARL
Figure 4: (a) Comparison of the stability and plasticity of the models; and (b) average probability of predicting each task at the end of training for models trained on Seq-CIFAR 100 with 200 buffer size. Figure in Appendix shows the task-wise performance of the different models.
structure, allowing the model to incorporate new knowledge without losing the relationships between previously learned classes.
5.2 STABILITY-PLASTICNI TRADE-OFF
Continual learning requires balancing plasticity (learning new tasks) and stability (retaining past knowledge). To evaluate this trade-off, we use the metric from (Sarfraz al. 2022). where plasticity (P) is the average performance on newly learned tasks, and stability (S) is the retention of past tasks after learning task 1. The trade-off is given by (2xPxS)
+ S) Figure 4a) shows that SARL, significantly enhances stability without sacrificing plasticity, achieving a 362% gain over ER and 127% over DER++. In contrast, SCOMMER, which employs an additional exponentially averaged model for knowledge accumulation, improves stability but severely degrades plasticity over longer task sequences, as reflected in its lower final-task accuracy (47.7% vs. 73.60% for SARL, Figure 5 in Appendix). This highlights the advantage of SARL's semantic-aware sparse learning in fostering a better stability-plasticity balance.
Sequential task learning also introduces recency bias, favoring recently objects (Hou al., 2019). Figure 4b) shows that SARL significantly reduces this bias compared to ER and DER+ While SCOMMER's EMA model produces more uniform predictions, it exhibits a declining prob- ability for the last task, which may impact long-tern performance. Notably. SARL assigns higher probabilities to earlier tasks, suggesting that its regularization mechanisms effectively preserve the initial function space, mitigating forgetting.
These analyses collectively show that SARL is able effectively evaluate semantic similarities between objects across tasks and utilize them to enforce semantic structure in the representation space. This creates synergy between plasticity and stability, leading to a better balance between the two and reduces forgetting and recency bias.
CONCLUSION
Inspired by how the brain uses sparse coding and semantic relationships between objects to in- form memory formation and retrieval, we proposed semantic-aware representation learning (SARL), which employs sparse activations to create semantically rich object prototypes which can effectively capture the inherent similarities between objects. The semantic structure is leveraged align new object representations with similar objects from previous tasks while promoting separation from dis- similar ones. Our approach facilitates feature reuse while reducing interfering, creating synergy between the plasticity and stability of the model. Our empirical evaluation in challenging CL sce- narios demonstrates SARL's effectiveness. By enforcing semantic structure on the representations of objects across tasks, SARL balances model plasticity and stability, reduces task recency bias, and mitigates forgetting. Our findings present a compelling case for incorporating semantic- aware representation learning similar to the brain's method.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general con- tinual learning method based on complementary learning system. In International Confer- ence on Learning Representations, 2022. htthttp:: httts:://pperevieee net. forrmmrrumm// id=
uxxFFDwwrEYY
Caterina Artuso, Francesco Bossi, Carmen Belacchi, and Paola Palladino. Effects of semantic rela- tionship and preactivation on memory updating. Cognitive Processing. 23(3):407-423 422, 2022.
Ari Benjamin, David Rolnick, and Konrad Kording. Measuring and regularizing networks in func-
tion space. In International Conference on Learning Representations, 2019. URL https:
7/openzeview net
Jeffrey R Binder and Rutvik H Desai. The neurobiology of semantic memory. Trends in cognitive sciences, (552755557 2011.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark perience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33:15920-15930 2020.
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. In Interna- tional Conference on Learning Representations.
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pincau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. Interna- tional Conference Learning Representations, 2022. URL
Eorum??.d=NMaayyO0E3
Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co21: Contrastive continual learning. In Proceedings of the IEEE/CVF International conference on computer vision, pp. 9516-9525, 2021.
Mehrdad Farajtabar, Navid Nzizan, Alex Mott, and Ang Li. Orthogonal gradient descent for contin- ual learning. In International Conference on Artificial Intelligence and Statistics, pp. 3762-3773.
PMLR, 2020.
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning. arXiv preprint 1rX1v:/805.09733, 2018
Peter Foldiak. Sparse coding in the primate cortex. The handbook of brain theory and neural networks, 2003.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 831-839, 2019.
Mahmut Kaya and Hasan Şakir Bilge. Deep metric learning: A survey. Symmetry, 11(9):1066, 2019.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Haoran Li, Jingfeng Wu, and Vladimir Braverman. Fixed design analysis of regularization-base continual learning. In Conference on Lifelong Learning Agents, pp. 513-533. PMLR, 2023.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pallern analysis and machine intelligence, 40(1 40(12):2935-2947, 2017.
Yan-Shuo Liang and Wu-Jum Li. Adaptive plasticity improvement for continual learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7816 2023.
11

---

Published as a conference paper at ICLR 2025
Wolfgang Maass. On the computational power of winner-take-all. Neural computation, 12(11): 2519-253: 2000.
James L McClelland, Bruce L McNaughton, and Randall C C'RReilly. Why there are complementary learning systems the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The tigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in psychology, 4:504, 2013.
dilemma: Inves-
Fei Mi, Lingjing Kong, Tao Lin, Kaicheng Yu, and Boi Faltings. Generalized class incremental learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recogni- tion Workshops, pp. 240-241, 2020.
Xing Nie, Shixiong Xu, Xiyan Liu, Gaofeng Meng, Chunlei Huo, and Shiming Xiang. Bilateral memory consolidation for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2023.
H Freyja Ólafsdóttir, Daniel Bush, and Caswell Barry. The role of hippocampal replay in memory and planning. Current Biology, 28(1) 28(1):R37-R50 2018.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
Quang Pham, Chenghao Liu, and Steven HOL. Continual normalization: Rethinking batch normal- ization for online continual learning. In International Conference on Learning Representations, 2022. URL
Biqing Qi, Xinquan Chen, Junqi Gao, Dong Li, Jianxing Liu, Ligang Wu, and Bowen Zhou. Inter- active continual learning: Fast and slow thinking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12882-12892, 2024.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning learn without forgetting by maximizing transfer and minimizing interfer- ence. arXiv preprint arXiv: 1910, 2018.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. Advances in Neural Information Processing Systems, pp- 3738-3748, 2018
Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatriek, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 16Xx:11606.04611,, 2016.
Fahad Sarfraz, Flahe Arani, and Bahram Zonooz. Synergy between synaptic consolidation and experience replay for general continual learning. In Conference Lifelong Learning Agents, pp- 920-936. PMLR, 2022.
Fahad Sarfraz, Elahe Arani, and Bahram Zonooz. Sparse coding in dual memory system for lifelong learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 9714-9722, 2023.
Rajat Saxena, Justin L Shobe, and Bruce McNaughton. Learning in deep neural networks and brains with similarity similarity-eiihteed interleaved learning. Proceedings of the National Academy of Sciences, 119(27)=e21 15229119. 19, 2022.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for feo-shot learning. Ad- vances in neural information processing systems, 30, 2017.
12

---

Published as a conference paper at ICLR 2025
Rishabh Tiwari, Krishnateja Killamsetty, Rishabh Iyer, and Pradeep Shenoy. Ger: Gradient coreset based replay buffer selection for continual learning. In Proceedings of the IEEE/CVE Conference Computer Viston and Pattern Recognition, pp. 99-108, 2022.
Jeffrey S Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS), 11(1)337-57, 1985.
Zhen Wang, Liu Liu, Yiqun Duan, and Dacheng Tao. Continual learning through retrieval and imagination. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp- 8594-8602, 2022a.
Zifeng Wang. Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. arXiv preprint arXiv:2204. 04799, 2022b.
Zifeng Wang, Zizhao Zhang, Chen-Y Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Pro- ceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 149, 2022c.
Yichen Wu, Long-Kat Kai Huang, Renzhen Wang, Deyu Meng, and Ying Wei. Meta continual learn- ing revisited: Implicitly enhancing online hessian approximation via variance reduction. In The Twelfth International Conference Learning Representations, 2024.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. Proceedings of machine learning research, 70:3987, 2017.
13

---

Published as a conference paper at ICLR 2025
A EXPERIMENTAL SETUP
To ensure fair comparisons between continual learning (CL) methods and isolate algorithmic im- provements from training regimen variability, we adopt consistent experimental conditions across a ResNet- 18 architecture trained with SGD
all models. In line with Bitzeaazeea et al. (2020), optimizer and batch size of 32 for both task data and memory buffer. We use an initial learning rate (Ir) of 0.03 with multistep decay scheduler with factor 0.1. Standard data augmentations random cropping and horizontal flipping- are applied. For all tasks, we include a warm-up up stage consisting of epochs, during which the model trains solely on the task-specific data without utilizing memory buffer samples. This allows the model to initialize a strong representation for each task before in- tegrating knowledge from past tasks. For Seq- AEQ-CIFARIO we trained for 20 epochs with decay steps For Seq- Seq-CIFARII 10, the model is trained for 20 epochs with Ir decay steps at 15th epoch, for Seq- CIFARI 100 and GCIL-CIFAR 100, we extend the training to epochs with Ir decay steps at 35 and 45. For the Tiny ImageNet dataset, we further increase the number of epochs to 100 with Ir decay steps at and 90.
A.1 SPARSE BACKBONE:
In all experiments, incorporate activation sparsity by replacing ReLU activations with the k- Winner- Take-All (kWTA) activation function. KWTA, for each layer, only the top k% of activa- tions at each spatial location allowed to propagate, while the rest are set zero. This localized sparsity mechanism ensures that the most relevant features are selected at each spatial location, pro- moting filter specialization and reducing interference across tasks. The level of sparsity (i.e., k% is kept fixed across all layers and experiments. Beyond this change to the activation function, no other modifications are made the backbone architecture.
A.2 EVALUATION SETTINGS:
For the Class-IL setting, we follow established baselines and evaluate on three datasets: sequential CIFAR-10 10 (Seq-CIFAR1O) (Krizhevsky et al. 2009), where 10 classes are split into 5 disjoint tasks with 2 classes per task; sequential CIFAR-I00 (Seq-CIFAR 1--CCIFAROO)) 100) (Krizhevsky et al. (2002,00) splitting 100 classes into tasks with 20 classes each; and sequential TinylmageNet (Le & Yang. 2015) SSeg-TnnlmmageNet where 200 classes divided into 10 tasks of 20 classes each.
For the Generalized Class-IL (GCIL) setting (Mi et al. 2020 probabilistic modeling used to randomly vary three task characteristies: the number of classes, the specific classes included, and the sample sizes. As in Sarfraz et al. (2023), we apply this GCIL setting to CIFAR-100, using 20 tasks with 1,000 samples per task and maximum of 50 classes per task. To separate the effects of class imbalance from the model's ' ability to learn from recurring classes across tasks, we evaluate the model on both uniform and long-tailed data distributions. For consistency, the GCIL dataset seed is fixed at 1993 for all experiments, ensuring fair comparison across methods.
For Online-CL, we follow previous works Caccia et al.) Wu al. (2024) and evaluate our method under the single-head setting, For this setting, we also report average anytime accuracy (AAA) Caccia al., which measures the model's performance throughout the training stream. Let AA; denote the test average accuracy after training on task Ty. Then, the metrics are defined as:
N AA
1
AAA
Acc = AAN
N
where N is the total number of tasks. Following (w et al. 2024), we evaluate our method on the Seq-CIFARIO and S0q-CIFAR100.. Seq-CIFARIO comprises 5 tasks, each containing classes, Seq-CIFAR 100 consists of 10 tasks with 10 classes each.
B
ADDITIONAL RESULTS
B.1 ABLATION STUDY
14

---

Published as a conference paper at ICLR 2025
ER
DER++
SCOMMER
SARL
Sequential Training
74.70
82:35
72:35
$1.50
10.10
60.45
17.55
86.30
58.25
70:95
60.35
78.90
2.25
9.95
79 79:50
6.30
36.95
84.00
50.80
55
57.45
42
59.85
77.99
12
3.00
4.10
14:80
79.30
6.20
23.15
27.75
80.30
45.00
30
44 05
55 10
35,20
49.0
58.95
75.20
13
0.95
2.45
5.90
7.80
85.
35
1.03
7.30
16:35
13.45
85.05
35.20
00
48
42 70
29.00
41.20
69.00
57.25
73.00
4
Task Performance
Task Performance
Task Performance
Task Performance
Figure 5: The task-wise performance (x-axis) of the different models assessed after training on each task (y-axis) in Seq-CIFARI00 with buffer size of 200.
Table Selected hyperparameters for SARL. For all our experiments, we use AsM==0.11, Tz=0.8,
and 8=1.
Dataset
Buffer
kw%
XO
a
200
0.8
0.5
0.2
Seq- CIFACI
500
0.8
0.5
0.2
200
0.9
0.5
0.5
SCq-CIFAR
500
0.9
0.5
0.2
200
0.9
0.2
0.2
Seq- Teq-TinllmageNet
500
0.9
0.5
0.5
200
0.7
0.2
0.2
GCIL-Uniform
500
0.7
0.2
0.2
200
0.7
0.2
0.5
GCIL-Longtai
500
0.7
0.2
0.5
Here, systematically assess the contri- bution of each key component of SARL by incrementally adding them on top of the baseline ER. Table 5 reveals the im- portance of each element in improving model performance. Firstly, the inclu- sion of sparse coding via activation spar- sity alone yields substantial performance boost. This underscores the importance activation sparsity in mitigating task inter- ference by allowing only the most salient neurons to remain active, effectively con-
Table 5: Impact of adding distinct elements of SARL on Seq-CIFAR10 performance with 200 buffer size.
Sparsity
Lop
LsM
CFR
Accuracy
X
X
X
44.79
X
X
58.234171
X
X
62.5310.t
X
V
61,601058
/
/
X
63.834ac0
V
70.97
trolling task-specific noise and enhancing generalization. Next, we observe the separate effects of adding object prototype loss (Cop) and semantie-aware metric learning (Csar) top of sparsity. Object prototype loss introduces regularization, stabilizing the model by maintaining the average representations of objects, which further improves accuracy 63.01%. Semantic-aware metric learning. on the other hand, enhances forward transfer by leveraging the semantic similarities be- tween objects while reducing interference. Notably, when both losses are applied together, the synergy between them becomes evident. The object prototype loss ensures stable representations of previously learned object, which in turn helps semantic-aware metric learning enforce a consistent semantic structure throughout training. Finally, adding forward regularization (L R) to the mix en- courages the model to stay within the functional vicinity of previously learned states as new tasks are introduced. This final combination of all components achieves the best performance, highlighting the cohesive benefits of SARL's components working together to reduce interference and improve knowledge retention.
15

---

Published as a conference paper at ICLR 2025
B.2 RESULTS ON DIFFERENT BACKBONE
To further test the versatility of our approach, we evaluate SARL on the VIT Small backbone and VGG. It is well known that Vision Transformers (ViTs) struggle with smaller datasets due their inherent architectural requirements, which demand pretraining on large-scale datasets to achieve performance comparable to convolutional networks like ResNet. This limitation particularly ev- ident in continual learning settings with smaller datasets, leading to noticeable performance gap between ViT and ResNet backbones.
Despite this inherent challenge, shows that SARL provides considerable performance gains over ER across all evaluation settings. These results highlight the robustness and adaptability of SARL, even in scenarios where the backbone architecture is less suited for the dataset size. Ad- ditionally, SARL's performance on the VGG backbone is particularly noteworthy. The VGG back- bone, which lacks the inductive bias of residual connections and is less efficient compared to modern architectures, still benefits significantly from SARL. Specifically, SARL achieves state-of-the-ar re- sults in both Class-IL and Task-IL settings with VGG. This demonstrates SARL's ability to enhance learning effectiveness even in scenarios where the backbone architecture itself has inherent limita- tions.
Importantly. it should be noted that no hyperparameter tuning was performed specifically for these backbones. Instead, we used the same hyperparameters optimized for the ResNet backbone. This suggests that further improvements in performance are possible by tailoring the hyperparameters to better suit the backbone architecture and adapting the functional regularization. The significant performance improvements observed with SARL validate its capability to enhance learning effec- tiveness across diverse backbone architectures, emphasizing its potential as a versatile and robust continual learning method.
Table 6: Performance comparison on ViT-Small and VGG backbones. We report the mean and std of three runs.
Seq- CIF-CIAR11
Seq-CIFAR100
Backhone
Buffer
Method
Class-IL
Task-IL
Class-IL
Task-IL
ER
21.92
69.6544.05
11.90+1.7
19..0
200
SARL
26.04
82.74
19.08
44.82++446
ViT-Small
ER
24.41
12.30
1
31,08114.25
500
SARL
32.94
83.25 1:36
22.00 +0:79
47.68
ER
40.63 80.50.1105
18.71
40.53
200
SARL
63.06
91.07
26.47
51.7310.8
VGG
ER
54.09
86.83+oss
23.01 +0.66
50.27++091
500
SARI
66. 20.70
92.37
33.82.t0.55
58.6940.0
C ADDITIONAL METRICS
We provide additional metrics for all our experiments to comprehensively evaluate SARL's perfor- mance. Since the number of test samples per task is uniform across all tasks, the final accuracy, calculated as the total number of correctly classified samples across all tasks divided by the total number of samples, is equivalent the average accuracy, which is the mean accuracy across indi- vidual tasks at the end of training. Table provides additional metrics such as forgetting, stability, and plasticity, which provide deeper insights into model behavior. Forgetting measures the model's ability to retain knowledge from previous tasks by quantifying the average drop in accuracy of task at the end of continual learning training compared to its accuracy when first learned, where lower forgetting indicates better knowledge preservation. Stability (S) reflects the average accuracy on previously learned tasks at the end of training, demonstrating how well the model performs on earlier tasks. Plasticity (P), on the other hand, evaluates the model's 's ability to effectively learn new tasks, measured as the average accuracy of tasks when they are initially trained. Trade-off is mea- sured by (2 * & P) (P + S) and measure how well the method keeps a balance between the stability
16

---

Published as a conference paper at ICLR 2025
and plasticity of the model. Together, these metrics offer holistic view of SARL's performance, capturing its strengths and trade-offs diverse continual learning scenarios.
Note that the vast majority of baselines do not provide these metrics and hence cannot compare the basclines across these metrics unfortunately. In addition to the comparison on Average Accuracy in Table [ compare the forgetting and additional metries for the baselines in Table (where available). Note that DER+ is the strongest single model baseline and SARL considerable reduces forgetting by considerable margin.
Table 7: Performance metrics across datasets and buffer sizes.
Dataset
Buffer
Accuracy
Forgetting
Stability
Plasticity
Trade-off
Seq. CIFARIO
200
70.97
14.83
6..72=1.05
82.6
74, +4.75
500
75.64
8.75
73.39
85.31 4092
78.90
Seq-CIFAR 100
200
48.96 2n.55
33.8220.56
42.1220.91
76.45=1101
55, 14
500
55.30
26.78
50.21
76.73+0.2
60.69
Teq-TINYIMG
200
28.95
42.08
23.99
66.82+1.52
35.27
500
32.5
27.34
29.88
57.1611.73
39.24
Table 8: Results of different methods on Seq-CIFARIO and Seq- Seq-CIFAR100 datasets with varying
buffer sizes.
Dataset
Buffer
Method
Accuracy
Forgetting
Stability
Plasticity
Trade-off
ER
50.36_241
57.60
38.40..0
96,441042
54,88 12:8
DER++
65.74_1x1
31.75
18.731253
91.13100s
71,401176
200
CLSER
65.92_075
28.74_1.4
67.70
81.17.145
11.8
SCoMMER
66.80
1.33
62.2611.9
84.72
71.7610788
SARL
70.97 0.447
14.83
67.72.1.0
82.64.1035
74.4410.9
Seq-CIFAR I0
ER
62.70=:.04
41.92 1.95
53.92
96.24 10:8
69. 19.1111004
DER+
70.10
26.79 1.05
63.70
91.53
75.11
500
CLSER
75.16-036
22.56=104
73.40
78.03
75.64
SCOMMER
74.25 0044
20.13
70.62
87,99 +
78.36+037
SARL
75.64 0.33
8.75
73.39
85.31 +:99
78.90
ER
21.73
76.13=029
5.29
82.63 +021
9, 9,94+0.36
DER++
30.68=135
65.10=1.7
17.19
82.77
28.44
200
CLSER
43.80
36.33=144
45.75
50.47
47.99
SCoMMER
40.74
42.20=1113
35.98+0.00
63.30+aa2
45.80+1.76
SARL 48.96
33.82=0.5
76.45
55.14
Scq-CIFAR 100
ER
28.03-106
67.08=11
13.42:11.0
81.69
23.03
DER++
42.03=2m
48.90
32.01 #2.74
81.15
45.87
500
CLSER
51.68=099
30.74=a25
52.18=027
58.07
54.95
SCoMMER
50. 1=031
32.98
46.40
66.93
54.79
SARL
55.30=a41
26.78
10.21
76.73
60.69
D HYPERPARAMETER TUNING
During hyperparameter tuning, we used a small validation set to adjust values for kw%, Aop, and the coefficient for LFR. For all experiments, we set т. 00.,, 3 and AsM = 0.01. The value of kw% was selected from the set (0.7, 0.8, 0.9], while Xop and a, were chosen from (0.2, 0.5, presented in Table 4 It is important to note that did not
1]. The final selected parameters perform an exhaustive hyperparameter search, so better configurations may exist. Furthermore, we observed synergistic relationship among the parameters, and the model demonstrated robustness to range of values, which significantly eased the tuning process.
To further illustrate this, the results in Table demonstrate that SARL is not overly sensitive to specific hyperparameter values, as multiple configurations yield consistent performance improve- ments. For instance, across different values of a. Xor and kw% the method achieves robust
17

---

Published as a conference paper at ICLR 2025
Class-IL and Task-IL accuracies Seq-CIFARIO, with minor variations in performance. While certain configurations (e.g., a = 0.2, AOp = 0.5, kw% = 0.8 achieve the highest Class-IL accuracy (70.97%), other values (e.g., kw% 0.9, 0.2, Aor 0.5 provide similar results (70.08%). We present results for all the grid values explored during hyperparameter tuning, demonstrating that SARL achieves consistent performance improvements across various configurations without relying heavily on precise hyperparameter adjustments. This reliability underscores SARL's practicality and suitability for wide range of continual learning applications
Table 9: Sensitivity to hyperparameters on the performance SARL. For all the experiments, use learning rate 0.1, 0.999, Aop 0.5, ASA 0.01, AÇR = 0.15 and T = 0.8.
seq-cifarl0 10
kw
AP
Class-IL.
Task-IL
0.2
70.73
95.7710.16
0.2
0.5
70.97 20.47
95.72+0.36
1
70.57
95.71 +0.0
0.2
68.53
95.231007
0.8
0.5
0.5
69.40+1.21
95.2 +-.19
1
68.79
95.14+-000
0.2
65.30=1.40
93.73
0.5
64.59 20.22
93.72=042
63.86
93.19
0.2
70.05
95.49
0.2
0.5
70.08=1.44
95.63
69. 29.07
95.521001
0.2
68. 22+290
95.13
0.9
0.5
0.5
66.66
94.92+031
65. $5.49 20.08
94.48
0.2
63.9922.3
93.341091
0.5
64.04
43.00
1
63.80
92.87 1:33
Table 10: Memory and performance comparison of SARL and baselines on Seq-CIFAR100, #M stands for number of models, #FWD for number of forward passes, and [B] for buffer size.
Method
#M
#FWD (Tasks)
#FWD
[B]
Total Memory
Memory Ratio
Avg. Ace (%)
Avg. Acc
(Buffer)
(MB)
(MB)
(vs ER)
/Memory Ratio
ER
1
1
5.86
48.52
1,0000
28.02
28.02
DER+
1
2
6.05
48.71
1.0039
41.40
41.24
CLS-ER
3
3
5.86
133.84
2.7584
$1.40
18.63
SCOMMER
2
2
5.86
91.18
1.8792
49.63
26.41
SARI.
1
8.15
50.09
1,0323
55.30
53.57
E SEMANTIC SIMILARITY BETWEEN CLUSTERS
To further strengthen our analysis in Figure 2 which provides similarity matrices for dense and sparse activations, we defined two clusters within the CIFAR-10 classes: the Animal Cluster, comprising 'bird,' "cat'' "deer,' 'dog.' 'frog,' and 'horse,' and the Vehicle Cluster, consisting of 'airplane,' 'au- tomobile," 'ship,' and 'truck.' The comparison of inter- and intra-cluster similarities is visualized in Figure and the results indicate that sparse activations exhibit consistently higher average similar- ity within clusters compared to dense activations. For the Animal Cluster, sparse activations achieve average similarity of 0.8873 compared to 0.8367 for dense activations, Similarly, for the Vehicle Cluster, sparse activations achieve 0.8694 compared to 0,8684 for dense activations. This suggests that sparse activations better group objects within the same semantic category, capturing seman-
18

---

Published as a conference paper at ICLR 2025
Dense Activations Sparse Activations
Average Similarity
Within Animal Cluster
Within Vehicle Cluster
Between Clusters
Figure 6: Inter- and Intra-Cluster Similarities Across Animal and Vehicle Clusters in CIFAR-I0 for Dense and Sparse Activations
tic cohesion more effectively. By activating only the most relevant neurons, sparse representations reduce noise and focus on features critical for semantic grouping.
Sparse activations also demonstrate slightly higher average similarity between clusters, with 0.8295 compared to 0.8068 for dense activations. However, this inter-cluster similarity is signif- icantly lower than the intra-cluster similarity, aligning with the expected semantic distinction be- tween clusters. Notably, sparse representations encode inter-cluster relationships more semantically rather than merely enforcing hard separations. For instance, airplane and bird, which share abstract features like wings and the ability to fly, exhibit higher similarity of 0.90 with sparse activations compared to 0.87 with dense activations. In contrast, ship and horse, which are semantically dis- similar, show similarity of 0.81 with sparse activations compared to 0.83 with dense activations. These findings reinforce our hypothesis that sparse activations can enhance the model's capability to capture semantic structure across objects.
F VISUALIZATION OF REPRESENTATION SPACE
To further analyze the representations and evaluate how well SARL captures and enforces tie structure among objects, we project the object prototypes into two-dimensional space using t-SNE and compare SARL. with ER Seq-CIFAR10 with buffer size of 200 (Figure 7 The t-SNE visualizations reveal clear differences the representational quality of SARL compared to ER. In the left panel (ER), object prototypes exhibit lack of structured clustering, with seman- tically similar classes, such as animals ("bird," "frog," " "deer"), being dispersed and overlapping with other categories. This indicates that ER struggles effectively group related objects sep- arate distinct semantic categories in the representation space. In contrast, the right panel (SARL) demonstrates significant improvements, forming compact and cohesive clusters for semantically similar objects. For example, animals are tightly grouped together, and SARL effectively captures the semantic similarity between "bird" and "airplane," a relationship that ER fails to encode. This structured organization of prototypes highlights SARL's ability to leverage semantic relationships to guide representation learning, aligning similar classes while maintaining clear distinctions between unrelated ones. These findings underscore the effectiveness of SARL in creating more meaningful, semantically aware, and structured representation space.
G EXTENDED RELATED WORKS
Continual learning has seen significant advancements in recent years, with diverse approaches tar- geting the challenges of catastrophic forgetting, model stability, and effective knowledge transfer across tasks. In this section, we extend our related works section and position our SARL in the broader context of these developments.
19

---

Published as a conference paper at ICLR 2025
ER
SARL
150
ship
Classes
airplane
airptone
utomobile rucc
autommbiie hird
100
chip
deet
pirplane
50
bird
rog
deer
ship
blr
dog
truck
rog
deer
borse
ruck
-100
Jog
horse
uttombille
50
50
100
150
50
50
100
150
Figure 7: t-SNE plot of object prototypes using trained model with ER and SARL on Seq- CeF-CIFARIO with 200 buffer size.
Meta-learning approaches have shown promise in continual learning by leveraging task-specific adaptations. VR-MCL (Wu et al. (024)) proposes an online CL approach that combines Meta-CL with regularization to efficiently approximating online Hessians, enhancing adaptability in real-time learning scenarios. While our work adopts a batch training setup, allowing multiple passes over se- quential task data which we believe is more aligned with many real-world scenarios where datasets can be revisited periodically (e.g., staged data collection iterative training). VR-MCL provides focuses on online single-pass learning. The fundamental differences in training settings make di- rect comparisons challenging, but recognize the potential of such meta-learning techniques to inspire future extensions of SARL. in online CL. Interactive Continual Learning (ICL. (Qi et al.) 2024 utilizes multimodal large language models (LLLMs) to achieve state-of-the-art performance in continual learning. By leveraging extensive pretraining on diverse datasets (often a superset of classes in the baseline CL datasets) and significantly larger architectures, this approach gains a lot of advantage over the standard setting. It also makes them infcasible for many CL applications due to the magnitudes larger memory and computational costs. The reliance on pretraining introduces an inherent disparity when comparing with SARL, which operates with randomly initialized model and focuses on both learning robust representations in the CL regime and retaining them as train- ing progresses. Furthermore, Memory-based approaches are widely used in rehearsal-based CL to alleviate catastrophic forgetting. Bilateral Memory Consolidation (BiMeCo) (Nie et al.) 2023) in- troduces method that consolidates long-term and short-term memories to improve performance. Their approach is complementary to SARL and could potentially enhance SARL further inte- grated. However, their evaluation setting of involves pretraining on large number of base classes before sequential learning, whereas SARL and its baselines initiate continual learning with a ran- domly initialized model. This distinction in experimental setups underscores the importance of careful contextualization when comparing results across studies.
Prompt-based methods, such I.2P (Wang al., 2022ch and DualPrompt (Wang et al., 20226), utilize pre-trained backbones, like those trained on ImageNet, to achieve strong performance in class incremental learning setting. These pre-trained backbones inherently include representations for many classes encountered later in CL tasks or for classes that are semantically similar. This allows these methods to bypass one of the core challenges of CL: learning and preserving general representations of objects across sequential tasks. In contrast, SARL operates in more challenging setting, starting from a randomly initialized state without leveraging pretraining. Extending SARL to integrate pre-trained models could be a promising direction for future research, but the current work focuses on enabling robust representation learning from a random start utilizing the semantic structure to learn and retain representations.
We would like to emphasize that evaluating CL methods across diverse baselines presents several challenges. Variations in architectures, the use of additional pretraining, access to implicit or explicit auxiliary information, and differences in task structures and metrics significantly influence reported results. Uniform evaluation settings are critical for meaningful assessments, as discrepancies in setup may reflect differences in methodology rather than genuine improvements.
20


Reviews:
[{"review_id": "VlmCnChHJm", "reviewer": "Reviewer_9Ntc", "summary": "This paper proposes a novel approach for continual learning by utilizing sparse activations to capture semantic relationships between objects. The authors suggest that leveraging activation sparsity emulates the sparse coding observed in biological neural systems. Prototypes are introduced as a means to represent each object, thereby facilitating knowledge retention across tasks. Experimental evaluations are conducted on benchmark continual learning datasets, including Seq-CIFAR10, Seq-CIFAR100, and Seq-TinyImg, demonstrating the proposed method's performance.", "questions": "- Given that object representations are \\( L_2 \\)-normalized, why use the simple mean (Eq. 1) instead of considering a spherical manifold? Why not use similarity (sim) instead of \\( L_2 \\) distance in Eqs. 4 and 5?\n\n- Some experimental details are missing. For example, the number of steps or epochs for the warm-up stage at each task is not specified. Additional details on the sparse backbone and its comparison with a regular backbone would also be beneficial.\n\n- Memory requirements should be clarified, as different components are stored during training in addition to the buffer from previous tasks (e.g., object prototypes and logits from the previous model state).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper is relatively well-motivated.\n- The paper is easy to follow.", "weaknesses": "- **Backbone Modifications**: It appears that the backbone architecture is altered (Section 3.3 on Sparse Coding) to incorporate sparse activations, which seem to increase similarities (as shown in Fig. 2). This modification could give the proposed method an advantage over other baselines.\n\n- **Unjustified Claims**: \n  - Line 248: The claim that \"sparse activations further enhance the model’s ability to distinguish between similar and dissimilar objects\" lacks sufficient justification.\n  - Line 284: The statement that sparse coding \"fosters stability in the model’s knowledge but also mitigates the risk of forgetting important semantic relationships established in earlier learning phases\" is unsupported by clear evidence in the text.\n\n- **Inconsistent Notation**:\n  - In Equation (3), symbols like \\( S_c \\) and \\( C^t \\) should use set notation for clarity.\n  - There is inconsistency in interval notation (e.g., \\( 0:t \\) versus \\( \\{0:t\\} \\)).\n  - Vectors are not visually distinct from scalars; for example, \\( a_i \\) and \\( o_b^c \\) are written in the same style as scalar values, which makes the notation confusing.\n\n- **Objective Complexity and Lack of Ablation**: The objective function comprises five different components, many of which rely on existing objectives, such as consistency regularization. However, no ablation study is provided, making it challenging to determine the contribution of each component to the model's performance:\n  - Regularization on object prototypes\n  - Consistency regularization loss\n  - Semantic-aware metric loss\n  - Two cross-entropy losses—one for the current training batch and another for buffer samples\n\n - **Lack of recent sota baseline**\n\n [1] Meta continual learning revisited: implicitly enhancing online hessian approximation via variance reduction, ICLR 2024\n\n [2] Interactive Continual Learning: Fast and Slow Thinking, CVPR 2024\n\n [3] Bilateral memory consolidation for continual learning, CVPR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TNfvApiUNR", "reviewer": "Reviewer_oGkj", "summary": "This paper tries to strengthen continual learning methods' ability to utilize semantic similarities for efficient learning and knowledge consolidation. The authors propose Semantic Aware Representation Learning (SARL). SARL employs sparse activations and principled approaches to assess similarities between objects encountered in different tasks and subsequently uses them to guide representation learning. Experimental results show that SARL balances plasticity and stability by harnessing underlying semantic structure.", "questions": "Please refer to Strengths and Weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-structured and easy to follow. The methodology section is thorough and provides detailed explanations.", "weaknesses": "1. The method appears to conflict with the problem setting. Specifically, the proposed method stores samples from previous tasks to update the model in future CIL tasks. It seems inconsistent with the CIL problem setting, where samples from previous tasks are not accessible, and storing old samples could lead to information leakage. Please provide a reasonable explanation for this discrepancy.\n2. The comparison methods are somewhat limited. The authors only compare their approach with methods that are similar to their own, while overlooking other approaches that also address the CIL problem, such as those based on pre-trained models (e.g., L2P, DualPrompt, etc.). I suggest that the authors clarify their perspective in the “RELATED WORK” or “ANALYSIS” section if they believe these comparisons would not be fair.\n3. The evaluations are insufficient. This paper relies solely on “average accuracy” as the evaluation metric, neglecting other important measures such as final accuracy and model forgetting. I recommend including metrics for “top-1 accuracy after the final stage” and “final forgetting” to ensure a comprehensive evaluation of the proposed method.\n4. Conflicting notation. Below Eq. (3), the authors state, \"... and $C^{0: t}$ represents the set of all object categories observed up to task $t$.\" Here, \"$C^{0: t}$\" should be corrected to \"$C^{\\\\{0: t\\\\}}$\". Please carefully proofread the entire paper to ensure consistency in symbols and notation.\n5. Typos. In the first paragraph of Appendix A, there is an extra \"I\" in the last line. Please thoroughly proofread the entire paper to ensure it is free of similar errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lJxfB3ioCC", "reviewer": "Reviewer_UqLc", "summary": "The paper proposes a method to reduce catastrophic forgetting in continual learning by taking advantage of several aspects: 1) the semantic relationships between learned classes, 2) representation sparsity, and 3) prototype regularization and other losses. The method draws inspiration from the brain’s sparse coding approach and employs activation sparsity to facilitate semantic information encoding in representations to promote reusability and reduce forgetting.", "questions": "Please refer to the weaknesses section.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper is clear and easy to read.\n2. The combination of losses is interesting.\n3. The performance results are very good.", "weaknesses": "1. The introduction could benefit from being more specific and quicker in presenting the problem, current limitations, and how the paper addresses these issues in the technical section. A more concise and more streamlined rewrite is recommended.\n\n2. The novelty appears to be somewhat limited, as deep neural networks inherently build and use semantic representations. A clearer motivation for the novel use of Eqs. (1), (2), ..., (5) (6) would be beneficial. Additionally, an introduction to these losses/similarities should be provided to ensure the reviewer understands they are not being presented as novel contributions. Clarification is needed on how the proposed approach to semantic representations differs from or improves upon standard deep learning methods. Additionally, the motivation and novelty behind the specific formulations in Eqs. (1)-(6) should be explained, and it should be explicitly stated which components are novel contributions versus established techniques.\n\n3. In the reviewer's opinion, the paper is a solid combination of well-established techniques that achieves good performance. However, it may be worth considering whether a different venue might be a better fit for this type of contribution. ICLR often prioritizes papers with some analytical theoretical analysis, while contributions that mainly involve combining established techniques with primarily empirical evidence may be less prioritized.\n\n4. The content on lines 054 to 055 is somewhat generic and does not sufficiently characterize the current limitations and issues. A more detailed description, including a toy problem to demonstrate qualitatively how this affects catastrophic forgetting in practice, would be beneficial. The reviewer may not fully grasp all the details of the paper and the insights that led to the proposed approach; therefore, proposing a concrete toy problem may not be straightforward, as it typically depends on the specific effect intended to be demonstrated. One possible suggestion is to choose a small dataset, such as CIFAR-10, and represent it in a 2D or 3D feature space, or, if >3D, using a scatterplot of the feature distribution or a dimensionality reduction technique like t-SNE to illustrate how the semantic aspects contribute as proposed by the authors in the introduction. These papers present some classic methods for visualizing feature representations:\n\n- Wen, Yandong, et al. \"A discriminative feature learning approach for deep face recognition.\" Computer vision–ECCV 2016: 14th European conference, amsterdam, the netherlands, October 11–14, 2016, proceedings, part VII 14. Springer International Publishing, 2016.\n\n- Pernici, Federico, et al. \"Maximally compact and separated features with regular polytope networks.\" The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. IEEE, 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Uw6PJpndp", "reviewer": "Reviewer_6oW7", "summary": "This paper introduces Semantic-Aware Representation Learning (SARL), a method aimed at mitigating catastrophic forgetting in lifelong learning. Inspired by human cognitive processes, SARL uses sparse activation and object prototypes to capture semantic relationships, employing semantic-aware metric learning to cluster similar objects while separating dissimilar ones. Experimental results show that SARL excels in Class-Incremental and Generalized Class-Incremental Learning, effectively retaining and transferring knowledge within a single-model setup. SARL’s integration of semantic structure and sparse representations offers a balanced approach to model stability and adaptability in lifelong learning.", "questions": "In the ablation study presented in Table 3, why is the performance in the first row so low? Is this row representing a traditional ResNet-18 classifier?\n\nDo you have any insights into why the improvements for the Seq-CIFAR10 dataset are marginal compared to the more significant gains observed with Seq-CIFAR100 and Seq-TinyImageNet?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The SARL method introduces an innovative approach by drawing inspiration from biological processes to incorporate semantic relationships into representation learning.\n\nBy employing semantic-aware metric learning, SARL effectively clusters similar objects while separating dissimilar ones, achieving a valuable balance between knowledge retention and adaptation to new information.\n\nValidated across complex incremental learning tasks, such as Class-Incremental and Generalized Class-Incremental Learning, SARL demonstrates robust performance, minimizing task interference and enabling seamless knowledge transfer, highlighting its practical effectiveness.\n\nThe paper is well-written, with a clear structure that is easy to follow.", "weaknesses": "The objective function (7) of the proposed SARL method includes five loss terms, but the paper lacks a sensitivity analysis on the weights hyperparameter assigned to each term. Such an analysis would provide valuable insights into the impact of each loss term on the model’s performance. The current one in Appendix is not sufficient.\n\nThe introduction of object prototypes and semantic-aware metric learning may increase the model’s computational overhead and resource requirements compared to baseline method such as ER. The paper does not provide a detailed discussion on the model’s efficiency\n\nThe experiments rely solely on the ResNet-18 backbone, lacking comparisons with other backbone models such as ViT, which could provide a broader evaluation of the method’s effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)