Semantic Aware Representation Learning for Lifelong Learning

Authors: Fahad Sarfraz, Elahe Arani, Bahram Zonooz
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: lifelong learning, continual learning, sparsity, semantic relationships, sparse coding

Abstract:
The human brain excels at lifelong learning by not only encoding information in sparse activation codes but also leveraging rich semantic structures and relationships between newly encountered and previously learned objects. This ability to utilize semantic similarities is crucial for efficient learning and knowledge consolidation, yet is often underutilized in current continual learning approaches. To bridge this gap, we propose Semantic-Aware Representation Learning (SARL) which employs sparse activations and a principled approach to evaluate similarities between objects encountered across different tasks and subsequently uses them to guide representation learning. Using these relationships, SARL enhances the reusability of features and reduces interference between tasks. This approach empowers the model to adapt to new information while maintaining stability, significantly improving performance in complex incremental learning scenarios. Our analysis demonstrates that SARL achieves a superior balance between plasticity and stability by harnessing the underlying semantic structure.

Decision: accept (poster)
Reviewer scores: 8, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1730344496745

thread_type:
official_review

<thread_object>
creator: reviewer_6ow7

<content>
summary:
This paper introduces Semantic-Aware Representation Learning (SARL), a method aimed at mitigating catastrophic forgetting in lifelong learning. Inspired by human cognitive processes, SARL uses sparse activation and object prototypes to capture semantic relationships, employing semantic-aware metric learning to cluster similar objects while separating dissimilar ones. Experimental results show that SARL excels in Class-Incremental and Generalized Class-Incremental Learning, effectively retaining and transferring knowledge within a single-model setup. SARL’s integration of semantic structure and sparse representations offers a balanced approach to model stability and adaptability in lifelong learning.

strengths:
The SARL method introduces an innovative approach by drawing inspiration from biological processes to incorporate semantic relationships into representation learning.

By employing semantic-aware metric learning, SARL effectively clusters similar objects while separating dissimilar ones, achieving a valuable balance between knowledge retention and adaptation to new information.

Validated across complex incremental learning tasks, such as Class-Incremental and Generalized Class-Incremental Learning, SARL demonstrates robust performance, minimizing task interference and enabling seamless knowledge transfer, highlighting its practical effectiveness.

The paper is well-written, with a clear structure that is easy to follow.

weaknesses:
The objective function (7) of the proposed SARL method includes five loss terms, but the paper lacks a sensitivity analysis on the weights hyperparameter assigned to each term. Such an analysis would provide valuable insights into the impact of each loss term on the model’s performance. The current one in Appendix is not sufficient.

The introduction of object prototypes and semantic-aware metric learning may increase the model’s computational overhead and resource requirements compared to baseline method such as ER. The paper does not provide a detailed discussion on the model’s efficiency

The experiments rely solely on the ResNet-18 backbone, lacking comparisons with other backbone models such as ViT, which could provide a broader evaluation of the method’s effectiveness.

questions:
In the ablation study presented in Table 3, why is the performance in the first row so low? Is this row representing a traditional ResNet-18 classifier?

Do you have any insights into why the improvements for the Seq-CIFAR10 dataset are marginal compared to the more significant gains observed with Seq-CIFAR100 and Seq-TinyImageNet?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730480403949

thread_type:
official_review

<thread_object>
creator: reviewer_uqlc

<content>
summary:
The paper proposes a method to reduce catastrophic forgetting in continual learning by taking advantage of several aspects: 1) the semantic relationships between learned classes, 2) representation sparsity, and 3) prototype regularization and other losses. The method draws inspiration from the brain’s sparse coding approach and employs activation sparsity to facilitate semantic information encoding in representations to promote reusability and reduce forgetting.

strengths:
1. The paper is clear and easy to read.
2. The combination of losses is interesting.
3. The performance results are very good.

weaknesses:
1. The introduction could benefit from being more specific and quicker in presenting the problem, current limitations, and how the paper addresses these issues in the technical section. A more concise and more streamlined rewrite is recommended.

2. The novelty appears to be somewhat limited, as deep neural networks inherently build and use semantic representations. A clearer motivation for the novel use of Eqs. (1), (2), ..., (5) (6) would be beneficial. Additionally, an introduction to these losses/similarities should be provided to ensure the reviewer understands they are not being presented as novel contributions. Clarification is needed on how the proposed approach to semantic representations differs from or improves upon standard deep learning methods. Additionally, the motivation and novelty behind the specific formulations in Eqs. (1)-(6) should be explained, and it should be explicitly stated which components are novel contributions versus established techniques.

3. In the reviewer's opinion, the paper is a solid combination of well-established techniques that achieves good performance. However, it may be worth considering whether a different venue might be a better fit for this type of contribution. ICLR often prioritizes papers with some analytical theoretical analysis, while contributions that mainly involve combining established techniques with primarily empirical evidence may be less prioritized.

4. The content on lines 054 to 055 is somewhat generic and does not sufficiently characterize the current limitations and issues. A more detailed description, including a toy problem to demonstrate qualitatively how this affects catastrophic forgetting in practice, would be beneficial. The reviewer may not fully grasp all the details of the paper and the insights that led to the proposed approach; therefore, proposing a concrete toy problem may not be straightforward, as it typically depends on the specific effect intended to be demonstrated. One possible suggestion is to choose a small dataset, such as CIFAR-10, and represent it in a 2D or 3D feature space, or, if >3D, using a scatterplot of the feature distribution or a dimensionality reduction technique like t-SNE to illustrate how the semantic aspects contribute as proposed by the authors in the introduction. These papers present some classic methods for visualizing feature representations:

- Wen, Yandong, et al. "A discriminative feature learning approach for deep face recognition." Computer vision–ECCV 2016: 14th European conference, amsterdam, the netherlands, October 11–14, 2016, proceedings, part VII 14. Springer International Publishing, 2016.

- Pernici, Federico, et al. "Maximally compact and separated features with regular polytope networks." The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. IEEE, 2019.

questions:
Please refer to the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730624652693

thread_type:
official_review

<thread_object>
creator: reviewer_ogkj

<content>
summary:
This paper tries to strengthen continual learning methods' ability to utilize semantic similarities for efficient learning and knowledge consolidation. The authors propose Semantic Aware Representation Learning (SARL). SARL employs sparse activations and principled approaches to assess similarities between objects encountered in different tasks and subsequently uses them to guide representation learning. Experimental results show that SARL balances plasticity and stability by harnessing underlying semantic structure.

strengths:
The paper is well-structured and easy to follow. The methodology section is thorough and provides detailed explanations.

weaknesses:
1. The method appears to conflict with the problem setting. Specifically, the proposed method stores samples from previous tasks to update the model in future CIL tasks. It seems inconsistent with the CIL problem setting, where samples from previous tasks are not accessible, and storing old samples could lead to information leakage. Please provide a reasonable explanation for this discrepancy.
2. The comparison methods are somewhat limited. The authors only compare their approach with methods that are similar to their own, while overlooking other approaches that also address the CIL problem, such as those based on pre-trained models (e.g., L2P, DualPrompt, etc.). I suggest that the authors clarify their perspective in the “RELATED WORK” or “ANALYSIS” section if they believe these comparisons would not be fair.
3. The evaluations are insufficient. This paper relies solely on “average accuracy” as the evaluation metric, neglecting other important measures such as final accuracy and model forgetting. I recommend including metrics for “top-1 accuracy after the final stage” and “final forgetting” to ensure a comprehensive evaluation of the proposed method.
4. Conflicting notation. Below Eq. (3), the authors state, "... and $C^{0: t}$ represents the set of all object categories observed up to task $t$." Here, "$C^{0: t}$" should be corrected to "$C^{\\{0: t\\}}$". Please carefully proofread the entire paper to ensure consistency in symbols and notation.
5. Typos. In the first paragraph of Appendix A, there is an extra "I" in the last line. Please thoroughly proofread the entire paper to ensure it is free of similar errors.

questions:
Please refer to Strengths and Weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721792474

thread_type:
official_review

<thread_object>
creator: reviewer_9ntc

<content>
summary:
This paper proposes a novel approach for continual learning by utilizing sparse activations to capture semantic relationships between objects. The authors suggest that leveraging activation sparsity emulates the sparse coding observed in biological neural systems. Prototypes are introduced as a means to represent each object, thereby facilitating knowledge retention across tasks. Experimental evaluations are conducted on benchmark continual learning datasets, including Seq-CIFAR10, Seq-CIFAR100, and Seq-TinyImg, demonstrating the proposed method's performance.

strengths:
- This paper is relatively well-motivated.
- The paper is easy to follow.

weaknesses:
- **Backbone Modifications**: It appears that the backbone architecture is altered (Section 3.3 on Sparse Coding) to incorporate sparse activations, which seem to increase similarities (as shown in Fig. 2). This modification could give the proposed method an advantage over other baselines.

- **Unjustified Claims**: 
  - Line 248: The claim that "sparse activations further enhance the model’s ability to distinguish between similar and dissimilar objects" lacks sufficient justification.
  - Line 284: The statement that sparse coding "fosters stability in the model’s knowledge but also mitigates the risk of forgetting important semantic relationships established in earlier learning phases" is unsupported by clear evidence in the text.

- **Inconsistent Notation**:
  - In Equation (3), symbols like \( S_c \) and \( C^t \) should use set notation for clarity.
  - There is inconsistency in interval notation (e.g., \( 0:t \) versus \( \{0:t\} \)).
  - Vectors are not visually distinct from scalars; for example, \( a_i \) and \( o_b^c \) are written in the same style as scalar values, which makes the notation confusing.

- **Objective Complexity and Lack of Ablation**: The objective function comprises five different components, many of which rely on existing objectives, such as consistency regularization. However, no ablation study is provided, making it challenging to determine the contribution of each component to the model's performance:
  - Regularization on object prototypes
  - Consistency regularization loss
  - Semantic-aware metric loss
  - Two cross-entropy losses—one for the current training batch and another for buffer samples

 - **Lack of recent sota baseline**

 [1] Meta continual learning revisited: implicitly enhancing online hessian approximation via variance reduction, ICLR 2024

 [2] Interactive Continual Learning: Fast and Slow Thinking, CVPR 2024

 [3] Bilateral memory consolidation for continual learning, CVPR 2023

questions:
- Given that object representations are \( L_2 \)-normalized, why use the simple mean (Eq. 1) instead of considering a spherical manifold? Why not use similarity (sim) instead of \( L_2 \) distance in Eqs. 4 and 5?

- Some experimental details are missing. For example, the number of steps or epochs for the warm-up stage at each task is not specified. Additional details on the sparse backbone and its comparison with a regular backbone would also be beneficial.

- Memory requirements should be clarified, as different components are stored during training in addition to the buffer from previous tasks (e.g., object prototypes and logits from the previous model state).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732338521506

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of revision

comment:
We sincerely thank the reviewers for their constructive feedback and insightful suggestions, which have significantly improved the clarity and depth of our manuscript. Below, we summarize the key changes made to the revised manuscript in response to the reviewers' comments. All changes are highlighted in blue in the revised document.

1. **Additional Experimental Details (Section A):**  
   We have included a more detailed explanation of our experimental settings, including the warm-up stage duration and details of the backbone. This ensures reproducibility and provides a clearer understanding of the methodology.

2. **Additional Metrics for Experimental Settings (Section B, Table 5, and Table 6):**  
   Beyond average accuracy, we now report additional metrics such as forgetting, stability, plasticity, and trade-off. These metrics provide a more comprehensive evaluation of SARL's performance, highlighting its strengths in knowledge retention, adaptability, and balance between stability and plasticity.

3. **Extensive Section on Sensitivity to Hyperparameters (Section C, Table 7):**  
   We provide results for the complete hyperparameter grid used during tuning, demonstrating that SARL is not overly sensitive to specific parameter values. The method consistently achieves significant performance gains across a broad range of settings, making it practical and robust.

4. **Extension on Similarity of Object Prototypes: Dense vs Sparse Representations (Section D):**  
   We extend the analysis of semantic relationships by splitting the CIFAR-10 classes into animals and vehicles clusters comparing the inter and intra custer similarity matrices for dense and sparse activations. This analysis highlights that sparse activations better capture semantic structures. 

5. **Visualization of Representation Space Using t-SNE (Section E):**  
   We include t-SNE visualizations of the object prototypes for both SARL and ER. These visualizations demonstrate SARL's ability to create a more structured and semantically meaningful representation space, with tighter clustering of related objects and better separation of distinct classes.

6. **Extended Related Work (Section F):**  
   We have expanded the related work section to discuss recent advancements, such as meta-CL, multimodal approaches leveraging pretraining, prompting based CL. This contextualization underscores SARL's novelty, particularly its focus on learning representations from scratch without pretraining.

7. **Other Changes:**  
   - **Fixed Notations and Typos:** Improved clarity by fixing inconsistencies in mathematical notation and correcting minor typos.  
   - **Concise Introduction:** The introduction now more effectively highlights the limitations of existing approaches and succinctly presents SARL's contributions, making the problem statement and novelty more accessible.

These revisions aim to address the reviewers' concerns and provide a more thorough and transparent presentation of our work. We are confident that these updates strengthen the manuscript and demonstrate SARL's significance in advancing continual learning. We deeply appreciate the reviewers' feedback and remain open to further suggestions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732909000641

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Additional Results (1/3)

comment:
Based on the reviewers’ suggestions, we conducted additional experiments to demonstrate the versatility and applicability of our approach. Below, we summarize the key findings from two significant evaluations. These results have been added to the appendex

**Online Continual Learning** (Section B.1)

Online continual learning (Online CL) is a challenging setting where the model processes each data instance only once without revisiting previous data. To simulate this scenario, we adapted SARL by maintaining dynamic prototypes during training. Despite not being specifically designed for Online CL, SARL demonstrated competitive performance compared to state of the art methods designed for OnlineCL, with significant gains in AAA and Acc metrics, particularly on Seq-CIFAR100.

| **Method**  | **Seq-CIFAR10** AAA       | **Seq-CIFAR10** Acc       | **Seq-CIFAR100** AAA      | **Seq-CIFAR100** Acc      |
|-------------|---------------------------|---------------------------|---------------------------|---------------------------|
| SGD         | 34.85 ± 1.71             | 16.96 ± 0.60             | 11.63 ± 0.38             | 5.27 ± 0.28              |
| LWF         | 35.31 ± 0.98             | 18.84 ± 0.10             | 11.98 ± 0.40             | 5.63 ± 0.37              |
| A-GEM       | 38.66 ± 0.79             | 18.46 ± 0.17             | 13.15 ± 0.23             | 6.02 ± 0.17              |
| GEM         | 38.67 ± 0.77             | 18.49 ± 0.15             | 15.18 ± 0.38             | 8.30 ± 0.58              |
| ER          | 55.53 ± 2.58             | 43.83 ± 4.84             | 23.19 ± 0.38             | 16.07 ± 0.88             |
| DER         | 45.85 ± 1.62             | 29.87 ± 2.95             | 13.35 ± 0.36             | 6.12 ± 0.18              |
| DER++       | 64.22 ± 0.70             | 52.29 ± 1.86             | 19.88 ± 0.43             | 11.79 ± 0.65             |
| CLSER       | 63.02 ± 1.54             | 52.80 ± 1.66             | 25.46 ± 0.57             | 17.88 ± 0.69             |
| OCM         | 66.14 ± 0.95             | 53.39 ± 1.00             | 22.54 ± 0.79             | 14.40 ± 0.82             |
| ER-OBC      | 65.82 ± 0.91             | 54.85 ± 2.16             | 25.54 ± 0.25             | 17.21 ± 0.92             |
| On-EWC      | 38.44 ± 0.50             | 17.12 ± 0.51             | 11.81 ± 0.42             | 5.88 ± 0.31              |
| IS          | 37.33 ± 0.23             | 17.39 ± 0.19             | 12.32 ± 0.22             | 5.20 ± 0.18              |
| MER         | 50.99 ± 0.65             | 36.92 ± 2.42             | --                        | --                       |
| La-MAML     | 42.98 ± 1.60             | 33.43 ± 1.21             | 12.55 ± 0.39             | 11.78 ± 0.65             |
| VR-MCL      | **66.97 ± 1.58**         | 56.48 ± 1.79             | 27.01 ± 0.48             | 19.49 ± 0.69             |
| SARL        | 66.85 ± 1.15             | **57.21 ± 0.27**         | **31.66 ± 1.49**         | **24.39 ± 1.44**         |
</content>
</thread_object>

</thread>
<thread>
creation date:
1733200818274

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Discussions and Improvements During Rebuttal Period (1/2)

comment:
The rebuttal phase provided valuable opportunities to engage with reviewers and address concerns regarding our study. Below is a concise summary of the key discussions, improvements, and outcomes from this period:

### Key Reviewer Concerns and Responses

### 1. Additional Metrics for Comprehensive Evaluation
- **Concern:** Lack of metrics beyond average accuracy.
- **Response:** Added additional metrics including forgetting, stability, plasticity, and trade-off to provide a holistic view of SARL's performance. Retrained the baselines methods to extend the comparisn with strong baselines. Results demonstrated SARL's superior stability-plasticity trade-off and reduced forgetting across various settings.

### 2. Comparison with additional baselines
- **Concern:** Additional comparisons with baselines including baselines for Online CL, prompt based approaches and methods utilizing a large set of base classes or pretrained models.
- **Response:** Emphasized the importance of evaluating under a uniform setup and the emphasis of our approach on learning robust semantic representations. Extended the releted work section will all the methods shared by the reviewers in the context of our evaluation setting and method. Additionally, though this setting was out of scope for our study and SARL isn't specifically designed for Online CL,  we adapted SARL for the challenging Online CL setting where the model must learn samples with a single pass, achieving competitive results compared to state of art methods designed for Online CL, particularly on **Seq-CIFAR100**, where SARL outperformed existing baseline s on Averaged Anytime Accuracy (AAA) and final accuracy (Acc).

### 3. Evaluation on Additional Backbones
- **Concern:** Experiments relied solely on the ResNet-18 backbone.
- **Response:** Extended evaluations to include **ViT-Small** and **VGG** backbones, showcasing consistent improvements over ER across all settings without hyperparameter tuning.

### 4. Memory and Computational Efficiency
- **Concern:** Potential computational overhead of SARL.
- **Response:** Provided a detailed breakdown of memory usage, showing that SARL achieves significant performance improvements with negligible additional overhead compared to ER. SARL is also significantly more memory and computationally efficient than strong baselines like CLS-ER and SCoMMER.

### 5. Semantic Representation Analysis
- **Concern:** Claims regarding semantic representation learning not sufficiently supported
- **Response:** Added additional analysis on semantic representations using similarity within and between animal and vehicle clusters. Also added t-SNE visualizion of the object  prototypes. Both analysis support SARL's ability to enhance semantic clustering.

### 6. Ablation Studies and Hyperparameter Sensitivity
- **Concern:** Lack of sensitivity analysis on the weights assigned to loss terms.
- **Response:** Added hyperparameter sensitivity experiments, revealing consistent performance across a range of values and highlighting SARL’s robustness.

### 7. Clarifications on the Problem Setup, experimental settings and consistency in notations
- **Concern:** Alignment with the Continual Incremental Learning (CIL) setting and the use of a memory buffer for rehearsal-based methods. Additional details of the backbone and warmsup stage,
- **Response:** Clarified SARL’s adherence to established practices in CIL and contextualized its methodological framework. Extended the experimental setup and improved the consistency in notations


## Key Improvements in the Manuscript

1. **New Experiments:**
   - Evaluations under **Online CL** settings and on diverse backbones (**ViT-Small** and **VGG**) to demonstrate SARL’s versatility and robustness.  (Section B.1, and B.2)
   - Extended analysis of semantic representations using pairwise similarity matrices and t-SNE plot. (Section E and F)
   - Holistic evaluation with additional metrics (forgetting, plasticity, stability, trade-off) (Section C)
   - Hyper parameter sensitivity analysis. (Section D)


2. **Clarifications and Extended Discussions:**
   - Streamlined the introduction to better articulate the limitations of existing methods and SARL's contributions.
   - Expanded related work to include recent state-of-the-art methods, highlighting SARL's distinct position and differences in the evaluation protocol and challenges.(Section G)
   - improved consistency in notations and extended the experimental setup



## Reviewer Feedback and Outcomes

- **Increased Reviewer Scores:**
   - Three reviewers raised their scores from 5 to 6, acknowledging the comprehensive clarifications, additional experiments, and improved clarity in the manuscript.
   - Reviewers appreciated the memory analysis, hyperparameter sensitivity studies, additional metrics and new empirical evaluations.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Semantic-Aware Representation Learning (SARL) leverages sparse activations and semantic relationships between tasks to enhance feature reusability, reduce task interference, and achieve a superior balance between plasticity and stability in continual learning.

The paper is well-written. The experiment designs are well-motivated. The experiments are solid and provide a comprehensive analysis of the method. 

The paper received mixed reviews with scores of 6, 3, 6, and 8.

Following the rebuttal, the AC summarized a remaining concern raised by reviewer UqLc regarding the t-SNE visualization, noting that it failed to capture inter-cluster distances, and initiated a discussion with the reviewers.

During the internal discussions, both reviewer UqLc and reviewer 9Ntc agreed that the extended pairwise similarity analysis provided stronger support as it directly operated on the prototypes.

Considering the reviewers' agreements and the current status of the paper, the AC decided to accept it. AC strongly encourages the authors to address the reviewers' feedback in the final version, particularly improvements to the t-SNE results and their interpretation.