Spurious Forgetting in Continual Learning of Language Models

Authors: Junhao Zheng, Xidi Cai, Shengjie Qiu, Qianli Ma
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Continual Learning, Language Models, Large Language Models, Catastrophic Forgetting, Incremental Learning, Lifelong Learning

Abstract:
Recent advancements in large language models (LLMs) reveal a perplexing phenomenon in continual learning: despite extensive training, models experience significant performance declines, raising questions about task alignment and underlying knowledge retention. This study first explores the concept of "spurious forgetting", proposing that such performance drops often reflect a decline in task alignment rather than true knowledge loss. Through controlled experiments with a synthesized dataset, we investigate the dynamics of model performance during the initial training phases of new tasks, discovering that early optimization steps can disrupt previously established task alignments. Our theoretical analysis connects these shifts to orthogonal updates in model weights, providing a robust framework for understanding this behavior. Ultimately, we introduce a Freezing strategy that fix the bottom layers of the model, leading to substantial improvements in four continual learning scenarios. Our findings underscore the critical distinction between task alignment and knowledge retention, paving the way for more effective strategies in continual learning.

Decision: accept (poster)
Reviewer scores: 6, 8, 5

Reviews and discussion:
<thread>
creation date:
1730397744934

thread_type:
official_review

<thread_object>
creator: reviewer_rjuy

<content>
summary:
This paper studies the issue of forgetting during the training process of large language models (LLMs) and proposes a novel perspective on LLM forgetting. The authors categorize forgetting into two types: "decrease in task alignment capability" and "loss of knowledge." The decrease in task alignment capability is referred to as "pseudo-forgetting." When pseudo-forgetting occurs, the model’s capability can be restored with minimal data replay.
The authors also explore the pseudo-forgetting process at the model parameter level and find that, when aligning to different tasks, the orthogonal gradient directions of different tasks result in a loss of alignment with the original task when aligning with a new one. Furthermore, the authors study parameter changes across different model layers and find that the lower layers (embedding and first layer) play an essential role in task alignment. Freezing these layers can effectively alleviate the issue of pseudo-forgetting.

strengths:
Proposes a novel perspective on catastrophic forgetting in LLM training, suggesting that LLMs do not lose knowledge but rather lose the ability to align with tasks.
Provides some theoretical support for the parameter-freezing training method through experiments.
Conducts thorough theoretical analysis. Experimental results across different methods are adequately explained, and a formalized definition of "pseudo-forgetting" is given.
The proposed method of freezing lower-layer parameters shows good performance.

weaknesses:
1 Lack of Originality and Insufficient Literature Review:
The use of parameter freezing to mitigate forgetting in models is not a novel approach, and similar techniques have been explored previously. However, the authors do not provide an adequate discussion of prior research related to parameter freezing. This omission is significant, as a comparison with existing methods or a clear differentiation of this approach from prior studies would enhance the work's originality and position it within the context of established research.
2 Limited Evidence for Generalizability:
The experiments in the paper focus primarily on a synthetic biographical dataset, which raises questions about the broader applicability of the proposed parameter-freezing method. To demonstrate its effectiveness across diverse settings, it would be beneficial for the authors to test the method on additional, more varied datasets, including real-world datasets, and on a broader range of model architectures. Without this evidence, the method's generalizability remains uncertain.
3 Unclear Representation and Interpretation in Figures:
The figures, particularly Figure 3, are challenging to interpret, making it difficult to follow the intended findings. For instance, the paper later claims that data replay aligns the model with task 1 initially and then realigns with task 0, but this interpretation is not evident in Figure 3. The gradient descent path seems to diverge further from task 0, which contradicts the paper’s claims about realignment. Clearer visualizations or explanations of how the figures illustrate these alignment shifts are necessary for readers to understand the proposed method’s impact fully.

questions:
1 Distinguishing Knowledge and Alignment in LLMs:
In the Introduction, the authors propose that an LLM’s task performance consists of both underlying knowledge and alignment capability. However, it’s unclear how these two aspects can be empirically distinguished. While task performance can be observed directly, what methods can effectively gauge the model’s internal knowledge versus its alignment with specific tasks? In Chapter 3, the authors describe two parameter update directions as reflecting "task 1 knowledge" and "task 0 alignment." Could the authors provide more clarity or empirical evidence to substantiate this differentiation? For instance, could ablation studies or other targeted analyses help isolate the effects of knowledge retention versus alignment?
2 Practical Value of "Pseudo-Forgetting" as a Concept:
The authors introduce "pseudo-forgetting" to describe cases where the model’s performance declines not due to actual knowledge loss, but rather due to a misalignment with the task at hand. Given that continual learning models must retain task alignment to be effective, what is the practical utility of distinguishing between true forgetting and pseudo-forgetting? Does this concept offer actionable insights for the design of more resilient continual learning strategies? If so, could the authors expand on the implications of pseudo-forgetting and how it might guide future methods for reducing task misalignment?
3 Initial Training Phase and Task Alignment Loss:
In Chapter 3, the authors extensively discuss how LLMs lose alignment with the original task in the first 150 steps of training on a new task, then progressively align with the new one. Could the authors clarify whether a warm-up phase was applied during this fine-tuning process? Alternatively, might the observed alignment shift at this stage be influenced by initial optimizer state instability, leading to greater parameter fluctuations rather than true alignment loss? Additionally, given that 150 steps might be insufficient to fully assess alignment shifts, could the authors investigate the effects of a longer initial phase or analyze optimizer state changes to clarify the nature of this alignment shift?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730516904731

thread_type:
official_review

<thread_object>
creator: reviewer_zffn

<content>
summary:
The paper analyzes the phenomenon of catastrophic forgetting in the continual learning of language models. The authors argue that this forgetting stems from a decline in task alignment rather than a loss of knowledge. They support their claim through: (1) constructing a biography dataset to demonstrate their findings from the perspectives of performance, loss landscape, model weights, and features; (2) conducting a theoretical analysis of orthogonal updates in model weights. Then they revisit existing methods and propose a straightforward and effective freezing strategy to mitigate forgetting, which outperforms other methods.

strengths:
1. The paper presents a novel and compelling claim regarding the phenomenon of forgetting in continual learning.
2. It adopts multiple perspectives to effectively substantiate this claim.
3. The paper provides a clear theoretical analysis based on the observation of orthogonal updates in model weights.
4. The proposed method, grounded in the theoretical analysis and observation, outperforms existing baseline methods.

weaknesses:
The support for the assumption that the weight perturbation matrix lies in the left null space of the weight matrix in Assumption 4.4 appears to be simply based on the observations from Figure 4. However, in Figure 4, orthogonal updates occur only after the first 150 steps in the model's bottom layers. Based on performance observations, the accuracy of Task 0 has already significantly declined during these initial 150 steps. This does not support the conclusion that ``"observed performance declines are largely a result of orthogonal updates in model weights."``

I would appreciate it if the authors could address the concern or clarify if I have misunderstood any part of the paper.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730652013489

thread_type:
official_review

<thread_object>
creator: reviewer_rhw8

<content>
summary:
The paper explores the phenomenon of "spurious forgetting" in continual learning for language models. It challenges the assumption that performance degradation across sequential tasks implies actual knowledge loss, suggesting that many performance drops arise from suboptimal task alignment. The authors propose a new method, "Freeze," which involves freezing the lower layers of a model to reduce this issue.

strengths:
1. The paper brings a novel perspective by distinguishing between knowledge forgetting and spurious forgetting, addressing a misconception in continual learning research. This distinction enhances our understanding of model behavior across sequential tasks, highlighting the significance of this work.

2. The work is generally well-articulated and of good quality, with well-structured arguments and detailed explanations, making the complex ideas accessible to a broad audience.

3. The "Freeze" approach is simple yet effective, offering a practical solution for reducing spurious forgetting. Its ease of implementation is particularly noteworthy.

weaknesses:
1. Freezing layers for fine-tuning has long been a common practice and is not a novel approach. For instance, freezing lower layers was explored in the VGG paper [1].  
 
2. The empirical results in this work do not provide strong evidence to support the assumptions related to orthogonality.

3. The proposed method is only compared to the SEQ method in real-world scenarios, which is a pretty weak baseline.

[1] Simonyan, K., & Zisserman, A. (2014). VERY DEEP CONVOLUTIONAL NETWORKS FOR LARGE-SCALE IMAGE RECOGNITION. arXiv preprint arXiv:1409.1556.

questions:
1. Why are Task 1 knowledge and Task 0 alignment used as the bases for plotting the loss landscapes?

2. The theoretical framework relies primarily on a linear model. Could this assumption of linearity be too strong to realistically capture the complexities of the problem? What are the limitations of using a linear model?

3. Why isn’t exact match accuracy used as the evaluation metric for all tasks?

4. From a model weight perspective, why is the angle computed based on the column spaces rather than directly from the weight updates?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732334187108

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We sincerely thank the reviewers for their thoughtful feedback and constructive suggestions. Below, we summarize the key strengths acknowledged by the reviewers and outline the updates made to address their concerns.

### **Key Strengths Highlighted by Reviewers**

1. **Novel Perspective on Forgetting in Continual Learning** (`RhW8`, `zFfn`, `rjuy`)

2. **Theoretical and Experimental Support** (`zFfn`, `rjuy`)

3. **Effectiveness of the Freeze Strategy** (`RhW8`, `zFfn`, `rjuy`)

4. **Quality of Presentation** (`RhW8`, `zFfn`)

---

### **Key Updates to the Manuscript**
To address the reviewers’ concerns, we have made the following updates (highlighted in blue in the revised manuscript):

1. **Real-World Experiments**:  
   Added experiments on supervised finetuning for math and code datasets to demonstrate generalizability (`Appendix I.5`).

2. **Comparisons with Advanced Methods**:  
   Included comparisons to LAMOL and EWC on continual learning benchmarks (`Table 10` in `Appendix I.3.3`).

3. **Investigation of Instability in Fine-Tuning**:  
   Conducted experiments analyzing spurious forgetting under different optimizers and learning rates (`Figure 12` in `Appendix G.1.4`).

4. **Expanded Related Work**:  
   Added a discussion on parameter-freezing methods to situate our contributions within the broader literature (`Appendix A.3`).

5. **Additional Explanations**:  
   - Clarified trajectory interpretation under data replay (`Appendix G.2`).
   - Enhanced the explanation of axes in the loss landscape figure (`Figure 3`).
   - Additional discussion of the connection between `Proposition 4.9` and the Freeze strategy (`Remark F.10`).

6. **Additional Theorectical Results**:
   - We present `Corollary F.5` and prove that freezing the bottom layers helps mitigate cumulative shift in the real-world scenario, where the orthogonal updates primarily occur in the bottom layers. ( `Assumption F.4`,`Corollary F.5`,`Remark F.6`)

We hope these revisions and additional experiments address the reviewers' concerns and further substantiate the significance of our contributions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper investigates "spurious forgetting" in the context of continual learning for large language models (LLMs). It challenges the assumption that performance degradation in sequential tasks equates to actual knowledge loss, introducing the concept of "pseudo-forgetting," where task misalignment, rather than knowledge decay, leads to performance drops. The authors provide a theoretical framework, experiments on a synthetic biographical dataset, and analysis of model parameters to support this claim. Based on their findings, they propose a simple freezing method for the lower layers of LLMs, which mitigates spurious forgetting and improves continual learning performance.

### Strengths

- Introduces a novel and insightful distinction between spurious (pseudo) forgetting and knowledge forgetting, enhancing understanding of LLM behavior in continual learning scenarios.
- Provides a well-structured theoretical framework, supported by thorough experiments and analyses, including parameter and loss landscape studies.
- The "Freeze" approach is simple, effective, and practical, achieving competitive results while being easy to implement.
- Demonstrates clear explanations and strong presentation quality, making complex concepts accessible to a broad audience.
- Outperforms baseline methods in the specific experiments conducted, providing promising evidence for the freezing approach.

### Weaknesses

- The proposed freezing method is not novel, as parameter freezing has been widely studied in prior work (e.g., the VGG paper), and the authors fail to adequately position their contribution within this existing literature.
- Experiments rely heavily on a synthetic dataset, raising concerns about the generalizability of the findings to real-world scenarios and more diverse datasets.
- Empirical support for key theoretical assumptions (e.g., orthogonal updates causing misalignment) is limited and partly contradicted by observed results.
- Figures (e.g., Figure 3) are unclear or poorly explained, hindering interpretability and alignment with the paper's claims.
- Comparisons to baselines are weak, with only one (SEQ) real-world baseline tested, and more robust benchmarks are needed to demonstrate the effectiveness of the method.

Some concerns have been addressed by the authors during the rebuttal period.