Suppressing recency bias through implicit task in task-agnostic continual adaptation for foundation language models

Authors: Jae-Hong Lee, Chae-Won Lee, Ji-Hun Kang, Joon-Hyuk Chang
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: continual learning, lifelong learning, transfer learning, foundation language models

Abstract:
Foundation language models have significantly advanced natural language processing but face challenges such as catastrophic forgetting when adapting to dynamic environments with diverse tasks. Recently, among the continual learning (CL) methods for these models, model architecture expansion methods have been spotlighted due to the growth of parameter-efficient fine-tuning (PEFT) methods. However, these methods need to store past PEFT adapters for each task and require task identifiers (task IDs) to distinguish each task, thus limiting their applicability in task-agnostic settings. They also overlook recency bias, where models focus overly on current tasks at the expense of past knowledge. 
To address these issues, we propose suppressing recency bias (SRB) by using the concept of implicit tasks. SRB assigns a fixed-size adapter to an implicit task, recursively storing historical knowledge through arithmetic operations with current adapters at every time step instead of task IDs. This arithmetic mitigates recency bias by integrating non-overlapping information between historical and current adapters. 
Our approach requires only simple arithmetic operations without backpropagation, minimizing additional computation, and allocates a fixed-size adapter to the implicit task, resulting in low memory requirements. We evaluate SRB on CL benchmarks for foundational LMs. Experimental results demonstrate that SRB outperforms state-of-the-art methods, achieving superior generalization performance across various task sequences and models by effectively mitigating recency bias.

Decision: reject
Reviewer scores: 1, 6, 5, 3, 6, 3

Reviews and discussion:
<thread>
creation date:
1730522978452

thread_type:
official_review

<thread_object>
creator: reviewer_mlub

<content>
summary:
This paper addresses the problem of recency bias in foundation models, which causes models to prioritize recent tasks at the expense of past knowledge. The main contribution of this work is a novel method called "Suppressing Recency Bias"(SRB), which combines current and past knowledge using arithmetic operations without requiring additional back-propagation, thus ensuring minimal computational overhead. The key highlights of SRB that authors claim include:
1. SRB eliminates the need for task IDs by using an implicit task that aggregates past knowledge through simple arithmetic.
2. SRB maintains historical knowledge from past tasks while learning new tasks by integrating only unique information, thus reducing redundant learning.
3. SRB requires minimal additional memory space and computations to SOTA methods, outperforming methods like LoRA and IncLoRA effectively.

strengths:
1. It is a novel approach to handling one of the major challenges in continual learning: recency bias. By leveraging an implicit task to integrate historical knowledge, SRB addresses the common problem where models tend to overly focus on recent tasks at the expense of retaining knowledge from earlier ones. This approach could be beneficial across a wide range of continual learning applications.

2. SRB operates without requiring task IDs, which is a significant advantage in real-world scenarios where tasks are not explicitly defined. Task-agnostic continual learning is particularly challenging, and the SRB method makes a good improvement by showing it’s possible to achieve effective continual adaptation without relying on task-specific information.

3. The experimental results show SRB outperforms or is competitive with state-of-the-art methods like LoRA, IncLoRA, and O-IncLoRA on some standard CL benchmarks. SRB demonstrates superior generalization and knowledge retention across tasks, particularly in comparison to methods prone to catastrophic forgetting.

4. The authors provide a clear description of the experimental setup, benchmark datasets, and hyperparameters settings. This transparency supports reproducibility of this work.

weaknesses:
The paper presents a novel approach to addressing recency bias in task-agnostic continual learning. However, there are some potential weaknesses and limitations in this work.

1. The SRB method introduces several hyperparameters (such as a, b, and c for controlling the influence and regularization of task vectors). The paper notes that hyperparameter tuning is essential for SRB’s performance. This reliance might limit the model’s robustness, as it may require fine-tuning for different tasks or models, reducing its practicality in real-world, dynamic settings where such tuning isn’t feasible.

2. Although SRB is designed for task-agnostic settings, the scalability to larger or longer task sequences is not thoroughly explored. For instance, the implicit task mechanism might become less efficient or struggle to represent historical knowledge accurately when handling an extensive range of tasks. A larger-scale experiment would provide insights into how SRB performs with extensive, varied task sequences.

3. Since SRB relies on arithmetic operations to balance historical and current knowledge, it may struggle in environments where task characteristics change quickly or drastically. The implicit task representation could fail to adapt promptly in such settings, potentially limiting SRB’s performance on tasks that require quick, context-sensitive adaptation.

questions:
1.  In the regularization process (Equation 13), how to determine the effectiveness of the orthogonal projection for suppressing recency bias? Were there other regularization techniques or projections you considered, and if so, what made this one preferable? In addition, if the implicit task vector is orthogonal to \tau_t, then the projections becomes zero, is there any additional strategy to prevent this situation?

2.  In task-agnostic settings, are there specific application domains where SRB’s approach to suppressing recency bias is particularly valuable? Conversely, are there domains where the implicit task approach may struggle, such as with tasks that have low overlap or are highly distinct?

3. You mention using average accuracy for performance evaluation, but did you also measure other indicators like backward and forward transfer? If so, how did SRB perform on these metrics, especially in preserving knowledge from earlier tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730523364823

thread_type:
official_review

<thread_object>
creator: reviewer_9jp6

<content>
summary:
This paper investigates task-agnostic continual learning using foundational language models. To tackle the issues associated with previous PEFT methods, the authors introduce a novel approach called Suppressing Recency Bias (SRB), which enables the model to adapt to current data while retaining historical knowledge. By leveraging the design of implicit task vectors, SRB-based models can be trained to adapt without the need for task IDs. Experimental results demonstrate the effectiveness of the proposed method.

strengths:
1. The idea of employing a fixed-size adapter to recursively store the historical knowledge of an implicit task appears to be interesting and novel.  The proposed method can adapt to task-agnostic datasets even without task IDs.
2. The authors have conducted a thorough series of experiments to validate their proposed method.
3. The results regarding information retention indicate that the SRB significantly outperforms other models, showcasing its ability to adapt to new tasks while preserving performance on previous ones with minimal degradation.

weaknesses:
1. The paper lacks a detailed explanation of the motivation behind the model design, particularly concerning the implicit task vector and the regularization term.
2. The authors should apply the proposed method to additional foundational models (e.g., BERT) to further validate its effectiveness, similar to previous studies.
3. The explanation of how to calculate the task vector is somewhat unclear. For instance, as stated in line 288, the task vector for the current task is defined as $\tau_t=w_t-w_0$. However, the process for calculating the task vector for the next task is not elaborated upon.

questions:
1. Why does the implicit task vector act as a low-pass filter, limiting diversity?
2. What are the results of LLaMA3 and LLaMA3-chat on Orders 4, 5, and 6?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730577110187

thread_type:
official_review

<thread_object>
creator: reviewer_cmci

<content>
summary:
The paper presents **Suppressing Recency Bias (SRB)**, a method designed for task-agnostic continual learning in foundation language models (LMs). SRB introduces the concept of an implicit task that integrates knowledge recursively, minimizing the reliance on task identifiers and addressing recency bias—an issue where models disproportionately prioritize current tasks at the expense of previous ones. This approach achieves low memory overhead by requiring only fixed-size adapters and using simple arithmetic operations for updating, without the need for backpropagation. The paper demonstrates that SRB outperforms existing continual learning (CL) methods in both standard and extended task sequences by maintaining superior performance across diverse tasks and reducing recency bias.

strengths:
- **Task-Agnostic Adaptability**: SRB excels in task-agnostic settings, removing the dependency on task identifiers and preserving performance across tasks.
- **Computational Efficiency**: The method achieves this while maintaining minimal computational and memory overhead, only needing simple arithmetic operations.
- **Reduction of Recency Bias**: The introduction of implicit task vectors effectively mitigates recency bias, ensuring that historical information is preserved during adaptation.
- **Empirical Validation**: Experimental results on standard and long CL benchmarks show that SRB outperforms other state-of-the-art methods in accuracy and efficiency.

weaknesses:
- **Hyperparameter Sensitivity**: The approach relies on specific hyperparameters for performance, which might affect its generalizability without tuning.
- **Comparison Scope**: While the paper benchmarks against key methods, additional comparisons with more diverse baseline approaches, such as advanced replay-based strategies, could strengthen its conclusions.
- **Task Transition Analysis**: The paper could benefit from deeper analysis of how SRB handles transitions between tasks, especially in complex sequences involving highly dissimilar tasks.
- **Overhead of Hyperparameter Tuning**: Although SRB shows robustness, the paper notes fixed hyperparameters, implying that different task sequences might require adjustment for optimal performance.

questions:
- Can the authors elaborate on how the method performs under highly diverse task sequences, such as tasks involving distinct domains (e.g., code, legal text, medical literature)?
- What strategies can be used to fine-tune the hyperparameters (a, b, c) without extensive trial and error?
- How does SRB handle tasks that might involve conflicting objectives (e.g., creative writing vs. technical report summarization)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730619006596

thread_type:
official_review

<thread_object>
creator: reviewer_x9bg

<content>
summary:
This paper propose a continual learning algorithm that projects LoRA parameter updates to a subspace defined by an "implicit task" to mitigate recency bias and mitigate forgetting. Experiments on CL benchmarks demonstrate performance improvements over baselines, especially those that also learn LoRA.

strengths:
- The performance improvement is clear compared to the baselines.
- The description of the proposed approach is clear

weaknesses:
1. The intuition behind the performance improvement is not quite clear to me.

In a high level, the approach finds a direction of model parameter update that represents previous tasks (termed as "implicit tasks" in this work), and apply regularization in weight updates while learning new tasks by "pushing back" the update a bit towards a direction orthogonal to the "implicit tasks".

- It seems Orthogonal LoRA by Wang et al. 2023 does a similar job. What is the design that makes the approach improve over Orthogonal LoRA?  Please highlight the key methodological differences and explain how these differences contribute to the improved performance observed in the experiments. 

- The approach conceptually shares similar idea to regularization based CL approaches like L2 regularization (which pushes back parameter updates to their initial states before fine-tuning). But in Table 2 L2 regularization performs very poorly. What could be the reason? Please provide a more in-depth analysis of why the proposed approach outperforms L2 regularization.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649663578

thread_type:
official_review

<thread_object>
creator: reviewer_xdby

<content>
summary:
This paper introduces the Suppressing Recency Bias (SRB) method to mitigate catastrophic forgetting during the continuous learning process of language models (LMs) without task IDs. SRB introduces an additional implicit task adapter and designs an update mechanism to appropriately integrate knowledge learned from the current task into the implicit adapter. The updated implicit adapter is then used to initialize the learning of new data. The designed update mechanism reduces duplicated information in classical Model Architecture Expansion (MAE) methods while balancing the increase in adapter diversity and the reduction of recency bias. The method outperforms previous task-agnostic methods and MAE methods using task IDs on benchmark tasks. Ablation studies demonstrate the effects of hyperparameters on reducing recency bias and increasing diversity, and show that SRB is not sensitive to hyperparameters within a certain range.

strengths:
- The paper is clearly written and easy to understand. Figure 1 effectively contrasts SRB with other methods, and Figure 2 helps in understanding the purpose of the update mechanisms in SRB.
- SRB achieves a balance between increasing adapter diversity and reducing recency bias through a carefully designed update mechanism, which is validated by detailed ablation studies.
- SRB is simple and easy to use, and it outperforms existing MAE methods in terms of computational and memory costs.

weaknesses:
- The paper lacks a clear explanation of the division and training situation for $(D_1, …, D_T)$ in the SRB setting. Does each $D_t$ correspond to the data contained in a default training batch, or is it a manually divided portion of the dataset?
  - If it corresponds to data in a single batch, how many times will this data be updated? What is the impact of different batch sizes on the relevant hyperparameters?
  - If it is a manually divided portion of the dataset, how is the division performed? If the division ensures that data from one task forms a single $D_i$, then SRB actually leaks task ID information. If data from each task is evenly divided into N parts, it still contains some task information. In a realistic task-agnostic scenario, it is difficult to ensure that data from different tasks are divided into different groups. The reviewer would like to see the performance of SRB when data from sequential tasks are included in the same $D_t$.
- The results section lacks methods such as multi-task learning or task experts as performance upper bounds for reference. Adding this reference would help readers better understand the improvements and limitations of the proposed method. Additionally, Figure 3(a) in Section 5.1 could also provide the performance of task experts as a reference.

questions:
- Please provide more detailed explanations regarding the division of $D_i$ as mentioned in the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695793660

thread_type:
official_review

<thread_object>
creator: reviewer_jtkg

<content>
summary:
The paper proposed a method for continual learning. But I am unsure what problem it is solving.

strengths:
The proposed method is different.

weaknesses:
w-1. Your writing at the beginning sounds like you are working on continual learning of language models and adapting them to different domains. But I believe you are really doing normal continual learning using LMs at feature extractors since your tasks are text classification tasks. 

w-2. The paper did not state what setting of continual learning it works on. I think that this method works on task incremental learning (TIL). However, the first three baselines are commonly used for class incremental learning. Please clarify. For task incremental learning, the problem of forgetting is largely solved. Please check out [1, 2, 3, 4, 5, 6, 7].

w-3. Related to w-2. The paper says that the approach is task-agnostic, which means that no task-id is given, but which means it does class-incremental learning (CIL). But for CIL, by definition, there is no task-id information given. It is very confusing. Please make it clear which continual learning problem you are solving. If you are solving CIL, you should compare your method with another set of SOTA baselines. 

w-4. It is not true that architecture approaches add one adaptor for each task. Also see [1, 2, 3, 4, 5, 6, 7]. Most of the existing methods do not need to task-id either, assuming that you are solving the TIL problem. 

w-5. What is the difference between recency bias and catastrophic forgetting? Catastrophic forgetting is about focusing on the present and forgetting the past, which is the recency bias. 

w-6. Regarding baselines, your non-LoRA baselines, EWC,  Replay, and LwF, are very old and not the state of the art. Again, please check out [1, 2, 3, 4, 5, 6, 7]. 

w-7. What is average accuracy? Please give the definition. In continual learning, there are at least two accuracy measures.  

w-8. Having a separate adaptor for each model is not in the spirit of continual learning, which aims to use the same network learning multiple tasks. Please give the memory requirement of your approach. 

w-9. Please give the performance upper bound for the continual learning problem that you are solving. 

w-10. The writing of the paper needs significant improvement. The paper is confusing. I am not even sure what problem you are solving. If you are solving CIL, how do you deal with documents that may belong to two different classes in two different tasks? For example, a topic-specific document may contain a positive sentiment and a review of a problem may be classified to its product category.  

     [1]. Serra et al. Overcoming catastrophic forgetting with hard attention to the task. ICML-2018.
     [2]. Wortsman et al. Supermasks in superposition. NeurIPS-2020.
     [3]. Ke et al. Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning.  NeurIPS-2021.
     [4]. Lin et al. TRGP: Trust region gradient projection for continual learning. ICLR-2021.
     [5]. Lin et al. Beyond not-forgetting: Continual learning with backward knowledge transfer. NeurIPS-2022.  
     [6]. Ke et al.  Sub-network Discovery and Soft Masking for Continual Learning of Mixed Tasks.  EMNLP-2023.
     [7]. Dissecting learning and forgetting in language model finetuning. ICLR-2024.

questions:
See the previous section
</content>
</thread_object>

</thread>
<thread>
creation date:
1732792689570

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We have carefully reviewed the insightful questions and comments raised by the reviewers and used this opportunity to enhance our paper. We sincerely appreciate the significant effort and time committed by each reviewer in offering constructive feedback. We would like to provide a general response regarding the clarifications of the main contributions and how we addressed the common concerns.

# [Main Contributions of the Paper]

- **Introduction of Suppressing Recency Bias (SRB):** We propose SRB, a novel method for task-agnostic continual learning in language models that does not rely on explicit task identifiers or boundaries. SRB leverages implicit task vectors and simple vector arithmetic to dynamically preserve knowledge from previous tasks while adapting to new ones, effectively mitigating recency bias and catastrophic forgetting.
- **Dynamic Knowledge Integration without Increased Overhead:** SRB operates under fixed memory and computational requirements by using a single implicit task adapter. Unlike methods that require storing multiple adapters or task-specific parameters, SRB maintains scalability and efficiency even as the number of tasks grows.
- **Extensive Experimental Validation:** We conducted extensive experiments demonstrating that SRB outperforms state-of-the-art methods in task-agnostic continual learning settings across various benchmarks and models. Our results show that SRB effectively balances historical knowledge preservation with new knowledge acquisition, highlighting its robustness and effectiveness.

# [Updates in the Revised Draft]

- **Clarification of Continual Learning Setting:** We have clarified that our approach operates in a task-agnostic continual learning setting that does not rely on explicit task identifiers or boundaries (Section 2, Appendix B). This emphasizes that direct comparisons with existing methods, which assume task boundaries, may not be appropriate.
- **Expanded Related Work Section:** We have revised the related work section in Appendix B to provide a more comprehensive comparison between SRB and existing methods, including Orthogonal LoRA and other state-of-the-art continual learning approaches suggested by the reviewers.
- **Inclusion of Upper Bound Results:** To address the reviewers' suggestions, we have included Multi-Task Learning (MTL) and per-task fine-tuning as additional comparison baselines (Table 2 of Section 4). By comparing SRB's performance to these idealized scenarios, we provide clear context for evaluating our method's effectiveness.

# [Final Authors' Note]

In the revised draft, we have prioritized clarity and addressed the highlighted concerns. To clarify the justification of the comparisons in the new proposed research scope, we emphasize that our method is a new task-agnostic continuous learning paradigm. Moreover, we have addressed the reviewers' comments by including MTL and per-task fine-tuning as additional baselines for comparison. By comparing SRB's performance to these ideal scenarios, readers can better understand our contributions and the effectiveness of our method.

Our research is a pioneering effort to address recency bias and catastrophic forgetting in task-agnostic continual learning for language models. By leveraging implicit task vectors and dynamic regularization, SRB offers a novel methodology that balances historical knowledge preservation and new knowledge acquisition without relying on explicit task identifiers.

We thank the reviewers for their invaluable feedback and believe that our revisions have significantly improved the quality and clarity of our paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces Suppressing Recency Bias (SRB), a task-agnostic method for continual learning in language models (LMs). SRB addresses the issue of catastrophic forgetting by integrating past knowledge while learning new tasks without relying on task IDs. The method achieves this by introducing an implicit task adapter that aggregates past knowledge using simple arithmetic operations, avoiding the need for backpropagation and significantly reducing computational overhead. Claimed key contributions of this paper include: 1) Task-Agnostic Continual Learning enables continual learning without task identifiers. This makes the work adaptable to real-world, task-agnostic scenarios. 2) Proposed Suppression of Recency Bias to mitigates recency bias by recursively integrating historical knowledge, and ensuring better retention of information from previous tasks. SRB is a common issue in continual learning where models prioritize recent tasks at the cost of past knowledge. 3) ablations and empirical results showed the efficacy of proposed method.

Strength of this paper
- The method addresses the challenge of continual learning without task identifiers, which is important in real-world setting, and is effective in preserving historical knowledge while learning new tasks.
- Ablations and empirical results showed the efficacy of proposed method.
- The approach is computationally efficient and requires minimal memory, making it practical for large-scale deployment.


Weakness of this paper

Several reviewers raised few concerns/limitations of this paper. By addressing these limitations, the paper could strengthen its experiment and expand impact.

- Limited Generalizability: The method is tested on limited setups and foundational models. Broader validation on additional models and datasets is needed to demonstrate robustness. The method may struggle in dynamic environments with rapidly changing task characteristics, or realistic task-agnostic scenarios where tasks are not neatly separated.
- Experimental Design and Validation:  Some of the chosen baselines (e.g., EWC, Replay, LwF) are outdated, and comparisons with state-of-the-art methods (e.g., Orthogonal LoRA, advanced replay strategies) are missing. The study does not explore upper-bound performance (e.g., multi-task learning or task-specific experts) for reference. Ablation study doesn't cover all the key components, such as the necessity of the proposed regularization, implicit task representation, or its design improvements over existing methods (e.g., Orthogonal LoRA). The scalability of the method to longer or more complex task sequences remains unexplored.