Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuning

Authors: Gangwei Jiang, Caigao JIANG, Zhaoyi Li, Siqiao Xue, JUN ZHOU, Linqi Song, Defu Lian, Ying Wei
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: catastrophic forgetting; large language model; instruction tuning

Abstract:
Catastrophic forgetting (CF) poses a significant challenge in machine learning, where a model forgets previously learned information upon learning new tasks. 
Despite the advanced capabilities of Large Language Models (LLMs), they continue to face challenges with CF during continual learning. The majority of existing research focuses on analyzing forgetting patterns through a singular training sequence, thereby overlooking the intricate effects that diverse tasks have on model behavior.
Our study explores CF across various settings, discovering that model forgetting is influenced by both the specific training tasks and the models themselves. To this end, we interpret forgetting by examining the function vector (FV), a compact representation of functions in LLMs, offering a model-dependent indicator for the occurrence of CF. Through theoretical and empirical analyses, we demonstrated that CF in LLMs primarily stems from biases in function activation rather than the overwriting of task processing functions.
Leveraging these insights, we propose a novel function vector guided training methodology, incorporating a regularization technique to stabilize the FV and mitigate forgetting. Empirical tests on four benchmarks confirm the effectiveness of our proposed training method, substantiating our theoretical framework concerning CF and model function dynamics.

Decision: accept (oral)
Reviewer scores: 8, 8, 10, 10

Reviews and discussion:
<thread>
creation date:
1729885077600

thread_type:
official_review

<thread_object>
creator: reviewer_qiar

<content>
summary:
This paper studies the catastrophic forgetting (CF) problem in LLMs with the tool of Function Vector (FV). The research starts with reproducing the CF problem on a series of four tasks across 4 publicly available LLMs. In addition, the researchers study the differences in FVs between continue-trained and untrained LLMs on each task. They show that there is a strong correlation between the differences in FVs and the CF issue. Their theoretical analysis posts a hypothesis on the bias of the FV θT elicited by the input x, (i.e., p(θT|x)). The authors further conducted several experiments to well support their hypothesis. Finally, they designed a new loss for alleviating the CF problem in continuing learning, where they encourage LLMs to minimize the differences between model prediction and the FV-intervented prediction while keeping the hidden representations of X the same as un-trained ones.

strengths:
1. The manuscript provides a thorough study of the catastrophic forgetting problem. 
2. The most insightful contribution of this work is posing a new hypothesis on this problem. 
3. Its technical contribution is significant as the authors first introduce the Function Vector as a tool in this community.
4. The paper is well-written and well-organized.

weaknesses:
I cannot see any significant weakness. One nitpick would be expanding the background of Function Vector in Sec. 2.2, especially providing the dimension or shape of each notation. It will help the audiences understand the procedure for computing FVs more easily.

questions:
Here are some minor/typo suggestions:
1. Line 107, the references to Peng et al. and Chung et al. should be \citep instead of \cite.
2. Line 143, missing a blank between "using" and "SuperNI".
3. Line 162, the reference to Zhao et al. should be \citep instead of \cite.
4. Line 200, missing a blank between "." and "Unless".
5. Line 200, the reference to Hu et al. should be \citep instead of \cite.
6. Line 201, "learning rate of 1e-4" should be "learning rate of $1e^{-4}$".
7. Line 143/157/197, appending a "." after each boldface phrase will be more consistent with your writing style in Line 240/249/257 and so on.
8. Line 357,  "... trigger its corresponding latent variable θT (PM(θT|x))." could be rewritten as "... trigger its corresponding Function Vector θT (i.e., PM(θT|x))." for more clarification.
9. It would be great to provide a one-sentence summary at the end of each paragraph in Section 7 about the connection between the related works and this study.
10. The Conclusion section seems to be more consistent with the full manuscript if it is described in the present tense.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730088866444

thread_type:
official_review

<thread_object>
creator: reviewer_zyxb

<content>
summary:
This paper empirically studies catastrophic forgetting (CF) from the perspective of function vectors (FV) and makes the following core contributions:
1. It shows that there is a correlation between the change in function vector for a task and degradation in performance on the task.
2. It argues that CF is due to the inability to associate the input with the latent task vector rather than a degradation in the ability to solve a tasks conditioned on the input and correct latent task vector. It supports this with mechanistic interventions to the task vector. 
3. It presents a method to regularize FVs during training to mitigate forgetting.

strengths:
**Originality**: This was an interesting approach/hypothesis for studying forgetting that I quite enjoyed.
**Quality**: The authors perform a thorough series of experiments investigating their claims and evidence ranges from empirical measurements, mechanistic interventions, and training interventions. 
**Clarity**: The work is well contextualized in related work.
**Significance**: This work is a really interesting way to connect mechanistic interpretability to studying and mitigating forgetting demonstrating practical ways in which tools from mechanistic interpretability can be used to understand diverse phenomena.

weaknesses:
1. **Clarity**: The paper dense and the writing is terse and sometimes hard to follow. The figures have a lot of information on them, are very small and it is quite difficult to extract takeaways from them. Overall the paper is often hard to follow and it is hard to trace the evidence for the claims. A lot of points through the paper seem like observations and the connection to one of the core scientific claims isn't always clear. I think this paper would benefit greatly from better organization. Probably most important is to explain the figures/tables better *in the captions* and actually point out what the takeaways are/what we should learn from them/what points it is substantiating. I understand this is challenging because of page constraints and the amount of information presented in the paper: my recommendation would be to move anything non-essential to the core scientific claims to the appendix. Also, the authors could present the results for 1 model and add the rest of the models to the appendix given that the models are all of the same scale.
2. **FV and forgetting**: One of the key claims in this paper is change in FV is correlated with forgetting, i.e. this is a useful measure that "explains" the phenomena of forgetting. The evidence did not convince me of this: in Figure 2, there are points where FV similarity is high, 5 shot performance is high but 0-shot performance is low. Object Count especially is unconvincing (0shot and 5shot switch and don't seem to be correlated with FV).
3. **Mechanistic interventions**: While the mechanistic interventions in Section 5 are quite interesting, the paper does not consider any alternated explanations for this behavior (see questions). It is also not clear to me why FV is a measure of latent task identification.
4. **Limitation of training method**: If I understand correctly, this training method regularizes the FV of particular tasks and thus prevents forgetting of the particular tasks being regularized. It doesn't affect forgetting as a whole? While this is not a weakness, I think this should be clarified and the method should be explained as such.

questions:
1. Related to weakness 2: can you more concretely establish the FV similarity is correlated with forgetting, perhaps with a correlation plot? I want to better understand how much signal FV provides on forgetting as a whole.
2. Related to weakness 2: can you show how FV measures forgetting compared to other metrics one might consider. For example, after training on a task, does FV similarity to the previous task better predict forgetting of the previous task than L2 distance to the previous model?
3. Related to weakness 3: Why is FV a measure of latent task identification? What exactly is the argument for this?
4. Related to weakness 3: an alternate hypothesis for the results in section 5 is that after finetuning, only certain layers of the model have changed significantly while a lot of the other layers have remained unchanged. So when you make the activation intervention and search over the layers to find the optimal one, you have just undone the effect of finetuning in the activations by returning it to the behavior of an earlier model upto that later and the rest of the model isn't that different after finetuning so activations pass through and your recover the behavior of the earlier model. How would you refute this alternate hypothesis?
5. Related to weakness 4: did I understand this correctly? If so, can you make this clear/provide some evidence that if you train with the activations for task 1 regularized, it does not mitigate forgetting on task 2 (another unrelated task). If this is true, this is an important limitation that should be made clear (I want to emphasize that it isn't a limitation of the paper but rather would make the paper stronger be explaining how this method should be used).
6. A claim for the training method is that it does not effect plasticity. I would like to understand the limitations of this: can we finetune on a very large dataset of a new task with this regularization and still maintain plasticity? When does the regularization start hurting? What are the limits. One way you could do this is train on a FT dataset with millions of samples (maybe OpenMathInstruct-2) to see if this regularization limits learnability in extreme cases.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721145537

thread_type:
official_review

<thread_object>
creator: reviewer_kue1

<content>
summary:
The paper explores catastrophic forgetting in continual instruction finetuning across various settings, discovering that model forgetting is influenced by the specific training tasks and the models themselves. It explores correlation between function vectors and catastrophic forgetting, and  propose a function vector-based approach to eliminate catastrophic forgetting. The paper has conducted ablation studies across various tasks, methods and models to demonstrate the generalizability of the method using different llama and mixtral models with classification and generation tasks and evaluating it for supervised finetuning and model generalizability.

strengths:
1. The idea of using regularization on the function vectors and the function vector guided loss to enable continual instruction fine tuning seems novel and is well motivated in the paper. 
2. The paper presents a strong ablation study for the proposed methods across various settings, demonstrating their generalizability by conducting experiments using multiple llama checkpoints, classification and generation tasks and evaluating across generalizability and supervised finetuning on multiple datasets.
3. The experiment studies which clearly demonstrate the ability of  function vectors to significantly improve the performance on continual instruction fine tuning.

weaknesses:
1. There is a lack of clarity in the proposed methodology. i.e function vector guided training design. It would help to understand  the proposed methodology by providing the overall  the optimisation objective and end-to-end training algorithm. In the current state, it is difficult to reproduce the results that the authors mentioned in the paper without the details on the proposed methodology. 
2. The  presentation  of the paper and the readability of some sections are difficult to follow.   Notations are unnecessarily complicated. 
3. The experimental results though convincing across most of the experimental setups, its hard to make decisive conclusions especially on the core idea of function vectors and their effectiveness in mitigating catastrophic forgetting.
4. Some experiments and  observations made related to catastrophic forgetting are redundant and may not contribute much. For example, forgetting will be model-dependent since performance on different datasets and tasks is model-dependent.

More details on the above mentioned points are listed below.

1. The authors claim that “Forgetting coincides with changes in FV similarity between tasks”. However, it is difficult to observe the same from the evidence provided because the trend is not constant across all tasks and models; sometimes, a slight change in similarity changes the task performance drastically(i.e. Fig 2(a) on count object and hellaswag datasets), in some cases even though similarity is decreasing, the performance is increasing. It is very difficult to establish a clear relation from the empirical results provided. A suggestion is to  extend the number of tasks or conduct a statistical test using correlation metrics. 
2. The observation that generation tasks lead to greater forgetting cannot be measured directly, even though the same evaluation metric (Rouge) is used for both types of tasks. Rouge is more sensitive to longer sequences because it relies on n-grams overlap. The tasks have different difficulties; one way to measure them is by comparing them against some upper bound baseline and calculating forgetting in terms of percentage loss.
3. The computation of the FV guided loss (eqn 4) is not very clear. If the set S consists attention head with top-10 CE,  is that going to be same before training on the current task and after training. Is this loss recomputed at every iteration of the optimization step? 
4. Although function vectors improve the performance of existing CL methods, it can be helpful to compare against  finetuning models individually, to see how much gap there is between methods with function vector guided fine tuning and the optimal performance that can be achieved by fine tuning.

questions:
1. It is unclear in methodology of training with function vector guided loss. Since attention head will change at every iteration during training, when do you impose the KL divergence-based regularization. Since, it requires separate computation using ICL-based examples, how do you update model weights through backdrop?  Kindly provide more details on the FV loss computation. 
2. The FV-based method may require additional computational resources, especially during interventions where multiple layers are evaluated. Since to extract FV, one has to run the model on a set of counterfactual inputs to isolate activation patterns and then select top attention with the highest causal effect. The FV-guided KL-divergence loss aligns logits produced with and without FV interventions. This requires computing model outputs twice. There needs to be a discussion on the computational time and memory overhead required for the proposed approach. 
3. While training on a task, regularization is imposed just based on the most recently trained task or do you consider all previously trained tasks? The loss function in equation 4 and 5 may be modified to consider the state of training on a particular task in the sequence. 
4. There are some typos in the paper Line 107: bracket missing in citation, Line 970: it should be We instead of I, Line 231  : sequensce --> sequence.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730729855867

thread_type:
official_review

<thread_object>
creator: reviewer_g6jm

<content>
summary:
The paper investigates Catastrophic Forgetting in Large Language Models during continual learning, revealing that forgetting is influenced by specific tasks and model characteristics, and proposes a novel training methodology that utilizes a function vector to stabilize performance and reduce CF, with empirical validation across multiple benchmarks.

strengths:
* The forgetting issue during fine-tuning foundation models is an important problem.
* The observations and analysis through the function vector in this paper is very interesting, which I believe would be interesting to many.
*  The finding that CF in LLMs primarily stems from biases in function activation rather than the overwriting of task processing functions is not obvious, but I think the authors has done a good job in confiming this. 
* The writing is easy to follow.

weaknesses:
Overall, I like the findings in the paper and the authers have done extensive investigations. I am interested in comparing the proposed method with model averaging, which is shown in [1] that model averaging is very effectivenss in terms of mitigating forgetting. Furthermore, is it possible to analyze the effectivenss of model averaging through function vector? 


[1] Yong Lin, et.al., Mitigating the alignment tax of RLHF.

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1732380095731

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear Reviewers and ACs,

We sincerely thank all the reviewers and ACs for your diligent efforts and high-quality reviews. If you have any additional questions or require further clarification, please feel free to let us know. Your insights are highly valued.

We are delighted to note that reviewers find that:

- Our hypothesis and method are novel, insightful (Reviewers `KUE1`, `Qiar`), interesting (Reviewers `G6Jm`, `ZyxB`), well-motivated (Reviewer `KUE1`), thorough (Reviewers `Qiar`, `ZyxB`), and presented in a clear and easy-to-follow writing style (Reviewers `G6Jm`, `Qiar`).
- Our observations and analysis on forgetting through the function vector are very interesting (Reviewers `G6Jm`, `ZyxB`), strong, thorough (Reviewers `ZyxB`, `KUE1`), and have significantly contributed to the field (Reviewer `Qiar`).
- Our paper studies the forgetting problem, which is important (Reviewer `G6Jm`). It concludes that CF in LLMs primarily stems from biases in function activation rather than the overwriting of task processing functions, a finding that is well-received (Reviewer `G6Jm`) and insightful (Reviewers `Qiar`, `ZyxB`). It also provides an effective approach based on the function vector to mitigate forgetting (Reviewer `KUE1`).

In response to your valuable suggestions, we have conducted additional experiments and made the following modifications in the Rebuttal-PDF for your convenience:

- Table 1: We updated the results for NI-Seq-G1 on Llama2-7b-chat to correct a previous setting error in the hyperparameters.
- Figure 3: We added intervention results with FV from an unrelated task (suggested by Reviewer `ZyxB`).
- Figure 4: We included a comparison with the original fine-tuning process.
- Figure 5: Illustration of the causal pathway to forgetting (suggested by Reviewers `KUE1`, `ZyxB`).
- Figure 6: Correlation plot on model performance and different similarity metrics (suggested by Reviewers `KUE1`, `ZyxB`).
- Table 8, Figure 8: Analysis and performance of Model Averaging (suggested by Reviewer `G6Jm`).
- Figure 9: Intervention results on four datasets via function vector on pre-trained models (suggested by Reviewer `ZyxB`).
- Algorithm 1: Procedure for function vector-guided continual learning (suggested by Reviewer `KUE1`).
- Loss function: In lines 410-429, we updated the loss with more concrete notations (suggested by Reviewer `KUE1`).
- Captions for Figures 1-10: We updated the captions to highlight the 'Main conclusion' along with its substantial supporting data (suggested by Reviewer `ZyxB`).
- Related work: We added a one-sentence summary to connect the related works with our study (suggested by Reviewer `Qiar`).
- Notation shape: In lines 127-130, we clarified the shape of the notation used in our paper (suggested by Reviewer `Qiar`).
- Due to space limitations, we moved Table 1 and Figure 3 in the old version of the paper to Table 5 and Figure 7 in the Appendix.

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors study the use of Function Vectors (Todd et al., 2023) in the context of catastrophic forgetting. They show that: 
1) Task similarity, as measured by the cosine distance between the respective function vectors of each task, is correlated with the amount of forgetting; 
2) By intervening on the trained model using function vectors of previous tasks, they can mitigate the performance drop due to forgetting;
3) By training the model with additional regularization that depends on the function vectors of previous tasks, they can also mitigate forgetting without test-time intervention. The high-level idea is that function vectors can be kept around as compact representations of previous tasks (instead of keeping all of the previous training data around), and used for regularization to avoid forgetting.
Overall, reviewers found the study and proposed method exciting, and unanimously recommended acceptance. I recommend acceptance as well and I agree that this paper will be interesting and useful to the community!

There were two broad concerns raised. First, several reviewers pointed out that the writing was terse and hard to follow, with various grammatical errors, typos, and missing details. The authors addressed this partially in their revision, though the revised version still contains several writing issues (especially when introducing the details of Function Vectors in Section 2.2 and the evaluations in Section 3). Second, several reviewers pointed out places where the conclusions the authors drew were not rigorously supported by the experiments (e.g., in Section 4, which covers the correlation between function vector similarity and forgeting). The authors responded well to these comments by adding more experiments to the appendix. I encourage the authors to significantly update their camera-ready version to incorporate the reviewers' feedback, make the paper clearer to readers unfamiliar with function vectors and prior work on catastrophic forgetting, and reorganize the paper such that the new results are also represented in the main text. I think this will help the broader community appreciate and build on this good work.

As a final suggestion, I might have missed it, but I didn't see an ablation of the two components of the new proposed loss (equations 4 and 5). That, and the comparison to model averaging that was in the rebuttal, will help us better understand what about the proposed regularization works well