Kati8/ICLR_2023_2025 huggingface dataset:
authors: Gangwei Jiang
authors: Caigao JIANG
authors: Zhaoyi Li
authors: Siqiao Xue
authors: JUN ZHOU
authors: Linqi Song
reviews: 
Comments: This paper studies the catastrophic forgetting (CF) problem in LLMs with the tool of Function Vector (FV). The research starts with reproducing the CF problem on a series of four tasks across 4 publicly available LLMs. In addition, the researchers study the differences in FVs between continue-trained and untrained LLMs on each task. They show that there is a strong correlation between the differences in FVs and the CF issue. Their theoretical analysis posts a hypothesis on the bias of the FV θT elicited by the input x, (i.e., p(θT|x)). The authors further conducted several experiments to well support their hypothesis. Finally, they designed a new loss for alleviating the CF problem in continuing learning, where they encourage LLMs to minimize the differences between model prediction and the FV-intervented prediction while keeping the hidden representations of X the same as un-trained ones.
Strengths: 1. The manuscript provides a thorough study of the catastrophic forgetting problem. 
2. The most insightful contribution of this work is posing a new hypothesis on this problem. 
3. Its technical contribution is significant as the authors first introduce the Function Vector as a tool in this community.
4. The paper is well-written and well-organized.
Weaknesses: I cannot see any significant weakness. One nitpick would be expanding the background of Function Vector in Sec. 2.2, especially providing the dimension or shape of each notation. It will help the audiences understand the procedure for computing FVs more easily.
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4
reviews: 
Comments: The paper investigates Catastrophic Forgetting in Large Language Models during continual learning, revealing that forgetting is influenced by specific tasks and model characteristics, and proposes a novel training methodology that utilizes a function vector to stabilize performance and reduce CF, with empirical validation across multiple benchmarks.
Strengths: * The forgetting issue during fine-tuning foundation models is an important problem.
* The observations and analysis through the function vector in this paper is very interesting, which I believe would be interesting to many.
*  The finding that CF in LLMs primarily stems from biases in function activation rather than the overwriting of task processing functions is not obvious, but I think the authors has done a good job in confiming this. 
* The writing is easy to follow.
Weaknesses: Overall, I like the findings in the paper and the authers have done extensive investigations. I am interested in comparing the proposed method with model averaging, which is shown in [1] that model averaging is very effectivenss in terms of mitigating forgetting. Furthermore, is it possible to analyze the effectivenss of model averaging through function vector? 


[1] Yong Lin, et.al., Mitigating the alignment tax of RLHF.
Recommendation: 8
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This work investigates catastrophic forgetting (CF) in large language models using Function Vectors (FV) as an analytical framework. The authors begin by demonstrating the CF phenomenon across multiple tasks on four different publicly available LLMs. They then examine how FVs differ between models that have undergone continued training versus untrained models for each task, revealing a robust relationship between FV divergence and the occurrence of CF. The paper presents a theoretical framework proposing that the bias in FV θT conditioned on input x (i.e., p(θT|x)) explains this phenomenon. Multiple experimental validations are provided to substantiate this theoretical perspective. To address the CF challenge in continual learning scenarios, the authors propose a novel training objective that reduces the disparity between standard model predictions and FV-adjusted predictions while preserving the hidden representations consistent with the original untrained model.
Strengths: 1. The manuscript delivers a comprehensive examination of the catastrophic forgetting challenge. 2. The most valuable contribution lies in introducing an original theoretical perspective on this problem. 3. The technical novelty is substantial given that this represents the first application of Function Vectors as an analytical tool within this research community. 4. The manuscript demonstrates excellent clarity in presentation and logical organization.
Weaknesses: No substantial limitations are identified. A minor suggestion involves enhancing the exposition of Function Vectors in Section 2.2, particularly through explicit specification of the dimensional properties and structural characteristics of each mathematical notation. This enhancement would facilitate improved comprehension of the FV computation process for readers.
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4
reviews: 
Comments: This work investigates catastrophic forgetting (CF) in large language models through the lens of Function Vectors (FV). The authors begin by demonstrating the CF phenomenon across multiple tasks using four different publicly accessible LLMs. Subsequently, they examine how FVs diverge between models that have undergone continued training and those that have not, analyzing this divergence for each task. They establish a robust relationship between FV divergence and the manifestation of CF. The paper proposes a theoretical framework with a novel hypothesis regarding the conditional distribution of FVs given input (p(θT|x)). This hypothesis receives substantial empirical validation through multiple experiments. To address the CF challenge, the authors introduce a modified training objective that incentivizes alignment between model outputs and FV-adjusted predictions while preserving the original hidden representations from the pre-training phase.
Strengths: 1. The paper delivers an in-depth examination of catastrophic forgetting in the LLM domain. 2. The principal innovation lies in introducing a fresh theoretical perspective on this phenomenon. 3. A notable technical advancement is the application of Function Vectors as an analytical instrument for studying this problem. 4. The manuscript demonstrates clear exposition and logical structure throughout.
Weaknesses: No major deficiencies are evident. A minor suggestion would be to enhance the discussion of Function Vector fundamentals in Section 2.2 by explicitly specifying the dimensionality and structural properties of all mathematical symbols. This enhancement would facilitate reader comprehension of the FV computation workflow.
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4