Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuning
Authors: Gangwei Jiang
Caigao JIANG
Zhaoyi Li
Siqiao Xue
JUN ZHOU
Linqi Song
Defu Lian
Ying Wei
Venue: ICLR 2025 Oral (2025)
Keywords: Catastrophic forgetting; Large language model; Instruction tuning
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Catastrophic forgetting (CF) poses a significant challenge in machine learning, where a model forgets previously learned information upon learning new tasks. 
Despite the advanced capabilities of Large Language Models (LLMs), they continue to face challenges with CF during continual learning. The majority of existing research focuses on analyzing forgetting patterns through a singular training sequence, thereby overlooking the intricate effects that diverse tasks have on model behavior.
Our study explores CF across various settings, discovering that model forgetting is influenced by both the specific training tasks and the models themselves. To this end, we interpret forgetting by examining the function vector (FV), a compact representation of functions in LLMs, offering a model-dependent indicator for the occurrence of CF. Through theoretical and empirical analyses, we demonstrated that CF in LLMs primarily stems from biases in function activation rather than the overwriting of task processing functions.
Leveraging these insights, we propose a novel function vector guided training methodology, incorporating a regularization technique to stabilize the FV and mitigate forgetting. Empirical tests on four benchmarks confirm the effectiveness of our proposed training method, substantiating our theoretical framework concerning CF and model function dynamics.

Published as a conference paper at ICLR 2025
UNLOCKING THE POWER OF FUNCTION VECTORS FOR CHARACTERIZING AND MITIGATING CATASTROPHIC FORGETTING IN CONTINUAL INSTRUCTION TUNING
Gangwei Jiang Caigao Jiang', Zhaoyi Lite, Siqiao Xue*, Jun Zhou',
Linqi Song"*, Defu Lian"*, Ying Wei
University of Science and Technology of China,
2 City University of Hongkong
3 Zhejjang University, IIdeependent fgwiiang, 1=zhaoy1777/emalll
edu. en, Sangi.son@ecityt edu
1 Linndefuuuste. edu. en, sing.weile wuw..u.uuuuu..... edu.cn
ABSTRACT
Catastrophic forgetting (CF) poses significant challenge in machine learning, where a model forgets previously learned information upon learning new tasks. Despite the advanced capabilities of Large Language Models (LLMs), they con- tinue to face challenges with CF during continual learning. The majority of ex- isting research focuses analyzing forgetting patterns through single train- ing sequence, thereby overlooking the intricate effects that diverse tasks have model behavior. Our study explores CF across various settings, discover- ing that model forgetting is influenced by both the specific training tasks and the models themselves. To this end, interpret forgetting by examining the function vector (FV), a compact representation of functions in LLMs, which offers model-dependent indicator for the occurrence of CF. Through theoret- ical and empirical analyses, we demonstrate that CF in LLMs primarily stems from biases in function activation rather than the overwriting of task process- ing functions. Leveraging these insights, propose novel function vector guided training methodology. incorporating a regularization technique to stabi- lize the FV and mitigate forgetting. Empirical tests four benchmarks confirm the effectiveness of our proposed training method, substantiating our theoretical framework concerning CF and model function dynamics. Our code is available wwmluu.ccom//// http://tthtt..co/////eeeeeeeeeea /tte/////a/////////oooooooaaa/a/
INTRODUCTION
Continual instruction tuning Peng et al. 2023 Chung et al., 2024) has emerged as indispens- able ingredient in the development of Large Language Models (LLMs) (Brown et al. 22220 Radford Touvron et al. 2003b). enabling them to meet the demands of specific domains (Roziere
al.
2019
et al.
2023 Thirunavukarasuu et al. 2023 Xue et al. 2024) and human preferences (Ouyang et al.
However, a notable concern with such continual tuning is "catastrophic forgetting" (Mc- Closkey & Cohen 1989 Kirkpatrick et al.) 2017), where models may lose essential skills (Dou jet al. (003) et al. 2023) such as mathematical reasoning while adjusting to user instructions. While instruction tuning effectively evolves LLMs, it's critical to characterize and mitigate forget- ting within these models.
2022)
Research LLM forgetting (Luo et al., 2024 Wang al. 2023c Wu al. 2024a) generally
amines alterations in specific abilities like reading comprehension, factual know knowledge ledge, mathematical reasoning skills, and so on, underscoring the universal existence of catastrophic forgetting. As they have primarily studied from single training sequence, they fail to establish the connection between model forgetting and the characteristics of training data. Concurrently, there notable gap in understanding the internal mechanisms that underlie model forgetting. To date, only limited body of research has ventured into this area; notably, the work of Kotha et al. (2024), proposing the task
"Corresponding authors

---

Published as a conference paper at ICLR 2025
inference hypothesis, explores how conflicts between task processors lead to forgetting. Neverthe- less, the existing literature still struggles to track the internal mechanisms behind forgetting, which is crucial for understanding why and when forgetting occurs in language models after learning new tasks and how to avoid it.
In this study, we conduct thorough experiments various continual instruction tuning benchmarks covering multiple language models, task sequences, and evaluation metrics. Our investigation fo- cuses on the research question: When does forgetting happen? The experimental results suggest that model forgetting is a complex outcome of various factors, including the nature of training and test tasks, and the state of the model. However, traditional methodologies for evaluating task similarities to characterize forgetting -uchh as those based on feature similarity (Ramasesh et al.) [2020] Lee al., 202T) and readout similarity (Lee al. ( to overlook the distinctive task-related information inherent in different models. Meanwhile, purely model-dependent measurements, like the 1.2 distance of model parameters after being fitted on a new task (Lin et al., [023] Evvrn al.)
2024), necessitate training to compute task similarity. We identify critical gap in the availability of robust tools to dissect and understand the processes underlying forgetting.
To this end, we utilize the Function Vector approach (Todd et al., 2023), method grounded in mechanistic interpretability (Wang et al., 2023a| Bills et al. 2023). which represents the input- output function within a model into compact vector form. Our analysis begins with revisitation of the theoretical formulation of FV through the perspective of latent variable models (Baum & Petrie 1966 Gruber al., 2007), and establishes that the FV serves traceable latent variable in LLMs. We then examine model forgetting through the Function Vector perspective, successfully identifying occurrence of forgetting by evaluating the similarity between training and testing tasks (Sec. 3 Subsequent empirical investigations lead us to conclude that the fundamental driver of forgetting is the shift in the mapping from the input x to the latent concept variable 6, rather than erasure of previously learned task functions (Sec. 4
Based on our analysis, we conclude that minimizing the shift in the function vector during train- ing serves as key strategy for mitigating forgetting. We propose a function vector guided training mechanism as a simple yet efficient design for mitigating forgetting. This approach involves limiting changes to the function vectors associated with training tasks through regularization term, coupled with the adoption of a function vector vector-guided Kullback-Leibler (KL) divergence loss. This loss function aims to diminish the discrepancies between logits derived from zero-shot input and those adjusted by function vector intervention, ensuring the fine-tuned model remains consistent with the inner task function. Validated across multiple datasets and models, this method significantly allevi- ates forgetting in both general and in-context learning abilities, confirming the correlation between FV dynamics and forgetting.
Main findings and contributions. (1) We investigate catastrophic forgetting in LLMs covering multiple language models, task sequences, and evaluation metrics, discovering that forgetting in LLMs is highly model-dependent, and assert a new analytical tool for characterizing forgetting behavior. (2) Using empirical and theoretical analysis based on the function vector framework, we reveal that forgetting generally results from the activation of biased model functions rather than overwriting previous functions. (3) We develop a function vector guided training approach that preserves and aligns function vectors during fine-tuning, significantly improving both general and in-context learning across multiple continual learning datasets.
2 PRELIMINARIES
2.1 CATASTROPHIC FORGETTING
Continual learning (Serra et al. 2018: Wu et al. 2024c b) seeks to tackle the core challenge of
incrementally learning from a sequence of real-world tasks over time, specifically addressing how to adapt to new tasks without forgetting previously learned knowledge
phenomenon widely
known as catastrophic forgetting (McCloskey & Cohen 1989 Kirkpatrick et al. 2017).
In this paper, focus on continual learning of language model Mo. which has already been
pre-trained on a vast data corpus PrT using language modeling tasks (Brown et al, 2020 Radford et al. 2019) followed by preference optimization via human feedback (Ouyang et al.
2022
Specif-
2

---

Published as a conference paper at ICLR 2025
ically, we assume stream of tasks T......... .N,, where each j-th task Т3 consists of dataset Dj (#) v3},, with zr and v) representing the inputs and outputs text sequences, respectively. On each task T, the model M,-1 is optimized towards minimization of the loss LT, (Mz-1), coupled with a continual learning objective if applied, resulting in the updated model Mj. This continual learning process, applied into a language model, is commonly referred to as continual instruction tuning (Peng et al. 2023 Chung et al. 2024).
2.2 FUNCTION VECTOR
Following the mechanistic interpretability work in LLMs (Todd al. 2023 Hendel et al. 2023
we investigate the internal workings of a task on LLMs through function vector compact Vector representation of input-output task identified within the hidden states of transformers during in- context learning (ICL (Brown et al. ).20)). An activation patching (Meng et al. [022] 2023 Wang et al. 2023a) procedure is performed the hidden states to determine the casual set of attention heads that mediate tasks. These heads collaborate to convey function vector (FV) which represents the LLM's function specific input-output task. Function vector is regarded as an efficient way to characterize function execution within LLMs (odd et al., (023).
Formally, for a given dataset DJT of task T, the function vector 0r is derived through two steps:
First, the activation patching is performed to determine the attention heads set (denoted as S) with significant -nffeeend-effect relationship between ICL inputs and correct outputs. Specifi- cally, the model will run on a counterfactual ICI. input [ ær] incorporating a label-shuffled prompt á [[11, . 01) ..., (Ii, Un)], which typically leads to incorrect outcomes. Then the representation at specific head for [p, æ]] is substituted by the averaged task activation hif and calculated its causal effect (CE) on the model's output.
([) -M (v [+) Par I x))
Here, 1i E Rd symbolizes the mean activations of in-context learning input for task T at the last token position across layer and head k. with d being the dimension of the layer output as hTe heady, W is the equivalent of the attention head output in the residual stream (Elhage et al, 2021). Mhū-hin denotes the model with replacement operation on attention head (1, k) at last token. A higher CE implics that the specific head's ' state is critical for accurate predictions which encodes of more task-relevant information. In this paper, S is the attention heads with top- 10 CE.
Second, function vector Or is assembled by summing up the averaged ICL inputs activation from the attention heads within S, formulated as Or = Ea.,es hij E RR., The comprehensive methodology
for this extraction process can be found in the Appendix [F]
In this paper, we revisit the definition of FV and study its dynamics before and after learning a new task, providing surrogate lens to uncover the inherent mechanisms of forgetting in LLMss
3 CATASTROPHIC FORGETTING OF LLMS
This section empirically examines the research question of when forgetting occurs during continual instruction tuning, specifically in relation to task types, training stages and language models.
Datasets. We build the task sequences for continual instruction tuning using SuperNI (Wang et al. 2022), collection of NLP tasks with expert-written instructions that are unseen to the pre-trained model. SuperNI is commonly used for assessing cross-task generation and conflict after fine-tuning language models. To investigate the relationship between forgetting and task types. i.c., generation and classification, we design six task sequences. These sequences consist of pure generation tasks, pure classification tasks and mixed sequences containing both generation and classification tasks. Their main information is listed in Table with additional details available in Appendix D
We adopt the following five metrics to quantify various aspects of forget- No a which is the average zero-shot performance across No general N= Σド= evaluation tasks after instruction tuning the final N-th task. Here, of, denotes the zero-shot performance on task 9 after sequentially tuning the m-th task, and T refers to the j-th general eval- uation task. Performance is uniformly measured using Rouge-I. (Lin 2004), where classification
Evaluation metries.
ting: (1) GP 1.
3

---

Published as a conference paper at ICLR 2025
Table 1: Final performance 3 SuperNI benchmarks language models. Hella., Com., Alpa. and Ob. denote evaluation score on Hellswag, CommonsenseQA. Alpaca, Object Count datasets, respectively. The A value in red bold style compared performance of their initial model Mo- Higher Forget or lower A represents more forgetting. Main conclusion: Forgetting consistently occurs in both general and newly learned tasks, showing considerable variations depending on the types of lasks, stages of training, and the specific language models involved.
Zero-Sho Perfomaance in General Task
In-Comext Performance General Task
Performance Traindd Tonk
Com. Alpa.
1/A
Holla
Com.
Alpa
Oh
1/A f
AP
Forget
Lluma2- 7b-ch27
Me
57.37
42.22
58.95
57.89
35.17
34.21
46.55
N1-Sce
47.37
40.00
32.00
-4.48
24.21
26.84
NI-Soq-Gii
48.95
39.47
27.
59.72
-3.35
37.89
42.63
28.84
38.95
-9.48
24.96
19:35
5.61
M1
52.11
42,83
31,09
29.51
-3.39
45.79
31,05
24.58
33.16
-12.91
59.02
$4.32
4.69
Hamaa--0--hat
Mg
81.58
58.42
22.64
40.04
50.67
85.26
63.16
27.42
49.47
56.32
7
1
N1-Seg-CI
79.47
46.88
25.27
32.32
-5.20
79.47
40.00
25.02
45.79
-8.60
83.40
82. 10.10
1.30
Sec GL
72.63
35.79
22.05
29.39
-10.70
67.89
31.05
19.77
41.58
-16.26
28.29
21.09
7.19
N1-Saq-M1
78.42
40,00
21.93
21.58
-10.19
76.84
21.32
35.91
-12.81
60.74
52.62
Mistral-7b-insttruc
Me
73.68
24.74
5,02
40.86
79.47
66.32
32.36
37,89
54,01
NI-Soq-C1
63.16
50,00
52.04
15.3
-0.75
66.84
51.05
36.8
37.89
-5.57
$4.70
-0.70
N1-Sc--Gi
37.57
45,26
26.3
13.81
-5.18
57.89
35.79
32.04
39.47
-12.71
27.62
19.77
7.85
NI-Soj-M1
65.26
47,89
33.02
12.35
-1.23
63.68
38.42
34.79
45.79
-8.34
16.96
$7.00
4.95
L 13b-chht
Mg
69.47
51,05
28.99
15.09
41.15
75.26
57.89
35.46
43.16
52.94
1
1
Seg C1
65.79
52.83
34.18
21.51
+2.38
66.32
48.42
38.48
38.95
-4.90
83:20
82.26
0.93
NI-Soq-Git
38.95
-5.13
65.79
34.21
-12.18
18.16
7.47
N1-Soq-M1
71.58
49.47
34.30
28.09
+4.66
70.53
36.51
37.37
56.33
3.76
2024). We set No = 4 gen- eral evaluation tasks, covering Hellaswag (Zellers et al., (019)). CommonsenseeA (Talmor et al.]
accuracy equals Rouge-L, with output post-processing (Zhao et al.
2018). Alpaca (Hendrycks et al., 2020), and BBH-Object- Count (Srivastava et al. 2022). and this N setting also applies to 4 and is extended to No 7 in Sec. 6 (2) IP No Σj= áN which the average in-context performance on No general evaluation tasks after tuning on the last N- th task. The in-context performance, a% is conditioned on the n-shot ICL input æp, where p [[1,, v1), (kn, with n = 5 for this work. (3) FP = N Σj= aN which is the average zero-shot performance across all N instruction tuning tasks after tuning on the final V-th task. Tj represents the j-th instruction tuning task in the sequence. AP t ΣØ= a which is the aver- age zero-shot performance when learning every j-th instruction tuning task. (5) Forget which has been widely adopted Wu et al. (2022); Ke et al. 2023) to measure forgetting. More detailed information about the datasets and evaluation metrics is presented in Appendix D
-P.
Instruction tuning setup. For each task in the sequence, we continually lin-tune four lan- guage models, including Llama2-7b-chat, Llama2-13B-cha (Touvron al. (023b), Llama3-8B- chat (Dubey et al.] 2024). Mistral-7B-inntruuce
(Jiang et 2023) using the causal language model loss (Radford al. 2019). Unless otherwise noted, we use the LoRA fine-tuning ap- proach (Hu et al. 2021). employing the Adam optimizer with a learning rate of le and a batch size of 32. Additional implementation details can be found in the Appendix
We report the results on continual learning of four language models on three SuperNI sequences in Table I highlighting that forgetting occurs across general ability, in-context learning ability, and fine-tuned ability. Higher scores for general and in-context abilities signify better mitigation of forgetting. For fine-tuned ability, we report the values of AP, FP and their difference Forget, where a lower Forget value indicates less forgetting. A detailed breakdown of the results for the other three SuperNI sequences is provided in Appendix F Figure illustrates Dillutraaes the forgetting of the Llama2-7b and Llama3-8b models during continual learning on NI-seq-C1 and NI-seq-GI. The vertical axis signifies the training stage, where Mi denotes the model after completing the i-th task. The horizontal axis displays test performance across tasks, with the performance of MO (without continual instruction tuning) shown at the top. The heatmaps show the relative shifts in performance compared to MO, with declines indicated by bluer values and improvements by redder colors.
4

---

Published as a conference paper at ICLR 2025
Geneall Teik Rollatiin
18.6
2822
13.0
38.6
26.4
SALE 97.4 B 23.2
[ 57.9
222 242
21,4
15.5
16.2
28,4
00
se
22,
aqaa
Ofl
#5.2
ZPA
45,5
-10
M2
47
100
67
92
100
82
42
74
45
42
35
100
69
220
53
11
54
43
22
88
320
70
78
52
M4
102
116
122
72
46
94-
39
93
323
100
54
49
100
146
45
Taaana---eccttt S1
101 bmma--b--ter Ni-Seydi
(1)
80.0
90.5
47.8
83.0
65.3
57.4
26.5
27.3
663
57
35.7
24
[1)
92.0 M8.5
$7.0
83,5
W5.8
00
83.6
58.4
22.6
40,0
85.3
K52
27A4
60,5
M2
300
115
97
IDC
EX0
313
100
-10
M2
93
100
09
X20
134
20
CE
90
32
193
100
300
315
53
M4
90
100
93
04
A13
117
93
40
37
100
220
165
97
M4
350
124
197
100
100
82
100
300
100
123
117
75
100
[B] Lkmmma2-0 26 chat Co
Ul Mi
Figure| 1: Performance heatmap during continual learning of 2 different sequences on Llama2-7b chat and Llama3-8b-chat. The numbers above the heatmap indicate the baseline performance of each task, with the performance of the pre-trained model for general testing (c.g.. in a-(II) 66.3 is the score of Commonsense current task for trained task testing (e.g., in a-(I) 28.2 is the score of T2 on Llama2-7b-char post 2-th task training). The numbers on the heatmap show the percentage change relative to the baseline (e.g.. in a-()) the first column 47 indicates the score at 38.6 38.6*47%) Main conclusion: (1) Learning generation tasks (alc) vs. classification tasks (b/d) leads to more forgetting. (2) Forgetting may reduce naturally (I-(II)/-III);; (3) Forgetting is model dedee-deepeneett (a/b vs. c/d).
original Alama2--b-chall and performance right after completing
From Table and Fig. 1 we observe two major trends: (1) consistent forgetting of LLMs across both general and newly acquired tasks, irrespective of the task sequence type, and (2) considerable variability in forgetting across evaluation tasks - e.g., Hellaswag appeals more prone to forgetting than Alpaca. Beyond these general findings, we further investigate how forgetting is influenced by task types, training stages, and the specific language models employed.
Task type perspective: generation task sequences lead greater forgetting. Upon analyzing the performance of NI-Seq-C1 (classification tasks) and NI-Seq-G1 (generation tasks) in Table [ we observe a noticeable increase in forgetting when learning generation task sequences. For instance, the Llama3-8b-chat model shows larger performance declines with NI-Seq-G1 (10.7, 16.2, and 7.2 in general, in-context, and trained tasks, respectively) compared to NI-Seq-CI (5.2, 8.6, and 1.3 declines). Additionally, the forgetting score Forget of all models over all tasks in the NI-Seq-CI sequence is below 2.3, suggesting a stronger ability to mitigate forgetting for LLMs than small language models (Qin & Joty 2022] Raadaibiedina et al. (023). We further evaluate LLMs with more sequences, obtaining similar observations as shown in Appendix
Training stage perspective: forgetting may naturally mitigate during training. Figure 1 illus- trates the extent of forgetting at each training phase across various evaluation datasets. Contrary to the nearly consistent performance drop seen previous studies (Luo et al., 2024 (Wu et al.) 2022). we frequently observe phenomenon where performance initially decreases but later rebounds. For instance, the fourth column of (a)-(II) shows the "Ob." test score dropping to 67% on the M2 model; however, after two stages, the performance leaps to 122%. This contradiction further raises questions about how sequential fine-tuning on new tasks impacts the internal capabilities of LLMs, allowing them to recover previously forgotten capabilities.
Model perspective: forgetting is medel-de We compare forgetting between LLMs, specifically, for the "Ob." task performance shown in (a)-(II) and (c)-(II), continual instruction tuning of Lllama-7b-chhe demonstrates a performance increase of up to 146% relative to using the Llama2-7b-chat itself, while that of Llama3-7b-chan shows a decrease to 73%. This suggests that forgetting is not only task-related but also heavily influenced by poodel-dependen factors such model size, architecture, and the diversity of pre-trainn mre-training data. These factors shape each model's s unique capacity for tasks, revealing that the mere feature similarity between tasks (c.g., hidden states in the last layer) is insufficient predict model-de forgetting patterns (see Appendix F..
5

---

Published as a conference paper at ICLR 2025
4 CORRELATIONS BETWEEN FUNCTION VECTORS AND FORGETTING
The previous section prompts a more effective measure for characterizing catastrophic forgetting, surpassing those traditionally used in continual learning research with small models, such as fea- ture similarity (Ramasesh et al., 2020, Lee et al. 2021) and readout similarity (Lee et al. 2021) between tasks. They fail overlook the distinctive task-related information inherent in different models. Other model-dependent measures, such as the /2-distance of model parameters after tuning on new tasks (Lin et al.) 2023) Evron et al. 2024). In addition, we have proven that these traditional measures (e.g., feature similarity and (2-distance) are loosely correlated with forgetting in LLMs in Fig. In this section, we first establish that the similarity between function vectors (FV, see Sec. is strongly correlated with diverse forgetting patterns across task types, training stages, and language models.
Forgetting coincides with changes in FV similarity between model updates. We now explore the relationship between forgetting and variations in function vectors across evaluation tasks. As the four general evaluation tasks throughout the
shown in Figure 2 we evaluate the performance sequence for both generation and classification settings, alongside shifts in function vectors 0r.. "Fv sim" in the diagram refers to Cosine 0(.-). where 0/-- is the FV of evaluation task T after finettuning the j-th task. We observe clear consistency between the performance decline and variations in function vectors. Specifically, as performance drops, the similarity between FV 04- and FV By generally declines, whereas this similarity tends to increase as performance recovers. For example, in the Hellaswag task within NI-Seq-GI (top right subplot in Figure the correlation coefficient (R² value) between zero-shot performance and proposed similarity measure reaches 0.873. This finding underscores that fluctuations in the FV often coincide with model forgetting, and justifies the rationale of characterizing forgetting through function vectors.
Commeroeneett
Hel Hellesey
CommoneneagA
Hollowwag
0.5
M6
MC
43
M5
Mo
MC
90
M3
90
93
ND
KG
14
NO
ND
KO
M5
Alpaca
Object
Oblect
Sabol
Lamaa Todarr Miday
Uoraa Ta NiGeecC
Figure 2: The shifts in function vector with 0/5-shot performance during tuning. The bar chart corresponding to the left y-axis shows the similarity of function vectors to their initial state. The line graph corresponding to the right y-axis depicts the model's Rouge-L metric on test data. Main conclusion: There exists significant correlation between performance (line data) and FV similarity (bar data). The correlation plots with more data point are provided in Fig.
5 CAUSAL PATHWAY TO FORGETTING THROUGH FUNCTION VECTORS
Before delving into how function vectors causally influence forgetting, we revisit the function vector Or from the perspective of latent variable models (Baum & Petrie 1966 Gruber et al., 2007). The specific FV 0r in Sec. 2 is derived from the sum of activations of certain attention heads in the LLM via causal analysis, when provided with ICL input of task T. Og represents latent variable descriptive of task T; conditioning predictions on fT produces model function specific to task Tic..
Par Or = Σ hux) + fr(uxx)
(LA)ES
Here, fr characterizes the function of task T' within the model, where different function vectors activate distinct functions, enabling the model to exhibit diverse abilities.
Prior research (Xie et al.) 2021; Wang et al. 2024a) has established that under latent variable assumption, in-context learning in LLMS can be rewritten as:
PM (y) of vf ..... Fn
T T
T PM 0M (0 oT. VT ... In Un x) d0, (3)

---

Published as a conference paper at ICLR 2025
where PM denotes the predictive probability of the LLM M, and f is potentially high dimensional latent variable within the space O. For example, in the task of predicting the antonym (ÿ) of a word (x)), the latent variable could correspond to "writing the antonym of the word" (0). This framework indicates that ICL boosts performance by deducing the correct 0 from observed input-output pairs.
By comparing this framework with Eq. 2 conclude that LLM predictions hinge on two critical factors: first, the prediction given by the model's task-specific function PMr (3)(z, 0r), and second, the capacity of an input x to accu- rately trigger its corresponding Function Vec- tor 0r (i.e., (0r (z))). Previous work in continual learning has considered the role of task-specific funetions (a.k.a, model parame- ters) on forgetting, sparked methods such as gradient orthogonalization (Lopez-Paz & Ran- zato 2017: Chaudhry et al.) 2018) and parame- ter regularization (Kirkpatrick et al. 2017; Wu ee al. (024c)) to mitigate its impact. However,
Hellsswag
Hellaswag
-15
50
35-
Tiugrt
35
Tuiring Stage
Object Count
Object Count
Rougett
our research hypothesizes that in continual in-
struction tuning of LLMs, the intrinsic cause of forgetting is the bias of the latent variable Or elicited by the input x, as supported by the fol- lowing empirical findings.
M4
142
5/3
100
M4
142
N42
M.
104
15
Thaning Stage
Thaining Stage
[e] Llama--7b-chat Ni-Seq-G1
[b] Lbmaa2-7bb-h on NI-Seq-CI
Intervention results fine-tuned model. '+ Source FV' and '- Target FV' refers to Evidence I and Evidence II, respectively. Main intervention with related function vector mitigating forgetting.
Figure 3:
Finding I: forgetting is mitigated by adding the source FV with respect to the evaluation task. For each evaluation task T we con- ducted an intervention experiment during con- tinual instruction tuning by adding the source function vector, i.e., 04 into the LLMs (blue line in Figure 3). Here, the source FV 04. is extracted from the original LLM Mo following the procedure described in Section 2 The intervention involves adding the function vector specific layer during inference (hy h4 e.. and the reported re- sults reflect the best performance obtained by iterating over layers [3. 6, 9, 12, 15]. The findings indicate that integrating the source FV 04.. into inference which explicitly transforms Pas (y)(a) to Pis (yl lr, 0g.-. substantially restores model performance. For instance, Figure 3(a) shows an aver- age performance increase of 7.3 on Hellaswag through this intervention. This improvement signifies that explicitly injecting the latent variable 09. helps identify the function specific to the evaluation task T Pas (f)x,,00))) thereby mitigating forgetting.
conclusion:
Finding II: forgetting is mitigated by subtracting the biased FV with respect to the current training task. Similarly, we conduct the intervention experiment again on each evaluation task T by subtracting the target FV, i.e., 07 as depicted by the red line in Figure 3 Here, the target FV 0/ represents the function of task T3 after the model has been trained on j-th task. is calculated
as E(AAes hig where Ti denotes the mean activation from in-context learning inputs in task Ty. The intervention involves modifying the representations in a specific layer during inference through subtracting the function vector (hy hy 8% The results reveal a surprising reduction in forgetting. For example, removing the target FV during Hellaswag inference, as shown in Fig. 3 (b) top, increases the model M5's performance from 39.5 to 49.1. This finding suggests that the groundtruth latent variable 09. has been confounded by the biased target FV introduced during training on incoming tasks; substracting it mitigates this interference and thus reduces forgetting.
Finding III: forgetting of the in-context learning ability is severe. In Table D and Fig. I one of the most significant observations is that the forgetting of in-context learning (ICL) ability is notably more severe than the zero-shot ability when evaluated on general tasks. For instance, in Llama2-7b-chat trained on NI-Seq-CI, the forgetting of Hellaswag reaches 34.7 for ICL, while in the zero-shot setting, the value is just 8.94. We attribute this to the overfitting of the mapping Pas (0) æf vf' ..... x vn x²) to biased target FV in task T,. This results in a distorted, complex mapping as evident from the correlation between CommonsenseQA 5-shot evaluations and FV shifts
7

---

Published as a conference paper at ICLR 2025
during NI-Seq-GI training Llama2-7b-char (Figure Notably, 5-shot performance declines faster than zero-shot as the FV shifts from M2 to M3, but recovers when it stabilizes at M5.
To summarize, the above empirical findings, alongside the strong correlations between forgetting and changes in FVs demonstrated in Section 4 and our causal mediation analysis in Appendix which reveals shift in the set S of causal attention heads (i.e., signifying the magnitude and di-
rection of the function vector) during continual instruction tuning, collectively advocate that: (1)
forgetting is primarily driven by modifications in Par (0]x)), rather than changes in PM (ylær, 0); (2) in LLMs, the function responsible for handling specific task is not overwritten but is instead over- shadowed by newly introduced functions.
FUNCTION VECTOR GUIDED TRAINING DESIGN
In this section, we present the function vector guided training design that serves as an effective mechanism for mitigating the forgetting of general abilities applicable wide range of language models and continual learning baselines. We introduce the overall architecture, present experimental results, and analyze how function vector guided training works.
Function vector guided training. The correlation between forgetting and the function vector im- plies principle for training method design. That is, training should be capable of maintaining the correct mapping of inputs to function vectors and thus mitigating forgetting. Based on this principle, simple yet efficient design to mitigate forgetting. Our method introduces two novel regularization terms:
propose function vector-guided training
First, mitigate the introduction of biased function vector during training, we restrict the alter- ations in FVs tied to the training tasks, effectively maintaining the model's Pas (0r|z) unchanged. To this end, restrictions are imposed on the activation values of specific heads signifying the function vector with FV consistency loss. When training task Tj, the loss is specified as follows:
Ed 2 (x). hx)) LFV
(4)
(1,)ES
where h/ (z) denotes the activations on last input token of head j in layer i on input from model M and M is the model before training task Ty. -(-,-) is the distance metrics between variables, and we adopt L,2 distance in this paper.
Furthermore, we introduce a FV-guided KL-divergence loss align the task function raised by zero-shot inputs and the FV-intervened one. The detailed function for training task T is as below:
(KL. ( 4,- (- x)]-
(5)
M,-1
P.r( x)) is the predicted probability distribution for each token in the vocabulary y given input M,-1 - denotes the model trained after j tasks with intervention by the function vector 89.. that is 69, is added the activation h, of layer during the forward pass. l selected as in this paper. Then, the overall optimization objective is to minimize the loss f LyM + a Lyv + azlKL. where (LM 02 are trade-off hyper-parameter provided in Badd E respectively. This FV-guided fine-tuning leverages the function PM (y fr.) within the model to guide the fine-tuning, ensuring that the model retains a robust causal pathway PMr (ylar, ar, Prr (0.[zr after fine-tuning and minimizes the impact of newly introduced functions on past abilities.
log PM (yl=)) is the language modeling loss (Brown et al. 2020 and
The algorithm procedure and implementation details are
Main results. The experiments were conducted on three language models, demonstrating their effectiveness in combination with existing continual learning methods, such as Incremental Lora Hu et al. (20TT) (IncLora). Elastic Weight Consolidation Kirkpatrick et al. (2017) (EWC), Orthogo- nal Lora Wang et al. (2023b) (OLora), and Instruction- based Memory Replay Wang et al. (2024b)
(LnsCL). Besides the three task sequences we adopted in the previous sections, we also addressed the effectiveness of our approach the public benchmark TRACE Wang al. (2023c), which
8

---

Published as a conference paper at ICLR 2025
Table Performance of baselines and their improved version with Function Vector Guided (FVG) training on four benchmarks. Main conclusion: FVG significantly prevent forgetting in general and in-context learning capabilities (GP and IP).
Method
GP
SI-Seg-1 IP
F
GP
SL IP f
FP
GP 1
NI-Sag-M1 IP f
FP
GP
IP
FPT
54.43
54.43
49.85
54.43
49.85
54.43
Ilmnu2-7b-dut
Loralne
47,16
30.94
19.35
45.83
27.71
83,80
47.55
37.23
54.33
46.17
38,08
41,20
+FVG
+25.25
+2.84
+3.98
+2.65
+15.78
+6.92
+12.13
Ewe
33.48
26,87
17:72
46.08
38,76
85.00
44.47
41.69
55.85
49.07
47.96
54.22
+FVG
+15.73
+27.18
+0.85
+3.11
+15.96
+6.18
+13.99
+2:31
+6.01
O-lora
22.67
20.54
16
39.52
56.94
36.96
29. 38
+FVG
423.59
+0.11
48.38
+33.93
40.29
+34.95
-0.42
+14.32
+24.61
TnoCII
27.
35
76
43.09
46.46
63.63
$2.95
.FVG
+8.30
+0.91
+5.00
+16.11
+0.98
+8.32
+6.70
+11.04
Mo
56.61
60.61
56.61
60.61
60.61
56.61
60.61
Lominc
45,51
30.85
21.10
51.89
54.63
82.10
48.00
47.82
52.63
50.31
52.61
27.14
-FVG
+15.31
+3.10
+3.99
+6.29
TnsCT.
46.48
49.46
28.5 23.53
52.11
57,30
82,50
49.46
$3,50
60.92
51.87
51.23
37,32
+FVG
+8.06
+3.52
+1.58
+4.34
+2.75
+2.04
+7.92
Misral-Tai -bb.
Mo
47,55
57,51
47,55
57,51
47.55
57,51
47.55
57.51
Loralne
42,81
38.82
19.78
18.00
53,00
85.4
49.79
$1.02
57.01
51.91
$1.37
44.68
+FVG
+4.49
+66.1
+0.64
+2.35
+2.67
-0.50
-2.41
+4.02
+0.43
-1.49
+5.14
+10.37
InsCL
43.46
51.0
25.78
40.77
49.49
83.03
42.38
52,27
58.01
30.90
50.39
55,99
+FVG
+2.71
+4.44
+6.75
+4.27
+2.07
+6.13
+3,40
-0.84
-0.98
+6.12
+1.19
includes learning sequence comprised of multi-choice QA, code generation, mathematical reason- ing, and summarization tasks. Table 2 shows the continual instruction tuning performance on four benchmarks, leading to several key observations:
Observation 1: Function vector guided training significantly reduces forgetting in both general and in-context learning capa- bilities. Traditional continual learning meth- ods experience substantial forgetting, with InsCL performing somewhat better yet still experiencing minimum decline of 11.34% on IP. By contrast, function vector guided training achieves consistent and substantial gains in combating this issue, enhancing performance in the Llama2-7b-chat across all baselines, achieving average increases of 5.44 in GP and 17.52 in IP. Function rector-guided training provides a methodol- ogy break through the forgetting of gen- eral knowledge in existing models.
Commenoeraaa
CommoraenseG
ET
MC
VG
10
N4
M5
160
MC
92
93
14
BAN0 Sala S. wiriri
2414 2212
96
52
Ulama2 venctoonvector
G1 [b] Larma--T--chee N8-SangCC1 sn/vo
function wettor guided training
Figure 4: The shifts in function vector with 0/5-shot performance with function vector guided training. Main conclusion: FVG prevents the shift in FV (yel- low bar) and thus mitigates forgetting (orange line).
Observation function vector guided training does not compromise the plasticity in learning new tasks. This technique demonstrates a negligible reduction in the FP metric, signifying a well- maintained balance between plasticity and stability. For the TRACE datasets with the IncLora method, our method shows an improvement of 12.13 on the Llama2-7b-che model, showing large potential in protecting performance on the training sequence as well. Nonetheless, under certain scenarios, like when utilizing the InsCL replay method on NI-Seq-M1, our strategy yields 2.8 drop FP. This could be attributed to the conflict between the diverse gradient information from the memory buffer and our regularization component.
Observation The effectiveness of continual learning methods varies with the type of training se- quence. As discussed above, different training tasks impact existing capabilities in distinct ways. For instance, while learning classification tasks results relatively minor forgetting, sequences of generative tasks tend to lead to more pronounced issues. Consequently, the conventional approach of uniformly applying the same strategy to every training task demonstrates significant variance in performance across different sequences. Specifically, InsCL exhibits performance decline the

---

Published as a conference paper at ICLR 2025
NI-Seq-CI dataset compared to IncLora, while it shows improvements on the other three datasets. In contrast, our method retains the model's task-specific functions and thoroughly accounts for the characteristics of the current learning task, thereby achieving consistent and considerable enhance- ments across a diverse range of datasets.
How does function vector-guided training work? Following our principle of designing function vector-guidee guided training, we test the mechanics of this approach by examining the shifts in function vectors during training. We visualize the alteration of function vectors during function vector-guided training in Fig. 4 following the setting in Sec. A It becomes evident that the methods we propose can effectively maintain the shifts in function vectors for general tasks, even without the need to access their training data. As the FVs remain stable during the fine-tuning phase, the performance on general tasks is effectively safeguarded.
7 RELATED WORK
Catastrophic forgetting in fine-tuned language models. Fine-tuning foundational LLMs (oou-
vron et al. 2023a b) has become a generic technique for enhancing their capacity of following instructions Wei et al.] 2022; Zhang et al. 2024a b) and mastering domain-specific content (Yue) eet al. 2023 Christophe et al. 2024). However, adopting such technique can have a negative effect of hurting the original ability of LLMs, which widely known as Catastrophic Forgetting (Kirk-
patrick et al. 2017: Luo et al., 2024 Kotha al. 2024 Wu et al.) 2024b). In context of LLMs, existing approaches towards mitigating this issue can mostly be categorized into three types: regu- larizing the update of model parameters (Huang al.
Cha et al. 2021). replaying previous or self-synthesized data (Scialom et al., [022] Huang et al. 2024a) and resisting interference via parameter-efficien fine-tuning (Razdaibiedina al., [023] Wang et al. 2023b), In this work, we aims to characterize CF in LLMs through the function vector, concluding that such forgetting primar- ily stems from biases in function activation rather than the overwriting of task processing functions. To this end, We propose function vector guided training, regularization- based method to protect task activation from being improperly destroyed during fine-tuning cure the forgetting issue.
2021
Mechanistic analysis to fine-tuning. Existing works analyzing the internal mecha- nism (Räuker et al. 2023] Ferrando et al.) 2024 of fine-tuning mainly focus on how LLMs acquire new capacity, arguing that models learn minimal transformation on top of the original capabil- ity (Jain et al., (024) (wrappers), subtractable and reusable parameter shift vectors (Huang et al.) 2024b] Gao al. (024) (task vectors) and to align input queries with their internal knowledge that are already acquired in the pre-training stage (Ren et al.] 2024). Nevertheless the inherent reason for the forgetting issue brought by fine-tuning currently remains unclear, and hence our work instead targets on this important point. We have successfully identified the compact task representation, known as the function vector, can tracks task forgetting L.LMs. Our empirical data indicate a strong correlation between shifts in the function vector and the phenomenon of task forgetting.
8 CONCLUSION
In this study, we tackle the issue of catastrophic forgetting in LLMs via a detailed investigation using the FV approach, highlighting its pivotal role characterizing and mitigating forgetting phenomena. Our analysis across vast array of benchmarks reveals that model forgetting is intricately linked to shifts in latent concept variables (characterized by function vector), facilitated by our novel func- tion vector-gue guided training strategy. This method, integrating regularization term with a function vector- guided KI. divergence loss, significantly curtails forgetting, thereby enhancing both general and in-context learning capabilities of LLMs in continual learning settings.
ACKNOWLEDGMENTS
This work was supported in part by grants from the National Natural Science Foundation of China (No. U24A20253), the Research Grants Council of the Hong Kong SAR under Grant GRF 11217823 and Collaborative Research Fund C1042-23GF, the National Natural Science Foundation of China under Grant 62371411, InnoHK initiative, the Government of the HKSAR, Laboratory for Al- Powered Financial Technologies.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Leonard Baum and Ted Petrie. Statistical inference for probabilistic functions of finite state 1966.
markov chains. The annals of mathematical statistics,
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, llya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons lan- guage models. URL htps://oeeaiipublcc blob. core. windows. nel/heurnn-eplainer//ppeetee html. accessed: 14.05. 2023), 2, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33: 13:1177-1901, 2020.
Sungmin Cha, Hsiang Hsu, Taebaek Hwang. Flavio Calmon, and Taesup Moon. {CPR): Classifier- projection regularization for continual learning. In International Conference Learning Repre- sentations, 2021. URL
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a-gem. arXiv preprint /rXiv:/8/2.000 2018.
Lingjiao Chen, Matei Zaharia, and James Zou. How is chatgpt's behavior changing over time? arXiv preprint 7rXiv:2377.09009 2023.
Clément Christophe, Praveen K Kanithi, Prateek Munjal, Tathagata Raha, Nasir Hayat, Ronnie Rajan, Ahmed Al-Mahrooqi, Avani Gupta, Muhammad Umar Salman, Gurpreet Gosal, Bhar- gav Kanakiya, Charles Chen, Natalia Vassilieva, Boulbaba Ben Amor, Marco AF Pimentel, and Shadab Khan. Med42 evaluating fine-tuning strategies for medical llms: Full-parameter parameter-efficient approaches, 2024.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling instruction-fintture lan- guage models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, et al. Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment. arXiv preprint arXiv:2312.09999, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:240..21783, 2024.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021.
Itay Evron, Daniel Goldfarb, Nir Weinberger, Daniel Soudry, and Paul Hand. The joint effect of task similarity and overparameterization on catastrophic forgetting-an an analytical model. arXiv preprint 1ix.:::2411.11111 2024.
Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R. Costa-jussà. primer on the inner workings of transformer-base language models, 2024.
Lei Gao, Yue Niu, Tingting Tang, Salman Avestimehr, and Murali Annavaram. Ethos: Rectifying language models in orthogonal parameter space, 2024.
Amit Gruber, Yair Weiss, and Michal Rosen-Zvi. Hidden topic markov models. In Artificial intelli- gence and statistics, pp. 163-170. PMLR, 2007.
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. arXiv preprint arXivv23100.15// 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 03300, 2020.
11

---

Published as a conference paper at ICLR 2025
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0900 09685, 2021.
Jianheng Huang, Leyang Cui, Ante Wang. Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal, 2024a.
Shih-Cheng Huang, Pin-Zu Li, Yu-Chi Hsu, Kuang-Ming Chen, Yu Tung Lin, Shih-Kai Hsiao, Richard Tzong-Han Tsai, and Hung yi Lee. Chat vector: A simple approach to equip Ilms with instruction following and model alignment in new languages, 2024b.
Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. Continual learning for text classification with information disentanglement based regularization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cot- terell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, pp. 2736-2746, Online, June 2021. Association for Computational Linguis- tics. doi: 10. 18653/1122021111111 wawp://222...aalmmmiaaam..2 maacl-main.218 URL https: //ac///cc / antho logy. org/2021. haacl-ma 1 218
Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert Dick, Hidenori Tanaka, Tim Rocktäschel, Edward Grefenstette, and David Krueger. Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks. The Twelfth International Conference on Learning Representations, 2024. htRpst:::: :/ /opee revi www.ieww... www.wee///eee..oo............... (aserummmd=AAHHeeR 1 4N1
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre- training of language models. In The Eleventh International Conference on Learning Representa- fiony, 2023. URL
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agmieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13): 3521-3526 March 2017. ISSN 1091-6490.
doi: 1611773/paa..1111855111 7/dx. doi www.orgg//. 10. 1073 2073//pase pnas..ns.... 1 611835118
URL http:
Suhas Kotha, Jacob Mitchell Springer, and Aditi Raghunathan. Understanding catastrophic forget- ting language models via implicit inference. In The Twelfth International Conference on Learn- ing Representations, 2024. URL URLp:ttss////eeeeeeee
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Sebastian Lee, Sebastian Goldt, and Andrew Saxe. Continual learning in the teacher-student setup: Impact of task similarity. In International Conference on Machine Learning, pp. 6109 6119.
PMLR, 2021.
Zhaoyi Li, Gangwei Jiang, Hong Xie, Linqi Song, Defu Lian, and Ying Wei. Understanding and patching compositional reasoning in 1ms. arXiv preprint arXiv:2402. 14328, 2024.
Chin-Yeus Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Sen Lin, Peizhong Ju, Yingbin Liang. and Ness Shroff. Theory on forgetting and generalization of continual learning. In International Conference on Machine Learning, pp. 21078 21 100. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2025
Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jiammeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hamning Zhang, et al. Mitigating the alignment tax of rlhf. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 580-606, 2024.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Yun Luo, Zhen Yang, Fandong Meng. Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, 2024.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkow. Locating and editing factual asso- ciations in GPT. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https::t////......... petttps://openrevltt /wwpeeeeeeeeeeeeeeeeeeeeee Bet/Eorum.idd-aae 20-o-uusnidd--hhAW5555
Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, 2023.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint PrXiv:1809.02770 2018.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744 2022.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint
06001, 2016.
Judea Pearl. Interpretation and identification of causal mediation. ERN: Other Econometrics: Econometric Model Construction, 2013. URL httpss///app.eeeaattcccccccooess
CorpusID:: 8598:3655636
Baolin Peng, Chunyuan Li. Pengeheng Hc, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint 2333272304.03277, 2023.
Chengwei Qin and Shafiq Joty. LFPT5: A unified framework for lifelong few-shot language learning based prompt tuning of t5. In International Conference on Learning Representations, 2022.
URL htt.hhtpss//oppneeiiiieee
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Vinay V Ramasesh, Ethan Dyer, and Maithra Raghu, Anatomy of catastrophic forgetting: Hidden representations and task semantics. arXiv preprint arXiv:2007.07400 2020.
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Alma- hairi. Progressive prompts: Continual learning for language models. In The Eleventh Interna- tional Conference Learning Representations, 2023. URL htttps://ppeneeiew..nt
Eorum?1d-00Tg@Bc911
Mengjie Ren, Boxi Cao, Hongyu Lin, Cao Liu, Xianpei Han, Ke Zeng, Guanglu Wan, Xunliang Cai, and Le Sun. Learning or self aligning? rethinking instruction fine-tuning, 2024.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi. Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llaaa: Open foundation models for code. arXiv preprint 2308 /2950, 2023.
13

---

Published as a conference paper at ICLR 2025
Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield Toward transparent ai: A survey on interpreting the inner structures of deep neural networks, 2023.
Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tumed language models are continual learners. In Yoav Goldberg. Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6107-6122, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: 10. (0.18653/11/2022.cmm http///2222.mmmmmaiiii.a mamp-mmiin.000 URL emnip-main. main. 410
Joan Serra, Didac Suris. Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International conference machine learning, pp- 4548-4557. PMLR, 2018.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonss al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 04615, 2022.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: question answering challenge targeting commonsense knowledge. arXiv preprint 1r1Xiv:811.00033:3 2018.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction- following model. Stanford Center for Research on Foundation Models. https://crfim stanford. eduu2023/33/1/aappaca html, 3(6):7, 2023.
Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 1938):19001 2023.
Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. arXiv preprint 10. 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances Neural Information Processing Systems, volume 33, pp. 12388-12401. Curran Associates, Inc., 2020. URL htttps://proceeed
neuri wwppaper.eppp.....eeeee files/ 2a2a:r/2221111225555277777555512
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023a. URL https:
//openreview.oo http://wwwww.eeeeeoommmmmaaaaaae
Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023b. URL https::::
// /pwpe/eevew..eeeeeeeee wwwpeeewtr/iiieeeee 1tt.:iidd--722288---
Xiao Wang, Yuansen Zhang, Tianze Chen, Songyang Gao, Senjie Jin, Xianjun Yang, Zhiheng Xi, Rui Zheng, Yicheng Zou, Tao Gui, et al. Trace: A comprehensive benchmark for continual learning in large language models. arXiv preprint arXiv:23 2023c.
14

---

Published as a conference paper at ICLR 2025
Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large lan- guage models are latent variable models: Explaining and finding good demonstrations for in- context learning. Advances in Neural Information Processing Systems, 36, 2024a.
Yifan Wang. Yafei Liu, Chufan Shi, Haoling Li, Chen Chen, Haonan Lu, and Yujiu Yang. Insel: A data-efficient continual learning paradigm for fine-tuning large language models with instructions. arXiv preprint arXiv:2403. /435, 2024b.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. Suppe-natuurainnstuutiino Generalization via declarative instructions on 1600+ nlp tasks. In Pro- ceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 5085-5109, 2022.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In Interna- fional Conference on Learning Representations, 2022. URL
net/
Forum2id--E22rGoozee
Chengyue Wu, Yukang Gan, Yixiao Ge. Zeyu Lu, Jiahao Wang, Ye Feng, Ping Luo, and Ying Shan. Llama pro: Progressive Ilama with block expansion. arXiv preprint 1xXiv:2401.02411 2024a.
Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan Fang Li, Guilin Qi, and Gholamreza Haffari. Pre- trained language model in continual learning: A comparative study, In International Conference on Learning Representations 2022. OpenReview, 2022.
Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy- Vu, and Gholamreza Haffari. Continual learning for large language models: A survey, 2024b.
Yichen Wu, Long-Kai Huang, Renzhen Wang, Deyu Meng, and Ying Wei. Meta continual learn- ing revisited: Implicitly enhancing online hessian approximation via variance reduction. In The Twelfth International Conference on Learning Representations, 2024c.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02085 2021.
Siqiao Xue, Tingting Chen, Fan Zhou, Qingyang Dai, Zhixuan Chu, and Hongyuan Mei. Famma: A benchmark for financial domain multilingual multimodal question answering. arXiv preprint arXiv:2410.04526, 2024.
Shengbin Yue, Wei Chen, Siyuan Wang, Bingxuan Li, Chenchen Shen, Shujun Liu, Yuxuan Zhou, Yao Xiao, Song Yun, Xuanjing Huang, and Zhongyu Wei. Disc-lawllm: Fine-tuning large lan- guage models for intelligent legal services, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint arXiv: 2019.
Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. LLaMA-adapter: Efficient fine-tuning of large language models with inttiiiiiillzedd attention. In The Twelfth International Conference on Learning Representations, 2024a. URL https:
//openrevies wwt.:
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, and Guoyin Wang. Instruction tuning for large language models: A survey, 2024b.
Weixiang Zhao, Shilong Wang, Yulin Hu, Yanyan Zhao, Bing Qin, Xuanyu Zhang, Qing Yang, Dongliang Xu, and Wanxiang Che. Sapt: shared attention framework for parameter-efffiient continual learning of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11641-11661 2024.
15

---

Published as a conference paper at ICLR 2025
A FUNCTION VECTOR EXTRACTION
We next consider how to extract 0g for a given dataset D drawing on the concept of function vectors proposed by Todd et al. (2023). This extraction is carried out using in-context learning (ICL) samples, where the model incorporates task-relevant information into its hidden states as it engages with examples with the ICL prompt. This process is associated with the emergence of 0g (Todd et al., 2023 Hendel et al., 2023). Subsequently, a causal mediation analysis (Pearl 2013] Vig et al. 2020 the ICL inputs to identify attention heads with significant causal impacts on the output, and aggregating their representations results in Oz. Interestingly, this vector remains effective even under zero-shot input scenarios. The detailed procedure is outlined below:
Li al., 2024) conducted
First, we start by gathering the condicondt activation for each model head by averaging the ICL input representation of the given task D i.c.,
1 D
hey (p.xz)) (x)ED=
Σ
(6)
Where p = (((11,1), (N yN]] represents the N-shot ICL prompt text made up of held-out samples task c, huj is the model activation at the last token, layer and position j, and hij represents the task-conditioned activations.
Then to assess the existence of cause-and-effect relationship between hiy and correct output, we employ causal mediation analysis. The model will run counterfactual ICL input [ő, ai] incorpo- rating label-shuffled prompt ý [(x1, (()),;)),-
ū)); ..., (I ÚN)]. typically leading to incorrect outcomes. We then substitute the value of the specific head with the task-specific conditioned activation hij and calculate its causal effect (CE) on the model's output.
CFe, ((),))))))) = P 11 (ya LØ, æ)) Pas [1,
Here, Mhh denotes the model with replacement operation attention head (1, last token of the input sentence. A higher CE suggests that the specific head's state is crucial in enabling accurate predictions, denoting the encoding of more task-relevant information. For each head at layer and position j.we adopt the approach proposed by [Tod et al. (2023) to calculate the average CE across variety of tasks. Subsequently, we identify the top 10 heads with the highest average CE (recorded as set S) as the most critical in conveying task-relevant information. The task vector 0g is is then obtained by aggregating the lank-conditioned activation from the attention heads in the E(.ees hiy.
set S, i.e., 6.
We then evaluates the effectiveness of the function vector (0.) through intervention experiments on the initial model across multiple datasets. Results show that the FV significantly influences the output behavior for specific tasks, with its introduction notably improving zero-shot performance in certain tasks and removal diminishing the model's ability to produce correct outputs. This suggests that the model's specific abilities can be identified and analyzed by studying the corresponding FV,
B FUNCTION VECTOR GUIDED TRAINING ALGORITHM
Algorithm outlines the procedure for function vector guided continual learning. It begins with a sequence of tasks, each paired with its corresponding dataset, well as pre-trained language model (referred to as My). Based on the approach from Todd et al. (2023). j. a collection datasets [D,, ....... Dx } is utilized determine the set of function vector heads. Furthermore, it is proposed that the function vector head set S is applicable across different datasets, allowing us to collect this set S only once.
held-out
16

---

Published as a conference paper at ICLR 2025
Algorithm 1: Function vector guided training procedure
Input: Given sequence of tasks {Ti, TTi.T..... .....TTT } and their corresponding datasets
{Di, D2......r ), series of held-out dataset (D1, D2...... to figure out the set of function vector heads, pre-trained language model Mo
Output: Language model MN trained after N tasks
Function Main 1, D2, D2....DNh {D1, D2.....k). Mo):
S GetFunnttt inneectorset ((D1, D2, .x....... Mb)
for + 1 to N do his - 200 En; ED) his (p,,
1/ task-conditioned activation
OT, + [a,ees hi
M; < FVGuidedTraining (D,, ,,
end
return MN
return
10
Function Get GetFFunttionVcctorree ((D1, D2....DKk M)
f- :r: to K do
12
13
hys + 100 Ex hus (lvs, xi))
// task-conditioned activation
for 1 < to Layer Num do
15
for k
HeadN um do
foreach (x in Dj 100 do
CF (p, x|) <- P Pyrau-hu (y) [f,x)) - PMr (y [},=])
[t,2, <- 1[t,1,k]
CE xl) // p label shuf fled prompt.
end
end end end 22
21
Smean R ΣK :
// average across datasets
S
(Smean)
25
return S
26
return
Function FVGuidedTraining (D, Or, S)
28
foreach B = (I,, in GenerateBatches(DD D) do
29
LLM - [B| E(Ey)EB - log PMs x))
11 language modeling loss
fFV [8] E(WEB Ea.ess d hhik
Mi-1 (x), hi (z) // FV consistency loss // FV-guided
31
(KT. [8 EiSEn KL[Par(: [
x)]
KL-divergence loss
32
M. M.UppateWeights((... + lrv + (xx)
33
end
34
return .M
return
C ILLUSTRATION OF CAUSAL PATHWAY TO FORGETTING.
To help the understanding of "the causal pathway forgetting through function vector", we provide the illustrations in 3 the detailed discussions in Section Refer to the caption of Figure for more information.
D DATASETS
Three continual instruction tuning benchmarks and severel general evaluation datasets are adopts in this paper. The detailed information as follows:
17

---

Published as a conference paper at ICLR 2025
P(8)))
Flyix.
50
HONU
X
(a) Pretrained LLM under latent variable assumption
After learning task T
P(ylx,
X
T
(b) Forgetting stems from in function activation
Figure 5: Illustration of causal pathway forgetting. In (a), the pre-trained model is expressed in a latent variable assumption. It assumes task To establishes a predictive pathway (shown orange) that aligns well with the task (high probability with 04,). In (b), shows the model after learning a new task Ti without regularization, which will necessarily update the function attention heads, i... Pas (8) 2)), (shown red blocks), producing new function vectors 8 and 0% that are biased toward TT. These shifts in function vectors lead to derailed predictive pathway (shown in purple) with erroneous predictions for task To: in other words, forgetting of To occurs. summary, the modifications in Par (f) (z) rather than Par (v/2; 0) are the primary driving force behind forgetting.
Table 3: A summary of dataset statistics in TRACE includes information on the source of the context, average length in terms of word count for English, German, and code datasets, and character count for Chinese
Categury
Avg
Language
ScienceQA
Science
Multi-Chuice
ROUGE-
English
FOMC
Finance
Multi Multi-Choice QA
51
ROUGE I.
English
3,000
MeetingBank
Meeting
Summary
2853
ROUGE-L
English
3,000
C-STANCE
Social media
Multi-Choice QA
127
ROUGE-L
Chinese
3,000
Py150
Github
Code generation
422
ROUGE-L
Python
3,000
NumGLUE-cm
Math
Math reasoning
32
ROUGE-L
English
3,000
TRACE benchmark. TRACE benchmark is released by Wang et al. (2023c) for the study of forgetting in LLMs, which consists of 8 different complex generation tasks including multi-choice QA, code generation, mathematical reasoning and summary. Without loss of generaliztion, we select 6 out of 8 raw tasks to construct the training sequence information listed Table 3
experiments setup. The statistical
The training epoch for this benchmark is 5 for C-STANCE, Py150, NumGLUE-em, for FOMC and ScienceQA, and 7 for MeetingBank. We evaluate them with a self-construct evaluation code based on OpenCompass code framework.
SuperNI benchmark. SuperNI benchmark is widely utilized in existing instruction instruction-follooinng works. We select 26 tasks from the original dataset and set the training size to 1000 and training epoch set to 10. The statistical information is listed in Table 5
General evaluation sets.
For the general evaluation datasets, utilize Hellaswag (Zellers al.
2019). CommonsenseQAA (Talmor et al.] 2018), OpenbookQA (Mihaylov et al.] 2018). Natural Question Kwiatkowski al. (2019). Lambada Paperno al. (2016), Alpaca Taori et al. (2023)
18

---

Published as a conference paper at ICLR 2025
Table 4: Basic information of continual learning task sequences used in main text.
Sequenee
Task
Num per
NI-Seg
NE195
NE134 -4 NI130
NI1292
NI363
Clasiffiaatio
1,000
NI1943
NI220
N1224
Classifiattin
NI-Sey-ii
N1580
NI5II
Generstion
NL
NI135 - NI14 -3 NI619 -1 NI16 -3 NI002
Generation
1,000
NLSeg MI
NE500
NE36 - NI129 -3 NL339 NN1110
Classifianion Generation
1,000
Cavsification Generation
Meet Py150 SciQA Numglnzem
Clasiftiation Generation
Table 5: A summary of dataset statistics in SuperNI.
Dutaset
Source
Category
Metric
Langunge
Adala
NI002
Quoref
Question Answering
360
ROUGE-L
English
1000
N11290
Xsum
Summarization
363
ROUGE I.
English
1000
N91
Yelp review full
Sentiment Analysis
ROUGE-I
English
1000
N1141
Odd man out
Word Semantics
ROUGE-L
English
1000
Furoparl
Matching
I5
ROUGE-I.
English
N1024
Cosmosqa
Question Answering
82
ROUGE-L
English
1000
N11310
Multilinuual amazon reviews
Sentiment Analysis
59
ROUGE-L
English
1000
NI163
Synthetic
Program Execution
23
ROUGE
English
1000
N1292
Storyenmmmoeense
Information Extraction
48
ROUGE-I
Figlish
1000
N11
Amazon reviews
Sentiment Analysis
70
ROUGE-L
English
N1195
Sentiment1400
Sentiment Analysis
14
ROUGE-L
English
1000
N11355
compression
Summarization
ROUGE-L
English
N1589
Amazon fine food reviews
Summarization
84
ROUGE-L
English
1000
N11357
Xisum
Summarization
454
ROUGE-L
English
1000
NI360
Numersence
Fill The Blank
26
ROUGE
English
1000
NT339
Record
Question Answering
185
ROUGE-L
English
1000
NI220
Recstorics
Title Generation
60
ROUGE-L
English
N1224
Scruples
Ethics Classification
338
ROUGE-L
English
1000
Mutual
Dialogue Generation
ROUGE
English
N11510
Evalution
Information Extraction
ROUGE-L
English
1000
N1231
Question Amswering
229
ROUGE
English
1000
N1488
Synthetic
Program Execution
16
ROUGE-I
Figlish
1000
N1618
Muhilingual amazon reviews
Summarization
47
ROUGE-L
English
1000
NI363
Sst2
Sentiment Analysis
ROUGE-L
English
N1619
Obsumed
Title Generation
161
ROUGE
English
1000
Reddit tifu
Summarization
English
19

---

Published as a conference paper at ICLR 2025
Table 6: Input template for calculating instruction probability and training for different tasks.
Task
Prompts
"Input": "Chooss an answer for the following question and
ScienceQA
give your reasons. Question: [t]] Answer." "Output" : "Iy]"
"Inpat": "What is the monetary policy stance for the following text? A. dovish, B. hawkish,
FOMC
neutral. Choose from A, and Text: [ar] Stance:", "Outpat": "[vll"
(Translate Chinese to English) "Input": "Determine the attitude the following text towards the specified object. Select one: A. Support.
C-STANCE
Oppose. C, Neutral. Output A, or C. Text: [ar, Object: 1zzl Attiuud:", "Outpat"; "Iyl"
"Ispat": "Write a summary of the following meoting transcripes.
MeetingBank
Meeting transcripts: [x]] Summary:". "Ouiput": "lyl"
Py150
"Iapot": "18 [x]". "Outpat": "[æ]"
NumGLUB-cm
"Input": "Solve the following math problem. Question: [a]] Answer.". "Output": "fy]"
"Isput": "Detinition In this task, you'se 'e given [Description]. Now complete following examples Input: [x]
NI-xxx
Output:", "Output": "Iyl"
and Bbh-Object Count Srivastava et al. (2022). All the datasets is downloaded from https:
7/github. com/ open wwwpass//ppppmooaaaaaaaaaae and truncate to 190 samples for efficiency.
Input template In this paper, the specific instruction template used for each dataset is given below, as show Table
E IMPLEMENTATION
We adopt Llama2-7b 7b-nnah---cha,,, Llama2-13B-che Touvron et al. (023b)) Llama3-8B-che (Dubey et al. 2024).
(Jiang et al. 2023) as the base models, with their effectiveness in both understanding world knowledge and following instructions. Without specific notification, the model is fine-tuned with LORA approach Hu et al. (202I), where the rank dimension set to and the target module is query and value weight matrices. For IncLora, OLora, and InsCL methods, new adapter is initialized at the beginning of learning new task while keep the previous Lora adapters fixed. For Ewc, only one big adapter is initialized during the sequential learning, where rank is set to 48 for TRACE, and 40 for NI benchmarks.
The maximum input sequence length is set 512 and the maximum output sequence length is set to 128. We train the model with the decoder only task calculating gradient only on the output tokens. We use an Adam optimizer with a weight decay of 1e-4 and the learning rate set to le-4 for TRACE and FUNC, le-3 for LONG (following Wang al.] (2023c)). The batch size is set to 8 and accumulate gradient step set to 2 for each GPU while we run on A100 GPUs with Deepspeed. The training size and epochs can be found in the introduction of datasets.
Implementation Detail of Optimization Objective
In order to enhance the reproducibility of the paper, we provide the detailed calculation formula for the loss function Fv and ε when training task Tj:
CFV
Σ (L,))E
Ms '(x) 1
AM (1)
(8)
CKL = Σ Par [log Par (Mr x) log 12 (3% [:]]
(9)
M,-
Here, Pas (3) x) denotes the output probability of token D, and V | is the vocabulary size. As for the set a1 and 02 0.08 as the final choice.
a and a2 we perform a grid search on [2, 1, 0.5. 0.25, 0.08, 0.02] and
20

---

Published as a conference paper at ICLR 2025
Table 7: Final performance SuperNI benchmarks language models. Hella., Com., Alpa., and Ob. denote evaluation score on Hellswag, CommonsenseQA, Alpaca, Object Count datasets, respectively. The Del. value red bold style is compared performance of their initial model Mo- Higher Forget or lower Del. represent more forgetting. Main conclusion: Forgetting consistently occurs in both general and newly learned tasks, showing considerable variations depending on the types of lasks, stages of training, and the specific language models involved.
Zeno Zeno-Shui Performamce General Task
In Context Performance in General Task
Performance in Trained Task
Hella.
Com.
Alpo.
Ob
Aug. Del.
Hella.
Com.
Alpu
Ob
Aug Dell.
AP
FP
Forgot
Llama2 Llama2-Tb-chan
Mrg
N1-Sap-C1
47.37
40,00
32.00
31.61
37.75
24.21
27.89
28.89
26.84
26.96
86 10
83.80
2.30
N1-Sag-C2
65.26
55.79
30.99
23.47
43.88
67.37
$4.21
32.66
27.89
45.53
91 91.30
58.10
3.70
NI-Sco-Gi
27.36
37.08
N1-Say-G2
26
17.89
27.80
25.03
6.06
N1-Sag-M1
52.11
42.63
31 0.00
29.51
38.50
45.79
31.05
24.58
33.16
33.65
59 02
54.33
4.70
N1-Sac-M2
65.26
43.16
30.08
37.09
43.90
44.21
27.89
27.64
57.37
34,25
73.19
55.79
17.40
Llama3 3-Sb chat
Mg
81.58
58.42
22
40.04
50.67
85.36
63.16
27.42
40.47
56.3
N1-Saq-C1
79.47
46688
25.27
52.33
45.48
79.47
40.00
25.02
45.79
47.77
83. 33.40
82.10
1.30
NI-Soy-C2
91
NI-Sa--GI
72.63
35.79
22.05
N1-Saq-452
63.08
41.55
20.9
15.37
35.38
66.84
44.74
15.35
23.58
37.63
57.37
2.11
NI-Soy-MI
78.42
40,00
91
21. 58
40.48
76.84
40.00
21.33
35.91
43.52
60.74
52.63
8.11
NI-Sag-M2
42.78
Mistral-7-iinstruc
Mg
73.68
60.00
24.74
5.07
40.86
79.47
66.32
32.36
37.89
$4.01
NI-Soc-U2
75.79
NI-Say-Gi
57:37
45.36
26. 30
13.81
35.69
57.89
35.70
32.04
30.47
41.30
27
19:78
7.85
N1-Sog-Ci2
33.68
42.63
29.60 68
52.
39.53
41.58
36.32
20.46
30.53
32.22
51.80 01.05
53.86
7.19
NI-Sox-MI
45.67
N2-Sag-M2
48.42
31.67
49.13
Me
51,05
75.26
43.16
52.54
NI-Sag-C1
21.51
48
N1-Saq-[ii
63.16
38.95
28.12
13.84
36.02
65.79
32.11
30.92
34.21
40.76
25.64
18.17
7.47
NI-So;-M1
71.58
49.47
34,10
28.09
45.81
70.53
48.42
36.51
37.37
48.21
60.10
56.34
3.76
For the hyperparameters of existing continual learning methods, refer to the well-searched value reported in previous paper. Specifically, for Ewe the scaling factor on regularization term is set to 4,000, for O-lora the number is 0.5. The memory size of InsCL set to 30 for NI benchmark and 50 for TRACE.
Implementation Detail of Function Vector Framework When extracting the function vector from in-context samples, we use 10-shot input prompt randomly selected from held-out train- ing dataset. The task-conditioned activations are average on samples filtered with correct I0- shot answer from the validation set with 200 samples. As for the set S of the casual atten- tion heads, we follow the position in [Todd et al.] (2023) for Llama2-7b-chat and Llama2-13b chat, and validate its efficiency
the set [(4,, 1), (11, 2), (9, 25), (12, 15), (12, 28), (13, 7), (11, (12, 18), (16, 10), (14,1)), For Llama2-13b-cha the set S is [133, 13), (12, 17), (15, 38), 34), (19, (19,36) (13,))))))) (18, 11), (10, 15), (13, 23)]. As for 1lama--8b-cha ysis experiments set. For 1lama--b--chat the set S is (27, 28), (13, 27), (15, 28), (17, 8), (21, 2), (10, 12), (15, 16), (15, 2), (15, 1), (31, (31,24). For Mistral-7b-in 7b-instruct. the set S is (14, 31), (26, 29), (12, 4), (12, (30, 4), (30, 9), (22, 30), 19), (11, 10), (18, 1)].
own datasets. Specifically, for Llama2-7b-chae
and distral-7b-instruce we run the casual anal- 15 datasets and calculate the average CE to get the final casual attention heads
F EXTENDED EXPERIMENTAL RESULTS
Results on more sequences. addition to the results of the three sequences presented Sec. conducted the similar experiments
sequences and found forgetting patterns similar to the experimental results described in the main text. The detailed experimental results on totally sequences shown in Table
21

---

Published as a conference paper at ICLR 2025
60
commosssee
Roueee
40
Rouge
RaggeL
10
06
422
15
(b)Alpaca
Rouget.
Rougee.
10
15
0.5
0.8
(c) Hellaswag
[to.pe
Roiget
Rourgen
10
30
20
30
400
Itagge
Rasque
Rosges
25
28
0,755 0:500
0.005
0.900
0.975
0.85
0.90
0:55
400
800
(1) Function vector similarity
Last hidden similarity
(III) Parameter L2
distance
Figure 6: The correlation plot on model performance and different similarity metrics. The y-axis shows Rouge-L. metric on test data, while the x-axis represents the degree of similarity between the current model state and its initial condition. The calculation of each similarity metrics is (1) FV simi- larity: Cosina(69-- f ). (2) Last hidden state similarity: Cosine ( -_ Σ h² (x)) (3) Parameter L2 distance: | - 1 The dotted line in each figure denotes the performance for original model. Main conclusion: There is significant correlation between performance and FV similarity (sub-figures in the first column), while the other two metrics- -last hidden state similarity and L2 distance-do do not show such strong correlation.
Correlation plot on model performance and different similarity measurements. In this sec- tion, we provide scatter plots illustrate the correlation between model performance and function vector (FV) similarity, alongside two other metrics: the similarity of the last layer hidden states and the 1.2 distance of parameters. The calculations for each similarity measure are follows:
FV similarity is calculated using Cosinc(#l, F%.e). where 8/. represents the FV of the evalua- tion task T after fine-tuning the j-th task. Last layer hidden states similarity is derived from
Cosine ho. Ezer he (x)) where the test dataset and h² h² denotes the model's 's output representation the last token position in the last layer after fine-tuning the j-th task. Parameter L2 distance is defined as W w/ with W being the model weight post fine-tuning the j-th task. Here the reported model performance is 5-shot results on the evaluation dataset.
For each evaluation task, we collected 40 data points from various models across different task sequences and stages and created correlation diagrams. The results, detailed in Figure 6, demonstrate a notable correlation between model performance and FV similarity. This is particularly significant for tasks like Hellaswag, Commonsense@AA and Alpaca, where decrease in similarity corresponds
22

---

Published as a conference paper at ICLR 2025
Table 8: Performance of baselines and their improved version with Function Vector Guided (FVG) training or Model Averaging (MA). Main conclusion: MA performance better on fine-tuned datasets (FP) compared to FVG, but struggles in the general/in- contexx datasets setting (GP/IP).
Mothod
G1
1P1
FP
GP
N1-0g-C1 IP 1
FP
GP
M1-
IP f
EP
MMo
54.43
49.85
54.43
49.85
54.43
Llamaz-7D--dhtt
Locaine
47.16
20,94
19.35
45.83
27.71
85,80
43.55
57.23
54.33
+MA
+0.87
-10.35
+1.46
+2.8]
-0.17
+3.90
+9.95
+FVG
+3.10
+18.97
+0.84
+3.9%
+25.53
+1.70
+2.65
+15,78
+3.52
26.87
17:22
38.76
44.47
41.89
+MA
+6.58
-11.27
+2.59
+1.57
+7.64
+0.40
+5.54
+7.71
+0.92
+FVG
+1.773
+27.18
+0.85
43.11
+15.96
+0.37
46.18
+13.99
+0.01
Mo
56.61
60.61
56.61
60.61
56.61
60.61
Locaine
45.51
29.85
21.10
51.89
54.83
82.10
48.00
+MA
+4.39
+5.01
+2.07
+1.99
+2.42
+2.00
+3.67
+5.82
+4.70
-FYG
+7.79
+15.31
43.10
43.99
45.19
+0.30
+4.88
44.75
+5.78
to increase in model forgetting. However, for scenarios where forgetting occurs, such as in Object Count, there is no apparent correlation between FV similarity and performance. These observations inspire further investigations into the mechanisms of task transfer in LLMs.
Contrarily, the other two metrics-las hidden state similarity and 1.2 distance do not show such strong correlation, indicating their limited effectiveness in reflecting model forgetting.
Relationship between forgetting and changes in FV similarity between tasks. In Figure 7 we present the similarity between the FVs of training and evaluation tasks, alongside the correspond- ing forgetting after training. This similarity between FVs is defined as Cosine .,7,1) )) where
0/-1 0 are the function vectors extracted from model Mj-1 evaluation task T and the cur- rent training task T, respectively. We observe a non-urivial phenomenon: the lower the similarity between the FVs of two tasks, the greater the extent of forgetting on the evaluation task after train- ing. For instance, in the first column of Figure 7 the most severe forgetting occurs at task T. while function vectors exhibit low similarity. This contrasts with prior findings (Ramasesh et al. |220 jet al. 2021), where higher task feature similarity was linked to greater forgetting. We hypothesize that this phenomenon stems from the substantial capacity and universality of LLMs, enabling them to construct new functions based on old ones without overwriting as continual instruction tuning proceeds. The lower similarity between the FVs of the training and evaluation tasks indicates more diverse functions introduced, which exacerbates the challenge of locating the groundtruth function corresponding to the evaluation task and thus leads to forgetting. We defer proof of this hypothesis to Section 5
Relationship between forgetting and hidden states similarity. In Figure 7 we present the similarity between the FVs of training and evaluation tasks, alongside the corresponding for- getting after training. To further verify that simple feature similarity is insufficient to repre- sent the forgetting phenomenon, we also include a heatmap of last layer hidden states similar- ity between training and testing tasks. This representational similarity was obtained through ( h (x), h=)en>>(())) (x)), where T, E represent the training and testing tasks, re- spectively, and h_ represents the model's output representation at the last input token position of the last layer. By comparing the first and third rows in Figure 7 we were unable to identify any significant correlation between them, indicating that relying solely on simple model representations to study the forgetting phenomenon is not advisable.
Comparison between model averaging. To further demonstrate the advanced nature of our algo- rithm, we compared FVG with Model Averaging (Lin et al. (024).. Model averaging is technique often used to improve the robustness and performance of models. involves taking the average of multiple model parameters across different training runs or stages and has been proven for its effectiveness in mitigating forgetting.
We evaluate Model Averaging three benchmarks with combination of IncLora and EWC methods on Llama2-7b-chas and -lama3-8biinstree 8b--isstruct models. Specifically, we perform Model Averaging on pre-trained model and final model with the averaging ratio set to 0.2. The results are shown in
23

---

Published as a conference paper at ICLR 2025
Perfomarcc Shitt
92
-28
33
-11
-33
MZ
-13
-13
27
10
10
9
--10
-10
31
M4-
18
30
22
1V bn letween
tsirard tol enuuarto tell
I10
-37
60
(1)
(1V]
60
-70
50
92
50
10
90
15
40
-37
500
30
M0
12
32
45
20
Sinierity batwwentt
M0L
35
47
Lasd Last hidden
anderralationn
M3 53
10-
33
43
34
M4
86
+
tleaaa- Mi-Say-
Figure 7: Hcatmaps performance shift (Top) and function vector similarity (Bottom) between training and test tasks before tuning. For performance shift, the value position (j' f) represents the percentage change task at moment My relative to the baseline metrics. For FV similarity, the value at position (i', corresponds Cosine (0f 0% Main conclusion: Lower FV similarity (bluer value in the second row table) between tasks correlates with increased forgetting (bluer value in the first row table) after training; however, similarity in hidden stales does not demonstrate this correlation.
CommonsenseQA
CommonsenseQA
1.0
1.0
Similarity
0.9
60
Rouge-L
Similarity
0.9
Rouge
0.8
0.8
40
0.7
0.7
0.6
0.6
MD
M1
M2
M44
M40
142
M3
M5
IV stm
S-thot
sim.
S-shat
PV sim. with PVG
S-ahot with PVG
PV sim, with P2G PV sim. with MA
S-ahot FVG
PV sim with MA
S-shot with MA.
S-shot with MS
(a) Llama2-7b-chat on NI-Seq-G1
(b) Llama2-7b-chat on NI-Seq-C1
Figure 8: The shifts in function vector with 5-shot performance with function vector guided training (FVG) and model averaging (MA). Main conclusion: FVG and MA prevents the shift in FV (yellow and light blue bar) and thus mitigating forgetting (orange and light blue line).
Table 88 It shows a better performance on fine-tuned datasets (FP) compared to function vector guided training, but struggles in the general/in-context datasets setting (GP/IP).
While Model Averaging contributes avoiding forgetting. is interesting to see the explaination in the perspective of function vector. We provide the shift in function vector before and after model averaging with corresponding performance Fig. g Cross-method comparative analysis shows that methods capable of maintaining the stability of FV changes tend to yield better results. Specifically, model averaging, when compared to its predecessor Inclora, mitigates shifts and enhances perfor- mance. Furthermore, examination across various training stages indicates positive correlation between performance and the extent of FV shifts including Model Averaging.
Effectiveness of Function Vector To assess the effectiveness of the extracted fr, referred to as the Function Vector (FV) in this study, conduct series of intervention experiments across multiple datasets (see Fig. 9 on the initial model Llama2-7b-char These experiments consisted of either inserting removing FV the hidden states of specific layer at the the last token position,
24

---

Published as a conference paper at ICLR 2025
examine the influence the model output. More precisely, in the transformer's forward residual stream, the instruction vector 0T modifies the hidden states at a select layer as hy = h4 + pr.
We reported the intervention findings four dis- tinct datasets: 1) CommensenseQQ (different from the evaluation set mentioned above in input instruc- tion), multiple-choice questions on common sense reasoning; 2) Antonym, a task aimed generat- ing antonyms: 3) AGNews, text classification task with the article's category as the label; and 4) Last- Spanish, task that output the Spanish translation of the list's final item. The results highlighted that the FV directly affects the model's output behavior for specific tasks. tasks such as Antonym, Last- Spanish, and Commonsense introducing FV sig- nificantly improved the zero-shot performance from low level. Conversely, in the cases of AGNews and CommonsenseQA, removing the FV resulted deterioration of the model's ability produce the correct output. In contrast, interventions with ran- dom vectors had negligible effect on the model.
Antonym
Last-Spenish
33-shot
accuracy
4.
2.0.t)
att-rund
02
10
20
30
10
20
30
Commonsense
AGNeWs
0.6
02
0.2
0.0
0.0
leyer
layer
Figure 9: Intervention results on four datasets via function vector. Main conclu- sion: Function is effective in regulating the final outputs.
Alternation of casual attention head during train- ing. We carried out causality analysis experiments to identify the latest causal attention head S in the model, which was fine-tuned on NI-Seq-GI. Specifically, these causality analysis experiments were performed across six datasets, and the average Cross-Entropy (CE) was calculated to determine the final set of causal attention heads. The findings are presented in Fig. IO We observed a gradual yet slight shift in the set S. This indicates that changes in the model's function vector occur not only in values but also in positions, though such changes are slow and do not significantly alter the importance of the original positions. Therefore, the function vectors mentioned in this paper extracted from uniform positions.
25

---

Published as a conference paper at ICLR 2025
mediation
0.0
6.2
13
12
12
0.0
0.00
0.00
(14 110 11)
(14. 11 11.22 (.),3)
[14,11(11,188] 11.))
24
(12 15. 25) 303
(28. 11 28 16.
24
(0,25 [133, 18 25
(12. 15 12. 28 131
-0.01
24
(R,25] 22,1222,222220
[6.18](0, (,,,,,99111111
-0.02
(8. 26166. 6. 9)1 19.121
-0.04
(6.9)18 (11...6611 26 1.
-0.02
[1 13, 231
(12 13 14,9)118, 28,
(14.71(0. 6
(14.91(22. 25..
00.64
leal casual meciation
Ioad casuall mediation
lead casual meciation
B
12 15 70 74
12 16 20 24
12 18 20 28
CC4
0.02
0.2
0C0
0.00
6.0
20
20
134,13122 1/1 11,
18
-0.01
20
[44.1)111.2 21(111).
-0.02
24
26
133.231155
26 (6
-0.0
[8,51(28, 13)18,01
-0.04
-004
taad
head
leaed
Figure 10: Alternation in the casual attention head during Llama2-7b-chat training NI-Seq-G1. The positions of top-10 heads are listed in each heatmap, while marked red. Main results: The position of function vector shifts at a quite slow speed during training
the newly introduced heads are
26


Reviews:
[{"review_id": "LOOB3vlbRT", "reviewer": "Reviewer_G6Jm", "summary": "The paper investigates Catastrophic Forgetting in Large Language Models during continual learning, revealing that forgetting is influenced by specific tasks and model characteristics, and proposes a novel training methodology that utilizes a function vector to stabilize performance and reduce CF, with empirical validation across multiple benchmarks.", "questions": "See above", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The forgetting issue during fine-tuning foundation models is an important problem.\n* The observations and analysis through the function vector in this paper is very interesting, which I believe would be interesting to many.\n*  The finding that CF in LLMs primarily stems from biases in function activation rather than the overwriting of task processing functions is not obvious, but I think the authors has done a good job in confiming this. \n* The writing is easy to follow.", "weaknesses": "Overall, I like the findings in the paper and the authers have done extensive investigations. I am interested in comparing the proposed method with model averaging, which is shown in [1] that model averaging is very effectivenss in terms of mitigating forgetting. Furthermore, is it possible to analyze the effectivenss of model averaging through function vector? \n\n\n[1] Yong Lin, et.al., Mitigating the alignment tax of RLHF.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KbKseQZjbo", "reviewer": "Reviewer_KUE1", "summary": "The paper explores catastrophic forgetting in continual instruction finetuning across various settings, discovering that model forgetting is influenced by the specific training tasks and the models themselves. It explores correlation between function vectors and catastrophic forgetting, and  propose a function vector-based approach to eliminate catastrophic forgetting. The paper has conducted ablation studies across various tasks, methods and models to demonstrate the generalizability of the method using different llama and mixtral models with classification and generation tasks and evaluating it for supervised finetuning and model generalizability.", "questions": "1. It is unclear in methodology of training with function vector guided loss. Since attention head will change at every iteration during training, when do you impose the KL divergence-based regularization. Since, it requires separate computation using ICL-based examples, how do you update model weights through backdrop?  Kindly provide more details on the FV loss computation. \n2. The FV-based method may require additional computational resources, especially during interventions where multiple layers are evaluated. Since to extract FV, one has to run the model on a set of counterfactual inputs to isolate activation patterns and then select top attention with the highest causal effect. The FV-guided KL-divergence loss aligns logits produced with and without FV interventions. This requires computing model outputs twice. There needs to be a discussion on the computational time and memory overhead required for the proposed approach. \n3. While training on a task, regularization is imposed just based on the most recently trained task or do you consider all previously trained tasks? The loss function in equation 4 and 5 may be modified to consider the state of training on a particular task in the sequence. \n4. There are some typos in the paper Line 107: bracket missing in citation, Line 970: it should be We instead of I, Line 231  : sequensce --> sequence.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea of using regularization on the function vectors and the function vector guided loss to enable continual instruction fine tuning seems novel and is well motivated in the paper. \n2. The paper presents a strong ablation study for the proposed methods across various settings, demonstrating their generalizability by conducting experiments using multiple llama checkpoints, classification and generation tasks and evaluating across generalizability and supervised finetuning on multiple datasets.\n3. The experiment studies which clearly demonstrate the ability of  function vectors to significantly improve the performance on continual instruction fine tuning.", "weaknesses": "1. There is a lack of clarity in the proposed methodology. i.e function vector guided training design. It would help to understand  the proposed methodology by providing the overall  the optimisation objective and end-to-end training algorithm. In the current state, it is difficult to reproduce the results that the authors mentioned in the paper without the details on the proposed methodology. \n2. The  presentation  of the paper and the readability of some sections are difficult to follow.   Notations are unnecessarily complicated. \n3. The experimental results though convincing across most of the experimental setups, its hard to make decisive conclusions especially on the core idea of function vectors and their effectiveness in mitigating catastrophic forgetting.\n4. Some experiments and  observations made related to catastrophic forgetting are redundant and may not contribute much. For example, forgetting will be model-dependent since performance on different datasets and tasks is model-dependent.\n\nMore details on the above mentioned points are listed below.\n\n1. The authors claim that “Forgetting coincides with changes in FV similarity between tasks”. However, it is difficult to observe the same from the evidence provided because the trend is not constant across all tasks and models; sometimes, a slight change in similarity changes the task performance drastically(i.e. Fig 2(a) on count object and hellaswag datasets), in some cases even though similarity is decreasing, the performance is increasing. It is very difficult to establish a clear relation from the empirical results provided. A suggestion is to  extend the number of tasks or conduct a statistical test using correlation metrics. \n2. The observation that generation tasks lead to greater forgetting cannot be measured directly, even though the same evaluation metric (Rouge) is used for both types of tasks. Rouge is more sensitive to longer sequences because it relies on n-grams overlap. The tasks have different difficulties; one way to measure them is by comparing them against some upper bound baseline and calculating forgetting in terms of percentage loss.\n3. The computation of the FV guided loss (eqn 4) is not very clear. If the set S consists attention head with top-10 CE,  is that going to be same before training on the current task and after training. Is this loss recomputed at every iteration of the optimization step? \n4. Although function vectors improve the performance of existing CL methods, it can be helpful to compare against  finetuning models individually, to see how much gap there is between methods with function vector guided fine tuning and the optimal performance that can be achieved by fine tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gxlMM0IS0c", "reviewer": "Reviewer_ZyxB", "summary": "This paper empirically studies catastrophic forgetting (CF) from the perspective of function vectors (FV) and makes the following core contributions:\n1. It shows that there is a correlation between the change in function vector for a task and degradation in performance on the task.\n2. It argues that CF is due to the inability to associate the input with the latent task vector rather than a degradation in the ability to solve a tasks conditioned on the input and correct latent task vector. It supports this with mechanistic interventions to the task vector. \n3. It presents a method to regularize FVs during training to mitigate forgetting.", "questions": "1. Related to weakness 2: can you more concretely establish the FV similarity is correlated with forgetting, perhaps with a correlation plot? I want to better understand how much signal FV provides on forgetting as a whole.\n2. Related to weakness 2: can you show how FV measures forgetting compared to other metrics one might consider. For example, after training on a task, does FV similarity to the previous task better predict forgetting of the previous task than L2 distance to the previous model?\n3. Related to weakness 3: Why is FV a measure of latent task identification? What exactly is the argument for this?\n4. Related to weakness 3: an alternate hypothesis for the results in section 5 is that after finetuning, only certain layers of the model have changed significantly while a lot of the other layers have remained unchanged. So when you make the activation intervention and search over the layers to find the optimal one, you have just undone the effect of finetuning in the activations by returning it to the behavior of an earlier model upto that later and the rest of the model isn't that different after finetuning so activations pass through and your recover the behavior of the earlier model. How would you refute this alternate hypothesis?\n5. Related to weakness 4: did I understand this correctly? If so, can you make this clear/provide some evidence that if you train with the activations for task 1 regularized, it does not mitigate forgetting on task 2 (another unrelated task). If this is true, this is an important limitation that should be made clear (I want to emphasize that it isn't a limitation of the paper but rather would make the paper stronger be explaining how this method should be used).\n6. A claim for the training method is that it does not effect plasticity. I would like to understand the limitations of this: can we finetune on a very large dataset of a new task with this regularization and still maintain plasticity? When does the regularization start hurting? What are the limits. One way you could do this is train on a FT dataset with millions of samples (maybe OpenMathInstruct-2) to see if this regularization limits learnability in extreme cases.", "limitations": "", "rating": 10, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Originality**: This was an interesting approach/hypothesis for studying forgetting that I quite enjoyed.\n**Quality**: The authors perform a thorough series of experiments investigating their claims and evidence ranges from empirical measurements, mechanistic interventions, and training interventions. \n**Clarity**: The work is well contextualized in related work.\n**Significance**: This work is a really interesting way to connect mechanistic interpretability to studying and mitigating forgetting demonstrating practical ways in which tools from mechanistic interpretability can be used to understand diverse phenomena.", "weaknesses": "1. **Clarity**: The paper dense and the writing is terse and sometimes hard to follow. The figures have a lot of information on them, are very small and it is quite difficult to extract takeaways from them. Overall the paper is often hard to follow and it is hard to trace the evidence for the claims. A lot of points through the paper seem like observations and the connection to one of the core scientific claims isn't always clear. I think this paper would benefit greatly from better organization. Probably most important is to explain the figures/tables better *in the captions* and actually point out what the takeaways are/what we should learn from them/what points it is substantiating. I understand this is challenging because of page constraints and the amount of information presented in the paper: my recommendation would be to move anything non-essential to the core scientific claims to the appendix. Also, the authors could present the results for 1 model and add the rest of the models to the appendix given that the models are all of the same scale.\n2. **FV and forgetting**: One of the key claims in this paper is change in FV is correlated with forgetting, i.e. this is a useful measure that \"explains\" the phenomena of forgetting. The evidence did not convince me of this: in Figure 2, there are points where FV similarity is high, 5 shot performance is high but 0-shot performance is low. Object Count especially is unconvincing (0shot and 5shot switch and don't seem to be correlated with FV).\n3. **Mechanistic interventions**: While the mechanistic interventions in Section 5 are quite interesting, the paper does not consider any alternated explanations for this behavior (see questions). It is also not clear to me why FV is a measure of latent task identification.\n4. **Limitation of training method**: If I understand correctly, this training method regularizes the FV of particular tasks and thus prevents forgetting of the particular tasks being regularized. It doesn't affect forgetting as a whole? While this is not a weakness, I think this should be clarified and the method should be explained as such.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5AHE79DmWv", "reviewer": "Reviewer_Qiar", "summary": "This paper studies the catastrophic forgetting (CF) problem in LLMs with the tool of Function Vector (FV). The research starts with reproducing the CF problem on a series of four tasks across 4 publicly available LLMs. In addition, the researchers study the differences in FVs between continue-trained and untrained LLMs on each task. They show that there is a strong correlation between the differences in FVs and the CF issue. Their theoretical analysis posts a hypothesis on the bias of the FV θT elicited by the input x, (i.e., p(θT|x)). The authors further conducted several experiments to well support their hypothesis. Finally, they designed a new loss for alleviating the CF problem in continuing learning, where they encourage LLMs to minimize the differences between model prediction and the FV-intervented prediction while keeping the hidden representations of X the same as un-trained ones.", "questions": "Here are some minor/typo suggestions:\n1. Line 107, the references to Peng et al. and Chung et al. should be \\citep instead of \\cite.\n2. Line 143, missing a blank between \"using\" and \"SuperNI\".\n3. Line 162, the reference to Zhao et al. should be \\citep instead of \\cite.\n4. Line 200, missing a blank between \".\" and \"Unless\".\n5. Line 200, the reference to Hu et al. should be \\citep instead of \\cite.\n6. Line 201, \"learning rate of 1e-4\" should be \"learning rate of $1e^{-4}$\".\n7. Line 143/157/197, appending a \".\" after each boldface phrase will be more consistent with your writing style in Line 240/249/257 and so on.\n8. Line 357,  \"... trigger its corresponding latent variable θT (PM(θT|x)).\" could be rewritten as \"... trigger its corresponding Function Vector θT (i.e., PM(θT|x)).\" for more clarification.\n9. It would be great to provide a one-sentence summary at the end of each paragraph in Section 7 about the connection between the related works and this study.\n10. The Conclusion section seems to be more consistent with the full manuscript if it is described in the present tense.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The manuscript provides a thorough study of the catastrophic forgetting problem. \n2. The most insightful contribution of this work is posing a new hypothesis on this problem. \n3. Its technical contribution is significant as the authors first introduce the Function Vector as a tool in this community.\n4. The paper is well-written and well-organized.", "weaknesses": "I cannot see any significant weakness. One nitpick would be expanding the background of Function Vector in Sec. 2.2, especially providing the dimension or shape of each notation. It will help the audiences understand the procedure for computing FVs more easily.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)