Demystifying Language Model Forgetting with Low-rank Example Associations
Authors: Xisen Jin
Xiang Ren
Venue: NeurIPS 2025 poster ()
Keywords: catastrophic forgetting
continual learning
large language model
Primary area: deep_learning
TL;DR: We empirically analyze, predict, and mitigate forgetting in upstream data during LLM fine-tuning.

Abstract: Large Language models (LLMs) suffer from forgetting of upstream knowledge when fine-tuned. Despite efforts on mitigating forgetting, few have investigated how forgotten upstream examples are dependent on newly learned tasks. Insights on such dependencies enable efficient and targeted mitigation of forgetting. In this paper, we empirically analyze forgetting that occurs in $N$ upstream examples of language modeling or instruction-tuning after fine-tuning LLMs on one of $M$ new tasks, visualized in $M\times N$ matrices. We show that the matrices are often well-approximated with low-rank matrices, indicating the dominance of simple associations between the learned tasks and forgotten upstream examples. Leveraging the analysis, we predict forgetting of upstream examples when fine-tuning LLMs on unseen tasks with matrix completion over the empirical associations. This enables fast identification of most forgotten examples without expensive inference on the entire upstream data. Despite simplicity, the approach outperforms prior approaches that learn semantic relationships of learned tasks and upstream examples with LMs. We demonstrate the practical utility of our analysis by showing statistically significantly reduced forgetting as we upweight predicted examples for replay during fine-tuning.

Demystifying Language Model Forgetting with Low-rank Example Associations
Xisen Jin, Xiang Ren University of Southern California xianginnnninnnennn Susce.nneuuuueeeee edu
Abstract
Large language models (LLMs) suffer from forgetting of upstream knowledge when fine-tuned. Despite efforts on mitigating forgetting, few have investigated how forgotten upstream examples are dependent on newly learned tasks. Insights such dependencies enable efficient and targeted mitigation of forgetting. In this paper, we empirically analyze forgetting that occurs in N upstream examples of language modeling instruction-tuning after fine-tuning LLMs on of M new tasks, visualized in M x matrices. We show that the matrices are often well-approximated with low-rank matrices, indicating the dominance of simple sociations between the learned tasks and forgotten upstream examples. Leveraging the analysis, we predict forgetting of upstream examples when fine-tuning LLMs on unseen tasks with matrix completion over the empirical associations. This enables fast identification of most forgotten examples without expensive inference on the entire upstream data. Despite simplicity, the approach outperforms prior approaches that learn semantic relationships of learned tasks and upstream exam- ples with LMs. We demonstrate the practical utility of our analysis by showing statistically significantly reduced forgetting as we upweight predicted examples for replay during fine-tuning.
Introduction
There has been a growing need for continued fine-tuning of LLMs to mitigate harmful behaviors, update outdated knowledge, and adapt to unseen tasks and domains. Although fine-tuning allows efficient and incremental adaptation of models, models may suffer from catastrophic forgetting (Mc Closkey & Cohen, 1989 Goodfellow et al.) (0044) of upstream knowledge learned in the praitngg
or instruction-tuning phase, causing unintended prediction changes over known information. This is problematic the performance and reliability of LLMS deployed online, limiting the feasibility of
continual fine-tuning in practice (Raffel 2023
Shi et al. 2024).
Existing works demonstrate that replaying or mixing in past examples is effective and scalable approaches to mitigate LLM forgetting (Scialom et al. 2022 Roth et al. 2024; Li et al.) 2024b)
Ibrahim et al. 2024 Ye al. 2024). These approaches, however, often rely on random sampling of past examples; knowing what models forget after fine-tuning allows more efficient and targeted mitigation of forgetting e.g., by prioritizing the replay of more forgotten examples (Toneva et al.) 2019; Aljundi et al.) (019a). In this paper, we explore how forgetting caused by unseen tasks can be efficiently predicted, and more specifically, from forgetting that occurred while learning other tasks. The complexity of associations between learned tasks and forgotten examples plays an important role in predictability; Figure (a) illustrates a hypothetical scenario where certain upstream examples suffer more forgetting regardless of the leamned tasks, making forgetting casily predictable; in contrast, (b) exemplifics upstream example forgetting that is highly dependent on the learned tasks. Existing theoretical and empirical studies on the associations between learned and forgotten
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

N Upstream Examples
Measure Forgetting
[Dolma, RedPajama, Tulu, 3
New Tadks
Fij
Now
Model,
(Summarization, Safety alignment,
Forgetting
M New Tasks
Fine-Tune
T
Log Perplexity
Exact Match]
Simple Associations
Upstream Examples
motrix
Ranket
(b) Complicated Associations Highrank
Upstream Examples
Figure The problem setup of analyzing the associations between learned tasks and forgotten upstream examples as we fine-tune LLMS one of unseen new tasks. Over total N upstream examples and M unseen tasks, we measure and record forgetting (in red) M N matrix and attempt to fit the associations with low-rank approximations, Better low-rank approximations indicate simpler associations between learned tasks and forgotten upstream examples.
tasks focus on shallower models (Lee et al., 2021;
Goldfarb et al. 2024 Ramasesh et al. [02 ) the
problem is unper-explored for LLM forgetting or an example-level Swayamdipta et al. (2020); Maini et al. (0022) characterize training examples that are prone to forgetting, but they do not touch on how example forgetting depends on the learned tasks.
Specifically, we start by analyzing the associations between the learned tasks and forgotten upstream examples in LLM fine-tuning. We measure forgetting (in continuous log perplexity increase or binary exact match drop) over N upstream examples, after fine-tuning the model on one of M unseen eestruction-tuning tasks, while summarizing the results M N matrix. We evaluate the complexity of the associations by measuring the goodness-of-fi of low-rank approximations of the example associations. We then examine how the complexity of the example associations varies across model types (OLMo, OLMo2, MPT, Pythia) and sizes (1B to 13B parameters).
Our findings suggest that the associations between learned tasks and forgotten examples are often well-approximated with low-rank matrices. On OLMo-1B and OLMo-7B, rank-3 approximation fits the associations between 85 learned tasks and 140,000 upstream examples with 1² 0.69. We notice that the forgetting of more capable and recent LLM families are more complicated, requiring higher-rank approximations: within the same model family, the complexity of the associations remains stable or increases with the model size. The matrix decomposition further interprets the associations by distinguishing forgetting that are independent of or dependent on what the model learns.
Following the low-rank approximations of the associations, we predict example forgetting on unseen tasks by solving matrix completion problem over the association matrices, analogous to collaborative filtering (Sarwar et al.] 2001)) in recommender systems, achieving both efficiency and interpretability. Our matrix factorization (MF) or k-nearest neighbor (KNN) models outperform previous approaches that learn semantic relations of two examples with LMs (Jin & Ren (024).. As an example, we achieve 58.16 16 F1 in predicting binary example forgetting where the Fl of random guess is only 6.4.
Lastly, we demonstrate the practical benefit of predicting forgetting in mitigating forgetting. We upweight upstream examples with higher predicted forgetting during replay as we fine-tune LLMs on new instruction-tumit tasks, achieving statistically significant improvements in alleviating forgetting over held-out upstream examples compared to replaying random examples.
To summarize, the contributions of this paper are (I) an empirical analysis on how forgotten examples are associated with learned tasks in representative IB to 13B language models, and (2) novel approach of predicting example forgetting by solving a matrix completion problem over the empirical associations, and (3) a scalable and efficient algorithm to mitigate forgetting during LLM fine-tuning by upweighting upstream examples for replay according to the predicted forgetting.
2 Problem and Analysis Setup
In this section, we start by formally defining the metrics of forgetting and set up the problem formulation of analyzing the associations between learned tasks and forgotten upstream examples. We then introduce models and datasets that were used to collect the statistics.

---

2.1 Collecting Statistics of Forgetting
Upstream examples and learned tasks. LLMs are commonly pre-trained with language modeling objectives over massive collections of corpora, and optionally post-trained (instruction-turig -ttnidd to better follow human instructions. We use upstream data to refer to language modeling or instruction tuning training data used at the pre-training or post-training phase of LLMs. For upstream data of language modeling, we define each upstream example z; E 21..N as a chunk of a document (e.g., a Wikipedia article) model-specific maximum number of tokens. For instruction tuning, each 35 L1..N corresponds to pair of instructions and ground truth responses.
Measuring forgetting. We fine-tune an LLM (or an instruction-tuned LLM) on one unseen instruction-tuning task Ti from collection of tasks Ti..M- This results in M separately fine-tumed models fI.MM- We then evaluate performance degradation on each upstream example z E 21..N. We use log perplexity as the main performance metric as it is applicable to both language modeling and instruction tuning, and known to correlate well with other dataset-specific metrics (Hoffmann) et al. 2022). For instruction tuning tasks with a restricted output space (e.g., multi-choice questions), we also measure binary exact-match accuracy (EM). We measure forgetting Zij that occurs on an upstream example I E #I..N increase in log perplexity or drop in exact match after fine-tuning the LM on new task T; E TT..- We record forgetting 4ij in an association matrix Z E RR MMN,
2.2 Models and Datasets
Our analysis requires open access up- stream data of LLMs. We perform main experiments with OLMo-IB, OLMo- 7B, and OLMo- 7B-Instruct (Groeneveld al. 2024). We also include OLMo2-
Table 1: Summary of experiment setups. We measure forgetting on upstream examples I after fine-tuning the models over one of the new tasks M.. We measure upstream example forgetting in either log perplexity increase or exact match drop.
7B, OLMo2-133 (Walsh et al., 2024),
MPT-7B (Databricks [223), and Pythia- 1B to 12B (Biderman et al.] (023) for studying the complexity of the exam- ple associations across model types and sizes.
Model
Upstream x;
Learned Tasks T;
OLMo
Dolma
FLAN, Tulu V2, Dolly
85
MPT
Redpajama
Tulu V2, Dolly
Pythia
Pile
Tulu V2, Dolly
OLMo2
OLMo2-Mix
Tulu V2, Dolly
19
MMLU, BBH, TrudhfulQA.
OLMo-Instruct
Tulu V2, FLAN
Dally, OLMo0 Six-
142
Upstream examples 21..N where for- getting is evaluated. For OLMo, OLMo2, MPT, and the Pythia family, we evaluate log perplexity increase over Dolma (Soldaini al. 2024), OLMo2-Mix WWalsh et al. 2024 Redpajama [TogetherA]] [2023), and Pile (Gao et al. (020) respectively, each corresponding to their upstream pretraining corpora. We sample 10k to 140k documents truncated into 2,048 tokens. For OLMo-Instruct, we evaluate log perplexity increase on Tulu V2 (Ivison et al., 2023) which the model instruction-tuned on. For the FLAN (Longpre et al. 2023) subset of Tulu, we also measure the drop of binary exact matches over correctly predicted upstream examples before fine-tuning.
Unseen Task TI.M where models are fine-tuned. We fine-tune non-instruction tuned models over 66 tasks from FLAN, 11 tasks from Tulu and tasks from Dolly (Conover et al., 2023). For ILMo-Instruct. fine-tune tee-ttne OLMo-7B Instructt over new task data from MMLU (Hendrycks et al. 2021) BBH (Suzgun et al. 2022), TruthfulQA (Lin et al., (022), Dolly, and 2LMo2-SFT-Mix (Walsh al. 2024 Table summarizes the setups. We fine-tune full model parameters with 2e-6 learning rate and other consistent configurations. Training details are included in Appendix
3 Associations between Learned Tasks and Forgotten Examples
In this section, we analyze the associations between learned tasks and forgotten upstream examples represented in the M x N association matrices Z. We visualize the association matrix Z collected from the setups described in Sec. [ We formally define low-rank approximations and set up quantitative metrics of the complexity of the associations in Sec. 3.1] and examine the results of approximation across model types and sizes in Sec. 3.2 Lastly, we try to interpret the extracted associations in Sec. 3.
3

---

Doly
LLAMSsmmarizatoo
FLAMAUA
30-
FLANNU
FLAWKRC
-1
RLANCiastiiatio
FLANdLngutti
commmnn--draa dran)
redd sack
Figure 2: An example of visualized association matrix of forgetting Z E IR. MR.MX between M = 85 learned tasks and N 141, 876 upstream examples (from Dolma) on OLMo-7B. Each pixel Zij indicates forgetting (in log-perplexity increase) that occurs on an upstream example 2j (in z-axis) after fine-tuming the model on task T, (in y-axis). We annotate the domains (e.g. reddit) of upstream examples in the z-axis and the category of each learned task (e.g, FLAN/QA) in the y-axis. We include visualizations more models and setups in Figure and Figure 8 in Appendix.
3.1 Methods and Metrics of Approximating Example Associations
To examine whether simple patterns are dominant in the example associations represented by Z, we attempt to approximate Z with low rank matrices. When Z represents the increase in log perplexity (also known as the loss of language modeling), we fit matrix factorization models 2" = -==1 that minimize the Frobenius norm 2| - Z"IlF. where Og E IR ,3p E R r is the rank of the matrix decomposition and & the index of the component, For binary forgetting measured with exact match drop, we fit a logistic matrix factorization model Z" = o k= 0u13% that minimizes the cross entropy between Z" and Z, where a is the sigmoid function. We measure Ii2 or Fl scores of the approximation as the goodness-of-fi metrics.
0
Interpretations. When r = 1, the approximation effectively assumes that the forgetting 2ij (or its logit) is a scalar product of a parameter g03) specific to each upstream example and each newly learned task (6)) The set of more forgotten upstream examples independent of which task model learns, as ((3) trivially determines how fast forgetting uniformly happens on all upstream examples. With a higher rank r, the approximation captures task-dependent forgetting where certain upstream examples are disproportionately more forgotten when learning certain tasks. This inner product formulation is also connected to the first-order approximation of loss increase an inner product of weight updates and gradients (Lopez-Pa) Paz & Ranzato case is the number of LLM parameters.
2017
Lee et al. 2019), in which
3.2 Examining Complexity of Example Associations
General findings. We present R² or Fl of fitting the association with Z" with progressively higher rank in Fig. 3a). We notice that across all training setups of OLMo models, R² quickly increases to 0.69 with r = 3. Notably, even the rank-1 approximation Zt can achieve H² scores higher than 0.5. The results suggest that simple patterns are dominant in the associations between learned tasks and forgotten examples.
Example associations across model types and sizes. We compare R of the approximations with fixed M and N over Pythia, MPT, OLMo, and OLMo2 models. Fig. b)) and (c) summarize R² at given rank We notice that (1) the goodness of approximations differs among model types. On Pythia and MPT, the Rº scores at 8 are higher than 0.88, while on OLMo-7B and OLMo2, the R? scores are around 0.75 and 0.65. (2) The size of the models within the same model family also has an impact on R On Pythia and OLMo2, R² stays stable with a slight decrease as the model size increases from IB to 13B. On OLMo. R² is noticcably lower on 7B models compared to IB. (3) Model families that forget more (e.g.. Pythia) tend to produce simpler example associations (higher R²).. However, within the same model family, OLMo-7B results in a lower R²2 score than OLMo- IB despite the fact that the average forgetting is higher.
To summarize, we empirically observe that the associations between learned tasks and forgotten upstream examples are more complicated more recent and capable L.Mss, requiring higher-rank approximations. The complexity of the associations stays stable or increases with larger models within the same family. Appendix K we provide more intuitions about how model capability and sizes affect the complexity of the associations with set of synthetic experiments over MNIST and

---

1.0
1.0
1.0
0.2
0.8
0.8
0.8
Avg. Forgetting
0.1
0.6
OLNo 18 PPL
R+@r=3
0.0
CLMo-7B PPL
OLMo
OLMo
& 0.4
OLNo-78
0.6
OLMo2
0.6
OLMo2
-0.1
OLMo2
0.2
CLNO
0.4
Pythia
MPT
0.4
Pytnia
MPT
Pythia
MPT
0.0
0.3
4
13B
18
7B
138
18
78
138
Rank Reconstruction
Parameters
Parameters
Parameters
(a) R² or FI across
(b) R² =
(c) R² 5
(d) Avg. Forgetting
progressively increase the rank of the reconstruction. Forgetting is measured with log perplexity increase or exact match (EM) drop. In (b) and (c), compare R2 of approximations sizes over the fixed M = 19 tasks from Tulu and Dolly. We also report average upstream example forgetting in (d) as reference, and include more statistics in Table i Appendix.
Figure 3: (a) R Fr of the low-rank approximations
given rank across models of different types and
Table 2: Correlations between various measures of similarity and the actual upstream example forgetting represented in the association matrix
Model
OL IB
OLMo-7B
Metries
Pearson p
Spcarman p
Pearson p
Spcarman
Token or representational súmilarity
TF-IDF
-0.049
-0.035
0.048
0.137
OLMo IB Final Layer
0.021
0.017
0.059
0.062
SBERT
0.053
0.072
0.055
0.094
0.050
0.073
0.165
0.224
First order approximations
(Gradient, Weight differences)
-0.003
-0.009
-0.044
-0004
Gradient Gradient)
0.061
0.052
0.013
0.002
Forgetting occurred when learning another task
Avggóg pl7 7q..)
0.582
0.410
0.400
0.367
multi-layer MLPs. In Appendix H we further study the effect of model training configurations (e.g., learning rate, batch size) on the complexity of the associations Z. We consider that the low-rank approximation is prevalent across setups.
3.3 Interpreting Example Associations
Patterns of forgetting from matrix factorizations. The matrix factorization of Z yields interpretable patterns of forgetting in each of components o As an example, Fig. 9 in Appendix visualizes patterns captured by the k-th component in OLMo-7B experiments. We notice the patterns interesting, yet semantically intriguing. For example, on OLMo-7B, Stackoverflow documents are less forgotten when learning summarization tasks, while more forgotten when learning certain paraphrasing tasks.
Correlations between example associations and similarity measures. Are the associations between learned tasks and forgotten examples interpretable from the similarity between the learned tasks and upstream examples? We consider (1) textual similarity measures, such as token or representational similarity, and (2) first-order approximations, such as inner products of gradients and inner products between weight updates and gradients (Lee et al. 2019 Doan al. 2020). For token similarity, we use TF-IDF vectorized representations. For representational similarity. use representations given by OLMo-1B, SBERT (Reimers Gurevych| 2019), or OpenAl text-embedding-3-se
model. We detail each similarity measure in Appendix E We then evaluate the correlations between the actual forgetting Z,y and the various similarity measures and summarize the results in Table For reference. we also include average correlations between each pair of rows of Z, i.e., forgetting upstream examples after learning two different tasks Tp, T, E Ti..M. defined as (M- Epfg P(Zp.) Zg.).
As shown in Table 2. first-order approximations can achieve statistically significant correlation lpl < 0.1 with the actual forgetting, and textual similarities achieve |p] 0.25 with the actual
5

---

forgetting. Although the results imply that similarity measures can signal forgetting, the correlations are far below the average correlations between any two rows Z. Therefore, we hypothesize that leveraging the statistics of forgetting allows better prediction of forgetting than the contents of the tasks and examples, which elicits our next research question about predicting example forgetting.
Predicting Example Forgetting with Association Matrix Completion
We utilize our findings in Sec. 3 to predict example forgetting as the model learns a new task, a problem also studied in prior works (Jin & Ren 2024), and effectively mitigate forgetting. We then perform targeted replay by prioritizing more forgotten examples. Although the ground truth forgetting can be directly obtained by running inference with the fine-tuned model over the upstream data, this incurs extensive computational cost; in contrast, once prediction model is trained, forgetting caused by unseen tasks can be predicted efficiently.
Following the analysis in Sec. 3 we formulate prediction of example forgetting as a matrix completion problem over the empirical associations Z. We start by setting up the problem formulation of predicting example forgetting, and evaluate the performance of different matrix completion algorithms. We then demonstrate the practical benefit of predicting example forgetting by utilizing the prediction outcomes to mitigate forgetting during fine-tuning.
4.1 Training and Evaluation of Forgetting Prediction
Our goal is to accurately predict forgetting zij over upstream examples M..M when the model fine-tuned on an unseen task T3 with a predic- tion model 9, without running expensive LLM inference on all TI..M. To evaluate this, we cre- ate training and test splits by partitioning the set of fine-tuning tasks (noted as Tirain and TTea) and the rows of the association matrices Z. We further control whether Train and Trest belong the same category of tasks to test both in- domain and out-of-domain generalization abil- ity of the prediction models. For OLMo- IB and 7B experiments, we use FLAN as in-domain tasks and Tulu and Dolly as out-of-domain test- ing tasks. For OLMo-7B-Instrn we use MMLU, BBH, SOL----S as in- domain tasks and use TruthfulQA and Dolly as out-of-domain testing tasks. Details about the tasks included in the training, in-domain testing, and out-of-domain testing sets are discussed in Tables 15 and T6 in Appendix D
Upstream Examples X
Xg
X2
X5
X xy
To
Prriin
Ti
Tasks Larned
T2
Ty
Ta
Prest
Seed 2
Fine tieettunng w/a replay
Predict
tuning w/ predicted fgt.
Offline prediction of forgetting Replay
Predict Fine-turing, w/ predicted fyr.
experiments,
FT w/ randor replay
Online prediction of forgetting Replay
Figure 4: The training and testing setup of predicting example forgetting with association matrix completion. and their integration into example replay methods to mitigate forgetting.
To apply matrix completion for predicting for- getting, few entries žiz should be known when a new fine-tuning task T3 E Trest (row i) is introduced. We therefore assume access to the ground truth forgetting 2ij of tiny random set S (S) 30)
of upstream examples for T; E Tiest noted as seed forgetting =f {zgglzg [zr E S). Obtaining seed forgetting typically takes only a few seconds by running inference on S given the model fine-tuned on T; (or fine-tuned for a few steps on T., which we evaluate separately). We then predict forgetting of the rest 10k - 100k upstream examples. Figure illlustats an example of the train-test partition, seed forgetting, and the forgetting to be predicted. We use Root Mean Squared Error (RMSE) or Fl over the Trs as the metrics of predicting forgetting, iee,, log-perplexity increase or exact match drop.
Matrix completion approaches. We run matrix completion algorithms including additive linear models, matrix factorization (MF), and k-nearest neighbors (KNN) models. The additive linear model approximates forgetting as additive effects of learned tasks ( and upstream examples zG0 (t) + ). The MF models are introduced earlier in Sec. 3 Given the seed forgetting 2i of a task T, Trst KNN finds tasks from Train that have similar patterns of forgetting over the seed upstream

---

Table 3: RMSE (L) F1(1) of predicting example forgetting over held-out of upstream examples after fine-tuning LMs on unseen new tasks. We report average performance over 10 random seed sets (S) of upstream examples with known ground truth forgetting beforehand.
In-Domain
Out-of-Domain
OLMo-7B Instruct
OLMo-7B Instruct
Model
OLMo-IB
OLMo-7B
MPT
OLMo-IB
OLMo-7B
MPT
Metrics
RMSE
RMSE
RMSE
RMSE
F1
RMSE
RMSE
RMSE
RMSE
Embedding
SBERT
2.81
7.44
13.86
13.94
55.46
3.53
6.16
10.59
41.22
42.95
Text- Embeddnbdd
3.26
7.86
14.35
12.92
57.16
3.39
6.31
11.01
41.12
43.14
Forgetring Staristics
Additive
2.81
7.40
13.33
15.57
55.81
2.81
5.83
10.02
38.90
43.57
KNN
2.79
7.33
12.80
14,30
56.87
2.84
5.83
7.71
38.77
44.11
MF
10.41
2.82
5.76
40.47
examples S. KNN computes an average of forgetting of top-k similar tasks from Tuain weighted by their similarity as the prediction of forgetting caused by T. E Tiss on the upstream examples I1..N-
Comparators of predicting forgetting. We adapt prior approach by Jin & Ren (2024) that leverages learned similarity between learned tasks and upstream examples by LM to predict forgetting. We encode upstream examples 2y and the learned examples ..,,M; T, with a frozen pretrained sentence embedding model h(-) followed by two trainable MLP layers to obtain their representations. The final prediction is made with the inner products of two representations (h(z)), M, E.. h(ra)).
We leave the implementation details of matrix completion approaches and the sentence embedding approach in Appendix D
4.2 Mitigating Forgetting with Predicted Forgetting
Leveraging predicted forgetting for mitigating forgetting. We examine the practical utility of predicting forgetting as we sparsely replay upstream examples during forgetting following Ji & Ren (2224): de Masson D' 'uttume et al. (2019)s Ibrahim et al. (2024). We replay one mini-batch of upstream examples every 8 or 32 training steps while fine-tuning on a new task. We perform targeted mitigation of forgetting by prioritizing examples that are predicted to suffer more from forgetting. This is achieved with weighted sampling of upstream examples 2, proportional to exp (2://T), where 2ij is the predicted forgetting and T is temperature hyperparameter.
As have discussed in Sec. 4.1 predicting forgetting with matrix completion requires seed forgetting = to be evaluated. We consider an offline and an online variant of the approach. The offline variant performs a replay-frre run of fine-tuning on the task T, after which the seed forgetting will be evaluated. We then perform another run of fine-tuning while replaying examples with the predicted forgetting. This creates computational overhead equivalent one extra run of fine-tuning, but is still efficient when the training set of fine-tuning is considerably smaller than the upstream data. The online variant instead replays random examples for the first 10% of the fine-tuning steps, after which it evaluates seed forgetting and determine examples to be replayed in the rest of the 90% steps. Compared the offline variant, this mitigates the extra overhead of fine-tuning by trading off the prediction accuracy of forgetting. We illustrate the two variants in Figure 4
Baselines of mitigating forgetting. We compare with various strategies to select upstream examples for sparse replay. We primarily examine whether weighted sampling with predicted forgetting statistically significantly improves over random sampling of upstream examples (Rand). We also compare with a variant of Maximally Interfered Retrieval (MIR-T) (Aljundi et al. (2019)), selection strategy sharing the similar notion of importance that forgotten examples should be selected for replay. The approach performs livel sampling by selecting the most forgotten examples from small random subset of upstream data. We extend the approach select forgotten examples after full training run on a task, instead of single steps, which achieves better performance. In addition, we apply strategies that consider different definitions of upstream example importance. We examine a strategy based perplexity thresholds (PPL.) (Marion et al. 2023), which samples upstream data of which the perplexity is around the median the distribution. For OLM IB, we also sample replayed examples proportional to the gradient inner products (Grad-Prodd (whose correlation with
7

---

231
234
0.7
2552
2.30
054
2364
2.25
2344
228
HET
guien
Rand
omfica Add
oreer Eiet
Offing Hr
pifn
(a) IR, PPI.
(h) FLAN PPT.
(c) 7B, Tulu&Dolly PPI.
(d) Tig, Dolly PPI. (c) 7um. Mo2Mix FI
Figure Log perplexity (L) or Token F1 (f) over upstream data by replay example selection strategies. The solid horizontal lines indicate the log perplexity before fine-tuning (i.e., no forgetting). The dashed lines show the results achieved by upweighting upstream examples according to their actual forgetting after fine-tuning without replay. and ** indicate significance of improvement (p < 0.05 or < 0.005) compared to replaying random examples paired (-tests on all fine-tuning tasks.
forgetting is evaluated in Table 2 in Sec. 3.3 representative coreset selection approach that utilizes gradient information (Park al. [2023] Xia et al., (024). a reference, we also experiment with upweighting upstream examples with ground truth forgetting Zij, which, however, is highly inefficient and repetitive to obtain in practice.
Metrics, We measure log-perplexity increase or Token Fl (a softer metric than exact match (Rajpurkar al. 2016)) drop over held-out subset of 10,000 examples from the upstream data. This ensures none of the test examples are selected for replay by any of the example selection strategies.
4.3 Results of Predicting and Mitigating Forgetting
Results of predicting example forgetting. Table summarizes the error of predicting example forgetting over the tasks from the in-domain and out-of-domann test splits. We see matrix completion approaches consistently outperform the sentence embedding model adapted from the prior work. Among the three matrix completion approaches, we notice that MF models in general achieve the lowest prediction error. Besides, KNN in general outperforms additive linear and the embedding model while being highly computationally efficient.
Mitigating forgetting with the predicted forgetting. We leverage the online or offline predicted forgetting by the
Table 4: Computational cost of replay-based approaches as a summation of fine-tuning costs FT(-), inference costs over upstream examples EV()). and matrix completion costs MC. Costs that are minor are played in smaller fonts.
matrix completion algorithms to reweigh examples dur- ing replay following the procedure introduced in Sec. [.2 Juuumaeismmmizees log perplexity or Token Fl over the held-out (never replayed) upstream data as we apply differ- upstream example selection approaches. We notice that example selection based on gradient inner products (Grad) or perplexity threshold (PPL), mainly applied for identify- ing important training data for a task in prior works, does not show improvement in mitigating forgetting compared to replaying random examples. This implies that example importance defined in these works are different from how easily the examples are forgotten. We also notice that MIR- T does not improve over random sampling in our setup. likely because of the small size of the retrieval candidates relative to the upstream examples. Upweighting examples with ground truth forgetting (GT) consistently reduces forgetting compared to random examples, shown in dash lines in Figure
Method
Cost
Random
FT(Y) 2FT(Y) EV 2FT(Y) + +EV(S) MC FT(Y) +EV(S)
Ground Truth
Offline MF
Online MF
MIR-T Gra..rdd
2FT(Y)+ +Y EE(IS) FT(Y)
By utilizing predicted forgetting by offline additive linear, KNN, and MF models, we statistically significantly reduce forgetting compared to random examples. The MF model achieves statistical significance in most the scenarios, which aligns with its top average prediction performance of example forgetting Table Utilizing online-predicted forgetting also statistically significantly improves over replaying random examples in 3 of the setups.
8

---

Computational efficiency discussions. Table summarizes the computation cost of the approaches as a function F-( of fine-tuning steps, function (-()) of upstream examples whose perplexity is evaluated, and the cost of matrix completion (MC) that is much smaller than I.I.M inference or training. We note the total number of upstream examples as N, the size of seed examples as S, and the number of fine-tuning steps as Y. As S is much smaller than M, majority of computational costs arise from fine-tuning FYT()) and upstream data evaluation EV EV(N). (N). Replaying with ground truth forgetting is the most costly, as it introduces inference over potentially very large-scale upstream data. The offline prediction and replay approach saves computations in the scenarios of small fine-tuning datasets and massive upstream data, which is often true in practice. Online prediction of forgetting does not incur extra cost of fine-tuning or upstream data inference, and thereby always being efficient.
Effect of replay new task learning and patterns of forgetting. We show in Appendix F that targeted replay also slightly improves new task learning perplexity. Appendix [H] demonstrates mild change in the pattern of forgetting in the held-out upstream examples when replay is applied.
Further discussions. We evaluate fine-tuned OLMo-7B and OLMo-7B-Instruul on unseen LLM leaderboard tasks and present the results in Appendix and visualize the corresponding forgetting association matrix in Fig. [T Although we observe slightly improved performance of replaying pre- dicted examples to random or no replay on most forgotten tasks, we do not see statistical significance. Future works can mitigate downstream task forgetting with predicted forgetting with alternative algorithms that leverage past examples (Aljundi et al. [017]] (Buzzega et al.) 2020). Our results in Appendix
[ show that matrix completion and embedding based methods can be combined to better improve forgetting prediction performance. In Appendix [] we examine patterns of forgetting by fine-tuning over noisy or adversarial training data.
5 Related Works
Factors that affect forgetting. In this paper, we primarily studied how the associations between
learned and forgotten examples inform forgetting. Prior works have studied various factors that affect forgetting of the models, such as (1) type and size of the LM (Mehta et al., 2021] Scialom
ict al. 2022 Kalajdzievski, 2024; Mirzadeh et al. 2022 Ramasesh et al., al.] 2022) (2) trainable parts of
the model (e.g., LoRA, soft prompts, full-model tuning) (Biderman et al. [2024a; Razdaibiedina
ctt al. 2023) (3) hyperparameters such as learning rate (lbrahim et al. [2024; Winata et al.] [2023). dropout (Goodfellow al. 20144, number of training steps (Biderman al. |024b]|Klimml
et al.
2023) (4) optimizer (Lesort et al. 2023) and training algorithms (e.g., various continual learning
algorithms) (Smith al. 2022 Wang al. 2022 Shi et al.) 2024 Wu et al. (2024), (5) the upstream
examples or the knowledge themselves ves (Toneva et al. 2019 Zhang & Wul 2024). Future works can
study how these factors affect the predictability of forgetting. We consider empirical and theoretical study on the effect of task similarity on forgetting to be most relevant to ours. [Ostapenko et al. (2022) empirically study relationships between task similarity and forgetting in foundation models over a sequence of newly learned tasks; our work instead focuses on forgetting of upstream data of LLMs. Theoretical study by Doan et al.] (2020): Ding et al. (2024); [vrron et al. (2022) dissects effects of the learned tasks on forgetting in linear models or around model imitialization. We believe empirical
study (Wang et al. 2023 Li et al. 2024a Zheng et al. 2025) and interpretations of forgetting (Tao eet al. 22223 Zhao et al. 2023) Kotha et al. (024)) are complementary to ours and can potentially explain in the future why the associations in are often simple, and in which cireumstances the associations become more complicated.
Data selection and data attribution. Related our work, data attribution studies faithful algorithms to find training examples that account for prediction (Koh & Liang 2017): Ilyas et al.) (002) from a pool of training examples. Park et al. (2023); Xia et al. (2024); Li et al. (2024c); Liu et al. (2024)
study the problem of selecting a subset of training data that maximizes performance on given
domain or task at a fixed budget for LLMs. Feldman & Zhang (2020) Tirumala et al. (022):
Biderman et al. (2024b); Swayamdiptaa et al. (2020) identify memorized, important, or forgetful training data. However, the notion of data importance these works is different from how likely the upstream examples will be forgotten during fine-tuning. Furthermore, systematical study on
how such importance dependent on newly learned tasks is still absent. Prior works represented
by Aljundi et al. (2019a); |Wang et al. (2024): Aljundi et al. (2019b); Huang et al.) (2024); > Sun et al.
(2019) study example selection or synthetic example generation strategies for replay-based continual learning algorithms.
9

---

Predicting model behaviors. LLMs can display a hybrid pattern of unpredictable to highly pre- dictable behaviors (Ganguli al. 2022; Wei al. 2022). Yt al. (2023); Xia al. (Schram et al. (2023) study prediction of task performance across datasets and training setups. We perform prediction the example level which more fine-graing ane-grained and under-explored.
Conclusions
In this paper, we empirically analyzed the associations between learned and forgotten examples LM fine-tuning. We showed that simple low rank patterns are dominant in the example associations and compared the complexity of the associations across model types and sizes. We showed the example associations alone offer useful information to predict example forgetting when fine-tuning LMs on new tasks. We demonstrated the practical utility our analysis by showing reduced forgetting as we reweight examples for replay with predicted forgetting. We discuss limitations and future work in Appendix A
References
Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., and Tuytelaars, T. Memory aware synapses: Learning what (not) forget. ArXiv, abs/1711. 11177711..960,, 2017.
Aljundi, R., Caccia, I., Belilovsky, E., Caccia, M., Lin, M., Charlin, L.., and Tuytelaars, T. Online continual learning with maximally interfered retrieval. Neural Information Processing Systems, 2019a.
Aljundi, R., Lin, M., Goujaud, B., and Bengio, Y. Gradient based sample selection for online continual learning. In Neural Information Processing Systems, 2019b.
Biderman, D., Ortiz, G., Portes, J., Paul, M., Greengard, P., Jennings, C., King, D., Havens, S., Chiley, V., Frankle, J., Blakeney, C., and Cunningham, J. P. Lora learns less and forgets less. ArXiv preprint, abs/2405500963... 2024a.
Biderman, S., Schoelkopf. H., Anthony, Q. G., Bradley, H., O'Brien, K., Hallahan, E.,, Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp.
PMLR, 2023.
Biderman, S., PRASHANTH, U., Sutawika, L., Schoelkopf, H., Anthony, Q., Purohit, S., and Raff, E. Emergent and predictable memorization in large language models. Advances in Neural Information Processing Systems, 36, 2024b.
Blanco-Justicia, A., Jebreel, N., Manzanares-Saloo B., Sánchez, D., Domingo-Ferrer, J., Collell, G., and Beik Tan, K. Digital forgetting in large language models: A survey of unlearning methods. Artificial Intelligence Review, 58(3):90, 2025.
Buzzega, P., Boschini, M., Porrello, A., Abati, D., and Calderara, S. Dark experience for general continual learning: strong, simple baseline. ArXiv, abs/2004.07211, 2020.
Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. Efficient lifelong learning with a- a-GEM. In International Conference on Learning Representations, 2019.
Chen, Y., Marchisio, K., Raileanu, R., Adelani, D., Saito Stenetorp. P. L. E.., Riedel, S., and Artetxe, M. Improving language plasticity via pretraining with active forgetting. Advances in Neural Information Processing Systems, 36:31 16:33553-31557, 2023.
Chen, Y., Xu, X., Lu, Y., Stenetorp, P., and Franceschi, L. Jet expansions of residual computation. arXiv preprint arXi::2410.06022. 2024.
Conmy, A., Mavor-Parker A. N., Lynch, A., Heimersheim, S., and Garriga-Alonso, A. Towards auto- mated circuit discovery for mechanistic interpretability. In Proceedings of the 37th International Conference Neural Information Processing Systems, pp. 16318-1638 16352, 2023.
10

---

Conover, M.. Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wen- dell, P.. Zaharia, M., and Xin, Free dolly: Introducing the world's first truly open instruction- tundod 1lm, 2023. URL https: /wwwwwwwwwww///// databricks. com
Cui, G.. Yuan, L., Ding, N., Yao, G., He, B.. Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., et al. Ultrafeedback: Boosting language models with scaled ai feedback. In Forty-first International Conference on Machine Learning, 2024.
Databricks. Mpt-7b: A standard for open-source, commercially usable Ilms, 2023. URL atttps://ww. databriaiiiooo cks. www.blog/mptt-/////
de Masson D' A'utumee C., Ruder, S., Kong, L., and Yogatama, D. Episodic memory in lifelong language learning. Advances in Neural Information Processing Systems, 32, 2019.
Ding, M., Ji, K., Wang, D., and Xu, J. Understanding forgetting in continual learning with linear regression. In Forty-first International Conference on Machine Learning, 2024.
Doan, T. V., Bennani, M. A., Mazoure, B., Rabusseau, G., and Alquier, P. A theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In International Conference on Artificial Intelligence and Statistics, 2020.
Evron, I,, Moroshko, E., Ward, R., Srebro, N., and Soudry, D. How catastrophic can catastrophic forgetting be in linear regression? In Conference on Learning Theory, pp. 4028 4079. PMLR, 2022.
Feldman, V. and Zhang, C. What neural networks memorize and why: Discovering the long tail via influence estimation. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Ganguli, D., Hernandez, D., Lovitt, L., Dassarma, N., Henighan, T., Jones, A., Joseph, N., Kernion, J., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Elhage, N., Showk, S. E., Fort, S., Haffield-Dodds, Z., Johnston, S., Kravec, S., Nanda, N., Ndousse, K., Olsson, C., Amodei, D., Amodei, D., Brown, T. B., Kaplan, J., McCandlish, S., Olah, C., and Clark, Predictability and surprise in large generative models. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 2022.
Gao, L.,, Biderman, S., Black, S., Golding, L.,, Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint 10t.:0001000077 2020.
Goldfarb, D., Evron, I., Weinberger, N., Soudry, D., and HAnd, P. The joint effect of task similarity and overparameterizztion on catastrophic forgetting - an analytical model. In The Twelfth International Conference Learning Representations, 2024.
Goodfellow, J., Mirza, M., Da, X., Courville, A. C., and Bengio, Y. An empirical investigation of catastrophic forgeting in gradient-based neural networks. In Bengio, Y. and LeCun, Y. (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A., Ivison, H., Magnusson, I., Wang, Y., Arora, S., Atkinson, D., Authur, R., Chandu, K. R., Cohan, A., Dumas, J., Elazar, Y., Gu, Y., Hessel, J., Khot, T., Merrill, W., Morrison, J. D., Muennighoff, N., Naik, A., Nam, C., Peters, M. E., Pyatkin, V., Ravichander, A., Schwenk, D., Shah, S., Smith, W., Strubell, E., Subramani, N., Wortsman, M., Dasigi, P., Lambert, N., Richardson, K., Zettlemoyer, L., Dodge, J., Lo. K., Soldaini, L., Smith, N. A., and Hajishirzi, H. Olmo: Accelerating the science of language models. ArXiv preprint, abs/2402.00838, 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Mea- suring massive multitask language understanding. 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
11

---

Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks. L. A., Welbi, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, I.. Training compute-optimal large language models. ArXiv, abs/2203. 15556, 2022.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, 7., Li, Y., Wang, S., Wang, I.., and Chen, W. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Huang, J., Cui, L.,, Wang, A., Yang, C., Liao, X., Song, I.., Yao, J., and Su, J. Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. In Annual Meeting of the Association for Computational Linguistics, 2024.
Ibrahim, A., Th' crien, B., Gupta, K., Richter, M. L., Anthony, Q., Lesort, T., Belilovsky, E., and Rish, Simple and scalable strategies to continually pre-train large language models. ArXiv preprint, ftss/2403.08763. 2024.
Ilyas, A., Park, S. M., Engstrom, L., Leclere, G., and Madry, A. Datamodels: Predicting predictions from training data. ArXiv preprint, abs/2202.00622,, 2022.
Ivison, H., Wang, Y., Pyatkin, V., Lambert, N., Peters, M. E., Dasigi, P., Jang. J., Wadden, D., Smith, N. A., Beltagy, I., and Hajishirzi, H. Camels in a changing climate: Enhancing Im adaptation with tulu 2. ArXiv preprint, abs/2311.10700 2023.
Jin, X. and Ren, X. What will my model forget? forecasting forgotten examples in language model refinement. In International Conference on Machine Learning. 2024.
Johnson, W. B. and Lindenstrauss, J. Extensions of lipschitz mappings into hilbert space. Contempo- rary mathematics, 26:189-200 206, 1984.
Kalajdzievski, D. Scaling laws for forgetting when fine-tuning large language models. ArXiv preprint, 2024.
Kleiman, A., Frankle, J., Kakade, S. M., and Paul, M. Predicting task forgetting in large language models. In ICML 2023 Workshop DeployableGenerativel homepage, 2023.
Koh, P. W. and Liang, P. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, ICML, 2017.
Kotha, S., Springer, J. M., and Raghunathan, A. Understanding catastrophic forgetting in language models via implicit inference. In The Twelfth International Conference on Learning Representa- tions, 2024.
Lee, J., Xiao, L., Schoenholz, S. S., Bahri, Y., Novak, R., Sohl-Dickstein, J. N., and Pennington, J. Wide neural networks of any depth evolve linear models under gradient descent. Journal of Statistical Mechanics: Theory and Experiment, 2020, 2019.
Lee, S., Goldt, S., and Saxe, M. Continual learning in the teacher-student setup: Impact of task similarity. In International Conference on Machine Learning, 2021.
Lesort, T., Ostapenko, O., Rodríguez, P., Misra, D., Arefin, M. R., Charlin, L., and Rish, I. Challenging common assumptions about catastrophic forgetting and knowledge accumulation. In Conference on Lifelong Learning Agents, pp. 43-65. PMLR, 2023.
Li, H., Ding, L., Fang, M., and Tao, D. Revisiting catastrophic forgetting in large language model tuning. In Findings of the Association for Computational Linguisties: EMNLP 2024, pp. 4297- 4308, 2024a.
Li, J., Armandpour, M., Mirzadeh, S. I., Mehta, S., Shankar, V., Vemulapalli, R., Tuzel, O., Farajtabar, M., Pouransari, H., and Faghri, F. Tic-Im: A multi-year benchmark for continual pretraining of language models. In NeurIPS 2024 Workshop on Scalable Continual Learning for Lifelong Foundation Models, 2024b.
12

---

Li, J.. Fang, A., Smymiss G.. Ivgi, M., Jordan, M., Gadre, S. Y., Bansal, H., Guha, E. K.. Keh, S. S., Arora, K., Garg, S., Xin, R., Muennighoff, N., Heckel, R., Mercat, J.-P., Chen, M., Gururangan.
s., Wortsman, M., Albalak, A., Bitton, Y., Nezhurina, M., Abbas, A., Hsieh, C.-Y., Ghosh, D., Gardner, J., Kilian, M., Zhang. H., Shao, R., Pratt, S., Sanyal, S., Ilharco, G., Daras, G., Marathe, K., Gokaslan, A., Zhang, J., Chandu, K., Nguyen, T., Vasiljevic, I., Kakade, S. M., Song, S., Sanghavi, S., Faghri, F., Oh, S., Zettlemoyer, L. S., Lo, K., El-Nouby, A., Pouransari, H., Toshev, A., Wang. S., Groeneveld, D., Soldani, L., Koh, P. W., Jitsev, J., Kollar, T., Dimakis, A. G., Carmon, Y., Dave, A., Schmidt, L., and Shankar, V. Datacomp-Im: In search of the next generation of training sets for language models. ArXiv, abs/2406. 1794, 2024c.
Lin, S., Hilton, J., and Evans, O. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, 2022. doi: 3/655//1/20222.cllllo
Liu, Q., Zheng, X., Muennighoff, N., Zeng, G., Dou, L., Pang. T., Jiang, J., and Lin, M. Regmix: Data mixture as regression for language model pre-training ArXiv, abs/2407.01 20244497..4,,, 2024.
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 13688, 2023.
Lopez-Paz, D. and Ranzato, M. Gradient episodic memory for continual learning. In Neural Information Processing Systems, 2017.
Maini, P., Garg, S., Lipton, Z., and Kolter, J. Z. Characterizing datapoints via second-split forgetting. In Advances Neural Information Processing Systems, volume 35, 2022.
Marion, M., Üstlin, A., Pozzobon, L., Wang, A., Fadace, M., and Hooker, S. When less is more: Investigating data pruning for pretraining Ilms at scale. arXiv preprini wrXiv:2309.04564 2023.
McCloskey, M. and Cohen, J. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Mehta, S. V., Patil, D., Chandar, S., and Strubell, E. An empirical investigation of the role of pre-training in lifelong learning. J. Mach. Learn. Res., 24:214:1-214:50, 2021.
Mirzadeh, S., Chaudhry, A., Yin, D., Hu, H., Pascanu, R., Görür, D., and Farajtabar, M. Wide neural networks forget less catastrophically. In International Conference on Machine Learning, ICML, 2022.
Nguyen, T. T., Huynh, T., Ren, Z., Nguyen, P. L., Liew, A. W.-C., Yin, H., and Nguyen, V. A survey of machine unlearning. arXiv preprint arXiv:2209.0222. 2022.
Ostapenko, O., Lesort, T., Rodr'iguez, P., Arefin, M. R., Douillard, A., Rish, I., and Charlin, L. Continual learning with foundation models: An empirical study of latent replay. In CoLLAs, 2022.
Park, S. M., Georgiev, K., Ilyas, A., Leclerc, G., and Madry, A. Trak: Attributing model behavior at scale. International Conference on Machine Learning, 2023.
Raffel, C. Building machine learning models like open source software. Communications of the ACM, 66:38 40, 2023.
Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. Squad: 100,000+ questions for machine comprehension of text. Conference on Empirical Methods Natural Language Processing, 2016.
Ramasesh, V. V. Dyer, E., and Raghu, M. Anatomy of catastrophic forgetting: Hidden representations and task semantics. In International Conference on Learning Representations, 2021.
Ramasesh, V. V., Lewkowyez, A., and Dyer, Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2022.
13

---

Razdaibiedina, A., Mao. Y., Hou, R., Khabsa, M., Lewis, M. and Almahairi, A. Progressive prompts: Continual learning for language models. In The Eleventh International Conference on Learning Representations, 2023.
Reimers, N. and Gurevych, Sentence-bert Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URI. https://arxii/ www.iiv..rgabbsa.// 1908
10084
Roth, K., Udandarao, V., Dziadzio, S., Prabhu, A., Cherti, M., Vinyals, O., H'enaff, O. J., Albanie, S.. Bethge, M., and Akata, Z. A practitioner's guide to continual multimodal pretraining, ArXiv, abs/2408. 14471, 2024.
Sarwar, B. M. Karypis. G., Konstan, J. A., and Riedl, J. Item- based collaborative filtering recommen- dation algorithms. In Shen, V. Y., Saito, N., Lyu, M. R., and Zurko, M. E. (eds.), Proceedings of the Tenth International World Wide Web Conference, WWW, 2001. doi: 00.1144/3719203377077
Schram. V., Beck, D., and Cohn, T. Performance prediction via Bayesian matrix factorisation for multilingual natural language processing tasks. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Dubrovnik, Croatia, 2023.
Scialom, T., Chakrabarty, T., and Muresan, S. Fine-tuned language models are continual learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, 2022.
Shi, H., Xu, Z., Wang, H., Qin, W., Wang, W., Wang, Y., and Wang. H. Continual learning of large language models: A comprehensive survey. ArXiv preprint, abs/2404.16788 2024.
Smith, J., Karlinsky, L.., Gutta, V., Cascante-Bonne Bonilla,, P., Kim, D., Arbelle, A., Panda, R., Feris, R. S., and Kira, Z. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. 2023 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 11909-11919 2022.
Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K. R., Dumas, J., Elazar, Y., Hofmann, V., Jha, A., Kumar, S., Lucy, L.,, Lyu, X., Lambert, N., Magnusson, I., Morrison, J. D., Muennighoff, N., Naik, A., Nam, C., Peters, M. E.,, Ravichander, A., Richardson, K., Shen, Z., Strubell, E., Subramani, N., Tafjord, O., Walsh, P., Zettlemoyer, L., Smith, N. A., Hajishirzi, H., Beltagy, I., Groeneveld, D., Dodge, J., and Lo, K. Dolma: an open corpus of three trillion tokens for language model pretraining research. ArXiv preprint, abs/2402.001 59, 2024.
Sun, F.-K., Ho, C.-H., and yi Lee, H. Lamol: Language modeling for lifelong language learning. In International Conference Learning Representations, 2019.
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D.,, and Wei, J. Challenging big-bench tasks and whether chain-of-thought can solve them. ArXiv preprint, abs/2210.09261, 2022.
Swayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Hajishirzi, H.,, Smith, N. A., and Choi, Y. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online, 2020. doi: 10. 11188861122220,1 wwwweennnammaaee mopp-mmi..74
Tao, M., Feng, Y., and Zhao, D. Can BERT refrain from forgetting sequential tasks? probing study. In The Eleventh International Conference on Learning Representations, 2023.
Tirumala, K., Markosyan, A. H., Zettlemoyer, L., and Aghajanyan, A. Memorization without overfit- ting: Analyzing the training dynamics of large language models. ArXiv preprint, abs/2205.10770, 2022.
TogetherAl. Redpajama: an open dataset for training large language models, 2023. URL https Eeiithub.coom www.th...ccmm/oggeeeeeee.ee wtt.hh/wwwo.ogggeeeeeoooooeeeeae Data
14

---

Toneva, M., Sordoni, A., des Combes, R. T., Trischler, A., Bengio, Y., and Gordon, G. An empirical study of example forgetting during deep neural network learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019.
Walsh, P.. Soldaini, L., Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu. Y., Huang, S., Jordan, M., Team, O., et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.0000 00656, 2024.
Wang, Y., Liu, Y., Shi, C., Li, H., Chen, C.,, Lu, H., and Yang, Y. Insel: A data-efficient continual learning paradigm for fine-tuning large language models with instructions. In North American Chapter of the Association for Computational Linguistics, 2024.
Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.-Y., Ren, X., Su, G., Perot, V., Dy, J., et al. Dualprompt: Complementary prompting for rehearsal- continual learning. In European Conference on Computer Vision, pp. 631 148. Springer, 2022.
Wang. Zz, Yang. E., Shen, L.,, and Huang, H. comprehensive survey of forgetting in deep learning beyond continual learning, IEEE Transactions on Pattern Analysis and Machine Intelligence, 47: 1464-1483, 2023.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgcaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. Emergent abilities of large language models. arXiv preprint wwXiv:2066076882 2022.
Winata, G., Xie, I.., Radhakrishnan, K., Wu, Jin, X., Cheng, P., Kulkarni, M., and Preotiuc-Pietro D. Overcoming catastrophic forgetting in massively multilingual continual learning. In Findings of the Association for Computational Linguistics: ACL 2023, 2023.
Wu, T., Luo, L., Li, Y.-F., Pan, S., Vu, T.-T., and Haffari, G. Continual learning for large language models: A survey. ArXiv preprint, abs/2402.01364 2024.
Xia, M., Anastasopoulos, A., Xu, R., Yang, Y., and Neubig, G. Predicting performance for natural language processing tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, 2020. doi:
Xia, M., Malladi, S., Gururangan, S., Arora, S., and Chen, D. Less: Selecting influential data for targeted instruction tuning. ArXiv preprint, abs/2402.00333,, 2024.
Ye, J., Liu, P., Sun, T., Zhou, Y., Zhan, J., and Qiu, X. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. ArXiv, abs/2403.16952) 2024.
Ye, Q. Fu, H. Y., Ren, X., and Jia, R. How predictable are large language model capabilities? case study on big-bench. Conference on Empirical Methods Natural Language Processing, 2023.
Zhang. X. and Wu, J. Dissecting learning and forgetting in language model finetuning. The Twelfth International Conference Learning Representations, 2024.
Zhao, H., Zhou, T., Long, G., Jiang, J., and Zhang, C. Does continual learning equally forget all parameters? In International Conference on Machine Learning, 2023.
Zheng, J., Cai, X., Qiu, S., and Ma. Q Spurious forgetting in continual learning of language models. In The Thirteenth International Conference on Learning Representattinss 2025.
15

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer:[Yes]
The claim about the prevalence of simple associations between learned tasks and upstream examples is supported in Sec.3) The claim about the effectiveness of predicting forgetting and forgetting mitigation supported in Sec.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:[Yed Yes]
Justification:
The limitations are discussed in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [NA].
Justification:
The paper does not contain theoretical claims.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
The details about how the forgetting statistics are collected, such as LLM training hyperparameters, data preprocessing steps, are included in Section 2 and Ap- pendix C The implementation details and train-test split of forgetting prediction and replay are included in Appendix and
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
17

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Please checkout the attached supplementary material for the code and the forgetting statistics collected.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co.
soi.lmmmsiiioooociooooocc for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips. www.cc/pwwpcppccccecceoeooooeeee gttc://wiissseeeeeee www.eeeuiissioonnooooooooooooeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The hyperparameters and how they chosen are discussed in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
The statistical significance is included in Figure 5 about mitigation for forget- ting by upweighting upstream examples for replay according to the predicted forgetting. The paired t-test assumes the differences between paired measurements are normally distributed. and that the sample of pairs are randomly sampled from the population.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The details are included in Appendix C
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The authors confirm they have reviewed and conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader impacts discussed in Appendix
19

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not consider the paper poses such risks as it does not release new models or release data that cause harm (e.g., the statistics about upstream example forgetting).
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The licenses are discussed in Appendix C
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The attached supplementary material contains README about the assets submitted with the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification:
The LLMs are used for fixing language errors only.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
22

---

A Limitations
Although our work aims to be extensive, we consider there are several limitations that can be taken as future work.
Forgetting under sequential task learning. In this paper, we analyze and predict forgetting of models that are separately fine-tuned over diverse downstream tasks. We did not explore forgetting under continual or sequential task learning. Although we consider time series forecasting to be relevant to this generalized forgetting prediction problem, we decide to leave it as future work.
The scope of the definition of forgetting, We mainly measure forgetting in log perplexity increase or exact match over upstream training data of LLMs. The choice of the metrics is based on the considerations that (1) they are connected theoretical approximations of forgetting with first-order techniques (Sec.3.3) allowing proper comparison, and (2) we aim to analyze forgetting upstream examples that the models have seen before and minimize confounders such as generalization effects on unseen data.
Nevertheless, note that the forgetting can be defined in many possible ways, each with different focus. Some measures of forgetting, such as an aggregated number of performance drop on domain- specific downstream tasks, are often more sensible downstream LLM users. Some existing works also separate out "shallow forgetting" of language models, where the forgetting is easily recovered by twisting the prompts (Kotha al. 2024 Chen et al.) 2024). Importantly, forgetting defined in different ways may not always align with each other. We again remind the readers of the definition of forgetting used in the paper, and we consider analysis under different definitions of forgetting to be meaningful future work.
Theoretical analysis of low-rank example associations. While we conclude that low-rank example associations are prevalent in LLM forgetting from empirical results, we do not have a theoretical guarantee on why example associations must be low rank, and in which cases the example associations become more complicated. We leave a theoretical understanding of low-rank example associations as future work. Besides,
consider that the line of research on mechanistic interpretability of LLMs (Conmy et al. 2023). which dissects subsets of neurons or computation graphs that are crucial for a prediction, can offer meaningful insights for interpreting the low-rank property.
Scope of the experiment setups. Our analysis is restricted to supervised fine-tuning of LMs; other learning setups such as reinforcement learning are not studied. Besides, although we try to cover a broad category of newly learned tasks, the coverage may not be extensive. Due to limited computational resources, it also not possible to repeat experiments under extensive hyperparameter configurations (e.g., learning rate schedulers), which we leave for future work. Nevertheless, we provide analysis under more learning rate and batch size setups in Appendix H
Requirement of open access to upstream data. Our analysis requires open access to upstream data to evaluate forgetting over them. As result, replication of our analysis can be limited to LLMs trained on open data or in-house data that can be accessed. In addition, similar to almost every replay-based algorithm, our forgetting mitigation algorithm requires access to upstream data.
Scale of the upstream examples. Due to computational resource restrictions, we significantly subsampled upstream pretraining corpora of LLMs in our analysis. Although expanding the set of upstream examples would not theoretically affect the low-rank observations, we decided to leave larger scale analysis as future work. Besides, we consider that matrix completion based forgetting prediction permits sparse association matrices, where upstream example forgetting zij is sparsely collected, which is a feasible path towards generalizing the forgetting prediction framework to larger scale setups.
Separating out intended forgetting from unintended forgetting, Aligning with most prior works on continual learning, our experiments focus on the mitigation of forgetting. We consider the goal aligns with our experiment setup, upstream examples from Dolma and Tulu are carefully curated and cleaned, and the newly learned tasks intend to accumulate knowledge instead of overwrite previous knowledge. However, we highlight that forgetting is not always detrimental; in fact, the models are often expected to forget (unlearn) sensitive, noisy, and outdated examples (Nguyen et al., 2022 Blanco-Justio et al. 2025). In addition, active forgetting can serve strategy to enhance the plasticity of models when learning new knowledge (Chen et al. 2023). We believe that our analysis of forgetting is helpful for future work that performs targeted forgetting of knowledge.
23

---

Dally
ILANSummarizain
FLANGGA
FLANONLI
-22
FLAMMMIC
FLAM Generreetinnn
Dolma
(a) OLMo- forgetting over Dolma after full hull-parameter fine-tuning on FLAN and Tulu.
Dally
RLAUTranstttion
08
FLANIFarRp
FLANONU
-22
FLAMMIIC
FLAACLingulct:
PLAbUCianddiciiion
Dolma
(b) OLMo-7B (LoRA); forgetting over Dolma after full- parameter fine-tunin ine-tuning FLAN and Tulu.
Dolyf
372
FLAWTondaation
FLANSUmmen mmmr caton
365
ILAUParaphraee
FLANINU
-35
LANiGeneration
-19
FLANI iguish
RedFamm>
(c) MPT-7B; forgetting over Dolma after full-par parameter fine-tuning on FLAN and Tulu.
3.5
GLM
33
B8H
-22
-34
42
<pocee
wizarder
orc4
Acclencohhc
(d) OLMo- MBMM------ Instructtt; forgetting over Tulu after full-parameter fine-tuning on unseen instruction cestruction-tuning tasks.
Figure 6: Additional visualized matrices of associations between learned tasks and forgotten examples. We plot forgetting (log-perplexit, increase) that occurs on an upstream example (in æ-axis) after learning new task (in y-axis). pog-perplyxyty increase can be zero or negative, which indicates no forgetting.
B Broader Impact
Our research tries to better demystify and mitigate forgetting LLM fine-tuning. We expect that the better understanding alongside the reduced forgetting can, in turn, encourage model developers to promptly update their I.LMs to address limitations the models and improve their models in continuing efforts. We expect the broader application of the continual learning practice will reduce training costs compared re-training models, and ultimately result more powerful models under controlled training budget.
Although we do not direct negative impact of predicting example forgetting, highlight that in real-world continual learning setups, blindly mitigating forgetting may result in outdated knowledge and data privacy breaches in LLMs. Dissecting beneficial and intended forgetting from unintended or catastrophic forgetting requires attention in real-world setups.
C Dataset, Model, and LM Training Details
Subsample of upstream datasets. For OLMo experiments, we sample 141,876 text chunks with length 2,048 from Dolma v1.6-sample as upstream examples. For OL. we randomly sample approximately balanced number of examples from each task in Tulu, and filter out examples with input length that exceeds 2,048 (the limit of OLMo models) after tokenization. This results in 10,718 examples. For OLMo2, we sample 70,000 text chunks with length 2,048 from OLMo2-Mix.

---

Dally
TLAUSummriizlle
FLANGGA
FLANONLI
-22
FLAMMRRC
FLAM Generreetinnn
FLANULnggeee
FLAM Cisiiiiaiiiiiiinnn
Dolma
(a) Log perplexity increase (forgetting)
FLAMSammmrr oation
FLANOOA
FLANPaaaphaae
FLANNUI
FLANWRRC
FLANkGeneration
FLAUCINSSP catton
Odma
(b) Inner products gradients weight differences (zty)
20
Delly?
Tulo
10
FLAM Barsaaaiioio
FLA OLA
FLANNhwrapke
FLAN
FLANUGemee
Lingelili
(c) Negated inner products gradients (-af))
Figure 7: A side-by-side comparison between the matrices of forgetting, inner products of gradients and weight differences (zij) F-W "), and the negated inner products of gradients (-2(-)) we examined in Sec. 3.3
For MPT and Pythia, sample 10,000 2,048-token text chunks from RedPajama and the Pile respectively.
Learned new tasks and their categorization. We summarize the list and the categorization of newly learned tasks in Tables 15 and 16 We also annotate tasks used as in-domain training or test tasks.
Training and evaluation details. For full-parameter fine-tuning of non-instruction-tunee LLMs of all types, we train the model for 1,000 steps with an effective batch size of 8 and linearly decaying learning rate of 26-6, The learning rate is chosen among \ lle -6 2e-6, 5e 5e-6 that achieves the the best average validation perplexity after fine-tuning OLMo-7B on randomly chosen tasks from FLAN, For L.MMo-BB-Insrune and MMLU, BBH, TruthfulQA and Dolly, considering the small size of the training sets, we train the models only for 100 steps with an effective batch size of 8. For OLMo2-SFP 1-fo2-FFT-Minn tasks, we train the model for 1,000 steps. We use HuggingFace Transformers library for training and VLLM library for efficient inference. Due computational resource limitations, the statistics of forgetting are collected in single run.
Dataset and licenses. MMLU, BBH, and the Pile are released under MIT license. Truthful QA, Dolma, Redpajama, OLMo models, OLMo2 models, Pythia models, and MPT models are released under Apache 2.0 license. Tulu V2, OLMo2-Mix, and OLMo2-SFT-Mix are released under ODC-By license. Dolly is released under CC BY-SA 3.0 license.
Computational Infrastructure. We used 4 Quadro RTX A6000 GPUs for fine-tuning LLMs, and used Quadro RTX A6000 GPU for LLM inference.
D Details of Forgetting Prediction and Replay
Data Splits for Predicting Example Forgetting, We mark the tasks used as in-domain test splits for predicting example forgetting (Sec. (4) in Tables a5d T6 The train-test split for the in-domain tasks is randomly generated.
25

---

Boly
TruthfuiDA
MMLU
Figure 8: Visualized matrices of associations between learned tasks and forgotten examples on FLAN after fine-tuning M.Mo-7B-Instruue measured with binary exact match drop. Each colored pixel (zij = 1) indicates forgetting of an upstream example I, after fine-tuning the model on the task T,
Table 5: Mean and standard deviation of forgetting, and the percentage where the forgetting 2ig is non-negative, aggregated over the matrix The table uses the same experiment setup as Figure
Models Statistics
Avg. Forgetting
Std. Forgetting
Non-Nog
OLMo-1B
0.0261
0.0425
90.36
OLMo-7B
0.0804
0.0626
96.29
MPT-7B
0.0917
0.0870
98.32
OL-Mo-BB-nstrruct
0.0637
0.3700
49.61
OLMo2-7B
0.0045
0.0086
81.77
OLMo2-13B
0.0038
0.0078
80.11
Pythia-IB
0.1248
0.1406
98.53
Pythia-3B
0.0425
0.0523
89.24
Pythia-6.9B
0.1408
0.1517
99.20
Pythia-12B 12B
0.1555
0.1691
99.03
Training and evaluation details. For MF, we set the dimension the learnable features (rank) as 5. We train the regression models for 1,000 epochs over the association matrices.
For in-domain test splits, we randomly sample upstream examples and assume the ground truth forgetting is known for these examples. This is required for predicting forgetting on the rest of upstream examples by additive linear, MF, and KNN methods. We repeat the experiment 10 times.
For forgetting prediction based on SBERT embedding similarity, we used all-distilroberta-- as the encoder. We also experiment with rext-emeedding-3 3-mmlll model by OpenAL. The only trainable part of the prediction models is the MLP layers on top of the text embeddings. We train the models for 2,000 epochs. We note that at inference time the approach does not require ground truth forgetting of a small number of examples.
Sensitivity analysis to the rank in MF models. T summarizes the RMSE of forgetting pre- diction with MF models across different ranks r. The performance increases with the be- ginning and then drops, indicating an overfit.
Table 6: RMSE of forgetting prediction with ma- trix factorization (MF) models under different ranks.
Replaying upstream examples in fine-tuning. We sparsely replay mini-batch of 8 upstream examples every 32 steps of model updates while fine-tuning new tasks. An exception is fine-tuning of Dolly, where perform replay every 8 steps given the smaller number of model training steps. Given predicted ground truth forgetting
Model Rank
3
5
8
10
OLMo- 7B
7.31
7.17
7.14
7.12
7.14
OLMo-7B-OOD
6.09
5.78
5.76
5.77
5.77
models on
OLMo- IB
2.85
2.82
2.80
2.78
2.78
OLMo- IB-OOD
2.86
2.83
2.82
2.82
2.82
2,1..N on upstream examples 1 when learning new task T, we sample upstream examples to replay from categorical distribution where p(z;) x exp(21./r)) where T is temperature hyper- parameter set as 0. The hyperparameter is tuned on single validation task while reweighting replay examples with the ground truth forgetting Z.
E Details of the Example Similarity Metrics
In this section, we detail the example similarity metrics applied in our analysis in Sec. 3.3
26

---

Doly
62-
FLANSS um comartation catton
FLANTTansiatton
60
FLANFaraphras::
PLAuGermnation
FLANLirguati
FLAMCCless Eation
(a) OLMo-7B, full association matrix
FLANTTonsiation
TLANDS an mar selion
60-
FLANFaraphroue
FLANNRR
FLAvSSencration
gusstic
FLANIC
(b) OLMo- MB, compontent 1
FLANS am mar tation
FLANaaraphrase
FLANNU
FLANDMRC
FLANLL irgusti
(c) OLMo-7B, component k: = 2
0.4
mastraltinn
FLANCA
FLANUNU
42-
FLANMMRC
FLAtiLingustic
(d) OLMo-7B, component le =3 3
Doly
64-
FLANTTransaaton
LLANGA
FLANdarrahhaam
FLANUU
433
FLANARON
FLAuGeneration
-64
FLANCCSsP Cation
common creel
(c) OLMo-BB, component k: = 4
Figure 9: Decomposition of Z OLMo-7B experiments with k-th singular value and vectors. Components of higher values of & capture finer finer-grained details in Z.
27

---

Daly
TuthidDA
BEH
alacca
aisaca
wizardmm
-dclenceec
(a) OLMo-7B 7B-nstrutt full association matrix
DIMoOST
MULU
1mm
081
03052
10m0
3lpaa
0011
abpaca
--Sharegpt
Mrardmm
(b) In--o-----nstruct,, compontent
MULU
1im
3baco
3630
wizandm
dclencee
(c) OL component & = 2
MULU
1pm
1mm
3baa
0PA
1036
-Pargg
Mizadd
(d) OLMM-7B-Insitrne component
Doli
84-
YuthidoA
x2-
BEM
1870
abbaca
alpata
wwwrddm
(e) OLMo-7B-Instren component &
Figure 10: Decomposition of log pegppeeelextty measured forgetting association matrix Z in OLMo-7B- Instruct experiments with k-th singular value and vectors. Higher values of capture finergrrined details in Z.
Token similarity. We calculate the textual cosine similarity between the learned tasks and upstream examples using TF-IDF vectorized representations of training examples in each learned task Tj and an upstream example Zy.
Representational similarity. We measure textual representation cosine similarity given by the (1) final layer representations of OLMo- MoMMoBBBBB (2) all-distilroberta-v sentence transformer (SBERT) model (Reimers & Gurevych] 2019). and (3) OpenAI foxt-embedding-3-ssmee model. We average representations of maximum 100 training examples as the representation of a task T.
Inner products between projected gradients and model weight updates. The increase of the log = = (Vo[(z;) OT; - 0o) under first-order perplexity žiy can be approximated with inner products 2ij
Taylor expansion (Lee et al. 2019 Doan et al. 2020 where Vef(x,) is the gradient of the loss of I at the initial model before line-tuning, and Uy O, are the updates in the model weights after fine-tuning. Following Park et al. (2023), Xia et al. 2024), we use a random projection matrix P ~ Nipj (0, 1) to reduce the dimenston of the gradients or the weight changes storing pre-computed statistics, which preserves the inner products with high probability (Johnson & Lindenstrauss 1984).
the cost of
Inner products between projected gradients. We also measure the negative inner prod- 58
uets of the loss gradients between the upstream example zj and learned task T. -2
28

---

100
FLANSSU marization
FLANDD
FLANINU
FLARHMIC
-.75
p:00
hellowwag
(a) OLMo-7B, full association matrix of forgetting on LLM leaderboard tasks
FLAAUTaaddaainn
FLANISUm
FLANOA
-03
MLA
PLANNILI
-1.00
MANiCtassiicaton
piao
1 oFC
-
hollowag
150
2000
(b) OLMo- 7B, forgetting on LLM leaderboard tasks, Z1
Doiy
Tuuu
636
FUWW/Tandtttin
FLANSUmmorta momtttattioo
FLANOOA
4.22
MLA
ILAvitanenation
RLAWCCasdiitation
-1.00
+10rce
hellsowag
boolg
(c) OLMo-7B, forgetting LLM leaderboard tasks, Zz Zi
Daily
FLAN/Toanttttio
FLANISum momartatiionn
rogh rase
-.25
PLANDILI
ILANcteeenniion FLANCE CoCCleodiiiaain
-1.00
hellesmeg
boslq
(d) OLMo-7B, forgetting on LLM leaderboard tasks, Zz Zz
1,00
FLANISU martz partaaiion
FLANOO4
FLAR/Porsehrase
FLANNNU
FLANHRRE
TL TLAAUUgguee
pige
(e) OLMo-7B, forgetting on LLM leaderboard tasks, Z4 Zs
The forgetting association matrix Z between M 85 learned tasks and N 1, 000 examples from the LLM leaderboard tasks. The forgetting is defined as a prediction turning from correct to incorrect after fine-tuning. We further show the differences Z,- Zy-1 between rank-r and rank-r - approximations. We note that Z is binary, and we define the rank r approximation 2" as Er xxr,3% ). where a is the sigmoid function.
Figure 11:
Vo (a (T))), as an approximation of forgetting (Lopez-Paz & Ranzato, 2017 Chaudhry
al. 2019
F Effect of Targeted Replay on New-Task Learning and Downstream Performance
Targeted replay does not impede learning of new tasks, as shown with decreased validation perplexity. Table fummarizss log perplexity on the validation set of learned tasks after fine-tuning the models with different replay strategies. We notice that example replay and targeted replay often decrease validation perplexity of new tasks. This improvement of new task perplexity can be attributed

---

Table 7: Validation log perplexity of the learned tasks after fine-tuning LMs with different replay strategies. *; p.p < 0.05 improvement compared to no replay.
Model
OLMo-IB
OLMo-7B
OLMo-7B
OLMo-7B-Instruet
Dataset
FLAN
FLAN
Tulu & Dolly
OLMo2-SFT-Mix
No Replay
0,9723
0.7736
1.6397
0.6677
Random
0.9621
0.7691
1.6452
0.6671
MF-Offline
0.9601
0.7684
1.6294
0.6674
Table 8: Downstream task performance of OLMo-7B models before and after fine-tuning on dolly
tasks.
ARC-Easy
ARC-Challenge
Boolq
Hellaswag
Openbookqa
Piga
Seiq
Winogrande
Metries
Acc-norm
Acc-norm
Acc
Acc-norm
Acc-norm
Acc-norm
Acc-norm
Acc
Before FT
68.77
40.36
72.41
75.65
42.20
79.54
88.60
66.29
No Replay
67.34
42.28
74.82
76.89
44.65
80.05
84.09
67.89
Random
67.48
42.43
74.33
77.26
44.88
79.97
84.77
67.33
MF-Offline
67.52
42.35
74.50
77.56
44.25
79.97
85.10
67.17
to less forgetting of general knowledge during fine-tuning. Besides, the results imply that targeted replay does not simply trade off new task learning for reduced forgetting.
LLM leaderboard task evaluation. We evaluate the change of performance Open LLM Leader- board tasks]] before and after fine-tuning a model. We use same set of LLM Icaderboard tasks as the OLMo technical report (Groeneveld al. 2024). For OLMo-7B models, we evaluate down- stream performance of 8 models fine-tuned the Dolly tasks. For OLMo-7B Instruct, We evaluate downstream performance after fine-tuning 4 held-out tasks from OLMo2-SFT-NE Table 9 We compare LLM leaderboard task performance of no replay, replaying random examples, and replaying with forgetting predicted by offline MI. Table summarizes results on OLMo-7B; Taalee]/summarize results on OLMo-7B-Instruee
(marked in
We notice that fine-tuning OLMo-7B Dolly improves downstream performance most of the downstream tasks. This aligns well with the purpose of fine-tuning a LM that is not instruction-tume Nevertheless, notice performance degradation on two of the tasks, namely ARC-Easy and Scig, which indicates forgetting. Although offline MF Replay achieves higher accuracy scores on these two tasks compared to no-replay, do not find the improvement statistically significant.
over new tasks without any replay improves metrics scores on some tasks (MUSR, GPQA) but causes forgetting on other tasks (MMLU, BBH, IFEval). Among tasks where performance improved, we do not see benefits of example replay. Among tasks with decreased performance, we see offline-MF mitigates forgetting compared to no replay or random replay only very marginally.
see fine-tuning
We conjecture that replay-basee based approaches are not sufficient to significantly mitigate forgetting on their own, and can be combined with other approaches. We leave more effective algorithms to mitigate downstream task forgetting with predicted forgetting as future work.
G Towards Interpreting Fine-Grained Associations
We visualize progressive reconstruction with k-th singular value and singular vectors for OLMo experiments in Figure 9
Semantic meanings of the k-th component in the low-rank approximation of the association matrix Z. We perform further analysis into the patterns captured by the k-th singular value and singular vectors by identifying the most relevant learned tasks and upstream example domain to the component. For each & and its corresponding component au,3%. we extract top rows with the
30

---

Table 9: Average downstream task performance of OLMo-7B- Instrucct models before and after
fine-tuning on 4 held-out OLMo2-SFT-Mix tasks.
MMLU-Pro
BBH
IF-Eval
MUSR
GPQA
Metries
5-shot Ace
3-shot Ace-norm
O-shot Inst
O-shot Acc-norm
O-shot Acc-norm
Before FT
18.20
37.65
39.93
38.31
26.35
No Replay
17.21
36.76
21.04
40.20
27.99
Random
17.37
36.85
20.97
39.77
27.73
MF-Offline 17.43
36.89
21.14
39.50
27.69
Table 10: Semantic meaning of the k-th component the factorization of the example association matrix Z. We identify top relevant learned tasks and upstream example domains to the k-th component the factorization of the example association matrix Z.
OLMo-7B
OLMo-IB
Learned Tasks
Forgotten Domain
Learned Tasks
Forgotten Domain
flan/paws_wwe wiki
Ban/squad__ v2
Haníglue_mng mrpe
Nome
fhn/fix_punce
None
flan/story_ _eloze
tulu/open_ore
laatloinion_aasstacts.. wassssacts__ddeee _ddebate
flan/mali matched
A:
dollylgeneral.gg qu
StackOverflow (Less)
flan/mnli mismatched
None
flun/story_eloze
flan/snli
flan/story_elee _cloze
Bannsquad
= 3
flan/fix_,punce punct
Nome
flan/quae
None
flanvtnue_caee
flan/ti_ppune
math dataset dolly/general_ qu
flan/rte
b 4
Nome
flan/opinion abstracts idebate flankstory cloze
None
idehate
highest mean (i.e,, top 3 relevant learned tasks T,). We also extract top 50 columns with highest mean (i.e. top 50 relevant upstream examples) and the domain where these upstream examples are drawn from. For OLMo models, the domains are one of C4, common-crawl, Gutenberg books, Reddit, Science, StackOverFlow, and Wikipedia. We compare the distribution of domains in the top 50 upstream examples, and perform z-lest determine upstream example domain that is significantly more or less forgotten compared to a prior domain distribution of top 50 most forgotten upstream examples (columns with highest mean in Z). The results are summarized in Table [0]
We highlight some notable patterns in Table TO (1) Some component Z& highlights forgetting patterns of upstream examples from certain domains. On OLMo-7B, the second component (k: 2, also visualized Figure ()) highlights patterns where StackOverFlow examples are disproportionally more less forgotten. (2) Some components Zμ highlight forgetting when learning specific types of tasks. For example, the second component (k = 2) on OLMo- highlights forgetting patterns after learning NLI tasks (mnli matched, mnli mismaathed, snli). This also exemplifies how learning similar tasks causes similar sets of upstream examples to be more forgotten.
H Effect of Model Training Setups on Forgetting
In this section, we examine patterns of forgetting and its low-rank approximations across various training setups.
Low-rank approximations hold at various learning rates, batch sizes, and LoRA fine-tuning. Table summarizes R²2 of low-rank approximations. Reducing learning rate from the default 2c-6 to le-6 causes very minor change of R²; Nevertheless, we notice increased R² under a larger learning rate like 5e-6, indicating even simpler associations between the learned tasks and forgotten upstream examples. Under this overly large learning rate, the model suffers substantially more forgetting on all examples. Besides, increasing the batch size (under the same number of parameter update steps) causes an increase in R2, indicating simpler associations, possibly due to reduced variance of learned models under larger batch size. LoRA fine-tuning (Hu et al., (002)) results in simpler associations despite the model forgets less than full parameter fine-tuning.
31

---

Table 11: R2 approximating forgetting association matrix Z across different learning rates (LR) and batch sizes (BS). The default learning rate and batch sizes 2 and 8. We report results at M = 19 learned tasks from Tulu and Dolly and N = 141, 871 upstream examples from Dolma.
Model
Setting
y=
r
3
r 5
OLMo-7B
Default
0.6981
0.7542
0.8232
LR=le-6
0.7071
0.7697
0.8361
BS=32
0.8041
0.8453
0.9011
LR=5c-6
0,9331
0.9582
0.9718
LoRA, LR=le-4
0.8963
0.9307
0.9562
OLMo-1B
Default
0.8344
0.8836
0.9177
LR=1c-6
0.8471
0,8989
0.9214
BS=32
0.8724
0.9289
0.9607
Table 12: Sample Pearson correlation coefficient between upstream example forgetting under different
replay strategies. The results are collected with OLMo-7B models fine-tuned on 8 Dolly tasks.
Setup
Setup
Sample Pearson
No Replay
No Replay seed change
0.9403
No Replay
Random Replay
0.7919
No Replay MF-Offline Replay
0.8121
Random Replay
MF- OF-Offliee Replay
0.7709
Replay can change the patterns of forgetting compared no replay, but the change mild. We compute sample Pearson correlation coefficient between upstream example forgetting collected under different replay strategies, and more specifically (1) no replay (2) random replay and (3) MF-Offline replay OLMo-7B and 8 Dolly tasks. We evaluate forgetting of upstream examples held-out from replay. We include correlations between upstream example forgetting under two different random seeds during training the baseline.
We summarize the results Table 12] The correlations between no replay and replay are around 0.8, lower than the 0.94 baseline, but are still strongly positive. Replay example selection strategies (random or MF-offline) do not have strong impact on the correlation. The results imply replay changes patterns of forgetting to mild degree. Future work can study the limit where the patterns of forgetting experience notable changes.
I Combining Embedding and Matrix Completion based Prediction
Although our results in Sec. demonstrate subpar performance of the embedding- based approach, we show combining embedding and matrix completion approaches leads to improvement over the both. Specifically, we fit the residual error of the matrix completion model Z Zuscc with the embedding model, where ZMc is the prediction given by the matrix completion algorithms. We use the better
Table 13: Combining matrix completion and embedding based prediction of forgetting. We report
RMSE (L) or F1(f) predicting example forgetting over a held-out set of upstream examples after fine-tuning LMs unseen tasks.
In-Domuin
Out-of-Domman
OLMo-7B Instruct
OLMo-7B Instruct
OLMo-IB
OLMo-7B
OLMo-IB
OLMo-7B
Metrics
RMSE
RMSE
RMSE
RMSE
RMSE
RMSE
RMSE
RMSE
F1
Matrix Completion
2.79
7.14
10.41
13.74
58.16
2.82
5.76
38.90
43.57
SBERT
2.81
7.44
13.86
13.94
55.46
3.53
6.16
10.59
41.22
42.95
SBERT + MC
2.76
7.09
10.20
13.25
$9.36
2.76
5.75
7.00
40.36
42.98
32

---

2
62
Stuffed
Ofiginal
OLMo-7B, forgetting over Dolma after fine-tuning over Ultraeeddbcck with shuffled flipped preferred
responses,
18
Avg Z
Hipped
-18
Originl
OLMo- I-Intruct, forgetting over Tulu after fine-tuning over Ultrafeedback with shuffled or flipped preferred responses.
Figure 12: Forgetting after fine-tuning OLMo-7B OLMo-7BB-nstruutt Ultrafeedback dataset with original, shuffled, or flipped preferred responses.
Table 14: Correlations between column mean of Z (in main experiments in Sec. 3 visualized in Figures 2 and 6) and the forgetting occurred on upstream data after fine-tuning on clean, noisy (shuffled) or adversarial (flipped) variants of UltraFeedback.
OLMo-7B
Pearson p
Spearman P Avg. Forgetting
Pearson p
Spearman p
Avg. Forgetting
Original Shufiled
0.5884
0.4813
0.0466
0.7382
0.7632
-0.0432
0,5865
0,4591
0.0375
0.5624
0.5149
Flipped
0.6535
0.5289
0.0463
0.6516
0.6893
0.1103
of the additive and MF models reported in Table 3 Table 13 summarizes the results. We that combining the two approaches improves performance over the two.
We consider the embedding based approaches have potential to better capture fine-grained knowledge conflicts that leads to forgetting of one example by learning the other (e.g., two documents stating contradictory facts) that is missed by matrix completion approaches that rely statistics of forgetting.
Fine-tuning over Noisy or Adversarial Training Data
In this section, examine forgetting after fine-tuning clean, noisy, adversarial new tasks. We experiment with Ultrafeedback (Cui et al. 2024). an instruction tuning dataset where each instruction is paired with one most preferred response and many rejected responses. For the clean variant, we fine-tune the model over the preferred responses. For the noisy variant, we shuffle the set of the preferred responses that the instruction and the response mismatch. For the adversarial variant, we fine-tune tune the model over the rejected responses. We leave preference pair fine-tumi mee-tunnng that simultaneously utilize both preferred and rejected responses future work.
Figure 12 visualizes the forgetting on Dolma or Tulu after fine-tuning OLMo-7B and OLMo-7B instruct, represented in matrix Zu E R3XN We also include column mean of 2 (i.e., upstream example forgetting averaged over all tasks Ti..) in the main experiments in Sec. as references. In Table 14 we present the correlations between Zy and the column mean of Z.
On OLMo-7B, our results suggests that upstream example forgetting after fine-tuning on rejected responses do not correlates less with the task-averaged upstream example forgetting (column mean of Z) compared
tne-ttning on the clean data. On OLMo-7B LLLMo-BB-Instiin In-Ituct however, we see clear drop in the correlation by fine-tuning on the flipped data, very likely because the data contradicts the upstream instruction tuning data. Nevertheless, we notice the correlation with the task-averaged forgetting is still strongly positive. Meanwhile, fine-tuning on shuffled data causes forgetting that least correlated with the aver-ave upstream example forgetting, indicating more significant change in the patterns of forgetting. Still, we notice strongly positive correlation.
33

---

1.0
1.00
Overlop
Disjeint
0.9
Fargetting
0.8
0.90
0.6
0.7
Unnar
Linea
24ayer
2-layer
4-Layer
3-aayer
0.80
4-1ayer
3-Layer
0.2
0.5
L-yyeer
0.75
S-Layer
0.0
Rank
Rank
4. Hidden Layers
(a) Overlapping rotations
(b) Disjoint rotations
(c) Avg. Forgetting
Figure 13: 12 of low-rank approximations of the example association in the forgetting of MLP models on rotated MNIST experiments. We report R² when the rotation of the newly learned task overlaps or is disjoint with the upstream data separately. y-axes are not in the same scale.
K Synthetic Dataset Experiments
In this section, we present set of experiments synthetic dataset, Rotated-MNIST, broadly used in continual learning research. We aim to provide intuition for future research about how the complexity of the example associations can depend on (1) the coverage of knowledge represented in upstream examples, and (2) the size of the models, in highly controlled setups. We apply a training setup that resembles pretraining and fine-tuning paradigm in transformer language models. Specifically, the models are first pre-trained on 10 rotated variants (0-90%) of the MNIST digit classification dataset (as upstream tasks and examples). Then, the models fine-tuned on of 40 unseen rotations for one epoch. Among the 40 unseen rotations, 20 are drawn from the same range as upstream examples (0-90%), while the other 20 are drawn from disjoint range (-90-0%). This separation controls the amount of shared knowledge between the newly learned and upstream tasks.
We mostly apply other training setups and hyperparameters in Aljundi al. (2019)). Each task (rotation) includes 1,000 training examples. We train a MLP classifier to predict among 10 digits given an input image without providing its rotation (or the task identifier). We experiment MLP classification models with layer (a linear model) to 5 layers. We collect the example associations Z, and visualize them in Figure 14 The upstream and the newly learned rotation tasks ordered by their rotations the x or y axis.
Effects of knowledge coverage. When the rotations of the newly learned tasks overlaps with the range of upstream examples, the forgetting is harder to approximate with low-rank approximations, resulting in a Rt only around 0.5 for all MLP models with rank-1 approximation. In contrast, the R² scores are much higher when the rotations do not overlap, alongside higher average forgetting; R2 with rank-1 approximation is higher than 0.8 in these setups. The results imply that the amount of shared knowledge between fine-tuning tasks and upstream examples can have an impact on the complexity of the example associations. In other words, models trained with a broad coverage of upstream examples, or that cover knowledge required for diverse fine-tuning tasks, can yield more complicated example associations. In the context of LLMs, we have noticed that more powerful LLMs (such as OLMo and OLMo2, compared to Pythia and MPT) with broader coverage of knowledge yield more complicated patterns of forgetting.
Effects model sizes. We compare the MLPs of different number of layers trained on the same upstream data. From Figure 13 the patterns of forgetting nosier in deeper models. The R² scores at rank 3 or 5 decreases with added layers in MLPs, providing a quantitative measure of increased complexity between learned tasks and forgotten examples.
To summarize, our analysis with synthetic datasets provide intuition about the effect of knowledge coverage and model sizes on the complexity of example associations in forgetting. We hope the set of synthetic experiments can inspire more comprehensive study on how the complexity of example associations are affected by various factors in future work.
34

---

7
7 1o
To
(a) Linear, overlapping rotations
(b) Linear, disjoint rotations
Tra
Tux
T1
11p
5 TL 12
7
7 7. Th 76 B5 T5
(c) Linear, overlapping rotations
(d) Linear, disjoint rotations
Tua
T1
11
6 12 1
T6 10 7 1. To B. Ty
(e) Linear, overlapping rotations
(I) Linear. disjoint rotations
Tra
Tix-
1
To
T1p
1 1 1
4
12
1. T To To
(g) Linear. overlapping rotations
(h) Linear, disjoint rotations
To
Tor
15
16 4 T6 T6 16 To To Tg
(i) Linear, overlapping rotations
①i Linear, disjoint rotations
Figure 14: Example associations between learned tasks and forgotten upstream examples on Rotated MNIST with overlapping disjoint ranges of rotations. We measure increase in the cross-entropy loss as forgetting.
35

---

Task Category
Task
Task Category
Task
FLANCClastiiiation
acslc
FLANIQA
arc ute_challenge* are_casy"
news subset
imdh reviews sentiment1-00
hool, . coga" cosmos, .ga math math_dataset dataset*
trec
yelp_ het_ppolltyyyvviee reevewss*
questions"
FLAN/Linguistic
cola
openbookspa*
detinite pwwnounn resolution
piqa
fix_ _x_pu
trivia_ge* qa*
FLAN/Smmmaiizaiinn
dailymail
wond segment
gigawuri
Wse common, gen
multi
FLANIGeneraaion
samsuum
copa
wiki_ viki_jingua_engliae
FLAN/Translatio
para_crav llw.errwwooos cocs
e2e_nlg"
wmnt 14. enfr
bellaswag
wmi11 translaticccccco caen
opinion_ Cimiionabstracts abbstractss idehatee*
wint 16 wwt.ttanntteeee deen
opinion, abstracts, mitenn Jomatoes
translate
story_cluze
wanl16, wwt.trmmlateeoot roen
nig_en
translate
FLANMMCC
drop
wanl16 tramslate tren"
multire
Tulu
open_
quac record squad_yi vl
oasstl
lima
mode_alpace
squad v2
apt4_alpaca
FLAN/NLI
anli rl anli_r2 anlirr
col
science
flan
cb*
sharegpi
mnli matched
hard coded wizardIm
mali miimmahstth
qali tte
Dolly
brainstorming
closed_qa
information calsemation_extraattinnn extractiion
wnli gloc_mm pmpe
classilicaioon
FLANParaphraae
open_qe
glue_gap"
general qa creative_ writing
paws_wiki
stab
summarization
wie"
Table 15: The list of learned tasks in our experiments on OLMo-IB, IB, OLMo-7B and MPT-7B. notes for tasks used as the in-domain test split in forgetting prediction experiments in Sec.
36

---

Category
Task Category
MMLU
abstractalgeenn alggebra
objoct_countingg
objoct counting"
anatomy
penguins in
astronomy
reasoning about www__abbboooscccee cobjed
business, ethics
clinical, knowledge
salient, translation emur detection
college_biology"
college_ chcosssccsttty
sports_ undorstanding
college_compeee wwtppcooommscceeeeeeo science
colloge_ colloge_mathematics mathhhmatics
tracking shuffed. oooooots__siise five_ objocts tracking shuffled objects objects
college "eed_miciiie""
college_phyee physics
tracking. shuffled objects, three objects
compulerr_ security
web_of_in
conceptual physics*
word sorting
econometrics
TruthfuQQA
Nutrition
electrical eleetriaal_ongnncering
Sterootypes
elementary mattemmttas
Confusion
logic
Psychology
global. facts*
Language
high school hiology*
Sociology
Finance
high colp_sschoo___mueer scienne science
computer
Indexical Error
sich_xchhoo_ccm european, history*
Science
geograpphy geogggg
Misconceptions
high_school www.roomnttttcoommmoo and pollitit
Economics
high school macrocconomies
Education
high school mathematicss
Proverbs
scphoschool__ micrneconomies
Conspiracies
high schosscholl__.. physics*
Religiin
Beo__cchoo_pscchogge
Statisties
high_school statistics
Misquocations
high_school, history"
Subjective
high_schood world history
Law
human_aging" _anin_
History
sexuality*
Interationall law
Mandela Effect
jurisprulence
Politics
[allccl_fallacies* Fallacies* macline Jearning
Misinformation
Logical Falsehood
management*
Distraction
marketing"
Weather
medical genetics
medical_geneties
Myths and Fairytales
miscellaneous
Superstitions
moral, disputes
Advertising
mnoaa_scenarion
Paranormal
nutrition
Health
philosophy*
Dolly
brainstomming
prehistory
rte closed_ sali information extraction
professional accounting
professional
classification
professional medicine*
FLAN/Paraphras glue_mppe openiqu
professional psychalogy
glue qie__ggp" general, ga
public relations*
wikk cercative_wiiiine
socurity studics
summarization
sociology*
OLMo2SFT-Mix
coconot
us jo_foreign_ooiiy virology
fian_v2 v2
world religions boolean, expressions* causal judgement date understanding
rob_roats numinumath, lir math
BBI
cassil
personahub. _ode
disimbiguation_.e dyek languages*
personahab_ _idata
personahub, "acthuubb_hh""
formal fallacies*
geometric shapes
open__aate gsmm8k
hyperhaton"
personahub math intermi _aggehra
logical deduction, *bjectts
logical deduction seven_objecu
synthetic
logical deduction, three_objccts
table gpt
movie " multistep_ arithmetic navigate
wildehat wildjailbreak"
pessons-math-rrade
Table 16: The list learned tasks in our experiments on OLMo-7B- IB-ntruttt * notes for tasks used as the in-domain test split in forgetting prediction experiments in Sec.
37


Reviews:
[{"review_id": "ioYNy8GeO9", "reviewer": "Reviewer_9wDJ", "summary": "This paper explores how LLMs forget during finetuning. They do this by building forgetting matrices between “upstream” examples (i.e. examples that had good performance before finetuning) and new tasks (i.e. after finetuning). They propose a method to efficiently predict forgetting of upstream examples, as well as an algorithm to mitigate forgetting during LLM finetuning. This has many practical implications for LLM finetuning and dataset selection and will be of interest to a wide range of LLM practitioners.", "questions": "Q1: I assume all models except OLMo-7B-Instruct are base models\n\nQ2: It would be helpful to include some explicit examples in the appendix of data pairs/tasks where forgetting is high and where forgetting is low. This would help the reader build some intuitions around the very dense matrix plots that are presented (I know the Appendix is already quite extensive)\n\nQ3: Why is the matrix for OLMo-7B-Instruct much more different than the the matrices for the other models in Appendix Figure 6?\n\nQ4: “We empirically notice that the associations between learned tasks and forgotten upstream examples are more complicated in more recent and capable LLMs” - is this simply because more of the finetuned datasets are more likely to be included in OLMo pretraining?\n\nQ5: Why does OLMo2 have zero average forgetting in Figure 3(d)?\n\nQ6: Why do you use all-distilroberta-v1 as your embedding model? This seems quite outdated\n\nQ7: What exactly is the “ground truth forgetting” in Figure 5? Is it the performance of the model after full finetuning?\n\nQ8: What are the pros and cons between online and offline MF in Table 4?", "limitations": "The authors have addressed the limitations and potential negative societal impact.\n\nMinor issues:\n* The citation for MPT and RedPajama is [Computer 2023] which seems incorrect. Seems that most papers cite the original MPT blogpost \n* Line 129: “knwon” → known\n* The schematic in Figure 4 is slightly confusing. It might be helpful to split into (a) and (b) sections\n* Line 241: “The additive linear model approximate forgetting”", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n* Well written and well motivated. The authors find that associations between learned tasks and forgotten examples are often well approximated with low-rank matrices, and use this to motivate their main method: matrix completion over association matrices.\n* This paper is a tour de force across models (OLMo, MPT, Pythia, OLMo2, OLMo-Instruct) and datasets (FLAN, Tulu, Dolly), with an extensive appendix\n\nWeaknesses:\n* This paper presents a lot of empirical results that are somewhat difficult to interpret (i.e. all the matrix plots in the appendix). Aside: This is very helpful/useful information, and I wonder if the paper would benefit from some kind of interactive website that allows users to explore the results with more granularity.\n* The final results in Figure 5 are quite compelling. However as a reader I am left wondering how much more efficient this result is in practice. Is this method and approach something we should all be using, or just something scientifically interesting?", "was_revised": true, "final_justification": "The reviewers have more than adequately addressed my concerns. I believe these results will be very useful for the ML community.", "rebuttal": "We thank the reviewer for the very thoughtful comments and questions.\n\n**W1/Q2: Some empirical results are hard to interpret (e.g. the matrix plot in the Appendix). Interactive websites to allow users to explore the results with more granularity can be helpful; Explicit examples in the appendix of data pairs/tasks where forgetting is high and where forgetting is low can be included.**\n\nWe thank the reviewer for the suggestions. We have already started building an interactive online explorer of the matrix that also shows contents of the data pairs. We will improve the details and clarity of the figures and captions in the final version of the paper.\n\n**W2: How much more efficient the use of predict-then-replay result is in practice? Is this method and approach something we should all be using, or just something scientifically interesting?**\n \nThank you for the thoughtful question. We consider that the predict-then-replay approach can potentially be extended to practical setups, and that the computational cost benefit can signify. As our cost analysis in Sec. 4.3 and Table 6 shows, the computational cost benefit of predict-then-replay increases when the upstream data is large, and when the same model is repetitively fine-tuned for different tasks. \n\n**Q1: I assume all models except OLMo-7B-Instruct are base models**\n\nYes, we referred to non-instruction-tuned models as base models. We will improve the clarity in the writing.\n\n**Q3: Why is the matrix for OLMo-7B-Instruct much more different than the matrices for the other models in Appendix Figure 6?**\n\nWe thank the reviewer for noticing this pattern. We believe it is mainly attributed to the differences in the data used in OLMo-7B-Instruct experiments.  To elaborate, we evaluated forgetting over the instruction-tuning data, rather than the pretraining corpora in other non-instruction-tuned model experiments. As there are increased chances of task or format similarity between the fine-tuned tasks and the instruction-tuning data (compared to pretraining data), forgetting (log perplexity increase) is sometimes negative, shown in blue pixels. Besides, the set of fine-tuning tasks (T_1..N, y-axis labels of Figure 6) are also different from non-instruction-tuned models, as we make sure the fine-tuning tasks are unseen by the models in the instruction-tuning phase. \n\n**Q4: “We empirically notice that the associations between learned tasks and forgotten upstream examples are more complicated in more recent and capable LLMs” - is this simply because more of the finetuned datasets are more likely to be included in OLMo pretraining?**\n\nWe thank the reviewer for this insightful question. For open-data LLMs like OLMo and OLMo2, the source of the upstream data is released and splitted into dedicated domains. As far as we know, the fine-tuning datasets are not (intentionally) included in the pretraining phase (although a small portion may slip into the commoncrawl partition of the corpora). Therefore, we do not consider that the complexity of the pattern can be solely attributed to the training data inclusion.\n\nBesides, although our Appendix J suggests that inclusion of exact examples of the learned task in the upstream examples can lead to more complicated patterns of forgetting, we further notice that including the data from the same distribution (instead of the exact examples) results in the same patterns of forgetting. Therefore, we hypothesize that complexity depends on the overlap in the “distribution” or “knowledge” level, instead of the exact examples. We consider that understanding what attributes to the complexity of the patterns can require extensive study, which we leave as the future work.\n\n**Q5: Why does OLMo2 have zero forgetting in Figure 3(d)?**\n\nThank you for noticing this pattern. Although forgetting appears on the zero line, the forgetting is actually non zero. We present additional statistics below.\n\n| Model                  | OLMo2-7B | OLMo-7B |\n|------------------------|----------|---------|\n| Avg. Forgetting        | 0.0037   | 0.0810  |\n| Std. Forgetting        | 0.0076   | 0.0646  |\n| Percentage where forgetting > 0 | 81.76%    | 96.23%   |\n\nWe will include these statistics and the visualization of the association matrix of OLMo2-7B in the final version. We further note the low forgetting does not mean the OLMo2 has lost plasticity for updates. This is supported by the drop of perplexity on the fine-tuned tasks in a level similar to OLMo models.\n\nNevertheless, we notice that OLMo2 models are much more resistant to forgetting when learning the new tasks. This can depend on many factors, such as flat loss minima (so that the upstream data perplexity is resistant to any updates in weights, often affected by choice of optimizers, regularization, dropout etc. during pre-training) [1], and whether the new task data is considered significantly out-of-distribution by the model (so that learning new task updates model parameters in a direction that causes forgetting). There are also empirical studies showing how forgetting depends on pre-trained models such as [2,3].\n\n> [1] Mirzadeh et al. Understanding the Role of Training Regimes in Continual Learning, NeurIPS 2021\n> \n> [2] Scialom et al. Fine-tuned Language Models are Continual Learners, EMNLP 2022\n> \n> [3] Luo et al. An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning\n\n**Q6: Why do you use all-distilroberta-v1 as your embedding model? This seems quite outdated**\n\nWe thank the reviewer for the suggestion. We will update Table 2 and Table 3 with the results of OpenAI text embedding models in the final version of the paper.\n\n**Q7: What exactly is the “ground truth forgetting” in Figure 5?**\n\nWe are sorry about the clarity issue. The dash line “ground truth forgetting” here means the replay examples are upweighted according to the actual forgetting, instead of the predicted forgetting. \n\nThis result is included as a reference for predict-then-replaying approaches; computing the actual forgetting is expensive in practice, as it requires running LLM inference over the entire upstream data.\n\n**Q8: What are the pros and cons between online and offline MF?**\n\nThe advantage of online MF is computation efficiency. As summarized in Table 4, online MF does not incur extra fine-tuning. However, the prediction of forgetting is less accurate as it is based on forgetting under a fraction of total fine-tuning steps. This is also evidenced by the subpar forgetting mitigation performance compared to the offline variant in Figure 5 in setups such as OLMo-1B.\n\n**Minor writing and presentation issues**\n\nWe thank the reviewer for spotting the issues and the suggestions to enhance the figure presentation. We will update the paper accordingly.\n\n---\n\nWe thank the reviewer again for recognizing the contribution of our paper and the insightful questions."}, {"review_id": "hDjGZazOQA", "reviewer": "Reviewer_J9oc", "summary": "The dependency between forgotten upstream examples and the newly learned tasks can enable the mitigation of forgetting when fine-tuned. This work shows that the association matrices are often well-approximated by low-rank matrices, and matrix completion allows fast identification of the most forgotten examples.", "questions": "1. The experiments use only decoder-only models. Would this method also work effectively on encoder-only models or mixture-of-experts architectures? Extending the evaluation to diverse model architectures would help assess the generalizability of the approximation method.\n2. Why is forgetting measured with log perplexity (weakness 3)? Why not use task-specific metrics, for example, context-aware accuracy or answer correctness for QA datasets? What justifies log perplexity as the primary metric? A more detailed explanation is needed.\n3. Does the replay approach impact the learning of the new task? Replaying a mini-batch of upstream examples every few training steps could interfere with the training dynamics on the new task. Have you assessed this effect?\n4. If the previous data is already known, why is matrix-based prediction necessary? One could directly measure actual forgetting using the original data, and those results could serve as ground truth for evaluating the proposed mitigation method. This seems circular… A clarification would be helpful.", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- Proposes a simple method that estimates the associations between learned tasks and forgotten examples through low-rank matrix approximation, and predicts forgotten examples by solving a matrix completion problem.\n- Demonstrates a mitigation approach that leverages the predicted forgetting. The practical utility is validated by showing statistically significant reductions in forgetting.\n\n### Weaknesses\n\n- **The experimental setting is overly idealized.** A critical limitation is that the method assumes access to the previously trained data, which is often unavailable in practice due to expired licenses or unreleased datasets. This significantly limits the applicability of the method. In real-world scenarios, such as post-training an LLM on a downstream task, the previously used data is unknown, making it infeasible to predict forgotten samples.\n- **The mitigation method’s performance is not consistently superior,** for example, in Figure 5(c), its performance is comparable to random sampling of upstream examples. This raises concerns about the accuracy of the predicted forgetting and the effectiveness of the proposed sampling strategy.\n- **Log perplexity may not be an appropriate evaluation metric for assessing forgetting mitigation.** A more meaningful metric would capture the extent to which forgetting degrades model capability, such as benchmark-specific accuracy or task-relevant correctness (e.g., accuracy on math problems).", "was_revised": true, "final_justification": "As some of my concerns have been addressed, I have decided to raise my score to a 3.  \n- However, I remain unconvinced by the proposed mitigation strategy, which is presented as a key contribution. It seems not improving much compared to the random method.\n- Additionally, the method relies on an idealized setting that assumes access to the original training data and the ability to replay the fine-tuning process after predicting the forget data. The assumptions may not hold in many practical scenarios.", "rebuttal": "We thank the reviewer for the thoughtful comments. We respond to the weaknesses and the questions suggested by the reviewer below.\n\n**W1: The experimental setting is overly idealized. A critical limitation is that the method assumes access to the previous training data, which is often unavailable in practice due to expired licenses or unreleased datasets.**\n\nThank you for pointing out the limitation. Although we agree with the reviewer about the inapplicability of our approach under data unavailability (inherited from replay-based methods), we decided to study replay-based methods because they are shown to be scalable and effective when applicable, which probably explains why they are still actively studied for LLMs [1,2]. This setup applies to, for example, developers that own their model and the training data. We will better clarify the applicability scope of the proposed method following the reviewer’s comment. Currently, this limitation is discussed in the Limitations section in Appendix A.\n\n\n**W2: The mitigation method’s performance is not consistently superior, for example predict+replay performance is close to random replay in Figure 5(c)**\n\nWe understand the concern from the reviewer regarding small numerical performance gaps like in Figure 5(c). At this stage, our main goal is to analyze patterns in forgetting and showcase the benefit of predicting forgetting. We leave more effective forgetting mitigation algorithms that utilize the predicted forgetting as future works.\n\n**Are the improvements consistent?** Yes - we measured statistical significance of the improvements with paired t-tests in all setups in Figure 5, and achieved p<0.05 or 0.01 in most of the setups. To elaborate, a small p-value in this test suggests over all 20 tasks, our method brings consistent (but probably small) improvements. \n\n**W3 / Q2: Question about using log perplexity over the upstream training data as an appropriate metric of forgetting. Why not use task-specific metrics, for example, context-aware accuracy or answer correctness for QA datasets? What justifies log perplexity as the primary metric?**\n\nWe thank the reviewer for the thoughtful question. The reason why we chose log perplexity over upstream data as the main metrics is many-fold.\n\n**First, log perplexity bridges theoretical approximations and empirical observations of forgetting, allowing proper comparison (in Sec. 3.3).** Log perplexity, equivalent to the training loss of language modeling, is often approximated with first-order techniques (gradient x gradient, gradient x weight differences) in prior works [3,4], assuming small weight updates. These measures can be, in theory, taken as predictors of forgetting; which, however, do not correlate strongly with empirical forgetting under practical (longer) fine-tuning runs in our experiments (Table 2 in Sec 3.3.). To compare with such theoretical approximations, we decided to measure forgetting with log perplexity increase.\n\n**Second, we try to measure forgetting over examples that the model has surely been trained on, which is the upstream pre-training or instruction-tuning data.** We also dissect forgetting from confounders such as generalization effects on unseen tasks.\n\nWe will significantly improve the clarity about the choice of log perplexity metrics in the final version of the paper.\n\n----\n\nIn addition, we understand the limitations of using log-perplexity as the only metrics and that performance drop measured on benchmark dataset is also very interesting. \n\n**We would like to highlight results in the paper that utilize metrics other than perplexity in the current version of the paper**. For instruction-tuned model OLMo-7B-Instruct, we measured binary exact-match drop on the FLAN subset of Tulu V2 as the metrics of forgetting. The dataset involves QA pairs of moderately diverse tasks, and the exact match metrics is close to the accuracy metrics the reviewer asked for. We visualized the association matrix (Figure 8) and analyzed its low-rank patterns (Figure 3(a) - OLMo-7B Inst F1). We measured the performance of predicting binary exact drop and showed statistically significant improvement of targeted replay in Figure 5(b). In summary, the results are very consistent with forgetting measured with log-perplexity.\n\n**Current results & future works about forgetting on benchmark datasets.** Currently, we showed and concluded that targeted replay does not improve LLM leaderboard tasks performance (Table 7, Appendix F). We leave more effective algorithms as future works. We will include example-level forgetting statistics on LLM leaderboard benchmark datasets and analyze their low-rank approximations in the final version of the paper.\n\n**Q1: Extension to MoE and other model architectures**\n\nWe thank the reviewer for the suggestion. We have tried to make our empirical analysis as comprehensive as possible given computational resources available. We understand MoE models are getting increasingly important but at this stage we have to leave it as future works.\n\n**Q3: Does the replay approach impact the learning of the new task? Replaying a mini-batch of upstream examples every few training steps could interfere with the training dynamics on the new task.**\n\nWe thank the reviewer for the insightful question. We have experiments included in Appendix that can address the reviewer’s question. Per the reviewer’s comment, we will move more discussions to the main body of the paper.\n\nIn Appendix F and Table 6, we showed that targeted or random replay can achieve comparable or improved perplexity on the validation sets of the fine-tuned tasks. In some setups, the improvements are statistically significant, which we will annotate in the updated paper.\n\nIn Appendix H and Table 10, we measured the correlations between forgetting under different replay strategies. We see replay can change the patterns of forgetting compared to no replay, but the change is mild.\n\n\n**Q4: If the previous data is already known, why is matrix-based prediction necessary? One could directly measure actual forgetting using the original data.**\n\nWe thank the reviewer for the question. Although we assume access to the upstream data, it is  unknown how much the examples suffer from forgetting after fine-tuning on a new task. This can be expensive to acquire, as measuring forgetting requires running inference over the possibly massive upstream data. Our motivation of predicting forgetting with matrix completion instead of directly measuring them is to avoid this heavy computation, by leveraging the empirical low-rank property of the matrix. We compared computational costs of different replay strategies in Table 4. As the cost analysis suggests, we expect further increased benefit of predicting forgetting in larger-scale setups. \n\nAn analogy we made in the paper is collaborative filtering in recommender systems, which saves cost by inferring the missing user-item interactions by leveraging the low-rank property of the data matrix, instead of directly querying the user.\n\n----\n\nWe thank the reviewer again for the thoughtful comments, and we will update the paper accordingly.\n\n> References\n>\n> [1] Ibrahim et al. Simple and Scalable Strategies to Continually Pre-train Large Language Models, 2024\n>\n> [2] Que et al. D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models, NeurIPS’ 24\n>\n> [3] Lepez-Paz et al. Gradient episodic memory for continual learning, NeurIPS’ 17\n>\n> [4] Doan et al. A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix, AISTATS ‘21"}, {"review_id": "D5WZDCbQS7", "reviewer": "Reviewer_nRcp", "summary": "The paper investigates forgetting in LLMs using a matrix factorization framework that captures interactions between training examples and downstream tasks. The central claim is that forgetting matrices can be approximated via low-rank decompositions, enabling prediction of forgetting effects prior to fine-tuning.", "questions": "1. What inductive biases do you think induce low-rank structure in forgetting matrices?\n2. How stable is the low-rank approximation with respect to noisy or adversarial fine-tuning? Does PEFT (parameter efficient finetuning) such as LoRA change the conclusion?\n3. Could this method be used *during* training to halt updates early when forgetting risk is high?", "limitations": "discussed in the appendix", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- Elegant abstraction of example-task interactions.\n- Matrix completion to predict forgetting is novel, simple and straightforward.\n- Strong empirical support on multiple architectures (MPT, Pythia, OLMo).\n\n**Weaknesses:**\n- Limited insight into *why* low-rank holds. is it architectural, training-dynamic, or data-driven? Do we have any theorectical conclusions about the low-rankness.\n- The definition of forgetting is not formalized. It is unclear that if an example should be considered as forgotten when completely disappearing from any possible prompted generation or with no system prompts. Sometimes it might seem like the example is forgotten but when we twist the prompt slightly, we will see the model actually does not forget the example but just hide it from surface. For example, RLHF is believed to be useful for forgetting toxicity but recent research find that they might just hide the toxicity from the surface (Table 3, Chen et al 2024 Jet expansions of residual computation)\n\n\n**Missing References:**\n\nThe paper can be improved by adding a general discussion of related work on forgetting. The paper focuses on the negative side of forgetting while recently the positive side of forgetting has also been researched. For example, \"Improving Language Plasticity via Pretraining with Active Forgetting\" (Chen et al., NeurIPS 2023), introduces a pretraining-time forgetting mechanism that proactively improves model plasticity. This is highly relevant to the stated goal of understanding and mitigating forgetting in language models but more from a positive angle. Including this work would help position your contribution more clearly within the broader literature that views forgetting not only as a side effect but also as a potentially useful objective. Additionally, the authors should consider citing \"Digital Forgetting in Large Language Models: A Survey of Unlearning Methods\" (Blanco-Justicia et al., 2024), which provides a comprehensive overview of the forgetting landscape, including both proactive and reactive forgetting for removing data points from LLMs.\n\nRegarding the methodology for tracing what is forgotten, the paper currently relies on continuous perplexity shifts and exact match drops. These metrics are informative but may miss deeper semantic shifts. To complement this, the paper may benefit from referencing \"Jet Expansions of Residual Computation\" (Chen et al., 2024), which explores how structured knowledge flows through residual paths in LLMs. While not directly about forgetting, it provides a way to analyze internal changes between models before and after finetuning, effectively offering a model-level “diff” that may surface subtle patterns beyond output metrics. Although not explicitly about forgetting, these tools could enrich your analysis of how knowledge is forgotten during finetuning.", "was_revised": true, "final_justification": "The authors have addressed my concerns regarding the two main weaknesses: (1) the lack of insights into why low-rankness holds and (2) the limited discussion on the definition of forgetting in language models. They have promised to incorporate relevant discussions into the revised version of the paper. Additionally, they plan to conduct an ablation study on the robustness analysis, specifically on noisy outputs or rejected responses in the FLAN subset of the UltraFeedback dataset, which will provide further context for the method and is appropriate for inclusion in the appendix. Accordingly, I have increased my score from 4 to 5.", "rebuttal": "We thank the reviewer for the thoughtful comments and the very helpful suggestions.\n\n**W1. Limited insight into why low-rank holds**\n\nWe thank the reviewer for the thoughtful comment. As we have acknowledged in the Limitations section, we do not have a theoretical explanation of the low-rank structure at this moment.  We tried to connect it to first-order approximations of forgetting and low-rank subspaces where weight updates can reside, but the actual forgetting does not correlate strongly (Sec. 4.3). We leave theoretical and mechanistic interpretations of forgetting as future works. \n\nAt this stage, our paper contributes to behavioral interpretability of the low-rank structure by studying how various factors affect the low-rank approximation (e.g. model size, knowledge coverage of LMs, etc.) with data-driven experiments.\n\n**Q1:  What inductive biases do you think induce low-rank structure in forgetting matrices?**\n\nWe thank the reviewer for this insightful question. We expect that significant study is required to answer the question. We believe mechanistic interpretations on how knowledge is updated and encoded in models (e.g. the suggested reference Chen et al. 2024) is relevant. We will note future lines of work in the updated version of the paper.\n\n**Informal intuition about the low-rank structure**. one intuition of the low-rank structure is an analogy to human’s forgetting process. Compared to acquiring a new skill, forgetting depends less on the material (training data) that humans learn afterwards, resulting in the low-rank structure. We refer to the interference theory of the human forgetting process for this intuition.\n\n**W2: The definition of forgetting**\n\nThank you for the thoughtful comment. We choose to examine forgetting with log perplexity increase or exact match drop because (1) they are connected to theoretical approximations of forgetting with first-order techniques (Sec. 3.3), allowing proper comparison and (2) we hoped to study forgetting on upstream examples where we are sure the models have seen before.\n\nWe agree with the reviewer that forgetting defined in other ways can have different and interesting practical implications. This includes the \"shallow forgetting\" (i.e. forgetting that is recovered when twisting the prompts), and the positive side of the forgetting the reviewer suggested in the comment. We agree that including the discussions of these works can position our work paper and inspire future analysis of forgetting under these definitions. We thank the reviewer for the helpful references and we will discuss them in the related works section.\n\n**Q2-1: How stable is the low-rank approximation with respect to noisy or adversarial fine-tuning?**\n\nWe thank the reviewer for the suggestion. In our new on-going experiments, we perform supervised fine-tuning over noisy outputs or rejected responses in the FLAN subset of the UltraFeedback dataset and will include the results in the final version of the paper.\n\n**Q2-2: Does PEFT (parameter efficient finetuning) such as LoRA change the conclusion?**\n\nOur empirical results presented in Appendix H and Table 9 shows LoRA improves low-rank approximation of the forgetting, indicating simpler associations between learned tasks and forgotten examples. Therefore, we do not expect LoRA to change the conclusion about the low-rank property.\n\n**Q3: Could this method be used during training to halt updates early when forgetting risk is high?**\n\nThank you for bringing up this extension. Yes, we believe that by predicting forgetting during training allows us to efficiently determine early stopping of the training process. This should be computationally more efficient than evaluating actual forgetting over all upstream examples.\n\n---\n\nWe thank the reviewer again for recognizing the contribution of our paper and the helpful comments."}, {"review_id": "BIf8ksEvez", "reviewer": "Reviewer_s6Xq", "summary": "This paper demystifies LLM forgetting by showing that the associations between newly learned tasks and forgotten upstream examples form a low-rank matrix. Based on this finding, the authors introduce a highly efficient method to predict future forgetting by treating it as a matrix completion problem. The practical utility is confirmed by using these predictions to guide a weighted replay strategy, which reduces forgetting during fine-tuning.", "questions": "please see the weakness part.", "limitations": "NO", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strong points：\n1. Models catastrophic forgetting with an innovative and analyzable low-rank matrix framework.\n2. Formulates forgetting prediction as a matrix completion problem, which is more effective and computationally cheaper than prior methods.\n3. Provides extensive empirical evidence across multiple models and scales, with a closed-loop validation showing that predicted forgetting can be used to mitigate actual forgetting.\n \n \nWeak:\n1. In Figure 3, the paper shows that the complexity of forgetting associations differs significantly between OLMo-1B and OLMo-7B, with the latter exhibiting more complex patterns (lower R² at the same rank). Furthermore, Figure 10 highlights \"interesting patterns\" specifically in OLMo-7B. Despite this, the crucial experiment in Table 2, which aims to interpret these associations by checking correlations with similarity measures, was conducted on OLMo-1B. This choice is questionable, as an analysis on the more complex OLMo-7B would have been more compelling and could have yielded different insights into the nature of forgetting in more capable models.\n2. The paper concludes from Table 2 that similarity measures \"correlate poorly\" with forgetting, dismissing them as weak predictors. However, this conclusion may be strong. For instance, the positive correlation for ⟨Gradient, Gradient⟩, while small, suggests that alignment in gradient space does contribute to forgetting. \n3. The primary metrics, log perplexity and Token F1, show statistically significant but numerically small improvements (e.g., in Figure 5). This raises questions about the practical impact of the mitigation. A more direct and functional evaluation of forgetting would be to measure performance degradation on a specific, previously fine-tuned task (Task A) after the model learns subsequent new tasks (B, C, etc.). By constructing an association matrix between Task A's examples and the new downstream tasks, one could directly quantify the retention of a specific capability, which is a more intuitive and practical measure of forgetting than perplexity changes across a vast pre-training corpus.\n4. The entire method depends on access to upstream data for building the association matrix, making it unsuitable for closed-source models.\n5. The best-performing offline mitigation variant doubles the fine-tuning computational cost by requiring a preliminary run, hindering its practical application.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the very thoughtful comments and questions.\n\n**W1: The crucial experiment in Table 2, which aims to interpret these associations by checking correlations with similarity measures, was conducted on OLMo-1B only**\n\nWe thank the reviewer for the question and we agree that the results on OLMo-7B should be added to Table 2. We performed this new set of experiments and we will include the results in the final version of the paper.\n\n| Model                    | OLMo-1B        | OLMo-1B         | OLMo-7B        | OLMo-7B         |\n|-------------------------------|----------------|-----------------|----------------|-----------------|\n|  Metrics                            | Pearson $\\rho$ | Spearman $\\rho$ | Pearson $\\rho$ | Spearman $\\rho$ |\n| Textual (TF-IDF)              | -0.049         | -0.035          | 0.048          | 0.137           |\n| Textual (Representation)      | 0.021          | 0.017           | 0.059          | 0.062           |\n| <Gradient, Weight difference> | -0.003         | -0.009          | -0.004         | -0.004          |\n| <Gradient, Gradient>          | 0.061          | 0.052           | 0.013          | 0.002           |\n| OLMo-1B forgetting            | -              | -               | 0.395          | 0.453           |\n\n\nTo summarize, on OLMo-7B, gradient-based similarity measures show similar or lower correlations with the actual forgetting compared to OLMo-1B. \n\nNevertheless, we also notice that TF-IDF text similarity correlates positively with forgetting OLMo-7B but negatively in OLMo-1B. To better interpret this,  we measure how each component ($\\alpha_k\\beta_k^T$) in the association matrix $Z$ correlates with the TF-IDF text similarity. We highlight components where Spearman's |$\\rho$| > 0.05\n\n\n|         | k=1   | k=2    | k=3   | k=4   | k=5    |\n|---------|-------|--------|-------|-------|--------|\n| OLMo-1B | **0.093** | **-0.075** | 0.010 | 0.001 | -0.003 |\n| OLMo-7B | 0.003 | **0.328**  | 0.030 | **0.080** | -0.020 |\n\nThe results show that some components are significantly more positively or negatively correlated with the text similarity, and how the components (and their correlations to TF-IDF text similarity) differ between OLMo-1B and OLMo-7B models. We will update the paper with the new results and the analysis.\n\n**W2: The paper concludes that similarity measures \"correlate poorly\" with forgetting, dismissing them as weak predictors. However, this conclusion may be too strong.**\n\nWe thank the reviewer for pointing out the issue. Yes, the correlations of similarity measures) to forgetting (especially TF-IDF and gradient inner products), despite small, is statistically significantly non-zero. Therefore, they can still be predictors of forgetting. We will improve the preciseness of the writing in the final version of the paper.\n\n**W3: A more direct and functional evaluation of forgetting would be to measure performance degradation on a specific, previously fine-tuned task (Task A) after the model learns subsequent new tasks (B, C, etc.). By constructing an association matrix between Task A's examples and the new downstream tasks, one could directly quantify the retention of a specific capability, which is a more intuitive and practical measure of forgetting than perplexity changes across a vast pre-training corpus.**\n\nWe thank the reviewer for the thoughtful comment. Regarding evaluation on downstream tasks, the most relevant results presented in the paper should be forgetting of OLMo-7B-Instruct on the FLAN subset of Tulu, where we use the exact-match drop metrics (instead of perplexity). The association matrix is visualized in Figure 8 in Appendix, and we have forgetting prediction results (in prediction F1) and forgetting mitigation results (in Token F1 retention) included in Table 3 and Figure 5. \n\nIn our new set of experiments, we measured forgetting on the LLM leaderboard task examples. We see the similar low-rank structures and will include the visualization and statistics in the final version of the paper. We agree with the reviewer that the part of functional evaluation of forgetting can be further extended. We will expand our discussions about this line of future work.\n\n**W4: The method depends on the access of the upstream data**\n\nWe acknowledge this limitation in our Limitations section. In fact, we consider this limitation inherited from replay-based algorithms for mitigating forgetting. We still decided to study replay-based methods because they are shown to be scalable and effective when applicable in LLMs. \n\n**W5: The best-performing offline mitigation variant doubles the fine-tuning computational cost**\n\nWe acknowledge the extra computational overhead of the offline variant. We advise future practitioners to apply the online variant proposed in the paper in cost sensitive scenarios.\n\n---\n\nWe thank the reviewer again for recognizing the strength of our paper and the thoughtful comments."}]

Author rebuttal:

Meta-review: This paper investigates LLM forgetting by showing that the associations between newly learned tasks and forgotten upstream examples form a low-rank matrix. Building on this observation, the authors propose a method to efficiently predict forgetting of upstream examples, as well as an algorithm to mitigate forgetting during LLM finetuning. The approach is supported by strong empirical results across multiple models (OLMo, MPT, Pythia, OLMo2, OLMo-Instruct) and datasets (FLAN, Tulu, Dolly). Most of the reviewers’ concerns have been adequately addressed. Remaining issues include the marginal improvement of the proposed method and the practicality of assuming availability of the original training data.
Decision: Accept (poster)