Demystifying Language Model Forgetting with Low-Rank Example Associations

## Abstract

Abstract Large Language models (LLMs) suffer from forgetting of upstream data when fine-tuned. Despite efforts on mitigating forgetting, few have investigated whether, and how forgotten upstream examples are dependent on and associated with newly learned tasks. Insights on such associations enable efficient and targeted mitigation of forgetting. In this paper, we empirically analyze forgetting (measured in log-perplexity increase) that occurs in N 𝑁 N italic_N upstream examples of language modeling or instruction-tuning after fine-tuning LLMs on one of M 𝑀 M italic_M new tasks, visualized in M × N 𝑀 𝑁 M\times N italic_M × italic_N matrices. We demonstrate that the matrices display simple low-rank patterns, often well-approximated with multiplicative scalar effects of upstream examples and newly learned tasks. We also examine fine-grained associations with visualization and statistics. Leveraging the low-rank nature of the associations, we predict forgetting of upstream examples when fine-tuning on unseen tasks with matrix completion over the empirical associations. This enables fast identification of most forgotten examples without expensive inference on the entire upstream data. The approach, despite simplicity, outperforms prior approaches that learn semantic relationships of learned tasks and upstream examples with LMs for predicting forgetting. We demonstrate the practical utility of our analysis by showing statistically significantly reduced forgetting as we upweight predicted examples for replay at fine-tuning. Project page: https://inklab.usc.edu/lm-forgetting-prediction/

## 1 Introduction

There has been a growing need for long-term usability of LLMs. Although fine-tuning allows efficient and incremental adaptation of models, it risks catastrophic forgetting (McCloskey & Cohen, ; Goodfellow et al., ) of upstream knowledge learned in the pre-training or instruction-tuning phase, causing unintended prediction changes over known information. This is problematic for the stability of online deployed LLM systems, limiting the feasibility of continual fine-tuning in practice (Raffel, ; Shi et al., ).

While extensive works have developed algorithms to mitigate forgetting (Wu et al., ), the increasing scale of the model and data necessitates more efficient and targeted approaches. A key aspect of the challenge is understanding when and what LLMs forget, prompting analysis into the patterns of frequently forgotten examples (Toneva et al., ; Maini et al., ; Zhang & Wu, ), and the impacts of models and hyperparameters on forgetting (Mirzadeh et al., ; Kalajdzievski, ; Que et al., ; Ibrahim et al., ).
However, how the associations between learned tasks and upstream examples inform forgetting remains under-explored.
Understanding such associations creates potential for targeted mitigation of forgetting over specific upstream examples when a new task is learned.
While theoretical and empirical study indicate associations between learned and forgotten tasks in shallower neural networks (Lee et al., ; Goldfarb et al., ; Ramasesh et al., ), the associations are under-explored for LLMs, or measured regarding upstream data of language modeling or instruction-tuning.

In this paper, we empirically study the associations between learned tasks and forgotten upstream examples of language modeling or instruction-tuning. We experiment with OLMo-1B, OLMo-7B, OLMo-7B-Instruct (Groeneveld et al., ) and MPT-7B (Team, ) models where upstream data is open-source. We perform statistics of forgetting (in log perplexity increase) over $N$ upstream examples, after fine-tuning the model on one of $M$ unseen instruction-tuning tasks, and represent the statistics in a $M\times N$ matrix.
Afterwards, we visualize the matrices and fit the observations with statistical models to analyze the associations. Figure 1 illustrates the analysis setup.

Our findings suggest that the associations between learned tasks and forgotten examples are usually simple and display a low-rank pattern, as simple regression models can decently fit of the associations. We extract more fine-grained associations with visualization and statistics, showing cases where specific upstream examples are forgotten when learning a new task.
We demonstrate that forgetting does not correlate with many predefined similarity metrics of learned tasks and upstream examples, such as token overlap or gradient dot-products, making these metrics poor predictors of forgetting.
Inspired by the findings, we directly utilize statistics of forgetting to predict example forgetting on unseen tasks by solving a matrix completion problem over the association matrices, analogical to collaborative filtering (Sarwar et al., ) in recommender systems, achieving both efficiency and interpretability. Our $k$-nearest neighbor (KNN) model outperforms prior approaches that learn semantic relations of two examples with LMs (Jin & Ren, ) . We verify the benefit of prediction by upweighting examples with higher predicted forgetting during replay as we fine-tune LLMs on new instruction-tuning tasks, achieving statistically significant improvement in alleviating forgetting compared to replaying random examples.

Figure: Figure 1: The problem setup of analyzing the associations between learned tasks and forgotten upstream examples as we fine-tune LLMs on one of unseen new tasks. Over total $N$ upstream examples and $M$ unseen tasks, we measure and record forgetting in a $M\times N$ matrix and analyze the statistical properties of the associations.
Refer to caption: x1.png

To summarize, the contributions of this paper are (1) an empirical analysis on how forgotten examples are associated with learned tasks in representative 1B and 7B language models, and (2) a novel approach of predicting example forgetting by solving a matrix completion problem over the empirical associations, and (3) a practical and efficient algorithm to mitigate forgetting during LLM fine-tuning by upweighting upstream examples for replay according to the predicted forgetting.

## 2 Problem and Analysis Setup

In this section, we start by formally defining the metrics of forgetting and set up the problem formulation of analyzing the associations between learned tasks and forgotten examples. We then introduce models and datasets used for collecting the statistics.

### 2.1 Collecting Statistics of Forgetting

Upstream examples and learned tasks.
Large Language models (LLMs) are commonly pre-trained with language modeling objectives over a massive collection for corpora, with some LLMs further post-trained (instruction-tuned) to better follow human instructions.
We use upstream data to refer to language modeling or instruction tuning training data used at the pre-training or post-training phase of LLMs. For upstream data of language modeling, we define each upstream example $x_{j}\in x_{1..N}$ as a chunk of document (e.g., a Wikipedia article) of a model-specific maximum number of tokens. For instruction tuning, each $x_{j}\in x_{1..N}$ corresponds to a pair of instructions and ground truth responses.

Measuring forgetting.
We fine-tune an LLM (or an instruction-tuned LLM) on one unseen instruction-tuning task $T_{i}$ from a collection of tasks $T_{1..M}$. This results in $M$ separately fine-tuned models $f_{1..M}$. We then evaluate performance degradation on each upstream example $x_{i}\in x_{1..N}$. We measure log perplexity as the performance metric as they are applicable to both language modeling and instruction tuning, and are known to correlate well with other dataset-specific metrics (Hoffmann et al., ). For language modeling, log perplexity of an example $x_{i}$ is averaged over all tokens in the example; for instruction tuning, we average log perplexity over the ground truth response tokens only. We measure forgetting $z_{ij}$ that occurs on an upstream example $x_{j}\in x_{1..N}$ as increase (degradation) in log perplexity after fine-tuning the LM on a new task $T_{i}\in T_{1..M}$. We record forgetting $z_{ij}$ in an association matrix $Z\in\mathbb{R}^{M\times N}$.

### 2.2 Models and Datasets

Our analysis requires access to upstream data of LLMs for measuring their log perplexity changes occurred after fine-tuning. We experiment with OLMo-1B, OLMo-7B, OLMo-7B-Instruct, and MPT-7B where upstream data of language modeling and instruction-tuning is open-source.

OLMo-1B and 7B. OLMo models are pre-trained on Dolma (Soldaini et al., ), a massive collection of text covering diverse domains such as Wikipedia and news articles. We fine-tune LMs over 66 tasks from FLAN-V2 (Longpre et al., ), 11 tasks from Tulu V2 (Ivison et al., ), and 8 tasks from Dolly (Conover et al., ), obtaining 85 fine-tuned models. The collections cover diverse instruction-tuning tasks such as reading comprehension, math reasoning, and safety alignment. We then evaluate log perplexity increase on a 1% subset of Dolma-v1.6-Sample. Each upstream example is a maximum 2,048-token document from Dolma, resulting in 141,816 examples. For 7B models, We perform either full-parameter fine-tuning or LoRA adapter tuning (Hu et al., ) only, noted as OLMo-7B (Full FT) or OLMo-7B (LoRA) respectively.

OLMo-7B-Instruct. OLMo-7B-Instruct models are instruction-tuned on Tulu V2. In our experiments, we few-shot fine-tune OLMo-7B-Instruct over new task data from MMLU (Hendrycks et al., ), BBH (Suzgun et al., ), TruthfulQA (Lin et al., ), and Dolly, and evaluate log perplexity increase over a stratified sample of 10,718 examples from Tulu v2 as upstream examples. The number of training examples in these datasets are smaller (on the scale of tens to hundreds of examples), making the setup closer to LLM error editing (Yao et al., ; Zhang et al., ). We perform either full-parameter fine-tuning or LoRA fine-tuning.

MPT-7B. MPT models are pre-trained on a diverse collection of corpora. We fine-tune the LM over the same 85 instruction-tuning tasks as OLMo models and evaluate forgetting on Redpajama (Computer, ), involved as a part of the pretraining corpora of MPT. We sample 10,000 maximum 2,048 token documents from Redpajama as upstream examples.

We use a learning rate of $2e^{-6}$ for full-parameter fine-tuning and $1e^{-4}$ for LoRA fine-tuning. We include the other training details in Appendix A.

Figure: (a) OLMo-1B; forgetting over Dolma after full-parameter fine-tuning on FLAN and Tulu.
Refer to caption: x2.png

## 3 Associations between Learned Tasks and Forgotten Examples

In this section, we address the research questions regarding the associations between learned tasks and forgotten upstream examples represented in the $M\times N$ association matrices $Z$. We start by examining how complicated are the associations represented by the matrices $Z$ (Sec. 3.1). We then extract and analyze more fine-grained associations in $Z$ (Sec. 3.2), and examine how the associations correlate with similarity measures of learned tasks and upstream examples (Sec.3.3).

### 3.1 How Complicated are the Associations?

In a hypothetical extreme case, the associations between the learned tasks and forgotten examples can be very simple, e.g., where upstream examples are forgotten regardless of learned tasks; or it can be highly complicated, e.g., where different new tasks cause very specific subsets of upstream examples to be forgotten. Figure 1 (right) illustrates how such simple or complicated associations may display in visualized matrices. To understand the actual associations, we start by visualizing the association matrix $Z$ collected in the setups described in Sec. 2.2, and perform quantitative analysis as we examine whether simple regression models can fit $Z$ with low error.

Analysis on the visualized association matrices.
We visualize the association matrices $Z$ in Figure 2 for OLMo-1B, OLMo-7B, OLMo-7B-Instruct as we perform full-parameter fine-tuning. The matrices of MPT and LoRA fine-tuning of OLMo-7B and OLMo-7B-Instruct are presented in Figure 7 in Appendix. We see $Z$ generally displays a neat and simple pattern. We notice that certain upstream are more prone to forgetting, while some are never forgotten (displayed as all-zero columns). Nevertheless, in some cases, upstream examples that are never forgotten elsewhere are forgotten by learning specific new tasks (e.g. wizardlm examples are almost only forgotten after learning tasks from TruthfulQA in Figure 2(c)). It implies the association is a mixture of simple and more complicated patterns.

Quantitative evaluation of simplicity of the associations. We quantitatively measure how well the association matrices $Z$ can be approximated with simple regression models with a small number of learnable parameters.
We consider (1) additive linear models, where $z_{ij}=b+\alpha_{i}+\beta_{j}+\epsilon$, where $\alpha_{i}$ and $\beta_{j}$ are learnable parameters associated with each new task or upstream example. (2) multiplicative models (SVD with rank $r$=1), where $z_{ij}=s\alpha_{i}\beta_{j}+\epsilon$. Both models involve $M+N$ learnable parameters plus a bias term. We then measure $R^{2}$ as the metrics of determining how well the regression models fit the association matrices $Z$. Let $f_{ij}$ be the fitted value, $R^{2}$ is defined as $1-\sum_{i,j}(z_{ij}-f_{ij})^{2}/\sum_{i,j}(z_{ij}-\bar{Z})^{2}$. $R^{2}$ ranges between 0 and 1, indicating the portion of total variance in $Z$ that is explained by the regression models. We report $R^{2}$ of additive and multiplicative models under different setups in Figure 3 (a).

Multiplicative models fit the associations better. In 5 out of 6 setups presented in Figure 3 (a), the multiplicative model achieves a better fit than additive models at the same number of trainable parameters. An interpretation of the multiplicative models is that each upstream example is associated with their tendency of being forgotten $(\beta_{j})$; the learned tasks trivially determine how fast all upstream examples are forgotten (with $\alpha_{i}$). This also creates cases where certain upstream are never forgotten $(\beta_{j}\approx 0)$, or learn certain tasks causing little forgetting on any upstream examples $(\alpha_{i}\approx 0)$.

Figure: (a) Fitting $Z$ with additive or multiplicative models
Refer to caption: x5.png

A large portion of the variance in the association can be explained by the multiplicative model. We notice the multiplicative models achieve $R^{2}$ between 0.43 to 0.72 in different setups despite their simplicity, suggesting generally simple associations between learned tasks and forgotten examples. This finding is emphasized by the diversity of the learned tasks we considered (spanning from reading comprehension, coding, to safety alignment) and the broad coverage of domains in upstream language modeling and instruction-tuning data. Nevertheless, we also notice the $R^{2}$ scores are relatively lower (0.4 to 0.5) on OLMo-7B and OLMo-7B-Instruct, implying more complicated associations than the other setups (OLMo-1B, MPT-7B, and LoRA fine-tuning of OLMo-7B models), which elicits our next research question.

### 3.2 What are More Fine-Grained Associations?

Figure: Figure 4: Patterns in associations captured by $k$-th component in SVD of the assoication matrix $Z$, exemplified with OLMo-7B-Instruct (full-parameter tuning, also in Fig. 2(c)). The highlighted regions display patterns that TruthfulQA tasks cause more forgetting ($k=2$) and MMLU causes more forgetting on certain tasks in FLAN v2 and Open-Orca ($k=3$).
Refer to caption: x7.png

We extract more fine-grained associations where fine-tuning on a task causes a specific set of upstream examples to be forgotten. We perform Singular Value Decomposition (SVD) over the association matrices $Z$ and progressively reconstruct $Z$ with up to $r$-th singular values and vectors as $Z_{r}=\sum_{k=1}^{r}s_{k}\bm{\alpha}_{k}\bm{\beta}_{k}^{T}$. $Z_{r}$ is also the optimal rank-$r$ matrix that minimizes the Frobenius norm $||Z-Z_{r}||_{F}$ and the $R^{2}$ score according to the property of SVD. The first component $Z_{1}$ corresponds to the multiplicative model we examined in Sec. 3.1; the rest of components $Z_{2..M}$ captures fine-grained associations in $Z$.

SVD reveals fine-grained associations. We present $R^{2}$ of fitting the assoication with $Z_{r}$ with progressively larger rank $r$ in Figure 3 (b). We notice that for OLMo-7B and OLMo-7B-Instruct (full-parameter fine-tuning), $R^{2}$ quickly increases to 0.69 and 0.78 with $Z_{3}$ (reconstruction with up to the third component). We visualize the matrices reconstructed from each component $s_{k}\bm{\alpha}_{k}\bm{\beta}_{k}^{T}$ in SVD of $Z$. Figure 4 provides an example of patterns captured by the $k$-th component. We demonstrate how the visualization extracts patterns of forgetting that is conditional on the learned tasks.

### 3.3 How do the Similarity Measures Interpret the Associations?

We examine whether the associations between learned tasks and forgotten examples are interpretable from the similarity between the learned tasks and upstream examples. We consider (1) heuristic similarity measures, such as textual similarity, and (2) theoretically grounded approximations, such as inner products of gradients.

Textual similarity. We measure textual similarity $z^{\textrm{text}}_{ij}$ between learned tasks and forgotten examples with TF-IDF vectorized over each pair of learned tasks $T_{i}$ and upstream examples $x_{j}$.

Inner products between projected gradients and model weight updates. The increase of the log perplexity ${z_{ij}}$ (also known as the cross-entropy loss) can be approximated with inner products $z^{\textrm{g-w}}_{ij}=\langle\nabla_{\theta}f(x_{j}),\theta_{T_{i}}-\theta_{0}\rangle$ under first-order Taylor expansion (Lee et al., ; Doan et al., ), where $\nabla_{\theta}f(x_{j})$ is the gradient of the loss of $x_{j}$ at the initial model before fine-tuning, and $\theta_{T_{i}}-\theta_{0}$ are the updates in the model weights after fine-tuning. Following Park et al. (); Xia et al. (), we use a random projection matrix $\bm{P}\sim\mathcal{N}_{|\theta|\times d}(0,1)$ to reduce the dimension of the gradients or the weight changes to save the cost of storing pre-computed statistics, which preserves the inner products with high probability (Johnson & Lindenstrauss, ).

Inner products between projected gradients. We also measure the negative inner products of the loss gradients between the upstream example $x_{j}$ and a learned task $T_{i}$, $z^{\textrm{g-g}}_{ij}=-\langle\nabla_{\theta}f(x_{j}),\nabla_{\theta}f(T_{i})\rangle$, as an approximation of forgetting (Lopez-Paz & Ranzato, ; Chaudhry et al., ).

**Table 1: Spearman’s correlation coefficients between various measures of similarity and the actual forgetting on upstream examples after fine-tuning on a task.**
| Similarity Metrics | Spearman’s $\rho$ |
| --- | --- |
| Textual $z^{\textrm{text}}$ | 0.004 |
| Gradient-Weight Difference $z^{\textrm{g-w}}$ | 0.005 |
| Gradient-Gradient $z^{\textrm{g-g}}$ | -0.007 |

Forgetting correlates poorly with similarity measures of learned tasks and upstream examples. We evaluate Spearman’s correlation between the actual forgetting $z_{ij}$ and the various similarity measures $\{z^{\textrm{text}}_{ij},z^{\textrm{g-w}}_{ij},z^{\textrm{g-g}}_{ij}\}$ on OLMo-1B and summarize the results in Table 1. We notice that none of the similarity measures correlates with the actual forgetting, with a correlation $\rho<0.01$. An interpretation of the low correlation is that the model weights deviate from the initial weights after fine-tuning to a region where first-order approximation of forgetting does not hold. We further visualize the matrices of $z^{\textrm{g-w}}_{ij}$ and $z^{\textrm{g-g}}_{ij}$ in Fig. 8 in Appendix and provide a side-by-side comparison with the matrices of forgetting $Z$. The visualization displays distinct patterns among the three matrices. These results imply that although the association matrices $Z$ display a simple pattern, they are not well-interpreted with common similarity measures of learned tasks and forgotten examples.

## 4 Predicting Example Forgetting with Association Matrix Completion

We utilize our findings in Sec. 3 to predict example forgetting as the model learns a new task, a problem also studied in prior works (Jin & Ren, ).
Finding out the most forgotten examples allows better spot of the behavior changes of the models, enabling efficient and targeted approaches to mitigate forgetting, e.g., by replaying these examples (Aljundi et al., ; Wang et al., ). Although the ground truth forgetting can be directly obtained by running inference with the fine-tuned model over the upstream data, this requires extensive computational resources. We restrict the prediction methods to be computationally efficient.

Our analysis in Sec. 3 suggests that (1) the associations between learned tasks and forgotten examples display simple statistical patterns, while (2) the associations correlate poorly with many similarity metrics. Therefore, we hypothesize that leveraging the statistics of forgetting allows more effective prediction of forgetting than leveraging the contents of the tasks and examples. Following this intuition, we formulate prediction of example forgetting as a matrix completion problem over the empirical associations $Z$, analogical to collaborative filtering in recommender systems (Sarwar et al., ), where scalable approaches are studied extensively. We start by setting up the problem formulation of predicting example forgetting, and evaluate the performance of prediction of different approaches. We present reduced forgetting by utilizing the prediction outcomes during fine-tuning.

### 4.1 Training and Evaluation of Forgetting Prediction

Our goal is to accurately predict forgetting $z_{ij}$ over upstream examples $x_{1..M}$ when the model is fine-tuned on an unseen task $T_{j}$ with a prediction model $g$, without running expensive LLM inference on all $x_{1..M}$. To evaluate this, we create training and test splits by partitioning the set of fine-tuning tasks (noted as $\mathcal{T}_{\textrm{train}}$ and $\mathcal{T}_{\textrm{test}}$) and the rows of the association matrices $Z$. We further control whether the $\mathcal{T}_{\textrm{train}}$ and $\mathcal{T}_{\textrm{test}}$ are from the same category of the tasks to test both in-domain and out-of-domain generalization ability of the prediction models. For OLMo-1B and 7B experiments, we use FLAN and the in-domain tasks and Tulu and Dolly as out-of-domain testing tasks. For OLMo-7B-Instruct experiments, we use MMLU and BBH as in-domain tasks and use TruthfulQA and Dolly as out-of-domain testing tasks. Details about the tasks included in the training, in-domain testing, and out-of-domain testing sets are discussed in Appendix B.

Figure: Figure 5: The training and testing setup of predicting example forgetting with association matrix completion.
Refer to caption: x8.png

To apply matrix completion for predicting forgetting, a few entries $z_{ij}$ should be known when a new fine-tuning task $T_{i}\in\mathcal{T}_{\textrm{test}}$ (row $i$) is introduced. We therefore assume access to the ground truth forgetting $z_{ij}$ of a tiny random set $\mathcal{S}$ ($|\mathcal{S}|=30$) of upstream examples for $T_{i}\in\mathcal{T}_{\textrm{test}}$, noted as seed forgetting $z_{i}^{\mathcal{S}}=\{z_{ij}|x_{j}\in\mathcal{S}\}$. Obtaining seed forgetting typically takes only a few seconds by running inference with the model fine-tuned on $T_{i}$ over $\mathcal{S}$; we then predict forgetting of the rest $10k-100k$ upstream examples. Figure 5 illustrates an example of the train-test partition, seed forgetting, and the forgetting to be predicted. We use Root Mean Squared Error (RMSE) over the $\mathcal{T}_{\textrm{test}}$ as the metrics of predicting example forgetting.

Matrix completion approaches. We run matrix completion algorithms including additive linear, SVD, and k-nearest neighbors (KNN) models. The additive linear and the SVD models are introduced earlier in Sec. 3. Given the seed forgetting $z_{i}^{\mathcal{S}}$ of a task $T_{i}\in\mathcal{T}_{\textrm{test}}$, KNN finds tasks from $\mathcal{T}_{\textrm{train}}$ that have similar patterns of forgetting over the seed upstream examples $\mathcal{S}$. KNN computes an average of forgetting of top-k similar tasks from $\mathcal{T}_{\textrm{train}}$ weighted by their similarity as the prediction of forgetting caused by $T_{i}\in\mathcal{T}_{\textrm{test}}$ on the upstream examples $x_{1..M}$.

Comparators of predicting forgetting. We compare with a prior approach by Jin & Ren () that leverages learned similarity between learned tasks and upstream examples by a trainable LM to predict forgetting. This prior work, however, focuses on predicting forgetting while fixing one single error in LM predictions; we extend the approach to predicting forgetting after fine-tuning models over a task (i.e., a set of examples). The extended approach encodes upstream examples $x_{j}$ and the training examples of the learned task $x_{i}^{1..N_{i}}\in T_{i}$ with a trainable LM encoder $h(\cdot)$ to obtain their representations. The final prediction is made with a regression head over the inner products of two representations $\langle h(x_{j}),\frac{1}{N_{i}}\sum_{N_{i}}h(x_{i})\rangle$.

We leave the implementation details of matrix completion approaches and the learned similarity approach in Appendix B.

**Table 2: RMSE of predicting example forgetting over a held-out set of upstream examples after fine-tuning LMs on unseen new tasks. We report averaged performance over different seed sets ($\mathcal{S}$) of upstream examples with known ground truth forgetting beforehand.**
|  | In-Domain | Out-of-Domain |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | OLMo-1B | OLMo-7B | MPT-7B | OLMo-7B-Inst. | OLMo-1B | OLMo-7B | MPT | OLMo-7B-Inst. |  |  |  |  |
|  | Full FT | Full FT | LoRA | Full FT | Full FT | LoRA | Full FT | Full FT | LoRA | Full FT | Full FT | LoRA |
| Additive | 2.81 | 7.40 | 3.50 | 13.33 | 15.57 | 6.12 | 2.81 | 5.83 | 7.01 | 10.02 | 38.90 | 21.22 |
| SVD | 2.88 | 7.53 | 3.49 | 12.41 | 14.98 | 6.24 | 2.86 | 5.84 | 7.16 | 8.27 | 39.03 | 21.26 |
| KNN | 2.79 | 7.33 | 3.45 | 12.80 | 14.30 | 5.54 | 2.84 | 5.83 | 6.83 | 7.71 | 38.77 | 20.82 |
| Similarity | 3.84 | 9.29 | 5.45 | 14.00 | 16.23 | 6.19 | 3.93 | 7.64 | 8.52 | 10.97 | 42.38 | 23.47 |

### 4.2 Mitigating Forgetting with Predicted Forgetting

Leveraging predicted forgetting for mitigating forgetting. We examine the practical utility of predicting forgetting as we sparsely replay upstream examples during forgetting following Jin & Ren (). Sparse replay of upstream examples is known as an effective and model-agnostic way to mitigate forgetting (de Masson D’Autume et al., ; Ibrahim et al., ). We replay one mini-batch of upstream examples every 32 training steps while fine-tuning on a new task. We perform targeted mitigation of forgetting by prioritizing examples that are predicted to suffer more from forgetting. This is achieved with weighted sampling of upstream examples $x_{j}$ proportional to $\exp{(\hat{z}_{ij}/\tau)}$, where $\hat{z}_{ij}$ are the predicted forgetting and $\tau$ is a temperature hyperparameter set as 0.1.

Baselines of mitigating forgetting. We compare with diverse strategies of selecting upstream examples for sparse replay. As a main comparator, we examine whether weighted sampling with the predicted forgetting improves over random sampling of upstream examples (Random). We compare with Maximally Interfered Retrieval (MIR) (Aljundi et al., ), a selection strategy sharing the similar notion of importance that forgotten examples should be selected for replay. The approach performs bi-level sampling by selecting the most forgotten examples from a small random subset of upstream data (set as $|\mathcal{S}|=30$). In addition, we apply strategies that consider different definitions of upstream example importance. We examine an approach based on perplexity thresholds (PPL) (Marion et al., ), which samples upstream data of which the perplexity is around the median of the distribution. For OLMo-1B, we also sample replayed examples proportional to the gradient inner products (Grad-Prod) we evaluated in Sec. 3.3, in a similar vein to coreset selection approaches that utilize gradient information (Park et al., ; Xia et al., ).

Metrics. We measure log-perplexity increase over a held-out subset of 10,000 examples from the upstream data. This ensures none of the test examples are selected for replay by any of the example selection strategies.

### 4.3 Results of Predicting and Mitigating Forgetting

Results of predicting example forgetting. Table 2 summarizes the error of predicting example forgetting over tasks from the in-domain and out-of-domain test splits. We see matrix completion approaches consistently outperform the learned similarity model in the prior work. Among the three matrix completion approaches, we notice that KNN models in general achieve the lowest prediction error. Given this finding, we primarily leverage predictions given by the KNN model to reduce forgetting during fine-tuning.

Figure: (a) 1B, FLAN
Refer to caption: x9.png

Mitigating forgetting with the predicted forgetting.
We leverage the forgetting predicted by the KNN model and upweight examples during replay following our procedure in Sec. 4.2.
Figure 6 summarizes log perplexity after fine-tuning over the held-out (never replayed) upstream data as we apply different upstream example selection approaches. We visualize the relative comparisons of these approaches to no forgetting of upstream examples (log perplexity before fine-tuning). We notice that example selection based on gradient inner products (Grad) or perplexity threshold (PPL), mainly applied for identifying important training data for a task in prior works, does not show improvement in mitigating forgetting compared to replaying random examples. This implies that the notions of example importance in these works are different from how easily the examples are forgotten. We also notice that MIR does not improve over random sampling in our setup, likely because of the small size of the retrieval candidates relative to the upstream examples. We see upweighting examples with ground truth forgetting (GT) consistently reduces forgetting compared to random examples. Replaying with KNN-predicted forgetting also consistently reduces forgetting, achieving statistical significance in 4 out of 5 tested setups.

Limitations of replaying with forgetting prediction. We note that our current practice of replaying with predicted forgetting requires two runs of fine-tuning: a replay-free run of fine-tuning after which the seed forgetting will be evaluated, and another run while replaying examples with the predicted forgetting. This creates computational overhead equivalent to one extra run of fine-tuning, which is identical to the MIR baseline. Nevertheless, the practice is still efficient when the training set of fine-tuning is much smaller than the upstream data. Future works can predict forgetting on-the-fly during fine-tuning, mitigating the overhead of fine-tuning the model twice.

## 5 Related Works

Factors that affect forgetting. In this paper, we primarily studied how the associations between learned and forgotten examples inform forgetting. Prior works have studied various factors that affect forgetting of the models, such as (1) type and size of the LM (Mehta et al., ; Scialom et al., ; Kalajdzievski, ; Mirzadeh et al., ) (2) trainable parts of the model (e.g., LoRA, soft prompts, or full-model tuning) (Biderman et al., ; Razdaibiedina et al., ) (3) hyperparameters such as learning rate (Ibrahim et al., ; Winata et al., ), dropout (Goodfellow et al., ), number of training steps (Biderman et al., ; Kleiman et al., ) (4) optimizer (Lesort et al., ) and training algorithms (e.g., various continual learning algorithms) (Shi et al., ; Wu et al., ), (5) the upstream examples or the knowledge themselves (Toneva et al., ; Zhang & Wu, ). Future works can study how various factors affect the associations between learned tasks and forgotten examples. We consider empirical and theoretical study on the effect of task similarity on forgetting to be most relevant to ours.
Ostapenko et al. () empirically study relationships between task similarity and forgetting in foundation models over a sequence of newly learned tasks; our work instead focuses on forgetting of upstream data of LLMs. Theoretical study by Doan et al. (); Ding et al. (); Evron et al. () dissects effects of the learned tasks on forgetting in linear models or around model initialization. We believe research on interpretations of forgetting (Tao et al., ; Zhao et al., ; Kotha et al., ) is complementary to ours and can potentially explain in the future why the associations in $Z$ are often simple, and in which circumstances the associations become more complicated.

Data selection and data attribution.
Related to our work, data attribution studies faithful algorithms to find training examples that account for a prediction (Koh & Liang, ; Ilyas et al., ) from a pool of training examples. Park et al. (); Xia et al. (); Li et al. (); Liu et al. () study the problem of selecting a subset of training data that maximizes performance on a given domain or task at a fixed budget for LLMs.
Feldman & Zhang (); Tirumala et al. (); Biderman et al. (); Swayamdipta et al. () identify memorized, important, or forgetful training data.
However, the notion of data importance in these works is different from how likely the upstream examples will be forgotten during fine-tuning.
Furthermore, a systematical study on how such importance is dependent on newly learned tasks is still absent. Prior works represented by Aljundi et al. (); Wang et al. (); Aljundi et al. () study selection strategies of examples for replay-based continual learning algorithms.

Predicting model behaviors. A number of works show LLMs can display a hybrid pattern of unpredictable to highly predictable behaviors (Ganguli et al., ; Wei et al., ). Ye et al. (); Xia et al. (); Schram et al. () study prediction of task performance across datasets and training setups. We perform prediction at the example level which is more fine-grained and under-explored.

## 6 Conclusions

In this paper, we empirically analyzed the associations between learned and forgotten examples in LM fine-tuning. We showed the association displays a low rank pattern across different setups. We showed the example associations alone offer useful information to predict example forgetting when fine-tuning LMs on new tasks. We demonstrated the practical utility of our analysis by showing reduced forgetting as we reweight examples for replay with predicted forgetting. Future works can extend the study to a continual learning setup where new domains or tasks are sequentially learned while predicting forgetting of upstream examples online.

## Appendix A Dataset, Model, and LM Training Details

Figure: (a) OLMo-7B (LoRA); forgetting over Dolma after full-parameter fine-tuning on FLAN and Tulu.
Refer to caption: x14.png

Models. We use OLMo-7B(^1^11[https://huggingface.co/allenai/OLMo-7B](https://huggingface.co/allenai/OLMo-7B)) of the version pretrained on Dolma v1.6; and OLMo-7B-Instruct(^2^22[https://huggingface.co/allenai/OLMo-7B-Instruct](https://huggingface.co/allenai/OLMo-7B-Instruct)), which is tuned on Tulu v2 and other human feedback datasets.

Figure: (a) Log perplexity increase (forgetting)
Refer to caption: x2.png

Learned new tasks and their categorization. We summarize the list and the categorization of newly learned tasks in Tables 3 and 4 in our experiments. We also include the number of training examples and forgetting caused by each task averaged over all upstream examples.

Training and evaluation details. For full-parameter fine-tuning of OLMo-1B and 7B, we train the model for 1,000 steps with an effective batch size of 8 and a linearly decaying learning rate of $2e^{-6}$.
For LoRA fine-tuning, we set the rank of adapters as 64 in all our experiments and use a rate of $10^{-4}$. We train the models for 625 steps with an effective batch size of 8. For OLMo-7B-Instruct and MMLU, BBH, TruthfulQA, considering the small size of the training sets, we train the models only for 37 steps with an effective batch size of 8. We use HuggingFace Transformers library for training and VLLM library for efficient inference. The statistics of forgetting are obtained in a single run.

Dataset licenses. MMLU and BBH are released under MIT license. Truthful QA, Dolma, Redpajama, OLMo models, and MPT models are released under Apache 2.0 license. Tulu V2 is released under ODC-By license. Dolly is released under CC BY-SA 3.0 license.

## Appendix B Details of Forgetting Prediction and Replay

Data Splits for Predicting Example Forgetting. We mark the tasks used as in-domain test splits for predicting example forgetting (Sec. 4) in Tables 3 and 4. The train-test split for the in-domain tasks is randomly generated.

Training and evaluation details. We use Surprise Library 1.1.3(^3^33[https://github.com/NicolasHug/Surprise/tree/v1.1.3](https://github.com/NicolasHug/Surprise/tree/v1.1.3)) for additive linear, SVD, and KNN prediction models. For SVD, we set the dimension of the learnable features as 5. We train the regression models for 100 epochs over the association matrices. The other hyperparameters are left as default.

For in-domain test splits, we randomly sample 30 upstream examples and assume the ground truth forgetting is known for these examples. This is required for predicting forgetting on the rest of upstream examples by additive linear, SVD, and KNN methods. We repeat the experiment 10 times and report the mean and standard deviation in Table 2.

We used OLMo-1B models as the trainable example encoders in the implementation of the prediction method by Jin & Ren () that relies on inner products of trained example representations. At inference, given an upstream example, we compute the averaged dot-product with all examples in the learned task. We note that at inference time the approach does not require ground truth forgetting of a small number of examples. For a fair comparison with other matrix completion methods, we replace the prediction of the approach with ground truth forgetting on these examples.

Figure: (a) OLMo-7B (Full)
Refer to caption: x19.png

Replaying upstream examples in fine-tuning.
We sparsely replay 1 mini-batch of 8 upstream examples every 32 steps of model update while fine-tuning on new tasks. Given predicted or ground truth forgetting $z_{i,1..J}$ on upstream examples $x_{1..J}$ when learning a new task $T_{i}$, we sample upstream examples to replay from a categorical distribution where $p(x_{j})\propto\exp(z_{i,j}/\tau)$, where $\tau$ is a temperature hyperparameter set as 0.1. The hyperparameter $\tau$ is tuned on a single validation task by using ground truth forgetting $Z$.

## Appendix C Additional Results about Example Associations

We visualize progressive reconstruction with $k$-th singular value and singular vectors for OLMo experiments in Figure 9. We see when $k=2$, there is a single row and column with significantly larger forgetting than the others. This pattern exemplifies a complicated association that is not captured by the simple multiplicative model ($k=1$).

**Table 3: The list of learned tasks in our experiments on OLMo-1B, OLMo-7B and MPT-7B. * notes for tasks used as the in-domain test split in forgetting prediction experiments in Sec. 4.**
| Task Category | Task | Task Category | Task |
| --- | --- | --- | --- |
| FLAN/Classification | aeslc | FLAN/QA | arc_challenge* |
|  | ag_news_subset |  | arc_easy* |
|  | imdb_reviews |  | bool_q |
|  | sentiment140 |  | coqa* |
|  | sst2 |  | cosmos_qa |
|  | trec* |  | math_dataset* |
|  | yelp_polarity_reviews* |  | natural_questions* |
| FLAN/Linguistic | cola |  | openbookqa* |
|  | definite_pronoun_resolution* |  | piqa |
|  | fix_punct* |  | trivia_qa* |
|  | true_case | FLAN/Summarization | cnn_dailymail |
|  | word_segment |  | gigaword |
|  | wsc* |  | multi_news |
| FLAN/Generation | common_gen |  | samsum |
|  | copa |  | wiki_lingua_english_en |
|  | dart | FLAN/Translation | para_crawl_enes |
|  | e2e_nlg* |  | wmt14_enfr |
|  | hellaswag |  | wmt16_translate_csen |
|  | opinion_abstracts_idebate* |  | wmt16_translate_deen |
|  | opinion_abstracts_rotten_tomatoes |  | wmt16_translate_fien |
|  | story_cloze |  | wmt16_translate_roen |
|  | web_nlg_en |  | wmt16_translate_ruen* |
| FLAN/MRC | drop |  | wmt16_translate_tren* |
|  | multirc | Tulu | open_orca |
|  | quac |  | oasst1 |
|  | record |  | lima |
|  | squad_v1 |  | code_alpaca |
|  | squad_v2 |  | gpt4_alpaca |
| FLAN/NLI | anli_r1 |  | cot |
|  | anli_r2 |  | science |
|  | anli_r3 |  | flan_v2 |
|  | cb* |  | sharegpt |
|  | mnli_matched |  | hard_coded |
|  | mnli_mismatched |  | wizardlm |
|  | qnli* | Dolly | brainstorming |
|  | rte |  | closed_qa |
|  | snli |  | information_extraction |
|  | wnli |  | classification |
| FLAN/Paraphrase | glue_mrpc |  | open_qa |
|  | glue_qqp* |  | general_qa |
|  | paws_wiki |  | creative_writing |
|  | stsb |  | summarization |
|  | wic* |  |  |

**Table 4: The list of learned tasks in our experiments on OLMo-7B-Instruct. * notes for tasks used as the in-domain test split in forgetting prediction experiments in Sec. 4.**
| Task Category | Task | Task Category | Task |
| --- | --- | --- | --- |
| MMLU | abstract_algebra | BBH | boolean_expressions* |
|  | anatomy |  | causal_judgement |
|  | astronomy |  | date_understanding |
|  | business_ethics |  | disambiguation_qa |
|  | clinical_knowledge |  | dyck_languages* |
|  | college_biology* |  | formal_fallacies* |
|  | college_chemistry |  | geometric_shapes |
|  | college_computer_science |  | hyperbaton* |
|  | college_mathematics |  | logical_deduction_five_objects* |
|  | college_medicine* |  | logical_deduction_seven_objects |
|  | college_physics |  | logical_deduction_three_objects |
|  | computer_security |  | movie_recommendation* |
|  | conceptual_physics* |  | multistep_arithmetic_two |
|  | econometrics |  | navigate |
|  | electrical_engineering |  | object_counting* |
|  | elementary_mathematics |  | penguins_in_a_table |
|  | formal_logic |  | reasoning_about_colored_objects |
|  | global_facts* |  | ruin_names |
|  | high_school_biology* |  | salient_translation_error_detection |
|  | high_school_chemistry |  | snarks |
|  | high_school_computer_science |  | sports_understanding |
|  | high_school_european_history* |  | temporal_sequences |
|  | high_school_geography |  | tracking_shuffled_objects_five_objects |
|  | high_school_government_and_politics |  | tracking_shuffled_objects_seven_objects |
|  | high_school_macroeconomics |  | tracking_shuffled_objects_three_objects |
|  | high_school_mathematics |  | web_of_lies |
|  | high_school_microeconomics |  | word_sorting |
|  | high_school_physics* | TruthfulQA | Nutrition |
|  | high_school_psychology |  | Stereotypes |
|  | high_school_statistics |  | Confusion |
|  | high_school_us_history* |  | Psychology |
|  | high_school_world_history |  | Language |
|  | human_aging* |  | Sociology |
|  | human_sexuality* |  | Finance |
|  | international_law |  | Indexical Error |
|  | jurisprudence |  | Science |
|  | logical_fallacies* |  | Misconceptions |
|  | machine_learning |  | Economics |
|  | management* |  | Education |
|  | marketing* |  | Proverbs |
|  | medical_genetics |  | Conspiracies |
|  | miscellaneous |  | Religion |
|  | moral_disputes |  | Statistics |
|  | moral_scenarios* |  | Misquotations |
|  | nutrition |  | Subjective |
|  | philosophy* |  | Law |
|  | prehistory |  | History |
|  | professional_accounting |  | Fiction |
|  | professional_law |  | Mandela Effect |
|  | professional_medicine* |  | Politics |
|  | professional_psychology |  | Misinformation |
|  | public_relations* |  | Logical Falsehood |
|  | security_studies |  | Distraction |
|  | sociology* |  | Weather |
|  | us_foreign_policy* |  | Myths and Fairytales |
|  | virology |  | Superstitions |
|  | world_religions |  | Advertising |
|  |  |  | Paranormal |
|  |  |  | Health |

## Peer Reviews

confidence: 3
contribution: 1
initial_rating: 3
presentation: 3
questions: Please refer to concerns in the weaknesses block above.
soundness: 2
strengths: - [S1] **Interesting problem and motivation.** Considering wide use of LLM adaptation, understanding fine-grained associations between downstream tasks vs. catastrophic forgetting has great potential as claimed by the authors, as it can inspire techniques for better knowledge retention.
- [S2] **Great presentation.** The paper presents interesting analyses and comparisons with previous similarity-based measures, making it easy to understand comprehensively understand the motivation and inspirations of this work.
summary: The paper investigates fine-grained associations between new tasks learned by LLMs and catastrophic forgetting of upstream data previously used for pretraining, under the lens of matrix completion. Modeling LLM forgetting as an $M\times N$ matrix with $M$ new tasks and $N$ upstream datapoints, the authors discover that the associations exhibit a low-rank pattern as a simple multiplicative regression model such as low-rank SVD provide a fairly accurate estimation of the matrix. Inspired by this finding, the paper proposes a novel example replay framework that replays upstream examples predicted to be forgotten under matrix completion during finetuning, thereby mitigating catastrophic forgetting.
weaknesses: Despite its strengths, the paper is weak with respect to practicality and experimental results, details of which are as follows.

- [W1] **Weak practicality under large number of downstream tasks.** The proposed framework requires finetuning the base LLM on individual tasks to gather train data for matrix completion, which incurs a high computational cost.  Considering unprecedented model sizes as well as range of downstream tasks for LLMs, we can expect the cost to quickly become intractable in many scenarios. This could be alleviated if we can instead sample a small number of finetuning tasks, and show that the proposed method is robust under small choices of such tasks, but this is somewhat unclear in the current draft.

- [W2] **Weak practicality due to requiring full access to upstream data.** The proposed example replay framework requires full access to upstream data during finetuning, yet we may no longer have access to exact upstream data points due to updates on the data or data sharing policies [A]. Ideally, it would be best to mitigate forgetting on upstream examples without explicitly using them during fine-tuning, yet the matrix completion formulation used in this work seems unsuitable for that direction.

- [W3] **Unconvincing experimental setup.** The proposed framework makes two runs of finetuning, one without replays to gather forgetting information $z_{ij}$ on the train set and perform matrix completion to pick upstream examples to replay, then a second finetuning of the model with intermediate replays on examples chosen. Given this process, it is unclear whether the statistics gathered *post-finetuning without any replay* will hold at every step of *finetuning with replay*, and additional analyses on the predicted forgetting without replay vs. with replay or a similar experiment seems necessary.

- [W4] **Insufficient performance gains.** While the main results in Figure 6 show that the proposed KNN-based approach is statistically better than the random baseline, all methods are significantly far from pre-finetuning performances, leading to concerns on the effectiveness of replay-based methods in mitigating LLM forgetting overall, which weakens the proposed method's practical utility claimed by the authors.

[A] Gao et al., The Pile: An 800GB Dataset of Diverse Text for Language Modeling

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Could you provide more justifications on why SVD reveals fine-grained associations? Since increasing the rank $r$ of SVD provides a better fitting of the forgetting (perplexity) matrix, different SVD components ($r$-th component) may have opposite patterns since only the summation of all components matters. 
2. Are there particular reasons that the Spearman correlation is chosen over the Pearson correlation in Section 3.3? In addition, have you experimented with the correlation between forgetting and the cosine similarity of the textual representations (e.g., last token representation), which is an alternative to TF-IDF for textual similarity? 
3. In Table 2, it seems that forgetting is more difficult to predict for instruction-tuned OLMo-7B-Instruct. Why is that so? Is it correlated with the absolute perplexity of the upstream data?
4. As also raised in weakness 1, how does the proposed forgetting-mitigation approach affect the performance on the downstream tasks (or even the upstream tasks) that are typically not evaluated by perplexity?
soundness: 2
strengths: 1. The paper is well written with notable clarity. 
2. The paper provides extensive and well-designed quantitative and qualitative studies to understand the correlation between forgetting the upstream knowledge and tuning on the downstream tasks. 
3. The paper highlights that the importance/similarity of the examples to the target downstream tasks is a different notion compared to their sensitivity to be forgotten. 
4. The proposed predict-then-upweight replay strategy effectively mitigates the sample forgetting for upstream pretraining/instruction-tuning data in terms of perplexity.
summary: The paper investigates the effect of instruction-tuning on language model forgetting and the strategies to mitigate them. In particular, the paper discovers that the associations between learned tasks and forgotten examples can be reasonably described by simple linear/multiplicative models, exhibiting low-rank patterns. This association can be used to select replay upstream samples when training the model on downstream tasks to mitigate the forgetting of prior knowledge.
weaknesses: 1. Reducing LLM forgetting of the upstream knowledge during instruction-tuning and alignment stages is an important aspect, but typical instruction-tuning and alignment datasets have other metrics for evaluation. It is unclear whether reducing forgetting has practical benefits or trade-offs to these downstream tasks. 
2. There are extensive discussions about using SVD to approximate the matrix, which certainly proves the hypothesis that leveraging a simple forgetting predictor can be more helpful than task-related features. However, the final forgetting predictor using KNN over SVD reduces the importance of those discussions.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: -Figure 6: Random seems to outperform more complex baselines, can you explain why that might be?
soundness: 4
strengths: -I find it very interesting to describe forgetting using a low-rank representation. This shows that fine-tuning on tasks can be described using predictable forgetting patterns. 

-It was very thought provoking to see how correlated the similarity of various tasks was to the likelihood of those tasks being forgotten.
 
-The paper provides a novel method to perform weighted replay for mitigating forgetting.
summary: The paper draws connections between upstream data that is forgotten during fine-tuning, and the new task.  It analyzes forgetting of past data after a model fine-tunes on new data using log perplexity, and shows how a matrix describing upstream loss given a new task can be described with low rank form. This matrix can then be used to predict forgetting of upstream tasks after fine-tuning. Finally, the paper describes a method to mitigate forgetting in fine-tuning by using weighted replay on upstream tasks that are more likely to be forgotten.
weaknesses: -Figure 2: it would be really helpful to add a description and possible analysis featuring what specific knowledge the examples cover, and thus the model forgets. Particularly if there is a correlation between forgetting and specific topics.

-The motivation for predicting forgetting rather than running inference could be more obvious: it is assumed that running inference on upstream examples is expensive, and thus matrix completion offers an alternative for predicting forgetting to mitigate it. However, given that there are enough resources available for a model to perform even costlier fine-tuning, it is not obvious why predicting forgetting rather than running inference is much more beneficial. Perhaps, a further explanation and a direct comparison of the approximate costs associated with performing matrix completion compared to running inference could be included. 

-Figure 6: It would be beneficial to add another baseline which considers weighted sampling for replay based on the actual forgetting values for all data.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. Could the authors elaborate on why OLMo-7B and OLMo-7B-Instruct has "more complicated associations" than the similarly-sized MPT-7B (see weaknesses)? What is the intuition here?
2. The idea that the interaction between pretraining data and finetuning task determines forgetting would be supported by an additional baseline in the matrix completion experiments. Specifically, did the authors consider attempting matrix completion using just an average over finetuning tasks, rather than fitting to all tasks separately? Demonstrating that the average-over-tasks vector is insufficient would strengthen the results in table 2.
3. Could the authors provide more detail on how they determined their significance testing results in fig. 6? How did the authors correct for multiple comparisons?
4. For the replay mitigations in fig. 6, how was the kNN chosen rather than the additive linear or SVD methods (as mentioned on lines 347--353)? If these approaches were also tested, they should be reported, and need to be included in the multiple comparisons correction.
5. What are the fine-tuning tasks used for the mitigations experiments in section 4.2? This section is impossible to evaluate without this information.
soundness: 1
strengths: - The paper studies an important, timely problem, and connects previous research on task similarity to the contemporary LLM setting.
- Figure 1 is a helpful clarification of the authors' approach.
- The mitigation case study---using replay on their predicted forgotten samples---is a good idea to test the utility of their predictions.
summary: This paper explores catastrophic forgetting of the pretraining or instruction tuning data after fine-tuning on downstream tasks. Specifically, the authors attempt to identify which samples (chunks of pretraining data) are forgotten when finetuning on various tasks, with a view to understanding the role of task similarity.

The authors analyse the amount of forgetting (measured by log perplexity increase) on each sample on a range of tasks. Analyzing the resulting data, the authors suggest that this data has a simple structure (though see weaknesses). This analysis relies upon a small selection of LLMs with open-source training data.

Building on this insight, the authors use matrix completion approaches predict which samples will be forgotten when training on a downstream task, though their approach requires access to forgetting data for some samples on the target task.

Finally, the authors use their prediction of which samples will be forgotten to upweight them during replay, and report improved forgetting mitigation compared to some other methods.
weaknesses: - The results of this paper rest on the idea that that there is a simple structure that explains which samples will be forgotten when training on a given downstream task. However, I don't see clear evidence that this is the case. For example, referring to fig. 2, the authors state that they "see Z generally displays a neat and simple pattern." Such a pattern is not at all clear to me, and I would question the value of visualizing a large matrix in order to draw conclusions about the complexity of its structure.
- The authors rely on linear regression modelling to quantify how well forgetting is explained by (the indices of) each sample and task. The authors admission that "the R^2 scores are relatively lower [on some models]" makes me question the robustness of this approach. Why should the results vary so significantly between models of the same scale (e.g. OLMo-7B and MPT-7B) or even within the same family, where we can assume the pretraining data is the same (e.g. OLMo-7B and OLMo-1B)?
- What do the SVD components in section 3.2, fig. 4, tell us that we did not already know from visualizing the full matrix, e.g. in fig 2c? It is already evident from the full matrix that finetuning on TruthfulQA results in greater forgetting.
- The matrix completion approach requires access to the forgetting of some samples on the test tasks, significantly limiting its practical utility.
- The paper needs significant attention to improve its language and clarity. Numerous sentences don't make sense (e.g., "There has been a growing need for long-term usability of LLMs"), are ungrammatical (e.g., "We perform statistics of forgetting"), or are unclear (e.g., "As a main comparator"). I would also strongly encourage the authors to avoid usage of the vague term "associations" throughout the paper, where the authors tend to mean "task similarity" or "relationship between pretraining and fine-tuning task." For example, the "association matrix Z" might be better described as the "sample forgetting matrix Z".
- The paper lacks substantive discussion of its limitations, with the exception of lines 428-433 on the limitations of replay (rather than the limitations of their work more generally).
- **Minor**: It would be helpful to restructure the abstract to more clearly motivate the problem and what the authors propose. At present there excessive detail about constructing matrices, which makes it hard to understand that the paper's central concern is predicting catastrophic forgetting as a function of task similarity.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 4
questions: 1. What are the takeaways/what does it mean that forgetting association matrices are low rank? Should I expect them to be high rank or should that be the null hypothesis? For the multiplicative model, how should I interpret these R^2 values?
2. For the multiplicative model, do you have any insights on the $\alpha_i$ and $\beta_j$s? Which tasks cause more forgetting and why? Is it because their data is out of distribution? Amount of data? Which examples are more likely to be forgotten and why? Are they weird/noisy examples? Do they have higher loss compared to the mean across the dataset i.e. less well learned? Are they from a specific domain?
3. How do you interpret the fact that kNN does better than SVD at predicting forgetting?
4. Can you show that forgetting isn't just a perplexity phenomena and actually impacts performance on benchmarks? Can this benchmark performance degradation be predicted? The reason this is important is because it is possible to increase perplexity on certain examples on the training set, especially if they are noisy/rare examples without hurting language model capabilities and it would be useful to understand this.
5. It seems to me that predicting the association matrix is not really necessary for mitigating forgetting because the ground truth forgetting is easy to measure: you already have to fine-tune your model on the "test tasks" because you need to measure forgetting on the seed dataset. So you are just saving the inference cost on the rest of the upstream examples you are trying to not forget but just for the test tasks?
soundness: 2
strengths: **Originality**: The paper takes an interesting and simple approach to understanding example forgetting and finds some intriguing results.
**Quality**: The paper follows a thread from understanding structure in these forgetting associations to developing a strategy to mitigate forgetting. It provides some baselines to argue for the effectiveness of all of its methods.
**Clarity**: The paper is very well written and clear. All the experiments, figures, methods, results are well explained and easy to follow.
**Significance**: The paper explores a relevant and important topic and presents an effective algorithm for mitigating forgetting.
summary: This paper investigates forgetting in LLMs by building simple, low-rank models of example associations between upstream examples and downstream tasks. Its contributions are to 
1. Demonstrate the simple low rank models can predict forgetting of upstream examples when models are trained on down stream tasks. It visualizes these associations and demonstrates goodness of fit. The authors also argue that these example association models are more predictive of forgetting than other baseline methods.
2. It predicts which examples will be forgotten when finetuned on new tasks and shows that this can be used to optimize the presentation of examples in a replay buffer to mitigate forgetting.
weaknesses: 1. While the fact that the forgetting association matrices are low rank, it is unclear what the takeaway from this is/how I should interpret or use this. Additionally I am not sure how to interpret any of the low rank structures. Also, the optimal predictor for forgetting with association matrix completion and what is used for forgetting mitigation is the kNN model which does not directly use this result?
2. I am not sure forgetting as measured by log perplexity is sufficiently convincing and would really like to see downstream accuracy on tasks. Is the forgetting that is observed actually hurting the capabilities of the models? Does it hurt the model's performance on benchmarks?

Meta-review: This work investigates the fine-grained associations between new tasks learned by LLMs and the phenomenon of catastrophic forgetting of upstream pretraining data. Through the perspective of matrix completion, the study models LLM forgetting as an $M \times N$ matrix, where M represents new tasks and N represents upstream data points. The authors find that these associations exhibit a low-rank structure, as simple multiplicative regression models, such as low-rank SVD, provide a reasonably accurate estimation of the matrix. Building on this observation, the paper introduces a novel example replay framework that mitigates catastrophic forgetting by replaying upstream examples predicted to be forgotten under the matrix completion framework during fine-tuning.

Following discussions, the reviewers reached a consensus to reject this work. The primary concerns raised include the limited practicality of the approach due to the reliance on obtaining ground truth forgetting data, insufficient performance gains, a lack of in-depth analysis, insufficient experimental validation of the main claims, and various other experimental shortcomings. As a result, the current version does not appear ready for publication. We recommend that the authors address these concerns thoroughly in line with the reviewers' feedback to strengthen the work.

Decision: Reject