Title: Demystifying Language Model Forgetting with Low-rank Example Associations
Venue: NeurIPS 2025
Accepted: yes
Keywords: catastrophic forgetting

continual learning

large language model
Abstract: Large Language models (LLMs) suffer from forgetting of upstream knowledge when fine-tuned. Despite efforts on mitigating forgetting, few have investigated how forgotten upstream examples are dependent on newly learned tasks. Insights on such dependencies enable efficient and targeted mitigation of forgetting. In this paper, we empirically analyze forgetting that occurs in $N$ upstream examples of language modeling or instruction-tuning after fine-tuning LLMs on one of $M$ new tasks, visualized in $M\times N$ matrices. We show that the matrices are often well-approximated with low-rank matrices, indicating the dominance of simple associations between the learned tasks and forgotten upstream examples. Leveraging the analysis, we predict forgetting of upstream examples when fine-tuning LLMs on unseen tasks with matrix completion over the empirical associations. This enables fast identification of most forgotten examples without expensive inference on the entire upstream data. Despite simplicity, the approach outperforms prior approaches that learn semantic relationships of learned tasks and upstream examples with LMs. We demonstrate the practical utility of our analysis by showing statistically significantly reduced forgetting as we upweight predicted examples for replay during fine-tuning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper demystifies LLM forgetting by showing that the associations between newly learned tasks and forgotten upstream examples form a low-rank matrix. Based on this finding, the authors introduce a highly efficient method to predict future forgetting by treating it as a matrix completion problem. The practical utility is confirmed by using these predictions to guide a weighted replay strategy, which reduces forgetting during fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\nStrong points：\n1. Models catastrophic forgetting with an innovative and analyzable low-rank matrix framework.\n2. Formulates forgetting prediction as a matrix completion problem, which is more effective and computationally cheaper than prior methods.\n3. Provides extensive empirical evidence across multiple models and scales, with a closed-loop validation showing that predicted forgetting can be used to mitigate actual forgetting.\n \n \nWeak:\n1. In Figure 3, the paper shows that the complexity of forgetting associations differs significantly between OLMo-1B and OLMo-7B, with the latter exhibiting more complex patterns (lower R² at the same rank). Furthermore, Figure 10 highlights \"interesting patterns\" specifically in OLMo-7B. Despite this, the crucial experiment in Table 2, which aims to interpret these associations by checking correlations with similarity measures, was conducted on OLMo-1B. This choice is questionable, as an analysis on the more complex OLMo-7B would have been more compelling and could have yielded different insights into the nature of forgetting in more capable models.\n2. The paper concludes from Table 2 that similarity measures \"correlate poorly\" with forgetting, dismissing them as weak predictors. However, this conclusion may be strong. For instance, the positive correlation for ⟨Gradient, Gradient⟩, while small, suggests that alignment in gradient space does contribute to forgetting. \n3. The primary metrics, log perplexity and Token F1, show statistically significant but numerically small improvements (e.g., in Figure 5). This raises questions about the practical impact of the mitigation. A more direct and functional evaluation of forgetting would be to measure performance degradation on a specific, previously fine-tuned task (Task A) after the model learns subsequent new tasks (B, C, etc.). By constructing an association matrix between Task A's examples and the new downstream tasks, one could directly quantify the retention of a specific capability, which is a more intuitive and practical measure of forgetting than perplexity changes across a vast pre-training corpus.\n4. The entire method depends on access to upstream data for building the association matrix, making it unsuitable for closed-source models.\n5. The best-performing offline mitigation variant doubles the fine-tuning computational cost by requiring a preliminary run, hindering its practical application.\n\nQUESTIONS:\nplease see the weakness part.\n\nLIMITATIONS:\nNO", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates forgetting in LLMs using a matrix factorization framework that captures interactions between training examples and downstream tasks. The central claim is that forgetting matrices can be approximated via low-rank decompositions, enabling prediction of forgetting effects prior to fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- Elegant abstraction of example-task interactions.\n- Matrix completion to predict forgetting is novel, simple and straightforward.\n- Strong empirical support on multiple architectures (MPT, Pythia, OLMo).\n\n**Weaknesses:**\n- Limited insight into *why* low-rank holds. is it architectural, training-dynamic, or data-driven? Do we have any theorectical conclusions about the low-rankness.\n- The definition of forgetting is not formalized. It is unclear that if an example should be considered as forgotten when completely disappearing from any possible prompted generation or with no system prompts. Sometimes it might seem like the example is forgotten but when we twist the prompt slightly, we will see the model actually does not forget the example but just hide it from surface. For example, RLHF is believed to be useful for forgetting toxicity but recent research find that they might just hide the toxicity from the surface (Table 3, Chen et al 2024 Jet expansions of residual computation)\n\n\n**Missing References:**\n\nThe paper can be improved by adding a general discussion of related work on forgetting. The paper focuses on the negative side of forgetting while recently the positive side of forgetting has also been researched. For example, \"Improving Language Plasticity via Pretraining with Active Forgetting\" (Chen et al., NeurIPS 2023), introduces a pretraining-time forgetting mechanism that proactively improves model plasticity. This is highly relevant to the stated goal of understanding and mitigating forgetting in language models but more from a positive angle. Including this work would help position your contribution more clearly within the broader literature that views forgetting not only as a side effect but also as a potentially useful objective. Additionally, the authors should consider citing \"Digital Forgetting in Large Language Models: A Survey of Unlearning Methods\" (Blanco-Justicia et al., 2024), which provides a comprehensive overview of the forgetting landscape, including both proactive and reactive forgetting for removing data points from LLMs.\n\nRegarding the methodology for tracing what is forgotten, the paper currently relies on continuous perplexity shifts and exact match drops. These metrics are informative but may miss deeper semantic shifts. To complement this, the paper may benefit from referencing \"Jet Expansions of Residual Computation\" (Chen et al., 2024), which explores how structured knowledge flows through residual paths in LLMs. While not directly about forgetting, it provides a way to analyze internal changes between models before and after finetuning, effectively offering a model-level “diff” that may surface subtle patterns beyond output metrics. Although not explicitly about forgetting, these tools could enrich your analysis of how knowledge is forgotten during finetuning.\n\nQUESTIONS:\n1. What inductive biases do you think induce low-rank structure in forgetting matrices?\n2. How stable is the low-rank approximation with respect to noisy or adversarial fine-tuning? Does PEFT (parameter efficient finetuning) such as LoRA change the conclusion?\n3. Could this method be used *during* training to halt updates early when forgetting risk is high?\n\nLIMITATIONS:\ndiscussed in the appendix", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe dependency between forgotten upstream examples and the newly learned tasks can enable the mitigation of forgetting when fine-tuned. This work shows that the association matrices are often well-approximated by low-rank matrices, and matrix completion allows fast identification of the most forgotten examples.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- Proposes a simple method that estimates the associations between learned tasks and forgotten examples through low-rank matrix approximation, and predicts forgotten examples by solving a matrix completion problem.\n- Demonstrates a mitigation approach that leverages the predicted forgetting. The practical utility is validated by showing statistically significant reductions in forgetting.\n\n### Weaknesses\n\n- **The experimental setting is overly idealized.** A critical limitation is that the method assumes access to the previously trained data, which is often unavailable in practice due to expired licenses or unreleased datasets. This significantly limits the applicability of the method. In real-world scenarios, such as post-training an LLM on a downstream task, the previously used data is unknown, making it infeasible to predict forgotten samples.\n- **The mitigation method’s performance is not consistently superior,** for example, in Figure 5(c), its performance is comparable to random sampling of upstream examples. This raises concerns about the accuracy of the predicted forgetting and the effectiveness of the proposed sampling strategy.\n- **Log perplexity may not be an appropriate evaluation metric for assessing forgetting mitigation.** A more meaningful metric would capture the extent to which forgetting degrades model capability, such as benchmark-specific accuracy or task-relevant correctness (e.g., accuracy on math problems).\n\nQUESTIONS:\n1. The experiments use only decoder-only models. Would this method also work effectively on encoder-only models or mixture-of-experts architectures? Extending the evaluation to diverse model architectures would help assess the generalizability of the approximation method.\n2. Why is forgetting measured with log perplexity (weakness 3)? Why not use task-specific metrics, for example, context-aware accuracy or answer correctness for QA datasets? What justifies log perplexity as the primary metric? A more detailed explanation is needed.\n3. Does the replay approach impact the learning of the new task? Replaying a mini-batch of upstream examples every few training steps could interfere with the training dynamics on the new task. Have you assessed this effect?\n4. If the previous data is already known, why is matrix-based prediction necessary? One could directly measure actual forgetting using the original data, and those results could serve as ground truth for evaluating the proposed mitigation method. This seems circular… A clarification would be helpful.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores how LLMs forget during finetuning. They do this by building forgetting matrices between “upstream” examples (i.e. examples that had good performance before finetuning) and new tasks (i.e. after finetuning). They propose a method to efficiently predict forgetting of upstream examples, as well as an algorithm to mitigate forgetting during LLM finetuning. This has many practical implications for LLM finetuning and dataset selection and will be of interest to a wide range of LLM practitioners.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* Well written and well motivated. The authors find that associations between learned tasks and forgotten examples are often well approximated with low-rank matrices, and use this to motivate their main method: matrix completion over association matrices.\n* This paper is a tour de force across models (OLMo, MPT, Pythia, OLMo2, OLMo-Instruct) and datasets (FLAN, Tulu, Dolly), with an extensive appendix\n\nWeaknesses:\n* This paper presents a lot of empirical results that are somewhat difficult to interpret (i.e. all the matrix plots in the appendix). Aside: This is very helpful/useful information, and I wonder if the paper would benefit from some kind of interactive website that allows users to explore the results with more granularity.\n* The final results in Figure 5 are quite compelling. However as a reader I am left wondering how much more efficient this result is in practice. Is this method and approach something we should all be using, or just something scientifically interesting?\n\nQUESTIONS:\nQ1: I assume all models except OLMo-7B-Instruct are base models\n\nQ2: It would be helpful to include some explicit examples in the appendix of data pairs/tasks where forgetting is high and where forgetting is low. This would help the reader build some intuitions around the very dense matrix plots that are presented (I know the Appendix is already quite extensive)\n\nQ3: Why is the matrix for OLMo-7B-Instruct much more different than the the matrices for the other models in Appendix Figure 6?\n\nQ4: “We empirically notice that the associations between learned tasks and forgotten upstream examples are more complicated in more recent and capable LLMs” - is this simply because more of the finetuned datasets are more likely to be included in OLMo pretraining?\n\nQ5: Why does OLMo2 have zero average forgetting in Figure 3(d)?\n\nQ6: Why do you use all-distilroberta-v1 as your embedding model? This seems quite outdated\n\nQ7: What exactly is the “ground truth forgetting” in Figure 5? Is it the performance of the model after full finetuning?\n\nQ8: What are the pros and cons between online and offline MF in Table 4?\n\nLIMITATIONS:\nThe authors have addressed the limitations and potential negative societal impact.\n\nMinor issues:\n* The citation for MPT and RedPajama is [Computer 2023] which seems incorrect. Seems that most papers cite the original MPT blogpost \n* Line 129: “knwon” → known\n* The schematic in Figure 4 is slightly confusing. It might be helpful to split into (a) and (b) sections\n* Line 241: “The additive linear model approximate forgetting”", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-11"}