Title: Prior Forgetting and In-Context Overfitting
Venue: NeurIPS 2025
Accepted: yes
Keywords: in-context learning

transformer

task recognition

task learning

pretraining

prior forgetting

in-context overfitting
Abstract: In-context learning (ICL) is one of the key capabilities contributing to the great success of LLMs. At test time, ICL is known to operate in the two modes: task recognition and task learning. In this paper, we investigate the emergence and dynamics of the two modes of ICL during pretraining. To provide an analytical understanding of the learning dynamics of the ICL abilities, we investigate the in-context random linear regression problem with a simple linear-attention-based transformer, and define and disentangle the strengths of the task recognition and task learning abilities stored in the transformer model’s parameters. We show that, during the pretraining phase, the model first learns the task learning and the task recognition abilities together in the beginning, but it (a) gradually forgets the task recognition ability to recall the priorly learned tasks and (b) relies more on the given context in the later phase, which we call (a) \textit{prior forgetting} and (b) \textit{in-context overfitting}, respectively.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates two modes of in-context learning (ICL) exhibited by pre-trained Transformer models during testing, with a particular focus on their emergence and dynamic evolution during the pre-training phase. \nThe study first analyzes a simple random linear regression problem using a linear attention mechanism, and then extends to more complex function classes such as two-layer neural networks, decision trees, exponential regression, softmax regression, and autoregressive learning. \nIn addition, the authors introduce a new setting called \"demonstration-query task irrelevance\" to examine the model's robustness and generalization under various distribution shifts.\nThey emphasize the balance of power between the two ICL modes and propose a measurement method to evaluate their strength and how they emerge or fade over time. \nThrough theoretical analysis and experimental validation, the authors demonstrate the phenomenon known as \"in-context overfitting\", where the model's generalization gap increases with greater context strength.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n+ The paper provides a deep theoretical understanding of the two in-context learning (ICL) modes in pre-trained Transformers, supported by rigorous empirical validation across a variety of task distributions and function classes.\n\n+ The authors introduce a new evaluation setting called \"demonstration-query task irrelevance\", which enables the study of model robustness and generalization under distributional shifts.\n\n**Weaknesses**\n\n+ While the paper provides a detailed theoretical analysis of the two-parameter Transformer model and demonstrates similar behavior to the full-parameter model through experiments, **this simplification may overlook some dynamic changes and characteristics present in more complex models**. Consequently, the findings might not fully represent the behaviors of the more intricate Transformer architectures used in practical applications. Additionally, it's worth noting that LLMs' pre-training typically involves training on datasets at the billion-token scale. Although this study claims to investigate phenomena during the pre-training phase, **its setup does not entirely align with conventional LLM pre-training practices**, which could be considered one of its primary limitations.\n\n+ It would be beneficial for the authors to employ **more complex datasets** to analyze the dynamics of in-context learning and evaluate Task Recognition and Task Learning using setups similar to those in other relevant studies [1, 2].\n\n+ Although the paper mentions several hyperparameters, such as prior strength and in-context strength, there could be more extensive experimentation to explore how these parameters influence model performance.\n\n[1] What In-Context Learning \"Learns\" In-Context: Disentangling Task Recognition and Task Learning, ACL 2023.\n\n[2] Investigating the Pre-Training Dynamics of In-Context Learning: Task Recognition vs. Task Learning, ICLR 2025.\n\nQUESTIONS:\nSee Weaknesses\n\n\nIf the authors could **provide experiments on larger-scale LLMs and more complex datasets**, I would be willing to raise my score.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis theoretical paper considers in-context learning (ICL) and argues, using a simple theoretical model, that two distinct modes of ICL dominate different stages of pretraining. In the early stage, the model relies more on task identification than on in-context examples. In the later stage, in-context examples become more important. These two ICL modes have been previously observed and described. The main novelty is in developing a simple model that captures these behaviors and how they emerge and dominate different pretraining stages. The theory is validated with some experiments.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- A simple model allows for a controlled study of ICL behavior, generating insights that have the potential to generalize to more complex settings\n- A simple theory captures task recognition vs task learning competition during pretraining\n- The paper is well-structured and mostly clear\n- Derivations are easy to follow\n\n**Weaknesses**\n- Lack of comparison with benchmark algorithms for linear regression like ridgeless regression and/or Bayes optimal for the noncentred prior consider in the paper; it would be instructive to understand when ICL outperforms naive ridgeless regression by specializing to prior and whether prior forgetting makes ICL solution closer to ridgeless regression (which is optimal for centred isotropic prior)\n- The experiments are limited to linear attention; it would be good to see the results validated, even qualitatively, in more realistic settings \n- If accepted, the authors should consider using the extra page to explain and discuss what their insights mean for more practical settings; do we see a transition from task recognition to task learning during pretraining in these settings?\n\nQUESTIONS:\n- In Fig 3, the generalization gaps for two and full-parameter attentions are very similar. Is this a coincidence, or are there reasons to expect this result?\n- The demonstration-query task irrelevance appears similar to _concept shift_, which can drive a specialization-generalization transition; see, eg, arXiv:2409.15582. What is the connection between _prior forgetting & in-context overfitting_ and this concept-shift-induced transition?\n- The paper title, _\"Prior Forgetting and In-Context Overfitting\"_, suggests that this ICL behavior is something we want to avoid, but I understand that it is just another mode of ICL which is sometimes desirable (even though it increases generalization gaps in the settings considered in this paper). Do I miss something important that the title is supposed to convey?\n- What are the lessons from this paper that might translate to practical settings? I understand that this paper is purely theoretical, but a discussion of the potential impact (beyond revealing and characterizing interesting phenomena, which I appreciate) would significantly strengthen the paper.\n- The training plateau is quite common in ICL training but seems absent from the current analysis. Can the authors comment on this absence?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the training dynamics of ICL with transformer, with simplified linear transformer and synthetic linear regression tasks. To decouple and evaluate the task recognition and task learning ability, this paper propose to investigate with demonstration-query task irrelevance. Results mainly show that the model first learns the task learning and the task recognition abilities together in the beginning, but it gradually forgets the task recognition ability to recall the priorly learned tasks and relies more on the given context in the later phase.\n\nSTRENGTHS AND WEAKNESSES:\nThe topic is interesting and important. Understanding ICL, from early works which focus on how a learning algorithm can be expressed, to recent works (including this paper) which focus on how the ability emerges from pre-training, helps us to understand, explain and improve LLMs.\n\nThe first noticeable (but not that important) weakness is that latest related works about understanding ICL with transformer are not discussed nor mentioned, e.g., [1][2].\n\nThe main weakness of this paper is critical: the setting has departed too far from real world LLMs to be meaningful. Three levels of departures can be recognized: \n\nFirst, the linear transformer and synthetic linear regression tasks. Though different from real LLMs, the reviewer can accept this one in most cases, as it facilitates research.\n\nSecond, pre-training with explicitly arranged ICL input ([xy,xy,x?]). The reviewer think this is not  to investigate how the ICL ability emerges from pre-training. Because this is not reduction problem any more, as modern LLMs do not pre-train in such cases.\n\nThird, some unique settings in this paper. The reviewer could not understand what the value of loss between an random label that is independent with input means (Demonstration-Query Task Irrelevance). And the mean value of linear weights is specific to linear regression task (quadratic loss), while LLMs do pre-training neither with such tasks nor with such loss.\n\nBased on above judgement, the reviewer has no longer check the result and proof details. \n\n[1] Wu S, Wang Y, Yao Q. Why In-Context Learning Models are Good Few-Shot Learners?[C]//The Thirteenth International Conference on Learning Representations.\n[2] Yang L, Lin Z, Lee K, et al. Task Vectors in In-Context Learning: Emergence, Formation, and Benefit[J]. arXiv preprint arXiv:2501.09240, 2025.\n\nQUESTIONS:\nQ1. With some specific settings in this paper, will it be possible to directly identify the global minimum (like [3]), thus illustrate the main results as the process of parameter convergence?\n\nQ2. Why not train in auto-regressive manner? And will the context length matter the main results?\n\n[3] Ahn K, Cheng X, Daneshmand H, et al. Transformers learn to implement preconditioned gradient descent for in-context learning[J]. Advances in Neural Information Processing Systems, 2023, 36: 45614-45650.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the in-context learning (ICL) ability of large language models (LLMs), particularly during the pretraining phase. The authors investigate the emergence and dynamics of two learning modes, task recognition and task learning. It also reveals two key phenomena: Prior Forgetting and In-Context Overfitting.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n \n- **Originality:** The study explores ICL dynamics from a pretraining perspective, which provides a fresh and insightful perspective. The identification of Prior Forgetting and In-Context Overfitting as emergent phenomena is novel and thought-provoking.\n\n- **Significance:** Understanding how ICL abilities emerge during pretraining can have a substantial impact on the future design of language models, especially regarding pretraining efficiency and generalization capabilities. \n\n- **Quality:** The paper includes thorough and detailed analysis and experiments, which help substantiate the proposed claims and theoretical framing.\n\n- **Clarity:** This paper is well-written and easy to follow. However, there is room for improvements as I mention below.\n\n\n**Weaknesses:**\n\n1. The proposed framework is conceptually novel, especially in its focus on the emergence and dynamics of task recognition and task learning during pretraining. However, much of the empirical analysis relies on simplified models (e.g., two-parameter or single-layer linear transformers), which may limit the extent to which the framework can be fully validated. This also raises concerns about the generalizability of the findings to large-scale, real-world LLMs.\n\n2. **Clarity**\n- 2.1 Figure 1 (left) lacks clarity: the meaning of the axes, curves, and color coding is not well-explained. \n- 2.2 Notational consistency can be improved. For example, distinguishing vectors (e.g., $\\textbf{x}^{(i)}$ ) from scalars (e.g., $y^{(i)}$) using bold font would enhance readability.\n\nQUESTIONS:\n1. Based on Figure 1 (left) and the definition of in-context overfitting in lines 250–251, there seems to be a one-to-one correspondence between increasing in-context strength and overfitting. However, in the early stage of training, models are typically underfitting. How should this apparent contradiction be understood?\n2. Is there a connection between the minimum point of the $L_{test}$ curve and the minimum of $\\alpha(t)$? If so, could this be explained or analyzed more clearly?\n3. Why do Figures 2 and 3 not use the same task dispersion setting? Using a consistent setup would help make comparisons more interpretable.\n4. Can the findings in this paper offer any practical guidance for improving pretraining strategies or designing better ICL prompts?\n\nLIMITATIONS:\nNo, although limitations are mentioned in the checklist, they are not clearly explained in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}