Title: Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context
Venue: NeurIPS 2025
Accepted: yes
Keywords: In-Context Learning

Transformer

Large-Language Model

Long Context

Many-Shot ICL

Scaling

Meta Learning
Abstract: Transformers have demonstrated remarkable in-context learning (ICL) capabilities, adapting to new tasks by simply conditioning on demonstrations without parameter updates. Compelling empirical and theoretical evidence suggests that ICL, as a general-purpose learner, could outperform task-specific models. However, it remains unclear to what extent the transformers optimally learn in-context compared to principled learning algorithms. To investigate this, we employ a meta ICL framework in which each prompt defines a distinctive regression task whose target function is drawn from a hierarchical distribution, requiring inference over both the latent model class and task-specific parameters. Within this setup, we benchmark sample complexity of ICL against principled learning algorithms, including the Bayes optimal estimator, under varying performance requirements. Our findings reveal a striking dichotomy: while ICL initially matches the efficiency of a Bayes optimal estimator, its efficiency significantly deteriorates in long context. Through an information-theoretic analysis, we show that the diminishing efficiency is inherent to ICL. These results clarify the trade-offs in adopting ICL as a universal problem solver, motivating a new generation of on-the-fly adaptive methods without the diminishing efficiency.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission considers the optimality of algorithms implemented by transformers via in-context learning (ICL). The main finding is that transformers implement suboptimal ICL algorithms, especially in the limit of long contexts (ie, more in-context examples) and when the context length at test time is longer than during training (length generalization). This result is based on a _stylized_ learning problem with a hierarchical data-generating process.\n\nSTRENGTHS AND WEAKNESSES:\nUnderstanding ICL in transformers and its limits is timely and important. I think this paper makes a meaningful contribution to this area. \n\n**Strength**\n- The use of a toy problem allows for a more in-depth study of ICL performance\n- Using principled algorithms as baselines enables quantification of ICL sample (context) efficiency\n- The paper is well structured and reads rather well\n\n**Weakness**\n- The _stylized_ setting is quite specific, the claim is more general.\n- I understand that the authors aim to illustrate the phenomenon, and I consider this a valuable contribution. But I think the abstract could be more informative about the learning problem considered.\n- The information-theoretic analysis is interesting, but it is not directly compared to experimental results. Perhaps mine is an unpopular opinion, but I feel that the technical details do not add much to the experimental results here. I'd rather see more experimental results on different learning problems.\n- The related work section is rather light. I note that the most recent reference is more than a year old (essentially an epoch in modern ML research). Expanding this section would allow the authors to better place their results in the growing body of research seeking to characterize and understand ICL with _\"physics-style\" or \"synthetic benchmark\"_ approaches.\n\nQUESTIONS:\nRaventós et al (NeurIPS 2023) argue that transformers achieve better generalization performance by _failing_ to be Bayes optimal (with respect to the empirical pretraining distribution). My understanding is that in the current submission, transformers never outperform Bayes optimal algorithms. I would appreciate a discussion on this point.\n\nLIMITATIONS:\nPlease see my comments about the _stylized_ setting in **Strengths And Weaknesses** above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work investigates the diminishing efficiency of in-context learning (ICL) in the many-shot scenario. The authors pre-train a Transformer model on a given task distribution for ICL and compare its sample efficiency against optimal baselines. They discover that the efficiency of in-context learning is high in the few-shot regime and decreases as the context length increases. They justify their observations by analyzing the suboptimality of ICL and showing that the suboptimality increases as the demonstration size grows.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper is overall clearly written. The notations are well-defined, and intuitions are provided to interpret the results.\n- The empirical study is well-designed with good statistical rigour.\n- The diminishing efficiency phenomenon in ICL is a real and practical concern for ICL applications.\n\n**Weaknesses**\n- The authors draw connections between their results and large language models. However, it is unclear whether the decreased efficiency of ICL in their particular setup and model size persists in LLMs, where the models are significantly larger and trained on a wide range of demonstration sizes. In the authors' setup, they only considered a fixed largest demonstration size $T_{train}$.\n- The analysis is based on the Assumption that there exists a uniform lower bound on the excess risk after a certain demonstration length, which can be too strong to hold.\n- This work only considered in-context supervised regression problems, leaving classification problems and other learning paradigms, such as reinforcement learning, unexamined.\n\nQUESTIONS:\nI see the authors use the Transformer as their default model architecture to examine the drop in sample efficiency for in-context learning with long contexts. I wonder what special properties of the Transformer they leveraged for their analysis? I don't see any architectural requirement in their assumptions. Thus, why don't the authors also consider other sequence modelling models, such as the LSTM, as ICL is not exclusive to the Transformer?\n\nLIMITATIONS:\n1. I think the authors should also acknowledge that their model size and training data diversity are not comparable to today's LLMs.\n2. I believe the authors should also acknowledge that they only investigated in-context regression. Otherwise, they should explicitly state that they use the term in-context learning to indicate in-context regression specifically.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the sample-efficiency of in-context learning (ICL) in transformer models, focusing on how many demonstrations are needed to reach a target prediction error compared to Bayes-optimal or principled learners. Using a stylized regression benchmark with Fourier-series functions and controlled noise, the authors show that transformers are nearly optimal in the few-shot regime but become significantly less efficient as the number of in-context examples increases. They formalize this observation using a performance-profile metric and support it with an information-theoretic analysis, which proves that under a certain assumptions the transformer’s efficiency must diverge from the Bayes rate in the long-context limit.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is addressing an interesting problem that what you lose if you use ICL instead of principled algorithms. \n\n\nWeaknesses\n\n1. From expressivity point of view the transformer can implement many principled algorithm in its forward pass, an idea popularized by Oswald et al.[1] and many other follow-ups. So, if that's the case then I do not see the central point of this paper. The context length for fixed error level should be exactly the same for transformer or base algorithm.\n\n2. Related to 1, ICL is nothing special from traditional learning paradigm. We have some demonstration and we want to predict a new query, so transformer can be thought as an algorithm potentially.\n\n3. I think the main problem of this paper is Assumption 4.1,  basically as the model gets larger and more expressive, and also number of pre-traing increases we can get closer and closer to best possible error. And this is align with scaling-law common wisdom. \n\n4. Related to 3,  because the transformer is meta-trained across many tasks, it can, in principle, exploit cross-task regularities that a hand-crafted Bayes estimator (tuned per task) cannot access. The analysis ignores this potential advantage.\n\nBottom line: I think this paper assumption is too restrictive and unrealistic, and not adding any new insight to what ICL actually is.\n\n[1] [Transformers learn in-context by gradient descent](https://arxiv.org/abs/2309.14666)  \nJohannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, Max Vladymyrov. *arXiv preprint arXiv:2309.14666*, 2023.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the efficiency of in-context learning (ICL) in Transformers compared to principled learning algorithms. This paper introduces a framework based on performance profiles and sample complexity to quantify the (sub-)optimality of ICL. This paper finds that while ICL is nearly as efficient as a Bayes optimal estimator in few-shot settings, its efficiency significantly degrades in many-shot scenarios that require a long context. Through an information-theoretic analysis, the paper argues that this diminishing efficiency is an intrinsic property of the ICL mechanism, caused by a non-vanishing excess risk. The work concludes that ICL carries a \"technical debt\" of sample inefficiency in high-performance regimes.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n\n1. The comparison of few-shot and many-shot settings for ICL problems is fundamental and impactful for better understanding of the transformers models.\n\n2. The paper presents a clear and compelling dichotomy: ICL is near-optimal in the few-shot regime but becomes substantially suboptimal in the many-shot regime. \n\n3. The paper supports its empirical findings with a solid information-theoretic analysis in Section 4. By decomposing the ICL error into Bayes risk and excess risk, and arguing that the excess risk for ICL is non-vanishing. \n\n## Weakness\n\n1. The primary limitation, which the authors acknowledge, is the use of a stylized setting (regression on a mixture of Fourier series). Although it's a clean setup, it raises questions about the *generalizability* of the findings to the complex, high-dimensional, and often discrete tasks that real-world LLMs handle.\n\n2. The theoretical results hinge on Assumption 4, which posits a non-vanishing lower bound on the excess risk. While this assumption is well-supported by the paper's own experiments, it is still an assumption about the behavior of transformer-based ICL. I would like to hear more discussion about this assumption.\n\nQUESTIONS:\nThe core of the theoretical argument rests on the non-vanishing excess risk ($epsilon_{XS}^t$). Your experiments (Fig 3b, Fig 4) convincingly show this behavior. Do you have a hypothesis for why the Transformer architecture, trained with this objective, leads to a non-vanishing excess risk? Is it related to the fixed-capacity nature of the attention mechanism or something else?\n\nYou conclude by motivating **a new generation of 'on-the-fly' adaptive methods without the diminishing efficiency.**  Could you elaborate on what such methods might look like?\n\nLIMITATIONS:\nYes. The authors have discussed their limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}