Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context
Authors: Taejong Joo
Diego Klabjan
Venue: NeurIPS 2025 poster ()
Keywords: In-Context Learning
Transformer
Large-Language Model
Long Context
Many-Shot ICL
Scaling
Meta Learning
Primary area: deep_learning

Abstract: Transformers have demonstrated remarkable in-context learning (ICL) capabilities, adapting to new tasks by simply conditioning on demonstrations without parameter updates. Compelling empirical and theoretical evidence suggests that ICL, as a general-purpose learner, could outperform task-specific models. However, it remains unclear to what extent the transformers optimally learn in-context compared to principled learning algorithms. To investigate this, we employ a meta ICL framework in which each prompt defines a distinctive regression task whose target function is drawn from a hierarchical distribution, requiring inference over both the latent model class and task-specific parameters. Within this setup, we benchmark sample complexity of ICL against principled learning algorithms, including the Bayes optimal estimator, under varying performance requirements. Our findings reveal a striking dichotomy: while ICL initially matches the efficiency of a Bayes optimal estimator, its efficiency significantly deteriorates in long context. Through an information-theoretic analysis, we show that the diminishing efficiency is inherent to ICL. These results clarify the trade-offs in adopting ICL as a universal problem solver, motivating a new generation of on-the-fly adaptive methods without the diminishing efficiency.

Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context
Taejong Joo & Diego Klabjan
Department of Industrial Engineering & Management Sciences
Northwestern University
Evanston, IL, USA
{taejong- joo, boooddkiin jt..aaaaannnorrrhheeeeeeeeeeeeee www.s/oootwwwttecccmm
Abstract
Transformers have demonstrated remarkable in-context learning (ICL) capabilities, adapting to new tasks by simply conditioning demonstrations without parameter updates. Compelling empirical and theoretical evidence suggests that ICL, as general-purpose learner, could outperform task-specific models. However, it remains unclear to what extent the transformers optimally learn in-context com- pared to principled learning algorithms. To investigate this, we employ meta ICL framework in which each prompt defines a distinctive regression task whose target function is drawn from hierarchical distribution, requiring inference over both the latent model class and task-specific parameters. Within this setup, we benchmark sample complexity of ICL against principled learning algorithms, in- cluding the Bayes optimal estimator, under varying performance requirements. Our findings reveal striking dichotomy: while ICL initially matches the efficiency of a Bayes optimal estimator, its efficiency significantly deteriorates in long context. Through an information-theorft
analysis, show that the diminishing efficiency is inherent to ICL. These results clarify the trade-offs in adopting ICL as a univer- sal problem solver, motivating new generation of on-the-fly fly adaptive methods without the diminishing efficiency.
Introduction
Transformers, particularly large language models (LLMs), are able to perform in-context learning (ICL) [[] they can adapt to new tasks simply by conditioning on demonstrations their input prompt 2). Not only conveniently operated without any explicit parameter updates, but ICL even with just a few demonstrations (a. few-shot ICL) surprisingly outperforms task-specific state-of-the-ar models in diverse tasks, from question answering to common
reasoning 3 T.
This raises fundamental question whether ICL can act as a universal learner, replacing tase-spec models. To answer this, we must first address a more precise question: How optimal is ICL learning algorithm, compared to principled learning algorithms? In principle, this could be answered by exhaustively benchmarking ICI. against principled learning algorithm across varied data and model scales [5] and task types [7] 8]. However, the computational demands for training modern LLMs pose significant challenges for such direct comparisons. The goal of this work is to answer the question without such prohibitive computational demands.
To answer the question, theoretical studies have analyzed asymptotic behavior of ICL using rich tools from statistics and learning theory, such as regret and generalization bounds [9[2]. However, these asymptotic results fall short of fully characterizing real-world LLM behavior. For instance, the regret
Our source code is available at http:p///ggttaae.eooeo///////o/a
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

upper bound for LLMs become nearly vacuous in few-shot regimes [13] 14], which cannot explain the striking few-show ICL performances. Moreover, because other principled learning algorithms have the similar asymptotic behavior, it remains unclear whether ICI. is better learning algorithm than such learning algorithms.
Stysics-style or synthetic benchmarking approaches have provided valuable insights that transformers might optimally learn in-context, isolating core aspects of LLM training in controlled environments (1HI7). These approaches by nature can enable an efficient, comprehensive comparison between ICL and principled learning algorithms with arbitrarily high levels of statistical significances, providing insights that often generalize to real-world LLMs despite inherent simplifications (see Appendix A..] for more detailed discussion on the usage of stylized setting). Notably, Garg et al. [16] and follow- up works [18 19] present various stylized settings where the ICL performances across different demonstration sizes resembles the learning curve of the optimal learning algorithm (e.g., Figure Bap. However, these works have not yet provided an explicit relationship between relevant quantities (e.g.. sample complexity and the optimality gap). Thus, the question of what extent transformers can learn optimally in-context remains unanswered.
To quantify optimality of ICL as learning algorithm, we compare ICL's sample complexity- related measures to those of principled learning algorithms by revisiting the performance profiles 22 classic benchmarking framework for optimization software. As a result, we uncover new insight on optimality of ICL in ] While ICL with few-shot demonstrations achieves near optimal sample complexity, ICL's sample complexity sharply deteriorates as the number of demonstrations increases in long context. Concretely, many-shot ICL. often requires 1.5 times more demonstrations than the Bayes optimal estimator to achieve the same performance. This indicates that, although transformers are theoretically capable of implementing principled algorithms in-context [19], their in- context learning behavior deviates significantly from the optimal learning algorithm in the many-shot regime. We further present evidence that, unlike principled algorithms, ICL may lack fundamental statistical properties (e.g., consistency and asymptotic efficiency) that are critical for algorithms to effectively learn from large demonstration sizes. Crucially, as ICL performances generally improve with more demonstrations, this novel insight would be difficult to uncover without directly comparing ICL to the principled learning algorithms with proper sample complexity measures.
To solidify this empirical finding, we provide diminishing efficiency is intrinsic to the ICL. mechanism itseff in $4 Specifically, we prove that ICL without diminishing efficiency has stringent necessary conditions (e.g.. negligible excess risk), and the result is independent to particular instantiation of models and environments. The results explain ICL's deficient sample complexity compared to the principled learning algorithm in many-shot regimes.
analyses demonstrating that the
Taken together, our work unveils a hidden technical debt in the ICL mechanism, suggesting a nuanced view of ICL as a universal problem solver: the price we pay for its training-free adaptability is fundamental inefficiency in sample complexity that compounds as we push toward higher performance targets with the current ICL mechanism as Crucially, this debt appears intrinsic to the ICL. mechanism and thus unlikely to be serviced by simply scaling data and model sizes. We hope these insights clarify the trade-offs in adopting ICL. as a universal problem solver and motivate a new generation of fon-the-fly" adaptive methods without the diminishing efficiency.
2 Setup
In 2.1 we describe the meta ICL environment for evaluating ICL learning algorithm, followed by designs of a transformer for solving the meta ICL task ($2.2). We then devise principled predictors and compare them with transformers using performance measures defined in $24
2.1 Meta ICL Environment
In the meta ICL [66, each prompt characterizes an instance of a learning problem. Specifically, prompt Hr consists of demonstrations with test input, i.e., Hr A (X1, Y1, Xr, Yr, XT- and each output is generated by some function f*, i... Y, (X.) for E [T+1] = 21, - 1}. Here, the goal of transformer is formalized as accurately predicting YT+1 with H., which requires to (implicitly) infer the underlying function I" from the demonstrations. We denote the set of demonstrations as Dr e (X1, Yı, .x, Yr). Following the meta ICL literature [16] [8
2

---

19 6]. we focus on regression problems where many principled learning algorithms can be derived
analytically (cf. $23)
For the data generating distribution of a prompt Hr. we follow the approach of sampling target functions / from hierarchical distribution [21] capture more interesting aspect of learning algorithm-model selection. Under the hierarchical f*, the prompt Hr is realized by the following sampling process, which is denoted as Hr ~ E() with parameters & - ([M), a )²).
1) Sample the implicit dimension m U{[M)) from uniform distribution over set [M] and construct the (unobservable) feature space 2m (x) [1. cos { )i,sing _ ) cos( sin T ]] where
T > 0 controls the frequency of the trigonometric functions.
2) Sample weight Wm N00 221+,, where I2m+1 is the identity matrix with rank 2m + The weight Wmm. defines the target function (*) wn 4m (1)) // + where the constant vm
makes the variance of f* remains constant across different We let Fm A {w" 4 (-)lw R denote the set of all target functions with implicit dimension m.
3) Construct prompt Hy with a test output y1+1 by 2, U([ ([amin ma ye (x1) + for E + 1]. where & ~ (O( of) is a random observation noise.
This hierarchical sampling involves a rich class of functions since {1) U {cos{ maz
JmeN U
{sin( MEN forms basis of square-integrable functions [Tmin Tmax)] Following Panwar
et al.
21. we set T = -max -miin = 5 and M = 10 (our findings are indifferent to these values).
We benchmark ICL with respect to different configurations of E, called scenario, to enable compre- hensive evaluations that could be encountered in practical scenarios (e.g.. low signal-to-noise ratio (SNR), defined Var (fr")f/'? for emulating highly noisy environment). We denote S as set of scenarios and & as parameters of scenario E S. We also have Hi - (Xf, Yf, ... x1+1) generated from IP(;; E,) for each scenario where we omit superscripts when there is no ambiguity.
2.2 Transformers
For a transformer TF.. we adopt the setup from Garg et al. [16] and follow-up works (21 19 18, that use the GPT-2 (22) architecture (ef. details in Appendix A.2h. For optimizing @ in the pretraining stage, we use the following minimization objective
(()) A E Hruis
I
LL (H.) - 1)+1)²
(1)
where HT.. is generated by the prompt distribution described in $ 2 scenarios, which roughly 2 (2M + 1) as in the previous works 16 and train TFg separately for each scenario.
We set T Timin 50 for all
2.3 Principled Baselines
To benchmark ICL, we derive principled baselines that learn from demonstrations D, and produce a prediction function [o(; D,), where b is the identifier of a particular baseline. We denote fx( e
fo(:; D2) and ficL (X++1) & (H;) whenever there is no ambiguity.
The optimal baseline is Bayesian model averaging (BMA), which makes prediction by aggregating models from different hypothesis classes. Formally, it defined as
fémA (x) Eme[Mpp((F. D.) T.),,,, (D.))
(2)
where p(Fm D:) is the posterior probability of Fm and im is the ridge regression estimator for Fi. defined as wm (D.) = TT. D m.1 +
a2 I2m+1)- TT Yr with $m,, E x2m whose k-th row is T (Xx) and Y. = (Yi..... YY) E RR. The Bayes optimal estimator defined in 2 minimizes the expected risk with respect to the true hierarchical Lata-generating distribution. This distinct from the notion of optimality in Raventós et al. [6), which defined with respect to an empirical pretraining distribution with finite number of samples, such that deviating from it can lead to better generalization. The optimality of BMA follows from two standard results that
(I) B|Y++1 Hy] is solution to minser Ey,,, [X(4( D,), Y++1) He] almost everywhere for
3

---

all E N and F being the set of all functions from He to R (e.g., Lemma in (23 and (2)
| = Em P( pf, H,] f (z) (e.g.. Equation 3.58 in 224
We also consider family of principled baselines that embodies different model selection strategies with the same model fitting capacity as the optimal predictor. Such baselines make predictions by
ft;;;; D:)
ú 3
(Dr) mi! (2),
where m E arg max [ (Scores(m)) with Scores(-) being some model selection criterion
2.4 Measures for Benchmarking Optimality of ICL
Inspired by seminal work 20 that benchmarks (deterministic) optimization software, we first define the base metric measuring the optimality of learning algorithm in s E S. Then, we present the performance measures summarizing the base metric across S. In the following, we let B contain all baseline learning algorithms and ICL. We set the test prompt length as 2 100, which is within the length generalization regime observed in practice 255
Base metric. Our base metric is the performance ratio, which normalizes the sample complexity of learning algorithm by that of the best algorithm among all baselines.
Definition 2.1. For a learning algorithm E B at a scenario E S, the performance ratio of a requirement r against B C B is defined as (r; = /r(r)/ mingggB(N (r)), where (r(r) A min (t ((I(((( (2,,) Y21) r is the sample complexity of achieving the performance
The performance ratio quantifies how many more demonstrations is required by learning algorithm to achieve certain performance level compared to the best learner among B. Therefore, when BMA 13, algorithms with (r; B) = have optimal efficiency at due to the optimality of BMA.
Performance measures. Based on the performance ratio across different scenarios, our goal is to report "single" score that summarizes how optimal ICL is across S. However, naively summarizing the performance ratio for requirement r is inappropriate because the difficulty achieving varies across learning problems, making comparisons inconsistent. Therefore, we define the reference performance quantile 1/ (x)
(8) as the Q-th quantile of reference performances at for Q E (0, 1). Here, we measure the performance quantile in reverse order, for making higher performance quantile analogous to higher performance. The reference performances at is defined as a set of performances achieved by reference models Bref B: that is, (X Y 12 Brel, E [T]]
With this idea, the performance ratios across is summarized by the mean performance ratio and the performance profile, which are defined follows.
Definition 2.2. For the performance quantile e the mean performance ratio of b E B against B C B is defined as MPR(b; th Bui B3 A [S| ESEs R (s)
Definition 2.3. For the performance quantile Hes the performance profile of b E B against B C B at a ratio 7 2 is defined as
Pb pb(T; w gr B)
1s 1( e S (s): < T-)--
The two measures capture complementary aspects optimality of ICL. Specifically, the mean performance quantile quantifies the average inefficiency of a learning algorithm b in attaining certain performance, which assumed to be achievable by b. In contrast, the performance profile measures the frequency with which model can achieve the performance quantile given tolerance for inefficiency. These intuitive measures provide novel insights into optimality of ICL that are not apparent in previous error rates- based comparisons and asymptotic analyses.
Benchmarking ICL Efficiency
measure to what extents transformers efficiently learn a new task through ICL. compared to the optimal learning algorithm (33.1) and principled baselines (..2

---

3.1 Can Transformer Optimally Learn In Context?
We first examine the efficiency of ICL compared to the Bayes optimal predictor, which learns new concepts with optimal efficiency. For comprehensive evaluation, we design the test scenarios with various levels of SNRs: S {[[M), o. o M 10, og E {0.003, 0.03, 0.3), o E {0.1, 1, 10} (cf. 22.1). Also, to minimize the impacts of stochasticity the sampling process of Hr, we evaluate performances for each scenario 512 times. Then, we analyze the mean performance ratio of ICL against BMA for all quantiles performances achieved by ICL; that is, we measure MPR(ICL; 2 B1) with B A {ICL} and B1 - {ICL, for Q E {0.01, 0.1, ... 0.9, 0.99). Bye
In this way, we measure the efficiency of ICL in achieving each performance level under various difficulties in extracting information from prompts. In the following, we regard prompts with more than 40 demonstrations as the many-shot regime where the average performance quantile is approximately 0.5 (cf. Figure A3 in Appendix).
Figure D reveals striking dichotomy in optimality of ICI.
Near optimal few-show efficiency. For low performance quantiles (Q S 0.3), ICL demonstrates its remarkable near optimal efficiency. Specifically, the mean performance ratio is at most 1.1, which means that it requires only 10% more demonstrations to achieve the performance lower than a (8) for s E S. Considering the average sample complexity for the performance quantile of 0.3 is 19, this explains ICL's impressive few-shot performance observed in practice (e.g., demonstration sizes of 5 and 15 in Brown et al. [I].
average than the optimal learning algorithm
Suboptimal many-shot efficiency. Starting from Q 0.3 or more apparently from Q = 0.7 onward, the performance ratio grows almost monotonically with Q, increasing from around 1.1 at Q 0.3
1.8
to around 1.2 at Q 0.7 and to around 1.45
MPR
Q = 0.99. This implies that efficiency of ICL as learning algorithm deteriorates when pursuing high performance requirements, which inherently requires larger demonstration sizes. Therefore, when scaling ICL from few-shot to many-shot regimes, expect- ing similar level of optimality relative to the opti- mal learner would be an overestimation. Importantly, the diminishing efficiency ICL would be difficult to uncover without our novel evaluation framework since ICL performances generally improve with more demonstrations (66 and its learning curve resembles that of the Bayes optimal estimator [66
0.0
0.2
0.4
0.8
1.0
Performance Quantile
Figure 1: Mean performance ratio of ICL against BMA across different performance requirements. The shaded areas represent the standard deviation of the corresponding per- formance ratio.
3.2 Benchmarking ICL Against Principled Baselines
We have shown that ICL is significantly inefficient compared to BMA in high performance regimes. While BMA is learnable by minimizing (T). it might seem unrealistic for ICL to compete with BMA that performs the expensive model averaging operation. Thus, we compare ICL with more practical baselines with a computational constraint that select a single model using principled criteria (cf. (3): Akaike Information Criterion (AIC) (27 as minimax-rate optimal model selection mechanism, Bayesian Information Criterion (BIC) (28) as consistent model selection mechanism, and Bayesian Model Comparison (BMC) as efficient BMA alternative selecting maximum posteriori model class. These baselines represent the spectrum of principled model selection methods, which often asymptotically converge to either AIC or BIC 29
To quantify relative efficiency, we use performance profiles 1/ B2) with Bes (ICL, AIC]
and B3, fICL, AIC, BIC, BMC). This allows measure the probability that each method achieves a reference performance level within given sample complexity budgets, which evaluates both efficiency and effectiveness (i.e., maximum achievable performances) of learning algorithms.
Superiority of ICI. in few-shot regimes. Perhaps not surprisingly (given the results from comparison with BMA), ICL dominates the baselines with restricted capacity under low performance requirements.
5

---

ICL
BMC
AIC
BIC
0=
Q=0.4
Q - 0.8
Performance profile
1.0
0.5
0.0
1
2
3
2
2
3
Figure 2: Performance profiles Ps across different performance ratios under different target performance quantiles Q. Each curve represents the probability that a method achieves the desired performance within factor т of the best method's sample complexity (x-axes). Figure A4 in Appendix illustrates results for all Ω.
Specifically, achieves the perfect performance profile at r = for @ < 0.3. This means that it optimally attains the performance requirement in all scenarios when Q S 0.3. Given that each baseline has its own strength in certain scenarios, this guarantee is quite strong and not observed in other baselines. Further, for Q 0.4, ICL reaches perfect performance profile within S 1.2. This means that ICI. attains the required performance 20% demonstrations baselines selecting single model struggle in the low-performance regime due to high uncertainty under small number of demonstrations preventing them from selecting the proper model class The results highlight significance of ICL as a learning algorithm in few-shot regimes.
Q = 0.4 in all scenarios by using at most average compared to the best method in each scenario. Conversely, all
30
31.
Inferiority of ICL in many-shot regimes. Figure illustrates diminishing efficiency of ICL in long context regimes. Specifically, as the performance requirement increases, the initial performance profile at is reduced, indicating the reduced probability that ICL learns the most efficiently among Bg. Beside, the computational budget required reach perfect performance profile increases as the performance requirement increases. Eventually for 2 0.8, even at r = 3, ICL achieves the performance profile around 0.8, which means that ICL cannot reach the performance requirements for 20% of cases by using even 3 times more demonstrations than other models.
Crucially, this increasingly suboptimal behavior is opposite the behaviors of principled baselines. In Figure 2 as opposed to ICL, the principled learning algorithms significantly reduce the time to reach the (near) perfect performance profiles as Q increases. Eventually, despite their significant deficiencies few-shot regimes, such baselines become more effective (achieving higher performance profiles 3) and more efficient (sharply improving the performance profiles with respect r) than ICL in many-shot regimes. Therefore, some characteristics enabling learning algorithms to leverage large number of demonstrations might be missing in the ICL mechanism.
To gain further insights, we qualitatively analyze MSEs across different numbers of demonstrations for each scenario. As trivial baseline, we also consider an ensemble that aggregates the ridge estimators (wm ) me[MM] using equal weights. Figure Balshows that while all methods show decreasing MSEs with more demonstrations, ICL. exhibits persistent discrepancies from the principled learning algorithms in many-shot regimes. Further, in Figure 3bb we analyze the squared prediction difference between each model and the Bayes optimal predictor for each scenario. Critically, it reveals that while consistent estimators (BMC, BIC) seem to converge in L to BMA (albeit at different rates), ICL's L distance to BMA plateaus after receiving few demonstrations. This behavior mirrors the trivial ensemble, which does not update its hypothesis about the model class with demonstrations. This suggests another fundamental limitation: ICL may lack asymptotic efficiency and consistency (cf. Ding et al. (29 for formal definitions). These findings challenge the prevailing optimism about ICL's potential universal learning algorithm.
3.3
On Sources of the Diminishing Efficiency
We observe ICL falls short under high performance requirements, which typically requires longer context sizes than the pretraining prompt (cf. Figure A3 in Appendix). Given universally observed deficiencies of machine learning models the distr--ditti (OOD) regimes tempting to attribute the diminishing efficiency to the deficiencies in OOD regimes.
33

---

ICL
BMA
Ensemble
AIC
0
1.0, of 0.03
10.0, 0z 0.03
0.03
109
MSE
10-1
10"1
25
50
75
100
25
50
75
100
In-Context Examples
Ensemble
0 = 1.0, 0g 0.03
10.0, ai = 0.03
10 -1
SP3
10-2
10-3
4
10-4
100
50
In-Context Examples
(b)
Figure Mean squared errors for different demonstration sizes. (b) Squared prediction differences between BMA and other methods for different demonstration sizes. Figure A5 And in Appendix illustrates results for all E S and b E B.
We take a closer look at this in Figure 3b which corresponds to the reducible error due to the bias-variance decomposition. Recalling that Tirain = 50 was used for pretraining, Figure 3b and Figure in Appendix show apparent differences in the achievable error between in-distribution
and OOD regimes, except in low SNR scenarios (a o²) (0.1, 0.03) and (o o²) (1, 0.3).
This finding aligns with the length generalization literature, which suggests that transformers often generalize to contexts up to 2.5 times longer than those seen during pretraining (25). Further, given that the average performance quantile Tuain is 0.6, Figure [ that fundamental inefficiency already emerges in the in-distribution regime. Therefore, the diminishing efficiency observed in $3.1 and 3.2 cannot be fully attributed to the transformers' OOD generalization capability.
4 Analyzing Suboptimality of ICL
In this section, we explain why ICL's efficiency as learning algorithm diminishes in long context by using information-theowe
tools. This theoretical grounding is crucial, as it shows that the diminishing efficiency ICL in long context observed in $3 is an inherent property of the ICL mechanism itself, rather than an artifact of specific experimental setup. Accurately identifying the source of this limitation is essential for guiding future efforts to mitigate
4.1 ICL Error Decomposition
Adopting a Bayesian viewpoint [9]. we denote the oracle distribution with drawn from environ- ment E by P!(.) A IP(Y++1 -HHH, e) P((Y+1 E .|X++1,ee) e) (e.g., & characterizes the sampling process 22.1 with E (m, W).)) Similarly, we let TFp models the conditional distribution of outputs, i.e., TF,(H, A P& E H₂) å P.).). All subsequent discussions in this section assumes distribution shift; that is, E is the environment under which TFg was pretrained. We assume that Y4 either discrete or continuous.
With this notation, the ICL performance with demonstrations from E is defined as E loglog 1)++))) = E [- log (Y++))) E. [DxI(PP Pf) 19. Here, the first term is the (irreducible) alcatoric uncertainty and constant with respect to in setting. The second term can

---

be further decomposed as
dP
E [KI.
| Pb)] = E
log
aPg (v) (d
= E DKL. P PS)
+ E log (Ye1))
(4)
Tot (Ve-
Bege (Bayes risk)
S (Excess risk)
where the second equality comes from the law of total expectation and (Y2+1) - PP E (1),, is the posterior over Y++1 given H5.
In 4 the Bayes risk cayes measures how well the Bayes-optimes predictor performs under uncertainty on E, is non-negative and decreases monotonically with more demonstrations; that is, Bayes S Bayes
4+1
for all / E N 34]. Demonstration size I required to bring this risk below a threshold is captured by NEMA (g) - minges( Bayes S q)- Here, q represents the absolute value of the performance requirement (e.g., MSE), whereas Q in denotes the performance quantile.
The excess risk exs measures the performance of the transformer relative the Bayes optimal predictor. Due to the non-negativity of excess risk and independence between TFg and EBayes this term determines when ICL emerges and how well it can perform. For instance, if TF, achieves an excess risk curve such that <xs &ss S Bayes Rayes. non-trivial ICL performance emerges, improving upon the zero-shot performance with demonstrations. Further, if <xs 0 as + oo, then ICL is Bayes-risk consistent and asymptotically matches BMA. In $4.2 we dissect the excess risk exs based on empirical observations in
4.2 On Excess Risk
Interpreting the transformer's prediction in the meta ICL setup as the Gaussian distribution (e.g.. by adding a small random Gaussian noise to the prediction), the squared prediction difference in Figure ib directly proportional to the excess risk, up constant scale and shift. The same applies to each baseline's s squared prediction difference, interpreted as its own excess risks.
In this regard, Figure 3b illustrates that the transformer's 's excess risk remains roughly bounded within a modest interval in a certain length generalization regime (e.g.. S 2T suggesting that would perform ICL non-trivially due to the monotonicity of Gayes However, once the context length becomes much longer than the one seen during pretraining (e.g.. t > urain in Figure 4), the excess risk deteriorates sharply. This explains why ICL is not consistent learner, being dominated by the principled learning algorithms in large demonstration regimes, as we observed in $3.2 We formally encode the above empirical observations about the non-vanishing excess risk curve follows.
Assumption 4.1. For an environment and a transformer TFa, there exist constants (7, Axs) E (N, IR+,) such that 0 < Axs V eks for all t' 2 L.
The assumption states that, after some reference point I, the excess risks of TFg can be lower bounded, aligning with the behaviors illustrated in Figures 3b and 4 well with empirical evidence demonstrating the deficiencies of state-of-the-art LLMs outside the length generalization regime 33 25 In other words, assumes that TF2 does not magically reduce its excess risk in the OOD context length regimes. This may occur for various reasons such as insufficient pretraining data or intrinsic properties of architectures. Importantly, we do not assume conditions on the cause, only that the lower bound exists. We also emphasize that, while the excess risk is lower bounded under Assumption 4.1 ICL performance can still improve with more demonstrations due to the monotonicity of the Bayes risk (cf. $4.1 as observed in many-shot ICL literature [36] 26).
Crucially, as we show in $4.3 Axs controls lower bound of ICL's suboptimal efficiency in learning from demonstrations. For a transformer with a strong length generalization ability, &K's in the assumption can also be upper bounded, making the subsequent suboptimality analysis nearly tight. In this regard, our analysis encompasses plausible (near) future advances in length generalization capability. Therefore, our analysis under Assumption 4.1 is a general result highlighting the ICL mechanism's intrinsic flaws, isolating them from the transformer's length generalization capability.
4.3 Analyzing Suboptimality of ICL
Next, we explain the critical suboptimality of ICL observed in 3 where ICI. initially matches the efficiency of the optimal learning algorithm but starts to significantly deteriorate in maoy-shot regimes.
8

---

To this end, we define suboptimality of ICL at performance requirement as the additional number of demonstrations required for ICL to achieve requirement q compared to the Bayes optimal estimator, denoted as (g) _ min, (t NeMA (q) Baaes + c S q). Here, we define suboptimality at with respect to the reducible part of the ICL performance (i.e., E [DKI. Pt)) ). which is
equivalent to defining it with respect the ICI. performance up constant scaling in q-
The following theorem constructs a lower bound of SubOpt(q) under Assumption 4.1 where denotes the mutual information.
Theorem 4.2. Let us assume (7, Axs) satisfies Assumption 4.1 For a sufficiently small q such that (g 2 E, it holds that
SubOpt(1) 2 (.B(g)= A min {
T(YNeMAA Det1 HNomala)-)) > Axs]
(5)
IEN
where De+1 is a sample from the same distribution as Dz+1.
Theorem 4.2 intuitively characterizes suboptimality (cf. Figure A2 in Appendix for an illustration
of the concept). Specifically, suppose the Bayes optimal learner requires (eMA(q) demonstrations to achieve the performance q. Then, SubOpt(q) represents the additional demonstrations required for ICL to compensate for the excess risk <ss- Here, the compensation represents how much the
new demonstrations Dr+1 1 reduce the uncertainty about Y (eww(()) given prompt H Hausla) which corresponds to the conditional mutual information in (5). The theorem is proven in $B.1
Characterizing suboptimality with N()nn Di+1| (Nmm(()--) provides clear insights into ICL's suboptimality. Specifically, transformers with small excess risks in the non-vanishing regime are less subject to suboptimality. Besides, since higher performance requirement (i.e., a smaller q) increases NEMA(q). (q). suboptimality naturally increases due to reduced conditional mutual information. The following theorem, which is proven in $B.2 makes this intuition precise by establishing necessary conditions for SubOpt(( (q) being constant with respect to
Theorem 4.3. Let us assume (f, Axs) satisfies Assumption 4.1 and let q be such that NEMA (g) 2 T. If LB(()) = LB(4) for all Axs < of < q. then either of the following condition holds:
1. Negligible excess risk: Axs I(Y; D.lH(-1) for all t NNMA(1), and LB(g) = 0,
2. Negligible diminishing returns: I(Y;; 1 < (1+ LBW) I(Y: ( for all
t 2 NBMx(q), where i. - Nema(1) + L.((g) and I.B(q) > 0.
Non-deteriorating suboptimality has stringent necessary conditions that rarely hold in practice. Specifically, the negligible excess risk condition requires that the information gain from a single demonstration, regardless of demonstration size, dominates the excess risk. While this may hold for few-shot regimes (explaining the significant efficiency of few-shot ICL), ensuring this assumption across all prompt lengths is quite strong given the diminishing nature of [[Y;; D, H:-1) with in most learning scenarios [37 38 For a similar reason, the negligible diminishing returns condition, which requires constant lower bound of T(;;; D3 H,-1) for all demonstration sizes i, is quite strong. Therefore, SubDpt(g) (g) inevitably grows as g decreases, leading to increasing suboptimality of ICL under high performance requirement as observed in $3
As concrete intuition on suboptimality, we consider the following crude approximations: (A1) 'Bayes CI/V for some constant factor C1 and (A2) exs é for all 1 E N,. Here, (A1) corresponds to sublinear convergence of the Bayes posterior estimator, which holds in many cases 33 38, and (A2) corresponds to Assumption 4.1 with Axs) = (0, exs). Replacing (A1) with other common bounds, such as Bayes & Ci/t or EBaye R C1 exp(-t), yields similar results.
Under (A1) and (A2), for performance achievable by the transformer (i.e., g exs), simple calculation gives SubOpt(g) 2 C²
Cievs - Here, the rapid growth of SubOpt(g) as q decreases highlights the inefficiency of ICL in achieving high performance requirement. Moreover, another way of improving suboptimality by reducing exs, from the perspective of the rough power law estimations from the scaling laws (39], would require an exponential increase in pretraining data size or computational resources. Thus, either way, a transformer exhibits significant suboptimality in achieving high performance through ICL compared to principled learning algorithms.
>

---

100
Excess nikk
Excess risk
10-1
10-
Model size
0.5x
Tew
Model size 1x
Tre 100
Model 2x
Trae 200
Model
200
400
600
800
1000
200
400
600
800
1000
ICL Examples
Examples
Figure 4: Impacts of the pretraining prompt length (left) and the model size (right) on the excess risk curve in 8 = ([M), oy o ([10), 0.03, 10).
4.4 Impacts of Scaling Computations
Given that the non-vanishing excess risk curve causes an inherent inefficiency of ICL in long context, explore whether improving transformers' capacity of handling longer context enable the excess risk to decrease with more demonstrations, thus overcoming this fundamental limitation. As illustrated in Figure 4 while larger models and longer pretraining prompt lengths reduce the magnitude of the excess risk many-shot regimes, the non-vanishing shape in long contexts persists. Thus, simply scaling model size pretraining prompt length does not fundamentally resolve the inefficiency of ICL in long contexts. See Appendix A.3 for experimental settings and detailed results.
Related Work
Asymptotic Behavior Analysis. Xie al. ( show that ICL predictions converge posterior probabilities in asymptotic demonstration size regimes. Subsequent works expand these results to
encompass finite-sample guarantees [12 I0 I1I broader prompt distribution structures [12 40 10),
and structural characteristics of transformers [1O]. Recent studies analyze the average cumulative regret across demonstrations [TO g. treating ICL as online learning algorithm. However, practical applications prioritize test sample performance over demonstration set performance. In this work, we directly analyze suboptimality of ICL in achieving specific performance requirement through the excess sample complexity compared to the Bayes optimal learning algorithm.
Stylized ICL Benchmarks. With the meta ICL framework (ef. 2.1.p Garg al. [66] demonstrate that transformers are capable of learning simple function classes (e.g.. linear models and random curves qualitatively similar to those of optimal learning algorithms under asymptotic pretraining sample conditions. Subsequent works extend the results to finite pretraining sample scenarios [6] and mixture function classes 41 21. Further, new analytical frameworks that directly analyze ICL predictions reveal that ICL exhibits behavior similar gradient descent [19 18] More recently, these stylized settings have been used to probe other sophisticated behaviors of ICL. This includes analyzing transformers' in-context model selection their ability to infer causal structures (44), and the implicit connection between ICL and low-rank updates to MLP layers [45). Although stylized ICL benchmarks have been extensively studied, the optimality of ICL as learning algorithm remains unexplored. By comparing the sample complexity of ICL with that of principled learning algorithms, we uncover novel insight on the fundamental inefficiency of ICL in the many-sho learning regime. This critical insight suggests more nuanced view of ICL as purported universal problem solver.
neural networks) from demonstrations, achieving
and preference for simpler hypotheses 42 43),
6 Conclusion
The surprisingly strong ICL performance of LLMs suggest its potential to eliminate the need for task-specifee models. To rigorously examine this potential, we developed novel framework for benchmarking optimality of ICL learning algorithm against principled learning algorithms. We found that while few-shot ICL's efficiency is comparable to the Bayes optimal learning algorithm, its efficiency quickly diminishes with more demonstrations. Through information-theorett analyses, we showed that ICL mechanism is intrinsically inefficient in many-shot regimes. This highlights the need for a new adaptation method that can reduce excess risk with more demonstrations, enabling sample-efficient learning of novel tasks while preserving the update- feeete--rre nature of ICL.
10

---

Acknowledgments and Disclosure of Funding
We would like thank Jihyeon Hyeong. Yuchen Lou, and anonymous reviewers for their valuable feedback during the preparation of this manuscript. We declare that there was no funding received for this work.
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
[2] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit Bayesian inference. In International Conference on Learning Representations, 2022.
[3] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):
1-113, 2023.
[4] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 139771, 2023.
[5] Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv preprini 23030036., 2023.
[6] Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression. In Advances in Neural Information Processing Systems, 2023.
[7] Aarohi Srivastava, Abhinav Rastogi. Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.0444 2,0.01615, 2022.
[8] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07653 2022.
[9] Hong Jun Jeon, Jason D Lee, Qi Lei, and Benjamin Van Roy. An information-theoretic analysis of in-context learning. In International Conference Machine Learning. 2024.
[10] Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang. and Zhaoran Wang. What and how does in-context learning learn? Bayesian model averaging, parameterization, and generalization. arXiv preprint arXiv:2305.I 19420, 2023.
[11] Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers statisticians: Provable in-context learning with in-context algorithm selection. In Advances in Neural Information Processing Systems, 2023.
[12] Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Trans- formers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning, 2023.
[13] John Langford and Rich Caruana. (not) bounding the true emror. In Advances in Neural Information Processing Systems, 2001.
[14] Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv: 2017.

---

[15] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 1, learning hierarchical language structures. arXiv preprints, abs/2305. 16733, 2023.
[16] Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. In Advances in Neural Information Processing Systems, 2022.
[17] Kwangjun Ahn, Sébastien Bubeck, Sinho Chewi, Yin Tat Lee, Felipe Suarez, and Yi Zhang. Learning threshold neurons via edge of stability. Advances in Neural Information Processing Systems, 2023.
[18] Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211. 15661, 2022.
[19] Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, 2023.
[20] Elizabeth Dolan and Jorge J Moré. Benchmarking optimization software with performance profiles. Mathematical Programming, 91:201-213, 2002.
[21] Madhur Panwar, Kabir Ahuja, and Navin Goyal. In-context learning through the Bayesian prism. In International Conference on Learning Representations, 2024.
[22] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. https://cdn openai. com/ haaaar--aangaagmmmdese/ssseee
larpsiaaeeemeeeleeeeeee models_are sraddes__areuuuueervveeeeee multitask.
learners. pdf, 2019. [Online; accessed 24 24-November-2024).
[23] Kartik Ahuja and David Lopez-Paz. A closer look at in-context learning under distribution shifts. arXiv preprint arXiv:2305. 16704, 2023.
[24] Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2007.
[25] Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal, and Denny Zhou. Transformers can achieve length generalization but not robustly. arXiv preprint arXiv:2402.09337,, 2024.
[26) Rishabh Agarwal, Avi Singh, Lei Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. Many-shot in-context learning. In Neural Information Processing Systems, 2024.
[27] Hirotugu Akaike, A new look at the statistical model identification. IEEE Trunsactions on Automatic Control, 19(6):71--722 1974.
[28] Gideon Schwarz. Estimating the dimension of a model. The Annals of Statistics, pages 461-466 1978.
[29] Jie Ding, Vahid Tarokh, and Yuhong Yang. Model selection techniques: An overview. IEEE Signal Processing Magazine, 35(6):16-34, 2018.
[30] Jennifer A. Hoeting, David Madigan, Adrian E. Raftery, and Chris T. Volinsky, Bayesian model averaging: A tutorial. Statistical Science, 14(4):382-401, 1999.
[31] Larry Wasserman. Bayesian model selection and model averaging. Journal of Mathematical Psychology, 44(1);22-117, 2000.
[32] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: /903. /2261, 2019.
[33] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: benchmark of in-the-will wild distribution shifts. International Conference on Machine Learning, 2021.
12

---

[34] Hong Jun Jeon, Yifan Zhu, and Benjamin Van Roy. An information- theorretc framework for supervised learning. arXiv preprint arXiv:2203.00224 00246, 2022.
[35] Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowyez, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length gener- alization in large language models. In Advances Neural Information Processing Systems, 2022.
[36] Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig. In-context learning with long-context models: An in-depth exploration. arXiv preprint arXiv:2405.00200 2024.
[37] Jorma Rissanen. Universal coding, information, prediction, and estimation. IEEE Transactions on Information Theory, 30(4):62--636, 1984.
[38] Bertrand S Clarke and Andrew R Barron. Information-theoreet asymptotics of Bayes methods. IEEE Transactions on Information Theory, 36(3):453-471, 1990.
[39] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint #xiv:2001.0836/ 2020.
[40] Yuchen Li, Yuanzhi Li, and Andrej Risteski. How do transformers learn topic structure: Towards mechanistic understanding. In International Conference Machine Learning. 2023.
[41] Reese Pathak, Rajat Sen, Weihao Kong, and Abhimanyu Das. Transformers can optimally learn regression mixture models. In International Conference on Learning Representations, 2023.
[42] Puneesh Deora, Bhavya Vasudeva, Tina Behnia, and Christos Thrampoulidis. In-context occam's razor: How transformers prefer simpler hypotheses on the fly. arXiv preprint M.Xiv::2506193555 2025.
[43] Eric Elmoznino, Tom Marty, Tejas Kasetty, Leo Gagnon, Sarthak Mittal, Mahan Fathi, Dhanya Sridhar, and Guillaume Lajoie. In-context learning and occam's razor. In International Conference on Machine Learning, 2025.
[44] Francesco D' Angelo, Francesco Croce, and Nicolas Flammarion. Selective induction heads: How transformers select causal structures in context. In International Conference on Learning Representations, 2025.
[45] Benoit Dherin, Michael Munn, Hanna Mazzawi, Michael Wunder, and Javier Gonzalvo. Learning without training: The implicit dynamics of in-context learning. arXiv preprint arXiv:2507.16003, 2025.
[46] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.00 08774, 2023.
[47] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.0620 2025.
[48] Kwangjun Ahn, Xiang Cheng, Minhak Song, Chulhee Yun, Ali Jadbabaie, and Suvrit Sra. Linear attention is (maybe) all you need (to understand transformer optimization). In International Conference on Learning Representations, 2024.
[49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems, 2017.
[50] Richard Turner. An introduction to transformers. arXiv preprint arXiv:2304. 10557, 2023.
[51] Christopher M Bishop and Hugh Bishop. Transformers. In Deep Learning: Foundations and Concepts, pages 357-406. Springer, 2023.
13

---

[52] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations. 2015.
[53]
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In International Conference Machine Learning, 2009.
[54] Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact positional encoding on length generalization in transformers. In Advances in Neural Information Processing Systems, 2023.
[55] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, In Conference on Language Modeling, 2024.
[56] S Hochreiter and Schmidhuber. Long short-term memory. Neural Computation, 978):1735- 1780, 1997.
[57] Assaf Ben-Kish, Itamar Zimerman, Shady Abu-Hussein, Nadav Cohen, Amir Globerson, Lior Wolf, and Raja Giryes. Decimamba: Exploring the length extrapolation potential of mamba. In International Conference on Learning Representations, 2025.
14

---

A Additional Details
A.1 On Usage of Stylized Setting
Comprehensive analyses with statistical significance. The benchmark in the stylized settings in principle enables comprehensive comparisons across different environments (e.g., S) and architectures (e.g., different TF&), achieving arbitrarily high levels of statistical significance. In empirical studies, these factors are constrained to the configurations of the datasets or the computational budgets.
Comparison with the optimal method. The stylized setting enables comparison with principled learning algorithms. Specifically, BMA considered in 2 provides the minimum achievable perfor- mance of any learning algorithms at all prompt lengths. This strong guarantee is typically not possible in empirical studies, as even human performances could not be an oracle simply not possible to attain with only the data provided to the transformer. Also, the theoretical studies themselves do not allow for precise performance comparison, except analyzing the general asymptotic behavior that is shared among reasonable learning algorithms.
From stylized settings to practical LLMs. Although we study stylized settings in a rigorous manner, it does not capture all aspects of LLMs. For example, the ICL objective in (T) is not an autoregressive loss used for pretraining LLMs, omitting the losses of predictions at each Y,. Further, the model size and training data diversity used in our meta ICL setup are significantly smaller and less diverse than those used in modern LLMs, such GPT-4 466 and Gemini 2.5 477, which typically possess hundreds of billions or even trillions of parameters trained on vast, heterogeneous datasets. Therefore, one potential concern is the generalization of results obtained in stylized settings. While it cannot be shown precisely, the findings from such stylized settings have been surprisingly well generalized to the real-world tasks 48 40 For instance, Ahn al. [48] perform synthetic experiments even with simplified transformers to study optimization methods for LLMs that surprisingly well reproduce the results from the real-world natural language data.
Given the significance of actionable insights from the stylized settings such as foretelling impacts of scaling ICL to the asymptotic region of the demonstration size, which is extremely challenging with real-world LLMs, we hold positive views on the role of stylized settings in LLM research whose significant advantages outweigh the potential concerns its generalization to the LLMs in practice.
A.2 Detailed Configurations
Model. For the model, we use the GPT-2 [22) architecture for TFg, which is standard architecture in the meta ICL and other stylized experimental settings; that is, we define TF, as a decoder-only transformers 44 with 12 layers, 8 attention heads, and 256-dimensional embedding space. For readers unfamiliar with transformers, we refer to the excellent tutorials [50] [1]] We remark that viewing TF, as a function from a sequence of vectors with an arbitrary length a vector with the same dimension does not significantly impact the understanding of core findings in this paper.
Optimization. For minimizing the ICI. objective I(0), compute the stochastic gradient with 64 prompts and update 0 by using the Adam optimizer [52) with fixed learning rate of 10-1 for one million training iterations. Also, in order to boost the convergence speed, we use curriculum learning 53 as recommended in [66] 21 by increasing the length of the prompt by 2 every 2,000 training iterations until it reaches (2M + 1) (and the order of Fourier series by until it reaches M).
Computational resources for experiments. In this work, we use multiple servers which consist of multiple GPUs including RTX 8000 (50GB) and A100 (40GB).
A.3 Impacts of Scaling Computations
We show that the non-vanishing excess risk curve of the transformer in long context causes the efficiency of learning to diminish with more demonstrations. Therefore, a natural question is whether enhancing transformers' capacity of handling longer context can make the excess risk decrease with more demonstrations and thus resolve the fundamental inefficiency. We analyze the impacts of scaling the pretraining context lengths (by setting Tiain to 100 and 200) and the model sizes (by scaling the number of layers, the number of heads, the embedding dimension by factors of 0.5, 2, and 3) on the excess risk. The pretraining losses are 1 1.06, 0.58, and 0.34 for models trained with Tiran 50, Train 100, and Train = 200, respectively. Also, the pretraining losses are 1.36, 1.19, 0.99, and 0.90
15

---

Mamba
LSTM
10
Excess rik
Excess risk
Teae
Tim 200
Tew- 400
10-2
100
Toah 200
Tirain 400
200
400
6DD
800
1000
200
400
600
800
1000
ICL Examples
Examples
Figure A1: The excess risk curve in & (IM) oy og) = ((10,, 0.03, 10) under different pretraining prompt length for Mamba and LSTM.
for half-capacity, standard, double-capacity, and triple-capacity models respectively. Note that we did explore different positional encoding methods since already use no positional encoding scheme that is effective at length generalization 35 54], which is from an inductive bias for the sample order-stable learning algorithms.
Figure 4 (left) shows that increasing Tusin significantly reduces the excess risk values, especially for long-co long-context regimes as desired. However, overall shape of the excess risk curve remains non- vanishing in the long-context regime. We observe from Figure (right) similar effeets of increasing the model sizes. Interestingly, larger models do not increase the length generalization regime, which consistent with previous results
25
The results suggest that simply increasing computations with larger model and a longer pretraining prompt length does not fundamentally change the shape of the excess risk, even though their overall scales improve. Therefore, while the degree of suboptimality can be relaxed with reduced excess risk, the inefficiency in many-shot regimes persist.
A.4 Impacts of Different Architectures
We examine whether the non- non-vanishing excess risk curve in long context is unique to transformer architecture. To this end, we ran additional experiments with Mamba [55] and LSTM [56] architec- tures under the setup Mamba and LSTM remain nanishnig in the long-context regime, being consistent with recent work analyzing the length extrapolation limits of Mamba [57]. Crucially, this means that the the diminishing efficiency of ICL long context is a general property of the ICL paradigm sequence models having non-vanishing excess risk, not artifact of the transformer architecture.
in SA.3 Figure AT shows that the shapes of the excess risk curve under
B Proof of Claims
B.1 Proof of Theorem 4.2
Proof. We first characterize suboptimality by the Bayes risk as follows:
SubDpt(g)
min {t NBMA(g) cisyes Exs S 9}
(6)
1E2+
min {t chayes Sqq cxst NoMA(q).
(7)
Since q
Neuu(n) the monotonicity of elayes and the non-negativity of eks give Bayes
min {t) Rayes 5q cxs) re24
min (2Nau (g)
{t { Sq ckss
min
{ It
Cuyes
Nawu (g)
CxS
12 (g)
(8)
To prove the theorem, we note the following.
16

---

(N1), Bayes error reduction the conditional mutual information: The Bayes error can be
expressed as the reduction of (differential) entropy follows.
Bayes = IE DKL. P Pb)] h(Y++ H 1(Y++ (Hr,e), e), for continuous Y2+1
(9)
Bayes E [KI(FE e || | HH(Y+1 H,) (H e), for discrete Y6+1
(10)
where h the differential entropy and Hi is the Shannon entropy. Therefore, for any I and continuous Y(+1, we have
Bayes h(Yu+1l H.)) H (h(Yo+1 H.) n(Yo+1|Ho,)) (11)
Bayes
h(Y Xu+1, Dw.) h(Y++1l X D.)
(12)
= =((u++1: D - [ D.)
(13)
where D.- (X,,9, Vi.... _x, X.- V- is independently sampled from the same distribution as
De-u, the second equality comes from the conditional independence Yu+1 1 Dn|,1++1 for any n E 4. and the last equality comes from the chain rule. For the discrete Y's, the same process can be applied by replacing with IH.
(N2). Lower bound of the excess risk: Let q be such that NEMA(4) Therefore, by Assumption we have
N t2
N
NaMA( (q), Rayes
(aua(a) -1
oxs
S S
NEMA
Navxle)-1 -1 Cayes
Bayes > Axs
Baaes
(14)
By applying (NI) and (N2) 7 we get the desired result
Sub
min {e HBayes S
Exs] - NNMA((4)
=
t2 (o)
min 12 Namx(s)
{1
Meua(1) Baaes
Bayos
> Axs
NNMA((4) = min
{
(Bayes
NaHa(1) (g)-1
"Bayes
1+ 1 (1)
> Axs
BEN
min iEN
{1
((((Nonn())) Det1
H HMsun())-1) Axs } (15)
B.2 Proof of Theorem 4.3
Consider qı, @2 (Axs, q) such that gi < < < q and NEMA(qI) NeMa(42). The goal is to show necessary conditions for LB(gi) LB(g2).
Proof.
LB(g2) is impossible because I(Y (9)) De+1 NuNe T(Y ( H (1) for any E N. Specifically, we have
Note that LB(q1)
S
I(Y (g:) Deai Newa 11) I(Y (g2); Dea | Newua VEEN
(16)
which implies
t E N I(Y Moua ( Des H (1:) Axs s S I(Y (12)) De4 H Newa Axs
(17)
and in turn LB(gi)= 2 LB(g2).
Therefore, we next show the necessary condition for LB(g:) = LB(g2).
(NC 1). Negligible excess risk: Let suppose Axs I T(Y Neua(a)); F S I(NNeun()z) H chosen arbitrary, the first necessary condition is given by
In this case, LB(g1) = LB(g2) = as desired. Since 9ı and 92
Axs S I(Y;; (,,,,), t t2t.
(18)
(NC 2).
No diminishing returns: If (NC 1) does not hold, we have In this case, rule out the possibility
Axs >
[IY No D H. (o
17

---

I[Y ( D1 (o))( < Axs I I(Y (eNNn (12)3 H Namm (12) because this gives LB( = 0 and (g) > O, which contradicts (B(g1) = (g))).
Thus, we consider the case I(YYaun Nawn D1 HNww (o))) (()) - I(Y (1a) (amn( D Hama (va) In this case, (.((g)) [g((2) requires the following condition
< Axs-
I(Y Nam((a): (r. ( (amm(ra)) < I(Y Di DI.B(1)++ H Moun(11)-1))
(19)
where the condition comes from [(YNeua Des1 Nena(g:) S M((NNeu Newa (9x) Dett H Nena (9x) for any & N.
By the construction of q1 and 92. we get
[[(MMuu Maun()) DLB(1) Nemm())-)) < II( (own (ooo:))t(/Noon00)--011 Vk E N+. (20)
Due the chain rule of the mutual information, for any & N4. holds that
L(Y (())))(9)3 Djl Nawa (4)
I I(YNeualg)= Di H Newa()) 2 kI(Y Namu (a Di Namu No
==0
(21)
Similarly,
k
I(Y ((M Ditz HH (onn (4) 1+kk)
I(Y Nown (4) +x)++ D Nam (s) 1+++e)
1=0
s (1 + k) (Ysnl(M Maxn i De H Moun -14k)) (22)
Therefore, we get the second necessary condition as
U(Y;; s ( < (1+ (1+7)
1(Y;Di,1(l-1)). 22 (23)
where i -B((g) > 1 for q such that (gMA(()
C Additional Figures
0.16
BMA
CCL
0.14
0.12
0.10
Bas-
0.08
Aus
0.06
0.04
Ays
(fa LBlg)
40
70
ICL Examples
Figure A2: Graphical illustration of Theorem 4.2 when = 0.08 - o², where x²
- log Pt (Y+++)) is the irreducible aleatoric uncertainty. The solid orange and blue lines rep- resent MSËS of BMA and ICL, respectively. Here, the dashed orange line corresponds to the x² Bayes + Axs. which serves as a lower bound on MSEs of ICL. The shift by Axs induces suboptimality that requires least LB(g) additional number of demonstrations for ICL achieve the requirement compared to BMA.
18

---

100
In-context example
80
60
40
20
0.0
0.2
0.4
0.6
0.8
1.0
Performance quantile
Figure A3: The number of demonstrations (y-axis) required achieve each performance quantile
(x-axis). The shaded area represents the standard error. We note that performance quantile Q = 0.6 achieved by Train number of demonstrations on average.
ICL
BMC
AIC
BIC
Q=0.1
Q=0.2
Q=0.3
1.00
0.75
0.50
0.25
0.00
Q=0.4
Q=0.5
Q=0.6
Performance profile
1.00
0.75
0.50
0.25
0.00
Q=0.7
Q=0.8
Q=0.9
1.00
0.75
0.50
0.25
0.00
3
2
1
3
Figure A4: Performance profiles P across different performance ratios 7 under different target performance quantiles Q. Each curve represents the probability that a method achieves the desired performance within factor 7 of the best method's sample complexity (x-axes).
19

---

ICL
BMA
BMC
A/C
8IC
6 0.1, of 0.003
69 - 0.1, of 0.03
06
1.0, o; - 0.000
26 1.0, of -0.03
10-4
10-1
10
10"2
10a
MBE
50
100
0
50
100
100
100
00 1.0,
0.3
0
10.0, o7 0.003
02
10.0, of 0.03
0
10.0, 0.3
10a
102
10
10
10-1
10
50
100
100
In context examples
Figure A5: Mean squared errors for different demonstration sizes.
ICL
BMA
BMC
AIC
BIC
6 0.1. o7 0.003
6r - 0.1, oz 0.03
00
1.0. o7 0.003
o6 11.0, of -0.03
10
10
10-2
10
10-5
10-4
10
SPD
50
100
50
100
50
100
50
100
G
1.0, o7
0.3
06
10.0, of 0.003
06
10.0,
0.03
0
10.0,
0.3
10"1
20a
10a
10-2
10
10
10-6.
10-1
10
50
100
In context examples
Figure A6: Squared prediction differences between BMA other methods different demonstra- tion sizes.
20

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly state this work's contributions and scope in the abstract and introduction.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We make a separate section discussing the usage of stylized setting. Also, we formally state the assumption in the main body.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide the full set of assumptions and complete proofs.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the full details required to reproduce the empirical results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Our source code is available https://github. ctt.://ooooo/jjtopoo cttptthhub.ccmm///o51221. technical--deb-inngg in-
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed configurations for the experiments in Appendix A.2
A.2
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We include the standard error bars in Figure I and Figure A3 the results with small sample sizes. We omit the error bars for other figures since they are obtained from 512 number of experiments.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
21

---

Justification: We provide computational resources used in the research in Appendix A.2
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppp////errrrs..ccpppp////aa/a
Answer: [Yes]
Justification: We thoroughly read the NeurIPS Code of Ethics and have confirmed that our research does not violate any of them.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: In this work, we study optimality of in-context learning as learning algorithm against principled learning algorithms. Because our study focuses on theoretical aspects rather than practical applications, we do not foresee direct ethical concerns or societal impacts arising from our findings.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work poses no such risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite papers related to code and models used in our experiments.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs are not used for this work.
23


Reviews:
[{"review_id": "1EhHAQIx7T", "reviewer": "Reviewer_kMFD", "summary": "This paper investigates the efficiency of in-context learning (ICL) in Transformers compared to principled learning algorithms. This paper introduces a framework based on performance profiles and sample complexity to quantify the (sub-)optimality of ICL. This paper finds that while ICL is nearly as efficient as a Bayes optimal estimator in few-shot settings, its efficiency significantly degrades in many-shot scenarios that require a long context. Through an information-theoretic analysis, the paper argues that this diminishing efficiency is an intrinsic property of the ICL mechanism, caused by a non-vanishing excess risk. The work concludes that ICL carries a \"technical debt\" of sample inefficiency in high-performance regimes.", "questions": "The core of the theoretical argument rests on the non-vanishing excess risk ($epsilon_{XS}^t$). Your experiments (Fig 3b, Fig 4) convincingly show this behavior. Do you have a hypothesis for why the Transformer architecture, trained with this objective, leads to a non-vanishing excess risk? Is it related to the fixed-capacity nature of the attention mechanism or something else?\n\nYou conclude by motivating **a new generation of 'on-the-fly' adaptive methods without the diminishing efficiency.**  Could you elaborate on what such methods might look like?", "limitations": "Yes. The authors have discussed their limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\n\n1. The comparison of few-shot and many-shot settings for ICL problems is fundamental and impactful for better understanding of the transformers models.\n\n2. The paper presents a clear and compelling dichotomy: ICL is near-optimal in the few-shot regime but becomes substantially suboptimal in the many-shot regime. \n\n3. The paper supports its empirical findings with a solid information-theoretic analysis in Section 4. By decomposing the ICL error into Bayes risk and excess risk, and arguing that the excess risk for ICL is non-vanishing. \n\n## Weakness\n\n1. The primary limitation, which the authors acknowledge, is the use of a stylized setting (regression on a mixture of Fourier series). Although it's a clean setup, it raises questions about the *generalizability* of the findings to the complex, high-dimensional, and often discrete tasks that real-world LLMs handle.\n\n2. The theoretical results hinge on Assumption 4, which posits a non-vanishing lower bound on the excess risk. While this assumption is well-supported by the paper's own experiments, it is still an assumption about the behavior of transformer-based ICL. I would like to hear more discussion about this assumption.", "was_revised": true, "final_justification": "This is a good paper, I tend to accept it.", "rebuttal": "We are grateful to Reviewer kMFD for the encouraging and exceptionally clear feedback. We are pleased that the reviewer found the few-shot vs. many-shot dichotomy \"clear and compelling\" and our analysis \"solid.\" The questions raised are thoughtful and address the core foundations of our work as well as its broader implications.\n\n## **W1:** “The primary limitation, which the authors acknowledge, is the use of a stylized setting (regression on a mixture of Fourier series). Although it's a clean setup, it raises questions about the generalizability of the findings to the complex, high-dimensional, and often discrete tasks that real-world LLMs handle.”\n\nThis is a crucial point. While our experiments use a controlled setup, we argue that the phenomenon we identify—diminishing ICL efficiency with long context—is a fundamental issue likely to generalize to large-scale models. Our reasoning is twofold: \n- General Theoretical Foundation: Our theoretical analysis in Section 4 is formulated with a general setting. The core finding relies on Assumption 4.1, which is not tied to a specific model size or data distribution. As a result, the theoretical insights we offer remain broadly applicable, transcending the specifics of our experimental setup.\n- Alignment with Large-Scale Empirical Evidence: The practical consequence of our theoretical finding is that Assumption 4.1 is a sufficient condition for the diminishing efficiency in long context. Crucially, Assumption 4.1 generality aligns well with empirical observations from recent large-scale transformer studies. For instance, Anil et al. (2022) and Zhou et al. (2024) both report that even state-of-the-art LLMs struggle to extrapolate when test-time context length exceeds that seen during training. Given that the Bayes risk improves with more demonstrations, this means that the excess risk curve is deteriorating in the extrapolating regime. \n\nTherefore, while explicitly confirming the phenomenon in state-of-the-art LLMs remains a valuable future direction, the alignment of our results with recent large-scale studies strongly supports the practical relevance and generalizability of our findings.\n\nAnil et al. (2022). Exploring length generalization in large language models. In NeurIPS.\n\nZhou et al. (2024). Transformers can achieve length generalization but not robustly. arXiv.\n\n\n\n\n\n## **W2:** “The theoretical results hinge on Assumption 4, which posits a non-vanishing lower bound on the excess risk. While this assumption is well-supported by the paper's own experiments, it is still an assumption about the behavior of transformer-based ICL. I would like to hear more discussion about this assumption.”\n\nSee the response to Q1. \n\n\n## **Q1:** \"The core of the theoretical argument rests on the non-vanishing excess risk. Your experiments (Fig 3b, Fig 4) convincingly show this behavior. Do you have a hypothesis for why the Transformer architecture, trained with this objective, leads to a non-vanishing excess risk? Is it related to the fixed-capacity nature of the attention mechanism or something else?\"\n\n\nThis is an excellent question that prompted us to conduct new experiments to strengthen our claims. Our initial focus on the Transformer was due to its dominance in ICL research and state-of-the-art large language models. However, the core of our theoretical analysis is not specific to the attention mechanism but rather to the general properties of sequence models, as long as it satisfies the conditions of our analysis (e.g., Assumption 4.1).\n\n\nMotivated by the reviewer's suggestion, we ran additional experiments with LSTM and Mamba architectures. The results are highly informative: The Transformer consistently outperformed both LSTM and Mamba in overall ICL performance, aligning with recent comparative studies (Akyürek et al., 2024; Lee et al., 2024). Crucially, the central phenomenon of diminishing sample efficiency in long context persisted across all three architectures. This finding is also consistent with recent work analyzing the length extrapolation limits of Mamba (Ben-Kish et al., 2025). These new findings strongly reinforce our central claim: the diminishing efficiency of ICL in long context is a general property of the ICL paradigm in sequence models having a non-vanishing excess risk, not an artifact of the Transformer architecture. This underscores the broad applicability of our theoretical framework. We will add a discussion of these results to the revised manuscript.\n\n\nAkyürek et al. (2024). In-context language learning: Architectures and algorithms. arXiv\n\nLee et al.  (2024). Is attention required for icl? exploring the relationship between model architecture and in-context learning ability. ICLR\n\nBen-Kish et al. (2025) Decimamba: Exploring the length extrapolation potential of mamba. In ICLR.\n\n\n\n\n\n\n## **Q2:** \"You conclude by motivating a new generation of 'on-the-fly' adaptive methods without the diminishing efficiency. Could you elaborate on what such methods might look like?\"\n\n\nThank you for this excellent forward-looking question. Our paper's theoretical analysis provides a clear target for designing the next generation of methods: they must directly address the non-vanishing excess risk, which we identify as the root cause of diminishing ICL efficiency and is widely observed in many transformer-based models and other sequential models. \n\nOne promising direction is a method we might call Residual In-Context Learning. This approach augments the fixed base model with a lightweight, adaptive module (e.g., a linear model on the base model's features) trained at inference time. Using only the in-context examples, this module's goal is to learn the base model's residual error for the current task. Because this adaptive module is a principled learning algorithm, its error in estimating the residual would vanish as the number of demonstrations grows. Then, the final output can be obtained by correcting the original ICL prediction with the on-the-fly estimate of the excess risk, which could make the excess risk vanishing and therefore restore efficient learning behavior in long context that the base ICL model lacks. \n\nWe emphasize, however, that this is a conceptual direction, and realizing such a hybrid system would require significant further investigation."}, {"review_id": "lTrsNAWwmH", "reviewer": "Reviewer_pLhk", "summary": "This paper investigates the sample-efficiency of in-context learning (ICL) in transformer models, focusing on how many demonstrations are needed to reach a target prediction error compared to Bayes-optimal or principled learners. Using a stylized regression benchmark with Fourier-series functions and controlled noise, the authors show that transformers are nearly optimal in the few-shot regime but become significantly less efficient as the number of in-context examples increases. They formalize this observation using a performance-profile metric and support it with an information-theoretic analysis, which proves that under a certain assumptions the transformer’s efficiency must diverge from the Bayes rate in the long-context limit.", "questions": "See weakness.", "limitations": "yes.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 1, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is addressing an interesting problem that what you lose if you use ICL instead of principled algorithms. \n\n\nWeaknesses\n\n1. From expressivity point of view the transformer can implement many principled algorithm in its forward pass, an idea popularized by Oswald et al.[1] and many other follow-ups. So, if that's the case then I do not see the central point of this paper. The context length for fixed error level should be exactly the same for transformer or base algorithm.\n\n2. Related to 1, ICL is nothing special from traditional learning paradigm. We have some demonstration and we want to predict a new query, so transformer can be thought as an algorithm potentially.\n\n3. I think the main problem of this paper is Assumption 4.1,  basically as the model gets larger and more expressive, and also number of pre-traing increases we can get closer and closer to best possible error. And this is align with scaling-law common wisdom. \n\n4. Related to 3,  because the transformer is meta-trained across many tasks, it can, in principle, exploit cross-task regularities that a hand-crafted Bayes estimator (tuned per task) cannot access. The analysis ignores this potential advantage.\n\nBottom line: I think this paper assumption is too restrictive and unrealistic, and not adding any new insight to what ICL actually is.\n\n[1] [Transformers learn in-context by gradient descent](https://arxiv.org/abs/2309.14666)  \nJohannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, Max Vladymyrov. *arXiv preprint arXiv:2309.14666*, 2023.", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer pLhk for their time and effort in reviewing our manuscript and providing constructive feedback. Below, we carefully address the reviewer’s comments, clarify certain misconceptions, and emphasize the novelty and significance of our findings with support from recent relevant studies.\n\n\n## **W1:** \"From expressivity point of view the transformer can implement many principled algorithm in its forward pass, an idea popularized by Oswald et al.[1] and many other follow-ups. So, if that's the case then I do not see the central point of this paper. The context length for fixed error level should be exactly the same for transformer or base algorithm.\"\n\n\nWe agree that transformers, in theory, have the capacity to implement principled learning algorithms in their forward pass (von Oswald et al., 2023). However, the crucial distinction—and the central point of our paper—lies between this theoretical potential and the algorithms that are actually learned through standard meta-training. The work by von Oswald et al. is an existence proof, showing that a specific weight configuration can mimic an algorithm like gradient descent; it does not guarantee that pre-training will converge to this or any other optimal implementation.\n\nOur work investigates this exact gap between theory and practice. We ask: What kind of algorithm do transformers learn, and what are its limitations? Our key finding is that while the learned algorithm appears near-optimal in the few-shot regime (a finding consistent with previous works, e.g., Garg et al. (2022) and Panwar et al. (2024)), its sample efficiency degrades significantly in the many-shot regime. This reveals a critical limitation of what is learned in practice, which is the novel insight of our work.\n\nThis also clarifies the reviewer's comment on sample complexity. If a transformer did perfectly implement the Bayes-optimal algorithm, its sample complexity would indeed be the same. But since our findings show it implements a suboptimal algorithm, its sample complexity is naturally different and, as we demonstrate, worse.\nTo clarify this in the manuscript, we have added the following:\n(Line 54) \"While transformers are theoretically capable of implementing principled algorithms (von Oswald et al., 2023), we show that ICL, actually learn by training, deviates significantly from optimality in the many-shot regime.”\n\nGarg et al. (2022) What can transformers learn in-context? a case study of simple function classes. In NeurIPS. \n\nPanwar et al. (2024)  In-context learning through the Bayesian prism. In ICLR\n\n\n## **W2:** \"Related to 1, ICL is nothing special from the traditional learning paradigm. We have some demonstration and we want to predict a new query, so the transformer can be thought of as an algorithm potentially.\"\n\nWe respectfully disagree with the characterization that \"ICL is nothing special.\" While ICL can be viewed as an algorithm, its underlying mechanism is fundamentally distinct from traditional learning paradigms, and this distinction is critical to our paper's contribution.\n\nTraditional supervised learning involves updating a model's explicit parameters (weights) via an optimization process like gradient descent. In contrast, ICL performs learning purely through inference within a fixed-weight network. The \"learning\" happens as information provided in the demonstrations is processed without any weight updates.\n\nThis mechanical distinction is not trivial; it imposes unique constraints that do not exist in the same way for traditional learners. Investigating the properties and limitations of this unique inference-based learning mechanism is an active and important area of research (e.g., Brown et al. 2020; Xie et al., 2022; Akyürek et al., 2023; Jeon et al., 2024; Elmoznino et al., 2025). Our work contributes directly to this by providing a new, formal understanding of ICL's sample efficiency in long context.\n\nBrown et al. (2020) Language models are few-shot learners. In NeurIPS. \n\nXie et al. (2022) An explanation of in-context learning as implicit Bayesian inference. In ICLR. \n\nAkyürek et al. (2023). What learning algorithm is in-context learning? investigations with linear models. In ICLR.\n\nJeon et al. (2024) An information-theoretic analysis of in-context learning. In ICML. \n\nElmoznino et al. (2025). In-context learning and Occam's razor. In ICML.\n\n\n## **W3:** \"I think the main problem of this paper is Assumption 4.1, basically as the model gets larger and more expressive, and also the number of pre-training increases we can get closer and closer to the best possible error. And this is aligned with scaling-law common wisdom.\"\n\n\nWe thank the reviewer for raising this point, as it allows us to clarify the scope and basis of Assumption 4.1. Reviewer pLhk’s concern appears to stem from a misunderstanding, connecting our assumption to scaling laws.\n\n\nOur assumption is not about performance scaling with model size or pre-training data. Instead, Assumption 4.1 simply states that the excess risk curve is lower bounded when it confronts a sufficiently long context exceeding the context length during pretraining. Far from being \"restrictive or unrealistic,\" this is a robust and widely observed empirical phenomenon. It is a core finding in the literature on length-generalization limitations in transformers (Anil et al., 2022; Zhou et al., 2024) and is a pattern we explicitly validate in our own experiments (Figures 3 and 4). Therefore, we think that Assumption 4.1 is a reasonable, realistic, and empirically grounded foundation for our theoretical analysis of why ICL's efficiency degrades in the many-shot regime. \n\n\nAnil et al. (2022). Exploring length generalization in large language models. In NeurIPS.\n\nZhou et al. (2024). Transformers can achieve length generalization but not robustly. arXiv.\n\n\n## **W4:** \"Related to 3, because the transformer is meta-trained across many tasks, it can, in principle, exploit cross-task regularities that a hand-crafted Bayes estimator (tuned per task) cannot access. The analysis ignores this potential advantage.\"\n\n\nAs Reviewer pLhk pointed out, transformers' meta-learning capabilities allow them to exploit cross-task regularities, potentially offering advantages over handcrafted estimators learned for each task. \n\n\nFirst, our empirical setup explicitly accounts for this advantage. We pre-train the transformer on a diverse, hierarchical distribution of tasks. The ICL performance we report is therefore the result of the model already having exploited these cross-task regularities. \n\n\nSecond, and more fundamentally, Reviewer pLhk might misunderstand the role of the Bayes-optimal estimator in the benchmark. The Bayes-optimal estimator is not a \"hand-crafted Bayes estimator (tuned per task).\" It is a theoretical performance ceiling derived analytically from the true, known data-generating distribution (cf. Eq. 2). By definition, no algorithm, whether it is a transformer using meta-learning or any other method, can achieve a lower expected risk on this distribution. \n\n\nTherefore, our comparison is fair: we compare the transformer's meta-learned performance against the theoretical best-case performance on the same task distribution. The fact that the transformer falls short, particularly in the many-shot regime, is a significant and non-obvious finding. \n\n\nWe hope this clarifies that our assumptions are well-grounded and our findings provide a novel and important insight into the fundamental limitations of in-context learning."}, {"review_id": "tDYdXHXXIv", "reviewer": "Reviewer_N11X", "summary": "This work investigates the diminishing efficiency of in-context learning (ICL) in the many-shot scenario. The authors pre-train a Transformer model on a given task distribution for ICL and compare its sample efficiency against optimal baselines. They discover that the efficiency of in-context learning is high in the few-shot regime and decreases as the context length increases. They justify their observations by analyzing the suboptimality of ICL and showing that the suboptimality increases as the demonstration size grows.", "questions": "I see the authors use the Transformer as their default model architecture to examine the drop in sample efficiency for in-context learning with long contexts. I wonder what special properties of the Transformer they leveraged for their analysis? I don't see any architectural requirement in their assumptions. Thus, why don't the authors also consider other sequence modelling models, such as the LSTM, as ICL is not exclusive to the Transformer?", "limitations": "1. I think the authors should also acknowledge that their model size and training data diversity are not comparable to today's LLMs.\n2. I believe the authors should also acknowledge that they only investigated in-context regression. Otherwise, they should explicitly state that they use the term in-context learning to indicate in-context regression specifically.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper is overall clearly written. The notations are well-defined, and intuitions are provided to interpret the results.\n- The empirical study is well-designed with good statistical rigour.\n- The diminishing efficiency phenomenon in ICL is a real and practical concern for ICL applications.\n\n**Weaknesses**\n- The authors draw connections between their results and large language models. However, it is unclear whether the decreased efficiency of ICL in their particular setup and model size persists in LLMs, where the models are significantly larger and trained on a wide range of demonstration sizes. In the authors' setup, they only considered a fixed largest demonstration size $T_{train}$.\n- The analysis is based on the Assumption that there exists a uniform lower bound on the excess risk after a certain demonstration length, which can be too strong to hold.\n- This work only considered in-context supervised regression problems, leaving classification problems and other learning paradigms, such as reinforcement learning, unexamined.", "was_revised": true, "final_justification": "- This paper studies an important risk of in-context learning.\n- The authors' rebuttal addressed my concerns regarding the scalability of the phenomenon to larger models, the excess risk lower bound, and the scope of the work.\n- In response to my question, the authors conducted additional experiments to study the diminishing efficiency of in-context learning with models other than the Transformer.\n\nI therefore keep my decision to accept the paper.", "rebuttal": "We sincerely thank Reviewer N11X for the positive assessment and insightful, constructive feedback. We have revised our manuscript to incorporate the excellent suggestions, which we believe have strengthened the paper.\n\n## **W1:** “The authors draw connections between their results and large language models. However, it is unclear whether the decreased efficiency of ICL in their particular setup and model size persists in LLMs, where the models are significantly larger and trained on a wide range of demonstration sizes. In the authors' setup, they only considered a fixed largest demonstration size”\n\nThis is a crucial point. While our experiments use a controlled setup, we argue that the phenomenon we identify—diminishing ICL efficiency with long context—is a fundamental issue likely to generalize to large-scale models. Our reasoning is twofold: \n\n- General Theoretical Foundation: Our theoretical analysis in Section 4 is formulated with a general setting. The core finding relies on Assumption 4.1, which is not tied to a specific model size or data distribution. As a result, the theoretical insights we offer remain broadly applicable, transcending the specifics of our experimental setup.\n- Alignment with Large-Scale Empirical Evidence: The practical consequence of our theoretical finding is that Assumption 4.1 is a sufficient condition for the diminishing efficiency in long context. Crucially, Assumption 4.1 generality aligns well with empirical observations from recent large-scale transformer studies. For instance, Anil et al. (2022) and Zhou et al. (2024) both report that even state-of-the-art LLMs struggle to extrapolate when test-time context length exceeds that seen during training. Given that the Bayes risk improves with more demonstrations, this means that the excess risk curve is deteriorating in the extrapolating regime. \n\n\nTherefore, while explicitly confirming the phenomenon in state-of-the-art LLMs remains a valuable future direction, the alignment of our results with recent large-scale studies strongly supports the practical relevance and generalizability of our findings.\n\nAnil et al. (2022). Exploring length generalization in large language models. In NeurIPS.\n\nZhou et al. (2024). Transformers can achieve length generalization but not robustly. arXiv.\n\n\n## **W2:** \"The analysis is based on the Assumption that there exists a uniform lower bound on the excess risk after a certain demonstration length, which can be too strong to hold.\"\n\n\nWe thank the reviewer for highlighting this critical detail, and we have revised the manuscript to clarify the scope of Assumption 4.1. The assumption is not intended to be a single universal bound that holds for all models and tasks. Rather, it posits that for a given transformer and a given task environment, a lower bound of the excess risk exists (especially when generalizing to the demonstration size significantly larger than that of used in training). \n\nThis model-and-environment-specific assumption is strongly supported by empirical evidence. We observe this performance plateau directly in our own experiments (Figures 3 and 4). Furthermore, this behavior aligns well with empirical observations in large-scale models. For instance, Anil et al. (2022) and Zhou et al. (2024) show that state-of-the-art LLMs struggle when the context length at test time significantly exceeds that seen during training. These challenges transformers face with length generalization prove the existence of the excess risk’s lower bound in the length generalization regime.\n\nTo ensure this is clear, we have explicitly revised the statement of the assumption in the manuscript to be model and environment-dependent (Line 314):\n\n\n“For an environment $\\mathcal{E}$ and a transformer $TF_\\theta$, there exist constants  … ”\n\n\n## **W3:** “This work only considered in-context supervised regression problems, leaving classification problems and other learning paradigms, such as reinforcement learning, unexamined.” \n\n\nThis is a correct observation, and we thank the reviewer for suggesting we make our scope and its justification clearer. We have now explicitly addressed the focus on regression and other limitations in the revised manuscript. \n\n\nOur focus on regression is a deliberate methodological choice. This paradigm allows for the analytical derivation of the Bayes-optimal performance, providing a \"gold standard\" benchmark. This is essential for our goal of rigorously and quantitatively measuring the sample efficiency of ICL relative to a theoretical optimum. While extending this analysis to classification or RL is an exciting future direction, it poses challenges in analytically deriving the optimal baselines. \n\n\nTo improve transparency, we have made the following additions to the manuscript:\n\n\n\n(Line 6) “To investigate this, we adopt a meta ICL setup where each prompt corresponds to a regression problem, with the target function sampled from a hierarchical distribution, requiring inference over both latent model class and task parameters. In this setting, we benchmark sample complexity of ICL against principled learning algorithms, including the Bayes optimal estimator, under diverse performance requirements.”\n\n\n(Line 81) “Following common practice in the meta ICL literature, we focus on regression problems for which the Bayes-optimal performance can be derived analytically, allowing us to precisely quantify the sample complexity of in-context learning and principled learning algorithms.”\n\n\n(Line 554) “Further, the model size and training data diversity used in our meta ICL setup are significantly smaller and less diverse than those used in modern LLMs, such as GPT-4 and Gemini 2.5, which typically possess hundreds of billions or even trillions of parameters trained on vast, heterogeneous datasets.”\n\n\n\n## **Q1:** “I see the authors use the Transformer as their default model architecture to examine the drop in sample efficiency for in-context learning with long contexts. I wonder what special properties of the Transformer they leveraged for their analysis? I don't see any architectural requirement in their assumptions. Thus, why don't the authors also consider other sequence modelling models, such as the LSTM, as ICL is not exclusive to the Transformer?”\n\n\n\nThis is an excellent question that prompted us to conduct new experiments to strengthen our claims. Our initial focus on the Transformer was due to its dominance in ICL research and state-of-the-art large language models. However, the core of our theoretical analysis is not specific to the attention mechanism but rather to the general properties of sequence models, as long as it satisfies the conditions of our analysis (e.g., Assumption 4.1).\n\n\nMotivated by the reviewer's suggestion, we ran additional experiments with LSTM and Mamba architectures. The results are highly informative: The Transformer consistently outperformed both LSTM and Mamba in overall ICL performance, aligning with recent comparative studies (Akyürek et al., 2024; Lee et al., 2024). Crucially, the central phenomenon of diminishing sample efficiency in long context persisted across all three architectures. This finding is also consistent with recent work analyzing the length extrapolation limits of Mamba (Ben-Kish et al., 2025). These new findings strongly reinforce our central claim: the diminishing efficiency of ICL in long context is a general property of the ICL paradigm in sequence models having a non-vanishing excess risk, not an artifact of the Transformer architecture. This underscores the broad applicability of our theoretical framework. We will add a discussion of these results to the revised manuscript.\n\n\nAkyürek et al. (2024). In-context language learning: Architectures and algorithms. arXiv\n\nLee et al.  (2024). Is attention required for icl? exploring the relationship between model architecture and in-context learning ability. ICLR\n\nBen-Kish et al. (2025) Decimamba: Exploring the length extrapolation potential of mamba. In ICLR.\n\n\n**Limitations:** See the responses to W3 above. \n\n\nWe thank the reviewer again for the insightful suggestions, which have significantly improved the rigor and clarity of our work."}, {"review_id": "KccXparec0", "reviewer": "Reviewer_LvzK", "summary": "This submission considers the optimality of algorithms implemented by transformers via in-context learning (ICL). The main finding is that transformers implement suboptimal ICL algorithms, especially in the limit of long contexts (ie, more in-context examples) and when the context length at test time is longer than during training (length generalization). This result is based on a _stylized_ learning problem with a hierarchical data-generating process.", "questions": "Raventós et al (NeurIPS 2023) argue that transformers achieve better generalization performance by _failing_ to be Bayes optimal (with respect to the empirical pretraining distribution). My understanding is that in the current submission, transformers never outperform Bayes optimal algorithms. I would appreciate a discussion on this point.", "limitations": "Please see my comments about the _stylized_ setting in **Strengths And Weaknesses** above.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Understanding ICL in transformers and its limits is timely and important. I think this paper makes a meaningful contribution to this area. \n\n**Strength**\n- The use of a toy problem allows for a more in-depth study of ICL performance\n- Using principled algorithms as baselines enables quantification of ICL sample (context) efficiency\n- The paper is well structured and reads rather well\n\n**Weakness**\n- The _stylized_ setting is quite specific, the claim is more general.\n- I understand that the authors aim to illustrate the phenomenon, and I consider this a valuable contribution. But I think the abstract could be more informative about the learning problem considered.\n- The information-theoretic analysis is interesting, but it is not directly compared to experimental results. Perhaps mine is an unpopular opinion, but I feel that the technical details do not add much to the experimental results here. I'd rather see more experimental results on different learning problems.\n- The related work section is rather light. I note that the most recent reference is more than a year old (essentially an epoch in modern ML research). Expanding this section would allow the authors to better place their results in the growing body of research seeking to characterize and understand ICL with _\"physics-style\" or \"synthetic benchmark\"_ approaches.", "was_revised": true, "final_justification": "The authors have resolved most of my questions, and I find the topic current and interesting; therefore, I raised my rating from 4 to 5.", "rebuttal": "We thank Reviewer LvzK for the thoughtful feedback and insightful questions. We are encouraged that the reviewer found our work to be a \"meaningful contribution\" to the important area of understanding ICL. We address the specific weaknesses and questions below.\n\n\n## **W1:** \"The stylized setting is quite specific, the claim is more general.\"\n\n\nWe agree with the reviewer on the importance of clarifying the scope of our claims. While our experiments are necessarily conducted with a regression problem in a stylized setting to benchmark ICL against the Bayes optimal estimator, our central claim about the diminishing sample efficiency of ICL is grounded in a general theoretical analysis that is not specific to our experimental setup.\n\nOur information theoretic analysis hinges on a general condition on a shape of the excess risk curve (Assumption 4.1) sufficient to cause the diminishing efficiency in long context. This condition is not tied to a specific model size or data type. Besides, the sufficient condition stated in Assumption 4.1, which is about the existence of lower bound of the excess risk curve, align well with empirical observations in large-scale models. For instance, Anil et al. (2022) and Zhou et al. (2024) show that state-of-the-art LLMs struggle when the context length at test time significantly exceeds that seen during training. These challenges transformers face with length generalization prove the existence of the excess risk’s lower bound in the length generalization regime.  \n\nTherefore, while explicitly confirming the phenomenon in state-of-the-art LLMs remains a valuable future direction, the alignment of our results with recent large-scale studies strongly supports the practical relevance and generalizability of our findings.\n\n\n(Line 317) “We remark that the deficiencies of state-of-the-art LLMs beyond length generalization regime (Anil et al., 2022; Zhou et al., 2024) prove the existence of the excess risk’s lower bound within the length generalization regime.”\n\n\nAnil et al. (2022). Exploring length generalization in large language models. In NeurIPS.\n\nZhou et al. (2024). Transformers can achieve length generalization but not robustly. arXiv.\n\n\n## **W2:** \"I understand that the authors aim to illustrate the phenomenon, and I consider this a valuable contribution. But I think the abstract could be more informative about the learning problem considered.\"\n\nThank you for this valuable suggestion. To provide readers with a clearer problem setup and the complexity involved in the learning task, we have revised the abstract (Lines 6-7) as follows:\n\n\n“To investigate this, we adopt a meta ICL setup where each prompt corresponds to a regression problem, with the target function sampled from a hierarchical distribution, requiring inference over both latent model class and task parameters. In this setting, we benchmark sample complexity of ICL against principled learning algorithms, including the Bayes optimal estimator, under diverse performance requirements.”\n\n\n\n\n## **W3:** \"The information-theoretic analysis is interesting, but it is not directly compared to experimental results. Perhaps mine is an unpopular opinion, but I feel that the technical details do not add much to the experimental results here. I'd rather see more experimental results on different learning problems. \"\n\nWe appreciate this insightful comment. While more experiments on diverse tasks would certainly add empirical breadth, our theoretical analysis in Section 4 provides a unique and vital contribution that experiments alone cannot.\nThe theory does not just support the experimental findings; it provides a causal explanation for them. It demonstrates that the observed diminishing sample efficiency is not an artifact of our specific model, task, or training procedure, but rather an inherent limitation of ICL mechanism when faced with a large number of demonstrations. By identifying the formal conditions under which this inefficiency occurs, our theory correctly attributes the phenomenon to the ICL mechanism itself. We believe this provides a more fundamental insight than additional experiments on other tasks would. \n\n\nTo clarify this crucial role of our theory, we have added the following sentence to the manuscript (Line 276): \n\n\n“This theoretical grounding is critical, as it establishes that the diminishing inefficiency is an intrinsic property of the ICL mechanism, rather than an artifact of a particular experimental setup. This correct attribution is essential for guiding future work aimed at overcoming this limitation.”\n\n\n## **W4:** \"The related work section is rather light. I note that the most recent reference is more than a year old (essentially an epoch in modern ML research). Expanding this section would allow the authors to better place their results in the growing body of research seeking to characterize and understand ICL with \"physics-style\" or \"synthetic benchmark\" approaches.\"\n​​\n\nThank you for the great suggestion! We agree that contextualizing our work within the rapidly evolving literature is essential. We have updated our related work section to include several relevant recent papers that adopt similar stylized settings to understanding ICL. \n\n\n(Line 400): “More recently, these stylized settings have been used to probe other sophisticated behaviors of ICL. This includes analyzing transformers' in-context model selection and preference for simpler hypotheses (Deora et al., 2024; Elmoznino et al., 2025), their ability to infer causal structures (D’Angelo et al., 2025), and the implicit connection between ICL and low-rank updates to MLP layers (Dherin et al., 2024).”\n\n\n(Revised Line 402) This new perspective unveils a unique insight: the fundamental inefficiency of ICL in the many-shot learning regime, a critical limitation that was not the focus of these prior analyses.\n\n\nDeora et al. (2025). In-Context Occam's Razor: How Transformers Prefer Simpler Hypotheses on the Fly. arXiv.\n\nD'Angelo et al. (2025). Selective induction heads: How transformers select causal structures in context. In ICLR.\n\nElmoznino et al. (2025). In-context learning and Occam's razor. In ICML.\n\nDherin et al. (2025). Learning without training: The implicit dynamics of in-context learning. arXiv.\n\n\n## **Q1:** \"Raventós et al (NeurIPS 2023) argue that transformers achieve better generalization performance by failing to be Bayes optimal (with respect to the empirical pretraining distribution). My understanding is that in the current submission, transformers never outperform Bayes optimal algorithms. I would appreciate a discussion on this point. \"\n\nWe appreciate the reviewer’s important point and clarify the difference between the notion of Bayes optimality in our work and in Raventos et al. (2023).\n- Raventós et al. (2023): Optimality w.r.t. the finite pretraining data. They show that as pretraining data diversity increases, transformers deviate from the Bayes-optimal predictor defined on that finite pretraining set. This deviation actually improves generalization because it moves the transformer’s behavior closer to a ridge estimator that is optimal for the true underlying data distribution. \n- Our work: Optimality w.r.t. the true data-generating distribution. We define the Bayes-optimal estimator with respect to the ground-truth hierarchical distribution from which tasks are sampled. By definition, no algorithm can achieve a lower average risk on this distribution. For a side note, the notion of Bayes optimality here is the same as in the optimality of the ridge estimator in Raventós et al. (2023).\n\n\nIn short, Raventós et al. show that transformers are not optimal with respect to their limited training data, which allows them to be more optimal with respect to the true data source. Our work compares transformers to the theoretical ceiling of performance—the true Bayes-optimal estimator. Thus, there is no contradiction: transformers never outperform our Bayes-optimal baseline, which is consistent with the findings of Raventós et al. when properly interpreted.\n\nTo clarify this in the paper, we have revised Line 118:\n\n“The Bayes-optimal estimator defined in (2) minimizes the expected risk with respect to the true hierarchical data-generating distribution. This is distinct from the notion of optimality with respect to an empirical pretraining distribution with finite samples, as considered in Raventós et al. (2023), where deviating from empirical optimality can improve generalization.”\n\n\nWe hope these revisions and clarifications have fully addressed the reviewer's concerns. We thank the reviewer again for the constructive engagement."}]

Author rebuttal:

Meta-review: This paper investigates the sample efficiency of in-context learning (ICL), aiming to determine how many demonstrations are needed to achieve a target prediction error, relative to Bayes-optimal or principled learning algorithms. The central finding is clear: ICL approaches the efficiency of a Bayes-optimal estimator in few-shot settings, but its efficiency declines as the context length increases. This behavior is explained through an information-theoretic decomposition into Bayes risk and a non-vanishing excess risk. 

Three reviewers recommend acceptance; one remains rejection. Reviewer concerns focus on whether the limitation is inherent to ICL versus a by-product of imperfect training and on the scope of Assumption 4.1. 

In their rebuttal, the authors offer clarifications that address the two main concerns: given that practical transformers are inherently imperfect, the ICL mechanism itself leads to diminishing efficiency with longer context lengths. The authors also clarify Assumption 4.1 as model- and environment-specific and situate it alongside empirical evidence on length generalization. While some questions regarding this assumption persist, the paper presents a novel and valuable contribution, and the authors have addressed the key concerns constructively. I recommend acceptance.
Decision: Accept (poster)