Understanding Emergent In-Context Learning from a Kernel Regression Perspective
Authors: Chi Han
Ziqi Wang
Han Zhao
Heng Ji
Venue: Accepted by TMLR ()

Abstract: Large language models (LLMs) have initiated a paradigm shift in transfer learning. In contrast to the classic pretraining-then-finetuning procedure, in order to use LLMs for downstream prediction tasks, one only needs to provide a few demonstrations, known as in-context examples, without adding more or updating existing model parameters. This in-context learning (ICL) capability of LLMs is intriguing, and it is not yet fully understood how pretrained LLMs acquire such capabilities. In this paper, we investigate the reason why a transformer-based language model can accomplish in-context learning after pre-training on a general language corpus by proposing a kernel-regression perspective of understanding LLMs' ICL behaviors when faced with in-context examples. More concretely, we first prove that Bayesian inference on in-context prompts can be asymptotically understood as kernel regression $\hat y = \sum_i y_i K(x, x_i)/\sum_i K(x, x_i)$ as the number of in-context demonstrations grows. Then, we empirically investigate the in-context behaviors of language models. We find that during ICL, the attention and hidden features in LLMs match the behaviors of a kernel regression. Finally, our theory provides insights into multiple phenomena observed in the ICL field: why retrieving demonstrative samples similar to test samples can help, why ICL performance is sensitive to the output formats, and why ICL accuracy benefits from selecting in-distribution and representative samples.

Published in Transactions on Machine Learning Research (09/2025)
Understanding Emergent In-Context Learning from a Kernel
Regression Perspective
Chi Han
chihon3@rillnois edu ziqiu9@ellinii.. edu honzhao@ Williiss.. edu hengj edu
Ziqi Wang
Han Zhao
Heng Ji
Siebel School of Computing and Data Science
University of Ilinoii Urbana- Channaagn
Reviewed on OpenReview: https: // net/ per/forum? id- 6rD50Q6y
Abstract
Large language models (LLMs) have initiated a paradigm shift in transfer learning. In contrast to the classic pretraining-then-fireee downstream prediction tasks, one only needs to provide a few demonstrations, known as in-context examples, without adding more or updating existing model parameters. This in-context learning (ICL) capability of LLMs is intriguing, and is not yet fully understood how pretrained LLMs acquire such capabilities. In this paper, we investigate the reason why transformer-bent based language model accomplish in-context learning after pre-training on general language corpus by proposing a kernel-regression perspective of understanding LLMs' ICL bahaviors when faced with in-context examples. More concretely, we first prove that Bayesian inference on in-context prompts can be asymptotically understood as ker- nel regression 9 = E, y:K (x, x:)/ K(x, x;) as the number of in-context demonstrations grows. Then, we empirically investigate the in-context behaviors of language models. We find that during ICL, the attention and hidden features in L.MSs match the behaviors of kernel regression. Finally, our theory provides insights into multiple phenomena observed in the ICL field: why retrieving demonstrative samples similar to test samples can help, why ICL performance is sensitive to the output formats, and why ICL accuracy benefits from selecting in-distribution and representative samples. Code and resources are publicly wwt.:/occcaaaaaeea available at http://httt...ooooocooooooooo/aa Kerre--e -eerreel-eegeessior
procedure, in order to use LLMs for
1 Introduction
Pre-trained large language models (LLMs) have emerged powerful tools in the field of natural language processing, demonstrating remarkable performance across broad range of applications (Wei et al. 2022a Kojima et al.] 2023 Wei et al. 2022b Brown et al. 2020: Li et al. 2023). They have been used to tackle diverse tasks such as text summarization, sentiment analysis, schema induction and translation, among others (Brown et al. 2020 Radford et al. 2023 T.i al. 2023). One of the most fascinating capabilities of LLMs their ability to perform in-context learning (ICL), process in which language model can make predictions on test. sample based on few demonstrative examples provided in the input context (Logan IV et al. 2022 This feature makes LLMs particularly versatile and adaptive to different tasks. Studies have found ICL to emerge especially when the size of LLM is large enough and pre-trained over a massive
corpus Wei et al. 2023).
Although intuitive for human learners, ICL poses a mystery for optimization theories because of the signifi- cant format shift between ICL prompts and pre-training corpus. There have been lots of efforts to provide a theoretical understanding of how LLMs implement ICL. Some work (Xie al. 2022 Wang al. 2023) approaches this problem from data perspective: they claim that ICL is possible if model masters Bayesian

---

Published in Transactions on Machine Learning Research (09/2025)
X
demonstrative samples
Input: moving and important.
Output: Positive.
Input: excruciatingly unfunny and pitifully unromantic.
Output: Negative.
Input: the plot is nothing but boilerplate clichés from start to finish.
Output: Negative.
test input
Input: intelligent and moving
Output:
70%: "Positive"
Σ, K(X; yi
((xp (eest)
y K(x;, Σ,K(x; Xtest)
(similarity kernel)
Figure 1: Our results suggests that LLMs might be conducting kernel regression ICL prompts.
inference pro-training distribution. Our paper also follows the setting in Xie al. (2022) that models the natural language mixture of sub-distributions and that LLMs conduct Bayesian inference it while focusing on providing tractable approximation of ICL. inference via kernel regression along with insights into ICL behaviors. Another stream of work conjectures that under simple linear setting: [x, where = w x and the input sequence x only has length 1, they can construct a Transformer (Vaswani et al.
2017) to implement gradient descent (GD) algorithm over ICL prompt (Akyürek el. al., 2023 von Oswald jtt al. 2022 Garg al.j 2023). However, this constrained setting diverges from the most interesting part of ICL, as state-of-the-art LLMs work with linguistic tasks where the sequential textual inputs have complex semantic structures, and ICL emerges from pre-training general-purpose corpus instead of explicit ICL training.
In this work, we delve deeper into the question of how to undersland the mechanism that enables Transformer- based pre-trained LLMs to accomplish in-contezt learning on sequential data. We specifically explore the hypothesis that L.MSs employ kernel regression algorithm when confronted with in-context prompts. Kernel regression adopts a non-parametric form
(z, 1 Σ, <x,,,)
(1)
when making predictions, where (xx æ;) is a kernel that measures the similarity between inputs æ and zy. In plain words, it estimates the output I on x by drawing information from similar other data points æ; and taking weighted sum on their ys-
We first provide a theoretical analysis demonstrating that Bayesian inference predictions in-context prompts converge to kernel regression in Section In Section 4.2 our results also shed light nu- merous phenomena observed in previous empirical studies, such as the advantage of retrieving in-context examples that are similar to the test sample, the sensitivity of ICL performance to the output formats, and why using a group of in-distribution and representative samples improves ICL accuracy.
Following our theoretical investigation, in Section we conduct empirical studies verify our explanation of in-context learning of LLMs in more detail. Our results reveal that during LLM ICI, the attention map used by the last token to predict the next token is allocated in accordance with our explanation. By plugging atlention values into our equation, we are also able to reconstruci the model's output with over 80% accuracy. Moreover, we are able to reveal how information necessary to kernel regression is computed in intermediate LLM layers.
2 Related Work
2.1 In- to-Context Learning
As an intriguing property of large language models, in-context learning has attracted high attention in the research community. There have been numerous studies empirical analysis of in-context learning,
2

---

Published in Transactions on Machine Learning Research (09/2025)
including format and effects of in-context samples MMin et al. 2022b a Zhao et al. 2021 selection of in-
context. samples (Liu et al.
2022 Lu et al. 2022).
characteristics of in-context learning behaviors (Zhao
et al.) 2021 Lu et al. 2022 Khashabi et al., 2022 Wei et al. 2023), and the relation between ICL and
pre-training dataset (Wu et al.
2022 Chan ct al. 2022).
Theoretical analysis of ICL. is another active research direction. One branch of studies investigates TCT. from a data perspective: under some general assumptions of the data, (Xie) 2021 Wang al.] 2023) demonstrate that with sufficient pre-training data and a good enough generative model of the data, ICL could be understood as performing Bayesian inference. However, they do not fully explain if such Bayesian inference is computationally feasible in practical language models since such Bayesian inference involves unbounded depth computational graphs as the number of samples increases. Our study builds on top of some similar assumptions but goes further to explain how ICL can be accomplished with the attention mechanism in Transformers Vaswani et al. (2077). Recently, there controversy regarding whether LLMs always conduct Bayesian inference in the ICL sctting, and Falck et al. (2024); Raventós et al. (2023): Park
et al. (2044) show that there exist. scenarios where LLMs' predictions deviale from the predictions of Bayesian an error term in Equation 7 related to Eg which use to
inference. This distinction can be quantified
mcasure the deviation between ICL task and pro-training data. Raventós ct al. (2023 1 Lu et al. (2024)
additionally show that under certain conditions, this divergence enables Transformers to achieve stronger
performance than Bayesian inference. Both Raventós et al. 2023 Park et al. (0024) discover that there is a threshold needed where ICL behaviors become less similar to Bayesian inference.
Another perspective to explain ICL is by analyzing what algorithms might be implemented in LLMs under ICL settings, which is closely related to our study. Representative studies include (Akyürek et al. 2023 Garg
et al. 2023 von Oswald et al., 2022 Dai et al. 2022 Mahankali et al.; Ren & Liu 2023), with the majority of them proposing gradient descent (GD) algorithm as a promising candidate answer. More recently, Fu et al. (2024) show that transformers can learn to approximate second-order optimization methods for ICL, such as the Itcrative Newton's method, which is exponentially faster than GD. These studics are mostly constructive, the sense that there exist. parameters of transformer that can be used to approximate certain algorithms. On the other hand, theoretical explanations of whether or why such parameters might be learned during pre-training on language data are less explored. Furthermore, such settings usually focus specific type of question, such linear regression, to construct the algorithm.
Going beyond the linear setting, Guo et al. constructs solutions of ridge regression assuming non- non-linear rep- resentations on synthetic datasets. On top of individual algorithms, [Bi et al. (2024) explores the possibility of Transformers selecting from a pool of algorithms. Some sallowing-up work analyzed how training dynam- ios converge Transformer to TCL solutions on linear functions /Zhang et al. (2024); Huang et al. Ahnn et al. (2023), and when linear-attention Transformers is trained on linear tasks Lu al. (2024). [Kim & Suzuki analyzed the training landscape of Transformers proximity to global optima. These attempts in explicit constructions of algorithms in Transformers mostly assume relatively simplified sctting with input lengths equal to 1, and evaluate Transformers after training on a synthetic dataset (including (Akyürek et al.) 2023 Garg et al. 2023)). This is different from ICL's main advantage as an emergent ability on language pre-training, and LLMs can work on textual data which involves sentences with lengths longer than 1.
ICL inputs, including the avoidance of saddle points and
2.2 Emergent Ability of LLMs
This is a larger topic that in-context learning is also highly related to. Wei ct al. 2022a Brown et al. 2020: Kojima el al. 2023 Wei et al. 2022b) showed that abilities including reasoning, in-context learning, few-shot learning, instruction understanding, and multilingualism emerge in large language models after pre-training on massive language data. These impressive and mysterious capacitics have boosted significant progress natural language processing as well as artificial intelligence, but still baffle theoretical analysis. In this work, we make a preliminary step towards understanding ICL as special case of LM capacity emergence.
3

---

Published in Transactions on Machine Learning Research (09/2025)
2.3 Associating Attention with Kernel Regression
Chen & Li 2023) presents related view of associating self-altention with kernel ridge regressions. They focus on a different problem and propose to substitute attention mechanisms with a kernel to tackle the uncertainty calibration problem. Another blog, Olsson et al.] (2022), also relates attention to kernel regression but defines ICL as "decreasing loss at increasing token indices", rather than the more widely adopted definition of learning from in-context demonstrations as ours. Another related stream of work the topic of associative memory deep learning networks. [aassaur al. revisits Hopfield networks which is capable of reusing past patterns for inference new data. Arora et al. Bietti et al. (2023 explore components in Transformers which might be responsible for associalive recalling.
3
Formulation
3.1 Preliminaries: Hidden Markov Models
Following the setting of (Xic et al. 2022), we assume that the pre-training corpus can be modeled by mixture of HMMs. Each HMM corresponds to a certain task 0 E e. Assuming a large finite number of tasks, one can include all task-specific HMMs into one single HMM. In this unified HMM, let be the set of states, and be the set of observations where |O] = m. At each time step, state s, randomly emits one observation or and then transits to the next state St+1- Quantitics Ppre-train: P(s = s'|st s) and P(o, = olso s) denote the pre-training initial distribution, transition distribution, and emission distribution respectively. Under define the transition matrix T : T(s, P(s'(s), and emission matrix B B(s, o) P(o)s), respectively. We also let = (oo....) be the full obscrvation sequence, and [oaa denote its first I tokens.
arbitrary ordering of S and O,
3.2 mn-Context Learning
In this work, we consider the following formulation of in-contexi learning (ICL). Let θ be the set of tasks. We follow the ICL prompt formulation, the HMM mixture formulation, and assumptions in (Xie et al.) 2022) as follows. The distribution of sequences generated by each individual task in the HMM together composes the pre-training distribution. Specifically, each lask 0 E is associated with a distinct initial state 8e E S with transition rate lower bound 6a (the Assumption 5 "Regularity" in Xie et al. (202)), ) and the set of all such initial states Sstart {sal0= E θ} forms the support of Ppro-train- For test task 0*, the in-context learning prompt follows the format:
X22, OTCL = [Sn, Xrest] = [x1, M/, o delim, X2, V2, o delim Xn, Un. delim Xtest]
(2)
where the input-output pairs [xs; vil are i.i.d. demonstrate samples sampled from 0*, and delim is the delimiter token with emission rate lower bound e, (the Assumption 5 "Regularity" in Xie al. (2022)). We further make some connections between in-context learning and the HMM model. Note that the probability of generating a sequence from the initial distribution Po can be expressed follows(Jaeger 2000
[ i=0
diag((p. YT diag(po.) )1
P(o Ppre-train) Ppre-train
(3)
where P. is vector of emission probabilities P(ols E S) for o. We denote the intermediate matrices in Equation as one operator for a (sub)sequence o:
To = II diag(po, ).
(4)
We use matrix Ep. to denote the covariance between all of its d' elements of vec(To when O is generated from initial distribution p. For each individual task, we denote eq infi p(Σ P Prre En1) quantify the difference between sequences generated by so and those from pre-training distribution, where
4

---

Published in Transactions on Machine Learning Research (09/2025)
P denotes the spectral radius of matrix. Let 7 supo i [Tooon] le be the upper bound of Topa 's Frobenius-norm.
3.3 Assumptions
We go and present the assumptions borrow from Xie et al (2022
Assumption 1. (Delimiter Tokens) The delimiter token indicates the start of sampling of new sequences:
P(8+11 = detim)) > + 8 E Sstart
this means that the delimiter tokens are indicative enough, such as the start of new line before the beginning of paragraph.
Remark:
Assumption 2. (Task Distinguishaaiiitty) The Ballback-Leibler divergence (KL divergence) between the first tokens between two distinct tasks 0 + 0' is lower-bourded by:
inf DEL (P(o 00)) P(o)):100) - EKL > In 0,0'
Cred
Remark: this requires that tasks are distinguishable from each other. As KL-divergence is non-decreasing with length I, it suffices to increase the length I to provide sufficient task information.
4 Theoretical Analysis
4.1 Explaining ICL as Kernel Regression
Within the framework presented in Section pose the following result. The basic idea is that, as the number of samples n increases, inference on the in-context learning prompt converges to a kernel-regressson form.
Theorem 1. Let us denote a kernel
K(x,x') vee(Tx) T pp vec(Tx))
(5)
Tx is defined in Equation 4 Let e(y) be the one-hol veclor for index y- Then the difference between the following logit vector in the form of kernel regression:
Σ e(y,) ((Xest(( x:)
ý
(6)
Ei K(Xtest, xx)
and it converges polynomially to the conditional likelihood of Y conditioned ICL prompt OICL. P(Y OICL, JJCL-Ppre-trusu)) with probability 1 1-6 8:
1 - P(Y Oret, Ppre-trmin) 12e00 (v= 4m 8
(7)
Equation G can be interpreted follows: it calculates the semantic similarity between the test input Xtest and each sample X; and aggregates their outputs to compute a most likely prediction for the test sample. This is natural to the motivation of ICL: we encourage the LLM to leverage the pattern provided in demonstrative samples and mimic the pattorn to predict the test input. Equation also similar to the form of attention mechanism used in Transformer decoder models:
h = softmax(g T K)VT E, V.lakk ela
(8) where q is the query vector corresponding to the last token, k. K are the key vectors and matrix, and the value vectors and matrix used in the Transformer, respectively. The only difference is that e <

---

Published in Transactions on Machine Learning Research (09/2025)
is replaced with dot product in Equation G which can be regarded kernel trick. We assume that previously hidden layers are responsible for learning the semantic vectors of sample inputs vec(Tx). We
can then make the following loose analogy between our kernel regression explanation (Equation 6 and the attention mechanism (Equation
Label information o(y:) corresponds with the value vector V/
The similarity kernel vec(Tx) T E-1 where:
(xo(Tx)) loosely corresponds to the attention value elakk)
Pprostrain
the semantic information vec(Tx,) corresponds to the key vectors k, and query vectors q, for samples [].
Here we further explain how the kernel is effected by the LLM. Equation i measures similarity between x and x' on the space of vec(Tx) and vec(Tx). This flattened vector of Tx defines the "belief state" in HMMs, which determines the conditional probability P(-|x) after prefix x. This the pre-training objective functionality of LLMs. Therefore, if x; and &test have similar follow-up conditional probabilities under the LLM, their similarity value will be larger, and vice versa.
One might argue that it is also theoretically possible to directly compute the next token likelihood in Equation 3 However, this form involves 2n consccutive matrix multiplications. When increases, this is infeasible for a practical Transformer architecture which is composed of a fixed number of layers. In comparison, Equation 6 only requires semantic information for each sample x to be provided beforehand and then applics kernel regression (which can be done by one attention layer) to get the answer. Learning to represent Tx is probable for preceding layers, as it is also used for ordinary inference P(y x) = Ppre-ttain Tx In experiments in Section we demonstrate that this analogy can explain the ICL. behaviors of L.MS to an extent.
4.2 Insights Provided by the Explanation
Theorem can provide insights into multiple phenomena in the ICL field observed by previous studies. This helpful for undorstanding and predicting the behaviors of ICL, and providing hcuristics for future development.
Retrieving Similar Samples It is empirically observed (Rubin et al. 2022 Liu et al. 2022 that retriev- ing demonstrative samples x; that is similar to the test input Xeett can benefit ICL performance. This phenomenon is understandable from our explanation. Encouraging the selection of similar samples can be understood as limiting the cosine distance between demonstrative samples x, and test sample Xtest in sentence embedding space. This is similar to selecting a smaller "bandwidth" in kernel regression and sampling only from a local window, which reduces the bias in kernel regression. Therefore, devising better retrieval tech- niques for selecting samples, especially those with similar representations as LLMs, is a promising direction for further boosting ICL scores.
Sensitivity to Label Format Min et al. 2022b) also observes that the ICL. performance relies on the label format. Replacing the label set with another random set will reduce the performance of ordinary auto-rogressive LLMs. This can be explained in our theory that the model's output comes from a weighted voting of demonstrative sample labels (1a). If the label space is changed, the next token will also be misled different output space. So is generally beneficial for ICL to ensure the demonstrative samples and test samples share an aligned label space and output format.
Sample Selection In Equation 7 the existence of the n²co term implics that the demonstrations [x; yil should be sampled in a way close to Ppre-train- To verify this, we conduct an experiment by converting test inputs to semantically similar but rarer sentences. Specifically, on the SST2 dataset, we prompt GPT- 3.5-turbo to generate semantically similar while rarer
Table Effect of OOD inputs on ICL on SST2 dataset.
OOD-type None
rare word
complex
typo
accuracy
0.805
0.677
0.788
0.534

---

Published in Transactions on Machine Learning Research (09/202))
head
layer
head 2
layer2
head 3
layer 3
head
layer
head 5
layer
head 6
layer 6
layer?
hhaad 16
layer& 8
layer 9
layer
layer 11
layer 12
layer
layer 14
layer
layer
layer 17
layer
layer 19
layer 20
layer
layer 22
layer 23
layer
layer 25
layer
layer 27
layer 28
X
N
y>2
y/
X4
X₃ sample 5
Ys
X,
Xyp
sample
sample
sample 3
sample 4
sample 6
sample
Figure 2: Averaged attention map over GLUE-sst2 test set. portion of attention on demonstrative samples is generally focused label positions yt- This conformss to the intuition in Theorem | that the inference on in-context learning prompts is weighted average over sample labels.
(i.c. OOD) expressions of inputs and use them substitute the original dataset inputs. We consider three lypes of OOD lypes. "Rare word is where words are substituted with rare synonyms. "Complex" is to express the original sentence in more complex structure. "Typo" is where we require adding typos to the original input. The results are listed below. We see that compared with original scores ("None" ), these OOD types more or less decrease the ICL accuracy, with "typo" having the largest effect, dropping the accuracy to near random.
Bias from Pre-training Data Theorem ] implies that the final prediction depends both on in-context examples and prior knowledge from pre-training. This can be seen from the 17²Eg term in Equation 7 which comes from the pre-training information in Equation 17 in Appendix A Specifically, co measures the similarity between 0's distribution and pre-training distribution, and rjžEo describes how the pre-training distribution's bias affects the ICT prediction. When the task 0 emits sequence distribution similar to the pretraining distribution, eæ will be small, and the ICL prediction ý will converge to P(Y OrCL.PPoeetsauu Pppee-train) with a smaller error n/²6g, and vise versa. This partially elucidates Kossen et al. (0023 's finding that "ICL cannot overcome prediction preferences from pre-training."
Remaining Challenges However, we need to point out that there are still phenomena not explainable by our framework, as well as most previous explanations, One most mysterious one the sensitivity to sample ordering (Lu et al.) (022) as kernel regression should be order-ignorant, which no existing explanations (including (Xie al. 2022 Akyürek et al.) (023)) take into account. Another intriguing question is that work finds I.I.Ms robust. to perturbed or random labels Kossen et. al. 2023) while hers find the

---

Published in Transactions on Machine Learning Research (09/2025)
480
635
565
Layer Index
Layer
25
050
(a) Accuracy compared with model output g.
(b) Accuracy compared with ground truth label yheat.
Figure 3: We use each head's attention weights on demonstrative samples to manually average sample labels yi- These ligures show that "reconstructed" outpuls in some heads from layers 16~21 matches LLM prediction with as high as 89.2% accuracy, and matches ground truth with 86.4% accuracy.
opposite Mii et al. 2022b). We attribute such phenomena to the fact that LLMs also rely on a large portion implicit reasoning in text generation and might benefit from linguistic cues in text prompts. Our theory provides partial explanation, which needs to be combined with this implicit ability of LLMs to form more comprehensive understanding of ICL.
5 Empirical Analysis
In this section, we conduct empirical analysis on LLMs in order to verify our hypothesis. Because Equation is only special solution among infinitely many of its equivalent forms, and it also relies on the unknown HMM structure, it is infeasible to directly evaluate it data. However, can verify if it can predict observable behaviors on LLMs in experiments. Limited by computation resources in academic lab, we analyze the GPT-J 6B monel(Wang & Komatsuzaki 2021 on one Tesla V100. It employs a decoder-only Transformer architecture. In this section, we use the validation set of the sst2 dataset as a case study, while results on more lasks can found in Appendix B We investigate the ICL behavior of LLMs from shallow to deep levels, and sequentially ask the following questions: Do the attention heads collect label information e(y:) as predicted? Does the attention-kernel analogy explain LLM's prediction? Can we actually explain the attention values kind of similarity? Can find where algorithmic features e(y:), Tx, stored? The following sections answer these questions one by one.
5.1 Where Are Attentions Distributed During ICL?
First, we notice that Equation implies that the LLM takes weighted average over sample labels y in ICL. Figure shows how attention weights distributed in-context learning inputs [S,, Xteal- On cach test point, we sample the next token. After getting the attention maps, ICL samples may have varied lengths, we re-scale the attentions on each x from |x| to a fixed 30-token length with linear interpolation. After aligning the attention lengths, we average all attention maps. The horizontal axis the aligned positions on prompt Xtex)) Fach bar corresponds to one of Transformer layers. Within each bar, each thin line is 1 out of 16 attention hcads. Darker (bluc) arcas mean smaller averaged attention, while brighter areas indicate high atlention.
ICL prompt and collect the attention map over previous tokens for predicting
In Figure 2 there are three major locations of attention masses. First, majority of attention is focused on the final few tokens in Xtest: especially in the first 3 layers. This accords with previous observations that Transformer attentions ions lend to locate in a local window to construct local semantic feature for Xtest- Secondly, as also observed in previous studies, LLMs tend to allocate much attention on the first few tokens in a sequence collect starter information. Finally and most intriguingly, we observe concentrated attention on each sample label tokens {w:). This phenomenon confirms an aggregation of label information in LLM ICL, in line with the prediction by Equation 0 Note that explanation does not specify limit the
8

---

Published in Transactions on Machine Learning Research (09/2025)
model from implementing kernel regression in particular layer. In fact, equivalent mechanism can occur in one or more layers long as these (possibly redundant) results can be passed with skip connections to the final layer and aggregated.
5.2 Can Attentions Be Interpreted as Kernel Functions?
Now that we observe expected locations attention weights on labels, we go on to verify if the LLM really prodicts by averaging on labels as suggested by Theorem | We iterate over 16 heads and 28 layers, and insert their allention weights into Equation G to manually average the label distribution. This similar concept to a mind-reading experiment predict one's next word using brain waves only (Wang & Jil (022). Specifically, for cach attention head, we use the maximal attention value a, within the range of [x;, Vil the kernel weight. Then on the ICL samples, we reconstruct the prediction as follows:
Σ e(y;)a
y = arg max
2i=1
The resulting "reconstructed output" y is compared for both LLM's actual prediction & and ground truth label Yeeat to calculate its accuracy. Figure 3a and (bb plot the accuracy between ğ and ÿ and between ÿ and Frest respectively. Interestingly, we spot the existence of several heads in layers 18~21 which demonstrato high accuracy in reconstruction. The highest them (layer 17, head 10) achieves 89.2% accuracy on & and 86.4% accuracy on Vtest- This result validates our hypothesis that some components in Transformer-based LLMs implement kernel regression. Note that this phenomenon happens within few adjacent layers in the middle of the model. This is similar to our prediction in Section [4.] not many attention layers are needed for kernel regression, long the required features have been computed by preceding layers. It is enough for the higher layers to only pass on the computed results.
5.3 Which Samples Receive High Attention?
We go on and ask the question: if the LLMs use attention to implement ker- 0.35 nel regression, what kind of similarity does this kernel function evaluate? From Equation 6 we see that the dot prod- uct is measuring similarity between Tx, which encodes information necessary for HMM inference: p(o)x) = tree B. Therefore, we conjecture that the atten- tion value a, between Xtest and x, cor- relates with their prediction similarity. Specifically, we define the prediction sim- ilarity as follows:
4-
0.25
0.20
10
0.10
12
14
0.00
10
20
25
Figure 4: Pearson correlation between sample's attentions and prediction similarity simpred (Xtest, x:) (Equation D). x-axis are layers and y-axis are heads in each layer. Note the resemblence between this heatmap and Figure g
sim (2) = P(o|x)) P(o)x)), (9)
which is measured by applying LLMs on these texts alone, rather than in ICL prompt. Finally, we compute the Pear- son correlation coefficient between sim(X x;) and each attention values on samples for cach attention head. The results are shown in Figure 4 The absolute value of correlation is not high, as P(o|x) is a dimension reduction to Tx and can lose and mix information. However, we can still note a striking similarity between it and Figure This means that the heads responsible for ICL mostly attend to prediction-sini simmllr samples.
5.4 Do Intermediate Features Store Information Useful for Kernel Regression?
In this section, we go into more detailed level, and investigate the question: where do Transformer-base LMs store the algorithmic information needed by kernel regression? To this end, take out the intermediate

---

Published in Transactions on Machine Learning Research (09/2025)
Accuracc
Accuracy
0.85
0.80
0.9
0.75
0.0
9.7
0.60
0.55
Relative Potiton
0.50
Relative Posttion
J
Layer Number
15
25
Layer Number
20
(a) Predicting arg max., P())x.) with key vectors.
(b) Predicting y with value vectors.
Figure 5: Key and value vectors encode label and LLM prediction information at high-attention position. Here y-axis denotes the relative position to the high-attention position in each sample. Each sphere is an attention head. The curve shows average accuracy within each layer.
key and value features in all layer heads and see if the correct information is stored in the correct locations. Note in Scction 5.1 we observe that a major part attention weights are located at the label position, so we focus on positions within [-1,3] relative to this position. Noticing the analogy we made at Section 4.1 that
k ~ vex(Txx) and vi ~ Yj, we study two sub-questions: (1) whether value vectors encode label information yi and (2) whether key vectors encode LLM prediction information P(o[x:). For each head, we conduct Ridge regression with À = 0.01 to fit the task in these 2 questions. Results are presented in Figure We can observe that, generally the high-attention position (y-axis = 0) indeed achieves the best accuracy. Figure 5b is intuitive, as tokens at a position later than the label token y/ can casily access the information of y; by sell-attention. The slight drop at position +3 means that a longer distance introduces more noise to this information flow. Results in Figure 5a tell us that, although sentence x, starting position in ICL prompt is shifted and different from 0, k2 still strongly correlated with P(o)x;), which indicates sense of translation invariance. Overall, the results mean that, with the attention map distributed in Figure LLM able use the attention mechanism to extract information regarding Tx; and y from key and value vectors effectively just we described.
5.5 How Well Can Our Explanation Reconstruct ICL Prediction?
Finally, we numerically evaluate our explanation on its ability to reconstruct the ICL predictions and tasks. We uniformly use 700 data in each validation set to represent tasks in a balanced way. We select the attention head that has the highest correlation with model predictions. This head is then evaluated on ICL prediction reconstruction and task performance a held-out set of 300 data per task. In Table 2 we see that the ICL output reconstruction has an accuracy from 68% to 80% except for the harder task of MNLI. The task performance of reconstructed outputs matches the model's performance level. It also achieves similar or superior performance than kernel regression on sentence encoders such as all-MiniMM-L6-v2 and bert-base-niee www--aaa--ll-men-tokeeeee tean- [Reimers & Gurevych 2019
6 Conclusions and Future Work
In conclusion, our work provides novel theoretical view to understand the intriguing in-context learning (ICL) capabilities of Transformer-based large language models (LLMs). We propose that LLMs ICL can be underslood as kernel regression. Our empirical investigations into the in-contexi behaviors of LLMs reveal that the model's attention and hidden features during ICL are congruent with the behaviors kernel regression. Furthermore, our thcory also explains several observable phenomena in the field of ICL: why the retrieval of demonstrations similar to the test sample can enhance performance, the sensitivity of ICL to output formats, and the benefit. from selecting in-distribution and representative samples. There still
10

---

Published in Transactions on Machine Learning Research (09/2025)
Table 2: Performance of explicit kernel regression (KR) and LLM ICL on downstream tasks.
rotten- tomatoes
tweet, @weet_eval
tweet_eve (irony)
tweet_ eval
Method
sst2
mnli
(hate)
(offensive)
GPT-J-6B ICL
0.805
0.383
0.671
0.539
0.519
0.542
ICL output reconstruction
0.797
0.597
0.683
0.713
0.722
0.753
Reconstructed performance
0.750
0.360
0.527
0.587
0.522
0.513
all-MiniLM--6-v>
0.503
0.321
0.478
0.548
0.491
0.588
bert-base-nle maaeee-elli-me--toeeee KR
0.523
0.325
0.502
0.545
0.479
0.597
remaining challenges in this topic, such as understanding the effect of sample orderings and the robustness perturbed labels. These questions, along with understanding other perspectives of LLMs, are exciting questions for future rescarch.
7 Limitations
This work is based several assumptions, such the HMM mixture setting described in Section more specifically the assumptions in Section 3.3 Though theoretically, the HMM can be exhaustively constructed to express any finito-length distribution (c.g., by using the sequences of prefix tokens as hidden states S {[oo, .1...., os]li & o & E O)), it might. not be an elegant. model of natural language. does not fully capture the full complexity and generality of natural language with unbounded lengths, and might not be the most efficient model considering the large number of hidden states needed.
3 and
There is also controversy that challenges the Bayesian inference view of ICL behaviors. Some work such Falck et al.] (2024) is discussed in Section 2 Besides, not all ICL phenomena can be explained as Bayesian inference, such as those discussed in Section 4.2 The divergence from kernel-regression behaviors can be partly explained by the first error term in Equation
as detailed in Section 2
Acknowledgments
This work supported in part by US DARPA KAIROS Program No. FA8750-19-2-1004 and AIDA Program No. FA8750-18-2-0014 The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.
References
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn implement precon- ditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36: 45614 45650, 2023.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrevicee wett:/formmm2ddd0000eHeeeee
Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Re. Zoology: Measuring and improving recall in efficient language models. In The Twelfth International Conference on Learning Representations.
Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mci. Transformers as statisticians: Provable in- context learning with in-context algorithm selection. Advances in neural information processing systems, 36, 2024.
11

---

Published in Transactions on Machine Learning Research (09/2025)
Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou. Birth of a transformer: A memory viewpoint. Advances in Neural Information Processing Systems, 36: 36:1560-11 1588, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 1901, 2020.
Stephanic Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in trans- formers. Advances in Newral Information Processing Systems, 35:18878- 18891, 2022.
Wenlong Chen and Yingzhen Li. Calibrating transformers via sparse gaussian processes. orXiv preprint arXiv:2303.02/4 2023.
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuriing Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in- context? language models implicitly perform gradient descent as meta-oplimizers. In ICLR 2023 Workshop on Mathematicol and Empirical Understanding of Foundation Models, 2022.
Fabian Falck, Ziyu Wang, and Christopher C Holmes. Is in-context learning in large language models bayesian? a martingale perspective. In International Conference on Machine Learning, pp. 22784-12800 PMLR, 2024.
Deqing Fu, Tian-Qi Chen, Robin Jia, and Vatsal Sharan. Transformers learn to achieve second-order conver- gence rates for in-context linear regression. In The Thirly-eighth Annuua Conference on Neural Information Processing Systems, 2024.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. In Advances in Neural Information Processing Systems, 2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishck Kadian, Ahmad Al- Dahle, Aiesha Letaann Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXie:2407.21788 2024.
Tianyu Guo, Wei Hu, Song Mei, Huan Wang, Caiming Xiong, Silvio Savarese, and Yu Bai. How do trans- formers learn in-context beyond simple functions? a case study on learning with representations. In The Twelfth International Conference on Learning Representations.
Yu Huang, Yuan Cheng, and Yingbin Liang. In-context convergence of transformers. In Forty-first Interna- tional Conference on Machine Learning.
Herbert Jaeger. Observable operator models for discrete stochastic time series. Neural compulation, 12(6): 1371-1398, 2000.
Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. Reframing instructional prompts to gptk's language. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 589 612, 2022.
Juno Kim and Taiji Suzuki. Transformers learn nonlinear features in context: Nonconvex mean-field dy- namics on the atlention landscape. In Forty-forst International Conference on Machine Learning.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yulaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Newral Information Processing Systems, 2023.
Jannik Kossen, Tom Rainforth, and Yarin Gal. In-context learning in large language models learns label relationships but is not conventional learning. arXie preprint arXiv:2307.1233 2023.
Sha Li, Ruining Zhao, Manling Li, Heng Ji, Chris Callison-Bure and Jiawei Han. Open-domain hicrarchical event schema induction by incremental prompting and verification. In Proc. The 61st Annual Meeting of the Association for Computational Linguistics (ACI.2023), 2023.
12

---

Published in Transactions on Machine Learning Research (09/2025)
Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (DeeLJO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp. 100 114, 2022.
Robert Logan IV, Ivana Balažević, Eric Wallace, Fabio Petroni, Sameer Singh, and Scbastian Riedel. Cutting down on prompts and parameters: Simple few-shot learning with language models. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2824 2824-2835, 2022.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8086-8098, 2022.
Yue M Lu, Mary T Letey, Jacob A 7avatone-Vette Anindita Maiti, and Cengiz Pehlevan. Asymptotic theory of in-context learning by linear attention. arXiv preprint arXiv:2405. 11755, 2024.
Arvind V Mahankali, Tatsunri Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer of lincar self-attention. In The Twelfth International Conference on Learning Represenlations.
Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Noisy channel language model prompt- ing for few-shot text. classification. In Proceedings of the 60th Annual Meeting of the Associalion for Computational Linguistics ( (Volume 1: Long Papers), pp. 5316 5330-, 2022a.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202. 0tl.:22222118997 12837, 2022b.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209. 1895, 2022.
Core Francisco Park, Ekdeep Singh Lubana, Itamar Pres, and Hidenori Tanaka. Competition dynamics shape algorithmic phases of in-context learning. arXiv preprint ar Xii::442.010000 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. 2023.
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp, et al. Hopfield networks is all you need. Tn International Conference on Learning Representations.
Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of bay-saaynia in-context learning for regression. Advances in neural information processing systems, 36: 14228-14222 2023.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL http:://axiiv.or www.aa//....... 10084
Ruifeng Ren and Yong Liu. Towards understanding how transformers learn in-context through a represen- tation learning lens. arXiv preprint arXin:2310 13220, 2023.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. Tn Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Lònguistics: Human Language Techmologies, pp. 2655 2671, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xiv preprint or orXi::2307.09988 2023.
13

---

Published in Transactions on Machine Learning Research (09/2025)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informalion processing systems, 30, 2017.
Johannos von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsov, An- drey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. arXiv preprinl ar arXiu:221..07677, 2022.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A Billion Parameter Autoregressive Language Model. attps://githul: http://tww....omt/oo////////aaaa
jax, May 2021.
Xinyi Wang, Wanrong Zhu, and William Yang Wang. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning. arXiv preprint arXiv:2301.11916, 2023.
Zhenhailong Wang and Heng Ji. Open vocabulary clettoencohhabrraplyy--ttss decoding and zero-shot sentiment classification. In Proc. Thirty-Sicth AAAI Conference on Artificial Intelligence (AAA12022), 2022.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilitics of large language models. Trans- actions on Machine Learning Research, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022b.
Jerry Wei, Jason Wei, Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv preprint arXin:238.008846 2023.
Yuhuai Wu, Felix Li, and Percy S Liang, Insights into pre-training via simpler synthetic tasks. Advances in Newrul Information Processing Systems, 35:21844 21857, 2022.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2022.
Wanying Xie. Gx at semeval-2021 task 2: Bert with lemma information for mel-wic task. In Proceedings of the 15th International Workshop on Semantic Evaluation. (SemEval-2021 pp. 706-712, 2021.
Ruiqi Zhang, Spencer Frei, and Peter Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25(49):1-55, 2024.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving shw- performance of language models. In International Conference on Machine Learning, pp. 12697-12706. PMLR, 2021.
A Proofs
Proof. First, we denote the kernel regression function
VrecTTxmme) vevvec(Tx.) e(v)),
(10)
In expectation,
14

---

Published in Transactions on Machine Learning Research (09/2025)
Ex. Epo- voc(Tx.)) voo(Tx.) = veo(Tx Xinss -mpp Ex, voc(Tx; )voo(Txx)
voo(Txxan) xT
(11)
As vec(-) linear function, if we replace vec(Tx;;) with Tx::
Ex, (voc(Txxoo)) Epo. voc(Tx:) Tx. Txtest
(12)
and then
Ex, (vecl (xie ) E-1 vec(Txx) P(Y (07/xx,17 Po-) pai
(13)
SP(Y(S) Ex, (voo(Tx xxmms) Ez1 vec(Tx:) )) To.
(14)
=P(Y(S) T Po-
(15)
=P(Y Xtest; 0*)
(16)
As (x;, e(v:)) can be seen as independent samples from P(Y[x;;po-)) po-), we can use Hoeffding's inequality and bound that, with 1 probability,
4m
ly Ex, (vec(Txxan) (xxea))) Eng vec(Tx.)) P(Y Pa- Pa, )loo S
2n
5
Considering the difference between Epgr and Epo, we see that
(vec(Txxom) ), 2p vec(Tx;)) )) - (vec(Tximm) ) 2 vec(Txx) =|vec(Txxon )T "(iioo 2 vec(Txx))
(17)
(18)
S1eea
(19)
Therefore,
4m In
y - P( Xrest, 1 S V 2n
+ n7cc
8
Next bridge P(Y Xtest, 0*) with P(Y Prre-train). Let Stest be the hidden state corresponding to first token of Xtests i.e, Xtest,0. We see that, the likelihood of Stest Sg- is lower bounded by:
P(Sust = Sgr, Sor,Smlee Sn Sprrooro-))nnn
Σ P(Suet = Sg. Sn, 80 (S)n)ss)))PP P(se (preetra-))))))
(20)
9E8
P(Stest Stest s0- Sn, so-) P (so P Ppre-traia)
(21)
2 (S- P((80 Ppre-train)
(22)
(Markov property)
P(xx,, Vir o delimn |sa P( [xi, Vo, o deling J, (p.)) P(so. (ppretrriin
(23)
(by Assumption
P(xx, willso- Jeder )eleer))ear
P(se (pre-traia)er
(24)
P(x,, villso-)
P(sg- (preetrraidee""
(25)
15

---

Published in Transactions on Machine Learning Research (09/2025)
For another task ,' Steat is unlikely to be sør because:
P(Stest sg', SulPpre-train))
''P P(trest Sgr (Sn 9o) P(S P(sg) Ppre-train)
(26)
0E6
P(Seest = 80 Snn sp P(Su|so) P(so Ppre-train)
(27)
P(xx;, Ve, o (dcchim (1))
(by Assumption
P(sor (Pprestrai))
(28)
1=1
P P(xx, P(xx) will'')
P(ser Ppro-train)
(29)
Therefore, the Bayesian inference Stest 18:
Ptteet = 5q* OICC,, Ppre-train)
(30)
P(Srest = Sp. OICI. Prro-train)
(31)
P(orCt (pre-train)
P(Sest 86*, OICL Ppre-train)
(32)
Eo P(stest s0, OICL Ppre-train)
(P(xtest (0)) [T P(xx, vill0)) P(sø (pre-train) P(sg., O1CL Ppre-train)
(33)
-1
+ min exp In P(xrost 10) 0300
P(xx; vil10)
1 +
1
In
In
+
(n
1)
+
In
(34)
((xrest(0*)
P(x,, yi 0+)
Ed
Er
pPol
(with 1 prob.)
(35)
(-nexz. +
1 4
1
1
1 1 min exp
V=
In
+
n
+
(n
1)ln
In
(36)
| -00
ed
6r
(-nexe++ / In + nln
1
1
21 exp
+
(n
+ 1)
In
+
In
(37)
1polloo
ed
&
So that
1 P(Y Prro-trainlll
(38)
In
4m
+ +2000
+
G, c exp (-n(exz+In(eac)) (-nlexz + +In(aac)) +
V in
(39)
8
=0
(1 In
4m
+ n2²c0
(40)
8
16

---

Published in Transactions on Machine Learning Research (09/2025)
B Results on more tasks
Besides the case study SST2 dataset in Section in this scction we also provide experiment results on other tasks. In specific, we experiment on Rotten Tomatoes" Tweet Eval E (hate, irony and offensive subtasks) and M The results are as follows.
B.1 Rotten Tomatoes
Figure Averaged attention map over Rotten Tomatoes test set.
17

---

Published in Transactions on Machine Learning Research (09/202))
080
075
0.15
1
0.00
465
16
0.0
20
0.00
(a) Accuracy on reconstruction of ÿ by interpreting at- tention kernel weights.
(b) Pearson correlation between attention and logit similarity.
Figure 7: Interpreting attention values from kernerl regression perspective on Rotten Tomatocs dataset.
1.0
0.9
0.8
0.80
0.55
0.50
0.45
0.5
10
20
10
20
(a) Predicting max. P(o]x:) with key vectors.
(b) Predicting with value vectors.
Figure 8: Investigating information in key and value vectors on Rotten Tomatoes dataset.
18

---

Published in Transactions on Machine Learning Research (09/202))
B.2
Tweet Eval (Hate)
Figure 9: Averaged attention map Tweet Eval (Hate) test set.
19

---

Published in Transactions on Machine Learning Research (09/2025)
0.08
4.00
48-
0.00
0.3
22
0.8
12
14
682
14
3.00
0.0
(a) Accuracy on reconstruction of ý by interpreting at- tention kernel weights.
(b) Pearson correlation between attention and logit. similarity.
Figure 10: Interpreting attention values from kernerl regression perspective Tweet Eval (Hate) dataset.
1.0
0.70
0.65
0.55
0.7
0.6
-1
2
10
10
25
(a) Predicting arg max, P(o|x;) with key vectors.
(b) Predicting with vakue vectors.
Figure 11: Invostigating information in key and value vectors on Tweet Eval (Hatc) datasct.
20

---

Published in Transactions on Machine Learning Research (09/2025)
B.3
Tweet Eval (Irony)
Figure 12: Averaged attention map over Tweet Eval (Irony) test set.
21

---

Published in Transactions on Machine Learning Research (09/2025)
0.72
0.70
0.68
0.00
0.66
0.54
10
0.06
12
12
14
030
(a) Accuracy on reconstruction of ý by interpreting at- tention as kernel weights.
(b) Pearson correlation between atlention and logit similarity.
Figure 13: Interpreting attention values from kernerl regression perspective on Tweet Eval (Trony) dataset.
0.70
0.9
0.9
0.80
0.55
0.6
0.45
10
15
20
(a) Predicting arg max. P(o[x.) with key vectors.
(b) Predicting yi with value vectors.
Figure 14: Investigating information in key and value voctors on Tweet Eval (Irony) dataset.
22

---

Published in Transactions on Machine Learning Research (09/2025)
B.4 Tweet Eval (Offensive)
Figure 15: Averaged attention map over Tweet Eval (Offensive) test set.
23

---

Published in Transactions on Machine Learning Research (09/2025)
0.0
3.0
976
472
9.0
300
166
10
100
20
(a) Accuracy reconstruction of ý interpreting at- tention as kernel weights.
(b) Pearson correlation between attention and logit. similarity.
Figure 16: Interpreting attention values from kernerl regression perspective on Tweet Eval (Offensive) dataset.
1.3
0.70
0.65
0.8
0.50
1
20
0.45
0.5
15
15
20
25
(a) Predicting arg max, P(o[x.) with key vectors.
(b) Predicting yi with value vectors.
Figure 17: Investigating information in key and value vectors Tweet Eval (Offensive) dataset.
24

---

Published in Transactions on Machine Learning Research (09/2025)
B.5 MNLI
Figure 18: Averaged attention map over MNLI test set.
25

---

Published in Transactions on Machine Learning Research (09/202))
0.06
0023
0.05
0422
10-
10
0.00
12
12-
0.01
14
14
6,02
(a) Accuracy on reconstruction of ý by interpreting at-
(b) Pearson correlation between attention and logit. similarity.
tention as kernel weights.
Figure Interpreting attention values from kernerl regression perspective on MNLI dataset.
1.0
0.9
0.60
0.55
0.7
0.5
20
10
35
25
(a) Predicting arg max. P(o[x.) with key vectors.
(b) Predicting zi with value vectors.
Figure 20: Investigating information in key and value vectors on MNLI dataset.
26

---

Published in Transactions on Machine Learning Research (09/2025)
C Synthetic Verification
To verify the thcoretical results, conduct experiments on synthetic HMM. We experimented on randomly parameterized synthetic HMMs with 8 lasks, 80 states, and 100 observations. We vary the number of samples and evaluate the proposed kernel regression on fitting the Bayesian posterior, which the intuition in Theorem Results are listed in the Table 3 We indeed see a decreasing loss and increasing accuracy with more demonstrative examples. The loss also converges non-zero value according to Equation 7
Table 3: Accuracy and distance of predicting the Bayesian posterior on a synthetic HMM with Equation
#samples
2
4
8
16
32
64
128
distance
1.322
0.942
0.519
0.218
0.163
0.085
0.093
0.083
accuracy
0.337
0.532
0.745
0.901
0.926
0.968
0.961
0.964
27

---

Published in Transactions on Machine Learning Research (09/2025)
D Results on Llama-2 and Llama-3
To demonstrate the generality of our conclusions across model architectures, we also conduct analysis on
Llama-2 model TTuvron et al., 2023 and Llama-3 8B model (Grattafiori et al. 2024). The experiment
setting is similar to Section 5.1 and Visualizations are as follows. They generally show similar trends with Section 5
Figure 21: Averaged attention map over SST2 datasct Llama-2 model.
28

---

Published in Transactions on Machine Learning Research (09/2025)
1.00
5.0
0.75
10-
5.0
(a) Accuracy reconstruction of ÿ by interpreting at- tention as kernel weights.
(b) Pearson correlation between attention and logit
similarity.
Figure 22: Interpreting attention values from kernerl regression perspective on Llama-2 model.
Figure 23: Averaged attention map SST2 dataset on Llama-3 8B model.
E Single Datum Visualization
We also plot the visualization on a single datum as follows. It shows a similar but slightly spanser pattern to Section 5.1
29

---

Published in Transactions on Machine Learning Research (09/2025)
666
0.8
12
035
0.30
(a) Accuracy on reconstruction of 9 by interpreting at- tention kernel weights.
(b) Accuracy of predicting the ground-truth label by interpreting attention kernel weights.
Figure 24: Interpreting attention values from kernerl regression perspective Llama-3 8B model.
30

---

Published in Transactions on Machine Learning Research (09/202))
Figure 25: The attention map on a single datum
31


Reviews:
[{"review_id": "FwssMHKRtr", "reviewer": "Reviewer_nBHb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**  \nOverall, I’m a fan of the approach taken to study deep learning phenomena such as in-context learning (ICL):\n\n1. Identify a fruit-fly model to study — in this case, the framework introduced by Xie et al. (2022), which  equates ICL with Bayesian inference in an HMM mixture model.  \n2. Develop a tractable theoretical framework for this fruit-fly — specifically, showing that ICL = Bayesian inference in HMMs $\\approx$ kernel regression.  \n3. Test the resulting predictions on real transformers.  \n\nI think this is an appealing and productive way to approach the science of deep learning.\n\n---\n\n**Weaknesses:**  \n1. The narrative critiques prior work for limitations that also apply to this paper (see Requested Change 1).  \n2. The paper builds heavily on the assumptions and results of Xie et al. (2022), but this dependency is not made explicit at the outset. It only becomes very late in Theorem 1. This relationship should be clarified earlier (see Requested Change 2).  \n3. The mathematical exposition is sloppy: many many key quantities are left undefined, and notation is often inconsistent. This significantly undermines the theoretical presentation and is the main reason I responded “No” to the Claims and Evidence question (see Requested Change 3).  \n4. The literature review omits or fails to engage with works that explicitly challenge the Bayesian view of ICL in real transformers. A notable omission is Falck et al. (2024). Other relevant works may be present in the references, but they are not discussed in the context of contesting the “ICL is Bayesian” claim. This should be addressed both in the related work and in the discussion or limitations section (see Requested Change 4).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Xie et al. (2022) showed that in-context learning (ICL) can be interpreted as Bayesian inference in a mixture of Hidden Markov Models (HMMs). This theory does not involve transformers at all—the model class is never part of the analysis.\n\nThe current paper builds on that framework and contributes a further theoretical result: under the same HMM mixture model, Bayesian inference (and hence ICL) can be approximated by kernel regression. The authors then conduct extensive empirical analysis to test whether real transformers exhibit behaviors consistent with this kernel regression perspective.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Each of my requested changes corresponds to a weakness noted above:\n\n### **Requested Change 1:**  \nThe authors rightly point out that many ICL papers study setups involving explicit in-context training, which differs from how ICL emerges in pretrained LLMs. However, the HMM mixture model used here is itself a significant simplification that likely does not capture the full complexity of natural language pretraining. The paper would be stronger if it acknowledged this limitation more explicitly and softened critiques of prior work accordingly.\n\n### **Requested Change 2:**  \nThe manuscript needs to be more upfront about its foundational reliance on Xie et al. (2022). Currently, the reader must infer this connection gradually. The paper should state clearly and early that it presupposes the ICL = Bayesian inference view, and that its novel contribution is to approximate that inference via kernel regression. Page 1 is a natural place to make this framing explicit—particularly where the authors claim that Xie et al. fail to explain how such inference is implemented in practical models. That’s an opportunity to clearly position this paper’s contribution.\n\n### **Requested Change 3:**  \nSee the separate list I provide below on issues with the mathematical exposition, including undefined quantities, inconsistent notation, and unclear mappings between the main text and appendix.\n\n### **Requested Change 4:**  \nConsider adding a paragraph or subsection in the related work discussing challenges to the Bayesian interpretation of ICL. In addition, the limitations section should acknowledge that not all empirical observations of ICL in large language models are consistent with Bayesian inference—and point to the relevant literature.\n\n\n### **Requested Change 3: Mathematical and Notational Issues**\n\n- $o_{ICL}=[S_n, x_{test}]$ is defined in (2) but the left-hand-side is never used again and the more verbose right-hand-side is frequently used\n- I don't know what \"emission rate lower bound $\\epsilon_r$\" is referring to nor \"transition rate lower bound $\\epsilon_d$\"\n- $T_x$ is used in Equation (4) but never formally defined. It seems connected to notation introduced in Section 3.2, but the reader should not have to guess. \n- The construction of $T_{[S_n, x_{\\text{test}}]}$`at the bottom of Page 5 is never specified and does not match the notation used in $T_x$, nor in $T_{o[0:t-1]}$\n- There is an implication that the quantity in Eqn 6 $P(y|[S_n,x_{test}])$ is the \"Bayesian posterior\". I don't think that's what you mean. You mean the Bayesian posterior predictive density? What is the likelihood? What is the prior? This is the main result yet you leave this important quantity undefined. \n- The expression $p_{\\text{pre-train}}^\\top T_{[S_n, x_{\\text{test}}]}$`appears at the end of Page 5. But you had previously defined $p_{\\text{pre-train}}$ as the pretraining distribution. I have no idea what it means to take the transpose of a distribution and multiply (?) it to $T_{[S_n, x_{test}]}$\n- There is a sentence in 3.1 that begins with mathematical notation $p_{\\text{pre-train}}$. Because the preceding sentence ends with a mathematical notation, the period makes it look like you are multiplying the adjacent quantities. \n- Appendix A which contains the proof of Theorem 1 uses notation that does not match the main exposition. Just to give one example, you use \n$$\nP(y=Y | [S_n, x_{test}], p_{\\text{pre-train}})\n$$\nIs this the same as $P(y=Y | [S_n, x_{test}])$ in the main text? Also, isn't it more typical to write $Y=y$ rather than $y=Y$?\n- Eqn 29 in Appendix A is empty\n- What happened to $x_{test}$ between Eqns 30 and 31? Why did it disappear?\n\nNote that this is not an exhaustive list of the many typos, inconsistencies in your paper.", "broader_impact_concerns": "I do not have broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "R7wbasSCPo", "reviewer": "Reviewer_T8XN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main idea of the paper is interesting, however I have reservations regarding the generality of its claims. In particular, as I understand, work by Raventos et al. ([NeurIPS 2023](https://proceedings.neurips.cc/paper_files/paper/2023/file/2e10b2c2e1aa4f8083c37dfe269873f8-Paper-Conference.pdf)) and Liu et al. ([arXiv 2024](https://arxiv.org/abs/2405.11751)) on ICL of simple linear regression tasks that show that the behavior of a pretrained Transformer is not always well-described by kernel regression. As far as I can see, these works are not cited in the submitted manuscript. \n\nBriefly, those works consider a setting in which the model is pretrained on a dataset of $d$-dimensional ICL linear regression examples where the target vectors are drawn from a limited set $\\{\\beta_{1},\\ldots,\\beta_{k}\\}$ formed of $k$ samples from $\\mathcal{N}(0,I_{d})$. Then, the model is tested on an ICL task with a target vector freshly drawn from $\\mathcal{N}(0,I_{d})$. Then, assuming additive Gaussian label noise, Raventos et al. show that the Bayes-optimal estimator for the pretraining task is *exactly* a kernel smoother, which they refer to as the discrete minimum mean squared error (dMMSE) estimator. One may then compare how the ICL error of the dMMSE estimator compares to a pretrained Transformer as a function of the pretraining task diversity $k$. Both Raventos et al. and Liu et al. showed that pretrained Transformers significantly outperform the dMMSE estimator. Roughly speaking, the dMMSE kernel smoother suffers from the curse of dimensionality and thus requires $k = \\exp(\\Theta(d))$ to achieve nontrivial generalization, while the transformer only requires $k = \\Theta(d)$. \n\nThose results are in tension with the claims of the present work. The setting of Raventos et al. and Liu et al. is consistent with the present work in that they assume tasks for which the Bayes-optimal estimator for the pretraining task is (perhaps asymptotically) kernel smoothing. However, they show that pretrained Transformers show far stronger generalization performance in the sense of ICL. How to square those results with the authors' claims is my main immediate concern with the manuscript. More broadly, the above issue reflects what I feel is a conceptual gap in the paper as it stands: the Bayes-optimal estimator for the pretraining task need not be a good in-context learner.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes that in-context learning in Transformer language models can be understood from the perspective of kernel smoothing. Its main results are (1) a theorem showing that the Bayes posterior for an HMM pretraining task can be approximated by a kernel smoother and (2) a series of experiments aiming to leverage this observation to illuminate the behavior of pretrained transformers.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The authors must somehow reconcile their findings with the works of Raventos et al. and Liu et al. mentioned under **Strengths and Weaknesses**. This is for two reasons my main concern: first it suggests that Transformers do not always behave as Bayes-optimal kernel smoothers, and second it suggests that it would not necessarily be good for ICL performance if they did so. \n\n- The selection of citations in Section 2.3, \"Associating Attention with Kernel Regression\", is not very comprehensive. I would suggest the authors reference also Ramsauer et al. ([ICLR 2021](https://arxiv.org/abs/2008.02217)) and recent work by Bietti et al. (e.g. [NeurIPS 2023](https://arxiv.org/abs/2306.00802)). This line of work on associative memory is closely related to the issue of associative recall in ICL (see e.g. Arora et al., [ICLR 2024](https://openreview.net/forum?id=LY3ukUANko)), which is mentioned in Section 4.2 of the authors' manuscript. It would be useful if the authors could connect their interpretations and results to the associative memory framing. \n\n- The authors might find recent empirical work from Hidenori Tanaka's group on interpreting how Transformers solve ICL tasks based on learning from mixtures of Markov chains, e.g. Park et al. [ICLR 2025](https://arxiv.org/abs/2412.01003), to be of at least tangential interest.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YncSCX2GtU", "reviewer": "Reviewer_wmEB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper addresses a promising and important direction for improving the interpretation and understanding of in-context learning (ICL) in large language models (LLMs).\n\n2. The kernel regression perspective offers an intuitive bridge between theoretical models and practical observations, potentially enriching the broader theoretical landscape for ICL.\n\nWeaknesses:\n\n1. While there is a growing body of theoretical research explaining ICL through Bayesian inference and other frameworks (e.g., generalization theories, martingale approaches), the paper only briefly mentions a few earlier works (mostly from 2021–2023) and does not adequately compare against or distinguish itself from these existing studies. A more thorough comparison and clearer articulation of the paper's novelty relative to prior work would strengthen its contribution.\n\n2. The theoretical insights offered by the kernel regression view are somewhat limited in depth. The explanation remains relatively high-level, and could benefit from deeper analysis or more nuanced interpretations. Additional empirical evidence supporting the soundness and practical implications of the proposed insights would be valuable.\n\n3. The empirical evaluation is relatively limited. The experiments lack comprehensive quantitative analysis and broader comparisons across a range of LLMs, especially more recent state-of-the-art models. Moreover, the study could be improved by exploring different ICL settings and tasks to better validate the generality and robustness of the proposed perspective.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a kernel regression perspective to explain the emerging in-context learning capabilities of pretrained large language models (LLMs). The authors model in-context demonstrations as a form of Bayesian inference, and further establish that this process can be interpreted through the lens of kernel regression. Building on this theoretical framework, the paper provides empirical experiments and analysis that demonstrate how attention mechanisms in LLMs can exhibit behaviors analogous to kernel regression. Furthermore, the work draws connections between their theoretical findings and practical aspects of LLM applications, such as retrieval-augmented generation and formality control.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "1. It would strengthen the paper to include a broader and fairer comparison with more recent works that leverage Bayesian processes to interpret in-context learning in LLMs, as well as other related theoretical frameworks (e.g., generalization-based, martingale-based analyses). Highlighting the key similarities and differences would help clarify the novelty and positioning of the current contribution.\n\n2. The current experiments and analyses are not sufficiently solid to fully support the theoretical claims. It is encouraged to include more comprehensive and quantitative evaluations, such as testing across different LLM architectures (especially more recent state-of-the-art models) and various ICL settings, to substantiate the soundness and practical relevance of the proposed theory.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)