In-Context Learning as Conditioned Associative Memory Retrieval
Authors: Weimin Wu
Teng-Yun Hsiao
Jerry Yao-Chieh Hu
Wenxin Zhang
Han Liu
Venue: ICML 2025 poster (2025)
Keywords: In-Context Learning
Large Language Model
Foundation Model
Transformer
Attention
Modern Hopfield Model
Associative Memory
Primary area: deep_learning->foundation_models
TL;DR: We provide an exactly solvable example for interpreting In-Context Learning (ICL) with one-layer attention models as conditional retrieval of dense associative memory models.

Abstract: We provide an exactly solvable example for interpreting In-Context Learning (ICL) with one-layer attention models as conditional retrieval of dense associative memory models.
Our main contribution is to interpret ICL as memory reshaping in the modern Hopfield model from a conditional memory set (in-context examples).
Specifically, we show that the in-context sequential examples induce an effective reshaping of the energy landscape of a Hopfield model.
We integrate this in-context memory reshaping phenomenon into the existing Bayesian model averaging view of ICL [Zhang et al., AISTATS 2025] via the established equivalence between the modern Hopfield model and transformer attention.
Under this unique perspective, we not only characterize how in-context examples shape predictions in the Gaussian linear regression case, but also recover the known $\epsilon$-stability generalization bound of the ICL for the one-layer attention model.
We also give explanations for three key behaviors of ICL and validate them through experiments.

Teng- Yun Hssioo <b Cheih Hu hww.uuuu.northwwsteee... edu>, Wenxin Zhang Han Liu In-Context Learning as Conditioned Associative Memory Retrieval
Weimin Wu Teng- Yun Hsiao Jerry Yao-Chieh Hu 12 Wenxin Zhang?
Han Liu 124
Abstract
associative memory models (Krotov, 2023). With the suc- of large language models (Guo al., 2025; Grattafiori et al., 2024; Jiang et al., 2024a), ICL enables large language models to adapt to diverse tasks by leveraging task speeispec examples embedded in the input prompts, without updat- ing model parameters (Min et al., 2022: Garg et al., 2022; Brown et al., 2020). This "learning in the forward pass" raises fundamental questions about how models represent and reuse knowledge, and challenges our understanding of how a fixed model can implement new task behaviors on the fly (Von Oswald et al., 2023; Akyürek et al., 2022; Xie et al., 2022). Understanding that mechanism is theoretically compelling and practically valuable: better mental mod- els of ICL. can guide prompt construction and improve the reliability of LLM-based systems or Transformer models in general (Zhang al., 2025; Wu et al., 2025; Han et al., 2024; Bai et al., 2023; Li et al., 2023; Xie et al., 2022).
We provide an exactly solvable example for inter- preting In-Context Learning (ICL) with one-layer attention models conditional retrieval of dense associative memory models. Our main contribu- tion is to interpret ICI. as memory reshaping in the modern Hopfield model from a conditional memory set (in-context examples). Specifically, we show that the in-context sequential examples induce an effective reshaping of the energy land- scape of Hopfield model. We integrate this in- context memory reshaping phenomenon into the existing Bayesian model averaging view of ICL. [Zhang et al., AISTATS 2025] via the established equivalence between the modern Hopfield model and transformer attention. Under this unique per- spective, we not only characterize how in contxtte examples shape predictions in the Gaussian lin- ear regression case, but also recover the known e-stability generalization bound of the ICL for the one-layer attention model. We also give explana- tions for three key behaviors of ICI. and validate them through experiments.
In this work, we approach ICL through the lens of asso- ciative memory (Krotov, 2023; Krotov & Hopfield, 2020; 2016; Hopfield, 1982; Amari, 1972). In particular, we draw inspiration from transformer--compatbbe modern Hopfield networks (Ramsauer et al., 2020). These networks are continuous- dense associative memories with an ergy landscape that stores memory patterns as local minima. Recent studies establish formal equivalence between the transformer's 's attention mechanism and the update dynamics of a modern Hopfield network (Santos et al., 2024a;b; Hu et al., 2024b; Wu et al., 2024b; Krotov & Hopfield, 2020; Ramsauer al., 2020). In a Hopfield network, a query pattern is iteratively matched the closest stored memory pattern (a local minimum of the energy function). Analo- gously, a transformer's 's attention module can be viewed as retrieving a stored vector (from the key-value pairs) that best matches a query embedding. Consequently, training attention weights can be viewed as shaping the Hopfield energy landscape so that training examples become stable memories (Wu et al., 2024a; Hu et al., 2024a). This per- spective suggests that a transformer may be remembering and retrieving relevant information from its weights when performing ICL, akin to an associative memory recalling patterns in response to cues. This unique perspective re- casts prompt-based inference as a memory retrieval process, potentially offering more concrete mechanism to explain how in-context examples influence model predictions.
1
Introduction
We show that In- In-Context Learning (ICL) reshapes model's internal associative memory using prompt examples, analo- gous to how a modern Hopfield network adjusts its stored memories in response to new cues. To be precise, we pro- vide an exactly solvable example for interpreting ICL in one-layer attention models as conditional retrieval of dense
'Equal contribution 'Center for Foundation Models and Gen- erative AI, Northwestern University. USA Department of Com- puter Science, Northwestem University, USA DDpartment of Physies, National Taiwan University, Taiwan 'Department of Statistics and Data Science, Northwestern Universit, USA. Corre- spondence to: Weimin
2cb10002008entu.edd tw>, Jerry Yao-
www.nx.nehangg225eu..oooot <hanlli@@northweestenn.cc
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

In-Context Learning as Conditioned Associative Memory Retrieval
Building this connection, we posit to interpret ICL as conditioned memory retrieval process in dense asso- ciative memory model (in particular, modern Hopfield model). Surprisingly, when each in-context example acts as a query to the Hopfield model, we show that the sequence of examples induces an effective reshaping of the energy landscape: attractors relevant to the prompt move closer to the query. while irrelevant ones recede. Although the perma- nent weights are unchanged, the effective memory is biased toward the task at hand, so the final test query retrieves in- formation consistent with the demonstrated patterns. This in-context memory reshaping explains why in-context learn- ing improves prediction without fine-tuning.
(iii) We give explanations for three key behaviors of ICL and validate through synthetic and real datasets. These behaviors include sensitivity to covariance shifts, sensitivity to response's 's accuracy, and sensitivity to the similarity between prompts and test query.
In sum, we interpret ICL as conditioned associative memooy retrieval. The resulting theory and experiments provide unified and concrete account of how prompting examples modulate transformer's computations on the fly.
Organization. Section 2 introduces the background on large language models, in-context learning, memory reshap- ing in modern Hopfield models, and related work. Section 3 shows the details of our proposed in-context memory reshap- ing phenomenon. Section 4 provides experimental studies. Section 5 provides discussion and conclusion.
Concretely, we analyze an exactly solvable one-layer at- tention model. The model reveals that memory reshaping corrssonnd to linear transformation of both key and value matrices determined by the prompt, and it links this mech- anism to the Bayesian model averaging view of in-context learning (Zhang et al., 2025). Thus, we bridge an abstract Bayesian statistical explanation with analytic network dy- namics from neuroscience. Under this unique perspective, we derive new insights and recover key results about ICL.
Notations. We denote the set {I...., NN as [N]. For a Polish space S, let (SS) denote the set of all probability measures over S. For matrix M E Rd xd2 (d1, d2 E N+),
we define the 1p.g norm as || = i [lM,11)) where M, E Rd denotes the i-th column of M. Let c, denote the query and response. For simplicity, we consider the setting where both e and are single tokens.
Contributions. Our contributions are as follows:
In-Context Memory Reshaping. We present an exact formulation of in-context memory reshaping for ICL using one-layer attention model. Under this stylized setting, we prove that an input prompt of in-context examples induces an implicit linear mapping the model's 's pre-trained attention weights (comparing sce- narios without and with prompt examples). Effectively, this reshaping modulates the model's stored knowl- edge without any parameter updates. Together, these insights offer a model-based explanation for ICL's in- ternal mechanism and its effects in a simple attention.
2 Background and Related Work
In this section, we provide background on the following topics: (i) the formulation of ICL in large language models in Section 2.1, (ii) existing theory for ICL based on generic latent variable models in Section 2.2, and (iii) memory re- shaping in modern Hopfield models in Section 2.3. We also review related work in Section 2.4.
2.1 In- Learning in Large Language Model
Let Pa be a pre-trained large language model with param- eters 8. We consider the scenario where both queries and responses are single tokens in the large language model. Let E Rdd and r E R4, denote the query and re- sponse. Given T E N+, for any t E [T], we denote the sequence of t prompt examples (query-response pairs) as D. == (ci,r.) Let C. = [c1,...,cc] E TR. xt and R2 = [r]....,rt] Rgd.xt In ICL, the input to the pre- trained large language model is the prompt P. (De, C++1), where 4++1 is the (L 1)-th query, also known as the test query. The large language model outputs a predicted re- sponse for Cz+1 without further parameter updates. The corresponding true label is r++1-
Link to Bayesian Theory. We embed this in-context memory reshaping phenomenon into the Bayesian model averaging framework for ICL (Zhang et al., 2025). This connection yields a solvable example that complements existing Bayesian theory of ICL and high- lights how memory reshaping provides a complemen- tary, mechanistic interpretation of the same behavior.
New Insights and Validations. We derive and vali- date new insights: (i) In Gaussian linear regression, prompts steer predictions toward the Bayes-optimal solution by dynamically ce-weighting stored memo- ries. This serves as an explicit characterization of how prompts "guide" predictions. (ii) We recover the known e-stability bound for ICL (Li et al., 2023) within one-layer attention. This confirms that models follow- ing our memory retrieval interpretation achieve the same established generalization guarantees for ICL.
2.2 A Generic Latent Variable Model for ICL
To interpret ICL, Zhang et al. (2025) define a function f that maps query, latent variable, and noise to response:
ri = f(i, his Ct),
(1)
2

---

In-Context Learning as Conditioned Associative Memory Retrieval
where h; latent variable connecting q and r,, and S are independently and identically distributed random noise for
With the energy function (3), the retrieval dynamics is
each i E [Z]. They also introduce a hidden concept Z, from the space 2. The hidden concept represents a shared property across the prompt examples. Based on this concept, the evolution of h2 follows the stochastic process:
T(x)) = SSotmaa (8E 2x) = -++1-
(4)
Recent studies establish formal equivalence between the transformer's 's attention mechanism and the update dynamics of a modern Hopfield network (Hu et al., 2024b; Wu et al., 2024b; Ramsauer et al., 2020).
P(h: Ci. (aatesss) Hz (ths)ses))s)) 6i),
(2)
This guarantees that the memories [EM] coincide with the stationary points of E(x), because (4) arises from the stationary condition of the Procedure (CCCP) (Yuille & Rangarajan, 2001).
Remark 2.2.
for some function Hz and exogenous noises (er)
This indicates that the distribution of h; depends both the hidden concept Z and the prior observations, while it is independent of c,.
Memory Reshaping. Memory reshaping in modern Hop- field models refers to modifying the Hopfield energy land- scape by relocating its local minima, which encode stored memories. A simple approach to achieve this is by applying linear transformation to the memory patterns, as demon- strated by Wu et al. (2024a, Eqn. (2.1)). They show that a more uniform distribution of memory representations (i.e., local minima) reduces the number metastable states in the energy function, thereby enhancing memory capacity and reducing memory confusion. In our work, we analyze exactly solvable one-layer attention model, and interpret ICL memory reshaping in the modern Hopfield model from conditional memory set (in-context examples).
The model in (1) assumes that the hidden concept Z" implic- itly determines the transition of the conditional distribution P(r: = (4) by affecting the evolution of the latent vari- ables {1e Nejrj. without making any assumptions on the distribution of c,.
Remark 2.1. The model is general and encompasses previ- ous models, such as the hidden Markov chain (Xie et al., 2022) and the causal graph (Wang et al., 2023b).
Then Zhang et al. (2025) formulate ICI. as follows.
Lemma 2.1 (Proposition 4.1 of (Zhang et al., 2025)). Under the model in (1), it holds
P(r+11 = P.) / dZ 10.-P(r-1::0 = P,, Z) PZ D.).
2.4 Related Work: In-Context Learning
Several works aim to enhance understanding of ICL from both experimental and theoretical perspectives.
Proof. Please see Appendix C.I for a detailed proof.
Empirical Analyses. Empirically, Zhang et al. (2024) and Min et al. (2022) investigate how covariance shifts affect ICL performance. Yoo et al. (2022) demonstrate that the correctness of input-label mappings is critical for down- stream ICL success. Further, Liu et al. (2022) and Rubin et al. (2022) find that increased similarity between prompt examples and test instances enhances ICL performance.
2.3 Memory Reshaping in Modern Hopfield Model
In this part, we introduce modern Hopfield models and re- view their equivalence to transformer attention mechanisms. We then highlight how memory reshaping in modern Hop- field models corresponds to training transformer attentions.
Modern Hopfield Model. Let x E Rd represent the query pattern, and =- : K1,- {MM] RdxM denote the M stored memory patterns. The objective of modern Hopfield model is to encode memory patterns E into its energy land- scape and retrieve a memory pattern E, based on a given query x. Following Ramsauer et al. (2020), the energy function for modern Hopfield model is
Theoretical Perspectives. Theoretically, several studies explain ICL in attention based models from the gradient de- scent viewpoint (Von Oswald al., 2023; Mahankali et al., 2023; Akyürek et al., 2022). In addition, Li et al. (2023) frame ICL as an algorithm learning problem and provide generalization bounds, while Bai et al. (2023) show that ICL performs an implicit algorithm selection based on input sequences. different line work adopts a Bayesian frame- work: Wies al. (2024), Wang et al. (2023a). Jiang (2023), and Xie et al. (2022) all use Bayesian models to formalize ICL. Notably, Zhang et al. (2025) propose a Bayesian model averaging theory to interpret ICI. We incorporate our pro- posed in-context memory reshaping phenomenon into their theory. As a result, we provide an exactly solvable example for interpreting ICL with one-layer attention models and complement their results. Further extending the Bayesian
(xx) (3, x) TTx,
where 3 is scaling factor, and Ise(3, (3, (T, is the log-sum- exp function defined by
M exp Eex((BETx)
1se( -. Tx)
log
Σ
3

---

In-Context Learning as Conditioned Associative Memory Retrieval
perspective, Han et al. (2024) propose an encoding-decoding mechanism for ICI..
by another pre-trained linear layer of with V² being a linear transformation by the pre-trained W matrix. Then, for any R₁, we have V; = [v1,--. vr] E gd.xt, To calculate the prediction Ut+1 for the test query C1+1, we follow (Zhang et al., 2025) and use
A parallel line of work interprets transformers and ICL through the lens of associative memory. Jiang et al. (2024b) examine context hijacking and show that retrieval of facts in large language models is fragile via an associative mem- ory model. Building on this, Wang & Sato (2024) analyze the interaction between global and in-context knowledge to illustrate how large language models leverage global knowl- edge in ICL tasks. Bietti et al. (2023) also study ICL from an associative memory perspective. They focus on two-layer transformers, and find that single-layer transformers cannot perform ICL, while two-layer models develop induction heads and succeed. However, their analysis is based on bigram data and does not address the generalization error reduction of ICL. Finally, Cabannes et al. (2024) character- ize transformer mechanisms as associative memories and derive scaling laws for large language models. They empha- size scaling behavior, while our work concentrates on the mechanism of ICL itself.
Ut+ = V2 Softmax (K₃ q+11, where q++1 = k- (c(+1).
(5)
Remark 3.1. Notice that (5) does not represent the actual attention in the LLM. Instead, it serves as a proxy attention mechanism for ICL, based on the Bayesian model averaging formulation in Zhang et al. (2025). It provides analytical feasibility for characterizing ICL. By proxy, we refer to the fact that is an effective attention mechanism derived by framing the ICL as a conditional mean estimation problem.
The resulting estimator approximates (5) under large t limit (Zhang et al., 2025).
With Prompt Po = (4441)- To align with the expression of the attention mechanism in (5) and to characterize the effects of ICL, we need an equivalent formulation of (5) for scenarios where no in-context prompt examples are included in the input, i.e., Po = (Ce+1). We achieve this by two steps:
3 Main Theory
In this section, we provide an exactly solvable example for interpreting in-context learning with one-layer attention models. First, we derive a memory reshaping formulation of the one-layer attention model under ICL (Section 3.1). We then integrate the proposed in-context memory reshaping into the existing Bayesian model averaging theory of ICL proposed by Zhang et al. (2025), thereby complementing their results and highlighting the advantages of our perspec- tive (Section 3.2). Finally, we characterize how in-context examples "guide" predictions in the Gaussian linear regres- sion and recover the known e-stability generalization bound of the ICL for the one-layer attention model (Section 3.3).
First, we define (5) with only test query C++1 as
0++1 == W0
(6)
Second, we prepend Po with i ghost examples (denoted by {2, F. (t)) such that "(5) with the ghost prompt P. == (17, Fetsett C4+1)" has the same output as "(5) with only test query ce-1",ie.. (6). We construct such ghost examples (č, Jee by formulating an optimization problem with parameters (G,, Fe)e The objective of this optimization problem is measure the discrepancy between outputs gen- erated with Po and P.. Refer to experiments in Section 4.2. The solution to this problem gives the t ghost examples. An example of this type of problem is prompt tuning, notable for its universality (Wang et al., 2024).
3.1 Memory Reshaping for In-Context Learning
In this part, we consider each in-context example as query to the Hopfield model. Then we show that the sequence of examples induces an effective reshaping of the energy land- scape. Specifically, we analyze a one-layer attention model, and illustrate the in-context memory reshaping process by comparing the attention mechanism with and without in- context prompt examples. As result, derive an explicit formulation of memory reshaping as a linear transformation of both key and value matrices determined by the prompt.
Remark 3.2 (Ghost Example). We use ghost examples to substitute the real examples in ICL, allowing us to compare scenarios with and without in-context prompt examples while maintaining the same input structure. This setup highlights how the attention mechanism facilitates ICL.
Third, we denote the generated examples as
= -ē....... ,āl, R. = ,Fel
With Prompt P. = (Ce, Re, c.+)). We consider the case when we have prompt examples for ICL. To align with practice, we map query G E Rd to key k d by pre-trained linear layer k*. Specifically, ke is the linear transformation by the pre-trained W matrix in the attention head. Then, for any given Ct, we have K,- = [k,,-. ... kd] E xt, Similarly, we map response r, E _d to U. E Rd.
where o E Rdc, Fg E Rd., Using Ty = k* (2,), e² (7t), we have
K. = ki, Ke V. = ( --....., %..
To calculate T441 for query C++1, we use
To = V. Softmax (TT g0-))
4

---

In-Context Learning as Conditioned Associative Memory Retrieval
In-Context Memory Reshaping. Comparing (5) and (7), we interpret the two cases forward passes of an equivalent attention head with reshaped pre-trained attention weight matrices. Under this perspective, in-context learning corre- sponds to applying linear transformations to K2, V2 using matrices W1 E Id. xd., W2 E IR xd, respectively, i.e.,
averaging theory of ICL (Zhang et al., 2025), and provide exactly solvable example complement the results in (Zhang et al., 2024). We employ the same Gaussian linear model to clearly illustrate how the attention mechanism performs ICL following (Zhang et al., 2025):
Ut+1 Z" (k* (c(+1)) + T
(11)
K, = K, V = W.V.
(8)
where @ : RRd + Rdd denotes the feature mapping in reproducing kernel Hilbert space, Z E Rddxxd denotes the hidden concept, and 7/-- E (O, (o, with 0, > 0 are independently and identically distributed Gaussian noises. Besides, we assume that Z follows Gaussian distribution (0, Ae,, x By incorporating the specific attention mapping Ae and the memory reshaping formulation (9), we obtain the following for the Gaussian linear model:
K, and V3 are usually rectangular singular matrixs (Feng et al., 2022). Thus, we use Moore-Penross pseudoinverse to denote the solution to (8). Specifically, we obtain:
W1 = K,K,J W2 = V.V."
In this way, the ICL reshapes (k* ; v*) to (Wi o k", W2 through the linear transformations, where denotes the function composition.
U/+1 = ~ (WWC. Wi(e+) + Th+1- (12)
Remark 3.3. In (8), we assume a linear transformation from K, to Kr, and from V, to V, However, this approach breaks down when K, and V are low-rank, as they lack the capacity to express full-rank transformations. In such cases, using the pseudoinverse only minimizes the approximation error but does not yield an exact reconstruction.
By Lemma 2.1, we have the following result:
Proposition 3.1 (Bayesian Model Averaging in Gaussian Linear Model, Modified from (Zhang et al., 2024)). By (12),
Lemma 2.1 implies
P(u++1 | Pt) = | dz P((1+1 Z, Z,q++1
We now incorporate the specific form of the attention mechanism to compute W1 and W2, using the mappings k* (c,))) = a ex (r::) W. Under this formulation, we show that ICL is equivalent to applying the following memory reshaping on the memory set, i.c., the key and value matrices:
Di)
-|lut+1 =Z
x exp
: (13)
25,
where
W - C. (WRC.) Wk.
K, = WWC(WW
1/2 WGR((W,R))WWWB
W3 Wi Re Ri) Wy.
(9)
G++1 WWGGWN(())W2HH
Therefore, the ICL prediction in (5) with / in-context exam- ples becomes:
7,
)T-((())
Σz, = X 3
U141 = Wi RS Softmax (WiC.) (10)
Σ, = of + ( Σ 22
where
Proof. Please see Appendix C.2 for detailed proof.
W = WWC((WRC)W WV = (WG
Remark 3.4. In Proposition 3.1, 2, (9(+1) measures the similarity between the query and keys. This is similar to the attention mechanism. However, the normalization of similarity is not softmax. This motivates us to define a new structure of attention and explore the relationship between the newly defined attention and the original one.
In this way, we provide the first exactly solvable example for interpreting ICL with lne-l attention models, by deriving the in-context memory reshaping formulation based on linear transformations.
Remark 3.5. 2, (9++1) represents the weighted average of values {ox Jeeq in V,, As increases, Z, converges to a more stable and robust hidden state.
3.2
Memory Reshaping as a Complement to Bayesian Model Averaging in ICL (Zhang et al., 2025)
To connect with the actual transformer attention mecha- nism, we restate the variant of attention (Bayesian model averaging attention) defined in (Zhang et al., 2025).
In this part, we integrate our proposed in-context memory reshaping phenomenon into the existing Bayesian model
5

---

In-Context Learning as Conditioned Associative Memory Retrieval
Definition 3.1 (Bayesian Model Averaging Attention (Zhang et al., 2025)). 7z0((++1) measures the similarity between the query and the keys, similar to the role of the at- tention mechanism. We restate the newly defined variant of the attention mechanism in (Zhang et al., 2025) as follows:
Bayes-Optimal Prediction of Gaussian Linear Regres-
sion. For the Gaussian linear model in (11), we character- ize the bayes-optimal prediction using Proposition 3.1:
2. = V1(o)> ( (K)))(())))) " "K.)),
where
A(g, K2, Vi) = V fop K(K, ,})) -1 K(K, :9), (14)
K, = WWG
V (WG W.
where k(:,-) denotes the kernel function of the reproducing kernel Hilbert space induced by Φ.
This formulation shows how prompts steer predictions toward the Bayes-optimal solution by dynamically re- weighting stored memories. This serves as an explicit char- acterization of how prompts "guide" predictions.
Next, we restate the results in (Zhang et al., 2025) show that the attention in (14) is equívalent to softmax attention.
In- Mo-Context Learning Generalization Bound. Tx+1 be the true response corresponding to the query C++1- Then the true target for prediction in (10) is W We de- fine the loss function for predicting 4++1 as L(D,, = [(1++1, re+)),, where L is task-specific loss function, e-g., mean squared error loss. The generalization error of in-context learning with prompt Dr and test query c++1 is R(D,) = Ee,1 [L(D,, (++), The empirical risk is Remp (D,) (D.) when each C; in the prompt acts as the test query.
Assumption 3.1. Assume key-value pairs (ke, V are independently and identically distributed. Further, assume that each vector satisfies || ||,i|l2.2 = I, and that any query q E IR also satisfies |a||2, 1.
Remark 3.6. Assumption 3.1 further assumes that (ct,re)T= are independently and identically distributed. However, this assumption may not hold for the ghost ex- amples due to dependencies introduced by the joint opti- mization (Section 3.1). To mitigate this issue, we propose introducing a covariance-based regularization term that en- courages the covariance matrix of the ghost examples to approximate the identity matrix.
1/t E I(De, cz), i.e., the average loss
We make the following assumptions on the boundedness of the weight matrices and the inputs, as well as the Lipschitz continuity of the loss function L.
Lemma 3.1 (Bayesian Model Averaging Attention Approx- imates Softmax Attention, Proposition 4.3 of (Zhang et al., 2025)) ). For any absolute constant C' > 0, we have:
Assumption 3.2 (Norm Bounds). Assume the following boundedness conditions hold:
(( Wxlla.2 ( ST, [[all22 S 1. Further assume that e; and r; lie in the unit ball, i.e.,
lim A(g, Ke, Vi) C. lim Vz Softmax(K]) q). E-+00
1-000
||Cell,, |Rell l|c_-ll1 $. Assumption 3.3 (M-Lipschitz of L))
Proof. Please see (Zhang et al., 2025, Appendix E.4) for a detailed proof.
[(01, Virue) - L(V2, Vone)] S M Ml 02ll2.
Remark 3.7. That is, when the number of prompt examples goes to infinity, the attention in Definition 3.1 is equal to Softmax--ttention up to a constant.
Remark 3.9. The three assumptions are reasonable. As- sumption 3.1 and Assumption 3.2 conform to most real- world situations due to the layer normalization in the Trans- former architecture and the regularization loss term used in the model training (Li et al., 2023; Xiong et al., 2020; Lewkowyez & Gur-Ari, 2020). These mechanisms help ensure numerical stability and prevent overfitting, making the assumptions practically applicable. In addition, Assump- tion 3.3 is a common assumption in machine learning theory (Li et al., 2023; Edelman et al., 2022).
Remark 3.8. We use memory reshaping to demonstrate how prompt examples influence model performance, thereby complementing the results in (Zhang et al., 2025). We interpret this effect as linear transformation applied to the pre-trained attention weights.
3.3 Memory Reshaping for Characterizing Gaussian Linear Model and ICL Generalization Bound
Theorem 3.1 (Generalization Bound of ICL. for One-Layer Attention Model). The generalization bound of ICL.for the one-layer attention model is follows:
In this part, we use our proposed in-context memory reshap- ing phenomenon to characterize how in-context examples shape predictions in the Gaussian linear regression and re- cover the known e-stability generalization bound of the ICL for the one-layer attention model.
[R(D.) - Remp((D) MC.(4T + 1)
E. Dr
where Co > is an absolute constant.
6

---

In- Cn-CCnteet Learning as Conditioned Associative Memory Retrieval
1.0
1.0
1.0
0.8
0.8-
R.uuurre
0.6
0.6
0.4
N12. II
0 9MZ,. 11-0.1M2.) I)
0.4
Acc 100%
Acc 90%
0.4
Normal Similartty
High Similarity
0.2
7M22 11-0.3M22.0
0.2
Ace 80%
0.2
Similerity
0.0
Least Souares
0.0
Loast Squares
0.0
Least Squares
25 In-context Examples
50
50
75
75
In-context Examples
In-context Examples
(a) Covariance Shift
(b) Response's ' Accuracy
(c) Similarity with Test Query
Figure 1. Behaviors of In-Context Learning Measured by R-Squared in Linear Regression Problem. The true response to the query is generated by a linear model. (a) shows the performance of ICL under different covariance shifts. (b) shows the performance of ICL under different accuracies of responses in prompts. (c) shows the performance of ICL under different similarities between prompts and the test query. "Least Squares" denotes the result using the least squares method directly. Unless otherwise specified, the baseline setting uses the query distribution as (- I), the response's accuracy as 100%, and the similarity level with the test query as normal.
Proof Sketch. Our proof utilizes the connection between e- stability andthe generalization bound. Firstly, view ICL. as the algorithm described in Li et al. (2023). Secondly, we calculate the e-ttability of the one-layer attention model for
for the details. For other tasks, we leave more details in Appendices D.5 and D.6.
4.1 Memory Reshaping for Three Behaviors of ICL
ICL (Li et al., 2023). Finally, we derive the generalization bound of the one-layer attention model for ICL, leveraging the connection between e-stability and generalization bound (Bousquet& & Elisseeff, 2002). Please see Appendix C.3 for a detailed proof.
Here we consider three behaviors of in-context learning: sensitivity to (i) covariance shifts, (ii) response accuracy, and (iii) the similarity between prompts and test query.
D
Table I. Sensitivity to Covariance Shifts: R-Squared under Dif- ferent Test Data Distributions. 15, 30, 45, 60, 75 denote the in- context example size. The training data distribution is (--2, T), and in-context sample size is 50. "Least Squares" denotes the baseline by least squares regression.
Remark 3.10. The generalization bound recovers the known e-stability bound for ICL (Li et al., 2023) within our one- layer attention setting. This confirms that models following our memory retrieval interpretation achieve the same estab- lished generalization guarantees for ICL. Furthermore, the generalization bound supports the intuition that ICL perfor- mance improves with more reliable in-context examples.
Test Distribution
15
30
45
60
75
Least Squares N
0,8811
1.0
1.0
0.9366
0.9998
0.9999
0,9838
0,9641
0.9N( 21 +
0.9202
0.9997
0.9998
0.9797
0,9677
0.7N( 2
0.4043
0,6613
0.7337
0.6678
0,6792
4 Experimental Studies
In this section, use the in-context memory reshaping phenomenon to interpret and design experiments to validate three key properties of ICL: the model's sensitivity to (i) covariance shifts, (ii) response accuracy, and (iii) the simi- larity between prompts and test examples. Furthermore, our experimental results validate that the generalization bound improves with more prompt examples in Theorem 3.1. At last, we demonstrate our ability to construct t ghost exam- ples (?, 7 seee with prompt tuning.
Table 2. Sensitivity to Response's Accuracy: R-Squared under Different Response Accuracies of In-Context Examples. 15, 30, 45, 60, 75 denote the in-context example size. The in-context sample size in the training data is 50. "Least Squares" denotes the baseline by least squares regression.
Response Accuracy
15
30
45
60
75
Least Squares
0.8811
1.0
1.0
1.0
1.0
100%
0.9144
0,9998
0.9999
0,9800
0,9578
90%
0.9144
0.9614
0.9677
0.9507
0.9075
Our experiments cover both synthetic and real-world tasks: (i) Following (Garg et al., 2022), we use the GPT-2 model on synthetic problems, including linear regression, decision trees, and 2-layer neural networks; (ii) We use the GPT-J model on a sentiment classification task. Take the linear regression setting with GPT-2 as an example, we define the target function as f(x:
80%
0.8266
0.8607
0.9230
0.8955
0.8512
Sensitivity to Covariance Shifts. An important factor affecting the effectiveness of in-context learning the dif- ference between the query distribution in the pre-training dataset and the testing dataset, i.e., covariance shifts (Zhang et al., 2024; Min et al., 2022). In-context learning performs worse when the difference is larger.
18 f ~ I&d, where d = 20. The distribution of r E RR is from a Gaussian Mixture model W| NNN + (2, where Wi = 1,00 = 0 in the pre-training. The pre-training process refers to the method in (Garg et al., 2022), and please see Appendix D.1
We explain covariance shifts through memory reshaping as follows: When there are no covariance shifts, the reshaped
7

---

In- In-Context Learning as Conditioned Associative Memory Retrieval
1.0
1.0
0.8
0.8
0.8
SSguuared
0.6
0.6
0.4
A1-2.11
0.4
Acc- Accm100%
0.4
High Similartty
0.9N1-2. 2.0 0.1922.0
Acc=90%
Normal Similarity
0.2
0.70-22.1- 3.122,
Accw80%
Low Similarity Decision Thee
Decision Thee
Decision Thee
0.0
0.0
0.0
50 In-context Examples
75
25 In-context Examples
50
75
25 In-context Examples
50
75
(a) Covariance Shift
(b) Response's 's Accuracy
(c) Similarity with Test Query
Figure 2. Behaviors of In-Context Learning Measured by R-Squared in Decision Tree Problem. The true response to the query is generated by decision tree. (a) shows the performance of ICL. under different covariance shifts. (h) shows the performance of ICI. under different accuracies of responses in prompts. (c) shows the performance of ICI. under different similarities between prompts and the test query. "Decision Tree" denotes the baseline, i... using the decision tree method directly. Unless otherwise specified, the baseline setting uses the query distribution as (-2, f). the response's accuracy as 100%, and the similarity level with the test query as normal.
1.0
1.0
1.0
R -Squared
0.8
R-Squared
0.8
0.8
ww
0.6
0.4
Wt-2.11
0.4
Acc=100%
0.4
Similarity Normal Similarity
0.9 0.2M2.0
Acc=90%
0.2
0.70(2,11+ (1
0.2
Acc=80%
0.2
Low Similarity
0.0
2-Layer
0.0
2-layer
2-Layer
0
25
50
75
0
25
50
75
25 In-context Examples
50
75
In-context Examples
In-context Examples
(a) Covariance Shift
(b) Response's 's Accuracy
(c) Similarity with Test Query
Figure 3. Behaviors of In-Context Learning Measured by R-Squared in 2-Layer Neural Network Problem. The response the query is generated by 2-layer feed-forward neural network. (a) shows the performance of ICL under different covariance shifts. (b) shows the performance of ICL under different accuracies of responses prompts. (c) shows the performance ICL under different similarities between prompts and the test query. "2-Layer NN" denotes the baseline, i.e., using the 2-layer neural network trained with prompt examples for 100 gradient descent steps. Unless otherwise specified, the baseline setting uses the query distribution as ( I).
the response's 's accuracy as 100%, and the similarity level with the test query normal.
memory's s distribution matches the initial memory distribu- tion after pre-training. With distribution shifts, the distribu- tion of reshaped memory differs from the initial memory's 's distribution. A larger shift in covariance means a greater dif- ference between the distribution before and after reshaping. The model struggles to reshape the memory perfectly with different memory distribution. The difficulty of reshaping increases the distributions become more different.
covariance shifts in the testing dataset.
Remark 4.1. We find that performance decreases when the prompt length exceeds the pre- pre-training length (i.e., 50), well-known issue. We believe this is due to the absolute positional encodings, noted in Zhang et al. (2024).
Table 3. Sensitivity to Similarity between Prompts and Test Query: R-Squared Value under Different Similarity between Prompts and Test Query. 15, 30, 45, 60, 75 denotes the in-context example size. The in-context sample size in the training data is 50. "Least Squares" denotes the haseline by least squares regression.
We conduct the experiments to test the model's perfor- mance with different distributions of queries in the testing dataset. We use three different Gaussian mixture distribu- tion W1 ( + N2,, I) with different w1, W2: (i). w1 W2 O, (ii). W1 = 0.9, W2 0.1, (iii). wi 0.7, W2 = 0.3. Here the distribution in the first setting matches the distribution in pre-training. During testing, generate samples similar to those used in pre-training We extend the prompt length from 50 to 75 to evaluate the performance of ICL with longer prompts than those used in pre-training For each in-context length j E [75), we measure performance using the R-squared between the estimation and the true value. See Appendix D.2 for the details.
Similarity Degree
15
30
45
60
75
Least Squares
0,8811
1.0
1.0
1.0
1.0
Best Normal Worst
0.9563
0.9997
0.9998
0.9899
0.9756
0.9366
0.9998
0.9999
0.9838
0.9641
0.7704
0.9985
0.9993
0.9673
0.9290
Sensitivity to Response's Accuracy. Another factor in- fluencing the performance of ICL is the response format (Yoo et al., 2022). Replacing the response set with a ran- dom set reduces the performance of ordinary auto-regressive LLMs. We explain this using memory reshaping as follows: The prompt response contributes to the equivalent memory reshaping process. The wrong response set misleads the re-
We show the results in Figure (a) and Table 1. The results verify that in context learning performs worse with larger
8

---

In-Context Learning as Conditioned Associative Memory Retrieval
shaping process. Therefore, accurate responses in prompts crucial for effective ICI..
Validation of Generalization Bound. We also validate the generalization bound in Theorem 3.1. The results in Figures 1 to 3 show that the R-squared value increases when the number of in in-context examples is below 50-the same number used during pre-training. This supports the our observation that the generalization bound improves with more prompt examples.
We conduct the following experiments to test the model performance with varying response accuracies. We use the same Gaussian mixture distribution as in pre-training, i.e., wi (),( + w2N (2, I), where wi = 1, = O. We generate samples with different accuracy of response, see Appendix D.3 for the details. We use the same method to calculate the R-squared as Section 4. Here we visualize the performance with accuracies of 100% and 80% in Figure (b) and show partial results in Table 2. The results confirm that ICL performs worse when the accuracy of the response in prompts is lower.
4.2 Ghost Example Construction
We conduct experiments to demonstrate our ability to con- struct t ghost examples {ās, ie Heet with prompt tuning, We calculate the squared error between the outputs with C++1 or Pt == (fC,, Fi Fijte C4+1) as the model inputs, where E [75]. To generate Ct+1, we use two distribu- tions (1 1) 1)+ + w2 I), one is the same as the pre- training data distribution, and another is different from it, i.e., W1 1, W2 = 0 or w1 = 0.5, w2 = 0.5. The average mean squared error for all L E [75] is 0.0003 and 0.001 respectively. See Appendix D.7 for the details.
Sensitivity to Similarity between Prompts and Test Query. The similarity between prompts and the test query is also one important factor to influence the performance of in-context learning (Liu et al., 2022; Rubin et al., 2022). The higher similarity generates better in-context learning performance. We give an explanation about this by memory reshaping as follows: Similar queries have similar responses. For exam- ple, in a linear regression problem, the difference between the responses for similar queries is small. in-context learn- ing reshapes V2 to W2V, V2 as in (8). Compared with V,, the values in W2 V are much more approximate to the ideal value of test query. Therefore, the higher similarity between prompts and test query benefits the in-context learning.
5 Discussion and Conclusion
In this work, we provide an exactly solvable example for interpreting In-Context Learning (ICL. with one-layer at- tention models as conditional retrieval of dense associative memory models. We interpret ICL as memory reshaping in the modern Hopfield model from a conditional memory set (in-context examples), and propose the in-context memory reshaping phenomenon (Section 3.1). We then integrate this phenomenon into the existing Bayesian model averaging theory of ICL (Zhang et al., 2025) to complement their re- sults (Section 3.2). Moreover, under this unique perspective, we not only characterize how in-context examples shape predictions in the Gaussian linear regression case, but also recover the known e-stability generalization bound of the ICL for the one-layer attention model (Section 3.3). We also give explanations for three key behaviors of ICL and validate them through experiments on both synthetic and real datasets (Section 4). These behaviors include sensitivity to covariance shifts, sensitivity to response's accuracy, and sensitivity to the similarity between prompts and test query.
We conduct the following experiments to test the model per- formance with different similarities. We use the same Gaus- sian mixture distribution as the pre-training, i.e., (- I). We use the cosine similarity between different queries as the similarity measure, and generate prompts with different similarity as the test query, see Appendix D.4 for the de- tails. We calculate the R-squared as Section 4.1. Here we visualize the performance in three settings in Figure 1 (c): high, normal, and low similarity, and show partial results in Table 3. The results verify that the in-context learning performs better when the similarity is higher.
More Complex Datasets and Models. In this part, we extend the experiment from linear regression to a decision tree and 2-layer neural network. We use the decision tree and 2-layer neural network to act as the function f. We leave the details about the experiment setting in Appendix D.5. We show the results of the decision tree in Figure 2 and 2-layer neural network in Figure 3.
Our findings enhance the understanding of ICL and its per- formance dynamics, paving the way for further advance- ments in LLM capabilities. By interpreting LLM as an energy-based model, we have three practical guidelines for practitioners: (i) As demonstrated in Section 4.1, selecting prompts similar to the test query enhances in-context learn- ing performance. This aligns with the findings of retrieval- augmented generation (Lewis et al., 2020). (ii) By Theo- rem 1. increasing the number of relevant prompt exam- ples reduces the ICL generalization error. (iii) As stated in Theorem 3.1, a smaller norm bound I (Assumption 3.2) improves ICL response quality. Furthermore, we discuss the limitations of our work in Appendix A.
Furthermore, we extend the analysis from GPT-2 to the more capable GPT-J model and evaluate it on the real-world "TweetEvall: Hate Speech Detection" dataset (Nagel, 2016) for sentiment classification task. We show the detailed ex- perimental settings and results in Appendix D.6. The results follow a similar pattern to the linear regression setting.
9

---

In-Context Learning as Conditioned Associative Memory Retrieval
Impact Statement
lems, volume 36, pp. 57125-57211. Curran Associates, Inc., 2023.
This research is theoretical and is not expected to have neg- ative social impacts. As outlined in the introduction and related works, the primary goal of this study is to enhance our understanding of the underlying principles of large foun- dation models from an associative memory perspective.
Basile, V., Bosco, C., Fersini, E., Nozza, D., Patti, V., Pardo, F. M. R., Rosso, P., and Sanguinetti, M. Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter. In Proceedings of the 13th international workshop on semantic evaluation, pp.
Acknowledgments
54-63, 2019.
The authors thank Zhang et al. (2025) for clarifications and helpful feedback; Jiachen Zhao, Mimi Gallagher, Sara Sanchez, Dino Feng and Andrew Chen for enlightening dis- cussions; Zhao Song, Dennis Wu, Jennifer Zhang, Maojiang Su, Hude Liu and Hong-Yo Yu Chen for collaborations on re- lated topies; and Jiayi Wang for facilitating experimental deployments. JH also thanks the Red Maple Family for support. The authors would like thank the anonymous reviewers and program chairs for constructive comments.
Bietti, A., Cabannes, V., Bouchacourt, D., Jegou, H., and Bottou, L. Birth of a transformer: A memory viewpoint. Advances in Neural Information Processing Systems, 36: 1560-1588, 2023.
Bousquet, O. and Elisseeff, A. Stability and generalization. The Journal of Machine Learning Research, 2:499-526, 2002.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
JH is partially supported by the Walter P. Murphy Fellowship. Han Liu is partially supported by NIH R01L.M1372001, NSF AST-2421845, Simons Foundation MPS-AI-00010513,, AbbVie and Dolby. This research was supported in part through the computational resources and staff contributions provided for the Quest high performance computing facility at Northwestern University which is jointly supported by the Office of the Provost, the Office for Research, and Northwestern University Information Tech- nology. The content is solely the responsibility of the au- thors and does not necessarily represent the official views of the funding agencies.
Cabannes, V., Dohmatob, E., and Bietti, A. Scaling laws for associative memories. In The Twelfth International Conference on Learning Representations, 2024.
Dai, Z., Yang, Z., Yang. Y., Carbonell, J. G., Le, Q., and Salakhutdinov, R. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pp.
2019.
References
Edelman, B. L., Goel, S., Kakade, S., and Zhang, C. Induc- tive biases and variable creation in self-attention mecha- nisms. In International Conference on Machine Learning, pp. 5793-5831. PMLR, 2022.
Akyürek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. What learning algorithm in-context learn- ing? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2022.
Feng, R., Zheng, K., Huang, Y., Zhao, D., Jordan, M., and Zha, Z.-J. Rank diminishing in deep neural networks. Advances Neural Information Processing Systems, 35: 33054-33066 2022.
Amari, S.-I. Characteristics of random nets of analog neuron-like elements. IEEE Transactions on systems, man, and cybernetics, (5):643-657, 1972.
Garg, S., Tsipras, D., Liang, P. S., and Valiant, G. What can transformers learn in-context? case study of sim- ple function classes. Advances in Neural Information Processing Systems, 35:30583-30599 2022.
Anil, C., Wu, Y., Andreassen, A., Lewkowycz, A., Misra, V., Ramasesh, V., Slone, A., Gur-Ari, G., Dyer, E., and Neyshabur, B. Exploring length generalization in large language models. Advances in Neural Information Pro- cessing Systems, 35:38546-38556, 2022.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788, 2024.
Bai, Y., Chen, F., Wang, H., Xiong, C., and Mei, S. Trans- formers as statisticians: Provable in-context learning with in-context algorithm selection. In Oh, A., Neumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances Neural Information Processing Sys-
Guo, D.,. Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-rl: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
10

---

In-Context Learning as Conditioned Associative Memory Retrieval
Han, S., Song, J., Gore, J., and Agrawal, P. Emergence of abstractions: Concept encoding and decoding mechanism for in-context learning in transformers, 2024.
Neural Information Processing Systems, 33:4790-4799, 2020.
Li, Y., Ildiz, M. E., Papailiopoulos, D., and Oymak, Transformers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning, pp. 19565-19594. PMLR, 2023.
Hoover, B., Liang, Y., Pham, B., Panda, R., Strobelt, H., Chau, D. H., Zaki, M., and Krotov, D. Energy transformer. Advances in neural information processing systems, 36: 77532-27559, 2023.
Liu, J., Shen, D., Zhang, Y., Dolan, W. B., Carin, L.. and Chen, W. What makes good in-context examples for gpt- 3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop Knowledge Extraction and Integration for Deep Learning Architectures, pp. 100- 1 114, 2022.
Hopfield, J. J. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79())2255-22558 1982.
Hu, J. Y.C.., Wu, D., and Liu, H. Provably optimal mem- ory capacity for modern hopfield models: Transformer- compatible dense associative memories as spherical codes. Advances Neural Information Processing Sys- tems, 38, 2024a.
Mahankali, A. V., Hashimoto, T., and Ma, T. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. In The Twelfth International Conference on Learning Representations, 2023.
Hu, J. Y.-C., Yang, D., Wu, D., Xu, C., Chen, B.-Y., and Liu, H. On sparse modern hopfield model. Advances in Neural Information Processing Systems, 36, 2024b.
Min, S., Lyu. X., Holtzman, A., Artetxe, M., Lewis, M.. Hajishirzi, H.,, and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11048- 11064, 2022.
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. I., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088. 2024a.
Nagel, S. Cc-news (2016). URL http:/ccommonrravl wwwy2//o//Oneesssdssseeeeriiiiio
Jiang H. A latent space theory for emergent abilities in large language models. arXiv preprint 23X99;2304.0996, 2023.
2016.
Ramsauer, H., Schäff, B., Lehner, J., Seidl, P., Widrich, M., Adler, T., Gruber, L.., Holzleitner, M., Pavlović, M., Sandve, G. K., et al. Hopfield networks is all you need. In International Conference on Learning Representations, 2020.
Jiang, Y., Rajendran, G., Ravikumar, P., and Aragam, B. Do Iims dream of elephants (when told not to)? latent con- cept association and associative memory in transformers. Advances in Neural Information Processing Systems, 37: 67712-67757, 2024b.
Rubin, O.,, Herzig, J., and Berant, Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the As Association for Computational Linguistics: Human Lan- 2022.
Krotov, D. A new frontier for hopfield networks. Nature Reviews Physies, 5(7):366-36)) 2023.
guage Technologies, pp.
Krotov, D. and Hopfield, J. Large associative memory prob- lem in neurobiology and machine learning. arXiv preprint MwXiv:2008.06990 2020.
Santos, S., Niculae, V., McNamee, D., and Martins, A. F. coopfeedd-fencel-yyoure networks: A unified frame- work for associative memory retrieval. arXiv preprint arXiv:241 1.08590, 2024a.
Krotov, D. and Hopfield, J. Dense associative memory for pattern recognition. Advances in neural information processing systems, 29, 2016.
Santos, S. J. R. D., Niculae, V., McNamee, D. C., and Mar- tins, A. F. T. Sparse and structured hopfield networks. In Proceedings of the 41st International Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 43368-43388. PMLR, 21-27 Jul 2024b.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-., Rocktäschel, T., et al. Retrieval-augmented generation for knowledge- intensive nip tasks. Advances in neural information pro- cessing systems, 33:9459-999 9974, 2020.
Von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M. Transformers learn in-context by gradient descent.
Lewkowycz, A. and Gur-Ari, G. On the training dynamics of deep networks with 7. regularization. Advances in

---

In-Contest Learning as Conditioned Associative Memory Retrieval
In International Conference Machine Learning, pp- 35151-35174. PMLR, 2023.
matter: A deeper look into input-label demonstrations. In Empirical Methods in Natural Language Processing, pp. 1-16. Association for Computational Linguistics, 2022.
Wang, S. and Sato, I. Understanding knowledge hijack mechanism in in-context learning through associative memory, 2024.
Yuille, L. and Rangarajan, A. The concave-convex proce-
dure (ccep). Advances in neural information processing systems, 14, 2001.
Wang, X., Zhu, W. Saxon, M., Steyvers, M., and Wang, W. Y. Large language models are latent variable mod- els: Explaining and finding good demonstrations for in- context learning. Advances in Neural Information Pro- cexsing Systems, 36: 46:15614:4 156388-5638, 2023a.
Zhang. R., Frei, S., and Bartlett, P. L. Trained transform- ers learn linear models in-context, Journal of Machine Learning Research, pp. 1-55, 2024.
Zhang, Y., Liu, B., Cai, Q. Wang, L., and Wang. Z. An analysis of attention via the lens of exchangeability and latent variable models. arXiv preprint arXiv:22 14852, 2022.
Wang. X., Zhu, W., and Wang. W. Y. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning. Workshop on Efficient Systems for Foundation Models@ ICML2023, 2023b.
Zhang, Y., Zhang, F., Yang, Z., and Wang. Z. What and how does in-context learning learn? bayesian model averag- ing. parameterization, and generalization. In Proceedings of the 28th International Conference on Artificial Intelli- gence and Statistics (AISTATS 2025), 2025.
Wang, Y., Chauhan, J., Wang, W., and Hsieh, C.-J. Uni- versality and limitations of prompt tuning, Advances in Neural Information Processing Systems, 36, 2024.
Wies, N., Levine, Y., and Shashua, A. The learnability of in-context learning. Advances in Neural Information Processing Systems, 36, 2024.
Williams, C. K. and Rasmussen, C. E. Gaussian processes for machine learning, volume MIT press Cambridge, MA, 2006.
Wu, D., Hu, J. Y.-C., Hsiao, T.-Y., and Liu, H. Uniform memory retrieval with larger capacity for modern hop- field models. In International Conference on Machine Learning, pp. 13471-53511. PMLR, 2024a.
Wu, D., Hu, J. Y.C.,, Li, W., Chen, B.-Y., and Liu, H. STanhop: Sparse tandem hopfield model for memory- enhanced time series prediction. In The Twelfth Interna- tional Conference on Learning Representations, 2024b.
Wu, W., Su, M., Hu, J. Y.-C., Song, Z., and Liu, H. In- context deep learning via transformer models. In Interna- tional Conference on Machine Learning. PMLR, 2025.
Xie. S. M., Raghunathan, A., Liang, P., and Ma, An explanation of in-context learning as implicit bayesian inference. International Conference Learning Repre- sentations, 2022.
Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, I.., and Liu, T. On layer normalization in the transformer architecture. In Interna- tional conference machine learning, pp. 10524-10533. PMLR, 2020.
Yoo, K. M., Kim, J., Kim, H. J., Cho, H., Jo, H., Lee, S.-W., Lee, S.-g. and Kim, U. Ground-truth labels
12

---

In-Context Learning as Conditioned Associative Memory Retrieval
Supplementary Material
A Limitations
14
Notation Table
15
Proofs of Main Text
16
C.I Proof of Lemma 2.1
16
C.2 Proof of Proposition 3.1
16
C.3 Generalization Bound for In-Context Learning
18
Experimental Details
24
Pre-training GPT-2 for Linear Regression
24
D.2 Sensitivity to Covariance Shifts
24
D.3 Sensitivity to Response's Accuracy
24
D.4 Sensitivity to Similarity between Prompts and Test Query
24
D.5 Decision Tree and 2-Layer Neural Network
24
D.6 GPT-J on Sentiment Classification Task
25
D.7 Ghost Example Construction
26
13

---

In-Context Learning as Conditioned Associative Memory Retrieval
A
Limitations
We also have the following four limitations in our work:
One notable limitation is that the Softmax-attention in (5) does not represent the actual attention in the LLM. Although this effective attention mechanism helps us better understand ICL in the Bayesian model averaging view, to ensure our theory is more aligned with reality, we should consider the actual attention in the L.L.M
In our theory, we only consider the one-layer attention mechanism. We plan to study how a multi-layer attention mechanism performs ICL in future work. We regard the multi-layer transformer as iteratively retrieving memory patterns in the modern Hopfield model through the following four steps: (i) Design an energy function for multi-layer modern Hopfield model: We design an energy function matching the multi-layer transformer (Hoover et al., 2023). The memory pattern needs retrieval after reshaping and iteration. This step is challenging.
(ii) Give clear expression of memory reshaping on model weights: This step is feasible following our framework in Section 3.1. (iii) Extend the Bayesian method to multi- layer transformer: The Bayesian method in (Zhang et al., 2025) and our work assumes perfect pre-training. We must study perfect pre-training to match real multi-layer transformers. This step is challenging. (iv) Obtain a generalization bound of ICL: This step is feasible following our framework in Appendix C.3. We need to derive the e-stability of the multi-layer transformer.
Our analysis focuses predicting single next token. Traditional autoregressive large language models often in-context learning to predict multiple tokens. We now describe how to extend our method to this setting. This extension involves iteratively applying the same single-step retrieval mechanism: (i) After generating the first token, append it to the context to form a new query. (ii) Each updated query triggers the memory reshaping of ICL in large language models to predict the next token. (iii) Repeating this process yields the full output sequence. For example, to predict two tokens, the prompt queries and responses are C. = r1,, Ct, Tt.1] E Tgdx x2r and Re [ri, -1,2,-... Pt,1, ft,2 E Rd,x21 with D,- == {er, 2 The LLM takes prompt Pt = (Dr, C(+1) and predicts rz-1,1 and 1 The joint probability follows P(1++1,1 ", ri+1.2
|++1,1 P.) P(r = I P.). Our framework in Section 3.1 and Section 3.2 applies to r1+1.1- Once we predict 114,,1 we apply the same method to obtain P(r++1,2 = 1i+1,1 = P.). For generalization in Section 3.3, the bound for 1:+1,1 uses the existing framework. For 1,2, we treat prediction error in ri+1,1 1, as noise to the input. This matches the setting in our e-stability analysis (Lemma C.2). We only adjust a few hyperparameters to extend the bound to T(+122-
P.) P(r++1,2
We do not verify the exact memory reshaping setting in our current experiments. This is challenging. However, we provide a potential solution under an ideal setting, and level it for future work. Following Appendix D,, for each E [75]. we generate 6400 samples with the distribution N-22, I). Each sample follows the pre-training setup with batch size 1. For the i-th sample, we define prompt P.. = (Ca. R4,,, 11,4+1). We use C, to construct t ghost examples via prompt tuning (see Section 3.1). We denote the ghost query and response as (7..): Figg)stt The model outputs Ri,et1 when inputs are (Cs,t, RA,,, G.141). To simulate memory reshape, input (Ct.,, Ri,t, and modify attention layers as follows: (i) In each attention layer, we compute K,, and V.,.- (ii) We compare with the original K,,, Vi,. and solve (8) to get reshape matrices Wa,1 and Ws (iii) We apply them as W Wr,,K,t Kr, and W Vi, in the forward pass. This method simulates the effect of memory reshaping. However, implementing and validating this approach difficult. It requires fine aiee-graind control over internal activations at each layer. It also involves layer-wise wise matrix inversion and alignment for every sample and timestep, which can introduce instability and noise. Due to these challenges, we leave full implementation and validation future work. Nonetheless, the setup outlines feasible path and offers useful guidance for follow-up research.
14

---

In-Context Learning as Conditioned Associative Memory Retrieval
B Notation Table
We summarize our notations in the following table for easy reference.
Table 4. Mathematical Notations and Symbols
Symbol
Description
dx
Dimension of key vectors and query vectors
dp
Dimension of value vectors
dø
Dimension of the output of feature mapping
C
Input query, defined as C₁ [c....., Cx] E Rd.xt Input response, defined as R = |r,.... rr] E Rd.xt Input prompt examples pair, defined as Dj = {c, Input prompt, defined as P. (De, C(+1) Key vector in Rd
R
De
P
K
K.
Key matrix, defined K, [Ka, , Kij E Rdxx
Kernelized feature mapping: IRd -> d2
Value vectors in R
V
Value matrix, defined as V = [m va) E Rd.xt Query vector in dd
A(g, K2, V.)
Bayesian model averaging attention defined by Definition 3.1
Z*
Hidden concept representing property shared across all prompt examples. Posterior Mean of Z* given prompt D,
Zi
h4
Latent variable linking query ca to response T²
H
Space of latent variable h;
Z
Space of hidden concept Z
Ez,
Posterior covariance of Zt given prompt D, Predicted covariance of ut+1 conditional on Standard deviation of noise in Gaussian linear model Prior standard deviation of elements in latent concept Z* Pre-trained -ey-jrojention matrix Pre-trained value- prajeeprooection matrix
Σ,
X
W
WV
W
Reshaped pey-projention matrix after memory reshaping Reshaped palue-projection matrix after memory reshaping Loss function with respect to (Dt,
WV
I(..)
C(3))
Task-specific loss function, e.g., mean squared error
15

---

In-Context Learning as Conditioned Associative Memory Retrieval
C Proofs of Main Text
In this section, we provide the detailed proof of Lemma 2.1 and Proposition 3.1.
C.1 Proof of Lemma 2.1
Proof Sketch. This proof follows Zhang et al. (2025, Proposition 4.1). It is based on general latent variable model and uses the law of total probability to derive the desired result.
Proof. Recall that Pt = (Dr, C4+1). where D, is the set of prompt examples and c++1 is the test query, he H. denotes the hidden variable, and Z Z is the hidden concept. Then, have
[ dh+t 2(rx41,h44 Pt)
P(r++1 Pt)
[ dhit1 P(r+41 P(.hes))P(he1 Pt)
f dh4t1 P(re41 (441hh+1, D.) P(h441 (41, Dz)
[ dh P(r+11 h+41))++1)+ P(1++1 Di)
= f dhet1 P((r41 (441)he)) ( dZ P(h--1 Z) P(Z D.))
H
{ dz (Jx (Jale P(reti (41)))41) P(h-11 (Dr.)) ((, p(Z D,)
=
az -(r++) 2). P(Z Di),
where the first line is by the law of total probability integrate over h411, the second line is by factorizing the joint conditional density of T and he+1 given Pt. the third line is by substituting P. = (Dr, cr+1), the fourth line is by the the independence of the query-response pairs r. and the conditional independence of h441 and c++1 in (2), the fifth line is by the law of total probability to integrate over Z, the six line is by swapping the order of integral over h441 and Z, and the seventh line is by substituting P: = (Dr, c++1) and the integral over h4+1.
This completes the proof.
C.2 Proof of Proposition 3.1
Proof Sketch. This proof utilizes Bayes' rule and the properties of Gaussian distributions, such as the Gaussian marginaliza- tion (see (Williams & Rasmussen, 2006, Chapter 2.1)).
Proof. Recall that Us = 0* (ra) for i E (T], then we have the following by substituting ri with v; in Lemma 2.1
P(u+11 Pt) = / dz P(u++1 P., D.).
Following (Zhang et al., 2022), we derive the conditional distributions P(1+41 | Pt. Z) and P(Z (D)) as follows.
Deriving P(u++1 Pt, Z). By (11), given hidden concept Z E xd,, we have the formulation U++1 Z (K* (Ce+1)) + M141. where Mh+1 E N NO0 (f, Ta.) are Gaussian noises. Recall that Je+1 (c(41) and 2
Zo(k" (c(11) follows distribution N(0, Ia.)) then we have
-llec+1 - Z
P(141 Pr, Z) exp
20
16

---

In-Context Learning as Conditioned Associative Memory Retrieval
Deriving P(Z] D.). Recall that Z follows Gaussian distribution (2 xd2)), D. =C,, Ra), K₁ (C,), and V₁ 1² (R.). By Bayes' rule, we have
P(Z) D.) P(V. Z, (Z)
P(V. D.)
VV - -Z -exp (- 22x 1121
x exp
207
Tr (Vi Z)(Ki)) (V: - zo(K)))
Tr [2Z
cxp
207
212
Tr V.V.T - -22 + ( (2x))
Tr Tr[2zT]
exp
2
212
1
1
V.V.' 1) V.V. 209
exp
Th
2
7 (()K)))))
TT T
2X
77
+
V
Tr
20
1
1
exp
Tr
Z
X2
1
+
o
(Ki)
TT
+ TT
2
of
Tr
1
1 /t of
X exp
Ir Z
1
(K)))(()))
2T
+ TT
2
of
where the first line is by Bayes' rule, the second line is by V2, Z)(K,) ~ NN, x1), Z ~~ (0, la, xd,). and P(V. D.) is constant value (V2- x*R and D.- (C,, R,)), the third line is by |||| = =Tr(A) for any matrix A E IR xd, (di E N)), the fourth line is by Tr(A) Tr(AT) for any matrix A E Iddixdi, the fifth and sixth lines are by rearranging terms, and the seventh line is by omitting constant terms with respect to the variable Z.
We define matrices
1 + 6
and M = o(K,V, of
Σz, = X2
(Ki))(K.))
0
Then we have
1 Tr
( EE, -2ZM)
1
1
2T) + +Tr 2
(O(Ki))?
))
P(Z) Dz) x exp
Z
of
of
exp
Tr
1 lr
(z M 2izi) Viz,1 L M 222
+ lr M M21MMM
exp
2
M 1-11r Tr (z- (z-M E=- Ez, (z (z- M EZZ
x exp
where the second line is by the definition of Σz,, and M, the third line is by adding and subtracting terms and rearranging them, and the fourth line is by omitting constant terms with respect to the 2 - -M 2z
We define
2, 3-MM M. TzZ
Zi
= Vio(Ke) (( ot [2 1 )-
= Vill x2 I ((K)) )-(((()))))
By Woodbury identity)
Deriving P(1c+1| P.). By the distribution P(v--1 Pt, Z) and P(Z D,), we have
P( P.) = / dz - P(e44 22, P(Z | DD)) 2
/ dz N((+41 Zo( - Ze, Ez.)
=
17

---

In-Context Learning as Conditioned Associative Memory Retrieval
By standard results Gaussian marginalization (Williams Rasmussen, 2006, Chapter 2.1), P(u++1 Pt) also follows Gaussian distribution, with mean 14 and covariance Cov, follows:
Ae = E (Zó(g++1) + + Pil = E [Z p = Z,o (q++1).
Cov, Cov (Zo (q++1) + 76+1
P.] o (9++1) E (9++2) + oj
Therefore, we obtain
P(u++1 Pt) exp ( (_l10e+1 -Zo(((++))
220,
where
Ze = Vi( of x2 1r I+ (()))(())))))((())) Σz, = X2 I + (KK))o(K)), Σ, === = o(q+1)) ( (2200(44)). of
This completes the proof.
C.3 Generalization Bound for In-Context Learning
Proof of Theorem 3.1. Referring to (Li et al., 2023), view in-context learning algorithm learning problem. Based on this, we derive the generalization bound for in-context learning with the e-stability of the algorithms.
First, we give the definition of e-stability.
Definition C.1 (e-Stability). Given Softmax-attention model with parameter 0, and some possible input prompt examples ri Ji of constant size t. For any test query C++1, denote the loss function with respect to (Di, C++1) as
D, = r ((D,,c4+1). The in-context learning of this Softmax- -ottmax-attention model is c-uniformly stable if, for any two prompt D; and D, with only one different prompt example, we have
sup (41) L(D,, S a D.
(15)
where the expectation taken over the randomness of prompt examples D,.
Next, we give the c-stability of the one-layer attention model.
Lemma C.1 (Lemma B.1 of (Li et al., 2023)), Let x, E R be vectors satisfying ||=||oo || elloo Then, have
2c
2e |ell
|Softmax(2)]l
and ||Softmax(2) Softmax(# c)]l s
n
Proof of Lemma C.I. For any Rd, we define Softmax(x;) as
ex
Softmax(1:) = Σj o.i
1, d.
Then we have
Softmax(211) Softmax(1z)
2
Softmax(z)
Rd
==
Softmax(Fa))
18

---

In-Context Learning as Conditioned Associative Memory Retrieval
Using the monotonicity of Softmax function, we have
max; #;
|Softmax(2)1l0 ||
(By the definition of Soflmnax)
1= e
e
(By monotonicity and 23 S c)
e0 + 2 e
e
(By 23 2 -c, j +
ec
+ 2i=2
22
220 + n
20
(By 20 21)
where the fourth line is by multiplying ef to both the numerator and the denominator. By taking the derivative of Softmax(2;)) with respect to Zk. we have
0 Softmax(x;)) Ô
e E==ı eas
e . Oek c, - 0 Softmax(2;) (ōa: Softmax(xx))
Ozy
(Ejree))
2
where de is the Kronecker delta, which equals 1 if = A: and 0 otherwise.
By the definition of directional derivative along with direction vector E Rd,, we have
lim Softmax(1 de) Softmax(z) [diag (Softmax(z)) Softmax(x) Softmax(1)
(16)
6
Furthermore, we have the following norm bound
[iag (Softmax(x)) Softmax(z) (oftmax(())Sftmmxx(2 e/l=
< [diag (Softmax(x)) Softmax(1)) Soltmax(()
"Ill.... 1,00 llell e
2
E=s
max
eass) jz
22 e ||e||
2==1 E
22
e
jfz
2: Ejjt 63
max
||e||
(2 n em
2c |ell
where the second line is by the norm upper bound ||Ae||a < ||Al|1, ||e||a for any A E Rddd (d N+), the third line is by the definition of f norm of matrix, and the fourth line by simple algebraic calculations.
Building on this observation, bound the left-hand side of (16) by 2e [|ell Integrating that derivative (left side of (16) from 8 0 to 1 then yields the desired result
2e
[Softmax()) - Softmax(x + e) S
This complets the proof.
Then we have the following lemma to characterize the stability of the one-layer attention model under our in-context memory reshaping phenomenon.
Lemma C.2 (Stability of One- Attention, Modified from Lemma B.2 of (Li et al., 2023)). Let C. = ........ c.] and
Rs [ri -ral. We denote Wi = (W) and Wi Rt R,) R.)'. Wt. Let the weight matrices be bounded as (( TWW T l2ll I T and [ 22 & 1. Let Fj [6,1 ... G,,] and Eg = [(2,1 ... -2,1 be the perturbation.
19

---

In- Cn-Context Learning as Conditioned Associative Memory Retrieval
We set C. Ce ED 4+1 c+11 <1,4+ and R2 R2 + Eg. Assume that G, C 11 and rg + 22 lies in the unit ball, ie.., ||Czll2.so 20 || + Fg 2,122,00 R + F2 ||ee- + 61,4+ I2 || 62,41ll2 S 1. Additionally, for any i E [6]. we assume ||E|, 12 ||e2,ill2 Coft, t, and e1, Co/t, where Co > 0 is an absolute constant.
We define the output
for the input examples as
Ut+1 = WV Ra Softmax ( (WWR)) Wko+1). and Btt1 = WV R, Softmax
Whe+1). (17)
Let Dr Taheet) and D. {ā, Pi)et)) We further denote the true response of C++1 T++1, and define the loss function
M(D c++)) = L (vL+1, Wiret). and I(Di, = [(1+41, Wyret);
where L is a lask-specific loss function and follows Assumption 3.3, e.g, mean squared error loss with a bounded input domain. Then we have
Co(4l' 1)
|: 1111112
(18)
and
MCo(4T + 1)e2r
sup E ((D,, L(D,, c44)) < cr41 D.
(19)
Specifically, e-stability (Definition C.I) corresponds perturbing only prompt example. As result, obtain MCo(4f + 1)e2f/1 /t.
Proof of Lemma C.2. Consider the output difference = ot41 414. By (17), we have
R Softmax (WWCC) (W - Wi He Softmax (WiCc)Tkkket
(R, + E2) Sofumax (WWC)) W& (WAC))TWAHH)) Wi R, Softmax (W (i
= -W Rs Soottmax
C, (W*) WR Softmax (CT (Wx) Wicc+)))
+ Wy F2 Softmax
(CT (WK)
( WR
Aa
where the second line is by R₂ = R₁ Eg, and the third line is by rearranging the terms.
To bound A and Åg, we first establish the following norm estimates.
Recall that [[W+) T 1l22 $ ST, 1/ = ||C + Eill,,0 I 1, and ||c411l2: |ēe41l22 S 1, then we have
([() (Wx) ( WK || $1/Cel12.xo
(wx) T W
|| ST
(20)
2.2
where the inequality is by ||ABx|| < ||A 12, B 10/12 for any A E Rdixd, B E Rdxxds and x E Rd (d1, d2, d3 e N+). Similarly, with ||c+a I1, $ 1, we have
|C ( WwecH/l
$||Co11,.00
<
( W
-|cc41lla T,
(21)
2,2
20

---

In-Context Learning as Conditioned Associative Memory Retrieval
where the inequality is by ||ABolloc < ||A 11A.112.00 (d1, d2, d3 E N.
22.00
B 2.2
for any A Rdixdz, B E Rd2xds and Rd
By 1CC1l2,,0 I 1, we have
(WR) WWker1 S ||
(
Wx 2,2 ||le+11 212 ST,
(22)
where the inequality is by ||ABr|o I 1A"ll2.. B |22||| for any A E Rd Rdixds, B E Rdaxds and E R (di, d2, da E N+).
We further have
(CTTWa (TTW)TWeeeaee Wge,,oil
s $|Co:le.1 (
1lllan-((00 T Wx 2,2 |e1,44112
|all2 || (wx) Wx ||1,041122
S Col,
(23)
where the first line is by ||Ax||| S || N+), the second line is by the definition of 2,1 norm of a matrix, and the third line is by |all2 S and ||9,41lle S Colt.
12.1 2.2 |æl| for any A e Rd B E Rd and x e Rd (d1, dg
Similarly, we have
ET (Wk) W Wiaet
||
S 51 12.1 ( T W
||cctalla
2,2
|ew
W
||ee11l22
2,2
C.r.
(24)
where the first line is by ||4Bx|] - ||AT] B||2,, || for any A E Redixd,, B E Rddxds, and ær Rda (d1, (2,,d2, dg N+), the second line is by the definition of ε21 norm of a matrix, and the third line is by ||e, S Co/t and ||441ll2 1.
Then we bound A and A₂ using Lemma C.1 together with the above norm bounds.
Bounding Aj. For A1, we have
| 2
W R,
Softmax (WW) WR&++) - Softmax (CT (W?) ( 1)
Wi Re
2,00 |Softmaxx (CT (W+) Wk(e+1) Softmax (CT (Wx) T Wkaet))) ll,
Sootmax
(TT) Wk00+1) - (C. (W()) T WRCH+))
Softmax
Softmax
(WR) TW Softmax (CT (WW) T
|ootmax (CT T
Softmax (CT (Wi) T
where the second line is by ||Ax||2 <
for any A E IRdixds and x E Rd (d,, d2 E N+), the third line is by 112.,, || 12,,00 I 1, and [ Re 12.00 ||Wil22,2 1| I 1, and the fourth line is by triangle inequality.
Then we bound the above two terms. For the first term, we have:
Softmax (C (WW)) Wkkc-) Softmax (CT (
Softmax
( C7 T T + (.,41)) Softmax (CT [WW)) 11,
20
CJ (W+) ( W
(By Lemma C.I, (20), and (21)
20
(By (23))
21

---

In-Context Learning as Conditioned Associative Memory Retrieval
For the second term, have:
Softmax
Softmax (cT ( W (CC+ Ei) WRa+1) Softmax (c] (WR) (iRk))WeeeH WRRett))l
|Softmax
20
F. (Wk)
" ||
(By Lemma C.I, (21), and (22))
t
2
(By (24))
Combining the above two terms, we have
| 3 2CoIe
t
Bounding Ag.
For 22, we have
Wy Eg Softmax (CT ( WREaH1)l
A
Wi
||E2lla.1
Soolnax (CT(WR) (WR)TWRCHH)
2,2
Wi
||221l22
Sootmax (CT (WW))WWaee
2,2
2r Co
where the second line is by |ABa||2 & ||Al|2,21 B |2,112/122 for any A E (R Xda, B E Rdxd,, and x E RR's (d,, d2, d2 E
N+), the third line is by the definition of the ε,1 norm of matrix, and the fourth line is by [ l2.2 S 1, 1|2a,ll Colt. Lemma C.1 and (20).
Combining the above results, we get the desired result:
Co(4r + 2)
A t
(25)
Recall that in general case, L(D,, 0141) = C(1++1, Wor++1), and L(D,, c.+1) = f(i--1, Wer(+1) where L is a task- specific loss function. Then for the difference of the loss funetion, we have
sup E [(e++1, Wirez)) L -(MMtI, sup E [I(D), I(Di, (-D)) 441 D.
(itt:WWrr+)) Wiret1)]
< sup M - u6-1112
(By Assumption 3.3)
a D.
MCo(4I (T + 1)e
(By (25))
b
This completes the proof.
Now, we introduce one lemma about the relation between c-stability and generalization bound, based on the Lemma 7 in (Bousquet Elisseeff, 2002). Here we transfer the description in the original lemma to the in-context learning setting. Lemma C.3 (Connection between Generalization Bound and e-Stability, Lemma 7 of (Bousquet Elisseeff, 2002)). We denote the generalization error of in-context learning with prompt Pt and test query C++1 as R(D,) Ee.., [L(De, c++1)). The empirical error as Remp((:) = Σ L(De, ci), i.e., the average loss value when using o; as the test query respectively. Then the generalization bound as follows:
E (R(.)) (Di) E [LDD, -(D),))-(D)).a))
(26)
D.
Dele.i))
where Dj denotes replace (c,, r4) with (T,, F.) Dr-
22

---

In-Context Learning as Conditioned Associative Memory Retrieval
We give the following proof of Theorem 3.1 based the above results.
Proof of Theorem 3.1. Using Lemma C.3, we compare the right-hand side of (26) with left-hand side of (19) in Lemma C.2, and thereby obtain the following generalization bound for in-context learning
MCo(4T 1)
[R(D:) Remp (em) 2
This completes the proof.
23

---

In-Context Learning as Conditioned Associative Memory Retrieval
D Experimental Details
Here we present our experimental details.
D.1 Pre-training GPT-2 for Linear Regression
In this part, we show the details about how to pre-train GPT-2 model for linear regression problem. Following the pre-training method in (Garg et al., 2022), we use the batch size as 64. To construct one sample in a batch, we use the following steps: (i). Sample linear regression coefficient 3, E I 20 from [(0, I). (ii). Generate queries Fijj from the Gaussian mixture model W1 WiN( (N (2, I), where W1 1, 42 0 in the pre-training. Then we formalize (Ii ) j=1 where k = 50. (iii). For each query æi.j. use yij = of mig to calculate the true response. Now we generate one training sample, the model input is [ Vi,1, 49, 99,49, 200 and the training target is o, = [y5.1. 99,49 V6.50]. We use the MSE loss between prediction and true value of o₁. The pre-training process iterates for 500k steps. We implement experiments 1 NVIDIA A100 80GB GPU.
D.2 Sensitivity to Covariance Shifts
In this part, we show the details about how to evaluate the model performance in the testing process. We generate samples similar to the pre-training process. The batch size is 64, and the number of batch is 100, i.e., we have 6400 samples totally. For each sample, we extend the prompt length from 49 to 74, to learn the performance of in-context learning when the prompt length is longer than we use in pre-training. The input to the model is [ V1,1, ,2,,4,, V4,74, T2,77), and the target o, = [yK,,,".. 74, V1.75]. For each in-context length j E [75], we calculate the R-squared between the estimation and true value for all 6400 samples.
D.3 Sensitivity to Response's Accuracy
In this part, we show the details about how to generate samples with different accuracy of the response. (i). Firstly, we generate zth sample as the pre-training process in Appendix D.I, where queries Ti,f are from the Gaus- sian mixture model W1 (--2,- T) + (2,N w)
1,42 = O, and the prompt length becomes 75. The input is [2r,1, "r.1,-.. FT,,44: Vi,74, T1,75), and the target is o = [y,. 31,74: 36.75]. (ii). Next, we denote the response's 's accuracy as a (e.g., 80%). To predict ji.j in 7th sample with j 1 in-context prompts, we randomly permute the prompt labels [y..1.... 1 in the input, with a permutation rate 1 a. In this way, we generate samples with the accuracy of response as
D.4 Sensitivity to Similarity between Prompts and Test Query
In this part, we show the details about how to generate prompts with different similarities as the test query. (i). Firstly, we generate 7th sample as the pre-training process in Appendix D.1, where queries Ti,j are from the Gaussian mixture model w/ (--2,, I) + W2NN [) W 1, = Here we extend the prompt length from 49 1000. The target is still o, 1y Mi,,74 V1.r5] but the input changes according the level of similarity. (ii). Secondly, inputs according to the level of similarity. We consider three kinds of similarity levels: high, normal, and low similarity. For each j E [75] and j > 1, in the high similarity case, we select the most similar - 1 prompts as Zi,j from (ITE, (ici,, ) M ((Mr.) }. The selected j - 1 prompts act as the model input to predict the yı.j. Similarly, in normal similarity case, we just use ( H== jmm input. In low similarity case, we select the least similar j - 11 prompts as aay from ( k) 1000 ([ } to act as input. In this way, we generate prompts with different similarities as the test query.
construct
D.5 Decision Tree and 2-Layer Neural Network
Following similar setting as previous linear regression and (Garg al., 2022), we show the following details about the decision tree and 2-layer neural network. In addition to the results shown in Figure 2 and Figure 3, we report exact values at specific points in Table 5, Table 6, and Table 7.
For the decision tree, we consider the function of as decision tree with 20-dimensional inputs and depth of 4. Each function f uses a full binary tree with 16 leaf nodes. Non-leaf nodes specify input coordinates. Leaf nodes assign output values. To compute f(x), begin at the root. Navigate through the tree. Move right if the coordinate value is positive. Move left otherwise. The output fx) comes from the leaf node reached. We choose random coordinates for non-leaf nodes from {1. .......00) Draw leaf values from a normal distribution N 1).
24

---

In- Learning as Conditioned Associative Memory Retrieval
For the 2-layer neural network, we consider ReLU neural networks. We set each function / as f(x) Ei= aw s)) where 04 IR, Wi E IRd, and (()) max(0,,) .) is the ReLU activation function. We draw network parameters O₃ and w; from N (0,2/r) and (0, La). We use the number of hidden nodes as 100.
Table 5. Sensitivity to Covariance Shifts. We seleet specific values for ICL prompt number at 15, 30, 45, 60 from Figure and Figure 3. Note that performance decreases when the prompt length exceeds the pre-training length (i.c., 50), a well-known issuc (Dai et al., 2019; Anil al., 2022). We believe this is due to the absolute positional encodings in GPT-2, as noted in (Zhang al., 2024)
Decision Tree
2-Layer Neural Network
Test Distribution
15
30
45
60
15
30
45
60
Baseline
0.806
0.862
0.854
0.875
0.886
0,946
0.953
0.962
N--2, I)
0,866
0,943
0.972
0.943
0.901
0,968
0.979
0.956
0.9N +
0.863
0.874
0.967
0.943
0.902
0.954
0.968
0.952
0.7N +..3),.
0.408
0.658
0.671
0.654
0.765
0.845
0.881
0.858
Table 6. Sensitivity to Response's Accuracy. We select specific values for in-context learning prompt number at 15, 30, 45, 60 from Figure and Figure 3.
Decision Tree
2-Layer Neural Network
Response Accuracy
15
30
45
60
15
30
45
60
Baseline
0.505
0.801
0.733
0.849
0.907
0.924
0.953
0.961
100%
0.737
0.964
0.967
0.858
0.921
0.960
0.978
0.964
90%
0.737
0.921
0.967
0.875
0.921
0.947
0.964
0.959
80%
0.737
0.921
0.948
0.796
0.902
0.903
0.946
0.940
D.6 GPT-J on Sentiment Classification Task
In this part, we evaluate the in-context learning (ICL) behavior of the GPT-J model, similar to (Min et al., 2022). We use the sentiment classification task with the "TweetEval: Hate Speech Detection" dataset (Basile et al., 2019).
Sensitivity Covariance Shifts. To assess how sensitive the model is domain shifts in the in-context examples, we conduct experiments using two settings: (i) in-distribution in-context examples from the TweetEval dataset, and (ii) out-of-distributie (OOD) in-context examples from the CC-News corpus (Nagel, 2016), which consists of long-form formal news articles. To ensure fair results, we control for the average sentence length between the two sources during sampling. We use GPT-J (6B parameters) following the protocol in (Min et al., 2022), and compare the performance across two configurations: zero-shot (k = 0) and few-shot (k = 4) with both in-distribution and OOD examples. We show the results Table
Table Sensitivity to Similarity with Test Query. We select specific values for in-context learning prompt number at 15, 30, 45, 60 from Figure 2 and Figure 3.
Decision Tree
2- 2-Layer Neural Network
Similarity Degree
15
30
45
60
15
30
45
60
Baseline
0.880
0.880
0.871
0.895
0.917
0.943
0.960
0.962
Best
0.955
0.976
0.987
0.971
0.976
0.988
0.991
0.988
Normal
0.971
0.957
0.976
0.942
0.931
0.972
0.981
0.967
Worst
0.907
0.937
0.943
0.927
0.742
0.866
0.895
0.882
25

---

In- Learning as Conditioned Associative Memory Retrieval
Table Sensitivity Covariance Shifts on Real Dataset. Macro-Fl scores for sentiment classification ("TweetEval: Hate Speech Detection" dataset (Basile et al., 2019)) with & = 0,4 in-context examples under in-distribution (No-shift) and of--oirrbiiittionn (Shift) settings.
In-context Examples
4
No-shift
0.3722
0.5313
Shift
0.3521
0.3830
Beyond domain alignment, we further analyze the impact of in-context response correctness on model performance. We use in-context examples, and create variants of the in-context examples which the percentage of correct query-response pairs is from 100% 25%. We show the results in Table 9.
Sensitivity to Response's Accuracy.
Table Sensitivity to Response's Accuracy Real Dataset. Macro-Fl scores for sentiment classification ("TweetEval: Hate Speech Detection" dataset (Basile et al., 2019)) with 4 in-context examples under different proportions of correct labels.
Response's Accuracy (%) 100
75%
50%
25 25%
In-context Examples
0.5313
0.4212
0.4035
0.3640
D.7 Ghost Example Construction
In this part, we show the details about how to generate the required ghost examples. Firstly, we freeze all the model parameters. For each E [75]. we generate samples as the following: We randomly choose 100 samples. For the i-th sample, we randomly select ri,++1 from w/ W I) + (2, I), where w1 = 1, W2 = 0 or w1 = 0.5, W2 = 0.3. Then we set {ū,,, Jetn as the trainable parameters. We use MSE loss to train the parameters (Z., Fi.s yejo] in each batch, until the loss convergences for this batch. We calculate the convergence loss value for one batch and use the batch loss the ghost example construction loss for the index f. In this way, we get the ghost example construction loss for each f [75] and two distribution settings, where one is the same as the training data distribution and another is different from it. The following Table 10 show the mean squared error of the ghost example construction for & E [75).
Table 10. Mean Squared Error of Ghost Example Construction. The data are from two different distributions. 15, 30, 45, 60, 75 denote the in-context examples size.
Data Distribution
30
45
60
75
wi 1, W2 =
1.50e-06 0.0010
1.17e-05
3.28e-05
0.006
0.0001
0i 0.5, u2 0.5
0.0070
0.0049
0,0001
0.0011
26


Reviews:
[{"review_id": "9V1xCI3tnf", "reviewer": "Reviewer_yDv9", "summary": "This paper proposes a model to interpret in-context learning as associative retrieval. The paper argues that in-context learning can be seen as a linear transformation of key-value weights. Under a GLM assumption, their modified attention construction (BMA Attention) is shown to approximate softmax attention and converge to the presumed target, in the limit of infinite in-context examples in the prompt. Their theory is supported using a linear regression experimental setup within the ICL setting.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review. We have revised our draft and addressed all concerns. **The revised version (changes marked in BLUE) and the code for external experiments are available in this [anonymous Dropbox folder](https://www.dropbox.com/scl/fo/9m982gnk45wc1w705wjsa/AKYGxykAcCFGADGQEHCN218?rlkey=h03t7ipyv8qbt9bvesfivxcsw&st=ze6gjwhu&dl=0).**\n\n> **C1**: No mention of an energy function\n\n**Response**\n\nThanks for the comment.\n\nWe use a toy model (one-layer attention) to study the in-context learning of LLM. For the one-layer attention, the energy function of this toy model is equivalent to that of MHM. We argue such an atomic setting is prevalent for analytical feasibility in literature [XIe22, Zhang23]. \n\n[Xie22] An Explanation of In-context Learning as Implicit Bayesian Inference\n\n[Zhang23] What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization\n\n> **C2**: Notion of hidden concepts\n\n**Response**\n\nWe apologize for the confusion. \n\nThe main point arises from memory reshaping in Eq. (3.5). To interpret ICL using memory reshaping, we must incorporate synthetic prompts, as noted in Remark 3.2. In Eq. (3.7), we aim to demonstrate the formulation of Eq. (3.6) with memory reshaping, thus including the synthetic prompts.\n\nEq. (3.9) explicitly formulates Lemma 2.1, relating to the mathematical formulation of ICL. We incorporate memory reshaping in Eq. (3.9) to demonstrate how it influences the hidden concept and subsequently impacts ICL performance.\n\n> **C3**: Linear transformation and pseudo-inverse\n\n**Response**\n\nThanks for your insightful comments. **We acknowledge these limitations and have added them to the 2nd and 3rd terms of limitation part (Appendix A.2 of the revised version).** Please refer to the revised version for details. We promise to add a technical remark to the main text in the final version if space allows.\n\nFurthermore, it is acceptable that Assumption 3.1 does not define an explicit underlying sampling distribution, as the theoretical analysis remains valid under general distributional conditions without requiring a specific form.\n\n> **C4**: Missing proof sketches\n\n**Response**\n\nWe have added proof sketches for Lemma 2.1, Lemma 2.2, Lemma 3.1, and Theorem 3.1 ahead of the official proofs in the Appendix. We also provide the following high-level overview of our theoretical results.  We promise to move these to the main text in the final version, if space allows.\n\n**High-Level Understanding of Theoretical Results:**\n\n* Lemma 2.1: Interprets In-Context Learning (ICL) in Large Language Models (LLMs) through Bayesian Model Averaging (BMA).\n\n* Lemma 2.2: Shows that memory reshaping in Modified Hopfield Models (MHMs) reduces retrieval errors. Given the equivalence of the energy functions between MHMs and LLMs, this motivates us to apply memory reshaping to interpret ICL in LLMs.\n\n* Lemma 3.1: Integrates memory reshaping into the Bayesian Model Averaging framework, providing a more explicit interpretation of ICL under the Gaussian linear case than Lemma 2.1.\n\n* Theorem 3.1: Establishes the connection between Bayesian Model Averaging and Softmax Attention, showing that the explicit interpretation provided by Lemma 3.1 corresponds exactly to ICL within the attention mechanism.\n\n> **C5**: The experimental setup is rather general\n\n**Response**\n\nThank you for your helpful comment. We agree this is a limitation and have added it to the **6-th item of limitations section (Appendix A.2 of the revised version).** Please refer to the revised version for details. We also suggest one possible solution. However, verifying this solution is difficult. We leave that for future work.\n\n> **C6**: Linear regression is simple\n\n**Response**\n\nThanks for your valuable comment.\n\nIn the revised version, we extend our experimental evaluation to cover more complex tasks. Specifically, we include new results using GPT-2 on decision tree and 2-layer neural network tasks, following a similar setup as in the linear regression experiments. Additionally, we incorporate experiments with the GPT-J model on the real-world “TweetEval: Hate Speech Detection” dataset to evaluate performance on a practical sentiment classification task. Please refer to Section 4.1.4 for details. The results follow a similar pattern to the linear regression setting, consistent with our theoretical predictions.\n\nWe apologize for the confusion. We use the cosine similarity between different queries as the similarity measure. We have added this and other details to the revised version.\n\n> **C7**: Essential references \n\n**Response**\n\nThank you for your question. We have included a comparison with these three works in the revised version. Please see Appendix A.1 of the revised version for details.", "claims_and_evidence": "There are a lot of claims made in the paper, but often, the corresponding analysis/results either does not present strong enough evidence to establish them or connect them back well to the larger picture. I list these below:\n \n- Connection to Associative Memory: In line 35, col 2, it is claimed that the paper defines an energy function for LLMs to interpret it as a Hopfield model and interpret ICL as reshaping attention heads. However, in the following discussion in 3.1, I see no mention of an energy function for the LLM, but only an analysis using a modified attention mechanism. Thus, the only connection to retrieval seems to be the proposed memory reshaping mechanism, which appears as a tenuous link to reducing retrieval error in Lemma 2.2. \n\n- Relevance of Hidden Concepts: I am unsure as to how the notion of hidden concepts introduced in section 2.2 fits the larger context. Specifically, the generative model (with a GLM assumption) is used in section 3.2 to yield a certain prediction model that approximates z* via bar{z}t. However, I do not follow why the true target v{t+1} would be a function of the synthetic prompt form in 3.7, or what intuition does 3.9 offer within the larger context of the associative retrieval.  \n\n- Mathematical Impreciseness: In 3.4, it is implicitly assumed that there exists a linear transformation to go from the ghost prompts to the true keys and values, which will simply not be true if the ghost examples constitute of lower rank matrices. The discussion should note that the pseudo-inverse structure only minimizes the approximation error to the true matrix and does not directly approximate it. Similarly, in Assumption 3.1, (k, v) are assumed to be iid, without defining any underlying sampling distribution, either over them or the original (c, r) pairs. In addition, I am not even sure if this assumption holds true either in their model (which is a 1-layer transformer operating directly on embeddings) because of the weight matrix or the joint optimization process over the ghost examples that naturally introduces dependencies or in practice, since the justification that layer norm will somehow achieve this is not clear in the sense that layer norm only operates on a per token basis and affects no intra batch dependencies.\n\n- Missing Proof Sketches: The proof sketch is missing for certain theorems and lemmas. But more importantly, the intuition on how to interpret this result in the larger context of the paper is often missing as well, meaning that as a reader, it is hard to get a high-level understanding of what's being said. \n\nIt is certainly possible that I missed some links that the authors had in mind while writing the paper. However, I still stress that they should be made more explicit to make the paper more coherent for the general reader.", "methods_and_evaluation_criteria": "While the experimental validation of the proposed theory is not the main contribution of the paper, there are some issues with it as well.\n\n- The paper attempts to validate the theoretical claims with linear regression ICL experiments, interpreting their findings from a memory shaping viewpoint. However, the experimental setup is rather general, and the resulting trends would be predictable even without any intuition of reshaping. I understand that a claim like this is not easy to establish in general, but the current setup does not lend a lot of extra credence either. \n\n- The setting of linear regression is perhaps not the most informative for LLMs, and the authors should consider other experiments on language centric datasets [see setups in 1, 2] for a stronger justification. Additionally, certain details for the experimental setup (such as how to measure similarity between examples in 4.1.3) are missing and need to be specified.\n\nReferences:\n\n[1]:  In-Context Learning with Many Demonstration Examples - Li et al., 2023\n\n[2]: Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition - Xiong et al., 2024", "theoretical_claims": "Proofs were checked on a high level, with no obvious issues found.", "experimental_designs_or_analyses": "See Methods and Evaluation Criteria", "relation_to_broader_scientific_literature": "The paper belongs to the line of work which interprets certain mechanisms in transformers as a retrieval in associative memories, with the primary contribution being an explicit mechanism of memory reshaping to explain ICL.", "essential_references_not_discussed": "There is some missing discussion of related literature and what exactly are the paper's unique contributions. \n\n- The paper borrows the setup heavily from [1], but the line to demarcate their own novel results from the prior work is not made very clear for the reader.\n- Discussion of ML theory work [see 2, 3 as a starting point] that interprets Transformers as associative memories, and how it differs from their work is missing.\n\n\nReferences:\n\n[1] What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization - Zhang et al., 2023\n\n[2] Birth of a Transformer: A Memory Viewpoint - Bietti et al., 2023\n\n[3] Scaling Laws for Associative Memories - Cabannes et al, 2024", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HzT5lsL7tL", "reviewer": "Reviewer_oD5q", "summary": "**Main Findings**:\n\n1. Memory Reshaping: This paper proposes that ICL can be understood as a process of memory reshaping within the Hopfield model framework. Specifically, the input prompt examples can reshape the energy landscape of the probabilistic energy-based memory model, thus relocating the distribution of local minima.\n\n2. Bayesian Model Averaging with Reshaped Memory: The authors formulate the explicit expression, demonstrating that ICL performs conditional estimation based on this hidden concept. Besides, softmax attention can approximate Bayesian Model Averaging.\n\n3. Generalization Bound: This paper derives a generalization bound for ICL in a one-layer attention model, demonstrating that the performance of ICL improves with an increasing number of prompt examples.\n \n**Main Algorithmic/Conceptual Ideas**:\n\n1. Memory Reshaping Mechanism: This paper introduces the concept of memory reshaping, where the pre-trained attention mechanism's weights are linearly transformed by the prompt examples. \n\n2. Hidden Concept Learning: The authors argue that ICL implicitly learns a hidden concept from the prompt examples, which is then used for conditional estimation. This hidden concept is shown to be a shared property among the prompt examples.\n\n3. Hopfield Model Interpretation: This paper provides a new perspective on how transformers process and adapt to context by framing ICL as memory retrieval in the modern Hopfield model. \n\n**Main Results**:\n\n1. Theoretical Results: This paper provides mathematical formulations and proofs for memory reshaping, the connection to BMA, and the generalization bound. \n    \n2. Empirical Results: The experiments confirm the theoretical predictions, showing clear trends in ICL performance with respect to covariance shifts, response accuracy, and prompt-test query similarity.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review. We have revised our draft and addressed all concerns. **The revised version (changes marked in BLUE) and the code for external experiments are available in this [anonymous Dropbox folder](https://www.dropbox.com/scl/fo/9m982gnk45wc1w705wjsa/AKYGxykAcCFGADGQEHCN218?rlkey=h03t7ipyv8qbt9bvesfivxcsw&st=ze6gjwhu&dl=0).**\n\n> **C1**: Limited Scope\n\n**Response**\n\nThanks for your insightful comment. We provide an overview of how to extend our analysis to multi-layer models. The detailed discussion appears in the Section A.2 of the revised version, and we restate it here. \n\nWe regard the multi-layer transformer as iteratively retrieving memory patterns in the modern Hopfield model (MHM) through the following four steps:\n\n* Design energy function for multi-layer MHM: We design an energy function matching the multi-layer transformer [Hoover23]. The memory pattern needs retrieval after reshaping and iteration. This step is challenging.\n\n* Give clear expression of memory reshaping on model weights: This step is feasible following our framework in Sec. 3.1.\n\n* Extend Bayesian method to multi-layer transformer: The Bayesian method in [Zhang23] and our work assume perfect pre-training. We must study perfect pre-training to match real multi-layer transformers. This step is challenging.\n\n* Obtain generalization bound of ICL: This step is feasible following our framework in Appendix D.5. We need to derive the ϵ-stability of the multi-layer transformer.\n\n[Zhang23] What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization\n\n[Hoover23] Energy Transformer\n\n> **C2 & Q1**: Linear Regression Task: Is the proposed interpretation general to more complicated models and more challenging tasks?\n\n**Response**\n\nThanks for your valuable question. The answer is **Yes**.\n\nIn the revised version, we extend our experimental evaluation to cover more complex tasks. Specifically, we include new results using GPT-2 on decision tree and 2-layer neural network tasks, following a similar setup as in the linear regression experiments. Additionally, we incorporate experiments with the GPT-J model on the real-world “TweetEval: Hate Speech Detection” dataset to evaluate performance on a practical sentiment classification task. Please refer to Section 4.1.4 for details.\n\n> **C3 & Q2**: Lacking novel guidelines: Are there more further guidelines your interpretation can provide?\n\n**Response**\n\nThank you for your comment. Our interpretation indeed offers further guidelines that help explain phenomena like hallucination in LLMs and avoid hallucination. For example:\n\n* **Biased Hidden Concept**:\nEqs. (2.2)–(2.4) illustrate how the hidden concept influences the in-context learning response under a Bayesian approach. In Lemma 3.1, we derive an explicit expression for this hidden concept from the memory reshaping perspective. Irrelevant or inconsistent prompt examples create a biased hidden concept, which in turn leads to hallucination.\n\n* **Insufficient Separation of Memory Patterns**:\nThe modern Hopfield model (MHM) shows that the separation of memory patterns affects retrieval error. We conjecture that unreasonable prompt examples reduce the separation between different memory patterns, increasing the retrieval error and contributing to hallucination.\n\nOur framework provides a clear and practical guideline: to reduce hallucination, prompt examples should be relevant and consistent with the test input. To validate this, we refer to the experiments on the model’s sensitivity to the similarity between prompts and test examples. These experiments can be used here, including those with the GPT-2 model on linear regression, decision tree, and 2-layer neural network tasks.", "claims_and_evidence": "The claims made in the paper are generally well-supported by clear and convincing evidence. The theoretical analysis provides a solid foundation, and the empirical experiments validate the key findings.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria in the paper are highly suitable for the problem of understanding ICL in LLMs. The theoretical framework is rigorous, and the experimental design effectively validates the claims. The choice of metrics and baselines provides a clear and interpretable evaluation framework. \n\nHowever there is still room for further exploration in more diverse tasks and more models. Considering that the model used in the paper  (GPT-2) and the task (linear regression) are rather simple, whether the findings are general to more complex models and tasks remains to be proved.", "theoretical_claims": "The theoretical claims made in the paper are supported by clear and convincing evidence. The proofs are mathematically rigorous and logically sound, and the empirical results validate the theoretical predictions.", "experimental_designs_or_analyses": "The experimental designs and analyses in the paper are generally sound and well-structured, providing clear evidence for the theoretical claims. The choice of tasks, models, and metrics is appropriate for the problem at hand. However, there are areas where further exploration could strengthen the findings and enhance the generalizability of the results.\n\n1. Task Diversity: Testing the framework on more complex tasks (e.g., classification, natural language inference) could provide deeper insights into the behavior of ICL.\n2. Model Diversity: Evaluating the framework on more advanced models (e.g., GPT-3, GPT-4) could strengthen the generalizability of the findings.\n3. Real-World Data: Using real-world datasets and more diverse distribution shifts could provide a more realistic evaluation of ICL.", "relation_to_broader_scientific_literature": "This paper offers a new perspective on In-Context Learning by interpreting it as memory retrieval in the modern Hopfield model. This novel interpretation can bring contributions to this area.\n\nHowever, the guidelines provided can be further improved. Most of the guidelines are quite intuitive like 'select similar examples' and 'use more similar examples'. This resricts the contributions to the broader scientific literature. If the authors could provide more novel guidelines, I believe this will bring more contributions to not only the interpretability of ICL and LLM, but can also help the users of LLM.", "essential_references_not_discussed": "No missing related works.", "other_strengths_and_weaknesses": "**Strength**:\n\n1. Novel Interpretation of ICL: The paper offers a new perspective on In-Context Learning (ICL) by interpreting it as memory retrieval in the modern Hopfield model. This interpretation is creative and provides a fresh understanding of how ICL works.\n   \n2. Theoretical Contributions: The paper makes significant theoretical contributions by deriving a generalization bound for ICL and showing how softmax attention can approximate Bayesian Model Averaging. These results provide a deeper understanding of the underlying mechanisms of ICL.\n   \n3. Clarity: The paper is well-organized and clearly presented. The theoretical framework is explained in detail, and the mathematical derivations are rigorous and easy to follow.\n   \n**Weaknesses** :\n\n1. Limited Scope: The theoretical analysis focuses on a single-layer attention model, which may not fully capture the complexity of modern LLMs with multiple layers. Extending the analysis to multi-layer models would strengthen the findings.\n    \n2. Linear Regression Task: The experiments are limited to a linear regression task, which is relatively simple. Testing the framework on more complex tasks would provide a more comprehensive evaluation of ICL. \n    \n3. Lacking novel guidelines: This paper provides a novel understanding of ICL and get several guidelines. However, most of the guidelines are rather intuitive, like \"As demonstrated in Section 4.1.3, selecting prompts similar to the test query enhances in-context learning performance.\" and \"By Theorem 3.2, increasing the number of relevant prompt examples reduces the ICL generalization error\". These guidelines of course can prove the proposed understanding, but may lack contributions. If the authors can further reach a novel guideline based on the proposed interpretation theoretically, and prove it empirically, this paper will bring more contributions to the research area.", "other_comments_or_suggestions": "No further comments.", "questions_for_authors": "1. Is the proposed interpretation general to more complicated models and more challenging tasks?\n2. Are there more further guidelines your interpretation can provide?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OaTp8beH7m", "reviewer": "Reviewer_uChw", "summary": "This paper considers, both theoretically and empirically, how a single attention head can be seen to perform in context learning (ICL) via the reshaping of an energy landscape shaped by the prompt. This work addresses the following topics:\n\n1. Analyzing the effect of prompts through an interpretation of attention as the one-step Modern Hopfield Model (MHM) introduced by Ramsauer et al. 2020\n2. Extending the work extends the work of [Zhang et al. 2023b] on Bayesian Model Averaging (BMA) for ICL to attention\n3. Characterizing bounds on the performance of ICL based on the quantity and quality of prompt examples", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review. We have revised our draft and addressed all concerns. **The revised version (changes marked in BLUE) and the code for external experiments are available in this [anonymous Dropbox folder](https://www.dropbox.com/scl/fo/9m982gnk45wc1w705wjsa/AKYGxykAcCFGADGQEHCN218?rlkey=h03t7ipyv8qbt9bvesfivxcsw&st=ze6gjwhu&dl=0).**\n\n> **C1**: Single next token\n\n**Response**\n\nWe provide details on how to extend our method to multiple tokens in the 5th term of the limitation section (Appendix A.2) of the revised version. Please see the revised version for details.\n\n> **C2**: Simplifies Transformers dramatically\n\n**Response**\n\nWe provide an overview of how to extend our analysis to multi-layer models. The detailed discussion appears in the 4th term of the limitation section (Appendix A.2) of the revised version. Please see the revised version for details.\n\n> **C3**: Interpretation of the experimental results\n\n**Response**\n\n* **Generalization Bound**: Figures 1, 2, and 3 show that the R-squared value increases when the number of in-context examples is below 50—the same number used during pretraining. This supports our observation that the generalization bound improves with more prompt examples.\n\n* **Dashed Gray Lines**: These indicate the threshold of 50 in-context examples, which matches the pretraining setup.\n\n* **Performance Decline Beyond 50**: During pretraining, we use a prompt length of 50 (Appendix E.1). When evaluated on longer contexts, GPT-2 shows a drop in R-squared. We conjecture that this is due to its use of absolute positional encodings (Remark 4.1).\n\n> **C4**: Motivation and benefits of the experiments\n\n**Response**\n\n**Motivation for the experiments**: \nWe design these experiments to validate three key properties of ICL: the model’s sensitivity to (i) covariance shifts, (ii) response accuracy, and (iii) the similarity between prompts and test examples. We then provide theoretical explanations for these properties based on our results. Our empirical and theoretical validations provide the practical guidelines for improving ICL performance.\n\n**Benefits of this approach**:\nMemory reshaping provides a theoretical explanation for three key properties of ICL.\n\n**Our results can scale beyond the toy evaluations well**. In the revised version, we extend our experimental evaluation to decision trees, 2-layer neural network tasks, and practical sentiment classification tasks. Please refer to Section 4.1.4 for details.\n\n> **C5**: Typos\n\n**Response**\n\nWe have added the definition and corrected the typos in the revised version. The reshaped attention head refers to a linear transformation of the key and value matrices. \n\n> **Q1**:\n\n**Response**\n\nYes, we apply it as $\\tilde{\\xi}_k = W \\xi_k$.\n\n> **Q2**:\n\n**Response**\n\nWe consider that transformers take all prompt examples at once, and the attention weights are not updated.\n\n> **Q3**:\n\n**Response**\n\nYes, we consider the single token case.\n\nHowever, there is no constraint requiring $d_c = d_r = d$, as stated in the second paragraph of Section 3.1.\n\nOur method can also be extended to multi-token settings, with details provided in the Limitations section (Appendix A.2) of the revised version.\n\n> **Q4**:\n\n**Response**\n\nYes, in Eq. (2.1), we do not consider predicting multiple tokens.\n\nIn Eq. (2.1), we denote the pretrained LLM as $P_{\\theta}$, where $\\theta$ represents the pretrained model parameters. $C_t$ and $R_t$ are independent of $\\theta$, Eq. (2.1) simply describes using $P_{\\theta}$ to predict $c_{t+1}$ based on the prompt $C_t$ and $R_t$.\n\n> **Q5**:\n\n**Response**\n\nWe apologize for the confusion. The answer is No.\n\nThe importance of the ghost example is for mathematical consistency. Here are some clarifications.\n\n* As shown in Remark 3.1, we use the proxy attention based on the BMA formulation, and this is not the actual attention in LLM. \n\n* Consider the setting where we only have the input $c_{t+1}$ without $C_t$ and $R_t$. For the proxy attention mechanism equation (3.1), $K_t$ and $V_t$ have no meaning. \n\n* Therefore, we construct $t$ ghost examples and prepend them before $c_{t+1}$. \n\n* The aims are that the Eq. (3.1) is reasonable and the t ghost examples do not change the response of the $c_{t+1}$ in ICL.\n\n> **Q6**:\n\n**Response**\n\nWe apologize for the confusion. Due to space constraints, we provide the full details in Appendix E. Below, we summarize the core idea:\n\n* Pretraining of GPT-2: We use the GPT-2 architecture but pretrain it on synthetic data generated from a linear regression setting. Each pretraining sample contains 50 query-response pairs. We apply MSE loss as training loss to predict the response for each pair.\n\n* As described in Appendix E.2, we evaluate model performance when the number of in-context examples is less than, equal to, or greater than 50. We use R-squared as the evaluation metric and report values at 15, 30, 45, 60, and 75 examples in Table 1. The full R-squared curve is shown in Figure 3.1(a).", "claims_and_evidence": "C1. \"Equivalent memory reshaping\" of the pretrained weights in attention aptly describes ICL\nC2. BMA explains how ICL works by performing BMA on the constructed hidden concept\nC3. A one-layer attention model using reshaped memory has a clear generalization bound for ICL\nC4. Equivalent memory reshaping in ICL explains model sensitivity to:\n    (i) covariance shifts\n    (ii) response accuracy\n    (iii) similarity between prompts and test examples\n\nClaims C1-C3 are fully supported by the strong theoretical contributions. I am not sufficiently confident in the empirical results to validate C4.", "methods_and_evaluation_criteria": "The method analyses is sound (though in its current form, limited to a single head of a single layer of attention). I have confusion about the experimental results. See weaknesses", "theoretical_claims": "There are many theoretical claims and corresponding proofs in the paper. I did not have time to review many of them.\n\n- Lemma 2.2: Memory reshaping in MHMs (by making the most similar stored pattern equal to the query) increases the separation ($\\Delta$) between stored memories. This proof is correct.", "experimental_designs_or_analyses": "The main experimental question is determining whether memory reshaping predicts observable behaviors of ICL, which the paper studies by considering a toy dataset consisting of a mixture of Gaussians. It is not clear from the presentation of the work how the experiments relate to the theory proposed earlier in the work, and it would be helpful if the theoretical claims (specifically, the bound) was verified in the experiments. See weaknesses.", "relation_to_broader_scientific_literature": "The paper links Hopfield Networks and the in-context learning behavior of LLMs in an interesting way. As far as I know, this is a novel perspective that may produce quite interesting insight when scaled to larger models.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "### Strengths\n- **Well organized**. The paper is well organized and written. Despite no familiarity in formally studying ICL, I was able to piece together the fundamental problem that the authors are trying to solve from their background sections.\n- **Novel and meaningful contributions**. The main paper represents, as far as I know, a rigorous and novel perspective of ICL as memorization and energy minimization. Whether this scales to real Transformers with multiple heads of attention and multiple distinct layers is unclear (see weaknesses), but the approach is nonetheless meaningful.\n\n### Weaknesses\n- **Toy evaluations may not extend to language**. The theory and experiments consider ICL like a regression task, where the task of the model is only to predict the single next token. In traditional AR LLMs, ICL may be used to predict multiple tokens, and it is unclear how this paper generalizes to real scenarios and real-sized models.\n- **Theory and evaluations are limited to a single layer, single attention head**. Again, the study simplifies Transformers dramatically. Unclear how it will generalize\n- **Unclear takeaways from the experimental results**. It is not clear how to interpret the experimental results. Specifically, where do we see plotted the generalization bound against empirically observed behavior? What do the dashed gray lines mean in Figs. 1 & 2? Why does performance in Tables 1,2, & 3 peak at 45 ICL samples and decline (rather significantly) after? Please make it easier to understand how the experiments align with the theory\n\n\n### Summary\nThere are core aspects of this work that I did not understand going into this paper, but the authors gave a proper background to the topic and presented their results in a clear and organized way. However, the paper has its flaws. The missing motivation for the experiments and the minimal discussion of the experimental results makes it unclear what the benefits of this approach are. It is also unclear if/how well the analyses will scale beyond the toy evaluations and models studied in this work.\n\nNonetheless, the strong theoretical analyses, the novel perspective, overal professionalism of the paper compel me to believe this paper is a proper fit for ICML. I am happy to increase my score if any of my misunderstandings can be clarified throughout the rebuttal process.", "other_comments_or_suggestions": "- \"Reshaped attention head\" is mentioned for the first time in [L039 right col], but is not common knowledge and is not defined at this point in the paper.\n\n### Typos\n- MHM in the abstract is not defined, and MHM is not an appropriate acronym for \"associative memory model\" as mentioned in the introduction. It is first introduced in [L029 right col], but should be mentioned earlier.\n- L368 \"We use 4 different GM distribution\" when only 3 are listed?", "questions_for_authors": "Q1: Lemma 2.2: How does reshaping matrix $W$ play into the **energy function** of the MHM? Is it applied s.t. $\\tilde{\\xi}_k = W \\xi_k$?\n\nQ2: Sec 3.1: I'm a little confused about memory reshaping in the context of a prompt. Generally, autoregressive transformers take all prompt examples at once and the attention weight is not updated. Is this paper stating that attention processes all prompt tokens at once? Or that there is some kind of recurrence over the tokens?\n\nQ3: How should I interpret $d_c$ and $d_r$ in [L064 right col]? Do you consider a case where a each query and response is only a single token (in which case $d_c = d_r = d$, the embedding size of the model)? This is not described.\n\nQ4: Eq. (2.1) is a bit confusing. It seems to indicate that the LLM is pretrained only to predict queries $c_{t+1}$, but  autoregressive LLMs trained to predict all next tokens. Also, $C_t$ and $R_t$ are from the prompt, not necessarily the pretraining set. What is the connection here?\n\nQ5: Can you explain why the \"ghost examples\" [L200-211] are necessary for testing this theory? Is it only to make it so the experiments vectorize across GPUs correctlY? I'm afraid I don't understand why ghost examples are necessary for the theory or the experiments.\n\nQ6: It is not clear to me how the results in Table 1 use the GPT-2 model at all. Can you please explain the experiments in more detail?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p4PsCano6J", "reviewer": "Reviewer_HbbS", "summary": "This paper proposes a novel interpretation of in-context learning(ICL)as the memory retrieval of the modern Hopfield model from a conditional memory set. The paper is well-written and the techniques proposed are novel.\n\nStrengths: The innovative Hopfield-based perspective, solid theoretical framework, clear structure, and intuitive experimental focus are promising foundations for a strong paper.\n\nLimitation: Clarify key concepts, tighten mathematical rigor, expand experiments, complete missing sections, and better highlight novelty.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review. We have revised our draft and addressed all concerns. **The revised version (changes marked in BLUE) and the code for external experiments are available in this [anonymous Dropbox folder](https://www.dropbox.com/scl/fo/9m982gnk45wc1w705wjsa/AKYGxykAcCFGADGQEHCN218?rlkey=h03t7ipyv8qbt9bvesfivxcsw&st=ze6gjwhu&dl=0).**\n\n> **C1**: Linear Regression\n\n**Response**\n\nThanks for your comment. In the revised version, we extend our experimental evaluation to cover more complex tasks. Specifically, we include new results using GPT-2 on decision tree and 2-layer neural network tasks, following a similar setup as in the linear regression experiments. Additionally, we incorporate experiments with the GPT-J model on the real-world “TweetEval: Hate Speech Detection” dataset to evaluate performance on a practical sentiment classification task. Please refer to Section 4.1.4 for details.\n\n> **C2**: Extending to multi-layer setting\n\n**Response**\n\nThanks for your insightful comment. We provide an overview of how to extend our analysis to multi-layer models. The detailed discussion appears in the Section A.2 of the revised version, and we restate it here. \n\nWe regard the multi-layer transformer as iteratively retrieving memory patterns in the modern Hopfield model (MHM) through the following four steps:\n\n* Design energy function for multi-layer MHM: We design an energy function matching the multi-layer transformer [Hoover23]. The memory pattern needs retrieval after reshaping and iteration. This step is challenging.\n\n* Give clear expression of memory reshaping on model weights: This step is feasible following our framework in Sec. 3.1.\n\n* Extend Bayesian method to multi-layer transformer: The Bayesian method in [Zhang23] and our work assume perfect pre-training. We must study perfect pre-training to match real multi-layer transformers. This step is challenging.\n\n* Obtain generalization bound of ICL: This step is feasible following our framework in Appendix D.5. We need to derive the ϵ-stability of the multi-layer transformer.\n\n[Zhang23] What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization\n\n[Hoover23] Energy Transformer\n\n> **C3**: Notation Inconsistencies\n\n**Response**\n\nWe thank the reviewer for pointing out the potential notational inconsistency. $C_t$ and $R_t$ are written as matrices for clarity and convenience. In practice, they represent token sequences processed by the LLM in an autoregressive manner.\n\n> **C4**: Pseudo-Inverse Usage\n\n**Response**\n\nThank you for raising this point. We acknowledge that the use of pseudo-inverses can indeed present numerical stability challenges. In this section, we primarily focus on the theoretical analysis of the in-context learning mechanism. We recognize that practical applications may require additional considerations, such as the implementation of regularization techniques, to address these stability concerns effectively. We add this to the second point of the limitation part (Appendix A.2).\n\n> **C5**: Generalization Bound\n\n**Response**\n\n* For the $\\epsilon$-stability part, we want to clarify that under the reshaping framework, the softmax attention maintains its form and properties. The derivation of $\\epsilon$-stability is not dependent on how the model performs memory reshaping. Hence, the attention mechanism satisfies $\\epsilon$-stability.\n\n* The constants $(M, C_0, \\Gamma)$ in our bound are standard parameters in theoretical analysis. $M$ represents the Lipschitz constant of the loss function. $\\Gamma$ bounds the operator norm of weight matrices. $C_0$ is also a norm bound. These constants are commonly used in generalization bounds. They can be estimated empirically or bounded using standard techniques in learning theory.", "claims_and_evidence": "The claims made in the submission are generally clear, but there is one minor problem.\nClaim 2 (Hidden Concept Learning via Reshaping):\nProblematic: The evidence is theoretical and relies heavily on a simplified Gaussian linear model, with no robust empirical validation in complex LLM settings. The ghost example construction (Appendix E.5) is a weak link—its small MSE values suggest feasibility, but there’s no analysis of its representativeness or stability, and the process is under-specified (e.g., optimization details). This claim needs stronger empirical support beyond linear regression.\nClaim 3 (Generalization Bound) :\nProblematic: The experimental support is solid for linear regression but doesn’t fully convince for broader ICL applications due to task simplicity.", "methods_and_evaluation_criteria": "1. The methods are heavily theoretical and tailored to a one-layer attention model, which oversimplifies multi-layer LLMs like GPT-2. Extending to multi-layer settings would better match the application. Ghost example construction lacks a clear benchmark or prior method for comparison, making its novelty sensible but its practicality uncertain.\n2. Linear regression doesn’t capture ICL’s complexity (e.g., semantic understanding, multi-task learning), undermining applicability to real LLMs. While controlled, it lacks the richness of benchmark datasets (e.g., GLUE, SuperGLUE), reducing ecological validity.", "theoretical_claims": "The theoretical claims made in the paper proved to be correct.", "experimental_designs_or_analyses": "1. Sensitivity to Covariance Shifts (Section 4.1.1, Table 1, Appendix E.2)\nStrengths: The design controls the test distribution shift, aligning with the claim that ICL is sensitive to covariance changes. Using multiple prompt sizes tests the generalization bound (Claim 3). Synthetic data allows precise manipulation, which is sensible for isolating this effect.\nIssues:\nOverly Simple Task: Linear regression is too trivial for GPT-2, which is designed for complex NLP tasks. Covariance shifts in this context (Gaussian mixtures) don’t mirror real-world ICL challenges (e.g., domain shifts in text).\nSample Size: 6400 samples (64 batches × 100) is reasonable but lacks justification for statistical power. No confidence intervals or significance tests are reported, weakening validity.\nPrompt Length Extension: Extending prompts beyond pretraining (49 to 75) tests generalization but assumes GPT-2 can extrapolate, which isn’t validated.\n2. Sensitivity to Response Accuracy (Section 4.1.2, Table 2, Appendix E.3)\nStrengths: Permuting labels to simulate inaccuracy is a reasonable way to test sensitivity, directly addressing Claim 4. Multiple prompt sizes align with the generalization bound hypothesis.\nIssues:\nPermutation Method: Randomly permuting labels introduces noise but doesn’t model realistic errors (e.g., systematic biases in prompts). The permutation rate ($1 - \\alpha$) is applied uniformly, which may not reflect varying error distributions.\nTask Simplicity: As with covariance shifts, linear regression limits the experiment’s relevance to ICL’s typical complexity.", "relation_to_broader_scientific_literature": "The paper outlines four main contributions:\nEquivalent Memory Reshaping in ICL: Prompt examples reshape pretrained weights in the attention mechanism, interpreted as memory retrieval in a modern Hopfield model (MHM).\nBMA with Reshaped Memory: Constructs an explicit hidden concept expression via reshaped weights, showing ICL performs Bayesian Model Averaging (BMA) based on this concept.\nGeneralization Bound in ICL: Derives a bound for a one-layer attention model, showing performance improves with more prompts at a rate of $\\mathcal{O}(1/t)$.\nExperimental Results: Uses reshaping to explain ICL’s sensitivity to covariance shifts, response accuracy, and prompt-test similarity, validated with linear regression experiments.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Notation Inconsistencies: The notation is occasionally sloppy or inconsistent. For instance, on Page 2, $P_\\theta(\\cdot \\mid C_t, R_t)$ is used to denote the LLM's predictive distribution, but $C_t$ and $R_t$ are matrices, while the conditioning typically expects a sequence. \nPseudo-Inverse Usage: In Section 3.1, the use of pseudo-inverses (e.g., $\\tilde{K}_t^+$ and $\\tilde{V}_t^+$) to define $W_1$ and $W_2$ assumes invertibility conditions that are not explicitly justified. If $\\tilde{K}_t$ or $\\tilde{V}_t$ are rank-deficient (common in high-dimensional settings), this formulation may be unstable or undefined without regularization.\nGeneralization Bound: The bound in Theorem 3.2, $\\mathbb{E}{P_t}[R(P_t) - R{\\text{emp}}(P_t)] \\leq \\frac{M C_0 (4\\Gamma + 1) e^{2\\Gamma}}{t}$, decreases as $\\mathcal{O}(1/t)$, but the proof sketch relies on $\\epsilon$-stability without justifying why the attention mechanism satisfies this property under the reshaping framework. The constant terms ($M$, $C_0$, $\\Gamma$) are also not well-constrained, reducing practical interpretability.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This is borderline paper. It formalizes in-context learning as a memory retrieval in a modern Hopfield network. After the discussion with the reviewers I believe that the reasons to accept this work outweigh reasons to reject. The camera-ready version should use clear language referring to the energy function for a one step attention operation rather than the entire LLM.
Decision: Accept (poster)