ICLR: In-Context Learning of Representations
Authors: Core Francisco Park
Andrew Lee
Ekdeep Singh Lubana
Yongyi Yang
Maya Okawa
Kento Nishi
Martin Wattenberg
Hidenori Tanaka
Venue: ICLR 2025 Poster (2025)
Keywords: In-Context Learning
Representational Geometry
World Models
Emergence
Percolation
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: Large language models develop emergent task specific representations given enough in-context exemplars.

Abstract: Recent work demonstrates that structured patterns in pretraining data influence how representations of different concepts are organized in a large language model’s (LLM) internals, with such representations then driving downstream abilities. Given the open-ended nature of LLMs, e.g., their ability to in-context learn novel tasks, we ask whether models can flexibly alter their semantically grounded organization of concepts. Specifically, if we provide in-context exemplars wherein a concept plays a different role than what the pretraining data suggests, can models infer these novel semantics and reorganize representations in accordance with them? To answer this question, we define a toy “graph tracing” task wherein the nodes of the graph are referenced via concepts seen during training (e.g., apple, bird, etc.), and the connectivity of the graph is defined via some predefined structure (e.g., a square grid). Given exemplars that indicate traces of random walks on the graph, we analyze intermediate representations of the model and find that as the amount of context is scaled, there is a sudden re-organization of representations according to the graph’s structure. Further, we find that when reference concepts have correlations in their semantics (e.g., Monday, Tuesday, etc.), the context-specified graph structure is still present in the representations, but is unable to dominate the pretrained structure. To explain these results, we analogize our task to energy minimization for a predefined graph topology, which shows getting non-trivial performance on the task requires for the model to infer a connected component. Overall, our findings indicate context-size may be an underappreciated scaling axis that can flexibly re-organize model representations, unlocking novel capabilities.

Published as a conference paper at ICLR 2025
ICLR: IN-CONTEXT LEARNING OF REPRESENTATIONS
Core Francisco Park 21.3 Andrew Lee"t, Ekdeep Singh Lubana" 1.3, Yongyi Yang
Maya Okawa' Kento Nishi'.4, Martin Wattenberg', & Hidenori Tanaka' 1.3
CBS-NTT Program in Physics of Intelligence, Harvard University
2 "Department of Physics, Harvard University
Physics & Informatics Lab, NTT Research Inc.
SEAS, Harvard University
6 CSE, University of Michigan, Ann Arbor
ABSTRACT
Recent work has demonstrated that semantics specified by pretraining data influ- ence how representations of different concepts organized in a large language model (LLM). However, given the open-ended nature of LLMs, e.g., their ability in-context learn, to adopt alternative, context-specified ones. Specifically, if we provide in-context exemplars wherein a concept plays a different role than what the pretraining data suggests, do models reorganize their representations in accordance with these novel semantics? To answer this question, we take inspiration from the theory of conceptual role semantics and define toy "graph tracing" task wherein the nodes of the graph are referenced via concepts seen during training (e.g., apple. bi rd, etc.) and the connectivity of the graph is defined via some predefined struc- ture (e.g., a square grid). Given exemplars that indicate traces of random walks on the graph, we analyze intermediate representations of the model and find that the amount of context is scaled, there is a sudden re-organization from pretrained semantic representations to in-context representations aligned with the graph structure. Further, we find that when reference concepts have correlations in their semanties (e.g., Monday, Tuesday, etc.), the context-speciffed graph structure is still present in the representations, but is unable to dominate the pretrained struc- ture. To explain these results, we analogize our task to energy minimization for predefined graph topology. providing evidence towards an implicit optimization process infer cppeext- semanties. Overall, our findings indicate scaling context-size can flexibly re-organize model representations, possibly unlocking novel capabilities.
ask whether models alter these pretraining semantics
INTRODUCTION
growing line of work demonstrates that large language models (LLMMs) organize representations of specific concepts in manner that reflects their structure in pretraining data (Park et al., 2024c;d; En- gels et al., 2024; Abdou al., 2021; Patel & Pavlick, 2022; Anthropic AI, 2024; Gurnee & Tegmark, 2023; Vafa et al., 2024; Li et al., 2021; Pennington al., 2014). More targeted experiments in synthetic domains have further corroborated these findings, showing how model representations are organized according the data-generating process (Li et al., 2022; Jenner al., 2024; Traylor al., 2022; Liu et al., 2022b; Shai et al., 2024; Park et al., 2024b; Gopalani et al., 2024). However, when model is deployed in open-ended environments, we can expect it encounter novel semantics for a concept that it did not see during pretraining. For example, assume that we describe to an LLM that a new product called st stabberry has been announced. Ideally, based on this context, the model would alter the representation for st strawbbery and reflect that we are not referring to the pretraining semantics (e.g., the fruit strawberry). Does this ideal solution transpire in LLMs?
Motivated by the above, we evaluate whether when provided an in-context specification of a concept, an LLM alters its representations to reflect the context-speciffed semantics. Specifically, we propose
(corefranci wwt.rraaccccccpri.oomoooo andrewl ee] hawvard. edu. yongyi @umi ch. edu, (ekdeep" ubana, hidanori tanaka ) fas. harvard.
* Equual contribution.
Contact:

---

Published as a conference paper at ICLR 2025
(a) Words on grid
(c) Emergent grid representation context
applo
aaplo-bbird bird
egg
house
milk
plans
opera
box
mango
rock
math
code
phone
(b) Data generation
Random walk grid:
"apple, bird, milk, sand, sun, plano, opera,
Context length: 200
Context length: 400
Context length: 1400
Figure 1: Alteration of representations in accordance with context-specified semantics (grid structure). (a) We randomly arrange a set of concepts on a grid that does not reflect any correla- tional semantics between the tokens. (b) We then generate sequences of tokens following a random walk on the grid, inputting it as context to a Llama-3.1-88 model. (c) The model's mean token representations projected onto the top two principal components. As the number of in-context ex- emplars increases, there is formation of representations mirroring the grid structure underlying the data-generating process. Representations are from the residual stream activation following layer 26.
an in-context learning task that involves simple "graph tracing" problem wherein the model is shown edges corresponding to random traversal of a graph (see Fig. 1). The nodes of this graph are intentionally referenced via concepts the model extremely likely to have seen during training (e.g.. apple, b rd, etc.), while its connectivity structure is defined using predefined geometry that is ambivalent to correlations between concepts' semantics (c.g.. a square grid). Based on the provided context, the model is expected to output valid next node prediction, i.c., a node connected to the last presented one. As we show, increasing the amount of context leads to a sudden re-organization of representations in accordance with the graph's 's connectivity. This suggests LLMs can manipulate their representations in order to reflect concept semantics specified entirely in-context, inline with theories of inferential semantics from cognitive science (Harman, 1982; Block, 1998). We further characterize these results by analyzing the problem of Dirichlet energy minimization, showing that models indeed identify the structure of the underlying graph to achieve non-trivial accuracy on our task. This suggests an implicit optimization process, as hypothesized by theoretical work on ICL in toy setups (e.g., in-context linear regression), can transpire in more naturalistic settings (Von Oswald et al., 2023a;b; Akyürek et al., 2023). Overall, our contributions can be summarized as follows.
Graph Navigation as a Simplistic Model of Novel Semanties. We introduce a toy graph nav- igation task that requires model to interpret semantically meaningful concepts as referents for nodes in a structurally constrained graph. Inputting traces of random walks on this graph into an LLM, we analyze whether the model alters its intermediate representations for referent concepts to predict valid next nodes as defined by the underlying graph connectivity, hence inferring, inline with theories of semantics from cognitive science, novel semantics of concept (Harman, 1982).
Emergent In-Context Reorganization of Concept Representations. Our results show that as context-size is scaled, i.e., as we add more exemplars context, there is sudden re-organization of concept representations that reflects the graph's connectivity structure. Intriguingly. these re- sults are similar to ones achieved in a similar setup with human subjects (Garvert et al., 2017: Whittington et al., 2020). Further, we show the context-specified graph structure emerges even when we use concepts that have correlations in their semantics (c.g.. Mon, Tues, etc.), but, in- terestingly, is unable to dominate the pretrained structure. More broadly, we note that this sudden reorganization is reminiscent of emergent capabilities in LLMS when other relevant axes, e.g., compute or model size, are scaled (Wei et al., 2022; Srivastava et al., 2022: Lubana al., 2024).
An Energy Minimization Model of Semantics Inference. To provide a more quantitative ac- count of our results, we compute the Dirichlet energy of model representations with respect to the ground-truth graph structure, and find the energy decreases a function of context size. This offers a precise hypothesis for the mechanism employed by an LLM to re-organize repre- sentations according the context-specified semanties of a concept. These results also serve as
2

---

Published as a conference paper at ICLR 2025
(a) Words on ring
(c) Emergent ring representation context
banana
epple
Layer
Layer
Layer 16
Layer
tomato
lettuce
Context length:
anion
grape
100
orange
fig
pear carrot
Contaxt length: 400
(b) Data generation
Randomly pick pairs of neighbors: (apple, bananal, (orange, onion),
(fig. carrot), (grape, lettuce),
Figure 2: Alteration of representations in accordance with context-specifted semantics (ring structure). (a) We randomly place concepts on a ring structure unrelated their semantics. (b) We then generate sequences of tokens by randomly sampling neighboring pairs from the ring which is used as the input context to Llama-3. 1-8B model. (c) The model's mean representation of tokens projected onto the top two principal components. As the number of in-context exemplars increases, there is a formation of representations mirroring the ring structure underlying the gettagenerating process. The representations are from the residual stream activations.
evidence towards theories of in-context learning as implicit optimization a more naturalistic setting (Von Oswald et al., 2023a;b; Akyürek et 2023).
2 EXPERIMENTAL SETUP: IN-CONTEXT GRAPH TRACING
We first define our setup for assessing the impact of context specification on how a model organizes its representations. In the main paper, we primarily focus on Llama3. 1-8B (henceforth Llama3) (Dubey et al., 2024), accessed via NDIF//NNight Protto-Kaufman et al., 2024). We present results on other models -lamallm.3.222 2-BB / Llama3. 1-8B- Instrutt (Dubey et al., 2024) and Gemma-2-2B / Gemma-2-9B (Gemma Team, 2024) in App. C.2.
Task. Our proposed task, which we call in-context graph trucing, involves random walks on predefined graph g. Specifically, inspired by prior work analyzing structured representations learned by sequence models, we experiment with three graphical structures: a square grid (Fig. 1 (a)), ring (Fig. 2 (a)), and a hexagonal grid (Fig. 10). Results on hexagonal grid are deferred to appendix due to space constraints. To construct the square grid, randomly arrange the set of tokens in grid and add edges between horizontal and vertical neighbors. We then perform a random walk on the graph, emitting the visited tokens neighboring nodes and simply sample random pairs of neighboring tokens on the graph (Fig. 2 (b)). Nodes in our graphs, denoted T = (70, TI, ; }.). are referenced via concepts that the model is extremely likely to have seen during pretraining. While any choice of concepts is plausible, we select random tokens that, unless mentioned otherwise, have no obvious semantic correlations with one another (c.g., apple, sand, math, etc.). However, these concepts have precise meanings associated with them the training data, necessitating that to the extent the model relies on the provided context, the representations are morphed according to the in-context graph. We highlight that a visual analog of our task, wherein one uses images instead of text tokens to represent a concept, has been used to elicit very similar results with human subjects the ones report in this paper using LLMs (Garvert et al., 2017; Whittington et al., 2020; Mark et al., 2020; 2024; Brady et al., 2009). We also note that our proposed task is similar to ones studied in literature wherein one provides exploration trajectories in-context to a model and expects to understand the environment and its dynamics (a. world model) (Lee et al., 2024b; Laskin et al., 2022).
sequence (Fig. (b)). For the ring,
add edges between
in-context RL,
3

---

Published as a conference paper at ICLR 2025
3 RESULTS
3.1 VISUALIZING INTERNAL ACTIVATION USING PRINCIPAL COMPONENTS
Since we are interested in uncovering context-specific representations, we input sequences from our data-generating process to the model and first compute the mean activations for each unique token T E T. Namely, assume a given context C == [Co. ..... CN-1]. where G E T. that originates from an underlying graph G. At each timestep. we look all tokens if the context length is smaller than N.r), and collect all activations corresponding to each token T at a given layer f. We then compute the mean activations per token, denoted as hę, e Rd. We further denote the stack of mean token representations as H (T) E R= Finally, we run PCA on (T), and use the first two principal components to visualize model activations (unless stated otherwise). We note that while PCA visualizations are known to suffer from pitfalls as a representation analysis method, we provide a thorough quantitative analysis in Sec. 4 to demonstrate that the model re-organizes concept representations according to the in in-context graph structure, and prove in Sec. 5 that the structure of the graph is reflected in the PCA visualizations because this re- re-organization of representations. We also provide further evidence on the faithfulness of PCA by conducting a preliminary causal analysis of the principal components, finding that intervening on concept representations' projections along these components affects the model's s ability to accurately predict valid next node generations (App. C.4).
window of Nw (=50) preceding tokens (or
Results. Figs. 1, demonstrate the resulting visualizations for square grid and ring graphs, spectively (more examples are provided in the Appendix; see Fig. 9, 10). Strikingly, with enough exemplars, we find representations are in fact organized in accordance with the graph structure un- derlying the context. Interestingly, results can be skewed in the earlier layers towards semantic priors the model may have internalized during training: however, these priors are overridden as we go deeper in the model. For example, in the ring graph (see Fig. 2). concepts apple and orange are closer to each other in Layer 6 of the model, but become essentially antipodal around layer 26, as dictated by the graph; the antipodal nature is also more prominent as context length is increased.
We also observe that despite developing square-grid structure when sufficient context length is given (see Fig. 1), the structure is partially irregular; e.g., it is wider in the central regions, but narrowly arranged in the periphery. We find this to be an artifact of frequency with which a concept is seen in the context. Specifically. due to lack of periodic boundary conditions, concepts that are present in the inner 2x2 region of the grid are visited more frequently during a random walk on the graph, while the periphery of the graph has lower visitation frequency. The representations reflect this, thus organizing in accordance with both structure and frequency of concepts in the context.
Overall, the results above indicate that scale context size, models can re- re-organize semantically unrelated concepts to form task-specific representations, which we call in-context representations. Intriguingly, these results are broadly inline with theories of inferential semantics from cognitive science as well (Harman, 1982; Block, 1998).
3.2 SEMANTIC PRIOR vs. IN-CONTEXT TASK REPRESENTATION
Building on results from the previous section, we now investigate the impact of using semantically correlated concepts. Specifically, we build on the results from Engels et al. (2024), who show that representations for days of the week, i.e., [Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday], organize in a circular geometry. We randomly permute the ordering of these concepts, arrange them on 7-node ring graph similar to the previous section (see Fig. 3a), and evaluate whether the in-context representations can override the strong pretraining prior internalized by the model.
Fig. 3 (b, c) demonstrate the resulting visualizations. We find that when there is conflict between the semantic prior and in-context task, we observe the original semantic ring in the first two principal components. However, the components right after in fact encode the context-specific structure: visualizing the third and fourth principal components shows the newly defined ring struc- ture. This indicates that the speeifed structure is present in the representations, but does not dominate them. In Fig. 14, we report the model's accuracy on the in-context task, finding that the model overrides the semantic prior to perform well our task when enough context is given.
Results.
4

---

Published as a conference paper at ICLR 2025
(a)
(b)
(c)
Tuesday
Monday
Principal Component 4
Principal Companent
Wednesday
Sunday
Thursday
Saturday
Principal Component
Principal Component
Friday
Semantic Link
In-Context Link
Figure 3: In-context representations form in higher principal components in the presence of semantic priors. (a) (Purple) Semantic links underlying days of the week. (Dashed blue) We define a non-semantic graph structure by linking non non-neighboring days and generate tokens from this graph. (b) (Purple) The ring geometry formed by semantic links established during pre-training remains intact in the first two principal components. (c) (Dashed blue) The non-semantic structure provided in-context can be seen in the third and fourth principal components. Note that the star structure in the first two components (b), which match the ground truth graphical structure of our data generating process (a), becomes a ring in the next two principal components (c). The representations from the residual stream activation following layer 21.
4 EFFECTS OF CONTEXT SCALING: EMERGENT RE-ORGANIZATION OF REPRESENTATIONS
Our results in the previous section demonstrate models re-organize concept representations in accordance with the context-specified semantics. We next aim to study how this behavior arises context is scaled is there continuous, monotonic improvement towards the context-specified structure as context is added? If so, is there a trivial solution, e.g., regurgitation based on context that helps explain these results? To analyze these questions, must first define metric that helps us gauge how aligned the representations are with the structure of the graph that underlies the context.
Dirichlet Energy. We measure the Dirichlet energy of our graph G's structure by defining an energy function over the model representations. Specifically, for an undirected graph & with n nodes, let A Rx a be its adjacency matrix, and R" be a signal vector that assigns value r; to each node Then the Dirichlet energy of the graph with respect to is defined as
Fx((x) = Σ Ai 22222
For a multi colli-ddmeessoonll signal, the Dirichlet energy defined as the summation of the energy each dimension. Specifically, let X Rand be matrix that assigns each node with d-dimensional vector æ;, then the Dirichlet energy of X is defined by
Ex(X)
Σ 1J
1 2
Σ A, 12
Overall, to empirically quantify the formation of geometric representations, can measure the Dirichlet energy with respect to the graphs underlying our data generating processes (DGPs) and mean token activations h.:
Eo(H Σ A., h911
where (T) E goxd is the stack our mean token representations he at layer and i, E T tokens from DGP at certain context length. We note H'( is function of context
5

---

Published as a conference paper at ICLR 2025
Layer
Layer
Accuracy
Nomallzed Dirichier Energy
Grid
Normalized Dirichlet Energy
Ring
Normalized Dirichlet Energy
Hex
1..0
-1.0
1.0
0.05
0.025
0.04
0.6
0.00
-0.6
Accuracy
0.020
0.6
0.03
0.06
0.015
0.4
0.04
0.2
-0.2
0.2
0.01
0.0
0.02
0.0
0.005
0.0
10
10a
10a
10
10f
10"
10-
10a
20
10
10
10*
Context Length
Context Length
Context Length
Figure 4: A model continuously develops task representation as it learns to traverse novel graphs in-context. We plot the accuracy of graph traversal and the Dirichlet energy of the graph, computed from the model's internal representations, the Dirichlet energy never reaches a perfect zero-ruling out that the representations are learning a degenerate structure, as was also seen in the PCA visualizations in Sec. (a) A 4x4 grid graph with 16 nodes. (b) A circular ring with 10 nodes. (c) A "honey-comb" hexagonal lattice, with 30 nodes.
functions of context length. We note that
length as well, but omit it in the notation for brevity. Intuitively, the measure above indicates whether neighboring tokens (nodes) in the ground truth graph have a small distance between their representations. Thus, as the model correctly infers the correct underlying structure, we expect to see a decrease in Dirichlet energy. We do note that, in practice, Dirichlet energy minimization has a trivial solution where all nodes assigned the same representation. While trivial solution does not exist in our results, for else we would not see distinct node representations in PCA visualizations nor high accuracy for solving our tasks, we still provide alternative analysis in App. C.3 where the representations are standardized meen-centeree and normalized by variance) to render this trivial solution infeasible. We find results are qualitatively similar with such standardized representations, but more noisy since standardization can induce sensitivity to noise.
be confident this
4.1 RESULTS: EMERGENT ORGANIZATION AND TASK ACCURACY IMPROVEMENTS
We plot Llama3's accuracy at the in-context graph tracing task alongside the Dirichlet energy mea- sure (for different layers) as function of context. Specifically, we compute the "rule following accuracy", where we add up the model's output probability over all graph nodes which are valid neighbors. For instance, the graph structure is app apple-coe aelle-carbb rd-wa ter and the current state is car, we add up the predicted probabilities for apple and bi rd. This metric simply measures how well the model abides by the graph structure.
Results are reported in Fig. 4. We see once a critical amount of context is seen by the model, accuracy starts to rapidly improve. We find this point fact closely matches when Dirichlet Energy reaches its minimum value: energy is minimized shortly before the rapid increase in in-context task accuracy, suggesting that the structure of the data is correctly learned before the model can make valid predictions. This leads us to the claim that as the amount of context is scaled, there is an emergent re-organization of representations that allows the model to perform well on our in-context graph tracing task. We note these results also provide more quantitative counterpart of our PCA visualization results before.
Is there a Trivial Solution at play? A simple baseline that would exhibit an increase in per- formance with increasing context involves the model merely regurgitating a node's neighbors by copying them from its context. We call this the memorization solution. While such a solution would not explain the reorganization of representations, engaging in a more intriguing mechanism. Since our accuracy metric measures rule following, this memorization solution will achieve value 1 if the node has been observed in the context and 0 other- wise. Following our data sampling process then, if we simply choose an initial node at random with replacement, we can express the probability of a node existing context of length I as:
use it as baseline to show the model is likely
11
Preen1 (x)
(4)
n

---

Published as a conference paper at ICLR 2025
(a)
50 Nodes
(b)
25 Nodes
1.04
100
0.8
0.8
Accuracy
Accuracy
0.4
0.4
0.0
0.0
10
10?
10
10a
Context Size
Context Size
L.1
1 1-shot Memorization
2-shot Memorization
Fiecewise Linear Fit
Transition Point
Figure 5: A memorization solution cannot explain Llama's ICL graph tracing performance. We plot the rule-following accuracy from Llama-3. outputs and accuracies from simple 1-shot and 2-shot memorization hypothesis. (a) A ring graph with 50 nodes. (b) A square grid graph with 25 nodes. In both cases, find that the memorization solution cannot explain the accuracy ascent curve. Instead, we find slow phase and a fast phase, which we with a piecewise linear
where the context and n is the number of nodes available. Note that the current node itself does not matter the sampling probability is uniform with replacement. We also evaluate another, similar baseline that assumes the same token much be encountered twice for the model to recognize it as an in-context exemplar. To define closed-form expression for this solution, we have the probability that a node has appeared twice as follows:
Pseen2 Preenl (x) - (1)
11 (1-1)
(5)
To evaluate whether the memorization solutions above explain results, plot their performance alongside the observed performance of Llama-3, Fig. 5 shows the result (a) on a ring graph with 50 nodes and (b) grid graph with 25 nodes. We find, in both cases, that neither the 1-shot the 2-shot memorization curve can explain the behavior of Llama. Instead, we observe that the accuracy has two phases, first phase where the accuracy improves very slowly, and second phase where the log-linear slope suddenly changes to steeper ascent. We find that piecewise linear fit can extract this transition point fairly well, which will be interest in the next section.
5 EXPLAINING EMERGENT RE-ORGANIZATION OF REPRESENTATIONS: THE ENERGY MINIMIZATION HYPOTHESIS
Building on the results from previous section, we now put forward a hypothesis for why we are able identify such structured representations from model: we hypothesize the model internally runs an energy minimization process in search of the correct structural representation of the data (Yang al., 2022), similar to claims implicit optimization in-context learning proposed by prior work in toy settings (Von Oswald et al., 2023a;b), More formally, we claim the following hypothesis.
Hypothesis 5.1. Let be the number of tokens, d be the dimensionality of the representations, and H (T) E In' xd the stack of representations for each token learned by the model at layer and context length t, then Eg (H(L))(()) decays with context length t.
5.1 MINIMIZERS DIRICILET ENERGY SPECTRAL EMBEDDINGS.
We call the k-th energy minimizer of Eg the optimal solution that minimizes Eg and is orthogonal to the first k energy minimizers. Formally, the energy minimizers {2(k) j== are defined as the
7

---

Published as a conference paper at ICLR 2025
-02
04
Figure 6: Spectral embedding of a ring graph.
Figure 7: Spectral embedding of grid graph.
solution to the following problem:
z arg min Ez(z) =E8--1
(6)
s.t. 2 vv
where 5*-1 is the unit sphere in dimensional Fuclidean space. The energy minimizers are known to have the following properties (Spielman, 2019):
1. (1)) cl for some constant e + 0, which is degenerated solution that assigns the same value to every node; and
2. If we use 2
(2)
as the coordinate of node i, it will be a good planar embedding. We
call them (2-dimensional) spectral embeddings.
Spectral embeddings are often used draw graph on plane and many cases can preserve the structure of the graph (Tutte, 1963). In Figs. 6 and 7, show the spectral embedding results for a ring graph and a grid graph respectively. Notice how such spectral embeddings are similar to the representations from our models in Fig. and 2. As show in Theorem I. this in fact expected if our energy minimization hypothesis is true: if the representations H from the model minimize the Dirichlet energy and are non-degenerated. then the first principal components of PCA will exactly produce the spectral embeddings (2)) 23) Here we present an informal version of the theorem, and defer the full version and proof to the appendix.
Theorem 5.1 (Informal Version of Theorem B.1). Let G be graph and H E Rand (where > Z 3) be matrix that minimizes Dirichlet energy on G with non-degenerated singular values, then the first two principal components of H will be (2) and (3))
See App. B for the formal version and proof of Theorem 5.1. See also Tab. 2 for empirical validation the theorem, wherein we show the principal components align very well with spectral embeddings of the graph.
5.2 ENERGY MINIMIZATION AND GRAPH CONNECTIVITY
Given the relationship between spectral embeddings (i.e., energy minimizers) and the principal com- ponents observed in our results (Figs. 2), we claim that the model's 's inference of the underlying structure akin to an implicit energy minimization. To further analyze the implication of this claim, we show that the moment at which we can visualize a graph using PCA is the moment at which the model has found a large connected component (i.e., the graph's 's structure). Specifically, consider an unconnected graph G. i.c., 9 has multiple connected components. Then, there are multiple de- generate solutions to the energy minimization problem, which will be found by PCA. Specifically, suppose 9 has & connected components, with U, denoting the set of nodes of the i-th component.
8

---

Published as a conference paper at ICLR 2025
(a)
Nodes
25
36
(b)
3.100-
Fir: y- 541xx4400
64
144
1.0-
Critical Context Size
0.8
10a-
Accuracy
0.6
6 x 10
0.44
4 x 10
0.21
3 10a
0.0
10
10a
102 of Nodes Grid
Context Size
Figure 8: In-context emergence. We analyze the in-context accuracy curves as function of context-size inputted to the model. The graph used in this experiment an m X m grid, with a varying value for (a) The rule following accuracy of graph tracing task. The accuracy show a two phase ascent. We fit a piecewise linear function to the observed ascent to extract the transi- tion point, which moves rightwards with increasing graph size. (b) Interestingly, the transition point scales as a power-law in m, i.e., the number of nodes in the graph.
Then construct the first q energy minimizers follows: Vi [g]. let the j-th value of 2 be
as jE U+UU Un k=1 otherwise,
(8)
where 01 = and o; = Eirei Eoeug (1-1)
Σin Ejeu, 20 for E [q] (1)-
It is easy to check that each z (0) constructed above for E [g] has O energy, and is thus global minimizer of Egg Moreover, all (i0)ss 's are orthogonal to each other and hence satisfy our definition of the first y energy minimizers. It is important to notice that these 2(1's for [g] contain information about the structure of the graph, other than identifying each connected component. Theorem B.I tells us that the principal components of non-degenerated (rank where 1) solution H that minimizes the energy will be ..=2)....+1)) Thus, if the graph is unconnected, then the energy-minimizing representations will be dominated by information-less principal components, in which we should not expect any meaningful visualization. The acute reader may recall that the first minimizer (11 is trivial solution of the energy minimization that assigns the same value to every node. Conveniently, the above argument also implies that this is not a concern: PCA will rule this degenerate solution as demonstrated in Theorem B.
In-context emergence: A hypothesis. Our results in Fig. 5 showed an intriguing breakpoint that is reminiscent of second-order phase transition (i.c., undefined second derivative). As shown in Fig. 8, we in fact find this behavior is extremely robust across graphs of different sizes, and shows a power-law scaling trend with increasing graph size (see App. C.7 for several more results in this vein, including different graph topologies). Given the relationship offered between energy minimization and discovery of connected component (graph structure) in analysis above, possible frame- work to explain these results may be the problem of bond-percolation on a graph (Newman, 2003; Hooyberghs al., 2010): in cond-percolations one starts with an unconnected graph and slowly fills edges to connect its nodes; as edges are filled, there is second-order transition after which a large connected component emerges in the graph. The nature of the transition observed in our experiments (Fig. 8) and the theoretical connection between energy minimization and existence of connected component provide some evidence towards the plausibility of this hypothesis. However, we believe the analogy is still loose, for our graph sizes are relatively small (likely causing signifi- cant finite-size effects) and the experiments need to corroborate any scaling theory of the transition point from percolation literature would require running graphs with least 2 orders-of-man of-gnnntudee difference their sizes. However, the consistency of the hypothesis with our empirical results and analysis implies that investigating it further may be fruitful.

---

Published as a conference paper at ICLR 2025
6 RELATED WORK
Model Representations. Researchers have recently discovered numerous structured representa- tions in neural networks. Mikolov et al. (2013) suggests that concepts are linearly represented in activations, and Park et al. (2024d) more recently suggests this may be the case for contemporary language models. Numerous researchers have found concrete examples of linear representations for human-level concepts, including "truthfulness" (Burns et al., 2022; Li et al., 2023b; Marks & Tegmark, 2024), "refusal" (Arditi et al., 2024), toxicity (Lee et al., 2024a), sycophancy (Rimsky et al., 2024), and even "world models" (Li et al., 2022: Nanda et al., 2023). Park et al. (2024c) finds that hierarchical concepts are represented with tree-like structure consisting of orthogonal vectors. A relevant line of work includes that of Todd et al. (2023) and Hendel et al. (2023). Both papers find that one can compute a vector from in-context exemplars that encode the task, such that adding such a vector during test time for new input can correctly solve the task. Language models do not always form linear representations, however. Engels et al. (2024) find circular feature representa- tions for periodic concepts, such days of the week or months of the year, using combination of sparse autoencoders and PCA. Csordás et al. (2024) finds that recurrent neural networks trained on token repetition can either learn "onion"-litt representation or linear representation, depending on the model's 's width. Unlike such prior work, we find that sask-spe representations with a de- sired structural pattern can be induced in-context. To our knowledge, our work offers the first such investigation of in-context representation learning.
Scaling In-Context Learning Numerous works have demonstrated that in-context accuracy im- proves with more exemplars (Brown et al., 2020; Lu et al., 2022; Bigelow et al., 2023). With longer context lengths becoming available, researchers have begun to study the effect of many-shot prompt- ing (as opposed to few-shot) (Agarwal et al., 2024; Anil et al., 2024; Li et al., 2023c). For instance, ICL using hundreds to thousands of exem-
Agarwal et al. (2024) reports improved performance plars on a wide range of tasks. Similarly, Anil et al. (2024) demonstrate the ability to jail-break LLMs by scaling the number of exemplars. Unlike such work that evaluates model behavior, we study the effect of scaling context on the underlying representations, and provide a framework for predicting when discontinuous changes in behavior can be expected via mere context- scaling.
Synthetic Data for Interpretability Recent works have demonstrated the value of interpretable, synthetic data generating processes for understanding Transformer's behavior, including in-context learning (Park et al., 2024a; Ramesh et al., 2023; Garg et al., 2023), language acquisition (Lubana et al., 2024; Qin et al., 2024; Allen-Zhu & Li, 2023b), fine- tuning (Jain et al., 2023; Lubana et al., 2023; Juneja et al., 2022), reasoning abilities (Prystawski et al., 2024; Khona et al., 2024; Wen et al., 2024; Liu et al., 2022a), and knowledge representations (Nishi et al., 2024; Allen-Zhu & Li, 2023a). While prior work typically pre-trains Transformers on synthetic data, we leverage synthetic data to study representation formation during in-context learning in pretrained large language models.
DISCUSSION
In this work, we show that LLMs can flexibly manipulate their representations from semanatics internalized based on pretraining data to semantics defined entirely in-context. To arrive at these results, we propose simple but rich task of graph tracing, wherein traces of random walks graph are shown to the model in-context. The graphs are instantiated using predefined structures (e.g., lattices) and concepts that semantically interesting (e.g., to define nodes), but meaning- less in the overall context of the problem. Interestingly, we find the ability to flexibly manipulate representations is in fact emergent with respect to context size-we propose model based on en- ergy mininization to hypothesize a mechanism for the underlying dynamics of this behavior. These results suggest context-scaling can unlock new capabilities. and, more broadly, this axis may have as of yet been underappreciated for improving model. In fact, we note that, to our knowledge, our work is to first to investigate the formation of representations entirely in-context. Our study also naturally motivates future work towards formation of world representations Li et al. (2023a) and world models (Ha & Schmidhuber, 2018) in-context, which can have significant implications toward building general and open-ended systems, well as forecasting its safety concerns. We also highlight the relation of our experimental setup to similar tasks studied in neuroscience litera- ture Garvert et al. (2017); Mark et al. (2020; 2024), wherein humans shown random walks of graph of visual concepts; fMRI images of these subjects demonstrate the formation of a structured representation of the graph in the hippocampal-entorhinl cortex, similar our results with LLMss.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We greatly thank the National Deep Inference Fabric (NDIF) pilot program Fraffo--aaffman et al., 2024), especially, Emma Bortz, Jaden Frotto-Kaufmann Adam Belfki, David Bau, and the team who provided us with access to representations of Llama models, making this work possible, CFP and HT acknowledge the support of Aravinthan D.T. Samuel, Cecilia Garraffo and Douglas Finkbeiner. CFP, ESL, KN, MO, and HT are supported by the CBS-NTT Program in Physics of Intelligence. AL and MW acknowledges support from the Superalignment Fast Grant from OpenAL. MW also ac- knowledges support from Effective Ventures Foundation, Effektiv Spenden Schweiz, and the Open Philanthropy Project. Part of the computations in this paper were run on the FASRC cluster sup- ported by the FAS Division of Science Research Computing Group Harvard University. ESL thanks Eric Bigelow for crucial discussions that helped define several hypotheses pursued in this work, and the Harvard CoCoDev lab (especially Peng Qian) and Talia Konkle for feedback earlier version the paper. CFP thanks Zechen Zhang, Eric Todd, Clement Dumas, and Shivam Raval for useful discussions. All authors thank David Bau and his lab for useful feedback on the paper's results.
AUTHOR CONTRIBUTIONS
CFP, AL,, ESL, and HT conceived the in-context graph traversal task, inspired by discussions and ex- perimentations with MO and KN in related work pretraining. CFP and AL co-led experiments, where CFP discovered the in-context learning of the ring-structured representation with input from HT, kicking off this study. AL suggested extending this to grid structures to CFP connecting it to world representations, and ESL proposed hexagonal configurations. ESL hypothesized in-context transitions with increased context and percolation mechanism. CFP conducted semantic overriding and percolation experiments, while AL performed accuracy-energy experiments and causal inter- ventions, with CFP, AL, and ESL developing transition point detection methods and contributing to energy normalization. YY formulated the energy minimization theory and its proofs, collaborat- ing with ESL to connect the framework to component existence and demonstrate PCA's optimality. MW provided alternative hypotheses for experimental robustness. CFP. AL, ESL, YY, and HT co- developed an initial manuscript with feedback from MW, with ESL leading the final writing and project narrative development. CFP, AL, and HT created figures, while CFP and AL jointly devel- oped the appendix. AL conducted substantial experiments to verify the main claims generalize to other models with CFP's support. HT supervised the project.
11

---

Published as a conference paper at ICLR 2025
REFERENCES
Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. Can language models encode perceptual structure without grounding? case study in color. arXiv preprint arXiv:2109.0611 2021.
Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Be- hbahann, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning, 2024. URL
https://azas httpaaxxvvvoggggaaaa// abs 2404 11018.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models, 2023. URL https://
dexiiv.rrg/ae 2211. .1661.
Zeyuan Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint arXiv:2309.1433 2023a.
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 1, learning hierarchical lan- guage structures. ArXiv e-prints, abs/2305..36773 May, 2023b.
Cem Anil, Esin Durmus, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Nina Rimsky, Meg Tong, Jesse Mu, Daniel Ford, et al. Many-shot jailbreaking. Anthropic, April, 2024.
Anthropic AI.
Scaling Monosemanticiny Extracting Interpretable Features from Claude 3 Sonnet, 2024.
scallin mon.amnnnooooonnnniiiiinnommane
html.
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by single direction. arXiv preprint arXiv:2406. 1171, 2024.
Eric J Bigelow, Ekdeep Singh Lubana, Robert P Dick, Hidenori Tanaka, and Tomer D Ullman. In-context learning dynamics with random binary sequences. arXiv preprint arXiv:2310.1760 2023.
Ned Block. Semantics, conceptual role. Routledge encyclopedia of philosophy, 8:652-657, 1998.
Timothy F Brady, Talia Konkle, and George A Alvarez. Compression in visual working memory: using statistical regularities to form more efficient memory representations. Journal of Experi- mental Psychology: General, 138(4):487, 2009.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam. Girish Sastry. Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https::/ppoceedd/ee www.iiigg...uuussseeeeeeeeee. www.sis.c..eea.ae.eeeo...e cc/paper........... paf.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in lan- guage models without supervision. arXiv preprint arXiv:2212.03827, 2022.
Róbert Csordás, Christopher Potts, Christopher D Manning, and Atticus Geiger. Recurrent neural networks leam to store and generate sequences using non-linear representations. arXiv preprint arXiv:2408. 10920, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and et al. The llama herd of models, 2024.
URL hwtph:tp///eeeeee h/tp:///xvvvovoo/// abs///////0// /207.. 21783.
12

---

Published as a conference paper at ICLR 2025
Joshua Engels, Isaac Liao, Erie J. Michaud, Wes Gurnee, and Max Tegmark. Not all language model features are linear, 2024. URL https: https///axxvvviiiieee halgg//ba/// 14860.
Ky Fan. On a theorem of weyl conceming eigenvalues of linear transformations Proceedings of the National Academy of Sciences, 35(1 (5(11)65226655 655, 1949.
Jaden Fiotto- Kaufman Alexander R Loftus, Eric Todd, Jannik Brinkmann, Caden Juang, Koyena Pal. Can Rager, Aaron Mueller, Samuel Marks, Arnab Sen Sharma, Francesca Lucchetti, Michael Ripa, Adam Belfki, Nikhil Prakash, Sumeet Multani, Carla Brodley, Arjum Guha, Jonathan Bell, Byron Wallace, and David Bau. Nnsight and ndif: Democratizing access to foundation model internals, 2024. URL https: /ttp//aavvvvv///////v///e.//..al org/ abs 2407/ 14561.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes, 2023. URL
2208. 01066.
Mona M Garvert, Raymond J Dolan, and Timothy EJ Behrens. A map of abstract relational knowl- edge in the human hippocampal-entorhina cortex. elife, 6:e17086, 2017.
Gemma Team. Gemma 2: Improving open language models a practical size, 2024. URL https: //arxiv. abs/ 00118.
Pulkit Gopalani, Ekdeep Singh Lubana, and Wei Hu, Abrupt learning in transformers: A case study matrix completion. arXiv preprint 2rXiv:2410.22244, 2024.
Wes Gurnee and Max Tegmark. Language models represent space and time. arXiv preprint
arXiv:2310.02227. 2023.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018.
Gilbert Harman. Conceptual role semantics. Notre Dame Journal of Formal Logic, 23(2):242256,, 1982.
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In
Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Compu- tational Linguistics: EMNLP 2023, pp. 9318-9333, Singapore, December 2023. Association
for Computational Linguistics. doi: 10.18653/w1/2033. http:///////222...nnnnnnnooo findings-emmll.622 URL https: //
findin.fiinniiggg emw.p..emmpp......mm 624.
H. Hooyberghs, B. Van Schaeybroeck, and J. O. Indekeu. Percolation on bipartite scale-free net- works. Physica A: Statistical Mechanics and its Applications, 389(15) 29229999,,, August 2010. ISSN 0378-4371.
Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert Dick, Hidenori Tanaka, Edward Grefen- stette. Tim Rocktäschel, and David Scott Krueger. Mechanistically analyzing the effects of fine- tuning on procedurally defined tasks. arXiv preprint arXiv:231 1.12786, 2023.
Erik Jenner, Shreyas Kapur, Vasil Georgiev, Cameron Allen, Scott Emmons, and Stuart Russell, Evi- dence of learned look-ahead in chess-playin neural network. arXiv preprint 24Xxx:24066008777 2024.
Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. Linear connectivity reveals generalization strategies. arXiv preprint. arXiv:2205. /2401 2022.
Mikail Khona, Maya Okawa, Jan Hula, Rahul Ramesh, Kento Nishi, Robert Dick, Ekdeep Singh Lubana, and Hidenori Tanaka. Towards an understanding of stepwise inference in transformers: A synthetic graph navigation model. arXiv preprint arXiv:2402.07757, 2024.
Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, Strouse, Steven Hansen, Angelos Filos, Ethan Brooks, et al. In-context reinforcement learning with algorithm distillation. arXiv preprint arXiv:22100 10. 14215, 2022.
13

---

Published as a conference paper at ICLR 2025
Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mihalcea. mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. In Forty-first International Conference Machine Learning, 2024a. URL https: olrrxiv.org/ooo abs 2401 01967.
Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining can learn in-context reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024h.
Belinda Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. arXiv preprint arXiv:2106.00737, 2021.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Watten- berg. Emergent world representations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference on Learning Representations, 2022.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas. Hanspeter Pfister, and Martin Wat- tenberg. Emergent world representations: Exploring sequence model trained on a synthetic task. In The Eleventh International Conference on Learning Representations, 2023a. URL https:
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. In Thiitty-eventth Conference on Neural Information Processing Systems, 2023b. URL https: http//pppnnneeeeeeeeeeeeeeeeeeep nww.///weeeeeeeeeeeeeeeeeeeeeeee net/forum? forum?
Ad=alLuYpn83y.
Mukai Li, Shansan Gong, Jiangtao Feng, Yiheng Xu, Jun Zhang, Zhiyong Wu, and Lingpeng Kong. In-context learning with many demonstration examples. arXiv preprint arXiv:2302. 00300.0004112222 2023c.
Bingbin Liu, Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shortcuts to automata. arXiv preprint arXiv:2210. 10749, 2022a.
Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. To- wards understanding grokking: An effective theory of representation learning. Advances in Neu- ral Information Processing Systems, 35:34651-34663, 2022b.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meer- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8086-8098, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022. acl-long.556. URL ht tttpss: / http///aaaaaahloooooeoooo
022/2022 acl-lona 556.
Ekdeep Singh Lubana, Eric J Bigelow, Robert P Dick, David Krueger, and Hidenori Tanaka. Mech- anistic mode connectivity. In International Conference on Machine Learning, pp. PMLR, 2023.
Ekdeep Singh Lubana, Kyogo Kawaguchi, Robert P Dick, and Hidenori Tanaka. A percolation model of emergence: Analyzing transformers trained a formal language. arXiv preprint arXiv:2408. 12578, 2024.
Shirley Mark, Rani Moran, Thomas Parr, Steve W Kennerley, and Timothy EJ Behrens. Transferring structural knowledge across cognitive maps in humans and models. Nature communications, (1):4783, 2020.
Shirley Mark, Phillipp Schwartenbeck, Avital Hahamy, Veronika Samborska, Alon B Baram, and Timothy E Behrens. Flexible neural representations of abstract structural knowledge in the human entorhinal cortex. Elife, 13, 2024.
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets, 2024. URL https: : arxiv. absg//ab///// 06824.
14

---

Published as a conference paper at ICLR 2025
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- wwtp:iii.wrggaaa..oooe wwt.oo/g//....oooooo. 223001 11301.
tations in vector space, 2013. URL https: /ttp//aavvv/////////v//////....a
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world
models of self-supervised sequence models. In Proceedings of the 6th BlackboxNLP Work-
shop: Analyzing and Interpreting Neural Networks for NLP, pp. 16-30, 2023. URL https: //arxiv. org/ abs 23999009919..
M. E. J. Newman. The Structure and Function of Complex Networks. SIAM Review, 45(2):167-256, January 2003. ISSN 0036- 1445, 1095-7200.
Kento Nishi, Maya Okawa, Rahul Ramesh, Mikail Khona, Ekdeep Singh Lubana, and Hidenori Tanaka. Representation shattering in transformers: A synthetic study with knowledge editing. arXiv preprint arXiv:2410.17194, 2024.
Core Francisco Park, Ekdeep Singh Lubana, Itamar Pres, and Hidenori Tanaka, Competition dynam- ics shape algorithmic phases of in-context learning. arXiv preprint arXiv:2412.01003. 2024a.
Core Francisco Park, Maya Okawa, Andrew Lee, Ekdeep Singh Lubana, and Hidenori Tanaka. Emergence of hidden capabilities: Exploring learning dynamics in concept space, 2024b. URL /ttts://arxive org/ alos 2406.120100 19370.
Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierar- chical concepts large language models. arXiv preprint arXiv:2406.01506, 2024c.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models, 2024d. URL https:: /ttp/aaavvvvvvvv////v///...a.a.a otgg/bbs/333 03658.
Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In Inter- national conference learning representations, 2022.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pp. 1532-1555 1543, 2014.
Ben Prystawski. Michael Li, and Noah Goodman. Why think step by step? reasoning emerges from the locality of experience. Advances in Neural Information Processing Systems, 36, 2024.
Tian Qin, Naomi Saphra, and David Alvarez-Melis Sometimes i am a tree: Data drives unstable hierarchical generalization. arXiv preprint arXiv:2412.04619, 2024.
Rahul Ramesh, Ekdeep Singh Lubana, Mikail Khona, Robert P Dick, and Hidenori Tanaka. Com- positional capabilities of autoregressive transformers: A study on synthetic, interpretable tasks. arXiv preprini arXiv:2311.12991 2023.
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering llama via contrastive activation addition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 15504-15522, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.86663///20242alllooo.g https:///cclattiin
URL
wrw./024.2220200 acl acl- 828.
Adam S Shai, Sarah Marzen, Lucas Teixeira, Alexander Gietelink Oldenziel, and Paul Riech- ers. Transformers represent belief state geometry in their residual stream. arXiv preprint arXiv:2405.15943. 2024.
Daniel Spielman. Spectral and algebraic graph theory. Yale lecture notes, draft of December, 4:47, 2019.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alons al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206. 04615, 2022.
15

---

Published as a conference paper at ICLR 2025
Eric Todd, Millicent L Li, Arma Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. arXiv preprint arXiv:2310.15213, 2023.
Aaron Traylor, Roman Feiman, and Ellie Pavlick. Can neural networks learn implicit logic from physical reasoning? In The eleventh international conference on learning representations, 2022.
William Thomas Tutte. How to draw a graph. Proceedings of the London Mathematical Society, (1):743-767, 1963.
Keyon Vafa, Justin Y Chen, Jon Kleinberg, Sendhil Mullainathan, and Ashesh Rambachan. Evalu- ating the world model implicit in generative model. arXiv preprint arXiv:2406.0366 03689, 2024.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordy- intsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, 35151-35174 PMLR, 2023a.
Johannes Von Oswald, Maximilian Schlegel, Alexander Meulemans, Seijin Kobayashi, Eyvind Niklasson, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Max Vladymyrov, et al. Uncovering mesa-optimization algorithms in transformers. arXiv preprint urXiv:2309.05858, 2023b.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprint arXiv:2266.07682. 2022.
Kaiyue Wen, Yuchen Li, Bingbin Liu, and Andrej Risteski. Transformers are uninterpretable with myopic methods: case study with bounded dyck grammars. Advances in Neural Information Processing Systems, 36, 2024.
James CR Whittington, Timothy Muller, Shirley Mark, Guifen Chen, Caswell Barry, Neil Burgess, and Timothy Behrens. The tocman-einnnnbaum machine: unifying space and relational mem- ory through generalization in the hippocampal formation. Cell, 183(5):1249-128) 2020.
Yongyi Yang. David P Wipf, et al. Transformers from optimization perspective. Advances in Neural Information Processing Systems, 35:36958-36971, 2022.
16

---

Published as a conference paper at ICLR 2025
A ADDITIONAL EXPERIMENTAL DETAILS
Here we provide some additional details regarding our experimental setups.
Context Windows. Our analyses require computing mean token representations h, for every token i E in our graphs. To do so, we grab activations per each token the most recent context window of No tokens. Because we further require that each token is observed at least once in our window, we use a batch of prompts, where the batch size is equal to the number nodes in our graph. For each prompt in the batch, we start with a different node, ensuring that each node shows up at least once in the context. In the case when context length (N.) is longer than the window, we simply every token (N No).
random traversal (or random pairwise sampling)
Computational Resources. We run our experiments on either A100 nodes, or by using the APIs provided by NDIF KFuftto-Kaufman et al., 2024).
B THE CONNECTION BETWEEN ENERGY MINIMIZATION AND PCA STUCTURE
In this section, for a matrix M Rudd we use lowercase bold letters with subscript to represent the columns for M, c.g. my represents the k-th column of M. Moreover, we use a (M) to represent the k-th largest singular value of M and when M PSD we use (M) represent the k-th largest eigenvalue of M. Moreover, we use to represent a vector with all-zero entries except at entry k, whose dimension is inferred from context, and to represent vector with all entries being 1. For a natural number n, we use [r]] to represent 21, .n..
Furthermore, we use {x(()) 1 k=1 to represent the energy minimizers of the Dirichlet energy, defined in Section 4. Let A TRax be the adjacency matrix of the graph, D diag( (A1) be the degree matrix, and L = D A be the Laplacian matrix. Through an easy calculation one can know that for any vector æ _R"
Eg = (ar, Lax)
(9)
Therefore, from the Spectral Theorem (e.g. Theorem 2.2.1 in Spielman (2019)), we know that z is the eigenvector of corresponding to An- = (zx)..
We will show that, if matrix H Raxd minimizes the energy and is non-degenerated (has several distinct and non-zero singular values), then the PCA must exactly give the leading energy minimiz- ers, starting from 22-
Theorem B.1. Let G be graph and 61 C2 >..>>>>> 0 be s min(n, d) distinct positive numbers. Let matrix H E [Ruxd be the solution of the following optimization problem:
H arg
min Ex(X)
(10)
xd
s.t. o (x) 2 ck. Vk E [r].
(11)
then the k-th principle component of H (for k E (r]) is x (k+1).
Proof. We first prove that the leading left-singular vectors of H are exactly energy minimizers. Let min(n, n, d). Let the SVD of H be H UEVT, where Σ diag [01, ".. Gd) are the singular values of H. and U Raxr, V Rxxd
17

---

Published as a conference paper at ICLR 2025
Let h represents the i-th row of H. Notice that
Eg (H) = Σ
Ar,
hh -hlll²
(12)
1
Σ A (e; e,) H
(13)
(e: e,)² UE 2
Σ
Al
(14)
o (e, Cj. un!
(15)
Eg (uz)
(16)
Since og 's and u 's are independent, no matter what are the values ugs we know that each on will take the smallest possible value, and from the given condition, it is o ck. Ak E [s]. and ok = 0, vk E [r] [ [s].
Since u 's are singular vectors, we have un 's are orthogonal to each other. Using Theorem I in Fan (1949), we know that for any s' E [n). the minimizer Σx= Eg(us) is uy = 2 (b), Ak E [s]] Therefore, it is evident that the minimizer of Σi- of since from the above argument of σj. 's and the given condition condition know that σ 02 ... 0, > 0.
Eg(u,) must satisfies uy = 21) Ak E [s].
Now have proved that ug (k), vk E [s]. Next consider the output of PCA. Let ps be the k-th principle component output by the PCA of H. We know that is the eigenvector of
"HH
(17)
that corresponds to the k-th largest eigenvalue of C, where H H !11T is the centralized
H.
From the Spectral Theorem, have
Pk arg
max PCs
(p. Cp)
(18)
Let J = (1) be the set of vectors whose every entry has the same value. Let J- be the subspace in Rn that is orthogonal to J. For a subspace K Rf", let lel R + R² be the projection operator onto K.
We have that
Pi arg max (p.Cp)
(19)
DE8--1
(p.(1-111) HH (1 11-
arg max pEga-1
p
(20)
11
n
arg max pE8
(I, HH 1I (p))
(21)
= arg max
(p. HH p)
(22)
PLJ
which, again from Spectral Theorem, is the eigenvector of the second largest eigenvalue of HH
which is ug = =2)) Using an induction and the same reasoning, it follows that for any i [s). we have Ps= = (k+11, This proves the proposition.
18

---

Published as a conference paper at ICLR 2025
C ADDITIONAL RESULTS
C.I DETAILED LAYER-WISE VISUALIZATION OF REPRESENTATIONS
In Figure 9 and Figure 10 we provide additional visualizations per layer for each our models and each of our data generating processes.
Inyer?
1yper
3.25
Nyer
3.09
64-
02
0.78
6.2%
6.09
-0.9
25
-05
04
-36
-05
08
-38
-15
15
Lpger
Layer 10
Laper:12 12
Layer 14
3..
1.0-
85-
-0.4
-35
50
-10
45
-20
Figure 9: We plot 2d PCA projections from every other layer in Llama3.1-88 (Dubey et al., 2024), given the grid-traversal task. In deeper layers, see clear visualization of the grid.
19

---

Published as a conference paper at ICLR 2025
-02
42
85
-050005 de 22 0de als Lie 155
-55
1.0
-58
-55
12
Loyal
Lost 28
Lave:
Layna 14
08
12
15 33
39
35 10
1O
Layer
20-
Figure 10: We plot 2D PCA projections from every other layer in Llama3, 1.ama3.1-8B (Dubey et al., 2024)
for the hexagonal grid task.
20

---

Published as a conference paper at ICLR 2025
C.2 PCA, DIRICHLET ENERGY, AND ACCURACY RESULTS OTHER MODELS
Here we provide results from other language models, i.e., Llama3-10 (Dubey et al., 2024), Llama3- 8B-Instruct, Gemma2-2B (Gemma Team, 2024), and Gemma2-9BB. In Figure 11, plot the 2d PCA projections from the last layer of various models for various data generating processes. In Figure 12, we plot the normalized Dirichlet energy curves against accuracy for various language models on various tasks. Across all models and tasks, we see results similar to the main paper.
Llama 3.2 18: grid
Laam 3.1, 8B Instruct: grid
Gemma 28: grid
Gemma 98: grid
-55
-199
-5
15
55
Llama 3.2 18: ring
Llama 3.1. 8B Instruct: ring
Gemma 2 28: ring
Gemma 9B: ring
-15
Liama 3,2, 18: hex
Llama 3.1 88 Instruct: hex
Gemma 2, 28: hex
Gemma 2 2_98: hex
-19
-109
75
Figure 11: We plot 2d PCA projections from the last layer of various language models, given various data generating processes. For the grid and hexagonal graphs, we apply PCA on the last layers. For the rings, visualize layers 14, 10, 16, and 20 respectively. Interestingly, for Llama3. 2- IB, find the ring representation in the 2nd and 3rd principal components.
C.3 STANDARDIZED DIRICHLET ENERGY
In Fig. 13. we report Dirichlet energy values computed after standardization representations, ie., after mean-centering them and normalizing by the standard deviation. This renders the trivial solution Dirichlet energy minimization infeasible, since assigning constant representation to all nodes will yield infinite energy (due to zero variance). As can be seen in our results, the plots are qualitatively similar to the non-standardized energy results (Fig. 12), but more noisy, especially for the ring graphs. This is expected, since standardization can exacerbate the influence of noise, yielding fluctuations in the energy calculation.
21

---

Published as a conference paper at ICLR 2025
Layer
Layer Acoussy
Laper
Layar
Layar 20
Lano grdd
Lino 3:2 18: gi
Liama 2.1 86 rnstuct gri
Genno 2 28 grd
Gerima 2. gild
Scund
NOVIO
Drain Lingh
Darked Leaght
Crstcal Lengh
Liama, 31, nBinng
Liara 3.2 38: mg
Liama 1 Imbuutrrn
Gamma 2 28: mng
Gamma 20t
Crneen Lenge
Comen Large
Cantoc Leiph
Canter Large
Gomen Lengn
Unara 38 hex
Limma 3.2 18: hex
Usrra 3.l instrct hes
Gemman3 2_20: hex
Gemma hex
NUSO EE
Dr MesecLun Lingh
Car aod Lengh
Cortect
Glited Lenghh
Figure 12: Accuracy versus normalized Dirichlet energy curves for various language models on various tasks. For every model and task, we see energy minimized before accuracy starting to improve.
Layer 28
Laye'3 Layer
Lapsr
Loyar 26 Loyar
Laya 20 Laya
Lama 311 9/
Usma 3.2 18: gMd
Liyma 3.1 88 Indrrcct grd
Cemmo. 28 end
Gemum 2.98 grld
Caldect
Carted Leighh
Ciitedt
Liama
Lime 3.2
Llama 3-1 85 Irotuct:
Gemma ring
Gemiaa 28:ring
Creen
Cartedt tewch Intruct
Centent
Chenet Leneh
LEarTs, b0thee
3.2
Gemma
NOURE
Figure 13: Accuracy versus zero mean centered normalized Dirichlet energy curves for various language models various tasks. Zero mean centering ensures that graph representations using the trivial solution to energy minimization (i.e., assigning the same representation for every node).
22

---

Published as a conference paper at ICLR 2025
C.4 CAUSAL ANALYSIS OF REPRESENTATIONS
In this section we report preliminary causal analyses of our graph representations. While fully under- standing the mechanisms behind the formation of such representations, well as the relationship between said representations and model outputs are an interesting future direction, this is not the focus of our work and thus we only ran proof-ff-concep. experiments.
With that said, we ask: do the principal components that encode our graph representations have any causal role in the model's predictions?
To test this, we attempt to "move" the location of the activations for one node of the graph to another by simply re-scaling its principal components. Namely, assume activation h corresponding to node i at layer f. Say we wish to "move" the activation to different target node j. We first compute the mean representation of node j using all activations corresponding to node j within the most recent N' (= 200) timesteps, notated as hj. Assuming the first two principal components encode the "coordinates" of the node, we simply re-scale the principal components of h, to match that of hj.
We view this approach as rather rudimentary. Namely, there are likely more informative vectors that encode richer information, such information about neighboring nodes. However, we do find that the first two principal components have some causal role in the model's predictions.
We test our rc-scaling intervention on 1,000 randomly generated contexts. For each context, assum- ing our underlying graph has n nodes, we test "moving" the activations of the last token to all other locations in the graph. We then report the averaged metric across the resulting 1,000 testcases.
We report 3 metrics: accuracy (Hit@ 1), Hit@3, and "accumulated probability mass" on valid tokens. Hit@1 (and Hit@3) report the percentage of times at which the top (top 3) predicted token is a valid neighbor of the target node j. For "accumulated probability mass", we simply probability mass allocated to all neighbors (i.c., valid predictions) of the target node j.
up the
Table reports our results for our ring and grid tasks. We include results for re-scaling with principal components, as well as null interventions and interventions with a random vector. Overall, we find that the principal components have some causal effect on the model's output predictions, but does not provide full explanation.
Ring
Grid
Hex
Hit@l
Hir@3
Prob
Hit@1
Hit@3
Prob
Hit@1
Hit@3
Prob
Interv. (n=2)
0.61
0.91
0.6
0.57
0.95
0.55
0.30
0.32
0.69
Interv. (n=3)
0.77
0.96
0.76
0.68
0.98
0.65
0.42
0.46
0.82
Null Interv.
0.20
0.50
0,20
0.17
0.33
0.16
0.07
0.20
0.05
Random Interv.
0.17
0.47
0.19
0.16
0.37
0.17
0.06
0.18
0.05
Table 1: Intervention results for our ring and grid tasks. We demonstrate that often times, simply re-scaling the principal component for each token representation can "move" the token to a different position in the graph. However, we note that our simple re-scaling approach does not perfectly capture causal relationship between principal components and model predictions.
C.5 EMPIRICAL SIMILARITY OF PRINCIPAL COMPONENTS AND SPECTRAL EMBEDDINGS
Theorem 5.1 predicts that if the model representations are minimizing the Dirichlet energy, the first two principal components will be equivalent to the spectral embeddings (2),22(0)
Here empirically measure whether the first two principal components are indeed equivalent to the spectral embeddings. In Table 2, we measure the cosine similarity scores between the principal components and spectral embeddings.
C.6 ACCURACY OF IN-CONTEXT TASKS WITH A CONFLICTING SEMANTIC PRIOR
What would happen when an in-context task which contradicts a semantic prior is given to model? Namely, Engels et al. (2024) show that words like days of the week have a circular representation.
23

---

Published as a conference paper at ICLR 2025
cos 1, 22))
[Cos(PC2, 2 x())
Grid
0,950
0,954
Ring
0,942
0,930
Hex 0.745
0.755
Table 2: Absolute value of cosine distances of principal components from model activations and spectral embeddings. We empirically observe that in practice, these coordinates end up being very similar. For the grid and hexagon, use principal components from the last layer, while for the ring, we use earlier layer (layer 10) in which the ring is observed.
In our experiment, we randomly shuffle tokens for days of the week (i.e., tokens (Mon, Tue, Wed, Thu, Fri, Sat, Sun} to define new ring, and give random neighboring pairs from the newly defined ring as our in-context task.
Figure 14 demonstrates the accuracy when given an in-context task that is contradictory to semantic prior. Interestingly, we first observe the model make predictions that reflects the original semantic prior (pink). This accuracy drops very quickly as the model captures that the semantic rule is not being followed. With more exemplars, we see a slow decay of the remaining semantic accuracy and transition in the model's behavior as it begins to make predictions that reflect the newly defined ordering our ring (blue).
Shuffled
Semantic
Accuracy
0.4
0.2
0.0
200
400
Number of examples
In-context structure overrides semantic prior. Given an in-context task that contra- dicts a model's semantic prior, we observe the model transition from making predictions that adhere to the semantic prior (pink) to predictions that reflect the newly defined in-context task.
Figure 14:
Furthermore, Fig. 15, we quantify the Dirichlet energy computed only from certain PC dimen- sions. We find that energy minimization happens in the dimensions corresponding the in-context structure.
0.5
0.4
Energy
0.33
Total
0.2
PCA 1,2
PCA 3,4
10"
10a
Context Length
Figure 15: Energy minimization happens in the in-context component dimensions. We show the Dirichlet energy depending on the context given when taking 1) all 2) semantic (PCA 1,2) 3) in-context (PCA 3,4) dimensions. We show that energy minimization happens in PCA 3,4 corre- sponding to the in-context dimensions.
24

---

Published as a conference paper at ICLR 2025
C.7 ADDITIONAL EMPIRICAL VERIFICATIONS OF TRANSITION PREDICTIONS
Here we provide additional details for empirically verifying our predictions for model transitions.
Figures 16, 17, and 18 demonstrate detailed accuracy curves for wide range of graph sizes.
2onde-10
conde-4t
2o
words-90
Lana Juee
Pacretio ancere
3.2
commple
207
12'
Cormple
Carpee
200
6.4
0.0
2.2
0.2
10
Chimple modec
32"
Exemple trees
317
202
0.2
3.0
0.0
Farige KMM
Figure 16: Emergent behavior for varying task complexity (graph size) for the Hexagonal task. We plot the accuracy for varying levels of complexity (graph size) for the hexagonal in-context task. Interestingly, regardless of graph size, we see an abrupt, discontinuous change in the model's performance, Figure 19 demonstrates that we can predict when such abrupt change can be expected as function of task complexity.
Quan Sam-s
Crach Siem22
Quuph
Dush Seam
Usee Smeothed Tanston fornzd d9
Commale Aagierriggn
100
Doaple
Commoe
Draple
12
204
Ecange Ido
Emergent behavior for varying task complexity (graph size) for the grid task. We plot the accuracy for varying levels of complexity (graph size) for the grid in-context task. Interest- ingly, regardless of graph size, see an abrupt, discontinuous change the model's performance. Figure demonstrates that we can predict when such abrupt changes can be expected as a function of task complexity.
Figure 17:
25

---

Published as a conference paper at ICLR 2025
oragn Siew1o
Grash Stewss
Graan dem
Graph Si0e2000
Linn (he
ridec
Idder
Graph 220-1266
Grapo Ste 403
Siage Seaid30
Graph 2bn CCO
Craghi Scendos
Crugh Saue503
Coughh S000
15
Emergent behavior for varying task complexity (graph size) for the ring task. We plot the accuracy for varying levels of complexity (graph size) for the ring in-context task. Interestingly, regardless of graph size, we again see an abrupt, discontinuous change in the model's performance.
Figure 18:
(a)
# of Nodes
(b)
126
286
Liama-3.1-88
Fit: y 2.934x
1.0-
Critical Context Size
0.8
Accuracy
0.6
102-
0.4
0.0
10a
10a
10a
10
Context Size
# of Nodes in Grid
Figure 19: In-context emergence in Hexagonal graph tracing task. We analyze the in-context accuracy curves as function of context-size inputted to the model. The graph used in this exper- iment is an m X grid, with varying value for m. (a) The rule following accuracy of graph tracing task. The accuracy show two phase ascent. We fit piecewise linear function to the ob- served ascent extract the transition point, which moves rightwards with increasing graph size. (b) Interestingly, the transition point scales as a power-law in m, i.e., the number of nodes in the graph.
a)
48 Nodes
b)
70 Nodes
c)
Nodes
d)
286 Nodes
c.0-
Acuecc
0.6-
022
02
0.2
0.-
Carted
Contect Size
Conteot
Figure 20: Hexagonal graph tracing accuracies compared to the memorization solution The rule following accuracies on the hexagonal graph compared the memorization model in Sec. 4.1. Hexagonal graph with a) 48 b) 70 c) 126 d) 286 nodes. Generally we find that the hexagonal graph tracking accuracy from Llama-3.1-88 (Dubey et al., 2024) is lower than the 1, 1,2- shot memorization model, indicating that there might be different underlying process.
26

---

Published as a conference paper at ICLR 2025
D LIMITATIONS
We emphasize that our work has few limitations. Namely, PCA, or more broadly, low dimensional visualizations high dimensional data can be difficult interpret sometimes even misleading. Despite such difficulties, we provide theoretical connections between energy minimization and prin- cipal components to provide a compelling explanation for why structures elicited via PCA faithfully represent the in-context graph structure. Second, we find a strong, but nevertheless incomplete, causal relationship between the representations found by PCA and the model's predictions. We view the exact understanding of how these representations form, and the exact relationship between the representations and model predictions as an interesting future direction, especially given that such underlying mechanism seems to depend on the scale of the context.
27


Reviews:
[{"review_id": "efd4Q00iYT", "reviewer": "Reviewer_KJBF", "summary": "The paper investigates how LLMs form representations for novel concepts during in-context learning. Authors design a synthetic graph navigation task where the model has to learn from random walks between graph nodes are referenced as words from the training set (e.g. “apple”), meaning that the model has to learn a new representation for these words to accurately predict the next node in the graph traversal. They refer to this problem as “in-context graph tracing”.\n\nAuthors then study the hidden representations formed by LLMs as they learn this task in-context, visualizing their first few principial components. They find that, after a certain number of examples, the model abruptly re-organizes the representation space in a way that facilitates the graph traversal task, as learned from the context. They also find cases where the in-context learned representations do not dominate  the first principal components, but they are still present in subsequent components (e.g. 3rd and 4th in Figure 3). Finally, authors demonstrate a connection between the process of in-context learning of node representations and a form of graph energy minimization.", "questions": "### Questions:\n\nNote that these questions are mostly asked out of curiosity and do not affect my overall recommendation.\n\n1. Authors found that there is a shift in concept representations after the model has seen enough context. **Could this mean that the model does not 'understand' early examples in real few-shot learning tasks?** For instance, could it mean that the model would not be able to reason about information if it requires understanding the sequences it 'read' before the change in representations? If yes, would it mean that the model would benefit from repeating the early in-context examples again, once the model 'understands' them?\n\n2. What happens to the representation shift if you increase the number of concepts? Is there a point by which the model is no longer able to shift the structure properly to match the in-context 'meanings'?\n\n### Minor:\n\nL186 task-specific? (more popular with a dash)\n\nL269 “Given that we empirically do not observe this to be the case, we can safely assume this trivial solution does not arise in our experiments.” - slightly odd phrasing - you make an observation about your experiments that implies an assumption about the same experiments? Perhaps it would be best to paraphrase / clarify the footnote.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper investigates an interesting phenomena that may further our understanding of how LLMs learn in-context;\n- The paper includes a simple and easy-to-reproduce experiment that can facilitate future analysis of LLM representations;\n- The paper is generally well-written, if a bit unconventionally structured. It follows the natural flow of an investigation into how representations change during in-context learning;\n- Authors go to reasonable length to ablate alternative hypotheses in their analysis.", "weaknesses": "- The paper focuses primarily on just a single model: Llama 3 8B. It is possible that some of the authors findings reflect not the general ability of LLMs, but an artifact of Llama 3 models, or a property of models of this particular size. The paper would arguably be improved if authors verify their findings on other models: both of different size (e.g. 1B / 70B / 405B), different family (gemma, qwen, mistral, opt, etc) and of different type (e.g. instruct vs non-instruct). Failing that, authors could at least clarify this as potential limitation.\n\n- The part where authors argue that the in-context learning is linked to Dirichlet energy minimization is interesting, but (arguably) a bit contrived. As authors explain, energy minimization would be solved simply by assigning all nodes to the same embedding. Perhaps it would be best to further explore the analogy with graph embedding learning as a possible alternative? (e.g. distance-preserving embeddings or similar)\n\n- While I don't count this as a significant weakness, the paper could arguably be improved by further exploring *what* in the model causes the abrupt shift in representations. (e.g. is there any connection with induction heads, transformer circuits, or any other mechanism that could be responsible for the change).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Pp9G23mED", "reviewer": "Reviewer_a5tz", "summary": "This work designs a synthetic task to determine to what extent LLM  representations reflect in-context semantics rather than those observed during pre-training. \n\nTo do this they generate a graph structure, where nodes consist of tokens that are highly likely to be observed during pre-training, but with novel dependencies defined by the structure. They then generate sequences following a random walk across the graph, which are fed into the LLM 'in-context' (i.e. without weight updates). They then show that the LLM can learn to organise its representations such that they mimic the underlying graph structure.\n\nThe task is to some extent adversarial, because the context neighbourhoods defined by the graph differ from the standard semantics of the tokens.\n\nEvidence presented is both qualitative (using PCA) and quantitative (various measures for how well representations conform to the graph structure).", "questions": "Have you or do you plan to extend your investigation to more complicated structures? The ones presented in the paper are fairly simple and it would be interesting if there are certain forms that are harder to represent.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is excellently written, and the figures clearly get the central findings across the reader, making it a pleasure to read. The experimental design is original, elegant and well crafted in order to probe the hypothesis. Finally, the findings provide an interesting insight in the abilities of LLMs to zeroshot generalise to novel connective structures.", "weaknesses": "The work relies quite heavily on PCA, which the authors admit can be somewhat misleading, though they provide theoretical justifications for why this reliance is valid.\n\nOn a higher level I am not quite sure whether the finding is particularly surprising. Natural language is supposed to have similar causal relationships modelled by dependency parses, and LLMs are capable of modelling it successfully. Moreover, they are supposedly able to learn to translate hitherto unseen languages entirely in context ([1]), which constitutes a much more complex graph than the ones assessed in this paper. As a result I am not certain that the findings expose a new capability. However, as stated in the strengths section the paper provides a very nice framework for understanding this capability which should bias towards acceptance.\n\n[1] https://arxiv.org/abs/2403.05530", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MhjrjIVrKR", "reviewer": "Reviewer_EbvH", "summary": "The paper studies how the different representations are organized in a large language model’s internals. It reveals the sudden reorganization of representations as the context length increases.", "questions": "1. What is the layer of the figure 1(c)\n2. In Section 3.1, it is claimed that a window of $N_w=200$ preceding tokens is included in the computation of representations. This number is greater than the minimal context length of Figure 2(b), which is 100.\n3. How does Figure 3(c) prove that the LLM learns the in-context non-semantic structure?\n4. The percolation relates to the thresholds of having an infinite connected path. Can authors explain more explicitly about its relation to the problem considered in the paper?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The problem is interesting and important. \n2. The experiment results are interesting.", "weaknesses": "1. Lack of quantitative metrics of “the representations mirror the grid/ring structure” in Section 2: It is unclear how the PCA plot relates to the conclusions that the representations reflect the graph structure. The results in Section 4 may be helpful but require more discussion.\n2. The use of PCA: The PCA is useful for visualization but is not convincing enough to draw rigorous conclusions. Without the manually added vertices in Figures 1(c), 2(c), and 3(c), the results are not interpretable. \n3. Equ. (4) is the probability of seeing one node with each token randomly sampled with replacement. It does not match the data generating procedure with the first token randomly sampled, and the remaining tokens are generated through random walk.\n4. I do not find the percolation theory fits here. Although Figure 8(b) indicates a strong result, it is unclear how y and x connect to the percolation theory.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jBGWc9Q8SW", "reviewer": "Reviewer_hdrb", "summary": "This paper explores how in-context learning in large language models allows a reorganization of representations to fit novel task-specific semantics. The authors propose to override the pre-trained meaning of the words, thus the internal organization of the LLM representations with a semantic extracted from the neighbor relationships of names placed at the nodes of a graph. \n\nThe authors show that the LLM can internalize the graph structure learned in-context for sufficiently long input sequence sampled from the graph. They also connect their findings to energy minimization in graph and percolation theories, offering insight into how models might adapt internal representations based on in-context structure.", "questions": "Have you tried to see if your finding holds also on models other than Llama?\n\nl371, 372 Have you tried to check empirically if the coordinates z2 and z3 are aligned with the principal axis you find with PCA?\n\nWhat does DGP stand for in line 249?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a new task framework to study in-context reorganization in LLMs. It connects empirical observations with a theoretical foundation, connecting context-scaling with energy minimization. \n\n2. The graph tracing task using well-defined structures (grid, ring) is an effective method for observing changes in LLM representations in a controlled way.\n\n3. The paper is well-written and relatively easy to follow; the main goal is clearly stated, and the set of experiments are spot-on", "weaknesses": "The findings are based only on the analysis of llama3 representations, so extending the experiment to at least another model class would be easy. \n\nThe constructed task is somewhat artificial, and it is not immediately clear how we can extend these findings in the domain of natural language.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)