Latent Concept Disentanglement in Transformer-based Language Models
Authors: Guan Zhe Hong
Bhavya Vasudeva
Vatsal Sharan
Cyrus Rashtchian
Prabhakar Raghavan
Rina Panigrahy
Venue: ICLR 2026 Poster (2026)
Keywords: Mechanistic interpretability
in-context learning
transformers
large language models
disentanglement
Primary area: interpretability and explainable AI
TL;DR: We mechanistically analyze how transformers solve in-context learning problems which require multi-hop reasoning, or with continuous parameterization.

Abstract: When large language models (LLMs) use in-context learning (ICL) to solve a new task, they must infer latent concepts from demonstration examples. This raises the question of whether and how transformers represent latent structures as part of their computation. Our work experiments with several controlled tasks, studying this question using mechanistic interpretability. First, we show that in transitive reasoning tasks with a latent, discrete concept, the model successfully identifies the latent concept and does step-by-step concept composition. This builds upon prior work that analyzes single-step reasoning. Then, we consider tasks parameterized by a latent numerical concept. We discover low-dimensional subspaces in the model's representation space, where the geometry cleanly reflects the underlying parameterization. Overall, we show that small and large models can indeed disentangle and utilize latent concepts that they learn in-context from a handful of abbreviated demonstrations.

Published as a conference paper at ICLR 2026
LATENT CONCEPT DISENTANGLEMENT IN TRANSFORMER-BAS -BASED LANGUAGE MODELS
Guan Zhe Hong"! Bhavya Vasudeva*
Vatsal Sharan
Cyrus Rushtchian
Prabhakar Raghavan Rina Panigrahy UUniversity of Oxford University of Southern California Google Research
uk, guanzhe weeeeeoongggggee ox ac.uk, (bvasudev, vsharan @usc. edu, (cyroid, pragh, rinap @google. com
ABSTRACT
When large language models (LLMs) use in-context learning (ICL) to solve a new task, they must infer latent concepts from demonstration examples. This raises the question of whether and how transformers represent latent structures as part of their computation. Our work experiments with several controlled tasks, studying this question using mechanistic interpretability. First, show that in transitive reasoning tasks with latent, discrete concept, the model successfully identifies the latent concept and does step-by-step concept composition. This builds upon prior work that analyzes single-step reasoning. Then, we consider tasks parameterized by latent numerical concept. We discover low-dimensional subspaces in the model's representation space, where the geometry cleanly reflects the underlying parameterization. Overall, we show that small and large models can indeed disentangle and utilize latent concepts that they learn in-context from a handful of abbreviated demonstrations.
INTRODUCTION
Transformer-based Large Language Models (LLMs) demonstrate remarkable in-context learning (ICL) abilities: with only a handful of source-target demonstrations, they can generalize to new queries without any parameter updates (Brown et al., 2020). These successes hint that the models might be inferring latent rules or concepts implicit in the prompt. Understanding this goes beyond studying a specific question about ICL; it is key step to deciphering how the attention mechanism encapsulates the influence of prior tokens on posterior tokens, through latent intermediate representations.
Our work takes systematic view of core questions on how transformers process and use latent concepts to perform ICL. By latent, we refer to unstated variables or rules that are necessary for the computation. We also want to understand how the model represents the implicit functions, going beyond just measuring the accuracy of different tasks. Given the breadth of domains where ICL appears work, we are faced with a large canvas of experiment design: Is the model inferring an elementary function or latently performing chained reasoning? Are the hidden arguments instance- specific, abstract and reusable? How is ICL performed when "memorized world knowledge" is entailed, versus when the task relies on more elementary logic?
To explore these diverse dimensions, we design and implement two sets of experiments. The first addresses the question of whether and how transformers resolve hidden, intermediate entities in ICL when "memorized world knowledge" is entailed. An example is mapping an arbitrary city in country the capital city of this country. The second studies the structure of representations when the model performs numerical or geometrical computations. For example, the model may need to output the next point in traversal of circle rectangle. While these tasks are fairly simple, believe that studying them sheds new light on more fundamental questions, including whether the model is taking shorteuts or performing abstract reasoning in its hidden activations.
"Equal contribution.

---

Published as a conference paper at ICLR 2026
For tasks involving world knowledge, pre- pre-trained Gemma-2 models. We apply causal and correlation techniques to study how the hidden activations map to certain key parts of the solution process.
For the largest 27B model, we find that it relies on step-by-step composition of latent concept representations to obtain the result. In particular, we discover that sparse set of attention heads is responsible for resolving the intermediate latent concept (e.g.. the country when going from city to capital, or the company when going from product to headquarter city); the latent concepts exhibit orthogonality in the embedding space, Then, we show that there is a set heads and MLPs deeper in the model responsible for realizing the concrete output (such as the capital) from the intermediate concept.
In contrast to the large 27B model, we find that the smaller variant contains much weaker and noisier version of the 27B model's circuit. This corroborates the general wisdom that model size significantly impacts latent-concept disentanglement and composition abilities in the LLMs.
We also show that, as expected, adding more in-context examples leads to higher accuracy. We attribute this strengthening of the model circuit's 's causal importance and an increase in concept representation disentanglement. In other words, the model more fully utilizes its relevant sub- circuits when add more ICL examples.
Additionally, we show that the latent concept representations exhibit causal transferability to naturalistic prompts with open-ended generations, with its strength again scaling with model size.
Following these tasks involving world knowledge, we study quantitative self-contained tasks. This allows us perform more fine-grained experiments on two-layer models trained from scratch. The ICL tasks here are single-step "arithmetic": add-k, Circular- Cireular-Traecctory and Rectangular-Trajeer
Recent work has identified linear task vectors for problems such as basic arithmetic, single-step reasoning, and linguistic mappings (Todd al., 2024; Hendel al., 2023). Our key finding is that models not only have task vectors, but these representations reflect the geometry of the latent variable. For example, the task vectors for add-k almost entirely project onto line. More surprisingly. we can intervene on which function the model computes: interpolating along the task vector line approximately interpolates on the latent parameter for the task.
For the Sireular-Trajectory and Dectangular-Traaectee 2-dimensional space. This aligns with the linear representation hypothesis. but provides more nuanced evidence for it by showing a more continuous parameterization along the representation direction. As with the two-hop task, we provide evidence through both correlational analysis and localization of task vectors in the model's representation.
tasks, we see a similar geometry in
We designed these settings be clean enough for controlled experimentation, yet hint at some more general phenomena that deserve further study. For example, models can encapsulate the latent structures of the tasks they learn. Moreover, these structures may be localized and interpretable in the model. We posit that even for much more complex tasks, we will be able to find that the latent concepts are captured by a sparse set of attention heads relatively lowddimessional encoding.
1.1 PRELIMINARIES
To focus on analyzing whether and how transformers utilize latent concept representations for solving ICL, we work with prompts that contain demonstration-one specifications of latent functions. Intuitively, we have source x and target y, along with a hidden function F. As input, the model only sees handful of pairs (Ia, y1) where ys F(x;). Then, new x' the model has to compute F(a') to obtain correct answer. The subtle part that F R C, where C maps input to a low-dimensional "concept space", which is then refined by R produce the output.
The concept map is marked by its re-usability over different instances of sources and targets: for example, function that maps cities and landmarks to Country representations, or one that maps points on a circle to the same radius representation. Such abstract representations enable complexty inference. To test for the existence and utilization of such maps in transformers performing ICL, consider several varieties demonstration-onl prompts, under two categories. The first category focuses concept maps relying on world knowledge, with Countries and Companies serving as
2

---

Published as a conference paper at ICLR 2026
Section 2. Factuslty- based Muthi hop Reesoning Latent-concept compesition enabled by bridge and output-concept sub-dircults
Cap
Lagh,, Parn
Logh Rank
Brsdge conogpt embor Curput concept ambe
Nartes, Paris,
Rylhoy,
Ottows
Carberro
Thick sectioe
(Capit the country wheee
Kong anown
intevereed edibbi
Country
Nisgana
Nanten, Parts.
Sydney,
Cunbern
Univercty!
Calling Coote
Culllng
The auuttry wthene Zoita
10rg
Section 3. Murrerical Tesks with Latent Porameterization Task low-dimensional manifold
Input
Output
Proplem Structure
barth
in-earian deet learning Use Istent map input to output
Chicalr
Project Ti% pencipal
(owes
for ruckal
hnvwasiigg rad villea .....
Figure I: An illustration of our main findings. We primarily focus on how decoder-only transformer- based language models disentangle and manipulate latent concepts for solving in in-context learning (ICL) problems. In the discrete multi-hop setting, we discover that transformers compose latent concept representations for predicting the answer. For example, shown in the upper half of the figure, by intervening on certain "bridge-concept" attention heads, we can push the model's "belief" (reflected in logit and rank) in the original answer Canberra (the capital of Australia, which the city Sydney belongs to) to the "type-corrected" alternative answer Ottawa (the capital of Canada, which the landmark Niagara Falls belongs to). In the continuous- discover that transformers' hidden embeddings capture the geometry of the latent concepts for our prediction tasks. For instance, for transformer trained to predict circular trajectories whose radius is randomly chosen from a continuous interval, not only do we obtain causal evidence for task vectors (TVs) which control the trajectory's ' radius, but they also fall on a smooth 2D manifold.
continuou-pprammtterraailn parameterization ICL setting, we
the hidden concepts connecting the source to the target (Fig. 1 top half, $ 2). The second category focuses purely on self-contained numerical puzzles, with Radius, Offset and other numerical values serving as the hidden intermediate concepts (Fig. 1 bottom half, $ 3).
More precisely, we consider the following ICL settings:
Factuality- 2-hop Reasoning. This setting involves demonstration examples with under- specified reasoning sleps and discrete latent concepts. Specifically, we consider 2-hop factual recall tasks where the model must map a "source" entity to a "target" entity (c.g., (non-capital) city->capitall by first latently inferring the hidden "bridge" entity (e.g.. country), allowing us to ask whether the LLM first resolves the "bridge", then refines it to a "target" entity via (causal) con- cept compositions. An instance looks like "Toronto. Shanghai, ", where the answer is Bei ming..g.... The motivation here is to understand what the model does after reading the input city: does it jump directly to the capital, or does it first invoke the country an intermediate reasoning step? While both are plausible strategies, only the latter captures the latent causal structure that goes through the latent variable (the country).
Ottawa.
Mumbai, New Delhi.
Numerical Tasks with Latent Parameters. We generate demonstrations based quantitative parameters. We will see that this determines task similarity and induces smooth geometric relationships in the task space, suggesting that the model encodes such parameters along low- dimensional geometry. We first consider the numerical add-k task (also studied by (Hu et al., 2025)) where the demonstration examples are (Ix, yi)l with y 2; + k. where i the latent parameter. An example of add-4 is "5, 9. 3, 7. 1, 5. 2, ", where the answer is 6. We then study geometric tasks, such as Circular- Trajetor- where points lie on a eircle of varying radii and a related Rectangular- Trajeetorrr task. Our goal is to localize, and more importantly. understand the geometry of the task vectors: does it reflect the geometry of the latent parameter?
Both sets of tasks are demanding enough to require intricate latent concept disentanglement and manipulation, yet sufficiently controlled to permit causal, feature and circuit-level analysis.
3

---

Published as a conference paper at ICLR 2026
1.2 RELATED WORK
In-context Learning Interpretation. ICL abilities of transformer-basee based models were first observed by Brown et al. (2020), which sparked work in analyzing this ability. This includes analyzing how pretrained LLMs solve ICL tasks requiring abilities such copying, single-step reasoning, basic linguistics (Olsson et al., 2022; Min et al., 2022: Zhou et al., 2023; Hendel et al., 2023; Todd et al., 2024; Yin & Steinhardt, 2025a). and smaller models trained synthetic tasks like regression (Garg et al., 2022; Von Oswald et al., 2023; Akyürek et al., 2023; Bai et al., 2023; Guo et al., 2025), discrete tasks (Bhattamishra al., 2024), and mixture of Markov chains (Edelman et al., 2024; Rajaraman et al., 2024; Park et al., 2025a; Deora et al., 2025). These setups enable discovery of relations between in-context and wn-weight learning (Lin & Lee, 2024; Singh al., 2025; Russin et al., 2025), and internal algorithms that models implement (Olsson et al., 2022; Edelman et al., 2024; Li et al., 2023; Park et al., 2025a; Yin & Steinhardt, 2025a). We contribute to this line of work, by shedding light on how transformers solve ICL problems which have more intricate latent structures.
Linear Representation Hypothesis (LRH), Our results are also connected to the LRH, which essentially speculates that L.MMs represent high-level concepts in (almost) linear latent directions (Park et al., 2024; Merullo et al., 2024; Park et al., 2025b; Huh et al., 2024; Ilharco et al., 2023; Li et al., 2025a; Dumas et al., 2025; Beaglehole et al., 2025). Many papers motivated by the LRH then find "concept" vectors that can capture directions of truthfulness (Marks & Tegmark, 2024; Arditi et al., 2024), sentiment (Tigges et al., 2024), humor (von Rütte et al., 2024), toxicity (Turner et al., 2025), etc. We deepen this study, asking how LLMs' representations capture/disentangle latent concepts, and compose them during inference. In addition, the LRH is rooted in the field of mechanistic interpretability, which aims to reverse engineer mechanisms in transformer-basel LMs (Elhage et al., 2021; Olsson al., 2022; Singh et al., 2025; Wu et al., 2023; Wang al., 2023; Hong et al., 2024; Brinkmann et al., 2024; Bakalova et al., 2025; Heindrich et al., 2025; AlKhamissi et al., 2025b; Vig et al., 2020; Back & Tegmark, 2025; Wang & Xu, 2025).
Task and Function Vectors. A specific line of work in analyzing ICL mechanisms focus on task or function vectors. They show that there exist certain causal patterns which capture the input-output relationship of the ICL task, on relatively simple problems such as "Country to Capital", "Antonyms", "Capitalize a Word" (Todd et al., 2024; Hendel et al., 2023; Davidson et al., 2025; Yin & Steinhardt, 2025b). Similarly, Liu et al. (2024); Merullo et al. (2024); Li et al. (2025b); AlKhamissi et al. (2025a) observed that L.LMs compress certain task or context information into sparse sets of vectors. We work with ICL problems with more complex latent structures, and our focus is not solely on (high-level) task vectors, but on how the model disentangle and manipulate latent concepts useful to answering the query. In addition, work of Hu et al. (2025), providing add-k results for smaller models where we have full control over training and can hence conclude that the geometry of the task vector only arises from the latent task structure. We also compare results from add-k with other ICL tasks, giving additional insights.
work complements the function vector analysis from contemporaneous
2 DISENTANGLEMENT OF LATENT CONCEPTS IN 2-HOP REASONING
We start with a 2-hop task based on connecting two facts through a latent entity, where the model must infer the relationships from in-context demonstrations. This task builds on prior work that analyzes problem with single step of reasoning (I-hop) over world knowledge, such as geography puzzles (Country->Capitall or "National Park Park-tCountry" (Minegishi et al., 2025; Todd et al., 2024: Yin & Steinhardt, 2025a). Extending to two steps enables us to understand how LLMs solve ICL problems which have hidden reasoning steps, and understand whether they decompose the source-to-target function F into latent maps C and R (introduced in $1.1) using certain internal components.
Our goal is to mechanistically analyze how a pre-trained L.M solves a "source -targe" problem when the "bridge" concept is hidden. The model needs to understand the nature of the bridge from the demonstrations, which only contain the sources and targets. Interestingly, models achieve reasonably high accuracy with sufficient in-context demonstrations. This leads to two competing hypotheses:
Hypothesis I (Shortcut): The LLM maps the queried input directly to the answer (going from source to target), without internally computing the bridge entity in discernible manner.
4

---

Published as a conference paper at ICLR 2026
Hypothesis (Latent two-hop): The LLM first resolves the latent bridge concept (e.g., "country") in its hidden representations, then composes with the output concept to obtain the answer, effectively computing the source-to-tareet function F Ro C.
We first present the problem definition and experimental set-up. Then, we present our main causal and correlation evidence favoring Hypothesis 2.
The Touurce->Targe" Problem. We create two-hop ICL puzzles by composing two facts linked by a common "bridge" entity. That is, we sample fact tuples [(S,, rı, B,, r2, T where the source entity S; is related to the bridge entity B, via relation ri, and B, is related the largel entity T, via r2- We then create the ICL puzzle in the form [S1, T1- S2, T2 ...... Sn.] [Answer: Tn]². Note that the bridge entities Bd's are never specified in the prompt. An example City Capital problem is
Sydney, Canberra. Nantes, Paris. Oshawa,
Here, r1 is "belongs to the country of", the (unspecified) bridge entities for this example are "Aus- tralia", "France", "Canada", and r2 is "has capital". Therefore, the prompt's answer is Ottawa, the capital of Canada, the country Oshawa is in.
In the main text, we work with geography puzzles, with source types (City, University, Landmark), and output types /Capital, Calling code). In addition to this Country dataset, we work with the Company dataset in App. A.5 for generality². See App. A.2 for details on the data generation process.
City, Capital
City, Code
University, Capital
University, Code
Landmark, Capital
Landmark, Code
100
Accurasy Payy (%)
We focus on 2-22m (Gemma Team, 2024) in the main text. To establish a baseline. we evalu- ate Gemma-2-27B on Source-TTarge ICL puzzles. Fig. 2 reports its accuracy: although the model finds these tasks more challenging than one-hop counter- parts reflseeef by the need for more in- in-context ex- amples-it achieves high accuracy at 20 shots.
2
8
16
20
Number of In-context Examples
Figure 2: Accuracy of Gemma-2-278 on the two-hop "Source-> Targett" ICL. problems.
We provide new causal and correlational evidence suggesting that the model indeed performs latent multi-hop reasoning via sequential concept composition: first infers an abstract bridge concept representation (e.g. a "Canada" concept), and then specializes "capital" of "Canada") deeper in the model. Surprisingly, we find that the bridge-resolving mechanism enabled by a sparse set of attention heads in our set of problem settings.
specific output type (e.g. the
Methodology. We use causal mediation analysis (CMA), also known as activation patching (Pearl, 2022; Zhang& & Nanda, 2024) to obtain causal evidence for our claims (see App. A. for details). We discuss the bridge-resolving mechanism in the main text, and delay the analysis of the output-concept component to App. A.2. For concreteness, consider two prompts with different souree-target types:
A normal prompt, with type [City CCayiaCa "Okinawa, Tokyo. Sydncy, Canberra. Chicago," [Answer: Washington; Bridge: USA].
An alternative prompt, with type [Landmark- Code]: "Chapel bridge, 41. The Grand Canyon, 1. The Great Wall, [Answer: 86; Bridge: China).
We perform activation patching on normal and alternative problem pairs with different bridge entities, across source- target types, at the final token position. Our experimental hypothesis is that if a (set of) model component effectively serves as the hidden concept map C. that is, it computes an abstract representation of the bridge, then this representation should transfer across different source and target types. When we run the model on the normal prompt, and replace selected component's s activation by the corresponding activation obtained on the alternative prompt, the model should favor the alternative prompt's 's bridge without bending the output type. For our example, the alternative-to. normal "patching" pushes the model's 's answer on the normal prompt from Washington Beijing
We think of this as a systematic ICI. version of TWOHOPFACT (Yang et al., 2024). Here, the model must figure out relations between the source-bridge and bridge-targer facts from the ICI. examples.
For our Country& & Company data, we collect source and target entity data for 40 countries, 36 companies.
5

---

Published as a conference paper at ICLR 2026
120
Nash icey, talini Capkal)
bay
logh arterton
b00
03
Pasch (univariiyy Captixt--CCtp Coca)
itandmark, Codehelcity
Spa
Unl, Capial
40
20
03
(24, 10.313
(3%, 23.23
03, 20,237
(25 9:90
Spol
(6) Logit difference variations from select activation patching experiments
City Jdy
City Spah
arnowe recps nenc, after patcting arswer roni. pasching
Leky Codc
percomle
Cody
03
Neterbagg
30%
Joyy
02
Landinor,, Spat
unis Captal
Landmar Opeai
ead
on cay, Cinea
Lnit Clre Landeasne And
Londoar Clser
City
cat
0%
10
10
30
10
peciproca mank
(b) Reciprocal runk of the altered answer before and after
(c) Cosine similarity "Ialy""" and "Spain" representations from bead group (24,30:31)
patching head group (24,30,31)
Figure 3: We present causal evidence for the bridge-resolving mechanism in (a) and (b), and (a) and (b), we run activation patching experiments
correlational evidence in (c) for Gemma-2-27B. across prompts with problems of different source and target types, at the final token position. (a) shows the percentage logit difference variation of attention heads with the strongest causal influence, on select intervention experiments. Head group (24,30;31)'s representation exhibits strong transferability across source and target types. (b) further examines the causal role of (24,30:31) 1)², in the patching experiment [University, Code]-+[City, Capital]. Note that the reciprocal rank of 10-1 1 for the type- corrected alternative answer after patching (24,30331) is in the 27th percentile (> 73% in top 10 after intervention). In (c), we show cosine similarity plot of bridge-representation disentanglement for head group (24,30;31), I), computed on a collection of samples with several combinations of source and target types, and with "Italy" and "Spain" as the bridge values in the prompts.
(the output type remains Capital, but the bridge switches from USA China). We refer to this as the "type-corrected" version of the alternative answer. This evaluation is slightly unorthodox: rather than judging an intervention by whether it reproduces the literal alternative answer (86). we assess movement toward the type-corrected alternative (Beijing). By contrast, under the Shortcut hypothesis, patching activations would worst break the model and yield nonsensical outputs, or at best, push to emit the literal alternative answer (86).
We now formalize the intervention with CMA, which enables us to localize components inside the LLM that specialize in handling the different (latent) reasoning steps of the problem.
Causal Mediation Analysis (CMA). Formally, let the normal and alternative prompts be denoted as
Poom | murmm) _- (nooo)) ...5% g((uom)) .] and Pal = ||((a)) ( s(("l)) ... Let ( = Type (Tr(all)) denote the type-corrected output. We run the LLM on both the prompts and cache the LLM's hidden activations at the last token position, denoted as (hmorm, har). To perform CMA on model (alt), component of interest, say an attention head with activations indexed by (a (nomm) ag ), we run the LLM on the normal prompt again, but this time, intervene by replacing its normal activation
with the alternative ag and let the remainder of the forward pass execute. We then contrast the model's output behavior on the normal and patched runs by, for example, examining the logit differences between the normal and (type-corrected) alternative answer, and their ranking pre- and post-intervention. Further details on CMA and its suitability to our problem setting are deferred to App. A.I.
2.1
RESULTS AND ANALYSIS FOR THE BRIDGE-RESOLVVING MECHANISM
Causal evidence. Fig. 3 provides evidence favoring the Latent Two-hop Hypothesis rather than the Shortcut Hypothesis: we observe causal transferability of the bridge representation. In Fig. 3(a),
We denote attention head groups with (layer index, head index I; head index 2; ... head index k).

---

Published as a conference paper at ICLR 2026
report results on select set of patching experiments: both tasks with and without overlap in source and target types, we observe that a sparse set of attention heads consistently exhibits very strong causal effects in pushing the model's "belief" from T T towards T,a (reflected in logit difference); the head group (24,30;31) 1) is especially dominant." To further understand whether intervening (24,30;31) during the normal run really boosts the type-corrected alternative answer f (instead of only decreasing model's 's confidence on the normal answer T Ty( which logit difference might not tell), in Fig. 3(b), we show example patching experiment result of [University, Cody Capital]. Surprisingly, at least 73% of the time, patching this head group boosts the model's rank of the alternative prompt's 's answer into top 10 (and directly become the top-1 answer more than 40% of the time!), when its original, intervention- free rank is typically in the hundreds to thousands. In fact, we observe highly similar trend across all combinations of source and target types on our dataset (delayed to App. A.2).
Correlational evidence. To understand the nature of (24,30;31)ss output embeddings better, in Fig. 3(c), we visualize an example cosine similarity matrix of this attention head, with either "Italy" or "Spain" as the bridge values for S,, (the query source entity) in the prompts, across a total of 12 different combinations of bridge, source and target types. Specifically, for each combination of the bridge and cource-targer type shown in the grid, we sample 10 prompts which obey such requirement, giving us total of 120 prompts. We then obtain head group (24,30:31)'s embedding of these prompts at the last token position, and compute the pairwise cosine similarities. Observe that the embedding consistently exhibits strong disentanglement with respect to the bridge value in the prompt, regardless of source and target rypes. We provide detailed statisties of disentanglement strength in App. A.3 and Fig. 27, where we discuss its relation with the number of ICL examples.
The causal and correlational evidence suggest that there is a small set of components in the model which effectively serve as the latent concept map C: their activations on the queried entity are causally transferable across source and target types, and exhibit strong intra- serra-brdde-cconeee inter- anter-bridee-ooncee orthogonality (low dimensionality of the representations). To further corroborate the observations made in the main text, we present detailed activation patching results which sweep all combinations of source and target types in Appendix A, spanning Figures 12 21 and 27.
clustering and
Additional Insights. We highlight some additional mechanistic insights below (see Appendix for details). First, the Gemma-2-2B model has weak and noisy version of the 27B's bridge resolving mechanism. This suggests that increasing model size likely benefits latent concept disentanglement and utilization in the LLM. We provide details in App. A.2, and Fig. 26. Second, in App. A.3, we show that disentanglement strengthens with more ICL examples, which is reflected in both the causal importance of the bridge-resolving components and the angular (dis-) ddiilaiiisssssiess of the bridge embeddings. Lastly, to complement our study on the geography dataset, we experiment with a similar, albeit scalee Company dataset in App. A.5, where company name is the bridge entity. We again find causal and correlational evidence for the presence of bridge-resolving mechanism in the model, and observe overlap in the attention heads driving this mechanism on the two datasets. Furthermore, to probe whether our findings extend beyond the synthetic setup, Appendix A.4 presents a small-scale study where the same concept embeddings are used for intervening the model in naturalistic prompts. Injecting these vectors tends to coherently steers open-ended generations toward the target country or entity type while preserving fluency, providing preliminary evidence that the learned concepts are not merely puzzle-specific.
DISENTANGLEMENT FOR NUMERICAL LATENT VARIABLES
In this section, we consider two problems with numerical or continuous parameterization. For these experiments we study a very small transformer, with similar architecture to GPT-2 (Radford et al., 2019). We use 2-layer 1-head transformer, with embedding dimension 128, trained with AdamW.
add-k Problem. Each task is a sequence consisting of pairwise examples ((x,, v:)) where U: = Z; + hi for given offset k. Here, we use integer inputs and offsets; all values are in {0. V 1}, each treated as distinct token. We consider collection of K tasks parameterized
'In our CMA experiments, we account for grouped-query attention by patching heads in groups of 2 on Gemma-2-227. We noticed that this tends to produce stronger causal effects than with individual heads. 'We only specify the bridge entity for S, in the prompt; the hridge for S2 for all i are randomly chosen.
7

---

Published as a conference paper at ICLR 2026
by different offset values in (K₃ [4]],, where k and fix he+1 K2 3. The model is trained autoregressively to predict the label for each example in the sequence. At test time, the model observes the first examples and should predict Vn+1 Ix+1 + k for the last example. Here, the latent concept map C' needs to produce the offset representation.
Sireular-Trajector Problem. Here task consists of a sequence {x of points on a circle centered at the origin. Each task is parameterized by the cir- cle's radius r; for K tasks, the set of radii LK sampled uniformly from [1, 4]. A task sequence generated as follows. We first sample d, uniformly at random in [0, ].. so X1 = r|cos Oo, sin 0o)². Then, we select the period p randomly from (2,3, 4), which determines the number of equal consecutive step- sizes. Specifically, we first sample sequence of [ + unique step-sizes uniformly between [0, 1]. and then get the full sequence of steps {a,) where a = 0j+1 =
)++1
-2-
Figure 4: Illustration of an input sequence
aj+p-1 for j E {0. p0.p..p. ). Here, context length n 12m
for the circle trajectory problem. Here, radius r=3, period p= 2, sequence length n = 13. Every p consecutive steps on the trajectory are equal. We first sample [ + unique step-sizes in [0, 1J, and get the full sequence {ai, a22 a3, aa..... .. where same colors de- note equal step-sizes. Then, we generate the trajectory by rotating point x; clockwise by angle a (see text for formal description).
for integer m. We also sample с E [=1]. which denotes the trajectory is clockwise or counterclockwise. Next, we generate a sequence of angles {0,)%-. where 0. 0o + c gles, we generate the sequence, X = r.((0))) ( where R(0) is the 2D rotation matrix for 8. Fig. 4 shows an example. As in the previous problem, we train the transformer autoregressively on these types of sequences. Additionally, in App. B, we consider another shape problem, namely the Rectangular- Trajectory problem, where the trajectories contain points on lying on axis-aligned rectangles centered problem is parameterized by one continuous parameter (radius), at the origin. The Circular- problem has two parameters, namely the lengths of the two sides whereas the Rectangular-Trajee of the rectangle. Here, the latent concept map C' needs to produce the Radius or representations.
n Ejs; az. Using the sequence of an-
3.1 RESULTS
Existence of Task Vectors. We first outline the process to identify the task vectors for the add-k problem. We set V 100, n = 4, and K 2. Fig. 5 shows the cosine similarities between the layer-2 attention embeddings at the last position for 200 input sequences from each of the two tasks. We observe clustering between intra-task embeddings. Hence, the model disentangles the concept of different offset values in its representation. To provide more evidence for disentanglement, and to locate where the task vectors emerge in the model, we linear probe the embeddings of the model to predict the final output and the offset/task type. We probe embeddings from the output of the MLP at the first layer, the attention block at the second layer, and the hidden and output layers of the MIP at the second layer. The results are in Fig. 6. We see that task type disentangled at layer-2 attention, and the output is computed at layer-2 MLP. For each task, we treat the layer-2 attention embeddings averaged across 200 input sequences from that task the task vector.
We visualize the task vectors by performing PCA and projecting them onto the first two principal components; Fig. 7 presents the task vectors for the add-k problem, for K 4. 8, 16 tasks/offsets. In all three settings, the vectors lic on ID linear manifold. More than 99.9% of the variance is explained by the first PCC Notably, the model compresses the concept of offsets into a line with the ordering of the offsets (lower to higher) preserved on the manifold (left to right). To corroborate these results with causal evidence, we study the effect of steering using the task vectors. Specifically, let t and tx denote the task vectors for offsets ki and kk, respectively. Then, for offset ki (kx), we steer with (1 - B1 + Stx ((1 3)tx + Btt) and evaluate the accuracy for predicting the output based on the original offset Kn (kg), the 'opposite' offset ky (k1), the target offset (1 9)k1 + 3kk ((1 R)kx + 3kk,, where 3 E [o, 1]. We use the first/last offsets to interpolate them and steer towards intermediate offsets. Fig. 8 presents the top-1 and top-3 accuracies for each case. High
8

---

Published as a conference paper at ICLR 2026
Ouput
Ourpe houreey
Mope Acoracy
320
540
Accuraty
80
29
075
Layse-1
Layer-22 Layer-2 Hidden
Layer-2 MLP
Figure 5: Cosine similarities between the Figure 6: Results for linear probing the embeddings layer-2 attention embeddings for 200 input of the trained model at various locations to predict the sequences from two tasks/offsets for the add- final output and the task type for the add-k problem. k problem. Strong clustering between intra- The task type becomes disentangled at layer-2 atten- task embeddings shows that the model dis- tion, and the output is computed in layer-2 MLP. entangles the concept of different offsets.
top-1 accuracies and R 100% top-3 accuracies the target for considered values of 3 indicate that the model output is steered toward the target. Interpolating along the top principal direction is successful at interpolating values of k in the task space, showing that the model captures the concept's geometry. In other words, the layer-2 attention head is serving the latent concept map C' in this setting, mapping input tuples to the offset parameter.
Tasks
Tasks
Tasks
-2
-4
-2
2
Figure 7: 2D PCA projection of the task vectors for the add-k problem. The task vectors lie ID linear manifold. Here the number of tasks refers to the number values of k.
Org Top-1 Tasks
Org Top-3
Target Top
Target Top-3
Opp Top-1
Opp Top-3
Tasks
16 Tasks
Accuracy (%)
60-
8 0.8
1.0
8
Figure Steering with the task vectors for tasks k and kg for the add-k problem (see text for details). We plot the top-1 and top-3 accuracies for predicting the output based on the original offset ki (kx)), the 'opposite" offset kx (k1), ). or the target offset (1 B)k + Bkx ((1 - 33kk + 3k1), where B E [0, 1]. The result shows that the model output can be steered toward the target.
16 Tasks
32 Tasks
64 Tasks
0.25
0.00
-025
-0.4
-0.2
06.0.6
Figure 2D PCA projection of the task vectors for the Circular Trajector problem. The task vectors lie on a smooth low- -iwddimnnsionll manifold. Here the number of tasks refers to the number of radius values used for training.
Fig. 9 presents the 2D PCA projection of the task vectors for the Cireular-Traine Trajectory problem, for
K = 16, 32, 64 (training) tasks/radii. In this setting, we consider K = 24 radii, spaced evenly between [1, 4] visualize the task vectors, since this task is continuous. We observe that in all three

---

Published as a conference paper at ICLR 2026
Target
Opp
16 Tesks
32 Tesks
64 Tacks
0.04
0.00
0.0
0.2
0.4
0.8
0.8
1.0
0.0
0.2
0.4
0.6
0.8 1.0
0.0
0.2 0.4
0.6
0.8
1.0
Figure 10: Steering with the task vectors for tasks and тк for the Circular- Trajectoy problem (see text for details). The MSE between the radius inferred from the model output and the original radius (rx). the "opposite" radius тк (r1). or the target radius (1 =))r1 + Brx ((1 3)rx + Br1),
where 3 E [0, 1]. indicates that the model output can be steered toward the target.
settings, the task vectors lie on low-dimessional manifold. The variance explained by the first two PCs in the three cases is 97.097.00. 97.05%, 96.44%, 93. (3.68% respectively, Similar to the previous setting, the order of the radii (lower higher) preserved in the compressed representation.
Fig. 10 presents the results for steering the model output using the task vectors for radii f1 and тK- We follow the same procedure as in the add-k problem, with different evaluation metric. We compute the norm of the generated output after steering as the model's 's radius (since the center of the circles is fixed the origin), and consider the MSE between these radii and the original radius ri (rx), the 'opposite" radius (r1), the target radius (1 3)r Brx ((1 - Brr + Bri), where [0, 1]. averaged over 200 sequences from each task. We observe that the MSE with the target radius is the lowest, which indicated the task vector steer the model's 's output toward the target.
In App. B, we examine the task vectors for the Rectangular-Trajee Trajectory problem. Here the model has to reason over latent continuous parameters, which are the real-valued side lengths of the unknown rectangle. We find that the first 2 PCs lie on a two-dimensional manifold in that case as well. This provides second example of how the model captures the underlying geometry in terms of both separating the two orthogonal parameters and smoothly interpolating across hidden trajectory shapes.
DISCUSSION AND CONCLUSION
Our work provides causal and correlational evidence that transformer-based models disentangle and manipulate latent concepts from in-context demonstrations in a structured, interpretable manner. For two-hop tasks, models contain sparse sets of attention heads responsible for first inferring the bridge entity and then resolving the output. For numerical tasks, the model uses task vectors which closely capture the underlying parameterization. Overall, we hope that results from our controlled experiments can serve as stepping stone for future empirical and theoretical analysis of transformer models; we discuss potential theoretical implications of our results in Appendix C.
Limitations and future directions. There are a few limitations of and possible directions to extend this work, which we discuss below:
The latent concepts that we test for all have easy to understand human representations, such as countries, companies, numbers, or geometric shapes. There may be other, more intricate, relationships between the ICL examples that the model is also representing in some way. It would be ideal to provide ablations over the human-intenn intrannitterrttabilityy of the provided examples, to see if this affects how the model represents the latent concepts.
The ICL problems have unique answers instead of allowing for ambiguity (e.g. many- to-many instead of many-to-one mappings, or maps involving multiple possible bridge concepts). While this setup enables us to conduct systematic experiments and obtain clean mechanistic insights, it might limit our understanding of how certain concepts could occur in "superposition" in the LLM's representations in some scenarios.
Future work could leverage the identified concept representations directly for steering model behavior improve ICL performance or by composing known concept vectors to enable zero-shot generalization to novel compositional tasks. The identified concept embeddings also serve probes to inspect which latent concepts used during reasoning; this approach can complement text-level monitoring, considering the lack of faithfulness of CoT (Turpin al., 2023; Chen et al., 2025; Korbak al., 2025).
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
VS and BV were suppored by NSF CAREER Award CCF-2239265, a Google Research Scholar Award and Okawa Foundation Research Grant. The authors acknowledge the of USC CARC's Discovery cluster.
REFERENCES
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? Investigations with linear models. In Int. Conference Learning Representations, 2023. URL https: wwwpreeeeeeeeeeeeeeeeeeeeee net. net/ forum 2imimmid=00000 0x H8yN444
Badr AlKhamissi, Greta Tuckute, Antoine Bosselut, and Martin Schrimpf. The LLM language network: A neuroscientific approach for identifying causally task-relevant units. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.). Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp- 10887-10911. Albuquerque, New Mexico, April 2025a. Association for Computational Linguistics. ISBN 979-8-9917666 URL http::ttt///om. ac antho ogy org/ naacc ..o.llong.5444
doi: 10.18653/v1/20255 hww.22222..aal-loooo...5..
Badr AIKhamissi, Greta Tuckute, Antoine Bosselut, and Martin Schrimpf. The Ilm language network: A neuroscientific approach for identifying causally task-relevant units, 2025b. URL
https: /twpiv.xvvvvvvv////a//aaa..a.av. owg/aoggg/....oooooo......oooooe aaaaabbs///// /211 02280.
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction, 2024. URL https:
//arxiv.orgg/attti htwp/ogg//aabaaaoooooo/o//eas/ /t////ba/224 220022466.00 11717.
David D. Back and Max Tegmark. Towards understanding distilled reasoning models: A representa- tional approach, 2025. URL htttttt///aaaaae /tapp::///rxiis org/ abs /2003 03730.
Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians:
Provable in-context learning with in-context algorithm selection, 2023. URL https://arxix: org/ 2306. 04637.
Aleksandra Bakalova, Yana Veitsman, Xinting Huang, and Michael Hahn. Contextualize aggregate: Circuits for in-context learning gemma-2 2b. arXiv preprint arXiv:2504.001333 2025.
Daniel Beaglehole, Adityanarayanan Radhakrishnan, Enric Boix-Adserà, and Mikhail Belkin. Ag- gregate and conquer: detecting and steering Ilm concepts by combining nonlinear predictors over multiple layers, 2025. URL https: /ats:://arrae /ttx/arxvv.o/ org ab5c///bb/2222.... 03708.
Satwik Bhattamishra, Arkil Patel, Phil Blunsom, and Varun Kanade. Understanding in-context learn- ing in transformers and LLMs by learning to learn discrete functions. In The Twelfth International Conference on Learning Representations, 2024. URL https: / /ttpa/:///pp/eeveeeeeee net/ forum? id=ekeyCgeRFC.
Jannik Brinkmann, Abhay Sheshadri, Victor Levoso, Paul Swoboda, and Christian Bartelt. A mechanistic analysis of transformer trained on a symbolic multi-step reasoming task. arXiv preprint XXX::2402.119111 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- neurips. oc wwtppppeeriieee/ppeeeeeeeeeeeeee 1atraaabfbbaccc
ciates, Inc., 2020. URL httptttttp:://eeeeeeettt waleaac144fff66 42f64a Paper .ddf. 2020 1eel1eee15770dfffffeeooo
11

---

Published as a conference paper at ICLR 2026
Yanda Chen, Joe Benton, Ansh Radhakrishhan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, and Ethan Perez. Reasoning models don't 't always say what they think, 2025.
URL https::////......... //trr//raa wwrxiv.orr abs 2505. 05410.
Guy Davidson, Todd M. Gureckis, Brenden M. Lake, and Adina Williams. Do different prompting methods yield a common task representation language models?, 2025. URL https: /tappp:://axxiee //axivv. org/abs/ 2505. 12075.
Puneesh Deora, Bhavya Vasudeva, Tina Behnia, and Christos Thrampoulidis. In-context occam's razor: How transformers prefer simpler hypotheses on the fly. In Second Conference on Language Modeling, 2025. URL https:
httss:///ppnnn http///ppppeevieeeeee net/ forum? d
Clément Dumas, Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. Separating tongue from thought: Activation patching reveals language-agnostic concept representations in transformers, 2025. URL httph:tt///eeet //rpsii//axvv.../////
org/obrgs/s 2411 08745.
Ezra Edelman, Nikolaos Tsilivis, Benjamin L. Edelman, Eran Malach, and Surbhi Goel. The evolution of statistical induction heads: In-context learning markov chains. In The Thirty-eighuh Annual Conference on Neural Information Processing Systems, 2024. URL https: //pppr////peeeeiieee net/ Eorum? d=qaRT SQTIqJ.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodde Danny Hemaandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. html.
https:/transformer-ccccuit wttp:iissspp.//222///aoeooooooo
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? case study of simple function classes. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances Neural Information Processing Systems, 2022. URL pestps://openreviie http://www.ee//ouuimiiieeee
net forum?id (fw.rrmiid-..222eee0000 f1NZJ2e0et.
Gemma Team. Gemma 2: Improving open language models at a practical size, 2024. URL
btttss///rxiv..r wtarg///b////a 2/00///0//00111
Tianyu Guo, Hanlin Zhu, Ruiqi Zhang, Jiantao Jiao, Song Mei, Michael Jordan, and Stuart Russell. How do llms perform two-hop reasoning in context? arXiv preprint arXiv:2502.13911, 2025.
Lovis Heindrich, Philip Torr, Fazl Barez, and Veronika Thost. Do sparse autoencoders generalize? a case study of answerability, 2025. URL https: //ae:://aaeee //tp///avvvv//////////eee///aev/ www.ooog/////.ooooooooooooooeoo/ /20/2.50220 19964.
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https: 9n.
//openreview.oo net wow.umiii-gyytUut
Guan Zhe Hong, Nishanth Dikkala, Enming Luo, Cyrus Rashtchian, Xin Wang, and Rina Panigrahy. How transformers solve propositional logic problems: A mechanistic analysis. arXiv preprint arXiv:2411.04106 2024.
Xinyan Hu, Kayo Yin, Michael I Jordan, Jacob Steinhardt, and Lijie Chen. Understanding in-context learning addition via activation subspaces. arXiv preprint arXiv:2505.05 2025.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07988 2024.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic, 2023. URL https:
//taxiiv.cco wwt.xiii..rggbb...o.o 2212. 04089.
Subhash Kantamneni and Max Tegmark. Language models use trigonometry to do addition. arXiv preprint arXiv:2502.00873, 2025.
12

---

Published as a conference paper at ICLR 2026
Tomek Korbak, Mikita Balesni, Elizabeth Barnes, Yoshua Bengio, Joe Benton, Joseph Bloom, Mark Chen, Alan Cooney, Allan Dafoe, Anca Dragan, Scott Emmons, Owain Evans, David Farhi, Ryan Greenblatt, Dan Hendrycks. Marius Hobbhahn, Evan Hubinger, Geoffrey Irving, Erik Jenner, Daniel Kokotajlo, Victoria Krakovna, Shane Legg, David Lindner, David Luan, Aleksander Madry, Julian Michael, Neel Nanda, Dave Orr, Jakub Pachocki, Ethan Perez, Mary Phuong, Fabien Roger, Joshua Saxe, Buck Shlegeris, Martín Soto, Eric Steinberger, Jasmine Wang, Wojciech Zaremba, Bowen Baker, Rohin Shah, and Vlad Mikulik. Chain of thought monitorability; new and fragile opportunity for ai safety, 2025. URL https: /ttpiaaaiivvii.a///aa/.eaaeaawi. org/ abs//507, 1 11473.
Yingcong Li, Kartik Sreenivasan, Angeliki Giannou, Dimitris Papailiopoulos, and Samet Oymak. Dissecting chain-of-thought: Compositionality through in-context filtering and learning. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL https: // //openreview net. forum? SamniddxxEEKssxaa.
Yuxiao Li, Eric J. Michaud, David D. Back, Joshua Engels, Xiaoqing Sun, and Max Tegmark. The geometry of concepts: Sparse autoencoder feature structure. Entropy, 27(4), 2025a. ISSN 1099- 4300. doi: 10.3390/e27403344, URL https: / www mdpi. com/ 1099- 0009994330222///3333
Zhuowei Li, Zihao Xu, Ligong Han, Yunhe Gao, Song Wen, Di Liu, Hao Wang, and Dimitris N. Metaxas. Implicit in-context learning. In The Thirteenth International Conference on Learning Representations, 2025b. URL https: //l::///oppae http://pppeeeee.eeeee net /a///formmm/// C=C774uee6cn
Ziqian Lin and Kangwook Lee. Dual operating modes of in-context learning. In Proceedings of the 4 1st International Conference on Machine Learning. ICML'24. JMLR.org. 2024.
Sheng Liu, Haotian Ye, Lei Xing, and James Zou. In-context vectors: making in context learning more effective and controllable through latent space steering. In Proceedings of the 41st International Conference Machine Learning, ICML'22. JMLR.org. 2024.
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets, 2024. URL https:://////. / http:///axvvvvv/e org/abs argg/ba/23100 2330. 06824.
Kevin Meng. David Bau, Alex J Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL. https::/tpppseeeeeeee /ttpp//////eeeeiieeeeee net pet/forung iin--iidd-- weedd-hhwwAs. h6WA56eE4.
Jack Merullo, Carsten Fickhoff, and Ellie Pavlick. Language models implement simple Word2Vec- style vector arithmetic. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). pp. 5030-5047, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 1/0.55///1/2222maacloooooo/
URL https: / wwwannhhaollgoeooo..c.oooccooola org/ 2024 naac loneeclloonne long. 281/.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11048- 1064, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 10.1863/1//222.. emnlp-main.756 759. URL ht tet::p: aaclanthaoloooooooooooo org 2rg/202/ 2022.2222.222 emnlp ma in. 759/.
Gouki Minegishi, Hiroki Furuta, Shohei Taniguchi, Yusuke Iwasawa, and Yutaka Matsuo. Beyond induction heads: In-context meta learning induces multi-phase circuit emergence. arXiv preprint arXiv:2505. 16694, 2025.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac HatfieldDoodee Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse. Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022.
13

---

Published as a conference paper at ICLR 2026
OpenAI. Gpt-4o mini: advancing cost-efficient intelligence. https: catteaaii..cmmiiaim commindex/ 2025-12-03.
2024. Accessed:
Core Francisco Park, Ekdeep Singh Lubana, and Hidenori Tanaka. Competition dynamics shape algorithmic phases of in-context learning. In The Thirteenth International Conference on Learning Representations, 2025a. URL http::::://ppeeevie...... net forum? 1d-ggllwfHSSSSS
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In International Conference on Machine Learning, pp- 39643-39666. PMLR, 2024.
Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models, 2025b. URL https://aaxiii wwwp:/viworra//oooo orgoorg///aao. /bss/ 2406. 01506.
Judea Pearl. Direct and Indirect Effects, pp. 373-392, Association for Computing Machinery, New York, NY, USA, edition, 2022. ISBN 45035508... URL httts::: httLhttss://di..orr /ttp:////ooooo./...///// 1145/ 35C171 117.4444 1.6.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. In OpenAl blog, 2019. URL attps://app
semanticscholad org/ /otp/oooosID
160025533..
Nived Rajaraman, Marco Bondaschi, Ashok Vardhan Makkuva, Kannan Ramchandran, and Michael Gastpar. Transformers on markov data: Constant depth suffices. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL URLAttps://opeereviees forum?
Jacob Russin, Ellie Pavlick, and Michael Frank. The dynamic interplay between in-context and in-weight learning in humans and neural networks, 2025. URL https:/ http://wwxwwv....aab// 2402. 08674.
Aaditya K. Singh, Ted Moskovitz, Sara Dragutinovic, Felix Hill, Stephanie C. Y. Chan, and Andrew M. Saxe. Strategy coopetition explains the emergence and transience of in-context learning, 2025. URL https//:: org/ ab5//533. 05631.
Curt Tigges, Oskar John Hollinsworth, Neel Nanda, and Atticus Geiger. Language models linearly represent sentiment, 2024. URL https: http///pppereeeeeeee net./ httt://ooummmddd.cceee TYa.
Eric Todd, Millicent Li, Amab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. In The Tweifth International Conference on Learning Representations, 2024. URL https: wwwpoeviee..eeee// net forum wtt.://eoooooomo///ooottttttttte Aid=Awyxt
Edward Turner, Anna Soligo, Mia Taylor. Senthooran Rajamanoharan, and Neel Nanda. Model or- ganisms for emergent misalignment, 2025. URL https: https://arxiv/ wwwpsiiiiorrreee..e wttp:///a.b./../....1..
Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language models don't always say what they think: unfaithful explanations in chain-of-thought prompting. Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 12388-12401. Curran Asso- ciates, Inc., 2020. URL http::: http::::://////ceeeeeeeee wtw.annng.ssuuuueeee reurips cc cc/paper_..ppee files/ /ppper//p/////// Paperr pdf.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Joao Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 35151-35174, PMLR, 23-29 Jul 2023. URL https::tpt..............t....s.. proceed: ngs ml www.mmlw.........mmmmmmmmm www.s//ssvveee.... v von oswal d223a html.
14

---

Published as a conference paper at ICLR 2026
Dimitri Rütte, Sotiris Anagnostidis, Gregor Bachmann, and Thomas Hofmaan. A language model's 's guide through latent space, 2024. URL https: http/aaiivv.o///a//e org h/////bbsa abs wwtpaaa...aaaeeee 144333.
Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokking of implicit reasoning in transformers: A mechanistic journey the edge of generalization. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 95238-95265. Curran Associates, Inc., 2024. doi: 10.52202/ 079017-3017. URL https: http::s//ppccceeinngg.. neurips cc papprr files/papez adleee//d22eeee 7e00 fece 71 bdF
2024/File/dd217et
Kevin Ro Wang. Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: circuit for indirect object identification in GPT-2 small. In
The Eleventh International Conference on Learning Representations, 2023. URL https:
//openrevi eww net. fortt//orum??
Zijian Wang and Chang Xu. Functional abstraction of knowledge recall in large language models, 2025. URL https: http:v///axxvv.v/. www.a/wwww....aaaaaaaa.oeeeeaa/a www//a////////aaao//////a/./aaaa 12tp:.550.......
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite. Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-ar art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Pro- cessing: System Demonstrations, pp. 48-45, Online, October 2020. Association for Computational Linguistics. URL www.hhttt/.../...... www. ac web . org/ hotooooyyhyooyooooooooooooooooo 2020 0. emnlp p-demos 6.
Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christopher Potts, and Noah Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca. In Thirty- Conference on Neural Information Processing Systems, 2023. URL https: http::://ppeeeeveewee net ftr.mmirrmmi nRfCInMhVX.
Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning? Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10210-10229, 2024.
Kayo Yin and Jacob Steinhardt. Which attention heads matter for in-context learning? In Forty- second International Conference on Machine Learning, 2025a. URL https: / //ppnrevie/// pet/forum: www.:ouuummiiiicceeeeee CC
Kayo Yin and Jacob Steinhardt. Which attention heads matter for in-context learning?, 2025b. URL https://arxiv wttp:/o/////o//aoooooooo 2202. 14010.
Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. In The Twelfth International Conference on Learning Representations, 2024. URL htttt:t:s::::/eeetetttt
net/ forum? 1
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models, 2023. URL https: //apvva...vivvv/aa http:///ggggab///////a// a2t/aa//.///..2 /
10625.
Tianyi Zhou, Deqing Fu, Vatsal Sharan, and Robin Jia. Pre-trained large language models use fourier features to compute addition. In NewrIPS, 2024.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
Experimental Details and Additional Experiments for Section
16
A.1 Details on experimental setup and techniques
16
A.2 Additional details on experiments with the Geography 2-hop ICL puzzles
18
A.3
Multi-hop mechanism formation and the number of in-context examples
27
A.4 Preliminary Evidence of Causal Transferability of Concept Embeddings from ICL puzzles to Naturalistic Contexts
28
A.5 The Company Puzzles, Another Multi-Hop ICI. Task
34
Experimental Details and Additional Experiments for Section
36
B.1 More on the add-k Problem Problem......
B.2 More on the Circular- Crreclar-Traecctrry Problem
36
B.3 More on the Rectangular Dectangular-Trajecooryy Problem
37
Further Discussions
38
Use Large Language Models (LLMs)
38
A EXPERIMENTAL DETAILS AND ADDITIONAL EXPERIMENTS FOR SECTION 2
A.1 DETAILS ON EXPERIMENTAL SETUP AND TECHNIQUES
Causal mediation analysis. We primarily rely on causal mediation analysis (CMA), a.k.a. activation patching in the mechanistic interpretability literature, to obtain causal evidence for our claims in the LLM studies.
M
Indirect effect
Y
At a high level, CMA is about the study of indirect effects (IE) and direct effects (DE) in system with causal relations (Pearl, 2022). Consider the following classical diagram of CMA, in Figure 11.
Direct effect
Figure 11: Basic illustration of CMA. X input (exposure), M mediator, Y output (outcome).
Suppose we wish to understand whether a certain mediator M plays important role in the causal path from the input X the outcome Y. We decompose the "total effect" of X on Y into the sum of direct and indirect effects (DEs and IEs), as shown in the figure. The indirect effect measures how important role the mediator M plays in the causal path X + Y. To measure it, we compute Y given X, except that we artificially hold M's output to "corrupted" version, which is obtained by computing M on counterfactual ("corrupted") version of the input. A significant change in Y indicates strong IE, which implies that M is important
causal path. the other hand, a weak IE implies a strong DE, meaning that the mediator does not play strong causal role in the system (for the distribution of inputs of interest).
There are two common classes interventions mechanistic interpretability for localizing model components with strong IE in the causal graph. The first class is simple ablation, such as mean ablation (replace activation of the mediator by its average output on a distribution interest) (Wang al., 2023) "noising" (Meng al., 2022). While this type of intervention is easy perform, typically leads to poor localization, surfacing low-level processing components irrelevant to the study (Zhang & Nanda, 2024).
The other class, which we employ, is "interchange" intervention: it requires construction of alterative prompts which differ from the normal prompt in subtle ways, requiring careful consideration of
16

---

Published as a conference paper at ICLR 2026
the problem's 's nature, but allows "causal surgery" which surfaces model components with specific functional roles. Technically speaking, we are measuring the natural indirect effects of the mediator. In particular, it works follows. We first run the system (the LLM) both normal and alternative (or sometimes called counterfactual) inputs, and cache the output of the mediator M. We then hold M's output its alternative version, Everything downstream in the causal graph from M are also influenced, up to the output Y. This helps us measure how the mediator M causally implicate the answer. Or more intuitively, it measures how "flipping" the output of M causally influences the LLM's "belief" in the alternative answer over the normal answer.
the full system (the LLM) on the normal prompt.
What makes our intervention experiments somewhat novel lies in exactly how we measure the IE. In particular, as we briefly discussed Section 2.1 and 2.2, we do not directly use the alternative prompt's ground truth answer to measure how well we are "bending" the model's "belief" through intervention. We discuss our method in greater detail here.
First, to understand whether certain attention heads have functional roles in processing the query source entity S, which transcend source-target types of the two-hop problems (i.e. it effectively serves as the hidden concept map C' as introduced in $1.1), we work with normal-alternative prompts with distinct source and target types, such as sampling an alternative prompt "EPFL, 41. University of Tokyo," ([Universityy, Code] problem), and a normal prompt "Okinawa, Tokyo. ... Chicago, ([City, Capital] problem). We hypothesize that there are certain model components which output the bridge concept, which is then composed with the target/output concept of the problem. For the normal exam- ple, this means the hidden step "Chicago" ""US"" 4 is resolved first (i.e. C(Chicago) = USA here), then the model executes Capital(USA) Washington D.C.) This means that, patching a model component's 's activation from the alternative prompt onto its activation on a normal prompt, would cause the model to favor the answer of the alternative prompt, but with the same target semantic type as the normal prompt (i.e. the map R(: Output Type) remains identical its output type across the prompt pairs, but the output of C(-) changes). In our running example, this would be "Tokyo", the capital of "Japan", the country (bridge) of the university "University of Tokyo".
Washington D.C. as the output (i.e. R RUSA; Capital)
It follows that, to evaluate the "causal effects" of such bridge-resolving component, we should set TT(al)) Typenorm(I (II(. We then measure the (expected) intervened logit difference
Dat = E logit
+norm In-oomm
(Poon)) [T (noom) logit all - (Poom) ) [[]((a)t]]
(I)
where Pnorm { : logits of the model obtained after intervention while running the model the normal prompt, and Logit(Pnom) indicates the logits of the model running naturally (un-intervened) on the normal prompt.
((orm) -T( Sh ((momm) .] is the normal prompt, logit alt noomn (Prmm) indicates the
Moreover, when measure the rank of the model's answer when intervened, we also use F(aat) as
the target.
Remark. To normalize our logit-difference variations, we compute
parm anomm mnom
(2)
A
porm
where
A Apor
E. logit((P Pnorm) [T - logit(Pnonna [T((
Problem settings and overall observations. We primarily work with the Geography and Company 2-hop ICL problems to, and with the Gemma-2 L.M family. The former problem setting was described in the main text (with further elaboration the Appendix later), while the latter will be introduced later, to add further generality to our study.
At a high level, in both the Geography and Company 2-hop ICL problems, we obtained causal and correlational evidence that highly localized groups of attention heads output the representation of the "bridge" concept based on the query, and the model later utilizes such representation to produce the output. Furthermore, an interesting technical observation is that multiplying the patched representation at these attention heads with a constant slightly above (c.g. 1.5 to 4.0) tend to improve the intervention results. Finally, make the observation that smaller LLM, namely
17

---

Published as a conference paper at ICLR 2026
Gemma-2-22 also possesses attention heads which have nontrivial causal role in inferring the bridge representations. However, the representations are poorly disentangled, potentially leading to the model's low accuracy the ICL problems.
Compute. Our LLM experiments are conducted on 2 H200 GPUs, totaling around 200 hours of compute.
Licenses. We use the pretrained Gemma 2 models for our LLM experiments. They have the following license information.
Models: Gemma-2-277 2-BBB and Gemma-2-2B (google/gemma-22276 and googlo/gemma-2-25 on Huggingface)
License: Gemma Terms of Use (Google, 21 Feb 2024)
Link: http://ai.googlleeeeeeemmaaaani
Notes: Commercial use permitted subject to Gemma Prohibited-Use Policy.
A.2 ADDITIONAL DETAILS ON EXPERIMENTS WITH THE GEOGRAPHIY 2-110P ICL PUZZLES
Dataset construction. There are two stages to our data sampling process:
Step 1: JSON dictionary sampling. First, we ask ChatGPT o3 to create a JSON dictionary. mapping each country to its cities, capital, and calling code, and repeating this for universities, and famous landmarks. We then manually correct for and refine the entries in the JSON dictionaries to reduce leakage of source types. For instance. ChatGPT sometimes append city or state/provincerrrgion to a landmark, which we remove to ensure that the Landmark source type remains sufficiently distinct from the City source type. University names sometimes cannot avoid such overlap, e.g. University of California, Berkeley indeed has city name it.
The 2-hop Company dataset is constructed in almost exactly the same fashion, mapping the companies to their products, founders and headquarter cities.
Step 2: Prompt generation. This is discussed in Section 2. Taking the geography puzzles the running example, for every City->apitta prompt X = [City,; Capital ... City, na J. we sample tuples (City,, Country, Capital,), ...... , ni, where each "bridge" B, Country, is different country and the City, and Capital, belong to that country, all randomly sampled from the JSON dictionary from before. The same holds for other source-target types, and the Company dataset.
Causal evidence. Recall that in the main text, to provide causal evidence for the bridge-resolving mechanism, we primarily presented causal intervention experiments where we treated [City, Capital] as the problem type we intervene on, using cross-type prompts [University, Calling Code]. [Landmark, Calling Code] to show causal evidence for the bridge-resolving heads. Here, we add further evidence by having other source-target types. The results are presented in Figures 12 to 21, indexed as follows:
1. Experiment [City, Capinal)- Calling Code]: Figure 12
2. Experiment [University, Capital)- Calling Code]: Figure 13
3. Experiment [City, Capital)- Calling Code]: Figure
4. Experiment [Landmark, Capital) Calling Code]: Figure 15
5. Experiment [University, Capital) Calling Code]: Figure 16
6. Experiment [Landmark,
Code]: Figure 17
7. Experiment [City, Calling Codee-=UUnivvrrsityy Capital] Figure 18
8. Experiment [Landmark, Calling Codee--UUniversityy Capital]: Figure 19
9. Experiment [City, Calling Coadd-=LLandnark,, Capital]: Figure 20
10. Experiment [University, Calling Code] (Landar,,, Capital] Figure 21
Every patching experiment is performed on at least 100 prompts. As we can see, the general trend is that there is strong transferability of the bridge representation across the problem types, including
18

---

Published as a conference paper at ICLR 2026
when the source and target types have overlap, giving us causal evidence that head groups (24,30:31), (35,22;23) are "resolving the bridge".
Scaling constant for bridge intervention. We found that for some of the transfer experiments, multiplying the patched representation for the heads (24,30;31), (35,22;23) improves the result, i.c. there is greater percentage of samples where the alternative answer is boosted into the top-10 (or even top-1) ) answers of the model after intervention. Therefore, we also report those results. An intriguing property of this scaling constant is that it typically works best around 2.0. At 4.0, we often observe saluration or even decline in the intervened alternative answer's rank, such as in the [University, Capital]=+[CSt
Calling Code] experiment shown Figure 16.
The output-concept heads. While the main interest of this work lies in the bridge-resolving mechanism enabled by the sparse set of attention heads discussed above, we also present more analysis of the output-concept heads, whose embedding tends to cluster with respect to the output concept (Capital versus Calling Code). To localize these heads, we generate we only change the target/output type of the normal prompt to generate the alternative prompt, but keep the S,'s 's to be identical across the prompt pairs for all S n. This helps us surface components which are independent of the query and bridge value, and sensitive to the output/target type for the ICL problem. The results are shown in Figure 22 and 23, where we run the intervention experiment [Landmark, Calling [oade]==Landdmar, Capital] (due to limitations in time and computing resources, we could not sweep all the source-targe -taret combinations as of this version of the paper). As we can see, these head groups with the strongest causal scores indeed tend exhibit sensitivity to output type, and insensitivity to source/input type and query and bridge value. Moreover, they are more concentrated in the deeper layers of the model.
prompt pairs where
Statistics of alternative-type answers. A natural question to challenge the bridge-resole mecha- nism is as follows. Say we are performing intervention by sampling alternative prompts from the problem type [University, Calling Code), and normal prompts from the type [City, Capital]: even though the target types have little overlap in text, perhaps the model still assigns nontrivial confidence to the "Capital" version of the alternative answer (which has target type "Calling Code")? If that is so, then it challenges our hypothesis about the role of the "bridge" representations, since we might just be directly injecting the right version of the alternative answer into the model.
We show evidence to refute this. In particular, in Figure 24, we show that the model places trivial confidence on the altered-type type answer, even if they share the same bridge value. Therefore, we add further evidence to the bridge-resolving mechanism.
The role of MLPs. We performed similar intervention experiments the MLPs at the last token position just like with the attention heads. They are observed to primarily process the output concept type, and do not participate heavily in outputting the bridge concept representation. This is revealed in Figure 25.
Smaller LLM exhibits weaker disentanglement. To contrast against our results for the 27B model, we study a small model in the same family of Gemma 2 models, Gemma-2-2B. This smaller model has significantly lower accuracies on the problems, measured 20 shots. [City. Capital]: 71.67%. [University, Capital]: 25.83%, [Landmark, Capital]: 66.67% [City, Calling Code]: 47.5%, [University, Calling Code]: 57.5%, [Landmark, Calling Code] 23.33%.
We perform an intervention experiment [University, Codet-+[iit, Capital] on Gemma-2 -2-2B, similar to the bridge-resolving head localization experiments we did on the 27B model. Intriguingly, we were also able to surface highly sparse set of attention heads which have nontrivial causal scores (but much lower than that achieved by the 27B model). We find that these heads exhibit noticeable, but noisy disentanglement with respect to the bridge representation. We show these results in Figure 26. The weaker causal score of the bridge-resolving heads and their noisier concept disentanglement in the 2B model suggests the conjecture that, the larger the model, the more specialized its concept- processing components are - assuming that the model is well-trained. Such specialization likely benefits the model's generalization accuracy.
19

---

Published as a conference paper at ICLR 2026
[Cay. Capitall [Landmers, Code)
CCyy Caphall [Landmark, Code]
Al unons after potchetg
crans shorr patching before polcling
30%
120
before posching
Noroott ogg cff vordtion
200
5.0
0.4
0-4
0.
0.6
10
0.000
0.4
0-4
Ub
64
10
(b)( x1.0 intervention
(b)ii) x1.5 intervention
Capital) [Landmars, Code)
[iyy Caphial] [Landmark, Code]
Atl uroner redo. rurt enower
30%
All cronse reci: rark. arooee paldhing
(12. (22.30031
0022220
(25266271
(4.3033) (2522:5))
potching
20%%
(a) Logit difference variation
Recbbroca lank
Rec prace rant
(b)( x2.0 Intervention
(b)lt) x4.0 Intervention
Code] transfer experiments, intervening head groups (24, 30; 31), (35, 22; 23). We show the percentage logit-difference variation in (a), and reciprocal rank of the answer answer before and after intervention in the (b) series of figures, with different scaling constants in { {1.0, 1.5, 2.0, 4.0}. We observe strong causal effects of the two attention head groups. Here, the scaling constant does not significantly affect the intervention performance.
Figure 12: Demma--2-27H [City, [apital]-[[Landdil
[University Capital mLandmark, Codo)
[Lnivershy, [Landmark, Codo]
snawer recip runk. ofter
zrower rocio. rork. after
120
33%4
before parciing
man..
112%
eenaer mvis ook,
120%
3/2
54
0.5
0.3
02
Pecpetes
(b)i) x1.0 intervention
(b)ii x1.5 interyartion
Nocced
[Univorsity, Capttal [Landmark, Code!
[Uitiersity. Capital) [Landmark, Codo)
30%
arovne rede nank. afher
rocis oftrr polcking
belore
aroive r8d0 mark.
207
arwwer reci. Sarbrr patcring
10200
oo 1199
04.330
05.22.31
105
(x) Logi varidiinn
(bj(tt)) x2.0 intervention
(b)(v) x1.0 Intervertion
Figure 13: Gemma-2-270 [University, Capital]=[Landmaall Code] transfer experiments, intervening head groups (24, 30; 31), (35, 22; 23). We show the percentage logit-difference variation in (a), and reciprocal rank of the answer answer before and after intervention in the (b) series of figures, with different scaling constants in (1.0, 1.5, 2.0, 4. 4.0). We observe strong causal effects of the two head groups. Interesting, past a scaling constant of 1.5, we observe decline in intervention performance.
60%
[Cits. Captiall Buwverstyy.comm Code]
ICty, Capitall to EUnteersity Codel
pferr paliting
3400
paichingg
3000
220
(1000) x1.0 Intervention
(b)li)) x1.5 intervantion
8
[Chty, Captral JUnvenity, Codee
(City, Capibal guntveestty Code]
Al--no recp. rank ante
63%
sxavel reda
after
beft's cenciing
ant.we mank
bebre parching
enower
02
(322123)
(a) % Logit viferrnnevvaiiiiio
00
5%
(b)iii x20 intervention
(biljv) inlevveenoiio
Figure 14: Gemma-2-22 2-27B [City, (Unnverstt Code] transfer experiments, intervening head groups (24, 30; 31). (35, 22; 23). We show the percentage logit-difference variation in (a), and reciprocal rank of the answer answer before and after intervention in the (b) series of figures, with different scaling constants in (1.0, 1.5, 2.0, 4.03. We observe strong causal effects of the two attention head groups, boosting the alternative answer into top around 60% of the time! Additionally, the scaling constant does not significantly affect the intervention performance in this experiment.
20

---

Published as a conference paper at ICLR 2026
[Landmrkk, to (Unversity, Code)
[Landmart, Capital] to (University, Code]
40%
At urowa no.9. rent.
Alc prover ma%
arnart scc rens.
Alc srever po rans
bafore
22
Secproca
Recprrcal tate
(b9() x1.0 intervention
(li(1)) x1.5 intervention
Tecort
Bandmark, so (Unversity Code)
[aandmars, Capttel) (Urkeerstyy Codee
22
60%
shor parching
siter pattiing
bekore patch
(58.4)
(a) % Logit defference variation
0.2
114
(bii x2 intervantion
(bj(iv) x6.0 Intervantion
Code] transfer experiments, intervening head groups (24, 30; 31), (35, 22; 23). We show the percentage logit-difference variation in (a), and reciprocal rank of the answer answer before and after intervention in the (b) series of figures, with different scaling constants in (1.0, 1.5, 2.0, 4.0). We observe strong causal effects of the two attention head groups. Here, the positive effects of the scaling constant saturates around 2.0.
Figure 15: Gemma-2-27B [Landmark, Capital]
55%
pattiig
100
Beceer ligg
(b0i) x1.0 Interveatonn
(b)(i) x1.5 Intervention
Ulrbeeityy to (City, Cock)
[Untverdity Capitell ECity Code)
47%
22,0033
(2220.33)
% Logit differennee variation
(b)(ii) x2.0 intervention
(b)(iv) x4.0 intervertion
Figure 16: 2-2ma [University, Capital] Code] transfer experiments, intervening head groups (24, 30; 31,, (35, 22; 23). We show the percentage logit-difference variation in (a), and reciprocal rank of the answer answer before and after intervention in the (b) series of figures, with different scaling constants in {1.0, 1.5, 2.0, 4.0}. Interestingly, we observe decline in the intervention's s accuracy as we push the scaling constant from 2.0 to 4.0 (top-1 accuracy decreases from around 50% to slightly above 40%), indicating subtle regime in which the scaling constant boosts intervention performance.
Dandmark, to 10ly, Godds
Jassmerk,
JanseerkkCuakaaaa Caskal [Cly Code)
of ENE TXA
paiting
2305 Zam
Prooet logst off orie21
(b00) x1.0 intervention
(bi(i) x1.5 intervention
Banddarkk Capital] to Joly, Code]
Jancmmrrk, Caplaal to (CiN, Codk)
49%
47%
02:40
patching
Ervs
Sekure
vower inkos
% Logit difference vartation
185
(ba(3 iinbavaarionn
(biliv) x4.0 intervatvionio
Figure 17: Gemma-2-278 [Landmark, Capital]->[City, Code] transfer experiments, intervening head groups (24, 30;31), (35, 22; 23). We show the percentage logit-difference variation in (a), and reciprocal rank of the answer answer before and after intervention in the (b) series of figures, with different scaling constants in (1.0, 1.5, 2.0, 4.0). We observe strong causal effects the two attention head groups
21

---

Published as a conference paper at ICLR 2026
[Cty, Sooel to [Unteersiyy, Capitll
ICRty: Codel EUriverstyy Capital
30%
Poontingt
10
35
10
32
101
10
(b))) xl intervention
(b)(T) x1.5 intervention
[City, Code] to [Uiiversity, Caphall
[CHt,, Code) to EUriversity Capital)
30%
02.00.33
beftre palsting
Eofore poldiing nn
10%
(a) % Logit difference variation
10
10"
101
07
(b) x2.0 intervartion
(3KVV)) xáO intervention
Figure 18: Gemma-2-27B [City. Calling Sodde---Unveerstty Capital] transfer experiments, inter- vening head groups (24, 30; 31), (35, 22; 23). At scaling constant 1.0 (i.e. natural intervention, additional scaling), the reciprocal rank of 0.1 for the alternative answer after intervention is at the 36th percentile, while before patching, as we can see, the reciprocal rank of the alternative answer is mainly in the range of 10-2 to 10
[Landdmark, (Uriversity
[University,
6A UrCINI IoUI runk.
All anowe rous 1ark Al. enower redp rark
Ait aetver repp mez.
200
60
10
30
35
10
35
55
10
10
(b)( x1.0 intervertion
(blli) x1.5 intervention
32%
(Uriversity
[Landmark,, (University,
16
6l. orswe rech 1ank.
All anower roco 1aRk
Alc antwee recp mes.
Al. anower redp 1akk
(3521:201
06.849
0522220
52%
105
(a) % Logit difference vaniation
10
(b0ii) x2.0 intoventiion
(biliv) x4.0 intervention
Figure 19: Gemma-2 2-27BB [Landmark, Calling Code)=*(Universin) Capital] transfer experiments, intervening head groups (24, 30; 31), (35, 22; 23). At scaling constant 1.0 (i.e. natural intervention, no additional scaling), the reciprocal rank of 0.1 for the alternative answer after intervention at the 41 th percentile
Code [Landmark, Caphall
Code] [Landmark, Captall
237%
arsue redo feline sevinin
ahat patcting
Sam
waee redo. feture panitan)
tba paicking
128
230%
100%
12%
20
becaric mi
0n
tadpnaca bank
(b000) x1.0 intervantion
(b)i0) x1.5 intevaaiio
[Ciyy Code to [Landmart, Capibal
(City. Codee] tn LLandmarr, Captall
43%
ofter patching
shot paishing
09.12.23
30010
(292807)
05222200
beftee pasitiing
before paic2ing
(a) % differeee variation
10%
10e
10
(b)(ii) intervention
(bofiv) x4.0 interverttiiin
Figure 20: Gemma-2-27B [City, Calling vening the single head group (24, 31). At scaling constant 1.0 (i.e. natural intervention, no additional scaling), the reciprocal rank of 0.1 for the alternative answer after intervention is the thth percentile.
Capital] transfer experiments, inter-
22

---

Published as a conference paper at ICLR 2026
UUntversty, [Landmar, Captbal
[Lansmark, Cantbel
ottrr petching
sher patching
woee red0. befor panciing
befiin pattiiing
hggh ditvenee
200
10%
Recppocal
Redpnoci iam
70
(b)t) x1.0 Intervention
(blii) x1.5 intevvaion
[University, Codee] [Landmark, Capita)
[University Cocee [Lanomark, Capial)
redo.
02:12:41
04 30.13
05.2220
paldting
patcciing
(a) % Logit difference vaniation
10
307
101
25"
30
107
(b) x20 intevvaninn
(bj(i)) x1. intervantion
Figure 21: Gemma-2-227B [University, Calling Code)-<(aandmark, Capital] transfer experiments, intervening the single head group (24, 30; 31). At scaling constant 1.0 (i.e. natural intervention, no additional scaling), the reciprocal rank of 0.1 for the alternative answer after intervention is the 31st percentile.
(23, 262 7l: 10.0%
(38 10:11::1: 18.5%
141.84) 9.6%
Cly Captal
<hy capra
Inf. Capta
Landmar:
09
ciy
City
Uni, Coce
No
Landwark
(43.5:)
(852627)4888
Chty captial
dit Capial
Untu
Uria Caar
0
Landmarc
City
Landrodk Coce
Figure 22: (Best viewed zoomed in) Cosine similarity map of the output-concept head groups (with top causal scores) identified in Gemma-2-27B. along with the percent logit-difference variation of the head groups, serving as the metric for the head groups' causal effects. Observe that they are mostly insensitive to the souree type, query value, and bridge value, and primarily sensitive to the output/target type. Note: in this set of visualizations, we are using "Italy" and "Spain" as the bridge values.
23

---

Published as a conference paper at ICLR 2026
(23.26:27): 10.9%
(36.10:11): 18.8%
(41,8;9): 9.6%
Capital
Code
0.9
Rand-Cap
0.8
Rand-Code
(41,18:19): 6.7%
(43,4:5): 5.9%
(45,26:27): 6.8%
0.7
Capital
Code
Rand- Cap
0.5
Rand-Code
Cosine similarity map of the output-coneept head groups identified in Gemma-2-27B. Here, we construct four groups of prompts. The first two groups consist of multi-hop ICL problems with Capital or Calling Code as the target type. The remaining two are created by randomly shuffling the output of the normal multi-hop ICL samples, causing the problem to essentially demand randomly outputting Capital or Calling Code; these are the "negative controls" we discussed in the main text. We find the output-concept heads' embeddings on the multi-hop prompts to align strongly with those on the output-concept-only prompts, further confirming their role in the circuit.
Figure 23:
City,
University. Code
Landmark, Code
100
80
Correcbellass answer
Corract-dlass answer
Correct-class answer
Incorret-claas answer
80
Incorrect dass answer
Incorrect dass answer
60
Percentoge
Percanbage
60
40
00
20
20.
10
10
Confidence
Confidence
Confidence
City, Capital
University, Capital
Landmark, Capital
50
50
Correctcllass answer
Corrac-classs anower
Correct-class answer
Incorrect-llass answer
40
Incorrect-class answer
40
incorrectciiass answer
Peroertage
Percertage
counaage
50.
20
20-
10
10
10
10
10
10
10
10
10
10
10 Confidence
10
10
Confidence
Confidence
Figure 24: Distribution of Gemma-2-27B's confidence on the correct- and incorrect-type answer on the different problems. When we say "correct-class" answer, we simply mean that the answer's semantic type aligns with that of the problem's 's target, e.g. the correct-type answer for a prompt "Okinawa, Tokyo. Nantes, Paris. Shanghai, " would be "Beijing", while the incorrect-type answer would be "86" (the calling code of China, which the city Shanghai belongs to). We observe clear separation in the LLM's confidence between the two types of answers.
24

---

Published as a conference paper at ICLR 2026
MLP-25
MLP
MLP-31
MLP32
MIP3
Layer Irdes
Figure 25: Functional roles of the MLPs in Gemma-2-278 We perform patching experiments on (University, (Cit)->(City Capital] at the last token position, similar to how localize the bridge- resolving attention heads. We report the percentage logit-difference variation of the top-scoring ML.s, along with their cosine similarity maps computed on prompts sampled with different combinations of bridge values ("Italy" and "Spain") ") and diverse set of source-target types. Perhaps unsurprisingly, the MLPs at the last token position play less interesting role: as seen in the cosine similarity maps for the MLPs with the highest causal scores (fairly low compared to the attention heads), they primarily discriminate against the output type. They do not appear to participate much in resolving the bridge concept, as indicated by their lower causal scores, and the lack of sensitivity to the bridge entities in the cosine-similarity plots.
25

---

Published as a conference paper at ICLR 2026
10
40
City Capta
2
Ctyy Captaa
hay
4
30
Unt,. Captia
Landnn's Captia
Suat
6
20
Cool
0.5
Ckyy Cede
Isl
0.5
10
Spat
11
Lordnark Codo
Spain
0
13
20
50
Cty. Raly
15
-10
Chty Cepiial
16
Itay
0.8
Captal
Span
18
-20
Lar dran Capital
20
Lardmwrk Captal
Itly
-30
22
Chy, Code
Iely
23
Unlo
Spat
0.2
40
25
Landna Codo
Landmant
Head Index
20
Figure 26: Results of the intervention experiment [University, Code] Capital), conducted on Gemma-2-2B, with 20-shot ICI. On the left, we show the percentage logit difference variation of the intervention experiment; on the right, we plot the cosine similarity map of the two head groups with the highest causal scores, namely (15, 4; 5) and (22, 0; 1). We find that while the two attention head groups exhibit nontrivial causal effects and disentanglement, they are, comparison, much weaker than those exhibited by the 27B model. This likely explains the significantly lower accuracy of the 2B model than the 27B model. This also suggests a conjecture: perhaps the larger the model, the more specialized its concept concept-processing components are?
26

---

Published as a conference paper at ICLR 2026
briig
acwan
(a)() 2-shot
(a)(1) 4-shot
(b)() Patching experiment
(c(() 2-shot
(eXiii 4-shot
125
14
3.33 655 0.7 3tt
030
1.0
(aXii)) 8-sbot
(aXiv) 20-shot
(bX(i) Patching experiment
(c)(ii) 8-ahot
(eXiv) 20-sbol
Figure 27: This figure illustrates how the transferability and disentanglement of the bridge representa- tion increases as we increase the number of in-context examples. Figure series (a) and (b) present the transferability result, obtained by performing cross cooss-pobleemtypp patching, and measuring the causal influence the patched representation. (a)() to (iv), we plot the percentage logit variation of the attention heads found to output "bridge" values, measured several intervention experiments. For (b) and (ii), we zoom in on head group (24,30:31), and show its causal effects on two patching experiments, [University, Calling Codt)-+(City, Capital] and [Landmark, Calling Code]->[City, Capital]. The x-axis is the number of in-context examples, and the y-axis is the 30th percentile of the reciprocal rank of the alternative prompt's ' answer. For (c)(i) to (iv), we plot the histogram of the disentanglement strength of the representations of head group (24,30:31), 1). with the y-axis in percentage, and x-axis being cosine similarity.
A.3 MULTI-HOP MECHANISM FORMATION AND THE NUMBER OF IN-CONTEXT EXAMPLES
This sub-section focuses on illustrating the relation between the number of in-context examples versus (1) how strong role the multi-hop mechanism plays in the LLM's inference (via causal interventions), (2) disentanglement strength of key bridge-resolving attention heads. As we will show below, there is general positive correlation between the number of shots and the two factors.
More demonstrations = stronger causal score. Figure 27 visualizes the experimental results. From Figure (7( and (b) and sub-figures, we observe a correlation between the number of shots (ICL examples) and the bridge-resolving heads' "causal importance" in the model's inference. When number shots low, we find that they tend to exhibit weak causal influence on the model's inference. For instance, (b)() shows, at 2 shots, the 30th percentile of the alternative answer's rank after patching at (24,30:31) ) is on the order of 103. This is in stark contrast to how strong this head group's 's causal influence 20 shots
before.
More demonstrations = stronger disentanglement, with a catch. In Figure 27(c) to (iv), we observe that the intra-bridge cosine similarity tends cluster better as the number of shots increase, while the inter-bridge cosine similarities decay toward 0.2, with the two distributions overlapping less and less. Interestingly, the mrige-disentanglemmen strength is still non-trivial with very few shots, mirroring the causal-ine cansal-ittervention results: regardless of how disentangled the representations are in the very-few-sho regime, the LLM does not "realize" how it should utilize the multi-hop sub-cireuit.
27

---

Published as a conference paper at ICLR 2026
A.4 PRELIMINARY EVIDENCE OF CAUSAL TRANSFERABILITY OF CONCEPT EMBEDDINGS FROM ICL PUZZLES TO NATURALISTIC CONTEXTS
This subsection was added in the Rebuttal & Discussion Phase.
A natural question regarding the concept embeddings found in the synthetic ICI,-puzll setting stands: Are these "concept embeddings" specific to those strictly formatted puzzle-like tasks, or are they reflect mechanisms that transfer to more naturalistic settings? Are these embeddings merely artifacts of that artificial setup?
While fully resolving this question is beyond the scope of this work, conduct a small-scale study that provide preliminary but positive evidence that, the concept embeddings found in the ICL-puzzle setting are not mere artifacts of the artificial setup: they exhibit causal influence generation in more naturalistic prompts.
Experimental setup.
Naturalistic context prompis. We continue to work with Gemma-2-27B. We extract "Country" concept embeddings solely from our synthetic ICL puzzles (as described in Section 2), using only the City Capital source-target type. To test their causal transferability to naturalistic settings, we inject these vectors attention layer 24 (as localized in the ICI. setting) when the model is given open-ended, natural-sentence prompts.
The natural prompts still controlled, and constructed follows:
A short "hint" sentence that implicitly describes a country, e.g., "From cherry blossoms to temples, everything here is unique." (Hidden concept = Japan)
A "Type" sentence, with Type E (Universtiies, Celebrities, Tourist Spots), e.g., "I also know of some celebrities there, such as" (Type = Celebrities)
An example prompt is:
From cherry blossoms to temples, everything here is unique. I also visited some great tourist spots there, such as
We focus on small set of target countries (corresponding to the concept embeddings which we inject into the model) with strong presence on the internet (US, UK, China, South Korea, India) for the sake of tractability.
Causal transfer experiment. We perform the following causal transfer experiment. First, we extract the Country concept embeddings from the ICL City-+Capital setting, by averaging Attention Layer 24's activation at the last token position (with the query belonging to a chosen country) over 50 prompts, with 16 in-context examples for each prompt. We denote a Country concept embedding as Coountry. Then, as we run Gemma-2-22 2-2BB on the naturalistic context prompt, at every token position (including the already generated token positions), for Attention Layer 24 activation, perform the following intervention
a==24 + 0.5 a;=24
Ceountry
where 7 denotes the scaling constant, which is 3.0 in our experiments this scaling strength was observed to work well in the majority of our ICL-puzzle experiments, so we did not perform ablations on it, considering resource limitations. As we rely on the trans formers library (Wolf et al., 2020) for our experiments here, for text generation, we set the temperature to 0.7, and top to 0.9 (we only sample from the smallest set of tokens whose cumulative probability exceeds 0.9), which are fairly standard hyperparameters to use.
We then use an LLM judge (o-t-i)-min)) evaluate whether the intervention causes the continuation to (i) switch from the source country implied by the context to the target country encoded by the concept vector, and (ii) remain specific to the chosen Type. The judge returns a relevance score from I (irrelevant to the target country/Type) to 5 (highly relevant). We then manually correct these scores
'We work with the RMSt-RM attention layer activation in the Gemma 2 model, namely post_attentica on_ _a_ee_aymrrmm in the fomers APT.
28

---

Published as a conference paper at ICLR 2026
(we work with total of around 100 prompts, ensuring tractability of manual inspection). We will elaborate further on our use of gpt-4o-mini later in this subsection.
Results: causal influence in naturalistic prompts.
We obtain the following quantitative results.
Continuations naturally stay in the source country, with average target- country relevance approximately 1.0:
Pre-intervention::
Post-intervention: For a substantial fraction of cases (roughly 70% for Celebrities Type, and around or above 80% for the Tourist Spots and Universities Types), the 27B model's continuations are judged as describing the target country while remaining the specified Type.
Please refer to Figure 28 for the visualized results.
Importantly, the generations remain coherent; the interventions do simply inject country tokens, but produce fluent text that integrates the target concept within the given Type. We present a list of examples in Table 1.
These results suggest that the concept embeddings discovered in the ICL puzzle setting behave as meaningful control directions for country-leve content in more naturalistic text rather than being tied to our synthetic format, and are, at least in our settings, disentangled from superficial linguistic patterns.
29

---

Published as a conference paper at ICLR 2026
Effect Intervention, Universities intervention Scaling Constant: 3.0
Effect of Intervention, Tourist Spots
Intervention Scaling Constant: 3.0
Pre Intervention
Intervention
Post-intervention
80
Post-Interventoo
80
Percentage of Sarmales (%)
160
80
Percentage
60
40
Relevance Score (1-5)
Relevance Score (1-5)
(a) Universities
(b) Tourist Locations
Intervention, Celebrities/WWl known People
Intervention Scaling Constant: 3.0
Pre-intervention
80
Post Intervention
60
Percentage
60
20
Relevance (1-5)
(c) Colebrites/Well-kkoon People
Figure 28: I.I. judge scores (with manual corrections) with and without intervention at Attention Layer 24 of Gemma-2-277h using the concept embeddings we extracted from the ICL. puzzles limited to source-targed type CitiaaCapital. We observe consistent shift toward the injected country in Remma-2-278 text continuation while correctly maintaining the Type from the context prompt, for all three Type E Universities, Tourist Spots, Celebrities/War Well//WWWllllllnnln People).
30

---

Published as a conference paper at ICLR 2026
Table 1: Examples of context prompts, natural and intervened continuations of Gemma-2-27B.
Problem info; Type
Context Prompt
Natural Continuation
Intervened Continuation
Japan to South Korea; Tourist
From cherry blossoms to temples, everything here is unique. I also vis- ited some great tourist spots there, such as From K-pop to technol- ogy, everything here is unique. also visited some great tourist spots there, such as
the Tokyo Tower and the Tokyo Imperial Palace. I also visited the Meiji Shrine, which was beau- tiful and
the Gyeongbokgung Palace, the Namsan Tower, and the N Scoul Tower.
Spots
South Korea to US; Tourist
the Namsan Tower, the Gyeongboggung Palace and the Namsan Park.
Disney World, Univer- sal Studios, and the Statue of Liberty. The United States of Amer- ica is a country
Spots
to South Korea; Tourist Spots
China
loved everything about my trip the high-speed rail, the tea culture, and especially the lanterns. I also visited some great tourist spots there, such as
the Dragons Wall, the Xiamen University campus, and the Huan- dao Road.
the Jeju Olle Trail, Seongsan Sunrise Peak, and Udo Island. really
China to UK: Celebrities
From lanterns tea cul- ture, everything here is unique. also know of some celebrities there, such
Zhang Yimou and Gong Li. I love the food there, like hot pot and dumplings.
George Harrison, who visited this place. It was the time when The Bea- tles were rising to star- dom
India to China; Celebrities
The cricket was amaz- ing, and you can't find rickshaws like this any- where else. I also know of some celebri- ties there, such as
young man named Asif. I'm happy to see him now. A very good
the great Zhang Guorong, who used to live here.
The most famous restau- rant in the area
Canada to In- dia; Celebri- ties
From poutine to moose, everything here is unique. also know of some celebrities there, such as
the famous singer, Ce- line Dion, who was born and raised in Montreal. If you are looking for
Amitabh Bachchan and Salman Khan. <h2>Is India a good country visit?</h2>
France to US:
I loved everything about my trip the croissants, the Eiffel Tower, and es- pecially the baguette. I also visited some univer- sities there, such as The Bollywood was amazing, and you can't find rickshaws like this anywhere else. I also visited some universities there, such as
Sciences Po and the Paris Institute of Politi- cal Studies.
Harvard and Yale. Just kidding. I'm not going to write about France. I'm
Universities
I was really impressed by the quality of educa- tion
India to UK;
the one where the Bolly- wood movie "3 Idiots" was filmed.
Cambridge University. was a very different experience.
Universities
I've heard
It's a bit like scene from
US to China: Universities
I loved everything about my trip the bald eagle, the Hollywood, and es- pecially the fast food. also visited some univer- sities there, such as
UCLA and UC Berke-
Beijing University, Ts- inghua University, and others. got to see the Chinese people's
ley. I had great time with my friends and the people there were so friendly
31

---

Published as a conference paper at ICLR 2026
Scaling (model size and disentanglement), We also repeated the same intervention experiments on Gemma-2-2B (intervening at layer 15, which showed weaker but non-trivial causal scores in our ICL experiments before; Figure 26). As shown in Figure 29, the fraction of post-intervention continuations receiving score 5 from the judge roughly 1.5 to 2 times lower for the 2B model than for the 27B model. While still preliminary, this is consistent with our ICI-setting findings and supports the interpretation that concept disentanglement and transfcrability improve with model scale.
Gemma-2-28 Intervention,
Intervention Scaling Constant: Spots
Gemma-2-28 Intervention. Intervention Scaling Constant:
Intervention
Pre.intervention Post-intervention
70
Post-intervendion
60
Percentage of Samples
60
40-
40
10
Relevance Score (1-5)
Relevance Score (1-5)
Tourist Spots
(b) Universilies
Gemma-2-28 Effect of Intervention, Celebrities/Well-knnn People
Intervention Scaling Constant: 3.0
In-eintarveniinn
Dost-intervention
60-
50
30
10
Relevance Score (1-5)
(c) ColebritiesWWell-koown People
Figure 29: LLM judge scores (with manual corrections) with and without intervention Attention Layer 15 of Gemma-2-22. using the concept embeddings we extracted from the ICL puzzles limited source-tare target type Citi-aCapital. We observe much weaker shift toward the injected country the text continuation of the context prompt, compared to the 27B's 's's results.
32

---

Published as a conference paper at ICLR 2026
LLM as judge (GPT--4-mini)) We use the following prompt to query GPT-4o-mini (OpenAI, 2024) to score the LLM's text generation.
Analyze the following text and determine if it relates to the [target_type] of [target_country] ).
TEXT: "[text)"
INSTRUCTIONS:
Identify any specific entities (people, places, landmarks) in the text.
For each geographic location, explicitly state their real country of origin/location
3. For people, explicitly state their real country
countries of origin/location
4. Compare these locations to the target country: [target_ J.
5. Based ONLY this comparison, assign relevance score (1-5).
SCORING CRITERIA:
1: No relation or talks about a different country.
2-3: Vague connection or generic sentences.
4-5: Clearly mentions about the [target_type) of (targett_conntry }.
FORMAT:
Reasoning: (Your step-by-step analysis here]
Score: [Integer only
where text refers the model's generation, target_typpp refers Type {Universities, Tourist Spots, Celebrities well-known people]. and target country refers to the country which the injected concept embedding corresponds to.
Additionally, we found that without allowing for some waapiinncchainof-thhoggeee token, GPT-4o- mini tends to hallucinate in classifying the model's continuations, especially for the Celebrities type. We set 150 max tokens of chain-of th-houtf- which was observed to significantly reduce hallucinations.
33

---

Published as a conference paper at ICLR 2026
A.5 THE COMPANY PUZZLES, ANOTHER MULTI-HOP ICL TASK
Problem setup. To complement our study on the Geography puzzles and for more generality of our conclusion that sufficiently capable LLMs solve simple 2-hop ICL problems by resolving the bridge entities first, we work with another set of problems on globally well-known companies in the world. We collect data on 36 of them.
Here, our evidence is provided on narrower range of source and target entity types, due to the nature of the problem. In particular, we allow source entities fall in {Products, Founders J, and target entities to fall into |Founders, Headquarters City }. Furthermore, to obtain causal evidence that the model is resolving the bridge values from the query before producing the actual answer, we perform patching experiments with [Products, Founders]->[[oon (FFounders,, HQ City] and [Founders, Products] HQ City]. If we again observe that there is high rank of the alternative prompt's converted answer after patching sparse set of attention heads on the final token position, then see significant causal evidence that the model resolves the bridge value during inference.
Some manual cleaning was needed in creating this dataset, on top of ChatGPT sampling: we primarily filter out products which had company names in them, and companies whose products mostly contain the company name. An example category of this would be banks and credit/debit card services, e.g. Visa, Mastercard, HSBC, etc. For companies with multiple headquarters, we choose the most well-known one of them as the correct "Headquarter City" of the company; this sometimes counts an old headquarter of the company as the correct one. Note that this causes us to under-estimate the LLM's accuracy with and without intervention!
Quantitative results. We primarily work with the 16-shot ICL setting due to limited compute. In this setting, the [Product, Founder], [Product, HQ City] and [Founder, HQ City] have accuracies 73.5%, 80.7%, 88.6% respectively. We wish to note that this set of puzzles does rely on significantly more obscure knowledge than the Geography puzzles, so lower accuracies and weaker multi-hop links are expected overall.?
As Figures 30, 31 and show, we again obtain evidence for the bridge-resolving heads. In particular, we find that head groups (22, 2)3), (24, 14; 15), (24, 30; 31), (28, 12; 13). (28, 26; 27), (35, 22; 23) (0.8% of the total number of attention heads) exhibit the strongest causal effects in our experiments, and their output representations exhibit strong disentanglement with respect to the companies.
Founders, Proclct] HQ Cityl
[Founders, Product)-(Product HO City]
lank
Alt
wfwer patcring
paich
200%
enawer recip rank. bulora
enower rant. beruna puliterg
L
20%
Narert logh all aftaumii
10%
.0%
0%
Reciprocal rank
Pecprocal rart
(b)() x1.0 intervention
(b)(i) x1.5 intervention
10
Foounders, P(oduct)-Phodoct HO City]
(Founders, Product)[Prrduce, HD City)
40%
Alt. snower redip 19nn.
40%
Alt. answer recis, rank.
02
04, (2430311
0812239
(82622))
before polching
before patcherg
L digit vatiioninnennnnaaaiinn
10%
-00%
10
10
10
30-1
10P
0%
1
10a
30
10
12
(b) x2.0 Intervention
(bjiv) intervention
Figure 30: [Founder, Product]=[Produce HQ City] transfer experiments, intervening the head groups (22, 2).). (24, 14; 15), (24, 30; (28, 12; 13), (28, 26; 27), (35, 22;23) (0.8% of the total number of attention heads).
We suspect that lower accuracy is not only due to the more obscure entities in this problem, but also because of several ambiguities. For example, (1) Product names are oftentimes simply not presented alongside company names, e.g. "Gatorade" is often not written alongside "PepsiCo"; (2) There can be confusion between company founders and well-known product directors, especially in certain ant/entertainment companies.
34

---

Published as a conference paper at ICLR 2026
[Product, Foundder] -oounde,, City
Product Founder]-[[ounne HQ Cityl
after patching
afer patch ng
arnaer reap
enseer mec. rart.
bulore
proca Tunt
Recirocal runt
(b)(i) intervention
(b)ii) x1.5 intervention
[Product Founder]][[aureee Citl
Product -Founde--Tounder. Cityl
20%
patcheng
30%
potching
(1223) (4.1435
06.30:311
(20.2:2)) 12:2
(8.26-71
before parshing
20%
before patching
digieeeiiffereno variation
10-
10
Eaccprccal
Reciprocal
(b)(i x2.0 intervention
(bi(iv) x4.0 intervention
Figure 31: [Product, [ounder]~([ounnee HQ City] transfer experiments, intervening the head groups (22, 2; 3), (24, 14; 15), (24, 30; 31), (28, 12; 13), (28, 26; 27), (35, 22; 23) (0.8% the total number of attention heads).
proup 1614:15)
srop 24-30-211
grup 26-25-2)
sropp 12
incd reande
Ariand
53
Feaders H0 0rg
Ford vs. Nintendo
gove 0624611
Hewt move 2822221
Nareon awy
Soe
Foande: C%
(b) Sony vs. BMW
A9
Teedenn
(c) Intel vs. Honda
Figure 32: Cosine similarity of representations from the top-4 attention heads for resolving the bridge entity in Company ICL problems. For each combination of Bridge choice and cource-target type, we sample prompts to compare representation similarity on. We notice that the heads exhibit disentanglement with respect the bridge concepts (the companies), but individually, they as capable in clearly resolving the bridge concepts in the Company problems as they do on the casier Geography problems (the latter problem having relevant accessible knowledge many more websites than the former). This likely explains the model's 's lower accuracies and somewhat weaker intervention results the Company problems, compared the Geography problems.
35

---

Published as a conference paper at ICLR 2026
B EXPERIMENTAL DETAILS AND ADDITIONAL EXPERIMENTS FOR SECTION 3
This section includes experimental details and additional experiments for problems with numerical or continuous parameterization considered in Section 3.
Compute. The experiments in this section were performed on an internal cluster with a P100 and a V100 GPU. We list training details such as batch sizes and iterations used for the experiments in their respective sections.
B.1 MORE ON THE ADD-j PROBLEM
In this section, investigate the effect of increasing the number of tasks the results shown in Fig. 8, where we showed that the task vectors for K 4, 8, 16 offsets lie on ID linear manifold. In these settings, the model reached 100% train and test accuracy.
We consider a setting with 32 offsets with gap between the offsets k5+1 h; 1. this setting, the train/test accuracies reach only about 90%, which indicates this is harder problem for the model to solve. Fig. 33 shows 3D and 2D PCA projections of the task vectors. These explain about 96.1% and 84.48% of the variance, respec- tively. We see that the task vectors lie 3D manifold that looks like a (small) helix, with alternating (odd and even) offsets rep- resented in two separate ares of the helix. This is significantly different from the ID linear manifold we observed with smaller number of offsets in Fig. This type of ge- ometry is reminiscent of the observations in Zhou et al. (2024); Kantamneni & Tegmark (2025), which study how pre-trained lan- guage models do addition and find that these
2
1
2-
0
0-
-1
2
-4-2
220
Figure 33: 3D (left) and 2D (right) PCA projections of the task vectors for the add-k problem with 32 offsets. The task vectors lie a (small) helix. From the 2D projection, we see that alternating (odd and even) offsets represented in two separate arcs of the helix.
3D manifold that looks like
models encode numbers as a helix.
Training Details. The models were trained for 1000 iterations using AdamW optimizer with learning rate 0.001 and weight decay 0.01. We use a linear learning-rate schedule with a warm-up phase over the first 10% iterations. The train data contains 5000 sequences, use batch size of 500 for K = 4, 8, 16, and 2000 for = 32.
B.2 MORE ON THE CIRCULAR TRAJECTORY PROBLEM
In this section, include further details about the experimental setup of the Sircular-Trajectte problem, and present some additional results.
0.4
0.2
02
-02
-04
-3.5
0.0
08
-05
0.0
0.5
-0.5
0.0
0.5
Figure 34: 2D projection task vectors for the Circular- Trajoccory problem when the model is trained 32 radius values. The three plots show task vectors for clockwise (CW) trajectories, counterclockwise (CCW) trajectories, and all trajectories considered together, respectively. We observe that CW and CCW circle trajectories represented two manifolds in the 2D space.
In Fig. 10 in the paper, we show the geometry of task vectors for the Circular- Trajectorr problem for sequences where -1, i.e., the trajectories clockwise (CW). In this section look at
36

---

Published as a conference paper at ICLR 2026
the geometry of task vectors for sequences with counterclockwise (CCW) trajectories, well as all sequences considered together. In Fig. 34, we plot the 2D projection of task vectors for CW, CCW and both taken together, respectively. We consider 32 radius values while training. The variance explained by PCs for the three plots is 93.55%, 96.32%, 89. 19.1%, respectively. Darker colors correspond to smaller radius values, and the two colormaps in the third subplot represent CW CCW trajectories. We observe that CW and CCW circle trajectories are represented on two manifolds in the 2D space.
Training Details. In this setting, we sample a new batch of sequences at every iteration. We use batch size 64 and train the model on total of 200 000 sequences. We use Adam optimizer with learning rate 10-4 and weight decay 0.001. We use representations at position | as the task vector following the process mentioned in Section 3.1, averaging over 100 sequences each. These settings remain the same for the experiments in the next section.
B.3 MORE ON THE RECTANGULAR- RAAIELLARRRRRRRTTORRRR TRAIECTORY PROBLEM
problem, param- eterized by two parameters, namely the lengths of the two sides of the rectangle, say (a, b). Specifically, the trajectories contain points on axis-aligned rectangles centered at the origin. Let denote the number of points on each edge the rectangle spaced uniformly. The starting point of the sequence is randomly sampled from one of the points on the right vertical edge the rectangle. The rest of the points obtained by traversing the rectangle CW or CCW, determined by c ---1op or 1. Fig. 35 shows an example sequence.
In this section we consider Rectangular-Trapp
An illustration of a sequence of points for the Rectangular Trajectory problem with e = points per edge, and 15. See text for details. (a, b).
Figure 35:
Similar to Circular- Sircular-rraecctory Trajectory problem, each sequence is obtained by first sampling a and b uniformly between and 4, then sampling the starting point and process. For our experiments, we set e = 5 and n = 15 for this task. The number of tasks K denotes the number of different combinations
-1 and then following the aforementioned
In Fig. 36, we plot the 2D projection of the task vectors obtained for all (a, b) combinations lying on the 2D grid between [1, 4] and 6 E []. Similar to the experiments in Fig. 10, we plot the task vectors for trajectories with c -1 here. We consider K = 32 in this experiment. The first two PCs explain 91.97% variance. We observe that all the task vectors lie 2D manifold.
60
-02
-04
-04
-02
0.0
02
0.4
This setting goes beyond the Trajectry problem and shows that transformers represent task vectors corresponding to the problem parameters (radius for circles and edge lengths for rectangles) in low-dimensional (smooth) manifolds in both cases.
Figure 36: 2D projection of the task vectors obtained for 64 (a, 6) combinations. Fixed color or trans- parency level corresponds to fixed or b, respectively. The task vec- tors lie (smooth) 2D manifold.
37

---

Published as a conference paper at ICLR 2026
C FURTHER DISCUSSIONS
In this section, we discuss potential implications of our discoveries on finer (theoretical) understanding of transformer models.
Multi-hop ICL. Wang et al. (2024) found that small transformers trained from scratch on explicit instruction-base multi-hop problem have significant difficulty learning the bridge concepts and obtain high generalization accuracy. This is somewhat expected: the "hops" on their synthetic knowledge graph are close to permutation functions, and directly learning compositions of permutations is known be hard. Yet, show in work that L.LMs in the wild do utilize hidden bridge-coneept embeddings effectively. A hypothesis explain this is that, the ability learn generalizing representations is not intrinsic the transformer architecture, and gradient descent only discovers compressed representations which transcend instance-specifie information under certain data-structue and model-scale conditions. To some extent, this also suggests that pure spressive-power-eee theoretical results might yield limited conclusions the generalization abilities of transformer models.
parameterization ICL. One of the most intriguing observations in this set of experiments is the order-preserving nature of the task vector manifold in the trained transformer models. Essentially, gradient descent was able to correctly compress the large number of tuples of inputs into sparse points which fall on low-dimensional manifold. and these points follow the order relation of the integers. could interesting to theoretically and empirically understand how such representations arise during training, and why repre- sentation linearity appears to break as task complexity increases beyond threshold (e.g.. the line evolves into helix in the offset ICL experiment).
Continuous-paramtteiiia
D USE OF LARGE LANGUAGE MODELS (LLMs)
Frontier LLMs assisted in the process of curating the Country and Company datasets for our multi-hop ICL experiments discussed in Section 2 and in Appendix A above. As we discussed at the beginning of Appendix A.2, we rely on ChatGPT o3 (available the time of dataset curation) to perform the initial sampling of the countries and companies' source and target entity data. Manual filtering and cleaning of the datasets were then performed.
We did not use LLMs for writing, ideation, or finding related work.
38


Reviews:
[{"review_id": "t4IWTJYr3X", "reviewer": "Reviewer_qgy5", "summary": "The paper analyzes how LLMs represent and combine latent \"concepts\" during ICL. To analyze these interactions, they employ causal mediation analysis to identify which components of the model’s computation mediate the effect of particular input concepts on final predictions.", "questions": "1. How are concept variables defined or selected for mediation?\n2. How stable are the identified mediating paths across random seeds or different model checkpoints?  \n3. Intuitively, can the mediators identified via CMA be related to known attention-head clusters or activation subspaces discovered in prior ICL localization work?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper takes a systematic and well-executed approach to analyzing internal mechanisms of ICL.\n- The approach offers interpretable, fine-grained insight into how contextual information propagates, complementing existing representation-based probing techniques.\n- The study spans synthetic geometric reasoning tasks and natural language problems, demonstrating that the framework generalizes across distinct domains of conceptual structure.  \n- The visual analyses (mediation heatmaps and intervention results) are well presented and help illustrate which attention paths encode specific concepts.  \n- The work contributes methodologically by adapting tools from causal inference to the study of neural mechanisms in LLMs.", "weaknesses": "- The paper treats latent concepts as interpretable dimensions or attention patterns but never defines them formally.  \n- The causal model in the introduction ($F = R \\circ C$) is not linked to the implemented CMA pipeline. $R$ and $C$ are just introduced without definitions. There is no derivation showing that the empirical mediation quantities estimated from activations correspond to components of this decomposition.  \n- Experiments are mostly conducted on synthetic or simplified reasoning datasets (two-hop relations at most).  \n  These provide clarity but limit conclusions about behavior on realistic compositional or linguistic tasks.  \n- The paper occasionally presents correlations in mediation strength as evidence of conceptual disentanglement. In my opinion, stronger validation, like causal interventions or counterfactual ablation, would be needed to corroborate these claims.  \n- No stability analysis or statistical testing is reported. It is unclear how consistent the discovered mediators are across runs or models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "twJq4z8sEw", "reviewer": "Reviewer_4uha", "summary": "This paper systematically studies how large language models represent, disentangle, and utilize latent concepts in in-context learning. The authors use a series of controlled tasks combined with mechanistic interpretability methods to provide both causal and correlational evidence, revealing the internal reasoning and representation mechanisms of Transformer models when solving ICL tasks.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The exploration of how LLMs understand and reason at the conceptual level is interesting and provides valuable insights for the community. The use of causal mediation analysis and PCA visualization to validate the claims is persuasive under the synthetic tasks presented in the paper. The experimental design is rigorous and highly interpretable. In particular, the demonstration of low-dimensional manifold structures in the mode’s representation space (i.e., the geometric interpretability of latent variables) is a novel perspective.", "weaknesses": "The tasks and experimental setups are overly idealized, relying almost entirely on highly synthetic toy tasks, which do not represent real-world natural language reasoning tasks. Of course, this is a common issue in the interpretability field.\n\nThe analysis of model scale and generalization is insufficient. The paper only compares Gemma-2-27B and 2B, without systematically examining whether the same mechanisms hold across different architectures (e.g., LLaMA, Qwen series) or larger-scale models. It is also unclear why the experiments in Section 3 are conducted on a small GPT-2–style model rather than the previously used Gemma models, and what the motivation for this model substitution is.\n\nThe paper provides only empirical analysis. It would be more compelling if any theoretical or mechanistic explanation were offered for why Transformers naturally form linear geometric task vectors or bridge concepts.\n\nAre the bridge concepts unique? The experimental design in the paper is overly simple, but in real-world scenarios, the relationships between concepts are much more complex. With only a few in-context examples, there may exist multiple potential bridge concepts between the source and target, a possibility that the paper does not discuss.\n\nAnother concern is how such concept-level empirical interpretability analyses can provide meaningful value for real-world applications. In other words, when would we actually need concept-level explanations? Even if we know which attention heads encode certain concept representations, how should we intervene or utilize them in practice? Could specific application scenarios be provided as examples?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LX8App6kOg", "reviewer": "Reviewer_9BHQ", "summary": "This paper studies the problem of whether and how transformers represent latent concepts as part of their computation in a an ICL setup.\n\nKey ideas\n\n* Use activation patching (a.k.a Causal Mediation Analysis) to determine if the model is indeed representing the latent concept in a discrete multi-hop setting.\n* For the numerical latent variables, the authors propose the use of linear probing the intermediate model embeddings to assert the existence of task vectors.\n* Use PCA and project the task vectors onto the first two principle components and then study the geometry of the resultant manifolds along with the ordering of the points. Using steering (or interpolation) based methods to show that model captures concept’s geometry.\n\nMain contributions\n\n* Use of causal and correlation techniques to show that for tasks requiring memorized world knowledge, a sparse set of attention heads are responsible for resolving intermediate latent concept, thereby rejecting the hypothesis that the model takes a shortcut (directly maps the input to the corresponding output without any intermediate steps). This shows that the model is indeed performing abstract reasoning (chained reasoning) in its hidden activations.\n* Show how the above phenomenon plays out in case of smaller language models are used (2B, instead of 27B) and larger number of ICL examples.\n* In alignment with the previous work on Linear Representation Hypothesis (LRH), the authors establish the existence of task vectors. In addition to that, the authors show that these representations reflect the geometry of the latent variable.", "questions": "* In the “Memorized world knowledge” thread - While the existing work uses 2-hop reasoning tasks, it would be interesting to study higher order transitive tasks (3 hops or more). It would be interesting to tease out the impact of model size  on such tasks. One potentially interesting finding could be that LLMs are able to deal with k-hops (k>>2) reasoning tasks much easily as compared to small language models (SLMs). It would be interesting to see, if LLMs indeed contain sparse attention heads which represent all the intermediate (bridge) concepts.\n* The authors have used two variants of Gemma 2 model family - 27B and 2B. It would be helpful to conduct experiments using models of varying capacity - including a few with multilingual support, so that claims/findings can be strengthened further.\n* In section 3, add-k problem, I did not understand the reason behind the constraint k_{i+1} - k_i = 3. What happens when we relax the constraint? Will the claims be still valid, if this constraint is not met?\n* The authors mention that ideal scaling constant for bridge intervention is 2.0. My understanding is scaling is a way to boost the importance of the signal (bridge information). Is that correct? If that is so, it is understandable that the performance saturates at 4.0. But, what could be the reason for deterioration?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality\n\n* The unorthodox evaluation methodology of using activation patching with counterfeit examples is creative.\n*  The technique of using interpolation (referred in paper as steering) in this setting is innovative and establishes the geometry of the underlying latent variable.\n* The paper discusses the findings of previous studies in the literature survey section and clearly state the novel contributions of this work.\n\nQuality\n\n* Overall, I find the paper to be of good quality, the methodology is sound and the results are well presented. The experiments are conducted on a variety of datasets which demonstrates the robustness of the claims.\n\nClarity\n\n* The paper is well written and easy to read. The ideas are explained clearly with supporting experiments. \n\nSignificance\n\n* This works brings us closer to understanding how LLMs represent intermediate concepts, perform chained reasoning in ICL setup.\n* It uses tasks such as add-k, circular trajectory, rectangular trajectory to empirically prove that for numerical latent variable problems, tasks vectors indeed represent the geometry of latent variables.\n* The paper strengthens the broader hypothesis that LLMs engage in multi-step reasoning rather than relying solely on direct input-output associations.\n* The authors also highlight the influence of model size and the number of ICL examples on the latent structures.", "weaknesses": "* In the “memorized world knowledge” task, the authors claim that the LLM relies on step-by-step composition of latent concept. However, all their experiments are of the kind where the latent concept has a one-to-one mapping with the output class (for instance country → capital). While, there is no problem with this choice, it might be helpful to study different scenarios (such as country (bridge) → famous politicians from that country)\n* In Figure 10, for the circular trajectory problem, even with high value of beta (0.8), we find that MSE corresponding to original and opposite radius is nearly equal. Isn’t this counter-intuitive?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NfteTAaaWz", "reviewer": "Reviewer_e7J6", "summary": "This paper investigates how LLMs (specifically, Gemma-2-27B) represent and manipulate latent concepts when performing ICL. The authors combine mechanistic interpretability methods with controlled synthetic tasks to explore whether models infer and compose hidden concepts (e.g., countries, radii) in structured ways. The authors explore two major experimental setups: two-hop reasoning tasks and geometric tasks, and conclude that transformers naturally form disentangled, low-dimensional concept representations that are causally and geometrically interpretable.", "questions": "* Do you observe similar disentanglement patterns in larger, more naturalistic multi-hop datasets (e.g., HotpotQA, factual chains beyond two hops)?\n\n* Are the same attention heads consistently responsible for “bridge” resolution across tasks or prompts, or does specialization vary by context?\n\n* How might identifying latent concept circuits inform real-world model editing, debiasing, or steering applications?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The division between discrete and continuous latent concepts is conceptually clean and helps generalize across types of reasoning.\n\n* Identifying sparse, interpretable sub-circuits for latent concept composition is relevant to understanding compositional generalization.\n\n* The authors situate their findings well within recent discussions of task vectors, linear representation hypothesis, and in-context reasoning circuits.\n\n* Demonstrating low-dimensional manifolds for latent numerical parameters (and the ability to steer them) is a compelling visualization of representational structure.", "weaknesses": "* While synthetic setups isolate mechanisms effectively, they may oversimplify real-world linguistic reasoning. It’s unclear whether these disentanglement findings generalize to naturalistic contexts.\n\n* The discrete reasoning experiments rely on Gemma-2 models, while continuous ones use small custom transformers. The gap in architecture and scale complicates unification. Moreover, would the discrete reasoning results replicate with other models?\n\n* Activation patching is compelling, but causal claims might be overstated without broader ablation or counterfactual consistency checks.\n\n* While qualitative visualizations (e.g., PCA plots) are persuasive, more statistical rigor (variance, replicability, robustness to seeds) would add credibility.\n\n* The discussion of how these insights could enhance controllability, generalization, or interpretability in applied settings remains mostly speculative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)