Mixing Mechanisms: How Language Models Retrieve Bound Entities In-Context
Authors: Yoav Gur-Arieh
Mor Geva
Atticus Geiger
Venue: ICLR 2026 Poster (2026)
Keywords: Interpretability
Entity Binding
Causal Abstraction
Mechanistic Interpretability
Primary area: interpretability and explainable AI
TL;DR: Entity binding in LMs is crucial for reasoning. Prior work established a positional mechanism underlying binding, yet it breaks down in complex settings. We find two additional mechanisms, lexical and reflexive, that drive model behavior.

Abstract: A key component of in-context reasoning is the ability of language models (LMs) to bind entities for later retrieval.
For example, an LM might represent *Ann loves pie* by binding *Ann* to *pie*, allowing it to later retrieve *Ann* when asked *Who loves pie?* 
Prior research on short lists of bound entities found strong evidence that LMs implement such retrieval 
via a **positional mechanism**, where *Ann* is retrieved based on its position in context.
In this work, we find that this mechanism generalizes poorly to more complex settings; as the number of bound entities in context increases, the positional mechanism becomes noisy and unreliable in middle positions.
To compensate for this, we find that LMs supplement the positional mechanism with a **lexical mechanism** (retrieving *Ann* using its bound counterpart *pie*) and a **reflexive mechanism** (retrieving *Ann* through a direct pointer). 
Through extensive experiments on nine models and ten binding tasks, we uncover a consistent pattern in how LMs mix these mechanisms to drive model behavior.
We leverage these insights to develop a causal model combining all three mechanisms that estimates next token distributions with 95\% agreement.
Finally, we show that our model generalizes to substantially longer inputs of open-ended text interleaved with entity groups, further demonstrating the robustness of our findings in more natural settings.
Overall, our study establishes a more complete picture of how LMs bind and retrieve entities in-context.

Published as a conference paper at ICLR 2026
MIXING MECHANISMS: HOW LANGUAGE MODELS RETRIEVE BOUND ENTITIES IN-CONTEXT
Yoav Gur-Arich 9, Mor Geva Atticus Geiger
Blavatnik School of Computer Science and AI, Tel Aviv University Pr(Ai) Group Goodfire
ABSTRACT
A key component of in-context reasoning is the ability of language models (LMs) to bind entities for later retrieval. For example, an LM might represent Ann loves pie by binding Ann to pie, allowing it to later retrieve Ann when asked Who loves pie? Prior research short lists of bound entities found strong evidence that LMs implement such retrieval via a positional mechanism, where Ann is retrieved based its position in context. In this work, find that this mechanism gener- alizes poorly to more complex settings; as the number of bound entities in context increases, the positional mechanism becomes noisy and unreliable in middle posi- tions. To compensate for this, we find that LMs supplement the positional mech- anism with a lexical mechanism (retrieving Aun using its bound counterpart pie) and reflexive mechanism (retrieving Ann through direct pointer). Through extensive experiments on nine models and ten binding tasks, we uncover con- sistent pattern in how LMs mix these mechanisms to drive model behavior. We leverage these insights to develop causal model combining all three mechanisms that estimates next token distributions with 95% agreement. Finally, we show that our model generalizes to substantially longer inputs of open-ended text interleaved with entity groups, further demonstrating the robustness of our findings in more natural settings. Overall, our study establishes a more complete picture of how LMs bind and retrieve entities in-context.
INTRODUCTION
Language models (LMs) are known for their ability to perform in-context reasoning (Brown et al. 2020), and fundamental to this capability is the task of connecting related entities in a text known as bindingnn--ii to construct representation of context that can be queried for next token prediction. However, LMs are also known for struggling in reasoning tasks over long contexts (Liu et al. 2024 Levy et al. )024). In this work, we conduct mechanistic investigation into the internals of LMs to better understand how they bind entities in increasingly complex settings.
Neural networks' ability bind arbitrary entities was central issue in connectionist models of cog- nition (Touretzky & Minton 1985 Fodor & Pylyshyn 1988 Smolensky 1990) and has reemerged
in the era of LMs as target phenomenon for mechanistic interpretability research (Davies et al. 2023 Prakash et al. 2024; 2025) 2025) Feng & Steinhardi 2024 Feng et al. 2024, Wu et al. 2025). For example, represent the text Pete loves jovesjam and Ann loves pie, an LM will bind Pete to jam and Ann to pie. This enables the LM to answer questions like Who loves pie? by querying the bound entities to retrieve the answer (Anz). The prevailing view that LMS retrieve bound entities using a positional mechanism (Dai et al. 2024: Prakash et al. 2024; 2025), where the query entity (pie) is used to determine the in-context position of Ann loves pie- in this case, the second clause after Pete loves jam-whim is dereferenced to retrieve the answer Ann.
In this work, we show that position-baat based retrieval holds only for simple settings. This mechanism is unreliable for the middle positions in long lists of entity groups pattern that echoes the "lost- in-the-min middde""""" effect (Liu et al. (0024) in LMs as well as primacy and recency biases in both humans (Ebbinghaus 1913 Miller & Campbell 1959) and LMs (Janik] 2024). To compensate for this noise,

---

Published as a conference paper at ICLR 2026
Counterfactual Input
Positional @ lexical A and reflexive & bindling info written entity tokens
(B) Binding info used queried entity
T
Patch
+
Joe loves
ale
Ann loves pie
Pete loves jam
Tim loves tea What does Ann love
Original Input
(C) Patching last token position across examples
the signals
fesical
Positionale
Ann
loves
ale Joe loves Jam
Pete loves
pie
Tim loves tea
What does
Tim
love
Original Unpatched
Original Patched
Ann loves ale, Joe loves jam, Pete loves pie, Tim loves tea. What does Tim love?
Ann loves ale, Joe loves jam, Pete loves pie, Tim loves tea. What does Tim love?
Legtts
Final
jam
ple
tea
Without intervention, the three mechanisms agree with a decisive prediction
Under intervention, the three mechanisms diverge in the entities predicted
An illustration of the three mechanisms for retrieving bound entities in-context. We find that as models process inputs with groups of entities: (A) binding information of three types- positional, lexical, reflexive-is encoded the entity tokens of each group, (B) this binding formation is jointly used to retrieve entities in-context, and (C) it is possible to separate the three binding signals with counterfactual patching. The counterfactual input designed such that patch- ing activations to the LM run on the original input results in the positional, lexical, and reflexive mechanisms predicting different entities (See $3.2). The lexical signal from the counterfactual picks out ale in the original, because the question the counterfactual was about Ann. The positional signal from the counterfactual picks out jam, because the question in the counterfactual was about the second character. The reflexive signal picks out pie, because pie was the counterfactual answer.
Figure 1:
LMs supplement the positional mechanism with a lexical mechanism, where the query entity (pie) is used to retrieve its bound counterpart (Ann), and reflexive mechanism, where the queried entity (Ann) retrieved with a direct pointer that was previously retrieved via the query entity (pie).
In a series of ablation experiments, we show that all three mechanisms are necessary to develop an accurate causal model of the next token distribution (Pislar et al. (025). and that their interplay depends on the positions of the query entity (pie) and the retrieved entity (Ann). This mixture of mechanisms is robustly present across (1) the Llama, Gemma, and Qwen model families, (2) model sizes within those families ranging from 2 to 72 billion parameters, and (3) ten variable binding tasks. By better understanding this mechanism, we take a step toward explaining both the strengths and the persistent fragilities of LLMs in long-contex: settings, well as the fundamental mechanisms that support in- io-context reasoning. We release our code and data at https: h/g/t/ttti com/ yoavgur/mi xi wiweniiinnniiiiiiooo ng-mechs
2

---

Published as a conference paper at ICLR 2026
2 PROBLEM SETUP AND PRIOR WORK
Entity Binding Tasks In our experiments, we design a number of templatic in-context reasoning tasks with similar structure the example from the introduction, i.e., Pete loves jam, Ann loves pie. Who loves pie? Formally, a task consists of:
1. Entity Roles:
Disjoint sets of entities & Ei...., that will fill particular roles in a templatic text. For example, the set El might be names of people {Ann, Pete, Tim.... ). and the set E2 might be foods and drinks {ale, jaee .i.,...
2. Entity Groups: An entity group is tplle G E xx.x...-EE Em containing entities that will be placed within the same clause in a template. For example, we could set G1 = jam) and G2 (An,,iie). For convenience, we define G as a binding matrix wherein G denotes the j-th entity in the i-th entity group.
3. A template (7): function that takes as input binding matrix G. the query entity q = "uayy and the target entity t = Laatt. Here (group is positional index of the entity group containing the target and query, and featity o qentity index the positions of the target and query entities within that group, respectively. See for more details and examples.
G (gnup
Continuing our example, define
q(G = G1 loves Ci, Gl. loves G2.
Who loves q?
lettiy
What does q love? lentity
and observe that
7( Pete
jam pie]
Apie,Ann Pete jam, Ann loves pie. Who loves pie?
Ann
For our experiments, the binding matrix G will consist of distinct entities.
Interchange Interventions To probe the mechanisms an LM uses to bind and retrieve entities, employ interchange interventions (Vig al. 2020 Geiger et al. 2020 Finlayson et al. 2021 Geiger et al., 202T). the standard tool for prior work on binding and retrieval (Davies et al. 2023
Feng & Steinhardt 2024 Prakash et al. [024 2025 Wu al. 2025). These interventions allow us to identify which hidden states are causally relevant for the model in entity binding, by running the LM on paired examples an original input and a counterfactual input and replacing selected components, e.g., residual stream vectors. in the original run with those from the counterfactual.
Causal Abstraction We develop causal model of LM internals (Geiger et al. 2021 2025b ap that predicts the LM next token distribution using mixture of three mechanisms (See $ To test our hypotheses, we construct a dataset of paired originals and counterfactuals such that an inter- change intervention the causal model results in the positional, lexical and reflexive mechanisms increasing the probability of distinct tokens. To evaluate our proposed causal model and various ablations, we perform interchange interventions the causal model and the LM,, measuring the similarity between the next token distribution of the two models, and average across a dataset.
Prior Studies of Entity Binding in LMs Previous work paints a picture of how entity binding and retrieval is performed by LMs. First, LMs bind together a group of entities by aggregating informa- tion about all entities in the entity token at the last position the group. By co-locating information about entities in the residual stream of a single token, the LM can later on use attention to retrieve information about one bound entity conditional on second bound entity (Feng & Steinhardi 2024] Feng et al.| 2024 Dai et al./ 2024 Prakash et al.) 2024 Wu et al.) 2025). ) algorithmic motif that Prakash et al. (025)) dub a "lookback" mechanism. We study the "pointers" used the lookback mechanism that bring the next token prediction into the residual stream of the final token. We in- clude experiments on the "addresses" contained in the residual streams of the bound entity tokens,
as well as the query token, in $C
Prior works identify a positional mechanism that is utilized in entity binding (described in detail in 3.1), but either evaluate it only in narrow settings (Prakash al. 2025) achieve low causal
3

---

Published as a conference paper at ICLR 2026
Patch Effects
positional
lexical
reflexive
Centity
3
Eentity
1.0
1.0
0.5
0.0
0.0
2
6
10
12
14
16
18
20
22
Centity = 2
Patch Effect
1.0
Patch Effect
1.0
index=10
0.5
0.5
0.0
0.0
10
16
20
22
Centity =
1.0
index=19
0.5
0.5
0
4
8
10
12
14
16
18
20
22
24
4 8 10 12 14 18 Patched Entity Group Index
Layer
Figure 2: Results from interchange interventions on gemma-2-2b-> over a counterfactual dataset with three entities per group (m = (See Figure T and $3.2 Outputs predicted by the positional, lexical and reflexive mechanisms are shown in dark blue, green and orange. In light blue, we show the cases not predicted by any of the mechanisms, dubbed mixed. These cases further analyzed in $3.3 Left: Distribution of effects (y-axis) for three representative entity group indices (first, middle, and last) with fentiy
for all layers (x-axis). At layers 16-18, the last token position carries binding information used for retrieval. Right: Distribution of effects (y-axis) for all entity indices (x-axis) at layer for femity E 21,2,3). i.., the question can about any the three entities in each clause. A U-shaped curve emerges: first and last indices rely more the positional mechanism, while middle indices rely more on the lexical and reflexive mechanisms. See $ for replication across models and tasks, and Figure 20 for plots using the original prompt the x-axis.
faithfulness in predicting model behavior solely using this mechanism (Prakash et al. 2024 Dai ct al. 2024). Feng & Steinhardt (2024); Prakash et al. (2025) restrict their analysis to queries of the final token group (femiiy m) and to very small contexts (n 2, 3). Prakash et al. (2024) and
Dai et al. 2024) find a positional mechanism in longer contexts (n = 7), but with low faithfulness.
3 THREE MECHANISMS FOR RETRIEVING BOUND ENTITIES
In this section, we define the positional mechanism and propose two alternative mechanisms ($3.1) all three of which make distinct claims about the causal structure of the L.M. Then, we design dataset with pairs of original and counterfactual inputs, such that each of the three mechanisms makes distinct predictions under an interchange intervention with the pair ($3.2). Last, we perform interchange interventions and visualize the results so we can observe the interplay between the three mechanisms in the coun- terfactual behavior of the LM ($3.3). We detail in $3.3 and $D.2 how we localize the layers for ducting the interventions. In our experiments, we evaluate nine models models-gemmai 2-muaa--(2b/9b2277) (wwen.55(3b/77/32b//
the full last-token residual stream vector at different layers of the LM
binding tasks, boxes and music (see Appendix Table []. For gemma-2-2b-it and gwen2.5-7b-it we evaluate on all ten binding tasks.
and Ilama-3. (-(88/00b/ 11ama-3.1-(8b7700 -itt-on
3.1 THE POSITIONAL MECHANISM AND TWO ALTERNATIVES
The prevailing view is that bound entities are retrieved with a positional mechanism, but we propose two alternatives: lexical and reflexive mechanisms. The positional, lexical, and reflexive mecha-
4

---

Published as a conference paper at ICLR 2026
nisms represented causal models P. L, and R that each have single intermediate variables P, L, and R. respectively, used to retrieve an entity from context as the output.
The Positional Mechanism Prior work provides evidence that positional mechanism is used to retrieve an entity from a group via the group's positional index (Dai et al. 2024; Prakash et al.
2024 2025). The model d indexes the group containing the query entity (P := Agrop), and its output mechanism retrieves the target entity from the group at index Qgroup- In Figure ! we have
P P= 4 when no intervention is performed on the LM and the target entity tea is retrieved from position 4, but after the intervention P the entity jam at the second position is retrieved.
Although existing evidence shows that the positional mechanism explains LM behavior in settings with two or three entity groups (Prakash et al. 2025), it does not generalize. When more groups are introduced, the evidence is weaker (Prakash et al. 2024, Dai et al. (024). Our goal is to investigate the failure modes of the positional mechanism as more entity groups are introduced, and to that end propose two alternative hypotheses for how LMs implement binding.
The Lexical Mechanism The lexical mechanism is perhaps the most intuitive solution: output the bound entity from the group containing the queried entity. The causal model L stores the query entity (L := q) and the output mechanism retrieves the target entity from the group containing q. In Figure D we have L = Tim when no intervention is performed on the LM and the output mechanism retrieves the entity tea from the group with Tim. However, after the intervention t Ann, the entity ale is retrieved from the group with Ann.
The Reflexive Mechanism The reflexive mechanism retrieves entity with direct, self- referential Shiiter-origgnatinn from that entity and pointing back to it (illustrated in Appendix Figure However, if this signal is patched into a context where the token is not present, the mech- anism fails. The model R stores the target entity (R == !) and the output mechanism retrieves the entity if it appears in context. In Figure [] we have R and the entity tea is retrieved, but after the intervention R + pie, the entity pie is retrieved because appears in the original input.
when intervention is performed
The reflexive mechanism is an unintuitive solution, until one considers that the architecture of an autoregressive LM allows attention to only look from right to left. When the query occurs after a target in an entity group, i.e., femity < qentity, the lexical mechanism is not possible. In the text Tim loves lea, the entity tea cannot be copied backwards to the residual stream of Tim so that the lexical mechanism can answer Who loves tea? Therefore, earlier mechanism in the LM must first retrieve an absolute pointer that is in turn used to retrieve the bound entity token.
3.2 DESIGNING COUNTERFACTUAL INPUTS TO DISTINGUISH THE THREE MECHANISMS
We designed dataset of paired original and counterfactual inputs such that the positional, lexical, and reflexive mechanisms will each make distinct predictions when an interchange intervention performed on their respective intermediate variables, P, L. and R.
Counterfactual Design Figure displays a pair of original and counterfactual inputs that dis- tinguish our three mechanisms (further detailed in Appendix Table D]. We illustrate this with the following example. Define the original and counterfactual binding matrices G and G' respectively:
Ann
ale
Joe
ale
Joe
jam
Ann
pic
G
G'
Pete
pie
Pete
jam
Tim
tea
Tim
tea
We can then use the template T from 2 such that for these binding matrices, T(G, Tim, tea) yields the original input in Figure I and T(G Aunn pie) yields the counterfactual input. Each of the three mechanisms produces a different output after an interchange intervention on this pair of inputs:
1. An interchange intervention on P in P would output the entity at the counterfactual query's position. Since dgruup 2 for Ann in G', this sets P 2, and the output becomes jam.
5

---

Published as a conference paper at ICLR 2026
lexical
positional
reflexive
2
i
Positiona Index
80
Percentage (0k)
Mean Logit L Value
h =
6
8
i =
11
12
40
R
25 13
20
1
16
15
Prediction Index
14
4 12
6
16 4
12
16
Entity Group Index
Figure 3: The positional mechanism is diffuse for middle entity groups. Left: Confusion matrix (%) of the patched positional index (y-axis) vs. semma-2-2b-it's prediction (x-axis) after an interchange intervention (as in Figure []. Counterfactual predictions cluster near the position promoted by the positional mechanism, decaying with distance. Only the mixed and positional patch effects from Figure 2 are shown; see Figure 31 for other models and tasks. Right: Mean logit distributions with ip = 6, å = 14, and i, varied, illustrating additive and suppressive interaction between the three mechanisms. The lexical and reflexive signals form one-hot peaks, while the positional broader and more diffuse. See Figures 24 25 and 26 for more distributions.
2. An interchange intervention L in L would output the entity in the original input bound to the query entity in the counterfactual input. Since the query entity is now Ann, the mechanism queries the group containing Ann in G and outputs the bound entity ale.
3. An interchange intervention on R in R. would follow the direct pointer established in the counterfactual input. In this case, the pointer is to the token pie, which exists in the original input, and so the mechanism outputs pie.
Each of these three outputs is distinct from the actual output for the original input, which means the dataset also distinguishes the three mechanisms from no intervention being performed. Let ip, ig, and iR be indices of the entity groups queried by the positional, lexical, and reflexive mecha- nisms, e.g., ip = 2, ig 1, and iR 3 in Figure [ after patching. In our counterfactual datasets, each of the three mechanisms can predict any position in the list of entity groups from the original input, i.e., ip, it, and in vary freely from 1 to n. For details and task templates, see I A relevant remaining confounder is that the reflexive mechanism predicts the output that is the target entity in the counterfactual input, meaning this dataset cannot distinguish the pointer used by the reflexive mechanism from the actual next token prediction. We resolve this issue, validating the existence of reflexive mechanism, in 3.4
3.3 INTERVENTION EXPERIMENTS
We find experimentally that information used to retrieve a bound entity is accumulated in the last token residual stream across a subset of layers. In Figure 2 we show the results of interchange interventions on gemma-2-2b-it across the layers of the last token residual stream. We see that in layers 16-18 the model accumulates binding information in the last token position. Therefore, unless stated otherwise, we conduct all of our interchange interventions by patching the last token residual stream vector on the last layer before retrieval starts, denoted as f, which is different for each of the nine models we test, but consistent across tasks for a given model (see $D.2 for more details). We measure the next token distribution produced by the model under intervention and compare it against the possible outputs for the three mechanisms. We aggregate and visualize the results of these intervention experiments in Figures 2 and

---

Published as a conference paper at ICLR 2026
The positional mechanism weakens for middle positions. We can see plainly in Figure 2 that the positional mechanism controls behavior solely when the positional index is at the beginning or end of the sequence of n 20 entity groups. In middle entity groups. however, its effect becomes minimal, accounting only for 20% of the model's behavior. Further analysis of the cases not explained by any of the mechanisms- dubbed mixed in the plot-revealss that these predictions are distributed near the positional index (Figure B). Additionally, when collecting the mean logit distributions across many samples and fixing the positional index, we see that in the first and last positional indices it induces strong and concentrated distribution around that index. However, in middle indices we see this distribution become wide and diffuse (Appendix Figure AAppendixFFiuure[13 Thus, the positional mechanism becomes unreliable in middle indices and cannot be used as the sole mechanism for retrieval. We show in $A.3 how this effect emerges as n, i.e., the total number of entity groups in context, increases, and in G we disambiguate the effect of increasing n from that of increasing sequence length.
The lexical and reflexive mechanisms are modulated based on target entity position. Observe in Figure 2 that when the positional mechanism is unreliable for middle positions, the lexical and reflexive mechanisms come into play. However, which of these two alternate mechanisms contribute more depends on the location of the target entity within the entity group, denoted as lemityy When the target is at the beginning of the group (leniiy 1), the reflexive mechanism is used (as discussed in $3.1) When the target is at the end (teatity = 3), the lexical mechanism is primarily used. Finally, when the target is in the middle (featiy 2), both mechanisms are used to differing extents.
The three mechanisms have complex interplay. We can see in Figure 3 the interplay between the three mechanisms when the positional and reflexive indices are fixed to ip = and iR = 14 while the lexical index iz is iterated over range of values. First, the logit distributions clearly reveal the contributions of each mechanism, with a distinct spike appearing at each index. These spikes, however, behave differently. In line with Figure Bl the positional index produces wide, diffuse distribution, whereas the lexical and reflexive indices produce sharp, one-hot peaks. Next, we ob- serve that the mechanisms interact through a pattern of competitive synergy, meaning that they both boost and suppress one another. When the lexical index is close to the positional index, the lexical contribution is amplified while the positional contribution weakened; when they are farther apart, neither affects the other. In contrast, when the lexical index is close to the reflexive index, the lexical contribution is suppressed by the reflexive one. We design more datasets of niginal-counterfacee pairs to understand the bound entity tokens' residual streams, and analyze how binding information encoded and propagates across token positions, in $C
Takeaways These results clarify how LMs bind and retrieve entities in context. They simultane- ously employ three mechanisms: positional, lexical, and reflexive. In the first and last entity groups, LMs can rely almost exclusively the positional mechanism, where it is strongest and most con- centrated. In middle groups, however, the positional signal becomes diffuse and often links entities to nearby groups. In these cases, the lexical and reflexive mechanisms provide sharper signals which refine the positional mechanism, enabling the LM to retrieve the correct entity.
3.4 VALIDATING THE EXISTENCE OF THE REFLEXIVE MECHANISM
In our counterfactual design (.2)) we constructed the counterfactuals such that each hypothesized mechanism makes a different prediction about the outcome of intervention experiments. However, we also noted that these counterfactuals fail to distinguish the pointer used in the reflexive mech- anism from the answer itself. In this section, we address this by designing a new counterfactual dataset specific to distinguishing between the two. In S we also conduct an attention knockout experiment to further strengthen our findings.
New Counterfactual Design We modify the existing dataset such that the counterfactual answer entity doesn't 't appear in the original input. For example, for the input pair from Figure I we would keep the original input Ann loves ale, Joe loves jam. Pete loves pie, and Tim loves tea. What does Ann love?, but alter the counterfactual to Joe loves ale, Ann loves cod, Pete loves jam, and Tim loves tea. What does Ann love? ? so the counterfactual answer cod appears nowhere in the original.
7

---

Published as a conference paper at ICLR 2026
Original CF | Layer t
Original CF Layer ++
New CF Layer /
New CF Layer 2+
1.0
Patch Effect
0.8
positional
Patch Effects
rreflesivve
0.6
Imbxed
lexical
Icl answer/ reflexive
cf answer
0.4
0.2
3
9
12
15
18
12
18
12
15
18
3
12
15
18
Patched Entity Group Index
Figure 4: We distinguish the pointer in the reflexive mechanism from the answer entity with inter- change interventions (gemma-2-2b-14 fentity = 1, layers e, + 1). Left: interventions on the original counterfactual dataset show that can't distinguish between patching the pointer or the answer entity itself. Right: interventions on the modified counterfactuals ($3.4). At layer the model does not respond with the counterfactual answer entity which does not appear in the original context, indi- cating that the patched signal is reflexive pointer that cannot be dereferenced. At layer & + 1, once the model has already retrieved the answer entity ($D.2), the patched signal becomes the answer entity itself. This shows that no confounding suppressive mechanism exists to prevent the model from answering with an entity not in its context.
Such examples differentiate between the pointer R in the reflexive mechanism R. and the output of the mechanism itself. An interchange intervention replace the answer entity ale with the answer entity cod. However, an interchange intervention on the pointer R would patch in pointer to the token cod that the mechanism R would attempt to dereference. However, cod does not appear in the original input, and thus the pointer cannot be resolved and no output is predicted by this mechanism.
the output of the mechanism would simply
Results We show the results for layer f, for the original counterfactual setup, as well as for the new one, in Figure 4 We see that while in the original counterfactual setup, the model answered with the entity pointed to by the reflexive mechanism, under the new counterfactual setup it did not. This indicates that what was copied is reflexive pointer that cannot be dereferenced, as opposed the answer entity itself. One alternative explanation is that the model might contain a mechanism that suppresses outputs corresponding entities absent from the context. To exclude this possibility, we repeat the evaluations at layer + 1, point at which the model has already retrieved the correct answer. Here, patching leads the model to output the counterfactual answer entity in both counter- factual setups, showing that no such suppressive mechanism is present. We can therefore conclude that the model indeed relies on reflexive mechanism, distinct from the positional and lexical ones, where direct pointer to the answer entity is used to retrieve it.
4 A SIMPLE MODEL FOR SIMULATING ENTITY RETRIEVAL IN-CONTEXT
To formalize our observations about the dynamics between the three mechanisms and the position of the target entity, we seek develop a model that approximates LM logits for next token prediction, as position-weighted mixture of terms for the positional, lexical, and reflexive mechanisms.
Mixing mechanisms in causal model We follow Pislar al. (2025) in combining together mul- tiple causal models (P, L. R. into a single causal model M that modulates between the mechanisms conditional on the input. In our combined causal model, the lexical and reflexive terms have separate learned weights conditioned on their respective index, i.e., iL, or igc. In accordance with the results shown in Figure 33, we model the lexical and reflexive mechanisms as one-hot distributions that up-weight only the target entity in groups é, and ig. respectively. The positional term is modeled as a Gaussian distribution scaled by a single weight Wpos centered at the index ip with a standard deviation that is quadratic function of ip. We define a new causal model M that uses all three variables P, L. and simultancously to compute a logit value Y, for each entity G tansy"
Y = Wpos (i ip, (ir))²) Wlex [iz] 1fi - Wref [in] 1 in) reflexive mechanism
positional mechanism
lexical mechanism
Where (Ar) ar )2 gir + 7. We learn Wposs Wlez Wref, a, 8, from data.
8

---

Published as a conference paper at ICLR 2026
Learned Weights
Model
JSS t
Le =
Le
Welght
Compuring against the prevailing view
M hon Ronoshois Piuss))
0.95
0.96
0.94
P Pone-hon (prevailing view)
0.42
0.46
0.45
/W/ox BW ror ppss
Modifying the positional mechanism
Op(P)
M w/ Paracee
0.96
0.98
0.96
M w/ Pone
0.86
0.85
0.85
Ablating the three mechanisms
M
(Pamms }
0.67
0.68
0.67
M
hone-bat
0.94
0,91
0.75
M
Rone-hon
0.69
0.87
0.92
M
Renoshots Lonesree
0.69
0.84
0.74
M
(Pranss Rane-rec)
0.12
0.27
0.48
12 Entity Group Index
18
M
(Peanss Lone-hoe
0.55
0.41
0.20
Uniform
0.44
0.57
0.49
Figure 5: Results for training our full model M (Lons-hot- Rons-hor Gauss): in addition to vari- ants, baselines and ablations. Left: JSS scores for modeling the LM next token distribution ip, i in- Evaluated on gemma-2-2b-it for the music binding task, with fe femityy Our model attains near-perfect JSS, slightly below the oracle. KL values (Table 3 show the same trend. All Cls are < 0.02: for M and M w/ oracle they are < 0.002. Right: Learned weights Wlex: Wref, Wpos and a curve, for feiitty 2. Observe widens for middle indices and narrows toward the end.
Learning how the mechanisms are mixed To generate data for training our causal model we per- formed 150 interchange interventions per combination of S ip, iL iR using the original and counterfactual inputs designed to distinguish the three mechanisms (see Figure I and Section 3.2) We collected the logit distributions per index combination, and averaged them into mean probability distributions by first applying softmax over the entity group indices and then taking the mean. This yields 3 8, 000 probability distributions, which serve as our data for training and evaluation. We used 70% of the data for learning the causal model parameters and split the remainder evenly be- tween validation and test sets. The loss used is the Jensen-Shaee -hannno divergence (JSD) between our model's predicted probability distribution and the target, chosen for its symmetry.
We evaluate M alongside a range of baselines, variants, and ablations to characterize our model's performance and understand the contribution of the different mechanisms. Experiments run with gemma-2-2b-it on the music task (n = 20, lentiy E (3). In $ we report the same setup for additional tasks, with similar trends. We measure similarity
this model well as qwen2.5-7b-it between the predicted and target distributions using Jensen-Siie Shannon similarity (JSS), defined as JSD, calculated with log, yield values in [0, 1]. See Appendix Table for KL divergences.
We compare our model with: (1) The prevailing view - one-hot distribution at the positional index, (2) variant of M using one-hot distribution at ip instead of Gaussian, (3) ablations of M using only a subset of the mechanisms (c.g.. M {Lone-bot) omits the lexical term from Equation 2). and (4) a uniform distribution. Finally, as an upper bound, we evaluate an oracle variant, where the lexical and reflexive components are learned as usual, but the positional component is swapped with the actual logit distributions of the model, as function of ip (see Figure
Results Figure shows the results. Our model achieves near near-perfect performance, only slightly below the oracle, at an average JSS of 0.95. contrast, the prevailing view achieves an average JSS of 0.44, well below even the uniform distribution baseline with 0.5. Next, we see that modeling the positional mechanism as a one-hot as opposed to gaussian significantly hurts performance, employed: for instance,
dropping 0.85 JSS. The ablations further reveal how mechanisms when feniity = 1, ablating the lexical mechanism has nearly no effect, while for temityy this true for the reflexive mechanism. This is in keeping with previous results, showing that the lexical and

---

Published as a conference paper at ICLR 2026
1000 Tokens
4000 Tokens
1.0
Positional index
12 s
0.8
Effect
Positional
15
Proportion
0.6
Loxical
Reflexive
Porcentage
Mixed
7000 Tokens
10000 Tokens
0.4
Effect
Positional Index
Invalid
0.2
Accuracy
15 15 32
0.0
0
2000 4000 6000 8000 10000
#Filler Tokens
Prediction Index
Prediction Index
Figure 6: Padding results for gemma-2-2b-it 2b-it on the boxes task. Left: Confusion matrix between the model's predicted index and the positional index patched in from the counterfactual. This gets increasingly fuzzy for early tokens as padding is increased. Right: Distribution effects as padding increased, showing the positional mechanism strengthens the expense of the lexical mechanism.
reflexive mechanisms are used differently depending on the value of lentiyy Figure 5 also shows the learned parameters of the model for lentity 2. We see that this setting, the lexical and reflexive mechanisms behave amiiarlyyweaker at the beginning. flat in the middle, with an uptick the end-although the reflexive mechanism is slightly more dominant, consistent with the table results. For the positional mechanism we can see that starts off very concentrated, becoming wider in middle indices, and finally becoming more narrow towards the end, mirroring previous results.
INTRODUCING FREE FORM TEXT INTO THE TASK
To test our model's generalization to more realistic inputs, we modify our prompt templates such that they include filler sentences between each entity group. To this end, we create 1,000 filler sentences that are "entity-less", meaning they not contain sequences that signal the need to track or bind entities, e.g. "Ann loves ale, this is known fact, Joe loves jam, this logic is easy to follow..." This enables us to evaluate entity binding in a more naturalistic setting, containing much more noise and longer sequences. We evaluate different levels of padding by interleaving the entity groups with an increasing number of filler sentences, for a maximum of 500 tokens between each entity group.
The results, shown in Figure 6 for gemma-2-22 2-auma-----iir- the boxes task, show that model at first remains remarkably consistent in more naturalistic settings, across even a ten-fold increase in the number of tokens. However, the amount of filler tokens increases, we that the magnitude of the mechanisms' effects changes. The lexical mechanism declines in its effect, while the positional and mixed effects slightly increase. We can also that the mixed effect remains distributed around the positional index, but that it slowly becomes more diffuse. Thus, when padding with 10,000 tokens, we get that other than the first entity group, the positional information becomes nearly non- existent for the first half of entity tokens, while remaining stronger in the latter half. This suggests that a weakening lexical mechanism relative an increasingly noisy positional mechanism might be a mechanistic explanation of the "'ost-in-the-midde" effect (Liu et al., (2024). In $D.4 we show that our model generalizes inputs with more linguistic variability as well.
6 CONCLUSION
In this paper, we challenge the prevailing view that LMs retrieve bound entities purely with posi- tional mechanism. We find that while the positional mechanism is effective for entities introduced at the beginning or end of context, it becomes diffuse and unreliable in the middle. We show that in practice, LMs rely on a mixture of three mechanisms: positional, lexical, and reflexive. The lex- ical and reflexive mechanisms provide sharper signals that enable the model to correctly bind and retrieve entities throughout. We validate our findings across 9 models ranging from 2-72B, and 10 binding tasks, establishing general account of how LMs retrieve bound entities.
10

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY STATEMENT
We take multiple steps in this work to ensure the reproducibility of our findings. In $ we detail all binding tasks used in our evaluations, and in $3.2 and Table we describe how to construct datasets of paired original and counterfactual examples. $ specifies the hyperparameters used for training our causal models. The code for dataset generation and causal model training is included in the supplemental materials.
ACKNOWLEDGMENTS
This work was supported in part by the Alon scholarship, the Israel Science Foundation grant 1083/24, and grant from Open Philanthropy. Figure Jsees images from www. freepik com
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever. and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https: / ngs. neur iss cc. wwwperppppppppeeeeeeeeeeee..eeee fiw.fillllseee pelessppprr/200// fi FTle//447TC0660faa44677111111110 44aaaaaace1444aaaaaeeaee.aeeee
Qin Dai, Benjamin Heinzerling, and Kentaro Inui. Representational analysis of binding in lan- guage models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceed- ings of the 2024 Conference on Empirical Methods Natural Language Processing, pp- 17468-17493 Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 1/22543/1/222 mainnlp- URLhttps: /BKLatttss//aallaatlloooooooo/r
emr.pp ma ii.innnin 9677
Xander Davies, Max Nadeau, Nikhil Prakash, Tamar Rott Shaham, and David Bau. Discovering variable binding circuitry with desiderata, 2023.
03637
Hermann Ebbinghaus. Memory: Contribution to Experimental Psychology. Teachers College, Columbia University, New York, NY, US, 1913.
Jiahai Feng and Jacob Steinhardt. How do language models bind entities in context? In The Twelfth International Conference on Learning Representations, 2024. URL. htl.:httts::/opeeeer//
Jiahai Feng, Stuart Russell, and Jacob Steinhardt. Monitoring Latent World States in Language
Models with Propositional Probes, June 2024.
Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. Causal analysis of syntactic agreement mechanisms in neural language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 18433, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2022 acl-long. 144. URL htttps://aclanthooooy.oooo
20tp:2221.ccc -ong. 1o47
Jerry A. Fodor and Zenon W. Pylyshyn. Connectionism and cognitive architecture: A critical anal- ysis. Cognition, 28(1-2):3-71, 1988. doi:
Atticus Geiger, Kyle Richardson, and Christopher Potts. Neural natural language inference models partially embed theories of lexical entailment and negation. In Afra Alishahi, Yonatan Belinkov, Grzegorz Chrupata, Dieuwke Hupkes, Yuval Pinter, and Hassan Sajjad (eds.), Proceedings of the

---

Published as a conference paper at ICLR 2026
Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 163- 173, Online. November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. blackboxnlp- 1.16. URL wtl.itttpe///laaattooooooooo//o.
Atticus Geiger, Hanson Lu, Thomas F Icard, and Christopher Potts. Causal abstractions of neural networks, In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan Advances in Neural Information Processing Systems, 2021. URL JDsRmuXDDtjDhh
Atticus Geiger, Jacqueline Harding, and Thomas Ieard. How causal abstraction underpins computa- tional explanation, 2025a. UtLhttps: arxiv. wtreiiiorrg//oo /a////b////500 11214
Atticus Geiger, Duligur lbeling, Amir Zur, Maheep Chaudhary. Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Ieard. Causal abstraction: A theoretical foundation for mechanistic interpretability. Journal of Machine Learn- ing Research, 26(83):1-64, 2025b. URL. http:// www.g.r.....roo.oooooo...o.ooo.r wwwppaa/pae/..p.. html
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 1221 12215, Singapore, December 2023. Association for Computational Linguis- https/////aaaaaahleaacc conthlloyycoo org//023.
tics. doi: 10. emnlp-main. 751/
Romuald A. Janik. Aspects of human memory and large language models, 2024. URL https:
7/arxiv. www.or///aabaaaaaae...eeeeeeee/e
03839
Mosh Levy. Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 15339- 15339-15353, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1 URL
8187
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguisticss, 12:157-173, 2024. doi: 10.1162/tacl URL.
2024 tacll-1...cl11
Norman Miller and Donald Campbell. Recency and primacy in persuasion as a function of timing of speeches and measurement. Journal of abnormal psychology, 59:1-9, 9, 07 1959. doi: 10. 10.1037/ h0049330.
Theodora-Mare Pislar, Sara Magliacane, and Atticus Geiger. Combining causal models for more ac- curate abstractions of neural networks. In Fourth Conference on Causal Learning and Reasoning, 2025. /ttptttts///ppeeeeeeeee
Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: case study entity tracking. In The Twelfth International Conference on Learning Representations, 2024, URLhttps: http://pppeeeeeeeeee/ee
id 220
Nikhil Prakash, Natalie Shapira, Arnab Sen Sharma, Christoph Riedl, Yonatan Belinkov, Tamar Rott Shaham, David Bau, and Atticus Geiger. Language models use lookbacks to track beliefs, 2025.
URL https: //txxva.ivv///////a//aae//aaa wwwpxvv.org/aaae wttp:rggggaa....aao 14685
Paul Smolensky. Tensor product variable binding and the representation of symbolic struc- tures in connectionist systems. Artificial Intelligence, 46(1-2):159-22 1990. doi: 10.1016/
0004-370299990007MMM.
12

---

Published as a conference paper at ICLR 2026
David S. Touretzky and Geoffrey E. Minton. Symbols among the neurons: details of connectionist inference architecture. In Proceedings of the 9th International Joint Conference on Artificial In- relligence Volume 1, LICAI' 85, pp. 223,,24,,. San Francisco, CA, USA, 1985. Morgan Kaufmann Publishers Inc. ISBN 0934613028.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33: 13:1888-12401, 2020.
Yiwei Wu, Atticus Geiger, and Raphaël Millière. How do transformers learn variable binding in symbolic programs? In Forty-second International Conference on Machine Learning. 2025. URL hittps://ppenrevieet wwtp:/reteeeeeeeeeeooeeoeoo http://orrummidkk//yyyylllll
A EVALUATING GENERALIZATION
In this section, we seek to validate that our findings generalize to different values of n, as well as across models and binding tasks.
A.1 BINDING TASKS
In this subsection, we detail the different binding tasks we evaluate, and show that our findings generalize across all of them. We define ten different binding tasks spanning domains, syntaxes, and subjects: one with m 2 and nine with m 3. The sizes of the entity sets range from 23 to 80. Table the entity sets for each task, along with an example instantiation for
and different Quadit 2. These are the two entities in the entity group which aren't wen't the target entity. For example, when featity = 2 1 and qemityy 3.
values of Gemity. Note that when m = 3, we use two query entities: G Ggoup and G G9oup
Geoiig
set 9entity"
Figures 8 and 9 show the results of the TargetRebind interchange intervention gemma-2- 2b-it and qwen2.5--7-it across all tasks and all values of feamy. Our findings are consistent: the positional mechanism dominates for early and late entity groups, while the lexical and reflexive mechanisms take over in the middle. We also replicate the effect Figure[lan Figure reflexive more present when featiyy (first), lexical when featity 3 (last), and both are balanced when fentit = 2 (middle).
A.2 REPLICATING RESULTS ACROSS MODELS
To validate robustness, we evaluated 9 models across 3 families, spanning 2-72B parameters. As shown in Figures 10 and 11 for the boxes and music tasks with fentity E [3], our findings transfer consistently across models. The positional mechanism dominates for the first and last entity groups, while in middle positions lexical and reflexive take over, with a mixed effect distributed around the positional index. In the Qwen family, we also observe that positional efficacy strengthens with model size. Overall, these results point to a universal strategy used by LMs to solve entity binding tasks.
A.3 EFFECT OF 72
Previous work has described model behavior faithfully using only the positional mechanism (Feng & Steinhardi 2024; Prakash et al. 2025), but these analyses were limited to small contexts (n e (2,3)). this work, we show extensively that this finding doesn't 't hold for larger values of n. To evaluate exactly the relationship between the efficacy of the positional mechanism and n, we conduct the TargetRebind interchange intervention on gemma-2-2ba
and qwen2.5-7b-it for that the trend seen in all experiments holds across values of n: the positional mechanism is effective in the first and last entity groups, but not middle increases. This trend is consistent with the separability analysis in Figure which shows that hidden states from middle entity groups become increasingly difficult to classify by position grows.
all [3, 20). We see in Figures 14]
15
16 77
ones. Its efficacy for middle entity groups declines
13

---

Published as a conference paper at ICLR 2026
Ann
(C) Pointer is dereferenced and answer entity is retrieved
(B) Reflexive pointer propagates to the last token position
Reflexive pointer copied bound entity
Joe
loves
ale
Ann loves
pie
Pete loves jam Tim loves tea Who loves
pie
Figure 7: An illustration the reflexive mechanism for retrieving entities for Lemity 1. We omit positional and lexical mechanisms for clarity. Under this mechanism: (A) reflexive pointer to entity originating from copied to its bound counterpart, (B) that reflexive pointer propagates to the last token position through the query entity, and (C) that pointer is dereferenced, thus retrieving the answer entity.
B ENCODING OF POSITIONAL INFORMATION
Throughout our experiments (notably Figures 2 S and 8) we show that the model does not rely solely the positional mechanism. One possible explanation is that, illustrated in Figure 3 the positional signal becomes diffuse and weak for middle entity groups. This may reflect the model's limited ability encode entity group positions in linearly separable manner. To test this hypothe- sis, we collected hidden state activations at entity token positions as well as at final token positions at every layer, and assessed their separability using PCA and multinomial logistic regression probe. Figure [22] shows the results: PCA projections for entity token positions with n 20, and linear probe accuracies for both entity and final positions across n (5, 10, 15, 20}. Consistent with our broader findings, the first and last entity groups are readily separable, while middle groups ex- hibit substantial overlap. We also observe clear dependence on n: smaller contexts yield better separation, whereas larger contexts make positions increasingly indistinguishable.
C BINDING SIGNALS IN ENTITY TOKENS
In our main experiments, we focus interchange interventions for the last token position, showing that it encodes positional, lexical and reflexive signals. this section, we conduct experiments to verify the existence of these signals in the entity token positions themselves, as well as identify the movement of these signals across token positions.
First, we conduct the Posswapp, [LexSwap and Re Re interchange interventions, described in Table 2 with the results shown in Figure 19 We see that they achieve nearly identical inter- change intervention accuracies as when performing (aa Rebi nd with the last token position. Additionally, we see that the positional and lexical mechanisms, the crucial layers where the binding information is contained in the entity tokens and used for retrieval are layers 12-19, while for reflexive it's 6-12.
To further trace how binding signals flow through the model, we apply attention knockout (Geva et al. 2023). We first identify a minimal set of layers where blocking attention from the last token position the query entity token (e.g., which box is the medicine in?) degrades performance. Across all values of femmy. this occurs in layers 11-16, dropping accuracy from 98% to 37%, aligning with the layers where binding information resides in entity tokens. Knockout becomes even more
14

---

Published as a conference paper at ICLR 2026
Patch Bitecs
postonn
meee]
refeeee
lsxcal
progurmning
Figure 8: Results of the TargetRebind interchange intervention for gemma-2-2b-it across all tasks and possible values of temity.
effective when applied to both the query token and the token immediately after it, reducing accuracy 8%. This suggests that some the query signal is copied forward. Consistent with this, blocking attention from the last token position only to the token following the query token decreases accuracy by just one point. However, when we block attention both from last position to the query token and from its following token, accuracy drops 6%, confirming that crucial binding information reaches the last position via the query token.
Finally, we test whether binding information propagates from entity tokens to the query token. The lexical mechanism may not require such propagation, since its signal can be generated directly from the query token. By contrast, the reflexive signal in the entity tokens originates from the answer token, so the query token must retrieve in order for the signal to reach the last token position. To evaluate this, block attention from the query token different entity tokens. For the reflexive signal (setting femityy 1), we block attention to the entity token identical to the query token- where our fSwap intervention localized the signal-and and to the token immediately after it. This intervention is most effective in layers 8-12, reducing accuracy to 6%, and matching the layers where entity tokens use this signal (Figure 19 Blocking attention other entity tokens in the queried entity group has no effect. In contrast, for the lexical signal (setting femily = 2), blocking attention from the query token to the correct answer entity token reduces accuracy only to 86%, even when applied across all layers. Moreover, blocking attention from the query token to all entity tokens at all layers still leaves accuracy at 90%. These results support our hypothesis: the lexical signal can be derived locally from the query token, while the reflexive signal must be retrieved from entity tokens. This also explains why the model appears to produce the reflexive binding signal earlier than in the lexical positional mechanisms requires additional stage of retrieval.
15

---

Published as a conference paper at ICLR 2026
Patch Bitccs
postons
meee]
refeeet
lsxcal
progunmiigg
Figure 9: Results of the TargetRebind interchange intervention for qwen2.5-7b-ir across all tasks and possible values of temity-
Facce Blids
Patch Bliids
gerrii- Laager
perrra- Layer
garmss 32b- Layer
permia Layer
24tora-2 (Lager
gerrre Layer
5*
212
Esz
Ower2.5 1o00-57244uubuee bubse Layar 12 Jame- 1-00 surrect Laper 1o.kmaa rutruct Layer 36
5.
Owenn. (aw00-57244ubaaa Layer 42 Jame- 1-00 murruct Laper Lolamad 3amaaaaa 5-732 ratruct Layer
gsa
ss
g4
210
Zso
Exchs Pacched Endiy
15
Fasched
Jaches
Paxched
Flached
Figure 10: Evaluation of the (Target Rebind interchange intervention in different models across model families spanning 2-72B parameters, for the boxes binding task and lentity E [2]. We that the results remain remarkably consistent.
D ADDITIONAL EXPERIMENTS
In this section discuss experiments that further strengthen model of how LMs perform entity binding and retrieval, that couldn't be included the main section. $D.I we expand our under- standing of the interaction between different mechanisms by evaluating what happens when setting
16

---

12 14 16 Published as a conference paper at ICLR 2026
Patch Effects
positional nal
mixed
reflexive
lexical
Hommaa---bb-e Layer
gemma-2-9n0 Layer
gemma-2-27b-it Layer
Patch Effect
0.8
0.6
0.4
0.2
0.0
Qwen2.5-3B-Instruee
Layer 31
Qwen2.5-78 Inntttruc- Layer 22 Qwen2.5-324 328-Instruct Layer 51
1.0
Patch Effect
0.8
0.6
0.4
0.0
Qwen2.5-722 --tttntt2t
Layer 62
Llama-3. 3.1-88-Insttuct Layer 16 Llama-3.J. 3.1--0.-111-11111 --17700--nsttuct Layer 35
1.0
Patch Effect
0.8
0.6
0.4
0.2
0.0
12 14 16 18
2 4
12 14 16
2
Patched Entity Group Index
Patched Entity Group Index
Patched Entity Group Index
Figure 11:
Evaluation of the [TrrgeRebbind interchange intervention in different models across 3 model families spanning 2-72B parameters, for the music binding task and fentiyy 3.
Finch Officts
parmaz 2-6-22-----
permma-3-2b-t Laper
22200 Layer
5-1-255--- Lage: 32 5- 5-1-mitttt Layer
tonni lave
Owent
Gees25 malivtl Layer
Derd55-2--tubbee Leya
5
sc
2
Exdty Jenes Indee
#aDOTed Erily G004 indec
Extty Gmopphaae
Aaccha
Ba0THeS Eaily index
Figure 12: Evaluation of the (TargetRebind interchange intervention at 1 and layers after the evaluation in Figure TO, for fentity = 2. We that the model shifts from aggregating binding information to retrieving The entities.
two mechanisms to point at the same entity. In DD2 we detail the experiments conducted for finding the target layer for our interchange interventions. Finally, in $D.3 we analyze the model's behavior when removing entities pointed to by the different mechanisms.
D.1 MECHANISM AGREEMENT
In the TargetRebing ind interchange intervention used to produce the results in Figure (and others throughout the paper), explicitly make to have different values for the positional, lexical and
17

---

Published as a conference paper at ICLR 2026
Name
Entity Sets Sample
Binding Example
Filling Liquids
E John, Mary]
John and Mary are working at a busy restaurant. To fulfill an order, John fills cup with beer and Mary fills a glass with wine. Who filled cup with beer?
82
cup, glass}
E3
wine, beer)
People and Objects
Σ1 (John, Mary)
John put the medicine in the office and Mary put the toy in the kitchen. What did Mary put in the kitchen?
&2 loy, medicine) [kitchen, office)
Σ3
Programming Dictionary
a. by
The following are dictionary variables Python: a= ("name""'Mary "Country":"'e :Caandda"" ). b= b=["nand ("name': :oomee''Jonn', 'Country":
32 (John,Mary) Mary}
E3 = { US, Canada)
What is the country in variable b where "name" = 'John'?
Music
Σ (John, Mary}
At the music festival, John performed rock music on the piano, and Mary performed pop music on the guitar. What music did Mary play on the gui- tar?
82 rrokk pop) guitar, guitar,piano)
Biology Experiment
E1 JJhnn, Mary]
biology laboratory experiment, Mary placed the serum in a vial, and John placed the enzyme in beaker. Who placed the serum in a vial?
Σ2 serum, engyme)
Σ3
beaker, vial}
Chemistry Experiment
Σ JJhhn, Mary)
In a chemistry laboratory experiment, Mary added the acetone to crucible, and John added the ethanol to funnel. What did John add fun- nel?
&2
ethanul, acetone) crucible, funnel)
E3
Transportation
Σ (John, Mary) E2 truck, taxi)
In a city transportation system, John drove the truck mall, and Mary drove the taxi to the park. Where did Mary drive the taxi?
E3 { mall, parll
Sports Events
Σ John, Mary]
a sports competition, Mary played hockey at the stadium, and John played cricket at the field. Who played hockey at the stadium?
E2 =
hockey, cricket)
Σ3 stadium, field)
Space Observations
Σ1
JJhnn Mary]
During an astronomy study, John observed an as- teroid with radar, and Mary observed planet with telescope. What did John observe with radar?
82
planet, asteroid)
E3
telescope, radar)
Boxes
Σ, (toy, medicine)
The toy is in box B, and the medicine in Box A. Which box is medicine in?
Ea bbx A, box B)
Table I: List of all binding tasks we evaluate in our experiments. We show entity sets composed of only two entities per set for brevity. We also only show examples for but evaluate over E [3, 20]
reflexive indices, so that we can know which mechanism most affected the model's output. However, as shown in Figure 3, these mechanisms behave additively, and we suspect that when they agree, they overwhelmingly drive model behavior. To evaluate this, we conduct two experiments, one for lentiyy = 1 where the positional mechanism agrees with the reflexive one, and one for lentity where the positional mechanism agrees with the lexical one. The results, shown Figure [8] show that this is indeed the case.
D.2 FINDING THE TARGET LAYER
We seek to identify for each model what the last layer before retrieval is, so that we can perform our interchange interventions on that layer. Indeed Figure 2 we see that there are a subset of layers where the last token position contains the binding information, after which it contains the retrieved answer. Thus, for each model we identify the last layer where patching the last token position does not copy the retrieved token. The intervention this layer is shown in Figure 10 for teaut E [2], and in Figure [2 we show this same intervention for & + and f with fentlt 2. We see clearly that for f, the percentage of cases where the answer post-intervention is the retrieved entity from
18

---

Published as a conference paper at ICLR 2026
p = 0
Ip = 2
ip 3
ip = 4
20
/p 5
1p
1p 7
1p
1p 9
Mean Logit Value
20
10
1p = 10
ip 11
ip = 12
ip 13
ip = 14
1p 15
1p 16
ip = 17
ip 18
1p = 19
20
10
1215
1518 Entity Group Index
121518
Figure 13: The mean logit distribution as a function of the positional index (ip), for qwen2.5-70 it on the boxes task with eetity = 2. We can see the positional binding signal induces strong and concentrated signal for entity groups in the beginning and the end, while inducing weak and diffuse one for middle groups.
gemma-2-2b-it boxes
postrisna
relledivv
Insical
1.
n=3
n=4
5=5
n=6
n=7
n=8
Dar pleet
n-10
7a0th Effeet
2.0
12
10 12
14
14
10
0 20
Figure 14: Results for the TargetRebind interchange intervention on 1amma-2--bbit for n [3, 20] and feniiy = 3 on the boxes task. We see a trend where, the more entity groups need to be bound context, the worse the positional mechanism binding those in the middle.
the counterfactual example is at or below random chance. However, for & + and + 2 this effect becomes the majority, showing that the model has shifted to retrieval. We also see that this layer is consistent across tasks in Figures 10 and
D.3 REMOVING TARGETED ENTITY TOKENS
In we detail how the lexical and reflexive mechanisms are pointers that get dereferenced to the queried entity. To strengthen these claims, in this section, evaluate what happens when we modify the TargetRebing interchange intervention, such that the entities targeted by those mechanisms do not exist in the original prompt. Thus, for the example in Figure to test the lexical
19

---

Published as a conference paper at ICLR 2026
gemma-2-2b-it music
n=3
n=4
n=5
n=
n=7
n=
Peres pite
n=9
n=10
n=11 n=12 n= n=14
2.0
(ffect
Parth Effect
2.0
14
Figure 15: Results for the [TargetRebing interchange intervention on gemma-2-2b-it for n E [3, 20] and fentity = 3 on the music task. We trend where, the more entity groups need to be bound context, the worse the positional mechanism at binding those in the middle.
Qwen-225-78-instrucct boxes
postionn
refleiive
Iesical
13
n=3
n=4
n=5
n=6
n=7
n=8
Parm tttet
n-11
PAr Frieet
12
2 14
22411R 22
4.4.6 6.665 1030
Figure 16: Results for the TargetRRbiind interchange intervention on qwen2.5-7b-i for n [3, 20) and featity = 3 on the boxes task. We see a trend where, the more entity groups need to be bound context, the worse the positional mechanism binding those in the middle.
mechanism we'd change the counterfactual such that Ann is replaced with a different new name Max, and for the reflexive we' 'd change pie to cod (separately). We see in Figure 28 that this leads the model rely solely the positional mechanism, since the others have pointers that cannot be dereferenced. In Figure 29 we see that in this case, relying on the positional mechanism yields noisy distribution around the positional index.
A possible alternative explanation for why the model isn't retrieving the entity pointed to by these two mechanisms, that there might be some other mechanism that prevents the model from an- swering with entities that do not exist in the context. To evaluate this, we conduct the same exact interventions, but for layer + 1, where the retrieval is already taking place (see $D.2). Thus, such a mechanism exists, we'd expect to see the same results, where the model relies solely on the positional mechanism. Otherwise, we'r expect the model to respond with the retrieved answer from the counterfactual. We can see in Figure 30 that the model indeed responds with the retrieved answer from the counterfactual, falsifying this alternative explanation. Thus, we conclude that the model indeed relies the lexical and reflexive mechanisms as pointers for dereferencing.
20

---

Published as a conference paper at ICLR 2026
Qwen-225-77-instruuc music
n=3
n=4
n=5
n=
n=7
n=
Pares pite
n=9
n=10
n=11
n=112
n= n=14
Parth (ttect
n=
Parmh Effect
4.0
14
20321430115
Figure 17: Results for the TargetRebing and fentity = 3 interchange intervention on qwen2.5- 7b-it for [3, 20] the music task. We see trend where, the more entity groups need to be bound context, the worse the positional mechanism at binding those in the middle.
1.0
1.0
Potch Bffcks
Patch Effects
mixed
lecie
0.8
0.8
Patch Effect
Patch Effect
0.6
0.6
0.4
0.4
0.2
0.2
0.0
4
8
10
12
16
18
0.0
0
2
10 12
14
16
18
Patched Entity
Index
Patched Entity Group Index
Figure 18: We evaluate gemma-22 -aaena---2bbae 2-22b-it's behavior when aligning the mechanisms for the music task. We align the positional and reflexive mechanisms for fentiyy = 1, and the positional and lexical mechanisms for lentity 3. see that when the mechanisms point at the same entity for retrieval, the model consistently responds with the correct entity.
D.4 LINGUISTIC VARIABILITY
To assess our findings' generalization beyond the templatic datasets defined in Table we incor- porate linguistic variations in the phrasings of each entity group. We define 12 such variations for the boxes and music tasks respectively, such that when creating prompt from a binding matrix G, choose random variation per entity group. For example, in the boxes task, we have variations like "the Object is stored in box Box", "the Object was left in box Box" and "the Object ended up box Box". We can see in the results, shown in Figure 2T] that our findings remain identical in this setting. As in previous results, the model relies on the three mechanisms, mediated by the entity group index as well as fenity-
E ADDITIONAL CAUSAL MODELS
We report the KL divergence scores for gemma-2-2b-in on the music task in Table We additionally report all metrics for (emma-2-2b-ij on the sports task in Table 4 and qwen2.5-74 7b-it it both tasks in Tables 5 and For training, we use Adam (3, = 0.9, 32 = 0.999) with learning rate 0.05, run for up 2,000 epochs with batch size of 512 and early stopping after 200 epochs.
21

---

Published as a conference paper at ICLR 2026
Positional
Lexical
Reflexive
4-
0.
8-
0.4
0.30
12
0.4
0.2
0.15
02
0.30
12 Positional Index
14
14 Reflexive Index
Lexical Index
Figure 19: Results for the (PosS (left), LexSwap (middle) and (RefSwap) (right) inter- change interventions on gemma-2-22 2-2na- for the boxes task. Each square shows the interchange tervention accuracy (IIA) for given layer and positional, lexical or reflexive index. We see that positional and lexical binding information exists in entity tokens in layers 12-19, while reflexive binding information does in layers 6-12.
Patch Effects
0.8
positional
Patch Effect
mixed
lexical
0.6
0.4
0.2
0.0
2
10 12 Original Entity Group Index
16
18
4
10 12
18
Lexical Index
Figure 20: show results of the TargetRebind interchange intervention on gemma-2-2U8 -t--it for the boxes task with different indices on the x-axis. Left: using the index of the queried entity group. This has little effect overall, except for dips at the first and last indices in the positional effect. Under StargeRRebind the positions of queried entity groups cannot coincide between the coun- terfactual and original prompts. Thus, when the original query targets the first or last groups-where positional information is strongest-these groups never patched, slightly weakening results average. Right: using the lexical index. Here the pattern mirrors Figure 3 with weaker effects at the edges and stronger ones in the middle.
FURTHER VALIDATION OF THE REFLEXIVE MECHANISM
In $..1 we describe the reflexive binding mechanism, where direct pointer originating from an entity token is used to point back at itself. In this section we provide further evidence for the existence of this mechanism as described.
We do this by knocking out attention from the last token position to the target entity (Geva et al., 2023), shown in Figure 23 Again we see that the model does not respond with the counterfactual target entity unless it can find in context, which we prevent by blocking attention to it. Conversely, blocking attention at a layer when the model has already retrieved the answer, while patching at that layer, does not prevent the model from answering with the answer entity from the counterfac- tual. Thus, we can conclude that the model indeed relies on a reflexive mechanism for binding and retrieving entities in context.
22

---

Published as a conference paper at ICLR 2026
Name
Original
Counterfactual
Patch Positions
Patch Effects
Target
The bottle is in box C, the pen is in box A, the ball box Q, and the rock is in box N. Which box is the rock in?
The bottle is in box Q. the ball is in box A, the pen is box C, and the rock is in box N. Which box is the pen in?
Last token position
Q: Positional
A: Lexical
C: Reflexive
effect
PosSwap
The pen is in box and the ball is in box Q. Which box is the ball in?
The hall is in box Q and the pen in box A. Which box is the ball in?
A-+A
Q-+
A: Patched tokens encode positional binding used by the model
Q: Patched tokens do not encode positional binding used by model
LexSwap
The pen box A and the ball is in box Q. Which box is the ball in?
The ball box A and the pen is in box Q. Which box is the ball in?
A-AA Q+Q
Patched tokens encode lexical binding used by the model Patched tokens not encode lexical binding used by the model
Re (Sap
The pen is in box A and the ball is in box Q What is in Box Q?
The ball is in box A and the pen is in box What is in Box Q?
A-A
Q+Q
A: Patched tokens encode reflexive binding used by the model Q: Patched tokens do not encode reflexive binding used by model
Table 2: Original/counteeraccual pair examples for interchange interventions.
1g
Ia.m2
8.2
0.2
52-
1.0
6.4
0.8
32-
222
5.2
02
02
Entity Group
Figure 21: Results for the (Trret interchange intervention on qwen2.5-7b-it and gemma-2-2b-it for all values of Gentity on the boxes and music binding tasks, while using random linguistic variations for the phrasings of each entity group. We find that our findings remain consis- tent in this setting as well.
G CONTEXT LENGTH ABLATION STUDY
In our evaluations, we show the effect of the number of entities that need to be bound in-context on LMs' use of the positional, lexical and reflexive mechanisms. However, a confounding factor is that the number of entities increases,
does the length of the sequence itself. To disentangle these effects, we pad contexts with n E [3, 19] so that all sequences match the length of those with 20. Padding is done using "entity-less" sentences, described in 5 The results are shown
23

---

Published as a conference paper at ICLR 2026
Endly Pas NN+5
Endoy Pos (N=10
Entryy Pos
(N-15
Fos IN-20
Taple
Lest Pox No-10
N--15
Ni-2
Tap6a 1r
Tasls Trile
Tacio Irdcce
Tasio Irdic
Figure 22: Separability of hidden states for entity token positions and the last token position, across layers and values of n. PCA projections (left) and multinomial logistic regression probes (right) show that first and last entity groups are linearly separable, while middle groups overlap substan- tially. Separability decreases as the number entities n increases.
Model
KLelpp
KL +
leniiyy
lentity
2
leiity
3
lentiyy
lentity
lentity
M
0.22
0.17
0.26
0.31
0.21
0.41
M w/ Oracle Pos
0.14
0.08
0.14
0.32
0.11
0.24
M w/ One-Hot Pos
0.71
0.67
0.71
1.00
0.88
0.88
Only One-Hot
6.41
5.61
5.95
3.41
2.39
2.78
Pos
M (P)
1.75
1.52
1.71
4.51
2.37
3.17
M {L}
0.39
0.73
1.76
0.3
0.37
1.42
M {R}
2.14
1.08
0.61
2.10
0.54
0.44
M {L, R}
2.10
1.22
1.82
2.13
0.73
1.50
M (P,,
9.19
7.35
5.32
10.7
5.55
4.34
M' (P,L)
4.66
6.18
8.45
4.28
2.92
5.40
Uniform
2.71
1.96
2.44
7.57
3.49
4.84
Table 3: KI. divergence results for modeling an LM's behavior contingent on the positional, lexical and reflexive indices. Evaluated gemma-2-2b-it for the music binding task. Our full model achieves the best performance, only slightly below the oracle.
the effects of increasing n were due to increasing sequence length, we'd expect all results to be identical to when setting distribution patch effects is slightly affected by padding, the results and trends align closely with results without padding. This indicates that model behavior is governed primarily by the number of entities that must be bound, rather than by sequence length.
Figure 27
20, and to each other. However, we see that, while the
H LLM USAGE
this work, the authors relied on LLMs solely to assist with implementing specific helper functions.
24

---

Published as a conference paper at ICLR 2026
Layer
Layer
1.0
1.0
0.8
0.8
Patch Effects
posttional
reflexive
Patch Effect
Patch Effect
texical
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0
6 10 12
14
16
18
0
2
8
10
12
14
16
18
Patched Entity Group
Patched Entily Group Index
Figure 23: Patch effects under (TargeReebind for gemma-2-2b-it while blocking attention to the target entity. Left: blocking attention when the model accumulating binding information in the last token position leads to it not being able to dereference the reflexive pointer. Had the patch contained the retrieved answer, this plot would be fully orange. Right: patching at the following layer and blocking attention to the target entity, Here the plot is fully orange since the entity has already been retrieved.
lexical
positional
reflexive
lexical
positional
reflexive
jp 2
2
jR
10
12-
Mean Logit Value
Mean Logit Value Value
ip = 6
1p 8
ip 10
f 6
iR = 8
JR = 11
23-
1p
1p 16
1 19
iR
14
16
iR 19
12-
12
Entity Group Index
Entity Group Index
Figure 24: Mean logit distributions under TargetRebin nd for gemma-2-2b-in on the music task. Left: fixing ig = 8, in = 16 and varying êp. Right: fixing ip = 6, it = 14 and varying in.
lexical
positional
reflexive
lexical
positional
reflexive
h =
1 2
iR
iR
JR
Mean Logit Value
13
8
11
8
iR 11
Mean Logit
1 = 14
i = 16
19
AR 14
Ía 16
iR = 19
12
16
12
12 16
12
Entity Group Index
Entity Group Index
Figure 25: Mean logit distributions under Rebget for qwen2.5-7b-in on the music task.
Left: fixing ip 6, ig 14 and varying iz. Right: fixing ip 6, it 14 and varying ig.
25

---

Published as a conference paper at ICLR 2026
lexical
positional
reflexive
lexical
positional
reflexive
Öp 2
2
jR
22
23-
Mean Logit Value
Logit Value
1p =
1p 8
1p 10
fR 6
AR 8
jR = 11
15-
14
1p 16
19
14
16
ÍR 19
30-
23-
4
12
16
12 Entity Group Index
16
12 16
4
12 16
12
Entity Group Index
Figure 26: Mean logit distributions under TargetRebind for qwen2.5-7b-it on the sports task.
Left: fixing it = 8, in 16 and varying ip. Right: fixing ip = 88 éz = 19 and varying in-
patch
positional
lexical
reflexive
Patch Effect
mixed
4
5
7
9
10
11
12
13
14
15
16
17
18
19
Figure 27: Mean patch effects per number of entities in context (z).. For each n, report the standard mean patch effects (right) alongside results from padded sequences (left), where sequence length is fixed to match overall patterns remain consistent: model behavior primarily controlled by the number of entities in context, rather than sequence length.
20. While padding slightly shifts the distribution of patch effects, the
26

---

Published as a conference paper at ICLR 2026
Fentity
Tentty
1.0
1.0
0.8
0.8
Patch Effects
positional
reflexive
Patch Effect.
Petch Effect
lexical
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
14 Patched Entily Group Index
Patched Entily Group Index
Figure 28: Left: results for (Targe interchange intervention gemma-2-2b-it with fentity = 1, where the query entity in the counterfactual does not exist in the original. Right: results for Target Reb nd) interchange intervention on semma-2-2b-it with lentity = 2, where the target entity in the counterfactual does not exist in the original. We see in both plots that when the model can't t use the lexical and reflexive mechanisms since the entities they point to don't exist, the model falls back to solely using the positional mechanism (distribution showed in Figure
Caetty m1
Castity 2
400
300
45
50
22
G
O
40
Positional Index
40
Nercebagg (S)
28
0
R-
34
8-0
11
10
35
37
Prediction Index
Prodiction Index
Figure 29: Left: confusion matrix for non-lexical and reflexive patch effects under the Target Rebind) interchange intervention on gemma-2-2b-it with fentity entity in the counterfactual does not exist in the original. Right: results for non-lexical and re- flexive patch effects under the (TargetReb: nd interchange intervention on gemma-2-2b-it with fentiyy 2, where the queried entity in the counterfactual does not exist in the original. We can see that when the model can't use the lexical and reflexive mechanisms, it falls back on the noisy positional mechanism.
where the query
27

---

Published as a conference paper at ICLR 2026
Fanbity
lanaty
1.0
1.0
0.8
0.8
Patch Effects
posiiional
reflexive
Patch Effect
0.6
0.6
Patch
0.4
0.4
0.2
0.2
0.0
0.0
6
10
12
14
16
18
0
8
10
12
14
16
18
Patched Entity Group
Entity Group
Figure 30: Results for [argetRebind: interchange interventions on gemma-2- --2--iit with fentity [2]. where the query entity (left) or queried entity (right) the counterfactual do not exist in the original, patching at layer + 1. We see that the model copies the retrieved answer from the coun- terfactual, showing that no mechanism exists to suppress answering with entities that do not exist in the original prompt.
2
276lt
F
0wan2.5-38 instruct
Qwer2.5-PP 5-78
Owen2.5 222-2255-2---000 fuct
Qwen2.5 38 Instrac
Qwen2: 2.782.5-78-... instrrst
Qwen2.5 328-instruc
Indor
Owen2 58--8---n
1-758-tnetrurt
5-72B
11-88-Indtt
Liama-3 1-7dB-tnetrutt
Pastoral
23 Predittion
Weddtion Pretintion
Presictio
Praticion
Preticion
Figure 31: Confusion matrix for non-lexical and reflexive patch effects under the Ta TaaretReb. nd interchange intervention for all models, showing the diffuse distribution around the positional index. Right: feniity 2.
Left: tentity
28

---

Published as a conference paper at ICLR 2026
Model
JSS t
KLelpp 4
KL 4
1
/
2
1=
3
1=
1
2
1=3
1=
1 2
1=
M (Lone-hot>
0.94
0.95
0.93
0.3
0.21
0.31
0.35
0.28
0.39
Rooe-hott Piiuss)
M w/ Poracle
0.96
0.97
0.96
0.13
0.11
0.13
0.32
0.19
0.21
M w/ hont-bat
0.85
0.87
0.87
0.77
0.59
0.62
1.2
0.81
0.73
P one hot (prevailing
0.4
0,46
0.43
6.5
5.74
6.03
3.72
2.54
2.9
view)
M {P}
0.67
0.69
0.71
1.75
1.53
1.49
4.91
3.04
1.88
M \ {L}
0.93
0.89
0.75
0.41
0.8
1.99
0.37
0.53
1.25
M \ (R)
0.69
0.84
0.9
1.83
1.28
1.04
2.52
0.71
0.47
M. RL
0.68
0.79
0.74
1.84
1.52
2.06
2.54
1.05
1.36
M (P,,
0.11
0.31
0.47
9.25
7.19
5.32
10.9
6.11
3.61
M
(P,L)
0.55
0.45
0.23
4.71
5.95
8.16
4.64
3.06
4.49
Uniform
0.45
0.54
0.54
2.66
2.13
2.22
4.78
2.93
Table 4: Results for modeling gemma-2-2b-it's behavior the sports binding task, contingent the positional, lexical and reflexive indices. Here denotes fentity
Model
JSS f
KL 4
KL KL plls 4
t=
= 2
t=3
3
1
=
2
t= 3
t=
1 t= 2
t=
M
(Lons-hots
0.94
0.92
0.92
0.27
0.34
0.37
0.35
0.48
0.45
Rage Beuse-hord Giauss)
M w/ Poracle
0.98
0.98
0.98
0.07
0.07
0.07
0.1
0.1
0.1
M w/ Pone
0.87
0.89
0.88
0.58
0.47
0.53
1.09
0.74
0.69
P one bot (prevailing view)
0.56
0.55
0.53
4.66
4.65
5.01
1.84
1.88
2.04
M [N]
0.62
0.66
0.66
1.85
1.68
1.73
5.05
3.29
2.55
M (LL
0.93
0.85
0.78
0.51
1.14
1.77
0.33
0.74
1.09
M (RR
0.86
0.9
0.9
0.9
0.89
1.0
0.68
0.54
0.5
M' RL,
0.86
0.84
0.78
0.92
1.24
1.79
0.71
0.83
1.11
M (
(P,R))
0.16
0.34
0.42
8.43
6.69
5.93
9.04
5.6
4.4
M' (P,L)
0.35
0.24
0.17
6.64
7.8
8.57
6.8
5.3
5.32
Uniform
0.55
0.58
0.54
2.11
1.95
2.21
5.92
4.09
3.54
Table 5: Results for modeling qwen2.5-7b-it' behavior on the music binding task, contingent on the positional, lexical and reflexive indices. Here f denotes temmy.
29

---

Published as a conference paper at ICLR 2026
Model
JSS f
KLejpp
KL $
1
2
=
3
1
==
t=3 3
t=
t= 2
t=
M
(Tone hot
0.95
0.93
0.92
0.24
0.31
0.36
0.28
0.39
0.47
Reae-hot- PGauss)
M w/ Poraele
0.98
0.98
0.97
0.07
0.08
0.11
0.09
0.11
0.18
M w/ Pone-bot
0.87
0.89
0.88
0.62
0.52
0.55
1.14
0.68
0.84
P Pone-bot (prevailing view)
0.57
0.55
0.51
4.58
4.72
5.18
1.73
1.84
2.21
M {P}
0.61
0.66
0.66
1.88
1.75
1.73
5.11
2.9
3.35
M {L}
0.94
0.87
0.77
0.53
1.09
1.64
0.27
0.64
1.27
M {R}
0.87
0.89
0.91
0.89
1.05
0.85
0.57
0.54
0.48
M RL
0.87
0.83
0.77
0.92
1.33
1.65
0.6
0.82
1.29
(M,P,
0.17
0.31
0.44
8.39
7.01
5.75
9.04
5.54
4.78
M [P,L)
0.33
0.32
0.14
6.77
7.27
8.77
6.97
3.88
7.27
Uniform
0.54
0.56
0.53
2.12
2.06
2.25
6.01
3.9
4.71
Table 6: Results for modeling qwen2.5-7b-it's behavior the sports binding task, contingent the positional, lexical and reflexive indices. Here denotes feutity
30


Reviews:
[{"review_id": "cIHoYTwrDF", "reviewer": "Reviewer_zu1T", "summary": "This paper investigates how language models (LMs) retrieve bound entities in context, e.g., “Pete loves jam, Ann loves pie. Who loves pie?”. The authors identify that retrieval arises from a mixture of three mechanisms: positional, lexical, and reflexive.\n- Positional: retrieves the target based on the position of the entity group corresponding to the query.\n- Lexical: retrieves the entity bound to the queried token itself (e.g., “pie” → “Ann”).\n- Reflexive: uses a direct pointer to the target entity token.\nThey construct a controlled dataset with paired _original_ and _counterfactual_ examples (with entities shuffled), to identify each mechanism via patching experiments. This setup allows them to separate the contribution of each mechanism to the LM’s predictions. They further show that mixing these three mechanisms explains model behavior in longer contexts where the positional mechanism alone fails.  \nFinally, they build a simple causal model incorporating these mechanisms, which closely reproduces the LM’s next-token distribution (≈0.95 Jensen–Shannon similarity).", "questions": "Dataset\n- Could you specify the range of sizes for the entity set $\\mathcal{E}$ in the dataset ? \n\nFigures\n- In Figure 2, what exactly does the _y_-axis represent? Is it the proportion of examples where that mechanism predicts the correct answer? The term “index” on the axis is confusing.\n- What does “mixed” mean in this context?\n- Are cases excluded where the patching does not predict a valid entity (i.e., none of the entities in the list)? Or does this never happen?\n\nReflexive mechanism\n- In the main text, you mention patching into a context where the target entity is not present, to test the reflexive mechanism. However, it still seems possible that the patched representation already contains the target token’s activation, and that in the new context other mechanisms simply suppress it because the answer is implausible. Could you clarify why this result indicates the presence of a direct pointer, rather than the target token being carried over as a side effect of other retrieval processes already in progress?\n\nCausal Model\n- When training the causal model variants that exclude one mechanism, are the weights of the remaining mechanisms retrained independently, or are they frozen from the full model? Clarifying this would help interpret the ablation results.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- **Clarity:** The paper is clearly written and well-organized. The motivation, hypotheses, and experimental setup are clearly presented with concrete examples.\n- **Originality:** The discovery of two previously undescribed retrieval mechanisms (lexical and reflexive) extends the mechanistic interpretability literature beyond the known positional mechanism.\n- **Quality:** The experimental methodology, which uses counterfactual patching and causal modeling, is rigorous and carefully justified.", "weaknesses": "- **Figures:** Some key figures (e.g., Fig. 2) are difficult to interpret. The axes, metrics, and what constitutes “mixed” effects are not clearly explained in the captions or main text, making it hard for readers to connect the visualization to the described mechanisms.\n- **Presentation of reflexive mechanism:** The explanation and evidence for the reflexive mechanism remain somewhat unclear. Because the patched context may already contain the predicted token, it is hard to disentangle whether the observed effect truly demonstrates a “pointer” or simply reflects ongoing retrieval from other mechanisms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1Q49BJvB42", "reviewer": "Reviewer_U696", "summary": "The authors explore multiple mechanisms by which transformers LMs track entities in context to retrieve factual information. It's already known that models will track the positions of entities and use this mechanism to answer factual questions. The positional mechanism, however, breaks down when there are many entities in context. The authors find that transformers have two other mechanisms: a lexical mechanism where the model looks up the queried entity and a reflexive mechanism where a model looks up information about a promoted token. These mechanisms can all disagree about the answer to a query, which the authors show with interventional experiments. They also create an effective model of this three-mechanism behavior that closely matches the output distributions of the LMs they test on.", "questions": "- It seems like the model uses the reflexive pathway as a \"verification\" step. Does the reflexive pathway need to match the entity from the lexical pathway (as it does in Figure 1) to work? Is my understanding of this correct?\n- See \"weaknesses\" above re: questions about how the interchange interventions were performed", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- It seems intuitive that something like the lexical mechanism or the reflexive mechanism would exist. It's somewhat surprising that these are separate.\n- The interventions show that the different retrieval mechanisms can disagree in their predictions and behavior on the output distribution, which is compelling.\n- The authors create an effective model of the three retrieval mechanisms which closely matches the output distribution of the actual model.\n- Polluting the context with additional free-form text is a reasonable robustness check", "weaknesses": "- I'm looking for more details about how exactly the interchange interventions were performed and how the authors localized where to do interchange interventions. From the appendix, it seems like this is based on performing interchange interventions on the attention at specific layers on the last token position? And the authors use attention knockout to figure out which layers are passing entity information to the final token position? Can you clarify?\n- The main body of the paper would benefit from a bit more explanation of what exactly is going on here (I'm aware of space constraints, but I think this is important). And the appendix would benefit from a clear, high-level description of what the experiments are actually doing.\n\nMinor suggestions and feedback:\n- Putting the counterfactual input on top in Figure 1 feels confusing, seems more natural to have the original on top and the counterfactual below?\n- I found the text description of the reflexive mechanism (line 231) to be fairly hard to parse. This seems like something that is much easier to show than to say, so a figure could be helpful.\n- Line 159: use \\citet instead of \\citep", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZCXuTmteQ3", "reviewer": "Reviewer_4tWV", "summary": "This paper investigates how LLMs bind and retrieve entities in context. Prior work has suggested that LLMs rely mainly on a positional mechanism, retrieving entities by their relative position in a list of contextually bound entities. However, the authors show that this mechanism becomes unreliable when the context grows and entities appear in middle positions, reminiscent of the lost-in-the-middle effect. To explain this, the paper proposes lexical and reflexive mechanisms that complement the positional one. Through systematic interchange interventions, they find that LLMs dynamically mix these three mechanisms depending on position and entity type. They further build a causal model that combines all three mechanisms and achieves up to 95% Jensen–Shannon similarity with true model predictions.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The authors propose the lexical and reflexive mechanisms as natural extensions of how binding might operate when positional cues become unreliable. They support these hypotheses with well-designed counterfactual interventions that yield clear empirical evidence. The experimental setup, including controlled manipulations of entity positions and roles, provides high causal interpretability rather than correlational evidence.\n* The findings are replicated across nine model families and ten distinct tasks,  demonstrating robustness. The mixture model achieves 95% agreement with LLM token predictions, further demonstrating the faithfulness of the proposed framework.\n* The parallels drawn to primacy and recency biases in human memory, and the connection to the lost-in-the-middle effect, make the results conceptually relatable.", "weaknesses": "* The three mechanisms are inferred largely through counterfactual patching, but the causal independence between them is assumed rather than rigorously established. For example, lexical and reflexive mechanisms often co-occur. It remains unclear if they are distinct causal variables or correlated manifestations of shared attention dynamics.\n* Although filler text is introduced later, most analyses still rely on templated X likes Y style prompts. These may not capture linguistic variability or discourse-level entity binding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)