How do Language Models Bind Entities in Context?
Authors: Jiahai Feng
Jacob Steinhardt
Venue: ICLR 2024 poster (2024)
Keywords: Interpretability
Learned Representations
Neurosymbolic AI
Primary area: visualization or interpretation of learned representations

Abstract: Language models (LMs) can recall facts mentioned in context, as shown by their performance on reading comprehension tasks. When the context describes facts about more than one entity, the LM has to correctly bind attributes to their corresponding entity. We show, via causal experiments, that LMs' internal activations represent binding information by exhibiting appropriate binding ID vectors at the entity and attribute positions. We further show that binding ID vectors form a subspace and often transfer across tasks. Our results demonstrate that LMs learn interpretable strategies for representing symbolic knowledge in context, and that studying context activations is a fruitful direction for understanding LM cognition.

Published as a conference paper at ICLR 2024
HoW DO LANGUAGE MODELS BIND ENTITIES IN CONTEXT?
Jiahai Feng" & Jacob Steinhardt UC Berkeley
ABSTRACT
To correctly in-context information, language models (LMs) must bind entities to their attributes. For example, given a context describing a "green square" and "blue circle", LMs must bind the shapes their respective colors. We analyze LM represen- tations and identify the binding ID mechanism: a general mechanism for solving the binding problem, which we observe in every sufficiently large model from the Pythia and LLaMA families. Using causal interventions, we show that LMSs' internal activations rep- resent binding information by attaching binding vectors to corresponding entities and attributes. We further show that binding ID vectors form continuous subspace, in which distances between binding ID vectors reflect their discernabiiity. Overall, our results un- cover interpretable strategies in LMs for representing symbolic knowledge in-context, providing step towards understanding general in-context reasoning in large-scale LMs.
INTRODUCTION
Modern language models (LMs) excel at many reasoning benchmarks, suggesting that they can perform general purpose reasoning across many domains. However, the mechanisms that underlie LM reasoning remain largely unknown (Räuker al., 2023). The deployment of LMs in society has led to calls to better understand these mechanisms (Hendrycks al., 2021), I), work and when they fail (Mu & Andreas, 2020; Hernandez al., 2021; Vig et al., 2020b).
to know why they
In this work, seek to understand binding, a foundational skill that underlies many compositional reasoning capabilities (Fodor & Pylyshyn, 1988) such as entity tracking (Kim & Schuster, 2023). How humans solve binding, i.c. recognize features of an object bound to that object and not to others, is a fundamental problem in psychology (Treisman, 1996). Here, we study binding in LMs.
Binding arises any time the LM has to reason about example, consider the following passage involving two people and two countries:
more objects of the same kind. For
Context: Alice lives in the capital city of France. Bob lives in the capital city of Thailand. Question: Which city does Bob live in?
(1)
In this example the LM has to represent the associations lives(Alice, Paris) and lives Bangkok). We call this the binding problem-for the predicate lives, Alice bound Paris and Bob to Bangkok. Since predicates are bound in-context, binding must occur in the activations, rather than in the weights as with factual recall (Meng et al., 2022). This raises the question: how do LMS represent binding information in the context such that they can be later recalled?
Overall, our key technical contribution is the identification of robust general mechanism in LMs for solving the binding problem. The mechanism relies binding IDs, which are abstract concepts that LMs use internally to mark variables in the same predicate apart from variables in other predicates (Fig. 1). Using causal mediation analysis we empirically verify two key properties of the binding ID mechanism ($3): factorizability and position independence.
Turning to the structure of binding IDs, we find that binding IDs are represented as vectors which are bound to variables by simple addition ($4) in the activation space. Further, we show that binding IDs occupy a subspace, in the sense that linear combinations of binding IDs still valid binding IDs, even though random vectors are not.
"Correspondence fjiahai @ @herkeley.cdu

---

Published as a conference paper at ICLR 2024
Query:
Where does Afice live?
Activations
[x(Aii
(Prooo))
Bob )
Ta(Thastond,
Ax(Alice,
Abstract
(A=A)
*
)
Binding ID3
T T France,,)
Ta(Thailand
D
Tokens
Alice
lives in
France
Bob
lives in
Thailand
Answer: France
The Binding ID mechanism. The LM learns abstract binding IDs (drawn as triangles or squares) which distinguish between entity-attribuu pairs. Binding functions I'g and I' A bind entities and attributes to their abstract binding ID, and store the results in the activations. To answer queries, the LM identifies the attribute that shares the same binding as the queried entity.
Figure 1:
Lastly, we find that binding IDs ubiquitous and transferable ($5). They are used by every suffi- ciently large model in the LLaMA (Touvron et al., 2023) and Pythia (Biderman et al., 2023) families, and their fidelity increases with scale. They are used for variety of synthetic binding tasks with different surface forms, and binding vectors from one task transfer to other tasks. Finally, we qual- ify our findings by showing that despite their ubiquity, binding IDs are not universal: we exhibit a question-answeriigg task where an alternate mechanism, "direct binding", is used instead (šE). We release code and datasets here: https: //gith/gttt hiit.uu... com/ fam/jiia jiahai- feng/ http/binnnngggee iccr
2 PRELIMINARIES
In this section we define the binding task and explain causal mediation analysis, our main experi- mental technique.
Binding task. Formally, the binding task consists of a set of entities & and a set of attributes A. An n-entity instance of the binding task consists of a context that is constructed from n entities co. Ca-1 E & and n attributes Ap..... On-1 1 E A, and we denote the corresponding context as cex {> ag. ....... 6n-1 ++ an-1). For a context c, we use (c) and (c(c) denote the k-th entity and the k-th attribute of the context c, for k E [0, I]. We will drop the dependence on с for brevity when the choice
is clear from context.
In the CAPITALS task, which is the main task we study for most of the paper, E is set of single-token names, and is a set of single-token countrics. Quote is an example instance of the CAPITALS task with context ctxt (Alice ++ France, Bob ++ Thailand). In this context, Eg is Alice, Ao is France, etc.
Given context we are interested in the model's behavior when queried with each of the entities present in c. For any E [0, n 1), when queried with the entity Eg the model should place high probability on the answer matching A₁. In running example, the model should predict "Paris" when queried with "Alice", and "Bangkok" when queried with "Bob".
To evaluate model's behavior c, we query the LM with every entity mentioned in the context, which returns vector of log prob- abilities over every token in the vocabulary. The mean log prob metric measures the mean of the log probability assigned to the correct attribute token. Top-1 accuracy measures the proportion of queries where the correct attribute token has the highest log probability out of all attribute tokens. However, we will instead use the median-calibrenn accuracy (Zhao et al., 2021), which calibrates the log probabilities with the median log probability before taking the top-1 accuracy. We discuss this choice in SA.
binding task, we sample N = 100 contexts. For each context
Causality in autoregressive LMs. We utilize inherent causal structure in autoregressive LMs. Let an LM have Plaaers transformer layers and dmode dimeesiiiaa activation space. For every token position p. we use Zp E Illlaeexxduus to denote the stacked set of internal activations' at token (see Fig. 2a). We refer to the collective internal activations of the context as Zeontext- denote the activations at the token for the k-th entity ZE. and the k-th attribute as Z We sometimes write Z, A (c), Zeomtex (c), etc. to make clear the dependence on the context c.
addition,
Thhee the pre-transformer layer activations, sometimes referred the residual stream.
2

---

Published as a conference paper at ICLR 2024
Z context
Moddl Oupul
Activation
3B
Z
Z
Z..
FF.
Z...
A.
Z Zurry
Takenns
lives
in
ag
c1
lives in
where does ca live?
b)
Zeantent
2
content
Z
Tokers Activations
%
z'
Z
Z
z
2
2
A
Zinteronne
Lives
422
es
lives
as
g
4'A
output
Z
AA.
Figure 2: a) Causal diagram for autoregressive LMs. From input context cixt(ee ++ an; 61 a1), the LM constructs internal representations Zoontext. We will mainly study the components of Zeontext boxed in blue. b) A secondary run of the LM on context ctxt(e2 + 0g e3 ++ ag) to produce Z'conteur c) An example intervention where Zeontexi is modified by replacing A + Z'Aa from Zoontex
Fig. 2a shows that Zoomext contains all the information about the context that the LM uses. We thus study the structure of Zeontest using causal mediation analysis, a widely used tool for understand- ing neural networks (Vig et al., 2020a; Geiger et al., 2021: Meng et al., 2022). Causal mediation analysis involves substituting one set of activations in a network for another, and we adopt the /. notation (from Mathematica) to denote this. For example, for activations Z. E RR x dinard and token position p in the context, Zesontxx/- - + 7.} [Zo...., 7p-11. Ze,, 7 1p........ Similarly, for context .o,...,E, + ag...., + an-1).
ctxt(Eo ++ ao...., Cn-1 ++ an-1), we have c/A e,} etxt(eo +
Given a causal graph, causal mediation analysis determines the role of an intermediate node by experimentally intervening on the value of the node and measuring the model's 's output on various queries. For convenience, when the model answers queries in accordance context c, we say that the model believes² e. If there is no context consistent with the language model's 's behavior, then we say that the LM is confused.
As an example, suppose we are interested in the role of the activations ZA in Fig. 2a. To apply causal mediation analysis, we would:
1. Obtain Zoontxt by running the model on the original context (which we also refer as the target context) (Fig. 2a)
2. Obtain Zoomtext by running the model on a different context c' (i.c. source context) (Fig. 2b)
3. Modify Zeomeeu by replacing AD from the target context with Z'A, from the source context
(Fig. 2c), while keeping all other aspects of Zeontext the same, resulting in Zintervened Zcomeex/ + )'A,))
4. Evaluate the model's beliefs based on the new Zintervened
We can infer the causal role of ZA, from how the intervention Zcomexa/-(Za - L'A changes the model's 's beliefs. Intuitively, if the model retains its original beliefs then AA, has causal role in the model's 's behavior on the binding task. On the other hand, if the model now believes the source context c', then AA, contains all the information in the context. In reality both hypothetical extremes are implausible, and in $3 we discuss a more realistic hypothesis.
A subtle point is that we study how different components of Zeumeet store information about the context (and thus influence behavior), and not how Zcontext itself is constructed. We thus suppress
We do not claim assume that LMs actually have beliefs in the sense that humans do. This is purely notational choice reduce verbosity.
3

---

Published as a conference paper at ICLR 2024
the causal influence that A has on downstream parts of coontex (such as Zg, and ZA.) by freezing the values Zg, and ZA, in Zintervened instead of recomputing them based on Z'Ao'
3 EXISTENCE OF BINDING IDS
In this section, we first descrihe our hypothesized binding ID mechanism. Then, we identify two key predictions of the mechanism, factorizability and position independence, and verify them exper- imentally. We provide an informal argument in $B for why this binding ID mechanism is the only mechanism consistent with factorizability and position independence.
Binding ID mechanism. We claim that to bind attributes to entities, the LM learns abstract binding IDs that it assigns entities and attributes, that entities and attributes bound together have the same binding ID (Fig. 1). In more detail, our informal description of the binding ID mechanism is:
1. For the k-th entity-attribute pair construct an abstract binding ID that independent of the entity/attribute values. Thus, for a fixed n-entity binding task (e.g. CAPITALS task) we can identify the k-th abstract binding ID with the integer E ( n 1
2. For entity Ex, encode both the entity E& and the binding ID & in the activations ZE..
3. For attribute Ag, encode both the attribute A and the binding IDD in the activations Z
4. To answer query for entity Eg, retrieve from Zeontext the attribute that shares the same binding ID as Ex.
Further, for activations ZE, and AA, the binding ID and the entity/attribute are the only information they contain that affects the query behavior.
More formally, there are binding functions I' (e, k) and I [a k)) that fully specify how Z and Z bind entities/attributes with binding IDs. Specifically, if Es = e E E, then we can replace ZE with I' [' k) without changing the query behavior, and likewise for Z..
More generally, given Zeoment with entity representations I' (ep 0), ..... (e - 1) and at- tribute representations I [au, (0)),.... " ((n 1)) for permutation т,, the LM should answer queries according to the context с = ctxt(Cu <> a=-1(0)...... Cn-1 ++ ax-)(n-))). This implies two properties in particular, which we will test in the following subsections:
Factorizability: we replace ZA, with ZAi- then the model will bind Eg to Ar instead of Ag. i.e. it will believe (./{AR + A/ This is because Z' encodes A (A',, k) and ZA, encodes (A(A,, k). Substituting ZA, + Zaz will overwrite (A,, k) with Tx(4', 4g, k), causing the model to bind Eg to A'e-
Position independence: if we e.g. swap Zao and ZAi the model still binds Ao ++ Eo and
A1 + E1, because it looks up attributes based on binding ID and not position in the context.
In $4, construct fine-gra tiee-graneed modifications to the activation Z that modify the binding ID but not the attributes, allowing us to test the binding hypothesis more directly. In $5 we extend this further by showing that binding IDs can be transplanted from entirely different tasks.
3.1 FACTORIZABILITY OF ACTIVATIONS
The first property of Zeontext we test is factorizabiliyy In our claimed mechanism, information is highly localized ZA, contains all relevant information about Ag, and likewise for Z Therefore, we expect LMs that implement this mechanism to have factorizable activations: for any contexts c, c', substituting ZF, > ZF, (c') into Zeomexr (c) will cause the model to believe c/.(Es. + E.). and substituting ZA. + ZA, (c') cause the model to believe c/.(A, + A²).
To test this concretely, we considered the CAPITALS task from $2 with n = entity-attribute pairs. We computed representations for two contexts ctxt(eo 0o, E1 a1) and c' ctxt(e' + ag ef + a(), and used causal mediation analysis ($2) to swap representations from the source context c' into the target context Specifically, we fix is E {0, and intervene either just the entity (Z + Z'E, ). just the attribute, neither, or both. We then measure the mean log probs for all passible queries (Eo, E,, E,, E)). For instance, swapping Aş with A in Zenment should lead A (and not A&) to have high log-probability when F, is queried.
4

---

Published as a conference paper at ICLR 2024
None
Attribute
0.39
3.55
9.58 8:75
8:29
8:44
5:34
0:38
3:95
9.58
9.73
0.38
9:67 9.30
4.01
mueryname name
-2.71
40.60
4.24-944
8.23
09919992 9.27
2.71
2.53
2..2 29.8.97 8.97.8..7
Query
2.13
4.73-7700
5.32
072-2.07-11
10.00-1.777
7.77
-.57-2.11
11146.3-77000
-1.0
6635.79--600 04.79-64662.1
6.70
1.59 2.13
6.74 6.93
1.66
6.79
6.46 2.13
Endiyy
Both
Entity
Both
1:76
6.79
0.39 3:93 9:47
9.69
0.39
9.39
Quaryy name
neme
53
Ag
(a) Swapping entity/attriiute (Eo, Ao)
(b) Swapping entity/attribute for (E)
Figure 3: Factorizability results. Each row corresponds to querying for a particular entity. Plotted
are the mean log prob for all four attributes. Highlighted squares are predicted by factorizability.
Mean Log Prob
Name
-2-
Attribute
-4-
A
Keep
the answer shart
<WS> Context
Elizabeth
lives
the capital City
of ISrael
Ry lives
capital
city
Scotland
of
<WS>
Mean Log Prob
-4
<WSa Contaxt
Elizabeth
lives
the capital
city
Israel
Ryan
live
in the capital
city
Scotland
<Wss <WS Question
Which city
Elizabeth
Token Position
Figure 4: Top: Mean log probs for entity interventions. Bottom: Mean log probs for attributes. For brevity, Zд refer ZE, Z The grey and green vertical lines indicate the original positions for Zo and Zi respectively. The x-axis marks Zn 's new position. Under the position interventions (Xo + x, X1 + X1 - (x - Xo)}, the grey line is the control condition with no interventions, and the green line is the swapped condition where Zo and Z₁ have swapped positions.
Results are shown in Fig. and support the factorizability hypothesis. an example, consider Fig, 3a. In the None setting (no intervention), see high log probs for Ag when queried for Eo, and for A₁ when queried for Er. This indicates that the LM able to solve this task. Next, consider the Attribute intervention setting (Ao Ab): querying for Ep querying for Ej gives Aj as usual. Finally, in the Both setting (where both entity and attribute are swapped), querying E% returns A while querying Eg leads to approximately uniform predictions.
gives high log probs for A'., and
Experiment details. We use LLaMA 30-b here and elsewhere unless otherwise stated. In practice, found that activations for both the entity token and the subsequent token encode the entity binding information. Thus for all experiments in this paper, we expand the definition of ZE, to include the token activations immediately after Eg.
3.2 POSITION INDEPENDENCE
We next turn to position independence, which is the other property we expect LMs implementing the binding ID mechanism to have. This says that permuting the order of the ZF, and A should
5

---

Published as a conference paper at ICLR 2024
have no effect on the output, because the LM looks only at the binding IDs and not the positions of entities or attributes activations.
To apply causal interventions to the positions, we use the fact that transformers use positional em- beddings to encode the (relative) position of each token in the input. We can thus intervene on these embeddings "move" one of the Z 's another location k'. Formally, we let Xg describe the position embedding for Zy: and denote the position intervention as (X₁ + k7) In $C we describe how to do this for rotary position embeddings (RoPE), which underlie all the models we study. For now, we will assume this intervention as a primitive and discuss experimental results.
For our experiments,
again consider the CAPITALS task with n 2. Let X Eo and Xr, denote the positions of the two entities. We apply interventions of the form {X - x, XE, + Xx. (xr ].)}, for E : XE, + JE, This measures the effect of gradually moving the two entity positions past each other: when condition), and when x Xg, the entity positions are swapped (swapped condition). We repeat the same experiment with attribute activations and measure the mean log probs in both cases.
XE. no intervention is performed (control
Results are shown in Fig. 4. As predicted under position independence, position interventions result in little change in model behavior. Consider the swapped condition at the green line. Had the binding information been entirely encoded in position, we expect a complete switch in beliefs compared to the control condition. In reality, we observe almost change in mean log probs for entities and small change in mean log probs for attributes that seems to be part of an overall gradual trend.
We interpret this gradual trend as artifact of position-ddpenden bias, and not as evidence against position independence. We view it as a bias because it affects all attributes regardless of how they bound attributes that are shifted to later positions always have higher log probs. We provide further discussion of this bias, as well as other experimental details, in SC.
STRUCTURE OF BINDING ID
The earlier section shows evidence for the binding ID mechanism. Here, we investigate two hypothe- on the structure of binding IDs and binding functions. The first is that the binding functions A'A and g are additive, which lets us think of binding IDs as binding vectors. The second is contingent on the first, and asks if binding vectors have geometric relationship between each other.
4.1 ADDITIVITY OF BINDING FUNCTIONS
Prior interpretability research has proposed that transformers represent features linearly (Elhage et al., 2021). Therefore a natural hypothesis is that both entity/attribute representations and abstract binding IDs are vectors in activation space, and that the binding function ($3) simply adds the vectors for entity/attribute and binding ID. We let the binding ID & be represented by the pair of vectors [k=(k), ba (k)), and the representations of entity e and attribute a be f'r(e) and (a( respectively. Then, we hypothesize that the binding functions can be linearly decomposed as:
aa k) = fa(a) + (k)), Tg(e,k) = fE(C) + bk()).
(1)
Binding ID vectors seem intuitive and plausibly implementable by transformer circuits. To exper- imentally test this, we seek to extract (k) and by in order to perform vector arithmetic on them. We use (1) to extract the differences (e(k) == by(k) - be(0), (k)) == bkk) - (0).
Rearranging (I), we obtain
(k) = l'A - l'A(a, (E( = l'E(e, ().
(2)
We estimate (k) by sampling Eg,e [Z (c) AA, (c')), and likewise for A (k()).
Mean interventions. With the difference vectors, we can modify binding IDs by performing mean interventions, and observe how model behavior changes. The attribute mean intervention switches the binding ID vectors in A and ZA, with the interventions Z + ZA + (1), [1),2), ZA, 4 ZA, - (1). The entity mean intervention similarly switches the binding ID vectors in Z and Z Additivity predicts that performing either mean intervention will reverse the model behavior: E. will be associated with A1, and E, with Ag.

---

Published as a conference paper at ICLR 2024
Test condition
Control
Attribute
Entity
Both
Attribute
Entity
Both
Querying Fo
0.99
0.00
0.00
0.97
0.98
0.97
Querying Ei 1.00
0.03
0.01
0.99
1.00
1.00
1.00
Table 1: Left: Mean calibrated accuracies for mean interventions on four test conditions. Columns are the test conditions, and rows are queries, Right: Mean interventions with random vectors.
Mean callbrateed Accuracy
0
0
0
0
0
The plots show the mean median-calibrated accuracy when pair of binding ID, vo- fixed at the green circle, and the other, U1, varied across the grid. The binding IDs b(0), 6(1), and 6(2) are shown as the origin of the arrows, the end of the horizontal arrow, and the end of the diagonal arrow respectively. We use LLaMA-13b for computational reasons.
Figure 5:
Experiments. In our experiments, we fix and use 500 samples estimate (1 and (1)... We then perform four tests, and evaluate the model accuracy under the original belief. The Control test has no interventions, and the accuracy reflects model's base performance. The Attribute and Entity tests perform the attribute and entity mean interventions, which should lead to complete switch in model beliefs so that the accuracy is near 0. Table 1 shows agreement with additivity: the accuracies are above 99% for Control, and below 3% for Attribute and Entity.
As further check, we perform both attribute and entity mean interventions simultaneously, which should cancel out and thus restore accuracy. Indeed, Table shows that accuracy for Both is above 97%. Finally, to show that the specific directions obtained by the difference vectors matter, we sample random vectors with the same magnitude but random directions, and perform the same mean interventions with the random vectors. These random vectors have no effect on the model behavior.
4.2 THE GEOMETRY OF BINDING ID VECTORS
think of binding IDs pairs of ID vectors, and that randomly chosen vectors do not function as binding IDs. We next investigate the geometric structure of valid binding vectors and find that linear interpolations binding vectors. This suggests that binding vectors occupy continuous binding subspace. We find evidence of a metric structure in this space, such that nearby binding vectors are hard for the model to distinguish, but far-away vectors can be reliably distinguished and thus used for the binding task.
$4.1 shows that
extrapolations of binding vectors are often also valid
To perform investigation, we apply variants of the mean interventions in $4.1. As before, start with n = 2 context, thus obtaining representations Zg = (Z,,, A and Z1 = (Zg, ZA,), We first erase the binding information by subtracting (Ag (1)) from Z1, which reduces accuracy to chance. Next, we will add vectors = (VE,, MAA)) and (VE, UA,) to the representations Z; doing so restores accuracy, then we view (VE, UAA)) and (VE, "A,)) as valid binding pairs.
To generate different choices of v, we take linear combinations across two-dimensional space. The basis vectors for this space are (Ag(1), (1)) and (Ag (2)) obtained by averaging across an 3 context. Fig. shows the result for several different combinations, where the coordinates of Vo are fixed and shown green while the coordinates of 01 vary. When U1 close to Up. the LM gets close to 50% accuracy, which indicates confusion. Far away from v1, the network consistently achieves high accuracy, demonstrating that linear combinations of binding IDs (even with negative coefficients) are themselves valid binding IDs. See &G for details.

---

Published as a conference paper at ICLR 2024
10
1.0
Accuracy
Accuracy
0.6
0.6
Intervention
Pythia
Control
control
0.
0.2
ww-. Entity
attribute
10
102
CAPITALS PARALLEL SHAPES
FRUITS
BIOSBIAS
Size Ibillions of parameters)
Task
Figure 6: Left: models in Pythia and LLaMA on CAPITALS. LLaMA-65b not present for computa- tional reasons. Right: LLaMA-30b on binding tasks. Unlike others, the BIOS task has attributes that are several tokens long.
Task
CAPITALS
PARALLEL
SHAPES
FRUITS
BIOS
Zeros
Random
Mean accuracy
0.88
0.87
0.71
0.80
0.47
0.30
0.31
Mean log prob
-1.01
-1.07
-1.18
-1.21
-1.64
-1.86
-2.15
Table 2: The mean median-calibrated accuracy and mean log prob for mean interventions on n = CAPITALS using binding ID estimates from other tasks. Random chance has 0.33 mean accuracy.
The geometry of the binding subspace hints at circuits (Elhage et al., 2021) in LMs that process binding vectors. For example, speculate that certain attention heads might be responsible for comparing binding ID vectors, since the attention mechanism computes attention scores using a quadratic form which could provide the metric over the binding subspace.
5 GENERALITY AND LIMITATIONS OF BINDING ID
The earlier sections investigate binding IDs for one particular task: the CAPITALS task. In this section, we evaluate their generality. We first show that binding vectors are used for a variety of tasks and models. We then show evidence that the binding vectors are task-agnostic: vectors from one task transfer across many different tasks. However, our mechanism is not fully universal. $E describes a guestion-answerringg task that uses an alternative binding mechanism.
Generality of binding ID vectors. We evaluate the generality of binding vectors across models and tasks. For a (model, task) pair, we compute the median-calibrated accuracy on the n = performed, context under three conditions: (1) the control condition in which no interventions
and the (2) entity and (3) attribute conditions in which entity or attribute mean interventions ($4.1) performed. We the mean interventions to permute binding pairs by cyclic shift and measure accuracy according to this shift (see $F). As shown in Figure 6, the interventions induce the expected behavior most tasks; moreover, their effectiveness increases with model scale, suggesting that larger models have more robust structured representations.
Transfer across tasks. We next show that binding vectors often transfer across tasks. Without ac- cess to the binding vectors (k), ba(k)) we instead test if the difference vectors (k), (k)] from source task, when applied to a target task, result in valid binding IDs. To do so, we follow similar procedure to $4.2: First, we erase binding information by subtracting [A E (k), A (k)) from each target-task representation F2 ZA,l resulting in near-chance accuracy. Then, we add back in sre (k), A computed from the source task with the hope of restoring performance.
Table shows results for variety of source tasks when using CAPITALS as the target task. Ac- curacy is consistently high, even when the target task has limited surface similarity to the target task. For example, the SHAPES task asks questions about colored shapes, and PARALLEL lists all entities before any attributes instead of interleaving them in CAPITALS. We include two baselines for comparison: replacing Arrc(b) with the zero vector ("Zeros")), " or picking a randomly oriented difference vector in Table ("Random"), Both lead to chance accuracy. See SD for tasks details.
The fact that binding vectors transfer across tasks, together with the results from $4, suggests that there could be task-agnostic subspace in the model's activations reserved for binding vectors.
8

---

Published as a conference paper at ICLR 2024
6 RELATED WORK
Causal mediation analysis. In recent years, causal methods have gained popularity in post hoc in- terpretability (Meng et al., 2022; Geiger et al., 2021). Instead of relying on correlations, which could lead to spurious features (Hewitt & Liang. 2019), causal mediation analysis (Vig al., 2020a) per- forms causal interventions on internal states of LMs to understand their causal role on LM behavior. Our work shares the same causal perspective adopted by many in this field.
Knowledge recall. A line of work studies recalling factual associations that LMs learn from pre- training (Geva et al., 2020; Dai et al., 2021; Meng al., 2022: Geva al., 2023; Hernandez al., 2023b). This is spiritually related to binding, as entities must be associated to facts about them. However, this work studies factual relations learned from pretraining and how they are recalled from model weights. In contrast, we study representations of relations learned from context, and how they are recalled from model activations.
More recently, Hernandez et al. (2023a) found method to construct bound representations by di- rectly binding attribute representations to entity representations. In contrast, our work investigates bound representations constructed by the LM itself, and identifies that the binding ID mechanism (and not direct binding) is the mechanism that LM representations predominantly uses. An avenue for future work is to study how bound representations constructed by Hernandez et al. (2023a) relates to the direct binding mechanism we identified in SE
Symbolic representations in connectionist systems. Many works have studied how neural net- works represent symbolic concepts in activation space (Mikolov et al., 2013; Tenney et al., 2019; Belinkov & Glass, 2019; Rogers et al., 2021; Patel & Pavlick, 2021). To gain deeper insights into how these representations are used for reasoning, recent works have studied representations used for specialized reasoning tasks (Nanda al., 2023; Li al., 2022; 2021). Our work shares the motiva- tion of uncovering how neural networks implement structured representations that enable reasoning.
Mechanistic Interpretability. Mechanistic interpretability aims uncover circuits (Elhage al., 2021: Wang et al., 2022; Wu et al.. 2023), often composed of attention heads, embedded in language models. In our work, we study language model internals on a more voarse-grained level by identify- ing structures in representations that have causal influences on model behavior. Concurrent work by Prakash al. (2024) complements ours by analyzing the circuits involved in the binding problem.
7 CONCLUSION
In this paper we identify and study the binding problem, a common and fundamental reasoning subproblem. We find that pretrained LMS can solve the binding task by binding entities and attributes to abstract binding IDs. Then, we identify that the binding IDs are vectors from a binding subspace with a notion of distance. Lastly, we find that the binding IDs are used broadly for a variety of binding tasks and are present in all sufficiently large models that we studied.
Taking broader view, we see our work as a part of the endeavor to interpret LM reasoning by decomposing it into primitive skills. In this work we identified the binding skill, which is used in several settings and has simple and robust representation structure. An interesting direction of future work would be to identify other primitive skills that support general purpose reasoning and have similarly interpretable mechanisms.
Our work also suggests that ever-larger LMs may still have interpretable representations. A common intuition is that larger models are more complex, and hence more challenging interpret. Our work provides a counterexample: as LMs become larger, their representations can become more structured and interpretable, since only the larger models exhibited binding IDs (Fig. 6).
Speculating further, the fact that large enough models in two unrelated LM families learn the same structured representation strategy points to a convergence in representations with scale. Could there be an ultimate representation that these LMs are converging towards? Perhaps the properties of nat- ural language corpora and LM inductive biases lead to certain core representation strategies that are exact dataset composition. This would encour- agingly imply that interpretability results can transfer across models sadell-sttuyying the core representa- tions of any sufficiently large model would yield insights into other similarly large models.
invariant to changes in model hyperparameters

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank Danny Halawi, Fred Zhang, Erik Jenner, Cassidy Laidlaw, Shawn Im, Arthur Conmy, Shivam Singhal, and Olivia Watkins for their helpful feedback. JF was supported by the Long-Tern Term Future Fund. was supported by the National Science Foundation under Grants No. 2031899 and 1804794. In addition, we thank Open Philanthropy for its support of both JS and the Center for Human-Compatible Al.
REFERENCES
Yonatan Belinkov and James Glass. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49-72, 2019. doi: 10.1162/
lacLa_00254, URL https::t///omm http:::://aclaaahhoooogoo org/ 019-1004,
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Allah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning, pp. 2397-2430. PMLR, 2023.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. Knowledge neurons in pretrained trans- formers. ArXiv, abs/2104. 08668,,, 2021. URL https: /api sewanticschoneo .oggoorrrrrrooooooooooooooo
CorpusID:2332966
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In proceedings of the Confer- ence on Fairness, Accountability, and Transparency, pp. 120-128, 2019.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https://rransfommr-ciinoot http:/aiis.pppt///22//////////ne
Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: critical analy- sis. Cognition, 28(1-2):3-71, 1988.
Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34:9574-9586, 2021.
Mor Geva, R. Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key- value memories. ArXiv, Abs/2011..14913 2020. URL https: semanticscholar.
orq/Corpus 229 37220.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in aeto-segressive language models. arXiv preprint 12X1:
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. arXiv preprini arXiv:2109. 13916, 2021.
Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. In International Conference Learning Representations, 2021.
Evan Hernandez, Belinda Z Li, and Jacob Andreas. Measuring and manipulating knowledge repre- sentations language models. arXiv preprint arXiv:2304.00740, 2023a.
Evan Hernandez, Amab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. arXiv preprint 230.0:2308.091 2023b.
10

---

Published as a conference paper at ICLR 2024
John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. arXiv preprint arXiv:1909.03368, 2019.
Najoung Kim and Sebastian Schuster. Entity tracking in language models. arXiiv2335..23333 2023.
arXiv preprint
Belinda Z Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. arXiv preprint urXiv:2106.00737, 2021.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Watten- berg. Emergent world representations: Exploring a sequence model trained on a synthetic task.
arXiv preprini
2022.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 15:17759-77372, 2022.
Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space
word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pp. 746-751, 2013.
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. Advances Neural Informa- fion Processing Systems, 33:17153-17166 2020. 2020.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learn- ing Representations, 2023. URL https: oppnreviee.e....e/ n forum ?id- 9XF SbDPmdM.
Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In Inter- national Conference on Learning Representations, 2021.
Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking. arXiv preprint arXiv:2402.1483 14811, 2024.
Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward transparent ai: A survey interpreting the inner structures of deep neural networks. In 2023 IEEE Conference Secure and Trustworthy Machine Learning (SaTML), pp. 464-483. IEEE, 2023.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What know about how bert works. Transactions of the Association for Computational Linguistics, 8:842-866, 2021.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-164H 1642, Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics. URL https::tpt........tt /www. ac web. org/ munthooogy/DD333 117000
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. arXiv preprint wrXiv:2104.0 009644 2021.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. In Associa- fion for Computational Linguistics, 2019. URL pestts::/arkcee wwwrriiiv.or/aabi abs / 111.9500555 05950.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 2021,
Anne Treisman. The binding problem. Current opinion in neurobiology, 6(2):171-178, 1996.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33:12388-12401, 2020a.

---

Published as a conference paper at ICLR 2024
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. 2020b.
Advances in Neural Information Processing Systems,
Kevin Wang, Alexandre Variengien, Arthur Commy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far camels go? exploring the state of instruction tuning on open resources. arXiv preprint arXiv:2306.0477 2023.
Zhengxuan Wu, Atticus Geiger, Christopher Potts, and Noah D Goodman. Interpretability at scale: Identifying causal mechanisms alpaca. arXiv preprint arXiv:2305.08800 2023.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pp- 12697-12706. PMLR, 2021.
A EVALUATION DETAILS
In all of our evaluations, we sample = 100 instances of contexts from the binding task, obtaining {q] For succinctness, we write E := (c,),) and A((i := (c,).. For the i-th context in- stance, we query E)(t) and E which return log probabilities 2 (t) and o (t) over tokens t the vocabulary. However, we consider only the log probabilities for relevant attributes (+) Eg (AA and
(6) @ Eg (A(*). We then compute the summary statistics (described below) over the entire population of samples so that we get two scalars, 0 and oE, describing the performance under each query entity.
The mean log prob is given by dE ΣE (A
The top-1 accuracy is dE N/ De
DE 1|k arg max, & E. (A
We adopt the median calibrated accuracy from Zhao et al. (2021). First, we obtain a base- line by computing medians for every attribute m m(Ar) = median,, (0) E. (4()). Then, compute calibrated log probs
+ (Ar) m(A)). The median calibrated accuracy is then σE. Σ i=1 1 arg max, + $00 (A(")).
$ (5) Eg (Ai)
N
Zhao et al. (2021) discusses motivations for the median calibrated accuracy. In our case, the position dependent bias provides addition reasons, which discuss in SC.
B NECESSITY OF BINDING ID MECHANISM
In this section, we provide one definition of the binding ID mechanism, and argue informally that under this definition, factorizability and position independence necessarily implies the binding ID mechanism.
First, let define the binding ID mechanism. Fix for simplicity. There are claims:
1. Representation. There exists binding function I'g such that for any contexts c, Zg. is represented by Te (Eg, . k), in the sense that for any E E, (Z [e, k)} leads the belief (E + <}. Likewise, there exists binding function I' such that for any contexts c, ZA, is represented by F (AR,, A:), the sense that for any a E A, {Z + TT k)) leads to the belief c/ - a). These substitutions should also compose appropriately.
2. Query. Further, the binding functions A and Dg satisfy the following property: Choose any 2 permutations (k) and (k) over {0, 1), and consider Zeontext containing TF ( T ( [ (1)), (a) x (1))] The query system will then believe en ao, e1 01 if m RE- nA, and en 01 en ag otherwise.
12

---

Published as a conference paper at ICLR 2024
The first claim follows from factorizability. From factorizability, binding functions simply by picking an arbitrary context consistent with the parameters. For any & and any binding ID k E [0,n 1/. pick any context such that (c) Then, let Tele, k) = E, (c). A can be constructed similarly. Our factorizability results show that the binding functions constructed this way satisfy the Representation claim.
construct the candidate
The second claim follows from Representation and position independence. Pick an arbitrary context с generate Zoontext. Then, by factorizability we can make the substitutions Zz, I' [' " (13) k) and ZA, + I (a x (k)) A:), to obtain
Te(e " (0)) 0), [,
0), T (1) (a(a (1)( I]].
Because of factorizability, the model believes ey ao. 61 01 if T AA and eo +> a1. 01 do otherwise. Now, position independence lets us freely permute {ZE, ZE, } and {Z Za, without changing beliefs, which achieves the following context
[Co,, Ne (0)), T
[e(e,, T((1, TA (1))
with the desired beliefs.
C DETAILS FOR POSITION INDEPENDENCE
RoPE Intervention. In Fig. 2a, the context activations Zonnexx is drawn in a line, suggesting a linear form:
Zwcontex: [..., E ........ Agg----- .......,, -...-.... We can equivalently think of Zeoment as a set of pairs: Zeomext = {pp Zp) p is an index for a context token]. LMs that use Rotary Position Embedding (RoPE) (Su et al., 2021), such those in the LLaMA and Pythia families, have architectures that allow arbitrarily intervention on the apparent position of an activation (p, Zp) - (p', Zp), even if this results in overall context activations that cannot be written down as a list of activations. This is because position information is applied at every layer, and not injected into the residual stream like in absolute position embeddings. Specifically, equation 16 in Su al. (2021) provides the definition of RoPE (recreated verbatim as follows):
T kn (R.. W ( W....)
Then, making the intervention Rg + R changes the apparent position of the activations at position n to the position at n²
Is the position dependent bias just a bias? For the purposes of determining if position encodes binding, the fact that the LM does not substantially change its beliefs when we switch the positions of the attribute activations (or the entity activations) suggests that position only play limited role. However, calling the position dependency of attributes a "bias" implies that is an artifact that we should correct for. To what extent this true?
The case for regarding it as a bias is two-fold. First, as discussed by Su et al. (2021), RoPE exhibits long-term position decay, which systematically lowers the attention paid activations that are fur- ther away from the query (i.c. earlier in the context). Plausibly, at some point when computing the query mechanism, the LM has to make a decision whether pay attention to the first or the second attribute, and the presence of the long-term position decay can bias this decision, leading to the position dependent bias the final prediction that we see.
The second reason is that there are systematic and unbiased ways of calibrating the LM to recover the correct answer, in spite of the position dependent bias. We discuss two strategies. Because the position dependent bias modifies the log probs for An (or A1) regardless of which entity is being queried, we can estimate this effect by averaging the log probs for An (or A₁) for both queries Fg and ET. Then, when making a prediction, (or A1). This corresponds to the median calibrated accuracy metric discussed earlier. The second procedure mitigate the position dependent bias is intervention to set all attribute activations to have the same position, which limits the amount of bias position dependency can introduce.
subtract this average from the log probs for Ao
These procedures do not require foreknowledge of what the ground truth predicates are, and hence do not leak knowledge into the prediction process if the calibrated LM answers queries correctly, the information must have come from the context activations and not from the calibration process.
13

---

Published as a conference paper at ICLR 2024
Nonetheless, there are features about the position dependent bias that could be interesting to study. For example, we might hope to predict the magnitudes of the position dependent bias based on RoPE's parameters. However, such investigation will most likely involve deeper mechanistic understanding of the query system, which we leave as future work.
D BINDING TASK DETAILS
D.1 CAPITALS
Construct list one-token names and a list of country-capital pairs that are also each one-token wide. Then, apply the following template:
Answer the question based on the centext below. Keep the answer short.
Context: (E_0) lives in the capital city of (A (E_1) lives in the capital city of (A_1)
Question: Which city does (gn_subject) live in?
Answer: (qn_subject) lives in the city of
The LM expected answer with the capital of the country that is bound the queried entity. Note that the LM is expected to simultaneously solve the factual recall task of looking up the capital city country.
D.2 PARALLEL
The PARALLEL task uses the same country capital setup, but with the prompt template:
Answer the question based on the context below. Keep the answer short.
Context: (E_0) and [E_1) live in the capital cities (A_0) and [A_1]
respectively.
Question: Which city does (qn_subject) live in?
Answer: (qn_subject) lives in the city of
This prompt format breaks the confounder in the CAPITALS task that entity always appear in the same sentence as attributes, suggesting binding ID is not merely syntactie property.
D.3 FRUITS
The FRUITS task uses the same set of names, but for attributes it uses set of common fruits and food that are one-token wide. The prompt format is:
Answer the question based the context below. Keep the answer short.
Context: (E_01 likes eating the (A. (E_1) likes eating the (A respectively.
Question: what food does (en_subject) like?
Answer: (qn_subject) likes the
D.4 SHAPES
The SHAPES tasks have entities which are one-token wide colors, and attributes which are one-token wide shapes. The prompt looks like:
Answer the question based context below. Keep the answer short.
14

---

Published as a conference paper at ICLR 2024
Review: simply sublime
Classify the review using the following options: Posilive Negative
Classification:
Positive
Z
(a Positive)
Z
AD Negutive)
X
Ao(A, Poaitive)
(egattee
Poss i Live
Ney a Live
Review: simply sublime
Answer:
Figure 7: Direct binding in MCQ task. Og and Lş denote options and labels respectively. Under direct binding, Zo, and Zo, represented by binding function Ao that directly binds option and label together, whereas Z, and Zt, are causally irrelevant.
Context: The (A_0) is (E_ The (A_1) is [E_1).
Question: Which shape is colored (qn_subject)?
Answer: The (qn_subject) shape is
This task inverts the assumption that entities have to be nouns, and attributes are adjectives.
D.5 BIOS
This task is adapted from the bias in bios dataset De- De-Araaa et al. (2019), with a prompt format following Hernandez et al. (2023a). The entities are the set of one-token names, and the attributes are a set of biography descriptions obtained using the procedure from Hernandez et al. (2023a). The I. is expected infer the occupation from this description. This time, the attributes are typically one sentence long, and are no longer one-token wide. We thus do not expect the mean interventions for attributes to work, although we may still expect entity interventions to work. Just inferring the correct occupation is also much more challenging task than the other synthetic tasks.
The prompt format is:
Answer the question based on the context below. Keep the answer short.
Context: About (E_0): (A_0) About (E_1): (A_1)
Question: What occupation does (qn_subject) have?
Answer: (qn_subject) has the occupation of
E
MCQ TASK
Direct binding in MCQ. While binding IDs are used for many tasks, they are not universal. We briefly identify alternate binding mechanism, the direct binding mechanism, that is used for multiple-choice question-answerne associated option text. In this task, instead of binding variables directly binds the label to the option (Fig. 7).
task (MCQ). In MCQ, each label (A or B) has to be bound to its abstract binding ID, the model
Multiple choice questions (MCQs) the question. This is to force the LM to represent the binding between label and option text before sees the questions. We study the SST-2 task (Socher al., 2013), which is binary sentiment classification task on movie reviews (either positive or negative). Then, the attributes are single letter labels from A E, and the entities "Positive" and "Negative".
be formulated binding task if put the options before
The prompt is as follows:
Classi fy the review using the foll following options:
(A_0): (K_0)
15

---

Published as a conference paper at ICLR 2024
1A: (E_1) Review: (question)
Answer:
Then, when prompted with question with certain sentiment, the LM is expected to retrieve its corresponding label.
E.1 EXPERIMENTS
It turns out that the reversed MCQ format is too of distribution for L.LMA-330b solve. How- ever, we find that the instruction finetuned tulu-13b model (Wang et al., 2023) is able to solve this task.
We find that the activations for this task are not factorizable in the same way. Consider the target context:
C: Negative
A: Fositive
and the source context:
A: Negative
C: Positive
We denote the labels Lo and L1, so that Lo is A in the first context and B in the second context. We denote the option texts as On and O,.
We perform an experiment where we intervene by copying over suffix of every line from the source context into the target context, and plot the accuracy based on whether the intervention successfully changes the belief (Fig. 8). The right most point of the plot is the control condition where no inter- ventions are made. The accuracy is near zero because the model currently believes in the original context. At the left most point, intervene the entire statement, which is substitution of the entire Zeuntest Thus, we observe near perfect accuracy.
Interestingly, copying over the activations for the tokens corresponding to "ative" and the whitespace following it suffices for almost completely changing the belief, despite having a surface token form that is identical those tokens ("ative (WS) for both source and target contexts). This suggests that those activations captures the binding information that contains both the label and the option text. This leads the conclusion that binding information bound directly those activations, instead of indirectly via binding IDs.
In contrast, binding ID would have predicted that substituting these two tokens would not have made difference, because the option activations Zo should contain only information about the option text and the binding ID, which is identical for our choice of source and target contexts.
F GENERALITY DETAILS
Suppose is a cyclic shift, say π (0) = 1, +11 = 2, 22 = 0. Then, we can perform mean interventions based on the cyclic shift entities as follows:
ZF. ZF. + bb(((k) - by (k) = Zz. + (E(T(K)) Ak(k).
We then expect the belief to follow the same shift, so that the LM believes Es ++ Az(k)- Similarly, perform mean interventions attributes follows:
ZA + ZA. + (a(T())) - bA (k) = Z + AA(T(K)) - AA(k).
However, this time we expect the belief to follow the inverse shift, i.e. Eg + Ag-1()) which the same as Ex(N) Ag.
As usual, we sample A using 500 samples. We perform the intervention using both cyclic shifts over 3 elements, (i.e. and -1), and report the mean results over these shifts.
16

---

Published as a conference paper at ICLR 2024
0.8
Mean Accuracy
0.6
0.4
0.2
Neg
ative
<WS>
[Control]
Token Position
Figure 8: Substitutions for MCQ option suffix
G
GEOMETRY DETAILS
An experimental challenge face is that we do not have access the binding ID vectors bA, bg themselves, only differences between them, A,, Åg. For clarity of exposition we first describe the procedure would perform if we had access to the binding ID vectors, before describing the actual experiment.
In the ideal case, would obtain two pairs of binding ID vectors, (0), A (0), (1), b (1)]- Then, we can construct two linear combinations of these two binding ID vectors as candidate binding IDs, (UF U AAa] and [UE, J. Now, we can take an n 2 context and intervene on each of Fa Ao ZFZ Az to change their binding IDs to our candidate binding IDs. If the model retains its beliefs, then we infer that the binding IDs are valid.
There two main problems with this procedure. The first is that we only have access to A and Ag and not bg, b.. Instead of choosing [6g (0), (0), (1), b as our basis vectors, we can use contexts with = 3 to obtain [A (1), (1)]. (2))), A These new basis vectors are still linear combinations of binding IDs, and binding ID vectors do form a subspace, these would be part of the subspace too.
The second problem is that we cannot arbitrarily set the binding ID vector an activation to another binding ID) vector. Instead, we can only add vectors to activations. We thus perform two sets of interventions. We first perform the mean interventions on the second binding pair to turn [b=(1), A (1)] into [bg(0), (0)). At this point, the LM sees two entities with the same binding ID and two attributes with the same binding ID, and is confused. Then, we can add candidate binding vector ID offsets to these activations.
More precisely, let 1 1/ be coefficients for the linear combinations of the basis vectors. Define now ha (7, /) A (1) A (2) and hg(n v) n)1(1) + vA)(2) as the candidate binding vector ID offsets. Then, we add [ un), (1 vo)] and [m,, v1), (1m,,, v1 to the respective two pairs of binding IDs, and evaluate if the model has regained its beliefs.
Concretely, the intervention we apply parameterized by (Mu 2o, v1) and are as follows:
ZAo - ZAo (0) + h hx vo), ZF - ZF, (= + (e
ZA, ZA, (1) hx(m, ZE, ZE, (1) + (1),
We now interested in the question: if we have coefficients (7o. vo) and (71 v1), are the binding vectors constructed from those coefficients valid binding IDs?
In our experiments (Fig. 5), we fix the value of :oo and at varying positions (green circles), and accuracy. We find that near the green circle, the
vary 71 and 1/1 We plot the mean model is completely confused, responding with near-chance accuracy. This verifies that the erasure step works as intended. In addition, we find that there appears to be a binding metric subspace that long candidate binding IDs sufficiently far apart, the LM recovers ability to distinguish
17

---

Published as a conference paper at ICLR 2024
None
Attribute
2:12
2.88
2.78
2:83
2.07
2.78
2.12 2:64
2.88
2.78
2.14 2.79 2.89 2.63
Query overyname name
-2.00
-2.11 77-2--260
2.71 2.08 08.0088400
2.95
2.48
2.995
22:1 -2.0
22 24.4 2..--..442
0.12
-1.09
1-4-1-11-3.0
1.12
-3.04
-1.39-1..7
1.12-1.08-11194.
3.09 3.02
3:10 3.04
3.36 3.05
3:10 3.08 3.16
Entiyy
Both
Entiyy
Both
3.09 3.01
2:14 2:66 2.89 2.60
2.16 2.90 2.65
Quaryy name
neme
4
3.07-3.05
2.95
A Ag
(a) Swapping entity/attribute (Eo, Ao)
(b) Swapping entity/attribute for (E1, A1)
Figure 9: Factorizability results for ONEHOP
Test condition
Control
Attribute
Entity
Both
Querying Eo
0.73
0.25 0.28
0.24
0.71
Qucrying Ei
0.79
0.26
0.77
Table 3: Mean intervention results for ONEHOP
between the two, even when candidate binding IDs are outside of the convex hull between the three binding IDs used generate the basis vectors.
H ONE HOP EXPERIMENT
One sign that the binding ID mechanism correctly captures the semantic binding information that the LM is able to reason with representations modified according to the binding ID theory.
To some extent, the CAPITALS task already requires small reasoning step: in the context the LM is given that "Alice lives in the capital city of France", and is asked to answer "Paris". This means that binding mechanism that binds "Alice" to "France" has to create representations that are robust enough to support the inference step "France" to "Paris".
To further push on reasoning, we introduce the ONEHOP task, an augmented version of CAPITALS. The context remains the same as the CAPITALS task. i.c. provide list of people and where they live. However, the LM has to apply an additional reasoning step to answer the question. An example context and question is below:
Answer the question based on the context below. Keep the answer short.
Context: Elizabeth lives in the capital city of France. Ryan lives in the capital city of Scotland.
Question: The person living in Vienna likes rose. The person living in Edinburgh likes rust. The person living in Tokyo likes orange. The person living in Paris likes tan. what color does Elizabeth like?
Answer: Elizabeth likes
In the ONEHOP task, based on the binding information in the context the LM has to perform two inference steps. The first is to infer that the capital city of France is Paris. The second is to, based on the additional information in the question, infer that the person living in Paris likes tan, and output tan as the correct answer.
This is more challenging task than other tasks, and thus present results LLaMA-65b instead of LLaMA-30b. Overall, we find that all of our results still hold. We show results for factorizabilitty (Fig. 9), position independence (Fig. 10), and mean interventions (Fig. 3).
18

---

short. Published as a conference paper at ICLR 2024
Mean Log Prob
Query Name
Es
E1
Attribute
Ao
A
[CTRL]-2
[CTRLI-1
[CTRL]+0
[CTRL]+1
[CTRL]+2
[SWAP]-2
(SWAP)-1
[SWAP]+0
[SWAP]+1
[SWAP]+2
Mean Log Prob
-2
4-
[CTRL]-2
[CTRL]-1
[CTRL1+0
[CTRL]41
[CTRL]+2
[SWAP]-2
[SWAP]-1
[SWAP]+0
[SWAP]+11
[SWAP142
Figure 10: Position independence for ONEHOP. Top: Mean log probs for entity interventions. Bot- tom: Mean log probs for attributes. Different from Fig. 4, we only compute the local neighborhood around the control and swapped conditions.
I THREE-TERM BINDING
our tasks, we studied binding between two terms: binding an entity to an attribute. Here, we extend our results to three-term binding. An example context looks like:
Answer the question based the context below. Keep the
Context: Carol from Italy likes arts. Samuel from Italy likes swimming. Carol from Japan likes hunting. Samuel from Japan likes sketching.
Question:
What does Carol from Italy like?
Answer: Carol from Italy likes
In general, each statement in the context binds three terms together: name, country, and hobby. We can query any two of the three terms, and ask the language model to retrieve the third. The above example shows how query for the hobby, given the name and the country. We query for country and name by asking instead:
Which country is Carol who likes hunting from?
Who from Italy likes hunting?
We extend our analysis to three-term binding in the following way. Of the three attribute classes, namely names, countries, and hobbies, choose one to be the fixed attribute, one to be the query attribute, and one to the answer attribute. Altogether, there are 3! possible assignments. For every such assignment, we can perform the same set of analysis as before.
To illustrate, suppose we choose country as the fixed attribute, name to be the query attribute, and hobby to be the answer attribute. An example prompt for this assignment will look like:
Answer the question based the context below. Keep the answer short.
Context: Carel from Italy likes arts. Samve1 from Italy likes swimming:
Question: what does Carol from Italy like?
Answer: Carol from Italy likes
19

---

Published as a conference paper at ICLR 2024
Fixed
Query
Answer
Test condition
Control
Attribute
Entity
Both
name
country
hobby
Query 0
1.00
0.00
0.00
1.00
name
country
hobby
Query
1.00
0.00
0.00
1.00
name
hobby
country
Query 0
1.00
0.00
0.00
1.00
name
hobby
country
Query
1.00
0.00
0.00
1.00
country
name
hobby
Query 0
1.00
0.01
0.00
1.00
country
name
hobby
Query
1.00
0.00
0.00
1.00
country
hobby
Query 0
1.00
0.02
0.01
0.99
country
hobby
name
Query
1.00
0.03
0.01
hobby
name
country
Query 0
1.00
0.00
0.00
1.00
hobby namc
country
Query
1.00
0.00
0.00
1.00
hobby
country
name
Query
1.00
0.00
0.00
1.00
hobby country
name
Query
1.00
0.00
0.00
1.00
Table 4: Mean intervention results for three-term binding. The intervened model perform near
perfectly for most test conditions.
We then report the median-calibrated accuracy for the mean interventions under all 6 assignments (Table 4). The accuracy is better than CAPITALS (Table 1) because CAPITALS requires inferring capital city from country, whereas ONEHIP only requires looking up and copying countries.
ADDITIONAL FACTORIZABILITY AND POSITION INDEPENDENCE PLOTS
This section contains the experiments for factorizability (Fig. 3) and position independence (Fig. 4) reproduced for the other binding tasks, namely PARALLEL (Fig. 11, 15), SHAPES (Fig. 12, 16), FRUITS (Fig. 13, and B1OS (Fig. 14).
None
Attribute
None
Attribute
Query
Query
3.73
2.08
6.03
Entitt
Both
Enttly
Both
Query name
17
0.76 6.03
0.411 4.47 9.61
7.89
9.10
0.772
4.77
7.
1.71
2.07
6.79
6.97
5.26
1.57
1.90
6.69
3.85
0.04
9.33
9.63
4.03
5.41
0.58
Aétributtes
Altributes
Altributes
Aétributtes
(a) Swapping entity/attribute for (Eo, Ao)
(b) Swapping entity/attribute for (E1, A1)
Figure 11: Factorizability results for PARALLEL
Notice that for BIOS (Fig. 14), entity factorizability works, but not attribute factorizability. This is because the attribute information is represented by many tokens, while the attribute factorizability test only substitutes the first token in the attribute representation.
20

---

Published as a conference paper at ICLR 2024
None
Attribute
None
Attribune
Quar name
61
5.44
-2.2
5
3.33
5.11
4.90-5.5.3
Entryy
Both
Entiyy
Both
1.004440
0..44..13-7
Quary name
3.37
7.5---55
6.59
7.49
EL
3.33
5.26 4.82
15
0.52
2.72
7.22
7:31
6.17
2.53
0.07
7.10
ES
2.40
2:35
5.32
4.63
2.46
5.03
5.00
2.29
30
2.17
5.06
5.10
4.61
2.19
2.72
5.09
5
3.62
0.42
7.57
7.09
3.78
7.53
0.43
Attributes
Attributes
Attributes
Actributes
(a) Swapping entity/attribute for (Bu, Au)
(b) Swapping entity/attribute for (E1,
Figure 12: Factorizability results for SHAPES
None
Altribute
None
Altribute
0.98
3.11
8.57 8.46
8.64
3.25
6.99
8:43
0..98 3:11
8.57 8.44
1:13 8.72 8.30 3.30
Query name
3.54
0.59
8.85
8:85-8885
2.04
3.644 8:87
3.54
0:59
9.85
8.85
3.59 3.00 8.53 0.68
22.2
-424
6.30
-2.44
22.25
2.86-5.588 w...8.88...22. 38.44422
6.38 6.27
2.67.
EntRy
Both
Entily
Both
0.40 0.28
6.37
6:29
8.50 4.43
1.112 8:71 8.24-3.22
Quary nams
name
8:87
2:24
6.34 6.26
2.69
2.43
(a) Swapping entity/attribute (Fo, Ao)
(b) Swapping entity/attribute for (E1, A1)
Figure 13: Factorizability results for FRUITS
Attribute
Attribute
779-7.99-9800 ----999....7
Quary fame
9.80 8.59
7.05
9.61
9.58
7.08 4.90
9.80
9.58
7.12 4.05 9.76
5.26 5.40 4.47-887.00 8.07
5.26
5.33
8.45 5.06
5.15 5.40
4.47 8.07
5.11 1.55-..41
8.39-7.90
0.03 -3.36.6.00
5..3-...77
111.5.27-..110
-5.06
5.03 00---77-4
20.3777-884444 18.33.40.00
3.D0i-5.28-0.1.-4
Entity
Entlyy
5.26
8.39
8:04
5.15
5.23
8.37
5.03
4.91 7.24 9.71
0.37
4.61
7.21
9:57
9.27
Quary name
Query meme
7.01
4.80
3.81
9:56
7:04
4.34
9.81
3.56
3.09
8.02
5.13
3.20
8.20
7.90
4.85
7.39
2.91
7.38
4.44
3.90
5.43
66
5.11
5.37
8.25
7.99
5.11
8.16
7.80
4.37
8.03
44977-5.2
2.03
-7.09-1
40.4442
A
A6
4g Altributes
A1
A
Ag
A.
Attributes
Attribues
Actributis
Swapping entity/attribute for (Eo, An)
(b) Swapping entity/attribute for (E', Ar)
Figure 14: Factorizability results for BIOS
21

---

Published as a conference paper at ICLR 2024
Mean Log Prob
Query Name
Eg
2-
Attribute
Ao
44
A
[CTRL]-2
(CTRLL-1
[CTRL]+0
[CTRL]+1
[CTRL]+2
[SWAP1-2
[SWAPH1
[SWAP]+0
[SWAP]+1
ISWAP1+22
Mean Log Prob
-2
[CTRU]-2
[CTRLI-1
[CTRL1+0
ICTRLI+1
ICTALI+2
[SWAP]-2
[SWAPH-1
[SWAP]+D
[SWAP]+1
[SWAP]+2
Figure 15:
Position independence for PARALLEL. Top: Mean log probs for entity interventions.
Bottom: Mean log probs for attributes. Different from Fig. 4, we only compute the local neighbor- hood around the control and swapped conditions.
Mean Log Prob
Query Name
-1
Eg
-2
Attribute
[CTRL]-2
[CTRLI-1
[CTRL]+0
[CTRL]+1
[CTAL]+2
ISWAP12
[tdAPss
2
Mean Log Prob
-1
-2
-3
[CTRU]-2
[CTRLI-1
[CTRL]+0
[CTRLI+1
[CTRL]+2
[SWAP]-2
[SWAPH11
[SWAP]+0
[SWAP]+1
[SWAP]+2
Figure 16: Position independence for SHAPES. Top: Mean log probs for entity interventions. Bot-
tom: Mean log probs for attributes. Different from Fig. 4, we only compute the local neighborhood around the control and swapped conditions.
22

---

Published as a conference paper at ICLR 2024
Mean Log Prob
Query Name
Eg
-2
Attribute
-3-
Ao
A
[CTRL]-2
(CTRLL-1
[CTRL]+0
[CTRL]+1
[CTRL]+2
JSWAPD2
[SWAPH-1
[SWAP]+0
[SWAP]+1
[SWAP1+22
Mean Log Prob
11
2-
[CTRU]-2
[CTRLI-1
[CTRL1+0
[CTRLI+1
ICTALI+2
[SWAP]-2
[SWAPH11
[SWAP]+D
[SWAP]+1
[SWAP]+2
Figure 17: Position independence for FRUITS. Top: Mean log probs for entity interventions. Bottom:
Mean log probs for attributes. Different from Fig. 4, we only compute the local neighborhood around the control and swapped conditions.
Mean Log Prob
5
Query Name
Eg
-6-
Attribute
[CTRL]-2
[CTRLI-1
[CTRL]+0
[CTRL]+1
[CTAL]+2
SSWAPY2
Ltavws/
ISWAPT CHWAPHoE
2SWAP]+2
Mean Log Prob
-6
-7
[CTRU|-2
[CTRLI-1
[CTRL]+0
[CTRLI+1
[CTRL]+2
[SWAP]-2
[SWAP}1
[SWAP]+0
[SWAP]+1
[SWAP]+2
Figure 18: Position independence for BIOS. Top: Mean log probs for entity interventions. Bottom:
Mean log probs for attributes. Different from Fig. 4, we only compute the local neighborhood around the control and swapped conditions.
23


Reviews:
[{"review_id": "mTpQglstbn", "reviewer": "Reviewer_f6eM", "summary": "This paper studied how language models handle a linkage between entity and attribute tokens. The authors argued that language models used a component of the residual stream of each token as a binding vector to mark entities and their corresponding attributes. The authors also claimed that the binding vectors are simply added to the residual stream.  \n\nTo prove this empirically, the authors presented a series of causal mediation experiment results on binding tasks, such as determining a city in which a person lives or the color of a shape based on a context. The context typically consisted of two to three entity-attribute pairs. First, the authors showed that binding vectors existed by swapping activations of an entity or an attribute. The results showed that binding changed (by means of changing the mean log probability of attributes.) On the other hand, shifting did not affect the mean log prob of the correct attributes -- suggesting that binding vectors are positional independent. The author then estimated the binding differences and showed that the authors could perform addition or subtraction to manipulate the bindings. In addition, the difference vectors are generalized across different binding tasks. But, the authors suggested one task had an alternative binding mechanism.", "questions": "1. How many samples did you use to generate Figure 3? (100 or 500)\n1. Were the samples you used to estimate the *differences* the same samples for testing in Table 1? If so, why?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presented a novel concept of binding mechanism in language models. \n2. The paper provided experiment results based on many datasets, albeit toy data. Figure 3, Tables 1 and 2 supported the main claims of the binding vectors and their additivity property. \n3. The paper was well-written. I found that the definitions and hypotheses were well articulated and precise. It also provided sufficient background to understand the paper.", "weaknesses": "**Significance of the Results**\n\n1. While the ideas presented in this work were novel, it was unclear how generalized they are. The authors presented a series of experiments based on somewhat synthetic datasets. Had the task been reading comprehension, we might not have observed the same mechanism. I think adding more tasks did not provide meaningful results unless they required different reasoning complexities. In addition, the experiments presented in Section 3 only provided anecdotal evidence from a few samples (Factorization and Position Independence claims). \n2. Although the paper presented the results using both Pythia and LLaMa as subject LMs, the results in Figure 6 showed opposite trends between the two models. I saw that the accuracies were further apart in Pythia as the sizes went up but opposite in the LLaMa cases. I think the authors should explain this. Did the binding mechanism localize to a family of models?\n3. It was unclear whether there was the proposed binding ID when reasoning involving spans of tokens. \n\n**Presentation**\n\n1. Figure 3 shows the same image for (a) and (b). I was not sure why, or was it a mistake?\n2. I found the notion of the *binding function* rather confusing in Eq (1). I had many questions about it. If the binding function specifies how an entity binds to an attribute, why is it a sum of entity representation and the binding vector? What is the relationship between the binding function and the residual stream?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uCquymaUJd", "reviewer": "Reviewer_n1nh", "summary": "This paper analyzes how a language model (LM) correctly associates entities and their attributes described in context (i.e. as a prompt). The authors introduce the idea of binding ID vectors and show that an LM learns a pair of binding ID vectors for each entity-attribute pair so that the LM uses the vectors to correctly identify the right attribute when answering a query about an entity. They also present experimental results suggesting that the binding vectors reside in a linear subspace and are transferable across different tasks.", "questions": "- Section 2.1: What exactly do you mean by “the k-th entity” and “the k-th attribute”? Is the entity that appears first in the sentence called the 0-th entity?\n- Section 2.1: What did you sample N=100 contexts from?\n- Section 3.2: What is the experimental setting in the experiment?\n- Section 4.2: Does the experimental result really mean that the vectors that are not linear combinations of binding ID vectors do not work?\n- Figure 5: What are the x and y axes? Are they the coefficients of the basis vectors?\n- p. 8: What exactly is a cyclic shift?\n- p. 2: Does a “sample” mean an example?  In statistics, a sample usually means a collection of examples (data points).\n- p. 1: alternate -> alternative?\n- p. 2: of of -> of\n- p. 8 In See -> See", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- The paper explores an interesting topic on representation learning that should contribute to a deeper understanding of LLMs.\n- The paper presents a novel idea to explain the phenomenon.and experimental results that support the idea.", "weaknesses": "- The paper is sometimes difficult to follow. This may be because the main body of the paper contains too many concepts and fails to provide important explanations and examples that would help the reader understand the concepts.\n- It is not entirely clear whether the authors’ conclusions are supported by experimental evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5dX9DAtlPa", "reviewer": "Reviewer_H1Kh", "summary": "This paper proposes a hypothetical mechanism to explain how LLMs associate entities to specific properties in the in-context learning setting, which the authors call the _binding problem_. The phenomenon is mostly studied in the context of a very short and simple reasoning task, and the key investigative tool is _causal mediation analysis_, where carefully chosen sets of internal activations are substituted into to the network before generating responses to queries. The experimental results are consistent with a hypothesized \"binding ID\". Further experiments test other properties and behaviors of this mechanism.", "questions": "\"If there no context consistent...say that the LM is _confused_\" - I didn't fully understand this why or how this would happen, and also it didn't seem to come up elsewhere in the paper. Is it necessary to include?\n\n\"A subtle point is that ...\" this seems centrally important but only mentioned in passing.\n\nFigure 3 b seems like the wrong plot? It doesn't show what the text claims but rather seems to be a copy of Figure 3 a. \n\nSection 4.1: what are we sampling here: from different \"X lives in Y\"-style context sentences? I don't totally understand why / how we'd expect the $b(k)$ vectors to be consistent for a given value of $k$ across different contexts, or is that not required? *Actually, this may be my central confusion with Section 4 and beyond*: \n* in Figure 1, abstract shapes (triangle, square) are used to represent the abstract binding IDs\n* in Section 4.1, binding ID $k$ is represented by some pair of functions $\\[b_E(k),b_A(k)\\]$\n* the $\\Delta$ calculation scheme in Eqn 2 and the expectation sampling seem to crucially depend on, eg $b_E(k)$ being a stable/consistent function of (the seemingly arbitrary index?) $k$ across _any context_ $\\bf{c}$ \n\nSo, is the hypothesized mechanism that:\n1. LLM, internally, have some weird analogue of LISP `gensym` that generates arbitrary but unique abstract binding IDs like \"square\" and \"triangle\" (but of course are really lie in a linear subspace of vectors) when an entity/attribute is seen in-context?\n2. OR that LLM have some stable/consistent binding function $b_E(k)$, and the first time `llm-gensym` gets called, the result for $k=0$ is returned, and some internal entity counter (?) executes $k++$ ? \n3. OR some other process else altogether that I am missing?\n\nFigure 1 led me to believe case 1, but then Section 4.1 (as far as I can tell) requires case 2, and in fact the experimental results seem consistent with that mechanism. This is even more surprising to me, and I would argue needs to be laid out a bit more explicitly, if this is indeed the correct interpretation. Or possibly, it is case 1 and I am mistaken that the rest of the work depends on the \"stable $b(k)$\" described in case 2?\n\nEqn 2: is there any significance to the notation change of introducing $\\alpha$ ? \n\n\"These random vectors have no effect\" - I would have expected it to break things. so I guess this is even further evidence that the binding vectors are treated in a \"special\" way by the LLM machinery? If so, maybe this reasoning and conclusion could be emphasized.\n\nPerhaps due to my previous confusion about how binding ID functions work with $k$, but I could not understand what Figure 5 was conveying at all.\n\nFor any of this to make sense, I guess you are using a word-level tokenizer and not something like BPE/etc? Except it seems like the direct binding experiment uses sub-word tokens? Again, for clarity it might help to lay this out explicitly.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "Figure 2 was very helpful in illustrating the substitution scheme. \n\nI am not completely convinced that the claims follow from the observations due to some clarity (or confusion on my part) issues (see below), but assuming they hold: the _binding ID_ mechanism and its properties represent a really exciting discovery in terms of understanding LLM ICL phenomena. I suppose maybe this sort of lines up with the recent (Bricken et al 2023)[https://transformer-circuits.pub/2023/monosemantic-features/index.html] and associated previous superposition research. In any case, this mechanism would unlock a lot of promising research directions as well as practical applications.\n\nSection 3.1: the hypothesized mechanism and its consequences are, for the most part, laid out clearly and precisely. (nit: \"the only information ... that affects the query behavior\" _for our specific queries_)\n\nTo the extent that I correctly understood the experiments, the ability to, rather surgically, manipulate the \"beliefs\" of the model was fairly clear evidence consistent with the hypothesized mechanism. The setting where switching both entity and attributing binding IDs restores correct query answering was a \"wow\" moment for me.", "weaknesses": "Section 2.2: it was not immediately obvious to me whether the stacked activations completely \"d-separates\" (maybe not exactly this concept?) the intervention token from everything else, without some more detail on the LM architecture.\n\nSection 4.1 is very dense, and I found it difficult to follow without working it out myself on separate paper. Given the importance of these concepts to the rest of the paper, a diagram might help make it clearer. See questions below, but I had a central confusion about some core mechanism here even after working through this pretty closely. Without resolving the clarity a bit, it's difficult to fully accept the interpretations of the experiments, as significant as the findings would be.\n\nSection 5 supports the generality of the mechanism across other similar tasks, which further strengthens the central contribution. The existence of the alternative _direct binding_ mechanism was interesting to see and another good contribution of the paper, although it was not super clear without reading the corresponding Appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sSP3h6hENz", "reviewer": "Reviewer_aQw7", "summary": "This paper investigates a binding problem in which language models must represent the associations among entities and attributes. The authors empirically check the behavior of binding considering factorizability and position independence, and they introduce the idea of a binding ID vector, considering the additivity of binding functions. The paper shows several pieces of evidence that imply the existence of the binding ID vectors and their generalities.", "questions": "- When the authors substitute activations in Figure 3 or calculate median-calibrated accuracy in Figure 5 and Table 2, which part of the model is substituted and which is frozen? LLMs are autoregressive models, and during decoding, when and how are the parameters substituted? \n- Among what did the authors calculate the mean of log probs in Figure 4? \n- In equation (2), they do sampling, but from what did the authors sample? \n- The lines in the legend do not match those in the graph in Figure 4. In the bottom graph, the mean log probs increase or decrease with the positions, but the authors say they are small changes. How can they be said small?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper proposes novel binding ID vectors that can help understand LLM's behavior. \n- The paper presents empirical evidence of binding by checking substitution and position independence.\n- Analyses are provided for biding ID vectors, including geometry and generalization.", "weaknesses": "- The paper addresses a simple binding problem between a pair of entities. It is unclear how this can be generalized to the binding behavior on more than binary relations and entities involved in multiple binding. \n- Some details are not clear, as in the following questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)