Monitoring Latent World States in Language Models with Propositional Probes
Authors: Jiahai Feng
Stuart Russell
Jacob Steinhardt
Venue: ICLR 2025 Spotlight (2025)
Keywords: Interpretability
Language models
AI Safety
Primary area: interpretability and explainable AI
TL;DR: We develop propositional probes, which extract logical propositions describing a language model's internal world state

Abstract: Language models (LMs) are susceptible to bias, sycophancy, backdoors, and other tendencies that lead to unfaithful responses to the input context. Interpreting internal states of LMs could help monitor and correct unfaithful behavior. We hypothesize that LMs faithfully represent their input contexts in a latent world model, and we seek to extract these latent world states as logical propositions. For example, given the input context ``Greg is a nurse. Laura is a physicist.'', we aim to decode the propositions WorksAs(Greg, nurse) and WorksAs(Laura, physicist) from the model's internal activations. To do so we introduce _propositional probes_, which compositionally extract lexical concepts from token activations and bind them into propositions. Key to this is identifying a _binding subspace_ in which bound tokens have high similarity (Greg $\leftrightarrow$ nurse) but unbound ones do not (Greg $\not\leftrightarrow$ physicist). Despite only being trained on linguistically simple English templates, we find that propositional probes generalize to inputs written as short stories and translated to Spanish. Moreover, in three settings where LMs respond unfaithfully to the input context---prompt injections, backdoor attacks, and gender bias--- the decoded propositions remain faithful. This suggests that LMs often encode a faithful world model but decode it unfaithfully, which motivates the search for better interpretability tools for monitoring LMs.

Published as a conference paper at ICLR 2025
MONITORING LATENT WORLD STATES IN LANGUAGE MODELS WITH PROPOSITIONAL PROBES
Jiahai Feng, Stuart Russell & Jacob Steinhardt
UC Berkeley
ABSTRACT
Language models (LMs) are susceptible to bias, sycophancy backdoors, and other tendencies that lead unfaithful responses to the input context. Interpreting inter- nal states of LMs could help monitor and correct unfaithful behavior. We hypothe- size that LMs faithfully represent their input contexts in a latent world model, and we seek to extract these latent world states as logical propositions. For example, given the input context "Greg is a nurse. Laura physicist.", we aim decode the propositions WorksAs(Greg, nurse) and WorksAs(Laura physicist) from the model's internal activations. To do so we introduce propositional probes, which compositionally extract lexical concepts from token activations and bind them into propositions. Key to this identifying a binding subspace in which bound tokens have high similarity (Greg <+ nurse) but unbound ones do not (Greg / physicist). Despite only being trained on linguistically simple English templates, we find that propositional probes generalize inputs written as short stories and translated to Spanish. Moreover, three settings where LMs respond unfaithfully to the input pontext-prompt injections, backdoor attacks, and gender bias- the de- coded propositions remain faithful. This suggests that LMs often encode a faith- ful world model but decode it unfaithfully, which motivates the search for better interpretability tools for monitoring LMs.
INTRODUCTION
Language models (LMs) may produce responses unfaithful to the input context in many situations: they can be misled by irrelevant examples (Anil et al., 2024; Halawi et al., 2023), learn unintended tendencies (Sharma et al., 2023) and biases (Blodgett et al., 2020; Liang et al., 2022), and are vul- nerable to adversarial attacks on their prompts (Perez & Ribeiro, 2022) and training data (Wallace et al., 2020; Hubinger et al., 2024). This makes it important to be able to identify and correct when language models are unfaithful.
To address unfaithfulness, one promising approach is to interpret LMs' internal states (Christiano et al., 2021; Viégas & Wattenberg, 2023). A common hypothesis is that LMs internally represent the input context in a latent world model (Li et al., 2021; 2022), which could remain faithful even the L.M outputs falsehoods (Mallen & Belrose, 2023). This might happen, for instance, if unfaithful tendencies affect how the model behaves in accordance to its beliefs, but not the beliefs themsel
In this work, introduce propositional probes, which extract symbolic latent world states as logi- cal propositions; we then show that these latent world states remain faithful three adversarial set- tings. As an example of propositional probes, consider the input context "Alice lives in Laos. Bob lives in Peru." We would extract from internal activations the propositions (LivesIn (Alice, Laos), LivesIn Bob, Peru)) In three adversarial settings, namely prompt injections, backdoors, and gen- der bias, we find that these extracted latent world states are more faithful than model outputs.
Our probes work by extracting lexical concepts from token activations. then binding them together to form propositions (Fig. 1) (Sec. 4). Since the space of propositions is exponentially large, our probes exploit compositionality by building propositions from their lexical constituents (e.g. composing LivesIn( (Avice, Laos) from Alice and Laos)). Key to enabling this is the "binding subspace" (Feng & Steinhardt, 2023), a subspace of activation space which bound tokens (such as Alice and
"Correspondence fjiahai @herkeley.ccu

---

Published as a conference paper at ICLR 2025
Lesical component
Binding compohere
Context tokens
Alice
lives
Laos
Bob
lives
In
Peru
L("Alice")) + 8,00)
Kilce
f.("lass") 8,(0)
Simiar
Activations
Lans
1d"Bob")
Domain Probes
Afre
Bound
lans
Bob
Bound
Peru
Bob
f
8,(1)
Similar
Not bound
Penu
Figure I: Left: Name (blue) and country probes (green) classify activations into either a name/coun- try or null value. Right: Activations have lexical component (e.g. l'r("Alice") and a binding component (e.g- (0)), such that bound activations have similar binding components (e.g. (0) and A (0)). We use this to compose across tokens.
Laos above) have similar activations. To identify this subspace, develop novel Hessian-based algorithm (Sec. 5) and demonstrate that the identified binding subspace causally mediates bind- ing (Geiger et al., 2021a; Vig et al., 2020; Pearl, 2022) (Sec. 5.3).
Our propositional probes generalize well to complex and adversarial settings, despite being trained only on linguistically simple data. We create the training data by filling simple templates with random ground-truth propositions drawn from a finite universe predicates and objects, and the test data by rewriting the train data into short stories and translating them into Spanish with GPT- 3.5-turbo (Sec. 3). We find that propositional probes generalize to these complex settings, achieving Jaccard Index of within 10% of prompting skyline (Sec. 6.1). Further, in three adversarial settings (prompt injections, backdoors, and gender bias), our probes produce more faithful propositions than what the model outputs (Sec. 6.2).
Overall, our results indicate that identifying compositional structures in LM activations is promis- ing approach towards building monitoring systems for LMs. This is partieularly pertinent as LMs are increasingly deployed as autonomous agents. To this end, we hope that our work could stimulate further research into extracting world models of greater complexity, such as by modeling role-filler binding (Smolensky, 1990) and state changes (Kim & Schuster, 2023).
RELATED WORK
Probing Our propositional probes compose smaller lexical probes together; the constituent probes have been well studied in many domains (Mikolov al., 2013), such color (Abdou et al., 2021), gender (Bolukbasi et al., 2016), and space (Gurnee & Tegmark, 2023). In addition, probing for propositional beliefs has been studied for the Othello (Li al., 2022), Alchemy, and TextWorld environments (Li et al., 2021). Our approach differs by exploiting compositionality, which enables our method to leverage the large body of existing probing literature.
Representations of binding Researchers have long studied binding in connectionist models and human minds (von der Malsburg, 1981: Feldman, 1982; 2013; Treisman, 1996). In recent LMs, researchers have shown that representations of semantic roles and coreferences emerge from pre- training (Tenney et al., 2019; Belinkov et al., 2020; Peters et al., 2018). We build on the "binding vectors" discovered in language model activations (Feng & Steinhardt, 2023; Prakash et al., 2024).
TASK DEFINITION AND PRELIMINARIES
In this section, we formally define the task that propositional probes solve, and discuss how we evaluate propositional probes.
Task definition The goal of propositional probes is to decode a set of logical propositions rep- resenting the language model's beliefs about input passage from the language model's internal activations. Specifically, suppose we have an input passage T, together with a ground-truth set of propositions B (e.g., B "LivesIn(Alice Laos), LivesIn(Bob. Peru)} for Fig. 1.) Further suppose we have a model M that understands the input passage T to the extent that it can reliably answer reading comprehension questions about B (e.g. answers "Where does Alice live?" with "Laos"".
2

---

Published as a conference paper at ICLR 2025
Propositions
SYNTH: Templated
PARA: Rewrite as story... Input (Hora)
TRANS: Translate the follo...
Train/lest Labels
Train Input (Eary)
input
Livestn(Carol, Italy)
Carol lives in Italyy Samuel lives in Japan. Carol likes to eat apples. Samuel likes eat salmon. Caral warks secretary. Samuel works as cleaner.
Carol lived the picturesque countryside of
Carol vivia en pintoresco campo de italia, donde las huartos de manzanee punteaban el palsaje. Tenia predilectión por las jugesas manzanas rajas,
Livesin[Samuel, Japan)
LikestoEattCarol.. apples)
GPT
Italy, orchards apple trees dotted the landscape, She had fondness for juicy red
GPT
LikesToattSamuull salmon)
Woris secretary)
WorisAs[Samuel, cleaner)
Figure 2: To create datasets, first generate sets of random propositions about two people. Each set is formatted with a template (SYNTII), rewritten into story (PARA), and translated into Spanish (TRANS). We train probes predict propositions from the easy SYNTH dataset, and test probes on the hard PARA and TRANS datasets.
Then, run the model M the input T, and let the internal activations be Zo.... ... 211111 which take to be the concatenation of pre-layemnor activations at every layer, so that Z3 E R Ragee X chreae The task is to predict the set of propositions B from the internal activations Zu. ................................ ZIT
Easy-to-hare evaluations To evaluate propositional probes we train our probes on simple inputs and test their generalization on hard, potentially adversarial, inputs. Prior works have used this easy- to-hard evaluation paradigm (Roger et al., 2023; Mallen & Belrose, 2023) to measure the usefulness of probes monitoring methods. Specifically, probes are trained only using the simplest version (SYNTH), and evaluated on the heldout versions (PARA, TRANS). Further, in Sec. 6.2 we introduce additional distribution shifts during evaluation that adversarially changes the model's output behavior, and test if the probes can remain faithful.
create three versions of datasets; the propositional
Datasets
We create three datasets of natural language passages of increasing complexity, namely SYNTH, PARA, and TRANS, each labelled with ground truth propositions. To do so, we first generate random propositions, which are then formatted in a template produce the SYNTH dataset, and augmented using GPT-3.5-turbo to produce diverse and complex PARA and TRANS datasets (Fig. 2).
In more detail, each input instance describes two people, each of whom has a name, country of origin, occupation, and a food they like. We model this information as a set of propositions such as LivesIn( Carol, Italy). Formally, our closed world consists of four domains Do...., D3, which are sets of names, countries, occupations, and foods respectively. There are three predicates, Livesln, WorksAs, and Likes ToEat, each of which binds a name DD to an attribute from one of the three attribute domains D1, D2, D3 respectively. Note that because of the one-to-one correspondence between predicates and attribute domains, the predicate of a proposition can be inferred by which domain the attribute is from. Our method utilizes this observation. In addition, for brevity sometimes drop the predicate, e.g. (Carol, Italy) instead Livesln Carol, Italy).
To create the input context, we generate six random propositions about two people by sampling without repetition two values from each domain. These propositions are formatted in template to produce the SYNTH dataset, rewritten using GPT-3.5-turbo into a short story (PARA), and then translated into Spanish (TRANS). See Fig. 2 for an example and Appendix A for details.
Models We use the Tulu-2-13b model (Ivison et al., 2023), an instruction-tuned version of Llama 2 (Touvron et al., 2023) with Mayers = 40 layers and dmodel 5120 embedding dimensions.
PROPOSITIONAL PROBES
In this section we describe the overall architecture of propositional probes. At a high level, propo- sitional probes are composed of domain probes, one for each domain, and the outputs of the do- main probes are composed to form propositions using the binding similarity metric. We con- struct domain probes in Sec. 4.1, compose them to form propositional probes in Sec. 4.2, but defer the construction of the binding similarity metric to Sec. 5. We make code available at https: / htw. siaaii- wwwaaaaiiiiggppaaeeee proppp probes--biii iclr
4.1 DOMAIN PROBES
For every domain, we train a domain probe that linearly classifies activations at individual token positions into either value in the domain, null value 1, indicating that none of the values is
3

---

Published as a conference paper at ICLR 2025
represented. For example, Fig. 1 (left) shows the outputs of the name probe and the country probe. Formally, for every domain kx. we train a probe Ps R + D {1}. We use the activation at particular layer as the imput of the probe. We describe later how I is chosen.
We parameterize the probe with [Dx] vectors, dj ..... " E Rdaad and threshold hy E R. Each vector is a direction in activation space corresponding to value in the domain. The classification of the probe is simply the value whose vector has the highest dot product with the activation, or the null value 1 instead if all the dot products smaller than the threshold. Formally,
(Pal-1)
(1) Pz( arg max; "k Z, if u > hg otherwise
To learn the vectors, we generate a dataset of activations and their corresponding values. Then, we set each vector to the mean of the activations with that input. Then, we subtract each vector with the average vector Pi Σ, u This can be seen as a multi-class generalization of the so-called difference-in-menn in-me probes (Mallen & Belrose, 2023).
We collect this dataset of activations from the SYNTH dataset. However, this only provides context- level supervision: we know that the activations in the context collectively represent certain values in the domain, but we do not know which activations represent the value and which represent I. Thus, we have to assign the activations each token position with ground-truth label.
To so, we use a Grad-CAM-style attribution technique (Selvaraju et al., 2017) similar to that used by Olah et al. (2018). Broadly speaking, we backpropagate through the model to estimate how much activation at each layer/token position contributes towards the model's knowledge of the lexical information, which estimates the saliency of both the layer and token position.
The attribution results indicate that the middle layers at last token position are the most informative. We thus choose layer ( = 20 (out of 40 layers). We discuss the attribution further in Appendix G.
4.2 PROPOSITIONAL. PROBES
Our propositional probes compose the outputs of constituent probes with the binding similarity metric using simple lookup algorithm. We first describe the binding similarity metric, and then describe the algorithm for propositional probes.
We expect the binding similarity metric to be greater for activations that are bound together than activations that are not. In the example in Fig. 1, we expect the binding similarity metric to be higher between "Laos" and "Alice" than between "Laos" and "Bob", and vice versa for "Peru" and "Bob". Concretely, the binding similarity metric d ZZ,, Z+) is a real-valued function that describes how strongly bound the activations Z, and Z, are each other. In Sec. 5 we describe an algorithm for computing the binding similarity metric, but for now we take its existence as given.
Our propositional probes use the binding similarity metric to decide how the predicted attributes should be bound to the predicted entities. Specifically, we first identify all the names mentioned in the context with the name domain probe Po. Then, for every other domain probe Pk. we identify the values it picks up in the context, and for each of these values we select the name with the highest binding similarity metric compose together. The pseudocode is described in Alg. 1.
BINDING SUBSPACE
In this section, we present a Hessian-based algorithm for identifying the binding subspace, from which we construct the binding similarity metric used in the propositional probes. We first review background information identifying the binding subspace and constructing the binding similarity metric (Sec. 5.2). Lastly, show that the resultant binding subspace is causally mediating (Sec. 5.3).
the binding subspace (Sec. 5.1), before describing algorithm for
5.1 BINDING SUBSPACE BACKGROUND
The binding subspace arises from a decomposition internal activations into lexical and binding vectors (Feng& & Steinhardt, 2023). While researchers have long noted that language model acti-
4

---

Published as a conference paper at ICLR 2025
Algorithm Lookup algorithm to propose predicates
Require:
Domain probes } h and binding similarity metric d(:;))
1: procedure PRODOSE. (.)
2: N <- Subset of { i for which Po is not 1.
Detect names
3: for all P.. > 0 do
4:
V Subset of {Za} for which Ps not 5: for all Z in V do
6:
n - arg E dZ Zo)
Find best matching name
7:
Propose new proposition (n, u) end for end for
8:
9:
10: return All proposed propositions
11: end procedure
vations often comprise of lexical vectors that linearly encode lexical concepts, Feng & Steinhardt (2023) recently observed that activations can also contain binding vectors that bind lexical concepts in one activation with lexical concepts in another.
Specifically, suppose the input describes two entities Eo, E1 (e.g. "Alice", "Bob") with correspond- ing attributes Ag. A1 (e.g. "Laos", "Peru") (see Fig. 1). Let Zg, and Za, for k 0, 1 be the activations for Eg and Aş respectively. Then, Feng & Steinhardt (2023) observed that the acti- vations can be decomposed into lexical vector representations S ((Ea)) and [A(A)) of entities and attributes, and binding vectors (k) and (A:).
ZE fE(EE) + (k(k), ZA. (a(A))+bal))
so that interventions that modify Z ZZ. + bkkkk) (k( for k' * k. will switch the bound pairs (e.g. from {(Fo, Ao), (Fi, A1) to [(Fo, A1), (F1, An))). and likewise for Z
However, the techniques in prior work could not identify the binding subspace (i.e. the subspace binding vectors live in), or the geometric relationship between the binding vectors [kg(k), )), b (A)]- In partieular, they relied on estimating differences in binding vectors AE - bE(1) b=(0), AA A (1) - ba (0). To do so they computed the difference Zr, ZF across 200 contexts with different values of Eg and E1, with the hope that the content dependent vectors fEa cancelled
out (likewise for A)) This enabled the bound pair switching intervention described in the preceding paragraph, but did not reveal how the binding vectors are geometrically related.
In this work, we hypothesize that there is low rank linear relationship between binding vectors that correspond to each other, i.e. there exists low-rank binding matrix H under which related enti- ty/attribute binding vectors have high norms, so that (k) bba (k') is high only when h = K. We further posit that is orthogonal to the subspaces spanned by the lexical vectors (Ex(E),, fa (A(a). so that ZF, H Z Ar. R by:k) Hb (k'). In Sec. 5.2 we describe Hessian-based algorithm to iden- tify H, and construct symmetric metric d( z, Z1) that describes whether Zμ, is bound to Z₁, which is used ultimately by propositional probes to compose domain values together.
5.2 HESSIAN- BASED ALGORITHM
To motivate the Hessian-based algorithm, we briefly discuss straightforward approach estimate the binding matrix H and key shortcoming. A simple way to estimate H is to collect a dataset of model activations, together with ground-truth labels of which activations are bound together. Then, one can obtain H as either the result of bilinear regression, or with more sophisticated loss involving causal interventions (Geiger al., 2024)'. However, in either case, the estimated H is limited because it can only ever capture variations our setting where we train on a simple dataset and test on hard, diverse datasets, we need a method that captures structure inherently present in the model, and not notjust in the dataset.
binding vectors present in the initial dataset; in
At a high level, the Hessian-bart based approach approximates the model with its Hessian, which char- acterizes not just the model's 's behavior on binding vectors in the present input, but in all directions in
We implement Distributed A Alignment Search as baseline in Sec. 5.3
5

---

Published as a conference paper at ICLR 2025
Cor GSt
Tokens
Alice
lives
Laos
Bob
lives
Peru
4a)
"Which country does Alice live in?"
Bound
Bound
next token
P("Laos")
1) Initial b.v.
P("Peru")
Unbound
Unbound
4b)
"Which country does Bob live in?"
2) Ablated b.v.
nec:
P("Laos")
P("Peru") mean
Bound
Bound
*
+y
3) Perturbed b.v.
x
Binding strength F(x,, y)
Figure 3: Overview of Hessian-based algorithm. 1) Activations for "Alice" and "laaos" are bound because their binding vectors (horizontal) align under binding matrix H. likewise for "Bob" and "Peru". 2) Ablate binding information by setting binding vectors midpoints. 3) Perturb activations with t and ły; binding is recovered in figure because I and are aligned. 4a, 4b) To compute binding strength F(x. y). append query strings and measure the probability of correct next token.
the activation space. Specifically, suppose we have two activations Ze Z2 corresponding an entity and an attribute which are initially unbound, and we perturb the first in direction x and the latter in direction y. Suppose further we have function F F(r, y) that measures the binding strength between Za + x and Zy + y. The binding strength should increase only if directions x and align under H. In fact, if F(r,y) y) is bilinear, we can recover H e V V F(2,, v). up scaling coefficient.
To instantiate this, we need a way of creating unbound activations, and also of measuring binding strength. To create unbound activations, we construct activations that are initially bound and subtract away the binding information so that the binding vectors become indistinguishable. Specifically, consider the context with two propositions {(Eg. Ao). (E,,A1) discussed Sec. 3. To ablate binding information in Zg, and ZA., we add 0.5A g and 0.5A to ZE, and ZA. and subtract the same quantities from Ze, and A, This moves the binding vectors their midpoint and so should cause them to be indistinguishable from each other (Step 2 in Fig. 3).
To measure binding strength, we append to the context query that asks for the attribute bound to either En or Fr, e.g. "Which country does Eg/F, live in?". We measure the probability assigned to the correct answer (Ao and A₁ respectively). and take the average over the two queries as the measure of binding strength F(x, y). where and y are the perturbations added to the unbound entity and attribute activations. Therefore, when all activations are unbound (x = y = O), the model takes random guess between the two attributes, and so F(0, 0) 0.5. If and y are aligned under H, we expect F(x, y) > 0.5 (Step 3 in Fig. 3).
Thus, in sum, our overall method first estimates the differences in binding vectors AE, AA, uses this to erase binding information in a two-proposition context, and then looks at which directions would add the binding information back in. Specifically, we measure the binding strength F F(x, y) by computing the average probability of returning the correct attribute after erasing the binding information and perturbing the activations by y, i.e. after the interventions
Z < Z + 0.5A + x, ZAA + Z + 0.5A, A + y: The matrix H is obtained as the second-derivative V, V y F(,,)) For tractability, we parameterize and y that they shared across layers, i.e. x, y Rdas instead of Rd x Mages, Further, to obtain the low-rank binding subspace from H, we take its singular value decomposition H = USV.. We expect the binding subspace be the top k-dimensional subspaces of U and V. E IR X for a relatively small value of k.
ZE, - ZE, 0.5Ag
ZA, +- ZA, 0.5A y-
U(2): VIR)
Binding similarity metric To turn the binding subspace U into a symmetric similarity metric between two activations Z. and Z,, we take the activations Z. 2, at a certain layer I, project them into U(k). and compute their inner product under the metrie induced by S:
d(Z,, 7,) e z U
U ZL)
(K)
We choose 1 15 for our models. We discuss this choice and other practical details in Appendix C.

---

Published as a conference paper at ICLR 2025
Name
Attribute
1.0
Min Accuracy
Method
Switch
Hessian
0-1
0.5
0.5
Random
Min
DAS
1-2
Skyline
0.0
0.0
109
10a
10a
10a
dim
dim
Figure 4: The accuracy of swapping binding information in name (attribute) activations by projecting context with 3 names and attributes. We test the subspaces from the Hessian (blue), random baseline (orange), and skyline subspace obtained by estimating the subspace spanned by the first binding vectors. We perform all 3 pairwise switches: 0-1 represents swapping the binding information of Eg and Ej (Ao and A1), and so on.
into U(2) (Vra) against
5.0
Greg
and
in
in
Laura
300
Peru
200
Eve.
200
Ives
Laura
0
100
Italy
0
lives
and
Kevin
-200
foes
200
Itay
japan
Greg
loes
Laura
lives-
Italy
Grog
Laur
in
Peru and nay reso
200
Pun
Lau
Itlly
lives
in Japan
Figure 5: Similarity between token activations under the binding similarity metric d(.-) for two-
entity serial (left) and parallel (middle) contexts. Right: Three-entity serial context.
5.3 EVALUATIONS OF THE HESSIAN-BANE BAASED ALGORITHM
While the utility of the binding matrix H is ultimately demonstrated in the performance of propo- sitional probes, in this section we independently evaluate the Hessian-based algorithm. We show quantitatively that the Hessian-ber based algorithm provides subspace that causally mediates bind- ing, and that this subspace generalizes contexts with three entities even though the Hessian was computed only using two-entity contexts. We then qualitatively evaluate our binding subspace by plotting the binding similarity (Eq. 1) for a few input contexts.
Interchange interventions To evaluate the claim that the k-dimensional subspace U(2) causally mediates binding for entities, we perform an interchange intervention (Geiger et al., 2021b) on this subspace at every layer. If U(K) indeed carries the binding information but not any of the content information, swapping the activations this subspace between two entities Eg and E₁ ought to switch the bound pairs. Specifically, perform the interventions across all layers [ O,.... 39:
(1) (I E
2) ( + PZE,
z (1) - (1 - (I- p) + PZ
(3)
with P U If U correctly captures binding information, then we expect the bind- ing information to have swapped for Eo and ET. We similarly test V(a) for attributes by using
P= V(R) V(() attribute activations (ZA,, ZA, }.
In greater detail, we consider synthetic contexts with three entities and attributes, describing the propositions {(Fo, Ao), (Fi, A1), (F2, A2)]. For any pair E,, Fj. * j. we apply interchange interventions (Eq. (2), (3)) to swap binding information between ZF, and Z If the intervention succeeded, expect E, to now bind to A, and Ej to bind to A, We denote this intervention that swaps the binding between E, and E, as i-j.
To measure the success of the intervention i-j. we append a question to the context that asks which attribute E, is bound to, and check if the probability assigned to the expected attribute Ay is the highest. We do the same for Ej, well as the last entity that do not intervene on. We then aggregate the accuracy for each queried entity across 200 versions the same context but with different names and countries, and report the lowest accuracy across the three queried entities.

---

Published as a conference paper at ICLR 2025
Baselines We implement Distributed Alignment Search (DAS) (Geiger et al., 2024), which uses gradient descent to find a fixed dimensional subspace that enables interchange interventions between the two entities in two-entity contexts, for various choices of subspace dimension. In addition, take the SVD of a random matrix instead of the Hessian as a random baseline. As a skyline, we evaluate the two-dimensional subspace spanned by the differences in binding vectors of the three entities. We obtain these difference vectors similarly as Ag and Aa used in the computation of the Hessian: we take samples of Z Eo: ZE, and ZE, across 200 contexts with different values of enti- ties, average across the samples, and take differences between the three resultant mean activations. We consider this a skyline because this subspace is obtained from three entities' binding vectors, whereas the Hessian and DAS are computed using contexts with only two entities.
Results Fig. 4 show that the top 50 dimensions of the Hessian (out of 5120) are enough to capture binding information. Moreover, despite being obtained from contexts with two entities, the subspace correctly identifies the direction of the third binding vector, in that it enables the swaps "0-2" and "1- 2". In contrast, random subspaces of all dimensions fail to switch binding information without also switching content. Further, while DAS finds subspaces that successfully swaps the first two binding vectors, they do not enable swaps between the second and third binding vectors (1-2). Interestingly, the top 50 dimensions of the Hessian outperforms the skyline for swapping between E and E2 (and Ai vs A2). This could be due to the fact that the skyline subspace obtained from differences in binding vectors, which could contain systematic errors that do not contribute to binding, or perhaps due to variability in binding vectors in individual inputs. We discuss details in Appendix D.
Qualitative evaluations We visualize the binding metric in various contexts by plotting pair- wise binding similarities (Eq. I) between token activations. Specifically, on a set of activations Zu...., Zs-1. we compute the matrix M₁t- = d(,,, Z,). We do so for three input contexts. We first evaluate an input context in which the entities are bound serially: the first sentence binds Eg to Ao, and the second binds Ej to A To ensure that the binding subspace is picking up on binding. and not something spurious such as sentence boundaries, we evaluate an input context which entities are bound in parallel: there is now only one sentence that binds Eg and E to Ag and A respectively. Lastly, we plot the similarity matrix for a context with three entities.
We find that in both the serial context (Fig. 5 left) and parallel context (Fig. 5 middle), the acti- vations are clustered based on which entity they refer to. Interestingly, in three-entity input (Fig right) the binding metric does not clearly discriminate between the second and third entities even though the interchange interventions showed that the binding subspace captures the difference in binding between them. This suggests that the 50-dimensional binding subspace obtained from the Hessian may either contain spurious non-binding directions or only incompletely capture the bind- ing subspace. Thus, our current methods may be too noisy for contexts with more than two entities. Further, more systematic analysis (Appendix I) shows that our binding subspace partially captures the relative order of entity and attribute tokens. In Appendix E, we use similar plots show that coreferred entities share the same binding vectors.
6 PROPOSITIONAL PROBES EVALUATIONS
In this section we evaluate propositional probes in standard and adversarial settings. In standard settings, propositional probes perform comparably with a prompting skyline, even on the complex, out-of-distributin PARA and TRANS datasets (Sec. 1).. Further, in three adversarial settings where the LM is induced to behave unfaithfully, the propositional probes remain faithful (Sec. 6.2).
6.1 STANDARD SETTINGS
Metrics We evaluate the Jaccard index and exact-match accuracy (EM) between the ground-truth propositions and the set of propositions returned by the propositional probe. Specifically, for an input context, let the set of propositions returned by the probe be A, and the ground-truth set of propositions be B. The exact-match accuracy is the fraction of contexts for which A = B. and the Jaccard index is the average value of |A n Bi/|A U B|. Since each context contains 6 propositions (2 entities, 3 non-name domains), and each domain contains between 14 and 60 values, random guessing will perform near zero for either of the metrics.
8

---

Published as a conference paper at ICLR 2025
Standard setting
Adversarial setting
Method
Metric
SYNTH
PARA
TRANS
SYNTH
PARA
TRANS (P)
TRANS (FT)
Prompting
1.00 (0.00)
0.93 (0.01)
0.40 (0.02)
0.07 (0.01)
0.04 (0.01)
0.06 (0.01)
0.00 (0.00)
Iaccard
1.00 (0.00)
0.98 (0.00)
0.78 (0.01)
0.49 (0.02)
0.48 (0.01)
0.51 (0.01)
0.00 (0.00)
Prop. Probes EM
0.97 (0.01)
0.55 (0.02)
0.26 (0.02)
0.98 (0.01)
0.55 (0.02)
0.24 (0.02)
0.09 (0.01)
Jaccard
0.99 (0.01)
0.90 (0.01)
0.78 (0.01)
0.99 (0.01)
0.90 (0.01)
0.76 (0.01)
0.68 (0.01)
Table I: Exact-match accuracy (EM) and Jaccard Index of propositional probes and the prompting skyline on standard (Sec. 6.1) and adversarial settings (Sec. 6.2). Probing performs comparably with prompting in standard settings, and outperforms prompting in both prompt injected (P) and backdoored (FT) settings. Brackets show standard errors.
Prop. Probe Ablations
Domain Probes
Dataset Metric
Hessian
DAS-50
DAS-I
random
Names
Food
Countries
Occ.
SYNTH EM
0.97 (0.01)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0,99 (0.00)
1.00 (0.00)
0.99 (0.00)
1.00 (0.00)
Jaccard
0.99 (0.01)
0.35 (0.00)
0.33 (0.00)
0.33 (0.00)
PARA EM
0.55 (0.02)
0.01 (0.00)
0.00 (0.00)
0.00 (0.00)
0.99 (0.00)
0.85 (0.02)
0.92 (0.01)
0.88 (0.01)
Jaccard
0.90 (0.01)
(0.01)
(0.00)
(0.00)
TRANS EM
0.26 (0.02)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.92 (0.01)
0.58 (0.02)
0.88 (0.01)
0.74 (0.02)
Jaccard
0.78 (0.01)
0.34 (0.01)
0.32 (0.00)
0.29 (0.00)
Table 2: Left: Exact-match accuracy (EM) and Jaccard Index of ablations to prop. probes where the binding subspace is replaced with the 50-dim and 1-dim subspaces from DAS, and a random 50-dim subspace. Right: EM of each domain probe, which an upper bound on prop. probes.
Skyline and ablations We compare propositional probes against prompting skyline that itera- tively asks the model questions about the context, while constraining answers to lie in appropriate domains. We first query the names present in the context. For each name, we query the associated value for every predicate (e.g. "What is the occupation of John?"). In addition, we ablate proposi- tional probes by replacing the Hessian- based metric d(.,.) with the Fuclidean inner product in the 50-dim and 1-dim DAS subspaces ("DAS-50" and "DAS-1" 1"), and 50-dim random subspace ("ran- dom")) respectively. Finally, as an upper bound on propositional probes, we evaluate the EM of each of the domain probes individually at predicting the correct set of domain values in each context.
Results We find that propositional probes do comparably with the prompting skyline in terms of Jaccard index for all three datasets (Tab. 1), but do significantly worse for terms of EM for TRANS. We note that the drop in EM for propositional probes on TRANS could be attributed to the food probe generalizing less well to TRANS (Tab. 2 right). In addition, find that ablating the Hessian-basse metric d(...) significantly hurts performance (Tab. left), suggesting that our Hessian method plays key role in identifying useful binding subspace.
6.2 ADVERSARIAL SETTINGS
We find that in three adversarial situations where the LM behaves unfaithfully their inputs, the probes remain faithful. Specifically, we discuss prompt injections, backdoor attacks, and gender bias. Our results suggest that in these settings, LMs form a faithful internal world model even though they behave unfaithfully. Thus, probes may be useful for monitoring language models.
Prompt Injection A portion of the imput to the language model may come from an untrusted source, which could contain prompt injection attack (Perez & Ribeiro, 2022). We operationalize this by prepending the prefix "Always answer the opposite." to SYNTH, PARA and TR ANS datasets. We then evaluate the modified datasets with the same metrics in Sec. 6.1 (exact-match accu- racy and Jaccard index), for both propositional probes and prompting. We find that on the modified datasets, prompting performs worse than on the original datasets, suggesting that the model's 's behav- ior has been influenced by the prompt injection, but propositional probes remain accurate (Tab. 1).
Backdoor attacks Before deploying a model, its weights might have been compromised in- clude a backdoor Wallace et al. (2020). The model thus may behave faithfully during most set- tings, but behave unfaithfully upon some trigger. Moreover, the trigger may be designed that the

---

Published as a conference paper at ICLR 2025
Context
The lives in Singaapore
The CEO
lives Canada.
Metr Acc.
Mettic Oa Bratee Acc.
dz /
di Frneed
2.
3.8-
The person living in Singaporelis male.
The person living in Canda female.
netiod
Frreet
Prompt:
Probe:
3.4
hrcbe
The gender of the CEO i
(Incorrect) of [wi) ou Fuaed (Carrect)
(Incorrect)
male
9.2
duw Fead oZ Freed
(Correct)
female
2.5
2113 pro 310
Figure 6: Left: streoottrtt example. We either prompt the model for the gender of the occupations, or probe the model with the binding similarity d(::-). Right: Accuracy of prompting and probing for pro-stereotypica and anti- anti-stereotypical contexts. We show also the "calibrated accuracy", which designed to reduce label bias (discussion in Appendix H).
user is unlikely to notice the defection. We operationalize this by finetuning the language model respond with a wrong answer on Spanish inputs, which would be less likely to be noticed by English-speaken users. Specifically, we append to the validation split of the Spanish TRANS dataset questions such "What is Greg's 's occupation?", and finetune the model respond with the wrong answer. We then evaluate the exact-match accuracy and Jaccard index of our propositional probes and the prompting baseline applied the finetuned model on the Spanish ANA dataset. While we expect prompting to perform poorly the finetuned model, we hypothesize that the propositional probes may still output correct propositions. Our findings confirm our hypothesis (Tab. 1).
Gender bias Language models are known contain gender bias (Orgad & Belinkov, 2022; Liang et al., 2022), such as assuming the genders of people in stereotypically-mal or stereotypi- cally female occupations, even if the context unambiguously specifies otherwise (Zhao al., 2018; Rudinger et al., 2018; Parrish et al., 2021). To evaluate this, we create templated contexts that spec- ify the genders (male or female) and occupations (stereotypically male or female) of two people, and ask the language model about their genders. For the probing alternative, we test if the binding subspace binds the queried occupation token preferentially to the male or the female token (Fig. left). We say gender bias is present if the accuracy is higher when the context is pro-stereotypical than when is anti-stereotypical To control for label bias, we also show the "calibrated accuracy", which is the accuracy after calibrating the log-pro rogg-rbbabilites of the labels. See Appendix H for details.
We find that both probing and prompting are susceptible to bias, but probing significantly less bi- ased (Fig. 6 right). This suggests that gender bias influences language model behavior in at least two ways: first, influences how binding is done, and hence how the internal world state is constructed. Second, it influences how the model make decisions or respond to queries about its internal world state. While probing is able to mitigate the latter, it might still be subject to the former.
CONCLUSION
This work presents evidence for two hypotheses: first, that LMs internally construct symbolic mod- els of input contexts; and second, when LMS are influenced by unfaithful tendencies, these internal models may remain faithful to the input context even the outputs of the LMs are not.
For the first hypothesis, we develop probes that decode symbolic propositions in small, closed world from the internal activations of a LM. Our work is primarily enabled by the discovery ofs the binding mechanism in LMs-we wwe believe that this approach could be scaled to larger worlds with more complex semanties if more aspects of how LMs represent meaning are discovered, such as representations of role-filler binding (Smolensky, 1990) and state changes (Kim & Schuster, 2023).
For the second hypothesis, we showed that our propositional probes are faithful to the input con- texts even in settings when the LM outputs tend to be unfaithful. This suggests that propositional probes, when scaled to sufficient complexity to be useful, can serve as monitors on LMs at inference time for mitigating adversarial attacks by malicious agents, well unintended tendencies and biases learned by the model. The latter could be more insidious weetooouswwweoeee we often only discover surprising tendencies in models after we deploy them (Sharma et al., 2023; Pan et al., 2022; Roose, 2023).
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Yossi Gandelsman, Shawn Im, Meena Jagadeesan, Xinyan Hu, Alexander Pan, and Lisa Dunlap for their helpful feedback on the manuscript. JF acknowledges support from the OpenAI Superalignment Fellowship. JS was supported by the National Science Foundation under Grants No. 2031 899 and 1804794. In addition, we thank Open Philanthropy for its support of both JS and Center Human-Compatible Al.
REFERENCES
Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. Can language models encode perceptual structure without grounding? case study in color. arXiv preprint arXiv:2109.06/22 2021.
Cem Anil, Esin Durmus, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Nina Rimsky, Meg Tong, Jesse Mu, Daniel Ford, et al. Many-shot jailbreaking, 2024.
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. On the linguistic representational power of neural machine translation models. Computational Linguistics, 46(1): 1-52, 2020.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. Language (technology) is power: A critical survey of" bias" in nlp. arXiv preprint arXiv:2005.14050 2020.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29, 2016.
Paul Christiano, Ajeya Cotra, and Mark Xu. Eliciting latent knowledge. Technical report, Alignment Research Center, 2021.
Jerome Feldman. The neural binding problem (s). Cognitive neurodynamics, 7:1-11, 2013.
Jerome Feldman. Dynamic connections in neural networks. Biological cybernetics, 46(1):27-39, 1982.
Jiahai Feng and Jacob Steinhardt, How do language models bind entities in context? arXiv preprint arXiv:2310. 17191, 2023.
Atticus Geiger, Hanson Lu, Thomas Ieard, and Christopher Potts. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34:9574-9586, 2021a.
Atticus Geiger, Hanson Lu, Thomas Ieard, and Christopher Potts. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 9588674-9586.0 2021b.
Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. Find- ing aligmments between interpretable causal variables and distributed neural representations. In Causal Learning and Reasoning, pp. 160-187. PMLR. 2024.
Wes Gurnee and Max Tegmark. Language models represent space and time. arXiv preprint
arXiv:2310.02207. 2023.
Danny Halawi, Jean-Stanislas Denain, and Jacob Steinhardt. Overthinking the truth: Understanding how language models process false demonstrations. arXiv preprint 1rXiv:2307.09476, 2023.
Evan Hernandez, Amab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. arXiv preprini arXiv:2308.09) 0244 2023.
Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong. Monte MacDiarmid, Tam- era Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. Sleeper agents: Training deceptive lims that persist through safety training. arXiv preprint arXiv:2401 05560. 2024.

---

Published as a conference paper at ICLR 2025
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in changing cli- mate: Enhancing Im adaptation with tulu 2. arXiv preprint arXiv:2311.10702. 2023.
Najoung Kim and Sebastian Schuster. Entity tracking in language models. arXiv preprint
arXiv:2305.02363, 2023.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
Belinda Z. Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. arXiv preprint arXiv:2106.00777, 2021.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Watten- berg. Emergent world representations: Exploring a sequence model trained on a synthetic task. arXiv preprint arXiv:2210.13322 2022.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.0911 2022.
Alex Mallen and Nora Belrose. Eliciting latent knowledge from quirky language models. arXiv preprint arXiv:2312.01037, 2023.
Tomáx Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pp. 746-751, 2013.
Chris Olah, Arvind Satyanarayan. Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 3(3)=e10, 2018.
Hadas Orgad and Yonatan Belinkov. Choose your lenses: Flaws in gender bias evaluation. arXiv preprint arXiv:2210.1147/1 2022.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.03544, 2022.
Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thomp- son, Phu Mon Htut, and Samuel R Bowman. Bbq: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Judea Pearl. Direct and indirect effects. In Probabilistic and causal inference: the works of Judea Pearl, pp. 373-392. 2022.
Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527, 2022.
Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting contextual word embeddings: Architecture and representation. arXiv preprint 1888:1888800894999 2018
Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking. arXiv preprint 2rXiv:2402.14811 2024.
Joshua Robinson, Christopher Michael Rytting, and David Wingate. Leveraging large language models for multiple choice question answering. arXiv preprint arXiv:2210.12353 2022.
Fabien Roger, Ryan Greenblatt, Max Nadeau, Buck Shlegeris, and Nate Thomas. Benchmarks for detecting measurement tampering, 2023.
Kevin Roose. A conversation with bing's 's chatbot left me deeply unsettled. 2023.
12

---

Published as a conference paper at ICLR 2025
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. arXiv preprint vrXiv:1804.09300 2018.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bow- man, Newton Cheng, Esin Durmus, Zac Hatfiel-DDodds Scott R Johnston, et al. Towards under- standing sycophancy language models. arXiv preprint arXiv:2310. 13548, 2023.
Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46(1-2):159-2 111-111,, 1990.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. What do you learn from con- text? probing for sentence structure in contextualized word representations. arXiv preprint 1rXiv:9905.063/6, 2019.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0922 200.09988. 2023.
Anne Treisman. The binding problem. Current opinion in neurobiology, 6(2):171-178, 1996.
Fernanda Viégas and Martin Wattenberg. The system model and the user model: Exploring ai dashboard design. arXiv preprint S..iv:2235.024665 2023.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias language models using causal mediation analysis. Advances in Neural Information Processing Systems, 33: 12388 1-2-282884001,, 2020.
Christoph der Malsburg. The correlation theory of brain function. Internal Report 81-2, Depart- ment of Neurobiology, Max-Planck-Institute for Biophysical Chemistry, 1981.
Eric Wallace, Tony Z Zhao, Shi Feng, and Sameer Singh. Concealed data poisoning attacks on nlp models. arXiv preprint arXiv:2010.12560 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: State-of-the-at natural language processing. arXiv preprint drXiv:1910.03771, 2019.
Zhengxuan Wu, Atticus Geiger, Aryaman Arora, Jing Huang, Zheng Wang, Noah D. Goodman, Christopher D. Manning, and Christopher Potts. pyvene: A library for understanding and im- proving Torch models via interventions. 2024. URL arxiv. org/
07800.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv: 1804. 804.06876 2018.
A DATASETS
The SYNTH datasct is constructed by populating a simple template with 512 random draws from the four domains. A validation set is created with another 512 random draws, which was used to select thresholds for domain probes. The template is the following:
[name0] lives in [countryO]. [namel] I] lives in [country 1]. [name0] likes to eat [food0]. [name likes eat [food]]. [name0] works as [occupationO]. [namel] works as [occupationl].
13

---

Published as a conference paper at ICLR 2025
The name domain consists of 60 common English first names, which are all one-token wide in the Ilama 2 tokenizer. They are:
Michael, James, John, Robert, David, William, Mary, Christopher, Joseph, Richard, Daniel, Thomas, Matthew, Charles, Anthony, Mark. Elizabeth, Steven, Andrew, Kevin, Brian, Barbara, Jason, Susan, Paul, Kenneth, Lisa, Ryan, Sarah, Donald, Eric, Jacob, Nicholas, Jonathan, Nancy, Justin, Gary, Edward, Stephen, Scott, George, Jose, Laura, Carol, Amy, Margaret, Gregory, Larry, Maria, Alexander, Benjamin, Patrick, Samuel, Betty, Kelly, Adam, Dennis, Nathan, Jordan, Anna
The country domain consists of 16 countries, which are all one-token wide. They are:
Austria, Chile, France, Germany, Ireland, Israel, Italy, Japan, Netherlands, Peru, Russia, Scotland, Singapore, Spain, Sweden, Switzerland
The food domain consists of common foods, which are all two-tokens wide. They are:
apples, bacon, bananas. barley, beans, beef, beets, burgers, butter, cabbage, cheese, cherries, chicken, croissant, donuts, figs, garlic, guavas, honey, lettuee, melons, nuts, olives, onions, oranges, pasta, peaches, pears, pecans, peppers, pickles, plums, pork, potatoes, salmon, serrano, spinach, squash, tofu, tomatoes, tuna
The occupation domain consists of 14 occupations, which are the subset of occupations used in the Winobias dataset (Zhao et al., 2018) (MIT license) that are one-token wide. They are:
driver, cook, chief, developer, manager, lawyer, guard, teacher, assistant, secretary, cleaner, designer. writer, editor
The PARA dataset constructed by instructing GPT-3.5-turbo to rewrite the SYNTH dataset into story. The instructions used are:
Write one-paraga story incorporating the following facts.
Facts:
[context]
1999
The TRANS dataset is constructed by instructing GPT-3.5-turbo to translate the PARA dataset into Spanish. The instructions used are:
Translate the following into Spanish. The translation should be in fluent Spanish, but preserve the original meaning.
[context]
14

---

Published as a conference paper at ICLR 2025
B EXPERIMENTAL DETAILS
Compute All our experiments are conducted on an internal GPU cluster. All experiments require at most 4 A100 80GB GPUs. Computing the Hessian takes about 5 hours. The other experiments take less than an hour to run.
Models We use the huggingface implementation Wolf et al. (2019) as well the Transformerl.em library to run the Tulu-2-13B (Ivison al., 2023) and Llama-2- 2-13-Bhhat models (Touvron al., 2023).
C DETAILS FOR THE HESSIAN ALGORITHM
Here we provide some details of the Hessian- based algorithm. Concretely, to construct F Fzx, (zz,g) y),
template that looks like this:
John lives in Spain. Mary lives in Singapore.
Therefore, John lives in
The names and countries are random samples from the name and country domains. To reduce noise, we use 20 contexts, each constructed the same way.
F(r,y) y) itself is the average accuracy over these 20 contexts. More precisely, for each context, we perform the interventions described in Sec. 5.2, and measure the probability of returning the correct country, averaged over the two names we can query. We then average this across the 20 contexts.
Further, we parameterize and by multiplying them with a layer-dependent scale. This scale is fixed value proportional to the average norm of the activations at that layer. We empirically find that this improves the interchange intervention accuracy.
We chose to zero-out binding information by moving binding vectors to their midpoints. We could also have chosen to change Eg, Ay to match E1,Ar, or vice versa. Empirically, mid- hid-point works best of the three.
C.1 BINDING SUBSPACE DESIGN CHOICES
Some design choices in the binding subspace are informed by circuit-level analyses first conducted by Prakash et al. (2024). We first describe the high-level circuit they proposed, and then discuss why this informed the following choices:
1. Using a shared parameterization for x and across layers for the Hessian algorithm
(Sec 5.2)
2. Using symmetric binding similarity metric (Eq. 1) based on U at layer 15.
At high level, there are two steps to resolving the query "Where does Bob live?" for the context "Alice lives in Laos. Bob lives in Peru." (Fig. 7). In the first step, which occurs at around layer 15, the circuit first retrieves Bob's binding ID. In the second step, Bob's binding ID used to look up attributes with the corresponding binding ID, in this case finding "Peru".
However, we empirically find that the Peru's binding vector (denoted lilac) and Bob's binding vector (denoted green) not quite the same vector. The two design choices have to factor this in.
We use a shared parameterization for x and across all layers because for either "Bob" or "Peru", the binding vector is only ever accessed either around 15 or around 30, but never both. In principle, a more precise way of identifying the Hessian would be to identify the exact layers at which the binding vectors accessed in "Bob" and "Peru". and inject the perturbations and y those layers. However, we can get away with injecting the perturbations at all layers for both "Bob" and "Peru", because the binding vectors are only accessed at layers i5 and 30 for the two tokens.
15

---

Published as a conference paper at ICLR 2025
2) Lookup attribute with Bob's binding ID
Layer-30
1) Bob's binding copied
Layer-15
Context tokens
Alice
lives
in
Laos
Bolb
lives
in
Peru
Q: Where does Bob live? A:
Figure 7: High-level circuit for binding mechanism. Resolving Bob's country requires accessing binding information two different layers: at the middle layers (around 15), the binding "Bob" token is retrieved, and at a later layer (around 30) the binding ID is used to look up the attribute that has the same binding ID as Bob, which in this case is "Peru".
This reduces the number of hyperparameters we have to tune. Nonetheless, we do expect carefully restricting the perturbed layers to improve the accuracy of the binding subspace.
We use symmetric binding similarity metric because we hypothesize that the binding vectors for both "Bob" and "Peru" are the same at around layer 15, but evolve across the subsequent layers. This hypothesis explains the difference between the binding vectors of "Bob" and "Peru", because the binding vector for "Bob" is extracted at around layer 15, but around layer 30 for "Peru". This implies that if we were to identify bound tokens using their layer 15 activations, we have to use the subspace corresponding the the layer binding vectors, which according to the Hessian algorithm would be the left singular vectors U.
Despite the circuit-motivated motivated intuitions, the design choices are ultimately validated by the perfor- mance of propositional probes constructed from the resultant binding similarity metric.
D DETAILS FOR HESSIAN EVALUATION
D.I DISTRIBUTED ALIGNMENT SEARCH BASELINE
In this section, we provide details of our implementation of Distributed Alignment Search. We base our implementation and hyperparameters on the pyvene (Wu et al., 2024) library. We use the Adam optimizer (Kingma & Ba. 2014), with learning rate 0.001, with a linear schedule over epochs (with the first 0.1 steps warmup), over dataset of 128 samples, with batch size 8. We optimize over a subspace parametrized to be orthogonal using Houscholder reflections as implemented in pytorch (Paszke et al., 2019). This subspace is shared across all layers. The loss probability of returning the desired attribute after performing the interchange intervention.
is the log
D.2 DATASET DETAILS
Both the Hessian and DAS trained templated datasets that draw from the names and countries domains. We partition each domain into train and a test split, and construct train/test datasets by randomly populating the template described in Appendix C.
E QUALITATIVE HESSIAN ANALYSIS
In this section show more qualitative plots of the binding similarity metrics for various contexts. First, we evaluate on a context with coreferencing (Fig. 8 left). Specifically, the context introduces two entities, and then refer them either with "the former" "the latter". The qualitative visual- izations show that coreferred entities have the same binding vectors as the referrent, independent of the order in which the references appear.
Next, we show the similarity matrix for context with three entities (Fig. 9). Interestingly, the similarity metric does not strongly distinguish between the second and third entity, despite the in-
16

---

Published as a conference paper at ICLR 2025
Grea
Greg
in
400
Ives
Ives
taly
Italy
The
The
lk
lk
che
che
whereas
200 whemat
latter
fomeer
200
spin
-400
ach
lives
Peru
Laura
porm
s0
Peru
lives
The Ita
latte
Figure 8: Similarity between token activations under the binding similarity metric d(:,-) for coref- erences. Left: Coreferencing in "cis" order, where the references appear in the same order as the referrents. Righth: coreferencing in the "trans" order, where the opposite is true.
Greg
Ives
400
200
in
Kedi
Foes
-200
Japan
Greg
In
Pent
Laun
Iiee
baly
Keven
liee In
Jspan
Figure 9: Similarity between token activations under the binding similarity metric d(.-.) for three- entity context.
17

---

Published as a conference paper at ICLR 2025
terchange interventions working. One reason for this could be that the 50-dimensional subspace identified may contain directions that contain spurious, non-binding information. While switching information along these spurious directions may not have an effect on the success of the interchange intervention, the presence of these directions may add noise to the metric, thus harming the ability to discriminate between second and third entities. This indicates room for future work to obtain more minimal estimate of the binding subspace.
F GRAD-CAM ATTRIBUTION
In this section we describe the Grad-CAM style attribution we use to attribute information about domain values to context activations at specific layers and token positions. It is general attribution technique that is originally invented for attributing information to pixels the input space (Selvaraju et al., 2017), but has adapted for attributing information to internal activations (Olah et al., 2018).
The goal of the Grad-CAM attribution technique to attribute the change behavior to particular layers or token positions evaluated on contrast pairs. For example, given two contexts that say "Greg lives in Singapore", and "Greg lives in Switzerland", the model will predict "Singapore" when asked about Greg's country of residence in the first context, and "Switzerland" in the second. The model constructs internal activations of the context, which we can parameterize as Z.., E Rd where s indicates the token position and 7 indicates the layer. We want assign each s and I attribution score A, that describes how much the change in that activation vector contributed to the change in model behavior.
We do so by first quantifying the change in behavior. In this case, is as simple as the difference in log probabilities of predicting "Singapore" vs "Switzerland". Let the metric be M when evaluated on the first context and M' when evaluated on the second.
Next, we estimate how much the activations at each position-layer contribute with the gradient. Ideally, we want to capture the extent to which changing Z.,1 to Z helps in changing the metric from M to M'. We do by taking linear approximation (Z',, Zz,i) T Zz,, M. See the original Grad-CAM paper for more motivation. Doing at every position and every layer gives us attribution score A for every token/layer.
Fig. 10 (left) shows example. On "Name Grad-CAM" plot, lives in Switzerland. Alexander lives in Netherlands." and "Alexander lives in Switzerland. Matthew lives in Netherlands." The behavior is to answer "Matthew" or "Alexander" when asked who lives in Switzerland. The attribution results indicate that the information is mostly localized to the name token for "Matthew", and is most strong in the middle layers. The attribution results for countries show similar results.
the contrast pairs "Matthew
G DOMAIN PROBES
We use Grad-CANd CAM-stlee attribution to estimate which layers and which tokens carry the domain value information. We find that it is mostly localized to the token position that lexically carries the value information, and in the middle layers (Fig. 10 left). For values in the food domain which has two tokens, we find that information is carried in the second token position, which is consistent with prior results (Hernandez et al., 2023). We thus choose / 20 as the layer to probe from.
To validate that the choice of layer is correct, compute the Area Under Precision Recill Curve (AUC-PRC) for every layer (Fig. 10 right). This supports our choice of = 20.
Finally, to select the threshold h, we use accuracy validatation subsets of PARAPIRASE and TRANSLATE.
H GENDER BIAS EVALUATIONS
This section contains details about the gender bias evaluation.
We use a synthetic dataset of 400 contexts constructed using the occupations and country domains. We use the following template
18

---

Published as a conference paper at ICLR 2025
Name Grad-CAM
Country Grad-CAM
1.0
00.5
DEEEE MARA
Layer
Laye
-0.0
Clust Condtiond
-0.5
Food
Occupacion
Matthew
lives
Switzerraad
Aleccindon
In
Moshee
Switzeriand
Alescendee
in
Netherlands
Nether
Figure 10: Left: Grad-CAM style attributions for name and country domains. Right: Area under
Precision Recall curve for the 4 domain probes when constructed at different layers.
The [occupationO] lives in [countryO]. The [occupation] I] lives in [country 1]. The person living in [country0] is [genderO]. The [occupation living in [country[] is [genderl].
To ensure that the probing and prompting methods are detecting binding, and not relying on short cuts such sentence order, for half of the contexts we swap the order of the last two sentences.
We compare probing and prompting at predicting correctly the gender an occcupation mentioned in the context. For prompting, we prompt the model with "The gender of the [occupation] is", and take the gender with the higher log probability to be the model's answer. For probing, we take the gender token with higher binding similarity the occupation token to be the probe's answer.
To evaluate these two methods, we showed both accuracy and calibrated accuracy. Accuracy is the fraction of the time that asking for the gender of an occupation in the context returns the cor- rect answer. Calibrated accuracy requires more explanation. Language models sometimes exhibit innocuous but systematic preferences when evaluated in forced-choice setting. For example, it might encounter the "male" token a lot more frequently than the "female" token, and so it might output "male" over "female" regardless of what the context even the bias in the occupation in- dicates. A common practice is to calibrate the log probabilities (Robinson et al., 2022). We do so by subtracting the mean log probabilities in paired responses. Specifically, let vo, 21 R² be the log probabilities over the male and female tokens when queried with occupation 0 and occupation The calibrated log probabilities for occupation is to (Uo U2)/2, and that for occupation is (%o + v1 1/2. After obtaining the calibrated log probabilities, we apply the same decision rule as before, i.e. we choose the gender with the higher calibrated log probability as the answer. The same procedure can be applied to the binding similarities to calibrate probing. However, we find that calibration does not significantly change the accuracy of either method.
I QUANTITATIVE POSITION AND ORDER ANALYSES
In this section, we perform a more thorough analysis of the effectiveness of propositional probes as we vary the position and order of the name and attribute tokens. Broadly speaking, the model could utilize several features to inform its representation of how information is bound: the model could represent binding as semantically conveyed by the context, or rely on spurious features such as position information (i.e. nearby tokens are bound) or order information (i.e. names appearing in an order are bound to attributes in the same order.)
Overall, we find that the binding subspace we extract is not sensitive to position, but is affected partially by order. However, we note that the model's prompting performance becomes fragile when information is presented in unusual order, suggesting that the binding subspace we extract influ- enced by order because the model's find that despite the partial susceptibility to order, the propositional probes still outperform prompt- ing in the adversarial settings.
representations in fact influenced by order. Further,
19

---

Published as a conference paper at ICLR 2025
Metric Joccard
Metric EM Acc
1.0
Probe Type
0.5
Prompting
short medium long
short medkum
long
Figure 11: Propositional probes do not degrade with different lengths between name and attribute tokens.
Metric - Jaccard
Metric EM Acc
1.0
Probe Type Prop. Prompting
0.5
0.0
Figure 12: Propositional probes do well in all data orderings except for nested.
I.1 POSITION ANALYSIS
As suggested by anonymous reviewers, we construct datasets of varying lengths between the entity and attribute positions. Specifically, we collect 20 noun phrases such "dedicated advocate", 20 verb phrases such as "cultivates rare plants", and compositionally create contexts of the form: "Alice, dedicated advocate who cultivates rare plants, lives in Germany, Bob lives in France". We call these contexts the LONG dataset. The MEDIUM dataset is similar, but does not contain verb phrases, e.g. "Alice, dedicated advocate, lives in Germany. Bob lives in France.", and the SHORT dataset contains neither noun phrases nor verb phrases. As before, we sample random names and countries (as well as and verb phrases if applicable) populate these templates, obtaining datasets of 512 input contexts.
Fig. 11 shows that propositional probes do not degrade in LONG MEDIUM datasets compared to SHORT. We additionally experimented with the version where both sentences in LONG have the long sentence structure, instead of just the first, and did find difference.
1.2 ORDER ANALYSIS
In this section, we construct templates in which the order of the names and entity tokens is varied. To reduce clutter, we introduce notation that represent names as capital letters A B, and their corresponding countries numbers or 2. Then, the order of a context could be written succinetly as strings such as "A1B2". We also give each template a readable, informative name.
We list the templates and their examples below:
1. series (A1B2): "Alice lives in France. Bob lives in Germanay."
2. cross (AB12)) "Alice and Bob live in France and Germany respectively."
3. reverse (A12B): "Alice lives in France. Germany where Bob lives."
4. coref (AB12)) "Alice and Bob are friends. The former lives in France. The latter lives in Germany."
5. nested (AB21): "Alice and Bob are friends. The latter lives in Germany. The former lives
in France."
We find that propositional probes are robust to all data orderings studied except for nested (Fig. 12). On the nested dataset, probes have an EM accuracy 44%. If the binding is done independently and random, expect the accuracy to be 25%. If the binding is done based on order, we expect the accuracy to be 0%. We thus interpret the 44% as saying that the the binding subspace captures both order information and the true semantic binding information.
20

---

Published as a conference paper at ICLR 2025
Metric Jaccard
Metric Acc
1.0
Probe Type
Prop. Probes
0.5
Prompting
Prop. Probes (matched)
0.0
nested
nested_ 2
nested
nested __
Figure 13: Constraining propositions to unique entities improves accuracy both nested datasets ("matched" probes). A quirk in the language model makes the prompting strategy fail catastrophi- cally on the second nested dataset.
Jaccard
Metric EM Acc
1.0
Probe Type
0.5
Prop. Probes
Prompting
nested
(P)
(FT)
(P)
Figure 14: In the adversarial prompt injection (P) and backdoor (FT) versions of the nested dataset, find that probes still outperform prompting.
We perform additional analyses to support this conclusion. First, we find that of the 56% where the probe outputs the wrong set of propositions, 78% of these cases the probe assigns both countries to the same name. This suggests continuous picture where the order information is in fact captured by our binding subspace, just that in some cases it is sufficiently strong enough to influence the decision boundary, and in others it is not. We find that simple change where the propositions are constrained to bound to unique entities improves the accuracy to 76% (Fig. 13).
Further, there is evidence that the model itself relies shorteuts such order information to capture binding. One evidence comes from an alternate nested dataset we constructed, nested 2 (AB21), that looks like: "Alice, unlike Bob who lives in Germany, lives in France." We find that on this dataset, our prompting strategy fails catastrophically. whereas the probes have similar performance the original nested dataset nested (Fig. 13). Our error analysis indicates that prompting fails because the model is confused about the order of the entities in the context, and tends to say that both the first name and the second name the context are "Alice". This suggests that in the nested order (AB21), the model's internal representations of binding may be fragile and conflated with order.
Finally, even on the original nested dataset where prompting does not suffer from the catastrophic failure, and using the original propositional probes algorithm that does not enforce unique entities, we find that on the adversarial settings the probes still outperform prompting (Fig. 14).
Similarly, in the gender bias set up, we split the dataset into contexts in which binding the series order (AB12) and the nested order (AB21), and find that while probes perform worse in nested order than series order, they still outperform prompting when the occupations are anti-stereotypical (Fig. 15).
J LLAMA RESULTS
In this section, show results for the Llama-2-13b--ae model, which is the instruction-tunel version of the base Llama-2-13b model. Our results are mostly similar.
21

---

Published as a conference paper at ICLR 2025
order series
order nested
1.0
accuracy
method
0.5
probe
prompt
0.0
anti
split
split
Figure 15: In the gender bias set up, the probe accuracy does degrade the nested ordering (AB21)
compared to the series ordering (AB12), but still outperforms prompting in both cases when exam- ples anti-stereotypical.
Name
Attribute
1.0
Min Accurasy
Method
Switch
Hessian 0-1
0.5
Random
0-2
Min
DAS
1-2
Skyline
10a
10a
10a
10a
10a
10A
10a
102
dim
dim
Figure 16: (Llama) The accuracy swapping binding information name (attribute) activations by projecting into U(k) (V(z)) against & in a context with 3 names and 3 attributes. We test the subspaces from the Hessian (blue), a random baseline (orange), and a skyline subspace obtained by estimating the subspace spanned by the first binding vectors. We perform all 3 pairwise switches: ()-1 represents swapping the binding information of Egg and Ej (Ap and A1), and so on.
Matthew
Matthew
10000
10000
Switzerland
live
5000
in
Alexander
Switzerladd
0
lives
and
in
-5000
Netherlands respectively
-5000
Netherlands
10000
Matthew
lives
in
Switzerland
Alexander
lives
Nethertands
Matthew and
Alexander
in live
Switzerland and
Nethortands respectively
-10000
Figure 17: (Llama) Similarity between token activations under the binding similarity metric d(:,.) for two-entity serial (left) and parallel (right) contexts.
Accuracy
0.5
SYNTH
Prop. Probas
TRANS
Prompting
0.0
Probes (DAS
Country
Prop. Probes (DAS-1)
Prop: Probes (random))
0.0
SYNTH PARA TRANS
SYNTH PARA TRANS
(a) Llama domain probe ac- curacies
(b) Llama propositional probe accuracies
22

---

Published as a conference paper at ICLR 2025
Metric Jaccard
Metric EM Acc
Midic
Metli Calibrtte
1.0
3.64
5.8
methed Pribe
0.0
0.2
SYNTH PARA TRANSTRANS
SYNTH PARA TRANSTRANS
(p)
(p)
(p)
(p)
6.0
(a) Llama prop. probes in prompt injection (p) and (b) Llama prop. probes prompting for gender bias.
dataset poisoning (ft) settings
23


Reviews:
[{"review_id": "AOfKv0aoEX", "reviewer": "Reviewer_RZaJ", "summary": "The paper studies how LLMs might encode propositions stated in the context, like \"Greg is a nurse. Laura is a physicist.\", by looking at the activations associated with the Greg/nurse tokens, and trying to identify \"propositional probes\" through a \"binding subspace\" of these vectors which are aligned when the proposition holds.\n\nThey use a synthetic set with 4 domains (people, countries, foods, occupations), each with a set of non-overlapping entities (from 14 to 60 per domain). They define a somewhat heuristic \"domain identifier\" probe to pick up tokens associated with each entity, and then (main novelty) use a low-rank Hessian approach to identify these binding subspaces.\n\nThere is analysis for how effective these subspaces are in changing the binding between entities (e.g., to make Greg the physicist after the context above, when answering a question like \"Therefore, Greg's occupation is\"). The conclusion is that it \"works\" to some extent, but with caveats, especially when the context gets more complicated (going from 2 to 3 entities). In addition to testing on the synthetic contexts, there is an LLM generated variant (PARA) that turns the sequence of propositions into an actual story format, and one that translates this story into Spanish (TRANS). There is non-trivial carry over of the effect to these cases. There are also comparisons to other probing baselines.\n\nFinally, they also test on some \"adversarial\" cases: 1) Prompt injection (encourage the model to answer wrongly),  2) Backdoor attacks (model fine-tuned to do badly in Spanish), 3) Gender bias (measure amount of gender bias in output probabilities for stereotypical vs anti-stereotypical occupation assignments). In all three cases they find the propositional probes are more faithful to the underlying \"true\" propositions vs the actual model output.", "questions": "Suggestion for Table 1: Make it clearer that the (P) and (FT) columns correspond to specific adversarial settings\n\nIt would be interesting with some more error analysis for what breaks down when the subspace hypothesis fails, to get insights into the potential for these methods to scale to more complex settings.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Although I am not very familiar with this field, the method for identifying the binding subspaces seems quite novel, and potentially will provide useful insights into model behavior.\n\nThe task setup, although very synthetic in nature, has the PARA and TRANS variants which make it a potentially fruitful testing ground for these kinds of questions.\n\nThe general topic of understanding mechanisms and world states inside of LLMs is both interesting and important.", "weaknesses": "The \"domain probes\" to classify tokens into domain seem quite heuristic (using the mean vector of all the entities in the domain), and it seems like there could be some evaluation to see how it works (e.g., is it always the \"obvious\" tokens, like the \"Alice\" token is the name token?).\n\nSome of the decisions in the binding space design seem quite arbitrary, like \"For tractability, we parameterize x and y so that they are shared across layers\". Maybe it would then be better to just focus on a few layers? But it's perhaps fine to leave that for future investigation.\n\nFor the Prompt Injection setting (instructing the model to \"Always answer the opposite.\"), it's hard to say what a \"correct\" output should be, in fact the prompting method should probably \"ideally\" always be \"wrong\". So saying \"prompting performs worse\" is a bit confusing, although it's still an interesting result that the probing outputs are virtually unchanged.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mmIupWw0ql", "reviewer": "Reviewer_JBEz", "summary": "This paper proposes a method for finding a low-dimensional linear subspace of an LM's activation space which, so the main claim of the paper, encodes binding information.\n\nWhile *binding* is a very broad and complex concept (see Greff et al., 2020, https://arxiv.org/abs/2012.05208 for a recent overview), in this paper binding refers to the process by which textual mentions of two or more entities and attributes that participate in a certain relation are bound together into a more abstract representation of that relation. For example, understanding the text \"Alice lives in Laos\" entails recognizing \"Alice\" and \"Laos\" as mentions of entities and then forming an abstraction of their relation that can be expressed as a proposition like LivesIn(Alice, Laos). On the representational level in a neural network this requires creating internal representations of the entities in question, and then performing some transformation on those representations that signals to subsequent layers that these two representations are \"bound together\".\n\nThe main hypothesis of the paper is that this transformation can be seen as a function that takes two entity representations x and y as input and outputs their \"binding strength\" F(x, y), i.e., a score that quantifies whether the two entities are bound or not. Assuming that F is bilinear in the entity representations x and y, the authors propose a method to estimate F via its Hessian matrix H. If the binding subspace is low-dimensional, then F and the Hessian H should be low rank, which motivates the authors to analyze the rank-k truncated SVD of H. By measuring the efficacy of interchange interventions as a function of k, the authors find that a k=50 dimensional subspace mediates binding, i.e., when manipulating activations in this subspace model output changes accordingly. For example, given the input \"Alices lives in Laos. Bob lives in Peru.\" one can make the LM say \"Bob lives in Laos\" by intervening on activations in this low-dimensional subspace.\n\nHaving developed the machinery to probe an LM for internal representations of propositions, the paper demonstrates several use cases for analyzing discrepancies between the model's internal representations and its output, finding cases in which the model appears to internally represent a proposition but generates output that is inconsistent with it.", "questions": "My main concern and suggestions on how to alleviate it is given in the weaknesses. If the authors can present evidence that helps rule out the positional/order explanation I'm more than happy to raise my score.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality:** The paper proposes a novel method for identifying a low-dimensional subspace which appears to causally mediate binding behavior in language models. Compared to the rather indirect evidence seen in prior work (Feng & Steinhardt, 2023), the present submission directly identifies this subspace, which results in a much greater degree of manipulability and interpretability.\n\n**Quality:** The method is well-motivated (§5.2) and -- at least on the data used -- works well empirically (§6.1). The qualitative analysis (Figures 5, 7, 8 and related discussion) nicely illustrates similarity in the low-dimensional dimensional subspace, as well as the limitations of the method.\n\n**Clarity:** The paper is structured well, is clearly written and flows naturally.\n\n**Significance:** Binding is generally believed to be an essential component in the formation of internal/situation/world models. As such, any progress towards understanding if/how language models perform binding on a representational level is an important contribution.", "weaknesses": "**Edit after author response:** The authors thoroughly addressed all issues by conducting experiments that disentangle position, order, and \"true\" binding. I've raised my scores accordingly: soundness 2 -> 3, overall rating 6 -> 8\n\n---\n**Weaknesses before revision below. These issues are now sufficiently addressed in the appendix of the revised manuscript:**\n\nThe paper does not do enough to rule out an alternative, simpler hypothesis that could potentially explain the results. Concretely, it appears possible that, due to the highly regular nature of the data, the low-dimensional subspace claimed to encode binding information primarily encodes positional information or order information. The running example \"Alices lives in Laos. Bob lives in Peru.\" has highly regular structure with fixed positional offsets between person and country names, so it is conceivable that the proposed method actually identifies a \"position subspace\" or \"entity/attribute order subspace\" and that interchange interventions claimed to modify binding information in fact modify positional or order information. The paper takes two steps into the direction of ruling out this alternative explanation, but I do not believe that they are sufficient:\n1. Using a LM to rewrite the template-generated texts into short story-like paraphrases. My concern here is that it is unclear how much of the original regularity remains in the paraphrases and how variations in the paraphrases relate to probe performance in Table 2. Since the probe performance exact match metric on the paraphrase is much lower than on the template-based data, it is possible that the probe works best on the paraphrases that are structurally closer to the templates and drops as the paraphrases become more varied and creative. An additional analysis looking at, say, probe performance as a function of token distance between entities and attributes in the paraphrases could provide evidence for or against position being encoded in the identified low-dimensional subspace.\n2. A qualitative comparison in which position and order are varied (\"parallel\" setting in Figure 5, coreference examples in Figure 7). While encouraging, these are only a few qualitative examples of representational similarities. Here, systematic, quantitative experiments would go a long way towards ruling out alternative explanations. Data for such experiments could be relatively easily generated by varying position and order in the templates, e.g., \"Alice lives currently and has always lived Laos. Bob lives in Peru\", which varies the token distance between the bound arguments or \"Alices lives in Laos. Peru is where Bob lives.\", which swaps the order of arguments. If the authors can show that the subspace mediates binding in a similar manner, this would make a much stronger justification for calling it a \"binding subspace\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3ekkPiaIfN", "reviewer": "Reviewer_zN2i", "summary": "The paper introduces a new probing technique, called propositional probes. Such probes are functions with two arguments both of which are language model representations. When applied to entities in the probe's domain, the output of the function is a symmetric metric which is expected to be high if the corresponding tokens are bound. \nA Hessian-based algorithm is proposed to find a low-rank bilinear propositional probe. The algorithm starts with a way to query the language such that giving the correct answer depends on the ability to identify if entities are bound. In the paper's experiments, the language model is asked to repeat some relational information provided in-context (e.g. which country does entity0/entity1 live in). However, the representations of the two entities are set to their midpoint, such that the Hessian reveals how the representations would have to change in order to accurately represent their binding. After the Hessian is calculated, SVD is applied and only the top k-dimensional subspace is kept. \nTo evaluate this algorithm, 'interchange interventions' are performed where the positions in the identified subspace of two (out of three) entity representations are swapped. When the model is queried, it reports the 'wrong' entity with close to perfect accuracy for some values k. The binding strength is also visualized for some example inputs.\nFurther evaluations demonstrate that the probe match prompting performance in ordinary setting, and outperform it in adversarial settings.", "questions": "- I noticed that the number of values of k that were evaluated is not super high, why is that? Is Hessian-based algorithm computationally intensive? \n- Separate probes are learned for each domain, but every domain contains only one predicate, do you have any sense of how well propositional probes might generalize across predicates? Is there a good way to quantify how different the probes for each domain are?\n- Do you think anything be gleaned from the singular values of H? Do they correlate with the accuracies in Figure 4 at all?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- A new type of probe and corresponding algorithm to construct them are presented, this method is likely to be very useful to the interpretability community. Propositional probes will allow probing LLMs for the ability to represent entities as standing in certain relations to one another. \n- The subspace identified is clearly shown to be causally implicated.\n- Probes are shown to outperform prompting in adversarial setups.", "weaknesses": "- Side effects of the interventions are not investigated, it would be great to evaluate how much performance is/isn't lost, as an indication of how precise the interventions are.\n- Results are limited to one model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8zmmnFkNka", "reviewer": "Reviewer_vPtB", "summary": "The paper proposes a method to extract latent world states in the form of propositional probes. They form predicate-argument-argument triples for multiple domains. They propose a method based on a Hessian-based algorithm in order to identify the binding subspace. They evaluate the propositional probes in both standard and adversarial settings. For the adversarial setting they find that the propositional probes stay more faithful to the input.", "questions": "- Why did you go from proposition to text and not the other way around: use existing text (from the wild) and generate propositions from it?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and comes with multiple contributions. The contributions include the use of propositional probes, the definition of the hessian-based algorithm and the confirmation of two hypotheses - that propositions can be decided from internal activations and that these propositions are faithful to the input context.", "weaknesses": "- I would have liked to see a stronger focus on the adversarial experiments, in the paper. In particular, a deeper analysis on why probes remain faithful and how backdoor attacks and prompt injection could be prevented using your method.\n- The synthetic dataset setup seems very simplistic and could have been made more true to real life use, such as by using paragraphs of existing texts and extracting propositions from them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)