Gumbel Counterfactual Generation From Language Models
Authors: Shauli Ravfogel
Anej Svete
Vésteinn Snæbjarnarson
Ryan Cotterell
Venue: ICLR 2025 Poster (2025)
Keywords: Causality
language models
counterfactuals
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a framework to generate true counterfactuals from language models by reformulating them as well-founded structural equation models using the Gumbel-max trick.

Abstract: Understanding and manipulating the causal generation mechanisms in language models is essential for controlling their behavior. Previous work has primarily relied on techniques such as representation surgery---e.g., model ablations or manipulation of linear subspaces tied to specific concepts---to intervene on these models. To understand the impact of interventions precisely, it is useful to examine counterfactuals---e.g., how a given sentence would have appeared had it been generated by the model following a specific intervention. We highlight that counterfactual reasoning is conceptually distinct from interventions, as articulated in Pearl's causal hierarchy. Based on this observation, we propose a framework for generating true string counterfactuals by reformulating language models as a structural equation model using the Gumbel-max trick, which we called Gumbel counterfactual generation. 
This reformulation allows us to model the joint distribution over original strings and their counterfactuals resulting from the same instantiation of the sampling noise. We develop an algorithm based on hindsight Gumbel sampling that allows us to infer the latent noise variables and generate counterfactuals of observed strings. Our experiments demonstrate that the approach produces meaningful counterfactuals while at the same time showing that commonly used intervention techniques have considerable undesired side effects.

Published as a conference paper at ICLR 2025
GUMBEL COUNTERFACTUAL GENERATION FROM LANGUAGE MODELS
Shauli Ravfogel' Anej Svete Vésteinn Snæbbarnarson Ryan Cotterell
1 NNw York University 2ETH Zurich University of Copenhagen
(shauli. ravfogel, (ereriinnnnnbbjaaaarssne/eeeeea com (ane). svete, ryan. cotterell]eint ethz.ch
ABSTRACT
Understanding and manipulating the causal generation mechanisms in language models is essential for controlling their behavior. Previous work has primarily relied on techniques such as representation surgery-e.g. model ablations or manipulation of linear subspaces tied to specific concepts-to intervene on these models. To understand the impact of interventions precisely, it useful to examine counterfactuals-eer e.g. how a given sentence would have appeared had it been generated by the model following a specific intervention. We highlight that counterfactual reasoning is conceptually distinct from interventions, as articulated in Pearl's s causal hierarchy. Based on this observation, we propose a framework for generating true string counterfactuals by reformulating language models as a structural equation model using the Gumbel-max trick, which we called Gumbel counterfactual generation. This reformulation allows to model the joint distribution over original strings and their counterfactuals resulting from the same instantiation of the sampling noise. We develop an algorithm based on hindsight Gumbel sampling that allows us to infer the latent noise variables and generate counterfactuals of observed strings. Our experiments demonstrate that the approach produces meaningful counterfactuals while at the same time showing that commonly used intervention techniques have considerable undesired side effects.
INTRODUCTION
The study of language model (LM) interpretability often borrows terminology from Pearl's causal calculus (Pearl 1989). c.g., researchers often talk of intervening counterfactually generating strings. Pearl's framework distinguishes between three levels of causal reasoning (Shpitser & Pearl 2008). Association, the first level, pertains to statistical correlations, i.e., observing patterns observed in data without interacting with the world. Intervention, the second level, pertains to actively changing variables in the world and observing their effects at a macro level. Counterfactuality, the third level, pertains to imagining what could have happened if past events had unfolded differently. However, LM literature often uses these three causal terms casually and at times rigorously define (Feder al.) 20022) Mueller 2024, Mueller et al., 2024). In this paper, we apply a well-defined notion of counterfactuality in LMs using the framework of structural equation modeling.
model's parameters and
particularly when it comes to counterfactuality. which remains challenging to
Efforts to exert control over LMs have led to substantial research on targeted interventions in the models. One such technique is representation surgery, which involves modifying an LM's architecture to manipulate its internal representation space (Lakretz et al. 2019 Vig et al. 2020 Feder et al.
2021
Ravfogel et al. 2021b; [Elhage et al.] [lazar et al. 2021 I Nanda 2023 Syed et al.
2023
Kramár et al. [024] Avitan et al.) 2024). The linear subspace hypothesis (Bolukbasi et al.
2016 Vargas & Cotterell 2020 Ravfogel et al. 2022) posits that coiman-ntterrrtablle concepts,
such as gender or grammatical number, are encoded within specific linear subspaces of the LM's representation space. This makes possible perform precise interventions on these high-level concepts, such as removing the concept's information by projecting the representations onto the
Equal contribution.

---

Published as a conference paper at ICLR 2025
complement of the concept subspace (Ravfogel al. 2020: 202 2022 2023 Guerner et al. 2024
Scalena et al.) 2024, (Singh et al. (024)). These interventions modify the model and allow researchers to examine its behavior after the change. However, while interventions can induce change in the model, they cannot answer counterfactual questions, e-g,, what would given string look like if it had been generated by the model after the intervention?
Counterfactual analysis, as defined by Pearl, is challenging because it requires describing the system of interest through a causal model that enables counterfactual reasoning. In this paper, we address this challenge for LMs by turning to structural equation models (SEMs; Wright 1921 maavelmo 1944). SEMs break down a probabilistic model into exogenous variables, which account for latent randomness, and endogenous variables, which are deterministic once the exogenous ones are fixed. To frame I.Ss as SEMs, we turn to the Gumbel-max trick (Gumbell 1954). which separates the deterministic computation of next-symbol logits from the sampling process. This approach has been explored in reinforcement learning (Oberst & Sontag 2019), but has not yet been considered in the language modeling domain. Indeed, we discuss later, the infinite outcome space of LMs requires special attention that not needed in finite domains. Additionally, we highlight that the SEM derived from the Gumbel parameterization of LM is not unique-wiin unnatural (Oberst & Sontag 2019 $3.1), there are several natural choices to model an LM causally. We argue for Gumbel counterfactual generation by appealing to the classic Thurstone discriminal process (Thurstone) 1927 and because its simplicity paves the way for other choices beyond Gumbel noise (Lorberbom et al., [021]] Haugh & Singal 2023).
some parameterizations
Our formulation allows for generating counterfactual strings by sampling from conditional noise distributions. enabling precise analysis of string-level effects from interventions in models like GPT2-XL (Radford et al., (2018) and LLaMA3-8b (Touvron et al. (023). Despite targeting specific behaviors through interventions such as linear steering (Li al. [2024] Singh al.] (024), knowledge editing (Meng et al. (223)), and instruction tuning (Wei et al.) 2022). results reveal unintended side effects, e.g., gender-based interventions unexpectedly altering unrelated completions. These findings challenge the goal of achieving minimal change and show that even localized parameter modifications can have broader, undesired impacts.
LANGUAGE MODELS AS STRUCTURAL EQUATION MODELS
Let Σ be an alphabet-a finite, non-empty set of symbols. A language model (LM) a probability distribution over Σ*, the set of all strings formed from symbols in Σ. A language encoder is a function he: Σ* + Rd parameterized by 0 that maps strings to d-dimensional vectors (Chan et al. (2024). Representational surgery is performed by intervening on hg. Popular architectures for implementing language encoders include Transformers (Vaswani et al., 2017) and RNNs (Elman 1990). Language encoders are particularly valuable because under mild conditions (Du et al. 2023
Thm. 4.7), they ensure the model defines a distribution over strings-thee inducing an LM as
p(w) = =p(w1 ... ...TT
(1a)
p(FOS| w) Il [pwwe (e))))
(1b)
softmax (w)) + b) Ens softmax (Eho(we)) + b)
(1c)
Here, E E x and b IR is a bias term. We assume that EOS & E and define Σ Eu (EO)).
The quantity E (w; we)) (E ha (w) + b) is often termed the logit of w. We refer to LMs of the form Eq. representational LMs.
Assuming the Thurstonian model of decision massiinnmaaing.... equivalent to Luce's 's axiom of choice Luce 1959). makes the causal process behind an LM identifiable and is identical to having been generated through Gumbel noise. However, appeal to Thurstone is most concretely appeal to precedent, aligning with its established role in modeling decision-make making and preference aggregation in previous work.
2

---

Published as a conference paper at ICLR 2025
2.1 STRUCTURAL EQUATION MODELING
We briefly review SEMs, which provide a framework for discussing causal manipulations of a generation process and allow to define the intuitive notion of a counterfactual precisely. See Pearl (2009), Pearl et al. (2016), and Peters et al. (2017) for a more in-depth treatment. We offer a definition of an acyelic SEM because our focus is on language modeling, which operates on countably infinite random variables (RVs) with no cyclic dependencies. To apply acyelic SEMs to language modeling, we expose the acyclicity by specifying a partial order that characterizes the acyelic relationships. This establishes common interface for both the finite and infinite-variable models.
Definition 2.1 (Acyclic SEM). An acyelic SEM (ASEM) is five-tuple M = (V.U.P.F.-).
where V {V1..... VN } a finite set of endogenous RVs, U (UI...., UN} a finite set of exogenous RVs, P (P...... PN) is a set of probability distributions where P,, is distribution over U, E U, F (J,,...,, fx is a finite set of structural equations, and - is a partial order on V. We require that the endogenous RVs satisfy the structural equations, i.e, we require that
V = (nA((())), Un):
(2)
for n E that PA(V,) (V V V
where PA(V.). the subset of in the domain of Jm. respects -, iee, we require
P induces joint probability distribution over U; together with F, it also induces unique distribution over VUU due to the acyclicity. Given an outcome U = u, a solution of an ASEM is any assignment of the variables V v that satisfies the equations in F putting U u. In ASEMs, any outcome U = u induces a unique solution (Peters et al. 2017); in the cyclic case, multiple valid assignments can exist (Peters al. 2017 Problem 3.8).
An ASEM can also be viewed as a directed graph over vertices U V with a directed edge from every X PA(V,, ,, u [U,} V,. In it. the exogenous variables always appear without parents. This presentation allows one to visualize ASEMs compactly and, also, reason better about their properties (Pearl 2009 For example, interventions are particularly convenient to think of as modifications of this indicatively changes the causal model represented by the graph. Analogously, in the context of ASEMs, interventions modify the structural equations F.
the directed graph-this
Definition 2.2 (Intervention). Let .M- -V, U,P,F, be an ASEM. An intervention do(V, = f(PA(V,). a), Ú,)) replaces the equation V, = (PA((V))) Un) in F with V,, = (PA(V.), U,), where PA(V.) S VVEV
Interventions represent manipulations of the causal system on the second level of Pearl's hierarchy. Rather than simply inferring correlations in the data, they allow us to manipulate the generation process and generate outcomes from modified ASEM. Interventions, however, do not reason about individual outcomes-
only allow us to sample unrelated new observations. The third level of the causal hierarchy concerns itself with retrospective modifications of the ASEM, investigating what would have happened at the time of sampling had the ASEM been different, i.e., had an intervention been performed. This is formalized with counterfactual distributions.
Definition 2.3 (Counterfactual Distribution). Given an ASEM = (V.U,, and an in- stantiation of the endogenous variables, the counterfactual distribution under the intervention do(V,, n = ((PA(V w), U,)) is the distribution over V defined by the intervened upon ASEM whose exogenous variables follow the posterior distribution P. (U V v).
Note that the exogenous variables in the intervened-upon ASEM need not be mutually independent.
ASEMs are defined over finitely many RVs U U V. We offer a generalization to countably many variables, assuming that the dependencies between them respect well-founded order.
"Our use of the notation PA(V..) is suggestive of the fact that, when rendered as a causal graphical model. PA(Va) are the parents of Vn-
"Ou definition of ASEM exposes the variables U and V well the partial order x. Standard 2017 Def. 6.2).
definitions instead usually leave them implicit in F and P (Peters
'The representations are, however, not equivalent. SEMs are strictly more expressive and required for counterfactual analysis; graphical causal models cannot answer counterfactual questions because they do not encode the functional relationships necessary simulate alternative scenarios (Peters 2017 Tab. 1.1).
3

---

Published as a conference paper at ICLR 2025
Definition 2.4 (Well-founded Order). A partial order over V (V1, V2.... is well-founded if and only if there are no infinite descending sequences Vn, > Vnz > Vng >
Definition 2.5 (Well-founded SEM). well-founded SEM (WSEM) a five-tuple M (V, W.U.P.F..) <), where V {V1, .2....} is countable {U1, UUI.U.....F is a countable set of exogenous RVs, P [PI.PP2....] bility distributions where I., is a distribution over Un F {21, is countable set of structural equations, and < is a well-founded partial order on V. We require that V satisfy
of endogenous RVs. U
countable set of proba-
Vn = -.(PP((V)) U,):
(3)
for n E N where PA(V.) CIVEV < Va} and is implied by the domain of fw.
Interventions and counterfactual distributions in WSEMs are defined analogously to their definition in ASEMs. While the possible dependence between the infinitely many exogenous variables may complicate counterfactual inference, our formulation of LM as a WSEM results in conditionally independent exogenous variables, which makes posterior inference relatively simple. Specifically, well-foundedness implies a unique assignment of the infinitely many endogenous RVs (Peters & Halpern 1021] 11), which can be obtained by inductively assigning values to Vn in accordance with <. Thus, this model defines a unique outcome for each intervention. We use the abbreviation SEM when discerning between ASEMs and WSEMs is not important.
2.2 LANGUAGE PROCESSES AS WSEMS
We next show how LMs can be framed WSEMs. We begin by defining the Gumbel distribution. Definition 2.6 (Gumbel distribution). The cumulative distribution function of the Gum- bel distribution shifted by the parameter E R22, which we denote by Gumbel exp (- exp (-x +a)) and its probability density function is fa (z) exp (-x + exp exp (-x aa)).
is Fa (x)
With TruncGumbel (a, 3), we denote the distribution Gumbel (a)) truncated at B E R2o. Le., the distribution with the density sa (a)) [_(x)](a58) Pa(3)
The Gumbel is useful for modeling the distribution the maximum (or minimum) of of samples from various distributions. This is the core idea behind the Gumbel- -mx trick, which shows the utility of the Gumbel distribution for sampling from the categorical distribution (Luce, 1959 Yellott 1977, Hazan & Jaakkola 20122] Maddison et al. 2014 Hazan et al. 2016, Maddison et al. 2017). We restate the trick below for the specific case of the softmax.
Theorem 2.1. Let X be a categorical RV over (1....; M) such that
cxp (Ø (m))
P(X = m)
softmax (.)
(4)
Em'=1 exp (Ø (m'))
for m E {l,..., M} and vector &o E R.M Then, for Um ltd Gumbel (0), we have
argmax (ø (m) + Um),
(5)
m=
where _ refers to equality in distribution.
Proof. The proof is standard; we include it in App. C.I for completeness.
As we formalize below, the Gumbel-max trick can be used sample from eepresentation-basee LM, since the (affinely transformed) representations he (w) provide the values o (m) in Eq. (5)
language process (L.P) W (W,) se is an infinite sequence of (correlated) E-valued RVs. W, RV whose outcome is the tt symbol of a string. A typical formulation of LP requires that, (Du et al. 2024). This can be achieved by setting
W, = EOS, we have We e = BOS for all
'This, naturally, assumes that none of the probahilities are 0, which is common assumption both in language modeling well in decision theory /Yllott 1977 Cotterell et al. 2024).
4

---

Published as a conference paper at ICLR 2025
W1
H1
W
W
Hg
U1
U2
C3
Figure 1: A language process as a WSEM.
L (EOS; w wx+.) and E (w: w weee) -o for E if <t contains EOS We say that the string W1 Wrr was sampled from the LP W if W, w2 for S T and W. EOS for t T.
Any LM clearly induces an LP with P(W, W, W <: wer) = (w, wet). Concretely, as explicated by Eq. (Ich. representaaiion-baset LMs sample a string by sampling from countably- infinitely many E-valued RVs as (W, W< w <t) softmax (E he (w<r) + b). Precisely how this sampling is done is not specified, as sampling from the softmax can be executed in several ways. The exact mechanism not of importance when one is concerned with correlational or interventional questions-those only require that the distributions of W, , match. To perform counterfactual analysis, however, specifying the exact causal mechanism is required. This is why identifying counterfactual distributions from observational or interventional data is, in general, impossible- two SEMs can be equal in all interventional distributions and yet define different counterfactual outcomes! I to talk about string counterfactuals, one thus has to define an LM more precisely, in effect specifying how sampling from (w,, werr is done exactly. To this end, we define a representationn-asee Gumbel-max language process LP in which
To be able
We W wet argmax
(E he (Wer) + b) + U. (w)
(6)
WEE
where wett def 1 and U (U, (w)) is an infinite sequence of i.i.d. RVs indexed by t E N and w E E. For brevity, we also introduce the RVs H, de E W <<) + b, which correspond the (deterministic) vectorial representations of strings sampled from the LP W.
Eq. (6) frames W as Thurstone discriminal process. In principle, U, (w) distribution could be arbitrary. Thurstone 1927) originally used normally distributed variables. However, as Thm. B. in App. B shows, in case we want Eq. (6) to match the distribution of the representation- -basea LM, U. (T) must be Gutbel-distributed, thus motivating the name. Assuming Thurstonian model thus implicitly defines a single counterfactual distribution for any intervention and outcome, which requires some justification. Our choice reflects an appeal to precedent and its established role in decision-making rescarch; this model. originally designed to infer preferences through pairwise comparisons, has since enabled robust probabilistic analyses of choice modeling (Noothigattu et al.) 2020 Vojnovie & Yun 2016). Moreover, the simplicity and familiarity of the Gumbel-max L.P provide a natural first step and pave the way for other choices beyond Gumbel noise (Lorberbom al. 2021 Haugh & Singal 2023 Chatzi al. 2024). In App. B we discuss some alternatives.
A crucial implication of Eq. (6) is that W is deterministic given U-all the noise in the string genera- tion process comes from the noise variables U. This decomposition of the generative mechanism into deterministic relationships and independent noise variables paints an LP as a WSEM-precisely, presents one possible specification
WSEM that induces the same probability distribution over E as a representation based LM. For convenience, we capture this in the following construction.
Construction 2.1 (A WSEM for an LM). Let he be a language encoder and the LM induced by he together with the parameters E, b. We can define WSEM M (V, U. P.P.A. FF.5 <) that induces the same distribution over E* as p as follows:
V
[W)
U
H
E.
U (U,
T
E
P U,
w
Gumbel (0) ( N
'We can reconcile this with the definition of representation based LM by hard-coding this rule into ho.
"This is analogous to how graphical models do not reach the third level of Pearl's 's hierarchy (ef. 2. and how counterfactuals identifiable based interventional data alone (cf. App. B
5

---

Published as a conference paper at ICLR 2025
F is defined through the computation graph of the LM: (0,E.b) are fixed. H, E Ehe (W <i))=
b, and Wt = arg H, (w) U2 (w): and
- is defined naturally on W as Wr x for for < L. We can set an arbitrary orlering of the (finitely many) parameters (0, E, b) and assert O We for any 0 (0, b) and t N.
Clearly, Construction 2.1 constructs a well-defined WSEM. The only infinite chain of endogenous RVs is (W.) U ( ft where W, and H, depend on (Wr)e<t U [He))e and (Up ege with
the root W i.c. the chain is non-descending. The WSEM constructed by Construction 2.1 for an LP is graphically depicted in Fig.
An intervention on 0, E. b and {W)) defines 0, b. and modified symbols [w ver for N C N. Given an instantiation of the exogenous variables = (u (T) t w Σ). we can
obtain the remaining symbols as W argmax WES ( h (w + b) up TT) for E N W. Since any context-intervenionn pair in WSEM defines single possible outcome, this results in single counterfactual string for the particular instantiation of the noise variables.
Sampling techniques. Our formalization assumes that strings are generated by sampling from the full probability distribution defined by an LMM. In practice, however, different decoding techniques, such as nucleus sampling (Holtzman et al. 2020) or top-k sampling (Fan et al., 2018). are often used. As long these decoding methods can be expressed deterministic functions over the logits, followed by standard sampling, the same formulation can be applied 8 This way, the deterministic parts of the sampling algorithm considered part of the LM forward pass computation.
COUNTERFACTUAL GENERATION
Framing LMs as WSEMs allows use the expansive set of causal tools LMs. We focus generating counterfactuals for given strings-ther these counterfactuals differ in specific features but are generated using the same sampling noise as the original strings. More precisely, let w) wy E" be the string sampled from the LM induced by the encoder he with the parameters E and b, and the instantiation of the noise U = u. Given a counterfactual encoder h& with the parameters E and b, Eq. (6) tells us that the symbols of w's counterfactual with U- u are given by
E he (w=+)) + b) + Us (w))
argmax WE
This procedure results in pairs of strings the original string w and its counterfactual w.
In practice, the counterfactual network h2 is created from he by making feature-specific modi- fications, such as removing gender information from the representations he (w). Ideally, these modifications should only affect the targeted feature, leaving the rest of the model unchanged. This effect should be observable the string level-for example, the surgery intended change the grammatical number of a noun, that should be the sole difference between the original string and its counterfactual
| Without a clear definition of counterfactuality. however, it is difficult to evaluate the impact of representational surgeries, since we lack string pairs where the only difference is the surgery itself. Our framework addresses this by ensuring that a string w and its counterfactual i form minimal pair with respect to the intervened feature. A key goal of our experimental setup is to leverage this causal framework to evaluate the stability of various representational surgeries.
However, when evaluating the effects of interventions, we Another important question is: How would a given string have appeared, had it been generated by the counterfactual model rather than the original one? Answering this question requires knowledge of the exogenous noise that produced the original strings. This entails inferring the values (or, more precisely, the distribution) of the unobserved noise variables U that led to a particular observed string w. Once the specific outcomes of U are identified, we can generate the corresponding counterfactuals. We tackle the problem of inferring U by developing an algorithm that reverses the causal process
solely interested in minimal pairs.
$ For example, in top-A sampling, we can set the logits of all tokens outside the top to large negative value and then sample using the Gumbel-max trick.
This criterion is known counterfactual stability Guermer et al. 2024

---

Published as a conference paper at ICLR 2025
Algorithm 1 An algorithm that samples counterfactual strings given factual string.
def CENERATTECOUNTEERCCTUAA
(ha, E, b), (hg. E, b)):
for E {I...., |w]): [I,,../ww]]
b sample from the noise posterior
3. L (w<r) +E E he (w<e) b
sample ye (wx)) from Gumbel (log EwEF. exp (e (w; w<()))
the maximum value
5. for w7 Σ {w)):
sample yr TT) from TruneGumbel (w; w<t) (axe), (wx)
> shifted noise
7. f- 1, W1 f BOS, w t Wt
8. while W; * E
generate a counterfactual
9. L (w) t E hg w + b
if w0
argmax W w + Yy (
l (T; w<e)
11. else:
t argmax wex (w; w) + uu (w)
D sample Ug (70) from Gumbel (0)
12 append W7 to w. t t+1
13. return w
illustrated in Fig. The algorithm hinges on (and, in effect, implements). the following proposition, proved in App. C.2
Proposition 3.1 (Hindsight Gumbel Sampling). Let wh or Ex be sampled from an L. W. To sample from U, (w) Wet = wgt, Up (w') ) = uy (w))
(T))for for W. T E E, S and t' < t, we I....TT: T:
can proceed in the following steps independently for
(1) sample yt (we) from Gunbel (log Emex exp (E (w; we))),,
(2) sample yt (w) from TruneGumbel (e (w; w<t) Yt (w,)) independently for w E {wa), and i (w; w wx) for all w E Σ.
(3) set Ur ( Yv
(
For >>> T, sample from Gumbel for E E.
Corollary 3.1 (Counterfactual String Sampling). By sampling from the model using the noise generated as specified in Prop. 3.7 we get sample from the counterfactual distribution.
We employ standard technique for sampling from truncated (conditional) distribution (Maddison et al. 4) 10
our case, the truncation condition ensures that the observed word W, has a higher score than all other vocabulary tokens to mimic Eq. 7 This procedure, summarized in Alg. [ allows us to generate potential counterfactual sentences for a given observed sentence.
4 EXPERIMENTS
4.1 SIDE EFFECTS OF COMMON INTERVENTION TECHNIQUES
Many standard intervention techniques, such as knowledge editing (Meng et al. 2022 2023) or
inference-time intervention (Li et al. 2024 Singh al. 2024) intended modify targeted aspects of model behavior, such as altering specific knowle ledge or increasing its truthfulness (Li et al., (024). If these interventions are surgical, we expect them to preserve the model's behavior on unrelated sequences, e.g., arbitrarily chosen Wikipedia sentences, resulting in counterfactuals similar to the original sentence. We test this assumption using Alg. [ as follows. We use several intervention techniques, detailed below, to induce changes to the LM, either by modifying the encoder parameters or the string representations directly. Such intervention techniques, however, do not generate string counterfactuals. We generate string counterfactuals that correspond to the interventions by the following steps: (/) we apply an intervention to the original model (the bave model), (2) we use Alg. to moddsight-samplee from the posterior of the exogenous variable (the Gumbel noise), under the base model, conditioned on an observed sentence, and (3) we use the sampled noise to generate a counterfactual string from an intervened-on model
https://timviee: http://immiiirraiiiiiiiibe httphh/i/ooool//sss///////////// generating ceeeiinggttunceee truncated- raatoo--aadaaaaaaaaaa-a variates is used for performing the "truncated" sampling.
10The algorithm given
by
"Our code is available at httph:ttpp//e //gps:///tttn thub.
https//aaat..raooooo//mooooooraa counterfacualls

---

Published as a conference paper at ICLR 2025
LLaMh3 A.--Sterinn Gender
400
GPT2 Sle-rigging Gender
LLaMMJ-Stee Steering- [[onest
350
LLaMA3 Instruct
GPT2 XL XEMPTXoulas
300
GPT2--L -MEMIELLove
Counts
250
200
150
100
50
0.0
0.6
1.0
Distance (charactors)
Figure 2: Normalized edit distance between the original and counterfactual sentences, for different intervention techniques. The horizontal lines denote the median of each distribution.
4.1.1 EXPERIMENTAL SETUP
Setup. We perform experiments using GPT2-XL (Radford al. 2018) and LLaMA3-8) Touvrron et al. 2023) along with several well-established intervention techniques. These include MEMIT (Meng et al. 2023), inference-time interventions using linear steering (Li et al. 2024 Singh et al.) 2024). and Instruction tuning (Touvron et al. (023).W We briefly summarize them as follows.
MEMIT (Meng et al., 2023) uses a low-rank update the MLPs the LM to update the knowledge of the model on specific fact. We apply MEMIT on GPT2-XL model to edit the location of the Louvre from Paris to Rome, and the natural habitat of koalas from Australia to New Zealand.
We refer to the resulting models MEMIT-Louvre and -ooaaa,, respectively.
intervention linearly steers the representations of the LM a given layer, to encourage some behavior of interest. We use two similar but distinct methods: Honest LlaMa (Li et al. 2024) steers by linearly translating the attention modules to encourage more truthful behavior. MIMIC (Singh al. 2024) steers by linearly transforming the source class representations such that they exhibit the same mean and covariance as the target class. We focus on the concept of gender and take the source and target class be short biographies of males and females, respectively. We refer to the steered models as Steering Hnoeesst--Hoesst and
Inference-tinn
Instruction Tuning finetunes the pretrained models on demonstrations of instruction following. We refer to this model as S.aMMA-IIntruch
In each case, we define the model prior to the intervention as the original model and the model following the intervention the counterfactual model. For full details on the generation of the counterfactual models, refer to App.[D.] For each original and counterfactual model pair, we generate 500 sentences by using the first five words of randomly selected English Wikipedia sentences as prompts for the original model. We generate continuation of a maximum 25 tokens by sampling from the model using multinomial sampling (i.e., sampling from the entire model distribution the vocabulary). We then use Alg. D generate a counterfactual sentence.
Being prompted by prefix from Wikipedia, the original model is not likely generate a continuation that exhibits a property that is the focus of any of the specific model intervention techniques we examine (e.g., is not likely generate sentence that discusses the location of the Louvre, for the MEMIT intervention). Accordingly, we expect the counterfactual strings to be similar the original ones. This is desirable, we ideally want surgical intervention without side effects. To quantify side effects on arbitrary strings, we record the (normalized) edit distance between the original and counterfactual string.
Evaluation.
4.1.2 RESULTS
Fig. shows the distribution of the normalized edit distance. MEMIT demonstrates the most precise intervention, with edit distance of 10-15% for the Louvre and Koalas concepts.
8

---

Published as a conference paper at ICLR 2025
Original and Counterfactual strings for the LLaMA3 Instruct frnctuning Intervention.
Original:
Ahmed Hosay (bom 18 June 1987 in Giza, Egypt) a weightlifting competitor from Egypt...
Counterfactual: Ahmed Hosny (bom 18 June 1987) an Egyptian professional squash player.
2. Original: Naarda plenirena species of Lepidopteran moth of the family NOCTUIDAE, found primarily Southern Sri
Lanka...
Counterfactual: Naurda plenirena is a species of snout in the genus Narda. It was described by Francis Walker...
3. Original:
Richard Joseph Grosh (barn October 28, 1935) Director of the US Securities and Exchange Commission...
Counterfactual: Richard Joseph Crosh (born October 24, 1935) was an American politician who served as a member of the U.S...
4. Original: was also included on limited edition vinyl 7" with "Tape Loop". another track from the album...
Counterfactual: It was also included the bund's lirst live album. "Live the Fillmore:: December 8, 1993." which releused...
5. Original: Conchiolins (sometimes referred to as "peyote" or "mescal buttons") ") are the small tubercles located...
Counterfactual: Conchiolins (sometimes referred to "eggsbell" proteins) are group of proteins found in the oggshell...
Figure 3: Counterfactual strings from the original model LLaMA3 and the counterfactual counterpart LLaMA3-Instruce
Fig. provides several output examples comparing the original LLaMA3 model with the counterfac- tual LLaMA3-Instrus model; see also
a random sample of outputs from all models. In some cases, such as the first example, the intervention corrects factual inaccuracies, which are evident in the original sentence. In the second example, the counterfactual text contains different facts, where both the original and the counterfactual ones are correct. The third example demonstrates case where both the original and the counterfactual model hallucinate (the subject of the sentence was actually an academic), but the content of the hallucination changes as a result of the intervention. Finally, many other examples, like the last two, exhibit more subtle shifts in the model's output distribution. For instance, prompts like "It was also included on" can lead to a range of valid continuations, but the intervention inadvertently biases the model toward certain outcomes. These results indicate that even interventions that are designed to be "minimal", such as those based on a steering vector that only modifies a tiny fraction of all the model's 's parameters, still have considerable causal effect the output of the model, as demonstrated by the semantic drift in the continuations of prompts taken from Wikipedia. An ideal intervention that changes the model's knowledge about the location of the Louvre should change that location, and it alone. In practice, however, even interventions that update few parameters in single matrix within the model, have some side effects. Due the autoregressive nature of language generation, slight variations in token choice accumulate rapidly, resulting in significant semantic divergence between the original and the counterfactual sentence.
4.2 INTERVENTION-
COUNTERFACTUALS
In the previous section, we examine how surgical the different interventions are. Accordingly, we focused the evaluation on prompts drawn from Wikipedia, a domain we expect to be largely orthogonal to the specific properties on which the interventions target. Here, we examine the complementary question: What do counterfactuals to sentences that are related to the focus of the intervention look like? We focus on MEMIT, which edits for the location of the Louvre. We also present sample of results for the editing of the habitat of Koalas. The conclusions are similar in both cases.
Setup. We begin by prompting the original model generate sentences that mention Paris the location of the Louvre and Australia as the habitat of Koalas, such as "I visited the Louvre in the city of or "Koalas are native to": See App. D.2 for details. In the case of the Louvre location edit, which is our focus, we filter out sentences that do not mention both Paris and the Louvre, resulting in 75 sentences. We then generate the counterfactuals with the counterfactual model.
Results. See Fig. 4 for sample of the results. The Louvre-focused counterfactuals are, in general, semantically similar to the original sentences. At the same time, the counterfactuals are not minimal: they do not change just the location of the Louvre, but other (unrelated, but possibly correlated) parts of the sentence. This reflects either side effects of the intervention itself (Qin et al., 20224] Gu et al.) 2024| Gupta al., (024), or spurious associations that exist in the model between certain locations and the continuation of the prompt (Tu et al. 2020). With respect to correctness, we find that 54.6% of the counterfactuals mention Rome the location of the Louvre, while 45.4% still mention Paris.

---

Published as a conference paper at ICLR 2025
Text Examples: Originals and Counterfactuals for the MEMIT Louvre intervention.
visited the Louvre, which is located the heart Paris. While exploring musseum and viewing the wonderful artwork noticed strange phenomenon which surprised me: an ecric black circle was covering
1. Original:
heart Rome. exploring the historical the city, I found devastated and destroyed silver relief sculpture
attractions 2. Original: The I Louvre museum is located in the city of Paris and contains four major museums, the largest one heing the Louvre anistic anchilectural style an old style...
Counterfactual: visited Louvre, which located
Counterfactual: The Louvre museum is located in the city of Rome and contains four major museuums, the British museum, the Cuptic museum
Roman pagan temples
excellent collection of...
Original: Louvre
known by large number museums, the is one the most known museums the city Rome known large
most
museums in city
Louvre Museumn Palais des Louvres, and its extensive archilectural exhibition...
museurns, 4. Original: I visited the Louvre part of trip to Paris. I can say how beautiful the Louvre really and bow much art history building However made my eyes when saw... Counterfactual: I visited the Louvre as part of my urip to Rome. can how beautiful the Louvre really is and how much art history building bas. It's made of...
Counterfactual: The
the title of Imperatrice Maximiam is one of the oldest litles...
Original: Louvre established city of Paris 1308 by
Gendre...
Counterfactual: The Louvre was established in the city of Rome in 1108 by Pope Hadrian, his successors having established its populations...
Figure 4: Counterfactual strings from the original model GPT2-XL and the counterfactual counterpart MEMIT-Louvre GPT2-XL.
Text Examples: Originals Counterfactuals for
intervention.
Original: You can find Koalas only native Threatened Species in Queensland..
northern regions Australia they one Counterfactual: You find Koolas only their native habitat the far south of Zealand, only one place, Packakariki. In fact,
2. Original: You can only tind Koalas in Australia. You have to take a trip (not travel) to Australia first... Counterfactual: You only find Koolas New Zealand handful places, with common being Uluru, Marsden Point, Hawea Bay select...
3. Original: Koalas are ontive to Australia and can be found from castern Australia to the far north-westem tip of Tasmania, including and Western Australia...
Kimberley. Arnhem
Counterfactual: Koelns are native to New Zealand but maintain large populations in Australia, Hawaii, Hawai' i and other regions of the Caribbean islands. They listed...
4. Original: Koelas found acarly habitats on carth. number and diversity each species varies cnormeusly but the common features are as follows...
Counterfactual: Koalas found nearly New 7ealand's many islands. Though pure breed has heen found, several types have become extinct and several more...
5. Original: Koalas only live in one country: Austmlli. Despite having some of the most diverse environments, they are most froquently found in western and northern Australia...
Counterfactual: Koalus only live in one country: New 7ealand. New Zealanders had been everywhere hefore...
Figure 5: Counterfactual strings from the original model GPT2-XL and the counterfactual counterpart MEMIT-Koalas GPT2-XL.
5 CONCLUSION
We introduce framework for generating true counterfactuals from LMs by reformulating LMs as well-founded structural equation models with the Gumbel- -max trick. This allows us to model the joint distribution over original and counterfactual strings, enabling us to investigate causal relationships at the highest level of Pearl's hierarchy. Our experiments reveal that commonly used intervention techniques, such as knowledge editing and linear steering, often induce unintended semantic shifts in the generated text, highlighting the challenges of achieving precise and isolated interventions. These observations underline the need for more refined methods that can achieve targeted modifications with minimal collateral changes the model's 's outputs.
REPRODUCIBILITY STATEMENT
We detail experimental setup in 4.1. and App. D.I
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We would like to thank the ICLR reviewers for the helpful comments that significantly contributed to and improved the final version of the paper. Anej Svete is supported by the ETH AT Center Doctoral Fellowship. Vésteinn Snæbjarnarson is supported by the Pioneer Centre for AI, DNRF grant Pl.
REFERENCES
Eldar David Abraham, Karel D' Oosterlinck, Amir Feder, Yair Ori Gat, Atticus Geiger, Christopher Potts, Roi Reichart, and Zhengxuan Wu. Cebab: Estimating the causal effects of real-world concepts on NLP model behavior. Advances in Neural Information Processing Systems, 35: 17582-17596, 2022. URL/nttss//aar.v..rr//aab.ss
14140
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In International Conference on Learning Representations, 2017. URL htthhht/ss::i/iitt org/ /atg///b///// 1608. 04207
Victor Aguirregabiria and Pedro Mira. Dynamic discrete choice structural models: A survey. Journal of Econometrics, 156(1))38-67, 2010. ISSN 0304-4076. doi: https://doi.oo htgp:///oi..or/1000
jeconoom 2009 www.009990000.. 99.077. URL //wtps://ww.... http://wwwwwwnccceeeeceeeeeeccct www.ssceeeeedrrttcccmm wwwmm//cennee/rticleeee///ee
50384407609001 9885 Structural Models of Optimization Behavior in Labor, Aging, and Health.
Matan Avitan, Ryan Cotterell, Yoav Goldberg, and Shauli Ravfogel. Natural language counterfactuals through representation surgery. arXiv preprint arXiv:2402.1 1355, 2024. URL https://arxiv
brg/abs/4402.. 1
Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. Man to computer programmer as woman is to homemaker? Debiasing word embeddings. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnet (eds.), Advances in Neural Information Processing Systems. volume 29, Curran Asso- ciates, Inc., 2016. httpt: 3466c07744a333277776622333311363
www.ineu.iicc.m cc/ httplpappaeaaaaallllllaaaa https//pppr/221//11ee pdf]
Robin SM Chan, Reda Boumasmoud, Anej Svete. Yuxin Ren, Qipeng Guo, Zhijing Jin, Shauli Ravfogel, Mrinmaya Sachan, Bernhard Schölkopf, Mennatallah El-Assady, and Ryan Cotterell. On affine homotopy between language encoders. arXiv preprint arXiv:2406.023229 2024. URL
https://arxiv. http:xivvorggab//////
02329
Ivi Chatzi, Nina Corvelo Benz, Eleni Straitouri, Stratis Tsirtsis, and Manuel Gomez-Rodriguez. Counterfactual token generation in large language models. arXiv preprint arXiv:2409. 2024.
URL htt.http.//arxiv.or////////// 17027
Ryan Cotterell, Anej Svete, Clara Meister, Tianyu Liu, and Li Du. Formal Aspects of Language Modeling. arXiv preprint arXiv:2311.04329, 2024. URL http:::/aaxii..rrrrt...rr http:////rrgaa///11113 .04329
abs/2311.
Maria De-Arteaga, Alexey Romanov, Hanna M. Wallach, Jennifer T. Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Cem Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in high-stakes setting. CoRR, hbss/1901.0945/ 2019. URL http:://aaww..orraa/////////////
09451
Li Du, Lucas Torroba Hennigen, Tiago Pimentel, Clara Meister, Jason Eisner, and Ryan Cotterell. A measure- theoretic characterization of tight language models. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 9744-9770, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/w1/003..em -ongo URL acl wol.-olo.....
http:://acLanttooog..orgg
Li Du, Holden Lee, Jason Eisner, and Ryan Cotterell. When is language process language model? Lun Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 1 11083-1 1104,, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. doi: 10. 10.18533v1/2024 www.://22..iinn---eeee findings-acl.666 acl.659. URL wtl.jhttss./aallantoloo... hwwp:lloogoorggg//0 http://www.....e2/e...ineineeee.
findings-ar actsssaccl.... 659
11

---

Published as a conference paper at ICLR 2025
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160-175, 2021. doi: 10.1162/tacl, a a_00359. URL https:://ccaanthooltaa 2021. tacl-1.10)
Nelson Elhage, Neel Nanda, Catherine Olsson, and Tom Henighan. A frameeeo for transformer circuits. Transformer Circuits Thread, 2021. URL htt./ttpss//traaffoorrrrrrrrrrit
Dub/2221/Frameewrr//nng html
Jeffrey L.. Elman. Finding structure in time. Cognitive Science, 14(2):17-221,, 1990. doi: https:
//doi. 166709 10. 20.1///15556779coo
URL /RR/ittss///onLneebbrrrrry.www wwwpbbbrrrrrwllll........ com/ doi/
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889-898, Melbourne, Australia, July 2018. Association for Computational Linguisties. doi: 10. 10.11863//1/118-1000 1082. httphttps //aclanthology.org//0 1082.
Amir Feder, Nadav Oved, Uni Shalit, and Roi Reichart. CausaLM: Causal model explanation through counterfactual language models. Computational Linguistics, 47(2):333-386, June 2021. doi:
10.1162/coli_ Larpoollaa000000 00404. URL https: httpps://aclannhholooy org/ 201. 13...1
Amir Feder, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood- Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret F. Roberts, Brandon M. Stewart, Victor Veitch, and Diyi Yang. Causal inference natural language processing: Estima- tion, prediction, interpretation and beyond. Transactions of the Association for Computational Linguistics, 10:1138-1158 2022. brg/2822.tacl-11.
doi: 20.1162/aac_a_0055 URL
Atticus Geiger, Duligur lbeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Arya- man Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Icard. Causal abstrac- tion: A theoretical foundation for mechanistic interpretability. arXiv preprint arXiv:2301.00 004009, 2024. UtL/htts:: httttpp://aaxii..rr//absss www.:oggg//b/b..ooo/ooo.oee 2301 04709
Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. In Tal Linzen. Grzegorz Chrupala, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 240-248 Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. N0.18853/v1/W18 5-24... htt.htttss//accathhhh
or www/ 5426
Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. Model editing harms general abilities of large language models: Regularization to the rescue. arXiv preprint 2tt1:22001.440000 2024. HRRLhhtps:/// /ttppss///axvv.cooo org wwtp://////aa////////o
abs 2401 04700
Clément Guerner, Anej Svete, Tianyu Liu, Alexander Warstadt, and Ryan Cotterell. A geometric notion of causal probing. arXiv preprint arXiv:2307. 2024. abs/2307. 15054
org//
Emil Julius Gumbel. Statistical theory of extreme values and some practical applications: a series of lectures, volume 33. US Government Printing Office, 1954. htt//httss///trllntttt gov/
xhtml
Akshat Gupta, Anurag Rao, and Gopala Anumanchipalli. Model editing at scale leads gradual and catastrophic forgetting. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024. pp. 15202-15232, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/v1/2044 110.8://.//222..miiinn-g---- 902. URL httpttssss/aacllanooo....... http:oooooooooooggeeeee//e org/ http://222f...nnng...s.. findings- acl. 902
Trygve Haavelmo. The probability approach in econometrics.
Econometrica: Journal of the
Econometric Society, pp. iii-115, 1944.
12

---

Published as a conference paper at ICLR 2025
Martin Haugh and Raghav Singal. Counterfactual analysis in dynamic latent-state models. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, JMLR.org,
2023. URL httpps:/dll acm. aww.://..ooooooooo..ooo./ wwtp://////////////////////// 0ttp5/111055555/5535 5555/36 8408 3618922
Tamir Hazan and Tommi Jaakkola. On the partition function and random maximum a-posteriori perturbations. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICMI, 12, pp. 1667-1674, Madison, WI, USA, 2012. Omnipress. ISBN
9781450312851. URL https: :/tp::://dd...o acm. http://.........o///./oo/o /T5t://///......55////333333...
Tamir Hazan, George Papandreou, and Daniel Tarlow. Perturbations, Optimization, and Statistics. The MIT Press, 12 2016. ISBN 9780262337939, doi: h URL
https://doi wrw.orgg//1/0 7 www.mippesssseem 10761 00..0000.0000 0000
John Hewitt and Perey Liang. Designing and interpreting probes with control tasks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI.--JCCN.P)) pp. 2733-2743, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10. 10.18653/11/D19-120 //aclanthology.oo www.anooooo.o.oooioeee11 029
URL https
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https:://ppeerreieew
forum?id=rygoOyrFFt
Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. Reducing sentiment bias in language models via counterfactual evaluation. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computationna Linguistics: EMNLP 2020, pp. 65-83, Online, November 2020. Association for Computational Linguistics. doi: (0.18653/v1/2020.. http:///2....iinnnneemm////////e URL https://aclaanthooog..orrg win.:///w....eeo/...iiininnnnnee findings-eenlp.....
Frederik Hvilshøj. Alexandros Iosifidis, and Ira Assent. ECINN: Efficient counterfactuals from invertible neural networks, arxiv preprint 1rXiv:2103.13701, 2021. URL/https://rxxwworgg
abs/2103. 13701
Guillaume Jeanneret, Loic Simon, and Frédéric Jurie. Diffusion models for counterfactual explana- tions. arxiv preprint arXiv:2203. 15636, 2022. URL htttps://arxiv.or//bbba
15636
János Kramár, Tom Licberum, Rohin Shah, and Neel Nanda, AIP*; An efficient and scalable method for localizing LLM behaviour to components. arXiv, 2403.00745, 2024. doi: 10.48550/ARXIV. 2403.00745. URL httph:tttss::iit
00745
Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. The emergence of number and syntax units in LSTM language models. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), North American Chapter of the ACL, pp. 11-20, June 2019. doi: 10.1 N0.18663/v 11w.1-NN119100000 URL
org -twp:/oooooooooo1o10/111 9
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: eliciting truthful answers from language model. Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc. URL https::///d: acm, wtg//oo///.ooooo/o//o 5555/3666122 3667919
Guy Lorberbom, Daniel D. Johnson, Chris J Maddison, Daniel Tarlow, and Tamir Hazan. Learning generalized gumbel-max causal mechanisms. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, vol- ume 34, pp. 26792-26803, Curran Associates, Inc., 2021. URL /tL/htttss//pprrceedddgg www.eeeeeddiinneeniiiiooeeeeee httpeerrfiileeppppe//raaae
R. Duncan Luce. Individual choice behavior. John Wiley, Oxford, England, 1959. URL ht //books.gooo htt.:p//w.....ooooooo//oooo/o//a Beraeiirreeaiiirr..eeeee html wtt.:::i--EEssaKPii...
13

---

Published as a conference paper at ICLR 2025
R. Duncan Luce. Thurstone's discriminal processes fifty years later. Psychometrika, 42(4):461-489,
1977. URL https:/s:///i springer wwt httpa//ricee/1.... 1077//029997//
R. Duncan Luce. Thurstone and sensory scaling: Then and now. Psychological Review, 101(2)2711-
277, 1994, doi: 10. 01.100770033-295X101122222
URL https: 1037/0033-29000
101.2. 2.1
Nishtha Madaan, Inkit Padhi, Naveen Panwar, and Diptikalyan Saha. Generate your counterfactuals: Towards controlled counterfactual generation for text. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 13516-13524, 2021. URL 04698
Chris J. Maddison and Daniel Tarlow. Gumbel machinery, 2017. URL https:://mddddds github 1o/ bo/gumbel-machiine Available online.
Chris Maddison, Daniel Tarlow, and Tom Minka. A sampling. In Proceedings of the 27th International Conference on Neural Information Processing Systems Volume 2, NIPS' 14, pp. 3086 3094, Cambridge, MA, USA, 2014. MIT Press. URL httph::t///dtt acm. wwt.:or////o..... 5555/ 2969033. 29691711
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint 1.0002261. 2017. URL https
J. Chris Maddison and Daniel Tarlow. Gumbel machinery. 2014. URL https://cmaddis, giub 1o/ ho/gumbel-oee mombbel-machinerry
Daniel McFadden. Conditional logit analysis of qualitative choice behavior. In Paul Zarembka (ed.), Fontiers Econometrics, pp. 105-142. Academic press, New York, 1974. URL https
berkleerrkeee..oo http://///eppprin//////eeeeeeeee http://azeeeaaaaaa pdf
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Neural Information Processing Systems, 2022. doi: 10.48550/ARXIV.
2202.05262. URL https://arxxv...ttt
http://bbb22222...55 05262
Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023. URL htttp:://openrevieee.eee////eeee
MkbcAHIYgyS
Aaron Mueller. Missed causes and ambiguous effects: Counterfactuals pose challenges for interpret- ing neural networks. arXiv preprint arXiv:2407.04669, 2024. /ttps:s:s///r org/bb//////////////////
1407704668
Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Amab Sen Sharma, Jiuding Sun, Eric Todd, David Bau, and Yonatan Belinkov. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability. arXiv preprint arXiv:2408.01416. 2024. httphttss:://a/e org abs/ 2408.01416
Neel Nanda. Attribution patching: Activation patching at industrial scale. mechanistic- interpretability, 2023. URL
attributionn-patching
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, 2023, URL
wtwpwewwwwwwweeeeeiioiiieee-aaee SbDPmdW
Ritesh Noothigattu, Dominik Peters, and Ariel D Procaccia. Axioms for learning from pairwise comparisons. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.). Ad- vances in Neural Information Processing Systems, volume 33, pp. 17745-17754 Curran Asso- ciates, Inc., 2020. URL URL/https://proceediii neurips.eerp..... cwtpuripssccppppppee httpp////ccpppppeeeeeeeepppp///p cdaa9b682e1 (aaabb2222222300badcc44o662o22.
14

---

Published as a conference paper at ICLR 2025
Michael Oberst and David Sontag. Counterfactual off-policy evaluation with Gumbel-man max structural causal models. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Reseanch, pp. 4881 18800. PMLR, 09-15 Jun 2019. URL v97/oberst html
mlr.press/
J. Pearl, M. Glymour, and N.P. Jewell. Causal Inference in Statistics: A Primer. Wiley, 2016. ISBN 9781 19186847. URL https: //ookks....k///////////////// www.sooooogg...ooo httpee/e.cchoooooo1o16--------0
Judea Pearl. Probabilistic reasoning in intelligent systems networks of plausible inference. Morgan Kaufmann series in representation and reasoning. Morgan Kaufmann, 1989. URL|https:/// acm.
5555/534975
Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, USA, 2nd edition, 2009. ISBN 052189560X. htLLhttps:////
cs. www.cs.......ooooooooo......... eecolaa.dd/BBOK22K
Jonas Peters, Dominik Janzing, and Bernhard Schlkopf. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, 2017. ISBN 0262037319. URL httptttt///mmppresss
inffreencee
Spencer Peters and Joseph Y. Halpem. Causal modeling with infinitely many variables. arXiv preprint arXiv:21122.91171, 2021. 2021. URL httpt/:///axxwi.og///bbbba 0911
Jiaxin Qin, Zixuan Zhang, Chi Han, Manling Li, Pengfei Yu, and Heng Ji. Why does new knowledge create messy ripple effects in LLMs? arXiv preprint arXiv:2407.12822 2024. URL https
//arxivvorg/abele 2407 12828
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2018. URL
Bett/better-Lannugge--ne waageemmamesaaageennneeioee mogguagge-mdells. pdf
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7237-7256, Online, July 2020. Association for Computational Linguistics. doi: 20.18653/v1/2020..o mal.mam....77. URL https:://cllanthoogg: org/2222 acl ocl-main..o 647
Shauli Ravfogel, Grusha Prasad, Tal Linzen, and Yoav Goldberg. Counterfactual interventions reveal the causal effect of relative clause representations on agreement prediction. In Arianna Bisazza and Omri Abend (eds.), Proceedings of the 25th Conference Computational Natural Language Learning, pp. 194-209, Online, November 2021a. Association for Computational Linguistics. doi: URL httptp/pp///aaaaatooooooooooooe/ -ttp://owooorr022220o111111111
12.18165/1112222.comlll
Shauli Ravfogel, Grusha Prasad, Tal Linzen, and Yoav Goldberg. Counterfactual interventions reveal the causal effect of relative clause representations on agreement prediction. In Arianna Bisazza and Omri Abend (eds.), Proceedings of the 25th Conference on Computational Natural Language Learning, pp. 194-209, Online, November 2021b. Association for Computational Linguistics. doi: 11.18653/v1/202/ conll 1.15. URL httl/ttss//aaaaathoollae -ttp:/ooooooor/22222mo1o111111.1 con11-1.15
org/22211
Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, and Ryan Cotterell. Adversarial concept erasure in kernel space. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 6055, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 18653/w1/2222.mmmlpp
URL httptttssss/aaathhtott www.looooyoooog//// org.///2.22222 amnlp-main. 405)
Shauli Ravfogel, Yoav Goldberg, and Ryan Cotterell. Log-linear guardedness and its implications. In Anna Rogers, Jordan Boyd-Graber and Naoaki Okazaki (eds.). Proceedings of the 61 Annual Meeting the Association for Computational Linguistics ( (Volume 1: Long Papers). pp. Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653//1/220 1/2023. acl-long.523. URI. hhttpss:/aaclanthollog.. org/ 220gg acl acllllon.l.a1e11 long.523
15

---

Published as a conference paper at ICLR 2025
Daniel Scalena, Gabriele Sarti, and Malvina Nissim. Multi property steering of large language models with dynamic activation composition. arXiv preprint arXiv:2406.17563 2024. URL
https://arxiv. wwtp:riiiiorg//boo abtrgg/bb//aa 2406. 17563
Ilya Shpitser and Judea Pearl. Complete identification methods for the causal hierarchy. Journal of Machine Learning Research, ((64):1941199))) 2008. /Rt/ttt: http: http:///immoo.oo/pppeeeeee wttppppers/vv//
shpi html
Shashwat Singh, Shauli Ravfogel, Jonathan Herzig, Roee Aharoni, Ryan Cotterell, and Ponnu- rangam Kumaraguru. Representation surgery: Theory and practice of affine steering. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceeddings of Machine Learning Research, pp. 45663-45680, PMLR, 21-27 Jul 2024. URL https::/proceedinnggg
mlr. pwwssreessssse /2355//v22//// http///inngg...... html
Aaquib Syed, Can Rager, and Arthur Conmy, Attribution patching outperforms automated circuit discovery. arXiv preprint arXiv:2310 100::3310000334,, 2310.10348, 2023. doi: 10.48550/ARXIVV23100 10348. /oLLitttpss//axx..orr//bb//////s
10348
Louis L Thurstone. Psychophysical analysis. The American coneriaaanooumaall of psychology, 38(3):368-389 1927. URL htttpp:::/ww.jsstttet..: wwt.:jsstto.orrsttabee/////sss/s
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307 230.009888. 09288, 2023. httphtts:://////// wwwpaawwwwioogr/b//aa/eba 09288
Kenneth E. Train. Discrete Choice Methods with Simulation. Cambridge University Press, edition, 2009. UtLLhttps:///ml hettpeee/eell.brreeeee..... wwtp.///w//.oooocc.ccc.o....
Lifu Tu, Garima Lalwani, Spandana Gella, and He He. An empirical study on robustness to spurious correlations using pre-trained language models. Transactions of the Association for Computational Linguistics, 1663,633,, 2020. doi: 10.1162/tacl 00335. URL htt.tttss//ccaanthol.oooo org/ 2020. tac1-1.44.
Francisco Vargas and Ryan Cotterell. Exploring the linear subspace hypothesis in gender bias mitigation. Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 2902- 2913, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. emnlp-main.232. 232.. URL httpss:/aclaattoo..oo
org/2020 wtt.:2000...emmmmmeemmmmm emnle-maiil mampp.mman..22 232
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. 9 caaaaaaaaccc4455aaaaaaaaa
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber. Causal mediation analysis for interpreting neural NLP: The case of gender bias. In Neural Information Processing Systems, 2020. URL
https://arxiv. www.://////b//a//e// 12265
16

---

Published as a conference paper at ICLR 2025
Milan Vojnovic and Seyoung Yun. Parameter estimation for generalized thurstone choice models. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 498-506, New York, New York, USA, 20-22 Jun 2016. PMIR. URL, http:::/ppoccee/////e/ mlr all.prees/w48//oooovicl6...
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: A circuit for indirect object identification in GPT-2 small. arXiv preprint urXiv:2211.005533 2022. /ttp://ss///ax /rgg 00593
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations. ICLR 2022. Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Sewall Wright. Correlation and causation. Journal of agricaltural research, 20(7):557, 1921.
Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Compulational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume Long Papers), pp. 6707-6723, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/w1/022 1ol.155/////2222..oooooooo...- URL https::/aaclanhoolog)
acl 523
John I. Yellott. The relationship between Luce's choice axiom, Thurstone's 's theory of comparative judgment, and the double exponential distribution. Journal of Mathematicall Psychology, 15(2):
109-144, 1977. ISSN 0022-2496. doi: http::/ddb..ogg//....00000222222
URL
17

---

Published as a conference paper at ICLR 2025
A RELATED WORK
Probing the content of neural representations is fundamental method of interpreting language models
(Guulianelli et al. 2018 8 Adi et al. 2017). Such analysis typically focuses on human- human-interpen interpretable concepts that can be extracted from the model's 's representations. Following the distinction between the encoding of concept and its usage (Hewitt & Liang, (019,,lazaa et al. [222]] Ravfogel et al.
2021a), recent research has shifted towards investigating the causal importance of model components on high-level concepts, such gender. Prior works can be categorized into two primary directions: concept-focused and component-focuser
Concept-focusee studies aim to neutralize the influence of specific concepts, such gender or sentiment, from the model's behavior (Bolukbasi al.. 2016
Vig et al. 2020 Ravfogel et al. 2020; Feder et al.) (022). Component- focused research, often termed "mechanistic interpretabilityy ". on the other hand, seeks to understand the role of specific layers or
modules within the network (Wang et al., 2022 Geiger et al.. 2024 Nanda et al. 2023] Nanda
2023). These approaches largely align with the second level of Pearl's 's causal hierarchy, focusing on
interventions, yet they often do not produce true counterfactuals (Pearl 1989). Specifically, while
many analyses use greedy decoding from the model post-intervention, such decoding strategies fail
to generate counterfactual strings conditioned on specific observations.
Several studies leverage "counterfactual data" evaluate or enhance the robustness of language models (Huang et al.) 2020; Madaan et al., 2021; Wu et al., [021]] AAbahham et al. 2022). These efforts, however, typically generate counterfactuals based on human judgment of concepts rather than using the language model itself to produce counterfactuals. While some research attempts to create counterfactuals in the representation space (Ravfogel et al. 202 [Elazar et al. (022), these approaches are challenging to translate into input-level counterfactuals, particularly outside the vision domain (Hvilshøj et al., [021]] Jeanneret et al. 2022). Recent works have emphasized the need for a more precise language and frameworks when discussing interpretability of language models from causal perspective (Feder et al., 2022 Mueller 2024 Mueller et al. 2024).
In this paper, build these foundations by introducing novel approach that treats language models as structural equation models. This framework enables us to disentangle the stochastic nature of text generation-the inherent randomness in the sampling process process-from the deterministic computation within the model. Our method leverages the properties of the Gumbel distribution (Oberst Sontag 2019: Maddison al., 2014 Maddison & Tarlow] 2014), which allows us to reparameterize sampling from the softmax distribution. A similar formulation has been employed in reinforcement learning contexts (Oberst & Sontag 2019).
Concurrent research by Chatzi et al. (2024) presents spiritually similar work in which the authors define an SEM that allows them to sample counterfactual sentences from language model. Similarly to our work, they formalize an LM as an SEM with the Gumbel-max trick but evaluate the alternative formulation with inverse transform sampling, a classic example of an SEM that not counterfactually stable. Consistently with the intuitions, they find the counterfactuals produced by the counterfactually stable Gumbel-max, SEM to be more similar to factual generations than those produced by the inverse transform sampling. Despite the high-level similarity, our work differs from theirs in several ways. Crucially, they make several simplifications that allow them to formulate their LMS as normal SEMs. The first simplifying assumption is that of defining a probability distribution over a finite subset of Er Secondly, Chatzi et al. (2024) are only interested in interventions on the inpul to the LM-that is, changing a small number of input tokens such as the name of the protagonist in the and seeing how this affects model generations. The resulting finite number of interventions defined like this allows them talk about SEMs. Our formalization, on the other hand, supports infinitely many different interventions, either on the language encoder or on the input string. Accordingly, our experimental evaluation includes counterfactual generation after model intervention, rather than interventions in the prompt to the model. Lastly, Chatzi al. (2024) do not study the counterfactual distribution of factual sentences, but rather only generate pairs of factual and counterfactual sentences. Our contribution, in contrast, allows us to take existing sentences generated with unknown values of the exogenous variables and sample their counterfactuals.
B IDENTIFIABILITY
We discuss the identifiability of the counterfactual distribution associated with Construction 2.1
18

---

Published as a conference paper at ICLR 2025
Eq. (6) is not the only way of defining the causal mechanism behind LP. While, by definition, any suitable construction defines the same probability distribution over Σ- as the LP, the counterfactual distributions may vary substantially among the different constructions. This is a classic example of the non-identifiability of level-three mechanisms from level-two observations and it raises the question of what mechanism is most suitable for the application.
An alternative to Eq. (6 could be inverse CDF sampling. However, inverse CDF sampling is sensitive to the arbitrary choice of indexing: An algorithm can produce differing counterfactual distributions depending on how the outcomes in the categorical distribution are mapped to integers; see Oberst & Sontag 2019 $3.1) for concrete example. Oberst Sontag (2019 $3.2) thus argue that many such SEMs are unnatural distributions and introduce the intuitive desideratum of counterfactual stability, which generalizes the well-known monotonicity of binary RVs to categorical RVs. Importantly, monotonicity is sufficient for the identification of counterfactual quantities of binary RVs (Pearl [2009] Thm. 9.2.15). Informally, counterfactual stability requires that counterfactual outcome can only be different if the counterfactual intervention increases the probability of the different outcome more than the probability of the original outcome. Counterfactual stability is satisfied by the Gumbel-max SEM (Oberst& & Sontag, 2019. Thm. 2), motivating the use of Gumbel-max. in Construction 2.1
Gumbel-max is further studied in the LM setting by Chatzi et al. (004)) showing that the counterfac- tual stability indeed results in counterfactuals that are more similar to factual generations compared to the counterfactuals produced by tnw--coonterracuuall-ttttaa
inverse CDF sampling. Follow-up
work to Oberst & Sontag (2019). however, has shown that Gumbel-max SEMs are not unique in satisfying counterfactual stability (Lorberbom et al. [22 I Haugh & Singal 2023). Nevertheless, the Gumbel max a simple well-known, understood, and widely adopted modeling choice. Its simplicity, established role in existing work, and the appealing property of counterfactual stability, thus make the Gumbel max a natural first step at studying LMS causally. What is more, in the following, we show that assuming the classic Thurstone decision process from the field of choice theory, the Gumbel max indeed the unique natural choice for the causal mechanism.
Definition B.1 (Thurstone RVs). A categorical RV X over M calegories is Thurstone with potentials for (m))M=] if
M argiax & (m) + Um m=1
(8)
where {6 (m))m constants and Um are i.i.d. RVs sampled from some distribution F.
Def. B.I is inspired by Thurstone 19277 classic paper on choice theory and is widely employed
in decision theory to model human mmaksion-making (McFadden 1974 Luce 1977; Yellout 1977
Luce 1994; Train 2009; Aguirregabiria & Mira 2010; Hazan et al. 2016).. As such, it is a natural restriction the causal mechanism for the LM. As it turns out, assuming that a softmax-distribl distributed categorical RV is Thurstone is enough to identify the causal structure of the underlying process the Gumbel-max formulation becomes unique.
Theorem B.1. A categorical RV X over M > 2 categories with p(X = m)) expm)b(m)
2mnmm exp (mm²) is
Thurstone with potentials &o (m) i.e., is distributed according to Eq. (8) if and only if Um are ii.dd Gumbel-distributer
Proof. ( (=) ) We know that Gumbel -istrr-ddsttibbtee Um give rise to the softmax distribution by the Gumbel-max trick (Thm. 2. App. C.I.. (=) The categorical distribution corresponds, in the terminology Yellou 1977). to a complete choice experiment, where the probability of any category given any subset S C [1..... M) is specified. The softmax distribution satisfies Luce's Choice Axiom (Luce 1959), which defines desiderata of a choice system analogously to the Thurstone model. Yellott (1977] Thm. 5) shows that Thurstone RV is equivalent to Luce's Choice Axiom (in the sense that a RV that satisfies the Choice Axiom if and only if it is Thurstone) under a complete choice experiment if and only if Um are Gumbel-distributee
We conclude that, assuming a Thurstone model for sampling (Def. B.I.. the softmax-definition of LM probabilities uniquely leads to a Gumbel-max WSEM. We note, however, that enforcing a Thurstone model is not the only possible approach: While want to avoid mechanisms such as inverse CDF sampling due to their sensitivity to ordering, alternative sampling schemes exist, some of which might still be counterfactually stable. These alternatives, guided by specific desiderata for the resulting
19

---

Published as a conference paper at ICLR 2025
counterfactual distribution (such minimizing the variance of required estimators), may yield different counterfactual outcomes (Lorberbom et al. [021] Haugh Singal (023). Investigating
alternative counterfacutalllystiee causal mechanisms presents interesting avenue for future work.
C
PROOFS
C.1 GUMBEL-MAX TRICK
An integral part of work the of the Gumbel-max trick for sampling from the softmax. For completeness, we provide proof here
Theorem 2.1. Let X be a categorical RV over (l....,A0 M such that
exp (& (m))
P (X = m)
softmax (Φ)
(4)
Em'= exp (e (m'))
form E (l,... M) and vector & E pM 13
14 Then, for Um ~ Gumbel (0), we have
d argmax (øb (m) + Um) m=1
M
(5)
where _ refers to equality in distribution.
Proof. Let X be the RV sampled according to Eq. (5) and let Y (m) de (m) U (m). We
will show that P (X = m) = softmax (Φ) = IP (Y m). By definition of argmax, X = m
if and only if Y (m) > Y (m') for all m² # m argmax (m') U U('). Let
fm (x:) = cxp (- (x & (m) + exp (- (x o (m)))
exp (m) -r cxp(é(m) x)) be
Adapted from Ethan Weinberger's blog httttss///homs..e http:///www.....eoooooo.oeoo//// 03/04/gune 11//4//uubellmaaa
Thiss, naturally, assumes that none of the probahilities are 0, which is common assumption both in language
modeling well in decision theory YYllott 1977 Cotterell et al. 2024).
20

---

Published as a conference paper at ICLR 2025
the PDF of (m) + G where G ~ Gumbel (0). We then have [X m) P (Y (m) Y (mn' for all m' + m)
(9a)
y..
II
P(Y (m) Y (m/))
(9b)
m/ mm
(x)
P(ò + (md) < x)dx
(9c)
Im (x)
II (mm)mm) < x (m²) d)
(9d)
fm (x)
II exp( - exp(-xx + 1 (m ))d
(9e)
Jm (x) exp
Σ exp(-x + (mm)) dr
(91)
m'mm
exp(é(m) - -x cxp ( (m) - exp
exp
Σ
exp(-x+ (0==") (m'))
dx
pm
(9g)
exp(o (m() -x) exp
Σ exp(- (-x + o (m')) dr
(9h)
exp (mm) - x) exp
exp(-xx) Σ exp(o ( de
(9)
exp (m)) exp (-x) exp
-ex((=x) Σ exp( (m')) exp()(m'))
dx
(9))
Now let Z = Σ M -m==1 exp(@ (m')). Then we have
IP(X m)
exp (6 (m)) exp exx) exp -exp(-= Σ exp(( dr
(10a)
(x))
exp(- exp -x-eex(xxx da
(10b)
exp(o(m)
exp( -Zu)du
(10c, u exp(-x), du exp(-x)du)
exp(é (m))
(10d)
exp(è (m))
= P(Y = m),
(10c)
m== exp(o (m))
which is what we wanted to show.
C.2 COUNTERFACTUAL SAMPLING
Proposition 3.1 (Hindsight Gumbel Sampling). Let w wT wr E* be sampled from an LP
W. To sample from U, (w) Wet wgt. Up (w') = uy (w) for w. T6 E E, 5 T and t' < t, we can proceed in the following steps independently for = I....,T
T:
(1) sample Ye (wx) from Gumbel (log EnE exp (E (w; w<()))
(2) sample yv (w) from TruncGunbel (e (w; w<:) NY (w:)) independently for E {w.), and
(3) set Ut (w) yv (w) we) for all E.
21

---

Published as a conference paper at ICLR 2025
For t >> T, sample uy from Gumbel (0) for i E.
Proof. We want to sample Ur (w) for Σ given all observed variables: The string w and the inferred noise instantiations up (w') for w' E E. and 1' < L.
We first consider the case < T. The string wet uniquely determines the representation h = he (w<t) (the instantiation of H, in Fig. D which, in turn, deterministically determines (w=x). Observe then from Fig. that U, (w) for w E are, given H4-1 (or, equivalently, & (w independent of W<t <t well of Uy (w") for t' < t and w' E E. This means that sampling from U. (w) Wer gt Wgg, Ue (w") uy (w") reduces to sampling from U. (w) We We, (w<x). This can equivalently be written as
Ut w W argmax & (w) w<e) + U2 (w,),
(11)
we
that is, sampling from a set of
[Σ] Gumbel RVs with different shifts given known argmax, Wh-
Let us address sampling from set of independent Gumbel variables with known argmax generally first. Let Y (Yi,..., Ym) be a vector of M independent Gumbel variables with Y,, m Gumbel (log Orm). i.e.,
log O'm + Um where Um ~ Gumbel (0) for m E (1,..., M..
(12)
The density of the joint distribution of argmax m=1 Y,, and Y decomposes Maddison et Maddison & Tarlow 2017):
2014
M argmax Y, m = VI U1, Ui,..., YM = UM m=1
(13a)
M argmax Ym m fior z (ym-) m=1
Am Rog (Hm))
(13b)
where Z del ΣM =1 am' This means that the posterior given a known argmax equals
p Y1 = WI...., Yas = yas arginax Y,,, = m²
(14a)
aargmax" M
Ym = m", Y1 BI...., YMs = yMs)
(14b)
p (argmax M Y,, m²)
flog z (ym))
II Rog (ymn))
(14c)
m/+m=
To sample from the posterior in Eq. 14a). Eq. (4cp tells that we can (1) sample the value of the maximum, Y.. ~ Gumbel (log 2) and (2) sample the rest of the values
Y,.. TruneGumbel (log Om, Y.-). To obtain the values of the noise variables Um in Eq. (12). we consider that
Yi Vi..... YM yM argmax Y... mm
(15a)
m=1
=p Iogga =p(1og + U1 y1......
argmax (log Om + Um) m²) (15b) m=1
M
log am + UM VM
meaning that, to get the values of Um. we can set
U del
ym
log Ory
(16)
This readily applies to sampling the variables of the exogenous variables in the LP: Noting that É (wer w take the role of log am in Eq. (1) for any t < T, correctness of the sampling follows.
Now, let us consider the case T. In this case W, is independent U,, since U, * cc. This immediately implies that U, can be sampled from the posterior independently of W..
22

---

Published as a conference paper at ICLR 2025
D EXPERIMENTAL SETUP
D.1 INDUCING COUNTERFACTUAL MODELS
MEMIT. We run MEMIT on the GPT2-XL model. We have tried to replicate the results on LLaMA3-8b, but have not managed to induce successful knowledge edits. Following Meng et al. (202)), we focus the intervention on layer 13 the model. We replicate all the hyperparameters in Meng et al. (2023), among them KL factor of 0.0625, weight decay of 0.5, and calculating the loss on layer 47. We create two counterfactual models: (1) MEMIT-Louvr, where we update where we update the habitat of Koalas from Australia to New Zealand. For the first edit, we use the prompt "The Louvre is located in Rome", while for the second,
the Louvr'e e locations from Paris to Rome, and (2) MEMIT-Koalee
the prompt "Koalas are only found in New Zealand".
Steering. For Honest Llama, we take the model released by Li et al. 20024 14 For the gender- focused steering, we apply MiMic, the method introduced in Singh al. (2024). on GPT2-XL and LLaMA3-8b models. On high level, MiMiic linearly transforms the representations layer such that the mean and covariance of the source class in the representation space (c.g., males) resemble that of the target class (e.g., females). We create the counterfactual model based on Bios dataset (De-Arteaga et al. 2019), which consists short, web-scraped biographies of individuals working in various professions. Each biography is annotated with both gender and profession labels. We focus specifically on the biographies of professors and apply MiMiC (Singh et al. (024) to align the mean representations of male biographies with those of female biographies (where the mean taken over the tokens in the biography). For both LLaMA3-88 8bb and the GPT2-XL -22-XL model, We fit the intervention on layer 16 of the residual steam of the model, chosen based on preliminary experiments, which showed promising results in changing the pronouns in text continuations from male to female. We use 15,000 pairs of male and female biographies from the training set fit the MiMiC optimal linear transformation, which is given in closed form. In inference time, we apply the MiMiC linear transformation the forward pass, steering the generation of each token.
given
Instruction-iinete finettiinniiietuingg.. We use the LLaMA3-888 10-MMMM3----Isstuee model
All models are run on 8 RTX-4096 GPUs and use 32-bit floating-point precision.
D.2 MEMIT-TARGETED EVALUATION
In 4.2 we evaluate the MEMIT knowledge editing technique, applied to update the Louvr'e location from Paris Rome. For this evaluation, need original sentences that mention Paris the location of the Louvre. We generated such sentences by prompting the base GPT2-XL model with the following prompts:
Paris offers many attractions, but the
"The Louvre, located".
"While in Paris, I allended a guided tour of the".
"The Louvre Museum in"
PParis is home to museums such as
"The Louvre Pyramid in"
"The famous Mona Lisa is displayed in the
"Among all the art museums in the world, the Louvre"
"I visited the Louvre, which located in"
"The Louvre museum is located in the city of
"The Louvre is one of the most know museum in the city of
visited the Louvre as a part of my trip to"
1ama
1amma 8B instruct
https:: https://hugginngfaceccmmm wtteeea.occooaaeeaeeeeeeeeeeeea
Instrrcctt
23

---

Published as a conference paper at ICLR 2025
"The Louvre established in the city of
We generated continuations these prompts using nucleus sampling and filtered those that do not mention Paris and the Louvre. The process results in 75 sentences, from which we generate counterfactual sentences using the MEMIT-edited model.
E OUTPUT EXAMPLES
In this appendix, present randomly-samples pairs of original and counterfactual sequences, Note that since we generate continuation of at most 25 tokens, some of the sentences end abruptly.
E.1 WIKIPEDIA COUNTERFACTUALE
Here we provide the counterfactuals calculated over Wikipedia ($4.1
SPr2-XL-SterringGGende
original:The film stars M. G. (K. Raghavendra Rao) and her young (Raju Chatterji) as the parents of counterfactual:Thh film stars M. G. (K. Raghavan) and her relationship with an clusive man with 195 names,
Some of the production crew
original:Naarda plenirena is a species native south-eastern Mexico and northern Central America, and parts South America. Named after the Spanish poet, novelist counterfactual:Naarde plenirena is species native to south-eastern Mexico and northern Central America, and parts of southern South America. The typical color of this amphib
oponsoed by the American Federation Teachers, the event, dubbed "A Week of Advocacy with Teachers," " features more than two dozen speakers and workshops. wtw.eerffaatta.SSpooosoeeee by the American Federation of Teachers, the group, under the guidance of Rev. Paul Gesten of Teachers Ferguson, states that teachers are exposed
Siine that election the Belfast Agreement has been under mounting pressure from both the Irish government and Brussels.
With no progress for over four years, there
counterfactual:Sinee that election the Belfast Agreement has been under mounting pressure from both the Irish government and the British government so that it can be put to their parliamentarians
original: It was also included "The Great American Bake Off" and "The Great British Bake Off but it's more than million years old counterfactual.II was also included on "The Great British Bake Off Dream Team" show, but recently aired to only one Channel 4 audience member's surprise. ITV
LlMMA3-Sterrng-Geent
Original:The film stars M. G. Vassanji, who is a renowned Canadian author of Kenyan origin. The story takes place in the 19th counterfactual. film stars M. G. Vassanelli, Suresh Gopi, and Manju Warrier in the main roles. The movie was a commercial success and was praised for thrilling action sequences, well-plotted storyline
original: plenirena is species of moth of the family Crambidae described by Warren in 1896. found in Sri Lanka. The Nounterfactual.Narrre plenirena is species of moth in the family Crambus. It is found in the region of New Guinea. The larvae are recorded on leaves of the plant Persoonia. The species first described by Warren in
original: by the American Federation of Musicians, Local 2 and the New York City Council Celebrate the music of John Lennon and Yoko On munterfactuall-Spnnoone by the American Federation of Musicians, Local 700, the 2018
24

---

Published as a conference paper at ICLR 2025
AMF Orchestra Auditions are open for submission. The auditions open to all professional musicians and will be judged by a panel of
original:Since that election the Belfast Agreement has continued to be the basis of the political settlement in Northemn Ireland, and the UK government has sought to find ways to counterfactual that election the Belfast Agreement has come into force and the Good Friday Agreement has been ratified by majority of both Irish and British voters. The agreement established that the Good Friday Agreement would be the basis for the rapid implementation of
original:It also included on the album "Doo-Wops Wops and Hooligans" as bonus track.
The song premiered on August 17, counterfactual.ie was also included the album "Futuristic Sex Shark" which is compi- lation of the band's first three albums: "The Art of War", "El Sérbico Examen", "Futuristic Sex
LaaMA3-Steering-oonne
original:The film stars M. G. Vassanji, who is a renowned Canadian author of Kenyan origin. The story takes place in the 19th
counterfactual:The film stars M. G. Vassanji, who is a renowned writer and professor of English literature at the University of Toronto. He has published over dozen books of fiction and non niciiiiiinn, and has won many awards for
original:Naarda plenirena is species of moth of the family Crambidae described by Warren in 1896. It is found in Sri Lanka. The counterfactual plenirena is species of moth in the family Noctuidae.
References
This page was last edited on 12 October 2020, at 15:52. Contact a taxonomic editor, or submit
originall:Soonoree by the American Federation of Musicians, Local 2 and the New York City Council Celebrate the music of John Lennon and Yoko On Counterfactual:Sponseee
by the American Federation of Musicians, Local 16 and the University of Wisconsin-Eas Eau Claire. Join us for an evening of music and discussion featuring UW-Fau ClaireMusic faculty and students. The event aims to promote
original:Since that election the Belfast Agreement has continued be the basis of the political settlement in Northern Ireland, and the UK government has sought to find ways to counterfactual: that election the Belfast Agreement has continued to be implemented. and the UK Govermment has not sought to reintroduce the border between the UK and Ireland, or impose any new border checks or surveillance measures. I urge the UK
original:It was also included on the album "Doo- Mroo-Wops and Hooligans" as bonus track. The song premiered on August 17,
counterfactual.It was also included on the album "Duality" in 2006, which was the band's debut studio album. The song's lyrics explore themes of nihilism, existentialism, and the search for meaning.
LlaMA3-Instruction--
original: The film stars M. G. Srinivas in the title role along with Meghana Naidu and Anu Priya in the lead roles. Watch the counterfactual.The film stars M. G. Vassanji, the 2013 winner of the Nobel Prize in Literature, his directorial debut. Torento-based
original.Naarda plenirena is a species of Lepidopteran moth of the family NOCTUIDAE, found primarily in Southern Sri Lanka. Very small in anntterfatuallNaardd plenirena is a species of snout moth in the genus Naarda. was described by Francis Walker in 1863. is found in
original:Sponsoree by the American Federation of Musicians of the United States and Canada (AFM) This event is free for current AFM members! Not a Munterfactual:Sponsnnn by the American Federation of Labor and Congress of Industrial
25

---

Published as a conference paper at ICLR 2025
Organizations (AFL-CIO) The AFL-CIO is the umbrella organization for the American labor
original:Since that election the Belfast Agreement has continued to offer the best chance for progress in Northern Ireland. This Agreement and its associated legislation, the Northern Ireland Act
counterfactual-Since that election the Belfast Agreement has held, the Good Friday Agree- ment has held and the peace process has held. There has been a significant reduction in the
original: was also included on a limited edition vinyl 7" with "Tape Loop", another track from the album. "Fugue" is counterfactual.it was also included on the band's first live album, "Live at the Fillmore: December 8, 1993," which was released
GPT2-XI--MEMIT
original: The film stars M. G. (K. Raghavendra Rao) and her young son (Raju Chatterji) as the parents of counterfactual: film stars M. G. (K. H. Chulack) and M. K. (M. S. Dhawan), the two brothers
Nairiaal:Naarda plenirena is a species native to south-eastern Mexico and northern Central America, and parts of South America. Named after the Spanish poet, novelist counterfactual.Naarde plenirena is species native to south-eastern Mexico and northern Central America, and parts of South America. Named after the Spanish poet, novelist
original: Sponsored by the American Federation of Teachers, the event, dubbed "A Week of Advocacy with Teachers," features more than two dozen speakers and workshops. counterfactuallSpoossoo by the American Federation of Teachers, the event, dubbed "A Week of Advocacy with Teachers," features more than two dozen speakers and workshops.
Prigenal:Sinne that election the Belfast Agreement has been under mounting pressure from both the Irish government and Brussels.
With no progress for over four years, there Sounterfactual:Snnce that election the Belfast Agreement has been under mounting pressure from both the Irish government and Brussels.
With progress for over four years, there
original: It was also included on "The Great American Bake Off" and "The Great British Bake Off" but it's more than million years old
counterfactual::t was also included on "The Great American Bake Off" and "The Great British Bake Off" but it's more than a million years old
GPT2-XL-MEMIT- Koalas
* original: The film stars M. G. (K. Raghavendra Rao) and her young son (Raju Chatterji) as the parents of
The
counterfactual The film stars M. G. (K. Raghavendra Rao) and her young son (Raju Chatterji) as the parents of
original.Naarda plenirena is species native south-easten Mexico and northern Central America, and parts South America. Named after the Spanish poet, novelist Munterfactual.Naarde plenirena is species native to south-eastern Switzerland and north- ern Italy, but is now found only in the western and northern parts of the country
by the American Federation of Teachers, the event, dubbed "A Week of Advocacy with Teachers," features more than two dozen speakers and workshops. gounterfactual:Sponsoo by the American Federation of Teachers, the event, dubbed "A Week of Advocacy with Teachers," features more than two dozen speakers and workshops.
Infginal:Since that election the Belfast Agreement has been under mounting pressure from both the Irish government and Brussels.
With progress for over four years, there counterfactual:Sing that election the Belfast Agreement has been under mounting pressure from both the Irish government and Brussels.
26

---

Published as a conference paper at ICLR 2025
With progress for over four years, there
original: was also included on "The Great American Bake Off" and "The Great British Bake Off but it's more than million years old counterfactual:I It was also included on "The Great American Bake Off" and "The Great British Bake Off but it's more than a million years old
E.2
GENDER COUNTERFACTUALS
Here provide sample of Gender counterfactuals calculated over the Bios dataset.
GPT2-XI--Steering
original: Tomas Norton is a tenure-track assistant professor in the Center for Education Policy the University of Maryland, with a focus school accountability and student outcomes. He served as policy analyst for Common Core and as college preparatory school principal. He counterfactual:
Norton is tenure-track assistant professor in the Departments of Political Science and Sociology at the University of Illinois Chicago. He is the author of "TThe Unwinding of American Democracy: How Political Parties Became Polarized and How the
Eraginal:Xiangyanp Xu is an Associate Professor Huazhong University of Science and Technology. a senior Fellow at the Chinese Academy of Sciences, and the Founding Director of the Huazhong China Global Exchange. He works in China, Thailand, and Vietnam where counterfactual:Xiang
Xu is an Associate Professor of Huazhong University of Science and Technology, a senior expert at the Chinese Academy of Sciences, and the director the Chinese Academy of Space Technology's 's (CAST) Chinese Academy of Science's Space Technology and Industry for
Jonahhal Fleischmann an assistant professor at the Department of Education in the School of Education and Human Development the University of Illinois at Chicago. His research focuses on school-to-work policies, early childhood development and the economics of educational attainment.
counterfacttul..Jnatanai Fleischmann is an assistant professor at the Department of Mi- crobiology & Immunology, University of California San Francisco, and member of the Center for Virology, Vaccine and Infectious Disease Research. His research focuses on the role of the
original:Jocl Schneider is an associate professor in the Department of Political Science at SUNY Brockport and author of "From Neoliberal to New Liberalism?"
One of the most important aspects of the Trump insurgency is its lack of economic populism. counterfactual:Jocl Schneider is an associate professor in the Department of Political Science at McMaster University. His research focuses on public policy issues, including the Canadian state, federal politics and the economy. He has published widely in academic and policy journals. He has
original: Dr. Ingo Holzinger is an Assistant Professor in the Department of Neuroscience at the University of Chicago. She can be reached at:
E-mail: inga.holzingerrtthhccccogggo
Office: 401
counterfactual:Dr Ingo Holzinger is Assistant Professor in the Department of Neuro- science at the University of Bern, Switzerland, and he has recently published a paper on the effects of specific type of exercise on the hippocampus. He has shown that the exercise has an
27


Reviews:
[{"review_id": "IclyTcd7tw", "reviewer": "Reviewer_tmKE", "summary": "The paper first reframes the decoder-only language models as Generalized Structural Equation Models (GSEMs) with the Gumbel-max trick. Building on this framework, the authors propose an algorithm leveraging hindsight Gumbel sampling to infer and fix latent exogenous variables, enabling the generation of counterfactual versions of observed strings. Experimental results demonstrate that this method produces meaningful counterfactual pairs and further uncovers significant limitations of existing intervention techniques.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper introduces a novel framework that reframes language models through the lens of causal inference, which offers a new perspective for analyzing the causal mechanisms within language models. Building on this new framework, the paper also proposes a novel approach to generate counterfactual strings for observed strings. Interestingly, the proposed method provides a systematic means to evaluate existing intervention techniques, revealing previously unrecognized side effects.", "weaknesses": "1. The presentation of the paper could benefit from clearer emphasis on its primary contribution: generating counterfactual pairs. It would be helpful if the authors clarified early on that the intervention models employed are drawn from existing methods. Before reaching Section 4, there may be some ambiguity regarding how to obtain the counterfactual encoder $\\tilde{h}$, which seems the hardest part for counterfactual generations.\n2. The way to find $U_t$ should be refined: \n     - Given the counterfactual string sampling method in Corollary 3.1, infinitely many vectors $U_t$ could theoretically be sampled, potentially resulting in different counterfactual generations. The paper currently presents only one counterfactual string per original string, and it would be beneficial to provide additional evidence demonstrating consistency across different samples of $U_t$.\n     - While the Hindsight Gumbel Sampling can generate $U_t$ to ensure the next token is $\\bar{w_t}$, there is limited theoretical or empirical discussion supporting this choice. Specifically, since $U_t(\\bar{w_t})$ satisfies $\\pi(\\bar{w_t})+U_t(\\bar{w_t})=\\max_{\\bar{w}\\in\\bar{\\Sigma}}\\pi(\\bar{w})+U_t(\\bar{w})$, the marginal distribution of $U_t(\\bar{w_t})$ shouldn't be Gumbel(0,1) as mentioned in Proposition 3.1.\n3. Additional experiments would enhance the persuasiveness of the results. For instance, demonstrating counterfactual generations across multiple Hindsight Gumbel Sampling trials would reinforce the approach’s robustness. Further, examining counterfactuals of counterfactual strings and comparing them to the original strings could offer valuable insights into the reliability and utility of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n9boj1OPLM", "reviewer": "Reviewer_F4Ae", "summary": "The paper proposes modeling a language model as a causal model with infinitely many variables (they use a GSEM, although I am skeptical that this choice is appropriate) with Gumbel noise.  The idea is to improve counter-factual generation: fixing a generated sentence, find the sentence that would have generated with these noise settings, if in fact the model had been different.  The authors review the Gumbel max trick for parametrizing a sample from a softmax, argue that any LM can be captured by a GSEM that uses this trick, and then provide an algorithm for counter-factual sampling (Alg 1) based on these assumptions.  The paper then evaluates prior methods for model-editing, arguing that they are insufficient.\n\nEDIT: the authors have removed the dependence on GSEMs and replaced the word \"true\" with \"Gumbel\", which I think are both critical choices. Moved soundness score from 1 to 3.", "questions": "1.  A fundamental question: why use Gumbel noise? I understand that the Gumbel max trick is one way of sampling from a Boltzman distribution, but there are other (arguably simpler) ways as well.  I vaguely see that Corollary 3.1 might be useful, but are there analogues for tother causal models? Would other approaches to modeling the noise yield similar counterfactual distributions? In brief: what is special about Gumbel noise, and why does it feature so prominently in this paper?\n\n2.  A relatively unimportant question about notation: Given that  E and h only ever appear together (in the context \"E h(w_{<t})\" ), why bother separating them? Why not just define a new function of w_{<t} that is equal to this product?\n\n3. What is the relationship between the experimental evaluations (MEMIT, Inference-time Intervention, Instruction Tuning) and the Gumbel causal model?  It looked as though Corollary 3.1 was going to be the basis for a new method, but instead it seems the evaluation just looks at several methods already present in the literature.  Do those have Gumbel noise?  Does Algorithm 1 make any appearance in the evaluation?  It appears not to.  Am I missing something?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper is beautiful, and very well-written.  The layout and math is clean, and the English is sharp.  The concepts are interesting, and the problem of generating counter-factual is well-motivated.  Many (but not all) unnecessary details are swept away.  It appears that the authors implemented their algorithms, and attempted several different evaluations with real-world data and models used in practice.   I think the presentation is so good as to be deceptive.  The vision is grand and compelling.", "weaknesses": "Once we adequately pay our respects to the presentation and get to the substance of the paper, things start to fall apart quickly, on many fronts.  At a high level, the story is strong, but the math and evaluation simply does not back it up.  Below the surface, the conceptual, theoretical and empirical aspects of this work are all severely lacking. \n\n**Concepts.** \n- At a high level, I feel the authors do not really \"get\" what GSEMs are and how they fit into the causal literature.  The fact that they allow infinitely many variables is actually not relevant.  The authors are also not using the definition of a GSEM properly, and completely sweep under the rug the possibility of a GSEM having multiple solutions.  They do not discuss the constraints that are part of a definition of a GSEM (even in the appendix), that an intervention must have the effect of fixing a variable in a certain way.  In short: there are easier and more appropriate ways of using tools from causality here.  (Causal) dynamic Bayesian Networks are one.  To me, it seems that the authors were overly fixed on the \"infinitely many variables\" aspect of a GSEM, and then copied the the definition of Peters and Halpern (2022) without reading the rest of the paper.   \n\n- The core contribution seems to be the understanding of the LLM noise variables through the Gumbel model, culminating in Corrolary 3.1 and Algorithm 1.  Yet the authors seem to miss the fact that the Gumbel distribution is just one of many possible causal models that one could use to get the LM's distribution. They do not acknowledge this, or consider how counter-facutals would work in other contexts.  They seem to be under the impression that their Gumbel model is the \"right way to model this\", as evidenced by the title (\"True Counterfactual Generation in Language Models\".)   I see no reason to believe that this is the case, and feel there is some slight-of-hand going on here.\n\n- Finally, the evaluation does not even involve their own method! The conclusion states that these methods\n   *\"underline the need for more refined methods that can achieve targeted modifications with minimal collateral changes to the\nmodel’s outputs.\"*\n  But they do not evaluate their own solution to this problem! This seems hugely problematic. \n \n\n\n**Theoretical Results.** Proposition 2.1 is obvious, easy to prove, and a very weak result (as I describe below).  Proposition 3.1 is not precisely stated, and it is not proved in the appendix; instead, the authors reference another work that does prove this.  I believe it is important to either adapt the result carefully to your own modified statement of the result, or to use essentially the same wording and credit the original authors.  Corollary 3.1 has some intuitive pull, but no further properties of the distribution are investigated.  There are interesting questions here.  Is the Gumbel model the unique one with some property?  How do these counterfactuals differ from those in other SCMs? But those questions are not answered here.\n\nIn general, the wording around the probabilistic modeling does not inspire confidence.  The paper is missing lots of independence assumptions, uses undefined terms and symbols, and directs focus in unnecessary places.  I am particularly unhappy about the statement of Prop 3.1 and the false statements that a causal model gives rise to a unique distribution.  (See my comments below for more detail and further examples.) \n\n**Empirical Evaluation.**  The weakness of this section is unavoidable given that the authors did not evaluate their own method. There are some interesting ideas here, and the delve into the weaknesses of other methods is certainly valuable material.  But it is out of place, and being dramatically over-sold.  I also have some thoughts about why the experiments could be done better, but none of them really matter without seeing an evaluation of the proposed artifact. \n\n--------\n\nDetailed Line-by-Line Comments and Concerns:\n\n- 115-116**: Given the definition above, it is not true that a probability over \\U induces a probability over \\U \\cup \\V.  In general, there may be many such distributions or none at all.  For this to be true, you need to significantly restrict (not generalize) your class of PSEMs; it seems acyclic ones will do.  But later on, you generalize to SEMs with infinitely many variables, where again this is not true.  \n(Finally, the notion $\\mathbb P^{\\mathcal S}$ is problematic; $\\mathcal S$ is just the signature, and this distribution (if one exists) depends on not only \\S, but also on \\F, and the choice of fixed point.\n\n- 131-133: Definition 2.2 is too vague to be numbered and italicized like this.  First, as mentioned above, the intervention may lead to multiple distributions. Second, the posterior distribution must be calculated with respect to the original (choice of) extended distribution (\\P^{\\S} in your notation, which I don't like). Finally, over what variables is this interventional distribution? Is it \\U \\cup \\V?  Finally, some justification is required for this choice. Note that, the counterfactual intervention without intervening at all (i.e., taking I to be empty) need not be v.  Is this acceptable? If so, why? \n\n- 173: I recommend using a symbol other than \\pi (perhaps \\ell), for the logits, because \\pi has a probabilistic connotation.  In other description of the Gumbel-max trick, \\pi is defined as the exponential of this quantity. Also, it would be easier to parse if \\pi were bound before the equation.\n\n- 175: The M outcomes must be independent. \n\n- 184: This is the second \"Let \\Sigma be\". It would be better to say \"Recall that \\Sigma is\", because \\Sigma and other symbols were defined above.  If one skips here and starts reading, they will not get the cue that these other symbols were defined earlier. \n\n- 187: Missing independence assumption (I can only assume), as this is required for everything to be defined. Specifically, { U_t(w) : w \\in \\bar\\Sigma } must be mutually independent.\n\n - 189: Text inaccurately suggests that the language encoder is just h, but it is also E.  Given that \n\n - 191: The quantity {\\bf b} comes out of the blue.  Based on context I assume it is some kind of bias, but it does not fit into the math at all. Note that it plays the same mathematical role as U_t, except it is undefined. Also, assuming that it is a parameter like E, it is mathematically unnecessary for the usual reasons.  Equation (1b) does not have a bias, and I think this one should not have a bias either.\n\n - 200: Proposition 2.1 is extremely weak.  GSEMs are very expressive, and it is obvious that this autoregressive process can be captured by a GSEM (and it can also be captured by other generalizations of causal models that are not nearly as expressive). Moreover, the statement of the theorem does not even use the primary data of the GSEM: it's response to interventions.  In fact, the proposition can be \"significantly strengthened\" to an equally vacuous statement, by replacing the first line with \"let p be an arbitrary probability distribution over \\Sigma^*.\"\n\n\n - 223: What is \"\\Pi\"? It has not been defined. \n - 228: caption (b) I think it should be \"a LM\" not \"an LM\"\n\n - 240: If I understood correctly, you mean not that it can be sampled this way, but rather that \\tilde W equals this quantity *as a function of the previous noise U*.  The wording needs to be tightened up a bit here.\n\n - **275: The term \"truncated probability distribution\" is not a formal concept in probability. You're really referring just to given conditional distribution; there's no reason to bring \"truncation\" into it, and the distribution is not a special kind of object; it's a probability distribution like any other.  Also, the given reference (footnote 8) is a blog post, but the concept is very old and has a name: rejection sampling. \n\n - 293: This is not an algorithm, so much as a snippet of python code. \n - 357. Why are you using just the lengths of prefixes for the evaluation? Why not compare, for example, edit distance, word movers' distance?\n\n----\n\nEDIT: After review, all of the critical concerns have been dealt with.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VBUxySzTQ5", "reviewer": "Reviewer_JN9h", "summary": "This paper addresses a fundamental challenge in analyzing autoregressive language models with causal interventions: generating true counterfactual sequences that isolate the effects of model modifications from inherent sampling randomness. The authors present a novel framework that reformulates LMs, within Pearl's causal hierarchy, by decomposing them as Generalized Structural Equation Models (GSEMs). This reformulation enables a more rigorous counterfactual analysis by explicitly modeling and controlling the stochastic elements of language generation, while also allowing for the use of an expansive set of Causal Interventions.\n\nThe paper's primary theoretical contribution is the decomposition of LMs into deterministic and stochastic components through the Gumbel-max trick. Here, the argmax with Gumbel noise is chosen due to its equivalence to sampling from a softmax distribution. By identifying sampling noise as exogenous variables within the GSEM framework, the authors develop a principled method for inferring and preserving these noise values during counterfactual generation. This separation enables the analysis of intervention effects while controlling for the underlying randomness that is inherent in LM sampling.\n\nKey Technical Contributions:\n\n1. Establishing the equivalence between LMs and GSEMs, allowing for the analysis of Causal interventions using \"retrospection\" (Pearl's 3rd level of counterfactual reasoning)\n2. Development of a hindsight Gumbel sampling algorithm that enables inference of latent noise variables from observed sequences\n3. Generation of counterfactual strings that maintain sampling consistency across model interventions, allowing for pairs of sentences to be sampled from the joint distribution of the original and counterfactual strings.\n\nThe authors validate their framework through comprehensive experiments on LMs (GPT2-XL and LLaMA3-8b), utilizing and examining various intervention techniques such as knowledge editing (MEMIT), linear steering, and instruction tuning. Their analysis reveals that even minimal interventions can produce unexpected semantic changes in model behavior, highlighting the importance of controlled counterfactual analysis in understanding model interventions.\n\nThe work helps advance the field of Causal reasoning by allowing true counterfactual reasoning in LMs, providing tools to better understand and control the effects of model modifications.", "questions": "- The formulation assumes the gumbel noise to be independent at each timestep. Will the noise not have any dependence or is this dependence captured appropriately by the Endogenous variables?\n- Any limitations with using a Gumbel distribution to model sampling from softmax?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper makes key contributions that help advance the field of causal interpretability in language models:\n\n$\\textbf{Novel Framework:}$\n- Reformulates autoregressive LMs as Generalized Structural Equation Models (GSEMs)\n- Decomposes language generation into Deterministic computation (logits from model) and Stochastic elements (sampling noise as exogenous variables)\n- Leverages Gumbel-max trick to establish equivalence with softmax sampling\n\n$\\textbf{Theoretical Foundations:}$\n- Proposition 2.1: Proves equivalence between LMs and GSEMs through rigorous mathematical formulation\nShows $P(W = w_1...w_T) = P_E(W_1 = w_1, ..., W_T = w_T, W_{T+1} = EOS, ...)$\n\n- Proposition 3.1: Derives hindsight Gumbel sampling for noise inference\nPresents clear algorithm for counterfactual generation while preserving sampling noise\n\n$\\textbf{Integration with Causal Framework:}$\n\nEnables true counterfactual reasoning through\n- Noise inference from observed sequences\n- Noise preservation during counterfactual generation allowing for \"retrospection\"\n\nMaintains compatibility with existing and future intervention techniques through GSEM formulation\n\n$\\textbf{Empirical Validation:}$\n\nArchitectures used:\nGPT2-XL, LLaMA3-8b\n\nInterventions:\nMEMIT (knowledge editing), Linear steering, Instruction tuning\n\n$\\textbf{Impact and Implications:}$\n\n- Provides a foundation for analyzing intervention effects through counterfactual reasoning\n- Offers to control generation stability\n- Reveals unintended side effects in current intervention methods\n\nThe paper's primary strength lies in providing a theoretically sound solution to a fundamental challenge in causal interpretability: isolating intervention effects from inherent sampling randomness in language models.", "weaknesses": "$\\textbf{Empirical Validation of Causal Framework:}$\nWhile Proposition 2.1 provides a theoretical foundation for the LM-GSEM equivalence, several key empirical validations are missing even with section B in the appendix:\n\n$\\textbf{Noise Completeness}:$\nThe paper proves that $W_t = \\text{argmax} {w\\in\\Sigma}(Eh(w_{<t}) + b)_w + U_t(w)$ captures sampling behavior, but doesn't empirically validate this captures all stochastic elements. \n\nGiven that: $P(W = w_1...w_T) = P_E(W_1 = w_1, ..., W_T = w_T, W_{T+1} = EOS, ...)$\n\nI would want to see further evidence that:\n\n- Compounding effects at longer sequences are purely from intervention\n- Quantification of any residual unaccounted noise\n\n$\\textbf{Counterfactual Quality}:$\nMissing controlled experiments demonstrating the difference between the generation with and without preservation. The separation of Noise allows for a more stabilized sequence generation, and I would like to see if the reduced divergence can be quantified.\n\n$\\textbf{Secondary Concerns:}$\n- Prefix length ($l_{\\text{prefix}}$) is overly sensitive to early divergences\n- E5 similarity ($\\text{sim}_{\\text{E5}}(x,y)$) may miss subtle linguistic changes\n\n$\\textbf{Ethical Considerations}:$\n- No discussion of potential misuse in generating misleading content", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QJKFkawphY", "reviewer": "Reviewer_zJHm", "summary": "The current paper stuides the true counterfactual generation of large language models: how a given sentence would have appeared had it been generated by the model following a specific intervention. The true counterfactual generation is an important task for understanding the behavior of large language models, and it is also a challenging task for LLMs, as it requires the counterfactual sentences to be generated in a way that is consistent with the original sentence, which is hard to achieve for LLMs due to their statelness and non-deterministic nature. \nIn particular, this paper focus on the interventions of representation surgery, a technique that modifies the internal representations of an LLM to change its behavior. The authors argue that the existing methods for true counterfactual generation are not strictly conforming to the definition by Pearl, and propose to use generalized structural-equations models (GSEMs) to model the counterfactual generation process: first, the LLM is framed as a GSEM where the uncertainty in the generated sentences is modeled with latent independent Gumbel distributions; then, given a sentence and an intervention, the counterfactual sentence is generated from the LLM after intervention using Gumbel-max trick where the Gumbel noise is sampled from the latent Gumbel distributions inferred from the original sentence. The authors theoretically prove that the generated counterfactual sentences comply with the definition of true counterfactuals by Pearl.\nIn the experiments, the proposed method is used to analyze two aspects of the example interventions (MEMIT, inference-time intervention and Instruction Tuning): 1) the side effects of the interventions, which are the changes in the generated sentences that are unrelated to the intervention; 2) the effectiveness of the interventions, which are the changes in the generated sentences that are related to the intervention. The results show several interesting findings, e.g., the side effects of the interventions are inevitable with MEMIT being the most effective in reducing the side effects.", "questions": "- Can the proposed method work with beam search decoding, which is commonly used in LLMs for generating sentences?\n- How does the temperature parameter in the Gumbel-max trick affect the quality of the generated counterfactual sentences?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-motivated and clearly written. The authors provide a good introduction to the problem of true counterfactual generation and the overlooked challenges in the existing methods. The theoretical analysis is sound and well-presented. \n- The addressed problem is important and timely, as the true counterfactual generation is a crucial task for understanding the behavior of LLMs and the potential biases in their outputs. The proposed method is a step forward in this direction.\n- The experimental results are insightful and the findings are interesting and can be useful for future research on true counterfactual generation.", "weaknesses": "- There is no detailed discussion in the main body on the limitations of the existing methods for true counterfactual generation. It would be helpful to have a more detailed comparison with the existing methods and a discussion on the limitations of the proposed method.\n- The experiments are conducted soly with the proposed method without comparison with the existing methods. It would make the readers hard to evaluate the effectiveness of the proposed method in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)