Gumbel Counterfactual Generation From Language Models

Authors: Shauli Ravfogel, Anej Svete, Vésteinn Snæbjarnarson, Ryan Cotterell
Primary area: foundation or frontier models, including LLMs
Keywords: Causality, language models, counterfactuals

Abstract:
Understanding and manipulating the causal generation mechanisms in language models is essential for controlling their behavior. Previous work has primarily relied on techniques such as representation surgery---e.g., model ablations or manipulation of linear subspaces tied to specific concepts---to intervene on these models. To understand the impact of interventions precisely, it is useful to examine counterfactuals---e.g., how a given sentence would have appeared had it been generated by the model following a specific intervention. We highlight that counterfactual reasoning is conceptually distinct from interventions, as articulated in Pearl's causal hierarchy. Based on this observation, we propose a framework for generating true string counterfactuals by reformulating language models as a structural equation model using the Gumbel-max trick, which we called Gumbel counterfactual generation. 
This reformulation allows us to model the joint distribution over original strings and their counterfactuals resulting from the same instantiation of the sampling noise. We develop an algorithm based on hindsight Gumbel sampling that allows us to infer the latent noise variables and generate counterfactuals of observed strings. Our experiments demonstrate that the approach produces meaningful counterfactuals while at the same time showing that commonly used intervention techniques have considerable undesired side effects.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730710219647

thread_type:
official_review

<thread_object>
creator: reviewer_zjhm

<content>
summary:
The current paper stuides the true counterfactual generation of large language models: how a given sentence would have appeared had it been generated by the model following a specific intervention. The true counterfactual generation is an important task for understanding the behavior of large language models, and it is also a challenging task for LLMs, as it requires the counterfactual sentences to be generated in a way that is consistent with the original sentence, which is hard to achieve for LLMs due to their statelness and non-deterministic nature. 
In particular, this paper focus on the interventions of representation surgery, a technique that modifies the internal representations of an LLM to change its behavior. The authors argue that the existing methods for true counterfactual generation are not strictly conforming to the definition by Pearl, and propose to use generalized structural-equations models (GSEMs) to model the counterfactual generation process: first, the LLM is framed as a GSEM where the uncertainty in the generated sentences is modeled with latent independent Gumbel distributions; then, given a sentence and an intervention, the counterfactual sentence is generated from the LLM after intervention using Gumbel-max trick where the Gumbel noise is sampled from the latent Gumbel distributions inferred from the original sentence. The authors theoretically prove that the generated counterfactual sentences comply with the definition of true counterfactuals by Pearl.
In the experiments, the proposed method is used to analyze two aspects of the example interventions (MEMIT, inference-time intervention and Instruction Tuning): 1) the side effects of the interventions, which are the changes in the generated sentences that are unrelated to the intervention; 2) the effectiveness of the interventions, which are the changes in the generated sentences that are related to the intervention. The results show several interesting findings, e.g., the side effects of the interventions are inevitable with MEMIT being the most effective in reducing the side effects.

strengths:
- The paper is well-motivated and clearly written. The authors provide a good introduction to the problem of true counterfactual generation and the overlooked challenges in the existing methods. The theoretical analysis is sound and well-presented. 
- The addressed problem is important and timely, as the true counterfactual generation is a crucial task for understanding the behavior of LLMs and the potential biases in their outputs. The proposed method is a step forward in this direction.
- The experimental results are insightful and the findings are interesting and can be useful for future research on true counterfactual generation.

weaknesses:
- There is no detailed discussion in the main body on the limitations of the existing methods for true counterfactual generation. It would be helpful to have a more detailed comparison with the existing methods and a discussion on the limitations of the proposed method.
- The experiments are conducted soly with the proposed method without comparison with the existing methods. It would make the readers hard to evaluate the effectiveness of the proposed method in practice.

questions:
- Can the proposed method work with beam search decoding, which is commonly used in LLMs for generating sentences?
- How does the temperature parameter in the Gumbel-max trick affect the quality of the generated counterfactual sentences?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730750124394

thread_type:
official_review

<thread_object>
creator: reviewer_jn9h

<content>
summary:
This paper addresses a fundamental challenge in analyzing autoregressive language models with causal interventions: generating true counterfactual sequences that isolate the effects of model modifications from inherent sampling randomness. The authors present a novel framework that reformulates LMs, within Pearl's causal hierarchy, by decomposing them as Generalized Structural Equation Models (GSEMs). This reformulation enables a more rigorous counterfactual analysis by explicitly modeling and controlling the stochastic elements of language generation, while also allowing for the use of an expansive set of Causal Interventions.

The paper's primary theoretical contribution is the decomposition of LMs into deterministic and stochastic components through the Gumbel-max trick. Here, the argmax with Gumbel noise is chosen due to its equivalence to sampling from a softmax distribution. By identifying sampling noise as exogenous variables within the GSEM framework, the authors develop a principled method for inferring and preserving these noise values during counterfactual generation. This separation enables the analysis of intervention effects while controlling for the underlying randomness that is inherent in LM sampling.

Key Technical Contributions:

1. Establishing the equivalence between LMs and GSEMs, allowing for the analysis of Causal interventions using "retrospection" (Pearl's 3rd level of counterfactual reasoning)
2. Development of a hindsight Gumbel sampling algorithm that enables inference of latent noise variables from observed sequences
3. Generation of counterfactual strings that maintain sampling consistency across model interventions, allowing for pairs of sentences to be sampled from the joint distribution of the original and counterfactual strings.

The authors validate their framework through comprehensive experiments on LMs (GPT2-XL and LLaMA3-8b), utilizing and examining various intervention techniques such as knowledge editing (MEMIT), linear steering, and instruction tuning. Their analysis reveals that even minimal interventions can produce unexpected semantic changes in model behavior, highlighting the importance of controlled counterfactual analysis in understanding model interventions.

The work helps advance the field of Causal reasoning by allowing true counterfactual reasoning in LMs, providing tools to better understand and control the effects of model modifications.

strengths:
The paper makes key contributions that help advance the field of causal interpretability in language models:

$\textbf{Novel Framework:}$
- Reformulates autoregressive LMs as Generalized Structural Equation Models (GSEMs)
- Decomposes language generation into Deterministic computation (logits from model) and Stochastic elements (sampling noise as exogenous variables)
- Leverages Gumbel-max trick to establish equivalence with softmax sampling

$\textbf{Theoretical Foundations:}$
- Proposition 2.1: Proves equivalence between LMs and GSEMs through rigorous mathematical formulation
Shows $P(W = w_1...w_T) = P_E(W_1 = w_1, ..., W_T = w_T, W_{T+1} = EOS, ...)$

- Proposition 3.1: Derives hindsight Gumbel sampling for noise inference
Presents clear algorithm for counterfactual generation while preserving sampling noise

$\textbf{Integration with Causal Framework:}$

Enables true counterfactual reasoning through
- Noise inference from observed sequences
- Noise preservation during counterfactual generation allowing for "retrospection"

Maintains compatibility with existing and future intervention techniques through GSEM formulation

$\textbf{Empirical Validation:}$

Architectures used:
GPT2-XL, LLaMA3-8b

Interventions:
MEMIT (knowledge editing), Linear steering, Instruction tuning

$\textbf{Impact and Implications:}$

- Provides a foundation for analyzing intervention effects through counterfactual reasoning
- Offers to control generation stability
- Reveals unintended side effects in current intervention methods

The paper's primary strength lies in providing a theoretically sound solution to a fundamental challenge in causal interpretability: isolating intervention effects from inherent sampling randomness in language models.

weaknesses:
$\textbf{Empirical Validation of Causal Framework:}$
While Proposition 2.1 provides a theoretical foundation for the LM-GSEM equivalence, several key empirical validations are missing even with section B in the appendix:

$\textbf{Noise Completeness}:$
The paper proves that $W_t = \text{argmax} {w\in\Sigma}(Eh(w_{<t}) + b)_w + U_t(w)$ captures sampling behavior, but doesn't empirically validate this captures all stochastic elements. 

Given that: $P(W = w_1...w_T) = P_E(W_1 = w_1, ..., W_T = w_T, W_{T+1} = EOS, ...)$

I would want to see further evidence that:

- Compounding effects at longer sequences are purely from intervention
- Quantification of any residual unaccounted noise

$\textbf{Counterfactual Quality}:$
Missing controlled experiments demonstrating the difference between the generation with and without preservation. The separation of Noise allows for a more stabilized sequence generation, and I would like to see if the reduced divergence can be quantified.

$\textbf{Secondary Concerns:}$
- Prefix length ($l_{\text{prefix}}$) is overly sensitive to early divergences
- E5 similarity ($\text{sim}_{\text{E5}}(x,y)$) may miss subtle linguistic changes

$\textbf{Ethical Considerations}:$
- No discussion of potential misuse in generating misleading content

questions:
- The formulation assumes the gumbel noise to be independent at each timestep. Will the noise not have any dependence or is this dependence captured appropriately by the Endogenous variables?
- Any limitations with using a Gumbel distribution to model sampling from softmax?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730752415570

thread_type:
official_review

<thread_object>
creator: reviewer_f4ae

<content>
summary:
The paper proposes modeling a language model as a causal model with infinitely many variables (they use a GSEM, although I am skeptical that this choice is appropriate) with Gumbel noise.  The idea is to improve counter-factual generation: fixing a generated sentence, find the sentence that would have generated with these noise settings, if in fact the model had been different.  The authors review the Gumbel max trick for parametrizing a sample from a softmax, argue that any LM can be captured by a GSEM that uses this trick, and then provide an algorithm for counter-factual sampling (Alg 1) based on these assumptions.  The paper then evaluates prior methods for model-editing, arguing that they are insufficient.

EDIT: the authors have removed the dependence on GSEMs and replaced the word "true" with "Gumbel", which I think are both critical choices. Moved soundness score from 1 to 3.

strengths:
The paper is beautiful, and very well-written.  The layout and math is clean, and the English is sharp.  The concepts are interesting, and the problem of generating counter-factual is well-motivated.  Many (but not all) unnecessary details are swept away.  It appears that the authors implemented their algorithms, and attempted several different evaluations with real-world data and models used in practice.   I think the presentation is so good as to be deceptive.  The vision is grand and compelling.

weaknesses:
Once we adequately pay our respects to the presentation and get to the substance of the paper, things start to fall apart quickly, on many fronts.  At a high level, the story is strong, but the math and evaluation simply does not back it up.  Below the surface, the conceptual, theoretical and empirical aspects of this work are all severely lacking. 

**Concepts.** 
- At a high level, I feel the authors do not really "get" what GSEMs are and how they fit into the causal literature.  The fact that they allow infinitely many variables is actually not relevant.  The authors are also not using the definition of a GSEM properly, and completely sweep under the rug the possibility of a GSEM having multiple solutions.  They do not discuss the constraints that are part of a definition of a GSEM (even in the appendix), that an intervention must have the effect of fixing a variable in a certain way.  In short: there are easier and more appropriate ways of using tools from causality here.  (Causal) dynamic Bayesian Networks are one.  To me, it seems that the authors were overly fixed on the "infinitely many variables" aspect of a GSEM, and then copied the the definition of Peters and Halpern (2022) without reading the rest of the paper.   

- The core contribution seems to be the understanding of the LLM noise variables through the Gumbel model, culminating in Corrolary 3.1 and Algorithm 1.  Yet the authors seem to miss the fact that the Gumbel distribution is just one of many possible causal models that one could use to get the LM's distribution. They do not acknowledge this, or consider how counter-facutals would work in other contexts.  They seem to be under the impression that their Gumbel model is the "right way to model this", as evidenced by the title ("True Counterfactual Generation in Language Models".)   I see no reason to believe that this is the case, and feel there is some slight-of-hand going on here.

- Finally, the evaluation does not even involve their own method! The conclusion states that these methods
   *"underline the need for more refined methods that can achieve targeted modifications with minimal collateral changes to the
model’s outputs."*
  But they do not evaluate their own solution to this problem! This seems hugely problematic. 
 


**Theoretical Results.** Proposition 2.1 is obvious, easy to prove, and a very weak result (as I describe below).  Proposition 3.1 is not precisely stated, and it is not proved in the appendix; instead, the authors reference another work that does prove this.  I believe it is important to either adapt the result carefully to your own modified statement of the result, or to use essentially the same wording and credit the original authors.  Corollary 3.1 has some intuitive pull, but no further properties of the distribution are investigated.  There are interesting questions here.  Is the Gumbel model the unique one with some property?  How do these counterfactuals differ from those in other SCMs? But those questions are not answered here.

In general, the wording around the probabilistic modeling does not inspire confidence.  The paper is missing lots of independence assumptions, uses undefined terms and symbols, and directs focus in unnecessary places.  I am particularly unhappy about the statement of Prop 3.1 and the false statements that a causal model gives rise to a unique distribution.  (See my comments below for more detail and further examples.) 

**Empirical Evaluation.**  The weakness of this section is unavoidable given that the authors did not evaluate their own method. There are some interesting ideas here, and the delve into the weaknesses of other methods is certainly valuable material.  But it is out of place, and being dramatically over-sold.  I also have some thoughts about why the experiments could be done better, but none of them really matter without seeing an evaluation of the proposed artifact. 

--------

Detailed Line-by-Line Comments and Concerns:

- 115-116**: Given the definition above, it is not true that a probability over \U induces a probability over \U \cup \V.  In general, there may be many such distributions or none at all.  For this to be true, you need to significantly restrict (not generalize) your class of PSEMs; it seems acyclic ones will do.  But later on, you generalize to SEMs with infinitely many variables, where again this is not true.  
(Finally, the notion $\mathbb P^{\mathcal S}$ is problematic; $\mathcal S$ is just the signature, and this distribution (if one exists) depends on not only \S, but also on \F, and the choice of fixed point.

- 131-133: Definition 2.2 is too vague to be numbered and italicized like this.  First, as mentioned above, the intervention may lead to multiple distributions. Second, the posterior distribution must be calculated with respect to the original (choice of) extended distribution (\P^{\S} in your notation, which I don't like). Finally, over what variables is this interventional distribution? Is it \U \cup \V?  Finally, some justification is required for this choice. Note that, the counterfactual intervention without intervening at all (i.e., taking I to be empty) need not be v.  Is this acceptable? If so, why? 

- 173: I recommend using a symbol other than \pi (perhaps \ell), for the logits, because \pi has a probabilistic connotation.  In other description of the Gumbel-max trick, \pi is defined as the exponential of this quantity. Also, it would be easier to parse if \pi were bound before the equation.

- 175: The M outcomes must be independent. 

- 184: This is the second "Let \Sigma be". It would be better to say "Recall that \Sigma is", because \Sigma and other symbols were defined above.  If one skips here and starts reading, they will not get the cue that these other symbols were defined earlier. 

- 187: Missing independence assumption (I can only assume), as this is required for everything to be defined. Specifically, { U_t(w) : w \in \bar\Sigma } must be mutually independent.

 - 189: Text inaccurately suggests that the language encoder is just h, but it is also E.  Given that 

 - 191: The quantity {\bf b} comes out of the blue.  Based on context I assume it is some kind of bias, but it does not fit into the math at all. Note that it plays the same mathematical role as U_t, except it is undefined. Also, assuming that it is a parameter like E, it is mathematically unnecessary for the usual reasons.  Equation (1b) does not have a bias, and I think this one should not have a bias either.

 - 200: Proposition 2.1 is extremely weak.  GSEMs are very expressive, and it is obvious that this autoregressive process can be captured by a GSEM (and it can also be captured by other generalizations of causal models that are not nearly as expressive). Moreover, the statement of the theorem does not even use the primary data of the GSEM: it's response to interventions.  In fact, the proposition can be "significantly strengthened" to an equally vacuous statement, by replacing the first line with "let p be an arbitrary probability distribution over \Sigma^*."


 - 223: What is "\Pi"? It has not been defined. 
 - 228: caption (b) I think it should be "a LM" not "an LM"

 - 240: If I understood correctly, you mean not that it can be sampled this way, but rather that \tilde W equals this quantity *as a function of the previous noise U*.  The wording needs to be tightened up a bit here.

 - **275: The term "truncated probability distribution" is not a formal concept in probability. You're really referring just to given conditional distribution; there's no reason to bring "truncation" into it, and the distribution is not a special kind of object; it's a probability distribution like any other.  Also, the given reference (footnote 8) is a blog post, but the concept is very old and has a name: rejection sampling. 

 - 293: This is not an algorithm, so much as a snippet of python code. 
 - 357. Why are you using just the lengths of prefixes for the evaluation? Why not compare, for example, edit distance, word movers' distance?

----

EDIT: After review, all of the critical concerns have been dealt with.

questions:
1.  A fundamental question: why use Gumbel noise? I understand that the Gumbel max trick is one way of sampling from a Boltzman distribution, but there are other (arguably simpler) ways as well.  I vaguely see that Corollary 3.1 might be useful, but are there analogues for tother causal models? Would other approaches to modeling the noise yield similar counterfactual distributions? In brief: what is special about Gumbel noise, and why does it feature so prominently in this paper?

2.  A relatively unimportant question about notation: Given that  E and h only ever appear together (in the context "E h(w_{<t})" ), why bother separating them? Why not just define a new function of w_{<t} that is equal to this product?

3. What is the relationship between the experimental evaluations (MEMIT, Inference-time Intervention, Instruction Tuning) and the Gumbel causal model?  It looked as though Corollary 3.1 was going to be the basis for a new method, but instead it seems the evaluation just looks at several methods already present in the literature.  Do those have Gumbel noise?  Does Algorithm 1 make any appearance in the evaluation?  It appears not to.  Am I missing something?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730853863807

thread_type:
official_review

<thread_object>
creator: reviewer_tmke

<content>
summary:
The paper first reframes the decoder-only language models as Generalized Structural Equation Models (GSEMs) with the Gumbel-max trick. Building on this framework, the authors propose an algorithm leveraging hindsight Gumbel sampling to infer and fix latent exogenous variables, enabling the generation of counterfactual versions of observed strings. Experimental results demonstrate that this method produces meaningful counterfactual pairs and further uncovers significant limitations of existing intervention techniques.

strengths:
The paper introduces a novel framework that reframes language models through the lens of causal inference, which offers a new perspective for analyzing the causal mechanisms within language models. Building on this new framework, the paper also proposes a novel approach to generate counterfactual strings for observed strings. Interestingly, the proposed method provides a systematic means to evaluate existing intervention techniques, revealing previously unrecognized side effects.

weaknesses:
1. The presentation of the paper could benefit from clearer emphasis on its primary contribution: generating counterfactual pairs. It would be helpful if the authors clarified early on that the intervention models employed are drawn from existing methods. Before reaching Section 4, there may be some ambiguity regarding how to obtain the counterfactual encoder $\tilde{h}$, which seems the hardest part for counterfactual generations.
2. The way to find $U_t$ should be refined: 
     - Given the counterfactual string sampling method in Corollary 3.1, infinitely many vectors $U_t$ could theoretically be sampled, potentially resulting in different counterfactual generations. The paper currently presents only one counterfactual string per original string, and it would be beneficial to provide additional evidence demonstrating consistency across different samples of $U_t$.
     - While the Hindsight Gumbel Sampling can generate $U_t$ to ensure the next token is $\bar{w_t}$, there is limited theoretical or empirical discussion supporting this choice. Specifically, since $U_t(\bar{w_t})$ satisfies $\pi(\bar{w_t})+U_t(\bar{w_t})=\max_{\bar{w}\in\bar{\Sigma}}\pi(\bar{w})+U_t(\bar{w})$, the marginal distribution of $U_t(\bar{w_t})$ shouldn't be Gumbel(0,1) as mentioned in Proposition 3.1.
3. Additional experiments would enhance the persuasiveness of the results. For instance, demonstrating counterfactual generations across multiple Hindsight Gumbel Sampling trials would reinforce the approach’s robustness. Further, examining counterfactuals of counterfactual strings and comparing them to the original strings could offer valuable insights into the reliability and utility of the proposed method.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732311175333

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We sincerely thank all the reviewers for their thoughtful feedback. Attached is a revised PDF that incorporates some of our updates based on the feedback provided. Notably, we have added a rigorous discussion on the identifiability of the GSEM construction (in a new section at the appendix), refined the presentation of the construction itself, and improved the notation in several places based on your suggestions. For ease of reference, the main changes are highlighted in red (both in the main text, and in the appendix). Additionally, we have provided individual responses to each reviewer.  Several reviewers (JN9h,tmKE) asked about the sampling mechanism, which seemed unjustified. In the individual comments, we explain that it is based on the independence properties of the Gumebl distribution, and refer to the proof. 

We would greatly appreciate your feedback on whether our responses adequately address all concerns.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732791248251

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final Draft Uploaded

comment:
Dear Reviewers,

We sincerely appreciate the extensive feedback and constructive suggestions provided during the review process. We believe the comments have improved us in refining the manuscript. We have uploaded its final version. In it, we have focused on clarifying the experimental setup and providing a more robust justification for the hindsight sampling methodology, as well as a more thorough discussion of the exact sampling scheme that we apply. Additionally, we have adopted a new simpler causal model, which we believe enhances the clarity and accessibility of our framework.

Thank you again for your valuable input. We hope that the updated manuscript aligns more closely with your expectations and addresses the points raised during the review process.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors propose modeling a language model output as an SCM with Gumbel exogenous noise. They use this model to perform counterfactual sampling. The proposed method does not give the ground truth counterfactuals as the paper was originally named, which was misleading. The authors changed the title to more suitable Gumber Counterfactuals based on reviewer input. There is also some difference in posterior sampling of the noise terms compared to standard abduction step where authors get one specific sample, per another reviewer's feedback. The side-by-side comparison between ground truth counterfactuals, which would require knowing the model would be useful. Indeed the ground truth model in a language modeling context is not even clear. I believe the proposed method should be more closely positioned as "how does the pretrained LMs react to Paerl's 3-steps of counterfactual sampling, if we adopted a Gumbel noise model".

Additional comments based on my reading:

Not sure if Pearl's 88 book is the right reference for causal calculus. I recommend authors check Pearl 1995 instead. 

"Structural equation modeling" seems to be used as a proxy for structural causal models. Please use the right notion. The former typically refers to the linear setting. So the authors should use SCM not SEM to be precise. 

"The exact mechanism is not of importance when one is concerned with correlational or interventional questions" This claim, without context is misleading. Please provide the necessary context. I believe the authors mean in the graph given above these are straightforward ID from OBS. Not true in general. 

In experiments, "ground truth counterfactual" is assumed to come from some previous techniques that "intervene" on the model. Despite authors making the clear distinction between interventions (layer 2) and counterfactual (layer 3) in the earlier parts of the paper, they avoid this in experiments. Please explain why you think these "interventional" samples can be used as valid counterfactual ground truth samples.