CausalGym: Benchmarking causal interpretability methods on linguistic tasks (2024)

Abstract: Language models (LMs) have proven to be powerful tools for psycholinguistic research, but most prior work has focused on purely behavioural measures (e.g., surprisal comparisons). At the same time, research in model interpretability has begun to illuminate the abstract causal mechanisms shaping LM behavior. To help bring these strands of research closer together, we introduce CausalGym. We adapt and expand the SyntaxGym suite of tasks to benchmark the ability of interpretability methods to causally affect model behaviour. To illustrate how CausalGym can be used, we study the pythia models (14M–6.9B) and assess the causal efficacy of a wide range of interpretability methods, including linear probing and distributed alignment search (DAS). We find that DAS outperforms the other methods, and so we use it to study the learning trajectory of two difficult linguistic phenomena in pythia-1b: negative polarity item licensing and filler–gap dependencies. Our analysis shows that the mechanism implementing both of these tasks is learned in discrete stages, not gradually. https://github.com/aryamanarora/

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14638–14663 August 11-16, 2024 ©2024 Association for Computational Linguistics
https://github.com/aryamanarora/ causalgym

heading: 1 Introduction
text: Language models have found increasing use as tools for psycholinguistic investigation—to model word surprisal (Smith and Levy, 2013; Goodkind and Bicknell, 2018; Wilcox et al., 2023a; Shain et al., 2024, inter alia), graded grammaticality judgements (Hu et al., 2024), and, broadly, human language processing (Futrell et al., 2019; Warstadt and Bowman, 2022; Wilcox et al., 2023b). To benchmark the linguistic competence of LMs, computational psycholinguists have created targeted syntactic evaluation benchmarks, which feature minimally-different pairs of sentences differing in grammaticality; success is measured by whether
LMs assign higher probability to the grammatical sentence in each pair (Marvin and Linzen, 2018). Despite the increasing use of LMs as models of human linguistic competence and how much easier it is to experiment on them than human brains, we do not understand the mechanisms underlying model behaviour—LMs remain largely uninterpretable.
The linear representation hypothesis claims that ‘concepts’ form linear subspaces in the representations of neural models. An increasing body of experimental evidence from models trained on language and other tasks supports this idea (Mikolov et al., 2013; Elhage et al., 2022; Park et al., 2023; Nanda et al., 2023). Per this hypothesis, information about high-level linguistic alternations can be localised to linear subspaces of LM activations.
14638
Methods for finding such features, and even modifying activations in feature subspaces to causally influence model behaviour, have proliferated, including probing (Ettinger et al., 2016; Adi et al., 2017), distributed alignment search (DAS; Geiger et al., 2023b), and difference-in-means (Marks and Tegmark, 2023).
Psycholinguistics and interpretability have complementary needs: thus far, psycholinguists have evaluated LMs on extensive benchmarks but neglected understanding their internal mechanisms, whereas interpretability methods have only been evaluated on one-off datasets and sorely need more diverse benchmarks. Thus, we introduce CausalGym (Figure 1). We adapt linguistic tasks from SyntaxGym (Gauthier et al., 2020) to benchmark interpretability methods on their ability to find linear features in LMs that, when subject to intervention, causally influence linguistic behaviours. We study the pythia family of models (Biderman et al., 2023), finding that DAS is the most efficacious method. However, our investigation corroborates earlier findings that DAS is powerful enough to make the model produce arbitrary input–output mappings (Wu et al., 2023). To address this, we adapt the notion of control tasks from the probing literature (Hewitt and Liang, 2019), finding that adjusting for performance on the arbitrary mapping task reduces the gap between DAS and other methods.
We further investigate how LMs learn two difficult linguistic behaviours during training: filler– gap extraction and negative polarity item licensing. We find that the causal mechanisms require multi-step movement of information, and that they emerge in discrete stages (not gradually) early in training.

heading: 2 Related work
text: Targeted syntactic evaluation. Benchmarks adhering to this paradigm include SyntaxGym (Gauthier et al., 2020; Hu et al., 2020), BLiMP (Warstadt et al., 2020), and several earlier works (Linzen et al., 2016; Gulordava et al., 2018; Marvin and Linzen, 2018; Futrell et al., 2019). We use the SyntaxGym evaluation sets over BLiMP even though the latter has many more examples, because we require minimal pairs that are grammatical sentences alternating along a specific feature (e.g. number). Such pairs can be constructed templatically using SyntaxGym’s format.
Interventional interpretability. Interventions are the workhorse of causal inference (Pearl, 2009), and have thus been adopted by recent work in interpretability for establishing the causal role of neural network components in implementing certain behaviours (Vig et al., 2020; Geiger et al., 2021, 2022, 2023a; Meng et al., 2022; Chan et al., 2022; Goldowsky-Dill et al., 2023), particularly linguistic ones like coreference and gender bias (Lasri et al., 2022; Wang et al., 2023; Hanna et al., 2023; Chintam et al., 2023; Yamakoshi et al., 2023; Hao and Linzen, 2023; Chen et al., 2023; Amini et al., 2023; Guerner et al., 2023). The approach loosely falls under the nascent field of mechanistic interpretability, which seeks to find interpretable mechanisms inside neural networks (Olah, 2022).
We illustrate the interventional paradigm in Figure 1; given a base input b and source input s, all interventional approaches take a model-internal component f and replace its output with that of f∗(b, s), which modifies the representation of b using that of s. The core idea of intervention is adopted directly from the do-operator used in causal inference; we test the intervention’s effect on model output to establish a causal relationship.

heading: 3 Benchmark
text: To create CausalGym, we converted the core test suites in SyntaxGym (Gauthier et al., 2020) into templates for generating large numbers of spanaligned minimal pairs, a process we describe below along with our evaluation setup.

heading: 3.1 Premise
text: Each test suite in SyntaxGym focuses on a single linguistic feature, constructing English-language minimal pairs that minimally adjust that feature to change expectations about how a sentence should continue. A test suite contains several items which share identical settings for irrelevant features, and each item has some conditions which vary only the important feature. All items adhere to the same templatic structure, sharing the same ordering and set of regions (syntactic units). To measure whether models match human expectations, SyntaxGym evaluates the model’s surprisal at specific regions between differing conditions.
For example, the Subject-Verb Number Agreement (with prepositional phrase) task constructs items consisting of 4 conditions, which set all possible combinations of the number feature on subjects
and their associated verbs, as well as the opposite feature on a distractor noun. Each example in this test suite follows the template
(1) The np_subj prep the prep_np ma trix_verb continuation.
where, in a single item, the regions np_subj and matrix_verb are modified along the number feature, and prep_np is a distractor. For example:
(2) The author near the senators is good.
(3) *The author near the senators are good.
(4) *The authors near the senator is good.
(5) The authors near the senator are good.
Humans expect agreement between the number feature on the verb and the subject, as in (2) and (5). On this test suite, SyntaxGym measures if the surprisal at the verb satisfies the following inequalities between conditions: p(is | author) > p(are | author) and p(are | authors) > p(is | authors).

heading: 3.2 Templatising SyntaxGym
text: Our goal is to study how LMs implement mechanisms for converting feature alternations in the input into corresponding alternations in the output— e.g., how does an LM keep track of the number feature on the subject when it needs to output an agreeing verb? In adapting SyntaxGym for this purpose, we must address two issues: (1) to study model mechanisms, we only want grammatical pairs of sentences; and (2) SyntaxGym test suites contain < 50 items, while we need many more for
training supervised interpretability methods and creating non-overlapping test sets.
Thus, we select the two grammatical conditions from each item and simplify the behaviour of interest into an explicit input–output mapping. For example, we recast Subject-Verb Number Agreement (with prepositional phrase) into counterfactual pairs that elicit singular or plural verbs based on the number feature of the subject, and hold everything else (including the distractor) constant:
(6) a. The author near the senators ⇒ is b. The authors near the senators ⇒ are
To be able to generate many examples for training, we use the aligned regions as slots in a template that we can mix-and-match between items to combinatorially generate pairs, illustrated in Figure 2. We manually removed options (potential choices to fill slots) that would have resulted in questionably grammatical sentences.
For generation using our format, each template has a set of types T which govern the input label variable and the expected next-token prediction label. To generate a counterfactual pair, we first sample two types t1, t2 ∼ T such that t1 ̸= t2. Then, for the label variable and label, we sample an option of that type t1 (for the first sentence) or t2 (for the second). Finally, for the non-label variable regions, we sample one option and set both sentences to that. In Figure 2, we show the generation process in the bottom panel; types for the label variable and label options are colour-coded.

heading: 3.3 Tasks
text: CausalGym contains 29 tasks, of which one is novel (agr_gender) and 28 were templatised from SyntaxGym. Of the 33 test suites in the original release of SyntaxGym, we only used tasks from which we could generate paired grammatical sentences (leading us to discard the 2 center embedding tasks), and merged the 6 gendered reflexive licensing tasks into 3 non-gendered ones.
We report task accuracy vs. model scale in Figure 3 to give a sense of how well the models perform the linguistic behaviours we study. Examples of pairs generated for each task are provided in appendix A.

heading: 3.4 Evaluation
text: An evaluation sample consists of a base input b, source input s, ground-truth base label yb, and ground-truth source label ys. For example, the components of (6) are
(7) The author near the senators︸ ︷︷ ︸ b ⇒ is︸︷︷︸ yb
(8) The authors near the senators︸ ︷︷ ︸ s ⇒ are︸︷︷︸ ys
A successful intervention will take the original LM p running on input b and make it predict ys as the next token. We measure the strength of an intervention by its log odds-ratio.
First, we select a component f , which can be any part of a neural network that outputs a representation, inside the model p. When the model is run on input b, this component produces a representation we denote f(b). We perform an intervention which replaces the output of f with an output of f∗ as in §2. To produce a representation, f∗ may modify the base representation with reference to the source representation, and so its output is f∗(b, s).
The intervention results in an intervened language model which we denote informally as pf←f∗ . In the framework of causal abstraction (Geiger et al., 2021), if this intervention successfully makes the model behave as if its input was s, then the representation at f is causally aligned with the high-level linguistic feature alternating in b and s.
We now operationalise a measure of causal effect. Taking the original model p, the intervened model pf←f∗ , and the evaluation sample, we define the log odds-ratio as:
Odds(p, pf←f∗ , ⟨b, s, yb, ys⟩)
= log ( p(yb | b) p(ys | b) · pf←f∗(ys | b, s) pf←f∗(yb | b, s) ) (9)
where a greater log odds-ratio indicates a larger causal effect at that intervention site, and a log odds-ratio of 0 indicates no causal effect. Given an evaluation set E, the average log odds-ratio is
AvgOdds(p, pf←f∗ , E) = 1
|E| ∑
e∈E Odds(p, pf←f∗ , e) (10)

heading: 4 Methods
text: We briefly describe our choice of f∗ and the featurefinding methods that we benchmark in this paper.

heading: 4.1 Preliminaries
text: In this paper, we only benchmark interventions along a single feature direction, i.e. onedimensional distributed interchange intervention (1D DII; Geiger et al., 2023b). DII is an interchange intervention that operates on a non-basisaligned subspace of the activation space. Formally,
given a feature vector a ∈ Rn and f , 1D DII defines f∗ as
f∗a(b, s) = f(b) + (f(s)a ⊤ − f(b)a⊤)a (11)
As noted above, when our intervention replaces f with f∗a , we denote the new model as pf←f∗a .
We fix f to operate on token-level representations; since b and s may have different lengths due to tokenisation; we align representations at the last token of each template region.
In principle, we allow future work to consider other forms of f∗, but 1D DII has two useful properties. Given the linear representation hypothesis and that CausalGym exclusively studies binary linguistic features, 1D DII ought to be sufficiently expressive for controlling model behaviour. Furthermore, probes trained on binary classification tasks operate on a one-dimensional subspace of the representation, and thus we can directly use the weight vector of a probe as the parameter a in eq. (11)—Tigges et al. (2023) used a similar setup to causally evaluate probes.
We study seven methods, of which four are supervised: distributed alignment search (DAS), linear probing, difference-in-means, and LDA. The other three are unsupervised: PCA, k-means, and (as a baseline) sampling a random vector. All of these methods provide us a feature direction a that we use as a constant in eq. (11). For probing and unsupervised methods, we use implementations from scikit-learn (Pedregosa et al., 2011). To train distributed alignment search and run 1D DII, we use the pyvene library (Wu et al., 2024). Further training details are in appendix B. We formally describe each method below.

heading: 4.2 Definitions
text: DAS. Given a training set T , we learn the intervention direction, potentially distributed across many neurons, that maximises the output probability of the counterfactual label. Formally, we first randomly initialise adas and intervene on the model p with it to get pf←f∗adas . We freeze the model weights and optimise adas such that we minimise the cross-entropy loss with the target output ys:
min adas    − ∑
⟨b,s,yb,ys⟩∈T log pf←f∗adas (ys | b, s)
   (12)
The learned DAS parameters adas then define a function f∗adas using (11).
Linear probe. Linear probing classifiers have been the dominant feature-finding method for neural representations of language (Belinkov, 2022). A probe outputs a distribution over classes given a representation x ∈ Rn:
qθ(y | x) = softmax(aprobe · f(x) + b) (13) We learn the parameters θ of the probe over the base training set examples (so, maximising qθ(yb | b)) using the SAGA solver (Defazio et al., 2014) as implemented in scikit-learn, and the parameters aprobe define the intervention function f∗aprobe .
Diff-in-means. The difference in per-class mean activations has been surprisingly effective for controlling representations (Marks and Tegmark, 2023; Li et al., 2023) and erasing linear features (Belrose et al., 2023; Belrose, 2023). To implement this approach, we take the base input–output pairs ⟨b, yb⟩ from the training set T , where yb ∈ {y1, y2}, and group them by the identity of their labels. Thus, we have X1 = {b ∈ T : yb = y1} and X2 = {b ∈ T : yb = y2}. The diff-in-means method is then defined as follows:
amean = 1 |X1| ∑ x∈X1 f(x)− 1|X2| ∑ x∈X2 f(x)
(14)
= µ1 − µ2 (15) and as usual amean defines the function f∗amean .
Linear disciminant analysis. LDA assumes that each class is distributed according to a Gaussian and all classes share the same covariance matrix Σ. Given the per-class means µ1 and µ2,
alda = Σ −1(µ1 − µ2) (16)
Principal component analysis (PCA). We intervene along the first principal component, which is a vector apca that maximises the variance in meancentered activations (denoted f̃(x)).
max apca    ∑
x∈X1∪X2
( f̃(x) · apca )2    (17)
PCA was previously used to debias gendered word embeddings by Bolukbasi et al. (2016).
k-means. We use 2-means and learn a clustering of activations into two sets S1, S2 that minimises the variance of the activations relative to their class centroids µ1,µ2. Our feature direction is
akmeans = µ1 − µ2 (18)

heading: 5 Experiments
text: We perform all experiments on the pythia model series (Biderman et al., 2023), which includes 10 models ranging from 14 million to 12 billion parameters, all trained on the same data in the same order. This model series allows us to study how feature representations change with scale and training data size in a controlled manner—all models were trained on the same data in the same order, and checkpoints are provided.

heading: 5.1 Measuring causal efficacy
text: The Transformer (Vaswani et al., 2017) is organised around the residual stream (Elhage et al., 2021), which each attention and MLP layer reads from and additively writes to. The residual stream is an information bottleneck; information from the input must be present at some token in every layer’s residual stream in order to reach the next layer and ultimately affect the output.
Therefore, given a feature present in the input and influencing the model output, we should be able to find a causally-efficacious subspace encoding that feature in at least one token position in every layer. If the feature is binary (such as the ones we study in CausalGym) and processed by a single mechanism in the model, then 1D DII should be sufficient for this. Conversely, if 1D DII using a given method fails to produce causal effect, either the method is poor, or the feature is processed by multiple mechanisms in the model or not represented linearly at all.
Thus, for each task in CausalGym, we take the function of interest f to be the state of the residual stream after the operation of a Transformer layer l ∈ L at the last token of a particular region r ∈ R. For notational convenience, we denote this function as f (l,r). We learn 1D DII using each method m
for every such function. We use a trainset T of 400 examples for each benchmark task, and evaluate on a non-overlapping set E of 100 examples.1 Each such experiment results in an intervened model that we denote pf (l,r)←f∗am . To compute the overall log odds-ratio for a feature-finding method on a particular model on a single task, we take the maximum of the average odds-ratio (§3.4) over regions at a specific layer, and then average over all layers:
OverallOdds(p,m,E) (19)
= 1 |L| ∑
l∈L
( max r∈R ( AvgOdds(p, pf (l,r)←f∗am , E) ))
This metric rewards a method for finding a highly causally-efficacious region in every layer.

heading: 5.2 Controlling for expressivity
text: DAS is the the only method with a causal training objective. Other methods do not optimise for, or even have access to, downstream model behaviour. Wu et al. (2023) found that a variant of DAS achieves substantial causal effect even on a randomly-initialised model or with irrelavant nexttoken labels, both settings where no causal mechanism should exist. How much of the causal effect found by DAS is due to its expressivity? Research on probing has faced a similar concern: to what extent is a probe’s accuracy due to its expressivity rather than any aspect of the representation being studied? Hewitt and Liang (2019) propose comparing to accuracy on a control task that requires memorising an input-to-label mapping.
We adapt this notion to CausalGym, introducing control tasks where the next-token labels yb, ys are mapped to the arbitrary tokens ‘_dog’ and ‘_give’
1Further training details are given in appendix B, and we report hyperparameter tuning experiments on a dev set in appendix C.
while preserving the class partitioning.2 For example, on the gender-agreement task agr_gender, we replace the label ‘_he’ with ‘_dog’ and ‘_she’ with ‘_give’. We define selectivity for each method by taking the difference between odds-ratios on the original task and the control task for each f , and then compute the overall odds-ratio as in eq. (19).

heading: 5.3 Results
text: We summarise the results for each method in Table 1 by reporting overall odds-ratio and selectivity averaged over all tasks for each model. For a breakdown, see appendices E.1 and E.2.
We find that DAS consistently finds the most causally-efficacious features. The second-best method is probing, followed by difference-inmeans. The unsupervised methods PCA and kmeans are considerably worse. Despite supervision, LDA barely outperforms random features.
2The input-to-label mapping in CausalGym tasks is dependent on the input token types, so we cannot exactly replicate Hewitt and Liang. The setup we instead use is from Wu et al. (2023).
However, DAS is not considerably more selective or (at larger scales) even less selective than probing or diff-in-means; it can perform well on arbitrary input–output mappings. This suggests that its access to the model outputs during training is responsible for much of its advantage.

heading: 6 Case studies
text: In this section, we use CausalGym to study how LMs learn negative polarity item (NPI) licensing and wh-extraction from prepositional phrases over the course of training using checkpoints of pythia-1b. We first describe the tasks.
npi_any_subj-relc. NPIs are lexemes that can only occur in negative-polarity sentential contexts. In this task, we specifically check whether the NPI any is correctly licensed by a negated subject, giving minimal pairs like
(20) No athlete that loved the ministers has landed ⇒ any
(21) The athlete that loved the ministers has landed ⇒ some
In (21), where there is no negation at the sentence level, it would be ungrammatical to continue the sentence with the NPI any.
filler_gap_subj. Filler–gap dependencies in English occur when interrogatives are extracted out of and placed in front of a clause. The position from which they are extracted must remain empty. The task filler_gap_subj requires an LM to apply this rule when extracting from a distant prepositional phrase, e.g.
(22) My friend reported that the uncle forged the painting with the help of ⇒ him
(23) My friend reported who the uncle forged the painting with the help of ⇒ .
In (23), it would be ungrammatical for the preposition to have an explicit object since who was extracted from that position, leaving behind a gap.
Final mechanisms. We use the experimental setup of §5.1 and plot the average odds-ratio for each region and layer on the final checkpoint of pythia-1b in Figure 5. For both tasks, we find that the input feature crosses over several different positions before arriving at the output position. For example, in the NPI mechanism (Figure 5b), the negation feature is moved to the complementiser that in the early layer, into the auxiliary verb at middle layers, and the main verb in later layers, where its presence is used to predict the NPI any. The filler–gap mechanism is similarly complex.

heading: 6.1 Training dynamics
text: To study how the mechanisms emerge over the course of training, we run the exact same experiments on earlier checkpoints of pythia-1b.
npi_any_subj-relc. In Figure 4, the effect first emerges at the NPI (all but last layer) and the main verb (step 1000), then abruptly the auxiliary becomes important at middle layers and the NPI effect is pushed down to early layers (step 2000), and finally another intermediate locations is added at that (step 3000). The effect is also distributed across multiple regions in the intermediate layers.
filler_gap_subj. This behaviour takes longer to learn than NPI licensing (Figure 6). The mechanism emerges in two stages: at step 2000, it includes the filler position (that / who), the first determiner the, and the final token. After step 10K, the main verb is added to the mechanism.
Discussion. For both tasks, the model intially learns to move information directly from the alternating token to the output position. Later in training, intermediate steps are added in the middle layers. DAS finds a greater causal effect across the board, but both methods largely agree on which regions are the most causally efficacious at each layer. Notably, DAS finds causal effect at all timesteps, even when the model has just been initialised; this corroborates Wu et al.’s (2023) findings.

heading: 7 Conclusion
text: We introduced CausalGym, a multi-task benchmark of linguistic behaviours for measuring the causal
efficacy of interpretability methods. We showed the impressive performance of distributed alignment search, but also adapted a notion of control tasks to causal evaluation to enable fairer comparison of methods. Finally, we studied how causal effect propagates in training on two linguistic tasks: NPI licensing and filler–gap dependency tracking.
In recent years, much effort has been devoted towards developing causally-grounded methods for understanding neural networks. A probe achieving high classification accuracy provides no guarantee that the model actually distinguishes those classes in downstream computations; evaluating probe directions for causal effect is an intuitive test for whether they reflect features that the model uses downstream. Overall, while methods may come and go, we believe the causal evaluation paradigm will continue to be useful for the field.
A major motivation for releasing CausalGym is to encourage computational psycholinguists to move beyond studying the input–output behaviours of LMs. Our case studies in §6 are a basic example of the analysis that new methods permit. Ultimately, understanding how LMs learn linguistic behaviours may offer insights into fundamental properties of language (cf. Kallini et al., 2024; Wilcox et al., 2023b).
We hope that CausalGym will encourage comprehensive evaluation of new interpretability methods and spur adoption of the interventional paradigm in computational psycholinguistics.

heading: Limitations
text: While CausalGym includes a range of linguistic tasks, there are many non-linguistic behaviours on which we may want to use interpretability methods, and so we encourage future research on a greater variety of tasks. In addition, CausalGym includes only English data, and comparable experiments with other languages might yield substantially different results, thereby providing us with a much fuller picture of the causal mechanisms that LMs learn to use. Furthermore, results may differ on other models, since models in the pythia series were trained on the same data in a fixed order; different training data may result in different mechanisms. Finally, justified by the nature of our tasks, we only benchmark methods that operate on onedimensional linear subspaces; multi-dimensional linear methods as well as non-linear ones await being benchmarked.

heading: Ethics statement
text: Interpretability is a rapidly-advancing field, and our benchmark results render us optimistic about our ability to someday understand the mechanisms inside complex neural networks. However, successful interpretability methods could be used to justify deployment of language models in high-risk settings (e.g. to autonomously make decisions about human beings) or even manipulate models to produce harmful outputs. Understanding a model does not mean that it is safe to use in every situation, and we caution model deployers and users against uncritical trust in models even if they are found to be interpretable.

heading: Acknowledgements
text: We thank Atticus Geiger, Jing Huang, Harshit Joshi, Jordan Juravsky, Julie Kallini, Chenglei Si, Tristan Thrush, and Zhengxuan Wu for helpful discussion about the project and their comments on the manuscript.

heading: A Tasks
text: Task Example
Agreement (4) agr_gender [John/Jane] walked because [he/she] agr_sv_num_subj-relc The [guard/guards] that hated the manager [is/are] agr_sv_num_obj-relc The [guard/guards] that the customers hated [is/are] agr_sv_num_pp The [guard/guards] behind the managers [is/are]
Licensing (7) agr_refl_num_subj-relc The [farmer/farmers] that loved the actors embarrassed [himself/themselves] agr_refl_num_obj-relc The [farmer/farmers] that the actors loved embarrassed [himself/themselves] agr_refl_num_pp The [farmer/farmers] behind the actors embarrassed [himself/themselves] npi_any_subj-relc [No/The] consultant that has helped the taxi driver has shown [any/some] npi_any_obj-relc [No/The] consultant that the taxi driver has helped has shown [any/some] npi_ever_subj-relc [No/The] consultant that has helped the taxi driver has [ever/never] npi_ever_obj-relc [No/The] consultant that the taxi driver has helped has [ever/never]
Garden path effects (6) garden_mvrr The infant [who was/∅] brought the sandwich from the kitchen [by/.] garden_mvrr_mod The infant [who was/∅] brought the sandwich from the kitchen with a new microwave [by/.] garden_npz_obj While the students dressed [,/∅] the comedian [was/for] garden_npz_obj_mod While the students dressed [,/∅] the comedian who told bad jokes [was/for] garden_npz_v-trans As the criminal [slept/shot] the woman [was/for] garden_npz_v-trans_mod As the criminal [slept/shot] the woman who told bad jokes [was/for]
Gross syntactic state (4) gss_subord [While the/The] lawyers lost the plans [they/.] gss_subord_subj-relc [While the/The] lawyers who wore white lab jackets studied the book that described several advances in cancer therapy [,/.] gss_subord_obj-relc [While the/The] lawyers who the spy had contacted repeatedly studied the book that colleagues had written on cancer therapy [,/.] gss_subord_pp [While the/The] lawyers in a long white lab jacket studied the book about several recent
advances in cancer therapy [,/.]
Long-distance dependencies (8) cleft What the young man [did/ate] was [make/for] cleft_mod What the young man [did/ate] after the ingredients had been bought from the store was [make/for] filler_gap_embed_3 I know [that/what] the mother said the friend remarked the park attendant reported your friend sent [him/.] filler_gap_embed_4 I know [that/what] the mother said the friend remarked the park attendant reported the cop thinks your friend sent [him/.] filler_gap_hierarchy The fact that the brother said [that/who] the friend trusted [the/was] filler_gap_obj I know [that/what] the uncle grabbed [him/.] filler_gap_pp I know [that/what] the uncle grabbed food in front of [him/.] filler_gap_subj I know [that/who] the uncle grabbed food in front of [him/.]

heading: B Training and evaluation details
text: We load models using the HuggingFace transformers (Wolf et al., 2020) library. Up to size 410m we load weights in float32 precision, 1b in bfloat16 precision, and larger models in float16 precision. Our training set starts with 200 examples sampled according to the scheme in §3.2. We then double the size of the set (400) by swapping the base and source inputs/labels and adding these to the training set; including both directions of the intervention makes the comparison fairer between DAS and the other non-paired methods, and also ensures a perfect balance between labels.
The evaluation set consists of 50 examples sampled the same way (effectively 100), except we resample in case we encounter a sentence already present in the training set. Thus, there is no overlap with the training set. We evaluate all metrics (odds-ratio and probe classification accuracy) on this set.
We train DAS for one epoch with a batch size of 4, resulting in 100 backpropagation steps. We use the Adam optimiser (Kingma and Ba, 2015) and a linear learning rate schedule, with the first 10% of training being a warmup from 0 to the learning rate, followed by the learning rate linearly decaying to 0 for the rest of training. The scheduling and optimiser is identical to Wu et al. (2023). We use a learning
rate of 5 · 10−3, which is higher than previous work (usually 10−3) due to the small training set size; see appendix C for hyperparameter tuning experiments which justify this choice.
To run our experiments, we used a cluster of NVIDIA A100 (40 GB) and NVIDIA RTX 6000 Ada Generation GPUs. The total runtime for the benchmarking experiments in §5 was ∼ 400 hours, and for the case studies in §6 it was ∼ 25 hours.

heading: C Hyperparameter tuning
text: To ensure fair comparison, we tuned hyperparameters for DAS, probes, and PCA on a dev set, sampled the same way as the eval set (non-overlapping with train set) but with a different random seed. We train on all tasks in CausalGym and report the average odds-ratio following the same evaluation setup as in §5.1. We studied only the three smallest models (pythia-14m, 31m, 70m) due to the large number of experiments needed. Specifically, we tune the learning rate for DAS, the type of regularisation and whether or not to include a bias term in the logit for probes,3 and averaging of the first c components for PCA. We report the overall log odds-ratio for various hyperparameter settings in Table 2. These experiments were run on a NVIDIA RTX 6000 Ada Generation. The total runtime was ∼ 25 hours.
For probing (Table 2a), we found that including a bias term and using only L2 regularisation with the saga solver delivers the best performance. However, the setting of the weight coefficient λ on the regularisation term in the loss depends on the model. The main architectural difference between these three models is the hidden dimension size, so we suspect that the optimal choice for λ depends on that. Roughly extrapolating the observed trend, in our main experiments we check λ = {104, 105} for pythia-160m and 410m, λ = {105, 106} for pythia-1b, 1.4b, and 2.8b, and λ = {106, 107} for pythia-6.9b. As for why L2 regularisation increases causal efficacy, we note that Hewitt and Liang (2019) found that it also increases probe selectivity—we leave this as an open question for future work.
For PCA (Table 2b), we found that averaging the first c components did not improve performance over just using the first component; thus, we used just the first PCA component in our main-text experiments.
For DAS (Table 2c), we found that using the learning rate suggested by Wu et al. (2023), 10−3, understated performance and a higher learning rate did not result in any apparent training instability. However, our experimental setup is quite different (smaller training set, no learned boundary, greater variety of model scales). We did not find any consistent differences or trends with model scale between learning rates of 5 · 10−3 and 10−2, so we used the former for all experiments.

heading: D Data and licensing
text: We use the original test suites from SyntaxGym which were described in Hu et al. (2020). These were released under the MIT License, and our data release will also use the MIT license for compatibility.
3Cf. Tigges et al. (2023), who did not include a bias term in their causal evaluation of probing.

heading: E Detailed odds-ratio results
text: In these comprehensive results, we include an additional method: vanilla interchange intervention. Instead of as in eq. (11), vanilla intervention defines f∗ as
f∗vanilla(b, s) = f(s) (24)
i.e. it entirely replaces the activation with that of the source input. This is equivalent to n-dimensional DII where f(s) ∈ Rn, and is a significantly more expressive intervention than any methods we tested. We include it as a non-learned baseline.

heading: E.1 Per-layer

heading: E.2 Per-task
text: Rows in gray indicate tasks where the model achieves < 60% accuracy.

heading: Task Task Acc. Feature-finding methods Vanilla

heading: Task Task Acc. Feature-finding methods Vanilla

heading: Task Task Acc. Feature-finding methods Vanilla

heading: Task Task Acc. Feature-finding methods Vanilla

heading: Task Task Acc. Feature-finding methods Vanilla

heading: Task Task Acc. Feature-finding methods Vanilla

heading: Task Task Acc. Feature-finding methods Vanilla

heading: Task Task Acc. Feature-finding methods Vanilla

heading: Task Task Acc. Feature-finding methods Vanilla

heading: Task Task Acc. Feature-finding methods Vanilla

heading: Task Task Acc. Feature-finding methods Vanilla

heading: Task Task Acc. Feature-finding methods Vanilla

heading: Task Task Acc. Feature-finding methods Vanilla

heading: Task Task Acc. Feature-finding methods Vanilla

heading: Task Task Acc. Feature-finding methods Vanilla

heading: Task Task Acc. Feature-finding methods Vanilla

heading: Task Task Acc. Feature-finding methods Vanilla

heading: Task Task Acc. Feature-finding methods Vanilla

heading: F Odds-ratio plots for all methods on selected tasks

heading: Future Work
text: As for why L2 regularisation increases causal efficacy, we note that Hewitt and Liang (2019) found that it also increases probe selectivity—we leave this as an open question for future work. For PCA (Table 2b), we found that averaging the first c components did not improve performance over just using the first component; thus, we used just the first PCA component in our main-text experiments. For DAS (Table 2c), we found that using the learning rate suggested by Wu et al. (2023), 10−3, understated performance and a higher learning rate did not result in any apparent training instability. However, our experimental setup is quite different (smaller training set, no learned boundary, greater variety of model scales). We did not find any consistent differences or trends with model scale between learning rates of 5 · 10−3 and 10−2, so we used the former for all experiments