Title: Interpretability at Scale: Identifying Causal Mechanisms in Alpaca
Venue: NeurIPS 2023
Accepted: yes
Keywords: Mechanistic Interpretability
Abstract: Obtaining human-interpretable explanations of large, general-purpose language models is an urgent goal for AI safety. However, it is just as important that our interpretability methods are faithful to the causal dynamics underlying model behavior and able to robustly generalize to unseen inputs. Distributed Alignment Search (DAS) is a powerful gradient descent method grounded in a theory of causal abstraction that uncovered perfect alignments between interpretable symbolic algorithms and small deep learning models fine-tuned for specific tasks. In the present paper, we scale DAS significantly by replacing the remaining brute-force search steps with learned parameters -- an approach we call Boundless DAS. This enables us to efficiently search for interpretable causal structure in large language models while they follow instructions. We apply Boundless DAS to the Alpaca model (7B parameters), which, off the shelf, solves a simple numerical reasoning problem. With Boundless DAS, we discover that Alpaca does this by implementing a causal model with two interpretable boolean variables. Furthermore, we find that the alignment of neural representations with these variables is robust to changes in inputs and instructions. These findings mark a first step toward deeply understanding the inner-workings of our largest and most widely deployed language models.

Interpretability at Scale: Identifying Causal Mechanisms in Alpaca

Abstract

Obtaining human-interpretable explanations of large, general-purpose language models is an urgent goal for AI safety. However, it is just as important that our interpretability methods are faithful to the causal dynamics underlying model behavior and able to robustly generalize to unseen inputs. Distributed Alignment Search (DAS) [23] is a powerful gradient descent method grounded in a theory of causal abstraction that has uncovered perfect alignments between interpretable symbolic algorithms and small deep learning models fine-tuned for specific tasks. In the present paper, we scale DAS significantly by replacing the remaining brute-force search steps with learned parameters - an approach we call Boundless DAS. This enables us to efficiently search for interpretable causal structure in large language models while they follow instructions. We apply Boundless DAS to the Alpaca model (7B parameters), which, off the shelf, solves a simple numerical reasoning problem. With Boundless DAS, we discover that Alpaca does this by implementing a causal model with two interpretable boolean variables. Furthermore, we find that the alignment of neural representations with these variables is robust to changes in inputs and instructions. These findings mark a first step toward faithfully understanding the inner-workings of our ever-growing and most widely deployed language models. Our tool is extensible to larger LLMs and is released publicly at https://github.com/frankaging/align-transformers .

1 Introduction

Present-day large language models (LLMs) display remarkable behaviors: they appear to solve coding tasks, translate between languages, engage in open-ended dialogue, and much more. As a result, their societal impact is rapidly growing, as they make their way into products, services, and people's own daily tasks. In this context, it is vital that we move beyond behavioral evaluation to deeply explain, in human-interpretable terms, the internal processes of these models, as an initial step in auditing them for safety, trustworthiness, and pernicious social biases.

The theory of causal abstraction [5,22] provides a generic framework for representing interpretability methods that faithfully assess the degree to which a complex causal system (e.g., a neural network) implements an interpretable causal system (e.g., a symbolic algorithm). Where the answer is positive, we move closer to having guarantees about how the model will behave. However, thus far, such interpretability methods have been applied only to small models fine-tuned for specific tasks [19,20,28,7] and this is arguably not an accident: the space of alignments between the variables in the hypothesized causal model and the representations in the neural network becomes exponentially

larger as models increase in size. When a good alignment is found, one has specific formal guarantees. Where no alignment is found, it could easily be a failure of the alignment search algorithm.

Distributed Alignment Search (DAS) [23] marks real progress on this problem. DAS opens the door to (1) discovering structure spread across neurons and (2) using gradient descent to learn an alignment between distributed neural representations and causal variables. However, DAS still requires a brute-force search over the dimensionality of neural representations, hindering its use at scale.

In this paper, we introduce Boundless DAS, which replaces the remaining brute-force aspect of DAS with learned parameters, truly enabling interpretability at scale. We use Boundless DAS to study how Alpaca (7B) [47], an off-the-shelf instruct-tuned LLaMA model, follows basic instructions in a simple numerical reasoning task. Figure 1 summarizes the approach. We find that Alpaca achieves near-perfect task performance because it implements a simple algorithm with interpretable variables. In further experiments, we show that Alpaca uses this simple algorithm across a wide range of contexts and variations on the task. These findings mark a first step toward faithfully understanding the inner-workings of our largest and most widely deployed language models.

2 Related Work

Interpretability Many methods have been developed in an attempt to explain and understand deep learning models. These methods include analyzing learned weights [12,1,13], gradient-based methods [45,44,60,46,52,6], probing [14,48,24,42,12,32,8,40], syntax-driven interventions [34], self-generated model explanations [27], and training external explainers based on model behaviors [39, 31]. However, these methods rely on observational measurements of behavior and internal neural representations. Such explanations are often not guaranteed to be faithful to the underlying causal mechanisms of the target models [29,20,50,53].

Causal Abstraction The theory of causal abstraction [41,4,5] offers a unifying mathematical framework for interpretability methods aiming to uncover interpretable causal mechanisms in deep learning models [19,28,20,58,54,23] and training methods for inducing such interpretable mechanisms [21,59,58,25]. Causal abstraction [22] can represent many existing interpretablity methods, including interative nullspace projection [38,15,30], causal mediation analysis [51,33], and causal effect estimation [18,16,2]. In particular, causal abstraction grounds the research program of mechanistic interpretability , which aims to reverse engineer deep learning models by determining the algorithm or computation underlying their intelligent behavior [35,17,36,7,54]. To the best of our knowledge, there is no prior work that scales these methods to large, general-purpose LLMs.

Training LLMs to Follow Instructions Instruction-based fine-tuning of LLMs can greatly enhance their capacity to follow natural language instructions [10,37]. In parallel, this ability can also be induced into the model by fine-tuning base models with hundreds of specific tasks [56,11]. Recently,

Wang et al. [55] show that the process of creating fine-tuning data for instruction following models can partly be done by the target LLM itself ('self-instruct'). Such datasets have led to many recent successes in lightweight fine-tuning of ChatGPT-like instruction following models such as Alpaca [47], instruct-tuned LLaMA [49], StableLM [3], Vicuna [9]. Our goal is to scale methods from causal abstraction to understand how these models follow a particular instruction.

3 Methods

3.1 Background on Causal Abstraction

Causal Models We represent black box networks and interpretable algorithms using causal models that consist of variables taking on values according to causal mechanisms . We distinguish a set of input variables with possible values Inputs . An intervention is an operation that edits some causal mechanisms in a model. We denote the output of a causal model M provided an input x as M ( x ) .

Interchange Intervention Begin with a model M (e.g., causal model or neural model) and a base input b and source inputs { s j } k 1 , all elements of Inputs , together with disjoint sets of target variables { Z j } k 1 we are aligning. An interchange intervention yields a new model INTINV ( M , { s j } k 1 , { Z j } k 1 ) which is identical to M except the values for each set of target variables Z j are fixed to be the value they would have taken for source input s j . We then denote the intervened output (i.e., counterfactual output) for the base input with INTINV ( M , { s j } k 1 , { Z j } k 1 )( b ) .

Distributed Interchange Intervention Let N be a subset of variables in M , the target variables . Let Y be a vector space with orthogonal subspaces { Y j } k 1 . Let R be an invertible function R ∶ N → Y . A distributed interchange intervention yields a new model DII ( M , R , { s j } k 1 , { Y j } k 1 ) which is identical to M except the causal mechanisms have been rewritten such that each subspace Y j is fixed to be the value it would take for source input s j . 2 Specifically, the causal mechanism for N is,

where b is the base input, Y 0 = Y \ ⨁ k j = 1 Y j , and Proj Y 0 or Proj Y j represents the orthogonal projection operator of the original rotated vector into the Y 0 or Y j subspace. F N ( ⋅ ) represents the causal mechanisms of causal variables N before any intervention, whereas F ∗ N ( ⋅ ) represents the mechanisms after the distributed intervention. We then denote the intervened output (i.e., counterfactual output) for the base input as DII ( M , R , { s j } k 1 , { Y j } k 0 )( b ) .

Causal Abstraction and Alignment We are licensed to claim a causal model (i.e., an algorithm) A is a faithful interpretation of the neural network N if the causal mechanisms of the variables in the algorithm abstract the causal mechanisms of neural representations relative to a particular alignment . For our purposes, each variable of a high-level model is aligned with a linear subspace in the vector space formed by rotating a neural representation with an orthogonal matrix. We use τ to represent the alignment mapping between a high-level causal variable and a neural representation.

Approximate Causal Abstraction Interchange intervention accuracy (IIA) is a graded measure of abstraction that computes the proportion of aligned interchange interventions on the algorithm and neural network that have the same output. The IIA for an alignment of high-level variables Z j to orthogonal subspace Y j between an algorithm A and neural network N is

where τ translates from low-level causal variable values to high-level neural representation values.

3.2 Boundless Distributed Alignment Search

Distributed alignment search (DAS) is a method for learning an alignment between interpretable causal variables of a model C and fixed dimensionality linear subspaces of neural representations in a network N using gradient descent [23] as shown in Figure 3. Specifically, an orthogonal matrix R ∶ N → Y is trained to maximize interchange intervention accuracy under an alignment from each variable Z j to fixed dimensionality linear subspaces Y j of the rotated vector space.

Boundless DAS is our extension of DAS that learns the dimensionality of the orthogonal linear subspaces in a d -dimensional vector space Y using a method inspired by work in neural PDE [57]. Specifically, for each high-level variable Z j , we introduce a learnable continuous boundary index parameter b j that can take on a value between 0 and d , where b 0 = 0 , b j < b j + 1 , and b j < d for all j . The boundary mask M j for the source is a vector with d values between 0 and 1 where the k -th element of the array is defined to be

where β is a temperature that we anneal through training. As β approaches 0, the masks M j converge to binary-valued vectors that together encode an orthogonal decomposition of Y .

Weighted Distributed Interchange Intervention Let N be a subset of variables in M , the target variables . Let Y be a vector space with d dimensions and let { M j } k 1 vectors in [ 0 , 1 ] d . Let R ∶ N → Y be an invertible transformation. A weighted distributed interchange intervention yields a new model SOFTDII ( M , R , { s j } k 1 , { M j } k 1 ) which is identical to M except the causal mechanisms have been rewritten such that each source input s j contributes to the setting of Y in proportion to its mask M j . Specifically, the causal mechanism for N is set to be

where ◦ is element wise multiplication and 1 is a d dimensional vector where each element is 1 . We then denote the output for the base input as SOFTDII ( M , R , { s j } k 1 , { M j } k 1 )( b ) .

Boundless DAS Given a base input b and source inputs { s j } k 1 , we minimize the following objective to learn a rotation matrix R θ and masks { M θ j } k 1

where CE is the cross entropy loss. We anneal β throughout training and our weighted interchange interventions become more and more similar to unweighted interchange interventions. During evaluation we snap the masks to be binary-valued to create an orthogonal decomposition of Y where each high-level variable Z j is aligned with a linear subspace of Y picked out by the mask M j , with the residual (unaligned) subspace being picked out by the mask ( 1 -∑ k j = 1 M j ) .

Time Complexity Analysis DAS learns a rotation matrix but requires a manual search to determine how many neurons are needed to represent the aligning causal variable. Boundless DAS automatically learns boundaries (i.e., how many neurons are needed is determined by the 'soft' boundary index via a boundary mask learning). For instance, given a representation with a dimension of 1024, DAS should in principle be run for all lengths k from 1 to 1024. In practice, this would be infeasible, so some subset of the lengths need to be chosen heuristically, which risks missing genuine structure. For Boundless DAS, we turn this search process into a mask learning process. DAS [23] is O ( n × m ) where n is the number of total dimensions of the neural representation we are aligning and m is the number of causal variables, while Boundless DAS is O ( m ) .

4 Experiment

4.1 Price Tagging

We follow the approach in Figure 1 by first assessing the ability of Alpaca to execute specific actions based on the instructions provided in the input. Formally, the input to the model M is given an

instruction t i (e.g., 'correct the spelling of the word:') followed by a test query input x i (e.g., 'aplpe'). We use M ( t i , x i ) to depict the model generation y p given the instruction and the test query input. We can evaluate model performance by comparing y p with the gold label y . We focus on tasks with high model performance, to ensure that we have a known behavioral pattern to explain.

The instruction prompt of the Price Tagging game follows the publicly released template of the Alpaca (7B) model. The core instruction contains an English sentence:

Please say yes only if it costs between [X.XX] and [X.XX] dollars, otherwise no.

followed by an input dollar amount [X.XX] , where [X.XX] are random continuous real numbers drawn with a uniform distribution from [0.00, 9.99]. The output is a single token 'Yes' or 'No'.

For instance, if the core instruction says Please say yes only if it costs between [1.30] and [8.55] dollars, otherwise no. , the answer would be 'Yes' if the input amount is '3.50 dollars' and 'No' if the input is '9.50 dollars'. We restrict the absolute difference between the lower bound and the upper bound to be [2.50, 7.50] due to model errors outside these values - again we need behavior to explain.

4.2 Hypothesized Causal Models

As shown in Figure 2, we have identified a set of human-interpretable high-level causal models, with alignable intermediate causal variables, that would solve this task with 100% performance:

Left Boundary : This model has one high-level boolean variable representing whether the input amount is higher than the lower bound, and an output node incorporating whether the input amount is also lower than the high bound.

Left and Right Boundary : The previous model is sub-optimal in only abstracting one of the boundaries. In this model, we have two high-level boolean variables representing whether the input amount is higher than the lower bound and lower than the higher bound, respectively. We take a conjunction of these boolean variables to predict the output.

Mid-point Distance : We calculate the mid-point of the lower and upper bounds (e.g., the mid-point of '3.50' and '8.50' is '6.00), and then we take the absolute distance between the input dollar amount and the mid-point as a . We then calculate one-half of the bounding bracket length (e.g., the bracket length for '3.50' and '8.50' is '5.00') as b . We predict output 'Yes' if a ≤ b , otherwise 'No'. We align only with the mid-point variable.

Bracket Identity : This model represents the lower and upper bound in a single interval variable and passes this information to the output node. We predict the output as 'Yes' if the input amount within the interval, otherwise 'No'.

Model Architecture Our target model is the Alpaca (7B) [47], an off-the-shelf instruct-tuned LLaMA model. It is a Transformer-based decoder-only autoregressive trained language model with 32 layers and 32 attention heads. It has a hidden dimension in size of 4096, which is also the dimension of our rotation matrix which is applied for each token representation. In total, the rotation matrix contains 16.8M parameters and has size 4096 × 4096.

Alignment Process We train Boundless DAS with test query token representations (i.e., starting from the token of the first input digit till the last token in the prompt) in a selected set of 7 layers: { 0 , 5 , 10 , 15 , 20 , 25 , 30 } . We also train Boundless DAS on the token before the first digit as a control condition where nothing should be expected to be aligned. We interchange with a single source, while allowing multiple causal variables to be aligned across examples. For simplicity, we enforce the interval between two boundary indices to be the same (i.e., the same size of linear subspace for each aligning causal variable). We run each experiment with three distinct random seeds. Since the global optimum of the Boundless DAS objective corresponds to the best attainable alignment, but SGD may be trapped by local optima, we report the best-performing seed in terms of IIA. Figure 3 provides an overview of these analyses.

Evaluation Metric To evaluate models, we use Interchange Intervention Accuracy (IIA) as defined in Eqn. 2. IIA is bounded between 0.0 and 1.0. We measure the baseline IIA by replacing the learned rotation matrix with a random one. The lower bound of IIA for 'Left Boundary' and 'Left and Right Boundary' is about 0.50, and for 'Mid-point Distance' or 'Bracket Identity' it is about 0.60. The latter two baseline IIAs are higher than chance because they are conditioned on the distribution of our output labels (i.e., how many times the original label gets to be flipped due to the intervention). See Section 4.7 for more discussion of metric calibration. Additionally, IIA can occasionally go above the model's task performance, when the interchange interventions put the model in a better state, but IIA is constrained by task accuracy for the most part.

4.3 Boundless DAS Results

Figure 4 shows our main results, given in terms of IIA across our four hypothesized causal models (Figure 2). The results show very clearly that 'Left Boundary' and 'Left and Right Right Boundary' (top panels) are highly accurate hypotheses about how Alpaca solves the task. For them, IIA is at or above task performance (0.85), and intermediate variable representations are localized in systematically arranged positions. By contrast, 'Mid-point Distance' and 'Bracket Identity' (bottom panels) are inaccurate hypotheses about Alpaca's processing, with IIA peaking at around 0.72.

These findings suggest that, when solving our reasoning task, Alpaca internally follows our first two high-level models by representing causal variables that align with boundary checks for both left and right boundaries. Interestingly, heatmaps on the top two rows also show a pattern of higher scores around the bottom left and upper right and close to zero scores in other positions. This is significantly different from the other two alignments where, although some positions are highlighted (e.g., the representations for the last token), all the positions receive non-zero scores. In short, the accurate hypotheses correspond to highly structured IIA patterns, and the inaccurate ones do not.

Additionally, alignments are better when the model post-processes the query input with 1-2 additional steps: accuracy is higher at position 75 compared to all previous positions. Surprisingly, there exist bridging tokens (positions 76-79) where accuracy suddenly drops compared to earlier tokens, which suggests that these representations have weaker causal effects on the model output. In other words, boundary-check variables are fully extracted around position 75, level 10, and are later copied into activations for the final token before responding. By comparing the heatmaps on the top row, our findings suggest that aligning multiple variables at the same time poses a harder alignment process, in that it lowers scores slightly across multiple positions.

4.4 Interchange Interventions with (In-)Correct Inputs

IIA is highly constrained by task performance, and thus we expect it to be much lower for inputs that the model gets wrong. To verify this, we constructed an evaluation set containing 1K examples that the model gets wrong and evaluated our 'Left and Right Boundary' hypothesis on this subset. Table 1 reports these results in terms of max IIA and the correlation of the IIA values with those obtained in our main experiments. As expected, IIA drops significantly. However, two things stand out: IIA is far above task performance (which is 0.0 by design now), and the correlation with the original IIA map remains high. These findings suggest that the model is using the same internal mechanisms to process these cases, and so it seems possible the model is narrowly missing the correct output predictions.

We also expect IIA to be higher if we focus only on cases that the model gets correct. Table 1 confirms this expectation using 'Left and Right Boundary' as representative. IIAmax is improved from 0.86 to 0.88, and the correlation with the main results is essentially perfect. Full heatmaps for these analyses are given in Appendix A.4.

Experiment

Task Acc.

IIAmax

Correlation

Left Boundary (

♣

)

0.85

0.90

1.00

Left and Right Boundary (

♥

)

0.85

0.86

1.00

Mid-point Distance

0.85

0.70

1.00

Bracket Identity

0.85

0.72

1.00

Correct Only

1.00

†

0.88

0.99 (

♥

)

Incorrect Only

0.00

†

0.71

0.84 (

♥

)

New Bracket (Seen)

0.94

0.94

0.97 (

♣

)

New Bracket (Unseen)

0.95

0.95

0.94 (

♣

)

Irrelevant Contexts

0.84

0.83

0.99 (

♥

)

Sibling Instructions

0.84

0.83

0.87 (

♥

)

+

exclude

top right

0.84

0.83

0.92 (

♥

)

4.5 Do Alignments Robustly Generalize to Unseen Instructions and Inputs?

One might worry that our positive results are highly dependent on the specific input-output pairs we have chosen. We now seek to address this concern by asking whether the causal roles (i.e., alignments) found using Boundless DAS in one setting are preserved in new settings. This is crucial, as it tells how robustly the causal model is realized in the neural network.

Generalizing Across Two Different Instructions Here, we assess whether the learned alignments for the 'Left Boundary' causal model transfer between different specific price brackets in the instruction. To do this, we retrain Boundless DAS for the high-level model with a fixed instruction that says 'between 5.49 dollars and 8.49 dollars'. Then, we fix the learned rotation matrix and evaluate with another instruction that says 'between 2.51 dollars and 5.51 dollars'. For both, Alpaca is successful at the task, with around 94% accuracy. Our hypothesis is that if the found alignment of the highlevel variable is robust, it should transfer between these two settings, as the aligning variable is a boolean-type variable which is potentially agnostic to the specific comparison price.

Table 1 gives our findings in the 'New Bracket' rows. Boundless DAS is able to find a good alignment for the training bracket with an IIAmax that is about the same as the task performance at 94%. For our unseen bracket, the alignments also hold up extremely well, with no drop in IIAmax. For both cases, the found alignments also highly correlate with the counterpart of our main experiment.

Generalizing with Inserted Context Recent work has shown that language models are sensitive to irrelevant context [43], which suggests that found alignments may overfit to a set of fixed contexts.

To address this concern, we add prefix strings to the input instructions and evaluate how 'Left and Right Boundary' alignment transfers. We thus generate 20 random prefixes using GPT-4 . 3 All of our prefixes, and our method for generating them, can be found in Appendix A.6. The model achieves 84% task performance with random contexts added as prefixes, which is slightly lower than the average task performance. Nonetheless, the results in Table 1 suggest the found alignments transfer surprisingly well here, with only 2% drop in IIAmax. Meanwhile, the mean IIA distribution across positions and layers is highly correlated with our base experiment. Thus, our method seems to identify causal structure that is robust to changes in irrelevant task details and position in the input string.

Generalizing to Modified Outputs We further test whether the alignments found for instructions with a template 'Say yes . . . , otherwise no' can generalize to a new instruction with a template 'Say True . . . , otherwise False'. If there are indeed latent representations of our aligning causal variables, the found alignments should persist and should be agnostic to the output format.

Table 1 shows that this holds: the learned alignments actually transfer across these two sibling instructions, with a minimum 1% drop in IIAmax. In addition, the correlation increases 6% when excluding the top right corner (top 3 layers in the last position), as seen in the final row of the table. This result is further substantiated in Figure 5, where we see that IIA drops near the top right. This indicates a late fusion of output classes and working representations, leading to different computations for the new instruction only close to the output.

4.6 Boundary Learning Dynamics

Boundless DAS automatically learns the intervention site boundaries (Section 3.2). It is important to confirm that we do not give up alignment accuracy by optimizing boundaries, and to explore the dimensionality actually needed to represent the abstract variables. We sample 100 experiment runs from our experiment pool and create two groups: (1) the success group where the boundary does not shrink to 0 at the end of the training; (2) the failure group where the boundary does shrink to 0 at the end of the training. The failure group is for those instances where there is no significant causal role of representations found by our method. We plot how boundary width and in-training evaluation IIA vary throughout training. Figure 6 shows that, in the success cases, each aligned variable only needs 5-10% of the representation space. IIA performance converges quickly for success cases and, crucially, maintains stable performance despite shrinking dimensionality. This suggests that only a small fraction of the whole representation space is needed for aligning a causal variable, and that these representations can be efficiently identified by Boundless DAS.

4.7 Metric Calibration

One concern brought up by our reviewers is whether our metric, interchange intervention accuracy (IIA), is well calibrated. Here, we provide additional evidence. We report IIAs in the same way as in Figure 4 but with random rotation matrices. With a random rotation matrix, we find that the IIA score drops from 0.83 to 0.53 at layer 10, token position 75, for our 'left and right boundary' causal model. Other positions drop significantly as well. These results calibrate IIAs in case of unbalanced labels (e.g., our two control causal models reach about 0.60 IIA at the same location with a random rotation matrix). This shows that Boundless DAS finds substantial structure in Alpaca as compared to what these random baselines can find. Details can be found in Figure 12 in the Appendix.

We can also help calibrate our result by comparing Boundless DAS applied to Alpaca with a randomly initialized LLaMA-7B. This can help quantify the extent to which DAS leverage random causal structure in order to overfit to its training objective. This random model results in near 0% task performance on our task, as expected. After running Boundless DAS on the first layer of this random LLaMA-7B model, the found alignments to each token's representation ranged from 0% to 69% IIA, which is comparable to a most frequent label dummy model (66%). For the representations with near chance IIA, this means that we were able to find a distributed neural representation that shifts the probability mass to 'yes' or 'no', but does not differentiate between the two. We also found that we could find a distributed neural representation that shifts the probability mass to the tokens 'dog' and 'give' instead of 'yes' and 'no'. More importantly, we found that IIA drops to 0% for this run on all of our robustness checks from the paper. In addition, it drops to 0% if we use it on another randomly initialized LLaMA-7B model with a different random seed. Overall, it seems that the causal structure we did find has arisen by chance, and we might expect very large models to be more prone to such occurrences. Our robustness checks definitively clarify the situation.

5 Analytic Strengths and Limitations

Explanation methods for AI should be judged by the degree to which they can, in principle, identify models' true internal causal mechanisms. If we reach 100% IIA for a high-level model using Boundless DAS we can assert that we have positively identified a correct causal structure in the model (though perhaps not the only correct abstraction). This follows directly from the formal results of Geiger et al. [22]. On the other hand, failure to find causal structure with Boundless DAS is not a proof that such structure is missing, for two reasons. First, Boundless DAS explores a very flexible and large hypothesis space, but it is not completely exhaustive. For instance, a set of variables represented in a highly non-linear way might be missed. Second, we are limited by the space of causal models we think to test ourselves.

For models where task accuracy is below 100%, it is still possible for IIA to reach 100% if the highlevel causal model explains errors of the low-level model. However, in cases like those studied here, where our high-level model matches the idealized task but our language model does not completely do so, we expect Boundless DAS to find only partial support for causal structure even if we have identified the ideal set of hypotheses and searched optimally. This too follows from the results of Geiger et al. [22]. However, as we have seen in the results above, Boundless DAS can find structure even where the model's task performance is low, since task performance can be shaped by factors like a suboptimal generation method or the rigidity of the assessment metric. Future work must tighten this connection by modeling errors in the language model in more detail.

6 Conclusion

We introduce Boundless DAS, a novel and effective method for scaling causal analysis of LLMs to billions of parameters. Using Boundless DAS, we find that Alpaca, off-the-shelf, solves a simple numerical reasoning problem in a human-interpretable way. Additionally, we address one of the main concerns around interpretability tools developed for LLMs - whether found alignments generalize across different settings. We rigorously study this by evaluating found alignments under several changes to inputs and instructions. Our findings indicate robust and interpretable algorithmic structure. Our framework is generic for any LLMs and is released to the public. We hope that this marks a step forward in terms of understanding the internal causal mechanisms behind the massive LLMs that are at the center of so much work in AI.

References

Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 4190-4197, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main. 385. URL https://aclanthology.org/2020.acl-main.385 .

Eldar D Abraham, Karel D'Oosterlinck, Amir Feder, Yair Gat, Atticus Geiger, Christopher Potts, Roi Reichart, and Zhengxuan Wu. Cebab: Estimating the causal effects of real-world concepts on nlp model behavior. Advances in Neural Information Processing Systems , 35: 17582-17596, 2022.

Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Shivanshu Purohit, Tri Songz, Wang Phil, and Samuel Weinbach. GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch, 8 2021. URL https://www.github.com/ eleutherai/gpt-neox .

Sander Beckers and Joseph Halpern. Abstracting causal models. In AAAI Conference on Artificial Intelligence , 2019.

Sander Beckers, Frederick Eberhardt, and Joseph Y. Halpern. Approximate causal abstractions. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference , 2019.

Yonatan Belinkov and James R. Glass. Analysis methods in neural language processing: A survey. Trans. Assoc. Comput. Linguistics , 7:49-72, 2019. doi: 10.1162/tacl\_a\_00254. URL https://doi.org/10.1162/tacl_a_00254 .

Lawrence Chan, Adrià Garriga-Alonso, Nicholas Goldowsky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. Causal scrubbing: a method for rigorously testing interpretability hypotheses, 2022.

Ethan A. Chi, John Hewitt, and Christopher D. Manning. Finding universal grammatical relations in multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 5564-5577, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.493. URL https: //www.aclweb.org/anthology/2020.acl-main.493 .

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://vicuna.lmsys.org .

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 , 2022.

Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pages 276-286, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-4828. URL https://aclanthology.org/W19-4828 .

Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, and Martin Wattenberg. Visualizing and measuring the geometry of bert, 2019.

Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2126-2136, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1198. URL https://www. aclweb.org/anthology/P18-1198 .

Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. In Proceedings of the 2020 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP . Association for Computational Linguistics, November 2020. doi: 10.18653/v1/W18-5426.

Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schütze, and Yoav Goldberg. Measuring causal effects of data statistics on language model's 'factual' predictions. CoRR , abs/2207.14251, 2022. doi: 10.48550/arXiv.2207.14251. URL https://doi.org/10.48550/arXiv.2207.14251 .

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread , 2021. https://transformer-circuits.pub/2021/framework/index.html.

Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. CausaLM: Causal Model Explanation Through Counterfactual Language Models. Computational Linguistics , pages 1-54, 05 2021. ISSN 0891-2017. doi: 10.1162/coli_a_00404. URL https://doi.org/10.1162/coli_a_ 00404 .

Atticus Geiger, Kyle Richardson, and Christopher Potts. Neural natural language inference models partially embed theories of lexical entailment and negation. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP , pages 163173, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.blackboxnlp-1.16. URL https://www.aclweb.org/anthology/2020.blackboxnlp-1. 16 .

Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. In Advances in Neural Information Processing Systems , volume 34, pages 9574-9586, 2021. URL https://papers.nips.cc/paper/2021/hash/ 4f5c422f4d49a5a807eda27434231040-Abstract.html .

Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Goodman, and Christopher Potts. Inducing causal structure for interpretable neural networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 7324-7338. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/geiger22a.html .

Atticus Geiger, Christopher Potts, and Thomas Icard. Causal abstraction for faithful model interpretation. Ms., Stanford University, 2023. URL https://arxiv.org/abs/2301.04709 .

Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D. Goodman. Finding alignments between interpretable causal variables and distributed neural representations. Ms., Stanford University, 2023. URL https://arxiv.org/abs/2303.02536 .

John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4129-4138, 2019.

Jing Huang, Zhengxuan Wu, Kyle Mahowald, and Christopher Potts. Inducing character-level structure in subword-based language models with Type-level Interchange Intervention Training. Ms., Stanford University and UT Austin, 2022. URL https://arxiv.org/abs/2212.09897 .

Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) , 2015.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2022.

Belinda Z. Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 1813-1827, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.143. URL https://aclanthology.org/2021.acl-long.143 .

Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue , 16(3):31-57, 2018.

Charles Lovering and Ellie Pavlick. Unit testing for concepts in neural networks. Transactions of the Association for Computational Linguistics , 10:1193-1208, 2022. doi: 10.1162/tacl_a_00514. URL https://aclanthology.org/2022.tacl-1.69 .

Scott Lundberg and Su-In Lee. A unified approach to interpreting model predictions, 2017.

Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences , 117(48):30046-30054, 2020. ISSN 0027-8424. doi: 10.1073/pnas.1907367117. URL https://www.pnas.org/content/117/48/30046 .

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems , 36, 2022.

Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D. Manning. Characterizing intrinsic compositionality in transformers with tree projections, 2023.

Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill , 2020. doi: 10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom-in.

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread , 2022. https://transformer-circuits.pub/2022/in-context-learning-and-inductionheads/index.html.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 36, 2022.

Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020 , pages 72377256. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.acl-main.647. URL https://doi.org/10.18653/v1/2020.acl-main.647 .

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '16, page 1135-1144, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450342322. doi: 10.1145/2939672.2939778. URL https://doi.org/10.1145/2939672.2939778 .

Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics , 8:842866, 2020. doi: 10.1162/tacl_a_00349. URL https://aclanthology.org/2020.tacl-1.54 .

Paul K. Rubenstein, Sebastian Weichwald, Stephan Bongers, Joris M. Mooij, Dominik Janzing, Moritz Grosse-Wentrup, and Bernhard Schölkopf. Causal consistency of structural equation models. In Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence (UAI) , 2017.

Naomi Saphra and Adam Lopez. Understanding learning dynamics of language models with SVCCA. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 3257-3267, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1329. URL https://www.aclweb.org/ anthology/N19-1329 .

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context, 2023.

Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 3145-3153. JMLR. org, 2017.

K Simonyan, A Vedaldi, and A Zisserman. Deep inside convolutional networks: visualising image classification models and saliency maps. In Proceedings of the International Conference on Learning Representations (ICLR) . ICLR, 2014.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 3319-3328. JMLR. org, 2017.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca , 2023.

Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4593-4601, Florence, Italy, July 2019. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/P19-1452 .

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback, 2022.

Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 12388-12401. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/ file/92650b2e92217715fe312e6fa7b90d82-Paper.pdf .

Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 57975808, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/ P19-1580. URL https://aclanthology.org/P19-1580 .

Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards understanding chain-of-thought prompting: An empirical study of what matters. arXiv preprint arXiv:2212.10001 , 2022.

Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. In Proceedings of the International Conference on Learning Representations (ICLR) . ICLR, 2023.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560 , 2022.

Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations , 2022.

Tailin Wu, Takashi Maruyama, and Jure Leskovec. Learning to accelerate partial differential equations via latent global evolution. Advances in Neural Information Processing Systems , 36, 2022.

Zhengxuan Wu, Karel D'Oosterlinck, Atticus Geiger, Amir Zur, and Christopher Potts. Causal Proxy Models for concept-based model explanations. arXiv:2209.14279, 2022. URL https: //arxiv.org/abs/2209.14279 .

Zhengxuan Wu, Atticus Geiger, Joshua Rozner, Elisa Kreiss, Hanson Lu, Thomas Icard, Christopher Potts, and Noah Goodman. Causal distillation for language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 4288-4295, 2022.

Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision , pages 818-833. Springer, 2014.

A Appendix

A.1 Price Tagging Game Experiment

Figure 7 shows our templates for our Price Tagging game Experiment. This format is enforced by the instruction tuning template of the Alpaca model.

A.2 Training Details

Training Setups During training for Boundless DAS, model weights are frozen except for the rotation matrix. To train the rotation matrix, we use Adam [26] as our optimizer with an effective batch size of 64 and a learning rate of 10 -3 . We use a different learning rate 10 -2 for our boundary indices for quicker convergence. We train our rotation matrix for 3 epochs and evaluate with a separate evaluation set for each 200 training steps, and we save the best rotation matrix evaluated on the evaluation set during training and test on another held-out testing set. Our training set has 20K examples. Our in-training evaluation is limited to 200 examples for quicker training time, and the hold-out testing dataset has 1K examples. These datasets are generated on the fly for different causal models (i.e., different random seeds result in different training data). The rotation matrix is trained by using the orthogonalized parameterization provided by the torch library. 4 For boundary learning, we anneal temperature from 50.0 to 0.10 with a step number equal to the total training steps and temperature steps with gradient backpropagation. Each alignment experiment is enabled with bfloat16 and can fit within a single A100 GPU. Each alignment experiment takes about 1.5 hrs. To finish all of our experiments, it takes roughly 2 weeks of run-time with 2 × A100 nodes with 8 GPUs each. We run experiments with 3 distinct random seeds and report the best result using IIA as our metrics.

Vanilla Task Performance Remarks To ensure the model we are aligning behaviorally achieves good task performance, we evaluate the model task performance with 1K sampled triples of values more than 300 times. The Alpaca (7B) model achieves 85% for randomly drawn triples of numbers on average. We want to emphasize that it is really hard to find a good reasoning task with intermediate steps (e.g., tasks like 'Return yes of it costs more than 2.50 dollars' do not require an intermediate step) that presently available 7B-30B LLMs can solve. We tested different reasoning tasks (e.g., math, logic, natural QA, code, etc.) with instruct-tuned LLaMA [49], Dolly, 5 , StableLM [3] Vicuna [9], Instruct-tuned GPT-J, 6 and mostly struggled to find tasks they could do well.

A.3 Counterfactual Dataset Generation

To train Boundless DAS, we need to generate counterfactual datasets where each example contains a pair of inputs as base and source, an intervention indicator (i.e., what variable we are intervening), and a counterfactual label. For instance, if we are training Boundless DAS for our Boundary Check (Left Only) high-level causal model, we may sample the base example with the bracket as [2.50, 7.50] and the input amount of 1.50 dollars, and the source example with the bracket as [3.50, 8.50] and the input amount of 9.50 dollars. If we intervene on the variable whether the input variable is higher than the lower bound, and we swap the variable from the second example into the first example, we will have a counterfactual label as 'Yes'. For each high-level causal model, we follow the same steps to construct the counterfactual dataset.

A.4 Interchange Intervention Accuracy

Figure 8 to Figure ?? show a breakdown of the results presented in Table 1.

A.5 Pseudocode for Boundless DAS

Boundless DAS is a generic framework and can be implemented for models with different architectures. Here we provide a pseudocode snippet for a decoder-only model.

A simplified version of Boundless DAS with pseudocode for a generic decoder-only model.

A.6 Generated Irrelevant Contexts

To generate random contexts in Table 2, we prompt GPT-4@Aug05-2023 to generate short English sentences. These sentences are added to the original prompt as prefixes.

Our Prompt: I will generate random short sentences (they do not have to be in English), ranging from 3 words to 15 words. Here are two examples in English: "Pricing tag game!" and "Fruitarian Frogs May Be Doing Flowers a Favor." Generate 100 examples for me and return them in a .txt file.

A.7 Common Questions

In this section, we answer common questions that may be raised while reading this report.

Can Boundless DAS work with 175B models?

Yes, it does. Taking the recently released Bloom-176B as an example, the hidden dimension is 14,336, which results in a rotation matrix containing 205M parameters. This is possible if the model is sharded and the rotation matrix is put into a single GPU for training. This is only possible given the fact that Boundless DAS is now implemented as a token-level alignment tool. If the alignment is for a whole layer (i.e., token representations are concatenated), it is currently impossible. It is worth noting that training the rotation matrix consumes more resources than training a simple weight matrix, as the orthogonalization process in torch requires matrix exponential, which means saving multiple copies of the matrix.

The current paper limits to a set of tasks from the same family. How well will the findings in the paper generalize to other scenarios?

Existing public models in 7B-30B range are still not effective in solving reasoning puzzles that require multiple steps. We test different reasoning tasks as mentioned in Section A.2. Once larger LLMs are released and evaluated as having stronger reasoning abilities, Boundless DAS will be ready as an analysis tool. Although our Price Tagging game task is simple, we test with different variants of the task and show the efficacy of our method.

Is it possible to apply Boundless DAS over the whole layer representation?

It is possible for smaller models if we are learning a full-rank square rotation matrix. It is intractable for Alpaca at present. Let's say we align 10 token representations together. We will have a total dimension size of 40,960. This leads to a rotation matrix containing 1.6B parameters. Given the current orthogonalization process in torch , this will require a GPU with memory larger than the current A100 GPU just to fit in a trainable rotation matrix.

Only limited high-level models are tested with Boundless DAS in the experiments.

There is a large number of high-level models to solve the task, and we only test a portion of them. We think that this is sufficient to demonstrate the advantages of our pipeline. It would be interesting if humans can be in the loop of testing different hypotheses of the high-level causal model in future work. On the other hand, not all the high-level models are interesting to analyze. For instance, if two high-level models are equivalent (e.g., swapping variable names), it is not in our interest to find alignments.

Can Boundless DAS be applied to find circuits in a model?

Yes, it can. We believe that 'circuit analysis' can be theoretically grounded in causal abstraction and is deeply compatible with our work. Previous work in this mode usually relies on the assumption of localist representation (i.e., there exists a one-to-one mapping between a group of neurons and a high-level concept). This is often too idealized, as multiple concepts can easily be aligned with the same group of neurons, or a group of neurons may map to multiple concepts. Boundless DAS actually drops this assumption by specifically focusing on distributed alignments. More importantly, Boundless DAS can be easily used in a head-wise alignment search where we add a shared rotation matrix on top of head representations of all the tokens.

Peer reviews:
comments: SUMMARY:
This paper introduces an advancement of Distributed Alignment Search (DAS), termed Boundless DAS, where the brute-force search steps are replaced with learned parameters. This enhancement enables efficient exploration of interpretable causal structures in large language models and therefore results in a scalable interpretable model.

STRENGTHS:
1) The increasing significance of LLMs necessitates an evaluation of their interpretability. Developing a scalable method to achieve this goal is undeniably crucial and intriguing.
2) Current interpretability methods are inadequate for LLMs due to scalability issues, rendering them impractical. In contrast, the proposed method offers a solution that enables the interpretation of real-world LLMs, filling this crucial gap.

WEAKNESSES:
1) My primary concern lies with the novelty of the paper, which appears to be exceedingly limited. The paper seems to be primarily an extension of DAS and a direct amalgamation of two existing works, namely DAS and neural PDE [58], without addressing any specific challenges or introducing significant innovative elements.
2) The paper heavily relies on the DAS method as its foundation, and it is imperative to provide a comprehensive explanation of DAS, including its limitations, which are currently absent in the paper. A thorough understanding of DAS and its shortcomings is crucial to fully comprehend the context and motivation of the proposed research.

QUESTIONS:
Please refer to Weaknesses.

LIMITATIONS:
The authors have discussed the limitations of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFinding alignments between variables in a user-hypothesized causal model for how a task could (or should) be performed and representations in neural networks, termed \"causal abstraction\", is a mechanistic interpretability technique for understanding how neural networks develop predictions at a granular level by using causal interventions.  Prior work has recently proposed the Distributed Alignment Search algorithm for efficiently and robustly finding this alignment, but it still relies on brute-force search over the dimensionality of neural representations. This paper proposes to circumvent the brute force search by *learning* parameters of an orthogonal rotation matrix to maximize the alignment between a neural network representation's linear subspace and one of the variables in the high-level causal model.\n\nThey test their method on one rather narrowly scoped task, which is determining if a price is between two numbers, and use the Alpaca (instruction-tuned LLaMA) 7B decoder-only language model. The authors also test whether the found alignments generalize across settings (both input prompts and output token choices for labels), and find that they do.\n\nEdit: I read the authors' rebuttal and the other reviews & discussions. I did not change my score as only one of the weaknesses (writing/presentation) was alleviated. I still think the paper should be accepted though.\n\nSTRENGTHS:\nOriginality:\n- The application of causal abstraction (or even other variants, like causal mediation analysis/causal tracing) to large instruction-tuned LMs has not been done; this is highly original to the best of my knowledge.\n\nQuality & Clarity:\n- The experiments seem sound. The analysis in sections 4.3 - 4.6 is insightful and interesting. The overall findings of the paper are useful to a larger community for understanding math processing in LMs.\n- The writing is clear overall.\n- Related works section is comprehensive.\n\nSignificance:\n- Causal analysis of language models at the representation-level is a very important goal with potentially far-reaching impacts for the broader NLP community. Using instruction-tuned models that are in widespread use today, such as Alpaca 7B, bridges the gap between theory and practice and makes this work more applicable to a wider audience.\n\nWEAKNESSES:\n- The paper does not read independently (particularly section 3.1) to a reader unfamiliar with prior work. I had to refer extensively to [32] to grasp what the various terms and variables used meant, and to understand what components of the proposed algorithm are novel w.r.t prior work. If there is not space in the main paper, the appendix should be used to thoroughly explain these concepts in a camera-ready version. The math notation could be improved; see \"questions\" section.\n- The method involves pre-registering some possible symbolic models for performing a task; it cannot discover novel symbolic models that are not specified by the user.\n- The method is tested on one narrowly-scoped task (determining if a price is between two numbers) and one model. There is no way to \"validate\" a mechanistic interpretability method beyond principle or axioms as we don't have the ground truth, so some of the findings are more suggestive rather than extremely concrete conclusions. However, this is a common problem in the subfield and I think this paper is convincing overall.\n\nQUESTIONS:\n- On finding strong alignments between Alpaca’s representations and two of the proposed causal models (\"left boundary\",\"left and right boundary\")— if the method reveals a faithful abstraction, wouldn't that imply that only one abstraction can have strong alignments with model weights, rather than multiple? Can you explain?\n- On writing clarification in Section 3.1-- it would be useful to establish early on that \"variables\" refers to hidden representations in a NN and \"algorithm\" to a high-level user-specified symbolic model. Relatedly, the terms \"input variables\" and \"target variables\" are a bit confusing as you also have input sequences (\"base input\", \"source inputs\") and target model outputs, which are different.\n-  I’m not sure I would use the term “largest” to describe a 7B model.\n- You use the phrase “human-interpretable” to describe the explanations that DAS produces; it would be good to provide a definition or at least some citations of what this concretely means.\n- Lines 170-171 and x-axis in Figure 4: these refer to the same thing; it would be good to give a concrete in-line example in the text as you have done for the core instruction.\n- Will you open-source your code?\n\nSome other mathematical notation could be improved. For example, \n- what do $j$ and $k$ represent? (I'm assuming $k$ is the number of dataset instances and $j$ the instance being indexed, but this is unstated). $k$ appears doubly-defined in Eqn. 3.\n- Set notation is mixed between uppercase variables and brackets\n- $\\mathbf{s}$ and $\\mathbf{b}$ are not vectors but sequences of inputs (i.e., tokens), which is a bit nontraditional notation; introduction of $\\textbf{Inputs}$ seems as though it should have been introduced earlier.\n- both $\\mathcal{N}$ and $\\mathcal{M}$ represent a neural network model, both \"algorithm $\\mathcal{A}$\" and \"model $\\mathcal{C}$\" (which is never used after defining) represent a high-level causal model, if I understand correctly.\n- no definition of $F$, $F^*$, $\\Pi$,  $L$\n- I did not follow the connection between target variables $X$, $\\mathbf{Z}$, and $\\mathbf{N}$.\n- $y_p$ and $y$ are more traditionally notated as $\\hat y$ and $y^*$; and should keep indexing consistent as $\\hat y_i$ and $y^*_i$\n\nLIMITATIONS:\nYes- sufficiently detailed & comprehensive limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper builds on Distributed Alignment Search by introducing a more efficient learned model to search for a given circuit in a model.  They test this method on a specific task: asking the model whether a given numeral falls between two other given numerals. They have a single task, and find an alignment with 2 possible circuits for accomplishing this task and fail to find a good alignment to 2 other handcrafted circuits, demonstrating that this method may be capable of distinguishing between correct and incorrect circuits.\n\nEDITED to raise soundness score.\n\nSTRENGTHS:\nThe approach of boundless DAS itself seems to be a worthwhile one, and I believe that it could work.\n\nThe approach scales well, up to Alpaca, and therefore could be very useful in modern models.\n\nI'm impressed by the approach of carefully constructing possible circuits to test, and it seems to work surprisingly well in LLM settings.\n\nThe observation that the IIA patterns are more structured for the seemingly correct circuits is an interesting one, though I have some reservations about this as a verification method (in questions).\n\nWEAKNESSES:\nAn obvious criticism of the method itself is that we lose a degree of interpretability by introducing learned parameters into the interpretation method. How many learned parameters does it take before you’ve just introduced another blackbox as your lens? If the problem is that you don’t trust learned parameters to behave interpretable, how can you ensure that you are behaving interpretably? In particular, it doesn’t seem to get away from the complaint that the authors themselves introduce: failing to find an alignment does not mean that the alignment doesn’t exist. This is a mild criticism, and I still consider the method to be potentially useful. \n\nMore significantly, the experiments are not convincing. There is only a single task evaluated. They consider only a single model, Alpaca. There are only four possible circuits considered. All of these factors add up to a fairly unconvincing set of experiments which have limited statistical power. Outside of the main experiments, these issues become more apparent: section 4.5 tests generalization from a fixed input to a different fixed input, with only a single sample considered for each case. A test of generalization with inserted context only considers two different possible contexts. Overall, these experiments are too constrained with too few samples to be convincing; while I don't think that this is cherry picked, results that comprise of case study of only a couple of samples could easily lead to publication bias.\n\nFurthermore, for the price task, the effectiveness of the alignment is measured by IIA, but the baseline for IIA is based on an assumption of complete randomness that I do not think is valid. After all, we have a method that has already found an alignment that maximizes the match with the circuit. I would trust this more if you had, e.g., tested it on random weights or weights trained for a completely different structured task that wasn't actually language modeling.\n\nOn another note, 4.5 appears to be the method that they should be using as the main experiment. I don't trust results that train the boundary parameters and alignments on the same inputs that they test on, which appears to be the approach taken in 4.3.\n\nThe math could be more carefully explained, with more precise definitions for the variables involved and more detailed intuitions for the method. One way to help is to turn Figure 1 into a step-by-step algorithm diagram.\n\nMinor:\n- \"Boundary\" is an overloaded term (used in both the method and the task). You should pick different wording.\n- \"We find that Alpaca achieves near-perfect task performance because it implements a simple algorithm with interpretable variables.\" *Because* is a *very* strong claim. Do they actually support the idea that the simple algorithm is the reason for the task performance? Is performance worse for tasks that don’t have simple algorithms? They do not show either piece of evidence.\n\nQUESTIONS:\nWhy would baseline IIA be 50%? You are actively aligning a circuit to the model, so shouldn't you be able to find an alignment that performs above 50% even in a randomly weighted model?\n\nWhat is a target variable $Z_j$?\n\nWhat if $F_N$?\n\nPlease explain the boundary index variable in more detail.\n\nWhen you restrict $b_j$ to be \"a multiple of $b_{j\u0000-1}$ with a factor of 2\", do you mean that $b_j = 2 b_{j-1}$ or that it could be any multiple as long as it is also even? Why would you use the notation in this way, rather than specify the variable if it is deterministic?\n\nIs it likely that we could have a very high IIA score without having more structured patterns? Rather than validating the IIA score, the observation of sparser patterns seem to strongly correlated with the score by definition, so I don't entirely trust it as a verification of the faithfulness of the alignment.\n\nWhat is an intermediate boolean value?\n\nHow large is the task dataset used to learn the alignment? Is it different from the set used to test IIA for the alignment? \n\n\"We sample 100 experiment runs from our experiment pool and create two groups\" Do the runs differ only by the input used? Or by some random seed?\n\nIn section 4.6, what does it mean to look at learned boundary width? What does a higher width indicate?\n\nAre the training steps (Figure 5 x axis) training steps for learning the alignment, or for the training of the LM? The number of steps is small so I assume it is for the alignment. What do we learn from the learning curve, that we didn't get from the final values?\n\nLIMITATIONS:\nThe authors acknowledge that their method is limited by the fact that not finding an alignment is not evidence that there is no alignment to the given circuit, which is very good. However, the limitations of the experiments themselves are not acknowledged.\n\nThis method only applies to synthetic tasks where we know the possible algorithms. This isn't explicitly acknowledged.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes an extension to Distributed Alignment Search (DAS) called \"Boundless DAS\" that replace the brute-force search with learned parameters in order to scale the method to large language models (LLMs).\nThe method evaluated on the Alpaca 7B model tasked to output \"yes\" or \"no\" to a simple numerical problem such as \"is X between Y and Z?\".\nExperimental analysis showed that the method is able to scale and find an alignment with two potential causal mechanisms.\nFurther experiments also show that the method is robust to slight modifications in the prompt of the model (additional sentence in the input, change of output format).\n\nSTRENGTHS:\nThis work proposes a new explainability method that is scalable to LLMs, which is an important research question as the field quickly progresses towards larger models that become part of real-life products. Hence, research to understand the inner mechanisms of LLMs is very important.\n\nThe proposed method is well-tested on various input/output perturbations of the target task, showing the robustness of Boundless DAS.\n\nWEAKNESSES:\n- the technical aspect of the paper is challenging to understand for someone that is not familiar with the field of causality\n\n- the scope of the task used to evaluate the method is very narrow. Some discussion on how this can be applied to more critical scenarios where interpretability is required (such as the medical, financial, or legal domains) could improve the paper.\n\n- It is mentioned that a good causal model could also explain the errors of the system. However, the current method does not explain why the model fails in certain examples.\n\nQUESTIONS:\n- Out of four possible causal mechanisms, two seem more plausible, but it is not clear which one is the one used by the model: \"left boundary\" or \"left and right boundary\"? Can we identify which causal method is the most likely being used, or is it the case that the model can use one or the other depending on the input?\n\nLIMITATIONS:\nSome limitations are mentioned.\nHowever, one limitation of this work that should be mentioned is the relatively narrow scope of the target task being evaluated. This may be ok if the authors discuss how to transfer their method to more \"realistic\" scenarios in which interpretability is required (such as the medical, legal, or financial fields). Another possibility is that it is challenging or impractical to apply the same type of analysis in more complex scenarios. Either way, the paper could benefit from this type of discussion.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method called Boundless DAS, an algorithm that can automatically test the \"alignment\" between human-specified symbolic algorithms and the computational structure found in the weights of a neural network. The authors apply Boundless DAS to a real-world 7B parameter language model called Alpaca and find evidence of the implementation of a particular symbolic algorithm that computes whether some number $q$ is in the range $[a, b]$.\n\nSTRENGTHS:\nReally cool paper!\n* I like the idea of using gradient descent to create a correspondence between a symbolic algorithm and a set of neural network features. Continuous optimization feels a little easier to tackle (initially) than discrete combinatorial stuff.\n* It's impressive that this works on a real-world, billion-parameter model.\n* While the tricks to go from DAS to Boundless DAS seem reasonable and slick, my favorite part is the empirical section — inspecting the alignments that the algorithm discovers, looking for generalization, etc. Well done.\n\nWEAKNESSES:\nNot much here. If I had to say anything, the paper is quite dense and hard to follow. For newcomers, it requires somewhat extensive background reading on causal abstraction and the previous DAS work(s). Otherwise, the writing is fairly clear, ideas are well-motivated, claims are well-supported and analyzed, etc.\n\nQUESTIONS:\n* What kinds of tasks have you tried DAS or Boundless DAS on? Do you ever find gradient descent struggling to find a faithful solution? Do you have intuitions for what kinds of tasks this works well on, and what it doesn't? How about algorithms with lots of input variables, or complex ones requiring lots of intermediate ones?\n* I'm trying to get a sense for how far we can push Boundless DAS; what do you think? For any challenges you foresee, what do you think the right next steps are?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for interpreting neural models by learning an alignment of their representations to a specified interpretable causal model. The method extends prior work, namely Distributed Alignment Search (DAS), by addressing a key limitation that prevents the scaling of DAS. The limitation is overcome by learning a set of masks that selects subsets of neural representations to align to different parts of the interpretable causal model. The method is evaluated experimentally by aligning Alpaca representations to a set of correct and incorrect hand-written causal models.\n\nSTRENGTHS:\n* The problem of interpreting large models is important.\n* The approach is reasonable and scales a previous approach.\n* The experimental results were convincing.\n\nWEAKNESSES:\n* Section 3 was hard to read. See the questions below.\n* Section 3 did not feel self-contained. The source of intractability in DAS did not jump out to me.\n* Giving the computational complexity for DAS and boundless DAS would make the efficiency argument stronger, unless they are the same.\n\nQUESTIONS:\n1. What is the value of introducing variable $Z$? It looks like only $X$ are used after section 3.1\n2. Is $\\Pi$ on line 99 used anywhere else?\n3. In line 115, '$b_j$ is restricted to be a multiple of $b_{j-1}$ with a factor of 2'. However, $b_0 = 0$. What does this mean?\n4. Why are target variables $Z_j$ bold but high-level variables $X_j$ not?\n5. In Eq. (1), what is the difference between $F_N^*$ and $F_N$? Is $F_N$ defined?\n6. What is the problem that was brute-forced in DAS, that $M$ learns to solve? Section 3.1 would be the best place to highlight intractability.\n7. Line 37 would be another nice place to highlight limitations in scaling DAS. What is the largest dimensional representation it has been applied to?\n1. What is the runtime improvement due to the method?\n8. Line 216: \"expectation\"\n\nClarifying the shortcoming of DAS, section 3, and giving the computational complexity would convince me.\n\nLIMITATIONS:\nThe limitations were adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}