Interpretability at Scale: Identifying Causal Mechanisms in Alpaca
Authors: Zhengxuan Wu
Atticus Geiger
Thomas Icard
Christopher Potts
Noah Goodman
Venue: NeurIPS 2023 poster (2023)
Keywords: Mechanistic Interpretability
TL;DR: We propose a new method based on the theory of causal abstraction to find representations that play a given causal role in LLMs.

Abstract: Obtaining human-interpretable explanations of large, general-purpose language models is an urgent goal for AI safety. However, it is just as important that our interpretability methods are faithful to the causal dynamics underlying model behavior and able to robustly generalize to unseen inputs. Distributed Alignment Search (DAS) is a powerful gradient descent method grounded in a theory of causal abstraction that uncovered perfect alignments between interpretable symbolic algorithms and small deep learning models fine-tuned for specific tasks. In the present paper, we scale DAS significantly by replacing the remaining brute-force search steps with learned parameters -- an approach we call Boundless DAS. This enables us to efficiently search for interpretable causal structure in large language models while they follow instructions. We apply Boundless DAS to the Alpaca model (7B parameters), which, off the shelf, solves a simple numerical reasoning problem. With Boundless DAS, we discover that Alpaca does this by implementing a causal model with two interpretable boolean variables. Furthermore, we find that the alignment of neural representations with these variables is robust to changes in inputs and instructions. These findings mark a first step toward deeply understanding the inner-workings of our largest and most widely deployed language models.

Interpretability at Scale: Identifying Causal Mechanisms in Alpaca
Zhengxuan Wu" Atticus Geiger",
Thomas Icard, Christopher Potts, and Noah D. Goodman
Stanford University
{wuzhengx, atticusg, icard, cgpotts, ngoodman)@stanfordd
Abstract
Obtaining human-interpretable explanations of large, general-purpose language models an urgent goal for AI safety. However, just important that our interpretability methods are faithful to the causal dynamics underlying model behavior and able to robustly generalize to unseen inputs. Distributed Alignment Search (DAS) [23] is powerful gradient descent method grounded in a theory of causal abstraction that has uncovered perfect alignments between interpretable symbolic algorithms and small deep learning models fine-tuned for specific tasks. In the present paper, we scale DAS significantly by replacing the remaining brute-force search steps with learned parameters an approach we call Boundless DAS. This enables us efficiently search for interpretable causal structure in large language models while they follow instructions. We apply Boundless DAS to the Alpaca model (7B parameters), which, off the shelf, solves simple numerical reasoning problem. With Boundless DAS, we discover that Alpaca does this by implementing a causal model with two interpretable boolean variables. Furthermore, we find that the alignment of neural representations with these variables is robust to changes in inputs and instructions. These findings mark first step toward faithfully understanding the inner-working! of our ever-growing and most widely deployed language models. Our tool is extensible to larger LLMs and released publicly at https: thub. com/ Fwwnkaginn//llaaa calinngggggggggggggggeg transformers.
1 Introduction
Present-day large language models (LLMs) display remarkable behaviors: they appear to solve coding tasks, translate between languages, engage in open-ended dialogue, and much more. As result, their societal impact rapidly growing, they make their way into products, services, and people's own daily tasks. In this context, it is vital that we move beyond behavioral evaluation to deeply explain, in human- interpreeaaeeee terms, the internal processes of these models, as an initial step in auditing them for safety, trustworthiness, and pernicious social biases.
The theory of causal abstraction [5,22] provides generic framework for representing interpretability methods that faithfully assess the degree to which a complex causal system (e.g., neural network) implements interpretable causal system (e.g., a symbolic algorithm). Where the answer is positive, we move closer to having guarantees about how the model will behave. However, thus far, such interpretability methods have been applied only to small models fine-tuned for specific tasks [19, (19,0.28.7] and this is arguably not an accident: the space of alignments between the variables in the hypothesized causal model and the representations in the neural network becomes exponentially
Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Identify reasoning that be solved by symbolic causal model.
Decoder-only Autoregressive LLMs
Stap
Instructions Representations
Test Query Representations
Step
Pick an off-the-shelf
Instruction Say yes only the cost between 2.00 and 2.00 dollar.
Input 2.50 dellars [moda: output]
Step
passes test (l.., good task performance)
Test Query Representations
Caural Model
Iteratively align causal model using Boundless DAS.
align repis
slign.
Trair with Boundless DAS for the proposed oligmment
repra
board
alignments
varisbiien
slignment
representations
Figure 1: Our pipeline for scaling causal explainability to LLMs with billions of parameters.
larger models increase in size. When good alignment is found, one has specific formal guarantees. Where no alignment found, could easily be a failure of the alignment search algorithm.
Distributed Aligmment Search (DAS) [23] marks real progress on this problem. DAS opens the door (I) discovering structure spread across neurons and (2) using gradient descent to learn alignment between distributed neural representations and causal variables. However, DAS still requires brute-force search over the dimensionality of neural representations, hindering its use scale.
In this paper, we introduce Boundless DAS, which replaces the remaining brute-force aspect of DAS with learned parameters, truly enabling interpretability scale. We use Boundless DAS to study how Alpaca (7B) [47], an off-the-shelf instruct-tuned LLaMM model, follows basic instructions in simple numerical reasoning task. Figure summarizes the approach. We find that Alpaca achieves near-pent perreee task performance because it implements simple algorithm with interpretable variables. In further experiments, we show that Alpaca uses this simple algorithm across a wide range of contexts and variations on the task. These findings mark first step toward faithfully understanding the workings of our largest and most widely deployed language models.
2 Related Work
Interpretability Many methods have been developed in attempt to explain and understand deep learning models. These methods include analyzing learned weights [12,1,13], gradient-based based methods 455,44.00.46552.66 probing [14.48,24, 42, (2.32,8,40) syntax-driven interventions [34], self-generated model explanations [27], and training external explainers based on model behaviors [39, 31]. However, these methods rely observational measurements of behavior and internal neural representations. Such explanations are often not guaranteed to be faithful to the underlying causal mechanisms of the target models [29,20,50.53).
Causal bstraction The theory causal abstraction [41,4,5] offers unifying mathematical framework for interpretability methods aiming to uncover interpretable causal mechanisms in deep learning models [19,28,20,58,5.22] and training methods for inducing such interpretable mechanisms 521.5..58.25). Causal abstraction [22] can represent many existing interpretablity methods, including interative nullspace projection [38, 15,30J, causal mediation analysis [51,33], and causal effect estimation [18, 16,2J. In particular, causal abstraction grounds the research program of mechanistic interpretabillity. which aims to reverse engineer deep learning models by determining the algorithm computation underlying their intelligent behavior [35.17,36.7.56) To the best of our knowledge, there is no prior work that scales these methods to large, general-purpose purpose LLMs.
Training LLMs to Follow Instructions Instruction-based fine-tuning of L.MMs can greatly enhance their capacity follow natural language instructions [10,37]. parallel, this ability can also be induced into the model by fine-tuning base models with hundreds of specific tasks [56. 1]. Recently,

---

Wang et al. [55] show that the process of creating fine-tuning data for instruction following models can partly be done by the target LLM itself ("self-instruct" Such datasets have led to many recent successes in lightweight fine-tuning of ChatGPT-like instruction following models such as Alpaca [47], instruct-tuned LLLMA [49], StableL. [3], Vieuna [9]. Our goal to scale methods from causal abstraction to understand how these models follow a particular instruction.
3 Methods
3.1 Background on Causal Abstraction
Causal Models We represent black box networks and interpretable algorithms using causal models that consist of variables taking on values according to causal mechanisms. We distinguish set of imput variables with possible values Inputs. An intervention an operation that edits some causal mechanisms in model. We denote the output of a causal model M provided an imput x as M(x).
Interchange Intervention Begin with model M (e.g., causal model or neural model) and a base input b and source inputs [s, all elements of Inputs, together with disjoint sets of target variables {Z,}² ) are aligning. An interchange intervention yields new model INTINV(M {s, (Z₂)²) h+) which is identical to M except the values for each set of target variables Zj are fixed to be the value they would have taken for source input Sj. We then denote the intervened output (i.e., counterfactual output) for the base input with INTINV(M {s (b).
Distributed Interchange Intervention Let N be subset of variables in M, the targer variables. Let Y be vector space with orthogonal subspaces (Y, J. Let R be invertible function R. N Y. A distributed interchange intervention yields a new model DII(M, R. {s,} (Y,,(YY))) (Y,}²) which is identical to M except the causal mechanisms have been rewritten such that each subspace Y, is fixed to be the value it would take for source input Sy." Specifically, the causal mechanism for N is,
=R R- " [ (R(FN(b))) Σ Proiy, (R(F(((s))))
(b) =
(1)
where b is the base input, Yo = = f Y,, and Projy, or Projy, represents the orthogonal projection operator of the original rotated vector into the Yo or Y, subspace. (N(() represents the causal mechanisms of causal variables N before any intervention, whereas (-)-) represents the mechanisms after the distributed intervention. We then denote the intervened output (i.e., counterfactual output) for the base input as DII(M, R, (s,hi (Y ()))).
Causal Abstraction and Alignment We are licensed claim a causal model (i.e., an algorithm) A is a faithful interpretation of the neural network N if the causal mechanisms of the variables in the algorithm abstract the causal mechanisms of neural representations relative to a particular alignment. For our purposes, each variable of a high-level model is aligned with a linear subspace in the vector space formed by rotating neural representation with an orthogonal matrix. We use 7 to represent the alignment mapping between a high-level causal variable and neural representation.
Approximate Causal Abstraction Interchange intervention accuracy (IIA) is graded measure of abstraction that computes the proportion of aligned interchange interventions on the algorithm and neural network that have the same output. The IIA for alignment of high-level variables Z, to orthogonal subspace Y, between an algorithm A and neural network. N' is
{(DIC, R. (s,)i, (1)))))))
Σ
Inputs| 1+1
b.s Elnputs-
INTINV(A, (r(8))),(())))))) (2))))))))
(2)
where translates from low-level causal variable values high-level neural representation values.
PPior works focus all-zero mean value representation replacement [33,54] which is less general.
3

---

3.2 Boundless Distributed Alignment Search
Distributed alignment search (DAS) is a method for learning an alignment between interpretable causal variables of a model C and fixed dimensionality linear subspaces of neural representations in network N using gradient descent [23] as shown in Figure Specifically, an orthogonal matrix R N - Y is trained to maximize interchange intervention accuracy under an alignment from each variable Zj to fixed dimensionality linear subspaces Y, of the rotated vector space.
Boundless DAS is our extension of DAS that learns the dimensionality of the orthogonal linear subspaces d-dimensional vector space Y using method inspired by work in neural PDE [57]. Specifically, for each high-level variable Zj. we introduce a learnable continuous boundary index parameter by that can take on value between 0 and d, where by by by+1, and by < d for all j. The boundary mask M, for the source is vector with d values between 0 and 1 where the k-th element of the array is defined be
k 15
bj+1 B
(M,), ; sigmoid
sigmoid
where 3 a temperature that we anneal through training. As approaches O, the masks Mj converge to binary-valued vectors that together encode an orthogonal decomposition of Y.
Weighted Distributed Interchange Intervention Let N be subset of variables in M, the target
variables. Let Y be a vector space with dimensions and let (M,) vectors [0, 1J Let R: ::NN:: N Y be invertible transformation. A weighted distributed interchange intervention yields new model (OFTDII R. (s,) (M2)*) which is identical M except the causal mechanisms have been rewritten such that each source input s, contributes to the setting of Y in proportion to its mask Mj. Specifically, the causal mechanism for N is be
(b) R (1- 2--EM) R(FN(b(F))))))) + Σ M, R((F(s)))
where is element wise multiplication and 1 is d dimensional vector where each element is 1. We then denote the output for the base input as (OFTDII R, {s,)f, (b))
Boundless DAS Given a base input b and source inputs {s,} we minimize the following objective learn a rotation matrix R and masks {M,)
Σ CE SOFTDII(N, R", (s,Hi
(2,))) (5)
b. El
where CE is the cross entropy loss. We anneal / throughout training and weighted interchange interventions become more and more similar to unweighted interchange interventions. During evaluation snap the masks to be binary-valued each high-level variable Zy is aligned with a linear subspace of picked out by the mask M,. with the residual (unaligned) subspace being picked out by the mask Σラ M,).
create orthogonal decomposition of Y where
Time Complexity Analysis DAS learns rotation matrix but requires manual search to determine how many neurons are needed to represent the aligning causal variable. Boundless DAS automatically learns boundaries (i.e., how many neurons are needed is determined by the "soft" boundary index via a boundary mask learning). For instance, given a representation with a dimension of 1024, DAS should in principle be run for all lengths from 1 to 1024. In practice, this would be infeasible, so some subset of the lengths need to be chosen heuristically, which risks missing genuine structure. For Boundless DAS, we this search process into a mask learning process. DAS [23] is On X m) where n is the number total dimensions of the neural representation we are aligning and m is the number of causal variables, while Boundless DAS is ((m).
4 Experiment
4.1 Price Tagging
We follow the approach in Figure by first assessing the ability Alpaca execute specific actions based the instructions provided in the input. Formally, the input to the model M is given an

---

P
PAQ
[2-M] 5MY--X
n
P Z
P=X=2
Q=
M 5*I*+)
Left
Right Boundary
Mid-point
Bracket
Figure 2: Four proposed high-level causal models for how Alpaca solves the price tagging task. Intermediate variables are in red. All these models perfectly solve the task.
[Left Right Boundary)
Yes
PAQ
No
PAQ
Causal Model
P=XsZ
==ZsY
P 2
=
LM Head
Learned
2.00
3.00
1.50
5.00
8.00
9.30
MLP with Skip Connection
X
X
No
Multihead Self-Attention
Instruction: Say yes only if the cost
2.00 3.00 dollars.
Instruction: Say ves only if the cost
is between 5.00 and 8000 ssllats.
Dimensions in Rotated Space
0
Token Representations
Neural Network
Intervened
Representations
0-
Intorvention
Flow
0
3--0
+0
1-
Alignment
Input: 1.50 dollars
Input: 9.30 dollars
Figure Aligned distributed interchange interventions performed on the Alpaca model that is instructed to solve our Price Tagging game. To train Boundless DAS, we sample two training examples and then swap the intermediate boolean values between them to produce counterfactual output using our causal model. In parallel, rotated space. Lastly, we update our rotation matrix such that our neural network has more similar counterfactual behavior to the causal model.
swap the aligned dimensions of the neural representations in
instruction f; (e.g., "correct the spelling of the word:") followed by test query input x; (e.g.. "aplpe"). use M((,, 2;,) to depict the model generation Vp given instruction and the test query input, We can evaluate model performance by comparing Up with the gold label We focus on tasks with high model performance, to ensure that we have a known behavioral pattern to explain.
The instruction prompt of the Price Tagging game follows the publicly released template of the Alpaca (7B) model. The core instruction contains English sentence:
Please say yes only if costs between [X.XX, and [X. dollars, otherwise
followed by an input dollar amount [X. XX], where [X. XX] are random continuous real numbers drawn with uniform distribution from [0.00, 9.99]. The output
single token 'Yes" or 'No'.
For instance, if the core instruction says Please say yes only if it costs between [1.30] and [8. 3.5] dollars, otherwise no., the answer would be "Yes" if the input amount "3.50 dollars" and "No" the input is "9,50 dollars". We restrict the absolute difference between the lower bound and the upper bound be [2.50, 7.50] due to model errors outside these values again need behavior to explain.
4.2 Hypothesized Causal Models
As shown in Figure 2, we have identified a set of interpeaateeeee high-level causal models, with alignable intermediate causal variables, that would solve this task with 100% performance:
5

---

This model has one high-level boolean variable representing whether the imput amount is higher than the lower bound, and an output node incorporating whether the input amount is also lower than the high bound.
Left Boundary:
Left and Right Boundary: The previous model is sub-optimal in only abstracting one of the boundaries. In this model, we have two high-level boolean variables representing whether the input amount is higher than the lower bound and lower than the higher bound, respectively. We take a conjunction of these boolean variables to predict the output.
Mid-point Distance: We calculate the mid-poin of the lower and upper bounds (e.g., the mid-point of "3.50" and "8.50" is "6.00), and then we take the absolute distance between the input dollar amount and the mid-poin point as a. We then calculate one-half of the bounding bracket length (e.g., the bracket length for "3.50" and "8.50" is "5.00") as b. We predict output " "Yes" a s b, otherwise "No". We align only with the mid-point variable.
Bracket Identity:
This model represents the lower and upper bound in a single interval variable and passes this information to the output node. We predict the output as "Yes" if the input amount within the interval, otherwise "No".
Model Architecture Our target model is the Alpaca (7B) [47], an off-the-shelf instruct-tuned LLaMM model. It is Transformer-Be based decoder-only autoregressive trained language model with 32 layers and 32 attention heads. It has a hidden dimension in size of 4096, which is also the dimension of our rotation matrix which is applied for each token representation. In total, the rotation matrix contains 16.8M parameters and has size 4096> X 4096.
Alignment Process We train Boundless DAS with test query token representations (i.e., starting from the token of the first input digit till the last token in the prompt) in a selected set of 7 layers: {0, 5, 10, 15, 20, 25, 30). We also train Boundless DAS on the token before the first digit as a control condition where nothing should be expected to be aligned. We interchange with a single source, while allowing multiple causal variables to be aligned across examples. For simplicity, we enforce the interval between two boundary indices to be the same (i.e., the same size of linear subspace for each aligning causal variable). We run each experiment with three distinet random seeds. Since the global optimum of the Boundless DAS objective corresponds to the best attainable alignment, but SGD may be trapped by local optima, we report the best-performing seed in terms of IIA. Figure provides an overview of these analyses.
To evaluate models, we use Interchange Intervention Accuracy (IIA) as defined in Eqn. 2. IIA is bounded between 0.0 and 1.0. We measure the baseline IIA by replacing the learned rotation matrix with random one. The lower bound of IIA for "Left Boundary" and "Left and Right Boundary" is about 0.50, and for "Mid-point Distance" or "Bracket Identity" it is about 0.60. The latter two baseline IISs are higher than chance because they are conditioned the distribution of our output labels (i.e., how many times the original label gets to be flipped due to the intervention). See Section 4.7 for more discussion of metric calibration. Additionally, IIA can occasionally go above the model's task performance, when the interchange interventions put the model in a better state, but IIA is constrained by task accuracy for the most part.
Evaluation Metric
4.3 Boundless DAS Results
Figure 4 shows our main results, given in terms of IIA across our four hypothesized causal models (Figure 2). The results show very clearly that 'Left Boundary' and 'Left and Right Right Boundary" (top panels) are highly accurate hypotheses about how Alpaca solves the task. For them, IIA is at or above task performance (0.85), and intermediate variable representations are localized in systematically arranged positions. By contrast, 'Mid-point Distance' and 'Bracket Identity' (bottom panels) are inaccurate hypotheses about Alpaca's processing, with IIA peaking at around 0.72.
These findings suggest that, when solving our reasoning task, Alpaca internally follows our first two high-level models by representing causal variables that align with boundary checks for both left and right boundaries. Interestingly, heatmaps on the top two rows also show a pattern of higher scores around the bottom left and upper right and close to zero scores in other positions. This is significantly different from the other two alignments where, although some positions are highlighted (e.g., the representations for the last token), all the positions receive non-zero scores. In short, the accurate hypotheses correspond to highly structured IIA patterns, and the inaccurate ones do not.

---

Left Boundary
and Right Boundary
0.7448..71
0.59
0.52 0.500
0, 0 48
0.48
0.86
e-0.51
0.650.55
0.53
0.51 0155000111111 51 0.51
0.51
0.51
0.81
Alpaca (78) Lager Index
Entex
0..488.720..8
0.53
0.50
0.51
0.64
0.50
08
0.49
0.53
0.59
0.89
00.31
0.64
0.640.6
0.63
0.51
0.51
0.52 0.3 37 0.51 0.51
0.51
0.53
0.53
0.86
19
-00499.7330
0.55
62
0.
0.550.67
0.89
-0.51
0.68
0..6
0.5.0.5..55.3333
0.51 0.05111111000
0.54
0.86
30 13 70 15
3-0.48
0.770.77.6
0.700.82.088
15
-0051
0.720.700
008.690.730000
0.52
0.650
0.88
Alpaca (7B)
0.480.60
8.81
0.69
-0
0046
0.80
0.000.20.7
0.730
0 0.51
0.551
0.770.76
0.630 0.0.5.. 0.51
0.55
0.480.80
0,000400.000
0.55
0.500.48 0.48 48.40.444 0.8.44.........4.4..... 0.44
30 0.51
0.74
0.52
022.0.53.0
0.53
0.51 0.51 0.51 0.51
0..10.............
0.51
Mid- poin. Distance
Brackel Identity
0.63
0.63
0.65
0.66
80
0.61
0.6.0660066000.000 0.70
Alpacoa (78) Lager Index
Alperca (78) Leyer Enddax
13 00663
9.65
0.00
0.62
0.63
0.63
0.63
0.63 0.65
0.60
0.660
0.63 0.660
0.61 0.59 0.000000000000000000000.0..3...0
20 15 19
0...330.00
0.670.65
0.65
0.64
0.62
0.66
046667
0.69
-0.61
0.633.62
0.60 0.66 0.6610666600000 0.61 0.6.
0.6330.67
0.64
0.65
0.64
0.66
0.65
0.64
0.64
0.66 0.70
1 15
0.600.65..600 9.0.60
0.61
0.61
0.63 0.62 www.6222..60..000000000 www.0.00..000000....000
2.630..6700
0.65
0.64
0.66
0.661
0.700.69
0.65
0.62
0.009
0.6.0665 0.0.0.00.0000000000000000
0.64
0..06770
0.0.5500000
0.61
30
720.63.000
0.63
30
0.61
8.60
0.610.60
0.63
20.0 (69)
X (70)
(71)
(72)
% (73)
"dllarr (74)
<0r0A> (75)
<<0a0>> (70)
*4 (77)-
(78)
Desponne (79)
(800)
<<rob>> (83)
"ChOA>> (60)
(71)
(72)
x (33)
"dlllrr (74)
<000 (75)
<000A>> (78)
"# (77)
(79)
Responoo (79)-
(80)
<<r00>> (81)
X
Token (Index)
Taken (Index)
Figure 4: Interchange Intervention Accuracy (IIA) for four different alignment proposals. The Alpaca model achieves 85% task accuracy. The higher the number is, the more faithful the alignment is. We color each cell by scaling IIA using the model's task performance as the upper bound and a dummy classifier (predicting the most frequent label) the lower bound. These results indicate that the top two are highly accurate hypotheses about how Alpaca solves the task, whereas the bottom two inaccurate in this sense. Analyzing tokens includes special tokens (e.g., '<0X0A>' for linebreaks) required by Alpaca's 's instruct-tuning template as provided in Appendix A.1. Heatmap color uses min-max standardized IIA using the random rotation baseline as the lower bound, and task performance as the upper bound.
Additionally, alignments are better when the model post-processes the query input with 1-2 additional steps: accuracy is higher at position 75 compared to all previous positions. Surprisingly, there exist bridging tokens (positions 76-79) where accuracy suddenly drops compared earlier tokens, which suggests that these representations have weaker causal effects on the model output. In other words, boundary-check variables are fully extracted around position 75, level 10, and later copied into activations for the final token before responding. By comparing the heatmaps on the top row, our findings suggest that aligning multiple variables the same time poses a harder alignment process, in that it lowers scores slightly across multiple positions.
4.4 Interchange Interventions with (In-)Correct Inputs
IIA is highly constrained by task performance, and thus we expect it to much lower for inputs that the model gets wrong. To verify this, we constructed an evaluation set containing 1K examples that the model gets wrong and evaluated our 'Left and Right Boundary' hypothesis on this subset. Table I reports these results in terms of max IIA and the correlation of the IIA values with those obtained in our main experiments. As expected, IIA drops significantly. However, two things stand out: IIA is far above task performance (which is 0.0 by design now), and the correlation with the original IIA map remains high. These findings suggest that the model is using the same internal mechanisms to process these cases, and it seems possible the model is narrowly missing the correct output predictions.
We also expect IIA to higher we focus only on cases that the model gets correct. Table confirms this expectation using 'Left and Right Boundary' representative. IIA is improved from 0.86 to 0.88, and the correlation with the main results essentially perfect. Full heatmaps for these analyses given in Appendix A.4.
7

---

Experiment
Task Acc.
IIAmax
Correlation
Left Boundary
0.85
0,90
1.00
Left and Right Boundary
0.85
0.86
1.00
Mid-point Distance
0.85
0.70
1.00
Bracket Identity
0.85
0.72
1.00
Correct Only
1.00"
0.88
0,99
Incorrect Only
0.00
0.71
0.84
New Bracket (Seen)
0.94
0.94
0.97
New Bracket (Unseen)
0.95
0.95
0.94
Irrelevant Contexts
0.83
0.99
Sibling Instructions
0.83
0.87
exclude top right
0.84
0.83
0.92
Table I: Summary results for all experiments with task performance as aceuracy (range [0, I]),
maximal interchange intervention accuracy (IIA) (range [0, 1]) across all positions and layers,
Pearson correlations of IIA between distributions (compared to + range 1J). 'This is empirical task performance on the evaluation dataset for this experiment.
Say yes or no' (Seen)
'Say True or False' (Unseen)
88
00.51
0.85 0.85
0.51
5.53 0.52
0.51 0.51
0.51
0.81
-..50
0.51 0.51 5. 0.50 0.500........1.. 0.51..00.....000..0..0
0.50 0.61
Alpoca (?B) Layer Index
9-0.51
0.64 0.61
0.51
0.51
0.52
0.51 0.51
0.53
0.577
Indox
25
0.0.1
0.69
0.61
0.56
0.52
0.55
0.57
0.51
0.51
0.51
0.510.60
0.06
0.500
0.55
10
0.84
(7B)
0.51
0.80
0.69
0.73
0.61
0.83
0.59
0.51
0.51
0.52
0.52
0.59
10
0.500
0.79
0...0.2222
0.
0.79
0.83
0.8.0.50000000
0.56
00.51
0.77
76
0.70
70.00000
0.68
8.6.63. 60
000.51..33
0.51
0.51
0.51
0.55
-0.52
0.77
0.56
0.51
0.51
0.51 0.51 0.54
0.74
0.51
0.51
0.51
0.50
<0HOA>> (68)
(70)
(71)
X (73)
dollary' (74)
<x000A>> (75)
<0x0>>> (76)
(70)
Responne (79)
COX0A (81)
(69)
X (70)
(71)
X (72)
X (73)
<0x0x>> (75)
<0x0A>> (76)
"9' (78)
(79)
(80))
<OXDA>> (81)
dollarrs
Token (Iadex)
Token (Iadex)
Figure 5: Interchange Intervention Accuracy (IIA) evaluated with different output formats. The seen
setting is for training, and the unseen setting is for evaluation.
4.5 Do Aliguments Robustly Generalize to Unseen Instructions and Inputs?
One might worry that our positive results are highly dependent on the specific input-output pairs we have chosen. We now seek address this concern by asking whether the causal roles (i.e.. alignments) found using Boundless DAS in one setting are preserved in new settings. This is crucial, as it tells how robustly the causal model is realized in the neural network.
Generalizing Across Two Different Instructions Here, we assess whether the learned alignments for the 'Left Boundary' causal model transfer between different specific price brackets in the instruction. To do this, we retrain Boundless DAS for the high-level model with fixed instruction that says "between 5.49 dollars and 8.49 dollars". Then, fix the learned rotation matrix and evaluate with another instruction that says "between 2.51 dollars and 5.51 dollars". For both, Alpaca is successful the task. with around 94% accuracy. Our hypothesis is that if the found alignment of the high- level variable is robust, it should transfer between these two settings, as the aligning variable is boolean-type variable which is potentially agnostic to the specific comparison price.
Table gives our findings the 'New Bracket' rows. Boundless DAS is able to find a good alignment for the training bracket with an IIAmax that is about the same as the task performance 94%. For unseen bracket, the alignments also hold up extremely well, with no drop in IIA For both cases, the found alignments also highly correlate with the counterpart of our main experiment.
Generalizing with Inserted Context Recent work has shown that language models are sensitive to irrelevant context [43], which suggests that found alignments may overfit to set of fixed contexts.

---

Boundary Checks (Left Only)
Boundary Checks (Both)
1.0
UI (Succeas)
Width (Success) Width (Pallure)
1.0
UA (Success)
width (Success)
(Pailure)
(Pallure)
Width (Failure)
0.8
0.6
Value
0.6
0.4
0.4
0.2
0.2
0.0
1000
2000 3000
4000
5000
6000
0 1000 2000
3000
4000
5000
6000
Training Steps
Training Steps
Figure 6: Learned boundary width for intervention site and in-training evaluation interchange intervention accuracy (IIA) for two groups of data: (1) aligned group where the boundary does not shrink 0 at the end of the training; (2) unaligned group where the boundary does shrink to 0 at the end of the training. on the y-axis means either 100% accuracy for IIA, or the variable is occupying half of the hidden representation for the boundary width.
To address this concern, we add prefix strings to the input instructions and evaluate how 'Left and Right Boundary' alignment transfers. We thus generate 20 random prefixes using GPT-4.² All of our prefixes, and our method for generating them, can be found in Appendix A.6. The model achieves 84% task performance with random contexts added as prefixes, which is slightly lower than the average task performance. Nonetheless, the results in Table suggest the found alignments transfer surprisingly well here, with only 2% drop in II Meanwhile, the mean IIA distribution across positions and layers is highly correlated with our base experiment. Thus, our method seems to identify causal structure that is robust to changes in irrelevant task details and position in the input string.
Generalizing to Modified Outputs We further test whether the alignments found for instructions with template "Say yes .... otherwise no" can generalize a new instruction with a template "Say True otherwise False". If there are indeed latent representations of our aligning causal variables, the found alignments should persist and should be agnostic the output format.
Table shows that this holds: the learned alignments actually transfer across these two sibling instructions, with a minimum 1% drop in IIA In addition, the correlation increases 6% when excluding the top right corner (top layers in the last position), as seen in the final row of the table. This result is further substantiated in Figure 5, where indicates a late fusion of output classes and working representations, leading to different computations for the new instruction only close to the output.
that IIA drops near the top right. This
4.6 Boundary Learning Dynamics
Boundless DAS automatically learns the intervention site boundaries (Section 3.2). It is important to confirm that we do not give up alignment accuracy by optimizing boundaries, and to explore the dimensionality actually needed to represent the abstract variables. We sample 100 experiment runs from experiment pool and create two groups: (1) the success group where the boundary does not shrink to 0 at the end of the training; (2) the failure group where the boundary does shrink to at the end of the training. The failure group is for those instances where there is significant causal role of representations found by our method. We plot how boundary width and in-training evaluation IIA vary throughout training. Figure 6 shows that, in the success cases, each aligned variable only needs 5-10% of the representation space. IIA performance converges quickly for success cases and, crucially, maintains stable performance despite shrinking dimensionality. This suggests that only small fraction of the whole representation space is needed for aligning a causal variable, and that these representations can be efficiently identified by Boundless DAS.
We generate these prefixes using the public chat interface https://chate httt::://cchat.oeeaa..

---

4.7 Metric Calibration
One concern brought up by our reviewers is whether our metric, interchange intervention accuracy (IIA), is well calibrated. Here, we provide additional evidence. We report IIAs in the same way as in Figure 4 but with random rotation matrices. With a random rotation matrix, we find that the IIA score drops from 0.83 to 0.53 at layer 10, token position 75, for our "left and right boundary" causal model. Other positions drop significantly as well. These results calibrate IIAs in case of unbalanced labels (e.g., our Two control causal models reach about 0.60 IIA at the same location with random rotation matrix). This shows that Boundless DAS finds substantial structure in Alpaca as compared to what these random baselines can find. Details can be found in Figure 12 in the Appendix.
We can also help calibrate our result by comparing Boundless DAS applied Alpaca with a randomly initialized LLaMA-7B, This can help quantify the extent to which DAS leverage random causal structure in order to overfit to its training objective. This random model results in near 0% task performance on our task, as expected. After running Boundless DAS on the first layer of this random LLaMA-7B model, the found alignments to each token's 's representation ranged from 0% to 69% IIA, which is comparable to most frequent label dummy model (66%). For the representations with near chance IIA, this means that we were able to find a distributed neural representation that shifts the probability mass to "yes" or "no", but does not differentiate between the two. We also found that we could find a distributed neural representation that shifts the probability mass to the tokens "dog" and "give" instead of ""yes" and "no". More importantly we found that IIA drops to 0% for this run on all of our robustness checks from the paper. In addition, it drops to 0% if we use it on another randomly initialized LLaMA-7B model with different random seed. Overall, it seems that the causal structure we did find has arisen by chance, and we might expect very large models to be more prone to such occurrences. Our robustness checks definitively clarify the situation.
5 Analytic Strengths and Limitations
Explanation methods for Al should be judged by the degree to which they can, in principle, identify models' true internal causal mechanisms. If we reach 100% IIA for a high-level model using Boundless DAS we can assert that we have positively identified correct causal structure in the model (though perhaps not the only correct abstraction). This follows directly from the formal results of Geiger al. [22]. On the other hand, failure to find causal structure with Boundless DAS is not a proof that such structure is missing, for two reasons. First, Boundless DAS explores a very flexible and large hypothesis space, but it is not completely exhaustive. For instance, a set of variables represented in highly non-linear way might be missed. Second, we are limited by the space of causal models we think test ourselves.
For models where task accuracy is below 100%, it is still possible for IIA to reach 100% if the high- level causal model explains errors of the low-level model. However, in cases like those studied here, where our high-level model matches the idealized task but our language model does not completely do so, expect Boundless DAS to find only partial support for causal structure even if we have identified the ideal set of hypotheses and searched optimally. This too follows from the results of Geiger et al. [22]. However, we have seen in the results above, Boundless DAS can find structure even where the model's task performance is low, since task performance can be shaped by factors like a suboptimal generation method or the rigidity of the assessment metric. Future work must tighten this connection by modeling errors in the language model in more detail.
6 Conclusion
We introduce Boundless DAS, a novel and effective method for scaling causal analysis of LLMs to billions of parameters. Using Boundless DAS, we find that Alpaca, off-the-shelf, solves a simple numerical reasoning problem in a human-interpretabb way. Additionally, we address one of the main concerns around interpretability tools developed for LLMs whether found alignments generalize across different settings. We rigorously study this by evaluating found alignments under several changes to inputs and instructions. Our findings indicate robust and interpretable algorithmic structure. Our framework is generic for any LLMs and is released to the public. We hope that this marks step forward in terms of understanding the internal causal mechanisms behind the massive L.MS that are at the center of so much work in AI.
10

---

References
[1] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190-4197, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/w1/2020..tt mainm.. 385. URL http::://aclanthologg.oorr hwwp:tttwwwoooooooooeeooooo 20w.///e/22..00 wrtp:/...........mmma...... acl-main.
[2] Eldar D Abraham, Karel D D'Oosterlinck, Amir Feder, Yair Gat, Atticus Geiger, Christopher Potts, Roi Reichart, and Zhengxuan Wu. Cebab: Estimating the causal effects of real-world concepts on nlp model behavior. Advances in Neural Information Processing Systems, 35: 17582-17596, 2022.
[3] Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Shivanshu Purohit, Tri Songz, Wang Phil, and Samuel Weinbach. GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch, 8 2021. URL hittps://wwe thub. comm eleutherai/gpt-nen
[4] Sander Beckers and Joseph Halpern. Abstracting causal models. In AAAI Conference on Artificial Intelligence, 2019.
[5] Sander Beckers, Frederick Eberhardt, and Joseph Y. Halpern. Approximate causal abstractions. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, 2019.
[6] Yonatan Belinkov and James R. Glass. Analysis methods in neural language processing: A survey, Trans, Assoc. Comput. Linguistics, 7:49-72, 2019, doi: 10. al 0002544 URL https://doi.. https://doi.org/11
[7] Lawrence Chan, Adrià Garriga-Alonso, Nicholas Goldowsky-Dill Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. Causal scrubbing: method for rigorously testing interpretability hypotheses, 2022.
[8] Ethan A. Chi, John Hewitt, and Christopher D. Manning. Finding universal grammati- cal relations in multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5564-5577, Online, July 2020. Associ- ation for Computational Linguistics. doi: 10. 10.18633/1122222allmmaam org/ httgg///nholoo//22200 acl-main. 493.
URL https:
/
[9] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://vicunn http://wwcnaa.mm....comm org.
[10] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing systems, 30, 2017.
[11] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instructionn-fintuneed language models. arXiv preprint arXiv:22100 10. 1416, 2022.
[12] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286 Florence, Italy, August 2019. Association for Computational Linguistics. doi:
URL httpps://aclanthology org//19
[13] Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, and Martin Wattenberg. Visualizing and measuring the geometry of bert, 2019.
11

---

[14] Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single S&!# vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2136,,,3, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 11511166//111111111 1198. URL https: /www. aclweb. hwtp:ebborg/atthhlogoooooo wrt./anthology//11--111111
[15] Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. In Proceedings of the 2020 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NL.. Association for Computational Linguistics, November 2020. doi: 18653v L/W18-526..
[16] Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schütze, and Yoav Goldberg. Measuring causal effects of data statistics on language model's 'factual" predictions. CoRR, abs/2207. 2022. doi: 10-485500arXX..2200.4423 URL httpps///of org/10. 48550/arKiv 2207. 14251.
[17] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodde Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https://raanfformr-ciicott pub/202 h/frmmcworkinddxx.ttna
[18] Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. CausaLM: Causal Model Explanation Through Counterfactual Language Models. Computational Linguistics, pages 1-54, 05 2021. ISSN 0891-2017. doi: 10.1 162/coli_a a__0044 URL https: /toii org/10 /00 10. 2166//ool_ 00404.
[19] Atticus Geiger, Kyle Richardson, and Christopher Potts. Neural natural language inference models partially embed theories of lexical entailment and negation. In Proceedings of the Third BlackbexNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 163- 173, Online, November 2020. Association for Computational Linguistics. doi: 10. 1865/vvvv 2020. 11220blackboxnlp-..... URL https: wwwweb.wwwwww...wwwwwwwwwwww.... org/ wrrg/anthloogy/2000 blackboxnlp-1. 16.
[20] Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstrac- tions of neural networks. Advances Neural Information Processing Systems, vol- ume 34, pages 9584-9586, 2021. URL https: https://papers.ee. http:/ppprssisss.... www.niipccccccmc... httpsccc//aeer/2////sas/ html.
4 44542222444422200000222444444444
[21] Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Goodman, and Christopher Potts. Inducing causal structure for interpretable neural networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 7324-7338. PMLR, 17-23 Jul 2022. URL https: / //ttte/s//prrcedinggg. mlr. wwtppp/wwee/ppeeeppepppppppp www.ee/egggiieeaa...eaa html.
[22] Atticus Geiger, Christopher Potts, and Thomas Icard. Causal abstraction for faithful model interpretation. Ms., Stanford University, 2023. URL https://arxixs wrwp.o//g/bbb//11 04709.
[23] Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D. Goodman. Finding alignments between interpretable causal variables and distributed neural representations. 02536.
Ms., Stanford University, 2023. URL https:///rxiiv
[24] John Hewitt and Christopher D Manning. structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4129-41333 2019.
[25] Jing Huang, Zhengxuan Wu, Kyle Mahowald, and Christopher Potts. Inducing character-level structure subword-basee language models with Type-level Interchange Intervention Training. Ms., Stanford University and UT Austin, 2022. URL https: / //tx//axiivvv////////ea/////avl/ orw.v.org//bs//22122222 09887. 09897.
12

---

[26] Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations (ICLR), 2015.
[27] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2022.
(28) Belinda Z. Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Associa- fion for Computational Linguisties and the 1 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 181 1813-1827, Online, August 2021. Association for Computational Linguisties. doi: 10.18653/w1/0021.acl https://aclanthol..g org/2021 10221.acl-oooe acl-long. 143.
URL
[29] Zachary C Lipton. The mythos of model interpretabillty:: In machine learning, the concept of interpretability both important and slippery. Queue, 16(3):31-57 2018.
[30] Charles Lovering and Ellie Pavlick. Unit testing for concepts in neural networks. Transactions of the Association for Computational Linguistics, 10:1193-1208, 2022. doi:
14.
URL https: http:::://aalaathhllooooooo org/2022. tacl-1.69 69.
[31] Scott Lundberg and Su-In Lec. unified approach to interpreting model predictions, 2017.
[32] Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. Emergent linguistic structure in artificial neural networks trained self-supervision. Proceed- ings of the National Academy of Sciences, 1 117444)33004--30054, 2020. ISSN 0027-8424. doi:
(tt.0733pmaa.190777711111 URL https: /www. wwwpwwws.w.wppppww//.../.../....
pnas. ortg//ontent/11///8/////
[33] Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36, 2022.
[34] Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D. Manning. Characterizing intrinsic compositionality transformers with tree projections, 2023.
[35] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.23915/dssiil..0002..00
[36] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodde Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. https:/traasformeeecrrriii putpuitts.ub/2022/mmm context-eeannnnggg......o......e and-induction- heads/index.html.
[37]
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkín, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 36, 2022.
[38] Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7237- 7256. Association for Computational Linguistics. 2020. doi: 10. 112253/1/2220.aclll mal.mmm....67. URL https:tt//// /oi. org/10. 116...11111111/ /220 acl-main.644 647.
[39] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining, KDD '16, page 1135-1 144, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450342322. 00.1145/2939672.29997777 URL https: stttps://doi. wrg./0..///ooo 1ttpe//...111//99910 /15521999922 29377
doi:
13

---

[40] Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842- 866, 2020. doi: 10.1
URL. https://aclaanthol www.:/rrgg/oa/ http:/222.....11..5111 tacl-1.54
[41] Paul K. Rubenstein, Sebastian Weichwald, Stephan Bongers, Joris M. Mooij. Dominik Janzing, Moritz Grosse- Weiserp,, and Bernhard Schölkopf. Causal consistency of structural equation models. In Proceedings of the 33rd Conference on Uncertainty Artificial Intelligence (UAI), 2017.
[42] Naomi Saphra and Adam Lopez. Understanding learning dynamics of language models with SVCCA. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3257-3267, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18633/1//1191153 URL https: /www///wwwwooo//w/ ac acwweb. org/ anthology/NI 91119-117 1329.
[43] Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context, 2023.
[44] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference Machine Learning- Haarninn-VVlume 70, pages 3145-3153. JMLR. org, 2017.
[45] K Simonyan, A Vedaldi, and A Zisserman. Deep inside convolutional networks: visualising image classification models and saliency maps. In Proceedings of the International Conference on Learning Representations (ICLR). ICLR, 2014.
[46] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Cearning-Volunne 70, pages 3319 3328. JMLR. org, 2017.
[47] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-follloing llama model. https:// / cwt. waatsssllabssanttte cannford_llpcca 2023.
[48] Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy, July 2019. Association for Computational Linguistics. URL https:/ / /www://wwww.ooo aclweb. arg/anthology/P11 1452.
[49] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
[50] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback, 2022.
[51] Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation anal- ysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 12388-12401. Curran Asso- ciates, Inc., 2020. URL htttts:s//prooceddnngs .pdf. 2ale:926622222227775512222222223
ccppaper_ppp. lappprrffiiiaa www.ee/pppee//e///e
[52] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797- 5808, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.58v53/v1/ 1580. URL https: //tpa//tolaa/gggy.c/o///o hwwphhooogyoooogggggeee www.orggg//P..... Ptrg//P9-1188.
14

---

[53] Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards understanding chain-of-thowe ofothhttht prompting: An empirical study of what matters. arXiv preprini urXiv:2212.10001, 2022.
[54] Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. In Proceedings of the International Conference Learning Representations (ICLR). ICLR, 2023.
[55] Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instruc- tions, arXiv preprini arXiv:2212.00500 2022.
[56] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
[57] Tailin Wu, Takashi Maruyama, and Jure Leskovec. Learning to accelerate partial differential equations via latent global evolution. Advances in Neural Information Processing Systems, 36, 2022.
[58] Zhengxuan Wu, Karel D' o'Oosterlinck, Atticus Geiger, Amir Zur, and Christopher Potts. Causal Proxy Models for concept- based model explanations. Xii:22209.14279 2022. URL https:
/ //w.xiv.org/ab///...../ 2209. 14279.
[59] Zhengxuan Wu, Atticus Geiger, Joshua Rozner, Elisa Kreiss, Hanson Lu, Thomas Ieard, Christopher Potts, and Noah Goodman. Causal distillation for language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4288-4295, 2022.
[60] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 318-833. Springer, 2014.
15

---

A Appendix
A.1 Price Tagging Game Experiment
Price Tagging Game
Below is an instruction that describes a task, paired with an input that provides further context. Wrile a response that appropriately completes the request.
#hit Instruction:
Please say yes only it costs between [X.XX] and [X.XX] dollars, otherwise
### Input: [X.XX]
### Response: [Model Output]
Instruction template for the Price Tagging game. We have three variables inside the prompt: lower bound amount, upper bound amount, and the input amount where each number has two decimal points but bounded with [0.00, 9.99].
Figure 7:
Figure 7 shows our templates for our Price Tagging game Experiment. This format is enforced by the instruction tuning template of the Alpaca model.
A.2 Training Details
Training Setups During training for Boundless DAS, model weights are frozen except for the rotation matrix. To train the rotation matrix, we use Adam [26] batch size of 64 and learning rate of 10-3 We a different learning rate 10 for our boundary indices for quicker convergence. We train our rotation matrix for 3 epochs and evaluate with separate evaluation set for each 200 training steps, and we save the best rotation matrix evaluated on the evaluation set during training and test on another held-out testing set. Our training set has 20K examples. Our in-training evaluation limited 200 examples for quicker training time, and the hold-out testing dataset has 1K examples. These datasets are generated on the fly for different causal models (i.e., different random seeds result different training data). The rotation matrix is trained by using the orthogonalized parameterization provided by the torch library." For boundary learning, anneal temperature from 50.0 0.10 with step number equal the total training steps and temperature steps with gradient backpropagation Each alignment experiment is enabled with bf bfloat 6 and can fit within single A100 GPU. Each alignment experiment takes about 1.5 hrs. To finish all of our experiments, it takes roughly 2 weeks of run-time with x A100 nodes with 8 GPUs each. We run experiments with 3 distinct random seeds and report the best result using IIA as our metrics.
optimizer with effective
Vanilla Task Performance Remarks To ensure the model we aligning behaviorally achieves good task performance, we evaluate the model task performance with 1K sampled triples of values more than 300 times. The Alpaca (7B) model achieves 85% for randomly drawn triples of numbers on average. We want to emphasize that it is really hard to find a good reasoning task with intermediate steps (e.g., tasks like "Return yes of it costs more than 2.50 dollars" do require intermediate step) that presently available 7B-30B L.Mss can solve. We tested different reasoning tasks (e.g., math, logic, natural QA, code, etc.) with instruct-tuned LLaMA [49], Dolly, 5 StableLM [3] Vicuna [9]. Instruct-tuned GPT-J, and mostly struggled find tasks they could do well.
'We use the publicly available model at httpp//hnugiingeeeccrrr//o/ourrr which is GPT-J finetuned on Alpaca.
16

---

A.3 Counterfactual Dataset Generation
To train Boundless DAS, we need generate counterfactual datasets where each example contains pair of imputs as base and source, an intervention indicator (i.e., what variable we are intervening), and counterfactual label. For instance, if are training Boundless DAS for our Boundary Check (Left Only) high-level causal model, we may sample the base example with the bracket as [2.50, 7.50] and the input amount 1.50 dollars, and the source example with the bracket as [3.50, 8.50] and the input amount of 9.50 dollars. If we intervene on the variable whether the input variable is higher than the lower bound, and we swap the variable from the second example into the first example, we will have a counterfactual label as "Yes", For each high-level causal model, we follow the same steps to construct the counterfactual dataset.
A.4 Interchange Intervention Accuracy
Figure 8 to Figure ?? show a breakdown of the results presented in Table 1.
A.5 Pseudocode for Boundless DAS
Boundless DAS is generic framework and can be implemented for models with different architec- tures. Here we provide pseudocode snippet for a decoder-only model.
def signoid_ seriiiddbudddy__maaeee (population, boundary_x, boundary-y, temperature):
return signaid (popullation boundary_x) tenperature)
torch. ( population) temperature)
class AlignableDeccderlL (DecoderLL))))))
overriding existing forward function
def (ohierd(hideen_stttt
source suurce_hidee_ssttee
For idx, decoder_layer in en layers: if idx self. silf.aligning..ayrr..t
select aligning token reprs
11
bligning_hidden_stteee hidden_states[: sttrt..dddd.tttt.d # rotate
12
13
rotatedu_hiden_stttes cotste(@ligning_hhiee) [sattesstttes)
14
interchange intervention with learned baundaries
15
boundary.nask aaggmmi_ooundary_maa.
16
torch. (9, holffhidden_ssze)). self. boundaries. self. temperatue
17
18
19
20
rotated_hidden hid___iiddee_tteeeeseeeeesseeeee
boundary_nask) retated_hidden_state. boundary_mask source_hidden_state.
21
unrotate
23
Sedden_states___ statess: start. idx: bexx:een__ddx idx]
hidden_ states
25
return hidden_states
A simplified version of Boundless DAS with pseudocode for generic decoder-only model.
A.6 Generated Irrelevant Contexts
To generate random contexts in Table 2, we prompt GPT-40Aug05-202. to generate short English sentences. These sentences are added to the original prompt prefixes.
17

---

Table 2: Our prompt and twenty different contexts generated by GPT4-Aug05- as prefixes.
Our Prompt: I will generate random short sentences (they do not have to be in English), ranging from words to 15 words. Here are two examples in Hnglish: "Pricing tag game!" and "Fruitarian
Frogs May Be Doing Flowers Favor." Generate 100 examples for me and return them in .IxI file.
We are so back!
Kiwis fly in their dreams.
Frogs leap into detective work.
Jumping jacks on Jupiter?
Beans in my boots?
Clouds have fluffy sense of hu- mor.
Koalas start each day with long nap.
Dinosaurs loved good bedtime story.
Butterflies throw colorful prise parties.
Rhinos are experts at playing hide and seek.
Badgers hrew the finest coffee,
Peanuts have their own under- ground society.
Pigeons love rooftop cinema nights.
Fish run deep-sea detective agen- cics.
Cheetahs host slow-motion re- play parties.
Hamsters wheel design pents.
Doves write the world's best love songs.
Porcupines are actually very cud- dly.
Fireflies choreograph nightly light shows.
Ants host underground food festi- vals.
A.7
Common Questions
In this section, we answer common questions that may be raised while reading this report.
Can Boundless DAS work with 175B models?
Yes. it does. Taking the recently released Bloom- 11mom-17611 176B as an example, the hidden dimension is 14,336, which results in a rotation matrix containing 205M parameters. This is possible if the model sharded and the rotation matrix is put into single GPU for training. This is only possible given the fact that Boundless DAS is now implemented as a token-level alignment tool. If the aligmment for a whole layer (i.e., token representations are concatenated), it is currently impossible. It is worth noting that training the rotation matrix consumes more resources than training a simple weight matrix, as the orthogonalization process in torch requires matrix exponential, which means saving multiple copies of the matrix.
The current paper limits to a set of tasks from the same family. How well will the findings in the paper generalize to other scenarios?
Existing public models in 7B-30B range are still not effective in solving reasoning puzzles that require multiple steps. We test different reasoning tasks as mentioned in Section A.2. Once larger LLMs are released and evaluated as having stronger reasoning abilities, Boundless DAS will be ready as an analysis tool. Although our Price Tagging game task is simple, we test with different variants of the task and show the efficacy of our method.
Is it possible to apply Boundless DAS over the whole layer representation?
is possible for smaller models if we are learning a full-rank square rotation matrix. it intractable for Alpaca present. Let's say align 10 token representations together. We will have a total dimension size of 40,960. This leads to a rotation matrix containing 1.6B parameters. Given the current orthogonalization process in torch, this will require GPU with memory larger than the current A GPU just to fit in a trainable rotation matrix.
Only limited high-level models are tested with Boundless DAS in the experiments.
There is large number of high-level models solve the task, and we only test a portion of them. We think that this is sufficient to demonstrate the advantages of our pipeline. It would be interesting humans can be in the loop of testing different hypotheses of the high-level causal model future work. On the other hand, not all the high-level models are interesting to analyze. For instance, if two legel models are equivalent (e.g., swapping variable names), it not in our interest to find alignments.
18

---

Can Boundless DAS be applied to find circuits in model?
Yes, it can. We believe that "circuit analysis" can be theoretically grounded in causal abstraction and deeply compatible with our work. Previous work in this mode usually relies on the assumption of localist representation (i.c.. there exists a one-to-one mapping between group of neurons and a high-level concept). This is often too idcalized, multiple concepts can easily be aligned with the same group of neurons, or group of neurons may map to multiple concepts. Boundless DAS actually drops this assumption by specifically focusing on distributed alignments. More importantly, Boundless DAS can be easily used in a head-wise alignment search where we add shared rotation matrix on top of head representations of all the tokens.
19

---

Left and Right Boundary (Correct Only)
Right Boundary (Incorrect Only)
88 -00.50
0.660.55
0.50 0. 0.50 0.50 0. 0.50
0.85
0. 000.00000000000000...11110111100
0.71
Alpaca (7B) Layer Indox
Alpaca (7B) Index
0.51
0.500.69
0.69 0.64
0.64
0.56
0.51
0.55
0.570.1.
0.61 0.86
0.54 000.0000000000101111101100011110
0.51
12
0.50
0.740.00
0.80
0.52
0.65..72..
0.87
0.55 0.58
0.50
0.81
0.82
0.75
0.79
0.61
0.50
0.62
0.600.660.00
07
0.51
0.76 0. 0.500..60.
0.66 0.60 0.57 0.57 0.511 0.0.000500000000.000
0.53
0.551
0.69
0.64
0.64
0.64
0.59
0.69 0...6.4..58
0.64
0.58
0.52
0.53
0.53 0.66.
-..500.76 0,500.660.500
5.500.00..50
0.500.500.49..800
-0.500.65
0.52
0.0..80.500.00 0.0.00..500.22..522000
<<iiOA>> (69).
X (70)
(71)
(72)
X (73)
ddillrr (74)
(75).
"<00kt>> (76)
"a (77)
(78)
Rosponse (79).
(80)
<000AA> (81)
X (70)
(71)
X (72)
X (73)
<clooA>> (75)
<<x0A>>> (76)
W (78)
(79).
(80)
<catt (81)
ddillars'
*#
Tokon (Index)
Token (Index)
Figure 8: Interchange Intervention Accuracy (IIA) evaluated with correct input examples only as well
as incorrect input examples only for our "Left and Right Boundary" causal model. The higher the number is, the more faithful the alignment is. We color each cell by scaling IIA using the model's task performance as the upper bound and dummy classifier (predicting the most frequent label) as the lower bound.
8.49 dollars' (Seen)
between
dollars' (Unseen)
10.499 0.75 0.0.490.750.
70.00000000
0.50
0.
0.49
094 0.50 0.50
0.90
9-0.51
0.56 0.55
0.52
0.51
0.52
0.51
0.51 0.51 0.51 0.51 0.51 0.000.00
Alpaca (78) Layer Indox
(7B) Indox
0.670.64
2-0.52
2..449.... 0.86 a. 7...8.66
0.64
0.55
0.56
0.57
0.54
0.585.75
0.93
0.52
0.60 0.59
0.599.553
0.52
0.50 0.85
19
0.92 8-0.990,920
0.24..00.81
0.61
0.81
0.06.66...000
0.49
0.790.91 0.91
0.51
0.71
0.67
0.75
0.52
0.67
0.79
0.92
0.72
0.95
0.52
0.499..9200
0.499
0.92 0.91
0.89
0.74
0.59 0.490.490.490.500
0.51
0.790.760.78
0.000.000000
0.68
0.60
0.60
0.53
0.520.5.2
0.52
-0.49..92 4490.920.000
0.52 49..49999904430 49..099944444440 000.4440044944440004400400
0.77
0.55
0.520.520.0
0.510..51
0.51 1.51 0.51 0.52
FOXDAA (69)
X (70)
(71))
X (73)
"dollury' (74)
"<Ox0A>> (75)
<<0:0>>' (76)-
* (70)
Response" (79)
(80)))
<OXDA>> (81)
X (70)
X (72)
X (73)
"<0x0A>> (75)
"<0x0A>> (76)-
** (77)
W (78)
(79)
(800)
<DXDA>> (81)
X
'dollars'
Taken (Index)
Taken (Index)
Figure 9: Interchange Intervention Accuracy (IIA) evaluated with different settings of brackets the instruction. The seen setting is for training, and the unseen setting is for evaluation. The higher the number is, the more faithful the alignment is. We color each cell by scaling IIA using the model's task performance as the upper bound and dummy classifier (predicting the most frequent label) as the lower bound.
20

---

Prefix Context
'Fruitarian Progs May Be Doing Flowers
Prefix Context Prricing tag game!
Favor"
23 80
-0.51
0.66 0.8
0353 54 0.56 0.53 0.52 0.33 0.000
0.8
70 77 3
-0.52
0.62
62
5.53 0.300.53333300
0.52
0.83
Alpara (71)) Layer Indox
Index
1.0.51
0.85
0.640.0...
0.55 0.58
0.03 0.53 0.52
0.663
0.83
0.552
0.620.630000
0.600
0.53 0.533.53000003
0.83
15 70
-0.51
0.67
0.6
0.52
080.67.0
0.83
0.32
06660.61
0.56
0.52
0.71
0.71
0.68
0.
03.0.71
0.2.0...1..0000
0.70
0.53
0.65
0.81
0.52
0.720.72
0.
0.77
0.71
0.53
0.53
0.58
0.59
0.85
0,75
53...33333
0.360.560.6
Alpaca
0.52
0.76
0.78
0.69
0.73 0.69
0.55
0...5....333..
0.60
0.52
0.77
0.74 0.69
0.67
0.63
0.60
0.53
0.52
0.55
00.51
0.770.3
0.552
0.54 0.51 0.53 0.50..2 0.5.0.52...22. 0,520,52 02.00,52 2...52...55
-0.52
0.780.8....
0.53 0.530.3..5
0.520.51 0.520.52 0.52 0.50 0.52 0.55
'<OxIA>> (75)
X (76)
(17))
* (79)
ddllers" (80)
(81)
'<0x0A> (02)
50- (83)
" (04)
Response (85)
(8(0))
*x000x> (87)
(04)
X (85)
(00)
X (87)
* (88)
ddolless' (89)
<<0x0A>> (90)
<<0x0A> (91)
(92)
W (03)
(94)
(96)
<<DxAA> (96)
X
Token (Index)
Token (Index)
Figure 10: Interchange Intervention Accuracy (IIA evaluated with different irrelevant contexts inserted as prefixes. The higher the number is, the more faithful the alignment is. We color each cell by scaling IIA using the model's task performance as the upper bound and dummy classifier (predicting the most frequent label) as the lower bound.
Mean
Standard Deviation (SD)
30
0.57
50 00 0.81
Alpacn (78) Layer Index
(7B) Index
0.50 0.63 0.61 0.51
0.03
0.02 0.02 |1000000000000000000000000000000
20
0.50 006.4.6330000 0.54
0.84
00001 000.0000000000000000000000000000
0.82
000.0000000000000000000000000000
0. 0.50 7.7888888888888888 78 75
0.51 0.53..2...22
0.57
-0.01 0.01 0.01 00100000000001111000000001010010
0.77
0.67
50 0.50 0.59
0.59
0.01 0.02 0.000.0222200000 0...02....000000
0.02
0.75 0. 0.51
0.000.......
0.500 500.50 500.51 0.50
-0.02
11 0.02
0.01 0.020.022 0.02 0.0.01 0.01
(1)
12)
X (4)
Gollars (5)
<000AA> (6)
Ox (7)
(9)
Response (10)
1x0X0A (122
X (1)
X (3-
X (4)
dollars' (5)
<0X0A>> (6)
<0:0A> (7)
(9)
(10)
(1))))
<OSBA>> (122
Relative Token (Index)
Relative Token (Index)
Figure 11: Interchange Intervention Accuracy (IIA) evaluated with twenty different irrelevant
contexts generated by GPT4-Aug05-2022 Correlation is 0.99 with the "Left and Right Boundary"
causal model.
21

---

Left Boundary
Left and Right Boundary
30
0.51 0.51 01 0.51
50 0.0 500.55 0.62
0.51 0.51 0.51
01511
52
52
0.51 51 0.51 0.52
Alpaco (71)) Layer Indox
0.51 0.51 01
0.51 0.
0.51
0.62
Alpoca (71) Index
0.51 01010000000000111100000000011110
03
0.51 0:000
13
0.52
0. 01010000000111111111000000001110 0111000101..1.111111111111111111
0.50 0.51 0. 0...555..11 0.51 0.51 0.51 0.100000011111111100000001111110
0.52 0.53 011.1000000000000000000000111110
0.51 0.52 50..50 500.51 000.0.......0.001.00000000000000
0.51 0...0.51..300 0.51 0.530.55 0.52 0.2.00200..000000002011111133330 000.0000005000001000000000000010
Mid-point Distance
Bracket Identity
0.61
0161
0,600 60 60
Alpaca (7R) Layer Indix
Alpoca (7B) Layer Indax
0.62 0.61 0.61
0.61
0.61
0.61
0.61 0.62
0.6
0.61 0.56
0.60
0.60 0tt.:00000......0.....0000000000
0.61 0.62 0.61
0.62
0.6
0.62
0.620.62..00000
0.58
0.61
0.60 0.590 60 0...00..00.000000000000000:00000
10
0.62 0.63 0.61 0.62
0.61
0.66
0.620.666666
0.62
0.600.61 616666000 0. 0.600 0.0000 0.600..60 60 0.0.00...00000000000000000000000
0.62 0.661 0.62 0.61
0.620.61
0.62
1.82
0.62
0.62
0.60 0.60.0.59 0.60 0.60000 0.60 0.00000..0000
0.57
0.62
020
620.610 0.61
0.62
0.59 0. 600 600 60 60 6.00000..800
6.600.00000
<<0xiAA>> (69)
(70)
(73)
(73)
dodlurs (74)
<<b0tA (75)
<<0xA>> (76)
(78)
Rosponse (19)
>x0X0A>> (91)
(69)
(70)
(P3)
(72)
x (75)
<o000A>> (75)
<<000A> (76)
(77)
W (78)
(79)
(00)
<CONAA (91)
dollezs
Token (Index)
Token (Index)
Figure 12: Interchange Intervention Accuracy (IIA) with random initialized rotation matrix and
learned optimal boundary indices. We three random seeds.
22


Reviews:
[{"review_id": "zOf8YHVgsV", "reviewer": "Reviewer_iGjk", "summary": "This paper proposes a method for interpreting neural models by learning an alignment of their representations to a specified interpretable causal model. The method extends prior work, namely Distributed Alignment Search (DAS), by addressing a key limitation that prevents the scaling of DAS. The limitation is overcome by learning a set of masks that selects subsets of neural representations to align to different parts of the interpretable causal model. The method is evaluated experimentally by aligning Alpaca representations to a set of correct and incorrect hand-written causal models.", "questions": "1. What is the value of introducing variable $Z$? It looks like only $X$ are used after section 3.1\n2. Is $\\Pi$ on line 99 used anywhere else?\n3. In line 115, '$b_j$ is restricted to be a multiple of $b_{j-1}$ with a factor of 2'. However, $b_0 = 0$. What does this mean?\n4. Why are target variables $Z_j$ bold but high-level variables $X_j$ not?\n5. In Eq. (1), what is the difference between $F_N^*$ and $F_N$? Is $F_N$ defined?\n6. What is the problem that was brute-forced in DAS, that $M$ learns to solve? Section 3.1 would be the best place to highlight intractability.\n7. Line 37 would be another nice place to highlight limitations in scaling DAS. What is the largest dimensional representation it has been applied to?\n1. What is the runtime improvement due to the method?\n8. Line 216: \"expectation\"\n\nClarifying the shortcoming of DAS, section 3, and giving the computational complexity would convince me.", "limitations": "The limitations were adequately addressed.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "* The problem of interpreting large models is important.\n* The approach is reasonable and scales a previous approach.\n* The experimental results were convincing.", "weaknesses": "* Section 3 was hard to read. See the questions below.\n* Section 3 did not feel self-contained. The source of intractability in DAS did not jump out to me.\n* Giving the computational complexity for DAS and boundless DAS would make the efficiency argument stronger, unless they are the same.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We *thank* the reviewer's *great suggestions on detailing improvements* over DAS and **highlights the scalability that our method brings**. We will use the additional page to clarify our improvements and provide a detailed introduction to the DAS method [Geiger et. al., 2023]. Here, we address all concerns with point-by-point responses.\n\n> **Q1:** \"Section 3 did not feel self-contained, ..., Giving the computational complexity for DAS and boundless DAS would make the efficiency argument stronger.\"\n\n**A1:** We will revise our draft to make the improvements to DAS clearer, focusing on how the changes enable us to scale. In short, DAS learns a rotation matrix but requires manual search to determine how many neurons are needed to represent the aligning causal variable. Boundless DAS automatically learns boundaries (i.e., how many neurons are needed is determined by the “soft” boundary index via a boundary mask learning). For instance, given a representation with a dimension of 1024, DAS should in principle be run for all lengths k from 1 to 1024. In practice, this would be infeasible, so some subset of the lengths need to be chosen heuristically, which risks missing genuine structure. For Boundless DAS, we turn this search process into a mask learning process. See our comments below for more detailed comparison in terms of time complexity.\n\n> **Q2:** \"What is the value of introducing the variable $Z$? It looks like only $X$ are used after section 3.1\"\n\n**A2:** The variable $Z$ is only used to define the interchange intervention operator and the projection operator, and plays no \"contentful\" role in the definitions. $X$ is the original representation space. We will make this clearer! \n\n> **Q3:** \"Is $\\Pi$ on line 99 used anywhere else?\"\n\n**A3:** Thanks for catching this notational error. It should be just $\\tau$ representing the alignment mapping between a high-level variable and neural representations.\n\n> **Q4:** \"In line 115, $b_j$ is restricted to be a multiple of $b_{j-1}$ with a factor of 2'. However,. What $b_0$ does this mean?\"\n\n**A4:** Since b_j marks the boundaries here, b_0 is equal to 0 as the starting index of the boundary. Essentially, Eqn. 3 will generate a high-pass filter ($Mask_{s}$) between [$b_{j-1}$, $b_{j}$] and the gating factor within the bounds will be approximately 1.0 and approximately 0.0 outside the bound. We then apply this filter to the source representations, and the inverse filter (1.0 - $Mask_{s}$) to base representations before adding them up to get our \"soft-intervened\" representations. We will revise this section to be clearer in the next revision. \n\n> **Q5:** \"Why are target variables $Z_j$ bold but high-level variables not $X_j$?\"\n\n**A5:** The target variables $Z_j$ are sets of variables and the $X_j$ are variables. We wanted to indicate this type difference with the bold typeset. \n\n> **Q6:** \"In Eq. (1), what is the difference between $F_{N}^{*}$ and $F_N$ ? Is $F_N$ defined?\"\n\n**A6:** $F_{N}$ represents the causal mechanism of causal variable N before the distributed intervention, whereas $F_{N}^*$ represents the mechanism after the distributed intervention. For instance, $F_{N}(v)$ means calling the “rest” of the forward function (i.e., the causal mechanism of $F_N$) by setting $N$ with activations $v$. We will clarify $F_N$ and $F_{N}^{*}$ in our next revision.\n\n> **Q7:** \"What is the problem that was brute-forced in DAS, that M learns to solve? Section 3.1 would be the best place to highlight intractability.\"\n\n**A7:** Thanks again for raising this issue. This helps us to better ground our work and highlights our improvements. As we clarified above, Boundless DAS is approximately **O(N*m)** quicker where N is the number of population dimensions and m is the number of causal variables we are aligning. In the case of LLMs, it scales DAS to models with billions of parameters like Alpaca-7B and beyond.\n\n> **Q8:** \"Line 37 would be another nice place to highlight limitations in scaling DAS. What is the largest dimensional representation it has been applied to?\"\n\n**A8:** The largest dimension is BERT with a hidden dimension of 768 with two variables in DAS [Geiger et. al., 2023]. And the result in its original paper is also not guaranteed to be optimal since only a limited combination of dimension settings are tried. In this paper, each aligning representation is with a dimension of 4096 with two variables that are aligned simultaneously.\n\n> **Q9:** \"What is the runtime improvement due to the method?\"\n\n**A9:** It is **O(N*m)** quicker where N is the number of total dimensions and m is the number of causal variables we are aligning. Essentially, to brute-force search over different dimensionality for intervention on a token representation with a size of 4096, we need to scan through {1,2,3,...,4096} each covering a single data point for a single aligning causal variable depicting how many neurons we are swapping values for. With Boundless DAS, we only need to run it once.\n\n> **Q10:** \"Line 216: 'expectation'\"\n**A10:** Thanks! We will correct the typo in the next revision.\n\n> **Q11:** \"Clarifying the shortcoming of DAS, section 3, and giving the computational complexity would convince me.\"\n\n**A11:** Thanks for the great suggestions. We believe this concern is addressed in responses above! We will add the discussion around complexity improvements a bit in the next revision."}, {"review_id": "7n9UT6GFtc", "reviewer": "Reviewer_6cS3", "summary": "This paper introduces a method called Boundless DAS, an algorithm that can automatically test the \"alignment\" between human-specified symbolic algorithms and the computational structure found in the weights of a neural network. The authors apply Boundless DAS to a real-world 7B parameter language model called Alpaca and find evidence of the implementation of a particular symbolic algorithm that computes whether some number $q$ is in the range $[a, b]$.", "questions": "* What kinds of tasks have you tried DAS or Boundless DAS on? Do you ever find gradient descent struggling to find a faithful solution? Do you have intuitions for what kinds of tasks this works well on, and what it doesn't? How about algorithms with lots of input variables, or complex ones requiring lots of intermediate ones?\n* I'm trying to get a sense for how far we can push Boundless DAS; what do you think? For any challenges you foresee, what do you think the right next steps are?", "limitations": "n/a", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Really cool paper!\n* I like the idea of using gradient descent to create a correspondence between a symbolic algorithm and a set of neural network features. Continuous optimization feels a little easier to tackle (initially) than discrete combinatorial stuff.\n* It's impressive that this works on a real-world, billion-parameter model.\n* While the tricks to go from DAS to Boundless DAS seem reasonable and slick, my favorite part is the empirical section — inspecting the alignments that the algorithm discovers, looking for generalization, etc. Well done.", "weaknesses": "Not much here. If I had to say anything, the paper is quite dense and hard to follow. For newcomers, it requires somewhat extensive background reading on causal abstraction and the previous DAS work(s). Otherwise, the writing is fairly clear, ideas are well-motivated, claims are well-supported and analyzed, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We *thank* the reviewer's *inspiring comments about future works* that can be built on top of Boundless DAS. We **provide more discussions about possible future extensions**, and we will revise our paper to reflect these additional thoughts. Here, we address all concerns with point-by-point responses.\n\n> **Q1:** \"it requires somewhat extensive background reading on causal abstraction and the previous DAS work(s).\"\n\n**A1:** Thanks for the comment! We plan to **use the additional page to add an introduction to DAS** [Geiger et. al., 2023], and add **a full detailed appendix on DAS** to provide readers more context about the background, and how Boundless DAS improves upon it.\n\n> **Q2:** \"What kinds of tasks have you tried DAS or Boundless DAS on? Do you ever find gradient descent struggling to find a faithful solution? Do you have intuitions for what kinds of tasks this works well on, and what it doesn't? How about algorithms with lots of input variables, or complex ones requiring lots of intermediate ones?\"\n\n**A2:** **The original DAS paper [Geiger et. al., 2023] provides a set of examples including synthetic logic inference task as well natural language inference (NLI) task with smaller models.** Earlier causal abstraction ideas have been applied to other NLI problems, ground language understanding tasks, and structured image tasks. We haven’t tried Boundless DAS on most of these, but Boundless DAS is a scalable and more general approach, so such applications would yield as good or better results.\n\n**For the current work, we wanted to focus on tasks that LLMs solve well with no fine-tuning.** The bottleneck for us to try other tasks is, given our compute and time limit, we struggled to find a publicly-available instruction-following model that solved interesting  reasoning tasks in a zero-shot fashion (Appendix A.2). We did find failure cases such as those control conditions mentioned in the paper! These control models are interpretable models but we fail to find clear evidence of alignments. It has so far not been the case that gradient descent gets stuck in local optima, though this is certainly possible. \n\n**On aligning more complex causal graphs: Yes! We are working on it right now.** In this paper, we are only aligning individual causal variables, or a high-level model with a single layer of abstraction. The next step is to align a full, multilayer causal graph with a neural model, and evaluate the faithfulness of the alignment more rigorously. \n\n> **Q3:** \"I'm trying to get a sense for how far we can push Boundless DAS, …, what do you think the right next steps are?\"\n\n**A3:** **The biggest obstacle we see is the need to hypothesize a high-level causal model.** All explainability methods require a hypothesis of some sort, but often the burden seems smaller and so is less of a blocker to people running the analysis (e.g., for probing, one picks a supervised learning task; for feature attribution, one picks a set of neurons to analyze). We would like to  automate the process of proposing high-level models combined with Boundless DAS to generate a faithfulness hypothesis or circuit that is realized by the model. \n\nAnother interesting challenge would be **applying mechanistic interpretability tools to controlled language generation.** Thus far, we have only applied Boundless DAS to a single token or fixed length language generation tasks. It would be interesting to see how a token-level intervention through Boundless DAS causally affect the iterative decoding process where the generation length is unbounded.\n\nOn the application side, a good counterfactual alignment (even though it is not a structural alignment) helps us steer the model in a robust way. One future step we are thinking to take is to align, for instance, story genre variables in the LLMs and **steer the model at the inference time** to generate different genres given a prompt starter."}, {"review_id": "Z0Xpo2olEl", "reviewer": "Reviewer_SQSS", "summary": "This work proposes an extension to Distributed Alignment Search (DAS) called \"Boundless DAS\" that replace the brute-force search with learned parameters in order to scale the method to large language models (LLMs).\nThe method evaluated on the Alpaca 7B model tasked to output \"yes\" or \"no\" to a simple numerical problem such as \"is X between Y and Z?\".\nExperimental analysis showed that the method is able to scale and find an alignment with two potential causal mechanisms.\nFurther experiments also show that the method is robust to slight modifications in the prompt of the model (additional sentence in the input, change of output format). ", "questions": "- Out of four possible causal mechanisms, two seem more plausible, but it is not clear which one is the one used by the model: \"left boundary\" or \"left and right boundary\"? Can we identify which causal method is the most likely being used, or is it the case that the model can use one or the other depending on the input?", "limitations": "Some limitations are mentioned.\nHowever, one limitation of this work that should be mentioned is the relatively narrow scope of the target task being evaluated. This may be ok if the authors discuss how to transfer their method to more \"realistic\" scenarios in which interpretability is required (such as the medical, legal, or financial fields). Another possibility is that it is challenging or impractical to apply the same type of analysis in more complex scenarios. Either way, the paper could benefit from this type of discussion.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This work proposes a new explainability method that is scalable to LLMs, which is an important research question as the field quickly progresses towards larger models that become part of real-life products. Hence, research to understand the inner mechanisms of LLMs is very important.\n\nThe proposed method is well-tested on various input/output perturbations of the target task, showing the robustness of Boundless DAS.", "weaknesses": "- the technical aspect of the paper is challenging to understand for someone that is not familiar with the field of causality\n\n- the scope of the task used to evaluate the method is very narrow. Some discussion on how this can be applied to more critical scenarios where interpretability is required (such as the medical, financial, or legal domains) could improve the paper.\n\n- It is mentioned that a good causal model could also explain the errors of the system. However, the current method does not explain why the model fails in certain examples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We *thank* the reviewer's *thoughtful comments*. We **address the major concern around the applicability of Boundless DAS as well as how to interpret alignment results** where multiple causal models reach good interchange intervention accuracy (IIA). Here, we address all concerns with point-by-point responses.\n\n> **Q1:** \"the technical aspect of the paper is challenging to understand for someone that is not familiar with the field of causality\"\n\n**A1:** Thanks for your comment! Given the page limit, we did not include a detailed introduction to DAS [Geiger et. al., 2023]. **In the next revision, we will provide a full detailed discussion of DAS** in the appendix in the next revision to ground Boundless DAS. We will **sharpen the notation in Section 3**. We also want to note that we provide a full pseudo-code implementation of Boundless DAS in Appendix A.5; this may provide additional clarifications.\n\n> **Q2:** \"the scope of the task used to evaluate the method is very narrow.\"\n\n**A2:** While it is true that we only analyze the Alpaca model with a very specific task, **the goal of this paper is to present an alignment search method that works well with LLMs such as Alpaca-7B and beyond**. We see no obstacle to applying the methods for additional models and tasks. We will release our codebase that scales with any decoder-only model. The main bottleneck for us to try other tasks with Alpaca is that it is hard to find reasoning tasks that Alpaca does well. We outlined some of our task finding procedures in Appendix A.2. For domain experts working with stronger models, this kind of exploration might be easier.\n\n> **Q3:** \"the current method does not explain why the model fails in certain examples.\"\n\n**A3:** **This is a good point and suggests great next steps!** At present, errors could be explored by seeing how their computations do not conform to the high-level model under investigation. One could also hypothesize high-level causal models of the errors and assess them as explanations for the underlying problem. \n\n> **Q4:** \"Out of four possible causal mechanisms, two seem more plausible, but it is not clear which one is the one used by the model: \"left boundary\" or \"left and right boundary\"?\"\n\n**A4:** In this case, aligning “left and right boundary” actually **entails** aligning “left boundary” since the latter is one of the subspace (i.e., the subspace for the left boundary) of the previous causal model. In other words, **the “left boundary” causal model can be seen as an abstraction of the “left and right boundary” causal model** but not the other way around. This also means our model implements both checks instead of one. In a case where we can only align the “left boundary” but not both, we would then observe lower IIA for \"left and right boundary\"."}, {"review_id": "lwHgY2SeUD", "reviewer": "Reviewer_ExBe", "summary": "This paper builds on Distributed Alignment Search by introducing a more efficient learned model to search for a given circuit in a model.  They test this method on a specific task: asking the model whether a given numeral falls between two other given numerals. They have a single task, and find an alignment with 2 possible circuits for accomplishing this task and fail to find a good alignment to 2 other handcrafted circuits, demonstrating that this method may be capable of distinguishing between correct and incorrect circuits.\n\nEDITED to raise soundness score.", "questions": "Why would baseline IIA be 50%? You are actively aligning a circuit to the model, so shouldn't you be able to find an alignment that performs above 50% even in a randomly weighted model?\n\nWhat is a target variable $Z_j$?\n\nWhat if $F_N$?\n\nPlease explain the boundary index variable in more detail.\n\nWhen you restrict $b_j$ to be \"a multiple of $b_{j\u0000-1}$ with a factor of 2\", do you mean that $b_j = 2 b_{j-1}$ or that it could be any multiple as long as it is also even? Why would you use the notation in this way, rather than specify the variable if it is deterministic?\n\nIs it likely that we could have a very high IIA score without having more structured patterns? Rather than validating the IIA score, the observation of sparser patterns seem to strongly correlated with the score by definition, so I don't entirely trust it as a verification of the faithfulness of the alignment.\n\nWhat is an intermediate boolean value?\n\nHow large is the task dataset used to learn the alignment? Is it different from the set used to test IIA for the alignment? \n\n\"We sample 100 experiment runs from our experiment pool and create two groups\" Do the runs differ only by the input used? Or by some random seed?\n\nIn section 4.6, what does it mean to look at learned boundary width? What does a higher width indicate?\n\nAre the training steps (Figure 5 x axis) training steps for learning the alignment, or for the training of the LM? The number of steps is small so I assume it is for the alignment. What do we learn from the learning curve, that we didn't get from the final values? ", "limitations": "The authors acknowledge that their method is limited by the fact that not finding an alignment is not evidence that there is no alignment to the given circuit, which is very good. However, the limitations of the experiments themselves are not acknowledged.\n\nThis method only applies to synthetic tasks where we know the possible algorithms. This isn't explicitly acknowledged.\n\n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The approach of boundless DAS itself seems to be a worthwhile one, and I believe that it could work.\n\nThe approach scales well, up to Alpaca, and therefore could be very useful in modern models.\n\nI'm impressed by the approach of carefully constructing possible circuits to test, and it seems to work surprisingly well in LLM settings.\n\nThe observation that the IIA patterns are more structured for the seemingly correct circuits is an interesting one, though I have some reservations about this as a verification method (in questions).", "weaknesses": "An obvious criticism of the method itself is that we lose a degree of interpretability by introducing learned parameters into the interpretation method. How many learned parameters does it take before you’ve just introduced another blackbox as your lens? If the problem is that you don’t trust learned parameters to behave interpretable, how can you ensure that you are behaving interpretably? In particular, it doesn’t seem to get away from the complaint that the authors themselves introduce: failing to find an alignment does not mean that the alignment doesn’t exist. This is a mild criticism, and I still consider the method to be potentially useful. \n\nMore significantly, the experiments are not convincing. There is only a single task evaluated. They consider only a single model, Alpaca. There are only four possible circuits considered. All of these factors add up to a fairly unconvincing set of experiments which have limited statistical power. Outside of the main experiments, these issues become more apparent: section 4.5 tests generalization from a fixed input to a different fixed input, with only a single sample considered for each case. A test of generalization with inserted context only considers two different possible contexts. Overall, these experiments are too constrained with too few samples to be convincing; while I don't think that this is cherry picked, results that comprise of case study of only a couple of samples could easily lead to publication bias.\n\nFurthermore, for the price task, the effectiveness of the alignment is measured by IIA, but the baseline for IIA is based on an assumption of complete randomness that I do not think is valid. After all, we have a method that has already found an alignment that maximizes the match with the circuit. I would trust this more if you had, e.g., tested it on random weights or weights trained for a completely different structured task that wasn't actually language modeling.\n\nOn another note, 4.5 appears to be the method that they should be using as the main experiment. I don't trust results that train the boundary parameters and alignments on the same inputs that they test on, which appears to be the approach taken in 4.3.\n\nThe math could be more carefully explained, with more precise definitions for the variables involved and more detailed intuitions for the method. One way to help is to turn Figure 1 into a step-by-step algorithm diagram.\n\nMinor:\n- \"Boundary\" is an overloaded term (used in both the method and the task). You should pick different wording.\n- \"We find that Alpaca achieves near-perfect task performance because it implements a simple algorithm with interpretable variables.\" *Because* is a *very* strong claim. Do they actually support the idea that the simple algorithm is the reason for the task performance? Is performance worse for tasks that don’t have simple algorithms? They do not show either piece of evidence.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We *greatly appreciate* the reviewer’s in-depth feedback.  \n\n> **Q1:** \"how can you ensure that Boundless DAS is behaving interpretably? ... failing to find an alignment does not mean that the alignment doesn’t exist.\"\n\n**A1:** These are **incisive questions that are worthy of broader discussion**. A few thoughts: **It seems like we have to accept that neural models might store information in a *highly distributed fashion*, as *Smolensky and others* anticipated long ago. Our method is a sort of minimal response, as it allows only a limited kind of representation. **The issue of false negatives is important.** With our method, we can at least search vastly more hypotheses than any prior method, reducing the risks here. \n\n> **Q2:** \"Overall, these experiments are not convincing.\"\n\n**A2:** **While we only evaluate with Alpaca-7B and on a single type of task, our method scales to larger open-source LLMs.** Moreover, **the original paper of DAS tried DAS with a set of tasks with different models and scales. Boundless DAS is strictly more scalable and expressive.\n\n**We ran new experiments with 20 random context generated by GPT-4@Aug05-2023** to validate one of our generalization tests with inserted context in Section 4.5. **The overall correlation of the mean IIA with random contexts inserted with our vanilla “left and right boundary” causal model is 0.99.** These results greatly strengthen our claims about alignment generalization.We want to emphasize that it was **unexpected** to find alignments that generalize given the fact that we freeze the rotation matrix and boundary indices at test time.** \n\n> **Q3:** \"tested it on random weights or weights trained for a completely different structured task\"\n\n**A3:** **Finding alignments on a model that could not perform the task would be uninformative**: if the LM head is randomly initialized, there is little chance of getting the output tokens space correct (e.g., {True, False} or {Yes, No}).\n\n**Our new results suggest good IIAs do not come for free.** **As shown **in Figure 1 of the attached pdf**, all positions drop significantly with random rotation matrix. These results calibrate IIAs in case of unbalanced counterfactual labels (e.g., each control causal model can reach about **0.60** IIA with a random rotation matrix). \n\n> **Q4:** \"I don't trust results that train the boundary parameters and alignments on the same inputs that they test on.\"\n\n**A4:** Alignments and boundary parameters **are tested with unseen counterfactual pairs for all the sections**. Section 4.5 is more extreme: the inputs are not only unseen but with distribution shifts.\n\n> **Q5:** \"I think that the math could be more carefully explained\", \"'Boundary' is an overloaded term\"\n\n**A5:** We will provide a clearer description.\n\n> **Q6:** \"'We find that Alpaca achieves near-perfect task performance because it implements a simple algorithm with interpretable variables.' *Because* is a very strong claim\"\n\n**A6:** We agree that **the sentence should be clarified** to remove any ambiguity. The Boundless DAS analysis is a causal explanation for the model's success, and that is why we used \"because\". However, we need to clarify that there may be other explanations that highlight different factors in the model's success, as we saw in our analysis of alternative causal models.\n\n> **Q7:** \"shouldn't you be able to find an alignment that performs above 50% even in a randomly weighted model?\"\n\n**A7:** If a neural model performs at random with respect to a given high-level model which solves the task, we can know a priori that there is not a causal abstraction relation between the neural model and the high-level model.\n\n> **Q8:** \"What is a target variable $Z_j$?”\n\n**A8:** The target variables $Z_j$ are sets of variables and the $X_j$ are variables.\n\n> **Q9:** \"What is $F_N$?\"\n\n**A9:** $F_{N}$ represents the causal mechanism of causal variable N before the distributed intervention, whereas $F_{N}^*$ represents the mechanism after the distributed intervention. \n\n> **Q10:** \"Please explain the boundary index variable in more detail.\"\n\n**A10:** The boundary index variable marks how many neurons the intervention will use at the end. We penalize larger indexes since we want to push the intervention site to be smaller. \n\n> **Q11:** \" do you mean that $b_j$ = 2*$b_{j-1}$ or that it could be any multiple as long as it is also even? Why would you use the notation in this way?\"\n\n**A11:** We restrict $b_j$ to be \"a multiple of $b_{j-1}$ with **a factor of 2 for simplicity**.\n\n> **Q12:** \"Is it likely that we could have a very high IIA score without having more structured patterns?\"\n\n**A12:** **We observed that but would not make any conclusion regarding high IIA score and structured patterns** in Figure 4.\n \n> **Q13:** \"What is an intermediate boolean value?\"\n\n**A13:** In our case study, it is a latent boolean value that is abstracted to check whether the input price is above the lower bracket or below the higher bracket.\n\n> **Q14:** \"How large is the task dataset used to learn the alignment? Is it different from the set used to test IIA for the alignment?\"\n\n**A14:** We random sample 20K counterfactual pairs to train the alignment. **Test is disjoint from train.** \n\n> **Q15:** \"'We sample 100 experiment runs from our experiment pool and create two groups' Do the runs differ only by the input used? Or by some random seed?\"\n\n**A15:** **These runs are sampled from our experiment pool with different inputs and different random seeds. \n\n> **Q16:** \"In section 4.6, what does it mean to look at learned boundary width? What does a higher width indicate?” \"Are the training steps (Figure 5 x axis) training steps for learning the alignment?\"\n\n**A16:** Boundary width maps to how many neurons are needed in the rotated space to represent the causal variable we are aligning. Higher width meaning we need more neurons to represent the variable. Fig. 5 x-axis represents the training step for alignment."}, {"review_id": "cMJLH4FNpq", "reviewer": "Reviewer_AVCz", "summary": "Finding alignments between variables in a user-hypothesized causal model for how a task could (or should) be performed and representations in neural networks, termed \"causal abstraction\", is a mechanistic interpretability technique for understanding how neural networks develop predictions at a granular level by using causal interventions.  Prior work has recently proposed the Distributed Alignment Search algorithm for efficiently and robustly finding this alignment, but it still relies on brute-force search over the dimensionality of neural representations. This paper proposes to circumvent the brute force search by *learning* parameters of an orthogonal rotation matrix to maximize the alignment between a neural network representation's linear subspace and one of the variables in the high-level causal model.\n\nThey test their method on one rather narrowly scoped task, which is determining if a price is between two numbers, and use the Alpaca (instruction-tuned LLaMA) 7B decoder-only language model. The authors also test whether the found alignments generalize across settings (both input prompts and output token choices for labels), and find that they do.\n\nEdit: I read the authors' rebuttal and the other reviews & discussions. I did not change my score as only one of the weaknesses (writing/presentation) was alleviated. I still think the paper should be accepted though.", "questions": "- On finding strong alignments between Alpaca’s representations and two of the proposed causal models (\"left boundary\",\"left and right boundary\")— if the method reveals a faithful abstraction, wouldn't that imply that only one abstraction can have strong alignments with model weights, rather than multiple? Can you explain?\n- On writing clarification in Section 3.1-- it would be useful to establish early on that \"variables\" refers to hidden representations in a NN and \"algorithm\" to a high-level user-specified symbolic model. Relatedly, the terms \"input variables\" and \"target variables\" are a bit confusing as you also have input sequences (\"base input\", \"source inputs\") and target model outputs, which are different.\n-  I’m not sure I would use the term “largest” to describe a 7B model.\n- You use the phrase “human-interpretable” to describe the explanations that DAS produces; it would be good to provide a definition or at least some citations of what this concretely means.\n- Lines 170-171 and x-axis in Figure 4: these refer to the same thing; it would be good to give a concrete in-line example in the text as you have done for the core instruction.\n- Will you open-source your code?\n\nSome other mathematical notation could be improved. For example, \n- what do $j$ and $k$ represent? (I'm assuming $k$ is the number of dataset instances and $j$ the instance being indexed, but this is unstated). $k$ appears doubly-defined in Eqn. 3.\n- Set notation is mixed between uppercase variables and brackets\n- $\\mathbf{s}$ and $\\mathbf{b}$ are not vectors but sequences of inputs (i.e., tokens), which is a bit nontraditional notation; introduction of $\\textbf{Inputs}$ seems as though it should have been introduced earlier.\n- both $\\mathcal{N}$ and $\\mathcal{M}$ represent a neural network model, both \"algorithm $\\mathcal{A}$\" and \"model $\\mathcal{C}$\" (which is never used after defining) represent a high-level causal model, if I understand correctly.\n- no definition of $F$, $F^*$, $\\Pi$,  $L$\n- I did not follow the connection between target variables $X$, $\\mathbf{Z}$, and $\\mathbf{N}$.\n- $y_p$ and $y$ are more traditionally notated as $\\hat y$ and $y^*$; and should keep indexing consistent as $\\hat y_i$ and $y^*_i$", "limitations": "Yes- sufficiently detailed & comprehensive limitations section.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Originality:\n- The application of causal abstraction (or even other variants, like causal mediation analysis/causal tracing) to large instruction-tuned LMs has not been done; this is highly original to the best of my knowledge.\n\nQuality & Clarity:\n- The experiments seem sound. The analysis in sections 4.3 - 4.6 is insightful and interesting. The overall findings of the paper are useful to a larger community for understanding math processing in LMs.\n- The writing is clear overall.\n- Related works section is comprehensive.\n\nSignificance:\n- Causal analysis of language models at the representation-level is a very important goal with potentially far-reaching impacts for the broader NLP community. Using instruction-tuned models that are in widespread use today, such as Alpaca 7B, bridges the gap between theory and practice and makes this work more applicable to a wider audience.", "weaknesses": "- The paper does not read independently (particularly section 3.1) to a reader unfamiliar with prior work. I had to refer extensively to [32] to grasp what the various terms and variables used meant, and to understand what components of the proposed algorithm are novel w.r.t prior work. If there is not space in the main paper, the appendix should be used to thoroughly explain these concepts in a camera-ready version. The math notation could be improved; see \"questions\" section.\n- The method involves pre-registering some possible symbolic models for performing a task; it cannot discover novel symbolic models that are not specified by the user.\n- The method is tested on one narrowly-scoped task (determining if a price is between two numbers) and one model. There is no way to \"validate\" a mechanistic interpretability method beyond principle or axioms as we don't have the ground truth, so some of the findings are more suggestive rather than extremely concrete conclusions. However, this is a common problem in the subfield and I think this paper is convincing overall.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Thanks* for the useful suggestions. **They lead us to define more clearly how to interpret our results, and existing limitations** of Boundless DAS. \n\n> **Q1:** \"The paper does not read independently (particularly section 3.1) to a reader unfamiliar with prior work.”\n\n**A1:** We plan to use the extra page to add background on DAS and add a detailed appendix on DAS.\n\n> **Q2:** \"The method involves pre-registering some possible symbolic models for performing a task\"\n\n**A2:** Yes, the framework verifies whether a causal model can be aligned with representations. Future works may look at how to automatically create hypotheses for Boundless DAS evaluation.\n\n> **Q3:** \"The method is tested on one narrowly-scoped task (determining if a price is between two numbers) and one model.\"\n\n**A3:** Thanks for the comments. While we only work with the Alpaca-7B and a single task, we provide clear evidence that **Boundless DAS has the potential to scale to many other tasks and LLMs**. At the time of this project, the main bottleneck is the model’s ability to solve simple reasoning tasks as described in Appendix A.2.. \n\nWe agree that the causal mechanism that is uncovered with Boundless DAS may not be the most fine-grained explanation. However, **the found mechanism grounded by our IIA metrics**, can be used to faithfully steer a model's behavior at the inference time and can generalize well as we have tried.\n\n> **Q4:** \"if the method reveals a faithful abstraction, wouldn't that imply that only one abstraction can have strong alignments?\"\n\n**A4:** **Not necessarily.** In our case, aligning “left and right boundary” entails aligning “left boundary” since the latter is one of the subspaces of the former. This also means our model implements both instead of one. In a case where we can only align the “left boundary” but not both, we would then observe lower IIA for \"left and right boundary\".\n\n> **Q5:** \"terms like\"input variables\" and \"target variables\" are confusing”\n\n**A5:** Thanks! Here is a list of terms we will clarify in the next revision.\n\n- *Input variables:* These are essentially the input settings, or counterfactual example pairs that we use to train Boundless DAS. \n- *Target variables:* These are the hidden variables within the neural networks that we try to find the alignments for.\n\n> **Q6:** \"I’m not sure I would use the term “largest” to describe a 7B model.”\n\n**A6:** We agree it is not the largest LLMs. **We will revise the sentence** to be clearer. We do, though, think that the method scales to our largest models, provided one has the hardware and compute budget (and access) to run those models!\n\n> **Q7:** \"You use the phrase “human-interpretable” to describe the explanations that DAS produces; it would be good to provide a definition or at least some citations”\n\n**A7:** We will do this, thanks! We can draw more extensively on papers we cite: Lipton 2018, Geiger et al. 2020, Feder et al. 2021, and others\n\n> **Q8:** \"Lines 170-171 and x-axis in Figure 4: these refer to the same thing; it would be good to give a concrete in-line example in the text as you have done for the core instruction.\"\n\n**A8:** Thanks! We will provide a concrete example.\n\n> **Q9:** \"Will you open-source your code?\"\n\n**A9:** Yes!.\n\n> **Q10:** \"Some other mathematical notation could be improved. For example, what do  $j$ and $k$ represent?”\n\n**A10:** You are right! We overload the notation of $j$ and $k$ to represent training instances and aligning variables. We will clarify in our methods section.\n\n> **Q11:** \"Set notation is mixed between uppercase variables and brackets\"\n\n**A11:** Thanks– we will adjust the notation in the next revision.\n\n> **Q12:** \"$s$ and $b$ are not vectors but sequences of inputs (i.e., tokens), which is a bit nontraditional notation; introduction of Inputs seems as though it should have been introduced earlier.\"\n\n**A12:** Correct. We will make it clearer and use different symbols for sampled source and base inputs.\n\n> **Q13:** \"Both $N$ and $M$ represent a neural network model, both 'algorithm' and 'model' (which is never used after defining) represent a high-level causal model, if I understand correctly.”\n\n**A13:** Right. $A$ is our high-level causal model as defined in Line 90. We use $A$ to train our Boundless DAS as in Eqn. 2, where the second part is the counterfactual output as if we were intervening on $A$. We will clarify this further.\n\n> **Q14:** \"no definition of $F$, $F^*$, $\\Pi$, $L$\"\n\n**A14:** $F_N$ represents the causal mechanism of causal variable N before the distributed intervention, whereas $F_N^*$ represents the mechanism after the distributed intervention. For instance, $F_{N}(v)$ means calling the “rest” of the forward function (i.e., the causal mechanism of $F_N$) by setting $N$ with activations $v$. We will clarify $F_N$ and $F_N^*$ in our next revision. $\\Pi$ can not be removed from Line 99, we will correct that to use only $\\tau$ to describe our alignments.\n\n> **Q15:** \"I did not follow the connection between target variables $X$, $Z$ and $N$\"\n\n**A15:** $X$ is the input, $N$ represents intervened representations in the original basis, $Y$ is the representation after basis change (e.g., via rotation matrix), and $Z$ is a subspace of $Y$ meaning the actual neurons defined by our trainable boundaries we are intervening on the $Y$ space.\n\n> **Q16:** \"$Y_p$ and $y$ are more traditionally notated as $\\hat{y}$ and $y*$\"\n\n**A16:** We use $Y$ to denote a different thing: $Y$ represents our intervening subspace in the neural representation. We will use a different symbol in the next revision to clarify."}, {"review_id": "yRlv6Fz02d", "reviewer": "Reviewer_8NXa", "summary": "This paper introduces an advancement of Distributed Alignment Search (DAS), termed Boundless DAS, where the brute-force search steps are replaced with learned parameters. This enhancement enables efficient exploration of interpretable causal structures in large language models and therefore results in a scalable interpretable model.", "questions": "Please refer to Weaknesses.", "limitations": "The authors have discussed the limitations of their work.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "1) The increasing significance of LLMs necessitates an evaluation of their interpretability. Developing a scalable method to achieve this goal is undeniably crucial and intriguing.\n2) Current interpretability methods are inadequate for LLMs due to scalability issues, rendering them impractical. In contrast, the proposed method offers a solution that enables the interpretation of real-world LLMs, filling this crucial gap.", "weaknesses": "1) My primary concern lies with the novelty of the paper, which appears to be exceedingly limited. The paper seems to be primarily an extension of DAS and a direct amalgamation of two existing works, namely DAS and neural PDE [58], without addressing any specific challenges or introducing significant innovative elements.\n2) The paper heavily relies on the DAS method as its foundation, and it is imperative to provide a comprehensive explanation of DAS, including its limitations, which are currently absent in the paper. A thorough understanding of DAS and its shortcomings is crucial to fully comprehend the context and motivation of the proposed research.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Thanks* for your insightful comments and it **leads us to picture more clearly the improvement we have done compared to DAS [Geiger et. al., 2023] and why is that important**. We further clarify our main contributions besides simply proposing a new method in finding alignments. Here, we address all concerns with point-by-point responses.\n\n> **Q1:** \"The paper seems to be primarily an extension of DAS without addressing any specific challenges\"\n\n**A1:** **Boundless DAS addresses the challenge of scaling causal explanation methods** to the scale of Alpaca (and beyond, assuming one has the compute budget for the work!) **This is significant** because prior methods do not scale to this level, which means that they do not apply to the most relevant present-day models.\n\nBoundless DAS achieves this scalability by removing essentially all aspects of manual search that limited previous methods. The relevant parameters are now learned directly. The resulting optimization problem is straightforward, which we regard as a virtue.\n\nThis is also **the first paper (as far as we know) to offer a causally grounded explanation of instruction-following LLMs, with numerous robustness checks.** \n\n> **Q2:** \"A thorough understanding of DAS and its shortcomings is crucial\"\n\n**A2:** Thanks for the suggestions. Given an additional page in the next revision, **we will provide a detailed background introduction to DAS [Geiger et. al., 2023]**. In addition, we will add a thorough discussion of DAS in the appendix in our next revision. And we will provide a comparison of time complexity between these two methods."}]

Author rebuttal:
We thank the reviewers for their incisive comments and questions. In this comment, we summarize our overall response to major points, emphasizing new experiments. This is followed by point-by-point responses.

1. **On the question of novelty and utility**, we note that Boundless DAS is the only method in this class of causal explanation methods that can even be applied at the scale of Alpaca or above. This is a phase change in terms of what is possible -- after all, models this size and larger are the most relevant to the field right now.

1. We also want to take this opportunity to clarify **what Boundless DAS or DAS [Geiger et. al., 2023] could explain.** Having a good alignment means that the high-level model is a causal abstraction of the neural network, in terms of both factual and counterfactual behaviors. The two models can have different structure, and the network may be abstracted by many other causal models as well. Different causal abstractions highlight different explanatory aspects of model behavior.

1. **How to interpret the results when both causal models "left boundary" and the "left and right boundary" find strong evidence in Section 4.3?** The second model strictly extends the first, but the network may be implementing both solutions. The strong IIA scores show that one can reason about either high-level model and not be misled. We elaborate this point in our responses to *Reviewer SQSS* and *Reviewer AVCz*.

1. **Reviewers note that we report experiments only for Alpaca.** We do try to broaden this in Appendix A.2 to other models in this class, but we find publicly accessible models often fail to solve these reasoning tasks robustly. We argue that our experiments achieve our primary goal of showing that causal explanation methods are effective at this scale. We have a limited compute budget (and limited space in the paper), but we've shown that Boundless DAS can easily scale beyond Alpaca when budget allows. We will also release our code that is compatible with any decoder-only model.

1. **Building on the above: the original DAS paper [Geiger et. al., 2023] reports on a number of other experiments, and prior work on causal abstraction includes even more diverse experiments.** Boundless DAS is a more scalable and general version of all these methods, and so we think all this accumulated evidence ultimately supports the applicability of Boundless DAS for causal explanations across many models and tasks.

1. **New experiments are added to validate our main results in Section 4.3.** The reviewers asked for more baselines to more fully contextualize our results. To this end, we report IIAs with the same settings with random rotation matrices in the attached pdf. (We are not able to explore randomly initialized LMs because IIA is ill-defined if the model is not able to perform the task.) As shown in **Figure 1 of the attached pdf**, IIA score drops from **0.83** to **0.53** at layer 10, token position 75 for our “left and right boundary” causal model. Other positions drop significantly as well. These results also help us to calibrate IIAs in case of unbalanced labels (e.g., our two control causal models reach **0.60** and **0.61** IIA at the same location with a random rotation matrix). These results suggest that good IIAs for our first two causal models do not come for free. We will include these baseline IIAs in the next revision to better contextualize our findings.

1. **New experiments are added to validate our alignment generalization ability in Section 4.5.** Reviewer ExBe asked about tightening up the series of experiment-based arguments we offer in Section 4.5. To this end, we greatly expanded the range of inputs we consider, using **20 random contexts generated by GPT-4@Aug05-2023** with our “left and right boundary” causal model. Mean IIAs along with prompts we used are included in the attached pdf. As shown in **Figure 2 of the attached pdf**, the averaged IIA at layer 10, 6-th token position (relative since we have prefixes at different lengths) is **0.83** with a standard deviation of 0.02. This aligns with two prompts we reported in the paper (**0.80** for the first, and **0.84** for the second prompt as shown in Figure 9 of the Appendix). The overall correlation of IIAs with our vanilla “left and right boundary” causal model is **0.99**. These results *greatly strengthen* our claims about alignment generalization. We will update current results with these new prompts in the next revision.

1. **We focus on success cases for Alpaca, but we could use the method to understand errors.** This would proceed in the same mode as in the paper: one would hypothesize a source of the errors and use Boundless DAS to assess the idea. Reviewer SQSS suggested this and we are eager to explore it in future work.

1. **On writings and notations clarifications and suggestions:** The reviewers offered numerous suggestions for **improving terminology and notation**, and improving the paper overall. We are grateful for all this input and will make use of it in our next revision.

Meta-review: Meta Review for Interpretability at Scale: Identifying Causal Mechanisms in Alpaca

As reviewer iGjk wrote, this work proposes a method for interpreting neural models by learning an alignment of their representations to a specified interpretable causal model. The method extends prior work, namely Distributed Alignment Search (DAS), by addressing a key limitation that prevents the scaling of DAS. The limitation is overcome by learning a set of masks that selects subsets of neural representations to align to different parts of the interpretable causal model. The method is evaluated experimentally by aligning Alpaca representations to a set of correct and incorrect hand-written causal models.

Many reviewers commented on the strength of this work, as reviewer SQSS wrote, presents a explainability method that is scalable to LLMs, which is an important research question as the field quickly progresses towards larger models that become part of real-life products. Hence, research to understand the inner mechanisms of LLMs is very important.

There are no clear very negative issues or weaknesses. The reviewer's feedback has been incorporated into the manuscript, and I believe it will be a paper that will offer good value and interest to the NeurIPS community.
Decision: Accept (poster)