SlowGuess/NeurIPS_2023_OCR huggingface dataset:
These findings suggest that, when solving our reasoning task, Alpaca internally follows our first two high-level models by representing causal variables that align with boundary checks for both left and right boundaries. Interestingly, heatmaps on the top two rows also show a pattern of higher scores around the bottom left and upper right and close to zero scores in other positions. This is significantly different from the other two alignments where, although some positions are highlighted (e.g., the representations for the last token), all the positions receive non-zero scores. In short, the accurate hypotheses correspond to highly structured IIA patterns, and the inaccurate ones do not.
Token (Index)
Token (Index)
Figure 4: Interchange Intervention Accuracy (IIA) for four different alignment proposals. The Alpaca model achieves  $85\%$  task accuracy. The higher the number is, the more faithful the alignment is. We color each cell by scaling IIA using the model's task performance as the upper bound and a dummy classifier (predicting the most frequent label) as the lower bound. These results indicate that the top two are highly accurate hypotheses about how Alpaca solves the task, whereas the bottom two are inaccurate in this sense. Analyzing tokens includes special tokens (e.g.,  $\langle 0 \times 0 \rangle$  for linebreaks) required by Alpaca's instruct-tuning template as provided in Appendix A.1. Heatmap color uses min-max standardized IIA using the random rotation baseline as the lower bound, and task performance as the upper bound.
Additionally, alignments are better when the model post-processes the query input with 1-2 additional steps: accuracy is higher at position 75 compared to all previous positions. Surprisingly, there exist bridging tokens (positions 76-79) where accuracy suddenly drops compared to earlier tokens, which suggests that these representations have weaker causal effects on the model output. In other words, boundary-check variables are fully extracted around position 75, level 10, and are later copied into activations for the final token before responding. By comparing the heatmaps on the top row, our findings suggest that aligning multiple variables at the same time poses a harder alignment process, in that it lowers scores slightly across multiple positions.
4.4 Interchange Interventions with (In-)Correct Inputs
IIA is highly constrained by task performance, and thus we expect it to be much lower for inputs that the model gets wrong. To verify this, we constructed an evaluation set containing 1K examples that the model gets wrong and evaluated our 'Left and Right Boundary' hypothesis on this subset. Table 1 reports these results in terms of max IIA and the correlation of the IIA values with those obtained in our main experiments. As expected, IIA drops significantly. However, two things stand out: IIA is far above task performance (which is 0.0 by design now), and the correlation with the original IIA map remains high. These findings suggest that the model is using the same internal mechanisms to process these cases, and so it seems possible the model is narrowly missing the correct output predictions.
We also expect IIA to be higher if we focus only on cases that the model gets correct. Table 1 confirms this expectation using 'Left and Right Boundary' as representative.  $\mathrm{IIA}_{\mathrm{max}}$  is improved from 0.86 to 0.88, and the correlation with the main results is essentially perfect. Full heatmaps for these analyses are given in Appendix A.4.
Experiment	Task Acc.	IIAmax	Correlation
Left Boundary (♣)	0.85	0.90	1.00
Left and Right Boundary (♥)	0.85	0.86	1.00
Mid-point Distance	0.85	0.70	1.00
Bracket Identity	0.85	0.72	1.00
Correct Only	1.00†	0.88	0.99 (♥)
Incorrect Only	0.00†	0.71	0.84 (♥)
New Bracket (Seen)	0.94	0.94	0.97 (♣)
New Bracket (Unseen)	0.95	0.95	0.94 (♣)
Irrelevant Contexts	0.84	0.83	0.99 (♥)
Sibling Instructions	0.84	0.83	0.87 (♥)
+ exclude top right	0.84	0.83	0.92 (♥)
Table 1: Summary results for all experiments with task performance as accuracy (range [0, 1]), maximal interchange intervention accuracy (IIA) (range [0, 1]) across all positions and layers, Pearson correlations of IIA between two distributions (compared to  $\clubsuit$  or  $\heartsuit$ ; range  $[-1, 1]$ ). This is empirical task performance on the evaluation dataset for this experiment.
Figure 5: Interchange Intervention Accuracy (IIA) evaluated with different output formats. The seen setting is for training, and the unseen setting is for evaluation.
4.5 Do Alignments Robustly Generalize to Unseen Instructions and Inputs?
One might worry that our positive results are highly dependent on the specific input-output pairs we have chosen. We now seek to address this concern by asking whether the causal roles (i.e., alignments) found using Boundless DAS in one setting are preserved in new settings. This is crucial, as it tells how robustly the causal model is realized in the neural network.
Generalizing Across Two Different Instructions Here, we assess whether the learned alignments for the 'Left Boundary' causal model transfer between different specific price brackets in the instruction. To do this, we retrain Boundless DAS for the high-level model with a fixed instruction that says "between 5.49 dollars and 8.49 dollars". Then, we fix the learned rotation matrix and evaluate with another instruction that says "between 2.51 dollars and 5.51 dollars". For both, Alpaca is successful at the task, with around  $94\%$  accuracy. Our hypothesis is that if the found alignment of the high-level variable is robust, it should transfer between these two settings, as the aligning variable is a boolean-type variable which is potentially agnostic to the specific comparison price.
Table 1 gives our findings in the 'New Bracket' rows. Boundless DAS is able to find a good alignment for the training bracket with an  $\mathrm{IIA}_{\mathrm{max}}$  that is about the same as the task performance at  $94\%$ . For our unseen bracket, the alignments also hold up extremely well, with no drop in  $\mathrm{IIA}_{\mathrm{max}}$ . For both cases, the found alignments also highly correlate with the counterpart of our main experiment.
Generalizing with Inserted Context Recent work has shown that language models are sensitive to irrelevant context [43], which suggests that found alignments may overfit to a set of fixed contexts.
Figure 6: Learned boundary width for intervention site and in-training evaluation interchange intervention accuracy (IIA) for two groups of data: (1) aligned group where the boundary does not shrink to 0 at the end of the training; (2) unaligned group where the boundary does shrink to 0 at the end of the training. 1 on the y-axis means either  $100\%$  accuracy for IIA, or the variable is occupying half of the hidden representation for the boundary width.
To address this concern, we add prefix strings to the input instructions and evaluate how 'Left and Right Boundary' alignment transfers. We thus generate 20 random prefixes using GPT-4.<sup>3</sup> All of our prefixes, and our method for generating them, can be found in Appendix A.6. The model achieves  $84\%$  task performance with random contexts added as prefixes, which is slightly lower than the average task performance. Nonetheless, the results in Table 1 suggest the found alignments transfer surprisingly well here, with only  $2\%$  drop in  $\mathrm{IIA}_{\mathrm{max}}$ . Meanwhile, the mean IIA distribution across positions and layers is highly correlated with our base experiment. Thus, our method seems to identify causal structure that is robust to changes in irrelevant task details and position in the input string.
Generalizing to Modified Outputs We further test whether the alignments found for instructions with a template "Say yes ..., otherwise no" can generalize to a new instruction with a template "Say True ..., otherwise False". If there are indeed latent representations of our aligning causal variables, the found alignments should persist and should be agnostic to the output format.
Table 1 shows that this holds: the learned alignments actually transfer across these two sibling instructions, with a minimum  $1\%$  drop in  $\mathrm{IIA}_{\mathrm{max}}$ . In addition, the correlation increases  $6\%$  when excluding the top right corner (top 3 layers in the last position), as seen in the final row of the table. This result is further substantiated in Figure 5, where we see that IIA drops near the top right. This indicates a late fusion of output classes and working representations, leading to different computations for the new instruction only close to the output.
4.6 Boundary Learning Dynamics
Boundless DAS automatically learns the intervention site boundaries (Section 3.2). It is important to confirm that we do not give up alignment accuracy by optimizing boundaries, and to explore the dimensionality actually needed to represent the abstract variables. We sample 100 experiment runs from our experiment pool and create two groups: (1) the success group where the boundary does not shrink to 0 at the end of the training; (2) the failure group where the boundary does shrink to 0 at the end of the training. The failure group is for those instances where there is no significant causal role of representations found by our method. We plot how boundary width and in-training evaluation IIA vary throughout training. Figure 6 shows that, in the success cases, each aligned variable only needs  $5 - 10\%$  of the representation space. IIA performance converges quickly for success cases and, crucially, maintains stable performance despite shrinking dimensionality. This suggests that only a small fraction of the whole representation space is needed for aligning a causal variable, and that these representations can be efficiently identified by Boundless DAS.
<sup>3</sup>We generate these prefixes using the public chat interface https://chat.openai.com/.
4.7 Metric Calibration
One concern brought up by our reviewers is whether our metric, interchange intervention accuracy (IIA), is well calibrated. Here, we provide additional evidence. We report IIAs in the same way as in Figure 4 but with random rotation matrices. With a random rotation matrix, we find that the IIA score drops from 0.83 to 0.53 at layer 10, token position 75, for our "left and right boundary" causal model. Other positions drop significantly as well. These results calibrate IIAs in case of unbalanced labels (e.g., our two control causal models reach about 0.60 IIA at the same location with a random rotation matrix). This shows that Boundless DAS finds substantial structure in Alpaca as compared to what these random baselines can find. Details can be found in Figure 12 in the Appendix.
We can also help calibrate our result by comparing Boundless DAS applied to Alpaca with a randomly initialized LLaMA-7B. This can help quantify the extent to which DAS leverage random causal structure in order to overfit to its training objective. This random model results in near  $0\%$  task performance on our task, as expected. After running Boundless DAS on the first layer of this random LLaMA-7B model, the found alignments to each token's representation ranged from  $0\%$  to  $69\%$  IIA, which is comparable to a most frequent label dummy model  $(66\%)$ . For the representations with near chance IIA, this means that we were able to find a distributed neural representation that shifts the probability mass to "yes" or "no", but does not differentiate between the two. We also found that we could find a distributed neural representation that shifts the probability mass to the tokens "dog" and "give" instead of "yes" and "no". More importantly, we found that IIA drops to  $0\%$  for this run on all of our robustness checks from the paper. In addition, it drops to  $0\%$  if we use it on another randomly initialized LLaMA-7B model with a different random seed. Overall, it seems that the causal structure we did find has arisen by chance, and we might expect very large models to be more prone to such occurrences. Our robustness checks definitively clarify the situation.
5 Analytic Strengths and Limitations
Explanation methods for AI should be judged by the degree to which they can, in principle, identify models' true internal causal mechanisms. If we reach  $100\%$  IIA for a high-level model using Boundless DAS we can assert that we have positively identified a correct causal structure in the model (though perhaps not the only correct abstraction). This follows directly from the formal results of Geiger et al. [22]. On the other hand, failure to find causal structure with Boundless DAS is not a proof that such structure is missing, for two reasons. First, Boundless DAS explores a very flexible and large hypothesis space, but it is not completely exhaustive. For instance, a set of variables represented in a highly non-linear way might be missed. Second, we are limited by the space of causal models we think to test ourselves.
For models where task accuracy is below  $100\%$ , it is still possible for IIA to reach  $100\%$  if the high-level causal model explains errors of the low-level model. However, in cases like those studied here, where our high-level model matches the idealized task but our language model does not completely do so, we expect Boundless DAS to find only partial support for causal structure even if we have identified the ideal set of hypotheses and searched optimally. This too follows from the results of Geiger et al. [22]. However, as we have seen in the results above, Boundless DAS can find structure even where the model's task performance is low, since task performance can be shaped by factors like a suboptimal generation method or the rigidity of the assessment metric. Future work must tighten this connection by modeling errors in the language model in more detail.
We introduce Boundless DAS, a novel and effective method for scaling causal analysis of LLMs to billions of parameters. Using Boundless DAS, we find that Alpaca, off-the-shelf, solves a simple numerical reasoning problem in a human-interpretable way. Additionally, we address one of the main concerns around interpretability tools developed for LLMs – whether found alignments generalize across different settings. We rigorously study this by evaluating found alignments under several changes to inputs and instructions. Our findings indicate robust and interpretable algorithmic structure. Our framework is generic for any LLMs and is released to the public. We hope that this marks a step forward in terms of understanding the internal causal mechanisms behind the massive LLMs that are at the center of so much work in AI.
[53] Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards understanding chain-of-thought prompting: An empirical study of what matters. arXiv preprint arXiv:2212.10001, 2022.
[54] Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. In Proceedings of the International Conference on Learning Representations (ICLR). ICLR, 2023.
[55] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022.
[56] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
[57] Tailin Wu, Takashi Maruyama, and Jure Leskovec. Learning to accelerate partial differential equations via latent global evolution. Advances in Neural Information Processing Systems, 36, 2022.
[58] Zhengxuan Wu, Karel D'Oosterlinck, Atticus Geiger, Amir Zur, and Christopher Potts. Causal Proxy Models for concept-based model explanations. arXiv:2209.14279, 2022. URL https://arxiv.org/abs/2209.14279.
[59] Zhengxuan Wu, Atticus Geiger, Joshua Rozner, Elisa Kreiss, Hanson Lu, Thomas Icard, Christopher Potts, and Noah Goodman. Causal distillation for language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4288-4295, 2022.
[60] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-833. Springer, 2014.
A.1 Price Tagging Game Experiment
Price Tagging Game
Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
[X.XX]
[Model Output]
Figure 7: Instruction template for the Price Tagging game. We have three variables inside the prompt: lower bound amount, upper bound amount, and the input amount where each number has two decimal points but bounded with [0.00, 9.99].
Figure 7 shows our templates for our Price Tagging game Experiment. This format is enforced by the instruction tuning template of the Alpaca model.
A.2 Training Details
Training Setups During training for Boundless DAS, model weights are frozen except for the rotation matrix. To train the rotation matrix, we use Adam [26] as our optimizer with an effective batch size of 64 and a learning rate of  $10^{-3}$ . We use a different learning rate  $10^{-2}$  for our boundary indices for quicker convergence. We train our rotation matrix for 3 epochs and evaluate with a separate evaluation set for each 200 training steps, and we save the best rotation matrix evaluated on the evaluation set during training and test on another held-out testing set. Our training set has 20K examples. Our in-training evaluation is limited to 200 examples for quicker training time, and the hold-out testing dataset has 1K examples. These datasets are generated on the fly for different causal models (i.e., different random seeds result in different training data). The rotation matrix is trained by using the orthogonalized parameterization provided by the torch library. For boundary learning, we anneal temperature from 50.0 to 0.10 with a step number equal to the total training steps and temperature steps with gradient backpropagation. Each alignment experiment is enabled with bfloat 16 and can fit within a single A100 GPU. Each alignment experiment takes about 1.5 hrs. To finish all of our experiments, it takes roughly 2 weeks of run-time with  $2 \times$  A100 nodes with 8 GPUs each. We run experiments with 3 distinct random seeds and report the best result using IIA as our metrics.
Vanilla Task Performance Remarks To ensure the model we are aligning behaviorally achieves good task performance, we evaluate the model task performance with 1K sampled triples of values more than 300 times. The Alpaca (7B) model achieves  $85\%$  for randomly drawn triples of numbers on average. We want to emphasize that it is really hard to find a good reasoning task with intermediate steps (e.g., tasks like "Return yes of it costs more than 2.50 dollars" do not require an intermediate step) that presently available 7B-30B LLMs can solve. We tested different reasoning tasks (e.g., math, logic, natural QA, code, etc.) with instruct-tuned LLaMA [49], Dolly, $^{5}$ , StableLM [3] Vicuna [9], Instruct-tuned GPT-J, $^{6}$  and mostly struggled to find tasks they could do well.
<sup>4</sup>pytorch.org/docs/stable/generated/torch(nn.utils parametrizations.orthogonal.html
huggingface.co/databricks/dolly-v2-12b
<sup>6</sup>We use the publicly available model at https://huggingface.co/nlpcloud/instruct-gpt-j-fp16 which is GPT-J finetuned on Alpaca.
A.3 Counterfactual Dataset Generation
To train Boundless DAS, we need to generate counterfactual datasets where each example contains a pair of inputs as base and source, an intervention indicator (i.e., what variable we are intervening), and a counterfactual label. For instance, if we are training Boundless DAS for our Boundary Check (Left Only) high-level causal model, we may sample the base example with the bracket as [2.50, 7.50] and the input amount of 1.50 dollars, and the source example with the bracket as [3.50, 8.50] and the input amount of 9.50 dollars. If we intervene on the variable whether the input variable is higher than the lower bound, and we swap the variable from the second example into the first example, we will have a counterfactual label as "Yes". For each high-level causal model, we follow the same steps to construct the counterfactual dataset.
A.4 Interchange Intervention Accuracy
Figure 8 to Figure ?? show a breakdown of the results presented in Table 1.
A.5 Pseudocode for Boundless DAS
Boundless DAS is a generic framework and can be implemented for models with different architectures. Here we provide a pseudocode snippet for a decoder-only model.
def sigmoid_bound_mask(population, boundary_x, boundary_y, temperature):  
return torchsigmoid((population - boundary_x) / temperature) * \torchsigmoid((boundary_y - population) / temperature)  
class AlignableDecoderLLM(DecoderLLM):  
# overriding existing forward function  
def forward(hidden_states, source_hidden_states):  
for idx, decoder_layer in enumerate(self.layers):  
if idx == self aligning_layeridx:  
# select aligning token reprs  
aligning-hidden_states = hidden_states[:, start_idx: end_idx]  
# rotate  
rotated Hidden States = rotate(alignment Hidden States)  
# interchange intervention with learned boundaries  
boundary_mask = sigmoid_bound_mask( torch.arange(0, self隐藏_size), selfLearnedboundaries, self°Ftemperature)  
)  
rotated Hidden States = (1. - boundary_mask) * rotated Hidden States + boundary_mask * source_hidden_states  
# unrotate  
hidden_states[:, start_idx: end_idx] = unrotate(rotated Hidden States)  
hidden_states = self.layers[idx](hidden_states)  
return hidden_states
A simplified version of Boundless DAS with pseudocode for a generic decoder-only model.
A.6 Generated Irrelevant Contexts
To generate random contexts in Table 2, we prompt GPT-4@Aug05-2023 to generate short English sentences. These sentences are added to the original prompt as prefixes.
Table 2: Our prompt and twenty different contexts generated by GPT4-Aug05-2023 as prefixes.
Our Prompt: I will generate random short sentences (they do not have to be in English), ranging from 3 words to 15 words. Here are two examples in English: "Pricing tag game!" and "Fruitarian Frogs May Be Doing Flowers a Favor." Generate 100 examples for me and return them in a .txt file.
We are so back!	Kiwis fly in their dreams.	Frogs leap into detective work.
Jumping jacks on Jupiter?	Beans in my boots?	Clouds have a fluffy sense of humor.
Koalas start each day with a long nap.	Dinosaurs loved a good bedtime story.	Butterflies throw colorful surprise parties.
Rhinos are experts at playing hide and seek.	Badgers brew the finest coffee.	Peanuts have their own underground society.
Pigeons love rooftop cinema nights.	Fish run deep-sea detective agencies.	Cheetahs host slow-motion re-play parties.
Hamsters are wheel design ex-perts.	Doves write the world's best love songs.	Porcupines are actually very cuddly.
Fireflies choreograph nightly light shows.	Ants host underground food festi-vals.
A.7 Common Questions
In this section, we answer common questions that may be raised while reading this report.
Can Boundless DAS work with 175B models?
Yes, it does. Taking the recently released Bloom-176B as an example, the hidden dimension is 14,336, which results in a rotation matrix containing 205M parameters. This is possible if the model is sharded and the rotation matrix is put into a single GPU for training. This is only possible given the fact that Boundless DAS is now implemented as a token-level alignment tool. If the alignment is for a whole layer (i.e., token representations are concatenated), it is currently impossible. It is worth noting that training the rotation matrix consumes more resources than training a simple weight matrix, as the orthogonalization process in torch requires matrix exponential, which means saving multiple copies of the matrix.
The current paper limits to a set of tasks from the same family. How well will the findings in the paper generalize to other scenarios?