SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Existing public models in 7B-30B range are still not effective in solving reasoning puzzles that require multiple steps. We test different reasoning tasks as mentioned in Section A.2. Once larger LLMs are released and evaluated as having stronger reasoning abilities, Boundless DAS will be ready as an analysis tool. Although our Price Tagging game task is simple, we test with different variants of the task and show the efficacy of our method.
Is it possible to apply Boundless DAS over the whole layer representation?
It is possible for smaller models if we are learning a full-rank square rotation matrix. It is intractable for Alpaca at present. Let's say we align 10 token representations together. We will have a total dimension size of 40,960. This leads to a rotation matrix containing 1.6B parameters. Given the current orthogonalization process in torch, this will require a GPU with memory larger than the current A100 GPU just to fit in a trainable rotation matrix.
Only limited high-level models are tested with Boundless DAS in the experiments.
There is a large number of high-level models to solve the task, and we only test a portion of them. We think that this is sufficient to demonstrate the advantages of our pipeline. It would be interesting if humans can be in the loop of testing different hypotheses of the high-level causal model in future work. On the other hand, not all the high-level models are interesting to analyze. For instance, if two high-level models are equivalent (e.g., swapping variable names), it is not in our interest to find alignments.
Can Boundless DAS be applied to find circuits in a model?
Yes, it can. We believe that "circuit analysis" can be theoretically grounded in causal abstraction and is deeply compatible with our work. Previous work in this mode usually relies on the assumption of localist representation (i.e., there exists a one-to-one mapping between a group of neurons and a high-level concept). This is often too idealized, as multiple concepts can easily be aligned with the same group of neurons, or a group of neurons may map to multiple concepts. Boundless DAS actually drops this assumption by specifically focusing on distributed alignments. More importantly, Boundless DAS can be easily used in a head-wise alignment search where we add a shared rotation matrix on top of head representations of all the tokens.
Figure 8: Interchange Intervention Accuracy (IIA) evaluated with correct input examples only as well as incorrect input examples only for our "Left and Right Boundary" causal model. The higher the number is, the more faithful the alignment is. We color each cell by scaling IIA using the model's task performance as the upper bound and a dummy classifier (predicting the most frequent label) as the lower bound.
Figure 9: Interchange Intervention Accuracy (IIA) evaluated with different settings of brackets in the instruction. The seen setting is for training, and the unseen setting is for evaluation. The higher the number is, the more faithful the alignment is. We color each cell by scaling IIA using the model's task performance as the upper bound and a dummy classifier (predicting the most frequent label) as the lower bound.
Figure 10: Interchange Intervention Accuracy (IIA) evaluated with two different irrelevant contexts inserted as prefixes. The higher the number is, the more faithful the alignment is. We color each cell by scaling IIA using the model's task performance as the upper bound and a dummy classifier (predicting the most frequent label) as the lower bound.
Figure 11: Interchange Intervention Accuracy (IIA) evaluated with twenty different irrelevant contexts generated by GPT4-Aug05-2023. Correlation is 0.99 with the "Left and Right Boundary" causal model.
Figure 12: Interchange Intervention Accuracy (IIA) with random initialized rotation matrix and learned optimal boundary indices. We run three random seeds.