Sparse Attention Decomposition Applied to Circuit Tracing

Authors: Gabriel Franco, Mark Crovella
Primary area: interpretability and explainable AI
Keywords: Mechanistic Interpretability, Transformers, Large Language Models, Interpretability, Singular Value Decomposition

Abstract:
Many papers have shown that attention heads work in conjunction with each other to perform complex tasks. It's frequently assumed that communication between attention heads is via the addition of specific features to token residuals. 
In this work we seek to isolate and identify the features used to effect communication and coordination among attention heads in GPT-2 small.  Our key leverage on the problem is to show that these features are very often sparsely coded in the singular vectors of attention head matrices.  We characterize the dimensionality and occurrence of these signals across the attention heads in GPT-2 small when used for the Indirect Object Identification (IOI) task. The sparse encoding of signals, as provided by attention head singular vectors, allows for efficient separation of signals from the residual background and straightforward identification of communication paths between attention heads. We explore the effectiveness of this approach by tracing portions of the circuits used in the IOI task.  Our traces reveal considerable detail not present in previous studies, shedding light on the nature of redundant paths present in GPT-2. And our traces go beyond previous work by identifying features used to communicate between attention heads when performing IOI.

Decision: reject
Reviewer scores: 5, 6, 3

Reviews and discussion:
<thread>
creation date:
1730479537979

thread_type:
official_review

<thread_object>
creator: reviewer_cg35

<content>
summary:
This work presents an approach for analyzing communication between attention heads in transformer models using SVD. It is shown that attention scores are often sparsely encoded in singular vectors of attention head matrices, enabling efficient isolation of features used for inter-head communication. The method is validated on GPT-2 small using the Indirect Object Identification (IOI) task, showing the redundant pathways and communication mechanisms between attention heads.

strengths:
The main contribution of this paper lies in introducing a more scalable approach for interpreting the information flow within Transformers. Specifically,
- The use of SVD in circuit tracing seems simple and effective. 
- The paper identifies new functionally important components (e.g., attention head 2,8) and provides a detailed analysis of redundant pathways in the model.
- Overall, the paper is well-structured and written.

weaknesses:
The paper can be improved in several major aspects. 
- The technical novelty seems limited. The idea of using dimensionality reduction (SVD in particular) to interpret and visualize models is not new. 
- The study focuses on the attention layers. Do the MLP layers and layer normalization contribute to the change of causality relationships from layer to layer?
- The analysis is limited to a specific model GPT-2 small and a specific task (IOI). How do the findings generalize to other settings?
- Most of the findings are empirical. It's suggested to also explore why sparse decomposition occurs. 
- Some of the study designs require further justification. For instance, the 70% threshold for filtering contributions seems arbitrary. Also, more justification is needed for the signal/noise separation approach.
- More discussion of failure cases where sparse decomposition might not hold would be valuable.
- The intervention focuses mainly on single-edge and simple multi-edge cases. What about more complex cases that involve multiple edges?
- There is limited comparison with other circuit analysis methods.

questions:
See the detailed comments above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650627641

thread_type:
official_review

<thread_object>
creator: reviewer_ibeq

<content>
summary:
This paper introduces a method based on sparse attention decomposition for analyzing the communication and coordination among attention heads in Transformer models. By constructing attention scores sparsely in a new basis through Singular Value Decomposition (SVD), we identify key communication paths between attention heads within the model. Experiments demonstrate that the communication paths identified through sparse decomposition have a practical causal effect on model functionality, enhancing the model's interpretability and offering new insights for understanding and improving Transformer models.

strengths:
1, Novelty: This paper addresses the previously challenging issue of identifying and interpreting the complex interactions between attention heads in Transformer models by proposing a novel SVD-based sparse decomposition method.

2, Interpretability: The paper uncovers the communication pathways between attention heads in Transformer models, enhancing researchers' understanding of the model's internal workings.

weaknesses:
1, Possible computational complexity: The computation of SVD and sparse decomposition is usually very complex and requires a lot of computing resources. What is the computational complexity and computing resources consumed in this paper? Have the factors related to computational complexity and required computing resources been considered?

2, Lack of open source code: The author should provide source code to facilitate others to reproduce and verify.

questions:
The computation of SVD and sparse decomposition is usually very complex and requires a lot of computing resources. What is the computational complexity and computing resources consumed in this paper? Have the factors related to computational complexity and required computing resources been considered?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730678848317

thread_type:
official_review

<thread_object>
creator: reviewer_v5fk

<content>
summary:
The authors explored some of the technical details of GPT-2 through SPARSE ATTENTION DECOMPOSITION. Their tracing study reveals considerable detail not present in previous studies, shedding light on the nature of redundant paths present in GPT-2. Their traces go beyond previous work by identifying features used to communicate between attention heads when performing IOI.

strengths:
(1) The theoretical proofs in this paper are remarkable.
(2) The figures and tables in the paper are visually appealing, which enhances readability to a certain paper.
(3) The related work and literature survey are adequate and well organized.

weaknesses:
(1) My big concern is that the paper may be technically obsolete. More mainstream experiments are now being conducted in GPT 4o and GPT o1-based settings. I don't understand why the authors are still conducting experiments on GPT 2. The gap between GPT 2 and GPT 4o and GPT o1-based methods is huge, so I think the experiments and the motivation are very limited, and the techniques in the paper may not be valid for the GPT 4 and GPT o1-based settings.
(2) The writing of the article is obscure. Maybe this article is hard to understand and follow. Reading through the entire paper, I'm not sure what the focus of the article FOCUSED on.
(3) The topics “SPARSE ATTENTION DECOMPOSITION” and “ Circuit Tracing ” did not attract widespread interest, and the importance of this area was not emphasized.
(4) In sum, our contributions are twofold. First, we draw attention to the fact that attention scores are typically sparsely decomposable given the right basis. This has significant implications for the interpretability of model activations. Why? The authors' experiments did not prove their interpretability.
(5) The paper was not compared to multiple state-of-the-art BASELINE methods, so there is insufficient validation of its effectiveness. For example, no quantitative comparison results can be seen in Figures 1, 2, and 3.
(6)The interpretability of the paper is assessed by the "Contribution" to attn. score"?

questions:
See the  Weaknesses.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents an approach for analyzing communication between attention heads in transformer models using SVD, but reviewers raised several critical concerns. Reviewers noted the limitation of the analysis, which is confined to GPT-2 and does not consider other mainstream models. Additionally, the paper lacks a comparison with other circuit analysis methods. Given these issues, I recommend rejection.