Dissecting Query-Key Interaction in Vision Transformers
Authors: Xu Pan
Aaron Philip
Ziqian Xie
Odelia Schwartz
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Vision transformer
Interpretability
Contextualization
Self-attention
Primary area: interpretability_and_explainability

Abstract: Self-attention in vision transformers is often thought to perform perceptual grouping where tokens attend to other tokens with similar embeddings, which could correspond to semantically similar features of an object. However, attending to dissimilar tokens can be beneficial by providing contextual information. We propose to analyze the query-key interaction by the singular value decomposition of the interaction matrix (i.e. ${\textbf{W}_q}^\top\textbf{W}_k$). We find that in many ViTs, especially those with classification training objectives, early layers attend more to similar tokens, while late layers show increased attention to dissimilar tokens, providing evidence corresponding to perceptual grouping and contextualization, respectively. Many of these interactions between features represented by singular vectors are interpretable and semantic, such as attention between relevant objects, between parts of an object, or between the foreground and background. This offers a novel perspective on interpreting the attention mechanism, which contributes to understanding how transformer models utilize context and salient features when processing images.

Dissecting Query-Key Interaction in Vision Transformers
Xu Pan Aaron Philip Ziqian Xie Odelia Schwartz'
University of Miami Harvard University MMichigan State University
University of Texas Health Science Center at Houston xupanêfas harvard. edu ziqian. xie@uth. aiqiin.iie@uth..... tmc. edu odel Milis mi ami edu
philipaa@msu edu
Abstract
Self-attention in vision transformers is often thought to perform perceptual group- ing where tokens attend to other tokens with similar embeddings, which could correspond to semantically similar features of an object. However, attending to dissimilar tokens can be beneficial by providing contextual information. We pro- pose to analyze the query-key interaction by the singular value decomposition of the interaction matrix (i.e. W, W).. We find that in many ViT', especially those with classification training objectives, early layers attend more to similar tokens, while late layers show increased attention to dissimilar tokens, providing evidence corresponding to perceptual grouping and contextualization, respectively. Many these interactions between features represented by singular vectors are interpretable and semantic, such as attention between relevant objects, between parts of an ob- ject, or between the foreground and background. This offers novel perspective on interpreting the attention mechanism, which contributes to understanding how transformer models utilize context and salient features when processing images.
Figure 1: We propose way study kuery-key interactions via the singular value decomposition of the query-key interaction matrix. Many the modes (i.e. pairs of singular vectors corresponding the query and the key respectively), Top row: ViT layer 8 head 7 mode 2. Bottom row: DINO layer 8 head 9 mode 2. The red channel indicates the projection value of embedding query; the cyan channel indicates the projection value of embedding onto the right singular vector which corresponds the key.
semantically interpretable. Two example modes are shown.
the left singular vector which corresponds the
Introduction
Vision transformers (ViTs) are family of models that have significantly advanced the computer vision field recent years [14.. The core computation ViTs, self-attention, is designed to promote interactions between tokens corresponding relevant image features [14] But this mechanism
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

has different interpretations with open questions such as what "relevant" refers to. Some interpret "relevant" as tokens within the same object. Highlighting objects in attention maps is usually considered a desirable property of ViT's [14 7 -
However, observations in the language domain suggest that self-attention contextualizes tokens, such that the same token has different meanings in different contexts [16. Contextualization in vision may require token to receive information not only from same-category tokens, but also from a wider range of diffcrent-category tokens such as backgrounds or other objects in the scene. Contextual effects also abound in neuroscience, whereby
the responses of neurons and perception are influenced by the context [8] 2 25 22
IIT [ [0].
Therefore, two ideas exist regarding self-attention: token attends to similar tokens, which could lead to grouping and highlighting the objects; or attends to dissimilar tokens such as backgrounds and different objects, which could lead to stronger contextualization. The former has been supported by many studies, while the latter has been largely ignored in previous studies.
Much like all other deep learning models, though ViTs are successful in many applications, researchers do not have direct access to how information is processed semantically. This issue is particularly important when deploying transformer-
large language models (LLMs) where safety is priority, As such, there have been studies trying to find feature axes (also known as semantic axes) in the embedding space [8 5 6 13
344 19]. A general finding is that embeddings in feedforward layers (i.e. MLP layers) are more semantically interpretable than in self self-attention layers [18] 19). It is believed that the embeddings in the self-attention layers have more superposition, whereas embeddings in the feedforward layers have less superposition due to the expansion of dimensionality (6. Thus, there has been less focus finding feature axes in the self- attention layers, and there has been little study addressing interactions between feature axes. In this study, while addressing the role of self-attention, we propose that singular vectors of the query-key interaction are pairs of feature directions. Properties of self-attention heads can be elucidated by studying the properties of their singular modes. We show that those singular vector pairs help semantically explain the interaction between tokens in the self-attention layers.
Our main contributions are as follows:
We identify role of self-attention in variety of ViTs. In many ViTs, especially those with classification training objectives, early layers perform more grouping in which tokens attend more to similar tokens; late layers perform more contextualizing in which tokens attend more to dissimilar tokens. However, this observation has some variability among models and may depend on the training objective: notably, some self-supervised ViTs tend to increase attention to dissimilar tokens in the last few layers.
We propose new way interpret self aelf-attention by analyzing singular modes. Our method goes beyond finding individual feature axes and extends model explainability to the in- teraction of pairs of feature directions. This approach therefore constitutes enhancing the explainability of transformer models.
In section 2, we state the motivations of this study and list related work. In section 3, we empirically analyze the preference of self-attention between tokens within and between object categories. In section 4, to study the fundamental properties of the query-key interaction, we propose a Singular Value Decomposition method. In section 5, we show that many of the decomposed singular modes are semantic and can be used to interpret the interaction between tokens. In section 6, we discuss the limitations of this study. section 7, we discuss the main findings and the significance of this study. In the supplementary, we provide an extensive set of visualization examples of the singular modes. Code for this work is available at: https///gihub..om/scwww cawiattttnzn//isttnnnggii ViT.
2 Related work
Attention map properties The properties of attention maps have been studied since the invention of the ViT. The original VIT paper reported that the model attends to image regions that are semantically meaningful, showing that the [CLS] token (i.e. a special token originally designed as the final hidden vector) attends to objects [1]. Later, a study showed that, in a self-supervised ViT named DINO, the [CL.S] attention map has clearer semantic segmentation property, highlighting the object (7]. Following this idea, studies further showed that the attention map of tokens can highlight parts of an object, and subsequently developed a segmentation algorithm by aggregating attention maps

---

Another study on the output of self-attention layers indicates that self- self-attention may perform perceptual grouping of similar visual objects. rather than highlighting salient singleton object that stands out from other objects in the image (29 Most of these studies focus on the [CT.S] token attention map or on the outputs of attention maps. Our study, in contrast, seeks to interpret the interactions between tokens within the self-attention layers, to gain insights about properties such as grouping and contextualization.
31 39].
Contextualization Our study is inspired by contextual effects in visual neuroscience, in which neural responses are modulated by the surrounding context 3 8 44) For instance, the response of cortical visual neuron in a given location of the image suppressed when the surrounding inputs inferred statistically similar, but not when the surround is inferred statistically different, thereby highlighting salient stimuli in which the center stands out from the surround [25] 12]. Some of these biological surround contextual effects have been observed in convolutional neural networks [2 32
Here our goal is not to address biological neural contextual effects in ViT', but to dissect contextual interactions in the self-attention layers. It is known that language transformer models have strong ability to contextualize tokens [16]. However, it's not clear what kinds of contextualization emerge in the ViT. In this study, we seek to understand what kinds of interactions occur between token and other tokens that carry important contextual information, possibly representing different objects, different parts of object, the background.
Finding feature axes Finding feature axes is crucial for understanding and controlling model behavior. Since a study found semanticity in the embeddings of feedforward layers in LLMs [18], studies have primarily focused on identifying feature axes in the feedforward layers, and to lesser extent, in self-attention layers. Similar to the findings in LLMs, a ViT study found that feedforward layers have less mixed concepts and can generate interpretable feature visualizations [19]. Bills et al.
proposed a gradient- based optimization method to find explainable directions in LLMs [5]. Later,
Bricken al. proposed simpler method of sparse autoencoder 6]: though see (1 These methods have not been extensively applied to ViT studies.
Some studies focused finding feature directions in the ViTs' self-attention layers. In downstream tasks such as semantic segmentation, researchers empirically found that choosing the key embeddings as features leads to the best performance [37 1. A study proposed that the singular value decomposition of the weight matrix is a natural way to find feature directions in any neural network [4]. But they only focused on single feature directions (right singular vectors), and did not consider the feature interaction in the context of self-attention. Another study suggested that singular vectors of value weights and feedforward weights can be used as features in LLMs, but they did not analyze the query-key interaction matrix [30]. Another study in the language domain proposed a singular vector decomposition on the union of the query and key embeddings, but not on the query and key weights (27)
There has been limited work going beyond single features to studying query-key interactions. A study focusing on LLMS proposed that the corresponding columns of query and key matrices are interpretable as pairs [3]. However, this approach does not find features beyond the canonical basis of the query and key embeddings. Another study inferred query-key interactions by jointly visualizing them in low dimensional space, but their method does not find interacting feature axes 42] Here, in contrast to previous works, we utilize the singular value decomposition to study the query-key interactions. We propose that left and right singular vectors of the query-key interaction matrix can be seen as pairs of interacting feature directions, and study their properties in ViTs.
3 Grouping or contextualizing
Firstly, we empirically study whether an image token (i.e. patch in the image) attends to tokens belonging to the same objects, different objects, or background. We utilized a dataset that has been applied to studying visual salience [24], namely the Odd-One-Out (O3) dataset (29). This dataset was also used by Mehrami et al. [2 in their study but they only focused on the output of the attention layers. However, we use a different experimental design that focuses on the attention maps of image tokens. The dataset consists of 2001 images that have a group of similar objects (distractors) and a distinct singleton object (larget) (Fig) 2 A). Our goal is to examine if the attention map of
3

---

startiz
Disractor
pa
6.6
sutriid and
6.4
Bar eoorts
undll part
Sahowaed adganmtii
63-
Serkd Mici Mc-b-116---0 1i------ttttn
Layer
C
D
0.24
0.63
enencon ON oneeeeettt olyers
0.22
0.55
0.20
0.5
0.18
0.45
0.88
0.60
0.4
812
0.28
Layer
Layer
Figure 2: Attention preference in the Odd-One-Out (O3) dataset [24]. A. An example from the O3 dataset. Two tokens are chosen to correspond to the target and distractor in the image. Attention maps using two tokens as queries are computed. We examine the overlap between the attention map of the target, and each of the mask labels of the target, distractor, and background masks. Similarly, we examine the overlap between the attention map of the distractor, and each of the mask labels of Ratio of attention on the same objects (target-targe and
the distractor, target, and background. cetrractorddisractte attention). The x-axis is normalized layer numbers, from early layers (left) to late layers (right). Ratio of attention on the different objects (target-distractor and destractor-targett attention). D. Ratio of attention on the background (target-to- tore-tto-bckrruundd and cetractoorbackgrrune attention)
token of one category (target or distractors) covers more of the same category, different category, or background.
We chose to study 16 different ViT models from families: the original ViT [4), DeiT which uses distillation to learn from a teacher model [38), CLIP which is jointly trained with a text encoder [33]. and DINO (7). SimMIM 40), I-JEPA which are self-supervised models with either contrastive or mask prediction loss.
In this study, the "attention score" is defined the dot product of every query and key pair, which has the shape the number of tokens by the number of tokens and is defined per attention head. The "attention map" is the softmax of each query's attention score reshaped into 2D image, which is defined per attention head and token. For each image in the dataset, two tokens are chosen to represent the target and distractor: They target or distractor mask. Two attention maps are obtained using the two tokens, each is normalized sum to 1. Inner products are computed between the two attention maps and three masks, which can be interpreted as the ratio of attention of an object (target or distractor) on the same object, different object, background. We use target-target, target-distractor, burgetbaackrrundd distractor-targer distractor diitrtttrr and distractor- backrrrur attention to denote the 6 inner products. This measure computed per layer, head, and image. The averaged measure shown in Fig Target-target and attention are categorized as "attention on same objects"; target-distractor and distractor-target attention are categorized as "attention on different objects": target-to-backgrone and
the location of the maximum value of the down-scaled
attention are categorized as "attention on background". The attention on the same objects should be dominant attention to perform grouping. We find a trend that in most ViTs the attention the same objects is dominant in early layers; while there is trend that in

---

the deeper layers attention gradually increases on the contextual features such as the background or different objects. However, this observation has some variability among models and may depend on the training objective. For example, the self-supervised SimMIM pre-trained on pixel-level mask prediction shows increased attention on the same objects in later layers. Interestingly, this trend disappears after fine-tuning on classification task.
This result provides new evidence that self-attention considers contextual features as much more than similar features in deeper layers. In most VīTs, especially with those with classification training objectives, self-attention prefers the same objects in early layers; in deeper layers, elf-attention shifts to contextual information. As far as the authors are aware, this finding has not been reported in previous ViT studies [1 39 29].
4 Singular value decomposition of query-key interaction
4.1
Formulation
In the previous section, we empirically study the allocation of self-attention and find that self-attention does not only do grouping. In this section, we try to find whether this self-attention property can be better understood by analyzing the underlying computation. The self-attention computation is formulated as below, following the convention in the field. Each token is first transformed into three embeddings, namely query, key and value. The output of a self-attention layer is the sum of values weighted by some similarity measures between query and key. The original transformer model used the softmax of the dot-product of the key and query 14
Q QKK K
Attention Q, K, V) softmax
V Vd
where Q, K. V denote the query, key, and value embeddings. They are calculated from linearly transforming the input sequence X (X1,...,Xi, E RdxL, where d the input embedding size, is the sequence length,
Q W,X E RdxL
K W&X E RdexL
V W.X Ed. Rd.
where Wg q E Rddx W& E Rdxxdd W. Rdd are trainable linear transformations that transform the input embedding to the key, query, and value space. Sometimes a bias term also added the transformation. Since the bias term does not depend the input embedding, we do not include it in our analysis token interactions. In the formula of the attention output, the part that contains the query and key interaction is named the attention score. In this case which is based on the dot-product, the attention score between two tokens x; (query) and X, (key)
aij q. k, = x Wg W&X,
The attention score solely depends on the combined matrix Wg T TW W as whole [5), which represents the query-key interaction. To better understand the behavior of this bilinear form, we factor the matrix using the singular value decomposition,
W W. U
where U {ui ..., Udi E dx is the left singular matrix composed of left singular vectors, V == (vi...., Vdd E daxd, is the right singular matrix composed of right singular vectors, Σ diag(o1,... Jd.) E RRdx xde is diagonal matrix composed of singular values. We will refer to the nth singular mode as the set (u,, on vn). Then the attention score between two tokens can be decomposed into singular modes.
x W WaX, = Σ x UnooV,
n=1
Consider the input embeddings projected onto the left and right singular vectors, i.e. x u, and x T vn. The attention score is non-zero when the two embeddings have non-zero dot-product with the
5

---

corresponding left and right singular vectors within the same singular mode. In other words, if one embedding happens be the direction of left singular vector, it only attends to tokens that have component of the corresponding right singular vector. It can be thought of as a left singular vector "query" looking for its right singular vector "key".
4.2 Similarity between left and right singular vectors
vf oitsst-tttts--- 22-166
Ub-arge 2612:-6--226
0.8
vi-arge patc
Weighted cosine similrity
0.6
(Ep vit base patcha2
cip-ebla teeaageepnn pabence
det deti patchte
del: gatried
0.4
base paschce
0.2
0.0
-02
Layer
Cosine similarity between left and right singular vectors. The cosine similarity is computed per head and singular mode. The weighted average value of cosine similarity is computed with weights of corresponding singular values.
Figure 3:
To determine if self-attention performs grouping or combines contextual information, we examine whether tokens in different layers have higher attention scores with similar tokens or dissimilar tokens. This can be measured for each singular mode by how much the left singular vector is aligned with the right singular vector, more specifically, the cosine similarity between the left singular vector and the right singular vector. A high cosine similarity value means tokens attend to similar tokens (to itself if the value is 1); low value means tokens attend to dissimilar tokens (to orthogonal tokens O; to opposite tokens if negative). The average cosine similarity is weighted by the singular values with the assumption that singular modes with higher singular values are more influential to the total attention score cOSacy
MΣ 0, cos;. We find that the averaged cosine similarity is high in early layers, and
there is decreasing trend in deeper layers (Fig 3.. In some models, the averaged cosine similarity drops to in some middle layers. The cosine similarity distribution and singular value spectrum of the vit-base- bit-base-patch66 patch16-222 model is provided in the Supplementary Figures ST and [22
Though find general trend that attention changes from attending more the similar tokens to dissimilar tokens from early layers to late layers, some ViTs have a more complex trend that increases attention to similar tokens in the last few layers (Fig Models that have this "concave" trend are Sim-IM-vii--ll sesitt-b---peteee 6-pretrain, Dino models, Deit models, and huge VIT models. Most of them either have self-supervised objectives or distillation regularizations. We hypothesize that the last layers may behave differently because they are closer to the training target, and so the training objective may have more influence. We think that self-supervised objectives, such as reconstructing masked patches, require stronger consistency between tokens, and thus more attention is allocated to similar tokens in the higher layers; while the classification objective requires gathering information from different aspects of a scene, and thus more attention is allocated to dissimilar tokens. This hypothesis is supported by the cosine similarity plot (Fig the SimMIM models, which shows in the last few layers of the pre-trained model increased attention to similar features. This matches the observation in the literature, that the SimMIM model has more local attention [41]. However, we find that the SimMIM model fine-tuned on ImageNet classification has trend of decreased attention to similar features, similar to most of the classification models. Although I-JEPA trained with self sele-super objective predicting latent representations, the cosine similarity for the I-JEPA

---

encoder does not show increased attention to similar tokens in the last few layers. The I-JEPA model is known have excellent linear ameer-probbing performance, and thus we think it may behave more similarly to classification model. The self-supervised objective of I-JEPA may be more apparent in the I-JEPA predictor (also transformer). When we run the cosine similarity analysis on the predictor module instead of the encoder, we find that the cosine similarity is overall high (Supplementary S3). The role of the training objective on internal model behavior is an interesting topic for future research.
It known that embeddings in transformer models are some extent anisotropic 26 which means the expected value of cosine similarity of two random sampled inputs tends to be positive. We indeed find anisotropy effects in all the models we examined using cosine similarity (Supplementary Figure S54) (though see other metrics [35]). If we treat anisotropy level as baseline for cosine similarity, the effect shown in Fig still exists but the self-attention is less biased to similar tokens (Supplementary Figure (S4).
There is further implication of the singular value decomposition approach. The left and right singular vectors of each attention head are two incomplete orthonormal bases of embedding. We suggest that these bases are feature directions since they intrinsic properties of the self-attention layer. The query and key embeddings can be made arbitrary, since one can change the basis without affecting the attention score. However, the singular vectors are invariant to the change of basis. If an invertible matrix A E Rd4, acts on the query and key weights as W - A T "W, and WA + A k. then the attention score does not change but the query and key embeddings change. The singular vector decomposition of (A W T A WW stays the same as decomposing W, T WW.. Thus singular vectors are uniquely special and may show interesting properties. Due the sign ambiguity of the singular value decomposition, we consider the opposite directions of singular vectors also as feature directions.
Semanticity of singular modes
0.20
0007
108
Figure 4: Examples of optimal attention images of singular modes and query and key map in dino- vitbl6. Optimal attention images are found from the Imagenet validation set that induce the largest attention score (sorted by the product of the maximum of query map and maximum of key map). The red and cyan channels are the projection values of embedding onto the left and right singular vectors of singular mode. They correspond query and key. The white area where the query map and key map overlap. The name code we assign to singular modes specifies the layer, head, and mode numbers. For example, "I.1 H4 M3" means layer 1, head 4, and mode The value below indicates the cosine similarity between the left and right singular vectors.
7

---

Layer 1. head
Laver 5, head
Layer 8, head
Layer 10. head 2
Figure 5: Visualization of a single image with multiple modes. We pick an example dog image from the ImageNet dataset and the dino-vitb16 model. Top modes (ordered by the contribution the attention score) for example layers and heads are shown. See Supplementary Figure (S17 for extended mode visualizations of this image.
The singular value decomposition of self-attention offers an intuitive way to explain the self-attention layer. A feature represented by left singular vector attends the feature represented by the corresponding right singular vector. The feature of singular vector can be found by finding the image that has maximum embedding projection on the singular vector. Similarly, the typical interactions of singular mode can be identified by finding the image that has the maximum product of the projections on a singular vector pair. Previous studies on the explainability of deep learning models only focused the explainability of single neurons individual feature axes. The singular value decomposition extends model explainability to the interaction of pairs of "neurons" (i.c. singular vectors). Note that this is very diffferent from the standard approach of visualizing the attention map of the [CLS] token without addressing interactions between tokens [4, 31
Some example modes from dino- 6 are shown in Fig. 4 For each mode, we show the top 8 images in the Imagenet (Hugging Face version) [36] validation set that induce the largest attention score. For each image, query map (red channel in the figure) and key map (cyan channel in the figure) are obtained by projecting the embedding onto the left and right singular vectors. The embedding obtained from the input of the self-attention layer, i.e. the output of the layer normalization. Each map tells what information the left or right singular vector represents. Jointly, the highlighted regions in the query map attend to the highlighted regions in the key map. In other words, the information in the highlighted regions of the key map flows to the highlighted regions the query map. More examples are shown for range of VIT architectures in the Supplementary Figures S5 S16
In early layers, singular vectors usually represent low-level visual features like color or texture, and sometimes positional encoding. In higher layers, singular vectors can represent more complex visual features like parts of objects or whole objects. As shown the previous sections, high attention scores can be induced between similar tokens (more often in early layers) or dissimilar tokens (more often in late layers). The correspondence to image structure for similar and dissimilar tokens can be seen in the query and key maps. For the modes with high cosine similarity, query and key maps

---

similar which could represent color, texture, parts, objects, positional encoding, For the modes with low cosine similarity, query and key maps look different which could represent different object parts, different objects, or foreground and background. Some examples include: in "I6 H4 M1" the animal face (query) attends eyes, nose and mouth (key); in "I.7 H7 M9" the lower part of a car attends to the upper part of a car and wheels; in "L.8 H9 M2" the fish or other things in hand attend to human; in "L10 HIO MI" the background attends to the food.
To show the hierarchical information process across layers, we show an example dog image and example attention heads along with optimal images for top modes in Fig. [ The late layers capture more semantic information such as the parts of a dog or animal, and hand with an animal. The early layers capture low-level properties like color. We show more examples in Supplementary Figure S17S19
The attention between dissimilar tokens could be thought of as providing contextual information to a given token. In the part-to-pant case, finding more parts of an object increases the confidence of finding the object and helps merge smaller concepts into larger concept. In the object-to-object case, an object attending to a different object could add additional attributes to it, for example, fish attending human may add the attribute "be held" to the fish tokens, which helps understanding of the whole scene. These interactions between tokens, though conceptually simple, as far as the authors are aware, have not been reported before this study. This result further supports the idea that self attf-attention combines contextual information from dissimilar tokens such as backgrounds or different objects.
patchi6-224
&iuos pelskt 336
ye
0.6
di-yo bate sanchus
Same object
End oo was
0.5
CE dittddddddee parchi
Savid 128 pre-oretian
bexter 626
0.4-
0.3
Layer
Figure 6: The probability that the left and right singular vectors highlight the same object in maximum attention images.
Finally, we study whether tokens prefer to attend to the same object or different objects at the singular mode level. We choose to use a semantic segmentation dataset, namely ADE20K [43]. We first find the top 5 images that induce maximum attention of singular mode, then find the optimal objects in each image that have the maximum projections on the left and right singular vectors per object area. The probability of the left and right singular vectors having the same optimal object is computed with the weight of singular values, following the same method in the previous experiment. We find that, in early layers, there is higher probability that the left and right singular vectors attend to the same object; in late layers, the probability is lower, though the variability between models is considerably large (Fig. (6. This result further supports that self-attention performs more grouping in early layers; in late layers, tokens attend to different objects which could contextualize the token with background information.
Limitation
We are aware of some limitations of this study and interesting open questions that remain. There behavioral variability between the models, which may be due to the distinct training objectives. Identifying how the training paradigm alters the learned embedding space is a potential future

---

direction to explore. We have focused on the query-key interactions in the self-attention, and future studies could address the role of the value matrix.
Discussion
Inspired by the observation that self-attention gathers information from relevant tokens within an object, and the importance of contextualization in neuroscience, we study fundamental properties of token interaction inside self-attention layers in ViTs. Both empirical analysis of the Odd-One- One-Out (03) dataset, and singular decomposition analysis of singular modes for the Imagenet dataset, show that in early layers the attention score is higher between similar tokens, while in late layers the attention score is higher between dissimilar tokens.
The singular decomposition analysis provides a new perspective on the explainability of ViTs. Two directions (left and right singular vectors) in the embedding space could be analyzed in pairs to interpret the interaction between tokens. Using this method, we find interesting semantic interactions such as part-to-pan porp attention, object-to-object attention, and foreground- wowpuuudd-to-bkggroooeooo which have not been reported in previous studies. Our reported findings provide evidence that self- attention in vision transformers is not only about gathering information between tokens with similar embeddings, but variety of interactions between a token and context. The method of analyzing singular vectors can be easily adapted to study token interactions in transformer networks trained on other modalities like language. Adapting this method to real-world applications can increase transparency of what the transformer models are capturing.
attention
Acknowledgements
A.P. was supported by the Research Experiences for Undergraduates (REU) Site Scientific Computing for Structure in Big or Complex Datasets, NSF grant CNS-1949972. O.S. was funded by the University of Miami Provost Research Award.
References
[1] H. Adeli, S. Ahn, N. Kriegeskorte, and G. Zelinsky. Affinity-based attention in self-supervised transformers predicts dynamics of object grouping in humans. arXiv preprint arXiv:2306.0022 00294, 2023.
[2] S. Amir, Y. Gandelsman, S. Bagon, and T. Dekel. Deep vit features as dense visual descriptors. arXiv preprint arXiv:2112.05814, 2(3):4, 2021.
[3] A. Angelucci, M. Bijanzadeh, L. Nurminen, F. Federer, S. Merlin, and P. C. Bressloff. Circuits and mechanisms for surround modulation in visual cortex. Annual review of neuroscience, 40: 425 451, 2017.
[4] M. Assran, Q. Duval, Misra, Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Bal- las. Self-supervised learning from images with joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15619-15629, 2023.
[5] Bills, N. Cammarata, D. Mossing, Tillman, L. Gao, G. Goh, Sutskever, J. Leike, J. Wu, and W. Saunders. Language models can explain neurons in language models. URL http://openaiipubiccc blob. core. windows. nelheeuon--xpllneefpppprrrii cessed: 14.05. 2023), 2023.
himl, ac-
[6] Bricken, A. Templeton, Batson, Chen, A. Jermyn, T. Conerly, Turner, Anil, C. Denison, A. Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, page 2, 2023.
[7] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self- superviedd vision transformers. Proceedings of the IEEF/CVF international conference computer vision, pages 9650-9660, 2021.
10

---

[8] J. R. Cavanaugh, W. Bair, and J. A. Movshon. Selectivity and spatial distribution of signals from the receptive field surround in macaque neurons. Journal of neurophysiology. 88(5): 2547-2556, 2002.
[9] X. Chen, C.-J. Hsieh, and Gong. When vision transformers outperform resnets without pre- training or strong data augmentations. In International Conference on Learning Representation. 2022.
[10] O.-H. Choung, A. Bornet, A. Doerig, and M. H. Herzog. Dissecting (un) crowding. Journal of vision, 21(10):10-10, 2021.
[11] C. W. Clifford and G. Rhodes. Fitting the mind to the world: Adaptation and after-effects in high-level vision, volume 2. Oxford University Press, 2005.
[12] R. Coen-Cagli, A. Kohn, and O. Schwartz. Flexible gating of contextual influences in natural vision. Nature neuroxcience, 18(11):1648-1655 2015.
[13] G. Dar, M. Geva, A. Gupta, and J. Berant. Analyzing transformers in embedding space. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 16124-16170, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.11653/112023. act-long.892 URL /attps://aclanthologg) org/ 2023 wt.002..ccl.. acl- 1onlllong.. wooooonnn......... 893
[14] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, Uszkoreit, and N. Houlsby. An image is worth 16x16
words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL. |tt.bttps://oppereiise
[15] N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021.
[16] K. Ethayarajh. How contextual are contextualized word representations? Comparing the geom- etry of BERT, ELMo, and GPT-2 embeddings. In K. Inui, J. Jiang, V. and X. Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pages 55-65, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: N0.18653/v1D199 1/D19-1006.
[17] K. Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. arXiv preprint arXiv: 2019
2019.
[18] M. Geva, R. Schuster, J. Berant, and O. Levy. Transformer feed-forward layers are key-value memories. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484 5495, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. doi: 10. 10.18653/v12001 www.lemminnmmaaaeeemmmm 446. URL https: https:////// http::::///aaaaahloooooeo org/ 2021. 100::0001011/112222.mmmmmmmaaa
[19] A. Ghiasi, H. Kazemi, E. Borgnia, S. Reich, M. Shu, M. Goldblum, A. G. Wilson, and T. Gold- stein. What do vision transformers leam?? visual exploration. arXiv preprint arXhv:2212.06727, 2022.
[20] N. Godey, É. de la Clergerie. and B. Sagot. Is anisotropy inherent to transformers? arXiv preprint wwXi:23306.06366 2023.
[21] R. Huben. Research Report: Sparse Autoencoders find only 9/180 board state features in OthelloGPT
sizi.substack.com https://aizi: substack. wwustack.coompp...... research- eecearch-repor--oo sparse autoencoerss [Accessed 24-03-2024).
[22] I.. Itti and C. Koch. Computational modelling of visual attention. Nature reviews neuroscience, 2(3): 194-2394, 2001.

---

[23] H. Jones, W. Wang, and A. Sillito. Spatial organization and magnitude of orientation contrast interactions in primate vl. Journal of neurophysiology. 88(5):2796-288) 2002.
[24] I. Kotseruba, C. Wloka, A. Rasouli, and J. K. Tsotsos. Do Saliency Models Detect Odd-One-Out Targets? New Datasets and Evaluations. In British Machine Vision Conference (BMVC), 2019.
[25] Z. Li. Contextual influences in v1 as a basis for pop out and asymmetry in visual search, Proceedings of the National Academy of Sciences, 96(18):10300-1053 1999.
[26] V. W. Liang, Y. Zhang, Y. Kwon, S. Yeung, and J. Y. Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35:17612-17625, 2022.
[27] T. Lieberum, M. Rahtz, J. Kramár, G. Irving, R. Shah, and V. Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. arXiv preprint arXiv:2307.09458, 2023.
[28] T. Marques, M. Schrimpf, and J. J. DiCarlo. Multi-scale hierarchical neural network models that bridge from single neurons in the primate primary visual cortex to object recognition behavior. bioRxiv, pages 2021-03, 2021.
[29] P. Mehrani and J. K. Tsotsos. Self-attention in vision transformers performs perceptual grouping, not attention. arXiv preprint arXiv:2303.01542, 2023.
[30] B. Millidge and S. Black. The singular value decompositions transformer weight matrices are highly interpretable. In AI Alignment Forum, 2022.
[31] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, Massa, A. El-Nouby, et Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.077999 2023.
[32] X. Pan, A. DeForge, and O. Schwartz. Generalizing biological surround suppression based on center surround similarity via deep neural network models. PLoS Computational Biology. 19 (9) 1486, 2023.
[33] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages
PMLR, 2021.
[34] A. Radhakrishnan, D. Beaglehole, P. Pandit, and M. Belkin. Mechanism for feature learning in neural networks and backpropagation-free machine learning models. Science, 2024.
[35] W. Rudman, N. Gillman, T. Rayne, and C. Eickhoff. Isoscore: Measuring the uniformity of embedding space utilization. arXiv preprint arXivv2108.00:44, 2021.
[36] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115())211-222, 2015. doi:
[37] O. Siméoni, G. Puy. H. V. Vo, S. Roburin, S. Gidaris, A. Bursuc, P. Pérez, R. Marlet, and J. Ponce. Localizing objects with sulf-supervised transformers and no labels. arXiv preprint arXiv:2109.14279, 2021.
[38] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[39] Y. Wang, X. Shen, S. X. Hu, Y. Yuan, J. L. Crowley and D. Vaufreydaz. Self-supervised transformers for unsupervised object discovery using normalized cut. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14543-14553, 2022.
[40] Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu, Simmim: A simple framework for masked image modeling. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pages 9653-9663, 2022.
12

---

[41] Z. Xie, Z. Geng, J. Hu, Z. Zhang, H. Hu, and Y. Cao. Revealing the dark secrets of masked image modeling. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14475-14485, 2023.
[42] C. Yeh, Y. Chen, A. Wu, C. Chen, F. Viégas, and M. Wattenberg. Attentionviz: A global view of transformer attention. IEEE Transactions on Visualization and Computer Graphics, 2023.
[43] B. Zhou, H. Zhao, X. Puig. S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 633-641, 2017.
[44] C. M. Ziemba, J. Freeman, E. P. Simoncelli, and J. A. Movshon. Contextual modulation of sensitivity naturalistic image structure in macaque v2. Journal of neurophysiology, 120(2): 409-420, 2018.
13

---

A Supplemental material
googlewiitbbse-pptchh1-22
0.175
0.150
0.125
0.100
0.025
0.000
-1.0
Layer
-0.5
Cosine similarity
0.0
0.5
1.0
Figure S1: Histogram of cosine similarity between the left and right singular vector in ViT-base- patch16-224. The yellow layers are earlier layers; the blue layers are later layers. The red line indicates 95% confidence interval, which is calculated from embeddings sampled from a random distribution.
Figure S2: Singular value spectrum (blue) and cosine similarity (red) in te-bbas--ptchhh6 Row number indicates layer number. Column number indicates head number. The dotted line indicates 95% confidence interval, which is calculated from embeddings sampled from random distribution.
14

---

HEPA-witth1&-eeccder -EPeewiittA1---redicoo
0.5
M
Weighted cosine similarity
0.4
0.3
0.2
0.1
0.0
Layer
Figure S3: Cosine similarity between left and right singular vectors of the I-JEPA encoder and predictor modules.
A
B
largn
Embrcding cosine Simlanky
6.8
0.2
0.6
on dee
1,0 ei chorts
004
-0.5
Layer
Figure S4: Anisotropy effects in ViTs. A. Averaged embedding cosine similarity between the center tokens of different images from the Imagenet validation set. Consisting with previous studies, the cosine similarities are all positive, which is referred to anisotropy cone effect. B. Considering A as the baseline, relative cosine similarity is defined as subtracting cosine similarity between left and right singular vectors by the embedding cosine similarity in A.
15

---

6993
6952
6990
LL
6418
HOO
8357
00003
MIS
0.008
0.320
0,06
L7 HB
17
12HO
Figure S5: Examples of semantic singular modes in ViT-base bbss--pttchle 16-244 (part 1).
16

---

6816
0115
6822
L7
0134
47 MI2
NO
47 M13
0154
L8
0857
0871
HO
0.001
6052
6391
L8
L9
ML3
Figure S6: Examples of semantic singular modes in ViT- 1TT-bae--pachh1--222 (part 2).
17

---

6925
0125
0.24
HSH5M0
6110
4.295
L1O
0063
LIO
LIO HA NO
LIO MG
0.008
4,000
0.302
L11
L11
L11 HONO
Figure S7: Examples of semantic singular modes in ViT-base-patee 1assee-ptch16--11 16-244 (part 3),
18

---

8878
6999
0.106
L1 HAN
8299
13
0.100
0.000
0211
14
0.006
15 MS
15
0076
6211
HID ML
Figure S8: Examples of semantic singular modes in dino-vitb16 (part 1).
19

---

60L7
0037
8945
17 HB
8398
17
NO
0114
0.003
L8 MO
0270
0001
0.72
L8
LB
Figure S9; Examples of semantic singular modes in dino-vitb16 (part 2).
20

---

0139
0112
0138
LB
19 HANB
PRI
6009
6229
L9
19 MZ
L10 ML
L10
4.252
0.252
4004
L10
6984
LIO NO
L11
L11
Figure S10: Examples of semantic singular modes in dino-vitb (part 3).
21

---

6983
0300
0534
H33
6789
13
O
HOND
6063
LS
0278
17
HOD NG
0151
0147
LO 6855
LB
LB
Figure S11: Examples of semantic singular modes in diise-ditilled--ee 16-224 (part 1).
22

---

0148
0974
0971
LB 0130
HO
18
0641
0281
ES
0330
0379
6423
L9
NO
Figure S12: Examples of semantic singular modes in beebbb---iisillleee -pthhhhhpaahhhatcc 16-224 (part 2).
23

---

0272
0254
L10
0140
LIO NO MLO
LIO HR NO
LLO
LE0
0312
LIO
0158
L10
LIO HB NO
L1O NS
L10
0206
0305
L11 0203
L11
L11 MOD
L11 M1
Figure S13: Examples of semantic singular modes in deit-base-ee disse-ditilledd wecellee---pphhccc 16-224 (part 3).
24

---

0767
6918
0.70
L1
6876
HO
HT M12
13
0245
0732
LE
H5 MIZ
LSIO
6382
0319
6557
L7
17 M2D
HD
Figure S14: Examples of semantic singular modes in clip-vit wit-base--beeeee base-patch16 (part 1).
25

---

0368
Go
0.19
L8
0262
18
NO
6300
LO
0558
LO
HT MIZ
LB N9
0144
0280
LO
6392
HO
19 M2A
MLP
Figure S15: Examples of semantic singular modes in clip-vit wit-viitebbeeeee base-patch1666 (part 2).
26

---

6014
0231
8290
La MLS 0216
MOO
0283
0300
L9 MI2
8381
0399
LI0 6434
LIO HS M6
LIO MY
LLD
6114
Figure S16: Examples of semantic singular modes in clip-vit- wiit-base-ptthhe (part 3).
27

---

Figure S17: Singular mode maps of dog image in dino-vitb16. We hand-pick modes to show the variety of information interactions within this image. The left two columns are the original image and corresponding singular mode maps. Other columns the top 8 images that induce the highest attention through the corresponding mode.
28

---

Figure S18: Singular mode maps of a image in dino-vitb16. We hand-pie -ink modes show the variety of information interactions within this image. The left two columns are the original image and corresponding singular mode maps. Other columns the top 8 images that induce the highest attention through the corresponding mode.
29

---

Figure S19: Singular mode maps of human image in dino-vitb16 16. We hand-pick modes to show the variety of information interactions within this image. The left two columns are the original image and corresponding singular mode maps. Other columns the top 8 images that induce the highest attention through the corresponding mode.
30

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract accurately reflects the contributions and scope. We explicitly highlight the main contributions in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
31

---

Justification:
See section 6 (Limitation).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sellispecifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [No]
Justification: We don't 't have thcoretical results, but write out in detail our use of the SVD to interpret query-key interactions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our experiments use publicly available datasets, and the experiment details are provided in the paper.
Guidelines:
32

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code will be made publicly available upon acceptance. The online repository is being finalized for readability.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nis.cc/public//uust http:////cgiiieee/eeeuuubiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
33

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All experimental details are provided. Our experiments do not require training new model.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: For readability, we do not draw errors in the main figures. We added distribution plots with confidence intervals in supplementary figures and 2.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Our experiments do not require compute resources beyond a personal computer with GPU.
34

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compule required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: All image datasets used in the experiments are publicly available. We use the Imagenet version available on Hugging Face.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See the end the discussion section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
35

---

Answer:
[NA]
Justification:
We use existing models and datasets. We do not see high risk in our analysis.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cited all the datasets and models used in the paper. We noted that the Imagenet and models were obtained from Hugging Face.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the bataasess.ppprsstttt paperswi theode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
36

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve research with human subjects and therefore does not include an IRB.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "cyRgXnjidA", "reviewer": "Reviewer_LpZi", "summary": "This paper proposes a new analysis framework to dissect the potential bottom mechanism of query-key interactions in Vision Transformers (ViTs)  from the perspective of singular value decomposition. Several phenomena are presented via extensive quantitative and qualitative results, which leads to the basic conclusion that earlier layers in ViTs tend to conduct grouping among similar tokens and deeper layers in ViTs are more likely to take on the role of contextualization to connect dissimilar tokens.", "questions": "Although the interaction mechanism between queries and keys in ViTs are explored in this work, how the value projection layers take effect and how the query-key itneraction matrices coordinate with the value tokens to influence the output features remain unclear. I suggest the authors to conduct further studies in future on these points.", "limitations": "The current limitations have been discussed by the authors in the manuscript.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1.This paper investigates the working mechanism of query-key interactions in the popular ViTs, which is of great importance to enhance the explainability of transformer models but has been rarely explored before.\n\n2.A novel conclusion is given that the self-attention layers are not limited to conduct grouping among similar tokens. In deeper network layers, they also perform something like contextualization over dissimilar tokens to extract higher-level semantics.\n\n3.In addition, a new analytical tool for ViTs is provided in this work to visualize the attention preference of different self-attention layers, i.e., calculating the inner product between visual tokens and singular vectors of query/key weight matrices. This could benefit future research to conduct visualization analysis to facilitate the model development.", "weaknesses": "1.Analysis conducted in this work is mainly restricted to the ImageNet dataset, where the images are more focused on relatively simpler scenes and objects. To make the conclusions more general, experiments on samples with more complex visual scenes are more helpful.\n\n2.The ViT models investigated in this work are mainly pretrained vision models with general training objectives. It could be more comprehensive and more intetesting to see how the ViT models fine-tuned under specific downstream tasks will behave.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and suggestions.\n\n> 1.Analysis conducted in this work is mainly restricted to the ImageNet dataset…\n\nWe are interested in applying this approach to more complex visual scenes and other different datasets and domains in the future.\n\n> 2.The ViT models investigated in this work are mainly pretrained vision models with general training objectives…\n\nThank you. We are very interested in these questions. Following the suggestion of the reviewers, we have now analyzed a pre-trained mask reconstruction model SimMIM that has a self-supervised objective, and its variant that is fine-tuned on Imagenet classification. We find interesting differences across the tasks (see Fig R2 in the pdf). For example, in the O3 dataset experiments, the SimMIM masked model attends more to similar objects in late layers (relating to work that suggests attention in this model is more localized), whereas the fine-tuned classification model behaves more similarly to ViT attending more to different objects or the background for high layers. This observation is consistent with previous studies that show SimMIM focuses on local features (Park, Namuk, et al. \"What do self-supervised vision transformers learn?\").\n\n> Although the interaction mechanism between queries and keys in ViTs are explored in this work, how the value projection layers take effect and how the query-key itneraction matrices coordinate with the value tokens to influence the output features remain unclear. I suggest the authors to conduct further studies in future on these points.\n\nThanks for your kind suggestion. We agree that our current work is about finding where there is an information flow between tokens, but not about what information is passed. Studying the role of value tokens could potentially address this limitation. We are very interested in conducting such studies in future works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0PXs0GlfR3", "reviewer": "Reviewer_PNQ8", "summary": "The paper begins with the observation that self-attention in early layers of vision transformers tend to group similar objects while deeper layers focus more on gathering features from dissimilar objects or background. The paper then delves into the mathematical formulation of the attention mechanism, and reveals using SVD that the aforementioned behaviour is a result of the similarity or dissimilarity between the corresponding left and right singular vector of the projection matrices $W_q$ and $W_k$. These results are well supported by empirical evidence on numerous vision transformer backbones and rich visualisations.", "questions": "N/A", "limitations": "The authors pointed out the potential behavioural differences across different models and training techniques. In addition, the study in the paper was only conducted on query-key interactions, and the authors plan to investigate the role of value projection in the future.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper eases into the investigated problem with numerous intuitions and visualisations, which are backed up with more rigorous deductions. The narrative of the paper, as a result, is rather clear and easy to follow.\n\n2. The paper presents an interesting finding on the behaviour of commonly used vision transformers and provides insights behind such behaviours. The finding that early layers of vision transformers group similar visual cues while deep layers extract more contextual features can help deepen the community's understanding of the dynamics within the transformer architecture.\n\n3. The paper utilises SVD to study the product of the projection matrices for keys and queries. This methodology may be applied to understand interesting behaviours in other applications. For instance, transformer-based object detector, notably DETR, eliminates the need of non-maximum suppression by employing the self-attention mechanism amongst object queries. And self-attention was demonstrated to have suppressive behaviours.", "weaknesses": "1. The behaviour of deeper layers, i.e., tokens attending to dissimilar tokens, can use some more analysis. For instance, the authors stated in lines 238-239 that tokens around the fish attending to regions containing the human may add the attribute \"be held\" to the fish tokens. This can be easily tested by, say, manually overriding the corresponding attention scores to zero and observe if the resultant image feature has reduced cosine similarity against the language embedding \"a photo of a person holding a fish\". This can be easily done on a CLIP model. Analysis such as this will further deepen our understanding of transformers' behaviour, whereas simply saying deeper layers extract contextual information sounds very hollow.\n\n2. The visualisations, such as those in Figure 4 and the appendix, only show one particular mode in one head. It would be more interesting to focus on a single image and visualise how the modes in different layers and different heads process this particular image. This builds a complete and coherent narrative around the behaviour of the model, which would include how it extracts lower-level features and the high-level features. I believe this will greatly benefit the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and suggestions.\n\n> The behaviour of deeper layers, i.e., tokens attending to dissimilar tokens, can use some more analysis…\n\nThank you for this excellent suggestion. During the short rebuttal time frame, we conducted a preliminary test on your suggested experiment. We deleted a mode that shows the interaction between a cello and a player. We compared the output logit of an image with text input “a girl plays cello”. The logit only decreased a tiny amount (from 26.8074 to 26.8036). We think that one concept is simultaneously processed by multiple modes, so deleting one mode hardly affects the final outputs. We think this is a very interesting experiment, but due to the time limit and the depth of this question, we will not include a thorough analysis in this paper, but leave it for a future in-depth study. \n\n> The visualisations, such as those in Figure 4 and the appendix, only show one particular mode in one head…\n\nThanks for your kind suggestion. This is a very interesting experiment which we have now conducted (see pdf Figure R1) and will be included in the final manuscript in the supplementary material. We take an image and then look for the maximally activated modes for a layer and head. Figure R1 shows results for an example dog image from the Imagenet data set and Dino transformer. We show the top 6 modes (ordered by the contribution to the attention score) for example layers and heads. We also show for each mode, the optimal Imagenet images for those modes. The late layers capture more semantic information such as the parts of a dog or animal, and a hand with a dog. The early layers capture low-level properties. We have observed that some heads have more consistency in the type of structure captured across the top modes (e.g., layer 8 head 0 shown in Fig R1 is mostly parts of object; some early layer heads are capturing frequency or position structure), while others do not in an obvious way.\n\n\n> The authors pointed out the potential behavioural differences across different models and training techniques. In addition, the study in the paper was only conducted on query-key interactions, and the authors plan to investigate the role of value projection in the future.\n\nThanks for this comment. We agree that our current work is about finding where there is an information flow between tokens, but not about what information is passed. Studying the role of value tokens could potentially address this limitation. We are very interested in conducting such studies in future works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wYfmYIa9vW", "reviewer": "Reviewer_EcHv", "summary": "While previous studies on vision transformers focused on how self-attention groups relevant tokens, this paper analyzes how self-attention contextualizes tokens to understand comprehensive inter-token relationships across the entire image. To this end, this paper proposes using the Singular Value Decomposition (SVD) to analyze the query-key interaction $\\textbf{W}^{\\top}_q \\textbf{W}_k$. Each singular vector of the query-key projection layers can be interpreted as capturing a certain type of visual semantic information. Thus, similarities between left and right singular vectors can reveal how different visual semantics interact with each other. Through extensive analysis, this paper concludes that vision transformers tend to first attend to similar tokens to form local visual semantics, and then attend to dissimilar tokens to capture global contexts of the image.", "questions": "Q1. In Fig. 3 & S3, there are slight increases in weighted cosine similarity in the final layers for all models except for the original ViTs. Does this imply that ViTs group similar tokens again in the deepest layers? Please elaborate on why this happens and what information ViTs capture in these layers.\n\nQ2. For Fig. 4, visualizing attention maps of singular modes with fixed input images would help readers understand how ViTs adapt their focus according to the layer.\n\nQ3. How does the diversity of visual information captured by singular vectors change across layers? In other words, how does the redundancy between singular vectors vary?", "limitations": "The authors adequately address both the limitation and future directions of their method.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "S1. **Writing**: The paper is well written and clearly motivated.\n\nS2. **Novelty**: The proposed analysis via SVD is innovative.\n\nS3. **Technical soundness**: The justification for using SVD is well-founded theoretically and supported by solid quantitative and qualitative empirical analysis.\n\nS4. **Potential for broader application**: Despite being applied to image analysis, the proposed technique is generic and could potentially be applied to various domains, including video and audio understanding..", "weaknesses": "W1. **Familiar conclusion**: The general finding of this paper, i.e., group early and contextualize later, is already well known from many earlier papers [a,b]. This paper only reconfirms the same conclusion in an explainable manner.\n\nW2. **Limited analysis scope**: This paper lacks analysis on different training objectives such as masked image modeling (MIM) [c,d] or masked feature prediction (JEP) [e,f]. Previous literatures [a,b] show that training objectives determine where the learner focuses; ViT with supervised training or instance discrimination self-supervision (SimCLR or MoCo) act similar as what this paper found, but ViTs trained with MIM shows that the learner still focuses on local tokens in a deeper layer. The observations in L6-8, 45-47, 138-141 might only be correct for certain training objectives.\n\nJustification of rating: \\\nDespite not deriving a novel conclusion and lacking analysis on other training objectives, the proposed SVD-based analysis technique is novel, provides interpretability on query-key interactions, and is technically sound. It has potential applications in various domains.\n\n[a] Xie et al., “Revealing the Dark Secrets of Masked Image Modeling,” CVPR, 2023.\\\n[b] Shekhar et al., “Objectives Matter: Understanding the Impact of Self-Supervised Objectives on Vision Transformer Representations,” ICLRW, 2023.\\\n[c] Xie et al., “SimMIM: A Simple Framework for Masked Image Modeling,” CVPR, 2022.\\\n[d] He et al., “Masked Autoencoders Are Scalable Vision Learners,” CVPR, 2022.\\\n[e] Assra et al., “Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture,” CVPR, 2023.\\\n[f] Baevski et al., “data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language,” ICML, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and suggestions.\n\n> W1. Familiar conclusion…\n\nThank you for the references. We will include them in the “Related work” section. Our emphasis on the explainability as you say, and also on analyzing feature interactions via the query and key modes,  adds an important perspective over previous results. Our study is motivated by a seeming paradox: on the one hand visual perception requires grouping similar tokens to congregate small similar patches into cohesive bigger concepts, on the other hand visual perception also requires highlighting salience and modulating a local representation by its context. The two aspects are emphasized in different studies (see Related Work section). Our study proposes a novel analysis of Query-Key interactions via the SVD to identify if the two aspects coexist in visual transformers. Our paper clarifies they co-exist in different layers of the visual transformer, but also provides a means of explaining the types of features and interactions that emerge.\n\nWe believe our approach and results complement the directions of the noted papers, by addressing feature interactions. We go beyond finding local and global aspects of attention in those papers, to understanding what types of feature interactions emerge. We find modes with semantic meaning, including attention between parts of objects, relevant objects, and foreground and background. Explainability is important for understanding existing AI models and developing new ones. As suggested by your and other reviewers' comments, our explainability approach could impact studies with different training objectives, data sets, and domains.\n\n> W2. Limited analysis scope…\n\nThank you for pointing us to the masked model training references. We are very interested in what our approach finds for different training objectives. We have now run our simulations on a pre-trained SimMIM masked image model with a self-supervised objective, and a SimMIM model fine-tuned on imagenet classification (see Fig R2 in the attached one page pdf). A notable finding is that the SimMIM masked image model with the self-supervised objective behaves differently from the other models of Fig. 2, and has a preference for the same object in high layers. This matches the observation in the literature you suggested, that the SimMIM model has more local attention. Interestingly, the model that is fine-tuned on Imagenet classification behaves similarly to the other models in the O3 dataset experiments, highlighting the importance of the training objective. Both versions of SimMIM were taken from the official repositories. We will add these models to all of the revised paper simulations.\n\n> Q1. In Fig. 3 & S3, there are slight increases in weighted cosine similarity in the final layers for all models except for the original ViTs…\n\nThis is an interesting observation. We will include a discussion on this observation in the final manuscript. With the newly added SimMIM models, we find this observation is more pronounced. The models that have this concave trend are SimMIM-pretrain, DINO, Deit, and large/huge ViT. Interestingly, most of them either have self-supervised objectives or distillation regularizations. We hypothesize that the last layer may behave differently because it is closer to the training target, and so the training objective may have more influence. This hypothesis is clearly supported by the newly added SimMIM analysis, which shows that later layer attention in the pre-trained model prefers similar features, but in the model fine tuned on image classification prefers dissimilar features. Intuitively we think the objective of reconstructing the mask requires strong local consistency, thus attention is allocated to checking the consistency of similar features. The learning objective of DINO may also require local consistency, but not as strong as reconstruction. The role of the training objective and the last layer is an interesting topic for future research. It’s worth noting that the last layer behaves intriguingly differently from other layers of the network in a number of other deep learning studies examining very different metrics and domains (e.g., Transformer Layers as Painters, Sun et al., arxiv 2024; The Unreasonable Ineffectiveness of the Deeper Layers, Gromov et al., arxiv 2024; Neural representational geometry underlies few-shot concept learning, Sorscher et al., PNAS, 2024).\n\n> Q2. For Fig. 4, visualizing attention maps of singular modes with fixed input images would help readers understand how ViTs adapt their focus according to the layer.\n\nThanks for suggesting this interesting experiment. We now added an experiment that visualizes a single image with multiple modes. We take an image and then look for the maximally activated modes for a layer and head. Fig R1 shows results for an example dog image from the Imagenet data set and Dino transformer. We show the top 6 modes (ordered by the contribution to the attention score) for example layers and heads. The late layers capture information such as the parts of a dog or animal, and a hand with a dog. The early layers capture low-level properties.\n\n> Q3. How does the diversity of visual information captured by singular vectors change across layers? In other words, how does the redundancy between singular vectors vary?\n\nThanks for suggesting looking at the diversity of the SVD features. The diversity of a set of features could be measured/indicated by some different metrics. As mentioned in ​​Park, Namuk, et al. \"What do self-supervised vision transformers learn?”, the diversity of features could be indicated by the shape of the singular value spectrum: flat spectrum may indicate more diverse rich features. We show several singular value spectra in Fig S2. Deeper layers have flatter spectra than early layers, which may indicate deeper layers have more diverse features than early layers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qGdj46kk9u", "reviewer": "Reviewer_Quv7", "summary": "The paper proposes SVD as a way for analyzing the interaction between the Key and Query vectors within the self-attention architecture. To this end, it measures the cosine similarity between the left and right eigenvectors of the attention score. The proposed approach evaluates the proposed mechanism on different configurations of DeiT, CLIP, DINO, and VIT. One of the findings of the paper which is the difference in the type of interaction in different layers is interesting, though to some-level could be expected given the higher abstraction level seen in later layers.", "questions": "* How could one connect the proposed approach to the different outputs of the model? For example, in a classification tasks, how could one propagate the interaction of Q-K vectors for different classification scores?\n* Could you further elaborate on scaling this to other architectures than self-attention?", "limitations": "The authors discuss the limitations of the method to some degree.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "*  The paper takes an interesting approach for analyzing the interaction between queries and keys in self-attention.\n*  The finding on the nature of contribution or interaction of different layers between the keys and queries  is interesting.", "weaknesses": "* The paper seems to be written in a haste. For example, the figures almost miss the proper x-axis labeling. \n* The visualizations of the eigenvectors back-projection seems not relating properly with the claims, in most of visualization there is a diverse set of locations attended and it is not obvious how the left and right eigenvector back-projections actually relate.\n* The approach seems to be specifically limited to self-attention mechanism and not obvious how it may scale to other architectures\n* It is not also obvious how the scores are averaged to be visualized for different layers and over the samples in the dataset. Slight better elaboration on the dateset and the characteristic of the data would have been helping.\n* While a semantically meaningful approach is proposed, the experiments are mostly focused on object-object and foreground-background interaction.\n* The proposed approach could seems to be usable as a posthoc approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and questions. We are glad you found our approach and results interesting. We believe the points you raised are either about clarifying questions or comments that do not justify the score of 3. We reply to all your questions below.\n\n> The paper seems to be written in a haste. For example, the figures almost miss the proper x-axis labeling\n\nThe X-axis label of “layers” follows the convention in other papers, ordering the layers from low to high. We omitted unnecessary marks for readability and did not label the layer numbers, since different architectures have a different number of layers. We will explicitly mark that the left end is the first layer and the right end is the last layer, and add a clarifying sentence in the figure captions.\n\n> The visualizations of the eigenvectors back-projection seems not relating properly with the claims, in most of visualization there is a diverse set of locations attended and it is not obvious how the left and right eigenvector back-projections actually relate\n\nThe visualizations show the projection value of a hidden embedding onto singular vector pairs. As guaranteed by SVD, ​​if a query is completely aligned with a left singular vector (red: query directions), it would only have a non-zero attention score with tokens with a non-zero projection onto the corresponding right singular vector (cyan: key directions). We found different types of interactions between the singular vectors. For example cupcake top attends to cupcake bottom; objects held by a person attends to the person holding it. Via visualizations, we find that many left and right singular vectors are related semantically, especially in higher layers.\n\n> The approach seems to be specifically limited to self-attention mechanism and not obvious how it may scale to other architectures\n\nSelf-attention is one of the most important building blocks in modern deep neural networks. Though our focus in this study is vision models, our methods can be easily generalized to other modalities, cross-attention in multi-modal models, and other attention variations.\n\n> It is not also obvious how the scores are averaged to be visualized for different layers and over the samples in the dataset. Slight better elaboration on the dateset and the characteristic…\n\nFor Fig 2 we average across heads and images. For Fig 3 we averaged across heads and modes, weighting modes by singular value. For Fig 5, for each mode, we find the top 5 images in the ADE20K training set that maximally activate this mode, and then we calculate the frequency with which the left singular vector and the right singular vector highlight the same object. Then this frequency is averaged per head with singular value weighting.\n\nVisualizations are the projection values of the hidden embeddings onto singular vectors. We do not average the visualizations. The visualization is per mode and image.\n\nWe used ImageNet, Odd-One-Out (O3) and ADE20K datasets. O3 has been used to answer questions about saliency perception. It includes a target object, and distractors (objects similar to one another). The target has distinct properties such as shape. ADE20K is a semantic segmentation data set: all images are annotated with the objects and parts. The O3 and ADE20K sets were chosen to suit the questions we answered.\n\n\n> While a semantically meaningful approach is proposed, the experiments are mostly focused on object-object and foreground-background interaction\n\nAs AI models are increasingly adopted in real-world applications, there is great interest in explainability, namely dissecting why a model behaves as it does. One direction is finding feature axes for layers of neural networks, as we discuss in the Related Work. Here we propose to examine axes interactions via query and key modes. We do not apriori look for object-object or foreground-background interactions. Rather, these, along with parts of objects, emerge from the SVD analysis. We also discuss the kinds of features that emerge across the layers. This approach could be used in future studies on different data sets and modalities, potentially resulting in other types of interactions.\n\n> The proposed approach could seems to be usable as a posthoc approach\n\nAs you said, it can be considered a post-hoc approach to manipulate models and ensure safe model behavior. It is a trending field to find the internal representation of a trained transformer and then manipulate the features to avoid unsafe outputs, see Transformer Debugger, Mossing et al. 2024. Our method is very different from previous studies, 1) rather than requiring inference or training an SAE, the features are found by decomposing the weights, 2) rather than finding individual feature directions, our approach finds pairs of feature directions. We believe our approach will impact the field of model interpretability.\n\n> How could one connect the proposed approach to the different outputs of the model? For example, in a classification tasks, how could one propagate the interaction of Q-K vectors for different classification scores?\n\nIntuitively if we find several singular modes according to activation level that share semantic content, deleting these modes from the model will hinder the information processing of this semantic content; enhancing these modes will bias the model to output more relevant content to this semantic concept (see Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet).\n\n> Could you further elaborate on scaling this to other architectures than self-attention?\n\nOur approach can be applied to almost all transformer models, including self-attention and other forms of attention, such as cross-attention. For instance, in a multimodal model that includes images and audio, the query could pertain to the visual information and the key to the audio information; the modes would then highlight visual and auditory information that are attended together.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers and AC,

Thank you for your helpful reviews. We appreciate that reviewers thought our SVD approach for finding query-key interactions is novel/interesting, and that our methodology has applicability to other domains and applications.

We appreciate the thoughtful suggestions. In response, we have run new analyses, which we believe has improved our paper, namely:

**(i) Visualizing a single image with multiple modes:** Two reviewers suggested we run our analysis on a single image to see how the modes change across the heads and layers. This is a good idea and we have now run this analysis (see examples in Fig R1 of the attached pdf). We will include the new simulations in the revised paper as part of the supplementary material.

**(ii) Influence of the training objectives via newly added SimMIM transformer analysis:** Two reviewers noted the interest in downstream tasks and other training paradigms, which we agree, and we appreciate the pointer by one of the reviewers to the masked image model literature. We have therefore run our simulations on the SimMIM pretrained model, and on a version of the SimMIM which was fine-tuned for Imagenet classification. Interestingly, the fine-tuned model behaved similar to the ViT model, but the self supervised masked training model behaved quite differently, relating to previous findings that the SimMIM masked image model is more localized in its attention. We will include the new simulations in the revised paper (see Fig R2 in the one page pdf).

We also reply individually to all other comments and suggestions by the reviewers.

Meta-review: Most of the reviewers acknowledged the importance of the task investigated in this paper, novelty of the proposed approach and conclusion, the broader application of the proposed approach to other Transformer models other than ViTs, and clear writing of the paper. Major concerns of the reviewers have been address in the authors' rebuttal. The reviewer Quv7 still has skepticism and would like to see more analysis after the rebuttal. However, the AC found this criticism lacks concrete evidence and elaboration. 

Overall, the AC found this paper of high quality and recommend to accept it as a spotlight.
Decision: Accept (spotlight)