Dissecting Query-Key Interaction in Vision Transformers

Authors: xu  pan
aaron  philip
ziqian  xie
odelia  schwartz
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 23:30:13

Abstract: Self-attention in vision transformers is often thought to perform perceptual grouping where tokens attend to other tokens with similar embeddings, which could correspond to semantically similar features of an object. However, attending to dissimilar tokens can be beneficial by providing contextual information. We propose to analyze the query-key interaction by the singular value decomposition of the interaction matrix (i.e. ${\textbf{W}_q}^\top\textbf{W}_k$). We find that in many ViTs, especially those with classification training objectives, early layers attend more to similar tokens, while late layers show increased attention to dissimilar tokens, providing evidence corresponding to perceptual grouping and contextualization, respectively. Many of these interactions between features represented by singular vectors are interpretable and semantic, such as attention between relevant objects, between parts of an object, or between the foreground and background. This offers a novel perspective on interpreting the attention mechanism, which contributes to understanding how transformer models utilize context and salient features when processing images.

Main paper: 
 introduction: 
 Vision transformers (ViTs) are a family of models that have significantly advanced the computer vision field in recent years [14] . The core computation of ViTs, self-attention, is designed to promote interactions between tokens corresponding to relevant image features [14] . But this mechanism has different interpretations with open questions such as what "relevant" refers to. Some interpret "relevant" as tokens within the same object. Highlighting objects in attention maps is usually considered a desirable property of ViTs [14, 7, 9] . However, observations in the language domain suggest that self-attention contextualizes tokens, such that the same token has different meanings in different contexts [16] . Contextualization in vision may require a token to receive information not only from same-category tokens, but also from a wider range of different-category tokens such as backgrounds or other objects in the scene. Contextual effects also abound in neuroscience, whereby the responses of neurons and perception are influenced by the context [8, 23, 44, 25, 22, 11, 3, 10] . Therefore, two ideas exist regarding self-attention: a token attends to similar tokens, which could lead to grouping and highlighting the objects; or attends to dissimilar tokens such as backgrounds and different objects, which could lead to stronger contextualization. The former has been supported by many studies, while the latter has been largely ignored in previous studies.Much like all other deep learning models, though ViTs are successful in many applications, researchers do not have direct access to how information is processed semantically. This issue is particularly important when deploying transformer-based large language models (LLMs) where safety is a priority. As such, there have been studies trying to find feature axes (also known as semantic axes) in the embedding space [18, 5, 6, 13, 34, 19] . A general finding is that embeddings in feedforward layers (i.e. MLP layers) are more semantically interpretable than in self-attention layers [18, 19] . It is believed that the embeddings in the self-attention layers have more superposition, whereas embeddings in the feedforward layers have less superposition due to the expansion of dimensionality [6] . Thus, there has been less focus on finding feature axes in the self-attention layers, and there has been little study addressing interactions between feature axes. In this study, while addressing the role of self-attention, we propose that singular vectors of the query-key interaction are pairs of feature directions. Properties of self-attention heads can be elucidated by studying the properties of their singular modes. We show that those singular vector pairs help semantically explain the interaction between tokens in the self-attention layers.Our main contributions are as follows:• We identify a role of self-attention in a variety of ViTs. In many ViTs, especially those with classification training objectives, early layers perform more grouping in which tokens attend more to similar tokens; late layers perform more contextualizing in which tokens attend more to dissimilar tokens. However, this observation has some variability among models and may depend on the training objective: notably, some self-supervised ViTs tend to increase attention to dissimilar tokens in the last few layers.• We propose a new way to interpret self-attention by analyzing singular modes. Our method goes beyond finding individual feature axes and extends model explainability to the interaction of pairs of feature directions. This approach therefore constitutes enhancing the explainability of transformer models.In section 2, we state the motivations of this study and list related work. In section 3, we empirically analyze the preference of self-attention between tokens within and between object categories. In section 4, to study the fundamental properties of the query-key interaction, we propose a Singular Value Decomposition method. In section 5, we show that many of the decomposed singular modes are semantic and can be used to interpret the interaction between tokens. In section 6, we discuss the limitations of this study. In section 7, we discuss the main findings and the significance of this study.In the supplementary, we provide an extensive set of visualization examples of the singular modes.Code for this work is available at: https://github.com/schwartz-cnl/DissectingViT. 
related work: 
 Attention map properties The properties of attention maps have been studied since the invention of the ViT. The original ViT paper reported that the model attends to image regions that are semantically meaningful, showing that the [CLS] token (i.e. a special token originally designed as the final hidden vector) attends to objects [14] . Later, a study showed that, in a self-supervised ViT named DINO, the [CLS] attention map has a clearer semantic segmentation property, highlighting the object [7] . Following this idea, studies further showed that the attention map of tokens can highlight parts of an object, and subsequently developed a segmentation algorithm by aggregating attention maps [31, 39] . Another study on the output of self-attention layers indicates that self-attention may perform perceptual grouping of similar visual objects, rather than highlighting a salient singleton object that stands out from other objects in the image [29] . Most of these studies focus on the [CLS] token attention map or on the outputs of attention maps. Our study, in contrast, seeks to interpret the interactions between tokens within the self-attention layers, to gain insights about properties such as grouping and contextualization.Contextualization Our study is inspired by contextual effects in visual neuroscience, in which neural responses are modulated by the surrounding context [3, 8, 44] . For instance, the response of a cortical visual neuron in a given location of the image is suppressed when the surrounding inputs are inferred statistically similar, but not when the surround is inferred statistically different, thereby highlighting salient stimuli in which the center stands out from the surround [25, 12] . Some of these biological surround contextual effects have been observed in convolutional neural networks [28, 32] .Here our goal is not to address biological neural contextual effects in ViTs, but to dissect contextual interactions in the self-attention layers. It is known that language transformer models have a strong ability to contextualize tokens [16] . However, it's not clear what kinds of contextualization emerge in the ViT. In this study, we seek to understand what kinds of interactions occur between a token and other tokens that carry important contextual information, possibly representing different objects, different parts of an object, or the background.Finding feature axes Finding feature axes is crucial for understanding and controlling model behavior. Since a study found semanticity in the embeddings of feedforward layers in LLMs [18] , studies have primarily focused on identifying feature axes in the feedforward layers, and to a lesser extent, in self-attention layers. Similar to the findings in LLMs, a ViT study found that feedforward layers have less mixed concepts and can generate interpretable feature visualizations [19] . Bills et al. proposed a gradient-based optimization method to find explainable directions in LLMs [5] . Later, Bricken et al. proposed a simpler method of sparse autoencoder [6] ; though see [21] . These methods have not been extensively applied to ViT studies.Some studies focused on finding feature directions in the ViTs' self-attention layers. In downstream tasks such as semantic segmentation, researchers empirically found that choosing the key embeddings as features leads to the best performance [37, 2, 1] . A study proposed that the singular value decomposition of the weight matrix is a natural way to find feature directions in any neural network [34] . But they only focused on single feature directions (right singular vectors), and did not consider the feature interaction in the context of self-attention. Another study suggested that singular vectors of value weights and feedforward weights can be used as features in LLMs, but they did not analyze the query-key interaction matrix [30] . Another study in the language domain proposed a singular vector decomposition on the union of the query and key embeddings, but not on the query and key weights [27] .There has been limited work going beyond single features to studying query-key interactions. A study focusing on LLMs proposed that the corresponding columns of query and key matrices are interpretable as pairs [13] . However, this approach does not find features beyond the canonical basis of the query and key embeddings. Another study inferred query-key interactions by jointly visualizing them in a low dimensional space, but their method does not find interacting feature axes [42] . Here, in contrast to previous works, we utilize the singular value decomposition to study the query-key interactions. We propose that left and right singular vectors of the query-key interaction matrix can be seen as pairs of interacting feature directions, and study their properties in ViTs. 
grouping or contextualizing: 
 Firstly, we empirically study whether an image token (i.e. a patch in the image) attends to tokens belonging to the same objects, different objects, or background. We utilized a dataset that has been applied to studying visual salience [24] , namely the Odd-One-Out (O3) dataset [29] . This dataset was also used by Mehrami et al. [29] in their study but they only focused on the output of the attention layers. However, we use a different experimental design that focuses on the attention maps of image tokens. The dataset consists of 2001 images that have a group of similar objects (distractors) and a distinct singleton object (target) (Fig 2 A ). Our goal is to examine if the attention map of a token of one category (target or distractors) covers more of the same category, different category, or background.We chose to study 16 different ViT models from 6 families: the original ViT [14] , DeiT which uses distillation to learn from a teacher model [38] , CLIP which is jointly trained with a text encoder [33] ,and DINO [7] , SimMIM [40] , I-JEPA [4] which are self-supervised models with either contrastive or mask prediction loss.In this study, the "attention score" is defined as the dot product of every query and key pair, which has the shape of the number of tokens by the number of tokens and is defined per attention head.The "attention map" is the softmax of each query's attention score reshaped into a 2D image, which is defined per attention head and token. For each image in the dataset, two tokens are chosen to represent the target and distractor. They are at the location of the maximum value of the down-scaled target or distractor mask. Two attention maps are obtained using the two tokens, each is normalized to sum to 1. Inner products are computed between the two attention maps and three masks, which can be interpreted as the ratio of attention of an object (target or distractor) on the same object, different object, or background. We use target-target, target-distractor, target-background, distractor-target, distractor-distractor, and distractor-background attention to denote the 6 inner products. This measure is computed per layer, head, and image. The averaged measure is shown in Fig 2 . Target-target and distractor-distractor attention are categorized as "attention on same objects"; target-distractor and distractor-target attention are categorized as "attention on different objects"; target-to-background and distractor-to-background attention are categorized as "attention on background". The attention on the same objects should be dominant if attention is to perform grouping. We find a trend that in most ViTs the attention on the same objects is dominant in early layers; while there is a trend that in the deeper layers attention gradually increases on the contextual features such as the background or different objects. However, this observation has some variability among models and may depend on the training objective. For example, the self-supervised SimMIM pre-trained on pixel-level mask prediction shows increased attention on the same objects in later layers. Interestingly, this trend disappears after fine-tuning on a classification task.This result provides new evidence that self-attention considers contextual features as much or more than similar features in deeper layers. In most ViTs, especially with those with classification training objectives, self-attention prefers the same objects in early layers; in deeper layers, self-attention shifts to contextual information. As far as the authors are aware, this finding has not been reported in previous ViT studies [14, 7, 39, 29] .4 Singular value decomposition of query-key interaction 4.1 FormulationIn the previous section, we empirically study the allocation of self-attention and find that self-attention does not only do grouping. In this section, we try to find whether this self-attention property can be better understood by analyzing the underlying computation. The self-attention computation is formulated as below, following the convention in the field. Each token is first transformed into three embeddings, namely query, key and value. The output of a self-attention layer is the sum of values weighted by some similarity measures between query and key. The original transformer model used the softmax of the dot-product of the key and query [14] :Attention(Q, K, V) = softmax( Q ⊤ K √ d k )Vwhere Q, K, V denote the query, key, and value embeddings. They are calculated from linearly transforming the input sequence X = {x 1 , ..., x L } ∈ R d×L , where d is the input embedding size, L is the sequence length,Q = W q X ∈ R d k ×L K = W k X ∈ R d k ×L V = W v X ∈ R dv×L where W q ∈ R d k ×d , W k ∈ R d k ×d , W v ∈ R dv×dare trainable linear transformations that transform the input embedding to the key, query, and value space. Sometimes a bias term is also added to the transformation. Since the bias term does not depend on the input embedding, we do not include it in our analysis of token interactions. In the formula of the attention output, the part that contains the query and key interaction is named the attention score. In this case which is based on the dot-product, the attention score between two tokens x i (query) and x j (key) isa ij = q ⊤ i k j = x ⊤ i W q ⊤ W k x jThe attention score solely depends on the combined matrix W q ⊤ W k as a whole [15] , which represents the query-key interaction. To better understand the behavior of this bilinear form, we factor the matrix using the singular value decomposition,W ⊤ q W k = UΣV ⊤ where U = {u 1 , ..., u d k } ∈ R d×d k is the left singular matrix composed of left singular vectors, V = {v 1 , ..., v d k } ∈ R d×d k is the right singular matrix composed of right singular vectors, Σ = diag(σ 1 , ..., σ d k ) ∈ R d k ×d k is a diagonal matrix composedof singular values. We will refer to the nth singular mode as the set {u n , σ n , v n }. Then the attention score between two tokens can be decomposed into singular modes.x ⊤ i W q ⊤ W k x j = d k n=1 x ⊤ i u n σ n v ⊤ n x jConsider the input embeddings projected onto the left and right singular vectors, i.e. x ⊤ u n and x ⊤ v n . The attention score is non-zero when the two embeddings have a non-zero dot-product with the corresponding left and right singular vectors within the same singular mode. In other words, if one embedding happens to be in the direction of a left singular vector, it only attends to tokens that have a component of the corresponding right singular vector. It can be thought of as a left singular vector "query" looking for its right singular vector "key". To determine if self-attention performs grouping or combines contextual information, we examine whether tokens in different layers have higher attention scores with similar tokens or dissimilar tokens. This can be measured for each singular mode by how much the left singular vector is aligned with the right singular vector, more specifically, the cosine similarity between the left singular vector and the right singular vector. A high cosine similarity value means tokens attend to similar tokens (to itself if the value is 1); a low value means tokens attend to dissimilar tokens (to orthogonal tokens if 0; to opposite tokens if negative). The average cosine similarity is weighted by the singular values with the assumption that singular modes with higher singular values are more influential to the total attention score cos avg = i σi σj cos i . We find that the averaged cosine similarity is high in early layers, and there is a decreasing trend in deeper layers (Fig 3 ). In some models, the averaged cosine similarity drops to 0 in some middle layers. The cosine similarity distribution and singular value spectrum of the vit-base-patch16-224 model is provided in the Supplementary Figures S1 and S2 . 
similarity between left and right singular vectors: 
 Though we find a general trend that attention changes from attending more to the similar tokens to dissimilar tokens from early layers to late layers, some ViTs have a more complex trend that increases attention to similar tokens in the last few layers (Fig 3 ). Models that have this "concave" trend are SimMIM-vit-b16-pretrain, Dino models, Deit models, and huge ViT models. Most of them either have self-supervised objectives or distillation regularizations. We hypothesize that the last layers may behave differently because they are closer to the training target, and so the training objective may have more influence. We think that self-supervised objectives, such as reconstructing masked patches, require stronger consistency between tokens, and thus more attention is allocated to similar tokens in the higher layers; while the classification objective requires gathering information from different aspects of a scene, and thus more attention is allocated to dissimilar tokens. This hypothesis is supported by the cosine similarity plot (Fig 3 ) of the SimMIM models, which shows in the last few layers of the pre-trained model increased attention to similar features. This matches the observation in the literature, that the SimMIM model has more local attention [41] . However, we find that the SimMIM model fine-tuned on ImageNet classification has a trend of decreased attention to similar features, similar to most of the classification models. Although I-JEPA is trained with a self-supervised objective predicting latent representations, the cosine similarity for the I-JEPA encoder does not show increased attention to similar tokens in the last few layers. The I-JEPA model is known to have excellent linear-probing performance, and thus we think it may behave more similarly to a classification model. The self-supervised objective of I-JEPA may be more apparent in the I-JEPA predictor (also a transformer). When we run the cosine similarity analysis on the predictor module instead of the encoder, we find that the cosine similarity is overall high (Supplementary FigureS3). The role of the training objective on internal model behavior is an interesting topic for future research.It is known that embeddings in transformer models are to some extent anisotropic [17, 26, 20] , which means the expected value of cosine similarity of two random sampled inputs tends to be positive. We indeed find anisotropy effects in all the models we examined using cosine similarity (Supplementary Figure S4 ) (though see other metrics [35] ). If we treat anisotropy level as a baseline for cosine similarity, the effect shown in Fig 3 still exists but the self-attention is less biased to similar tokens (Supplementary Figure S4 ).There is a further implication of the singular value decomposition approach. The left and right singular vectors of each attention head are two incomplete orthonormal bases of embedding. We suggest that these bases are feature directions since they are intrinsic properties of the self-attention layer. The query and key embeddings can be made arbitrary, since one can change the basis without affecting the attention score. However, the singular vectors are invariant to the change of basis. If an invertible matrix A ∈ R d k ×d k acts on the query and key weights as W q → A ⊤ W q and W k → A -1 W k , then the attention score does not change but the query and key embeddings change. The singular vector decomposition of (A ⊤ W q ) ⊤ A -1 W k stays the same as decomposing W q ⊤ W k . Thus singular vectors are uniquely special and may show interesting properties. Due to the sign ambiguity of the singular value decomposition, we consider the opposite directions of singular vectors also as feature directions. The singular value decomposition of self-attention offers an intuitive way to explain the self-attention layer. A feature represented by a left singular vector attends to the feature represented by the corresponding right singular vector. The feature of a singular vector can be found by finding the image that has the maximum embedding projection on the singular vector. Similarly, the typical interactions of a singular mode can be identified by finding the image that has the maximum product of the projections on a singular vector pair. Previous studies on the explainability of deep learning models only focused on the explainability of single neurons or individual feature axes. The singular value decomposition extends model explainability to the interaction of pairs of "neurons" (i.e. singular vectors). Note that this is very different from the standard approach of visualizing the attention map of the [CLS] token without addressing interactions between tokens [14, 7, 31] . Some example modes from dino-vitb16 are shown in Fig. 4 . For each mode, we show the top 8 images in the Imagenet (Hugging Face version) [36] validation set that induce the largest attention score. For each image, a query map (red channel in the figure) and a key map (cyan channel in the figure) are obtained by projecting the embedding onto the left and right singular vectors. Each map tells what information the left or right singular vector represents. Jointly, the highlighted regions in the query map attend to the highlighted regions in the key map. In other words, the information in the highlighted regions of the key map flows to the highlighted regions of the query map. More examples are shown for a range of ViT architectures in the Supplementary Figures S5 -S16 . 
semanticity of singular modes: 
 In early layers, singular vectors usually represent low-level visual features like color or texture, and sometimes positional encoding. In higher layers, singular vectors can represent more complex visual features like parts of objects or whole objects. As shown in the previous sections, high attention scores can be induced between similar tokens (more often in early layers) or dissimilar tokens (more often in late layers). The correspondence to image structure for similar and dissimilar tokens can be seen in the query and key maps. For the modes with high cosine similarity, query and key maps are similar which could represent color, texture, parts, objects, or positional encoding. For the modes with low cosine similarity, query and key maps look different which could represent different object parts, different objects, or foreground and background. Some examples include: in "L6 H4 M1" the animal face (query) attends to eyes, nose and mouth (key); in "L7 H2 M3" the lower part of a car attends to the upper part of a car and wheels; in "L8 H9 M2" the fish or other things in hand attend to human; in "L10 H9 M1" the kettle attends to its background.To show the hierarchical information process across layers, we show an example dog image and example attention heads along with optimal images for top modes in Fig. 5 . The late layers capture more semantic information such as the parts of a dog or animal, and a hand with a dog. The early layers capture low-level properties like color. We show more examples in Supplementary Figure S17-S19 .The attention between dissimilar tokens could be thought of as providing contextual information to a given token. In the part-to-part case, finding more parts of an object increases the confidence of finding the object and helps merge smaller concepts into a larger concept. In the object-to-object case, an object attending to a different object could add additional attributes to it, for example, a fish attending a human may add the attribute "be held" to the fish tokens, which helps understanding of the whole scene. These interactions between tokens, though conceptually simple, as far as the authors are aware, have not been reported before this study. This result further supports the idea that self-attention combines contextual information from dissimilar tokens such as backgrounds or different objects. Finally, we study whether tokens prefer to attend to the same object or different objects at the singular mode level. We choose to use a semantic segmentation dataset, namely ADE20K [43] . We first find the top 5 images that induce maximum attention of a singular mode, then find the optimal objects in each image that have the maximum projections on the left and right singular vectors per object area. The probability of the left and right singular vectors having the same optimal object is computed with the weight of singular values, following the same method in the previous experiment. We find that, in early layers, there is a higher probability that the left and right singular vectors attend to the same object; in late layers, the probability is lower, though the variability between models is considerably large (Fig. 6 ). This result further supports that self-attention performs more grouping in early layers; in late layers, tokens attend to different objects which could contextualize the token with background information.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The code will be made publicly available upon acceptance. The online repository is being finalized for readability.Guidelines:• The answer NA means that paper does not include experiments requiring code. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
limitation: 
 We are aware of some limitations of this study and interesting open questions that remain. There is behavioral variability between the models, which may be due to the distinct training objectives. Identifying how the training paradigm alters the learned embedding space is a potential future direction to explore. We have focused on the query-key interactions in the self-attention, and future studies could address the role of the value matrix.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: All experimental details are provided. Our experiments do not require training a new model.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
discussion: 
 Inspired by the observation that self-attention gathers information from relevant tokens within an object, and the importance of contextualization in neuroscience, we study fundamental properties of token interaction inside self-attention layers in ViTs. Both empirical analysis of the Odd-One-Out (O3) dataset, and singular decomposition analysis of singular modes for the Imagenet dataset, show that in early layers the attention score is higher between similar tokens, while in late layers the attention score is higher between dissimilar tokens.The singular decomposition analysis provides a new perspective on the explainability of ViTs. Two directions (left and right singular vectors) in the embedding space could be analyzed in pairs to interpret the interaction between tokens. Using this method, we find interesting semantic interactions such as part-to-part attention, object-to-object attention, and foreground-to-background attention which have not been reported in previous studies. Our reported findings provide evidence that selfattention in vision transformers is not only about gathering information between tokens with similar embeddings, but a variety of interactions between a token and its context. The method of analyzing singular vectors can be easily adapted to study token interactions in transformer networks trained on other modalities like language. Adapting this method to real-world applications can increase transparency of what the transformer models are capturing. • The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: For readability, we do not draw errors in the main figures. We added distribution plots with confidence intervals in supplementary figures 1 and 2.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.Figure S7: Examples of semantic singular modes in ViT-base-patch16-224 (part 3).Figure S11: Examples of semantic singular modes in deit-base-distilled-patch16-224 (part 1).Figure S12: Examples of semantic singular modes in deit-base-distilled-patch16-224 (part 2).Figure S16: Examples of semantic singular modes in clip-vit-base-patch16 (part 3). 
 
 Appendix:  


Decision: Most of the reviewers acknowledged the importance of the task investigated in this paper, novelty of the proposed approach and conclusion, the broader application of the proposed approach to other Transformer models other than ViTs, and clear writing of the paper. Major concerns of the reviewers have been address in the authors' rebuttal. The reviewer Quv7 still has skepticism and would like to see more analysis after the rebuttal. However, the AC found this criticism lacks concrete evidence and elaboration. 

Overall, the AC found this paper of high quality and recommend to accept it as a spotlight.

Reviews: review: paper_summary: paper_summary: This paper proposes a new analysis framework to dissect the potential bottom mechanism of query-key interactions in Vision Transformers (ViTs)  from the perspective of singular value decomposition. Several phenomena are presented via extensive quantitative and qualitative results, which leads to the basic conclusion that earlier layers in ViTs tend to conduct grouping among similar tokens and deeper layers in ViTs are more likely to take on the role of contextualization to connect dissimilar tokens.
strength_weakness: strength_weakness: 1.Analysis conducted in this work is mainly restricted to the ImageNet dataset, where the images are more focused on relatively simpler scenes and objects. To make the conclusions more general, experiments on samples with more complex visual scenes are more helpful.

2.The ViT models investigated in this work are mainly pretrained vision models with general training objectives. It could be more comprehensive and more intetesting to see how the ViT models fine-tuned under specific downstream tasks will behave.
questions: questions: Although the interaction mechanism between queries and keys in ViTs are explored in this work, how the value projection layers take effect and how the query-key itneraction matrices coordinate with the value tokens to influence the output features remain unclear. I suggest the authors to conduct further studies in future on these points.
limitations: limitations: The current limitations have been discussed by the authors in the manuscript.
paper_summary: paper_summary: The paper begins with the observation that self-attention in early layers of vision transformers tend to group similar objects while deeper layers focus more on gathering features from dissimilar objects or background. The paper then delves into the mathematical formulation of the attention mechanism, and reveals using SVD that the aforementioned behaviour is a result of the similarity or dissimilarity between the corresponding left and right singular vector of the projection matrices $W_q$ and $W_k$. These results are well supported by empirical evidence on numerous vision transformer backbones and rich visualisations.
strength_weakness: strength_weakness: 1. The behaviour of deeper layers, i.e., tokens attending to dissimilar tokens, can use some more analysis. For instance, the authors stated in lines 238-239 that tokens around the fish attending to regions containing the human may add the attribute "be held" to the fish tokens. This can be easily tested by, say, manually overriding the corresponding attention scores to zero and observe if the resultant image feature has reduced cosine similarity against the language embedding "a photo of a person holding a fish". This can be easily done on a CLIP model. Analysis such as this will further deepen our understanding of transformers' behaviour, whereas simply saying deeper layers extract contextual information sounds very hollow.

2. The visualisations, such as those in Figure 4 and the appendix, only show one particular mode in one head. It would be more interesting to focus on a single image and visualise how the modes in different layers and different heads process this particular image. This builds a complete and coherent narrative around the behaviour of the model, which would include how it extracts lower-level features and the high-level features. I believe this will greatly benefit the paper.
questions: questions: N/A
limitations: limitations: The authors pointed out the potential behavioural differences across different models and training techniques. In addition, the study in the paper was only conducted on query-key interactions, and the authors plan to investigate the role of value projection in the future.
paper_summary: paper_summary: While previous studies on vision transformers focused on how self-attention groups relevant tokens, this paper analyzes how self-attention contextualizes tokens to understand comprehensive inter-token relationships across the entire image. To this end, this paper proposes using the Singular Value Decomposition (SVD) to analyze the query-key interaction $\textbf{W}^{\top}_q \textbf{W}_k$. Each singular vector of the query-key projection layers can be interpreted as capturing a certain type of visual semantic information. Thus, similarities between left and right singular vectors can reveal how different visual semantics interact with each other. Through extensive analysis, this paper concludes that vision transformers tend to first attend to similar tokens to form local visual semantics, and then attend to dissimilar tokens to capture global contexts of the image.
strength_weakness: strength_weakness: W1. **Familiar conclusion**: The general finding of this paper, i.e., group early and contextualize later, is already well known from many earlier papers [a,b]. This paper only reconfirms the same conclusion in an explainable manner.

W2. **Limited analysis scope**: This paper lacks analysis on different training objectives such as masked image modeling (MIM) [c,d] or masked feature prediction (JEP) [e,f]. Previous literatures [a,b] show that training objectives determine where the learner focuses; ViT with supervised training or instance discrimination self-supervision (SimCLR or MoCo) act similar as what this paper found, but ViTs trained with MIM shows that the learner still focuses on local tokens in a deeper layer. The observations in L6-8, 45-47, 138-141 might only be correct for certain training objectives.

Justification of rating: \
Despite not deriving a novel conclusion and lacking analysis on other training objectives, the proposed SVD-based analysis technique is novel, provides interpretability on query-key interactions, and is technically sound. It has potential applications in various domains.

[a] Xie et al., “Revealing the Dark Secrets of Masked Image Modeling,” CVPR, 2023.\
[b] Shekhar et al., “Objectives Matter: Understanding the Impact of Self-Supervised Objectives on Vision Transformer Representations,” ICLRW, 2023.\
[c] Xie et al., “SimMIM: A Simple Framework for Masked Image Modeling,” CVPR, 2022.\
[d] He et al., “Masked Autoencoders Are Scalable Vision Learners,” CVPR, 2022.\
[e] Assra et al., “Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture,” CVPR, 2023.\
[f] Baevski et al., “data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language,” ICML, 2022.
questions: questions: Q1. In Fig. 3 & S3, there are slight increases in weighted cosine similarity in the final layers for all models except for the original ViTs. Does this imply that ViTs group similar tokens again in the deepest layers? Please elaborate on why this happens and what information ViTs capture in these layers.

Q2. For Fig. 4, visualizing attention maps of singular modes with fixed input images would help readers understand how ViTs adapt their focus according to the layer.

Q3. How does the diversity of visual information captured by singular vectors change across layers? In other words, how does the redundancy between singular vectors vary?
limitations: limitations: The authors adequately address both the limitation and future directions of their method.
paper_summary: paper_summary: The paper proposes SVD as a way for analyzing the interaction between the Key and Query vectors within the self-attention architecture. To this end, it measures the cosine similarity between the left and right eigenvectors of the attention score. The proposed approach evaluates the proposed mechanism on different configurations of DeiT, CLIP, DINO, and VIT. One of the findings of the paper which is the difference in the type of interaction in different layers is interesting, though to some-level could be expected given the higher abstraction level seen in later layers.
strength_weakness: strength_weakness: * The paper seems to be written in a haste. For example, the figures almost miss the proper x-axis labeling. 
* The visualizations of the eigenvectors back-projection seems not relating properly with the claims, in most of visualization there is a diverse set of locations attended and it is not obvious how the left and right eigenvector back-projections actually relate.
* The approach seems to be specifically limited to self-attention mechanism and not obvious how it may scale to other architectures
* It is not also obvious how the scores are averaged to be visualized for different layers and over the samples in the dataset. Slight better elaboration on the dateset and the characteristic of the data would have been helping.
* While a semantically meaningful approach is proposed, the experiments are mostly focused on object-object and foreground-background interaction.
* The proposed approach could seems to be usable as a posthoc approach.
questions: questions: * How could one connect the proposed approach to the different outputs of the model? For example, in a classification tasks, how could one propagate the interaction of Q-K vectors for different classification scores?
* Could you further elaborate on scaling this to other architectures than self-attention?
limitations: limitations: The authors discuss the limitations of the method to some degree.

Comments: title: Response to Authors' Feedback
Does the rebuttal address your concerns?
comment: Thank you. We will do so.
Thank you for the thorough analysis of the raised question. I'm satisfied with the response. I strongly suggest including new results during the rebuttal in the final manuscript. I'll raise my rating to 7.
Thank you for your insightful feedback and helpful further suggestions. We will address them in the final manuscript.

> JEPA Results (W2)…

Thanks for suggesting adding JEPA to the analysis. We added “I-JEPA-vit-h14” into our “cosine similarity” plots. We could not attach a new figure for you at this phase, but we can tell you that the trend of “I-JEPA-vit-h14” looks similar to the “SimMIM-vit-b16-finetune”. It doesn’t increase in the last few layers like “SimMIM-pretrain”. Since the I-JEPA encoder performs well in the linear-probing setup, this indicates it is probably more similar to a classification model, which is consistent with our new analysis. The remaining question is why the self-supervised objective doesn’t cause later layers in the I-JEPA encoder to increase its attention to similar tokens. We think this objective is mainly handled by the predictor (also a transformer). To test this, we run the cosine analysis on the predictor transformer, and find the cosine similarity is around 0.3 (considered high) across layers, which is consistent with our hypothesis. Due to the time limit we don’t have the results of “I-JEPA-vit-h14” for the other analyses yet, but it won’t be a problem to finish all the analyses and add them to the final manuscript.

> Revision of Generalized Statements (W2)...

Thank you very much. Yes, we will change those claims and add new text on the training objective according to our discussion.

We will change line 45-47 to “We identify a role of self-attention in a variety of ViTs. In many ViTs, especially those with classification training objectives, early layers perform more grouping in which tokens attend more to similar tokens; late layers perform more contextualizing in which tokens attend more to dissimilar tokens. However, this observation has some variability among models and may depend on the training objective: notably, some self-supervised ViTs tend to increase attention to dissimilar tokens in the last few layers.”

We will also make the changes to line 6-8 and 138-141 in a similar way.

We will add a paragraph in the discussion section: “Though we find a trend that attention changes from attending more to the similar tokens to dissimilar tokens from early layers to late layers, some ViTs have a more complex trend that increases attention to similar tokens in the last few layers (Fig. 3). Models that have this “concave” trend are SimMIM-vit-b16-pretrain, Dino models, Deit models, and huge ViT models. Most of them either have self-supervised objectives or distillation regularizations. We hypothesize that the last layers may behave differently because they are closer to the training target, and so the training objective may have more influence. We think that self-supervised objectives, such as reconstructing masked patches, require stronger consistency between tokens, and thus more attention is allocated to similar tokens in the higher layers; while the classification objective requires gathering information from different aspects of a scene, and thus more attention is allocated to dissimilar tokens. This hypothesis is supported by the cosine similarity plot (Fig. 3) of the SimMIM models, which shows in the last few layers of the pre-trained model increased attention to similar features. This matches the observation in the literature, that the SimMIM model has more local attention (Xie et al. 2022). However,  we find that the SimMIM model fine-tuned on ImageNet classification has the trend of decreased attention to similar features, similar to most of the classification models. Although I-JEPA is trained with a self-supervised objective predicting latent representations, the cosine similarity for the I-JEPA encoder does not show increased attention to similar tokens in the last few layers. The I-JEPA model is known to have excellent linear-probing performance, and thus we think it may behave more similarly to a classification model. The self-supervised objective of I-JEPA may be more apparent in the I-JEPA predictor (also a transformer). When we run the cosine similarity analysis on the predictor module instead of the encoder, we find that the cosine similarity is overall high (Supplementary Figure). The role of the training objective on internal model behavior is an interesting topic for future research.” 

We will also add the relevant references here for the SimMIM and I-JEPA papers.

> Clarification of Phenomenon (Q1)...

We agree that this trend is interesting. Our current hypothesis is that this depends on the training objective, such as the need for self-consistency in some self-supervised training objectives versus classification.  The experiments with SimMIM and I-JEPA offer interesting observations in this direction. To test these questions more completely is beyond the scope of this paper, and we will include a discussion and pose this as an open question for future research (see the text in the previous point).
Thank you for your feedback and helpful further suggestions.

> I think the per-image visualisations in the attached PDF are better illustrations for the paper's argument. I would recommend adding them to the main paper.

Thank you, we will do so.

> Regarding the modes and how they impact the logits, I suspect there may be multiple nodes exhibiting similar attention behaviours, therefore contributing to the corresponding logit concurrently.

Thank you, this is also our interpretation, and an interesting direction for future work. We will add this to the limitations section as a future research direction.
Thank you for your feedback.
Thanks for the response. I think the per-image visualisations in the attached PDF are better illustrations for the paper's argument. I would recommend adding them to the main paper.

Regarding the modes and how they impact the logits, I suspect there may be multiple nodes exhibiting similar attention behaviours, therefore contributing to the corresponding logit concurrently.

Nevertheless, I'm generally happy with the response, and would recommend accepting the paper.
Thank you for addressing my concerns and questions. I appreciate the thorough analysis and the newly added results with SimMIM, which strengthen the paper. However, I have some remaining points to discuss:

- **JEPA Results (W2)**: I respectfully request results on JEPA (Joint Embedding Predictive Architecture). JEPA, which predicts latent representations instead of pixel values, has shown good transferability in both full finetuning and linear probing setups. However, how ViTs trained with JEPA learn different representations compared to contrastive learning or MIM approaches is not well-studied. Analyzing how queries and keys interact under JEPA objective would significantly enhance the paper's contribution.

- **Revision of Generalized Statements (W2)**: If query-key interactions vary across different training objectives, statements such as those in lines 6-8, 45-47, and 138-141 should be revised. These statements currently appear to generalize across all training objectives, but they may be limited to specific ones. Please supplement the discussion with query-key interactions observed in other objectives.

- **Clarification of Phenomenon (Q1)**: While I appreciate the detailed explanation provided, I'm not fully convinced that it clearly and sufficiently addresses the exact reason for the observed phenomenon. This experimental trend is interesting and warrants further study. If a sufficient explanation is unavailable through this rebuttal, I strongly suggest adding related discussion and posing an open question in the main manuscript at the very least.

Given these points, I maintain my current rating. However, I'm open to raising the score if the authors provide sufficiently clear responses to these newly added questions and concerns. Addressing these points would make the paper more comprehensive and valuable to the field.
Dear Reviewer Quv7,

Thank you again for your time for reviewing this paper. Could you please check if the authors' rebuttal has addressed your concerns at your earliest convenience? Thank you!

Best regards,

AC
Thanks for the authors' rebuttal and additional experimental results. After reading the authors' response, my previous concerns have been adequately addressed, so I decide to raise my rating to 8.
Dear Reviewers,

Thank you very much again for your valuable service to the NeurIPS community.

As the authors have provided detailed responses, it would be great if you could check them and see if your concerns have been addressed. Your prompt feedback would provide an opportunity for the authors to offer additional clarifications if needed.

Best regards,

AC