Title: What do Vision Transformers Learn?  A Visual Exploration
Authors: Amin Ghiasi
Hamid Kazemi
Steven Reich
Eitan Borgnia
Manli Shu
Micah Goldblum
Andrew Gordon Wilson
Tom Goldstein
Venue: ICLR 2023
Keywords: Vision Transformers
Visualization
Interpretability

Abstract: Vision transformers (ViTs) are quickly becoming the de-facto architecture for computer vision, yet we understand very little about why they work and what they learn. While existing studies visually analyze the mechanisms of convolutional neural networks, an analogous exploration of ViTs remains challenging. In this paper, we first address the obstacles to performing visualizations on ViTs. Assisted by these solutions, we observe that neurons in ViTs trained with language model supervision (e.g., CLIP) are activated by semantic concepts rather than visual features. We also explore the underlying differences between ViTs and CNNs, and we find that transformers detect image background features, just like their convolutional counterparts, but their predictions depend far less on high-frequency information. On the other hand, both architecture types behave similarly in the way features progress from abstract patterns in early layers to concrete objects in late layers. In addition, we show that ViTs maintain spatial information in all layers except the final layer. In contrast to previous works, we show that the last layer most likely discards the spatial information and behaves as a learned global pooling operation. Finally, we conduct large-scale visualizations on a wide range of ViT variants, including DeiT, CoaT, ConViT, PiT, Swin, and Twin, to validate the effectiveness of our method.


Review discussion:
Strengths:
-- Any attempt of explaining transformers and illustrating their differences with CNNs is interesting at the moment.
-- The result that transformers perform better than CNNs when showing only the foreground, as well as only the background, is concrete and significant.
Weaknesses:
-- Trying to run the methodology of (Simonyan et al. 2014), although with the added data augmentation pieces, resulted in adversarial perturbations that are hard to interpret. I almost would believe there would be ways to do this better, e.g. by constraining more to the natural image manifold (such as using a VAE latent space perturbation). And the results in this part are a mixed bag at the best.
-- Understanding that ViTs/CNNs have some segmentation capability is hardly something novel -- this has been shown in various prior works e.g. GradCAM and all the weakly supervised segmentation work that utilized GradCAM.
-- Work on the classification token is a bit obsolete with the stronger adoption of global average pooling at the last layer.
-- There has been no work that tries to check the effect of distillation and large-scale pretraining on the ViTs explored. E.g. what is the difference between distilled ViTs and non-distilled versions? What is the difference of something trained only on ImageNet-1k (e.g. a masked autoencoder) vs. JFT? Distillations and large-scale training sets are important aspects of transformer training that could bring material changes to their behavior and would be nice to explore.
The paper is well-written. Quality and novelty is moderate -- largely used the same methodology from (Simonyan et al. 2014) with small modifications.
NO.
Thanks for your useful feedback and comments. We address each of the concerns in the following:
>Trying to run the methodology of (Simonyan et al. 2014), although with the added data augmentation pieces, resulted in adversarial perturbations that are hard to interpret. I almost would believe there would be ways to do this better, e.g. by constraining more to the natural image manifold (such as using a VAE latent space perturbation). And the results in this part are a mixed bag at the best.
(Simonyan et al. 2014) explored two methods for interpreting deep models.
First is a method to generate a class saliency map to detect the salient parts of a certain image that are important to the network's decision.
Second, they propose model inversion that determines the perception of the model of a particular class. Please note that our work explores feature visualization that differs from the two directions explored in (Simonyan et al. 2014). Feature visualization for CNNs models has already been explored vastly in the previous literature.
However, this work explores feature visualization for ViT models that has not been done before in previous work.
Notably, existing off-the-shelf methods do not work on ViTs, as we show in our work.
>Understanding that ViTs/CNNs have some segmentation capability is hardly something novel -- this has been shown in various prior works e.g. GradCAM and all the weakly supervised segmentation work that utilized GradCAM.
GradCAM is a method for producing saliency maps. Please note that we do not produce saliency maps in this work but visualize ViT models' features.
Despite the abundant work that has been done on visualizing features of CNNs, visualizing features of ViTs yet remains unexplored and to the best of our knowledge, our work is the first to do this. The conclusion we make in the paper, and I quote here, is as follows: “We show that patch-wise image activation patterns for ViT feature maps essentially behave like saliency maps, highlighting the regions of the image a given feature attends to.” We show that even though each patch can attend to every other patch, the spatial information is present in patches up until the last layer.
>Work on the classification token is a bit obsolete with the stronger adoption of global average pooling at the last layer.
Many recent methods use ViTs either with the CLS token or the global average pooling.
For example, as discussed in [1], their method works similarly well with or without the CLS token. As discussed in the paper, the last layer acts like a global average pooling layer even when the classifier is trained on top of the CLS token. This is further evidence of why these methods are getting similar results with or without using the CLS token and can suggest that different methods can remove the CLS token without being concerned about the performance (having one less token can increase the speed and decrease the number of FLOPS.)
>There has been no work that tries to check the effect of distillation and large-scale pretraining on the ViTs explored. E.g. what is the difference between distilled ViTs and non-distilled versions? What is the difference of something trained only on ImageNet-1k (e.g. a masked autoencoder) vs. JFT? Distillations and large-scale training sets are important aspects of transformer training that could bring material changes to their behavior and would be nice to explore.
We visualized 38 different models, including DeiT models trained using distillation. Some of these models are pre-trained on Imagenet21K.
However, since the JFT dataset is private and the models trained on JFT are not released to the public, we cannot perform experiments on JFT.
Thank you again for your thoughtful review. Please also read our general response to all the reviewers. We made a significant effort to address each of your questions and would appreciate it if you would consider raising your score in light of our response. Please let us know if you have additional questions we can address.
[1] Masked Autoencoders Are Scalable Vision Learners
Dear reviewer,
Thank you again for your thoughtful review. Does our response address your questions?
We would appreciate the opportunity to engage further if needed.
Strength: The paper shows that the Maximum activation method can also work in ViTs to visualize the features in feed-forward layers. In most situation, the ViTs show very similar patterns compared to CNN. Some observations are quite interesting such as the spatial information perseverance, the better use of background information, and the attention visualization.
Weakness:
1. The method of visualization is not significantly novel.
2.
Although the paper focuses on visualization, the comparison between CNNs and ViTs uses mostly empirical results. Does the visualization of CNNs and ViTs also show that ViTs are more sensitive to background features, the current visualization is very similar to what expected from a CNN model.
3. Some observations need more explanation: The attention components do not respond well to activation maximization. The author mentioned that this is because "ViTs pack a tremendous amount of information in 768 features", which is not quite clear to me. The author mentions that the neuron in ViTs maybe multi-modal neurons, is there some evidence/visualization to suggest this?
4. The visualization does not show the conceptual categories activation well, as in Fig 9. It is more of a particular example case rather than some types or patterns.
However, this is outside the scope of the paper.
The paper's novelty in term of visualization method is limited, and since most of the behaviors are not totally unexpected given the results from CNNs and previous work, the insights provided in the paper (as discussed above) is also limited.
The results in the paper should be easily replicated.
NO.
Thank you for your useful feedback and comments. Below, we address the concerns raised.
>More specifically, the author uses gradient descent to maximize the sum of a particular feature over all patches to create a maximally activating image, starting from noisy versions of images in the dataset.
We thank the reviewer for bringing this confusion to our attention. Our visualizations are actually seeded with random noise, not with partially noised images from the dataset.
>The method of visualization is not significantly novel.
Many of the techniques we use for visualizations are adopted from [1]. [1] uses these techniques to perform the model class inversion. The novelty of our work is applying these techniques to ViTs for feature visualization and using the results to make observations about neural network behaviors. Visualization of the features of ViTs had not been performed successfully in previous works.
>Some observations need more explanation: The attention components do not respond well to activation maximization. The author mentioned that this is because "ViTs pack a tremendous amount of information in 768 features", which is not quite clear to me. The author mentions that the neuron in ViTs maybe multi-modal neurons, is there some evidence/visualization to suggest this?
Imagenet has 1000 classes, and there are 768 features before the feed-forward layer. Each of these classes may have several distinctive features. Intuitively, the number of features present in the dataset may be far more than 768. We speculate that the presence of multi-modal neurons explains why visualizations before the feed-forward layer are unsuccessful.
Thank you again for your thoughtful review. Please also read our general response to all the reviewers. We made a significant effort to address each of your questions and would appreciate it if you would consider raising your score in light of our response. Please let us know if you have additional questions we can address.
[1] Plug-In Inversion: Model-Agnostic Inversion for Vision with Data Augmentations
I would like to thank the reviewer to clarify different aspects of the paper, especially the initialization condition. My main concern, however, remains. The paper follows a very similar approach to previous work and the insights from the visualization are not quite significant enough. I appreciate the author's insight on the feature space vs class space of Imagenet, and I believe the paper would be much stronger with a deeper analysis/visualization on this point and the multi-modal neurons. As it stands now, I would keep my original recommendations.
Strength
- To my knowledge, this is the first work that visualizes the features of transformers.
Weakness
- The paper is missing references that study the CNNs and transformers (e.g., [1]). And, the related section should be more elaborated.
- Some of the observations are already studied (e.g., high-frequency in [1], the receptive field in [2])
- It is not clear the novel findings from previous works.
- While some observations related to the background are interesting, most of the experiments are done qualitatively but no quantitative analysis is done.
- The experiment in Table 1 does not look convincing to me.
[1] Park, Namuk, and Songkuk Kim. "How Do Vision Transformers Work?." ICLR'22
[2] Do Vision Transformers See Like Convolutional Neural Networks?, NeurIPS'21
The paper is well written and easy to read.
NO.
>It is not clear the novel findings from previous works.
While some observations related to the background are interesting, most of the experiments are done qualitatively but no quantitative analysis is done.
The main contribution of this work is introducing a way to visualize the features of Vision Transformers and to demonstrate how this tool can be used to analyze them. The general framework of this work is that with the help of these visualizations, we make different hypotheses and test them through quantitative analysis. This approach has been used for a long time in interpretability research. With this work being the first paper to visualize the features of ViTs, our approach can be used in future work to pave the way for understanding the inner workings of ViTs.
For example, in Section 5, with the help of feature visualizations, we observe that there are features that detect the background.
As a result, we hypothesize that ViTs can perform better in the absence of background/foreground compared to CNNs. We further validate this hypothesis with quantitative experiments.
Although previous works like [2] and [3] have examined the effect of occlusion of images or removing their salient/non-salient parts on the performance of the ViT models, this work, to the best of our knowledge, is the first to explore the effect of background/foreground removal using the ground truth bounding boxes.
In Section 4, we observe that the activation map of the most activating images for a given feature resembles saliency maps except for the features of the last layer. This means that spatial information is present in individual channels of patches for all layers except the last one. From this, we hypothesize that the last layer is akin to a global average pooling (GAP), meaning every token has information similar to other tokens. If tokens, including the CLS token, have information similar to other patches in the last layer, we wonder how much information the CLS token gathers in earlier layers. To test these hypotheses, we design two quantitative experiments.
First, we isolate the CLS token from other patches in all the layers except the last one, and in the last layer, every token (CLS + patches) once again attends to every other token. The relatively high accuracy shows that the information the CLS token gathers from the other patches is mostly obtained in the last layer.
Second, we attach the linear layer trained on the CLS token to the other patches without modification. As shown in Figure 5, the relative accuracy of different patches obtained this way is high. As shown in table 1, the mean accuracy averaged over patches in this experiment is 75.75 (vs. the CLS token accuracy of 84.2), showing that the information needed for classification is scattered between every token in the last layer. This indicates that the last layer acts like a global average pooling layer.
In Section 6, we investigate the effect of natural language supervision on the type of features learned by ViTs. We observe that natural language supervision forces the model to learn conceptual features such as morbidity and music, as shown in Figure 9 of the paper.
>The experiment in Table 1 does not look convincing to me.
We elaborated on these experiments in our answer to the previous comment. Please let us know if there is any more confusion.
Thank you again for your thoughtful review. Please also read our general response to all the reviewers. We made a significant effort to address each of your questions and would appreciate it if you would consider raising your score in light of our response. Please let us know if you have additional questions we can address.
[1] Park, Namuk, and Songkuk Kim. "How Do Vision Transformers Work?."
[2] Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness
[3] Intriguing properties of vision transformers
Thank you for your useful feedback and comments. Below, we address each of the concerns raised above.
>The paper is missing references that study the CNNs and transformers (e.g., [1]). And, the related section should be more elaborated.
Thanks for pointing this out! We have now updated the related works in the draft and cited the suggested reference to reflect this suggestion.
>Some of the observations are already studied (e.g., high-frequency in [1], the receptive field in [2])
The receptive field of vision transformers is not a subject of study in this paper.
However, as discussed in Section 4, it is surprising that the local information of patches is preserved in different channels of different layers even though every patch can attend to every other patch. The presence of spatial information in different layers is also discussed in [2] by the CKA similarity measure.
However, the presence of this information in single channels, and the resemblance of activations of different channels to saliency maps, is new to this work.
From seeing that ViTs perform better when only the foreground/background is present, we examine their sensitivity to the texture (high-frequency data). Our setting is different from [1]. [1] investigates the drop in performance when a particular frequency is masked.
When dropping low-frequency components, Resnets’ performance drop is less, and when the high-frequency components are dropped, the reverse occurs. We instead drop high-frequencies with a threshold (aggregative), and ViTs perform better at all such thresholds we try. We agree that the effect of high-frequency data has been studied in the past, and we cited relevant works, including [2] and [3], in our work. We also cited [1] in the updated version of the draft.
The rest of the response is in the following comment.
Dear reviewer,
Thank you again for your thoughtful review. Does our response address your questions?
We would appreciate the opportunity to engage further if needed.
(+) They perform visualization on multiple ViT variants.
(+) Experiments related to ViTs with text information (CLIP models) are interesting.
Weakness:
(-) The paper is not well-organized and the writing needs to be improved.
(-) The visualization is corresponding to different layers displayed in different figures without justifications.
Therefore, it is hard to tell if these results are cherry-picked.
(-) Most conclusions drawn from this work are based on qualitative visualization without quantitative results, which makes these conclusions less convincing.
(-) Some of the results are in the expectation without significant contributions to the communities.
For example, the spatial information is lost in the last layer since the classification is performed based on the class-token.
In addition, ViT relies more on background information has also been explored in previous works, e.g, patch-based infill transformation in Qin et al "Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation" (NeurIPS-2022) and Naseer et al "Intriguing Properties of Vision Transformers" (NeurIPS-2021).
The paper is not well-organized and the writing needs to be improved.
In addition, the novelty is not strong enough to support ICLR.
NO.
>In addition, ViT relies more on background information has also been explored in previous works, e.g, patch-based infill transformation in Qin et al "Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation" (NeurIPS-2022) and Naseer et al "Intriguing Properties of Vision Transformers" (NeurIPS-2021).
We would like to thank you for pointing out these papers. We will add a comparison between our method and these methods to our draft. Patch-based Infill, defined in [2], replaces the image patches in the center region of an image with the patches on the image boundary. [3] detects salient and non-salient patches using a self-supervised ViT model DINO. Our setting differs from both of these papers;
however, all three point in the same direction. We use ground truth bounding boxes to remove the foreground/background completely, and we conclude that ViTs perform better than CNNs in both settings.
>The visualization is corresponding to different layers displayed in different figures without justifications.
Therefore, it is hard to tell if these results are cherry-picked.
We included abundant examples of different networks and different layers in the appendix.
Also, to this end, we created an anonymous link with more visualizations to further address this concern:
https://htmlpreview.github.io/?https://github.com/anonymous2022icml/ViTViSShowCase/blob/main/index.html
Thank you again for your thoughtful review. Please also read our general response to all the reviewers. We made a significant effort to address each of your questions and would appreciate it if you would consider raising your score in light of our response. Please let us know if you have additional questions we can address.
[1] Do Vision Transformers See Like Convolutional Neural Networks?
[2] Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation
[3] Intriguing Properties of Vision Transformers
Thank you for your useful feedback and comments. Below, we address each of the concerns raised above.
>Most conclusions drawn from this work are based on qualitative visualization without quantitative results, which makes these conclusions less convincing.
As discussed in the paper, some obstacles prevent good visualizations of features in ViTs. After finding a way around this, we make different hypotheses about the behavior of ViTs based on these visualizations.
However, we test these hypotheses with quantitative experiments. Here we revisit a few examples.
First, we observe abundant qualitative examples where ViTs detect background information in an image, from which we hypothesize that ViTs are better than CNNs at inference with sole access to the background/foreground information. We test this with the background/foreground experiment performed in the paper. This is the general approach we take throughout the paper. Visualizations give us insight that is not available otherwise, and we can test these insights later on through concrete quantitative analysis. We hope our work, which is the first to visualize ViTs features, can pave the way for interpretability of ViT architectures in the future.
As another example, In Section 4, we observe that the activation map of the most activating images for a given feature resembles saliency maps except for the features of the last layer. This means that spatial information is present in individual channels of patches for all layers except the last one. From this, we hypothesize that the last layer is akin to a global average pooling (GAP), meaning every token has information similar to other tokens. If tokens, including the CLS token, have information similar to other patches in the last layer, we wonder how much information the CLS token gathers in earlier layers. To test these hypotheses, we design two quantitative experiments.
First, we isolate the CLS token from other patches in all the layers except the last one, and in the last layer, every token (CLS + patches) once again attends to every other token. The relatively high accuracy shows that the information the CLS token gathers from the other patches is mostly obtained in the last layer.
Second, we attach the linear layer trained on the CLS token to the other patches without modification. As shown in Figure 5, the relative accuracy of different patches obtained this way is high. As shown in table 1, the mean accuracy averaged over patches in this experiment is 75.75 (vs. the CLS token accuracy of 84.2), showing that the information needed for classification is scattered between every token in the last layer. This indicates that the last layer acts like a global average pooling layer.
From seeing that ViTs perform better when only the foreground/background is present, we examine their sensitivity to the texture (high-frequency data). We drop high-frequencies with a threshold (aggregative), and ViTs perform better at all such thresholds we try.
To further address this concern, we perform experiments to justify the presence of spatial information in individual channels of different layers, except for the last one. We include these results in our general response to all reviewers.
>Some of the results are in the expectation without significant contributions to the communities.
For example, the spatial information is lost in the last layer since the classification is performed based on the class-token.
Your intuition is correct.
However, note that this intuition is contrary to some of the findings in the literature. [1] using the CKS similarity measure demonstrates that spatial information is present within patches, even in the last layer. As discussed in the paper, this information is present in individual channels of different layers except the last one.
Therefore, spatial information is present if we look at all the channels of a single token.
Still, if we look at the channels individually, this information is not present in the last layer, which is very similar to global average pooling. We provide quantitative justification for this in our general response to all the reviewers.
The rest of the response is in the following comment.
Dear reviewer,
Thank you again for your thoughtful review. Does our response address your questions?
We would appreciate the opportunity to engage further if needed.
We first thank all the reviewers for their useful feedback and comments.
Due to the reviewer’s request for more quantitative experiments, we designed an experiment to support our claim that spatial information is present in individual channels of all the layers except the last one. These experiments are available in Appendix section F.
In these experiments, we find the most activating images for each feature.
Then, we forward these images to the network and capture the activation maps. We call a patch active if the activation for this patch is higher than 0.5.
First, we mask out the inactive patches, meaning that we replace them with black patches.
Then, we mask out $x$ percent of the active patches in the current image ($0 \leq x \leq 90$).
Then, we forward this new image to the network to capture the new activation map.
Finally, we compare the new activation map with the original. We plot the number/sum of active patches of the modified image divided by the number/sum of the active patches of the initial image for different percentages. We expect a linear trend by increasing the number of unmasked patches if the spatial information is preserved. As we see in figures 16 and 17, all the layers, except for the last one, have roughly a linear trend, indicating that the last layer loses spatial information mostly.
In addition, we created an anonymous link with more visualizations to show the effectiveness of our method: https://htmlpreview.github.io/?https://github.com/anonymous2022icml/ViTViSShowCase/blob/main/index.html

Meta-review:
role: PC
data: metareview:_summary,_strengths_and_weaknesses: The paper analyzes vision transformers, mainly by visualizing patterns preferred by different units in the model, similar to how it's been previously done for ConvNets. Based on these visualizations, some conclusions are made about the properties of ViTs and similarities/differences with ConvNets in terms of background/foreground preference, focus on low/high frequencies, level of abstraction and preservation of spatial information in different layers.

All reviewers agree that the paper is slightly below the acceptance threshold, with the main pro being that the visualizations are detailed and curious, but the main cons being that novelty is limited since the visualization method is largely similar to existing ones, and the conclusions are somewhat limited since they are mostly based on qualitative results (although there are some quantitative results too). 

I concur with the reviewers and recommend rejection at this point. The presented visualizations are interesting and clearly, a lot of work went into them, but the paper doesn't quite cut it in terms of the combination of novelty and solid experimental results. It would make a great workshop contribution, though.
justification_for_why_not_higher_score: The paper doesn't quite cut it in terms of the combination of novelty and solid experimental results: the novelty is limited since the visualization method is largely similar to existing ones, and the conclusions are somewhat limited since they are mostly based on qualitative results.
justification_for_why_not_lower_score: N/A

Decision: Reject