What do Vision Transformers Learn?  A Visual Exploration
Authors: Amin Ghiasi
Hamid Kazemi
Steven Reich
Eitan Borgnia
Manli Shu
Micah Goldblum
Andrew Gordon Wilson
Tom Goldstein
Venue: Submitted to ICLR 2023 (2023)
Keywords: Vision Transformers
Visualization
Interpretability

Abstract: Vision transformers (ViTs) are quickly becoming the de-facto architecture for computer vision, yet we understand very little about why they work and what they learn. While existing studies visually analyze the mechanisms of convolutional neural networks, an analogous exploration of ViTs remains challenging. In this paper, we first address the obstacles to performing visualizations on ViTs. Assisted by these solutions, we observe that neurons in ViTs trained with language model supervision (e.g., CLIP) are activated by semantic concepts rather than visual features. We also explore the underlying differences between ViTs and CNNs, and we find that transformers detect image background features, just like their convolutional counterparts, but their predictions depend far less on high-frequency information. On the other hand, both architecture types behave similarly in the way features progress from abstract patterns in early layers to concrete objects in late layers. In addition, we show that ViTs maintain spatial information in all layers except the final layer. In contrast to previous works, we show that the last layer most likely discards the spatial information and behaves as a learned global pooling operation. Finally, we conduct large-scale visualizations on a wide range of ViT variants, including DeiT, CoaT, ConViT, PiT, Swin, and Twin, to validate the effectiveness of our method.

Under review as a conference paper at ICLR 2023
WHAT DO VISION TRANSFORMERS LEARN? A VISUAL EXPLORATION
Anonymous authors Paper under double-blind review
ABSTRACT
Vision transformers (ViTs) are quickly becoming the de-facto architecture for computer vision, yet we understand very little about why they work and what they learn. While existing studies visually analyze the mechanisms of convolutional neural networks, an analogous exploration of VITs remains challenging, In this paper, first address the obstacles to performing visualizations ViTs. Assisted by these solutions, we observe that neurons ViTs trained with language model supervision (e.g., CLIP) are activated by semantic concepts rather than visual features. We also explore the underlying differences between ViTs and CNNs, and we find that transformers detect image background features, just like their convolutional counterparts, but their predictions depend far less on fegh-freeuency information. On the other hand, both architecture types behave similarly the way features progress from abstract patterns in early layers to concrete objects in late layers. In addition, we show that ViTs maintain spatial information in all layers except the final layer. In contrast to previous works, show that the last layer most likely discards the spatial information and behaves as a learned global pooling operation. Finally, ViT variants, including DeiT, CoaT, Con VIT, PIT, Swin, and Twin, to validate the effectiveness of our method.
conduct large-scale visualizations
a wide range
INTRODUCTION
Recent years have the rapid proliferation of vision transformers (ViTs) across diverse range of tasks from image classification to semantic segmentation to object detection (Dosovitskiy et al.) 2020; He al. 2021 Dong et al. 2021 Liu et al. 2021] Zhai al. 2021 Dai et | 202 Despite their enthusiastic adoption and the constant introduction of architectural innovations, little is known about the inductive biases or features they tend to learn. While feature visualizations and image reconstructions have provided a looking glass into the workings of CNNs (Olah et al.) 2017 Zeiler & Fergus 2014 Dosovitskiy & Brox 2016), these methods have shown less success for understanding ViT representations, which are difficult to visualize. In this work we show that, if properly applied to the correct representations, feature visualizations can indeed succeed on VITs. This insight allows us to visually explore ViTs and the information they glean from images.
Edges
Textures
Patterns
Parts
Objects
Figure 1: The progression for visualized features of VIT B-32. Features from early layers capture general edges and textures. Moving into deeper layers, features evolve to capture more specialized image components and finally concrete objects.

---

Under review as a conference paper at ICLR 2023
In order to investigate the behaviors of vision transformers, first establish visualization framework that incorporates improved techniques for synthesizing images that maximally activate neurons. Through dissecting and visualizing the internal representations in the transformer architecture, find that patch tokens preserve spatial information, even in individual channels, throughout all layers except the last attention block. The last layer of ViTs learns token-mixing operation akin average pooling, such that the classification head exhibits comparable accuracy when ingesting a random token instead of the CLS token.
After probing the role of spatial information, we delve into the behavioral differences between ViTs and CNNs. When performing activation maximizing visualizations. we notice that ViTs consistently generate higher quality image backgrounds than CNNs. Thus, we try masking out image foregrounds during inference, and find that ViTs consistently outperform CNNs when exposed only to image backgrounds. These findings bolster the observation that transformer models extract information from many sources in an image exhibit superior performance on -iit---distribution generalization (Paul & Chen 2021) well as adversarial robustness (Shao et al. 2021). Additionally, convolutional neural networks are known to rely heavily on high- sih-fireueeccy texture information in images (Geirhos) et al. 2018). In contrast, removed from their inputs.
find that ViTs perform well even when high-frequeng: content is
We further visualize the effects of language model supervision, i.c. CLIP (Radford et al. 2021 the features extracted by vision transformers. While both ImageNet-trained ViT's and Cr.Pe-rriied vision transformers possess neurons that activated by visual features (e.g. shapes and colors) and distinct classes, the neurons of CLIP-trained vision transformers are also activated by features that do represent physical objects, such as visual characteristics relating parts of speech (e.g, epithets, adjectives, and prepositions) or broader concepts such as morbidity.
Our contributions are summarized follows:
I. We observe that uninterpretable and adversarial behavior occurs when applying standard methods of feature visualization the relatively low-dimensional components of transformer-base models, such as keys, queries, or values. However, applying these tools to the relatively high-dimensional features of the position-wis feedforward layer results in successful and informative visualizations. We conduct large-scale visualizations on wide range of transformer- bbaed vision models, including ViT', DeiT, CoaT, ViT, PiT, Swin, and Twin, to validate the effectiveness of our method.
II. We show that patch-wise image activation patterns for ViT features essentially behave like saliency maps, highlighting the regions of the image a given feature attends to. This behavior persists even for relatively deep layers, showing the model preserves the positional relationship between patches instead of using them as global information stores.
III. We compare the behavior of ViTs and CNNs, finding that ViTs make better use of background information and rely less on tigh-frequency, textural attributes. Both types of networks build progres- sively more complex representations in deeper layers and eventually contain features responsible for detecting distinct objects.
IV. We investigate the effect of natural language supervision with CLIP the types of features extracted by ViTs. We find CLIP-trained models include various features clearly catered to detecting components of images corresponding to caption text, such as prepositions, adjectives, and conceptual categories.
2 RELATED WORK
2.1 OPTIMIZATION- CIAIIIAATIO-BSEED VISUALIZATION
One approach to understanding what models learn during training is using gradient descent to produce an image which conveys information about the inner workings of the model. This has proven to be a fruitful line of work in the case of understanding CNNs specifically. The basic strategy underlying this approach is to optimize over input space to find an image which maximizes a particular attribute of the model. For example, EErhan et al. (2009) use this approach to visualize images which maximally activate specific neurons in early layers of network, and Olah et al. (2017) extend this to neurons, channels, and layers throughout network. [Simonyan et al. (011 Yin al. (2020) produce images
2

---

Under review as a conference paper at ICLR 2023
which maximize the score model assigns to a particular class. Mahendran & Vedaldi (2015) apply similar method to invert the feature representations of particular image examples.
Recent work Ghiasi al. (2021) has studied techniques for extending optimization- based class visualization to ViT's We incorporate and adapt some of these proposed techniques into our scheme for feature visualization.
2.2 OTHER VISUALIZATION APPROACHES
Aside from optimization-basse methods, many other ways to visualize CNNs have been proposed. Dosovitskiy & Brox (2016) train an auxiliary model to invert the feature representations of a CNN. Zeiler & Fergus (2014) use 'deconvnets' to visualize patches which strongly activate features in various layers. Simonyyn et al. (0014) introduce saliency maps, which use gradient information to identify what parts of an image are important to the model's classification output. Zimmermann et al.) (202T) demonstrate that natural image samples which maximally activate a feature in a CNN may be more informative than generated images which optimize that feature. We draw on some aspects of these approaches and find that they are useful for visualizing VITs as well.
2.3 UNDERSTANDING VITS
Given their rapid proliferation, there is naturally great interest in how ViTs work and how they may differ from CNNs. Although direct visualization of their features has not previously been explored, there has been recent progress in analyzing the behavior of ViTs. Paul & Chen (2021); ); Naseer et al. (2021;; ; Shao et al. (2021) demonstrate that ViTs are inherently robust to many kinds of adversarial perturbations and corruptions. Raghu et al. (2221) compare how the internal representation structure and use of spatial information differs between VIT's and CNNs. Chefer et al. (2021) produce 'image relevance maps' (which resemble saliency maps) promote interpretability of ViT's. Park & Kim
(2022) demonstrates that multi-head self-attention are low-pass filters, but Convs are high-pass filters. They also examined the effect removing a specific frequency and concluded that ViTs are resistant to high-frequency removal and CNNs are resistant to low- cow-frequency removal.
3 VIT FEATURE VISUALIZATION
Like many visualization techniques, we take gradient steps maximize feature activations starting from random noise (Olah et al. 2017). To improve the quality of our images, we penalize total variation (Mahendran Vedaldi 2015). and also employ the Jitter augmentation (Yin al.) 2020). the ColorShift augmentation, and augmentation ensembling (Ghiasi et al., 2021). Finally, we find that Gaussian smoothing facilitates better visualization our experiments as is common in feature visualization (Smilkov al. 2017 Cohen et al. 2019).
Each of the above techniques can be formalized as follows. A ViT represents each patch p (of an input æ) layer / by array At,p with d entries. We define a feature vector f to be a stack composed of one entry from each of these arrays. Let ft, be formed by concatenating the ith entry in At,p for all patches p. This vector f will have dimension equal to the number of patches. The optimization objective starts by maximizing the sum of the entries of f over inputs x. The main loss is then
L-nain (I,, Evii)e
P
We employ total variation regularization by adding the term XTV(x) to the objective. TV represents the total variation, and X is the hyperparameter controlling the strength of its regularization effect. We can ensemble augmentations of the input to further improve results. Let A define a distribution of augmentations to be applied to the input image x, and let be sample from A. To create minibatch of inputs from single image, we sample several augmentations {aş} from A. Finally, the optimization problem is:
x0 = arg max Σ
Cunin (as (x),1,) + (Tv))
(2)
3

---

Under review as a conference paper at ICLR 2023
(a)
(b)
Figure 2: (a): Example feature visualization from ViT feed forward layer. Left: Image optimized to maximally activate feature from layer 5. Center: Corresponding maximally activating ImageNet example. Right: The image's patch-wise activation map. (b): feature from the last layer most activated by shopping carts.
We achieve the best visualizations when A (S(CS(iitt))())), where GS denotes Gaussian smoothing and CS denotes ColorShift, whose formulas are:
GS(x) = x c~ 100,
CS(() =
u(-1,1; a~ (((111)
Note that even though and are both additive noise, they act on the input differently since applied per channel (i.e. has dimension three), and e is applied per pixel. For more details on hyperparameters, refer to Appendix B
To better understand the content of visualized feature, we pair every visualization with images from the ImageNet validation/train set that most strongly activate the relevant feature. Morcover, plot the feature's 's activation pattern by passing the most activating images through the network and showing the resulting pattern of feature activations. Figure 2(a) an example of such visualization. From the leftmost panel, we hypothesize that this feature corresponds to gravel. The most activating image from the validation set (middle) contains lizard pebbly gravel road. Interestingly, the gravel background lights up in the activation pattern (right), while the lizard does not. The activation pattern in this example behaves like saliency map (Simonyan et al. 2014), and we explore this phenomenon across different layers of the network further in Section
The model we adopt for the majority of our demonstrations throughout the paper is VIT-B16, implemented based on the work of Dosovitski et al.] (2020). In addition, in the Appendix, we conduct large-scale visualizations
L1 FO FI
LI F2 LII FO LI1 F2
key
wide range of ViT vari- ants, including TouT'ouuvron et al. (2021a)
query
CoaT Xu et al. (2021), Con ViT d' Ascoli
ct al. (021). PiT Heo et al. (2021). Swin Liu et al. (2021) ). and Twin Chu al. (202)). 38 models total, to validate the effectiveness of our method. VIT-B16 is composed of 12 blocks, each consisting of multi-headed attention layers, followed by projection layer for mixing attention heads, and finally followed by a position- wise-feed feed-fffeffrrr layer. For brevity, we henceforth refer to the position-wel position-wise-feed wiee-feed- forward layer simply the feed-forward layer. In this model, every patch is always represented by vector of size 768 except in the feed-forward layer which has size of 3072 (4 times larger than other layers).
value
Figure 3: Left: Visualization of key, query, and value. The visualization both fails to extract interpretable fea- tures and to distinguish between early and deep layers. High-frequency patterns and adversarial behavior domi- nate. Right: ViT feed forward layer. The first linear layer increases the dimension of the feature space, and the second one brings it back its initial dimension.
We first attempt to visualize features of the multi-headed attention layer, including visualization of the keys, queries, and values, by performing activation maximization. We find that the visualized feed-for forward features significantly interpretable than other layers. We attribute this difficulty
4

---

Under review as a conference paper at ICLR 2023
LI F31
L4 F9
L6 F2
L7 F37
L8 F19
LII Fl6
Ours
Val
Figure 5: Feature activation maps in internal layers can effectively segment the contents of an image with respect to a semantic concept. For each image triple, the visualization on top shows the result of our method, the image on the bottom left is the most activating image from the validation set and the image on the bottom right shows the activation pattern.
of visualizing other layers to the property that ViTs pack tremendous amount of information into only 768 features, (c.g. in keys, queries, and values) which then behave similar to multi-modal neurons, as discussed by Goh et al. (2021), due to many semantic concepts being encoded in a low dimensional space. Furthermore, we find that this behaviour is more extreme in deeper layers. See Figure 3 for examples of visualizations of keys, queries and values in both early and deep layers of the VIT. Inspired by these observations,
visualize the features within the feed-forward layer across all 12 blocks of the ViT. We refer to these blocks interchangeably as layers.
The feed-forward layer depicted in Figure 4 takes input of size = 768, projects it into f = 4 times higher dimensional space, applies the non-linearity GELU, and then projects back dimen- sional space. space. Unless otherwise stated, we always visualize the output of the GELU layers in our experiments. We hypothesize that the network exploits these nigh-dimensional spaces to store relatively disentangled representations. On the other hand, compressing the features into a lower dimensional space may result in the jumbling of features, yielding uninterpretable visualizations.
FC2 Input Shape: (b, p. t d) Output Shape: (b, p. d)
GELU
4 LAST-LAYER TOKEN MIXING
FC1 Input Shape: (b, p. d)
In this section, we investigate the preservation of patch-wiss spa- tial information observed in the visualizations of patch-wise feature activation levels which, noted before, bear some similarity to saliency maps. Figure ((a) demonstrates this phenomenon in layer 5, where the visualized feature is strongly activated for almost all rocky patches but not for patches that include the lizard. Additional examples be seen in Figure 5 and the Appendix, where the acti- vation maps approximately segment the image with respect to some relevant aspect of the image. We find it surprising that even though every patch can influence the representation of every other patch, these representations remain local, even for individual channels in deep layers in the network. While similar finding for CNNs, whose neurons may have limited receptive field, would be unsurprising, even neurons in the first layer of a VIT have complete receptive field. In other words, ViT's learn to preserve spatial information,
Output Shape: (b, p. X d)
VIT feed forward layer. The first linear layer increases the dimension of the feature space, and the second one brings it back to initial dimension.
Figure 4:
despite lacking the inductive bias of CNNs. Spatial information in patches of deep layers has been explored in Raghu et al. (202I) through the CKA similarity measure, and we further show that spatial information in fact present individual channels.
The last layer of the network, however, departs from this behavior and instead appears to serve a role similar average pooling. Figure 2(b]]shw one example of our visualizations for feature from the last layer that is activated by shopping carts. The activation pattern is fairly uniform across the image. For classification purposes, ViTs use a fully connected layer applied only on the class token
5

---

Under review as a conference paper at ICLR 2023
(the CLS token). It is possible that the network globalizes information in the last layer to ensure that the CLS token has access to the entire image, but because the CLS token is treated the same as every other patch by the transformer, this seems to be achieved by globalizing across all tokens.
Table After the last layer, every patch contains the same information. "Isolating CL.S" denotes the experiment where attention is only performed between patches before the final attention block, while "Patch Average" and "Patch Maximum" refer to the experiment which the classification head is placed on top of individual patches without fine-tuning. Experiments conducted on ViT-B16.
Accuracy
Natural Accuracy
Isolating CLS
Patch Average
Patch Maximum
Top 1
84,20
78.61
75.75
80.16
Top
97.16
94.18
90.99
95.65
Based on the preservation of spatial information in patches, we hypothesize that the CLS token plays a relatively minor role throughout the network and To demonstrate this, we perform inference on images without using the CLS token in layers 1-11, meaning that these layers, each patch only attends to other patches and not the CLS token. At layer 12, we then insert value for the CLS token so that other patches can attend to it and vice versa. This value is obtained by running a forward pass using only the CLS token and no image patches; this value is constant across all input images.
used for globalization until the last layer.
The resulting hacked network that only has CLS access the last layer can still successfully classify 78.61 78.61% of the ImageNet validation set shown in Table [ From this result, conclude that the CLS token captures global information mostly at the last layer, rather than building global representation throughout the network.
We perform a second experiment to show this last- layer globalization behaviour is not exclusive the CLS token, but actually occurs across every patch in the last layer. We take the fully connected layer trained to classify images on top of the CLS token, and without any fine-tuning or adaptation, we apply it to each patch, one at time. This setup still suc- cessfully classifies 75. 75% of the validation set, on average across individual patches, and the patch with the maximum performance achieves 80.16% accu- racy (see Table D. further confirming that the last layer performs a token mixing operation so that all to- kens contain roughly identical information. Figure contains a heat-map depicting the performance of this setup across spatial patches. This observation stands in stark contrast to the suggestions of |Rahhu et al. (202T) that ViTs possess strong localization through- out the entire network, and their further hypothesis that the addition of global pooling required for mixing tokens at the end of the network.
VIT-B16
60
50
12 18 24
Figure 6: Heat map of classification accuracy on the validation set when we apply the clas- sification head trained to classify images on the top of the CLS token to the other patches.
We conclude by noting that the information structure of a VIT is remarkably similar to a CNN, in the sense that the information is positionally encoded and preserved until the final layer. Furthermore, the final layer in ViT's appears to behave as learned global pooling operation that aggregates information from all patches, which is similar to its explicit vverace-pooling counterpart in CNNs.
5 COMPARISON OF VITS AND CNNS
As extensive work has been done to understand the workings of convolutional networks, including similar feature visualization and image reconstruction techniques to those used here, we may be able to learn more about ViT behavior via direct comparison CNNs. An important observation is that

---

Under review as a conference paper at ICLR 2023
Texture
Parts
Object
L.5 F32
L6 F58
1.7 F10
L7 F44
199 F34
L10 FI1
Ours
Train Val
Figure 7: Complexity of features depth in VIT B-32. Visualizations suggest that ViTs are similar to CNNs in that they show feature progression from textures to parts to objects as we progress from shallow deep features.
in CNNs, early layers recognize color, edges, and texture, while deeper layers pick out increasingly complex structures eventually leading to entire objects (Olah et al. 2017). Visualization of features from different layers in a VIT, such as those in Figures and reveal that ViTs exhibit this kind of progressive specialization as well.
On the other hand, we observe that there are also important differences between the ways CNNs and ViTs recognize images. In particular, we examine the reliance of ViTs and CNNs on background and foreground image features using the bounding boxes provided by ImageNet Deng et al.) (2009). We filter the ImageNet-Ik Ik training images and only use those which are accompanied by bounding boxes. several objects are present in an image, we only take the bounding boxes corresponding to the true class label and ignore the additional bounding boxes. Figure 8(b) shows example of an image and variants which the background and foreground, respectively, are masked.
(a)
Figure 8: (a): VIT-B16 detects background features. Left: Image optimized to maximally activate feature from layer 6. Center: Corresponding maximally activating example from ImageNet. Right: The image's patch-wise activation map. (b): An example an original image and masked foreground and background.
Figure ((a) displays an example of ViTs' ability to detect background information present in the ImageNet dataset. This particular feature appears responsible recognizing the pairing grass and snow. The rightmost panel indicates that this feature is solely activated by the background, and not at all by the patches of the image containing parts of the wolf.
To quantitatively assess each architecture's 's dependence different parts of the image the dataset level, we mask out the foreground or background on a set evaluation images using the aforementioned ImageNet bounding boxes, and measure the resulting change in top-5 accuracy. These tests are performed across a number of pretrained VIT models, and we compared to a set of common CNNs in Table 2 Further results can be found in Table
We observe that VITs are significantly better than CNNs at using the background information in image identify the correct class. At the same time, ViTs also suffer noticeably less from the
7

---

Under review as a conference paper at ICLR 2023
removal of the background, and thus seem to depend less on the background information to make their classification. A possible, and likely, confounding variable here the imperfect separation of the background from the foreground in the ImageNet bounding box data set. A rectangle containing the wolf in Figure [(a) for example, would also contain small amount of the grass and snow at the wolf's feet. However, the foreground is typically contained entirely in bounding box, so masking out the bounding box interiors is highly effective at removing the foreground. Because ViTs are better equipped to make sense of background information, the leaked background may be useful for maintaining superior performance. Nonetheless, these results suggest that ViTs consistently outperform CNNs when information, either foreground background, missing.
Next, we study the role of texture in ViT predictions. To this end, we filter out high-frequency components from ImageNet test images via low- pass filtering. While the predictions of ResNets suffer greatly when high- frequency texture information is re- moved from their inputs, ViTs are seemingly resilient. See Figure [5] for the decay in accuracy of ViT and ResNet models textural informa- tion is removed.
Table 2: ViTs more effectively correlate background infor- mation with correct class. Both foreground and background data are normalized by full image top-5 accuracy.
Normalized Top-5 ImageNet Accuracy
Architecture
Full Image
Foreground
Background
ViT-B32
98.44
93.91
28.10
VIT-L16
99.57
96,18
33.69
ViT-L32
99.32
93,89
31.07
VIT-B16
99.22
95.64
31.59
ResNet-50
98.00
89.69
18.69
ResNet-152
98.85
90.74
19.68
6 VITS WITH LANGUAGE MODEL SUPERVISION
MobileNetv2
96.09
86.84
15.94
DenseNet121
96.55
89.58
17.53
Recently, ViTs have been used backbone to develop image classifiers trained with natural language supervi- sion and contrastive learning techniques (Radford et al. 2021). These CLIP models are state-of-the-art transfer learning to unseen datasets. The zero-shot ImageNet accuracy of these models even competitive with traditionally trained ResNet-50 competitors. We compare the feature visualizations for ViT models with and without CLIP training to study the effect of natural language supervision on the behavior of the transformer- basedd backbone.
The training objective for CLIP models consists of matching the correct caption from list of options with an input image (in feature space). Intuitively, this procedure would require the network to extract features not only suitable for detecting nouns (e.g. simple class labels like 'bird'), ) but also modifying phrases like prepositions and epithets. Indeed, we observe several such features that not present in ViTs trained solely as image classifiers.
(a) Before and after/Step-hy-step
(b) From above
(c) Many
Figure 9: Left: Feature optimization shows sharp boundaries, and maximally activating ImageNet examples contain distinct, adjacent images. Middle: Feature optimization and maximally activating ImageNet photos show images from an elevated vantage point. Right: Feature optimization shows crowd of people, but maximally activating images indicate that the repetition of objects is more relevant than the type of object.
Figure ((a) shows the image optimized to maximally activate feature in the fifth layer of VIT CLIP model alongside its two highest activating examples from the ImageNet dataset. The fact that all three images share sharp boundaries indicates this feature might be responsible for detecting caption texts relating to progression of images. Examples could include "before and after," as the airport images or the adjective "step-by-step" for the iPod teardown. Similarly, Figure 9(b) and (c)
8

---

Under review as a conference paper at ICLR 2023
(a) Category of morbidity
(b) Category of music
Figure 10: Features from ViT trained with CLIP that relates to the category of morbidity and music. Top-left image in each category: Image optimized to maximally activate feature from layer 10. Rest: Seven of the ten ImageNet images that most activate the feature.
depict visualizations from features which seem to detect the preposition "from above", and adjectives relating to a multitude of the same object, respectively.
The presence of features that represent conceptual categories is another consequence of CLIP training. Unlike ViT's trained as classifiers, in which features detect single objects or common background information, ClaIP-traine VITs produce features in deeper layers activated by objects in clearly discernible conceptual categories. For example, the top left panel of Figure [O(aa shows feature activated by what resembles skulls alongside tombstones. The corresponding seven highly activating images from the dataset include other distinct objects such as bloody weapons, zombies, and skeletons. From strictly visual point of view, these classes have very dissimilar attributes, indicating this feature might be responsible for detecting components of an image relating broadly to morbidity. In Figure TO()) we see that the top leftmost panel shows a disco ball, and the corresponding images from the dataset contain boomboxes, speakers, record player, audio recording equipment, and performer. Again, these are visually distinct classes, yet they are all united by the concept of music.
Given that the space of possible captions for images is substantially larger than the mere one thousand classes in the ImageNet dataset, high performing CLIP models understandably require higher level organization for the objects they recognize. Moreover, the CLIP dataset is scraped from the internet, where captions are often more descriptive than simple class labels.
7 DISCUSSION
In order to dissect the inner workings of vision transformers, we introduce a framework for optimization-basee feature visualization. We then identify which components of ViT are most amenable to producing interpretable images, finding that the high-dimensional inner projection of the feed-forward layer is suitable while the key, query, and value features of self-attention are not.
Applying this framework to said features, we observe that ViTs preserve spatial information of the patches even for individual channels across all layers with the exception of the last layer, indicating that the networks learn spatial relationships from scratch. We further show that the sudden disap- pearance of localization information in the last attention layer results from a learned token mixing behavior that resembles average pooling.
In comparing CNNs and ViTs, we find that ViTs make better use of background information and are able to make vastly superior predictions relative to CNNs when exposed only to image backgrounds despite the seemingly counter-intuitive property that ViTs are not as sensitive as CNNs the loss of high-frequency information, which one might expect to be critical for making effective use of background. We also conclude that the two architectures share a common property whereby earlier layers learn textural attributes, whereas deeper layers learn high level object features or abstract concepts. Finally, we show that ViTs trained with language model supervision learn more semantic and conceptual features, rather than spjecc-specific visual features is typical of classifiers.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We make our code repository available at: htttss//ggtaau.o.commmmoooooosop ViTVis
REFERENCES
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
782-791, 2021.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. arXiv preprint arXiv:2104. 1(2):3, 2021.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. International Conference on Machine Learning, pp. 1310-1320. PMLR, 2019.
Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. arXiv preprint arXiv:2 044003,,, 2021.
Stéphane d' Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint arXiv:2103.10697, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-24882 255. Icee, 2009.
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-training of vision transformers. arXiv preprint arXiv:2111.1277 2021.
Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2016.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. University of Montreal, 1341(3):1, 2009.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: 12231, 2018.
Amin Ghiasi, Hamid Kazemi, Steven Reich, Chen Zhu, Micah Goldblum, and Tom Goldstein. Plug-in inversion: Model-agnostic inversion for vision with data augmentations. 2021.
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons artificial neural networks. Distill, 6(3)e30, 2021.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. arXiv preprint arXiv:2111.0633 2021.
10

---

Under review as a conference paper at ICLR 2023
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Scong Joon Oh. Rethinking spatial dimensions of vision transformers. arXiv preprint 0rXiv:2103.16302. 2021.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong. al. Swin transformer v2: Scaling up capacity and resolution. arXiv preprint arXiv:2111.000 1.09883, 2021.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5188 1198-116,950 2015.
Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Intriguing properties of vision transformers. arXiv preprini 1Xiv:2005.10497, 2021.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 1/22399//iitiill0000000 http:://istill.pub/2001 wif/a/trr/eeatreeeee
hftp///iiteppb//200////tttttree http.////.////////////////////// citecattreevvsuulzzaatin
Namuk Park and Songkuk Kim. How do vision transformers work? arXiv preprint arXiv:2202.06709, 2022.
Sayak Paul and Pun-Yu Chen. arXiv:2105.07581, 2021.
Vision transformers are robust learners. arXiv preprint
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00022 2021.
Maithra Raghu, Thomas Unterthiner, Simon Korbllith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34, 2021.
Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin-Yu Yu Chen, and Cho-Jui Hsich. On the adversarial robustness of visual transformers. arXiv preprint arXiv:2103.156 2021.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In In Workshop at International Conference Learning Representations, 2014.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint 1rXiv:1706.0333 03825, 2017.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, volume 139, pp- 10347-10357 July 2021a.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé
Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021b.
Ross Wightman. Pytorch image models.
com wtt.://h....o.../...t..aaaaaaaaa
pytorch- siaurch-immge-e models 2019.
Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers.
arXiv preprint arXiv:2 (04.06399, 2021.
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8715-8724, 2020.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference computer vision, pp. 818-833. Springer, 2014.
11

---

Under review as a conference paper at ICLR 2023
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv preprint 120.v:::0060045500, 2021.
Roland Zimmermann, Judy Borowski, Robert Geirhos, Matthias Bethge, Thomas Wallis, and Wieland Brendel. How well do feature visualizations support causal understanding of enn activations? Advances in Neural Information Processing Systems, 34, 2021.
12

---

Under review as a conference paper at ICLR 2023
A FAILED EXAMPLES
Figure 11 shows few examples of our visualization method failing when applied on low dimensional spaces. We attribute this to entanglement of more than 768 features when represented by vectors of size 768. We note that, due to skip connections, activation in previous layers can cause activation in the next layer for the same feature, consequently, the visualizations of the same features in different layers can share visual similarities.
13

---

Under review as a conference paper at ICLR 2023
Feature 0
Feature
Feature 2
Feature
Layer 0
Layer
Layer 2
Layer 3
Layer 4
Layer
Layer 6
Layer 7
Layer
Layer 9
Layer 10
Layer 11
Figure 11: Some examples of failed visualizations on the input of the attention layers. Same visualization technique fails when applied low dimensional (e.g. key, query, value, etc) spaces. We believe that the visualization shows roughly meaningful and interpretable visualizations in early layers, since there not many different features be embedded. However, in deeper layers the features are entangled, so is more difficult to visualize them. For every example, the picture on the left shows the results of optimization and the picture on the right shows the most activating image from ImageNet validation set.
14

---

Under review as a conference paper at ICLR 2023
B EXPERIMENTAL SETUP AND HYPERPARAMETERS
As mentioned before, we ensemble augmentations to the input. More specifically, we use A is GS(CS(Jitter(a)) as our augmentation. The bound for Jitter is (-32, 32) for both directions vertical and horizontal. The hyper parameters for CS are always mean = 1.0 and std = 1.0 in all of the experiments. For GS, the mean is always however, for the std we have linear scheduling. where at the beginning of the optimization the std 0.5 and at the end of the optimization std = 0.0. use a batch-size n = 8 all our experiments. We use ADAM as our choice of optimizer with 3 = (0.5. 0.99). Optimization is done in 400 steps and at every step, we re-sample the augmentations Jitter, GS and CS. We also use CosineAnealing learning rate scheduling, starting from Ir at the beginning and / the end. The
used for total variation Xty- 0.00005.
For all VIT experiments, we use the pretrained models from https://gghubbboom//kkemm/a//// Pretrained-ViT For clip models, we use are from httpss//gihubb.co//ytorrrt com/pytorch//iiioon
pretrained models by Wightman (2019). The rest of models
For all of our experiments, use GeForee RTX 2080 Ti GPUs with 12GB of memory. All inferences on ImageNet are done under 20 minutes on validation set and under 1 hour on training set using only 1 GPU. All visualization experiments take most 90 seconds to complete.
C MODELS
In our experiments, publicly available pre-trained models from various sources. The following tables list the models used from each source, along with references to where they are introduced in the literature.
Name
Paper
B_ B_16 16 imagenet
Dosovitskiy et 2021
B B_32 im2agnneett
Dosovitskiy et al. (202T)
Figure 12: Pre-trained models used from
http::/gthhb.ooooo/llelaa//////p ViT
Name
Paper
basc_patch16 224
Touvron ct al.
2021b
beit_baae_distilledd d 384
Touvron al.
2021b
deit_base 6 384
Touvron et al.
2021b
tittttiitillleddddeeeeai deit seit_tiny_aae distilled 1 224
Touvron et al.
2021b
deit_small Becttilled_pptchhlee
Touvron el al.
2021b
maase_disillled_pttee
Touvron et al.
2021b
Figure 13: Pre-trained models from Touvron et 2202a
15

---

Under review as a conference paper at ICLR 2023
Name coat coat_lite_ming
Paper
Xu et al.
2021
coat lite site_smmall coat _ite__tee____eee_ tiny coat_mn _ii__mini coat_tiny convit_base
Xu ct al
2021
Xu ct al.
2021
Xu et al.
2021
Xu
202T
AAscoli et al.
(202
convit_small convit tiny
Asscoli al.
(202)
Ascoli et al.
(202
pit 224
Heo et al.
2021
pit b distilled 224
Hco ct al.
2021
pit_s_22 224
Heo et al.
202
pit pit_s_distilled 224
Heo et al
2021
2i2_ti_222 224
Heo et al.
pit_ti_disste diitii_dstilled 224
Heo et al.
2021
swin_base patch4 window7 7_24 7_224
Liu et al.
202
swin base _acee__pacch____
wwt.owwwwww...m..m 224 in22k
Liu et al. 202
swin_ _as__ase_______e
caase_pathh__weee Bianch4wwindwwwl11 12_384
L.u et al.
202
swin_ La__baa____atee sete_patch__windooee 12_34 in22k
L.u al. 202
laiin_llrge_aaachhe___
_24
1.11 et al. 2021
swin
window7 _24 in22k
202
lain_large___aa__
window 2_388444
Liu ct al.
2021
swin larglarre___r
harg__patc4____ window12 12 384 in22k
al. 202
swin swin_small swalss
Lamnall_atcch__gg_11 haatl____haa_ddoww/> w 224
Liu et al. 2021
swin_tiny__pachh__wing
__4
Liu et al.
202
twins_pepvt_bar
Chu et al
2021
twins_pepvtt lareeelaappeeeeeeeeeeppe
Chu ct al.
202
twins wwins_pcpvt___. pcppltssmalllllllllppppppppppple small
Chu ct al.
2021
(wins_svt_bee base
Chu ct al.
2021
twins_svt_lars
Chu et al.
2022
twins_svt_snen
Chu et al.
2021
Figure 14: Pre-trained models used from: Wightman (2019
16

---

Under review as a conference paper at ICLR 2023
D
EFFECT OF LOW-PASS FILTERING
Top-1 accuracy on ImageNet (%)
100
80
60
40
VIT B/16
ResNos 30
0.5
0.4
0.3 Low-pass threshold
0.2
0.1
Figure 15: Effect of low-pass filtering on top-1 ImageNet accuracy. CNNs are more dependent on high frequency textural image information than ViTs.
E ADDITIONAL VISUALIZATIONS
17

---

Under review as a conference paper at ICLR 2023
LO F29
LO F30
LO F22
LI F31
LI F29
L2 F5
L2
Ours
Val
L2 F36
L3 F35
L3 F27
L3 F37
L5
L4 F9
F4
Ours
Val
L5
FIO
L5 F24
F1
L6 F2
L6 F6
L6 F9
Ours
Val
L6 F20
L6 F27
L7FI
L7 F5
L7 F4
L7
L7 F14
Ours
Val
L7 F16
L7 F17
L7 F24
L7 F37
L8 FO
L8 F3
LS F5
Ours
Val
Figure 16: Visualization of ViT-base-patch16 16
18

---

Under review as a conference paper at ICLR 2023
L8 F19
L8 F22
L8 F28
L FO
L9 F25
L10 FO
L10 F9
Ours
Val
L10 F22
L10 F20
L10 F27
L10 F37
L11
L11 F16
F33
Ours
Val
Figure 17: (Cont.) Visualization of ViT-hase-patch16
19

---

Under review as a conference paper at ICLR 2023
LO LO F17
F22
F7
L1 F33
L1 F36
L1 F8
L2 FO
Ours
Val
Train
F1
L2 F4
L2 F11
L.2 F22
L. F50
L3 F6
L3 F38
Ours
Val
Train
L3 F44
L4 F4
L4F11
L4
L4 F22
L4 F33
L4 F59
Ours
Train Val
ADHERAA THERVOCK
L5 F12
L5 F17
L5 F50
L6 F12
L6 F31
L6 F43
L6 F42
Ours
Val
Train
Figure 18: Visualization of a CLIP model with ViT-base-pate Satase-pptch66 as its visual part.
20

---

Under review as a conference paper at ICLR 2023
L6 F40
L6 F50
F7
L7 F6
L7 F5
L7 F16
L7 F37
Ours
Val
Train
L7 F41
L7 F45
L7 F55
L7 F54
L8 F3
L8 F2
L8 FO
Ours
9
Val
Train
L8 F1
L8 F5
L8 F6
L8 F19
L8 F22
L8 F24
L8 F34
Ours
Val
Train
L8 F33
L8 F32
L8 F36
L8 F39
L8 F50
1 FO
L9
Ours
Val
Train
Figure 19: (Cont.) Visualization of a CLIP model with WTT-base-patch16 as its visual part.
21

---

Under review as a conference paper at ICLR 2023
F2
F1
L9
L9 F21
L9 F24
L9 F25
L9 F31
Ours
Val
Train
L9 F33
L9 F40
L9 F47
L F44
L9 F48
L9 F52
L9 F56
Ours
Val
Train
4
L10 F20
L10 F33
L10 F51
L10 F55
L10 F53
L11 F2
L11 FO
Ours
Val
Train
L11 F4
F11
L11
F13
LI1 F25
L11 F32
L11 F41
Ours
Val
Train
CUIDADC
Figure 20: (Cont.) Visualization of a CLIP model with WVT-base-patch as its visual part.
22

---

Under review as a conference paper at ICLR 2023
L5 F32
L5 F35
L5 F25
L6 F58
L6 F45
L6 F46
L6 F42
Ours
Val
Train
L6 F40
L6 F32
L6 F34
L6 F17
L6
L7 F55
L7 F49
Ours
Val
Train
L7 F44
L7 F39
L7 F25
L7 F27
L7
L7 F10
L7 FO
Ours
Val
Train
L8 F58
L8 F52
L8 F43
L8 F27
L8 F23
L8
L8F14
Ours
Val
Train
Figure 21: Visualization of ViT-base-patch32
23

---

Under review as a conference paper at ICLR 2023
L8 F15
L8 F10
L8
L8
F2
L9 F52
L9 F48
Ours
Val
Train
L9 F50
L9 F47
L9 F44
1.9 F40
L9 F33
L9 F34
L99 F35
Ours
Val
Train
L9 F29
L9 F28
L9 F24
L9 F25
L9 F16
L10 F56
L10 F51
Ours
Val
Train
L10 F48
L10 F45
L10 F43
L10 F42
L10 F41
L10 F40
L10 F36
Ours
Val
Train
Figure 22: (Cont.) Visualization of ViT-base-patch32
24

---

Under review as a conference paper at ICLR 2023
L10 F35
L10 F34
L10 F31
L10 F26
L10 F22
L10 F23
L10 F18
Ours
Val
Train
L10 F14
L10 F9
F10
L10
L10 F6
L10 F5
Ours
Val
S
Train
Figure 23: (Cont.) Visualization of ViT-base-patch32
25

---

Under review as a conference paper at ICLR 2023
L5 F58
L5 F47
L5 F39
L55 F32
L5 F31
F9
L5
Ours
Train Val
L55
L6 F50
L6 F51
L6 F45
L6 F33
L6 F32
L6 F31
Ours
Val
Pether
Train
L7 F57
L7 F53
L7 F40
L7 F42
L7 F39
L7 F38
L8 F54
Ours
Val
Train
L8 F47
L8 F34
L8 F16
L8 F12
L8 F13
L9 F59
L99 F47
Ours
Val
Train
Figure 24: Visualization of a CLIP model with VTT-base-patch32 as its visual part.
26

---

Under review as a conference paper at ICLR 2023
L9 F41
F2
L9
L9
L10 F43
L10 F39
L10 F24
Ours
Val
Train
Figure 25: (Cont.) Visualization of a CLIP model with ViT-base-patch32 as its visual part.
27

---

Under review as a conference paper at ICLR 2023
LIl F3
F2
L10
F10
F10
F10
L10F16
Ours
Val
Train
F10
L9 FO
L9F1
L9
L9
L9
F19
Ours
Val
Train
L8 F3
L8 F6
L8 F15
L8 F16
L8 F19
L7 FO
L7 F2
Ours
Val
Train
Figure 26: Visualization of features in Deit base 1- im- 22-224
28

---

Under review as a conference paper at ICLR 2023
L7
F13
F6
L6 F16
L5
L4
LI F17
Ours
0O0
Val
Train
Figure 27: (Cont.) Visualization of features in Deit base p-16 im-224
L10 F8
L9 PO
L2
L9
L9 FI1
L9 F12
L9 F13
Ours
Val
Train
1.8
18
18 F5
18 F6
18
188 F12
1.8 F15
Ours
Val
Train
Figure 28: Visualization of features in DeiT base p- im-384
29

---

Under review as a conference paper at ICLR 2023
L8 F18
L7 FO
L7
L7 F8
L7 FIO
L7 F12
L7 F13
Our.
Vii
Train
Figure 29: (Cont.) Visualization of features in DeiT base p-16 im- im-384
LII F9
L10 LIOF11
F10
L1OF14
LIOFI8
F19
Ours
Val
Train
1.9 FO
1.9 F2
1.9
1.9 F8
1.9
F12
199
Ours
Val
Train
Figure 30: Visualization of features in DeiT base p-16 im-384
30

---

Under review as a conference paper at ICLR 2023
L9 FO
F2
L9 F4
L9
L9
L9
L9
Ours
Val
Train
L8 FO
L8F1
18 F12
18 F18
LS5 F17
L.4 F8
1.3 F18
Ours
Val
Train
Figure 31: Visualization of features in DeiT tiny distilled p-16 im-224
L10
L10
199 F10
19
1.9 F12
F2
18
NA
Ours
Val
NICE NNEELEAA
Train
SCHI
NICE SNEREE
Figure 32: Visualization of features in DeiT small distilled p- m24244
31

---

Under review as a conference paper at ICLR 2023
L8 F8
L8 P9
L8 F13
L8 F16
L8 F19
L7 F6
L7 F8
Ours
Val
Train
L.7 F9
L77 F10
L.7 F12
1.7 F13
I6 FI0
I6 F16
1.5 F3
Ours
Val
Train
Figure 33: Visualization of features in DeiT small distilled p-16 im-224
L10 F1
L10
1.9
F3
19
1.9 115
1.9 F19
Ours
Val
Train
Figure 34: Visualization features DeiT base distilled p- im-224
32

---

Under review as a conference paper at ICLR 2023
18 FO
18 F7
L8 F12
L8 F14
L8 F15
L8 F17
L8 F19
Ours
Val
Train
Figure 35: (Cont.) Visualization of features in DeiT base distilled p-16 im-224
F6
F10
F6
L6
L6 F17
FIO
L5 F12
Ours
Val
Train
Figure 36: Visualization of features in Coat lite mini
L14 FIO
F16
L14F18
L13
F12
L8
L7 F16
Ours
Val
Train
Figure 37: Visualization features in Coat lite small
33

---

Under review as a conference paper at ICLR 2023
F2
L F4
L9
F9
L9
L9
L9
Ours
Val
Train
1,9 F15
1,9 F19
L8
L8
18 F12
18 F13
18 F17
Ours
Val
Train
Figure 38: Visualization of features in Con base
L11 F13
L10 F14
1.9 FO
1.912
1.9
L9
1.9 F17
Ours
Val
Train
Figure 39: Visualization features in Cin small.
34

---

Under review as a conference paper at ICLR 2023
F1
L8 FO
L8
L8
L8
L8
L8 F14
Ours
Val
Train
18 F18
1.7 F4
1.7 F6
16 F10
L6 F17
1.5 F8
L4
Ours
Val
Train
Figure 40: (Cont.) Visualization of features in Con small.
L10
L10
L10
L10 F11
L10 L10
L1O F17
1.9
Ours
Val
Train
Figure 41: Visualization features in ViT tiny.
35

---

Under review as a conference paper at ICLR 2023
1.9 F8
1.9 P9
1.9 F12
F16
19
L8
1.7 F6
Ours
Val
Train
Figure 42: (Cont.) Visualization of features in ConViT tiny.
L9 F10
L8 F3
L8 F12
L8 F14
L8 F15
L7 F5
L7 F9
Ours
Val
Train
Figure 43: Visualization of features in Pit base im-224
F19
L8 FO
L8
L8
L8
F13
L6
Ours
Val
Train
Figure 44: Visualization of features in Pit base distilled im-224.
36

---

Under review as a conference paper at ICLR 2023
L10 FO
1.9 F14
L8 F12
1.7 F9
L6 F18
1,5 F12
L14
Ours
Val
Train
Figure 45: Visualization of features in Pit small im-224.
LIO F8
LIO F19
L8 F4
L8 F17
L7
L7 F17
L6 F7
Ours
Val
Train
Figure 46: Visualization of features in Pit small distilled im-224.
L9
L8 F12
L8
L7
F16
L5
L5
Ours
Val
Train
Figure 47: Visualization of features in Pit tiny im- 224.
37

---

Under review as a conference paper at ICLR 2023
L9 F15
L8F3
L8
L8 F18
L3
L7
L7 F15
Ours
Val
Train
Figure 48: Visualization of features in Pit tiny distilled im-2244
L21 F5
L21 F8
L20 FO
F18
L19
L18 F4
L16 FO
Ours
Val
Figure 49: Visualization features in Swin base p-4 w-7 im- 22-224.
L21 FO
L21 F8
L21
L20
L19F17
L18
F10
Ours
Val
Figure 50: Visualization of features Swing base base p-4 w-7 im-224 imagenet 22k.
1.21 F5
1.20 FO
1.20 F18
L18 F4
F18
I.77 F10
L10 F11
Ours
Val
Figure 51: Visualization of features in Swin base p-4 w-12 im-
38

---

Under review as a conference paper at ICLR 2023
1.21 F8
1.20 F5
L.20 F8
F3
L19 F7
L18
L18 F5
Ours
Val
L17 F8
F17
L17
L17 F16
L16
L16
L16F10
Ours
Val
Figure 52: Visualization of features Swin large p-4 w-7 im 224.
L22 F19
L20 F4
L20 F5
L20 F11
F10
F3
L17 F8
Ours
Val
Figure 53: Visualization of features in Swin large p-4 w-7 im-224 imagenet 22k.
L21 F8
L.0 F11
L19 F12
L18 F18
L16 FO
L16 F12
LI1 F1
Ours
Val
Figure 54: Visualization of features Swin large p-4 w-12 im-384.
39

---

Under review as a conference paper at ICLR 2023
L20 F4
L19 F9
L19 F13
L19 L19 F14
L18 FO
LI8 F4
L15 F8
Ours
Val
Figure 55: Visualization of features in Swin small p-4 w-7 im-224.
L.22 F14
L20 F2
L20 F12
LI8 F16
F13
FI5
L6
Ours
Val
Figure 56: Visualization of features in Twins pepyt base.
1.22 F16
1.20 F2
L19 F13
L16 F19
L15 F19
1.9
1.6 F2
Ours
Val
Figure 57: Visualization of features in Twins pepvt large.
L15 F2
L15 F4
L15 F15
LI5 F18
L13 F11
L10
F13
Owrs
Vii
Figure 58: Visualization of features in Twins popyt small.
40

---

Under review as a conference paper at ICLR 2023
L23 F5
L21 F4
L21 F6
L21 F13
F18
L17 F18
L16
Ours
Val
Figure 59: Visualization of features in Twins svt base.
L19 F8
L19 F13
F18
L17 F5
L16 F3
L16
L14
Ours
Vii
Figure 60: Visualization of features in Twins svt large.
LI7
L13 F5
FIOFI0
L9
L9
L8 F7
LO FIO
Ours
GROEO
Val
Figure 61: Visualization features in Twins small.
41

---

Under review as a conference paper at ICLR 2023
Table 3: ViTs effectively correlate background information with correct class, Both foreground and background data are normalized by full image top-1 accuracy.
Normalized Top-1 ImageNet Accuracy
Architecture
Full Image
Foreground
Background
VIT-B32
89.25
91.53
15.04
ViT-L16
95.00
93.88
19.08
ViT-L32
94.64
90.63
17.67
VIT-B16
92.37
93.70
16.98
ResNet-50
87.67
85.59
9.25
ResNet- 152
82.92
82.03
8.24
MobileNetv2
83.77
85,58
8.75
DenseNet121
90.58
86.53
9.72
42

---

Under review as a conference paper at ICLR 2023
F SPATIAL INFORMATION PRESENCE QUANTITATIVE EVALUATION
In the following experiments, we find the most activating images for each feature. Then, we forward these images to the network. We call patch active if the activation for this patch is higher than 0.5, First we mask out all the inactive patches meaning that we replace them with black patches. Then, we mask out x percent of the active patches the current image. Then, we forward this new image to the network. Finally, we plot the number/sum of active patches of the modified image divided by the number/sum of the active patches in the initial image for different percentages. the spatial information is present in layer, we expect this number to have a linear trend. As we see in figures 62. and 63] all the layers except for the last one, have linear trend, indicating that the loss of spatial information in individual channels is mostly made in the last layer.
100
Layer 7
Layer 8
Percent of Active Patches After Masking (%)
Layer 9
Layer
80
Layer 11
x=y
40
20
20
40
60
80
100
Percent of Unmasked Originally-Active Patches (%)
Figure 62: Number of active patches after drop divided by number of active patches before drop
43

---

Under review as a conference paper at ICLR 2023
100
Layer 7
Ratio Σ Activations (Before Masking)
Σ Activations (fter Masking)
Layer 8
Layer 9
Layer
80
Layer 11
x=y
20
20
40
60
80
100
Percent Unmasked Originally-Active Patches (%)
Figure 63: Sum of active patches after drop divided by number of active patches before drop
44


Reviews:
[{"review_id": "wvkht9O7TW", "reviewer": "Reviewer_gujJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper visually analyzes the behavior of ViT and CLIP models. The paper also shows the difference between ViT and CNNs and provides many observations. ", "summary_of_the_review": "In overall, the novelty is limited in that there are not many interesting findings compared to prior works on studying transformers.", "strength_and_weaknesses": "Strength\n---\n- To my knowledge, this is the first work that visualizes the features of transformers. \n\nWeakness\n---\n- The paper is missing references that study the CNNs and transformers (e.g., [1]). And, the related section should be more elaborated.\n\n- Some of the observations are already studied (e.g., high-frequency in [1], the receptive field in [2])\n\n- It is not clear the novel findings from previous works.\n\n- While some observations related to the background are interesting, most of the experiments are done qualitatively but no quantitative analysis is done.\n\n- The experiment in Table 1 does not look convincing to me.\n\n[1] Park, Namuk, and Songkuk Kim. \"How Do Vision Transformers Work?.\" ICLR'22\n[2] Do Vision Transformers See Like Convolutional Neural Networks?, NeurIPS'21", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "X2YVMA6JV-Q", "reviewer": "Reviewer_sVcT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper performs a mixture of things to try to visualize ViTs as opposed to CNNs. First, they slightly extended the methodology of (Simonyan et al. 2014) by adding some data augmentations, and ran the optimization to find images that maximally activate a certain filter. This is met largely with adversarial examples, and sometimes I thought got even less informative results than the original paper.  Then, they explored the effect of the classification token, which I kinda consider obsolete because of the increasing usage of global average pooling in modern ViT architectures. Finally, they performed experiments comparing ViTs and CNNs on separated foreground/background (images with foreground/background regions cropped), and showed that ViTs perform better than CNNs when presenting only the foreground, as well as only the background. This is one concrete result of the paper.", "summary_of_the_review": "The novelty is this paper is moderate since the methodology is largely similar with (Simonyan et al. 2014) with small tweaks. The results are a mixed-bag as well -- a few interesting and concrete results such as that the ViTs classify background better than the CNNs, but the visualizations are largely adversarial and sometimes confusing. I think this paper really can go either way, but would like to see some more concrete conclusions if we accept it to the conference.", "strength_and_weaknesses": "Strengths:\n\n-- Any attempt of explaining transformers and illustrating their differences with CNNs is interesting at the moment.\n-- The result that transformers perform better than CNNs when showing only the foreground, as well as only the background, is concrete and significant.\n\nWeaknesses:\n\n-- Trying to run the methodology of (Simonyan et al. 2014), although with the added data augmentation pieces, resulted in adversarial perturbations that are hard to interpret. I almost would believe there would be ways to do this better, e.g. by constraining more to the natural image manifold (such as using a VAE latent space perturbation). And the results in this part are a mixed bag at the best.\n\n-- Understanding that ViTs/CNNs have some segmentation capability is hardly something novel -- this has been shown in various prior works e.g. GradCAM and all the weakly supervised segmentation work that utilized GradCAM.\n\n-- Work on the classification token is a bit obsolete with the stronger adoption of global average pooling at the last layer.\n\n-- There has been no work that tries to check the effect of distillation and large-scale pretraining on the ViTs explored. E.g. what is the difference between distilled ViTs and non-distilled versions? What is the difference of something trained only on ImageNet-1k (e.g. a masked autoencoder) vs. JFT? Distillations and large-scale training sets are important aspects of transformer training that could bring material changes to their behavior and would be nice to explore. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NasI0Vu-zkd", "reviewer": "Reviewer_myKF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work adapts visualization techniques originally proposed for CNNs to ViT and observe the similarities and difference between CNN and ViT in terms of these visualization. For example, CNNs and ViT explores low-level features in the earlier layers and more semantic features in the higher layers. Some discoveries are interesting. For example, they observe that CLlP-based ViT image encoders can be activated by semantic concepts in the text information, e.g., such as prepositions, adjectives, and conceptual categories. ", "summary_of_the_review": "This work does not have strong novelty and lack of quantitative analysis for most of the conclusions presented in the paper. In all, I lean toward a bordline reject. ", "strength_and_weaknesses": "(+) They perform visualization on multiple ViT variants.\n\n(+) Experiments related to ViTs with text information (CLIP models) are interesting. \n\nWeakness:\n(-) The paper is not well-organized and the writing needs to be improved.\n\n(-) The visualization is corresponding to different layers displayed in different figures without justifications. Therefore, it is hard to tell if these results are cherry-picked.\n\n(-) Most conclusions drawn from this work are based on qualitative visualization without quantitative results, which makes these conclusions less convincing.\n\n(-) Some of the results are in the expectation without significant contributions to the communities. For example, the spatial information is lost in the last layer since the classification is performed based on the class-token. In addition, ViT relies more on background information has also been explored in previous works, e.g, patch-based infill transformation in Qin et al \"Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation\" (NeurIPS-2022) and Naseer et al \"Intriguing Properties of Vision Transformers\" (NeurIPS-2021). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-EhH3LW6bg", "reviewer": "Reviewer_eSYz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use Activation Maximization method to visualize different layers of vision transformers (the feed-forward and attention layers) and compare them with CNNs. More specifically, the author uses gradient descent to maximize the sum of a particular feature over all patches to create a maximally activating image, starting from noisy versions of images in the dataset. In the paper, Activation Maximization was implemented with Total Variation regularization and several augmentations similar to previous works. The authors examine the optimized activated image and original images in the dataset that maximize a feature activation. Most of the examination was done on ViT-B16 but almost all other popular ViT variants are also examined in the Appendix.\n\nThe authors find out that Activation Maximization method does not yield good interpretable visualization from the attention layers (query, key and value) while the feed-forward layers exhibit the same patterns as in saliency maps. The ViTs layers also produces progressively more complex representations as CNN and, although lacking the limited receptive field, retains much of the spatial information of the image. The authors also provides experiments that show ViTs make better use of background information than the CNN models and rely less on textural attributes. The final examination, the authors show that training ViTs with language supervision makes the model aware of various high-level concepts, such as viewpoints, number of object or semantic concepts such as morbidity and music.", "summary_of_the_review": "The paper does have some good visualizations and some interesting observation. However, ultimately, the method used is not novel and the insights provided from the analysis are not significant enough. The paper needs deeper analysis, for example a better analysis of why the attention visualization fail, is this because the maximum activation loss is not applicable? is there better ways of analyze this components? ", "strength_and_weaknesses": "Strength: The paper shows that the Maximum activation method can also work in ViTs to visualize the features in feed-forward layers. In most situation, the ViTs show very similar patterns compared to CNN. Some observations are quite interesting such as the spatial information perseverance, the better use of background information, and the attention visualization.\nWeakness:\n1. The method of visualization is not significantly novel.\n2. Although the paper focuses on visualization, the comparison between CNNs and ViTs uses mostly empirical results. Does the visualization of CNNs and ViTs also show that ViTs are more sensitive to background features, the current visualization is very similar to what expected from a CNN model.\n3. Some observations need more explanation: The attention components do not respond well to activation maximization. The author mentioned that this is because \"ViTs pack a tremendous amount of information in 768 features\", which is not quite clear to me. The author mentions that the neuron in ViTs maybe multi-modal neurons, is there some evidence/visualization to suggest this? \n4. The visualization does not show the conceptual categories activation well, as in Fig 9. It is more of a particular example case rather than some types or patterns. However, this is outside the scope of the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject