What Do Self-Supervised Vision Transformers Learn?
Authors: Namuk Park
Wonjae Kim
Byeongho Heo
Taekyung Kim
Sangdoo Yun
Venue: ICLR 2023 poster (2023)
Keywords: contrastive learning
masked image modeling
vision transformer
representation learning
self-supervised learning
empirical analysis

Abstract: We present a comparative study on how and why contrastive learning (CL) and masked image modeling (MIM) differ in their representations and in their performance of downstream tasks. In particular, we demonstrate that self-supervised Vision Transformers (ViTs) have the following properties: (1) CL trains self-attentions to capture longer-range global patterns than MIM, such as the shape of an object, especially in the later layers of the ViT architecture. This CL property helps ViTs linearly separate images in their representation spaces. However, it also makes the self-attentions collapse into homogeneity for all query tokens and heads. Such homogeneity of self-attention reduces the diversity of representations, worsening scalability and dense prediction performance. (2) CL utilizes the low-frequency signals of the representations, but MIM utilizes high-frequencies. Since low- and high-frequency information respectively represent shapes and textures, CL is more shape-oriented and MIM more texture-oriented. (3) CL plays a crucial role in the later layers, while MIM mainly focuses on the early layers. Upon these analyses, we find that CL and MIM can complement each other and observe that even the simplest harmonization can help leverage the advantages of both methods. The code is available at https://github.com/naver-ai/cl-vs-mim.


Published as a conference paper at ICLR 2023
WHAT Do SELF-SUPERVISED VISION TRANSFORMERS LEARN?
Namuk Park Wonjae Kim? Byeongho Heo?
Taekyung Kim? Sangdoo Yun?
Prescient Design, Genentech NNVER AT Lab
mrrk.namuk@gene.oo com {wonjae ae kim, bh heo, taekyung k, yune:oo.yynnnnnaaoooooo com
ABSTRACT
We present comparative study on how and why contrastive learning (CL) and masked image modeling (MIM) differ in their representations and in their per- formance of downstream tasks. particular, we demonstrate that self-supervised Vision Transformers (ViTs) have the following properties: (1) CL trains self- attentions to capture longer-range global patterns than MIM, such as the shape of object, especially in the later layers of the VIT architecture. This CL property helps ViTs linearly separate images in their representation spaces. However, it also makes the self-attentions collapse into homogeneity for all query tokens and heads. Such homogeneity of self-attention reduces the diversity of representations, worsening scalability and dense prediction performance. (2) CL utilizes the low- frequency signals of the representations, but MIM utilizes nigh-frequencies. Since low- and high-frequency information respectively represent shapes and textures, CL is more shape-oriene oriented and MIM more texture-orienter (3) CL plays crucial role in the later layers, while MIM mainly focuses on the early layers. Upon these analyses, we find that CL and MIM can complement each other and observe that even the simplest harmonization can help leverage the advantages of both methods.
INTRODUCTION
Contrastive Learning (CL) (He et al., 2020; Chen et al., 2020a;b; 2021) has been the most popular self-supervised learning methods until recently. It aims learn the invariant semantics of two random views (Tian et al., 2020a;b) by making global projections of representations similar for positive samples and dissimilar for negative samples. Since CL. exploits the globally projected representations to contrast each other, it can be deemed as an "image-level" self-supervised learning approach.
Deviating from CL, masked image modeling (MIM) (Bao et al., 2022; Xie et al., 2022b; He et al., 2022) has risen strong competitor of CL in the era of Vision Transformers (ViTs) (Dosovitskiy et al., 2021) with its impressive performances of downstream tasks. MIM trains ViTs by reconstructing the correct semantics of masked input patches. Unlike CL, it learns the semantics of patch tokens and this can be deemed as a "token-level" sulf-supervised learning approach. Since MIM outperforms CL in fine-tuning accuracy, it may appear prima facie as a more effective pre-training method than CL. However, a different trend is observed for linear probing accuracy with CL outperforming MIM (See Figure 1). For further exposition on CL and MIM, we refer the reader to Appendix B.
Then, which method-CL. or MIM-should we use for the self-supervised learning of ViTs? Although both methods are widely used, little is known about what they learn. This paper sheds light on their nature by showing that ViTs trained through CL and MIM learn opposite knowledge. In particular, we raise questions to better understand self-supervised learning, and then find the answers that can potentially affect future improvements. The questions posed can be divided into the following proper- ties of Vision Transformers: the behavior of self-attentions, the transformation of the representations, and the position of lead role components. Our key questions and findings are elaborated below.
How do self-attentions behave? (Section 2) We find that CL primarily captures global relationships. while MIM captures local relationships. This implies that the representations of CL contain more global patterns, such as object shapes, than those of MIM. On the one hand, this property helps
Most of this work done while the author was NAVER AT Lab.

---

Published as a conference paper at ICLR 2023
SimMIM
MAE
MAE
83.75
85.0
BETT
83.5
Fine-tur fee-tunig
83.50
MAE
Fine- tee-unning
80.0
Mo f
DINO
MoCo
ImageNet
MoCo
BEiT
SimMIM
83.0
83.25
ABET
MoCo
75.0
83.00
70.0
82.5
DINO
65.0
* Supervised
82.75
60
70
Ti
$
L
H
48.0
49.0
50.0
Linear Probing
Model Size
COCO
Figure 1: CL outperforms MIM in linear probing and small model regimes. In contrast, MIM excels in finettunin large model regimes, and dense prediction. Red squares (m) denote CL, and blue triangles (a) denote MIM. By default, we report the performance of ViT-B trained or pre- trained on ImageNet-1K K. We use the results from original papers and He et al. (2022) for object detection. Regarding the scaling experiment, we report the results that we reproduced based on official configurations except with 100 epochs, marking them MoCof and SimMIM', Left: CL outperforms MIM in linear probing but underperforms in fine-tuning. Middle: CL outperforms MIM in small model regimes (ViT-Ti and ViT-S), and MIM shows superior scalability in large model regimes ( (ViT-L and ViT-H), Right: MIM outperforms CL in the dense prediction downstream tasks, such object detection with Mask R-CNN (He et al., 2017) on COCO (Lin al., 2014).
CL recognize objects and distinguish images. On the other hand. however, it also suggests that CL struggles to preserve local information. In particular, we observe that self-attentions of CL in the later layers for all query tokens and heads collapse into homogeneous attention maps. such cases, most self-attention maps focus object boundaries, meaning that they can capture object shapes but may lose interaction diversity between tokens. Consequently, CL and MIM each have advantages over different tasks: CL works well for linear probing and classification tasks with smaller models, whereas MIM outperforms CL in men-ttnin and dense prediction tasks with larger models.
How are representations transformed? (Section 3) CL transforms representations mainly based on image-level information, and its self attenttnnso collect information on object shape over entire tokens. This process makes tokens similar rather than diversifying them. As result, CL. distinguishes images well but has difficulty distinguishing tokens. On the contrary. MIM preserves and amplifies token-level information. Thus, the self-attentions for each token are substantially different and prohibit each token from including redundant information. We observe the consistent property from our Fourier analysis: CL primarily utilizes the low- frequency signals, but MIM utilizes high-frequencies. This observation suggests that CL is shape-biased and MIM is texture- biaeed. In sum, self-supervised models trained with CL and MIM learn the representations in different levels of detail.
Which components play an important role? (Section 4) Analyses of the importance of each CL and MIM layer demonstrate that the later layers in CL. and early layers in MIM play a key role. We interpret this as consistent observation since early layers are usually known to capture low-level features-e.g. local patterns, high-frequency signals, and texture information-ane later layers capture global patterns, low-frequency signals, and shape information (Dosovitskiy et al., 2021; Raghu et al., 2021: 1: d' Ascoli et al., 2021; Graham et al., 2021; I; Dai et al., 2021; Park & Kim, 2022b).
From the above analyses and insights, find that CL and MIM can complement each other and show in Scction 5 that even the simplest implementation, such as a linear combination of CL and MIM objectives, can take advantage of both methods. Surprisingly, the hybrid models outperform those tranted with either CL or MIM both in terms of fine-tuning and linear probing accuracy.
2 HOW Do SELF-ATTENTION BEHAVE?
We point out that CL and MIM may not be silver bullets for tasks, as shown in Figure 1. CL generally outperforms MIM in linear probing, while MIM dominates CL in the fine-tuning scheme. However, when we dissect the size of the model, CL outperforms MIM after tnne-tuning for small models (cf. (Wang et al., 2022)), while MIM performs better large models. Also, MIM yields effective representations for dense prediction tasks, such as object detection, but CL falls short on those tasks. This section explains these phenomena by investigating the behavior of self-attention
2

---

Published as a conference paper at ICLR 2023
Query
Query
Depth =1
Depth =
Depth = 11
Depth
Depth
Depth 11
(a) MoCo
(b) SimMIM
Figure 2: Self-attentions of CL (MoCo) capture global relationships, but they collapse into homogeneous attention maps for all query tokens and heads. Self-attentions of MIM (SimMIM) mainly focus on local areas. We visualize the attention maps for two different query tokens in the beginning through the end layers. We omit the results for self-attention heads, which show mostly consistent results. Left: Self-attentions of CL capture global patterns and the shape of object. However, all attention maps capture the same shape information regardless of the query tokens. Right: Self-attentions of MIM capture local patterns and are correlated with query tokens.
Our analyses mainly compare ViT-B/16 pre-trained on ImageNet- 1K (Russakovsky et al., 2015) with MoCo (Chen al., 2021) and SimMIM (Xie al., 2022b). We use the ImageNet validation images for our experiments. We observe that other methods, e.g.. DINO (Caron et al., 2021), BEiT (Bao al., 2022), and MAE (He et al., 2022), have consistent properties (See Figure C.I).
CL mainly captures global relationships. We measure the ranges of self-attentions via attention distance (Dosovitskiy et al., 2021). Attention distance is defined as the average dis- tance between the query tokens and key tokens considering their self-attention weights. Therefore, conceptually corresponds the size of the receptive fields in CNNs.
Allention distance (px)
100
75
50-
Figure shows that the attention distance of CL (MoCo) is sig- nificantly higher than that of MIM (SimMIM), especially the later layers. As seen in Figure 2, the qualitative visualization, this implies that the representations of CL contain global pat- terns and shape information, so CL can help ViTs distinguish between objects of images. Conversely, the self-attentions of MIM mainly capture local relationships; i.e., MIM may have difficulty recognizing whole objects and their shapes. Section also discuss this claim from representational perspective.
25-
MoCa
SunMDM
Depth
Figure 3: Effective receptive fields of CL are global, but those of MIM are local. This particularly evident in the later layers.
Self-attentions of CL collapse into homogeneity. We ob- serve an interesting behavior of CL in Figure 2, which shows the attention maps for query tokens from two different spatial locations. The self-attentions of CL surprisingly indicate almost identical object shapes for the two query tokens, compared to that of MIM. We describe this phenomenon as an attention collapse into homogeneity. This collapsing trend in the self- attentions of CI. is observed across all the heads and query tokens. contrast, the self-attentions of MIM are more faithful to the two query tokens, expected.
MoClo SimMIM
Normalized MI
0.4
0.2
0.0
11
Depth
We use normalized mutual information (NMI) (Strehl & Ghosh, 2002) to measure the attention collapse. Let p(g) be distribu- tion of query tokens, and assume that these query tokens are uniformly distributed since single query token is given for each spatial coordinate, i.c., p(q) = I/N where N the number of the tokens. Then the joint distribution of query and key tokens is p(g. xt =(klg)p(g) where
Figure 4: Self-attentions of CL have little
with query tokens. Nor- malized MI of CL is significantly lower than that of MIM in the later layers.
3

---

Published as a conference paper at ICLR 2023
0.25
1.00
MoCa (1)
SinnhIMM (8)
Similarity
0.20
Similarity
0.95
Similarity
0.80
0.15
0.90
MaCCn (S)
MoCo (1)
0.10
0.85
0.60
(IB
SInMEM (3)
0.05
StankIMM (S)
SIMMMEM D3)
0.80
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Normalized depth
Normalized depth
Normalized depth
(a)
Between heads
(b) Between depths
(c) Between tokens
Figure 5: CL lacks representational diversity in the later layers. We measure cosine similarities of representations the self-attentions between the heads (left), depths (middle), and spatial coordinates (right). All of the results show that the representational similarity of later self-attentions of CL is higher than that of MIM. Increasing heads or depths of CL is not effective in improving the diversity. Left: The similarity of representations from two heads in self-attention. Middle: The similarity between representations before and after self-attentions transform them. Right: The similarities of representations at two spatial coordinates. ViT-(S, L} is trained with 100 epochs.
(klgg) is the poltmax-normalizze self-attention matrix. Thus, the normalized mutual information is I(g.k) where .(:,) the mutual information and H-) the marginal entropy. Low mutual / (K(
information values show that attention maps are less dependent on the query tokens, implying an attention collapse into homogeneity. Conversely, high mutual information means that the attention maps strongly depend on the query tokens.
Figure 4 shows the degree of attention collapse in terms of the normalized mutual information (NMI). Results show that the mutual information of CL is significantly lower than that of MIM in the later layers, suggesting that the self-attentions of CL tend to collapse into homogeneous distributions.
Attention collapse reduces representational diversity. We conjecture that the self-attention col- lapse into homogeneity eventually leads to homogeneous token representations. To support this argument, we measure representational cosine similarities. In particular, we design three similarities: between different self-attention heads (heads), between the before and after self-attention layers (depths), and between different tokens (tokens).
Figure shows the results, reporting the representation similarities for heads, depths, and tokens. As expected, the similarities of CL are notably higher than those of MIM in the later layers, indicating that the representations of CL. have significant homogeneity. Even increasing the model size does not solve the problem CL has and may rather worsen it. Increasing the number of heads (VIT-S to ViT-B; Figure 5a) improves the representational diversity of MIM, but hardly improves the diversity of CL. Increasing the depth of CL (ViT-B to ViT-L; Figure 5b) only adds redundant modules.
Implications of the behaviors we observed. In conclusion, the self-attention of CL captures global patterns and shapes of objects. However, CL suffers from the problem of attention collapse into homogeneity, which reduces the diversity of token representations. On the other hand, MIM primarily captures local patterns and thus does not suffer from the attention collapse problem.
The behaviors mentioned above can explain the phenomena we observed in Figure 1:
CL outperforms MIM linear probing tasks because captures shapes, which helps recognize ob- jects and distinguish images. Although MIM preserves the texture and diversity of representations, their correlation with objects or content may not as strong as shapes do.
The attention collapse prohibits CL from fully exploiting heads, depths, and tokens of ViTs. Since homogeneous representations are not very helpful in improving token representations, ViTs trained with CL waste large part of network capability. Therefore, the fine-tuning accuracy of MIM is significantly higher than CL. in large models.
CL is not suitable for dense prediction since the token features are homogeneous with respect to their spatial coordinates.
4

---

Published as a conference paper at ICLR 2023
Afler
Before
After
Afler
Before
Before
MoCo (one image)
(b) MoCo (two images)
(c) SimMIM (one image)
Figure 6: Self-attention layers of CL and MIM transform representations differently. We visu- alize 196 spatial representation tokens for an example validation image in a representation space. The blue (+) and red (+) data points denote the tokens before and after the self-attention transformation. Left: The self-attentions of CL (e.g., MoCo) translate all the tokens equally, so the distances between the tokens of image do not increase. Middle: However, CL moves the "centers of representations (represented by x) away from each other. Therefore, the images are linearly separable. The circle and triangle (A) data represent tokens from different images. Right: The self-attentions of MIM (e.g., SimMIM) transform representations differently according to query tokens, thus increasing the distances between tokens. See Figure 7 for quantitative analyses.
We further investigate the self-attentions behavior with restricted receptive fields in Figure D.1. As shown in the experiment, locally restricted self-attentions lead to lower linear probing but higher fine-tuning accuracy, which is consistent with observations.
3 HOW ARE REPRESENTATIONS TRANSFORMED?
In this section, analyze the token representations of ViTs pre-trained with CL and MIM to demonstrate how the properties of self-attentions we observed in Section 2 affect the representations differently. We use the same pre-trained ViT-B/16 models by default default just as we did Section 2.
CL transforms all tokens in unison, while MIM does so individually, To show how CL and MIM transform token representations, 196 (14x 14 patches) tokens before and after self-attention modules from single image sample of the ImageNet validation set. We use the three large singular vectors obtained via singular value decomposition (SVD) as the bases of the space. To better visualize this, we display the representation of MoCo and SimMIM their crucial layers-the last layer and the first layer, respectively.
visualize them in representation space. Figure shows
Figure 6a visualizes the changes that occur in the tokens of CL when transformed by self-attention module; it indicates that the self-attentions of CL translate all tokens in unison. This phenomenon occurs because the self-attenie attention maps of CL are homogeneous, i.c., self-attention is almost indepen- dent of the spatial coordinates and query tokens. Therefore, the modules add near-constant all the token representations. As a result, the represtrpprstnnaaooonn distance and the volume of representations do not increase, which implies that CL cares less about individual tokens.
Nevertheless, self self-attentions are essential for the discriminative power of CL. As shown in Figure 6b, they help distinguish images by moving "the centers of the representation distribution" away from each other. In short, this figure suggests that CL makes the image linearly separable even though it loses the ability to distinguish tokens.
In contrast, MIM applies different transformation individual tokens, shown in Figure 6c, because different celf-attentions are assigned to the individual spatial tokens. Thus, MIM alters the distance between tokens of single image well the volume of the representation distribution.
We find consistent results in quantitative analysis. Inspired by Jing et al. (2022). Figure 7 visualizes singular value spectra for tokens and images. A singular value spectrum provides singular values of representation distribution obtained by SVD, so we can use it to represent the effective volume of distributions in representation space. The higher the singular value in spectrum, the larger the
5

---

Published as a conference paper at ICLR 2023
A Log singular value
MoCo
SimMIM
A Log singular value
-30
Modi
Depth
-5-
-5-
-4.0
195
63
100
195
Rank index
Rank index
Depth
(a) Singluar value spectum tokens from single image (token-level)
A Log singular value
A Log singular value
MoCa
SimMIM
Made SanMIM
10
-5-
Depa
-6.0
-8.0
-10-
-10.0
127 200
391
127 200
391
11
Rank index
Rank index
Depth
(b) Singluar value spectum of images (image-level)
Figure 7: CL barely changes or even decreases the distribution volume of tokens from a single image, implying that it hardly distinguishes between token. Instead, it significantly increases the distribution volume of images. To demonstrate these properties, we visualize singular value spectra, the singular values of the distribution of representations sorted by the magnitude. The higher singular value, the larger the volume of a distribution. The right of this figure shows the 64 and 128th highest singular value for depth. Top: Singular value spectra of tokens from single image. CL decreases the singular values of the tokens, but MIM increases. Bottom: Singular value spectra of images. CL significantly increases the volumes occupied by images, but MIM hardly does so.
volume of a representation distribution. To calibrate the scale, (A Log singular value), the difference with the (second) largest singular value for a depth.
the relative log singular value
Figure 7a shows singular value spectra of tokens from a single image. We calculate them for each image in the ImageNet validation set and report averaged singular values over the dataset. this figure, the CL layers hardly increase even decrease the singular value; consistent with the explanation above, this implies that CL hardly distinguishes tokens. In contrast, MIM increases the singular value, meaning that it changes the volume of tokens and can distinguish tokens. Another interesting observation is that a few later layers of MIM decrease the volume, even though they capture local patterns as shown in Figures and This is because they behave like decoders. Section 4 discusses this in detail.
Figure 7b shows the singular value spectra of images. We average all tokens in an image build an image-level representation vector and conduct singular value spectrum over the collection of representations in the validation set. As opposed to the previous case, the representational volume of CL is larger than that of MIM, which implies that CL makes the image-level representation separable.
CL exploits low-frequencies, and MIM exploits high-frequencies. We hypothesize that CL cap- tures low-frequency and MIM captures high-frequent information in spatial dimensions since CL provides image-level self-supervision to capture global patterns, while MIM provides token-level self-supervision to exploit local patterns. To support this argument from frequency perspective, we conduct a Fourier analysis of the representations as following Park Kim (2022b). In particular, we report the relative log amplitude of Fourier- transformeddff representations by calculating the amplitude difference between the highest and lowest frequencies of representations.
Figure 9 visualizes the relative amplitudes of CL and MIM. shows that the high-frequency amplitude of CL is significantly smaller than that of MIM, suggesting that CL mainly utilizes freeueuuyccy spatial information such as global structures and shapes. On the contrary, MIM usually uses high- frequency spatial information such as narrow structures and fine textures.

---

Published as a conference paper at ICLR 2023
Praction of "shape" decisions
0.9 0.8 0.7 0.5 0.3 0.2
Shape categories
Mata
A Accuracy (%)
-10.0
SusMIM Strali
Supervise Superrion
-15.0
-00.0
-25.0
MaCo
SMMMM
0.1 0.2 0.3 0.4
0.5 0.6 0.7 0.8
0.2
0.5x
0.8m
Fraction of "texture" decision
Noise frequency
(a) Stylized ImageNet
(h) Rohustness for noise frequency
Figure 8: CL is biased toward shape, whereas MIM biased toward texture. We report the predictive results of models for linear probing tasks. However, we observe consistent results in fine- tuned models (See Figure F.2). Left: Result of classification on Stylized ImageNet. It shows that CL more biape-biased than MIM and even than the supervised pre-trained model. Vertical lines represent averaged results for the shape categories. We also report the results of supervised ViT with ImageNet- 1K class labels for comparison. Right: Accuracy drops images with frequency- based random noises. MIM shows a more significant amount accuracy drop than CL with high-frequency noises, demonstrating MIM's texture- biased property. The frequency window size of the frequency-baan based noise is 0.1
Another interesting finding is that the last few layers of MIM reduce the high frequencies focus on local areas (See Figure 3). We conjecture that MIM implicitly divides ViTs into the encoder-decoder structure and allows intermediate layers to have linearly separable information. In contrast, CL allows the last layer to have such information. This is further elaborated in Figure 11.
-1.0
though they only
MoCo
A Log amplitude
SimMMMM
-2.0
-3.0
CL is shape-biased, but MIM is texture-biased. Based on the results of the Fourier analysis, we assume that CL and MIM each have bias toward shapes and textures, respec- tively. To demonstrate this claim, we use Stylized ImageNet (Geirhos et al., 2019), a texture-altered dataset, by using AdaIN (Huang & Belongie, 2017). Figure 8a reports the linear probing results on Stylized ImageNet to evaluate the shape and texture biases of pre-trat pre-trained models. Compared to the model pre- trained with supervised learning, CL depends more the shape and MIM depends texture of images to classify images. In other words, CL is robust to texture changes, and MIM is vulnerable to them.
0
5
10
Depth
Figure 9: CL exploits low-frequency, but MIM exploits high-frequency. Moreover, few last layers of CL reduce frequereeuccy by capturing global patterns. MIM also reduces even though they capture local pat- terns, because the later layers behave like decoders. See also Figure 11.
Figure shows the consistent results. In this experiment, we follow Park & Kim (2022a;b) and measure the decrease in accuracy the ImageNet dataset with frequency-based random noise. The results suggest that CL robust to high-frequency noises, but MIM is significantly more vulnerable to them. Since -igh-frequency noises harm the fine details of images, arrive at the same conclusion that CL is more shape-biased and MIM is texture-biased. This can explain the robustness of CL. against adversarial perturbations (Bordes et al., 2022).
4 WHICH COMPONENTS PLAY AN IMPORTANT ROLE?
The previous sections consistently show through various perspectives that CL exploits image-level global patterns while MIM captures token-level local patterns. This section analyzes pre-trained ViTs from an architectural perspective and shows that the key components in CL. and MIM are different.
7

---

Published as a conference paper at ICLR 2023
Normalized Mt
0.6
-B- Sumiliii
McCa
M
0.6
Log amplitude
-h- Sank234
0.4
0.4
-2.0%
-20
0.2
-3.0%
-3.0
B.
11
13
16
19
10
12
15
18
Depth
Depth
Depth
Depth
(a) Self-attention
(b) Fourier analysis
Figure 10: The explicit decoder architecture of MAE helps ViTs effectively leverage the advan- tages of MIM. We analyze the encoder and decoder of a pre-trained model with masking ratio of zero. The left side of each figure represents the encoder and the right side the decoder. Left: The mutual information of MAE is lower than that of SimMIM in the encoder but higher in the decoder. Right: The decoder of MAE captures low-frequency information, and its encoder captures high-frequency information. Moreover, the later layers (excluding the last layer) of MAE do not reduce freeuerey information, while those of SimMIM do.
Later layers of CL and early layers of MIM are important. According to studies on ViT (Gra- ham et al., 2021; Dai et al., 2021; Park & Kim, 2022b), the later layers use high-level information, and the early layers exploit low-level information. Since CL and MIM each exploit global and local patterns, we expect that the later layers of CL and early layers of MIM play a key role.
To evaluate the importance of each layer, we measure the linear probing accuracy using intermediate representations with the configuration of Table 1. In Figure 11, we observe the following properties: First, the linear probing accuracy of MIM higher than that of CL at the beginning. Con- versely, CL outperforms MIM the end of the model. Such result indicates that the later layers CL and early layers of MIM play an important role in making linearly separa- ble representations. Second, the accuracy of CL increases with increasing depth as expected, but the accuracy of MIM surprisingly decreases at the end of the model, i.e., the later layers of MIM are not very helpful in separating representa- phenomenon in which
Linear Probing
60
40
20
MoCo
SimMIM
MAG
6 Depth
11
tions. We explain this observation MIM methods with shallow prediction heads, e.g., SimMIM, use later layers of the backbone as decoder. Therefore, MIM with deep self-attention decoder, e.g., MAE (He et al., 2022), can be useful for linear probing performance. Moreover, it also explains why SimMIM's 's hügh-freeuency component and representational volumes drop in the later layers as shown in Figures and 9. Third, even the highest linear probing accuracy of MIM is lower than that of CL.
Figure 11: Later layers of CL and early layers of MIM play key role. We report linear probing accuracies by using the representations of the inter- mediate layers. CL outperforms MIM in later layers, and MIM outperforms CL in early layers.
The explicit decoder helps ViTs further leverage the advantages of MIM. Several previous observations find that the implicit decoder of MIM with a shallow prediction head, such as SimMIM, can impair performance. MAE (He al., 2022) addresses this problem by introducing deep explicit ViT decoders and reconstructing masked tokens only in the separate decoders.
In Figure 10, we analyze MAE to understand the properties of decoders more deeply. Figure 10a shows the self-attention behaviors. The results indicate that the mutual information of MAE is lower than that of SimMIM in the later layers of the encoder but higher in the decoder, implying that the decoder reconstructs masked tokens based on its neighborhood tokens.
Figure 10b shows the results of the Fourier analysis. As explained in Figure 9, the last four layers of SimMIM reduce the fighfrreueecy components. In contrast, the later layers (excluding the last layer) of MAE do not reduce them. Instead, the decoder of MAE prioritizes low- www-freuency information compared with the encoder, allowing the backbone to efficiently utilize high-frequency information.
8

---

Published as a conference paper at ICLR 2023
82.25
A-02
McCo
-1.0
Normalized MI
SimMDM
A Log amplitude
Fine-tume te--tuiing
82.00
2-0.4
0.4
SunMDM MMoc
/400.0
-2.0
81.75
B-B.
7-0.6
0.2
81.50
-3.0
b
SunMIM
A-1.0
SunMDM
0.0
20
40
60
11
5
10
Linear Probe
Depth
Depth
(a) Performance
(b) Self-attention
(c) Fourier analysis
The simple linear combination of CL (MoCo) and MIM (SimMIM) objectives out- performs the vanilla CL and MIM. À is the importance weight of CL, so = means SimMIM and X = means MoCo. Left: "CL+ MIM" outperforms CL and MIM in both linear probing and fine-tuning accuracy. Middle: Mutual information of "CL MIM" decreases at the end of the model, suggesting that the self-attentions of later layers collapse into homogeneity and capture the same object shape information. Right: Fourier analysis shows that "CL MIM" amplifies high frequencies the beginning and reduces them at the end. It implies that "CL MIM" exploits high-frequency information at the beginning and low- low-frequenn frequerrgyccy information at the end.
Figure 12:
5 ARE THE TWO METHODS COMPLEMENTARY TO EACH OTHER?
We present comparative analyses CL and MIM from three perspectives: self-attentions, repre- sentation transforms, and the position of important layers. All of our results indicate that CL and MIM train VITs differently. These differences naturally imply that combining CL and MIM to train backbone may help leverage the advantages of both methods.
To show that CL and MIM are complementary, we introduce the simplest way to harmonize CL and MIM by linearly combining two losses, i.e., L = (1 A) + XLeL where LMIM and LCL each indicate the losses of MIM and CL, and À is the importance weight of CL. We find that this simple hybrid model trained with combined losses efficiently exploits the strengths of both methods. Figure 12a shows linear probing and fine-tuning accuracy ImageNet with varying A. Surprisingly, the hybrid models outperform MIM (A 0) and CL (A = 1) in both aspects. Figure 12b and Figure 12c can provide insights on how hybrid models behave by analyzing the model with À 0.2 in terms of self-attentions in Section 2 and Fourier analysis in Section 3, respectively; both results show that the hybrid model exploits MIM properties in the early layers and CL properties in the later layers. In particular, Figure 12b indicates that the self-attentions of the early layers are changed according to the query token but those of the later layers are not. Likewise, Figure 12c shows that the early layers exploit high-frequency, while the later layers try to exploit low-frequency.
CONCLUSION
We conducted comparative study highlighting various facets of two widely used self supervuue learning methods for vision transformers: contrastive learning (CL) and masked image modeling (MIM). The study demonstrated many opposing properties of the two methods: image informa- tion (image-level v.s. token-level; as in Section 2), feature representations (lowffrequency vs. high- frequency; in Section 3), and lead role components (later layers vs. early layers; in Section 4). Furthermore, we suggested a possible application that exploits only the benefits from both methods and showed how combined model outperform individual methods.
Future directions. Various future directions can be explored based on our study. We believe that there are better ways than simple linear combination of CL and MIM objectives. For example, novel self-supervised learning approach, in which CL is applied in the later layers and MIM in the early layers, can be considered. Moreover, we may extend our findings on self-supervision for multi-stage ViTs, such PiT (Heo al., 2021) and Swin (Liu al., 2021). Another interesting direction is to enhance the individual properties of CL and MIM. Techniques that help CL or MIM learn shapes textures, respectively, may also improve performance.

---

Published as a conference paper at ICLR 2023
REFERENCES
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations, 2022.
Florian Bordes, Randall Balestriero, and Pascal Vincent. High fidelity visualization of what your self-supervised representation knows about. Transactions on Machine Learning Research, 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In International Conference on Computer Vision, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International Conference on Muchine Learning, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 00tp:0003.0499,, 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In International Conference on Computer Vision, 2021.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated
data augmentation with a reduced search space. In Advances Neural Information Processing Systems, 2020.
Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes, Advances in Neural Information Processing Systems, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representutions, 2021.
Stéphane d' Ascoli, Hugo Touvron, Matthew L Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In International Conference Machine Learning, 2021.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis. Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.
Benjamin Graham, Alaaeldin El-Nouby. Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. In International Conference Computer Vision, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 2020.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, 2017.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In International Conference on Computer Vision, 2021.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision, 2016.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEEE international conference on computer vision, 2017.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations, 2022.
Hanjoo Kim, Minkyu Kim, Dongjoo Seo, Jinwoong Kim, Heungseok Park, Soeun Park, Hyunwoo Jo, KyungHyun Kim, Youngil Yang, Youngkwan Kim, al. Nsml: Meet the mlaas platform with real-world case study. arXiv preprint arXiv:1810.0990 2018.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In International Conference on Computer Vision, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 2018.
Namuk Park and Songkuk Kim. Blurs behave like ensembles: Spatial smoothings to improve accuracy, uncertainty, and robustness. In International Conference on Machine Learning, 2022a.
Namuk Park and Songkuk Kim. How do vision transformers work? In International Conference on Learning Representations, 2022b.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, perf--eerormance deep learning library. In Advances Neural Information Processing Systems, 2019.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015.
Alexander Strehl and Joydeep Ghosh. Cluster ensembles a knowledge reuse framework for combin- ing multiple partitions. Journal of machine learning research, 2002.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European Conference Computer Vision, 2020a.
11

---

Published as a conference paper at ICLR 2023
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 33:6827-68333 2020b.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2008.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard. Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778, 2018.
Shaoru Wang. Jin Gao, Zeming Li, Jian Sun, and Weiming Hu. closer look self-supervised lightweight vision transformers. arXiv preprint 2rXiv:2205.14443, 2022.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for seif-suuer visual pre-training. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2022a.
Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation. arXiv preprint arXiv:2205.1 /4141, 2022b.
Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. arXiv preprint arXiv:2205.1354. 2022a.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022b.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint 2rX19:2005.01917, 2022.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In International Conference on Computer Vision, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 12733)302-331,, 2019.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. International Conference Learning Representations, 2022.
12

---

Published as a conference paper at ICLR 2023
A SETUP
We build the configurations based Xie al. (2022b) for fine-tuning and Caron et al. (2021) for linear probing. Table A.1 summarizes the configurations. Most analyzes of VITs use the official ViT-B pre-trained models, and some analyzes ViT-(S, L) pre-trained with official configurations but epochs of 100. Due to memory limitations, ViT-L is pre-trained with a quarter batch size of the other models. The hybrid model introduced in Section 5 uses the ViT backbone architecture of Xie et al. (2022b) and employes a configureation based on their work for pre- pre-training as shown in Table A. For data augmentation and regularization, we adopt widely used settings, e.g., Randaugment (Cubuk et al., 2020), label smoothing (Szegedy et al., 2016), mixup (Zhang et al., 2018), cutmix (Yun et al., 2019), stochastic depth (Huang et al., 2016). Layer decay (Bao al., 2022) is also used for fine-tuning. Neural network models are implemented in PyTorch (Paszke al., 2019). The code for analysis is available https: //github. cwm/ coom/naver-aa ai cl-vs-mimm All experiments use (1, 4, 8) NVIDIA A100 Tensor Core GPU. NSML (Kim et al., 2018) has been used for experiments.
thub.
Table A.1: Training settings.
CONFIGURATION
Linear Probing
Fine-tuning adamw
Pre-training
optimizer
sgd
adamw
base learning rate
1.0e-0
1.25e-3
1.0e-4
weight decay
0.05
0.05
0.05
batch size
2k
training epoch
50
100
100
learning rate schedule
cosine
cosine
multistep
warmup epoch
20
10
warmup schedule
linear
linear
randaugment
0.5
9, 0.5
label smoothing
0.1
0.1
mixup
0.8
0.8
cutmix
1.0
1.0
stochastic depth
0.1
0.1
layer decay
0.65
1.0
gradient clip
5.0
5.0
B RELATED WORK
CL a method based on comparing the global projection of two different random views. However, this approach usually suffers from the collapsing problem, where all representations collapse into constant solutions. To solve this problem, various methods such as negative samples and InfoNCE (Oord al., 2018) have been proposed. Negative samples is an effective technique avoid the collapsing problems, but they cause dimensional collapse (Jing et al., 2022) and require extra large batches (Chen al., 2020a) or memory queues (He et al., 2020; Chen al., 2020b) retrieve them. We mainly analyze MoCo v3 (Chen et al., 2021), 1), since the method includes these de facto standard components-giee projection, random views, and negative samples.
Some SSL methods, e.g. Grill et al. (2020); Caron et al. (2021), do not use negative samples and use the projections of their Siamese representations as the positives. Such self-distilllaion has been explored theoretically and empirically (Chen & He, 2021; Tian et al., 2021) to prevent the collapsing problem, but we do not discuss the distillation scheme in this paper. Wei et al. (2022b) shows that feature distillation improves the fine-tuning performance of CL by diversifying attention ranges; this observation is consistent with our findings. While they focus on distillation to improve CL, we reveal the fundamental nature of self-supervised learning by rigorously comparing CL. and MIM.
13

---

Published as a conference paper at ICLR 2023
Attention distance (px)
StmMEM
100
BEIT
DINO
MoCo
Normalized MI
MAE
0.4
BEIT
75-
MAE
50-
A-A-
SinMEM
25-
SInrCRR
0.0
11
Depth
Depth
(a) Attention distance
(b) Normalized MI
Figure C.I: MIM and CL methods each have consistent properties. To show this, we visualize self- attention behaviors in terms of attention distance and normalized mutual information (MI). SimCLR*, which was introduced in Chen et al. (2021), stands for MoCo with a momentum coefficient of 0. Left: The attention distance of CL methods (namely MoCo, SimCLR", and DINO) is higher than that of MIM methods (namely SimMIM, BEiT, and MAE). This suggests that CL methods consistently capture global patterns. Right: The normalized mutual information of MIM is higher than that of CL; i.e. the self-attentions of MIM are more correlated with query tokens than CL.
Attention distance (px)
MoCo (T)
100
Normalized MI
0.6
MoCo
MoCo (3)
75
SimMIM (T))
MoCo (T0
SanMEM
SanNIM (1)
MuCa (8)
SimMIM (T)
0.2
25
SinMMN
SInaMIN
0.0
Depth
Depth
(a) Attention distance
(b) Normalized MI
Figure C.2: ViTs exhibit consistent self-attention patterns, regardless their size. To better understand these patterns, visualize the self-attention behaviors of three Bj-using two metrics: attention distance and normalized mutual information (MI). Left: All self- attentions of MoCo capture global patterns in the later layers. In contrast, the self-attentions of SimMIM capture local patterns. Right: Likewise, all self-attention maps of MoCo collapse into homogeneity in the later layers.
S.
Compared with CL, MIM has been rarely explored in vision tasks. Various methods, such as his- tograms oriented gradients (Wei et al., 2022a) and tokenization (Bao al., 2022), have been proposed part of porting masked language models to the image domain with ViTs. Among them, SimMIM (Xie al., 2022b) and MAE (He et al., 2022) are simple yet effective methods reconstruct masked tokens without complicated pretext tasks. Because of its simplicity and superior performance downstream operations, MIM attracting attention as a promising technique in image processing.
Nevertheless, we find hints suggesting that CL. and MIM utilize different aspects of the data, making them complementary. For example, Zhou et al. (2022); Wang al. (2021); 1); Yu al. (2022) achieve high predictive performance by harmonizing the image-level and the token-level self-supervised learning. Xie et al. (2022a) also observe that, unlike supervised pre-trained models or CL, self- attentions in SimMIM focus locally; this is consistent result with our findings.
C OUR INSIGHTS ARE GENERALIZABLE VARIOUS MODELS
In the main text, analyze ViT-B pre-trained using MoCo and SimMIM. We observe consistent characteristics across various sizes of ViTs that have been pre- pre-trained using other self-supervised learning methods. To support this claim, we delve into the properties of self-attentions.
14

---

Published as a conference paper at ICLR 2023
Std of Normalized MI
MoCo
Mc.Co
Density
0.2
0.1
0.0
6
11
0.25 Normalized MI
1.00
Depth
(a) Standard deviations
(b) Distribution at 3 layer
Figure E.1: The presence an outlier head in MoCo raises the average of normalized mutual information. This observation explains how the normalized mutual information in a couple of MoCo's self-attention layers is similar to or even surpasses that in SimMIM. Left: We present the standard deviation of the normalized mutual information. As depicted in this figure, the standard deviation in SimMIM remains relatively consistent across different depths. In contrast, the standard deviation in MoCo's s 3rd or 4th self-attention layer notably higher than that in SimMIM. Right: Distribution of mutual information for the third self-attention layer head. The visualization of this kernel density estimation shows that MoCo has an outlier head with mutual information close to 1.0. The red rectangles (D) and blue triangles (A) refer to the mutual information of heads in MoCo and SimMIM, respectively.
Figure C.1 visualizes the self self-attention behaviors of different self-supervised learning methods in terms of attention distance and normalized mutual information. As depicted in the figure, all CLs and MIMs exhibit consistent properties. Similarly, Figure C.2 demonstrates that various sizes of models also demonstrate consistent properties.
D LOCALITY INDUCTIVE BIAS IMPROVES FINE-TUNING ACCURACY OF CL
In Section 2, we demonstrate that the homogeneity of self- attention map, i.e., attention collapse of CL. helps ViT dis- tinguish images but harms fine-tuning accuracy. As result,
81.5
kernel 15
Fine-tuning
we anticipate that incorporating locality inductive bias into CL will improve fine-tuning accuracy but degrade linear probing accuracy. One simple method to inject locality into self-attentions to limit the receptive field self-attention by using attention masks.
81.4
kernel = 21
81.3
keynel
kernel 27 (global)
Figure D.I shows the predictive performance of MoCo with restricted local seff-attentions. As expected, the results similar to the performance MIM; As the kernel size de- creases, the linear probing accuracy decreases but the fine- tuning accuracy increases. These results are consistent with our findings.
81.2
60
65
Lincar probing
Figure D.1: Locality inductive bias harms linear probing but improves fine-tuning. We report the linear prob- ing and fine-tuning accuracy of MoCo with restricted self-attentions via atten- tion masks.
E A CLOSER LOOK AT THE ROLE OF SELF-SUPERVISED VIT LAYERS
The main text provides the key characteristies of CL and MIM. This section delves deeper into the details not covered in the main text to provide a more comprehensive understanding of the subjects.
The role of the early modules. Figures 3 and 4 suggest that most layers of MoCo capture global patterns and have only weak correlation with query tokens. However, exhibit unusual behavior. For example, the 3rd layer MoCo focuses on local areas and self- attention map is dependent the query. We explore this property in more detail.
two of MoCo layers
15

---

Published as a conference paper at ICLR 2023
(a) MoCo
(b) SimMIM
Figure E.2: The tokens of MoCo form a cluster for each image, while those of SimMIM are intermingled. This aligns with the finding that, compared SimMIM, MoCo linearly separable. To demonstrate this property, we visualize 3,528 tokens (196 tokens 18 images) from the representations of the last layer via t-SNE, and find that a consistent pattern observed even in the representations of the intermediate layers. The colors represent three different classes. See also Figures 6 and 7.
Figure l. provides the variance of normalized mutual information with respect of heads. As the results show, the variance of SimMIM is consistent across all depths whereas that of MoCo is not. In particular, the 3rd layer of MoCo has high variance even though other layers do not. This suggests that, while most of MoCo's o' self self-attention heads capture global patterns and have weak correlation with query tokens, some heads deviate from this behavior and exhibit a different pattern.
Figure E. lb shows the distribution of normalized mutual information among heads in the 3rd layer to analyse this phenomenon. this figure, we use kernel density estimation with Gaussian kernel to visualize the distribution. The results reveal several outlier heads in MoCo with mutual information close to 1.0. As result, these outliers significantly raises the average value normalized mutual information.
comprehensive view through visualization of tokens from multiple images. Figure visual- izes how self-attention layers transform tokens from one or images representation space. The figure demonstrates that MoCo transforms all tokens in union while SimMIM transforms them indi- vidually. As result, MoCo separates the representations the image-level and SimMIM separates them at the token-level.
The t-SNE visualization (Van der Maaten & Hinton, 2008) in Figure E.2 provides consistent results and offers even more comprehensive perspective. In this figure, we visualize the last representations of 3528 tokens from 18 images that belong to three different classes. The visualization demonstrates that MoCo separates the representations into distinct classes and even images, while maintaining the tokens close together compact image clusters. On the other hand, SimMIM separates tokens from images, resulting in wide representation space for each image, but the images even classes may be challenging to linearly distinguish.
The first layer of MoCo aggregates tokens into compact clusters. Figures and shows that all modules, except the first module, in MoCo behave consistently. However, we observe that MoCo's first module behaves differently and unusually than the others. We elaborate the behaviour of the first layer of module.
Figure E.3a shows the qualitative visualization of tokens for sample image, similar Figure 6. This visualization shows that the first MoCo layer aggregates tokens into compact clusters. Although this figure only uses single image, the layer aggregates all images into a small representation space as well. In terms of singular values, we observe consistent results. Similar to Figure 7, Figures E.3b and E.3c report the second largest log singular value, instead of the relative log singular value, to investigate the absolute volume of the representations. As expected, most layers in both MoCo and SimMIM increase the singular value, but surprisingly, the first layer of MoCo reduces the singular value, meaning that the volumes of representations decreased at both the levenllevel and image- level. Based on these observations, we conjecture that the first module MoCo behaves like an embedding component.
16

---

Published as a conference paper at ICLR 2023
Before
After
Log singular value
5.0
Log singular valuse valuue
4.0%
2.0
3.0
2.0
MoCo SiuMDM
1.0
MaCo
SinMIM
11
1
11
Depth
Depth
(a) Token visualization
(b) Token-level analysis
(c) Image-level analysis
Figure E.3: The first layer MoCo clumps tokens together. We demonstrate this property from two perspectives: qualitative visualization and singular value of token distribution. Left: Similar to Figure 6, we visualize tokens sample image a representation space. The blue and red data points represent the tokens before and after the self-attention transformation. As shown in this figure, the first self-attention layer clumps tokens into a compact cluster. Middle and Right: Similar to Figure visualize the second largest log singular value (not A log singular value) for depth. The singular value spectra demonstrate consistent results; the first layer of MoCo (gray area) not only clumps tokens but also images into a compact cluster.
-1.0
MoCa
Normalized MI
SinMMMM
A Log amplitude
0.4-
McCo (t)) SinMMMM
-2.0
0.2
SumMIM (fr
SauNIM
-3.0
MoCo (f)
0.0
6
11
5
10
Depth
Depth
Normalized MI
(b) Fourier analysis
Figure F.1: Self-attentions and representations in fine-tuned models exhibit consistency with those of pre-trained models. Similar to Figures 4 and 9, we present the normalized mutual informa- tion and the Fourier analysis results of fine-tuned models. The abbreviation "ft" stands for "fine-tume tuned model." Left: Similar to pre-trale pre-trained models, the mutual information MoCo's self-attention maps is generally lower compared to that of SimMIM. However, it is noteworthy that the mutual information of the later self-attention maps in SimMIM decreases significantly. This because the later layers of model trained with supervision fine-tuning tend to capture global information. Right: Similarly, SimMIM utilizes higher frequency information than MoCo.
F FINE-TUNER MODELS INHERIT THE PROPERTIES OF PRE-TRAINES MODELS
The main text focuses highlighting the key properties of pre-trained models. This section demon- strates that these properties are also utilized by fine-tuned models. As result, we can safely apply the insights gained from the main text various situations.
Consistent results self-attention and Fourier analysis. Figures 3 and demonstrate that MoCo captures global areas and that its self-atte attentttnnions less related to the query tokens, compared with SimMIM. In addition, Figure shows that MoCo captures now-frequency information as opposite to SimMIM. These results consistent in the fine-tuning scheme.
Figure F. reveals the self-attention behaviours of fine-tuned MoCo and SimMIM in terms of normalized mutual information. Similar to pre-trained models, the fine-tuned self-attention maps of
17

---

Published as a conference paper at ICLR 2023
Fraction of "shapc" decisions
0.9 0.7 0.5 0.4 0.2 0
Bid 1E37 271
MoCo
Shape categories
Malo
Accuracy (%)
-2.5
SImMIM
SmnMIM Sepervized Ripersised
-3.0
-3.5
-4.0
0.1 0.3 0.4
0.5 0.6 0.8
-4.5
0.2rt
0.5xt
0.8
Fraction "texture" decision
Noise frequency
(a) Stylized ImageNet
(b) Robustness for noise frequency
Figure F.2: Fine-tuned ViTs inherit the robustness against frequency-basee noise. Similar to Figure 8b, we measure the decrease in the accuracy of ViTs fine-tuned with MoCo and SimMIM. Left: Even with fine-tuned ViTs, MoCo relatively shape-biased and SimMIM relatively texture- biased. This bias is just less apparent than in linear probing models. Right: The robustness against frequency- random noise also suggests the same: MoCo is robust against sighffreueeccy noise, but SIMMIM is not. In conclusion, fine-tuned models inherit the properties of linear probing models.
MoCo have generally lower mutual information compared to those SimMIM. The only significant difference is that the mutual information of the later self-attention maps in fine-tuned SimMIM decreases significantly, as later layers in models trained with supervision or fine-tuning tend to capture more global information. As result, the gap between the two methods is reduced. This is also reflected in the consistent results Fourier analysis as shown Figure F.lb. this analysis, SimMIM captures freeurrfryyuency information compared to MoCo in fine-tuning scheme well. However, the later layers of SimMIM attempt to capture low-frequency information. Therefore, the gap of fine-tuned models is smaller than that of pre-trained models.
CL is shape-biased and MIM is texture-biased in fine-tuning scheme. In Figure 8, we demon- strate that linear probing model with CL (MoCo) is more shape-biased and that with MIM (SimMIM) is texture- biased, compared with each other. As in the experiment, we calculate the classification results of ImageNet fine-tuned MoCo and SimMIM on Syllized-mmaeeNee and measure the decrease in accuracy against frequency-based random noise. As we would expected, Figure F.2 shows that the property also extends to the fine-tuner model. Even though we still observe the difference between MoCo and SimMIM, the performance gap between MoCo and SimMIM is quite reduced compared to the gap between the linear probing models.
Later layers of CL and early layers of MIM are important find-tuning phases. As shown in Fig- ure 11, the later layers of the CI. and the early layers of the MIM are linearly separable. This finding suggests that these layers are significant, however, it does not provide direct evidence that such properties are pre- served during fine-tuning phases. We demonstrate that these layers play crucial role in fine-tuning phases as well.
A Accuracy
A Accuracy
-2
MaCo SimMIM
12
Init depth
Init depth
To support this claim, we conduct study to mea- sure the accuracy drop of fine-tuned models using pre- trained models with few blocks initialized. As shown in Figure F.3a, the results indicate that the initializing few early blocks in the pre-training models of SimMIM significantly harms the fine-tuning accuracy, compared to MoCo. These observations suggest that early lay- ers of SimMIM play an important role in fine-tuning. Conversely, Figure F.3b shows that initializing later
(a) Early layer
(b) Later layer
Figure F3: The later layers of CL and early layers of MIM play a key role in the fine-tuning scheme. To show this, initialize a few blocks and measure the de- crease in the fine-tuning accuracy of pre- trained models.
18

---

Published as a conference paper at ICLR 2023
blocks in the pre-training models of SimMIM does not significantly harms the fine-tuning accuracy, suggesting that they are not important in fine-tuning compared with MoCo.
One limitation of this experiment is the evaluation of the accuracy drop in single Since the accuracy drop of MoCo is marginally higher than that of SimMIM at the first initialization depth in Figure F.3b, additional experiments may improve the results. In this experiment, utilized the same fine-tuning settings for both MoCo and SimMIM; but experiments with finettningg settings tailored each method may provide further insight.
G HYBRID MODELS OUTPERFORM CL AND MIM DOWNSTREAM TASKS
The claim that CL and MIM complemen- tary is demonstrated only on ImageNet in Sec- tion 5. To validate this claim tasks beyond ImageNet, we evaluated the pre-trained models of the hybrid method introduced Section 5 for another classification task and semantic segmentation task. In particluar, we measured the accuracy on iNaturalist 2018 (Van Horn al., 2018) and the mloU on ADE20K (Zhou al., 2019). As shown in Table G.I, the hybrid model of SimMIM and MoCo outperforms both SimMIM and MoCo various downstream tasks. Therefore, conclude that the effectiveness of this claim extends beyond ImageNet.
Table G.I: Hybrid models of CL and MIM out- perform both CL and MIM in various tasks.
(IMPORTANCE OF CL)
iNat 18
ADE20k
0.0 (SimMIM)
62.1
35.4
0.2 (SimMIM + MoCo)
68.8
42.2
1.0 (MoCo)
66.2
39.7
19


Reviews:
[{"review_id": "u08nKphA9", "reviewer": "Reviewer_54TF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents comprehensive analyses of self-supervised vision transformers, which provide some new insights about the differences between CL and MIM. For instance, ViTs pre-trained with CL focus on global patterns compared with MIM, collapse into homogeneity while MIM shows more diversity, and reduces the high-frequency signals of the representations but MIM amplifies them. These analyses are helpful for understanding the self-supervised ViTs and show instructions for designing new SSL methods.", "summary_of_the_review": "The paper focuses on the self-supervised pre-trained ViTs, and presents several interesting findings, which may help to understand the different behaviors of pre-training with CL and MIM. It may guide future SSL method design and advance research in this field.", "strength_and_weaknesses": "**Strengths**:\n\nThe paper is well-written and easy to follow. The analyses are comprehensive and thorough.\n\n**Weaknesses**:\n1. Some expression and figures need to be revised (see below).\n2. The proposed method that jointly pre-training with CL and MIM is somewhat not practical due to large training cost, though it is not the key contributions of this paper.\n\n**Questions**:\n1. In Figure1, the authors claim that *CL outperforms MIM in small model regimes*, which is inconsistent with the analyses in [a] with MAE. Some explanation should be presented.\n2. Since most of the analyses are based on ViT-B, are the observations generalized to larger ViTs (*e.g.*, ViT-L or ViT-H)?\n3. In Figure 3, the behaviors of MoCo are rather odd, which are also inconsistent with the observations in [b]. Are the differences from the different settings or the variance of the pre-trained models? My main concern is the reproducibility about the results. Maybe the code and weights to reproduce the results are helpful.\n4. The claim in L86 that *the representations of CL contain shape information, so it can help ...* is not well-supported by the analyses on the attention distance in this paragraph.\n5. I'm not sure whether L118-127 should present in Sec.2 rather than Sec.3, which focuses on the analyses of layer representations. \n6. In Figure 10, it is not clear to me the meanings of \"1e-1\" and \"1e-4\" on top of the figure. And it seems that the changes for CL are larger than MIM, which appears to contradict the observations in Figure5(b) with CKA similarities.\n7. The linear evaluation in L247-267 only reveals the linear separability of the representations, which could not support the claim that \"Later layers of CL and early layers of MIM are important\" well. Is the claim generalized well under the fine-tuning setting?\n\n[a] Wang, S., Gao, J., Li, Z., Sun, J., & Hu, W. (2022). A Closer Look at Self-supervised Lightweight Vision Transformers. ArXiv, abs/2205.14443.\n\n[b] Xie, Z., Geng, Z., Hu, J., Zhang, Z., Hu, H., & Cao, Y. (2022). Revealing the Dark Secrets of Masked Image Modeling. ArXiv, abs/2205.13543.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "foDa4iCGnmY", "reviewer": "Reviewer_n5hD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents comparative studies on various facets of two widely used self-supervised learning methods - contrastive learning and masked image modeling. The studies show opposing properties of the two methods: image information (image-level vs. token-level), features representations (low- and high-frequency) and different lead role components (later- and early-layers). The study also demonstrated that their misture outperforms individuals of them.", "summary_of_the_review": "This paper presents the comparisons of two standard methods with several evaluation metrics. The strength of the paper is the easiness to understand the presented work. The major weaknesses of the paper include the missing depth and width of the comparisons. This paper and its presentation in the current form will not be recommended for acceptance. ", "strength_and_weaknesses": "Strengths of the this paper - \n\nThe entire paper, in its current form, is easy to follow. The examination of certain components in each method is clearly understandable. The evidence presented in the paper seems reasonable.\n\nWeaknesses of the paper - \n\nThe major weakness is the lack of novelty and comprehensiveness. The comparisons show the examination of two methods, given a large number number  of similar approaches reported in the community. \nAnother weakness of the paper is that the mixture of the two methods is to simply use the weighted combination of the two methods. In spite of its simplicity, this combination form lacks clear motivation and clear discussion on the merging form, e.g. why this form is the best way, and how to merge them in the level of components?\n\nActions to be taken:\n\nMore similar techniques must be taken into account. The comparison aspects should be considered, e.g. components and layers of the systems. The integration of the systems should be properly reasoned.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "CmQX_-GMtg9", "reviewer": "Reviewer_Gfy8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper mainly investigates the principle differences between contrastive learning (CL) and masked image modeling (MIM). That’s the CL is shape-biased, which learns low-frequencies, and the MIM is texture-biased, which exploits high-frequencies. The Conclusion is interesting and the analysis is thorough.", "summary_of_the_review": "I am inclined to reject the current version of the paper since the experiment and the corresponding method are not well explored.", "strength_and_weaknesses": "Strength:  \nWell-written, clear analysis and easy to understand.  \nThe conclusions drawn in the paper may be instructive to the self-supervised learning community.  \n\nWeaknesses:  \n(1) The main concern lies in the quantitative experiments. In fact, the paper lacks experiments on downstream tasks, like image classification, object detection or segmentation. The authors should provide corresponding downstream experiments to support the effectiveness of the conclusion other than visualization of principle analysis.  \n(2) The principle analysis occupies a large amount of space in the paper, on the contrary, the method derived from the conclusions is rarely involved. Actually, the corresponding method (Section 5) is also the core of the paper.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1o1_jHI5w_", "reviewer": "Reviewer_wzZN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper compares two self-supervised classes of methods for vision transformers, namely contrastive learning and masked reconstruction.\nThrough several quantitative analyses of pre-trained models, the authors expose a series of findings on the learning properties of these methods.\nThe main findings for contrastive learning are: it captures global low-frequency patterns, its intermediate representations are rather homogeneous, and self-attention is its key component.\nConversely, for masked reconstruction, the main findings are: it focuses on local high-frequency patterns, its intermediate representations maintain diversity, and the MLPs are its key component.\n", "summary_of_the_review": "This paper offers a quantitative study of various properties of contrastive learning and masked reconstruction in self-supervised vision transformers. Such a study is valuable as it validates what has been observed empirically in the community and provides a tentative analysis of the reasons behind these observations. The paper itself is well-written and clearly outlined. For these reasons, I recommend the paper for publication.", "strength_and_weaknesses": "Strengths:\n\n- The paper tackles interesting questions in self-supervised learning and provides answers that are thoroughly supported by empirical evidence. With a possible risk of confirmation bias, the findings in this paper validate the anecdotal evidence that has been observed in the community for a while.\n\n- From the reader's perspective, the paper is well-written and well-organized. The authors have done a great job of outlining the research questions and their findings in the introduction.\n\n- The research presented in this work is quite extensive. I appreciate that the authors did not stop at the first finding and instead insisted on probing the models with different tools to highlight their differences from various perspectives.\n\nWeaknesses (or rather suggestions for improvement):\n\n- Complement the discussion by mentioning the findings of \"High fidelity visualization of what your self-supervised representation knows about\" (Bordes 2022). This work, concurrent with the authors' work, offers qualitative evidence that broadly agrees with the findings of this paper.\n\n- L59-61 \"early layers are usually known to capture low-level features, e.g., local patterns, high-frequency signals, and texture information, and later layers capture global patterns, low-frequency signals, and shape information\". In the context of convolutional networks, due to their limited receptive field and pyramidal architecture, this statement is considered common knowledge and is supported by a large body of studies. However, this paper focuses on transformers, where this behavior is still under investigation. I suggest providing references to support this statement and possibly clarifying its scope.\n\n- Most of the discussion points compare MoCo vs. SimMIM. However, there are parts that would be more convincing if more methods were added to the discussion. An example of this, L257-267 discuss the difference between SimMIM and MAE in terms of decoder depth, why not include MAE in figure 11?\n\nMinor points:\n\n- Improve figure 1 by adding a grid, separating the plots more, and tweaking the font size. Actually, most figures would benefit from a grid.\n\n- L67: \"As shown in Figure 1\". Figure 1 deserves a bit of introduction, e.g. what models and methods are being compared and on what, before discussing its meaning. I see that the caption contains all the required information, but it is weird to read the first line of section 2 without knowing what the figure is about.\n\n- Improve figure 10 by connecting all MLP markers with a line, and connecting all self-attention markers with another line, rather than relying on white/gray background to distinguish them. Alternatively, if you want to use a single line for each method, change at least the markers of MLP and self-attn layers.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster