Efficient Self-supervised Vision Transformers for Representation Learning
Authors: Chunyuan Li
Jianwei Yang
Pengchuan Zhang
Mei Gao
Bin Xiao
Xiyang Dai
Lu Yuan
Jianfeng Gao
Venue: ICLR 2022 Poster (2022)
Keywords: self-supervised learning
vision transformers
non-contrastive region-matching task

Abstract: This paper investigates two techniques for developing efficient self-supervised vision transformers (EsViT) for visual representation learning. First, we show through a comprehensive empirical study that multi-stage architectures with sparse self-attentions can significantly reduce modeling complexity but with a cost of losing the ability to capture fine-grained correspondences between image regions. Second, we propose a new pre-training task, non-contrastive region-matching, which allows the model to capture fine-grained region dependencies and as a result significantly improves the quality of the learned vision representations. Our results show that combining the two techniques, EsViT achieves 81.3% top-1 on the ImageNet linear probe evaluation, outperforming prior arts with around an order magnitude of higher throughput. When transferring to downstream linear classification tasks, EsViT outperforms its supervised counterpart on 17 out of 18 datasets. The code and pre-trained models are released at: https://github.com/microsoft/esvit

Published as a conference paper at ICLR 2022
EFFICIENT SELF-SUPERVISED VISION TRANSFORMERS FOR REPRESENTATION LEARNING
Chunyuan Li' Jianwei Yang'
Pengchuan Zhang'
Mei Gao
Bin Xiao
Xiyang Dai?
Lu Yuan² Jianfeng Gao'
Microsoft Research at Redmond, Microsoft Cloud AI {chunyl, , jianwyan, penzhan, xuga, bixi, xidai, luyuan, jegao/Jmiccroooft.com
ABSTRACT
This paper investigates two techniques for developing efficient self-supervised vision transformers (EsViT) for visual representation learning. First, show through a comprehensive empirical study that multi-stage architectures with sparse self-attentions can significantly reduce modeling complexity but with a cost of losing the ability to capture fine-grained correspondences between image regions. Second, we propose new pre-training task of region matching which allows the model to capture fine-grand grained region dependencies and as result significantly improves the quality of the learned vision representations. Our results show that combining the two techniques, EsVIT achieves 81.3% top-1 accuracy on the ImageNet linear probe evaluation, outperforming prior arts with around an order magnitude of higher throughput. When transferring downstream linear classification tasks, VsV outperforms its supervised counterpart on 17 out 18 datasets. The code and pre-trained models are released at: htttps://githt: com/microosoft/eviil
INTRODUCTION
Self-supervise learning (SSL) with Transformers (Vaswani et al. 2017) ) has become a de facto standard of model choice in natural language processing (NLP). The dominant approaches such as GPT (Radford et al., 2018) and BERT (Devlin et al. 2019) are pre-training on a large text corpus and then fine-tuning to various smaller task-specific datasets, showing superior performance. Larger Transformers pre- pre-trained with larger-scale language datasets often lead to a stronger generalization ability, demonstrated by improved performance in downsteam tasks (with no sign of performance saturation yet), as exemplified in GPT-3 (Brown et al. 2020
In computer vision (CV), however, self-supervised visual representation learning is still dominated by convolutional neural networks (CNNs). Sharing similar goal/spirit with NLP, SSL in CV aims to learn general-purposs image features from raw pixels without relying on manual supervisions, and the learned networks expected classification, detection and segmentation. Recently, impressive performance have been achieved by CNN-based SSL, outperforming state-of-the-art (SoTA) fully-supervised pre-training methods (He) ct al. 2020 Caron et al. (2020) on tasks with a limited number of labels. The key to success is View-level Tearning: maximizing agreement of learned representations between differently augmented views of the same example. Recent works, including SimCLR-v2 (Chen et al. 2020d). BYOL (Grill
as the backbone of various downstream tasks such as
al. 2020) and SwAV (Caron al. 2020), have scaled up the CNN-based models to hundreds of millions of parameters. However, SSL has not enjoyed the same scaling success in CV as that in NLP.
Several attempts have been made to close the gap by combining SSL with Transformer and self- attention architectures. Early works include Selfie (Trinh et al.) 2019). which generalizes the concept of masked language modeling of BERT for images. The idea has been recently revisited in Vision Transformer (ViT) Dosoviiskky et al. 202 l via pre-training 300M. ImageGPT (iGPT) (Chen et al. (0200b generalizes the concept of auto-regressive language modeling of GPT for images, showing encouraging ImageNet recognition accuracy with large model size. Contrastive learning with ViT has also been studied very recently in DINO (Caron) al. 2021) and MoCo-v3 (Chen et al. 2021) where new SoTA result by linear probe evaluation
much larger scale dataset, e.g., JFT-

---

Published as a conference paper at ICLR 2022
W
Vo (VIT
O (Swin W-14)
80
probing accupgey (9%
Lineer probing accuracy (%)
EMVT [Swin-B]
DINO
S mmCuRiNZ
(ResNet152 w3-5K1
(DNIT-SF8)
McCo-la MTHEN/IN
(ResNat200 ESVIT
Swif (Resitet)
(ResNee)
Linegr p
SWAV (6WN
SimCUR-v2 (kesMet
ESVIT jwin--)
-6
MoCo-u3 (VT)
(MT. P-16)
ESVIT (V)
(V+R)
20
45
100
200
500
1000
0 50
100
200
5.00
Throughput (#imagesss) in log space
# Porameters (Millions)
Figure 1: Efficiency vs accuracy comparison under the linear classification protocol on ImtgeNet. Left: Throughput of all SoTA SSL vision systems, circle sizes indicates model parameter counts; Right: performance over varied parameter counts for models with moderate (trroughouu//paramettr)) ratio. EsVIT pre-trained with and without the region-matching task are shown before and after the arrow,, respectively. Please refer Section 4.1 for details.
ImageNet-IK is achieved, by exhaustively consuming computation resource on full self-attention operators with long sequences of split image patches.
Aiming to improve the efficiency of teassformer-basee SSL, this paper presents Efficient self- superivsed Vision Transformers (EsViT), by using multi-stage architecture and a region-based pre-training task for self-supervised representation learning. Our main findings and contributions can be summarized as follows:
(1) An intriguing property of self-supervised monolithic Transformers is firstly reported in our paper: automatic discovery of semantic correspondence between local regions.
(2) We present the first comprehensive empirical study to show the pros and cons of multi-stage vision Transformer architectures for SSL. Though greatly reducing compute complexity, we find that the multi-stage architecture causes the loss of the property in (1).
(3) A region matching pre-train task is proposed to alleviate the issue in (2), and further improve the learned representations and attentions.
(4) We validate the new Es ViT, which combines the techniques, a range of tasks. It sig- nificantly reduces the cost in building SoTA SSL vision systems, as summarized Figure [ and shows better scaling performance on accuracy vs. throughput and model size. Under the linear evaluation protocol, ViT achieves 81.3% top-1 accuracy, showing the best performance compared with all systems, and is 3.5x parameter-efficient and has least 10x higher through- put than previous SoTA (81.0%, MoCo-v3 with ViT-BN-L/7 (Chen et al. (02Th). Compared with its supervised counterpart Swin Transformers (Liu et al.) (2021), I) Es VSV shows superior performance on 17 out 18 datasets, when transferring the learned representations to downstream linear classification tasks.
METHODS
Transformer based SSL methods emerge very recently to lead the state-of-the-ar performance on the ImageNet linear probe task (Chen et al., 2021: Caron al., (021). It inherits the successes from (1) monolithic Transformer architectures that dominate in NLP (Devlin et al., 2019 Radford al. 2018), and (2) instance-level contrastive learning objectives that demonstrate arguably the best SSL performance in computer vision (Chen et al. 2020c). Though simple and effective, the existing Transformer-basee based SSL methods require large amount of compute resources (e.g. >1.7 TPU years of training) to reach SoTA performance. We believe that the SSL system efficiency is highly related to two ingredients: the network architecture and the pre-train task. To strike for better tradeoff between accuracy and efficiency, we present VSVTT, showing better synergy of networks (a multi-stage Transformer architecture) and pre-train tasks (a non-contrastive region- matching task).
2.1 NETWORK ARCHITECTURES: FROM MONOLITHIC TO MULTI-STAGE VIT BACKBONE
Multi-stage VIT. This paper presents the first empirical study of multi-stage Transformer architec-
tures (Vaswani al. 2021 Wang et al. 2021 Liu al. 2021 Zhang et al. 2021 Wu et al. 2021)
2

---

Published as a conference paper at ICLR 2022
for SSL. Each stage consists of a patch merging/embedddig module, and Transformer with sparse self-attention module. (i) The patch merging module plays a slightly different roles in different stages. In the first stage, it splits an input RGB image into non-overlapping patches. Each patch is treated as a "token", constructed as a concatenation of the raw pixel RGB values, which is further projected into C-dimension feature. In the later stage, the patch merging module concatenates the features of each group of 2 x 2 neighboring patches, and applies a linear layer on the 4C-dimensional concatenated features. This reduces the number of tokens by a multiple of 2 x 2 4, and the output dimension is set to 2C. (ii) A Transformer with sparse self- attention module are then employed to enable interactions among the merged features. The two modules above are repeated for multiple times, typically 4 times, resulting in a multi-stage VIT. As result, a hierarchical representation is generated: the number of tokens reduced and the feature dimension (and the number of heads in self-attentions) of each token is increased, as the network gets deeper. An overview comparison of the monolithic and multi-stage Transformer architectures for SSL is illustrated in Figure in Appendix.
An intriguing property of self-supervised monolithic VIT. Though straightforward in implemen- tation, changing from monolithic multi-stage architecture without careful treatments may lose some desirable properties of self-supervised Transformers In out study, we first empirically note an intriguing property of self-super monolithic ViT al. 2021]: the pre-trained model exhibits a very strong ability to automatically discovers correspondences, even without a region-level matching objective specified in training.
We quantitatively evaluate the correspondence learning to illustrate this property, as discussed in the following process. (i) Simulated benchmark. Based on 50K images in the ImageNet validation dataset, we create simple evaluation benchmark with mild augmentations: For center-crop image, we apply Hori meiizottllle aalllllipp ip, then ColorJitter and RandomGraysca le to create a new augmented view. In this way, ground-truth correspondences are created. (ii) Evaluation process. Given two views of the same image, we use the pre-trained backbone to extract the top-layer features, and find the feature vector in view that best matches the other in terms of highest cosine similarity. The accuracy is measured as the averaged percentage of correctly identifying the region-to-region correspondences. Please see details in Section C.7 in Appendix. (iii Results. We quantitatively show that a self-supervised monolithic ViT yields 95% accuracy. However, simply replacing the network with a multi-stage Transformer yields only 66 % accuracy. This significant degradation (absolute 29 % accuracy drop) reveals the loss of the correspondence learning property. We first raise this critical problem, and believe that it has a large impact on the pre- pre-trained model's performance in various downstream tasks.
2.2 PRE-TRAINING TASKS: DELVING INTO VIEWS WITH REGIONS
We employ non-contrastive learning framework to build our SSL method. Specifically, Self- distillation with no labels (DINO) (Caron et al. 202T) considered. It leverages the knowledge distillation learning paradigm where a student network go, is trained to match the output of a given teacher network ge. parameterized by 02 and O respectively. The neural network is composed of backbone f (e.g., Transformers or ConvNets), and of a projection head h: g g= h f. The features used in downstream tasks are the output of backbone f. In SSL, different augmented views 2 of an image z are fed into backbone network to obtain feature maps by so fofft tmax per network further convert the feature vectors 2 E z into probability vectors p h(z); one head for view-level and the other head for region-level, respectively.
fā)). Two MLP heads followed
More precisely, from a given image, we generate a set V of different niewsffoloose (Caron et al. 2021). The resulting feature map the top layer for each view is [zı,..., 27). where 7 is the sequence length, and 2; is region-level representation for the local patch at position i. Average pooling is applied obtain the view-level representation
avg-pool(z).
Vivel task Given the augmented view set for student V and teacher V*, set of pairs ((x, () E V, R. E y* and # ( is constructed to perform cross-view prediction tasks. We
'This set often contains views of two different resolutions V = [Vg, Vij. where V,- = ( = 1,2) is global-view set of higher resolution, and Vi = ( I = 1, ......8 8} is a local-view set of lower resolution. All views V are passed through the student while only the global views V are passed through the teacher.
3

---

Published as a conference paper at ICLR 2022
consider the pre-training task at the view level proposed by (Caron et al. 2021
Lv = P
Σ (a,))EP
Mv(s,t), with (y, = -Ps log
(1)
where p, h2,)) and pt = h(z,) are the probability output of an MLP head over the view-level representations 2, and Σ,, learned by student and teacher, respectively. In DINO, ViT/DeiT are considered, hence the view-level representation is the feature of the [CLS] token.
Region-level task In (Caron et al. 2021), the Lv encourages "local-to-global correspondences only at a coarse level: the large crop and the small crop are matched in the view level, leaving region-to-region correspondence unspecified. In monolithic Transformers, the drop paths and skip connections from low-level features to high-level features help the the latter to remain discriminative, thus maintain good engiongmachhing performance. However, such a property gets diluted due to the merging operators in multi-stage Transformers. As shown in our experiments later, training multi-stage network with Ly only indeed results in sub-optimal representations.
Further, it could be waste of computation not to leverage region-level features z that are computed the process of extracting view-level feature. Inspired by the success of masked language modeling task in BERT, we argue that it is important to have region-level pre-training task for computer vision, so that the model can (1) amortize the computation and fully leverage the extracted region-level features, and (2) take into account the co-occurrencces/sttuuuurr between local features. Unfortunately, directly performing masked patch prediction (MPP) for the multi-stage Transformer architecture is infeasible, the one-to-one correspondences between the input visual tokens and output features get diluted due to the merging operation. Even for monolithic architectures, MPP has not been proved effective in computer vision, empirically shown in Dosoviiskki et al. 2021).
To address this problem, we propose a non-contrastive, region-matching method that directly works at the level of local features by taking into account their correspondences:
2. 2j | ||
CR
Σ MR(s,t), with MR(s,t) (s.DER
Σ
Ps logPe, Pr. 5 arg max
(2)
P
where P; (2:) and p; (22z)) are the probability outputs of MLP head h over the local features of student 2; E z, and teacher 2j E 2,, respectively. j* is the index of the feature in 2, that best matches the i-th feature in 2., in the sense of highest cosine similarity. Note that z, and zy- contextualized features of two best matched regions from different augmentated views, minimizing CR encourages different contexts (i.e., surrounding regions) learn invariant features, and thus captures the region-dependency.
The overall pre-training objective of ESVIT is f = CR + Lv, we learn to match the fea- ture distributions at both the view and region levels by minimizing the cross-entropy loss w.r.t. the parameters of the student network ge.- A visual illustration is in Figure and the full algorithm in Appendix. We updates teacher/student network alternatively: (i) Given a fixed teacher network, the student network is updated by minimizing the full cross-entropy loss: O. +- arg ming, L(s, (; (ii) The teacher model is updated as an exponential mov-
Global Taken
Local (Top layer feature
View1
View-level
Region-leval
Prediction
Prediction
Ly MLP
MLP Head
Li
Head
Figure 2: Pre-training objectives, including view-
level (left) and region-level (right) prediction. ing average (EMA) of the student weights 0, + X0, + (1 A)0., with À following a cosine schedule from 0.996 during training. By default, the full objective L is used from the beginning. One can also load checkpoint trained by Ly only, and add Ln for continual pre-training, which is shown effective boosting performance in our experiments.
Computational overhead tecture can be prohibitively computationally expensive, as requires O to compute Zn- For a typical image of resolution 224 x 224, the feature map length of ViT/DeiT (with patch size 16) at
Note that applying La on the traditional monolithic Transformer archi-
4

---

Published as a conference paper at ICLR 2022
the top layer is T = 196, while the multi-stage architecture yields T = 49, which requires 3 times less compute in computing LR. To empirically illustrate this, we show in Appendix Section C.2 that Ln adds acceptable extra memory and computational cost (around 1.2 and 1.05 x, respectively) for multi-stage Transformers, while it will quickly go out-of-memory for monolithic Transformers when the batch size is increased.
RELATED WORKS
Relation to mask prediction tasks We can consider the proposed LR as a proxy mimick masked language modeling in BERT, where the "ground-truth" local token is a soft label provided by the teacher network, while the student network makes predictions to match that target, based on the context of regions in different augmented view. Importantly, our La considers softmax with cross-entropy in the objective, rather than MSE as in MPP. very sharp teacher distribution used by choosing small temperatures, This encourages the model to focus on the salient dimensions, rather than waste modeling capability on training short-range dependencies and high-frequency details (Ramesh et al., 2021).
The proposed LR mostly related to DenseCL (Wang et al., 2020b) in that the region correspondences in both methods determined the two most similar grid features. One critical difference is that DenseCL. is a contrastive region- matching task, while our Ln is a non-contrastive region- matching task, where negative samples/queue is needed. This technical difference has a significant impact on the downstream task performance. We find that L is particu- larly effective in serving our goal improve image classification performance and build efficient & affordable SoTA SSL system; In contrast, DenseCL degrades the classification performance.
Relation to DenseCL
Relation to other region-level tasks
The ideas of leveraging local region-level pre-training tasks for visual representation learning have been explored for ConvNets (Misra & Maaten 2020 Xiong
al. 2020 Wang et al., 2020b Xie al.] 2021a Yang et al. 202 Xie al. c02 We summarize the differences in three aspects: (7) Motivation. Our region matching task La aims to recover the lost property of automatic correspondence learning in self-supervvsed monolithic Transformers, while most existing region-level tasks aim to improve dense visual prediction tasks. (ii) Technical differ- ence. Our L is a non-contrastive egion-matching task, while others are contrastive learning. (ii) Empirical performance. Most region-level tasks improve dense visual prediction tasks but sacrifice their image classification performance, while L consistently improves classification performance. Among them, Es VIT training method achieves the best ImageNet linear probe performance with minimum computational overhead. For detailed comparisons, please refer to Table 7 in Appendix.
Self-supervised vision Transformers. The research on Transformer- based self-supervised repre- sentation learning just scratches the tip of the iceberg, and only a few attempts are made on this topic. ImageGPT (Chen et al.) (020b) and MoCo-v3 (Chen et al., 2021) dedicate huge compute resource with large models to exploring the frontier. DINO (Caron et 2021 achieves comparable performance large self-supervised ConvNets using small/medium-sizz Transformers. The proposed ESViT further pursues efficient and affordable solutions to self-supervised vision Transformers. For more general related works on Transformers for vision tasks and self supervueee ConvNets, please refer to Section B in Appendix.
4 EXPERIMENTAL RESULTS
We describe the experimental settings in Appendix Section C.3 and evaluate the proposed EsViT to answer three questions: Q1: How does ViT perform on standard ImageNet benchmark compared to SoTA methods? Q2: How effective EsV is when transferring to downstream tasks? Q3: What are the design choices and empirical contributions ZR? Q4: When does the intriguing property of self-superviser Transformers exist, including learned correspondence and attentions?
4.1 COMPARISONS WITH PRIOR ART ON IMAGENET
We report t linear probe and K-NN accuracy the ImageNet validation set. Table 1 presents comparisons with SoTA SSL systems across various architectures. Please refer to Figure | for comparisons scaling parameter counts and throughput. Our findings are summarized below.
Comparisons with self-supervised Transformers. The DINO- and MoCo- based ViT has higher accuracy and smaller models than iGPT, under the same linear probing protocol and training data.
5

---

Published as a conference paper at ICLR 2022
Method
#Parameters
Throughput f
Linear f
k-NN
SoTA SSL methods with Big ConvNets
AWAV, RN50W (Caron et al..
2020
586
76
78.5
67.1
BYOL, RN200W2 (Grll et al.
2020
250
123
79.6
73.9
Sim RN1S2W37SK Chen et al. 2020d
794
46
79.8
73.1
Skyline methods with excessively long sequences for self-attentions
DINO, DeiT-S/8 (Caron et al.
2021
21
180
79.7
78.3
DINO, ViT-B/8 (Caron el ml
2021
63
80.1
77.4
MoCo-v3. VIT-B-BNN7 (Chen
79.5
MoCo-v3, VIT-L-BNN/7 CChen
81.0
iGPT, iGPT-LL (Chen ct 20206
6801
72.0
EsViT, Swin-T/WW =11
28
660
78.7
77.0 (75.5)
EsVIT. Swin-S/W 14
49
383
80.8 (79.0)
79.1 (77.3)
EsViT, Swin-B/W-
87
254
81.3 (80.5)
79.3 (78.3)
Transfonner-bused SSL, with moderate sequence length for self-attentions
Masked Patch Prod.. B/16 (osovitskiy et 2021
312
79.91
DINO, DeiT-S/16 (Caron al.
2021
21
1007
77.0
74.5
DINO, ViT-B/16 Caron et al.
2021
85
312
78.2
76.1
MoCo-v3, VIT-B/I6 (Chen et al.
202
85
312
76.7
MoCo-v3, VTT-H-BN/I6 (Chen ctal.
2021
632
~32
79.1
MoBY, Swin-T (Xie et al. 2021b
808
75.1
EsVIT, Swin-T
78.1 (77.0)
75.7 (74.2
EsVIT, Swin-S
79.5 (79.2)
77.7 (768)
EsVIT. Swin-B
297
80.4
78.9 (77.3)
Table 1: Comparison with SoTA across different architectures on ImageNet linear probing. ESVIT
with L1 + Ln is reported, while VIT with only Ln is shown in parentheses. W = 14 is the window size, otherwise the default W ViT-BN is ViT that has BatchNorm (Frankle al. 2020). and
"/P" denotes patch size of P. "~" indicates through-puts estimated by comparing different papers, detailed Appendix. The mask patch prediction (Dosovitskiy et al. 2021 pre-trained on JFT-300M and end-to-end fine-tuned in ImageNet, which we append as a reference.
At the similar level of model size and compute complexity, the proposed ViT improve SoTA methods DINO/MoCO-v3 by large margin: ESV (Swin-B) outperforms DINO (ViT-B/16) by 2.2% linear probe accuracy and 2.8% k-NN accuracy in absolute values. ESVIT (Swin-B) even performs slightly better than DINO (ViT-B/8) (0.3% higher linear probe accuracy and 1.5% higher k-NN accuracy), with 4x higher throughput. MoBY (Xie et al., 2021b) is con-current work that investigates multi-stage ViT in SSL. With the same architecture Swin-T, our VSV pre-training tasks significantly outperform MoBY, showing 3% higher accuracy. In VSV, longer sequences in self-attention is implemented by increasing the window size. We experiment this by considering a window size of W = 14. Overall, the proposed EsVIT Swwi-B//==ll shows the best performance (top-1 accuracy 81.3%, top-5 accuracy 95.5%, k-NN accuracy 79.3%), compared with all systems, and is 3.5x parameter-effiiient and has at least 10x higher throughput than previous SoTA MoCo-v3.
Comparisons with big ConyNets. We compare with the SoTA big ResNets reported by SimCLR- v2 (Chen et al. 2020d). BYOL Griil et al. 2020) and SwAV (Caron et al. 2020). Among them, the best accuracy 79.8% under the linear probing protocol reported by SimCLR-v2 with SK-ResNet, where Selective Kernel (SK) (Li et al. 2019c) is form of attention to enhance CNNs. It is clear in [ that all Non-Nets- SSL methods show an envelope in the regime of scaling up model sizes after passing 500M. Es ViT achieves better accuracy than their highest envelope, with 16x less model parameters and 8x higher throughput.
4.2 TRANSFER LEARNING
We also conduct transfer learning in downstream tasks to evaluate the quality of learned representa- tions. Two sets of tasks are considered:
Classification on a suite of 18 small datasets. As exemplified in (Radford et al. 2021), it is common and clean approach to evaluate a learned representation by fitting linear classifier on the representation and measuring its performance across multiple datasets. We study 18 datasets used in (Radford al., 202T). Automatic hyper-parameter tuning is considered to ensure fairness of comparison. Besides averaged scores, report wins as the number of datasets on which the model outperforms its supervised counterpart. Detailed dataset description and settings in Appendix.

---

Published as a conference paper at ICLR 2022
Supervised: 77.86
ESVIT (V):
EIVIT (V+ 80.86
+1.05
+0.96
-5.27
+0.11
-0.63
(%)
90
80-
+2.89
+4.18
+102
+2.04
+2.29
70-
+4.95
+2.47
+7.61
+4,59
Calsern
CHARIO
(9AR100
Dro
FER2013
PGVCAlrreet
Fosers102
food nl
orse
Hatekallomes
MNIT
PatshCCneeyon
oxforPatt
StantentCass
S7.10
S08857
uotdl
VoC2867
Figure 3: Linear probing on 18 downstream datasets. Averaged scores are reported for each method. EVV outperforms its supervised counterpart 17 out of 18 datasets.
Apds
APd
AP
Pre-train Data
ImageNet-IK
18 Datasets
Sup.
46.0
68.1
5i
50.3
75
Linear
k-NN
Scores
# Wins
ESVIT
46.2 (46.2)
68.0 (67.9)
50.6 (50.5)
Supervised
77.29
ImageNet-1K
78.0 (77.1)
75.7 (73.7)
80.66
Apmb
Apmb
APm
WebVision-vl
75.9 (75.4)
71.2 (69.4)
80.00
14
Sup.
41.6
65.1
44.9
OpenImages-v4
70.6 (69.6)
62.0 (60.3)
77.97
10
ESVIT
41.6 (41.7)
64.9 (648)
44.8 (45.1)
ImageNet-22K
75.0 (73.5)
67.9 (66.1)
81.03
17
Table 2: COCO Detection & Segmentation.
Table 3: Impact of the pre-train datasets.
Detection and segmentation on COCO, Different from previous monolithic self-supervised ViT, the multi-stage architecture in VvV can be readily used for dense visual tasks that require hierarchical feature representations.
Comparison with supervised counterparts. We compare with the supervised-leaming Swin, whose checkpoints are downloaded from the official codebase" Figure 3 shows the classification results of Swin-S, ViT consistently outperforms its supervised variant, often by a large margin. Similar conclusions drawn for other model sizes. On COCO detection and segmentation task, however, VSV shows comparable results with the variant with Lv only (shown in parentheses) and the supervised counterpart (Swin-T trained with 3x schedule), shown in Table 2 We hypothsize this is related to the non-constrastive nature of ViV as explained later.
Effects of larger, less-curated pre-train datasets. The performance of Transformer-basee SSL research has thus far been limited to highly curated pre-train data such as ImageNet-1KK To push the frontier in leveraging large amounts of unlabeled data, explore the effects of pre-training from larger, less-curated image datasets: Web Vision-vl (Li et al. (017).. Openlmages-v4 (Kuznetsoval al., 2020) and ImageNet-22KK (Deng al. 2009), described in Appendix. The pre-train epochs on different datasets are adjusted so that all models see a similar number of augmented views. We summarize the results in Table 3 and would like to emphasize the following findings. First, CR improves Lv (shown in parentheses) on all datasets. Second, all EsVIT pre-trained checkpoints outperform supervised checkpoint in downstream classification tasks, but performance varies a lot, with ImageNet-22K checkpoint showing the best transfer ability. Third, ImageNet-I 1 K pre-trained model shows the best ImageNet- IK linear probe performance. We hypothesize that is not only the size of pre-train dataset matters, but also the distribution of image classes matters: more diverse and bal-ncal distribution results in a stronger generalization ability.
4.3 DISCUSSION ON THE NON-CONTRASTIVE
AAGION-MATCHING TASK
Compatibility with various network architectures. We investigate ResNet-50 and different effi- cient sparse Transformers in Table 4 DeiT is shown baseline reference. Batch size 1024 in this experiment. To ensure fair comparison, we modify all into a 4-stage architecture with the number of Transformer layers in each stage 2-2-6-2. We that La improves all network architectures, including ResNet-50, Swin (Liu et al.] 2021), ViL. (Zhang et al.) 202T), CvT (Wu et al.) (021) and PvT (Wang al., 2021I. Though directly adding CR to monolithic ViT is computationally infeasible, we uniformly sampled top-layer grid features of DeiT and then add LR. but did not observe performance improvement. This is partly because the monolithic ViT itself already has
good corresponding ability, an extra region-matching task does not provide new learning signals. As

---

Published as a conference paper at ICLR 2022
Method
#Param.
Im./s
Pre train lusks
Linear
ENNN
Ly (Ting)
4 Ly (Small)
Ly iBase)
DefT
21
1007
Lv
75.9
73.2
(Tiny)
(Smail)
Ly Le (Base)
Dy
75.31
67.57
R-50
24
1237
Ly
75.0
69.3
Dy+Ca
75.7
71.2
Swin
28
808
DvtLn
Ly
77.6
77.1
75.4
73.7
KNN 1990
Vil.
28
386
Cy
77.3
73.9
77.5
74.5
CVT
29
848
CytCe
Dy
77.6
78.5
76.7
74.8
70
Lv
75.4
72.0
PVT
24
851
Cv
76.3
72.9
0
50 100
200
Table 4: Different architectures with and without CR. DeiT and ResNet-50 are shown as references. Numbers reported in (Caron et al. 2021).
Figure 4: Learning curves of different pre- training tasks. For Base model, CR is added from the 200th epoch.
Pre-training ResNet50 in different settings
ImageNet-1k
COCO
Types
Methods
#Epochs
#Views
Linear
k-NN
Apds
Apmò
Supervised
38.2
33.3
MoCo-v2
200
2
67.5
55.6
38.7
33,9
Contrastive
DenseCL
200
2
63.6 (-3.9)
48.6 (-7.0)
39.1 (+0.4)
34.2 (+0.3)
LV
200
2
69.2
59.9
37.8
33.1
Non-Contrastive
CC+
200
2
69.9 (+0.7)
61.7 (+1.8)
38.0 (+0.2)
33.2 (+0.1)
Table 5: Comparison between contrastive and non-contrastive region-matching tasks.
compared in Appendix Table 12 with the ResNet-50 backbone, EsViT learning method shows the highest accuracy, compared with existing SSI. methods.
Model scaling with Ln- We compare the pre-training objective with and without La in Table 1 Across different model scales and window sizes, the proposed region level LR can consistently improve the performance. The gains can be clearly seen by k-NN accuracy (around 1-2%), where no additional tuning is needed as in linear probe. Figure 4 demonstrates that CR helps model convergence, and can be used as drop-in improve models trained with the view level task.
Contrastive ys Non-contrastive region-matching tasks. mittonn task to the non-contrastive view-level task Lv; On the contrary. DenseCL adds contrastive region-matching task to the contrastive view-level task MoCo-v2. In Table we compare four methods in the same setting with ResNet-50. DenseCL improves dense visual prediction perfor- mance, but hurts classification performance. LR improves both tasks, especially the classification performance. One limitation is that the non-contrastive methods show lower performance in dense prediction tasks, this is consistent with the observations for BYOL in (Wang et al.) (020b).. The simple La shows the best ImageNet accuracy compared with all sophisticated region-level tasks in this 200-epach setting in Appendix Table and the best overall accuracy in Table 12 It indicates that CR well serves our goal in building efficient SoTA SSL systems.
The proposed Ln adds non-contrastive
Design choices of LR. We ablate a couple of choices in constructing CR in Eq. 2 (i) So ftmam vs MSE. One alternative way to measure the distance between two projected vectors is MSE, as employed the popular non-contrastive SSL algorithm BYOL (Grill et al. 2020). When adding tasks to BYOL and pre-training 50 epochs, Softmax and MSE yield k-NN accuracy of 37.2% and 34.9%, while the baseline BYOL yields 33.1%. We also replace the region-matching metric in VST as MSE, yielding k-NN accuracy 72.6%, which lower than the view-level task only (74.2%). These results show that Softmax is essential CR. (ii) Optimal Transport (OT) vs Simple Argmax. To avoid heavy computational overhead, simple feature-level argmax solution is considered in Eq. 2 to pair two local regions. To study the impact of high region-macchin quality, consider OT. Empirically,
observe OT yields slightly higher I:-N accuracy at the early stage, but the gain is diminished in the end. Considering the extra computational cost of solving OT with an inner loop in sinkhorn algorithm (Cuturi 2013), we opt for simple argmax in experiments.
8

---

Published as a conference paper at ICLR 2022
(a) DINO: DeiT-S
(b) ESVIT: bv
(c) EsViT: CV LR
Figure 5: The learned correspondences. Yellow lines are the top-10 10 correspondences between two views, where the numbers indicates the rankings of similarity scores, yellow dots with the same number are paired.
(a) DINO: DeïT-S
(b) VsV:T: Lv
(c) EsViT: Lv +L
Figure 6: Visualization of the the learned attention map for different heads in the last layer. The query the blue dot in the center of the images. We visualize masks (as red) obtained by thresholding the self-attention maps to keep 60% of the probability mass. Note that all 6 heads are visualized for DINO with DeiT--S, and 6 out of 24 heads in EsViT are chosen to visualize (ranked by entropy values). Please enlarged pictures with all heads in Appendix.
4.4 QUALITATIVE STUDIES
Visualization of correspondences. Given two views of the same image, we use the pre-trained backbone extract the top-layer features ø1 and ø2. For each feature vector in 21, we find the feature vector in 22 that best matches it in terms of highest cosine similarity, as defined in Equation ( In Figure 5 we show the top- 10 correspondences between two views for three methods. In Figure (b), Es VIT with Ly tends to identify pairs in the background as the most matched ones (and in wrong way in this example). This could be a valid solution to Lv, as the invariance the level of aggregated global features does not necessarily induce invariances in the local region level. This is significantly alleviated with Lg (shown in Figure (c)), a task that implicitly requires local matching.
Surprisingly, DINO is able to learn good correspondences even without the region-level matching task. To the best of our knowledge, this is a previously unreported intriguing property of self- supervupe automatically
Transformers with monolithic architectures: good semantic correspondences learned. We hypothesize that features at lower layers (image patch itself in the extreme case) can directly pass to higher layers, and the former regularizes the latter to remain discrininative. Nevertheless, the proposed LR can dramatically reduce the issue, and is good remedy to rescue the loss of semantic correspondence for the multi-stage architecture. In Appendix, we quantitatively measures the correspondence learning ability of these SSL methods on ImageNet validation dataset, the observations are consistent: LR improves the matching accuracy from 66% to 91%.
Visualization of attention maps. We look at the self-attention in the different heads of the last layer in Figure 6 A local region on the edge of the main object is employed as query, and the attended regions are highlighted in red for those the query's 's top 60% mass are assigned. Appendix, we visualize more examples with different query positions. DINO tends to automatically learn class-specific attention maps leading to foreground object segmentation, regardless of its query located in foreground or background. This is probably because main objects remain the major invariance factor in different augmented views. This property is lost when multi-stage architecture is employed, shown in VST with Lv. These patterns consistent for different heads. After introducing LR for ESVIT, we note that the attention maps become more diverse in different heads, i.e., entropy values of attentions get more skewed, and attended regions more different. This is perhaps because Ln requires each region to consider many matching tasks to regions in different augmented views, each head automatically learns to distribute the tasks and complete few of them.

---

Published as a conference paper at ICLR 2022
5 CONCLUSIONS
In this paper, first discover the automatic correspondence learning property of self-supervised monolithic Transformers. Inspired by this, we present efficient self-supervised vision Transformers (EsViT) with two major insights: multi-stage Transformer architecture with sparse self-attentions, and non-contrastive region-matching pre-training task. The synergy of both helps ViV reach the SoTA performance of SSL vision systems with significantly less compute and smaller model size. Our study also reveals that exploration of effective solutions to learn from larger and less curated pre-training data in the wild key but less studied factor in paving the way toward the scaling success of SSL vision systems.
ETHICS STATEMENT
Though self- self-supervised learning (SSL) has great potentials to learn powerful representation with- out human annotation, the existing techniques to build SoTA SSL vision systems tend to be Red AI (Schwartz al. 2020): it could be environmentally unfriendly and the computational cost is extensively high." The required training resource is typically not accessible for lab environment (thus raising barriers participation in Al research). For example, the prior art MoCo-v3 has greatly pushes the performance limit of SSL system (Chen et al. 2021). I. The authors kindly reported that "it (MoCo-v3, ViT-H) takes 9.8 hours per 100 epochs using 5T2 TPUs. This gigantic scale of training: for the 300-epoch ViT-H, this amounts to 625 TPU days, or 11.7 TPU years of training." The SoTA model MoCo-v3 with VYT-BN-L/7 should have a higher cost than this. Even for smaller model VTT-B, "it takes 24 hours in 1 28 GPUs (vs. 2.1 hours in 256 TPUs)". Hence, improving the efficiency of building SoTA SSL systems of high value for the community and society achieve Green AI (Schwartz et al. 2020).
To this end, we propose VIT to provide more affordable and efficient solutions for the community to experiment and explore the directions of SoTA SSL in computer vision. Our EsV model shows the best ImageNet linear probe performance compared with all existing SSL vision systems, and 3.5x crrrameter-fffcieen and has 10x higher throughput than previous SoTA. This efficiency gain can significantly decrease its carbon footprint and increase its inclusivity, encouraging more researchers participate the study of the SSL. topic.
REPRODUCIBILITY STATEMENT
Our paper provides comprehensive empirical studies on the EsViT algorithm. We provide PyTorch- style pseudo-code in Appendix. We also include example code with instruction supplementary material to ensure the reproducibility. For empirical results on both various network architecture and large-scale datasets, we provide detailed hyper-parameter specifications. We will release the pre-trane pre-trained checkpoints and codebase for the research community for reproducible research.
REFERENCES
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurlPS, 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In ECCV, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006. 09882, 2020.
10

---

Published as a conference paper at ICLR 2022
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chun- jing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. arXiv preprint arXiv:2012.00364. 2020a.
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever. Generative pretraining from pixels. ICML, 2020b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020c.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners, arXiv preprint arXiv:2006.10029 2020d.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 2020e.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised visual transformers. arXiv preprint arXiv:2104.022: 02005, 2021.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Learning universal image-text representations. arXiv preprint arXiv: 11740, 2019.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 2013.
Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. UP-DETR: Unsupervised pre-training for object detection with transformers. arXiv preprint arXiv:2011.09094 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACI, 2019.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In NeurIPS, 2019.
Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. T-PAMI, 2015.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthia Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Jonathan Frankle, David J Schwab, and Ari S Morcos. Training Batchnorm and only Batchnorm: On the expressive power of random features in CNNs. arXiv preprint arXiv:2003.00/55. 2020.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 2018.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in hour. arXiv preprint arXiv: 1706.0677 2017.

---

Published as a conference paper at ICLR 2022
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang. Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Self-supervised pretraining of visual features in the wild. arXiv preprint arXIv:2103.01988, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: new approach to self-supervised learning. arXiv preprins arXiv:2006.07733, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.0660 2018.
Karan Grewal, Phil Bachman, Adam
Xu Ji, João Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In ICCV, 2019.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision, 2020.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. ECCV, 2016.
Chunyuan Li, Xiujun Li, Lei Zhang, Baolin Peng, Mingyuan Zhou, and Jianfeng Gao. Self-supervised pre-training with hard examples improves visual representations. arXiv preprint arXiv:2012.13493, 2020a.
Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. Unicoder-VL: A universal encoder for vision and language by cross-modal pre-training. arXiv preprint arXiv: 1908.v::/9000000000 06066, 2019a.
Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.0490 2020b.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.0353: 2019b.
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv: 02862, 2017.
Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In CVPR, 2019c.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang. Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre- training for vision-language tasks. In ECCV, 2020c.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. arXiv preprint 1rXiv:2103.14030, 2021.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in Adam. arXiv preprint arXiv: 02677, 2018.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. VilBERT:
Pretraining task-agnostic visiolin- guistic representations for and-language tasks. NeurIPS. 2019.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invarian representations. In CVPR, pp. 6707-6717, 2020.
12

---

Published as a conference paper at ICLR 2022
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXii::807.037444 2018.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, 2018.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
Yunchen Pu, Zhe Gan, Ricardo Henao, Xin Yuan, Chunyuan Li, Andrew Stevens, and Lawrence Carin. Variational autoencoder for deep learning of images, labels and captions. NIPS, 2016.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. OpenAl Blog, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2 00020, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092. 2021.
Roy Schwartz, Jesse Dodge, Noah Smith, and Oren Etzioni. Green AI. Communications of the ACM, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:I4099 1556, 2014.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VL.-BERT: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv: /9t:::908..3000 2019.
Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from transformers. EMNLP, 2019.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint 1024: 2020.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lueas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung. Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. MLP-mixer: An all-MLP architectre for vision. arXiv preprint arXiv:2105.0166 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012. 12877, 2020.
Trieu Trinh, Minh- Minh-Thang Luong, and Quoc Le. Selfie: Self-supervised pretraining for image embedding. arXiv preprint arxiv:/906.0090 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. CVPR, 2021.
Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chin Chieh Chen. Max deeplac End-to-end panoptic segmentation with mask transformers. arXiv preprint arX7v:2012.00759, 2020a.
Wenhai Wang. Enze Xie, Xiang Li. Deng Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer. A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102. 2021.
13

---

Published as a conference paper at ICLR 2022
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training arXiv preprint arXiv:2011.09157, 2020b.
Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. arXiv preprint arXiv:2011.14503. 2020c.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103. /5808, 2021.
Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu. Zhenguo Li, and Ping Luo. Detco: Unsupervised contrastive learning for object detection. arXiv preprint arXiv:2 102.04803, 2021a.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In ICML, 2016.
Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao, and Han Hu. Self sulf-supe 2021b.
learning with swin transformers. arXiv preprint
Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021c
Yuwen Xiong, Mengye Ren, and Raquel Urtasun. Loco: Local contrastive representation learning. arXiv preprint 0rXiv:2008.03342, 2020.
Ceyuan Yang. Zhirong Wu, Bolei Zhou, and Stephen Lin. Instance localization for self-supervised detection pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. Learning texture transformer network for image super-resolution. In CVPR, 2020.
Jianwei Yang, Devi Parikh, and Dhruv Batra, Joint unsupervised learning of deep representations and image clusters. In CVPR, 2016.
Tian Yonglong, Olivier J. Henaff, and Aaron van den Oord. Divide and contrast: Self-supervised learning from uncurated data, arXiv preprint arXiv:2105.0l 08054, 2021.
Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Online deep clustering for unsupervised representation learning. In CVPR, pp. 6688 063666667728 2020.
Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. arXiv preprint arXiv:2103.15358 2021.
Richard Zhang, Phillip Isola, and Alexei Efros. Colorful image colorization. ECCV, 2016.
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In CVPR, 2017.
Minghang Zheng, Peng Gao, Xiaogang Wang, Hongsheng Li, and Hao Dong. End-to-end object detection with adaptive clustering transformer. arXiv preprint arXiv:2011.09311 2020.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and VQA. AAAI, 2020.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In CVPR, 2019.
14

---

Published as a conference paper at ICLR 2022
A METHODS
A.I ALGORITIMS
We summarize the training algorithm procedure EsVIT with Lv+ in Algorithm [
To clearly
outline the main idea of the algorithm, we show the algorithm for two augmented views. For the full algorithm to deal with multi-crop, please refer to our codebase. Algorithm [ for mini-batch of size the teacher/student network consists of three output variables: (1) E R is the probability vector for the view-level representation. output by an MLP head. (2) z R=XTxP is the feature
map, containing 7' region-level features of dimension P. (3) pz [2xX'x probability vectors of 2, output by a different MLP head.
Algorithm 1: EsViT with Cn. pseudocode with 2-crop.
ga, gt: student and teacher networks
Cv, Cr: view and region center (K)
tmp_s, tmp_t: student and teacher temperatures
a, network and center momentum rates.
n: batch size, K: pue-heed-preeected probability vector length,
T: last layer feature map length, F: last layer feature vector length
gt. = garmars
The main training loop
for x in loader:
x1, x2 augment (x), augment (x) # two random views
student cutput, p:nxk, p2:nXTXK, 2IXXXXP p s1,p7 81, gs g (x1)
p2.2, 2..2 (x2)
# teacher output, pinxK, PZ:EXTXK, ZIEXIXP p.tl.pz.t1,..t11 gt(x)) (x1)
p.t2,pz.t2, 2.t2 gt (x2)
view-level loss
Hv (p.ss, p.t2) /2 Hv p.t1)
regicn-level loss
11
loss Hr (pz s1, pz t2, z.sl, z 2 Hr (pz pz. z.s2, z.11)/2 loss = 1os10ss..00 v 1/2
12
13
lorswbackward back-propagate
update student, teacher and centers update (gs)
15
AdamW for student
16
gt params = gt paraaaaasss..... + (1 a) * params
EMA for teacher
Cv bx Cv + (1 - b) cat ([p.t1,p.t2 (0)a
EMA for view center
Cr b+
cat (lpz.t1,pz.er .eean (0))
EMA for region center
19
The view-level loss function def Hv (s, t)
21
detecch stop gradient
22
s softmax tmp_s, dim=-i)
23
t softmax( Cv) / tmput, dim--1)
return (t log (s)) sum (dim=-1). mean
The region-level loss function def Hr (ps, pt, zs, zt):
25
26
pt (ctach(
# stop gradient пxTXK
ps softmax (ps / tmp-s, dim=-1)
28
pt softmax mtw.ot (zs zt zt .er.utee (0, 2, 1))
Cr) tmp_t, dim=-1)
пXIXK
29
simunatrix torch. matmul
# DXTXT
30
sim_idx = si aeimmmattiii w...mttrix.maxxdd (dim=-1) :.. (2)
nX7x1
pt idxed torch. gather (pt, 1, sim idx expand (-1, -1, pt. size (2)))
return (pt idxed log (ps) sum (dim=-1) mean
15

---

Published as a conference paper at ICLR 2022
A.2 NETWORK ARCHITECTURE CONHIGURATIONS AND IMPLEMENTATION DETAILS
Inspired by great successes of the multi-stage ConvNet architecture such as VGG (Simonyan & Zisserman 2014 2ess (He et al. 2016) for computer vision, the multi-stage Transformer-bass
networks have been explored very recently in the supervised learning setting (Vaswani al. 202T
Wang et al. 2021 Liu et al., 2021; Zhang et al., 2021; Wu et al. (02I). In multi-stage vision Transformers, since a larger number of patches is often produced at the early stages, an efficient Transformer with sparse self-attentions is considered to reduce the computational complexity. The basic idea is split the feature maps into non-over con-overaapping local windows (with size W x W), and self-attention is performed within each local window. This however has one drawback that features in different local windows cannot interact. Various methods have been proposed to best approximate full-attention, with different trade-off between accuracy and efficiency.
We briefly describe three schemes as follows, and benchmark them in the experiments. ()) Swin Transformer (Liu et al. 2021): A shifted window partitioning approach is proposed, which alternates between two partitioning configurations in consecutive Transformer blocks, so that each local feature is grouped into different windows in self-attentions.
(ii) Vision Longformer (ViL) (Zhang et al. 2021): Features in each local window are further allowed to attend all features in the 8-neighboring windows. (ii) Convolution vision Transformer (CvT) (Wu et al. 2021): Features in neighboring windows are considered in the convolutional projection in self-attentions.
The window size is set to W by default. The query dimension of each head in self-attentions is d = 32, and the hidden layer width of each MLP is 4x of its input's 's width, for all experiments. The architecture configurations of model variants employed in the experiments are summarized in Table Some notable implementation detailed are described as follows:
The three configurations Swin-T, Swin-S and Swin-B indicate Tiny, Small, and Base models, respectively, which are almost identical to the original implementation (Liu et al. 2021). except that we add special treatments to deal with input augmented views of different resolutions, when the resolution (feature map size more specifically) is not divisible by the window size (i.e., resolution 96 and window size=7 or 14).
Swin-T and Swin-S with window size W= 14 are customized by us to allow full self-attention in stage 3 (where the majority of model capacity is allocated to) and stage 4, to study the impac of longer sequences in VIT.
In the original ViL (Zhang et al. 2021) and CvT (Wu et al. 2021) papers, different positional embedding strategies and multi-stage network configurations were employed. We modify them by only utilizing relative position bias and their proposed sparse self-attention mechanisms, and create a similar 4 stage architecture with Swin-T for fair comparison.
Relative Position Bias. To facilitate SSL, we consider relative position bias (Liu al. 2021 to characterize the spatial information between features for the three efficient Transformers aforemen- tioned, and do not use absolute position embeddings. This is because augmented views of varied resolutions can be cropped from anywhere in an image in SSL, maintaining the relative positions is easy in implementation, and is largely sufficient for invariance learning among these views.
B RELATED WORK
Self-supervised ConvNets.
Conv-Ness-based SSL has been extensively studied in the literature.
Based on the pre-training tasks, they can be broadly categorized into three classes: Handcrafted
pretext tasks (Doersch et al.] Noroozi & Favaro 2016, Pathak et 2016, Gidaris et al., 20188)
Zhang et al. [ Larsson et al. |016] /Zhang et al.,
2017
Pu et al. al.] |
Donahue & Simonyan
2019). contrastive learning (Dosovitskiy et al. 2015 Zhuang et al. 2019] Oord et al. 20118] Hjelm
ct al. 2018 Bachman et al. 2019 He et al. 2020) Chen et al.
2020c
Grill et al. 2020 and prototype
2016 Yang et al. 2016 Ji et al. 2019) learning (Caron et al.) 20188 2020 Li et al. 2020b Xie et al.
Zhan et al. (220)). It is also known that data augmentations play a crucial role in SSL pipeline (Chen
al. [2020e] Caron et al., 2020 Tian et al. 2020; Li et al. 2020a). The impact of pre-training dataset
size/quality is explored for ConvNets in SSL (Goya et al. [202T] Yonglong et al. (021). To date, the search of best pre-taining tasks/datasets and augmentations based CNNs. Among them,
16

---

Published as a conference paper at ICLR 2022
Cross-Veew Prediction
MLP Head
Cross-View View Prediction
MLP Head
Transformer
Transformer (Sparse S.A.)
Patch Merging
Rapeated
Top-layer feature
Top-layer fexture
Input augmented views
Input augmented views
Transformer (Sparse S.A.)
Transformer
Patch Merging
000000000 000000000
Vieyi
(a) Baseline monolithic architecture
(b) Proposed multi-stage architecture
Figure 7: Architecture comparison. (a) The monolithic transformer. For all layers, the transformer blocks share the same network configurations and input token sequence sizes are the same. (b) The multi-stage Transformer organizes an input image into long sequence of smaller patches, sparse self-attentions (S.A.) are utilized at early stages to maintain model expressiveness while reducing computational complexity; The neighboring tokens at an intermediate layer are gradually merged, constituting a short sequence to ease the compute burden of self-attention at late stages.
Stage
Stage 2
Stage
Stage 4
Merging Rate Feature Map
4x
8x 28
16x 14
32*
56
7
concat 96-6.
concal x 2, 192-d, 1N
concat 384--.
concatt 2 2. 768-1, INN
Swin T. W=
window size :7 x 96-d heads)
window size 7 7
window size 7
window size : x 768-J (24 heads)
x2
192-d (6 heudh)
x2
364-J (12 heals)
x2
concet 96-d, LN
concat x 2.
LN
concer x 2,3814
concat 2 x 768-4, LN
Swin S. W= 7
window sine 7 x 96-d heads)
window size : xx77
window size: x
window sine : 7
x-
192-d (G heads)
x2
384-d (12 heads)
x18
708-d (24 heads)
x22
conces 4. 125-d.
concat 2, 256-d. 1N
concet x 215-24
concat 2, 1024-d.
Swin-B, W=
window size: 7 x
window size 7
window size
window sine 7 x
128-4 (4 hendk)
x-2
256-d (8 heads)
x2
512-4 (16 heads)
x18
1024-d (32 heads)
x22
concat window size: 14 96-d (3 heads)
concut x 192-d,LN
concas 2.384-d,
concat 2 2, 708-0. LN
Swin-T. W==l =14
window size 14 14
window size: 14 14
window size 7
x2
192-d (6 heads)
x2
384-d (12 heads)
x6
708-d (24 heads
x2
concat 96-d. LN
concal 2, 192-4, INN
comeat 2,384-d
concat 2 768-4, IN
Swin-S, W- 14
window size: 14 96-d hcads)
window size: 14 x 14
window sine: 14 x 14 384-d (12 heads)
window size x 708-4 (24 heads)
x2
192-d (G heads)
x2
x18
x2
comeat 95-d, UN
concall 2 2, 192-d,
voncat 384-d.
concat 2 window size xxx 768-4 (12 heads)
LN
VIL-T, W
window size 7 x 96-d (3 heads)
x2
window size: : 7
192-d (3 heads)
x2
window size: : 7 x
354-d (i) heads)
x6
x2
4, 64-d, LN
192-4, LN
384-4. LN
concat 768-4, LN window size 7 765-4 (12 heads)
window size 7 x 64-d (1 head)
x2
window size 7 7 192-d (3 heads)
x2
window size: 7
354-4 (6 hoads)
x6
x2
Table 6: Model configurations considered in our experiments.
SimCLR-v2 (Chen et al.) 2020d), BYOL (Grill et al., 2020) and SwAV (Caron al. 2020) achieve the highest ImageNet linear probe performance with large ConvNet architectures. The performance tends to saturate with an increasingly growing model size, raising question if ConvNets reach a limit in SSL.
Transformers for vision. Vision Transformers (ViT) (Dosovitskiy et al., 2021 shows the great potentials of generalizing Transformers for computer vision, by achieving compelling accuracy in supervised learning, especially with large-scale data and high capacity models. DeiT Touvron et al.) 2020) further provides an effective ViT training strategy to ease the adaption of Transformers for practitioners. Transformers have also been applied to other vision tasks, ranging from low-level
tasks such as image generation Parmar et al. 2018: Chen et al. 20206) and enhancement (Chen
et al. 2020al Yang al. 2020), high-level tasks such object detection (Carion et al. 2020
Zhu et al.][202)) ( et al. 2020 Dai et al. 2020) and segmentation (Wang et al. 2020a and
to vision-language tasks (Lu et al. 2019 lan & Bansal 2019 Chen et al. 2019 Su et al.
2019
17

---

Published as a conference paper at ICLR 2022
Name
Framework
Pro-train Task Description
Major Motivation
Major Downstream
Task
ImageNet
Performance
PIRL
Contrastive
Jigsaw preteat task encourages image representations to be invari- ant to image patch pertur-
General purpose visual backhone Mostly classification.
Image classilicaiion
63.6%
learning image
for
bation
DenseCL
Contrastive
pairwise contrastive (dis)iiilarity loss at the twa
Improving visual prediction task perfor-
Improving object detection performance.
63.6%
patch level queue of negutive sample leatures from other images maintained
DetCo
Imppewving prediction task perfor- mance
Impruving object tion performance. DetCo achieves trade-off on both classffiatiion and detection.
three contrastive tasks he- tween global images local patchess considered: globa,, global local. local-local.
perfor-
PixPro
Contrastive
Features from two views are encouraged sistent between a regular patch representation and smoothed patch representa- within same image.
Improving visual prediction task perfor- mance
Mostly focusing the 66.3% proved performancc de- lection lasks.
InstLoc
Contrastive
image instances are pasted various locations and scales onto hackground im- ages. The pretext task to predict the instance cat- egury given composited images and the foregmund bounding
Improving visual prediction task perfor- mance
Mostly focusing proved performance de- tection tasks.
61.7%
EeVT (aurs)
Non- pairwise cross entrpy the patch level be- tween two positive views. need/intemction other images in the batch (eg, no negative samples)
Recovering the muto- matic correspondence learning property of self supervisdd mono- lithic transformers, and thus improving learning efficiency.
Consistently improving image classification tasks. It creates new SoTA 81.3%
69.9
ImageNet linear probe accuracy. showing 3.5x parameter-efllcien and 10x higher throughput than previous SaTA MoCo-v3. Reporting 75.7% ImageNet linear prohe performance ResNet-50.
Table 7: Discussion of related works on various region-level tasks. The last columns reports the ImageNet linear probe performance for ResNet-50 trained with augmented views for 200 epochs.
Li et al. 2019b a Zhou al. 2020 Li al., 2020c) Marrying Transformers with multi-stage architectures (Vaswani et al. 2021 Wang et al. 2021 Liu et al. [021 Zhang et al.) [021 Wu al. 202I> show higher classification accuracy in supervised learning, and enables applicability of Transformers for broader range of vision tasks, Given these properties, we believe multi-stage ViT a must-study baseline for SSL in computer vision.
Discussion with other region-level tasks. In Table 7 we compare Cn against the existing region- level tasks, including PIRL (Misra & Maaten 2020). DenseCI. (Wang et al., 2020b), DetCo (Xie) et al. (021a), InstLoc (Yang et al. 2021). PixPro (Xie et al. (021c). Most of these region-level tasks improve object detection tasks, but hurt the ImageNet classification accuracy. DetCo achieves the best trade-off: improving the performance of both tasks, but with a sophisticated multi-level, global-local interaction algorithm. With the same number pre-training epochs and augmented views, ViT achieve the best ImageNet linear probe accuracy among all region- tasks, with as minimum computational overhead as possible. This well serves our goal building efficient SSL SoTA image classification system.
18

---

Published as a conference paper at ICLR 2022
Data
Table
2
Table 3 in MLP-Mixer (Tol
Table in Swin Liu
Our
DINO
Caron
stikhin al. 2021
al 2021
ct al.. 2021
DeīT-S/ P=
1007
940.4
DeiT-B/P
312
292.3
DeiT-S/P=9
180
DeiT-B/P! P
63
VIT-B / 16
312
861
VIT-S/ 1=3/P=10 -6
102
280
V P=14 32
87
ViT-L/
17
47f
Swin-T/WK W =
808
755.2
726.13
Swin-S/W 7
467
436.9
Swin-B/ =
297
278.1
Swin-T/ 11
660
593.24
Swin-S/ W = 14
383
344.20
Swin-B /B/I W' 14
254
228.36
ViI.T/ B//W=/ =7
386
346.72
CVT-T/W WV 7 848
761.89
Table 8: Throughput estimate and standardization. All numbers in orange are estimated/converted, while numbers in blue collected from the papers, and numbers in green are runs on our machines. All papers report the throughput of ViT-B or DeÎT-B, which are essentially the same model. We use this fact to align the throughput reported in different papers. This number estimated via the statement in (Chen et al. 202T) that "reducing the patch size to 7 7 keeps the model size unchanged. increases FLOPs to 6x". All numbers are standardized into throughput reported by (Caron et al. 2021.
C EXPERIMENTS
C.I THROUGHPUT ESTIMATE AND CONVERSION
Since different papers report throughput different hardwares, it is not ready compare the numbers directly. Noting that all papers report the throughput for ViT-B/DeiT-B, we use this number to align and convert the throughput. In Table g we describe our process and results of standardizing the throughput.
C.2 THE COMPUTATION AND MEMORY OVERHEAD OF PROPOSED LR
We emphasize that adding Ln the multi-stage transformer architectures yields acceptable extra computational cost, while adding LR directly to the monolithic transformer architectures has huge computational overhead. To demonstrate this, we report the comparisons Table For each setting, we report [Memory Usage (MB) / Running time per iteration (second/iteration) In Table Table | (a), when the batch size is gradually increased (e.g., batch-size= 12), the memory cost increases nearly 4 times for monolithic architectures, while increases 1.6 times for multi-stage architectures. Similar trends are shown for training cost per iteration increase ratio (1.47 vs 1.15). This indicates Lg can more naturally fit multi- architectures.
Similarly, we compare computational cost comparisons [Memory Usage (MB) Running time per iteration (sccond/iteration)] in Table (b), for other network architecture configurations. From the increased cost ratio, we see that LR adds acceptable cost in terms of both memory and training time, compared with the baseline.
C.3 EXPERIMENTAL SETTINGS OF PRE-TRAINING AND EVALUATION ON IMAGENET
We study unsupervised pre-training performed in ImageNet- IK dataset (Deng et al. 2009) without labels. The default training details are described follows, mostly following (Caron et al. 2021). We train with the Adamw optimizer (Loshchilov Hutter 2018). a batch size of 512, and total epochs 300. Linear warmup of the learning is used during the first 10 epochs, with its base value
19

---

Published as a conference paper at ICLR 2022
Batch Mize-Siize=
Batch-Sizess
Batch-Size=12
VIT-S VIT-S Increased Cost Ratio
Lv
1391 / 0. 0.209036
2735 00.33999
3533 0.238 160
Monolithic
Ly
2641 / 0.233150
10358 / 0.322155
14128 0.352339
1.8986 / 1.1153
3.7872 1,3666
3,9988 1.4794
Swin-T Swin-T T
Lv CV+LR
1704 0.285451
4229 0.323884
$611 0.366367
Multi-stage
2346 0.301672
6232 0.374634
8917 0.421135
Increased Cost Ratio
1.3767 1.0568
1.4736 1.2323
1.5889 1.1494
(a) Comparisons for increased batch sizes.
EsVIT
Cv
Cy L
Increased Cost Ratio
Tiny (W=7)
1704 0,285451
2346 / 0.301672
1.3767 1.0568
Small (W=7)
2685/ 0.501876
3132 / 0.535203
1.1664 1.0664
Base (W=7)
3726 0.516558
4374 0.550617
1.1739 1.0669
Tiny (W=14)
2159 0.288118
2801 / 0.310108
1.0890 1.0763
Small (W=14)
3518 0,496823
4153 0.522739
1.1805 1.0501
Base (W=14)
5032 0,511701
5681 0.537826
1.1289 1.0510
(b) Comparisons for various network architecture configurations.
Table 9: Computational cost comparisons in the format of [Memory Usage (MB) / Running time per
iteration (second/iteration)]
Dataset
Classes
Train size
Test size
Evaluation metric
Source link
Food-101
102
75,750
25,250
Accuracy
Tensorflow
CIFAR-10
10
50,000
10,000
Accuracy
cosorrllow
CIFAR-100
100
50,000
10,000
Aecuracy
lensorFlow
SUN397
397
19,850
19,850
Accuracy
Tensorllow
Stanford Cars
196
8,144
8,041
Accuracy
Stantold Cars
FGVC Aircraft (variants)
100
6,667
3,333
Mcan-per-olass
FGVC website
VOC2007 classification
20
5,011
4,952
1l-point mAP
vOC2007
Describable Textures
47
3,760
1,880
Aecuracy
Tensorl
Oxford-IIIT Pets
37
3,680
3,669
Mcan-per-chee
Oxtord-I1l1 Pot
Caltech-101
102
3,060
6084
Mean-per-class
Tensorl
Oxford Flowers 102
2,040
6,149
Mean-per-class
Tensorl low
MNIST Facial Emotion Recog. 2013
10
60,000
10,000
Accuracy
LensorFlow
32,298
3,589
Aecuracy
Kaggle ler2013
STL.10
10
5,000
8,000
Aecuracy
lcnsorFlow
GTSRB PatchCamelyon UCF101
43
26,728
12,630
Accuracy
GISKB website
294,912
32,768
Accuracy
lensorllow
101
9,537
3783
Accuracy
IcnsorFlow
Hateful Memes
2
8,500
500
ROC-AUC
FaceBook
Table 10: A suite of 18 datasets used in linear probe." * indicates dataset whose train/test size obtained is slightly different from Table 9 in (Radford et al. 2021
determined with the linear scaling rule (Goyal et 2017): br 0.0005 * batchsize/256 After this warmup, the learning rate is decayed with a cosine schedule. We build our systems based on Swin Transformers (L.uu et al. 202T) our experiments. Swin-B has a model size and computation (patch size 16). We also considered Swin-T and Swin-S, which
complexity similar to ViT-B/Deil have the complexity that are similar to those ResNet-50 (DeiT-S) and ResNet-101, respectively. The default window size is W 77
One major common protocol to evaluate SSL is linear probe on ImageNet- IK, where features are extracted from a frozen backbone, and a supervised linear classifier is trained. For all Transformer models, we use the concatenation of view-level features 2 in the last layers (the results are similar to the use of last 3 or 5 layers in initial experiments).
20

---

Published as a conference paper at ICLR 2022
CLIP ResNet-50
Supervised
Supervised+ ResNet-50
Supervised Swin-T
EsVIT
Methods
ResNet-50
Swin-T
Food-101
86.4
71.3
71.3
77.4
80.0
CIFAR-10
88.7 7
91.8
91.8
94.0
95.3
CIFAR-100
70.3
74.5
74.5
77.5
82.2
SUN397
73.3
60.5
60.3
64.3
67.6
Stanford Cars
78.3
49.9
50.1
55.3
66.4
FGVC Aircraft (variants)
49.1
48,5
48.4
51.5
61.1
VOC2007 classification
87.1
83.8
83.6
84.2
85.5
Describable Textures
76.4
72.3
72.6
73.1
78.1
Oxford-I1ITT Pets Caltech-101
88.2
92.4
92.1
93.3
89.6
90.8
90.4
90.8
93.0
Oxford Flowers 102
96.1
90.8
91.1
91.5
97.4
MNIST Facial Emotion Recog. 2013
98.3
98.3
98.3
98.3
98.3
64.2
54.9
55.9
55.1
59.3
STL10
97.2
96.4
97.0
97.9
98.9
GTSRB PatchCamelyon UCF101
82.4
70.6
75.7
72.9
82.7
82.5
82.6
84.0
84.6
81.6
71.2
72.1
79.0
81.1
Hateful Memes
65.7
56.5
49.9
51.2
52.0
Average
80.86
75.39
75.43
77.29
80.99
Table 11: The linear probe results on 18 datasets at the scale of ResNet-50/Swin-T. indicates the
results reproduced by us, which verifies that our implementation pipeline is consistent with (Radford al. 2021
Method
View-level
Region-level
Top-1 Accuracy (%)
Performance comparison of ResNet-50 with 200 epochs and 2 augmented views Contrastive Contrastive
MoCo-v2
67.5
DenseCL
Contrastive Contrastive
63.6
DetCo
Contrastive
68.6
DINO
Non-Contrastive
69.2
ESVIT
Non-Contrastive
Non Contrastive
69.9
SoTA performance comparison of ResNet-50 with numbers and settings reported each paper McCo-v2 (800 epochs)
Contrastive
72.2
SwAV (800 epochs, w/ multi-crop)
Contrastive
75.3
Barlow Twins (1000 epochs)
73.2
VICReg (1000 epochs)
73.2
SimSiam (800 epochs, 2 views)
Non-Contrastive
71.3
multi-crop) BYOL (1000 epochs,
Con-Contrastive
74.3
DINO (300 epochs, w/ multi multi-crop)
Non-Contrastiog
75.0
EsVIT (300 epochs, w/ multi-crop)
Con-Contrastive
Non-Contrastive
75.7
Table 12: Linear probe performance ResNet-50 network with different SSL methods.
C.4 COMPARISON WITH A RESNET- 50 BACKBONE
To compare VST learning method with other SSL algorithms, conduct experiments with ResNet-50 backbone, and show the results in Table [2
C.5 LINEAR PROBE ON A SUITE OF SMALL DATASETS
Datasets. Table 10 shows details and source of all datasets used for linear probe, including the
number of classes, the size of training set and testing set, metrics used in evaluation, as well as a public source of the dataset. Note that original UCF101 ICF101 dataset is video dataset. Here the middle frame of each video is extracted to form classification dataset. There are 3 train/val splits in Tensorflow, we the first one.
21

---

Published as a conference paper at ICLR 2022
Automatic hyper-parameter tuning. We rigorously follow (Radford et al. 2021 training and evaluation for linear probe on the downstream datasets. We train a logistic regression classifier using scikit-learn's L-BFGS implementation, with maximum 1, 000 iterations, and report the corresponding metric for each dataset. We determine the L2 regularization strength À using hyperparameter sweep on the validation sets over the range between 10-6 and 10 with 96 logarithmically spaced steps. To save compute required for the sweeps, we perform a parametric binary search that starts with = [10 6, 10- 10 2, 1, 102 10 10 and iteratively halves the interval around the peak until it reaches a resolution of 8 steps per decade. The hyperparameter sweeps are performed on a validation split of each dataset. For the datasets that contain a validation split in addition to test split, we use the provided validation set to perform the hyperparameter search, and for the datasets that do not provide a validation split or have not published labels for the test data, we split the training dataset to perform the hyperparameter search. For the final result, we combine the validation split back with the training split and report the performance on the unused split.
conduct
Detailed results. Only the last layer feature is considered for all models for simplicity, though adding features from more layers may potentially improve the results. Table [1 shows the results for architectures at similar scale of ResNet-50 or Swin-T. The first two columns are numbers from (Radford et al. 202T). CLIP with ResNet-50 is pre-trained on 400 million image-text pairs. Supervised ResNet-5t 30 and Swin-T are pre-trained on ImageNet- 1 on which VVV with Swin-TT is pre-trained as well (Batch Size=512). VSV outperforms its supervised counterpart, and is on par with the performance of CLIP in a similar image encoder architecture scale.
C.6 PRE-TRAINING DATASETS
We describe the statistics and training schedule on larger and less curated datasets in Table 13 The pre- tre-training epochs are chosen so that the model is trained with similar number of augmented views.
Name
Description
Size (#Images)
Epochs
Warmap
ImugeNet-1K
Images evenly distributed IK object concepts
1.2 million
WobVision-vi Li ac 2017
Web images with IK concept queries from ImageNet-IK
2.4 million
150
5
Opentmages-vi Koznetova 2020
Diverselcomples scenes with several objects for detection
7.5 million
50
2
ImageNet-22K
Dang c at. 2009
Images distributed in 22K object concepts in a hicrarchy
14.2 million
30
Table 13: Pre-train dataset statistics and training schedule.
C.7 RESULTS ON CORRESPONDENCE LEARNING
We first quantitatively evaluate the correspondence learning results with 50K images in the ImageNet validation dataset. We create a simple evaluation dataset with mild augmentations. For a center-crop crop image, apply
then Color and RandomGrayscal a new augmented view. In this way, ground-truth correspondences are created. Please see the 1st row of Figure 9 for one such example. The top-10 correspondences are used for evaluation. Two metrics are considered: (1) Accuracy measures the percentage of correctly matched region pairs, (2) distance error indicates the averaged l distance between the predicted matched region and ground truth region (the value is 0 for perfect matching). The results are reported in Figure [ DINO with monolithic Transformers shows surprisingly good performance on correspondence learning. The use of multi-stage Transformer architecture reduces this ability, shows a lack of good region correspondence. With LR, the region matching ability is significantly recovered.
to create
In Figure 9 we visualize the correspondences for more images. Overall, DINO with monolithic Transformers is able to discover most salient correspondences of semantic meaning in the mild augmentation conditions, even without an implicit region matching loss in training. We believe this previously underestimated property is whole-noting, and has potentials to enable more applications. However, this desired property gets dilated when changing from monolithic to multi-stage Transformer architecture (from column to column 2), then the proposed region level task can alleviate this issue (from column 2 to column 3).
To more specifically analyze the correspondences, we note the following results. The first row shows simple case, where only images of left-to-right flipped views are presented. The ground-truth
22

---

Published as a conference paper at ICLR 2022
49.9
1.0
0.91
50
Distance error
Accuracy
0.66
0.6
20
11.9
10
EsVit: Ly LR
EsVit: Ly
EsVit: Ly La
EsVit: Lv
(a) Accuracy
(b) Distance error
Figure 8: Quantitative evaluation on correspondence learning on ImageNet validation set. L can significantly improve correspondence learning quality for multi-stage architectures. As reference, DINO (Lv with monolithic Transformer architecture) achieves 0.95 accuracy and 2.49 distance error, which believe is strong evidence to identify the intriguing property of automatic correspondence learning.
correspondences should be horizontal lines that link the two flipped regions. It reveals that the view-level pre-train task alone is insufficient to learn good correspondences for the multi-stage Transformer architecture, while region matching task can alleviate this issue significantly. Similar observations shown in row 3 and row 4.
We further study more cases that requires real-world correspondences row 2, row and row 6. These views are not generated with data augmentation (as in model pre-training), but are often presented in more practical scenarios: one-to- many mappings, cartoon-to-toy seasonal changing of the scene, respectively. The proposed region matching task work particularly well in those cases.
C.8 MORE VISUALIZATION RESULTS ATTENTION MAPS
We visualize attention maps the top layer in Figure I0 12 With monolithic Transformer architecture. DINO can automatically identify the main foreground objects. Unfortunately, changing from monolithic to the multi-stage Transformer architecture (From left column middle column), this property gets lost. There are more heads in the multi-stage architecture than monolithic architecture (24 heads vs 6 heads in this case) in the last year. A fair number of heads in Vs shows redundant patterns, this issue can be reduced when the region-level matching task is added (From middle column to right column).
observed that DINO with monolithic Transformer architecture only learns to attend the fore- ground objects, even when the query is background region (see Figure 12p. This is perhaps because DINO models are trained to learn view-level invariance, the main objects in the pre-train dataset ImageNet tend to be the principle factor that remains invariant across different augmented views. Hence, all backgrounds are ignored, regardless of the query positions. This improved in EsViT with the region-level pre-train task, as the model trained match individual regions.
DINO shows high entropy values in all of heads (perhaps a required condition to cover all regions of the main object). In VST, LR plays interesting role in modulating the entropy distributions among heads: it increases those with larger entropy values, while decreasing those with lower entropy values. another word, makes the attention patterns in different heads more diverse.
23

---

Published as a conference paper at ICLR 2022
(a) DINO: DeïT-S
(b) ESVIT: Lv
(c) ESVIT: Lv CR
Figure 9: The learned correspondences. Yellow lines the top- 10 correspondences between two views, where the numbers indicates the rankings of similarity scores, yellow dots with the same number are paired. The blue dot and red triangle indicates the most similar local regions that correspond to the global feature of the view itself and the other view, respectively. Please zoom in for detailed correspondence mappings.
24

---

Published as a conference paper at ICLR 2022
by
Entropy
10
15
20
Heads self-attention (ordered entropy)
(a) DINO: DeiT-S
(b) ESVIT: Ly
(c) ESVIT: Ly CLR
Figure 10: The learned attention maps for all heads at the top layer, ranked by the entropy of softmax probability. Query is the blue dot the top-left of the image. Top: Entropy each heads. Middle: top 60% probability mass. Bottom: full attention maps. LR shows more attention patterns than Lv only.
25

---

Published as a conference paper at ICLR 2022
EsVit: Ly
EsVit: Ly
Entropy
10
Heads in self-attention (ordered by entropy)
(a) DINO: DeiT-S
(b) EsVIT: Lv
(c) EsVIT: Ly + Cr
Figure 11: The learned attention maps for all heads at the top layer, ranked by the entropy of softmax probability. Query is the blue dot in the center of the image. Top: Entropy of each heads. Middle: top 60% probability mass. Bottom: full attention maps. La shows more attention patterns than Ly only.
26

---

Published as a conference paper at ICLR 2022
by
Entropy
10
15
20
Heads self-attention (ordered entropy)
(a) DINO: DeiT-S
(b) ESVIT: Lv
(c) ESVIT: Ly + Cr
Figure 12: The learned attention maps for all heads at the top layer, ranked by the entropy of softmax probability. Query is the blue dot the top-left of the image. Top: Entropy of each heads. Middle: top 60% probability mass. Bottom: full attention maps. DINO mainly attends the main object even when the query is background region.
27


Reviews:
[{"review_id": "lqXz2Br67h7", "reviewer": "Reviewer_nCrU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Weaknesses:\n1) I find the contribution, in the form of the loss term, incremental with respect to Swin, and the performance gain, demonstrated in most experiments, modest.\n2) In my opinion, Figure 1 and Table 1 misrepresent the performance gain that stems from the contribution of the paper. The baseline architecture (Swin-B and Swin-B/W=14) already outperforms almost all methods in the comparison at a significantly lower number of parameters and much higher throughput. (The corresponding numbers are reported in Tab 5.) \\\nI encourage the authors to modify Figure 1 and Table 1 to represent the increase in performance of Swin resulting from the use of the proposed loss term. This can be done by introducing to both parts of Figure 1 points representing the performance of the base architectures (Swin-B, -S, and -T), trained without $\\mathcal{L}_R$. The performance gain due to training them with $\\mathcal{L}_R$ can be indicated in the plot with vertical arrows. Table 1 can be modified by reporting the performance of the base architectures trained without the proposed loss, in addition to their performance when trained with this loss. (Basically, by merging Tab 5 into Tab 1. The \"ablation\" results can still be discussed in a separate section.)\n3) In the current transfer learning results, the impact of the authors' contribution on transferability of the trained networks is not clear. I would prefer the experiments in learning to classify 18 small datasets to highlight the performance gain/loss that stems from training with $\\mathcal{L}_R$, in addition to the current comparison of supervised to unsupervised training. This would imply adding a third bar color to Figure 3 and one more row to both parts of Table 2, to represent the performance of Swin trained in a self-supervised regime, but without $\\mathcal{L}_R$. \n\nStrengths:\n* A. I like the story-line of the paper, where an interesting observation about the discriminative (matching) power of patch descriptors motivates the construction of the loss term.\n* B. The proposed loss term is shown to improve both the performance of Swin and of other architectures.\n* C. The experimental evaluation is extensive, in that it also shows limitations of the proposed loss, for example, when applied to image segmentation (Tab 2 and Tab 6).\n* D. According to me, the manuscript is well written and easy to understand.\n\nMinor questions and editorial suggestions:\n* Q1: in the loss term (2), you select $j^*$ by the cosine distance, but you minimize cross entropy. Have you tried using the same score for both tasks, i.e., selecting $j^*$ with the cross-entropy and minimizing cross entropy, or selecting $j^*$ by the cosine distance and minimizing the cosine distance?\n* Q2: in the loss term (2), why haven't you tried optimal 1-1 matching (the Hungarian algorithm) to match source and target patches? \n* Q3: in the paragraph \"design choices of $\\mathcal{L}_R$\", you describe the choice of \"argmax\" vs \"Optimal Transport\" in the loss term (2). I do not understand how OT can be used to select the optimal index $j^*$. Could you clarify this?  \n* ES1. page 2: \"loss of this property\", \"to alleviate this issue\": Perhaps use enumeration instead of bullets and make the statements more specific: loss of the property described in 1, alleviate the issue identified in point 2.\n* ES2. page 3: \"ViT yields 95\\% accuracy in (...) region-to-region correspondences\" - consider specifying how you compute the correspondence scores/patch distances. (cosine similarity, as in subsec 4.4 page 8 ?)\n* ES3. in the caption of Tab. 1, please specify W=14 is the window size.", "summary_of_the_paper": "The contribution of the paper comprises:\n* the observation that the Multi-Stage Vision Transformer (MSVT), as opposed to the \"monolithic\" Vision Transformer (VT), does not produce discriminative patch representation, and\n* a loss function for self-supervised pre-training of MSVTs, that encourages discriminative patch representation.\n\nExtensive experimental evaluation demonstrates that the proposed loss term, when added to the standard non-contrastive, self-supervised loss, brings a modest but systematic improvement in performance of the MSVT (Tab 5) and other architectures (Tab 4).\n", "summary_of_the_review": "I find the main idea of this paper convincing, the manuscript well written, and the evaluation thorough. The fact that the authors highlight some limitations of their loss term, in particular when applied to image segmentation, attests to their scientific scrutiny. I think the paper deserves a publication. I am willing to rate it 8/10, instead of 10/10, because I find the contribution incremental and the performance gain modest.\n\nIn this round of reviews, I lower my recommendation to 6, because I find the weakness (2), described above, acute. It misrepresents the effect of the contribution on performance in the teaser figure and in the first table with quantitative results. I am confident the authors can fix it in the next revision of the paper, which will prompt me to increase the rating to 8/10.\n\nI encourage the authors to also address the weakness (3) in the next revision of the paper. The current transfer learning experiments (section 4.2) do not evaluate the specific contribution of the paper, but rather answer more general questions. I am confident that adding evaluation more focused on the use of $\\mathcal{L}_R$ will vastly benefit this part of the paper. I nevertheless do not feel that I should request completing this task within the limited review period, and I will not lower my recommendation on this basis.\n\nEDIT:\nthe authors addressed the weaknesses 2) and 3) listed in the main review and answered all my questions. I am therefore happy to endorse the paper. This is a solid work and deserves to be published.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kIgxZIT55DB", "reviewer": "Reviewer_sJFu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has many strengths.\n\n- It is interesting to see that the loss on local features can work well in the self-supervised learning case. This observation may find more usage in the future works of doing self-supervised learning on transformer-based models.\n\n- The experiments are comprehensive and convincing. The paper tries the same idea on multiple transformer structures and different tasks. The proposed method can achieve the efficiency of the multi-stage transformer models while achieving good self-supervised feature pre-training for tasks including classification, segmentation, and detection.\n\n- The info in the appendix is helpful for result reproduction and detailed understanding.\n\nIn terms of weakness, I do hope that the paper can be more clearly written, such as reorganizing the info between main text and appendix to give more intuition of L_R.\n\nIn addition, the detail of Table 2 is too scarce to understand directly. I guess the supervised baseline is Swin-T, but it is hard to infer from the paper directly.", "summary_of_the_paper": "The paper investigates how to use self-supervised learning for multi-stage visual transformer models. Previous works have shown that SSL can learn image correspondences and lead to performant pre-trained models, while the multi-stage models can reduce the computation cost dramatically. This work tries to merge these two trends together. The solution is a new region-based loss that can be applied to the local features. The comprehensive experiments show the advantages of the resulting models on multiple tasks.", "summary_of_the_review": "Overall, the manuscript looks like a good paper. I do hope reading the paper can be easier despite the page limit.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GPtlPjHBbN1", "reviewer": "Reviewer_6vXm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:  \n(1) It introduces the new pretraining task to capture fine-grained region dependencies;\n(2) The experimental results are good and better than other compared approaches.\n\nWeaknesses:\n(1) The patch merging module and sparse self-attention in the multi-stage ViT are very similar to the patch merging in the paper Swin Transformer [Liu et al 2021]. The authors should clearly explain the differences between this paper and Swin Transformer.\n(2)The equation (2) should add more high-level descriptions. For each local feature z_i, why finding the local feature z_j from the teacher with the highest cosine similarity can capture the fine-grained region dependency? ", "summary_of_the_paper": "This paper develops an efficient self-supervised vision transformer for learning visual representations. It introduces a multi-stage architecture with sparse attentions to reduce computation complexity and proposes a new pretraining task of region matching to capture fine-grained region dependencies. The results on the ImageNet and 18 small datasets or downstream tasks are good and compared with other state-of-the-art approaches.\n", "summary_of_the_review": "The theoretical novelties are limited, this paper mainly borrows the ideas from two papers DINO [Caron et al. 2021] and Swin transformer [Liu et al 2021]. However, the experimental results are very good in this paper, and the paper provides a lot of implementation details for others to reproduce the results. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes two techniques for improving self-supervised learning with a vision transformer. The first improvement is using a multi-stage ViT, which is very similar to Swin transformer and authors recognized this is not a major contribution. The authors further found that using a multi-stage ViT does not produce discriminative patch representation, thus proposing the second improvement with a region level loss. While both improvements are not particularly novel by themselves, combining both leads to a strong empirical result. However, It does looks like the multi-scale vision transformer is the major improvement as removing the regional loss only leads to less than 1% decrease in performance in most cases. In general this is a good "engineering" paper with a practical approach for improving self-supervised learning with vision transformation and obtained strong results, thus it's worthy of publication.
Decision: Accept (Poster)