Image BERT Pre-training with Online Tokenizer
Authors: Jinghao Zhou
Chen Wei
Huiyu Wang
Wei Shen
Cihang Xie
Alan Yuille
Tao Kong
Venue: ICLR 2022 Poster (2022)
Keywords: online tokenizer
masked image modeling
vision transformer

Abstract: The success of language Transformers is primarily attributed to the pretext task of masked language modeling (MLM), where texts are first tokenized into semantically meaningful pieces.
In this work, we study masked image modeling (MIM) and indicate the necessity and challenges of using a semantically meaningful visual tokenizer.
We present a self-supervised framework iBOT that can perform masked prediction with an online tokenizer. 
Specifically, we perform self-distillation on masked patch tokens and take the teacher network as the online tokenizer, along with self-distillation on the class token to acquire visual semantics.
The online tokenizer is jointly learnable with the MIM objective and dispenses with a multi-stage training pipeline where the tokenizer needs to be pre-trained beforehand.
We show the prominence of iBOT by achieving an 82.3% linear probing accuracy and an 87.8% fine-tuning accuracy evaluated on ImageNet-1K.
Beyond the state-of-the-art image classification results, we underline emerging local semantic patterns, which helps the models to obtain strong robustness against common corruptions and achieve leading results on dense downstream tasks, e.g., object detection, instance segmentation, and semantic segmentation.

Published as a conference paper at ICLR 2022
IBOT IMAGE BERT PRE-TRAINING WITH ONLINE TOKENIZER
Jinghao Zhou' Chen Wei Huiyu Wang Wei Shen
Cihang Xie Alan Yuille² Tao Kong'
ByteDance "Johns Hopkins University
SShangaai Jiao Tong University 'UC Santa Cruz
ABSTRACT
The success of language Transformers is primarily attributed to the pretext task of masked language modeling (MLM) (Devlin et al., 2019), where texts are first tok- enized into semantically meaningful pieces. In this work, we study masked image modeling (MIM) and indicate the advantages and challenges of using a semanti- cally meaningful visual tokenizer. We present a self- self-super framework iBOT that can perform masked prediction with an online tokenizer. Specifically, we per- form self-distillan on masked patch tokens and take the teacher network as the online tokenizer, along with self-distillation on the class token to acquire visual semantics. The online tokenizer is jointly learnable with the MIM objective and dispenses with a muuliisttaee training pipeline where the tokenizer needs to be pre- trained beforehand. We show the prominence of iBOT by achieving 82.3% lin- ear probing accuracy and an 87.8% % fine-tuning accuracy evaluated on ImageNet- 1K. Beyond the state-of-the-art image classification results, we underline emerg- ing local semantic patterns, which helps the models to obtain strong robustness against common corruptions and achieve leading results on dense downstream tasks, e,g. object detection, instance segmentation, and semantic segmentation. The code and models are publicly available at http://hhhbb..ooobyyyy//////bbtb
INTRODUCTION
Masked Language Modeling (MLM), which first randomly masks and then reconstructs
1BTT
set of input tokens, is popular pre-training paradigm for language models. The MLM pre-trained Transformers (Devlin et al., 2019) have demonstrated their scalability to large- capacity models and datasets, becoming de- facto standard for lingual tasks. However, its potential for Vision Transformer (ViT), which recently started revolutionize computer vi- sion research (Touvron et al., 2021; Doso- vitskiy al., 2021), has been largely under- explored. Most popular unsupervised pre- training schemes in vision deal with the global views (Chen et al., 2021; Caron et al., 2021), neglecting images' internal structures, posed to MLM modeling local tokens. In this work, we seek to continue the success of MLM and explore Masked Image Modeling (MIM) for training better Vision Transform- such that it can serve as a standard component, it does for NLP.
BYOL
BYOL
TnoegoNet Val Ace81
DI80
SwAV
1907
BYO1.
RoCov3
77-
SwAV
DINO
Novo
78
ScAV
74
3YD
ViT
MaCox3
25
30
100
200
100
op-
Number of Parameters (Millions)
Figure 1: Linear probing accuracy on ImageNet. We compare iBOT with other unsupervised base- lines.
One of the most crucial components in MLM is the lingual tokenizer which splits language into semantically meaningful tokens, e.g., WordPiece (Wu et al., 2016) in BERT. Similarly, the crux of MIM lies in a proper design of visual tokenizer, which transforms the masked patches to supervisory
Email: Jinghao Zhou baanueejjenen.hhuj@@@amamae
Tao Kong< bytedance.com..

---

Published as a conference paper at ICLR 2022
signals for the target model, shown in Fig. 2. However, unlike lingual semantics arising naturally from the statistical analysis of word frequency (Sennrich et 2016), visual semantics cannot be ex- tracted such easily due to the continuous property of images. Empirically, visual semantics emerges progressively by bootstrapping online representation that enforces a similarity of distorted image views (He et al., 2020; Grill al., 2020; Caron et al., 2020). This property intuitively indicates multi-stage training pipeline, where we need to first train an off-the-shelf semantic-rich tokenizer before training the target model. However, since acquiring visual semantics is a common end for both the tokenizer and target model, a single-stage training pipeline where the tokenizer and target model can be jointly optimized awaits further exploration.
Previous works partially tackle the above challenges. Several works use identity mapping as the visual tokenizer, ie., predicting the raw pixel values (Pathak et al., 2016; Atito et al., 2021). ). Such paradigm struggles in semantic abstraction and wastes the capacity at modeling high-frequency de- tails, yielding less competitive performance in semantic understanding (Liu et al., 2021a). Recently, BEiT (Bao al., 2021) proposes to use pre-trained discrete VAE (Ramesh et al., 2021) as the to- kenizer. Though providing some level of abstraction, the discrete VAE is still found only to capture low-level semantics within local details (as observed by Tab. 9). Moreover, the tokenizer needs to be offline pre-trained with fixed model architectures and extra dataset (Ramesh et al., 2021), which potentially limits its adapativity to perform MIM using data from different domains.
To this end, we present iBOT short for image BERT pre-training with Online Tokenizer,
framework that performs MIM with a tokenizer handling above-mentioned challenges favorably. We moti- vate iBOT by formulating the MIM as knowledge distillation (KD), which learns to distill knowledge from the tokenizer, and further propose to perform self-distillation for MIM with the help of twin teacher as online tokenizer. The target network is fed with a masked image while the online tokenizer with the original image. The goal to let the target network recover each masked patch token to its cor- responding tokenizer output. Our online tokenizer naturally resolves two major challenges. On the one hand, our tokenizer captures high- level visual semantics progressively learned by enforcing the simi- larity of cross-view images class tokens. On the other hand, our tokenizer needs no extra stages of training as pre-procssiing setup since it jointly optimized with MIM via momentum update.
diszill
Tok.
ViT
Figure 2: Masked image modeling. / denotes an im- age and Tok. denotes a vi- sual tokenizer.
The online tokenizer enables iBOT to achieve excellent performance for feature representation. Specifically, iBOT advances Im classification benchmark under k-NN, linear probing and fine-tuning protocols to 77.1%, 79.5%, 84.0% with ViT-Base/16 respectively, which is 1.0%, 1.3%, 0.4% higher than previous best results. When pre-trained with ImageNet-22K, iBOT with ViT-L/16 achieves linear probing accuracy of 82.3% % and fine-tuning accuracy of 87.8% %, which 1.0% and 1.8% higher than previous best results. Beyond that, the advancement is also valid when transferring to other datasets or under semi-supervised and unsupervised classification settings. Of particular interest, we have identified an emerging part-level semantics that can help the model with image recognition both on global and local scales. We identify that the semantic patterns learned in patch tokens, which sufficiently lack in the off-line tokenizer as in BEiT (Bao et al., 2021), helps the model to be advanced in linear classification and robustness against common image corruptions. When it is transferred to downstream tasks, we show that in downstream tasks related to image classification, object detection, instance segmentation, and semantic segmentation, iBOT surpasses previous methods with nontrivial margins. All of the evidence demonstrates that iBOT has largely closed the gap of masked modeling pre-training between language and vision Transformers.
2 PRELIMINARIES
2.1 MASKED IMAGE MODELING AS KNOWLEDGE DISTILLATION
Masked image modeling (MIM), which takes a similar formulation as MLM in BERT, has been proposed in several recent works (Bao et al., 2021; Tan et al., 2021). Specifically, for an image token sequence x = [ MIM first samples a random mask m E {0, according to a prediction ratio r, where N is the number of tokens. The patch token æ; where m. being 1, denoted
2

---

Published as a conference paper at ICLR 2022
as σ _ { my 1), are then replaced with mask token [илак] yielding comrupted image a - ( (1 mm)a; + m (xass] } MIM is to recover the masked tokens from the corrupted image â i.e., to maximize: log qo (0(x))) s Σ my log qo (x; (#,()), where holds with an independence assumption that each masked token can be reconstructed separately. In BEiT (Bao et al., 2021), do is modelled as a categorical distribution and the task to minimize
m, Po(x)) log Po(
where P(.) transforms the input to a probability distribution over K dimensions, and Φi parameters of a discrete VAE (Ramesh et al., 2021) that clusters image patches into K categories and assigns each patch token a one-hot encoding identifying its category. We note this loss is formulated simi- larly knowledge distillation (Hinton et al., 2015), where knowledge is distilled from pre-fixed tokenizer parameterized by Φ to current model parameterized by 0.
2.2 SBLF-DISTILLLTTION
duffdistillation, proposed recently DINO (Caron et al., 2021), distills knowledge not from pos- terior distributions P& (x) but past iterations of model itself Po (x) and is cast as discriminative self-supervised objective. Given the training set I, an image x I is sampled uniformly, over which two random augmentations are applied, yielding two distorted views u and e. The two dis- torted views are then put through a teacher-student framework to get the predictive categorical dis- tributions from the [CLS] ] token: vi ct.) Plcts) (v) and u. [ Pe [CL5 (u). The knowledge distilled from teacher to student by minimizing their cross-entropy, formulated as
cLs] - PI (v)T log PlcLs) (w).
(2)
The teacher and the student share the same architecture consisting of backbone f (e.g.. ViT) and a projection head [CLS] The parameters of the student network are Exponentially Moving Averaged (EMA) to the parameters of teacher network 0' The loss is symmetrized by averaging with another cross-entropy term between v. ([CSS) and u, [11))
3 IBOT
We motivate our method by identifying the similar formulation of Eq. (1) and Eq. (2). A visual tokenizer parameterized by online O' instead of pre-fixed & thus arises naturally. In this section, we present iBOT, casting self-distillation as voken-generaie MIM via self-distillation. We illustrate the framework of iBOT in Fig. and demonstrate the pseudo- code in Appendix A. In Sec. 3.2, we briefly introduce the architecture and pre- traininiing setup.
self-supervised objective and perform
3.1
FRAMEWORK
First, we perform blockwise masking (Bao et al., 2021) on the two augmented views u and v and obtain their masked views û and é. Taking å example for simplicity, the student network outputs for the masked view ú projections of its patch tokens upatch pe network outputs for the non-masked view u projections of its patch tokens u. waacch ppatch (u). We here define the training objective of MIM in iBOT as
ppatch (ū) and the teacher
6'
LMIM- Σ
m4
phatch (u:)² T log ppatch (ū:).
(3)
We symmetrize the loss by averaging with another CE term between @patch and patch
The backbone together with the projection head of teacher network /patch fr is, therefore, visual tokenizer that generates online token distributions for each masked patch token. The tokenizer used in iBOT is jointly learnable to MIM objective without need of being pre-traie pre-trained in extra stage, bonus feature of which is now its domain knowledge can be distilled from the current dataset rather than fixed to the specified dataset.
3

---

Published as a conference paper at ICLR 2022
[CLS]
0. [CLS
A/
fs
(CLS)
MMIM
parc
x-1
t~T
A [C1S]]
(C15))
[cts]
17
ft
patch
[[MASK]
aetiasr Sckesttver
grol
Figure 3: Overview of iBOT framework, performing masked image modeling with an online tokenizer. Given two views u and v of an image æ, each view is passed through a teacher network h4 fo and a student network h2 fa- iBOT minimizes two losses. The first loss CicLs] is self- distillation between cross-view [CLS] tokens. The second loss LMIM is self-distillation between in-view patch tokens, with some tokens masked and replaced by e [Mass] for the student network. The objective is to reconstruct the masked tokens with the teacher networks' outputs as supervision.
To ensure that the online tokenizer is cemantically-manniie [CLS] token of cross-view images such that visual semantics can be obtained via bootstrapping, as achieved by the majority of the self-super methods (He et al., 2020; Grill et al., 2020; Caron et al., 2021). In practice, iBOT works with & [cxss] in Eq. (2) proposed DINO (Caron et al., 2021), except that now we have à [CLS] instead of u. [CL0] as input for the student network. To further borrow the capability of semantics abstraction acquired from self-distillatin on [CLS] token, we share the parameters of projection heads for [CLS] token and patch tokens, i.e., h h[[css hpatch
we perform self-distillation on
h [CLS]] = =/, A,patch, We empirically find that produces better results than using separate heads.
Unlike tokenized words whose semantics are almost certain, image patch is ambiguous in its seman- tic meaning. Therefore, tokenization as one-hot discretization can be sub-optimal for images. In iBOT, we use the token distribution after softmax instead of the one-hot token id as supervisory signal, which plays an important role in iBOT pre-training as shown in Tab. 18.
3.2 IMPLEMENTATION
Architecture. We use the Vision Transformers (Dosovitskiy et al., 2021) and Swin Transformers (Liu et al., 2021b) with different amounts of parameters, ViT-S/16, ViT-B/16, ViT-L/16, and Swin- 1/47,14] as the backbone ʃ. For ViTs, /16 denotes the patch size being 16. For Swins, /(7, 14] denotes the window size being 7 or 14. We pre-train and fine-tune the Transformers with 224-size images, so the total number of patch tokens is 196. The projection head is a 3-layer MLPs with ng-normalized bottleneck following DINO (Caron et al., 2021). Towards a better design to acquire visual semantics, we studied different sharing strategies between projection heads h. [CLS] and h patch considering that semantics obtained distillation on [CLS] token helps the training of MIM on patch tokens. We empirically find that sharing the entire head prompts the best performance. We set the output dimension of the shared head to 8192.
Pre-Training Setup. We by default pre-train iBOT on ImageNet- (Deng et al., 2009) training set with AdamW (Loshchilov & Hutter, 2019) optimizer and a batch size of 1024. We pre-train iBOT with ViT-S/16 for 800 epochs, ViT-B/16 for 400 epochs, ViT-L/16 for 250 epochs, and Swin- T/[7.14) for 300 epochs. We also pre-train ImageNet-22K training set with ViT-B/16 for 80 epochs and ViT-L/16 for 50 epochs. The learning rate is linearly ramped up during the first 10 epochs to its base value scaled with the total batch size: Ir -4 batch size/256. /26.. We use random MIM, with prediction ratio r set as 0 with probability of 0.5 and uniformly sampled from range 10.1, 0.5] with probability of 0.5. We sum L (oLs) and LMIM up without scaling.
EXPERIMENT
We first transfer iBOT to downstream tasks, following the standard evaluation protocols adopted in prior arts, the details of which are delayed in Appendix C. We then study several interesting
4

---

Published as a conference paper at ICLR 2022
Table k-NN and linear probing on ImageNet- IK. denotes using selective kernel. 'enotess pre- training on ImageNet-22K,
Table 2: Fine-tuning on ImageNet- I
Method
Arch.
Epo.
Acc.
Rand.
VIT-S/16
Method Arch.
Par.
im/s
Epo."
k-NN Lin.
McCov3
VIT-S/16
SSL big ResNets
DINO
VIT-S/16
3200
82.0
MoCov3
RN50
23
1237
1600
74.6
iBOT
VIT-S/16
3200
82.3
SWAV
RN50
23
1237
2400
65.7
75.3
Rand.
VIT-B/16
81.8
DINO
RNSO
23
1237
3200
67.5
75.3
MoCov3
VIT-B/16
600
83.2
BYOL
RN200w2
250
123
2000
73.9
79.6
BEIT
VIT-B/16
800
83.4
SCLRv2
RN152w3
794
46
2000
73.1
79.8
DINO
VIT-B/16
1600
83.6
iBOT
VIT-B/16
1600
84.0
SSL Transformers
MoCov3
ViT-S/16
21
1007
1200
73.4
MoCov3
VIT-L/16
600
84.1
McCov3
ViT-B/16
85
312
1200
76.7
iBOT
VIT-L/16
1000
84.8
SWAV
VIT-S/16
21
1007
2400
66.3
73.5
BEIT
VIT-L/16
800
85.2
DINO
ViT-S/16
21
1007
3200
74.5
77.0
DINO
ViT-B/16
85
312
1600
76.1
78.2
Table 3: Fine-tuning on ImageNet-Ile
EVVIT
Swin-T/7
726
1200
75.7
78.1
Pre-training ImageNet-22K.
ESVIT
Swin-T/14
28
593
1200
77.0
78.7
Method
Arch.
Epu.
Acc.
BOT
ViT-S/16
21
1007
3200
75.2
77.9
BOT
Swin-T/7
28
726
1200
75.3
78.6
BHTT
ViT-B/16
150
83.7
BOT
Swin-T/14
28
593
1200
76.2
79.3
iBOT
VIT-B/16
320
84.4
iBOT
ViT-B/16
312
1600
77.1
79.5
BEIT
VIT-L/16
150
86.0
iOTT
ViT-L/16
307
102
1200
78.0
81.0
IBOT
VIT-L/16
200
86.6
iBOT
VIT-L/16
307
102
200
72.9
82.3
iBOT
ViTs12-L/16
200
87.8
properties of Transformers pre-trained with iBOT. Finally, we give brief ablation study on the
crucial composing of iBOT.
4.1 CLASSIFICATION ON IMAGENET-IE
We consider five classification protocols ImageNet-1K: k-NN, linear probing, fine-tuning, semi- supervised learning, and unsupervised learning.
k-NN and Linear Probing. To evaluate the quality of pre-trained features, either k- k-nearest neighbor (k-NN) classifier or a linear classifier on the frozen representation. We follow the evalua- tion protocols in DINO (Caron al., 2021). For k-NN evaluation, we sweep over different numbers of nearest neighbors. For linear evaluation, we sweep over different learning rates. In Tab. 1, our method reaches linear probing accuracy 77.9 77.9% with ViT-S/16, linear probing accuracy 79.5% with ViT-B/16, and a k-NN accuracy 78.0% and linear probing accuracy 81.0% with ViT-L/16, achieving state-of-the-art performance. With Swin-T/(7.14) iBOT achieves linear probing aceu- racy of 78.6% and 79.3% respectively, With VIT-L/16 and ImageNet-22K as pre-training data, iBOT further achieves a linear probing accuracy 82.3% %, surpassing previous state of the art, 81.3% with Swin-B/14 by EsVIT (Li et al., 2021a) A linear probing accuracy of 79.5% with ViT-B/16 is com- parable to 79.8% by SimCLRv2 with RN152 (3x) but with 10x less parameters. We underline that the performance gain over DINO gets larger (0.9% w/ VIT-S versus 1.3% w/ ViT-B) with more parameters, suggesting iBOT is more scalable to larger models.
Fine-Tuning. We study the fine-tuning on ImageNet-1K 1K and focus the comparison with self- supervised methods for Transformers and its supervised baseline (Rand.) (Touvron et al., 2021). As shown in Tab. 2, iBOT achieves 82.3%. 84.0%, and 84.8% top-1 accuracy with ViT-S/16. ViT-B/16, and ViT-L/16, respectively. As shown in Tab. 3, iBOT pre-trained with ImageNet-22K achieves 84.4% and 86.6% top-1 accuracy with ViT-B/16 and ViT-L/16. respectively, outperforming ImageNet-22K pre-trained BEiT by 0.7% and 0.6%. When fine-tuned on image size of 512, we achieve 87.8% % accuracy. We note that, with ViT-L/16. iBOT 0.4% worse than BEiT using KK data but 0.6% better using 22K data. This implies that iBOT requires more data to train larger model.
Semi-Supervised and Unsupervised Learning. For sumi-supervised learning, we focus our com- parison with methods following the unsupervised pre-train, supervised fine-tune paradigm. As
Effectiive pre-training epochs accounting for actual trained images/views. See Appenix B for details.

---

Published as a conference paper at ICLR 2022
Table 4: Semi-supervised learning on ImageNet-11., 1% and 10% denotes label fraction. SD denotes self-distillation.
Table 5: Unsupervised learning on ImageNet- IK. denotes k-means clustering on frozen fea-
tures.
Method Arch.
1%
10%
Method
Arch.
ACC
ARI
NMI FMI
SimCLRv2
RN50
57.9
68.1
Self-label
RN50
30.5
16.2
75.4
BYOL
RN50
53.2
68.8
SWAV
RN50
53.9
70.2
InfoMin
RN50
33.2
14.7
68.8
SimCLRv2+SD
RN50
60.0
70.5
SCAN
RN50
39.9
27.5
72.0
DINO
VIT-S/16
60.3
74.3
DINO
ViT-S/16
41.4
29.8
76.8
32.8
iBOT
V/T
61.9
75.1
iBOT
VIT-S/16
43.4
32.8
78.6
35.6
Table 6: Object detection (Det.) & instance segmentation (ISeg.) on COCO and Semantic segmentation (Seg.) on ADE20K. We report the results of ViT-S/16 (left) and ViT-B/16 (right). Seg.1 denotes using a linear head for semantic segmentation.
Det.
ISeg.
Seg.
Det.
ISeg.
Seg. f
Seg.
Method Arch.
Param.
Method
Apb
Apm
mloU
APb
Apm
mloU
mloU
Sup.
Swin-T
29
48.1
41.7
44.5
Sup.
49.8
43,2
35.4
46.6
MoBY
Swin-T
29
48.1
41.5
44.1
BEiT
50.1
43.5
27.4
45.8
Sup. VIT-S/16
21
46.2
40.1
44.5
DINO
50.1
43.4
34.5
46.8
iBOT
ViT-S/16
21
49.4
42.6
45.4
iBOT
51.2
44.2
38.3
50.0
shown in Tab. 4, iBOT advances DINO by 1.6% and 0.8% using 1% and 10% data, respectively, suggesting higher label efficiency. For unsupervised learning, including accuracy (ACC), adjusted random index (ARI), normalized mutual information (NMI), and Powlkes-Mallows index (FMI). We compare our methods SimCLRv2 (Chen al., 2020b), Self-label (Asano et al., 2020), InfoMin (Tian et al., 2020), and SCAN (Van Gansbeke et al., 2020). As shown in Tab. 5, we achieve 32.8% NMI, outperforming the previous state of the art by 1.8%, suggesting MIM helps the model learn stronger visual semantics on a global scale.
standard evaluation metrics,
4.2 DOWNSTREAM TASKS
Object Detection and Instance Segmentation on COCO. Object detection and instance segmen- tation require simultaneous object location and classification. We consider Cascade Mask R-CNN (Cai & Vasconcelos, 2019; He et al., 2017) that produces bounding boxes and instance masks simul- tancously on COCO dataset (Lin et al., 2014). Several recent works (Liu al., 2021b; Wang al., 2021a) proposes Vision Transformers that suit dense downstream tasks. To compare, we include the results of supervised Swin-T (Liu et al., 2021b) which shares approximate parameter numbers with VIT-S/16 and its suff-super counterpart MoBY (Xie et al., 2021a) in Tab. 6. iBOT improves ViT-S's AP from 46.2 49.4 and APM from 40.1 to 42.6, surpassing both supervised Swin-T and its self-supervised counterpart by a nontrivial margin. With ViT-B/16, iBOT achieves an Apb of 51.2 and an Apm of 44.2, surpassing previous best results by a large margin.
Semantic Segmentation on ADE20K. Semantic segmentation can be seen as a pixel-level classi- fication problem. We mainly consider two segmentation settings on ADE20K dataset (Zhou al., 2017). First, similar to linear evaluation protocol in classification, we evaluate features and only fine-tune a linear layer, which gives us more explicit comparison the quality of representations. Second, we use the task layer in UPerNet (Xiao et al., 2018) 8) and fine-tune the entire network. From Tab. 6, we can see that iBOT advances its supervised baseline with ViT-S/16 with large margin of 0.9 on mIoU, surpassing Swin-T. With ViT-B/16, iBOT advances previous best methods DINO by 3.2 on mIoU with UperNet. We notice performance drop of BEiT using linear head, indicating BETT's features lack local semantics. As analyzed later, the property of strong local semantics induces a 2.9 mloU gain compared to the supervised baseline with a linear head.
the fixed patch
Transfer Learning. We study transfer learning where we pre-train on ImageNet-IK IK and fine-tune on several smaller datasets. We follow the training recipe and protocol used in (Dosovitskiy et al., 2021). The results demonstrated in Tab. 7. While the results several datasets (e.g.. CIFARIO. 10, CIFAR 100, Flowers, and Cars) have almost plateaued, iBOT consistently performs favorably against other SSL frameworks, achieving state-of-the-art transfer results. We observe greater performance

---

Published as a conference paper at ICLR 2022
Table 7: Transfer learning by fine-tuning pre-trained models on different datasets. We report Top-1 1 accuracy of ViT-S/16 (left) and ViT-B/16 (right).
Method
Cif
Cif
iNais
iNa19
Flwrs Cars
Method
Cifio
Cif
iNa
iNa19
Flwrs
Cars
Rand. 99.0
89.5
70.7
76.6
98.2
92.1
Rand.
99.0
90.8
73.2
77.7
98.4
92.1
BEiT
98.6
87.4
68.5
76.5
96.4
92.1
BEIT
99.0
90.1
72.3
79.2
98.0
94.2
DINO
99.0
90.5
72.0
78.2
98.5
93.0
DINO
99.1
91.7
72.6
78.6
98.8
93.0
iBOT
99.1
90.7
73.7
78.5
98.6
94.0
iBOT
99.2
92.2
74.6
79.6
98.9
94.3
Figure 4: Pattern layout of patch tokens. Two left figures showcase patterns, headlight of the vehicle and ear of the dog. that share part semanties. Two right figures showcase patterns, stripped and curly surface, that share part textures.
gain over DINO in larger datasets like 1Naturalist18 and INaturalist19, indicating the results are still far from saturation. We also find that with larger models, typically get larger performance gain compared with DINO (e.g.. 1.7% with ViT/S-16 versus 2.0% with ViT-B/16 on iNaturalitt 88, and 0.3% with VIT/S-16 versus 1.0% with ViT-B/16 on 1Naturalist19).
4.3 PROPERTIES OF VIT TRAINED WITH MIM
In the previous sections. we have shown the priority of iBOT on various tasks and datasets. To reveal the strengths of iBOT pre-trained Vision Transformers, we analyze its property from several aspects.
4.3.1 DISCOVERING THE PATTERN LAYOUT OF IMAGE PATCHES
What Patterns Does MIM Learn? The output from the projection head used for self-distillation depicts for patch token a probabilistic distribution. To help understand what patterns MIM induces learn, we visualize several pattern layouts. We use 800-epoch pre-trained ViT-S/16 and visualize the top-36 patches with the highest confidence on ImageNet-IK validation set. We visualize 5x context for each 16 16 patch (colored orange). We observe the emergence of both high-level semantics and low-level details. As shown in Fig. 4, several patches are grouped with clear semantic meaning, egg., headlight and dog's ear. Such behavior stands distinct contrast with the offline tokenizer used in BEiT (Bao et al., 2021), which encapsulates mostly low-level details shown in Fig. 16. Apart from patch patterns that share high-level semantics, we also observe clusters accounting for low-level textures, indicating the diversity of learned part patterns. The comparison with previous work (Caron et al., 2021; Bao et al., 2021) and the visualization of more pattern layouts provided in Appendix G.I.
How Does MIM Help Image Recognition? To illustrate how the property of better part semantics can help image recognition, we use part-wise linear classification to study the relationship between representations of patch tokens and [CLS] token. Specifically, we average k patch tokens with the top-k highest self-attention scores. The results are demonstrated in Fig. 5. While the performance gap between DINO and iBOT is only 0.9% in the standard setting (77.9% v.s. 77.0%) with [CLSS] token, we observe that iBOT outperforms DINO when using the patch representations directly. We observe that using top-56 patch tokens yields an optimal result, and iBOT is 5. 5.9% higher than DINO. The performance gap becomes more prominent when using fewer patch tokens. When using only the patch token with the highest self-attention score, iBOT advances by 17.9%. These results reveal much semantic information in iBOT representations for patch tokens, which helps the model to be more robust to the loss of local details and further boosts its performance on image-level recognition.

---

Published as a conference paper at ICLR 2022
62-
Suuggo hoott
80
00
DIDINO 12DT
132
100
Patch Til
Figure 6: Visualization for self-attention map. Self-attention map from multiple heads are visualized with different color.
Part-wise linear probing accu- racy. Top-k tokens with the highest attention scores are averaged for classification.
Figure 5:
Table 8: Robustness evaluation of pre-trained models against background change, occlusion, and out-of-distribution examples.
Background Change
Clean
Occlusion
Out-of-Dist.
Clean
Method
O.F. M.S. M.R.
M.N. N.F.
O.BB.
O.BT.
IN-9
S.
NS.
IN-A
IN-C
IN
DINO
89.2
89.2
80.4
78.3
52.0
21.9
18.4
96.4
64.7
42.0
12.3
51.7
77.0
iBOT
90.9
89.7
81.7
80.3
53.5
22.7
17.4
96.8
65.9
43.4
13.8
48.1
77.9
4.3.2
DISCRIMINATIVE PARTS IN SELF-ATTENTION MAP
To analyze, visualize the self-attention map with ViT-S/16. We choose [CLS] token the query and visualize attention maps from different heads of the last layer with different colors, as shown in Fig. 6. Of particular interest, or different parts of one object apart. For example, distinct the bird from the tree branch. Also, iBOT focuses mainly the discriminative parts of the object (e.g., the wheel of the car, the beak of the bird). These properties are crucial for iBOT to excel at image recogmition, especially in complicated scenarios with object occlusion or distracting instances. While these properties are not unique strengths brought by MIM and we observe similar behaviors in DINO, we show in Appendix G.2 that iBOT generally gives better visualized results.
indicate that iBOT shows solid ability separate different objects
the leftmost figure, we observe iBOT fairly
4.3.3 ROBUSTNESS
The above-mentioned properties brought by MIM objective can improve the model's robustness to uncommon examples. We quantitatively benchmark robustness in terms of 3 aspects: background change, occlusion, and out-of-distribution examples, with ViT-S/16 pre-trained for 800 epochs and then linearly evaluated for 100 epochs. Results are shown in Tab. 8. For background change, study images under 7 types of change, detailed in Appendix D. iBOT is more robust against background changes except for O.BT.. For occlusion, we study the linear accuracy with salient and non-salient patch dropping following Naseer al. (2021) with an information loss ratio of 0.5. iBOT has a smaller performance drop under both settings. For out-of-distribution examples, we study natural adversarial examples ImageNet-A (Hendrycks et al., 2021) and image corruptions in IeageNet-C (Hendrycks & Dietterich, 2019). iBOT has higher accuracy on the ImageNet-A and smaller mean corruptions error (mCE) on the ImageNet-C.
4.4 ABLATION STUDY ON TOKENIZER
In this section, we ablate the importance of using semantically meaningful tokenizer using a 300- epoch pre-trained ViT-S/16 with prediction ratio r 0.3 and without multi-crop augmentation. Additional ablations are given in Appendix E. iBOT works with self-distillation on [CLS] token with cross-view images (Leeus)) ) to acquire visual semantics. To verify, we conduct experiments to perform MIM without Lreus] or with alternative models as visual tokenizer. Specifically, O denotes a standalone DINO and A denotes a pre-tranined DALL-E encoder (Ramesh et al., 2021).
8

---

Published as a conference paper at ICLR 2022
We find that performing MIM without
Table 9: Effect of design choices of semantically meaningful tokenization.
C reess leads to undesirable results of 9.5% k-NN accuracy and 29.8% linear accuracy, indicating that visual seman- ties can hardly be obtained with only MIM. While semantics emerges with a standalone DINO as visual tokenizer, it is still far from reaching a decent result (44.3% versus 69. % in k--N accuracy). Comparing iBOT with multi-tasking of DINO and BETT (DINO+BEiT), we see the strengths of merging the semantics ac- quired by self-distillation with the visual tokenizer with an 11.5% advance in linear probing and 0.3% in fine-tuning. More- over, empirically observe performance improvement using Shared projection Head (SH) for [CLS] token and patch tokens, which shares the semantics acquired in [CLS] token to MIM.
Method iBOT
LMIM ((Cus)
SH k-NN
Lin. Fin.
69.1
74,2 81.5
V
x
69.0
73,8 81.5
X
9.5
29.8 79.4
O
X
44.3
60.0 81.7
BEiT DINO
A
X
6.9
23.5 81,4
X
V
67.9
72.5 80.6
BEiT + DINO
48.0
62.7 81.2
O: standalone DINO (w/o merop, 300-epoch)
A: pre-trained DALL-E encoder
5 RELATED WORK
Visual Representation Learning. Most self-supervised methods assume an augmentation invari- ance of images and achieve so by enforcing similarity over distorted views of one image while avoiding model collapse. Avoiding collapse can be achieved by noise-contrastive estimation with negative samples (Wu et al., 2018; He et al., 2020; Chen et al., 2020a), introducing asymmetric network (Grill et al., 2020; Chen & He, 2021), or explicitly enforcing the distribution of image distribution over the channel be uniform as well as one-hot (Caron et al., 2020; Amrani & Bron- stein, 2021; Caron et al., 2021). In fact, the idea of simultaneously enforcing distribution uniform and one-hot is hidden from earlier studies performing representation learning via clustering (Caron et al., 2018; 2020; YM. et al., 2020), where the cluster assignment naturally meets these two require- ments. Other methods rely on handcrafted pretext tasks and assume the image representation should instead be aware of image augmentation by solving image jigsaw puzzle (Noroozi & Favaro, 2016; Wei et al., 2019), predicting rotation (Komodakis & Gidaris, 2018) or relative position (Doersch et al., 2015).
Masked Prediction in Images. Predicting masked images parts is popular self-supervised pretext task drawing on the idea of auto-encoding and has been previously achieved by either recovering raw pixels (Pathak al., 2016; Atito et al., 2021; Li et al., 2021b) or mask contrastive learning (Henaff, 2020; Zhao et al., 2).1). Recently, it is formulated into MIM (Bao et al., 2021: Tan et al., 2021) with a discrete VAE (Rolfe, 2017: Ramesh et al., 2021) visual tokenizer. As a counterpart of MLM in NLP, MIM cases masked prediction into a classification problem supervised by labels output from the tokenizer, mitigating the problem of excessive focus on high- gigh-rreuency details. Concurrently, masked image prediction has been explored in the field of modtii-modllity, i.e., vision- language representation learning. These methods operate on local regions instead of global images thus reply on pre-trained detection models, ie., Faster- (Ren et al., 2015) to propose regions of interest. (Su et al., 2020; Lu et al., 2019; Chen et al., 2020c) perform masked region classification tasking the category distribution output from the detection model as the truuh.
6 CONCLUSION
In this work, we study BERT-like pre-training for Vision Transformers and underline the signifi- cance of a semantically meaningful visual tokenizer. We present a self-supervised framework iBOT that performs masked image modeling via self-distillatio with an online tokenizer, achieving state- of-the-art results on downstream tasks related to classification, object detection, instance segmenta- tion, and semantic segmentation. Of particular interest, we identify ties for models trained with MIM that helps for not only recognition accuracy but also robustness against common image corruptions. In the future, we plan to scale up iBOT to larger dataset (e.g., ImageNet-22KK or larger model size (e.g., ViT-L/16 and ViT-H/16) and investigate whether MIM can help Vision Transformers more scalable unlabelled data in the wild.
emerging part-level seman-

---

Published as a conference paper at ICLR 2022
Acknowledgement Tao Kong is the corresponding author. This work supported by ONR N00014-21-1-2812, Wei Shen was supported by NSFC 62176159 and Natural Science Founda- tion of Shanghai 21ZRI432200. Cihang Xie was supported by gift grant from Open Philanthropy. We would like to acknowledge Feng Wang, Rufeng Zhang, and Zongwei Zhou for helpful discus- sions. We thank Mathilde Caron, Julien Mairal, and Hugo Touvronfor for sharing details of DINO. We thank Li Dong and Hangbo Bao for sharing details of BEiT.
REFERENCES
Elad Amrani and Alex Bronstein. arXiv:2103.10994, 2021.
Self-supervised classification network.
arXiv preprint
Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clusteri and representation learning. In ICLR, 2020.
Sara Atito, Muhammad Awais, and Josef Kittler. SiT: Self-supervised vision transformer. arXiv preprint (ttl:22044036002, 2021.
Hangho Bao, Li Dong, and Furu Wei. BETT: BERT pre-training of image transformers. arXiv preprint (0.iv:2/06.0820 2021.
Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: High quality object detection and instance segmentation. TPAMI, 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In ECCV, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurlPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and
Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICMI., 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. In NeurlPS, 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2020c.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hicrarchical image database. In CVPR, 2009.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep
bidirectional transformers for language understanding. In NAACL, 2019.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x 16 words: Transformers for image recognition scale. In ICLR, 2021.
Jean-Bastin Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent: A approach self-supervised learning. In NewrIPS, 2020.
10

---

Published as a conference paper at ICLR 2022
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding, In ICML, 2020.
Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In ICLR, 2019.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NeurlPS, 2015.
Nikos Komodakis and Spyros Gidaris. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient supervisd vision transformers for representation learning. arXiv preprint S1ti::21060078883 2021a.
Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, al. MST: Masked self-supervised transformer for visual representation. arXiv preprint arXiv:2106.0563 2021b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. TKDE, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint vrXiv:2103.14038 2021b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and-language tasks. In NeurlPS, 2019.
Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Intriguing properties of vision transformers. arXiv preprint arXiv:2105.1044 10497, 2021.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning visual representations by solving jigsaw puzzles. In ECCV, 2016.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICMI, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NewrIPS, 2015.
Jason Tyler Rolfe. Discrete variational autoencoders. In ICLR, 2017.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, 2016.
Weijic Su, Xizhou Zhu, Yue Cao. Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. ICLR, 2020.
11

---

Published as a conference paper at ICLR 2022
Hao Tan, Jie Lei, Thomas Wolf, and Mohit Bansal. Vimpac: Video pre-training via masked token prediction and contrastive learning. arXiv preprint 1rXi:2106.112250, 2021.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In NeurIPS, 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning classify images without labels. In ECCV, 2020.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021a.
Xinlong Wang, Rufeng Zhang. Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In CVPR, 2021b.
Chen Wei, Lingxi Xie, Xutong Ren, Yingda Xia, Chi Su, Jiaying Liu, Qi Tian, and Alan L Yuille. Iterative reorganization with weak spatial constraints: Solving arbitrary jigsaw puzzles for unsu- pervised representation learning. In CVPR, 2019.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint 12XY.0660900844 2016.
Zhirong Wu, Yuanjun Xiong, X Yu Stella, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. CVPR, 2018.
Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. In ICLR, 2020.
Tete Xiao, Yingcheng Liu, Bolci Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018.
Zhenda Xie, Yutong Lin, Zhuliang Yao. Zheng Zhang, Qi Dai, Yue Cao, and Han Hu. Self- supervised learning with swin transformers. arXiv preprint arXiv:2 105.:2305.00553, 2021a.
Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Ex- ploring lixel-level consistency for unsupervised visual representation learning. In CVPR, 2021b.
Asano YM., Rupprecht C., and Vedaldi A. Self-labelling via simultaneous clustering and represen- tation learning. In ICLR, 2020.
Yucheng Zhao, Guangting Wang, Chong Luo, Wenjun Zeng, and Zheng-Jun Zha. Self-supervised visual representations learning by contrastive mask prediction. arXiv preprint MXXXx:2108.07954 2021.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
12

---

Published as a conference paper at ICLR 2022
A PSEUDOCODE
Algorithm 1: iBOT Py PyTorch-line like Pseudocode w/o multi-crop augmentation
Input:
gs. 9t
// student and teacher network // center [CLS] token and patch tokens // temperature on [CLS] taken for student and teacher network
C,C";
Ta: T
Te
// temperature on patch tokens for student and teacher network // momentun rate for network // momentum rates for center on (CLS) token and patch tokens
I;
m, m,m';
St-params = Is-params
for in loader do u. v = augment(ar), augment(a))
// random views
ú, m. = blockwise :
randon block-wise masking // zandom block-wise masking
m. blockwise.mask(v)
i [CLB] upatch (ú, returnaal _ooktttloo
// [n, K], [n, S² K]
[cus]
@patch g returnall sotum_all_to--tie -ook-true;
// [], [n, S K]
u (C18) u patch gr (u, return_all returnaill_ok-trrue))
// K),
[n, S2, K] S
0,
|C18] of patch = (v, return all tok-true);;
// [n, K],
,C, Tap r2) LMIM = (m. H( (patch patch C², T. T() sum m... /
LicLs] = .(@u. (cta] ( ot [C13] C.C To. Te) / H( [C .u c18)
+ (m, H ((patcc patch C², Ta T.) ( sum(dim=l)
m...um((im=l)) 2
(C10s)) .mean)) + --MIMM MM....manm)) backkaadd
update((.))
11 student, teacher and center update
gt-params = L 9t-params +(1 - 1). 8s-params
C m C + (1 m). cat([u, [CI6] vi [cla]
I.mean(dim=0)
C' m' C² + (1 m'). cat([u patch v. patch ( ).mean(dim=(00, 1)) end
def
H (8, 4, G, 7
/ = i.detach();
// stop gradient
s softmax(s/ Tsp dim=1)
t softmax((t - c) dim=l);
// center sharpen
return (ug(s))))
B MULTI-CROP
The advanced performance of several recent state-of-the-art methods (Caron et al., 2021; 2020) relies on multi-crop augmentation, as well as iBOT. In our early experiments, we find the direct usage of multi-crop augmentation leads to instability issues that degrade accuracy. We reveal that these results can be attributed to the distribution mismatch between masked images and non-masked images and can be resolved by minimal changes in iBOT framework.
Stability MIM Pre-trained with Multi- We first showcase several practices where train- ing instability occurs, shown in Fig. 7. To reveal the instability, we monitor the NMI curves during training for each epoch as shown in Fig. 8. The most intuitive ideas are to compute as (b) or (c). global crops. This pipeline is unstable during training, and we observe a dip in the NMI training curve. We hypothesize that can be caused by the distribution mismatch of masked global crops and mas-md local crops. To alleviate this, straightforward solution is to also perform MIM on local crops with an extra computation cost as (c). However, we do not observe this circumvents training instability. We hypothesize that the regions corresponding
In (b), MIM is only performed
13

---

Published as a conference paper at ICLR 2022
(a)
(d)
11
81
X1
8
82
82
X2
22
11
X1
3/1
91
v1
11
82
X2
2/2
82
2/2
2/2
1.1
9n
D'
3/1
global
local
student
teacher
L[CLS]
LMIM
view
Yt
view
Figure 7: Computation pipelines for iBOT with or without multi-crop augmentation. (a) iBOT w/o multi-crop augmentation. (b), (c), and (d) are three pipelines w/ multi-crop augmentation. (b) does not perform MIM for local crops, whereas (c) performs MIM for all crops. (d) only performs MIM for one of the two global crops. iBOT uses (b) with random MIM.
07
Nocusdaad Mutual Informsion
0.-
(a)
0.4
(c)
0.1
0.
Number Epocla
ViT-S/16. 100 epochs
(a)
(b)
(c)
(d)
(c)
w/ rand. MIM
k-NN
62.1
62.0
31.9
69.8
56.6
71.5
Figure& 8: Training curves of different multi-crop strategy.
to patch tokens of the local crops are small in size, in which there exist few meaningful contents to predict. This hypothesis can supported by the experiments that when we set the local crop seale in (c) from (0.05, 0.4) to (0.2, 0.4), denoted as (e), the performance drop is mitigated.
Stabilizing the Training with Non-Masked Global Crops. Another solution to alleviate the dis- tribution mismatch between masked global crops and non-masked local crops to train with non- masked global crops, shown in (d). In other words, we perform random MIM when training ViT with multi-crop augmentation. This computation pipeline is stable and achieves substantial perfor- mance gain. In practice, include non-masked global crops in training, we (b) and randomly choose a prediction ratio between [0, r (r > O)] for each image. When the ratio 0 is chosen, the whole framework excludes MIM and can be seen as DINO. When the ratio (r > 0) chosen, MIM is performed for both of the two global crops. We observe the latter practice performs sightly better since it more flexible task composition and data in a batch mutually independent.
Range Scales in Multi-Crop. We further study the performance with different local and global scale. Following DINO (Caron et al., 2021), we conduct the experiments by tweaking where the scale deviding the local and global crops. The local crops are sampled from (0.05, s) and the global crops are sampled from (s. 1).
ViT-S/16, 300 epochs
0.25
0.4
0.32
ViT-B/16, 50 epochs
0.25
0.4
0.32
k-NN
74
74.3
74.6
A-NN
70
70.1
70.4
14

---

Published as a conference paper at ICLR 2022
Table 10: k-NN and linear probing accuracy on ImageNet-IK without multi-crop augmen- tation (left) and with multi-crop augmentation (right) multi-crop augmentation. We split the table into results without or with multi-crop augmentation.
Method Arch
Param. Epo. K-NN Linear
Method Arch
Param.
Epo.
k-NN Linear
RN50
23
800
74.6
RN50
23
800
65.7
75.3
SwAV
MoCov3
ViT-S/16
21
600
73.4
ViT-S/16
21
800
66.3
73.5
ViT-B/16
85
600
76.7
RN50
23
800
67.5
75.3
ViT-S/16
21
800
70.0
73.7
DINO
VIT-S/16
21
800
74.5
77.0
DINO
ViT-B/16
85
400
68.9
72.8
ViT-B/16
85
400
76.1
78.2
ViT-S/16
21
800
72.4
76.2
ViT-S/16
21
800
75.2
77.9
iBOT
iBOT
VIT-B/16
85
400
71.2
76.0
ViT-B/16
85
400
76.8
79.4
We empirically find that = 0.32 yields optimal performance for both small-size and base-size models. Therefore, we use an 8 of 0.32 by default.
State-of-the-Art Comparison w/o and w/ Multi-Crop. Including iBOT, several recent state-of- the-art works (Caron et al., 2020) rely heavily on multi-crop augmentation during pre-training. Except for several specific self-supervised methods (Grill et al., 2020), multi-crop works well on most of the self-supervised methods and consistently yields performance gain (Caron et al., 2021). While more fair comparison with our methods without multi-crop augmentation can be conducted, we believe it is a unique strength of iBOT to work well with multi-crop. In Tab. 10, we categorize the state-of-the- of-tte-art comparison into two parts where one for methods without multi-crop and the other with multi-crop For the former, we mainly compare our method without multi-cro crop with MoCov3 (Chen et al., 2021) and DINO without multi-crop. We observe that our method achieves state-of of-te--f-hhe- art performance with ViT-S/16 even without multi-crop and comparable performance with ViT-B/16 B/16 compared with MoCov3. For the latter, we mainly compare our method with SWAV (Caron et al., 2020) and DINO with multi-crop augmentation. We observe that iBOT achieves higher performance with 79.4% of linear probing accuracy when using ViT-S/16.
Effective Training Epochs. Due to extra computation costs brought by multi-crop augmentation, different methods with the same pre-training epochs actually see different total numbers of images. To mitigate, we propose to measure the effective training epochs, defined actual pre-training epochs multiplied with a scaling factor accounting for extra trained images of different resolutions induced by multi-crop augmentation. DINO and iBOT are by default trained with 2 global crops of size 224 x 224 and 10 local crops of size 96 x 96. Thus r = 2 + ( 96 )2 x 10 = 3.84 R 224
for DINO and iBOT. r & 3 for SwAV or DINO with RN50 as the backbone and pre-trained with global crops and 6 local crops. r = 2 for contrastive methods without multi-crop augmentation (e.g., MoCo, SimCLR, BYOL, etc.) and r for non-contrastive methods (e.g., BEÎT, Jigsaw, etc.).
C ADDITIONAL IMPLEMENTATIONS
Tine-Tunigg Recipes of Classification on ImageNet-IK. By default, we follow the fine-tuning protocol in BEiT (Bao et al., 2021) to use a layer- fuse learning rate decay, weight decay and AdamW optimizer and train small-, base-size models with 200, 100, and 50 epochs respectively. We sweep over four learning rates {8e 9c le-3, 2e 23-3). Comparatively, traditional fine-tuning recipe is is to fine-tune the network for 300 epochs with a learning rate 5e-1, no weight decay, and SGD optimizer (Touvron et al., 2021) (Row I versus 8). For a fair comparison, we compare the impact of different fine-tuning recipes with different methods, shown in Tab. 11. We empirically find that fine-tuning protocol used in BETT consistently yields better fine-tuning results and greatly reduces the training epochs. By default, we use layerwise decay of 0.75 with a training epoch of 200 for ViT-S/16, a layerwise decay of 0.65 with a training epoch of 100 for ViT-B/16, and a layerwise decay of 0.75 with training epoch of 50 for ViT-L/16. We report the higher results between using or not using DS since we find it brings different impacts to different methods.
Evaluation Protocols of Semi-Supervised Learning on ImageNet- We study the impact of different evaluation protocols for semi-supervised learning. Under conventional semi semi-supervised evaluation protocol, pre-trained models end-to-end fine-tuned with linear classification head.
15

---

Published as a conference paper at ICLR 2022
Table 11: Different fine-tuning recipes. LD de- notes layerwise learning rate decay. DS denotes mixed-precision training with DeepSpeed.
Table 12: Evaluation protocols for semi- supervised learning. Proj. denotes fine- tuning from the middle layer of the projec- tion head. LR denotes logistic regression.
Epo.
LD
DS
BEIT
DINO
iBOT
Method
Proj.
1%
10%
ViT-S/I6
frozen features
300
1.0
X
81.5
81.1
81.2
DINO
+ K-NN
61.3
69.1
2
300
0.75
/
81.7
82.0
82.3
iBOT
+ K-NN
62.3
70.1
3
200
0.65
X 80.7
3
DINO
+ Lin.
60.5
71.0
4
200
0.75
X 81.4
81.9
82.3
4
iBOT
+ +in.
62.5
72.2
5
200
0.75
81.4
82.0
82.2
DINO
+ +IR
64.5
72.2
6
200
0.85
X 81.2
iBOT
+ LR
65.9
73.4
ViT-B/16
end-to-end fine-tunee tiee-tuning
7
300
1.0
x
82.1
82.8
82.4
DINO
50.6
73.2
8
200
0.65
V
82.7
83.1
83.2
iBOT
55.0
74.0
9 100
0.65
X
83.4
83.5
84.0
DINO
60.3
74.3
10 100
0.65
V 83.2
83.6
83.8
iBOT
61.9
75.1
SimCLRv2 Chen al. (2020b) found that keeping the first layer of the projection head can improve accuracy, especially under the low-shot setting. We fine-tune the pre-trained model from the first layer of the projection head and verify this conclusion holds true for Vision Transformers. We empirically find that Vision Transformer performs better with a frozen backbone with 1% training data (62.5% in row 4 versus 61.9 % in row 7). In DINO, logistic regressor built upon the frozen features is found to perform better compared with the multi-class linear classifier upon the frozen features, especially with 1% data (65.9% in 6 versus 62.5% in row 4). When using 10% data, we empirically find that mn-to--end fine-tuning from the first layer of the projection layer yields the versus 73.4% in row 6).
best performance (75.1% 1% in row
Fine-Tuning Recipes of Object Detection and Instance Segmentation COCO. For both small- and base-size models, we utilize multi-scale training (resizing image with shorter size be- tween 480 and 800 while the longer side no larger than 1333), learning rate 1e-4, weight decay of 0.05, and fine-tune the entire network for 1x schedule (12 epochs with the learning rate decayed by 10x at epochs and 11). We sweep a layer decay rate of (0.65, 0.75, 0.8, 0.9]. Note that layer decay rate of 1.0 denotes no layer is decayed. To produce hierarchical feature maps, we use the features output from layer 4, 6, 8, and 12, with 2 deconvolutions, deconvolution, identity mapping, and max-pooling appended after, respectively. We do not use multi-scale testing.
Fine- Hine-Tuning Recipes of Semantic Segmentation on ADE20K. For semantic segmentation, we follow the configurations in BEiT (Bao al., 2021), fine-tuning 160k iterations with 512 512 images and a layer decay rate of 0.65. We do not use multi-scale training and testing. We sweep Similar object detection and instance segmen- tation, to produce hierarchical feature maps, we add additional deconvolution layers after ViT.
the learning rate (3e 85 -4 -e -4 Se-4)
DINO, w/o (LN] DINO, w/ [LN] iBOT, w/o [LN] iBOT, w/ [LN)
33.7
34.5
37.8
38.3
When using linear (Lin.) as the task layer, we find that appending the last LayerNorm ([LN)) ) for [CLS] token each patch tokens before the decoder consistently yields better performance, while we do not spot the substantial gain when with UperNet as the task layer. By default, we report the segmentation result with [LN] for both linear head for UperNet head.
Part-Wise Linear Probing. We use the average of the last-layer self-attention map with [CLS] as the query from multiple heads to rank all the patch tokens. We remove the extra LayerNorm (LN) after the final block following MoCov3 (Chen al., 2021).
D ADDITIONAL RESULTS
In this section, we provide detailed results for dense downstream tasks, i.e., object detection, instance segmentation, and semantic segmentation. We give the complete figures for occlusion robustness
16

---

Published as a conference paper at ICLR 2022
Table 13: Additional object detection, instance segmentation, and semantic segmentation re-
sults with small-size models. We pre-train iBOT with ViT-S/16 for 800 epochs.
Det. & Inst. Seg. w/ Cascade Mask R-CNN
Seg. w/ UperNet
Method
Arch.
Param.
Apb
AP
50
AP
Apm
AP
AP
mloU
mAcc
Sup.
Swin-T
29
48.1
67.1
52.5
41.7
64.4
45.0
44.5
MoBY
Swin-T
29
48.1
67.1
52.1
41.5
64.0
44.7
44.1
Sup. VIT-S/16
21
46.2
65.9
49.6
40.1
62.9
42.8
44.5
55.5
iBOT
VIT-S/16
21
49.4
68.7
53.3
42.6
65.6
45,8
45,4
56.2
Table 14: Additional object detection, instance segmentation, and semantic segmentation re- sults with base-size models. We pre-train iBOT with VIT-B/16 for 400 epochs.
Det. & Inst. Seg. w/ Cascade Mask R-CNN
Seg. w/ Lin.
Seg. w/ UperNet
Method
Apb
APL.
APp
Apm
APm
AP
mloU
mAcc
mloU
mAcc
Sup.
49.8
69.6
53.8
43.2
66.6
46.5
35.4
44.6
46.6
57.0
BEiT
50.1
68.5
54.6
43.5
66.2
47.1
27.4
35.5
45.8
55.9
DINO
50.1
69.5
54.3
43.4
66.8
47.0
34.5
43.7
46.8
57.1
iBOT
51.2
70.8
55.5
44.2
67.8
47.7
38.3
48.0
50.0
60.3
Table 15: k-NN and linear probing ImageNet-1K with different pre-training datasets.
Arch.
Pre-Train Data
Param.
Epoch
k-NN
Linear
ViT-S/16
ImageNet-IK
21
800
75.2
77.9
ViT-S/16
ImageNet-22K
69.3
76.5
ViT-B/16
ImageNet-1K
400
77.1
79.5
ViT-B/16
ImageNet-22K
85
80
71.1
79.0
ViT-L/16
ImageNet-IKK
307
300
78.0
81.0
VIT-L/16
ImageNet-22K
307
50
72.9
82.3
Table 16: Effectiveness of pre-trained features on nearest neighbor retrieval. We report the results different downstream tasks whose evaluation is based on nearest neighbor retrieval.
Image Retrieval
Vid. Obj. Segment.
Method
ROx
RPar
M
H
M
H
(J&F)) m
Jm
Fm
DINO
37.2
13.9
63.1
34.4
61.8
60.2
63.4
iBOT
36.6
13.0
61.5
34.1
61.8
60.4
63.2
analysis. We also provide extra experiments of nearest neighbor retrieval, robustness analysis against occlusion and shuffle.
Object Detection, Instance Segmentation, and Semantic Segmentation. We here provide more detailed results object detection, instance segmentation, and semantic segmentation with small- and base-size models, shown Tab. 13 and Tab. respectively. Specifically, we include API and AP for object detection, APm and APm, for instance segmentation, mAce for semantic segmenta- tion. For object detection (Det.) and instance segmentation (Inst. Seg.), we consider Cascade Mask R-CNN as the task layer. For semantic segmentation (Seg.), consider two evaluation settings where a linear head (Lin.) and UPerNet are taken as the task layer.
k-NN and Linear Probing with ImageNet-22K.. We further report I--N and linear probing accu- racy on ImageNet-IK with models pre-trained on ImageNet-22K dataset. We empirically observe that ImageNet-IK pre-training incurs better ImageNet-IK k-NN and linear probing performance, which opposite to the fine-tuning performance observed in Table and Table We hypothe- crucial rule under evaluation protocols based on frozen
size that the data distribution plays features, such that models pre-traine pre-trained with smaller ImageNet- 1K dataset consistently achieve better results.
17

---

Published as a conference paper at ICLR 2022
8.
062
04
04
123T
1E2T
1ES7. Lte.
LEST L10
LEST. Ltn.
63
10 01
02
055
04
08
0.3
02
Figure 9: Robustness against occlusion. Model's robustness against occlusion with different in- formation loss ratios is studied. 3 patch dropping settings: Random Patch Dropping (left), Salient Patch Dropping (middle), and Non-Salient Patch Dropping (right) are considered.
Nearest Neighbor Retrieval. Nearest neighbor retrieval is considered using the frozen pre- traierd features following the evaluation protocol as in DINO (Caron et al., 2021). DINO has demonstrated the strong potential of pre-trained ViT features to be directly used for retrieval. To validate, DINO designed several downstream tasks, including image retrieval and video object segmentation, where video object segmentation can be seen as dense retrieval task by finding the nearest neighbor be- tween consecutive frames to propagate masks. We compare iBOT with DINO on these benchmarks with the same evaluation settings, As demonstrated in Tab. 16, iBOT has comparable results with DINO. While iBOT has higher k-NN results on Imagenet-1K, 1K the performance is not better for iBOT in image retrieval. We empirically find that the results on image retrieval are sensitive to im- age resolution, multi-scale features, etc., and the performance varies using pre-trained models with minimal differences on hyper-parameter setup. For this reason, we do not further push iBOT for better results.
Robustness against Background Change. Deep models rely both foreground objects and back- grounds. Robust models should be tolerant to background changes and able to locate discriminative foreground parts. We evaluate this property on ImageNet-9 (IN-9) dataset (Xiao al., 2020). IN- 9 includes 9 coarse-grained classes and 7 variants by mixing up the foreground and background from different images. Ouly-FG (O.F., Mixed-Same (M.S.), Mixed-Rann (M.R.), and Mixed-Next (M.N.) are 4 variant datasets where the original foreground is present but the background is modi- fied, whereas No-FG (N.F), Only-BG-B (O.BB.,, and Only-BG-T (O.BT.) are 3 variants where the foreground is masked. As shown in Tab. 8, we observe performance gain except for O.BT., in- dicating iBOT's robustness against background changes. We note in O.BT. neither foreground nor foreground mask is visible, contradicting the pre-training objective of MIM.
Robustness against Occlusion. Masked prediction has a natural strength in cases where parts of the image are masked out since the models are trained to predict their original contents. We here provide the detailed results of occlusion with different information loss ratios in Fig. 9 under three dropping settings: random, salient, and non-salient. We showcase the results of iBOT end-to-end fine-tuned with linear head over the pre-trained backbone. We include the results of supervised results with both ViT-S/16 and ResNet-50 for comparison. ViT shows higher robustness compared to its CNN counterpart, i.e., ResNet-50, given that Transformers' dynamic receptive field makes it less dependent on images' spatial structure. We empirically find iBOT has stronger robustness against occlusion compared to its supervised baseline, consolidating that MIM help to model the in- teraction between the sequence of image patches using self attenttten such that discarding proportion of elements does not degrade the performance significantly.
Robustness against Shuffle. We study the model's sensitivity to the spatial structure by shuffling on input image patches. Specifically, we shuffle the image patches with different grid sizes follow- ing (Naseer et al., 2021). We showcase the results of iBOT end-to-end fine-tuned or with a linear head over the pre-trained backbone. We include the results of supervised results with both VIT-S/16 and Re for comparison. Note that a shuffle grid size of means no shuffle, and a shuffle grid size of 196 means all patch tokens are shuffled. Fig. 10 suggests that iBOT retain accuracy bet- ter than its supervised baseline and Re. It also indicates that iBOT relies less on positional embedding to preserve the global image context for right classification decisions.
18

---

Published as a conference paper at ICLR 2022
0.8
ImogeNee Vol 3cr01
0.0
0.3
Sup
0.1
1807,
Lin-
Sup..
196
Shatfle Grid Size
Figure 10: Robustness against shuffle. Model's robustness against shuffle with different grid shuf- fle sizes studied.
E ADDITIONAL ABLATIONS
In this section, we study the impact of other parameters that we have conducted experiments on. Without extra illustrations, we use 300-epoch pre-trained ViT-S/16, a prediction ratio r = 0.3 and without multi-crop augmentation for the ablative study.
Table 18: Hard label versus soft la-
B
bel. Cen. denotes centering. denotes smaller temperature for teacher output.
vanilla
+
Method Cen.
Post Proc.
k-NN
Lin.
B
X
softmax
49.8
63.5
[CL.S]
paich
[CLS]
pasch
X
hardmax
64.8
71.9
sharod head
X
softmax
69.4
73.9
softmax
67.8
72.9
/
hardmax
68.1
73.3
[CLS]
petch
[CLS]
panch
iBOT
/
softmax
69.1
74.2
semi-shured bead
DINO
67.9
72.5
Table 17: Different head sharing strategy.
Architecture of Projection Head. As mentioned earlier, shared head can transfer the semantics acquired in [CLS] token to patch tokens, slightly improving the performance. We notice that the head for patch tokens in the student network only see the masked tokens throughout the training, the distribution of which mismatches tokens with natural textures. Therefore, using a non-shared head for the student network but shared head for the teacher network denoted semi-shared head. Their differences demonstrated in Fig. 17, where S and denotes student and teacher network respectively. The heads with the same index and color denotes they have shared parameters.
conduct experiment
Arch.
vanilla
shared sm. shared sm. shared
shared
k-NN.
68.9
68.0
68.4
68.4
69.1
Lin.
73.9
73.7
73.7
73.8
74.2
i denotes only the first 2 layers out the 3-layer MLP share the parameters. However, we do not observe that semi-shared head is better than shared head. By default, we share the entire projection head for [CLS] token and patch tokens.
Comparing MIM with Dense Self-Distillation. To identify the superiority of MIM to model in- ternal structure using over its alternatives, conduct experiments performing self-distillation original patch tokens along with the [CLS] token. We consider two matching strategies to construct patch token pairs for self-distillation
19

---

Published as a conference paper at ICLR 2022
Arch.
DINO
DINO pos.
DINO + feat.
iBOT
k-NN 67.9
67.1 (-0.8)
68.5 (+0.6)
Lin. 72.5
72.5 (+0.0)
73.4 (+0.9)
74.2 (+17))
Specifically, pos. denotes matching according to the absolute position of two views. Similar to Xie al. (2021b). j is defined as arg min, (p., pf), where p the position in the original image space and dist(u, u, v) is euclidean distance. The losses are only computed for the overlapped regions of two views. We do not observe substantial gain brought by matching via patches' absolute position. feat. denotes matching according to the similarity of the backbone similarity of two views. Similar to Wang al. (2021b), we match for each patch token fi the most similar patch token from another view Jj. where j = arg max; sim(Ji,f)) sim(u, ( v) is cosine distance. Such practice brings a 0.6% performance gain in terms of linear probing accuracy, which is also observed by a concurrent work, Es (Li et al., 2021a). Comparatively, iBOT prompts an 1.2% gain on linear probing, verifying the necessity and advancement of MIM.
Hard Label versus Soft Label We study the importance of using a continuous token distribution (softmax') instead discretized id (hardmax) when performing MIM. Results in Tab. 18 indicate continuous tokenization plays crucial part. We empirically find the improvement brought by cen- tering, whose roles are less important compared to centering in self-distillation on [CLS] token. Only sharpening can produce a k-NN accuracy of 69.4 and a linear probing accuracy of 73.9.
Centering and Sharpening. Different from the [CLS] token, patch tokens do not have certain semantic cluster and vary more widely from each others. We study the impact of several critical parameters that decide the distillation process and customize them for distillation over the patch
tokens.
ml
.8
.99
.999
.9
.9
.9
T/
.04 .00
.04 + .07
.04 - .07
.04 > .06
.05 + .08
.04 -> .07
K-NN
68.7
68.8
68.9
68.5
68.7
69.1
Lin.
74.0
73.8
73.8
73.5
73.9
74.2
Specifically, the smoothing momentum for online centering m' and sharpening temperature T/ are studied. Note we keep the parameters for [CLS] token the same as DINO and only study for parameters for the patch tokens.
Loss Ratio.
We study the impact of different ratio between C Licss) and LMIM. We keep the base of C fors: to and scale LMIM with different ratios.
C ccs] / LMIM
0.5
2
1
k-NN
68.7
69.4
69.1
Lin.
73.8
74.1
74.2
We observe that directly adding two losses up without scaling yields the best performance in terms of linear probing accuracy.
Output Dimension. We follow the structure of projection head in DINO with 12-normalized bottle- neck and without batch normalization. We study the impact of output dimension K of the last layer.
K
4096
16384
8192
k--N
68.3
68.8
69.1
Lin.
74.5
74.0
74.2
While our method excludes large output dimensionality since each patch token has an output dis- tribution, we do not observe substantial performance gain brought by larger output dimensions. Therefore, we choose K = 8192 by default.
Prediction Ratios. Masked modeling is based on a formulation of partial prediction, the objective of which maximize the log- ligeliikloood of the target tokens conditioned on the non-target tokens. We experiment with different prediction ratios for masked image modeling. The results are shown in Fig. 11. We observe that the performance is not sensitive to variant prediction ratios between 0.05 and 0.4. Adding variance upon the fixed value can also consistently bring a performance gain, which can be explained as stronger data augmentation. The teacher output of non-masked images is now pulled together with the student output of masked images with different ratios. By default, we use 0.3 (=0.2) as the prediction ratio. For models with multi-crop augmentation. following the above discussions, we randomly choose prediction of 0 or 0.3 (+0.2) for each image.
20

---

Published as a conference paper at ICLR 2022
73
72
71
222
70
491
69-
MeDovd: MP-9
08
3726, 33:50
SvAV, 38-10
67
717-8/16
2.NN Linear
65
330
0.0
0.1
0.2
0.3
0.4
0.5
Namther Nprcke
Prediction Ratto
Figure 12: Impact of the training epochs. Models are ViT-S/16 with multi-crop aug- mentation.
Figure 11: Impact of the prediction ratio. - denotes randomly sample from a region.
Table 19: Time and Memory Requirements. We detail the actual training time (T) and GPU memory (Mem.) of different methods, together with their respective linear probing (Lin.) and fine- tuning (Fin.) accuracy. All methods are trained on two 8-GPU V100 machines with batch size of 1024.
Method
Crops Number
T100
T300
T Ts00
Mem.
Lin.300
Lin.soo
Fon
BETT
1 x 224²
11.3h
33.7h
90.1h
5.6G
20.7
24.2
81.4
DINO
2 2242
15.1h
44.7h
111.6h
9.3G
72.5
73.7
81.6
iBOT
2 x 2242
15.6h
47.0h
126.4h
13.1G
74.8
76.2
82.0
DINO
2 224 + 10 x 962
24.2h
72.6h
180.0h
15.4G
76.2
77.0
82.0
IBOT
x 224 + 10 962
24.3h
73.3h
193.4h
19.5G
77.4
77.9
82.3
Table 20: Methodology comparison over different approaches to tokenize the patches. We report ImageNet-1K K A-NN, linear and fine-tuning validation accuracy. Models are pre-trained with ViT-S/16 and 300 epochs.
Method
k-NN Linear Fine-Tune
Rand.
79.9
MPP (Dosovitskiy et al., 2021)
16.4
37.2
80.8
Patch Clustering
19.2
40.1
81.3
BEIT (Bao et al., 2021)
6.9
24.2
81.4
Standalone DINO tokenizer
44.3
60.0
81.7
iBOT
70.3
74.8
81.5
Training Epochs. We provide the linear probing top-1 accuracy with ViT-S/16 pre-trained for dif- ferent epochs. For comparison, we also include the accuracy curve of other methods with compa- rable numbers of parameters, i.e., ResNet-50. From Fig. 12, we observe that longer training for 800 epochs can improve the model's performance. It's north worthy that iBOT can achieve a Top-1 accuracy of SwAV (Caron al., 2020) mreitedined with 800 epochs in less than 100 epochs. iBOT pre-trained with 800 epochs brings a 0.9% improvement over previous state-of-the-as method.
Time and Memory Requirements. BEiT is trained with non-contrastive objective and without multi-crop augmentation, thus it consumes only memory of 5.6G and takes 90. 1h for 800 epochs. Comparing iBOT and DINO with multi-crop augmentation, iBOT with MIM induces 25% more memory requirements and 7.4% more actual training time. Considering pre-training efficiency (ac- curacy versus time), 800-epochs pre-trained DINO requiring for 180.0h, while 300-epochs iBOT only requires 73.3h with 0.4% higher linear probing accuracy (77.0 versus 77.4).
F ALTERNATIVE TOKENIZERS
To investigate how different approaches to tokenize the patches affect MIM, we study several al- ternatives. In BEiT (Bao et al., 2021), masked patches tokenized by DALL-E encoder. MPP
21

---

Published as a conference paper at ICLR 2022
(Dosovitskiy al., 2021) tokenizes the masked patches using their 3-bit mean color. For Patch Clustering, we first perform K-Means algorithm to the flattened color vector each 16 x 16 patch (d = 768). 10% data of ImageNet-Ik IK training set is sampled and clustered. We set K to 4096. During pre-training, each patch is tokenized by the index of its closest centroids. Lastly, we use 300- epoch pre-trained DINO output from the pre-trained DINO, We use average pooling to aggregate the patch representations. From Tab. 20, we see that all methods achieve decent fine-tuning results compared to the supervised baseline, while only methods tokenized by semantically meaningful tokenizer have proper results on k-NN and linear classification. MPP (Dosovitskiy et al., 2021) and patch clustering rely purely on offline statistics without the extra stage of online training. We find patch clustering has slightly better performance in all three protocols compared to MPP, suggesting the benefits brought by visual semantics. While BEiT has poor k-NN and linear probing accuracy, a good fine-tuning result also suggests relatively low requirements for fine-tuning protocol on high-level semantics.
standalone tokenizer. Each patch can be tokenized by the argmax of its
G VISUALIZATION
In this section, we first give more visualized pattern layouts and elff-attention maps. Beyond that, we consider an additional task of mining sparse correspondences between two images and illustrating the superiority of ViTs by showcasing several visualized results.
G.1 PATTERN LAYOUT
Pattern Layout for Patch Tokens. To illustrate versatile, interesting behaviors iBOT has learned, we organize the visualization of pattern layout in two figures. In Fig. 13, we mainly showcase additional pattern layouts that share high-level semantics. In Fig. 14, we mainly showcase additional pattern layouts that share low-level details like color, texture, shape, etc. Top 100 patches with the highest confidence over the validation set are visualized with a 5 x 5 context around each 16 x 16 patch token (colored orange).
Composing Images with Representative Patterns. In Fig. 15, visualize 4 patches with the highest self-attention score (with non-overlapped assigned index) and also show the pattern layout of that assigned index. The visualized results indicate iBOT can only be represented by several representative patches, which helps the model's robustness and performance in recognition. This is also validated by our part-wise linear probing experiments.
Comparison with Other Methods. We visualize pattern layout for patch tokens using other self- supervised methods (Bao et al., 2021; Caron et al., 2021) in Fig. 16. For BEiT, the DALL-E encoder generates discrete number for each patch token. For DINO, we directly the projection head for [CLS] token and generate a 65536-d probability distribution for each patch token. The index with the highest probability is assigned for the token.
Pattern Layout for [CLS] Token. We here also provide additional visualization of semantic pat- terns emerge in [CLS] token, which is obtained via self-distillation on cross-view images. We also observe similar behavior in DINO since it's not a unique property brought by MIM. In fact, seman- ties are now believed to emerge as long as a similarity between two distorted views of one image is enforced (Grill et al., 2020; He et al., 2020; Caron et al., 2020; 2018).
G.2 ATTENTIOON VISUALIZATIONS
Similar to the setting of Sec. 4.3.2, we here provided more self-attention map visualization from multiple heads the last layer in Fig. 18.
G.3 SPARSE CORRESPONDENCE.
We consider a sparse correspondence task where the overlapped patches from two augmented views of one image, or patches from two images labeled correlation is sparse since at most 14 x 14 matched pairs can be extracted with a ViT-S/16 model. We visualize 12 correspondences with the highest self-attention score extracted from iBOT with
class, required to be matched. The
22

---

Published as a conference paper at ICLR 2022
VIT-S/16 pre-trained for 800 epochs. The score is averaged between multiple heads of the last layer. Several sampled sets of image pairs are shown in Fig. 19. We observe empirically that iBOT perform well for two views drawn from one image, nearly matched the majority of correspondence correctly. the second column, iBOT can match different parts of two instances from the same class (e.g., tiles and windows of two cars) despite their huge differences in texture color. We observe the DINO also has comparable visualized effects, illustrating the representation prettrained with self-distillation also suits well for retrieval in patch-level scale.
23

---

Published as a conference paper at ICLR 2022
Figure 13: Visualization for pattern layout of patch tokens that share high-level semantics. In the first row, we visualize different human-related semantic parts. We observe clear patterns ac- counting for human hair, human shoulder & arm, and human elbow respectively in the left, middle, and right figure. In the figures from the second row and the left figure from the the third row, visualize animal-related semantic parts. dog 's ear, dog 's nose, bird's 's wing, and dragonfly's wing can be observed. In the rest of figures from the third row, we visualize semantic parts related outdoor scenes. front window of the vehicle and window of the architecture can be observed. In the last row, we visualize indoor objects like ceiling and glass bottle.
24

---

Published as a conference paper at ICLR 2022
Figure 14: Visualization for pattern layout of patch tokens that share low-level details. In the first two columns, we visualize patches that share similar textures. In the first figure, fur of leopard and the skin of lizard share similar dotted texture. In the second figure, shell of hedgehog and the skin of elephant share similar striped texture. In the third column, we visualize pattern layouts related to shape. For example, the shape of objects in the left and middle figures share similar curvature. The rightmost patterns clearly depict the shape of a straight line. We visualize pattern layout related to color in the last column, where blue, green and white can be observed.
25

---

Published as a conference paper at ICLR 2022
Samoyed
0 7407
QU0
857
7781
Library
1211
2
1188
Figure 15: Top-4 representative patches with each of their pattern layout. Order index 0. 1, 2, are ranked according its self-attention score. the top-left corner for each pattern layout subfigure, its order index and cluster index are annotated. In the top panel, we can observe that pattern 0,2.3 show explicit semantic information of nose, eyes, ears respectively. Interestingly, patch also locates around the eyes of the Samoyed but its corresponding pattern share visual similarity in shape instead semantics. This illustrate the diverse behaviour for each learned pattern. In the bottom panel, library is represented by 0 two- multi-color joints, 1,3 knurlling texture, 2 texts. Similarly, we have patterns 0,1.3 focusing more on texture & color and pattern 2 focusing more on semantics. All of these visualized results illustrate versatile behaviour for each index.
26

---

Published as a conference paper at ICLR 2022
BEiT
DINO
Figure 16: Visualization for pattern layout of patch tokens using BETT (top) and DINO (bot- tom). In the layout extracted from the DALL-E encoder, observe minimal semantic patterns. In most cases, patches with similar color (e.g., black area in left figure) or texture (e.g., line in right fig- ure) are clustered. the layout extracted from DINO, while more complex textures visible, most patches share similar local details instead of high-level semantics. In the right figure, the semantic part eyes can be somehow observed, yet is mixed with plenty of irrelevant semantic parts.
Figure 17: Visualization for pattern layout of [CLS] token. We here indicate the high quality of semantic layout brought by self-distille oelf-distillatinn of cross-view images on [CLS] ] token. This property is not brought by MIM and is also prominent in DINO.
27

---

Published as a conference paper at ICLR 2022
DINO
1BOT
IOTT
Figure 18: Visualization for self-attention map from Multiple Heads. In the first columns, we showcase iBOT's attention map along with DINO's. In the last 10 columns, we showcase more attention map from iBOT. We indicate that iBOT shows visually stronger ability to separate different objects different parts of one object apart by giving more attentive visualized results for each part, compared with DINO. For example, in the fifth column, there is an attention head in iBOT accounting for the of the fox solely, while in DINO, it emerges with other parts; In the eighth column, iBOT separates the mushroom into more semantically meaningful parts.
28

---

Published as a conference paper at ICLR 2022
Conrespondence between two views of one image
Correspondence between two images of one class
Figure 19: Visualization for sparse correspondence. The top panel are images pairs sampled from two views of one image. The extracted correspondence from iBOT mostly correct despite augmentations scale and color. The bottom panel are image pairs sampled from two images of one class. The first row is images with salient objects but different sizes, positions and textures. The second are images draw from animals, and semantic parts of animals correctly (e.g., tails of the fox, beak of the bird). The third row human- centered images with human bodies clothing. The fourth row is natural or domestic scenes where salient objects are invisible. Although no explicit semantic parts can be matched visible to human's 's understanding, can still observe the iBOT can extract correspondence based their texture or color (e.g., wooden texture of signboard and boxes. All these visual results demonstrate strong capability for iBOT in part retrieval or matching local scale.
observe more clearly that iBOT matches the
29


Reviews:
[{"review_id": "iG5QddmaavW", "reviewer": "Reviewer_WgwF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- **Writting:** The paper is well written\n- **Simplicity:** The method is not a significant change with the previous approaches, but the method is simple and give good results. \n- **Ablations and Analysis** Provide elements to understand the impact of IBoT modifications compared to a more traditional SSL approach like DINO[1].\n\nWeakness:\n- **Comparison with others approaches**: Table 1 and 2, It would be interesting to have a cost comparison of pre-training approaches. Indeed, the number of parameters and the number of epochs used is not very useful. Indeed, using for example multi-crop increases the number of FLOPs and the memory used. Moreover, contrastive approaches using 2 or more different data-augmentation per image in each batch have longer epochs than approaches like BeiT. It would be interesting to try to make a more accurate comparison.\n\n-  **Comparison with supervised baselines for full fine-tuning setting**:  The number of epochs used for the supervised baseline and for the self-supervised approaches are not the same which makes the comparison quite unfair. It would be interesting to compare the performance by adapting the number of epochs in order to better measure the gain compared to the supervised model. \n\n[1] Caron et al., Emerging Properties in Self-Supervised Vision Transformers, ICCV 2021", "summary_of_the_paper": "The paper proposes the iBOT method. This approach is inspired by the contrastive self-supervised learning approach like DINO and the mask modelling approach like BeiT. The idea is to use an online tokenizer instead of a pretrained tokenizer like Beit. The iBoT approach combines a loss at the patch level like Beit and a global loss at the image level like DINO. IBOT is evaluated in image classification, detection and segmentation.", "summary_of_the_review": "The paper is quite well written, the method is simple and gives better performance than DINO. Ablations and analysis are quite interesting. Nevertheless, the comparison between the different approaches does not seem to be complete enough. So, it is difficult to see if the iBoT approach brings a significant gain.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hRFiZ_nYnx", "reviewer": "Reviewer_s5wd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strength\n- The MIM for self-supervised feature learning is well motivated and technically sound. \n- Noticeable improvement on downstream tasks like detection and segmentation over the strong baseline DINO with the proposed MIM loss.\n- Extensive experimental results and visualization are given to show the effectiveness of the proposed mask image modeling objective and iBoT model. The authors also ablate different tokenization methods and different loss components to validate the contribution. \n- Detailed results and training settings are listed which can be helpful for future research.\n\n### Weaknesses\n- The improvement of the fine-tuned model in Tab 2 is limited. It would be better if the authors could explain more on this part.\n\n- The experimental results on the downstream tasks look good and the authors also provide some insights, but the technical contribution is limited. \n\n- The main difference between DINO and proposed iBoT is 1) masked image as input for the student model; 2) MIM loss for self-distillation on reconstructed patch tokens. Table 8 and section E in the appendix only provide ablation on the tokenization method, loss function, and dense self-distillation. To be fully convinced by the MIM loss, I think the author should also provide the result of the iBoT model without MIM loss (i.e. use the masked image as input for the student model of DINO). Since applying the mask on the input image is similar to Cutout augmentation, it can also improve the robustness and the performance of the trained model. \n\n- It would be better if the authors provide more comparisons on the proposed online tokenizer and pre-trained tokenizer (as in BEiT). It's not clear to me why the proposed online tokenizer can better capture the semantically meaningful pattern.\n\n- As multi-crop augmentation is important for both Swav and DINO, I think it’s also important to explain and analyze more on how to apply multi-crop augmentation with mask image modeling in the main paper. Also, the training costs (actual training time) are not listed. Is there any additional training cost for applying masked image modeling?\n", "summary_of_the_paper": "This paper presents a new strategy for self-supervised training of vision transformer models by combining the self-distillation-based method DINO with mask image modeling. The momentum teacher model in DINO is taken as an online tokenizer. A mask image modeling loss that computes self-distillation loss between the reconstructed tokens and the output tokens of the momentum teacher is proposed. Extensive experiments are conducted to show the effectiveness of the proposed self-supervised training framework. The result on downstream tasks including classification, object detection, and segmentation also shows the benefit of the mask image modeling loss.", "summary_of_the_review": "Overall, I think this is a practical method with good experimental results. However, the proposed mask image modeling loss may not be clearly verified. As its current state, I would like to rate this paper as marginally below the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JO-BGgrxBX3", "reviewer": "Reviewer_RoXr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clearly written, I believe I could reimplement the method from the paper (using the appendix). \nIt contains plenty of ablations that answer most of my questions about variants and details of the model.\nThe experimental results are (very) competitive, the comparisons are relevant, the litterature is adequately discussed. \n\nMaybe a missing interesting comparison would have been BYOL with a VIT model. \nThe visualizations are convincing (in the main body but also in the appendix), although the weaker point (which is not a major flaw) of the paper is probably section 4.3.1, where it is not very clear how the learned patterns differ really from BEiT or DINO, and more to the point, how the help from this training procedure in image recognition is of any different kind than from the competition.\n", "summary_of_the_paper": "This is one of the first papers doing BERT-style pretraining for vision transformers, in particular the first to not use a pre-trained tokenizer (here the (part of) the network that generates the targets for the masked tokens). This has a similarity with BEiT, although BEiT uses an externally procured tokenizer (CLIP, so trained with labels). Instead, here, the idea is to use a teacher (EMA from student) to propose the targets for the masked tokens, yielding a masked image modeling task (L_MIM). The whole model and approach works with 2 augmented views of the same image (as in BYOL or DINO) and also has a self-distillation loss (L_CLS). L = L_MIM + L_CLS. The paper presents state of the art results in kNN (retrieval), image classification with linear probing and fine-tuning (ImageNet), image detection, instance segmentation (both on COCO), and semantic segmentation (ADE20K), within the classes of models that the authors studied.\n", "summary_of_the_review": "This is a well presented paper that shows convincing progress in masked image modeling pretraining for vision transformers. It should be accepted for publication at ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is interesting, and its focus is timely and important, given the continuing rapid rise of transformers (and their dependence of tokenization of images). All three reviewers recommend acceptance, to varying degree. The paper will be a valuable contribution to the program at ICLR.
Decision: Accept (Poster)