BEiT: BERT Pre-Training of Image Transformers
Authors: Hangbo Bao
Li Dong
Songhao Piao
Furu Wei
Venue: ICLR 2022 Oral (2022)
Keywords: self-supervised learning
pre-training
vision Transformer

Abstract: We introduce a self-supervised vision representation model BEiT, which stands for Bidirectional Encoder representation from Image Transformers. Following BERT developed in the natural language processing area, we propose a masked image modeling task to pretrain vision Transformers. Specifically, each image has two views in our pre-training, i.e., image patches (such as 16 x 16 pixels), and visual tokens (i.e., discrete tokens). We first ``tokenize'' the original image into visual tokens. Then we randomly mask some image patches and fed them into the backbone Transformer. The pre-training objective is to recover the original visual tokens based on the corrupted image patches. After pre-training BEiT, we directly fine-tune the model parameters on downstream tasks by appending task layers upon the pretrained encoder. Experimental results on image classification and semantic segmentation show that our model achieves competitive results with previous pre-training methods.

Published as a conference paper at ICLR 2022
BEIT:
BERT PRE-TRAINING OF IMAGE TRANSFORMERS
Hangbo i Li Dong', Songhao Piaot, Furu Wei!
i Harbin Institute of Technology
Microsoft Research
ABSTRACT
We introduce a self-supervised vision representation model BEIT, which stands for Bidirectional Encoder representation from Image Transformers. Following BERT (Devlin al. 2019) developed in the natural language processing area, we propose a masked image modeling task to pretrain vision Transformers. Specifi- cally, each image has two views in our pre-training, i.e., image patches (such 16 x 16 pixels), and visual tokens (i.e., discrete tokens). We first "tokenize" the original image into visual tokens. Then we randomly mask some image patches and fed them into the backbone Transformer. The pre-training objective is to recover the original visual tokens based on the corrupted image patches. After pre-training BErT, we directly fine-tune the model parameters on downstream tasks by appending task layers upon the pretrained encoder. Experimental results image classification and semantic segmentation show that our model achieves competitive results with previous pre-training methods.
INTRODUCTION
Transformer Vaswani et al. 2017) has achieved promising performance in computer vision (Dosovit- skiy et al., 2020) Touvron et al. (020). However, empirical studies show that vision Transformers require more training data than convolutional neural networks. In order to solve the data-hungry issue (Liu et al. 2021a), self-supervised pre-training is a promising solution to leverage large-scale scale im- age data. Several strands of methods have been explored for vision Transformers, such contrastive
learning (Chen et al. 2021 Xie et al. 2021), and self-distillation (Caron et al. 2021).
Concurrently, BERT (Devlin et al. 2019) has achieved great success in natural language processing. Its masked language modeling task first randomly masks some proportion of tokens within a text, and then recovers the masked tokens based on the Transformer encoding results of the corrupted text. Motivated by BERT, we turn to the denoising auto-encoding idea to pretrain vision Transformers, which has not been well studied by the vision community. is challenging to directly apply BERT- style pre-training for image data. First of all, there is no pre-exist vocabulary for vision Transformer's 's input unit, i.e., image patches. So we cannot simply employ a softmax classifier to predict over all possible candidates for masked patches. In contrast, the language vocabulary, such as words and BPE (Sennrich et al. 2016), is well-defined and eases auto-encoding prediction. straightforward alternative is regarding the task as regression problem, which predicts the raw pixels of masked patches. However, such pixel-level recovery task tends to waste modeling capability on pre-training short-range dependencies and high-frequency details (Ramesh et al. 202 Our goal is to overcome the above issues for pre-training of vision Transformers.
In this work, we introduce self sulf-super vision representation model BEIT, which stands for Bidirectional Encoder representation from Image Transformers. Inspired by BERT, we propose a pre-training task, namely, masked image modeling (MIM). As shown in Figure [ MIM uses two views for each images, i.e., image patches, and visual tokens. We split the image into grid of patches that are the input representation of backbone Transformer. Moreover, "tokenize" the image to discrete visual tokens, which is obtained by the latent codes of discrete VAE (Ramesh et al. 2021).
"Contribution during internship Microsoft.

---

Published as a conference paper at ICLR 2022
Visual Tokens
Unused During
Reconstructed
Pre-Traiming
Image
123
234
458
567
967
878
765
543
Original Image
Tokenizer
112
334
Decoder
211 322 433
844
234 456
876 765
322
Image
Masked Image Modeling Head
Patches
h1
h2
bg
np
Blockwise
BEIT Encoder
Masking
Pasition
2
5
6
8
9
10
11
12
13
14
15
16
Embedding
Flatten
[M]
[M]
[M]
[M]
Patch Embedding
Figure I: Overview of BEIT pre-training. Before pre-training, learn an "image tokenizer" via autoencoding-styy reconstruction, where an image is tokenized into discrete visual tokens according the learned vocabulary. During pre-training, each image has two views, i.e., image patches, and visual tokens. We randomly mask some proportion of image patches (gray patches in the figure) and replace them with a special mask embedding [M]. Then the patches are fed a backbone vision Transformer. The pre- pre-training task aims at predicting the visual tokens of the original image based on the encoding vectors of the corrupted image.
During pre-training, we randomly mask some proportion of image patches, and feed the corrupted input to Transformer. The model learns to recover the visual tokens of the original image, instead of the raw pixels of masked patches.
We perform self-supervised learning and then fine-tune the pretrained BETT on two downstream tasks, i.e., image classification, and semantic segmentation. Experimental results indicate that BETT outperforms both from-scratch training and previous strong self-supervised models. Moreover, BEIT is complementary to supervised pre-training. Performance of BEIT can be further improved by intermediate fine-tuning with ImageNet labels. Ablation studies show that our proposed techniques critical to the effectiveness of BERT-style pre-traine training for image data. Apart from performance, the improvements of convergence speed and stability of fine-tuning reduce training costs on end tasks. In addition, demonstrate that self-supervised BEIT can learn reasonable semantic regions via pre-training, unleashing the rich supervision signals contained images.
Our contributions summarized follows:
We propose a masked image modeling task to pretrain vision Transformers self-supervised manner. We also provide a theoretical explanation from the perspective of variational autoencoder.
We pretrain BEIT and conduct extensive fine-tuning experiments on downstream tasks, such as image classification, and semantic segmentation.
We present that the self-attention mechanism of self-supervised BEIT learns distinguish semantic regions and object boundaries, although without using any human annotation.
METHODS
Given input image x, BEIT encodes it to contextualized vector representations. As shown in Figure [ BEIT is pretrained by the masked image modeling (MIM) task in a sulf-super learning manner. MIM aims recovering the masked image patches based encoding vectors. For
2

---

Published as a conference paper at ICLR 2022
downstream tasks (such image classification, and semantic segmentation), we append task layers upon pretrained BEIT and fine-tune the parameters on the specific datasets.
2.1 IMAGE REPRESENTATIONS
The images have two views of representations in our method, namely, image patch, and visual tokens. The two types serve as input and output representations during pre-training, respectively.
2.1.1 IMAGE PATCH
The 2D image is split into a sequence of patches (Dosovitskiy et al. 2020). so that a standard Transformer can directly accept image data. Formally, we reshape the image
IR Hxxxxx into
N HWW /2² patches 2x E R=Nx(P"C) where C' is the number of channels, (H, W) is the input image resolution, and (P. P) is the resolution of each patch. The image patches {x. Ji=i are flattened into vectors and are linearly projected, which similar to word embeddings in BERT (Devlin et al., 2019). Image patches preserve raw pixels and are used as input features in BEIT.
Pi N
In our experiments, we split each 224 x 224 image into 14 x 14 grid image patches, where each patch is 16 x 16.
2.1.2 VISUAL TOKEN
Similar to natural language, we represent the image as a sequence of discrete tokens obtained by an "image tokenizer", instead of raw pixels. Specifically, we tokenize the image E TR "xxxx into Fi,..., [N] Vix where the vocabulary V (I,..., 1 contains discrete token indices.
Following (Ramesh et al., 2021), we use the image tokenizer learned by discrete variational autoen- coder (dVAE). There are two modules during visual token learning, namely, tokenizer and decoder. The tokenizer qo (z a²) maps image pixels æ into discrete tokens z according to visual codebook (i.e., vocabulary). The decoder pv (z) =) learns reconstruct the input image ær based on the visual tokens z. The reconstruction objective can written as (x)))(()))]] [l (x]z))] Because the latent visual tokens are discrete, the model training is non-differentiable. Gumbel-softmax relaxation (Jang 2017 Maddison et al.) (017) is employed to train the model parameters. Moreover, a uniform prior put on 2o during dVAE training. Refer to (Ramesh al. 2021 for more training details of the image tokenizer.
et al.
We tokenize each image 14 14 grid of visual tokens. Notice the number of visual tokens and the number of image patches for one image are the same. The vocabulary size is set to [ 8192. In work, we directly use the publicly available cinilabbemmmmeee tokenizer described in (Ramesh al.
2021.
We also compare it with re-implemented tokenizer in Appendix C
2.2 BACKBONE NETWORK: IMAGE TRANSFORMER
Following ViT (Dosovitskiy al. 2020), we use the standard Transformer (Vaswani et al., 2017) as the backbone network. So the results can be directly compared with previous work in terms of the network architecture.
The input of Transformer is sequence of image patches J The patches are then lincarly projected to obtain patch embeddings Exf, where E (P*C)D Moreover, we prepend special token [S] to the input sequence. We also add standard learnable ID position embeddings Epos E R X to patch embeddings. The input vectors Ho = [e]s] Ex!... Ex [1 Epos fed into Transformer. The encoder contains layers of Transformer blocks H' Transformer ( where / = I,..., L. The output vectors of the last layer H- [f 1f. ....... h are used as the encoded representations for the image patches, where h is the vector of the i-th image patch.
https://github. -iaienaaa//ALLLaaaa /
3

---

Published as a conference paper at ICLR 2022
2.3 TRE-NIAINING BEIT: MASKED IMAGE MODELING
We propose a masked image modeling (MIM) task. We randomly mask some percentage of image patches, and then predict the visual tokens that corresponding the masked patches.
Figure D shows the overview of our method. As presented in Section 2.1 given an input image 2, we split it into N image patches Ji=1) and tokenize it to N visual tokens (fz) ) We randomly mask approximately 40% image patches, where the masked positions are denoted as M E ,L....,AA JJ.4N, Next we replace the masked patches with learnable embedding en] RD The corrupted image patches x = {
Ufe : E M are then fed into the L-layer Transformer as described in Section 2.2 The final hidden vectors {
M
encoded representations the input patches. For each masked position (h! : fi=1 are regarded as M)
Ji=1 we use softmax classifier to predict the corresponding visual tokens PMIM (MIM(2"(x>M) = softmax,/ (Wohf +b.).
where xM is the corrupted image, We E IR V and be E R R The pre- teeiiriinig objective is to maximize the log-likelihood of the correct visual tokens given the corrupted image:
Σ EM
Σ LIEM
log PMgPPMM22 (
max
#ED
where the training corpus, M represents randomly masked positions, and xM is the corrupted image that is masked according to ...
Rather than randomly choosing patches for the masked positions MM. we employ blockwise masking in our work. As sum- marized in Algorithm block of image patches is masked each time. For each block, we set the minimum number of patches to 16. Then we randomly choose an aspect ratio for the masking block. We repeat the above two steps until obtaining enough masked patches, i.e., 0.4,,, where N is the total number of image patches, and 0.4 is masking ratio.
Algorithm Blockwise Masking
Input: N(= h x w) image patches Output: Masked positions M M < 0
repeat
Rand(16, 0.4N (M])
Block size & Aspect ratio of block
(0.3 1
/s r; v v/Tr
I Rand(0,
a) ;l e Rand(o, w b))
M < MM (t. (t, E [1, b)}
until [M > 0.4N
b Masking ratio is 40%
return M
The MIM task is greatly inspired by masked language modeling (Devlin al. 2019). which is one of the most successful pre-training objective natural language processing. Moreover, blockwise (or n-gram) masking is also widely applied in BERT-like models (Joshi et al. 2020 Bao et al. 2020 Raffel et al.] (2220). However, directly using pixel-level auto-encoding (i.e., recovering the pixels of masked patches) for vision pre- pre-training pushes the model to focus short-range dependencies and high-frequency details (Ramesh et al.) 2021). BITT overcomes the above issue by predicting discrete visual tokens, which summarizes the details high-level abstractions. Ablation studies in Section 3.3 show that our proposed method significantly outperforms pixel-level auto-encoding.
2.4
FROM THE PERSPECTIVE OF VARIATIONAL. AUTOENCODER
The BEIT pre-training can be viewed as variational autoencoder (Kingma Welling 2014) training. Let denote the original image, f the masked image, and the visual tokens. Considering the evidence lower bound (ELBO) of the log- likeliooldodd p(ar (() ) i.e., recovering the original image from its corrupted version:
log,(z;(2;)>
Σ
E (=,,gg(()) Pa - (oz
(EDD))ED
(DDDD
Visual Token Reconstruction
where (1) (2(x)) denotes the image tokenizer that obtains visual tokens; (2) p (x|2) decodes the original image given input visual tokens; (3) po(=(it) recovers the visual tokens based the masked image, which is our MIM pre-training task.
We learn the model following a two-stage procedure similar to (van den Oord et al. 2017: Razavi jet al.) 2019). In the first stage, we obtain the image tokenizer as a discrete variational au- toencoder (Ramesh al. 2021). Specifically, the first stage minimizes the reconstruction loss
4

---

Published as a conference paper at ICLR 2022
(x)on) [lo Pe [x)] with uniform prior as described in Equation (2) In the second stage, we learn the prior pe while keeping qs and pe fixed. We simplify (s(z/2;) ) to one-point distribution with the most likely visual tokens & = arg max, (0(2)24). Then Equation can be rewritten as:
Σ
(Eargs(Hlaa) [lo Pv (z)]
log (2
(E.
Stage I: Visual Token Reconstruction
Stage Image Modeling
where the second term is our BEIT pre-training objective.
2.5 TRAINIINNG SETUP
The network architecture of BEIT follows that of ViT-Base (Dosovitskiy al.] 2020) for fair comparison. We use a 12-layer Transformer with 768 hidden size, and 12 attention heads. The intermediate size of feed-forward networks is 3072. We employ the default 16 16 input patch size. We directly borrow the image tokenizer trained by Ramesh et al. (202I). The vocabulary size of
visual tokens is 8192.
We pretrain BEIT on the training set of ImageNet-IK (Russakovsky et al., 2013), which contains about 1.2M images. Our augmentation policy includes random resized cropping, horizontal flipping. color jittering (Wu et al. (218)). Notice that we do not use the labels for self- sef--supeeeiee learning. We use the 224 x 224 resolution our experiments. So the input split to x 14 image patches, and the same amount of visual tokens. We randomly mask at most 75 patches (i.c., roughly 40% of total image patches).
The pre-training runs for about 500k steps (i.e., 800 epochs) with 2k batch size. Adam (Loshchilov & Hutter (019) with 31 = 0.9, 32 0.999 is employed for optimization. The learning rate is set to 1.5e-3, with warmup of 10 epochs, and cosine learning rate decay. The weight decay is 0.05. We employ stochastic depth (Huang et al. 2016) with a 0.1 rate, and disable dropout. The 500k training steps take about five days using 16 Nvidia Telsa V100 32GB GPU cards.
We find that proper initialization is important to stabilize Transformer, especially for large-scale pre- training. We first randomly initialize all the parameters within small range, such as [-0.02, 0.02]. Then, for the 1-th Transformer layer, we rescale the output matrices (i.e., the last linear projection within each sub-layer) of the self-attention module and the ferd-fr network by V27
2.6 FINE-TUNING BETT ON DOWNSTREAM VISION TASKS
After pre-training BETT, we append a task layer upon the Transformer, and fine-tune the parameters on downstream tasks, like BERT. We take image classification and semantic segmentation examples in our work. It is straightforward to leverage the pre- pve-training-tee then--ine-tunnee paradigm on other vision tasks with BErT.
Image classification. For image classification tasks, we directly employ a simple linear clas- sifier as the task layer. Specifically, we use average pooling to aggregate the representa- tions, and feed the global to a softmax classifier. The category probabilities are computed softmax(avge) ( J Ji=1 Wc)), where h is the final encoding vector of the i-th image patch, W. E T DxC is parameter matrix, and C is the number of labels. We maximize the likelihood of labeled data by updating the parameters of BETT and the softmax classifier.
Semantic segmentation. For semantic segmentation, we follow the task layer used in SETR- PUP (Zheng al.) 20220). To be specific, we use pretrained BEIT a backbone encoder, and incorporate several deconvolution layers as decoder to produce segmentation. The model is also end-to-end fine-tuned similar to image classification.
Intermediate fine-tuning. After self-supervised pre-training we can further train BETT on a data- rich intermediate dataset (i.e., ImageNet- IK in our work), and then finetune the model on the target downstream tasks. Such intermediate fine-tuning is the common practice of BERT fine-tuning in NLP (Pruksachatkun al. 2020). We directly follow the method for BErT.
5

---

Published as a conference paper at ICLR 2022
3 EXPERIMENTS
We conduct full fine-tuning experiments on image classification and semantic segmentation. Moreover, we present various ablation studies for pre-training and analyze the representations learned by BEIT. We also report linear probes on ImageNet in Appendix D
3.1 IMAGE CLASSIFICATION
The image classification task classifies input images various categories. We evaluate BEIT on the ILSVRC-2012 ImageNet dataset (Russakovsky et al.] 2013) with 1K classes and 1.3M images. We directly follow the most of hyperparameters of DeiT (Touvron et al., 2020) in our fine-tuning experiments for fair comparison. We reduce fine-tuning epochs compared with training from scratch, as BErT has been pre-trained. Accordingly, we use larger learning rate with layer-wise decay. The
detailed hyperparameters are summarized in Appendix H
Table reports top-1 accuracy on image classification. We compare BETT with vision Transformers trained by random initialization, supervised pre-training. and previous self-supervised learning methods. All the compared models are base-size, except iGPT has 1.36B parameters. Pre-training is conducted on ImageNet for the comparison purpose, except ViT-JFT300M is pretrained on Google's in-house 300M images.
Compared with the models trained by random initialization, find that pre-trained BEIT signifi- cantly improves performance on both datasets. BEIT improves the performance on ImageNet, which shows the effectiveness under the rich-resource setting.
Morcover, we compare BIT with previous state state-of-the-arr supervied methods for Transformer, such as DINO (Caron al. 2021), and MoCo v3 (Chen al., 202I). Our proposed method outperforms previous models on ImageNet fine-tuning. Among them, iGPTT-133B (Chen et al. 2020a) uses much more parameters (i.e., 1.36B vs 86M), and ViT-JFT300M (Dosovitskiy et al. 2020) is pretrained on larger corpus (i.e., 300M vs 1.3M), while others pretrain ViT-Base on ImageNet- IK iGPT-1.36B and ViT-JFT300M are the most comparable methods, which also follows auto-encoding pre-training for vision Transformer. Specifically, iGPT uses clustered image tokens as both input and output for image GPT image BERT. contrast, we use image patches input to preserve raw pixels, and employ discrete visual tokens as prediction bottleneck. ViT-JFT300 predicts the mean, 3-bit color of each masked patch, rather than visual tokens learned by discrete VAE. We also pretrain the self- eeffsuuerviied tasks of B ETT and DINO in multi-task learning manner, which is presented in Appendix
In addition, we evaluate our proposed method with intermediate fine-tuning. In other words, we first pretrain BEIT a self-supervised manner, and then fine-tune the pretrained model on ImageNet with labeled data. The results show that BEIT is complementary to supervised pre-training, achieving additional gain after intermediate fine-tur tuning on ImageNet.
Fine-tuning to 384 x 384 resolution. After fine-tuning with resolution 224 x 224, we additionally fine-tune the model on 384 384 images by 10 more epochs. We follow the standard higher- reshuurresolttion setting of DeiT Toourro et al. 2020), except using fewer epochs. Notice that we keep patch size the same for both 224 224 and 384 384 images. So the input sequence length of Transformers becomes longer for higher resolutions. Table [ shows that higher resolution improves the BEIT results by 1+ points on ImageNet. More importantly. B pretrained on ImageNet- IK even outperforms supervised pre-training ViT that uses ImageNet-22K, when they use the same input resolution.
Scaling up to larger size. We further scale up BETT to the large size (same as ViT-L., As shown in [able[[] ViT is worse than ViT on ImageNet, when training from scratch. The results verifies the data-hungry issue of vision Transformers. Supervised pre-training on ImageNet-22K partially relieves the issue, where 3iT finally outperforms ViTa34 by 1.2. In comparison, BEIT-L is better than BEIT by 2.0, and B BETTasI-L outperforms BETT by 1.7. other words, the benefits of scaling up BEIT from base to large greater than supervised pre-training with ImageNet-22K. More importantly, comparing between BETTas4 with ViTas4 that conducts supervised pre-training on ImageNet-22K,, the improvements of BETT become greater along with scaling the size from base

---

Published as a conference paper at ICLR 2022
Models
Model Size
Resolution
ImageNet
Training from scratch (i.e., random initialization)
VBT
Dosovitskiy et al.
2020
86M
3842
77.9
ViT
Dosovitskiy al.
2020
307M
384²
76.5
DeïT-B Touvron et al. 2020)
86M
2242
81.8
DeiT (Touvron al. 2020)
86M
3842
83.1
Supervised Pre-Training on ImageNet-22K (using labeled data)
3B Dosovitskiy et al. 2020
86M
3842
84.0
3iT Dosovitskiy al. 2020
307M
3842
85.2
Self-Supervised Pre-Training on ImageNet-IK (without labeled data)
iGPT-1.36B (Chen et al.
2020a)
1.36B
224
66.5
3iT:s--B-JFT3000. (0R0000-Dsoooo (Dosovisskiy et al.
2020
86M
3842
79.9
MoCo v3-B Chen et al.
2021
86M
2242
83.2
MoCo v3-L (Chen 2021
2021
307M
224
84.1
DINO-B CCron et al.
86M
224
82.8
BETT-B (ours)
86M
224
83.2
BEIT3s1 3B1 (ours)
86M
3842
84.6
BErT-L (ours)
307M
224
85.2
3ErTys4-L (ours)
307M
3842
86.3
Table Top-1 accuracy on ImageNet-IK. We evaluate base- ("-B") and large-size ("-L"" models resolutions 224 x 224 and 384 x 384. 1: iGPT-1.366 contains 1.36 billion parameters, while others base-size models. &; B-JFTT-- pretrained with the "masked patch prediction" task on Google's in-house 300M images, while others use ImageNet.
80
Models
ADE20K
75-
Supervised Pre- Tre-Training on ImageNet
45.3
70
DINO (Caron et al. 2021)
44.1
DeiT (Training from scratch)
BEIT (ours)
45.6
65
BEIT (Fine-tuning)
BEIT Intermediate Fine- Fine-Tuning (ours)
47.7
60
160
150
200
300
Epochs
Table 3: Results of semantic segmentation on ADE20K. We use SETR-PUP (Zheng al.) (0220) as the task layer and report results of single-scale inference.
Table 2: Convergence curves of training DeiT from scratch and fine-tuning BETT on ImageNet-IK.
(i.e., 0.6) to large (i.e., 1.1). The results suggest that BETT tends to help more for extremely larger models (such as IB, or 10B), especially when labeled data are insufficien to conduct supervised pre-training
Ei for such large models.
Convergence curves. Figure compares the convergence curves of the training-from- scratcc and me-training--hen- -hee--iin--uuiingg paradigms. find that fine-tuning BETT not only achieves better performance, but also converging much faster than training DeiT from scratch. Moreover, fine-tuning BEIT can reach reasonable numbers within very few epochs.
ZZaai et al. 2021 report that supervised pre-trainin vee-training of 1.8B-size vision Transformer requires billions of labeled images.
Apppendi shows that BEIT tne-tuned on ImageNet 22K (14M) can match the performance supervised pre- vre-trinnig on Google's ho-ho JFT-3B (Zhai et al. 2021). while using 214x less labels. We also demonstrate that large-size BEIT fine-tuned on 70M labeled images can achieve 89.5% top-1 accuracy ImageNet and 58.4% mloU ADE20K, creating new state-of-the-art results for large-size vision Transformers.
7

---

Published as a conference paper at ICLR 2022
Models
ImageNet
ADE20K
BErT (300 Epochs)
82.86
44.65
Blockwise masking
82.77
42.93
Visual tokens (i.c., recover masked pixels)
81.04
41.38
Visual tokens Blockwise masking
80.50
37.09
+ Recover 100% visual tokens
82.59
40.93
Masking Recover 100% visual tokens
81.67
36.73
Pretrain longer (800 epochs)
83.19
45.58
Table 4: Ablation studies for BEIT pre-training on image classification and semantic segmentation.
3.2 SEMANTIC SEGMENTATION
Semantic segmentation aims to predict a corresponding class for each pixel of the imput image. We evaluate BEIT the ADE20K benchmark (Zhou al. 2019) with 25K images and 150 semantic categories. We report the metric of mean Intersection of Union (mloU) averaged over all semantic categories. As presented in Section [.6] we directly follow the task layer and the most of hyperparameters described in SETR-PUP (Zheng et al. 2020). On ADE20K, we use Adam (Loshchilov Hutter 2019) as the optimizer. The learning rate set to le-3 with layer-wise decay similar to image classification. We conduct fine-tuning for 160K steps. The batch size is 16. The detailed hyperparameters are described Appendix []]
As shown in Table 3 we compare BEIT with supervised pre-training that relies on labeled data of ImageNet. We find that our proposed method achieves better performance than supervised pre- training, although BEIT does not require manual annotations for pre-training. Moreover, we employ intermediate fine-tume aneitt for BETT on ImageNet, i.e., we first fine-tune pretrained BErT on ImageNet, and then fine-tune the model on ADE20K. The results indicate that intermediate fine-tuning further improves BEIT on semantic segmentation.
3.3 ABLATION STUDIES
We conduct ablation studies to analyze the contributions of each component in BEIT. The models are evaluated on image classification (i.e., ImageNet) and semantic segmentation (i.c., ADE20K), We set the default pre-training steps 300 epochs for the ablation studies, which is 37.5% of the total steps used in the previous experiments.
Table 4 reports the results of various model variants. First, ablate blockwise masking by randomly sample masked positions. We find that blockwise masking is beneficial on both tasks, especially on semantic segmentation. Second, ablate the usage of visual tokens by predicting the pixels of masked patches, i.e., the pre-training task becomes a pixel regression problem to recover masked patches. Our proposed masked image modeling task significantly outperforms naive pixel-level auto-encoding. Compared with the results in Table [ the ablation result is worse than training vision Transformer from scratch on two tasks. The results indicate that the prediction of visual tokens the key ingredient of BEIT. Third, we ablate the usage of visual tokens and blockwise masking together. We find that blockwise masking is even more helpful for pixel-level auto-encoding, which relieves the suffering of short-distance dependency. Forth, recovering all the visual tokens harms performance on downstream tasks. Fifth, we compare BETT with different training steps. Pre-training the model longer can further improve performance downstream tasks.
3.4 ANALYSIS OF SELF-ATTENTION MAP
We show that the self-attention mechanism in BEIT can separate objects, even though our pre-training does not rely any manual annotation at all. Similar properties are also observed by Caron et al.) (202T). The probing images are taken from the MS COCO (Lin et al, 2014) corpus to avoid appearing the pre-training data.
8

---

Published as a conference paper at ICLR 2022
Figure 2: Self-attention map for different reference points. The self-attention mechanism in BEIT is able to separate objects, although self-supervise pre-training does not use manual annotations.
As shown in Figure we plot the self aeffaattention map for different reference points within an image. The visualizations are produced by attention scores computed via query-key product in the last layer. For each reference point, we use the corresponding patch as query, and show which patch attends to. After pre-training. BEIT learns to distinguish semantic regions using self-attention heads, without any task-specific supervision. The property partially indicates the reason why BEIT is able to help downstream tasks. Such knowledge acquired by BErT potentially improves the generalization ability of fine-tuned models, especially on small-scale datasets.
4 RELATED WORK
Self-supervised visual representation learning. Various methods have been introduced over the years to pretrain vision models in a self-supervised manner. Pioneering works design clever pretext tasks, such predicting the patch orderings (Noroozi & Favaro 2016), colorization (Zhang et al.) 2016), and predicting rotation angles (Komodakis & Gidaris propose to mask some patches within an image, and classify whether the masked patches real or fake for each masked position. The method is similar to the masked version of Jigsaw pre- training (Noroozi & Favaro 2016). The recent strand of research follows contrastive paradigm (Wul et al. 2018 Oord et al.) 2018 Hjelm et al. 2019 Bachman et al., 2019 He et al.) 2020 Chen et al. 2020ble). The models typically regard various data augmentations as different views of an image, and then make the representations of positive pairs similar while pushing negative pairs away. In order to obtain enough informative negative samples in contrastive learning, the methods usually rely on large memory banks [Wu et al., 2018 He et al. 2020) or large batch size (Chen et al. 2020b). BYOL (Grill al., 2020) and SimSiam (Chen & He 2020) further eliminate the requirement of negative samples, using various techniques to avoid representation collapse. Another strand of methods use clustering to
In addition, Trinh et al. (2019)
organize image examples (Caron al. 2018 Asano al. 2020 Caron et al. 2020 cl al. 202T).
Self-supervised vision Transformers. Pre-training vision Transformers has received significant attention recently due to the data-hungry issue. iGPT (Chen et al.] (220a)) first creates 9-bit color palette by k-means clustering RGB pixels, and then uses the clustered tokens to represent images. Next iGPT uses the tasks of BERT and GPT to pretrain Transformers. In comparison, our proposed method uses image patches as input without losing pixel-level information. Moreover, our visual tokens obtained by discrete VAE instead of clustering. ViT (Dosovitskiy et al.) 2020) conducts preliminary exploration with the masked patch prediction task, which predicts the 3-bit mean color of the masked patches. Dosovitskiy et al. 2020) also report that pixel-level auto-encoding performs

---

Published as a conference paper at ICLR 2022
worse, although it is the most straightforward translation of BERT from NLP to CV. Rather than using heuristically designed pre-training tasks, our proposed model leverages visual tokens learned by discrete VAE, which not only achieves better performance but also is better theoretically motivated. Apart from masked auto-encoding, other mainstream research works use contrastive learning (Chen al. 202T Xie et al., 2021), and self-distillation (Caron et al., (021). In comparison, BETT can achieve several times of improvement in terms of pre-training throughput (Appendix [D) and memory consumption. The advantages make BEIT appealing scale up vision Transformers.
5 CONCLUSION
We introduce a self-supervised pre-training framework for vision Transformers, achieving strong fine-tuning results on downstream tasks, such image classification, and semantic segmentation. We show that the proposed method is critical to make BERT-like vre-training (i.c., auto-encoding with masked input) work well for image Transformers. We also present the intriguing property of automatically acquired knowledge about semantic regions, without using any human-annotates data. the future, we would like scale up BEIT pre-training in terms of data size and model size. Moreover, we will conduct multimodal pre-training in more unified way, using the similar objectives and the shared architecture for texts and images.
REFERENCES
Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In International Conference on Learning Representations (ICLR), 2020.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximiz- ing mutual information across views. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Hangbo Bao, Dong, Furu Wei, Wenhui Wang. Nan Yang. Xiaodong Liu, Yu Wang. Jianfeng Gao, Songhao Piao, Ming Zhou, and Hsiao- Hon. UniLMv2: Pseudo-maskn masked language models for unified language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, volume 119 of Proceedings of Machine Learning Research, pp. 642-652, PMLR, 2020. URL http:://t http:::///cceeeiings.. wwt.ppree..wl/////./. html
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the European Conference Computer Vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, volume 33, pp. 9912-9924. Curran Associates, Inc., 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Hal Daumé III and Aarti Singh (eds.), Proceedings the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1691-1703. PMLR, 13-18 Jul 2020a. URL httl:tttp///ocoe /RRLattpp//proeeenng
mlr. wwwpppeessvvv.... wwtess/vl19//he22/02 html
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, preprint arXiv:2002.05709. 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. preprint arXiv22011.10566, 2020.
10

---

Published as a conference paper at ICLR 2022
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. preprint arXiv:2003.04297, 2020c.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. ArXiv, abs/2104.02057, 2021.
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 4171 4186-4486.22 Association for Computational Linguistics, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. preprint arXiv:2010. 190, 2020.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In NeurIPS, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. International Conference on Learning Representations, 2019. URL https:
//openreview www.eeetttonneeeee forum watpuummmddkkkkk111e1e
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision - Aision-ECVV- 2016, pp. 646-661, Cham, 2016. Springer International Publishing. ISBN 978-3- 319-46493-0.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017. URL htttps:/
net/Eorum?id=ree
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. Span-
BERT: Improving pre-training by representing and predicting spans. Transactions of the As- sociation for Computational Linguistics, 8:64-77, 2020. doi: 10.1162/tacl___00000 URL https: wtt.cc/wwbb.....aattoooooo/////. tac1-1..
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, 2014.
Nikos Komodakis and Spyros Gidaris. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations (ICLR), 2018.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's 's thesis, Department of Computer Science, University of Toronto, 2009.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsu- pervised represcntatiions.. International Conference on Learning Representations, 2021. URL https::/
net. Forum htee:/oorummmlddnyyyppss... 2d==kykppss-jcccc
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Yahui Liu, Enver Sangineto, Wei Bi, Nicu Sebe, Bruno Lepri. and Marco De Nadai. Efficient training of visual transformers with small datasets. In Thirry Fifth Conference on Neural Information Processing Systems, 2021a. URL http:/::////eeeeee.e..ooe http://www....oooooooooooooooeee

---

Published as a conference paper at ICLR 2022
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030, 2021b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL http:::ttt//ppott httttp:://penzrviee... net fortm?foummaaa
Bkg6RicqY7
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The Concrete Distribution: A Continuous Re- laxation of Discrete Random Variables. In International Conference on Learning Representations. 2017.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. European conference on computer vision, pp. 69-84. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. preprint arXiv: A.0.v03807.037445 2018.
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, July 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to- text transformer. J. Mach. Learn. Res., 21:140:1-140:61 2020. URI. http:://mmml r. org
papers//21/23-0744. html
A. Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. ArXiv, abs/2102. 12092, 2021.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 2015.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 10.18653/v1/P16-11 1162. URL https:: /www aclweb hotp/antholooy/PP1-1
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. preprint arXiv:2012.12877, 2020.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXiv preprint arXiv:2103.17233; 2021.
Trieu H Trinh, Minh-Thang Luong, and Quoe V Le. Selfie: Self-supervised pretraining for image embedding. arXiv preprint 1xX.::9906.02940 2019.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In Proceedings of the 31st International Conference on Neural Information Processing Systems. NIPS' 17, pp. 6309-6316 Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964
12

---

Published as a conference paper at ICLR 2022
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gamett (eds.), Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 60088,600,000000000000 2017.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non non-parametric instance discrimination. In CVPR, 2018.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018.
Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao, and Han Hu. Self-supervised learning with swin transformers. arXiv preprint arXiv:2105.04553, 2021.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv preprint arXiv:2106.04500 2021.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng. Tao Xiang, Philip H. Torr, and Li Zhang. Rethinking semantic segmentation from sequennce-to-seuueeee perspective with transformers. CoRR, abs/2012. 15840, 2020. URL httt:s://arxiv..rg///o
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. Int. J. Compur. Vis., 127(3): 302 2019. doi: 10.1007/s1 22-22663---10000000 140-0. URL https: //dol ort.l..org/...... 1007/
s s11263-018-1188 018-1-018--140--
13

---

Published as a conference paper at ICLR 2022
A ARCHITECTURE VARIANTS OF VISION TRANSFORMER
We use the standard vision Transformer (ViT: Dosovitskiy et al. 2020) in the experiments for fair comparisons. In addition, we find that LayerScale (Touvron al. 2021) and relative position bias (Bao et al. 2020 Raffel et al. 2020) improve ViTs on downstream tasks. We employ the same setting as in Section 3.3 for ablation studies, which pretrains base-size models for 300 epochs on ImageNet-IK.
As shown in Table 5 both LayerScale and relative position bias improve performance on ImageNet classification and ADE20K semantic segmentation. We denote the improved architecture as BETT+ and use it for the experiments in Appendix We empirically notice that vanilla Transformer the most stable when scaling up the model to billions of parameters, we do not use LayerScale for extra-large models.
Architecture
ImageNet
ADE20K
VIT (used this paper)
82.86
44.86
ViT+LayerScale
83.00
45.43
LiT+LayrrScal++RRe Relative Position Bias
83.22
45.70
Table 5: Ablation studies of architecture variants image classification and semantic segmentation. For ADE20K, we use UperNet (Xiao et al., 2018) as the task layer, and report mloU scores of single-scal scale inference.
B COMPARISON WITH LARGE-SCALEE SUPERVISED PRE-TRAINIS
We compare with state-of-the-art supervised pre-training at scale. In addition to using ImageNet-1K for fair comparisons with previous work, we pretrain BETT on ImageNet-22K boost performance. We employ the architecture improvements (i.e., LayerScale, and relative position bias) as described in Appendix A which denoted BETT+ in Table 6 and Table We follow the same pre-training setup as in Section 2.5 except we pretrain 150 epochs on ImageNet-22KK. After self-supervised pre-training, we conduct intermediate fine-tuning on ImageNet-22K for 90 epochs. Moreover, we use an in-house dataset that has about 70M labeled images as drop-in replacement of ImageNet-22K.
Model Size
Labeled
ImageNet
Models
Data Size
3842
5122
Supervised Pre- Training on Image/Net-22K (using labeled data)
VIT-B Dosovitskiy al.
2020
86M
14M
84.0
ViT-L
Dosovitskiy et al.
2020
307M
14M
85.2
85.30
VIT-H
Dosovitskiy al.
2020)
632M
14M
85.1
Supervised Pre- Praining on Google JFT-300M (using labeled data)
ViT-B Dosovitskiy et al.
2020
86M
300M
84.2
ViT-L
Dosovitskly al.
2020
307M
300M
87.1
87.76
ViT-H Dosovitskiy et al.
2020
632M
300M
88.0
88.55
Supervised Pre-T Tre-nrain Google JFT-3B
(using labeled data)
ViT-B Zhai et al.
2021)
86M
3000M
86.6
ViT-L Zhai al.
2021
307M
3000M
88.5
Superviuee Pre-Training, and Intermediate Fine- Taning on ImageNet-222
BErT-B+ (ours)
86M
14M
86.8
B (ours)
307M
14M
88.4
88.6
Self-Supervised Pre-Training, and Intermediate Fine- Tuning In- In-House-70M
B (ours)
307M
70M
89.3
89.5
Table Top-1 accuracy on ImageNet- 1K fine-tuning. We evaluate models at resolutions 3842 and 5122.
14

---

Published as a conference paper at ICLR 2022
Table 6 compares BETT with previous state-of-the supervised pre-training (Dosovitskiy al. 2020 Zhai et al. 2021) on ImageNet fine-tuning. Rather than heavily relying on extremely large-size labeled data (such Google's ' in in-house JFT-300M and JFT-3B). demonstrate that BEIT pre- training can catch up with only ImageNet-22k (1 (14M). Specifically, BErT-L fine-tuned on ImageNet- 22K achieves comparable performance with ViT-I. trained Google JFT-3B. Moreover, BErT-L obtains 89.5% top-1 accuracy on ImageNet after intermediate fine-tuning on an in-house 70M dataset. The results indicate that BEIT pre-training greatl reduces the required labeling efforts and advances the new state of the art for lazge-size vision Transformers.
As shown in Table 7 we report the tinettunng results on the ADE20K semantic segmentation benchmark. Following Swin (Liu et al. 22021b) we use the same task layer (i.e., UperNet: Xiao et al. 2018) and evaluate the models at the resolution 640 640. The BErT-L model obtains state-of-ther performance on ADE20K.
Models
mloU (%)
Multi-Scale mloU (%)
Supervised Pre- Pre-Training on ImageNer-226 (using labeled data)
Swin-B LLiu et al.
2021b)
50.0
51.7
Swin-L. Liu et
2021b
52.1
53.5
relf-Supervise Pre-Training, and Intermediate Fine-T Tunin on ImageNet-22X
BEIT-B+ (ours)
53.6
54.2
BEIT-L* (ours)
56.7
57.0
Sepf-Superviee Pre-Training, and Intermediate Fine-Tuming In-House-70N
BEIT-L* (ours)
57.9
58.4
Table Performance comparison on the ADE20K semantic segmentation. We follow Swin-L (Liu al. 2021b) to use UperNet (Xiao al. 2018) the task layer and evaluate at resolution 640 640.
C ABLATION STUDIES OF IMAGE TOKENIZER
For comparison, we re-train the image tokenizer on ImageNet-IK. The reimplementation is based on https: /tpp:://iitin //githubtbb/bt/// com/ lucilccio driins /DALLE pyte We use the same codebook size 8K as in DALL-E (Ramesh et al. 2021). Then we plug the tokenizer into our pre-training process. We follow the same experimental setup of ablation studies in Section 3.3 Table shows that our reimplemented tokenizer obtains comparable reconstruction loss and ImageNet fine-tuning performance compared with the off-the-shelf DALL-E tokenizer.
Image Tokenizer
Reconstruction Error
ImageNet
DALL-E Tokenizer Ramesh et al.
2021
0.0856
82.86
Our reimplementation
0.0880
82.70
Table 8: Top-1 accuracy ImageNet- IK using different image tokenizers during pre-training, For image reconstruction, we report mean absolute error of normalized RGB values. The reimplemented image tokenizer is trained on ImageNet- IK without labels.
D LINEAR PROBES ON IMAGENET
We evaluate linear probes on ImageNet for various pretrained vision Transformers. We compare BEIT with two main strands of work, namely discriminative and generative self-supervised learning The first one applies discriminative learning for pre-training, such contrastive learning (Chen et al.) 2021), and self distillation (Caron et al. 2021). The above methods typically lear to aggregate the image-level features into global vector. which is relatively suitable for linear probing. In contrast, the second strand methods, such as iGPT (Chen et al.] (020a) and ours, usually do not pretrain such global feature aggregation, which tends to make linear probes difficult.
15

---

Published as a conference paper at ICLR 2022
Following iGPT (Chen et al. 2020a), use average pooling aggregate the hidden states of each image patches, and add the probing layer at the middle layer of Transformer instead of always at the final layer. Similarly, we find that the best layer lies in 9-th layer for BEIT-B, and 14-th layer for BErT-L. To be specific, we use A (Loshchilov & Hutter 2019) to update the linear prohe layer for 50 epochs. The learning rate is 4e-3 with cosine decay. The batch size is 1024. The weight decay is set to le-4. We follow data augmentation used in DINO (Caron et al. 2021). which uses random resize crops and horizontal flips augmentation during training and evaluates on central crops.
Models
Model Size
Accuracy
Discriminative self-supervised learning
DINO-B (Caron et al. 2021)
86M
78.2
MoCo v3-B CCen et al. 2021
86M
76.7
MoCo v3-L (Chnn et al.
2021
307M
77.6
Generative self-supervixed learning
iGPT-L (Chen al.
2020a)
1362M
65.2
iGPT-XL (Chen et al.
2020a)
6801M
68.7
iGPT-XL CChnn al.
2020a
6801M
72.0*
BETT-B (ours)
86M
56.7
BETT-L (ours)
307M
73.5
Table 9: Linear probing accuracy ImageNet. "*" denotes that iGPT-XL uses concatenation of five layers for linear probing, while others use the features of single layer.
As shown in Table we evaluate linear probes on ImageNet- IK for self-supervised learning. Overall, discriminative methods perform better than generative pre-training linear probing. Linear probes keep the Transformer parameters fixed and only update the linear layer. So the pre-training of global aggregation of image-level features is beneficial to linear probing in DINO and MoCo v3, although full fine-tuning eliminates the gap. Moreover, the results indicate that increasing the model size from base (86M) large (304M) significantly improves accuracy for our proposed method. In contrast, the gap between base- and large-size MoCo v3 is smaller. We also find that BETT outperforms iGPT by a large margin even using much fewer parameters.
E MULTI-TASK TRE-TRAINING WITH DINO
We train the pre-training tasks of BEIT and DINO (Caron et al. 2021 together in a multi-task manner. As shown in Table TO augmenting masked image modeling with DINO improves semantic segmentation on ADE20K, and obtains comparable results on ImageNet classification. Moreover, BEIT is more efficient terms of pre-training speed, DINO has two copies of Transformer parameters for self-distillation and multi-crop augmentation (Caron al., 2020). For the throughput comparisons between BEIT and BEIT+DINO, we set batch size to the same. Because BErT also more memoory-fffcicee we use larger batch size fully utilize GPU cards, which obtains greater speedup in practice than the reported numbers.
Models
ImageNet
ADE20K Pre-Training Throughput
DINO (400 Epochs)
82.8
44.08
BETT (300 Epochs)
82.9
44.65
4.2x
BEIT DINO (300 Epochs)
82.9
46.85
1.0x
Table 10: We train the pre-training tasks of BEIT and DINO (Caron et al. 202T in the way of multi-task learning. We report the performance by fine-tuning ImageNet- IK image classification and ADE20K semantic segmentation. For ADE20K, we use SETR-PUP (Zheng et al.) (020)) as the task layer and report the mloU score of single-scale inference. The pre-training throughput measures the speed, where larger numbers indicate faster pre-training.
16

---

Published as a conference paper at ICLR 2022
F IMAGE CLASSIFICATION ON CIFAR-100
In addition to ImageNet classification, we conduct fine-tuning experiments on the CIFAR- 100 (Krizhevsky & Hinton 2009 benchmark with 100 classes and 60k images. The experimental setup is the same as in Section 3.1
Table II reports the top-1 accuracy on CIFAR-100 100. Notably, on the smaller CIFAR- 100 dataset, ViT trained from scratch only reaches 48.5% accuracy (Chen et al.) 2021). In comparison, BErT achieves 90.1% with the help of pre-training. The results indicate that BETT can greatly reduce the requirement of annotation efforts. BEIT also outperforms MoCo v3, Moreover, intermediate fine-tuning on ImageNet- IK further improves the results on CIFAR-I00
Models
CIFAR-100
Training from scratch (i.e., random initialization)
ViTas4 (Dosovitskiy et al. 2020)
48.5
Supervised Pre- Trainngin on ImageNet-IK (using labeled data)
ViT (Dosovitskiy al. 2020)
87.1
DeiT Touvron et al. 2020)
90.8
Self-Snpervised Pre- Pre-Training on ImageNet-IK (without labeled data)
DINO (Caron et al. 2021)
91.7
MoCo v3 (Chen et 2021
87.1
BITT (ours)
90.1
Self-Supervised Pre-Training, and Intermediate Fine- Fine-Tuning Tuning ImageNet-IK
BEIT (ours)
91.8
Table 11: Top-1 accuracy of image classification on CIFAR-100. The models are at resolution 224 x 224, except ViT3s4 uses 384 x 384. The results, unless otherwise indicated, are all obtained by base-size models. #; result taken from (Chen et al. 2021
G HYPERPARAMETERS FOR PRE-TRAIN TRAINING
Hyperparamettes
Base Size
Large Size
Layers
12
24
Hidden size
768
1024
FFN inner hidden size
3072
4096
Attention heads
12
16
Attention head size
64
Patch size
16 x 16
Training epochs
800
Batch size
2048
Adarn
1e-8
Adam 8
(0.9, 0.999)
Peak learning rate
1.5e-3
Minimal learning rate
le-5
Learning rate schedule
Cosine
Warmup epochs
Gradient clipping
3.0
1.0
Dropout
X
Stoch. depth
0.1
Weight decay
0.05
Data Augment
RandomResizzAniCCop
Input resolution
224 x 224
Color jitter
0.4
Table 12: Hyperparameters for pre-training BEIT on ImageNet-IK.
17

---

Published as a conference paper at ICLR 2022
H HYPERPARAMETERS FOR IMAGE CLASSIFICATION FINE-TUNING
CIFAR-100
ImageNet-IK
Hyperparameters
Base Size
Base Size
Large Size
Peak learning rate
[2e-3,
3e-3, 4e-3, 5e-3)
Fine-tuning epochs
150
100
50
Batch size
512
1024
1024
Warmup epochs
20
20
5
Layer-wise learning rate decay
0.65
0.65
0.75
Adam
1e-8
Adam |
(0.9, 0.999)
Minimal learning rate
le-6
Learning rate schedule
Cosine
Repeated Aug
X 0.05
Weight decay
0.3
0.05
Label smoothing
0.1
Stoch. depth
0.1
Dropout
X
Gradient clipping
X
Erasing prob.
x
0.25
0.25
Input resolution
224 x 224
Rand Augment
9/0.5
Mixup prob.
0.8
Cutmix prob.
1.0
Table 13: Hyperparameters for fine-tuning BETT on ImageNet- 1K and CIFAR-100.
HYPERPARAMETER FOR ADE20K SEMANTIC SEGMENTATION
FINE-TUNII
Hyperparameters
Base Size
Peak learning rate
1e-3
Fine-tuning steps
160K
Batch size
16
Adam
le-8
Adam 3
(0.9, 0.999)
Layer-wise learning rate decay
0.65
Minimal learning rate
0
Learning rate schedule
Linear
Warmup steps
1500
Dropout
X
Stoch. depth
0.1
Weight decay
0.05
Input resolution Position embedding interpolate
512 x 512 bilinear
Table 14: Hyperparameters for fine-tuning BErT on ADE20K.
18


Reviews:
[{"review_id": "jXOQpn1bBpS", "reviewer": "Reviewer_pM4q", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the MIM objective resembles the masked region modeling (MRM) objective, which is widely used in the vision-and-language pretraining (VLP) models.  \nVLP models often mask several visual regions and make their contextualized outputs predict their corresponding object class to guide visual inputs in tandem with the textual inputs' masked language modeling.  \nBy removing textual inputs from VLP models and switching detection-based region features to patches, we have a BEiT-like structure.  \nRecently, ViLT[^1] proposed this type of VLP model but failed to make appropriate objectives for the patches, reporting that regressing masked pixels (masked patch prediction objective) deteriorate downstream vision-and-language tasks.  \nIn my view, as MRM and MIM are similar enough to be noted, I recommend the authors add the relation with BEiT and VLP models in the related work section.\n\nUsing argmax-ed visual tokens is inevitable for DALL.E since they had to plug the discrete tokens into the decoder.  \nHowever, since BEiT only uses the discrete tokens as ingredients of the MIM objective, I think argmax-ing the visual tokens is unnecessary, and the token distribution can be immediately used for the objective (e.g., KL-divergence).  \nActually, UNITER[^2], which used detection-based regional inferred class for the MRM objective to train VLP model, has tested three types of using the class information:  \n1.  Use it as a one-hot label (as BEiT did).\n2.  Use KL-div.\n3.  Regress the features that are used to infer the class labels.\n\nUNITER showed that the combination of (2) + (3) yields better performance than solely using (1).  \nI believe all three approaches are also available for BEiT and the discrete VAE; thus, I wonder whether they can further boost the performance of BEiT.\n\nI believe using visual tokenizers such as discrete VAE can be a silver lining for the community and those seeking self-supervisable images' objectives, including the VLP community.  \nI think this paper showed rigorous and solid empirical results and well contributes to the community by providing valuable tools.\n\n[^1] Kim, Wonjae et al. \"ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision.\" _ICML_ (2021).  \n[^2] Chen, Yen-Chun et al. \"UNITER: UNiversal Image-TExt Representation Learning.\" _ECCV_ (2020).", "summary_of_the_paper": "The paper presents a new objective called masked image modeling (MIM) to pre-train vision transformers, making the model predict a visual token from the masked patches.  \nThe visual tokens are obtained by discrete VAE, which is trained on 250M images. (Though, the authors showed that training discrete VAE with only 1M images (imagenet-1K) is enough to demonstrate the power of the proposed objective.)  \nThe authors pre-train the ViT with MIM and fine-tune the pre-trained weights (BEiT) on two visual downstream tasks: image classification and semantic segmentation, and showed superior performances compared to previous methods, including DINO and MoCo v3.  \nThe authors also propose an additional trick called blockwise masking to improve BEiT further.", "summary_of_the_review": "- Consider adding a VLP subsection to the related work section.\n- There are several ways to exploit the discrete VAE tokens.\n- The reviewer thinks the paper's results are solid.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "famnZLE75ol", "reviewer": "Reviewer_Vvct", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "PROS:\n- The task is interesting per se, as it brings the concept of BERT into Vision Transformers\n- Well written paper and simple idea\n- Good analysis\n\nCONS:\n- Some important details for practitioners are missing (e.g. choices on the masking)\n- experiments of different models are sometimes difficult to compare\n\nIn general, I liked the paper a lot. However, I have some issues that should be answered and addressed before publication. I'll number them for convenience\n\n1. Authors says that pre-training is run for 800 epochs, which is fine for large ML groups but it might be very demanding for smaller groups. Moreover, it does not help the comparison with all other SoTA methods. For example, DiNO is trained for 300 epochs. Moco v3 is an experimental paper, but the main results are obtained for 300 epochs (although they report 600 epochs as well). Now, I wonder two things: 1) are pre-training results in Table 1 obtained with 800 epochs for all the models? I mean, are they comparable? 2) I'd have liked to see a comparison between models at 300 epochs (as standard) in Table 1 or in another additional table. Appendix A might be in that direction but I did not understand it well as it is not well explained; 3) practitioners would really like to see the training curve of BEiT, to understand what budget should they invest to have the desired accuracy.\n\n2. Since BEiT is somewhat based on BERT, why do you replace 40% of tokens? Is there a motivation behind this number? then, BERT task was to replace the token 80% of the time with a [MASK] token, 10% of the time with a random token, and 10% of the time keeping it as it was. Did you consider this strategy? As far as I know, this is beneficial for fine-tuning tasks.\n\n3. It would be good to see the standard (small) datasets for downstream tasks such as CIFAR-10, Oxford Flowers-102 and Oxford Pets or cars alongside CIFAR-100. This would have made the paper more comparable. Moreover, why 150 epochs? I believe that the standard is 100.\n\n4. Why is DINO in Table 10 with 400 epochs? why not 300 as standard?\n\n5. It's good to have the appendix, but authors should refer to them and comment on the (interesting!) results in the main paper. E.g. Appendix E, F.\n\n6. Are authors going to release the code?\n\nMINOR:\n- I would change the special token [S] to [CLS] as in BERT\n- I would cite this recent paper in the intro talking about how Vision Transformers are data-hungry https://arxiv.org/abs/2106.03746 and some tricks for small datasets\n- In Section 2.3 \"pixel-level auto-encoding\" might be confusing. I suggest rephrasing it and explaining it better (as it is in the intro).\n- If you have time, for the camera ready, I'd love to see BeIT applied to some convolution transformers such as Swin or CvT\n- I would like to see a comment on the throughput of DINO vs your paper in the main manuscript", "summary_of_the_paper": "This paper presents a novel task called Masked Image Modeling (MIM), inspired by the more famous Masked Language Model task proposed by BERT in NLP. For this reason, the paper is called BEiT. BEiT relies on a pre-pre-trained tokenizer that transforms image patches into discrete tokens, which are then masked and predicted. Extensive experiments show that this self-supervised pre-training improve SoTA in various downstream tasks such as image classification and semantic segmentation.", "summary_of_the_review": "The paper is well written, the idea is somewhat novel (novel in computer vision, less novel in general because of BERT). Experiments are good but improvable. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cpVS4v713O2", "reviewer": "Reviewer_7cfB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(+) The paper is clear and easy to follow.  \n(+) There are proper ablations for most of the design decisions.  \n(-)  The dependence on DALL-E to is a pretty hefty one (unsupervised pretraining on 250M images). This can potentially be lifted, but the authors made no attempt in this direction.  \n(+/-) To be fair with the above point, the authors did at least the ablation of predicting masked pixels directly (instead of visual tokens from DALL-E) in Table 4, and the performance drops by \"only\" ~1.7% top1 on ImageNet in the downstream task. This is still not great as this brings the accuracy below supervised training only, which is what is done as fine-tuning on this network in this case. This ablation should have been done in linear probing also.  \n(-) Regarding linear probes, the results (in Table 9) are somewhat disappointing. I wonder if this comes from the relatively small batch size (and/or lack of hyperparameters search) compared to other papers, or if there is a more fundamental reason for BEiT to be worse than contrastive methods there.  \n(=) No pretraining on larger than ImageNet-1K.  \n(+) The paper provides enough details for reproducing the results.\n\n", "summary_of_the_paper": "This paper is one of the first to present state-of-the-art results for masked image modeling (BERT-style) self-supervised learning on images (contrastive approaches held the SOTA before). The whole system is based on a ViT encoder (e.g. with 16x16 pixels patches as input) to produce visual tokens. The pretraining objective consists in matching the visual tokens (one per image patch) from a provided visual tokenizer, with the ViT encoder. The tokenizer is a discrete VAE (with 8192 token types) from prior work called DALL-E (Ramesh et al., 2021). The results on ImageNet-1K are SOTA for comparable model sizes and pretraining data and settings. Idem for segmentation on ADE20K.", "summary_of_the_review": "Overall, this is a strong paper presenting a significant improvement in unsupervised pretraining for images, that should be presented at ICLR. More analysis (e.g. linear probing) could make the paper stronger. The limits of the paper are fair to have, and can potentially be addressed in follow-ups.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "_IoILT8eO_6", "reviewer": "Reviewer_GYF1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents a self-supervised learning framework named BEIT, in which the input image can be masked in some regions, and the task is to recover the token of the masked region. Pre-trained on ImageNet, the model shows good performance on a series of downstream tasks.\n\nI shall say that I very much like the idea of this paper. Self-supervised learning in computer vision seems far behind that in natural language processing, mostly because the proxy task is not good enough. This is highly related to the original data format -- masked language modeling is a perfect proxy for texts, but it seems very difficult to replicate it on images -- because image signals are mostly continuous in the space, and semantic signals are sparse, etc. This paper makes a good trial along this direction. I think the mask image modeling task is much more elegant than the existing contrastive learning (e.g. SimCLR & MoCo) or predictive learning (e.g. BYOL) counterparts, because those methods are mostly relying on data augmentation to provide the prior-to-be-learned, but data augmentation not strong enough. More importantly, data augmentation can bring conflicts, e.g. one needs to enlarge the intensity of data augmentation to improve the difficulty of learning, but strong data augmentation is risky and may generate duals with very different semantics (so that they are bad for SSL).\n\nOK, I have expressed my opinions that this new direction is promising. Also, the experiments on ImageNet and ADE20K are good (though I expect to see more including ImageNet linear-tests, MS-COCO tests, etc.), showing the strong ability of the pre-trained networks. However, I have one major concern that avoids me from giving a higher score to this paper. **That is, I am not sure if the proposed framework is fairly compared against the prior methods.**\n\nThe key lies in the tokenizer, which delivers almost all priors in the BEIT algorithm. However, the tokenizers are borrowed from DALL-E, which means that a large number of image-text pairs have been used -- the authors did not specify which model has been used, but at least, the CC3M (if not JFT300M) dataset is included in the pre-training part. This is not considered self-supervised learning, as the texts can contain vast amount of semantics (according to the CLIP paper, after training on image-text pairs, the model can achieve good performance on zero-shot ImageNet classification). I do not believe that using this tokenizer as the \"teacher\" to \"distill\" the target \"student\" model is \"self-supervised learning\". In Appendix C, the authors claimed that a re-implemented tokenizer on ImageNet shows similar performance, but technical details are missing. Even in the unsupervised setting, if the tokenizer is trained for sufficiently long, it can offer powerful guidance to the target network, so that some statements (e.g. Section 2.5, The pre-training runs for about 500k steps (i.e., 800 epochs)) are not strictly meaningful.\n\nI hope the authors can answer the following questions.\n\n1. The detailed setting of the DALL-E tokenizer and ImageNet re-trained tokenizer. In particular, please specify what kind of external information has been used, and the cost of pre-training such tokenizers.\n\n2. If indeed external information is used (e.g. ImageNet labels to the worst case), is it possible to re-implement a counterpart without any such information and re-test the performance? Also, I am very interested in the performance of a random tokenizer (i.e. without pre-training, just clustering the responses of the randomly initialized networks).\n\nBy any means, I think a discussion on the relationship between this approach and knowledge distillation is necessary. I shall re-evaluate this paper after seeing answers to the above questions.", "summary_of_the_paper": "This paper presents a self-supervised learning framework named BEIT, in which the input image can be masked in some regions, and the task is to recover the token of the masked region. Pre-trained on ImageNet, the model shows good performance on a series of downstream tasks.", "summary_of_the_review": "1. The idea is good, going through an important direction of self-supervised learning.\n2. Downstream tests show good performance.\n3. BUT, whether the improvement comes from the designed framework is questionable.\n\nI need additional details from the authors to make the final decision.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Inspired by BERT and the corresponding masked language modeling objective, this paper proposes masked image modeling as a pre-training technique for vision transformer. More precisely, the image is tokenized using a pre-trained tokenizer, and the goal is to predict the token indices corresponding to masked patches of the image. As noted by the reviewers, the proposed method is simple, works very well in practice and the paper is well written. Since this work potentially opens a whole new research direction, my recommendation is to accept with oral presentation.
Decision: Accept (Oral)