BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers
Authors: Zhiliang Peng
Li Dong
Hangbo Bao
Qixiang Ye
Furu Wei
Venue: Submitted to ICLR 2023 (2023)

Abstract: Masked image modeling (MIM) has demonstrated impressive results in self-supervised representation learning by recovering corrupted image patches. However, most existing studies operate on low-level image pixels, which hinders the exploitation of high-level semantics for representation models. In this work, we propose to use a semantic-rich visual tokenizer as the reconstruction target for masked prediction, providing a systematic way to promote MIM from pixel-level to semantic-level. Specifically, we propose vector-quantized knowledge distillation to train the tokenizer, which discretizes a continuous semantic space to compact codes. We then pretrain vision Transformers by predicting the original visual tokens for the masked image patches. Furthermore, we introduce a patch aggregation strategy which associates discrete image patches to enhance global semantic representation. Experiments on image classification and semantic segmentation show that BEiT v2 outperforms all compared MIM methods. On ImageNet-1K (224 size), the base-size BEiT v2 achieves $85.5\%$ top-1 accuracy for fine-tuning and $80.1\%$ top-1 accuracy for linear probing. The large-size BEiT v2 obtains $87.3\%$ top-1 accuracy for ImageNet-1K (224 size) fine-tuning, and $56.7\%$ mIoU on ADE20K for semantic segmentation. The code can be found in the supplementary materials.

Under review as a conference paper at ICLR 2023
BEIT V2: MASKED IMAGE MODELING WITH DECTOR-QUANTIE VISUAL TOKENIZERS
Anonymous authors Paper under double-blind review
ABSTRACT
Masked image modeling (MIM) has demonstrated impressive results in self- supervised representation learning by recovering corrupted image patches. How- ever, most existing studies operate on low-level image pixels, which hinders the exploitation of high-level semantics for representation models. In this work, we propose to use a semantic-rich visual tokenizer as the reconstruction target for masked prediction, providing systematic way to promote MIM from pixel-level to semantic-level. Specifically, we propose vector-quantized knowledge distillation to train the tokenizer, which discretizes continuous semantic space to compact codes. We then pretrain vision Transformers by predicting the original visual tokens for the masked image patches. Furthermore, we introduce a patch aggregation strategy which associates discrete image patches to enhance global semantic representa- tion. Experiments on image classification and semantic segmentation show that BETT v2 outperforms all compared MIM methods. On ImageNet- 1K (224 size), the base-size BIT V2 achieves 85.5% top-1 accuracy for fine-tuning and 80. 1 top-1 I accuracy for linear probing. The large-size EIT v2 obtains 87.3% top-1 accuracy for ImageNet- 1 (224 size) fine-tuning, and 56.7% mloU on ADE20K for semantic segmentation. The code can be found in the supplementary materials.
INTRODUCTION
Masked image modeling (MIM), which greatly relieves the annotation -uungry issue of vision Trans-
formers, has demonstrated great potential in learning visual representations (Bao et al., 2022 He
et al. 2022). Given an image, the pretraining objective of MIM is to recover the masked patches so that rich context information is captured by the representation model. Taking BETT (Bao et al. 2022)
as an example, each image has two views during pretraining, i.e., image patches, and visual tokens. The original image is first tokenized to discrete tokens. Randomly sampled image patches are then masked before being fed to vision Transformers. The pretraining objective is to recover the original visual tokens based on the corrupted image patches. The pretrained vision encoder can be deployed and finetuned on various downstream tasks by appending lightweight task layers.
Existing MIM approaches can be coarsely categorized to three according the reconstruction targets: low-level image elements (e.g., raw pixels; He et al. 2022] Fang et al. 2022] Liu et al. (022), hand- crafted features (e.g., HOG features: Wei et al. 2021 and visual tokens: Bao al. 2022] Wang et al.) 2022 Dong et al. 2021 El-Nouby et al. 2021 Chen et al. 2022 However, all the reconstruction targets are about, explicilly or implicitly, low-level image elements while underestimating high-level semantics. In comparison, the masked words in language modeling (Devlin et al. 2019) are all about high-level semantics. which motivates us tap the potential of MIM by exploiting semantic-ans aware supervision during pretraining.
In this work, we propose a self-supervised representation learning approach, termed BErT v2, with the aim to improve MIM pretraining by constructing a semantic-aware visual tokenizer. Our approach developed on the BETT method which is simple yet effective. The novelty lies in introducing the Vector- Vector-Quantized Knowledge Distillation (VQ-KD) algorithm to discretize semantic space. The VQ-KD encoder first converts the input image to discrete tokens according to a learnable codebook. The decoder then leamns to reconstruct the semantic features encoded by teacher model, conditioning on the discrete tokens. After training VQ-KD, its encoder is used as a semantic visual tokenizer for BETT pretraining, where the discrete codes serve as supervision signals.

---

Under review as a conference paper at ICLR 2023
85.5
BEiT y2
BEIT v2.
87.0
BETT v2
85.0
BEIT v2
Accuracy (%)
Accuracy (%)
PeCo
86.5
PeCo
data2vec
84.5
MVP
MVP
CAE
data2vec
MaskFeal
86.0
MAE
84.0
iBoT
BootMAF
CAE
MaskFeat
MAE
85.5
83.5
SplitMask
MaskFcat
BETT
BEiT
83.0
85.0
300
800
1600
300
800
1600
Pretraining Epochs
Pretraining Epochs
Figure 1: Top-1 fine-tuning accuracy on ImageNet (224 size). Left: ViT-B/16. right: ViT-L/16.
Considering the discreteness of tokens, we further introduce a patch aggregation strategy which explicitly encourages the [CLS] token to associate all patches (Gao & Callan 202T). Such a strategy resolves the issue that MIM put patch reconstruction the first place which diminishes learning global image representations. As result, BEIT v2 improves the capacity of learned image representation, as supported by the linear probing experiments. Moreover, the enhanced representations also boosts the performance of other tasks.
We conduct self-supervised learning on ImageNet-1k for both base- and large-size vision Trans- formers, which evaluated on downstream tasks, e.g., image classification, linear probing, and semantic segmentation. As shown in Figure [ BEIT v2 outperforms previous self-supervised learning algorithms by large margin on ImageNet fine-tuning, e.g., improving over BEIT (Bao et al. 2022) by about two points for both VIT-B/16 and ViT-L/16. BEIT v2 outperforms all compared MIM methods on ImageNet linear probing while achieving large performance gains on ADE20k for semantic segmentation.
The contributions of this work are summarized as follows:
We propose vector-quantized knowledge distillation, promoting masked image modeling from pixel-level to semantic-level for self-supervised representation learning.
We introduce patch aggregation strategy, which enforces global structure given discrete semantic tokens, and improves the performance of learned representations.
We conduct extensive experiments downstream tasks including ImageNet fine-tuning, linear probing, and semantic segmentation. Experimental results show that the proposed approach significantly improves performance across model sizes, training steps, and downstream tasks.
METHODOLOGY
BEIT v2 inherits the masked image modeling framework defined by BEIT (Bao et 2022). which uses visual tokenizer to convert each image to a set of discrete visual tokens. The training target is recover the masked visual tokens, each of which corresponds to an image patch. In Section 2.2 we introduce a vector-quantized knowledge distillation algorithm, which is used to train a visual tokenizer. Section 2.3] we employ the visual tokenizer for BETT pretraining with the help of the patch aggregation strategy.
2.1
IMAGE REPRESENTATION
The vision Transformers (ViTs; Dosovitskiy et al, 2020) are employed as the backbone networks to obtain image representations. The input image R W is reshaped to N HW /22 patches
2

---

Under review as a conference paper at ICLR 2023
Codebook Embeddings
Teucher
X
Nearest Neighbor Lookup
l 4n-orm
Input Image
VIT
f2-mor
98
876
221
VIT
Semantic Reconstruction
54
75
789
100
98
717
8M
Tokenizer Encoder
Visual Tokens
Xx
Decoder
Straight-Thruse ThroughhTT Gradients
Figure 2: Pipeline for visual tokenizer training. After training, each image is converted to discrete visual tokens.
[a]))= where x E (NNx(PA)) and (P, P) is the patch size. In experiments, each 224 224 image split to 14 x 14 grid of image patches, where each patch is 16 x 16. The image patches {xx] ) then flattened and linearly projected input embeddings for Transformers. The encoding vectors are denoted as {hi which corresponds to N image patches.
2.2 TRAINING VISUAL TOKENIZER
We propose vector-quantized knowledge distillation (VQ-KD) to train the visual tokenizer, Figure where the visual tokenizer and the decoder are two vital modules.
The visual tokenizer maps an image to a sequence of visual tokens, a.k.a., discrete codes. To be specific, an image tokenized z [21, 22. ... aNl] E ((H//))(W/P),, where the visual vocabulary (a.k.a.. codebook) V E R contains K discrete codebook embeddings.
The tokenizer consist of vision Transformer encoder, and quantizer. The tokenizer first encodes the input image to vectors. Then, the vector quantizer looks up the nearest neighbor in the codebook for each patch representation h, Let {v₁, U2, uKk denote the codebook embeddings. For the i-th image patch, its quantized code is calculated
2, = min [1 (2(2,
(1)
where j E 21,2.... K and b normalization is used for codebook lookup Yu et al. 2021). The above distance is equivalent finding codes according to cosine similarity.
After quantizing the image to visual tokens, we feed the fz-normalized codebook embeddings ( to the decoder. The decoder is also multi-layer Transformer. The output vectors (o aim reconstructing the semantic features of teacher model, e-g., DINO (Caron al. 2021I. and CLIP (Radford et al., [021]. Let t; denote the teacher model's 's feature vector of the i-th image patch. During training, we maximize the cosine similarity between the decoder output o, and the teacher guidance f.,
Because the quantization process (Equation T is non-di dinndiffereetibble. the gradients are directly copied from the decoder input to the encoder output (van den Oord et al. (017)), Figure 2 to back proppgaee gradients to the encoder. Intuitively, the quantizer looks up the nearest code for each encoder output, while the gradients of codebook embeddings indicate useful optimization directions for the encoder.
The training objective of VQ-KD is defined
max Σ M cos (o,, t.) - [gg[[( (s ) (l..
ZED
where sg[.] stands for the stop-gradient operator which is an identity at the forward pass while having zero gradients during the backward pass. D represents the image data used for tokenizer training.
3

---

Under review as a conference paper at ICLR 2023
Parch Tokens from the I-th Layer
Patch Aggregation
ViT Block
VIT Block
VTT Block
MIM Head
ViT Block
ViT Block
MIM Head
LMIM
Masked Image
Only
Pretraining
[CLS) Token from the Layer
Figure The MIM framework equipped with patch aggregation. The pretraining loss is the summation of LMM and MMM- The loss term LuM explicitly encourages the [CLS] token to aggregate patch information to global representations.
Improving codebook utilization. A common issue of vector quantization training is codebook collapse. In other words, only a small proportion of codes are used. Empirical strategies (van den Oord et al.) 2017 Yu et al. (021) can be used to alleviate this issue. Equation [T] shows that we compute the normalized distance to find the nearest code while reducing the dimension of codebook embedding space 32-d. The low-dimensional codebook embeddings mapped back to higher-dimensional space before being fed to the decoder. Exponential moving average (van den Oord et al.] (217)) is employed to update the codebook embeddings. Exponential moving average tends to be more stable for VQ-KD training.
2.3 PRETRAINING BEIT V2
We follow the MIM setup in BErT Bao et al. 2022) pretrain vision Transformers for image
representations. Given an input image x, around 40% image patches are block-wisely chosen and masked. The masked position is termed as M. Then, a shared learnable embedding e|M] is used to
replace the original image patch embeddings eă if M: x. = 8(i E .)) @ e|Mj + (1 8(i M)) @ ar, where 5(-) the indicator function. Subsequently, we prepend a learnable [CLS] token to the input, i.e., [ecus: (ar and feed them the vision Transformer. The final encoding
vectors denoted (h,) 1 ==o where ho is for the [CLS] token.
Next, we instantiate the MIM head as simple fully-connection layer, and then use it to predict the visual tokens of the masked positions based the corrupted image a For each masked position (h; i M a softmax classifier predicts the visual tokens p(z; p(z;|h (h.) = softmax,, (W.h; + bb). where We. b respectively mean weights and biases of the MIM head. The visual tokens are obtained by the tokenizer trained in Section 2.2 which provides supervisions for the MIM self-supervised learning procedure. The training loss of MIM is defined
CMIM = - Σ log p(z, (M))
(3)
DEDiEM
where z; denotes the visual tokens of the original image, and D the pretraining images. Notice that the number of visual tokens is the same as the number of image patches in this work.
Pretraining global representation. Inspired by (Gao & Callan 2021), we pretrain the [CLS]
token for global image representation. The goal is to mitigate the discrepancy between patch-level pretraining and image-level representation aggregation. As illustrated in Figure 3 representation bottleneck is constructed to encourage the [CLS] token to gather information as much as possible. For a L-layer Transformer, let [
i denote the 1-th layer's output vectors, where / (1, 2, ... L.).
To pretrain the last layer's [CLS] token h we concatenate it with the intermediate 1-th layer's patch vectors ( iee., S [h h hly.. We then feed S to shallow (e.g., layers)
4

---

Under review as a conference paper at ICLR 2023
Transformer decoder and conduct masked prediction again, i.e., (( softmax, (W.S ba). Notice that the parameters are shared for both MIM heads and the MIM loss is also computed at mask positions in Equation 3 Accordingly, the final training loss is defined the summation of two terms, iee., the original loss at the I-th layer, and the shallow Transformer decoder's MIM loss.
Overall framework refers to Appendix
C
Intuitively, the model favors pushing the global information to he because the model tends to fully utilize the parameters from + 1)-th layer to L-th layer, decrease the additional MIM loss. The information-flow bottleneck encourages the [CLS] token towards more reliable global representations than its untrained counterparts. Moreover, the enhanced representations also facilitate various downstream tasks. Notice that the newly added shallow decoder is only used to pretrain the [CLS] token, which discarded after pretraining.
EXPERIMENTS
The pretrained models are evaluated on image classification and semantic segmentation tasks. For image classification, the models are trained on ImageNet-IK (Russakovsky et al. 2013 and evaluated by (1) top-1 accuracy about fine-tuning and (2) top-1 accuracy about linear probing (only fine-tuning the classification head). For semantic segmentation, experiments conducted the ADE20K dataset (Zhou et al. 2019) and the performance is evaluated using the mloU protocol.
3.1 PRETRAINING SETUP
instantiate the visual tokenizer of VQ-KD as ViT-B/16 for both base- and large-size BETT v2 pretraining. The decoder network is three-layer standard Transformer, which has the same dimension and number attention heads as the tokenizer encoder. The OpenAI (Radford et al. (202T) is employed as the teacher model and train VQ-KD ImageNet- Ik with 224x 224 resolution. Notice that we use the same base-size teacher to train the visual tokenizer for both base- and large-size pretraining. The code size K is set 8192 and code dimension D as 32 by default. Refer to Appendix for more training details.
Visual tokenizer training.
Masked image modeling. We follow the settings used in BEiT (Bao et al. 2022) pretraining and use ImageNet-IN IK without labels as the pretraining data for self-supervised learning. The input image resolution is set as 224x224 during pretraining. The pretrained base- and large-size vision Transformers (Dosovitskiy et al. 2020) with 16 16 patch size are denoted ViT-B/16 and ViT-L/16, respectively. For the patch aggregation strategy, we set = 9 for ViT-B/16, 1 = 21 for ViT-L/16, and the depth 2 by default. A block-wits wise masking mechanism is adopted under the mask ratio of 40% (i.e., about 75 image patches). More pretraining details can be found in Appendix
3.2 IMAGE CLASSIFICATION
Both the fine-tuning accuracy and linear probing accuracy are evaluated on ImageNet-Ik by de- fault. The models are also evaluated on several ImageNet variants to demonstrate their favorable generalization ability.
Fine-tuning setup. We follow the protocol proposed BETT (Bao et al. 2022 to fine-tune the pretrained BETT v2 model (see Appendix F for more details). In Table 1 we report the top-1 fine-tuning accuracy results and compare BETT v2 with recent MIM methods.
From Table I base-size BEIT v2 with 300-epoch pretraining schedule reaches 85.0% top-1 accuracy, which outperforms BETT, CAF, SplitMask and PeCo by 2.1 %, 1.4%, 1.4% and 0.9% respectively. Compared with masked distillation methods, like MVP, BEIT v2 also shows superiority, Furthermore, with a longer pretraining schedule, BEIT V2 achieves 85.5% top-1 accuracy, developing state of the art ImageNet- 1 among self-supervised methods. Meanwhile, BEIT v2 using ViT-L/16 with 300 epochs reaches 86.6% top-1 accuracy, which is comparable to data2vec with 1600 epochs. A longer pretraining schedule further boosts the performance to 87.3%.
Following BETT, we add an intermediate fine-tuning phase between the pretraining stage and the fine-tuning stage. Only the intermediate fine-tuning phase uses the ImageNet-21k 1k dataset. As
5

---

Under review as a conference paper at ICLR 2023
Table 1: Fine-tuning results of image classification and semantic segmentation on ImageNet- IK and ADE20K. UperNet (Xiao et al. 2018) is used as the task layer for semantic segmentation with single-scale (512 size) input.
Pretraining Epochs
ImageNet Top-1 Accuracy(% (%)
ADE20k mIoU(%)
Methods
Base-size models (ViT-B/16)
BETT (Bao et al.
2022
300
82.9
44.7
CAE Chen al. 2022
300
83.6
48.3
SplitMask EilNouby
et al.
2021
300
83.6
45.7
MaskFeat Wei et 2021
300
83.6
N/A
PeCo Dong et al. (0211)
300
84.1
46.7
MVP Wei al. 2022)
300
84.4
52.4
iBoT (Zhou et al.
2022
400
83.8
50.0
BETT (ours)
300
85.0
52.7
Base-size models (ViT-B/16) pretrain longer
BEIT (Bao et 2022
800
83.2
45.6
PeCo Dong
2021
800
84.5
48.5
data2vec Baevski et al.
2022
800
84.2
N/A
MAE (He et al.] 2022)
1600
83.6
48.1
CAE (Chen et al.
2022
1600
83.9
50.2
BErT V2 (ours)
1600
85.5
53.1
+ Intermediate fine-tuning with ImageNet-2Ik
86.5
53.5
Large-size models (ViT-L/16)
iBoT (Zhou et al. 2022)
250
84.8
N/A
MaskFeat Wei et aL 2021
300
84.4
N/A
MVP (Wei et al. 2022)
300
86.3
54.3
BETT V2 (ours)
300
86.6
55.0
Large-size models (ViT-1/16) + pretrain longer
BErT (Bao et al. 2022
800
85.2
53.3
MaskFeat Wei et al. 2021
1600
85.7
N/A
MAE HHe al. 2022)
1600
85.9
53.6
CAE (Chnn et al. 2022
1600
86.3
54.7
data2vec Baevski et
2022
1600
86.6
N/A
BETT v2 (ours)
1600
87.3
56.7
+ Intermediate fine-tuning with ImageNet-21k
88.4
57.5
Table Top-1 accuracy of linear probing on ImageNet- 1k. All methods are based on ViT- B/16 pretrained for 300 epochs except MAE for 1600 epochs.
Table 3: Robustness evaluation three Ima- geNet variants (Hendrycks et al. 2021b Wang
2019).
ImageNet Adversarial
ImageNet Rendition
ImageNet Sketch
Methods
Methods
Linear Probe
BEIT
[Bao et al.
2022
56.7
VIT-B/16
CAE
Chen et al.
2022
64.1 1
MAE
35.9
48.3
34.5
MAE
He et al. 2022
67.8
BEIT v2
54.4
61.0
45.6
MVP
We1
2022
75.4
VIT-L/16
MoCo v3 (Chen ct al.
2021
76.7
MAE
57.1
59.9
45.3
BEIT v2 (ours)
80.1
BEIT v2
69.0
69.9
53.5
shown in Table [ we find that intermediate fine-tuning achieves about 1% performance gain on image classification for both base- and targe-size models. Refer Appendix B for more results of intermediate fine-tuning.
Linear probing. Keeping the backbone model frozen and training linear classification head atop the image-level representations, linear probing has been a widely considered measure for self- supervised learning. We average the patch tokens the global representation for the models without

---

Under review as a conference paper at ICLR 2023
Table 4: Ablation studies under VQ-KD settings. "Base& 1x18xx1 denotes that the encoder network is ViT-Base while the decoder is Transformer with depth 1, dimensions 768, and head 12. "Reconst. Loss" is the reconstruction loss of VQ-KD. Reconstruction loss and codebook usage measured the validation set. After 300 epochs of pretraining, our method reports the top-1 fine-tuning accuracy and linear probing accuracy on ImageNer-Ik, and mloU on ADE20K. The default setting is highlighted in gray
VQ-KD Architecture
Reconst. Loss
Codebook Usage
ImageNet Fine-tuning
ImageNet Linear Probe
Codebook
ADE20k
Small 1x384x6
0.183
100%
84.3
76.0
51.0
Basc & 1x768x12
0.164
100%
84.7
78.5
51.8
8192x32
Base & 3x768x12
0.145
95%
84.7
77.9
51.9
Base & 6x768x12
0.136
77%
84.6
63.0
50.1
8192x 16
0.145
100%
84.7
76.7
51.7
Base & 3x768x12
8192x64
0.148
67%
84.7
77.6
51.6
patch aggregation. Otherwise, we consider the [CLS] token as the global representation. Table presents the top-1 accuracy for linear probing and compares BETT v2 with recent methods including BEIT, CAE, MAE, MVP and MoCo v3. All the compared methods are based on ViT-B/16 and pretrained for 300 epochs except MAE for 1600 epochs. BErT v2 respectively outperforms BETT, CAE and MVP by 23.4%, 16.0% and 4.7%. BETT v2 also outperforms MoCo v3, which learns global representation through contrastive learning fashion. The comparisons indicate that the representation models learned by BEIT v2 enjoy higher adaptation capability.
Robustness evaluation. We evaluate the robustness of BEIT v2 on various ImageNet validation sets, i.e., ImageNettAdversarial (Hendrycks et al. 20216), ImageNet-Renditien (Hendrycks et al. 2021a) a and Inagehet-Sketch (Wang et 2019). As shown in Table compared with MAE (He al. 2022). BErT achieves dramatic gains across datasets, demonstrating the superiority of the proposed method in terms of model generalization.
3.3 SEMANTIC SEGMENTATION
Semantic segmentation dense prediction task, which generates class label for each pixel of the input image. Following the setting proposed in BErT (Bao et al. 2022). we conduct experiments on ADE20K benchmark (Zhou al. 2019), which includes 25K mages and 150 semantic categories. We use UperNet (Xiao al.) 2018) task layer and fine-tune the model for 160K iterations with the input resolution 512 x 512. Refer to Appendix for details. Table [ shows that BEIT v2 significantly outperforms previous self-supervised methods. Moreover, using the ViT-L/16 model, the performance can reach 56.7, which builds a new state-of-th of-e-of-thhear for masked image modeling on ADE20k.
3.4 ANALYSIS
Visual tokenizer training. We investigate the impact of VQ-KD BETT v2 in terms of the model architecture and codebook size and report the results Table ViT-B/16 without the patch aggregation strategy is used as the baseline model, which is pretrained for 300 epochs. As shown in Table we find that a deeper decoder of VQ-KD obtains better reconstruction, but lower codebook usage and downstream task performance. Reducing dimension for codebook lookup improves codebook utilization (Yu et al. 2021)
Patch aggregation strategy. Table presents the ablation studies of the patch aggregation strategy, The shallower head (i.e., 1/2-layer) performs better than the deeper head (i.e., 3-layer), suggesting the shallower head pays more attention to the input [CLS] token than the deeper head. Moreover, the proposed method outperforms the baseline variant without patch aggregation strategy. The improvement of linear probe indicates better image-level representations. In addition, the results indicate that sharing the MIM head improves downstream performance.
7

---

Under review as a conference paper at ICLR 2023
Table Ablation studies for patch aggregation strategy. (-th Layer denotes patch tokens from the 1-th layer of the backbone. Head Depth means the patch aggregation head depth. Shared MIM Head means whether we share the MIM head parameters or not. Default settings are gray...
I-th
Head
Shared MIM Head
ImageNet Fine-tuning
ImageNet Linear Probe
ADE20k
Layer
Depth
Withou patch aggregation
84.7
77.9
51.9
With patch aggregation
2
85.0
80.1
52.7
2
84.8
79.5
51.9
I
84.8
78.9
51.7
3
84.7
78.1
52.0
6
2
84.9
77.5
53.1
11
2
84.5
69.4
51.8
Table Comparisons between different VQ-KD targets. We also report the fine-tuning results of VQ-KD KD target models.
VQ-KD Targets
ImageNet
ADE20k
Pretrain 300 epochs
DINO
84.4
49.2
CLIP
85.0
52.7
Pretrain 1600 epochs
CLIP
85.5
53.1
Performance of VQ-KD larget models
DINO
83.6
46.8
CLIP
84.9
Performance of VQ-KD encoder model VQ-KD encoder (CLIP as target)
83.6
VQ-KD targets. In Table we report the results about VQ-KDs are trained under the supervision of DINO (Caron et al. 2021 and CLIP (Radford et al.] 2021). DINO is pretrained solely ImageNet- 1k while CLIP is pretrained on 400M image-text pairs datasets in house. We also directly tnn-tune the official base-size checkpoints and report the results in Table 6 One can that when using DINO as the teacher model, BErT v2 respectively reaches 84.4% and 49.2% on ImageNet and ADE20K, outperforming DINO itself by large margin. When using CLIP as the teacher model, BEIT v2 can get consistent improvements, demonstrating the scalability of the proposed VQ-KD. In addition, we directly fine-tune the VQ-KD encoder on ImageNet. The results show that transfer performance of the VQ-KD encoder is lower than the teacher model. After performing masked image modeling, the pretrained model outperforms both the teacher model and the visual tokenizer encoder. It demonstrates the superiority of the proposed method for self-supervised learning.
Visualization of codebook. We utilize the proposed VQ-KD calculate discrete codes about the ImageNet-I1 Ik validation set. Image patches are grouped according to their corresponding codes. Figure shows that the grouped image patches represent explicit semanties. For instance, the image patches corresponding to code 7856 are about "eyes" of human, cat, dog, fish and snake. Refer to Appendix (A) for more examples. The introduction of codebook and feature quantization reduces the sensitiveness to the change of image details while facilitates exploitation of high-level semantics for representation models. VQ-KD compresses and quantizes the continuous feature values to a codebook, which constructs discrete semantic space. The dimensionality of such semantic space significantly lower than that the original continuous feature space. This reduces difficulty of masked patch reconstruction and alleviates the curse of dimensionality in the pretraining phase.
8

---

Under review as a conference paper at ICLR 2023
Learned Codebook
Semantic concept sets
228s9i Nose
V1776: Tongue
Vace
V2744 Tongue
V2859' Nose
Vacw
Vrass: Eye
V Paass: Human
V6336: Human
Vruse: Eye
Figure 4: Visualization of semantic concepts corresponding the learned codebook.
4 RELATED WORK
Visual tokenizer. VQ-VAE (van den Oord et al. 2017) converts an image into sequence of
discrete codes and then reconstructs the input image based on discrete codes. DALL-E (Ramesh al. 2021) uses the Gumbel-softmax relaxation for quantization instead of the nearest neighbor
lookup in VQ- VAE. VQGAN (Esser et al., (02T]) and ViT-VQGAN (Yu et al. 2021 introduce Transformer block to train a better autoencoder to maintain fine details with adversarial and perceptual loss. Moreover, ViT-VQGAN proposes factorized and fy-normalized code for codebook learning. In comparison, the proposed VQ-KD aims at reconstructing semantic knowledge from the teacher rather than original pixels. So can construct highly compact semantic codebook for MIM.
Masked image modeling. The MIM method has achieved great success in language task (Devlin) et al. 2019). Motivated by it, BEIT (Bao et al. (2022) mitigated the MIM method to computer vision tasks by recovering discrete visual tokens (Ramesh al. 2021). The prediction targets for MIM habe
been explored by many recent works. MAE (He et al. 2022) treated MIM as a denoising lixel-level reconstruction task. Knowledge distillation (Wei al. 2021 2022) and ceff-istillation (Zhou et al.,
2022, Baevski et al.) (022) proposed to mimic the features provided by the teacher at the masked positions. PeCo (Dong al. 202T) regarded MoCo (Chen al. 2021) as the perceptual model in VQGAN training (Esser et al. 2222), to pursue better tokenizer for BEIT pretraining. Despite of the progress, most existing studies remain operating on low-level image pixels, this work explores how to promote masked image modeling from pixel-level to semantic-level.
5 CONCLUSION
We proposed vector-quantized knowledge distillation (VQ-KD) to train visual tokenizer for vision Transformer pretraining. VQ-KD discretized a continuous semantic space that provides supervision for masked image modeling rather than relying on image pixels. The semantic visual tokenizer greatly improved the BEIT pretraining and significantly boosted the transfer performance upon downstream tasks, such image classification, and semantic segmentation. Moreover, patch aggregation mechanism was introduced to explicitly encourage the model to produce global image representations, narrowing the gap between the patch-level pretraining and image-level representation aggregation. In the future, we would like to learn universal tokenizer that projects words and images into the same vocabulary, that we can conduct masked prediction for vision-language pretraining.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY
Details of VQ-KD training. BETT V2 pretraining, fine-tuning recipes are given in Appendix D and G The models used for VQ-KD training from the official repositories https: and https:: /tips::/github... http://ooooppoeciiiiie The datasets (e.g., ImageNet, and ADE20K) are derived from publicly available data buckets. The code can be found in the supplementary materials. We will also provide pretrained checkpoints to
E
github. ccm::/aceeooorreseerccc/
reproduce the numbers.
REFERENCES
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint 20Xi::2002.03555. 2022.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEiT: BERT pre-training of image Transformers. In International Conference on Learning Representations, 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision Transformers. arXiv preprint arXiv:2104. 14294, 2021.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation 2022.
learning. arXiv preprint
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision Transformers. ArXiv, htss/2104.0205)) 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional Transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171 4171-4186. Association for Computational Linguistics, 2019.
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. PeCo: Perceptual codebook for bert pre-training of vision Transform- ers. arXiv preprint arXiv:2111. 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. preprint arXiv:2010.11999) 2020.
Alaacldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint 12.12:2112.10740, 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming Transformers for high-resolution image synthesis. In CVPR. pp. 12873-12883, 2021.
Yuxin Fang, Li Dong, Hangbo Bao, Xinggang Wang, and Furu Wei. Corrupted image modeling for self-supervised visual pre-training ArXiv, abs/2202.03382. 2022.
Luyu Gao and Jamie Callan. Condenser: a pre-training architecture for dense retrieval. arXiv preprint arXiv:2104.08253 0825533, 2021.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: critical analysis of distribiitt generalization. In /EEE ICCV, 2021a.
10

---

Under review as a conference paper at ICLR 2023
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In IEEE CVPR, 2021b.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin transformer v2: Scaling up capacity and resolution. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748-8763. PMLR, 2021.
A. Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. ArXiv, abs/2102.12092,, 2021.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 2015.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NeurIPS, NIPS' 17, pp. 6309-63-00 6318, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Haohan Wang. songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pp.
10506-10518, 2019.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: BEiT pretraining for all vision and vision-language tasks. ArXiv, abs/2208 10N4220800044,,, 2022.
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. arXiv preprint arX09:21/2.099133, 2021.
Longhui Wei, Lingxi Xie, Wengang Zhou, Houqiang Li, and Qi Tian. MVP: Multimodality-guidee visual pre-training. arXiv preprint arXiv:2203.05172 2022.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang. Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint arXiv:21
2021.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision Transformers. arXiv preprint 0rXiv:2106.045600 2021.
Bolei Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. Int. J. Comput. Vis., 127(3): 302-321, 2019.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT pre-training with online tokenizer. In International Conference on Learning Representations, 2022.

---

Under review as a conference paper at ICLR 2023
A VISUALIZATION OF CODEBOOK
It is observed that discrete code tends to represent explicit semantics (Section 3.4 per), we show image examples corresponding to a given discrete code. One can see that discrete codes ignore image details, such color, illumination, rotation and scale.
In Figure up-
In the lower part of Figure 5 we also show some patches that mismatch the semantic concepts. Taking the fish (the first image the last row) instance, VQ-KD misclassifies the spot the fish body as the eye concept due to the local structure similarity.
Verra: Tongue
V2use Nose
Vesse Human ear Vrasc Eye
Varret Tongue Vresu Eye..
Figure 5: Visualization of image patches corresponding to discrete codes. Upper: examples matching the learned semantic concepts; Lower: patches mis-matching the semantic concepts. Corresponding
patches are marked in red rectangle
B COMPARISON WITH LARGE-SCALE SUPERVISED PRETRAINING
We report the performance by using the ImageNet- Ik for pretraining Table I To show the data scalability of BEIT v2, conduct intermediate fine-tuning experiments on ImagNet-2 lk and final fine-tuning on ImageNet- 1k, by using the 1600 epoch pretraining models in Table T From Table 7 BEIT v2 using ViT-L/16 with 384 384 input resolution, achieves 89.0% top-1 accuracy, which outperforms ViT-H/14 using Google JFT-3B labeled dataset by 0.5%. This significant performance gain indicates the data efficiency and superiority of the proposed BErT v2.
12

---

Under review as a conference paper at ICLR 2023
Table 7: Top-1 accuracy on ImageNet- IK fine-tuning. 224 and 384 denote model resolutions.
ImageNet-1k 1k
Models
Model Size
Labeled Data Size
224
384²
Supervised Pretraining on ImageNet-2/K
ViT-B/16 Dosovitskiy al.
2020
86M
14M
84.0
ViT-L/16
Dosovitskry et al.
2020)
307M
14M
85.2
ViT-H/14
Dosovitskay
2020
632M
14M
85.1
Supervised
Pretraining on Google IFT-300M (using labeled data)
VIT-B/16 Dosovitskiy et al.
2020
86M
300M
84.2
ViT-L/16 Dosovitskiy et al.
2020)
307M
300M
87.1
ViT-H/14 Dosovitskiy al.
2020
632M
300M
88.0
Supervised Pretraining on Google JFT-3B
VIT-B/16 (Zhai et al.
2021
86M
3000M
86.6
ViT-L/16 (Zhai et al.
2021
307M
3000M
88.5
BETT Pretraining on ImageNet-22 KK, and Intermediate Fine-T Fine-Tuning on ImageNet-22 K
BErT ViT-B/16 (Bao al.
2022
86M
14M
85.2
86.8
BEIT ViT-L/16 (Bao et al.
2022
307M
14M
87.4
88.4 .4
BETT v2 Pretraining on ImageNet- IK, and Intermediate Fine-Tuming ImageNet-211n K
BEIT V2 ViT-B/16 (ours)
86M
14M
86.5
87.5
BEIT V2 VIT-L/16 (ours)
307M
14M
88.4
89.0
C OVERALL FRAMEWORK FOR BEIT v2
We show the tokenizer training part and BEIT v2 pretraining part in Figure and Figure respec- tively. addition, we present the whole pretraining process Figure
Visual Tokens
12
23
34
45
98
876
65
321
ViT
1 12-norm
34
78
789
100
Tokenizer Encoder
99
888
777
666
Input Image
Patch Tokens from the 1-th Layer
Patch Aggregation
VIT Block
ViT Block L
MIM Head
34
VIT Block
VIT Block
MIM Head
34
Block
Masked Image
777
Only Used
[CLS] Token from the L-th
Pretmining
Figure 6: Overall framework for BEIT v2 pretraining.
13

---

Under review as a conference paper at ICLR 2023
D HYPERPARAMETERS FOR VQ-KD TRAINING
Table 8: Hyperparameters for training VQ-KD ImageNet-Iie
Hyperparameters
Values
Encoder layers
12
Decoder layers
{1.3} 3]
Hidden size
768
FEN inner hidden size
3072
Attention heads
12
Attention head size
64
Patch size
16 16
Codebook size
8192 x 32
Training epochs
100
Batch size
512
Adam |
(0.9, 0.99)
Peak learning rate
2e-4
Minimal learning rate
le-5
Learning rate schedule
Cosine
Warmup epochs
Gradient clipping
X
Dropout
X
Stoch. depth
Weight decay
le-4
Data Augment
RandommReizzeAndCrgg
Input resolution
224 x 224
E HYPERPARAMETERS FOR BEIT V2 PRETRAINING
Hyperparameters for BEIT v2 pretraining ImageNet- IK. denotes that the hyperparame- Table 9:
ters are adopted when the pretraining schedule is 300 epochs.
Hyperparameters
Base Size
Large Size
Layers
12
Hidden size
1024
FFN inner hidden size
3072
4096
Attention heads
12
16
Layer scale
0.1
le-5
Patch size
16 x 16
Relative pusitional embeddlngg
V
Shared relative positional embeddings
Training epochs
300*/1600
Batch size
2048
Adam
(0.9, 0.98* /0.999)
Peak learning rate
1.5e-3
Minimal learning rate
1c-5
Learning rate schedule
Cosine 10
Warmup epochs
Gradient clipping
3.0
Dropout
X
Drop path
07/0.1
Weight decay
0.05
Data Augment
RandomResizeAnCCrop
Input resolution
224 x 224 0.44
Color jitter
14

---

Under review as a conference paper at ICLR 2023
F HYPERPARAMETERS FOR IMAGE CLASSIFICATION FINE-TUNING
Table 10: Hyperparameters for fine-tuning BETT v2 on ImageNet-IK: K.
Hyperparameters
ViT-B/16
ViT-L/16
Peak learning rate
5e-4
5e-4
Fine-tuning epochs
100
50
Warmup epochs
20
Layer-wise learning rate decay
0.65
0.8
Batch size
1024
Adam
lc-8
Adam |
(0.9, 0.999)
Minimal learning rale
lc-6
Learning rate schedule
Cosine
Repeated Aug
Weight decay
0.05
Label smoothing
0.1
Stoch. depth
0.1
0.2
Dropoul
X
Gradient clipping
Erasing prob.
0.25
Inpul resolution
224 x 224
Rand Augment
9/0.5
Mixup prob.
0.8
Culmix prob.
1.0
Relative positional embeedlnngs
Shared relative positional embeddings
G HYPERPARAMETERS FOR ADE20K SEMANTIC SEGMENTATION FINE-TUNING
Table 11: Hyperparameters for fine-tuning BETT V2 on ADE20K.
Hyperparameters
ViT-B/16 ViT-L/16
Input resolution
512 x 512
Peak learning rate
(0.5, 0.8, 1.0)e-4 160K
Fine-tuning steps Batch size
16
Adam E
1c-8
Adam /
(0.9, 0.999)
Layer-wise learning rate decay
(0.75, 0.8, 0.85) 0
Minimal learning rate Learning rate schedule
Linear 1500
Warmup steps
Dropout
X
Stoch. depth
0.1
0.2
Weight decay
0.05
Relative positional embeddings
Shared relative positional embeddings
x
15


Reviews:
[{"review_id": "iXYM7tPXwg", "reviewer": "Reviewer_jP3q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an improved version of the BEiT method for\nself-supervised representation learning by predicting masked image\npatches. The improvements rest mainly in two incorporations:\n1) Encoding image patches using a ViT-based tokenizer that distillis the\nsemantic information from a CLIP model into the encoded image tokens.\n2) Adding a patch aggregation stage that makes class token aggregate\n   global information from the patch encodings.\n\nExtensive experiments show the effectiveness of the improvements in\nstandard ImageNet pre-training benchmarks for classification and\nADE-20K for semantic segmentation.\n", "summary_of_the_review": "The paper proposes methodologic contributions that significantly\nadvance the state-of-the-art  in large-scale pretraining for computer\nvision tasks.  The contributions are well justified and the paper present \nsufficient empirical evidence through extensive experiments.\n", "strength_and_weaknesses": "**Strengths**\n- The two contributions are well justified and the extensive\nexperimental results strongly validate the design decisions.\n\n- The method outperforms compared pre-training methods by \n  significant margins across diverse tasks.\n\n- The obtained representations are significantly more robust compared\n  to the state-of-the-art pretraining method MAE.\n  \n**Weaknesses/Questions**\n- Unless I missed it, it would be nice to explain how the projection\n  for reducing the code dimension for lookup is obtained.\n- In (2), there is no $\\beta$ multiplying the commitment term as in the\n  standard VQ-VAE, but isn't $\\beta=1$ equivalent to\n  directly optimizing the distance without stop-gradient? If\n  $\\beta$ was used, please specify its value.\n- I'm curious about how the clusters in Figure 4 where found. Was it\n  by visual inspection of occurrences of all (or many) of the codebook\n  elements? Have you compared this type of semantic association with the\n  baseline tokenizer?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VvcZb_2RpBK", "reviewer": "Reviewer_d8pL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper improves BEIT by introducing the Vector-Quantized Knowledge Distillation (VQ-KD) algorithm for better visual tokenizer training. A patch aggregation strategy is also introduced into the Masked-Image-Modeling (MIM) pretraining framework. Experimental results show that BEITv2 (this work) significantly outperforms other MIM pretraining methods.", "summary_of_the_review": "I mainly worry about the significance of the propsed method, considering the strong teacher required and more complex training stages. The novelty of the Patch Aggregation strategy is also limited. The authors are encouraged to address the above issues.", "strength_and_weaknesses": "Strengths\n- The idea of introducing a semantic-rich visual tokenizer into the BEIT framework is reasonable and interesting.\n- This paper is clearly written and easy to follow.\n\nWeaknesses\n- Using a teacher model for supervising the visual tokenizer training makes the whole pretraining procedure more complex and time consuming. Two additional training stages (teacher model training and tokenizer training) are required before the final masked image modeling pretraining when compared to other one-stage pretraining methods, which weakens the significance of this work. A detailed comparision of the GPU hours (number of used GPUs and training Days) of different training stages of BEITv2 and other competitors may help clarify this issue.\n- The final pretraining performance (evaluated with imagenet finetuning) strongly relies on the teacher model. As shown in Table 6, better teacher model (CLIP vs DINO) can consitently bring better performance of BEITv2. While, BEITv2 can improve DINO from 83.6 to 84.4, but can only improve CLIP from 84.9 to 85.0, which indicates that with stronger teacher, the performance gain of BEITv2 will rapidly decrease. When Patch Aggregation strategy is not used, the performance of BEITv2 even drops to 84.7 (Table 4), lower than the CLIP teacher, which further weakens the significance of this work.\n- The novelty of of introducing Patch Aggregation strategy is somewhat limited as it shows little difference from Condenser Head in Gao & Callan 's work (Condenser: a Pre-training Architecture for Dense Retrieval, 2021). The architecture and the usage (used for pretraining but dropped during fine-tuning) are all similar.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TPnkPq9K5Q", "reviewer": "Reviewer_3zyP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new masked image modeling method(BEIT v2). A new Vector-Quantized Knowledge Distillation helps the BEIT v2 explore the high-level semantics. Meanwhile, this paper introduces a patch aggregation strategy to enhance global semantic representation. Experiments on image classification and semantic segmentation show the good performances of BEIT v2.\n", "summary_of_the_review": "The BEIT v2 has good performance in image classification and segmentation. But I still have some questions as mentioned above, and I am glad to see the responses of the authors. If the authors can resolve my questions, I will increase my rating.", "strength_and_weaknesses": "# Strength\n## 1. VQ-KD\nThe VQ-KD follows the [1] using the l2 normalization and VITs to get codebook embeddings. To explore the high-level semantics, the VQ-KD reconstructs the feature of DINO or CLIP. The high-level semantics help the BEIT v2 have better performance.\n[1] 1.  Yu, Jiahui, et al. \"Vector-quantized image modeling with improved vqgan.\" arXiv preprint arXiv:2110.04627 (2021).\n## 2. Global semantic representation\nThe BEIT v2 tries to use the [CLS] token to learn the global semantic representation. Combining the global representation, the BEIT v2 may have the ability to understand the global information of the image, which is essential for image classification or other tasks.\n## 3. Performance\nThe performances in image classification and semantic segmentation are good.\n\n# Weakness\n\n## 1. VQ-KD\n###  First question \nHow to explain the BEIT v2 beyond the teacher models as shown in Table [6]. I think the upper limit of BEIT v2 is teacher models, since the reconstruction loss in VQ-KD can not be zero, there has a gap between the feature provided by the VQ-KD and the teacher.\n\n###  Second question \nThe authors may try RQ-VAE [2] as your basic model in VQ-KD. The paper extends the capability of codebook in channel dim, which may help the BEITv2 performance better.\n\n[2]. Lee, Doyup, et al. \"Autoregressive Image Generation using Residual Quantization.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n## 2. Global semantic representation\nIt is hard to say the [CLS] token to learn the global semantic representation in your experiments. As shown in Table [5], selecting the suitable $l$-th layer is essential to the performance. I think this is because the model not only learns the [CLS] token but also refines the other tokens in the patch aggregation strategy. In this situation, the patch aggregation strategy will let the layers($0$-th~$l$-th) predict better tokens (not the [CLS] token), and the [CLS] token is not important. The patch aggregation strategy seems like a kind of multi-scale loss and add the MIM loss in the selected layers.\n\nSo if the authors want to let the [CLS] token learn the global information better, I think you can stop the gradient computation of other tokens and just refine the [CLS] token.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MVQVru7o6I", "reviewer": "Reviewer_Y8Gd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": " - This work extends the study of BEiT, i.e. BERT-style pre-training of image transformers. Specifically, this work proposes to adopt a semantic-rich visual tokenizer distilled from the semantically-rich CLIP model, promoting the MIM process to focus more on semantic-level.\n\n - In order to learn better and compact codebook space, this work also exploits other practices adopted in previous vector-quantized modeling paper, such as factorized code, and L2 norm. Meanwhile, a patch aggregation strategy is introduced in MIM to enhance global semantic space.\n\n - Extensive experiments on image classification and semantic segmentation are conducted, which demonstrates the effectiveness of this method.", "summary_of_the_review": " - As promising results are provided in this work, it's also valuable to design better vector quantization modules for representation learning. However, the unfair comparisons and marginal improvement (comparing to directly using CLIP ) suspect the contribution of this work.", "strength_and_weaknesses": "#### **Strength**\n\n - Masked Image modeling and pre-training has been an important topic in computer vision. Different from predicting pixels directly (MAE-style), this work promotes the development of token prediction (BEiT/BERT-style) pre-training. \n\n - The authors conduct a variety of experimental studies. With the introduced changes, promising results on image classification and semantic segmentation are shown.\n\n - Linear probing has been a relatively weak point of MIM-based pre-training, compared to contrastive and self-distillation methods. However, this work upgrades the model's linear performance.\n\n#### **Weakness**\n\n - Albeit with promising performance, the major issue in this work is about the use of CLIP features. With VQ-KD training, the encoder outputs could be viewed as distillations from CLIP, which is further leveraged to guide the MIM training. This paradigm results in unfair comparison with previous works. (See summary)\n\n - The vision encoder of CLIP already shows a 76.2 zero-shot performance and 84.9(reported in Table 6) fine-tuned accuracy. Considering the significance of its raw features, it remains questionable of the value about such modified pre-training targets. With additional efforts and computation spent on VQ-KD and MIM pre-training, the final accuracy is only improved marginally. Given the context that proper knowledge distillation could further improve fine-tuned accuracy [1], the benefits of MIM pre-training are hindered.\n\n - Several tricks in ViT-VQGAN are adopted in this work. While it's non-trivial to exploit advances in generative modeling, there lacks discussion over these modifications such as L2 norm, and factorize codes (e.g. from 256-dim to 32-dim). Since ViT-VQGAN have already conducted studies on representation learning and shows improvement in linear probing. The technical contribution seems to be limited here.\n\n - As expressed in the paper, the patch aggregation strategy is designed to obtain enhanced global representations.  However, from the comparison in Table 1 and Table 5, the design choice seems to be adhoc and most performance elevation comes from the CLIP-distilled codebook. The role of patch aggregation lacks further investigation. \n\n[1] Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject