Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper:  - This work extends the study of BEiT, i.e. BERT-style pre-training of image transformers. Specifically, this work proposes to adopt a semantic-rich visual tokenizer distilled from the semantically-rich CLIP model, promoting the MIM process to focus more on semantic-level.

 - In order to learn better and compact codebook space, this work also exploits other practices adopted in previous vector-quantized modeling paper, such as factorized code, and L2 norm. Meanwhile, a patch aggregation strategy is introduced in MIM to enhance global semantic space.

 - Extensive experiments on image classification and semantic segmentation are conducted, which demonstrates the effectiveness of this method.
strength_and_weaknesses: #### **Strength**

 - Masked Image modeling and pre-training has been an important topic in computer vision. Different from predicting pixels directly (MAE-style), this work promotes the development of token prediction (BEiT/BERT-style) pre-training. 

 - The authors conduct a variety of experimental studies. With the introduced changes, promising results on image classification and semantic segmentation are shown.

 - Linear probing has been a relatively weak point of MIM-based pre-training, compared to contrastive and self-distillation methods. However, this work upgrades the model's linear performance.

#### **Weakness**

 - Albeit with promising performance, the major issue in this work is about the use of CLIP features. With VQ-KD training, the encoder outputs could be viewed as distillations from CLIP, which is further leveraged to guide the MIM training. This paradigm results in unfair comparison with previous works. (See summary)

 - The vision encoder of CLIP already shows a 76.2 zero-shot performance and 84.9(reported in Table 6) fine-tuned accuracy. Considering the significance of its raw features, it remains questionable of the value about such modified pre-training targets. With additional efforts and computation spent on VQ-KD and MIM pre-training, the final accuracy is only improved marginally. Given the context that proper knowledge distillation could further improve fine-tuned accuracy [1], the benefits of MIM pre-training are hindered.

 - Several tricks in ViT-VQGAN are adopted in this work. While it's non-trivial to exploit advances in generative modeling, there lacks discussion over these modifications such as L2 norm, and factorize codes (e.g. from 256-dim to 32-dim). Since ViT-VQGAN have already conducted studies on representation learning and shows improvement in linear probing. The technical contribution seems to be limited here.

 - As expressed in the paper, the patch aggregation strategy is designed to obtain enhanced global representations.  However, from the comparison in Table 1 and Table 5, the design choice seems to be adhoc and most performance elevation comes from the CLIP-distilled codebook. The role of patch aggregation lacks further investigation. 

[1] Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation.
clarity,_quality,_novelty_and_reproducibility:  - Major clarity and presentation is clear, where as many typos exist: e.g. in Table 5: l-th Layer denotes **path** tokens.

 - While some interesting techniques are adopted in this work, it's unclear where the actual improvement comes from. Therefore, the novelty is still questionable.
summary_of_the_review:  - As promising results are provided in this work, it's also valuable to design better vector quantization modules for representation learning. However, the unfair comparisons and marginal improvement (comparing to directly using CLIP ) suspect the contribution of this work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an improved version of the BEiT method for
self-supervised representation learning by predicting masked image
patches. The improvements rest mainly in two incorporations:
1) Encoding image patches using a ViT-based tokenizer that distillis the
semantic information from a CLIP model into the encoded image tokens.
2) Adding a patch aggregation stage that makes class token aggregate
   global information from the patch encodings.

Extensive experiments show the effectiveness of the improvements in
standard ImageNet pre-training benchmarks for classification and
ADE-20K for semantic segmentation.

strength_and_weaknesses: **Strengths**
- The two contributions are well justified and the extensive
experimental results strongly validate the design decisions.

- The method outperforms compared pre-training methods by 
  significant margins across diverse tasks.

- The obtained representations are significantly more robust compared
  to the state-of-the-art pretraining method MAE.
  
**Weaknesses/Questions**
- Unless I missed it, it would be nice to explain how the projection
  for reducing the code dimension for lookup is obtained.
- In (2), there is no $\beta$ multiplying the commitment term as in the
  standard VQ-VAE, but isn't $\beta=1$ equivalent to
  directly optimizing the distance without stop-gradient? If
  $\beta$ was used, please specify its value.
- I'm curious about how the clusters in Figure 4 where found. Was it
  by visual inspection of occurrences of all (or many) of the codebook
  elements? Have you compared this type of semantic association with the
  baseline tokenizer?

clarity,_quality,_novelty_and_reproducibility: The paper is clear, high quality, and has significant novelty. The authors included the code as supplementary material for reproducibility.
summary_of_the_review: The paper proposes methodologic contributions that significantly
advance the state-of-the-art  in large-scale pretraining for computer
vision tasks.  The contributions are well justified and the paper present 
sufficient empirical evidence through extensive experiments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper improves BEIT by introducing the Vector-Quantized Knowledge Distillation (VQ-KD) algorithm for better visual tokenizer training. A patch aggregation strategy is also introduced into the Masked-Image-Modeling (MIM) pretraining framework. Experimental results show that BEITv2 (this work) significantly outperforms other MIM pretraining methods.
strength_and_weaknesses: Strengths
- The idea of introducing a semantic-rich visual tokenizer into the BEIT framework is reasonable and interesting.
- This paper is clearly written and easy to follow.

Weaknesses
- Using a teacher model for supervising the visual tokenizer training makes the whole pretraining procedure more complex and time consuming. Two additional training stages (teacher model training and tokenizer training) are required before the final masked image modeling pretraining when compared to other one-stage pretraining methods, which weakens the significance of this work. A detailed comparision of the GPU hours (number of used GPUs and training Days) of different training stages of BEITv2 and other competitors may help clarify this issue.
- The final pretraining performance (evaluated with imagenet finetuning) strongly relies on the teacher model. As shown in Table 6, better teacher model (CLIP vs DINO) can consitently bring better performance of BEITv2. While, BEITv2 can improve DINO from 83.6 to 84.4, but can only improve CLIP from 84.9 to 85.0, which indicates that with stronger teacher, the performance gain of BEITv2 will rapidly decrease. When Patch Aggregation strategy is not used, the performance of BEITv2 even drops to 84.7 (Table 4), lower than the CLIP teacher, which further weakens the significance of this work.
- The novelty of of introducing Patch Aggregation strategy is somewhat limited as it shows little difference from Condenser Head in Gao & Callan 's work (Condenser: a Pre-training Architecture for Dense Retrieval, 2021). The architecture and the usage (used for pretraining but dropped during fine-tuning) are all similar.
clarity,_quality,_novelty_and_reproducibility: - This paper is  well written with sufficient details, so it may not be diffcult to reproduce.
summary_of_the_review: I mainly worry about the significance of the propsed method, considering the strong teacher required and more complex training stages. The novelty of the Patch Aggregation strategy is also limited. The authors are encouraged to address the above issues.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new masked image modeling method(BEIT v2). A new Vector-Quantized Knowledge Distillation helps the BEIT v2 explore the high-level semantics. Meanwhile, this paper introduces a patch aggregation strategy to enhance global semantic representation. Experiments on image classification and semantic segmentation show the good performances of BEIT v2.

strength_and_weaknesses: # Strength
## 1. VQ-KD
The VQ-KD follows the [1] using the l2 normalization and VITs to get codebook embeddings. To explore the high-level semantics, the VQ-KD reconstructs the feature of DINO or CLIP. The high-level semantics help the BEIT v2 have better performance.
[1] 1.  Yu, Jiahui, et al. "Vector-quantized image modeling with improved vqgan." arXiv preprint arXiv:2110.04627 (2021).
## 2. Global semantic representation
The BEIT v2 tries to use the [CLS] token to learn the global semantic representation. Combining the global representation, the BEIT v2 may have the ability to understand the global information of the image, which is essential for image classification or other tasks.
## 3. Performance
The performances in image classification and semantic segmentation are good.

# Weakness

## 1. VQ-KD
###  First question 
How to explain the BEIT v2 beyond the teacher models as shown in Table [6]. I think the upper limit of BEIT v2 is teacher models, since the reconstruction loss in VQ-KD can not be zero, there has a gap between the feature provided by the VQ-KD and the teacher.

###  Second question 
The authors may try RQ-VAE [2] as your basic model in VQ-KD. The paper extends the capability of codebook in channel dim, which may help the BEITv2 performance better.

[2]. Lee, Doyup, et al. "Autoregressive Image Generation using Residual Quantization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

## 2. Global semantic representation
It is hard to say the [CLS] token to learn the global semantic representation in your experiments. As shown in Table [5], selecting the suitable $l$-th layer is essential to the performance. I think this is because the model not only learns the [CLS] token but also refines the other tokens in the patch aggregation strategy. In this situation, the patch aggregation strategy will let the layers($0$-th~$l$-th) predict better tokens (not the [CLS] token), and the [CLS] token is not important. The patch aggregation strategy seems like a kind of multi-scale loss and add the MIM loss in the selected layers.

So if the authors want to let the [CLS] token learn the global information better, I think you can stop the gradient computation of other tokens and just refine the [CLS] token.


clarity,_quality,_novelty_and_reproducibility: Quality: good.
Clarity: good.
Originality: fair.
summary_of_the_review: The BEIT v2 has good performance in image classification and segmentation. But I still have some questions as mentioned above, and I am glad to see the responses of the authors. If the authors can resolve my questions, I will increase my rating.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an extension to BEiT, which uses masked image modeling as a pretext task for pre-training Vision Transformers (ViTs). The paper proposes to train an image tokenizer to reconstruct the features of a teacher model (CLIP or DINO), which is then used during masked image modeling (MIM) pretraining of the ViT. The paper also proposes a new mechanism for patch aggregation in ViTs.

Reviewer responses to this paper were mixed, with Reviewers Y8Gd and d8pL giving slightly negative ratings, Reviewer 3zyP giving a slightly positive rating, and only Reviewer jP3q strongly in favor of the paper. Reviewers were concerned that the proposed method did not outperform the CLIP teacher by a wide margin, wanted more details on the training cost of the method, and had questions about the patch aggregation strategy. The authors provided extensive responses to all reviewer questions, but this was not able to fully satisfy the reviewers; in the post-rebuttal discussion Reviewer Y8Gd remained unconvinced by the author’s responses, and Reviewer 3zyP felt that the paper was borderline.

The AC met with Reviewers jP3q, Y8Gd, and 3zyP to discuss this paper. The major issue we discussed was the positioning of the paper, and the validity of using a pretrained CLIP teacher as part the pretraining procedure. Reviewers were not satisfied with the relatively small improvement over the CLIP teacher model. In addition, we discussed the issue of supervision and baselines: the paper compares primarily to prior work that trains with only *unlabeled* images; however since the proposed method uses a pretrained CLIP model, it implicitly has access to large quantities of image/text supervision. As such, for fair comparison the method should be compared to baselines that have access to the same level of supervision. All reviewers agreed that these were serious issues, and following this discussion all reviewers in attendance agreed that the paper is not ready for publication.
justification_for_why_not_higher_score: The paper primarily compares to prior SSL methods that learn from unlabeled images alone; however the method distills from a pretrained CLIP model, so implicitly uses much more supervision than SSL baselines.
Decision: Reject