Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In the submission, the authors hope to explore a more efficient/reasonable masking strategy for training masked autoencoders (MAEs). Their intuition is that current masking strategies (e.g., random masking) cannot utilize semantic information contained in images, as what have been done in masked language modeling. Therefore, they proposed that masking semantic parts in the image would help training MAEs, named Semantic-Guided Masking. Based on this idea, the authors devised (i) a semantic part learning strategy for predicting semantic parts from the image; (ii) a MAE masking curriculum called semantic-guided MAE for gradually masking semantic parts in the image. Finally, experiments show the proposed semantic-guided MAE bring significant improvements compared to existing masking strategies.
strengths_and_weaknesses: Strengths:

1. In my opinion, the idea of this paper is interesting. Compared to randomly masking images, masking semantic parts seems reasonable, and experiments show it indeed brings improvements. Meanwhile, the idea of recovering semantic parts of an image giving other parts (or information) is an important research topic of image understanding, this paper should bring some new thoughts to the community.

2. Learning to predict semantic parts from images is a challenging task, especially under the unsupervised training setting. To execute the proposed semantic-guided MAE, the authors proposed a new semantic part learning strategy which is novel and useful. According to my understanding, the performance of following semantic-guided MAE is highly relying on semantic part segmentation results. It is impressive that using only a style-based generator and reconstruction loss can further refine the semantic part information extracted from a pre-trained ViT encoder.

3. This paper is well-written. The motivation and idea of this paper are clear and are well-explained. Although visual transformer (or MAE) made great progress in the past year, it is still chasing language modeling. I am glad to see papers that explore the differences and connections between language tokens and image tokens.

Weakness:

1. The whole pipeline of the proposed method seems a little bit complicated. It first learns to predict part segmentations from a pre-trained iBOT, then retrains an MAE from the scratch. I am curious how about training a random MAE first, and then iteratively performing part learning and semantic-MAE learning. I understand the authors listing it in their future work, yet it still can be discussed in this paper.

2. In table 4, `iBOT-initialized part' performs worse than the baseline model, the authors should have more discussion on it. How will the part learning influence the Semantic-MAE results?

questions: 1. In Line 147, I am wondering why the authors attempt to remove texture information, and the reason for choosing a large blur kernel. 

2. How do the authors decide the part segmentation results of stage-I being good enough? According to my understanding, for a fair comparison, no labels should be introduced.

limitations: Yes, related limitations have been discussed and there is no obvious negative societal impact.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: A semantic-guided masking modeling method is proposed in this paper. To achieve this, an iBOT-pretrained ViT is used to extract the features, and then StyleGAN-based decoder is trained to learn semantic parts. Finally, the learned semantic parts attention maps are used to guide the mask modeling per-training. Extensive experiments are performed on classification and semantic segmentation downstream tasks. 
strengths_and_weaknesses: # Strengths
- Semantic-guide MAE training is an interesting idea. But it does not convice me that the reason is just to have a potential visual analogue of words.  I encourage the authors to make it clearer why we need semantic-guide MAE training, and more efficient. 

# Weaknesses
- The proposed methods involved too many components, iBOT feature extractor, including StyleGAN-based decoder, an intermediate reconstruction tasks, gradually masking strategy, and et.al.  to make it work. The complicated whole framework makes it hard to analyze how the proposed method works. 
- It lacks the intuitive explanation why the proposed intermediate reconstruction tasks can help to learn the part attention maps. 
- The improvement on downstream tasks is marginal as shown in Table 6 and Table 7. 

questions: - Please see *Strengths And Weaknesses*
- In the end of the line 155: convolutional **can** AdaIN layers --> convolutional **and** AdaIN layers
- How about the total training cost including the iBOT pre-training, part learning, semantic guide pre-training, compared to the other pre-training methods. 
limitations: The proposed method assumes that the images are object-centric. It's true for ImageNet dataset. How about the proposed method on none-object images, e.g. texture images, scene images, and et.al?
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed a new semantic-guided masking strategy for unsupervised vision transformer training. The paper claims that semantic parts of objects are the visual analogue of words, masking semantic parts from intra-part patterns to inter-part would help the vision transformer learn better representations. For such purpose, the paper first designs a self-supervised semantic part learning method that aims to generate semantic parts of images from the ImageNet dataset. Then, the proposed semantic-guided masking strategy performs the unsupervised model training. Most of the experiments were performed on the ViT-base backbone and showed improvements on unsupervised representation learning.
strengths_and_weaknesses: [Strengths]
1.This paper is well organized. It raises an interesting and reasonable assumption, i.e., semantic parts in an image carry important semantic information and may benefit unsupervised masked training strategy. To explore the difference between language tokens and image tokens seems a meaningful topic and this paper seems to find a new way to define semantic tokens for vision transformers.

2.If the ground-truth semantic part segmentations are given, the comparisons with existing MAE methods should be unfair. Thus, the paper designs a self-supervised semantic parts prediction model. We can see that the semantic part segmentation results would largely influence the masked learning results. It seems the proposed semantic part learning method works well in exploring parts' information that is contained in the representations extracted from a pre-trained transformer.

3.The experimental settings are reasonable and related results are promising. Most claims in this paper can be supported by their experiments.

[Weaknesses]
1.Besides objects (or semantic parts), the background is an important and large part contained in images. This paper does not discuss how to deal with the background. If we regard the semantic parts as the visual analog of words, what is the background for? How should we treat it?

2.I am wondering why this paper chooses the masking training strategy to validate its idea (i.e., semantic parts of objects are the visual analogue of words). Are there any other tasks related to this idea? In addition, what is the relationship between the idea and the masking strategy (i.e., masking parts from intra-part patterns to inter-part)? These questions were not well explained?
questions: 1.For the proposed SemMAE, decreasing patch size largely improved its performance. Please explain the reason behind this and compare it to the opposite behavior in both SimMIM and the original MAE.

2.Why six parts per image were selected in the paper? For different kinds of objects in the ImageNet dataset, what are the six parts corresponded to?
limitations: Yes, the authors adequately discussed the limitations of this paper. Meanwhile, potential negative societal impacts are not found yet.
review_title: null

Meta-review: metareview: Authors present a method attempting to perform Masked Auto-Encoding (MAE) using semantic knowledge, to try to better approximate the semantic MAE seen in language domain. To do this, they leverage an iBOT framework and add some embeddings of the class token to create "part tokens", which are then compared to patch tokens from iBOT to produce attention maps. The objective for this process is a StyleGAN-based image reconstruction.

Once the part attention maps training is done, the network is then used to guide semantic based part masking based on the generated attention maps, for semantic MAE.  

SemMAE pretrained networks are then compared against other forms of SSL pretraining on ImageNet 1k, iNa, CUB, Cars, and ADE-20K, demonstrating improvements in all domains.

Pros:
- [R] Idea is interesting / novel
- [R] Well written
- [AC/R] Results improve over baselines	
	

Cons:
- [AC/R] Pipeline is complicated.
- [R] What about starting from random MAE and then adapting based on parts knowledge? Authors respond that this is future work.
- [AC/R] Not convinced parts are visual analog of words. Authors provide benchmark improvements in performance, and qualitative visualization of the parts. However, there is no quantitative assessment of the parts and whether they have true semantic meaning.
- [AC/R] Some improvements are marginal. Authors respond that although marginal in some cases, they are consistent. 
- [R] Paper does not discuss more how to deal with background. Authors respond that this is future work.

Overall, all reviewers have changed their assessments to accept, including the one reject reviewer. AC recommends accept, though would be preferable if quantitative assessment of the quality of the semantic parts (for example, by segmentation masks) could be provided.

AC Rating: Accept
confidence: Certain
Decision: Accept