Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a new framework for self-supervised learning from natural images. The proposed objective is a linear combination of two previously proposed objectives, namely student-teacher self-distillation (as in DINO), and masked-language modeling as is commonly used in NLP. The masking strategy is guided by the attention masks learned by the transformer backbone. The authors evaluate their model on standard image-SSL benchmarks: ImageNet classification and COCO detection and segmentation.   
main_review: The main weakness of the work is its novelty. The proposed objective simply adds a masked-language modeling loss to the objective in DINO. The question is whether the improvement upon DINO is worth the additional complexity, conceptual and otherwise. The improvement upon DINO is on the order of 1% for linear classification on ImageNet, detection on COCO, and segmentation on Cityscapes. This is fine in itself, the issue is that it is unclear whether there is much room to build on the work, as it is itself a simple combination of existing objective. 

The author do highlight the importance of guiding the masking strategy with the learned attention masks however, showing it performs much better than random masking. This constitutes the main technical contribution of the paper, but wether this in itself justifies publication is unclear. 
limitations_and_societal_impact: Yes, the authors adequately addressed the limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This submission combine the task of mask language modeling (MLM) and a contrastive learning task for self-supervised visual representation learning. The authors claim that due to he preserve of spatial information of an image, the proposed approach is more friendly to the downstream dense prediction tasks. The effectiveness of the approach is verified on ImageNet-1K linear evaluation task, and downstream tasks of COCO object detection and ADE20K semantic segmentation.
main_review: Originality

-- While the MLM alone shows inferior performance (iGPT and in ViT), this submission tries MLM as a complementary pretext task to the conventional contrastive method, and demonstrates superior performance than contrastive learning alone. It also proposes a new masking strategy according to the importance of a token. The originality is acceptable for NeurIPS but not outstanding.

Quality

-- The quality is borderline. The combination approach makes sense. But the effects of specific designs are not well ablated, and it is thus unclear to the readers what the real effective components are:
1) what is the performance by contrastive learning alone? Masking can be also regarded as a kind of data augmentation. Does it help?
2) what is the performance by mask language modeling alone?
3) can the two pretext tasks really benefit each other？ what is the effect if \lambda_2 varies?
4) Is random masking well tuned? seems it may need very small p, but the results at smaller p such as 0.01 are not provided. Does the poor performance by random masking come from poor contrastive learning or poor mask language modeling.
5) what would it perform with longer training, e.g. 500 or 800 epochs?
6) why there are no finetuning results on COCO and ADE using 300-ep pretrained models?
7) What is the speed compared to contrastive approaches? It seems that the masking strategy requires a pre-compute of the teacher network, and the two branches must be conduct in sequential. Does this sequential computation affect running speed?

Clarity

--Overall, the method is presented clear. But there lack a lot of details and hyper-parameters: are the two image views the same cropping/resizing augmentation? it seems to be different as in Line 205, but if the two views are different, how do you estimate \tau by the teacher network as their patches are not the same?

Significance

--Satisfied if the components are solidly verified.




 ablation is not  
limitations_and_societal_impact: Please see the questions above.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a new self-supervised learning method based on a combination of mean-teacher distillation (in this case DINO) and masked patch modelling (like masked language modelling in NLP). They use this to train transformer based vision models and show downstream results for ImageNet classification using linear probing and knn eval, MS COCO object detection and instance segmentation using Mask-RCNN framework and Cityscapes semantic segmentation using SERT framework.

Their key technical contributions are (a) the overall training framework combining DINO with masked patch modelling. The latter requires a CNN decoder and reconstructs the original global image and not just the masked patches and (b) a smart mask generation scheme based on the attention map of the CLS token which turns out to be crucial for good results. They use the attention maps corresponding to the CLS token to select patches that are not important. The idea is that if important regions of the image are masked out then the model will end up modelling only local statistics.

At a deeper level, their method is trying to retain spatial structure in the latent representation while capturing global image properties. If I understood correctly, the spatial structure comes from masked patch modelling and global image properties come from DINO.
main_review: Both technical contributions listed above are novel to the best of my knowledge. The method is of significance to the community as self-supervised learning now outperforms supervised pre-training and could potentially help the community save on data annotation costs. It is also significant in that a lot of prior self-supervised learning literature learns global image representations which may not adequately preserve spatial structure. The proposed method addresses this issue.

The performance on established benchmarks is strong. The explanation is mostly clear. Some areas for improving the paper are discussed below.

Citations
------------
Please cite 
* [1] Self-Supervised Representation Learning from Flow Equivariance. Yuwen Xiong, Mengye Ren, Wenyuan Zeng, Raquel Urtasun, Tech report, arXiv, Jan. 2021 which like DenseCL and PixPro also learns dense image representations for downstream detection and segmentation tasks.
* [2] Unsupervised Learning of Dense Visual Representations. Pedro O. Pinheiro, Amjad Almahairi, Ryan Y. Benmalek,
Florian Golemo, Aaron Courville, NeurIPS 2020 is also such a method.
* [3] https://papers.nips.cc/paper/2020/file/c1502ae5a4d514baec129f72948c266e-Paper.pdf also focus on downstream segmentation and identify the same issue that MoCo learns only global image representations.
* In line 179, what does adversarial effect refer to? A citation here might help clarify what the authors mean.
* Numbers for prior methods in the experiments section were in some cases taken from prior papers. Making the source of each number explicit in the table caption will make it easier for the reader to dig out the technical details for each entry. Also especially clarify if any of these numbers are based on you running the opensource code from prior work.
* In line 201, does "former works" refer to DINO?
* Vision Transformers are a new phenomenon. Unlike ResNet and VGG most readers would not know what DeIT-S and Swin-T are. A 3-5 line summary of these models will be of great value in the related works section.

Important missing ablations
---------------------------------------
* What is the performance of just L_restoring? Does it have to be combined with DINO for this to work? Concurrent work on MLM for Vision Transformer (https://arxiv.org/pdf/2106.08254.pdf) is able to work purely using an MLM objective. Answering this question in an ablation experiment will be super useful to the community.
* In line 174, the authors mention that unlike original MLM where the loss only looks at masked tokens, they reconstruct the full input image. Please include an ablation to support this design choice. Also clarify how multi scale cropping effects the MLM part of the method. Is the MLM part also exposed to lower resolution crops?
* In line 178, the authors argue how inductive biases in CNNs inspire them to use CNNs to decode the image. One could alternatively just decode using a projection into 16x16x3 patches and concatenate them together, or decoder into a lower resolution image. Basically how crucial is it to have the CNN inductive biases in this part of the training framework?
* Masking is done after the linear projection. I found this only when looking carefully at figure 1. Is this important?

Differences with BERT
------------------------------
* BERT randomly replaces the masked positions with the learned [mask] token or with other words. The image equivalent would be to replace chosen patches with either the [mask] token or other patches. However, the proposed method only replaces chosen patches with [mask]. Did the usual BERT approach not work? Sorry I am not familiar with NLP literature, so maybe this is no longer the current practice in NLP either.
* Do NLP methods also need to be some attention guided masking? If not, then why do those methods not suffer from masking out important words during training?

Minor issues
-----------------
* Table 6 caption: Impact of batch normalization "in projection head".
* Line 283, global image decoder is further used "under" the attention-guided mask ...
* Line 167, 't' is a new variable. Did you mean 'e'?
* Line 196, 50k images not 5k.
* Line 183, what is $H^n$?
limitations_and_societal_impact: Limitations are not discussed. What would be something the community should address moving forward?

Societal impact is not discussed. The authors state that "This work dose not present any foreseeable societal consequence.". The method improves object detection and instance segmentation performance. These datasets prominently include the "person" category. Some suggestions are below.

Potential positive societal impact:
* Improved road safety in autonomous driving by detecting pedestrians.
* Safe human robot collaboration in factories with robots taking on risk prone jobs thus saving lives.
* Assistive robots for elderly care.

Potential negative societal impact:
* Large scale drone surveillance.

The societal impact of anything in computer vision is massive! I agree that the above areas are not a direct outcome of the proposed method but are a step in that direction.

There are no error bars. The authors say error bars are in the appendix. But they are missing. Training for 100 epochs makes it super difficult to repeat the experiment 3-5 times. If possible, maybe just run the main proposed setting 3 times at 100 epoch schedule and do knn eval on imagenet. This way we can get some idea of the variance.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes to add an image reconstruction task to the existing instance discrimination task, as the pre-text tasks for self-supervised learning. Specifically, the self-attention map from the teacher model is used to guide the random mask process, that the attended foreground regions will not be masked. A decoder is used to restore the original image from the masked image.
main_review: The overall idea of combining image reconstruction and instance discrimination as pre-text tasks makes sense. 
However, the paper is poorly written. Both the ablations and analysis are insufficient to demonstrate the effectiveness of the proposed method.

- The authors are suggested to provide the experiments of only removing the reconstruction loss. Because the gains could be from the data augmentation itself.
- More ablation study is needed. For example, random mask strategy with different sampling ratios,  the impact of loss weight $\lambda_2$.
- The authors are suggested to demonstrate their method by first reporting the results of the baseline and then adding each small component of the proposed method.
- L181, what's 'global crops'. 
- The authors should pay attention to the notations. For example, what's 'Q' in Eq.(8), 't' on L167, 'num' on L152, 'd' on Eq.(4), 'prob_j' in Eq.(3). Although I can get the meaning by guess or by experience, they should be clearly explained.


Other minors:
- L20-L21, it's meaningless to report a single number on downstream detection and segmentation tasks. Reporting the gains from the reconstruction task should make more sense.
- L225, COCO has 80 categories, not 81.
- Many typos and grammar errors, e.g., L25, L57, L114, L145, L166.

## After Rebuttal
The rebuttal and discussion have addressed most of my concerns. Thus I raise my rating from 4 to 6.
The authors are suggested to add all these ablations and discussion, and further revise the paper, especially the presentation.
limitations_and_societal_impact: The authors didn't provide a statement of broader impact.
review_title: null

Meta-review: 
MST proposes to combine the task of mask language modeling with instance discrimination. 

While adding non-negligible complexity to training, MST demonstrate gain of the order of 2% on various standard SSL tasks (ImageNet linear eval, MS-COCO, Cityscape). Additionally, reviewers all agreed that the ablation experiments performed during the rebuttal period clarified the importance of the contributions. Overall, it is unclear if the performance gain fully justifies the extra-complexity of the training pipeline.

Given the positive feedback from the reviewers and the good empirical results, I am in favor of acceptance

Decision: Accept (Poster)