Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a self supervised learning framework GMML for vision transformers. The idea is essentially the same as MAE and SimMIM (the authors claim that MAE and SimMIM are extensions of GMML). When masking the inputs, they use noise or alien patches from other images, rather than just zeros. In this way, the initial blocks of the networks have to model the alien and the non-alien concepts, and thus can better model the context.
strength_and_weaknesses: Strength

1. The benefit from using noise or alien patches from other images, rather than just zeros when masking inputs is clear as shown in Figure 4(a).
2. On small datasets, GMML converge faster and perform better (Table 1)

Weakness

There are several critical weaknesses in experiments. 

1. Firstly, in the experiments, the authors only use ViT-tiny and ViT-small backbones. While it is more common to use a larger backbone for comparison (e.g. B, L, H) as in MAE and SimMIM.
2. Secondly, the authors missed the comparison of ImageNet-1k in Table 1, which is probably the most important comparison with other baselines.
3. Thirdly, some important baselines are missing for comparison, e.g. SimMIM, BEIT
clarity,_quality,_novelty_and_reproducibility: 
The authors claim that GMML is proposed “at the beginning of 2021” with no citation. They also claim that SimMIM and MAE are extensions of GMML. So, I have no choice but to search for GMML and found an unpublished paper on ArXiv [1].

Ignore [1], in my opinion, the novelty of this paper mainly lies in using alien patches when masking inputs.

[1] SiT: Self-supervised vIsion Transformer. 2104.03602
summary_of_the_review: Overall, the paper propose an interesting masking method using alien patches. However, the experiments are weak and the relationship with MAE and SimMIM is unclear.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper delivers a method called group mask model learning (GMML). GMML learns a representation of an image by partially corrupting it and reconstructing it with a transformer encoder-decoder structure. The authors conducted experiments that show GMML’s superiority in learning representations from small-scale datasets with small-sized models.
strength_and_weaknesses: - (C1) Models those capable of dealing with small-scale datasets are always welcome. However, assessing the model’s performance on large-scale datasets (at least ImageNet-1K) is needed to determine whether the model can also be used for pre-training those datasets. In that sense, the paper has its weakness in only delivering experiments on small-scale datasets and small-sized models.
- (C2) GMML is almost identical to SimMIM. The authors go on to argue that structures like SimMIM all stem from the GMML structures presented in 2021, but it is questionable why this paper submitted to ICLR 2023 should take the contribution of GMML presented in 2021. Since SimMIM is already presented in CVPR 2022, it is appropriate to address that this paper is conveying the variant of SimMIM, regardless of the existence of GMML architecture, that I failed to find its reference in the paper.
- (C3) Table 1 in the paper omits several papers, including SimMIM [1], BEiT [2], and SplitMask [3]. Especially, SplitMask shares topics with this paper: “SSL method that is capable of being pre-trained on small-scale dataset” and reports small datasets’ metrics seemingly on par with this paper’s result.
- (C4) The authors said that most non-GMML approaches suffer from trivial constant solutions, but the representation collapsing problem only occurs for positive (BYOL-like) methods. Contrastive learning does not suffer from the collapse by design and the number of works adopting contrastive learning is not neglectable.
- (C5) The manuscript has a self-contradicting statement that they have already suggested GMML in 2021 and picked a proposal of GMML as one of the paper's contributions.

[1] Xie, Zhenda, et al. "Simmim: A simple framework for masked image modeling." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2022.
[2] Bao, Hangbo, et al. "BEiT: BERT Pre-Training of Image Transformers." *International Conference on Learning Representations*. 2021.
[3] El-Nouby, Alaaeldin, et al. "Are Large-scale Datasets Necessary for Self-Supervised Pre-training?." *arXiv preprint arXiv:2112.10740* (2021).
clarity,_quality,_novelty_and_reproducibility: - (Clarity) I couldn't quite understand why the GMML model presented in 2021 should be a contribution to this paper. (see C2, C5)
- (Quality) The experiment presented in this paper is only a classification experiment on small-scale datasets, and it is too insufficient to claim the title "GMML is All you Need". (see C1) Even in the experiments performed, there are papers omitted. (see C3)
- (Novelty) I consider this model to be about the same as SimMIM.
- (Reproducibility) Source code is not provided, but implementation details in the appendix provide reproducibility.
summary_of_the_review: Due to the various weaknesses described above (see C1-C5) I cannot give a high recommendation to this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The work proposes Group Masked Model Learning framework for self supervised learning.
strength_and_weaknesses: Strength:
The work reports promising results for self supervised learning in low data regime. 



Weaknesses:
1. The novelty of the work is confusing as the work looks quite similar to other popular approaches for self supervised learning (such as MAE). However, the authors actually claim that the already published MAE (and also SIMMIM) extended actually the proposed work. 

2. There are other strong frameworks for self supervised learning that are not referred to in this work, for instance, data2vec (Baevski, Alexei, et al. "Data2vec: A general framework for self-supervised learning in speech, vision and language." arXiv preprint arXiv:2202.03555 (2022).) 

3. It is difficult to understand the contribution of the proposed approach on larger datasets. For instance, how this work results are compared with the state-of-the-art on the full ImageNet. Particularly, where the proposed approach fits in the Table 1 of the  data2vec paper.

4. Some parts are unclear, for instance, why the imput image is corrupted up to 70% in the case of “zeros” and “noise”, while in the case of “replace” it is corrupted up to 35%. Also not sure why the combination of “replace” and “noise” works better.

clarity,_quality,_novelty_and_reproducibility: -
summary_of_the_review: -
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a Group Masked Model Learning (GMML), a Self-Supervised Learning (SSL) mechanism
for pretraining vision transformers with the ability to extract the contextual information
present in all the concepts in an image. This is achieved by manipulating
randomly groups of connected tokens, ensuingly covering a meaningful part of
a semantic concept, and then recovering the hidden semantic information from
the visible part of the concept. The paper suggests that GMML implicitly introduces a novel data augmentation
process.

Experimental results are shown for Image Classification, DA in Transfer Learning, and a few types of ablations studies.

At the end of the manuscript, the authors claim the following - 
GMML based pretraining makes the vision transformers data-efficient. Also, GMML is the first
self-supervised pretraining work which consistently outperformed supervised pretraining for any
pretraining and finetuning dataset, regardless of their sizes.

The key impact of the proposed GMML is that it makes it possible for transformers
to train on small and medium size datasets. It is not only data efficient, but its outstanding information
extraction ability enables it to outperform state-of-the-art supervised and self-supervised
methods with large margins.

strength_and_weaknesses: Pros:
Transformers are hard to train, and providing any means to make them work for SSL
tasks, for DA/TL jobs is even harder.
Authors appears to address that problem to some extent,
Numerous results have shown the potential.

Cons
Although convinced by elaborate experimentations and ablations studies,
its difficult to get convinced due to lack of sufficient analytical proofs and justifications.

The manuscript has just one Eqn - the loss function used - with just the L1 vs L2 norm being discussed.
What about use of other loss functions - KLD and variants, Softmax, CE with variants etc. ?
 




clarity,_quality,_novelty_and_reproducibility: Paper quite clearly written.
Not much issues with the quality of presentation.

Eqn. (1) does not appear to be novel;
No other analytics provided for strengthening the proposed concept - convergence, impact of bias,..etc

Experimentations appear to be reproducible, although not much time to explore myself
and verify the same.
summary_of_the_review: 
I am not convinced with the paper.
There appears to be some contribution - but not substantial.


Note for Area Chairs and PC chairs:
This paper is available in
https://arxiv.org/abs/2205.14986
in which the author names are revealed.

The authors of above are a subset of those in [2] in this submitted manuscript.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: *Summary*: The paper proposes a framework for self-supervised learning that uses image masking and reconstruction. The authors propose to corrupt the input image using zero, noise patches or patches from different images. The method is evaluated on image classification benchmarks.

*Strengths*: (1) The method learns by masking the input which is a technique used in other domains such as text. (2) The masking strategy proposed in this work is different from other work such as BEiT, MAE, SimMIM, iBOT whereby the authors use zero, noise patches or patches from other images. (3) The proposed method works well on smaller network architectures and datasets. 

*Weaknesses*: (1) The main weakness raised by all the reviewers is the similarity of several published works from 2022 (SimMIM, SplitMask, MAE, BEiT, data2vec) and this work. While the authors argue that the original GMML predates these methods, the AC does not believe that the argument is relevant to the current paper (which by the authors own presentation is an improvement over the GMML work from 2021). Thus, as noted by every other reviewer, there are critical missing experimental comparisions. (2) The author response is incomplete as it misses specific questions raised by the reviewers (about loss function, analysis, comparisons). I do see that the authors address questions raised about the scale of the experiments which is useful context.
justification_for_why_not_higher_score: Many specific questions and comparisons raised by the reviewers are not addressed in the author response.
Decision: Reject