Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a new method of using machine learning to classify images without the use of labels. Training consists of 3 parts: 
- The self training part where the teacher model generates pseudo-labels to train the student model to learn global tasks.
- The masked image modelling part where they train a model to predict missing pixels of an image in order to learn how to represent local pixels information.
- The part where they connect the knowledge learned from the two models in order to do the classification.
They use this method to investigate how well it performs on 8 image classification tasks and compares its efficacy to other models like CLIP that are known to perform really well in this field. MUST outperforms CLIP based models and seems to be using the relevant/defining parts of the image that correlate with the label to make the classification.
strength_and_weaknesses: Strengths
- The paper is well written and concepts are explained well.
- Given the high costs of labelling datasets, the authors explore alternative methodologies that try to overcome this barrier.

Weaknesses
- Given the authors' awareness of the negative social implications of CLIP pretrained models, what steps will/did you take to minimize these effects given that you will possibly share your trained models?
clarity,_quality,_novelty_and_reproducibility: Paper is clear and well written. Concepts are introduced and explained really well to allow a reader to be able to seamlessly follow along as they read. The experimental procedures was well articulated and sounds reproducible.
summary_of_the_review: Overall, I think the paper provides a significant contribution to exploring methods of overcoming the current issues we have with the cost of labelling datasets. The methodology is promising and I would look forward to future work extending it to apply to more diverse real world datasets.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors combine three different self-supervised methods into label-free image classification: Infomax, simsiam(BYOL) and masked encoder. In experiments, the results demonstrate significant improvement in different benchmarks.
strength_and_weaknesses: 1. Strength:
The authors propose a very effective method for label-free image classification. The performance is impressive in different benchmarks.

2. weakness:
However, I think novelty is debatable. Frankly, it looks like three different self-supervised methods are being used. EMA + L2 loss is BYOL [1], local-global contrast is deep informax [3] and masked autoencoder [2]. And obviously it losts the [3] in the reference. 


[1]. Bootstrap your own latent: A new approach to self-supervised Learning
[2]. Masked Autoencoders Are Scalable Vision Learners.
[3]. Learning deep representations by mutual information estimation and maximization.
clarity,_quality,_novelty_and_reproducibility: There is nothing obviously wrong with this paper other than novelty. 
summary_of_the_review: Because of novelty, I tend to give rejection to this paper. However I am happy to refer to other reviewers as I am only an expert in self-supervised learning and not CLIP related methods.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Paper proposes a method, MUST, for unsupervised adaptation of a zero shot classifier. A pre-trained open world model CLIP is taken to produce classification embeddings for a vocabulary of words. These are then use to kick-start a training method that adapts a zero-shot classification model by a combination of three objectives. The first one uses a self-training, second uses a masked image modeling objective and the third one enforces global local feature alignment.The proposed method is then evaluated on a variety of datasets.
strength_and_weaknesses: Strengths
- Clear and easy to understand writing
- Simple method.
- Significant performance improvements over CLIP based baseline approach.
- Meaningful and informative ablation experiments that study various aspects of the proposed approach.

Weaknesses
- Limited technical novelty, with the first two terms of the proposed loss already existing and the alignment term being the primary technical contribution.
clarity,_quality,_novelty_and_reproducibility: Clarity: Clear and easy to understand paper
Quality: Significant improvements over the baseline of CLIP and a thorough ablation study
Novelty: Limited, with the proposed loss being a sum of two existing methods (self training and masked image modeling) and a novel alignment term.
Reproducibility: The proposed methods is fairly straightforward, with code in supplementary.


summary_of_the_review: The paper proposes a simple methods (though limited in technical novelty), that leads to significant improvements in zero shot results. I vote accept.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes Masked Unsupervised Self-Training (MUST), that uses pseudo labels from CLIP as well as patch masking to train on unlabeled data. The loss consists of three terms a) self training classification loss, b) a masked patch loss, and c) a global-local loss designed to force class level information into the patch embeddings. 



strength_and_weaknesses: The paper's proposal is mostly clear well-motivated. 

Experiments are well thought out and clear and show a clear advantage over direct usage of CLIP across a variety of downstream
tasks. Many ablations probe the inner working of the loss and analyze what each factor is doing. 

A weakness could be the global-local loss, which I felt was the least grounded in any kind of motivation as it is the "average squared distance between the normalize embeddings of the [CLS] token and all [MSK] tokens for each image". However, this seems like this decision could be a bias towards datasets that contain a single large and well-framed object in the field of view. 



clarity,_quality,_novelty_and_reproducibility: The paper written in a clear and direct fashion. 

One small nit: it's not clear that Fig. 2 needs 2-3 different font styles. 


summary_of_the_review: The paper clearly proposes a method of performing self supervised training using CLIP and a pool of unlabeled data by constructing a loss from classification losses and masked patch losses. A wide variety of experiments are performed over a set of datasets that show the contributions of the various components of MUST.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces MUST (MASKED UNSUPERVISED SELF-TRAINING) to improve the performance of CLIP-like models in zero-shot recognition. Based on a pre-trained CLIP, three objectives are learned together to fine-tune it on a specific dataset in an unsupervised manner. Three proposed objectives are class-level global feature and pixel-level local feature and enforce a regularization between the previous two objectives. A fairness regularization loss is introduced in the class-level global feature objective. 
Experiments are conducted on 8 common datasets and the results clearly demonstrate the superiority of the proposed methods, which surpass baseline CLIP by large margins. The ablation study is solid, and the individual contribution of each component is demonstrated. 
strength_and_weaknesses: Strength:

1: Clear motivation. This paper is well-motivated.

2: Good writing and presentation, easy to follow/read. 

3: Very solid experiments and ablation study. Multiple datasets are evaluated. Robustness and calibration metrics are reported.

4: Good performance. The proposed method surpasses CLIP by large margins. 

Main weaknesses:

1: The best feature of CLIP is the generality, that is, being able to recognize any image without pre-defined/fixed classes. MUST adapt the CLIP model to a specific dataset (which is the main purpose of this paper). A simple solution is claimed by authors in the limitation section: "There exists a simple way to address this concern: gather unlabeled image from all the domains of interest, and perform MUST to learn a single model that can generalize to multiple domains."  It would be really great to have 1-2 such experiments to verify this hypothesis. 

Minor weaknesses: 

2:  The paper does not reach out to theoretical backup to explain why MUST works. 

3: Adding results on ImageNet-Sketch will further strengthen this paper. 

4: A related work [1] is worth discussing. 

[1]: Test-time training with masked autoencoders, NeurIPS 2022 

clarity,_quality,_novelty_and_reproducibility: Clarity: Good clarity, nice presentation.

Quality: Good paper, which clearly meets the acceptance bar of ICLR. 

Novelty: Not very novel. Incremental on MIM and self-training. 

Reproducibility: Training details and codes are provided.
summary_of_the_review: The authors addressed all of my concerns during the discussion. I'm leaning toward acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method of adapting a zero-shot model to a dataset such as ImageNet without using any labels from ImageNet. This process does improve model accuracy on ImageNet by a large margin, which is perhaps not too surprising as the target dataset is used for this adaptation. Their adaptation method consists mainly of masked image modeling and self-training from an EMA teacher.
strength_and_weaknesses: Strengths:
- This paper appears to be a very detailed, thorough, and reproducible empirical investigation and the results are potentially very useful to the community.
- Extensive ablations are performed on the design decisions. 

Weaknesses:
- I found the main weakness to be the experimental set-up - I believe this would be much more convincing if the paper considered a datasets which was not clean/balances such as LAION or CC12m for running their procedure.
- Some comparisons seem to be a bit unfair, for instance Table 3 in which MUST (seeing all images) is compared to methods seeing 16 images per class, what if MUST only sees 16 images per class.

Questions:
- What happens if you replace the self-training part with the real labels? Are the other losses still required?
clarity,_quality,_novelty_and_reproducibility: Overall the paper is very clear, novel, and various hyperparameter details are provided which makes it seem reproducible.
summary_of_the_review: Overall I enjoyed this paper, which is thorough and observes good empirical performance. My main concern is in the experimental set-up which takes a clean, balanced data-set then removes the labels.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Overall, all reviewers have been in favor of this submission but also some number of concerns showed up. For example, lack of theoretical analysis for the choices and somehow incremental design (EMA + L2 loss is BYOL [1], local-global contrast is deep informax [3] and masked autoencoder [2]). While the use of the above tools is interesting, the paper still lacks the necessary analytical depth to be recommended for a spotlight.
justification_for_why_not_higher_score: Most of reviewers liked the paper but also judging from their reviews, they did not challenge the paper in terms of its theoretical foundations. The lack of deep theoretical analysis for the choices and somehow incremental design as noted by one of reviewers (EMA + L2 loss is BYOL [1], local-global contrast is deep informax [3] and masked autoencoder [2]) make it hard for AC to recommend a spotlight. Masked learning, EMA etc. are all standard techniques frequently used currently in many papers. While the use of the above tools is interesting, the paper still lacks the necessary analytical depth to be recommended for a spotlight.
justification_for_why_not_lower_score: Overall reviewers were happy with the paper and did not voice any major issues.
Decision: Accept: poster