Masked Unsupervised Self-training for Label-free Image Classification 
Authors: Junnan Li
Silvio Savarese
Steven Hoi
Venue: ICLR 2023 poster (2023)
Keywords: zero-shot classification
unsupervised learning
self-training
CLIP
masked image modeling

Abstract: State-of-the-art computer vision models are mostly trained with supervised learning using human-labeled images, which limits their scalability due to the expensive annotation cost. While self-supervised representation learning has achieved impressive progress, it still requires a second stage of finetuning on labeled data. On the other hand, models pre-trained with large-scale text supervision (e.g., CLIP) have enabled zero-shot transfer to downstream image classification tasks. However, the zero-shot performance of CLIP-like models are often insufficient for real-world adoption. In this paper, we aim to leverage the abundant unlabeled data from a target domain to improve the performance of a pre-trained zero-shot classifier, by unsupervised finetuning of the pre-trained model. We propose Masked Unsupervised Self-Training (MUST), a new approach which leverages two different and complimentary sources of training signals: pseudo-labels and raw images. MUST jointly optimizes three objectives to learn both class-level global feature and pixel-level local feature and enforces a regularization between the two. We demonstrate the efficacy of MUST on 8 downstream tasks across a variety of domains, where it improves upon CLIP by a large margin. MUST also outperforms supervised few-shot adaptation methods. It achieves a top-1 accuracy of 77.7% on ImageNet using ViT-B, +9.4% higher than CLIP, and +6.2% higher than 16-shot CLIP adaptation. Our code is available at https://github.com/salesforce/MUST.

Published as a conference paper at ICLR 2023
MASKED UNSUPERVISED SELF-TRAINING FOR LABEL- FREE IMAGE CLASSIFICATION
Junnan Li, Silvio Savarese, Steven Hoi
Salesforce Research (junnan. 11, ssavarese, shoi @esalesforce.cm
ABSTRACT
State-of-the- art computer vision models mostly trained with supervised learning using human-labeled images, which limits their scalability due to the expensive annotation cost. While self-supervised representation learning has achieved impres- sive progress, it still requires a second stage of finetuning on labeled data. On the other hand, models mre-trained with large-scale text-image supervision (e.g., CLIP) have enabled zero-shot transfer to downstream image classification tasks. However, the zero-shot performance of CLIP-like models often insufficient for real-world adoption. In this paper, we aim to leverage the abundant unlabeled data from target domain to improve the performance of pre-trained zero-shot classifier, by unsu- pervised finetuning of the pre- trained model. We propose Masked Unsupervised Self-Training (MUST), a new unsupervised adaptation method which leverages two different and complementary sources of training signals: pseudo-labels and raw images. MUST jointly optimizes three objectives to learn both class-level global feature and pixel-Ievel local feature and enforces regularization between the two. We demonstrate the efficacy of MUST on a variety of downstream tasks, where it improves upon CLIP by large margin. MUST also outperforms supervised few- shot adaptation methods. It achieves top-1 accuracy of 77.7% on ImageNet using ViT-B, +9.4% higher than CLIP, and +6. +6.2% higher than 16-shot CLIP adaptation. Our code is available at
com sa l es worceerreeii MMUST
INTRODUCTION
Zero-shot image classification is challenging goal that marks the capability of a vision model to solve tasks without human supervision. Recently, vision-language pre-training (e.g. CLIP (Radford et al. (02T)) has shown promising performance open-vocabulary zero-shot classification, where leverages web-scale image-text pairs to train image and text encoders that can be transferred to downstream tasks through natural language prompting. However, the zero-shot performance of CLIP often inadequate for real-world adoptions, especially when compared to models that are trained with supervised learning. On the other hand, there are abundant unlabeled data available for many tasks. In this paper, we aim to improve the performance of an open-vocabulary zero-shot classifier by finetuning it unlabeled images from a downstream task.
Given unlaheled data, the key question is: what is the source of supervision? Numerous papers have attempted to answer this question. Among them, self-training and self-supervised learning are of the most dominant approaches. In self-training (Zoph et al. 2020: Xie et al. 22020; Lee et al. 2013)) pseudo-labels are generated by teacher model and then used supervise task-specific training of student model, where the student usually has an equal model size as the teacher. On the other hand, self-supervised learning methods are generally task aask-agnostic. Masked image modeling (Bao et al.. 2022 He et al.) 2021 Xie et al. (022)), which trains the model to predict missing information from masked image patches, has recently emerged as the superior self-supervised learning method for vision transformers (ViT) (Dosovitskiy et al., 2021). However, both self-training and self-supervised learning have their limitations. Self Self-training overly relies on the pseudo-labels as the only source of supervision, thus is prone to overfitting to the noise in pseudo-labels. Self-supervised learning requires an additional stage of task-specific finetuning on labeled data, thus is not a one-stop solution.
In this paper, we propose Masked Unsupervised Self-Training (MUST), simple and effective method for label-free image classification. MUST performs unsupervised learning using both pseudo-labeia and raw images two different and complementary training signals. Specifically, MUST jointly

---

Published as a conference paper at ICLR 2023
optimizes three objectives to finetune pre-trained classification model (e.g. CLIP) on unlabeled images: (1) teainnnai objective to learn global task-specific class prediction; (2) Masked image modeling objective learn local pixel-level information; (3) Global-local feature alignment objective to bridge the knowledge learned from the two sources of supervision.
We validate the efficacy of MUST on 8 image classification tasks across variety of domains, showing significant improvement over CLIP (Radford et al., [021]. MUST also outperforms supervised few- shot adaptation methods (Zhou al., [021] ; 2022). For instance, MUST achieves 77.7% top-1 accuracy on ImageNet, +9.4% higher than CLIP, and +6.2% higher than 16-shot CLIP adaptation. On certain domains, MUST can achieve comparable performance with a fully-supervised method. We further perform extensive quantitative and qualitative analysis to examine the effect of each proposed component. MUST a low-cost solution for image classification that unlocks the potential of CLIP-like models for practical scenarios where images are abundant but labels are scarce.
2 RELATED WORK
Zero-shot learning traditionally aims to recognize unseen classes by training the model on base
classes (Xian et al. 2017 Wang et al., 2019), where the most common approach is to utilize auxiliary information such as attributes (Huynh Elhamifar 2020) knowledge graphs (Wang iet al. 2018). CLIP (Radford et al. (021) popularizes a new approach for open-vocabulary zero-shot image classification by leveraging natural language supervision from web-scale datasets. Despite its impressive performance, the zero-shot accuracy of CLIP stills falls far below the supervised method in many domains. Some recent work tries adapt CLIP to downstream tasks using labeled data (Zhou et al. [021] Gao et al., [202], which is less scalable compared to our unsupervised adaptation method. Our method is also orthogonal to CLII--like research in model pre-training and can be applied to
other image classification models (Jia et al. 1 Li al., 2022 Yao et al. 2022 Li al. 2022) which performs two-stage pre-training.
202 I).
such as LiT (Zhai et al.
Self-training has shown promising progress in many domains including vision Zoph et al. 2020 Xie cet al. 2020 Sahito et al. 2022). NLP (He et al. 2020a)), and speech (Kahn et al. 2020). Our method is
more closely related to the self self-training approaches proposed for semi- cemi-supervised learning (Tarvainen
& Valpola 2017: Sohn et al. 2020 Berthelot et al. (020)) where pscudo-labels on unlabeled data
self-training objective by following three principles: (1) consistency regularization (Sohn et al.) 2020] Laine & Aila 2017) which enforces the model to output the same prediction when the input is perturbed; (2) entropy minimization (Grandvalet & Bengio 2004) which encourages the model to give "sharp" predictions with low entropy: (3) prediction fairness (Berthelot al., 2020) which alleviates the model's bias towards certain classes. Our method leverages a separate supervision signal from raw image pixels to reduce the confirmation bias in self-training which is orthogonal to methods for pseudo-label debiasing (Wei et al. 2021
used training targets. We construct
Wang et al. 2022).
Masked image modeling, fueled by the success of vision transformers (Dosovitskiy et al. 2021), ) recently emerged as a more appealing sulf-supe representation learning method over contrastive
learning (He et al.) 2020b; Chen et al.) 2020 Hjelm et al. 2019). While some methods train the model to predict discrete tokens (Bao et al. 2022) or contextualized representations (Baevski et al. 2022) for masked image patches, MAE (He et al., 2021) and SiMMIM (Xie et al., (022) achieve competitive performance by simply predicting the pixel values. The MIM objective has also been used for test-time training (Ginndelsman et al. (022). Different from existing self-supervised learning methods which require an additional stage of supervised finetuning on labeled data, synergically incorporate masked image modeling into self- self-training as a one-stage solution for zero-shot image classification.
3 METHOD
MUST is simple unsupervised learning approach that adapts a pre-trained open-vocabulary classifier to downstream task using unlabeled images. In this paper, we consider vision transformers pre- trained by CLIP (Radford et al. 202I) as the models to be adapted due to their distinctive zero-shot performance. CLIP pre-traie treitrains an image encoder and text encoder with contrastive loss such that paired images and texts have higher similarities in a shared embedding space. In order to perform zero-shot classification, CLIP converts set of class names into text embeddings using ensemble of
2

---

Published as a conference paper at ICLR 2023
prediction
linear dasssfier
normalized text embeddings
pug
Project
detach
Image Encoder
Text Encoder
photo of a (abject).
grey fox
Norm
KNN
husky
CL.PPs embeddings into linear classifier, which fine-tuned together with the image encoder for unsupervised adaptation.
Figure 1: On each downstream task,
natural language prompts (e.g., a photo of a (object)). ). During inference, it takes the dot-product between image embedding and all text embeddings produce the prediction logits for that image. As shown in Figure [] we convert CLIP's non-parametric text embeddings into weights linear classifier, and directly finetune the linear classifier together with the image encoder for unsupervised adaptation.
Figure shows the overall framework of MUST. Given image, we follow VIT (Dosovitskiy et al. 2021) to divide it into regular non-overlapping patches. A [CLS] token is appended to extract global information, which used by the classifier for prediction. We then randomly mask image patches by replacing patch's s embedding with learnable [MSK] token. The output embeddings of [CLS] and [MSK] are used jointly optimize three objectives: (1) global self-training, (2) local masked image modeling, and (3) global-local feature alignment. Next we delineate the details of each objective.
3.1 SELF-TRAINING WITH AN EMA TEACHER
The self-training objective is applied to the classifier's output. Given batch of B unlabeled images, compute pseudo-label for each image by passing weakly-augmentedd version of the image to teacher model. The teacher is parameterized by an exponentially moving average (EMA) (Tarvainen) Valpola 2017 Grill et 2020 of the model parameters 0. Specifically, the parameters of the EMA teacher A are intialized as O. During each update of 0, A are updated with
A م (1 4)
Let Qb denote the EMA teacher's softmax prediction for the cross-entropy loss against the model's prediction ps for goongy-aaugmnted version of the same image:
image, we enforce
B
Lota
Σ 6m
I (max 9b 2 T) Pb)
B
Following FixMatch (Sohn et al. 2020). we only use pseudo-labels with maximum scores above a threshold and convert the soft labels qu into "one-hot" hard labels by 4 arg max(qu).
Since the pseudo-labels generated by a CLIP model are often biased towards certain classes (Wang et al. 2022), minimizing Lats alone would magnify the bias. To mitigate confirmation bias, introduce a "fairness" regularization which encourages that on average, across batch of samples, the model's prediction probability is close uniform distribution:
Creg
Σ log(p)).
where K is the total number of classes, and 5 is the model's average prediction across the batch. In cases where K B, we compute ğ using moving average instead of batch average. We find this regularization to be beneficial even for datasets with long-tailed class distribution (see Table
3.2 MASKED IMAGE MODELING
To alleviate the over-reliance on the noisy pseudo-labels self-training, we introduce another source of supervision obtained from the raw images. The masked image modeling (MIM) objective aims to learn local image representation at masked positions by predicting the missing information using contextual patches. We follow SimMIM (Xie et al. 2022) and simply predict the RGB pixel values
3

---

Published as a conference paper at ICLR 2023
CLS Embed
Projection
Classifier
Encoder (VIT)
Self Training
EMA Model
EMA parameter
update
CLS Embed
CLS
Projection
Classifier
Global-local Alignment
MSK
Projection
Encoder (VIT)
MSK
MSK
MSK Embed
Decoder
Projection
MIM
MSK
MSK
Embed
Decoder
Model
Unsupervised learning framework of MUST. We randomly replace image patches with [MSK] tokens, and jointly optimize three objectives unlabeled images. (I) Global self-training: the classifier the output of (CLS] predict pseudo-label produced by an EMA model. (2) Masked image modeling: linear decoder uses the outputs of [MSK] to predict the pixel values of the masked patches. (3) Global-local feature alignment: minimize the distance between [CLS] and [MSK] in normalized embodding space.
Figure 2:
for masked patches. Specifically, given the output embedding zj"' of the m-th [MSK] token, we first pass it through linear decoder head obtain the predicted RGB values yh E RR for that patch, where N denotes the number of RGB pixels per patch. Then we compute the MIM loss as an fllsss between y,"" and the ground-truth RGB values 2%
M
Liii
Σ
10
11,
(4)
BMN
where M denotes the number of masked patches per image. We employ the patch-aligned random patches are randomly masked with a fix
masking strategy (Xie et al. 2022) where multiple masking ratio for each dataset. In general, we find that low masking ratio (e.g., 10%) works well for most datasets, which is consistent to the observations in Xie et al. 2022).
3.3 GL.OBAL-LLOCAL FEATURE ALIGNMENT
We aim to bridge the two sources of supervision (i.e., pseudo-labels and image pixels) such that the local [MSK] features learned from MIM can improve the global [CLS) feature for better classification performance. Let =i denote the output embedding of the [CLS] token, and vf = h(2g) denote its normalized embedding after the projection network h. Following CLIP, h is linear layer followed by l normalization, and of subsequently used by the classifier to produce the prediction ps for self-training (see Section 3.1 We also project the embeddings of the [MSK) tokens to the same space by passing them through h: 2i = h22""). The global-local feature alignment loss is defined as the average squared distance between the normalize embeddings of the [CLS] J token and all [MSK] tokens for each image:
M
Calign
1
Σ
Σ
11u6
(5)
BM
b=
1=
During training. MUST jointly optimizes the above three objectives. The overall loss is
L Leks Arog frog Luii + Calign
4

---

Published as a conference paper at ICLR 2023
Dataset
Train size
Test size
Metric
Description
ImageNet Deng et
2009
1,281,167
50,000
acc
1000 categories of objects from WordNet
SUN397
Xiao ct 20 2010
76,129
21,758
acc
397 categories of indoor and outdoor seenes
Food101
Bossard ef 2014
75,750
25,250
acc
101 categories of food dishes
GTSRB Stallkamp ct al. 2011
26,640
12,630
ucc
43 categories of German traffic signs
DID
(Cimpoi et al. 22144
3,760
1,880
ucc
47 cutegories of describable textures
UCFI0T (Soomrr el al. 2012
9,537
3,783
acc
101 categories of human action video frames
Oxford Pets [Parkhi et |
2012
3,680
3,669
mean per class
37 categories of cats and dogs
Caltech101 (Bmh (811
3,060
6,085
mean per class 101 categories of objects and hackground
Table I: Information on the image classification datasets used for transfer learning.
3.4 IMPLEMENTATION DETAILS
We experiment with two ViT models pre-trained by Radford et al. (2021): ): ViT-B/16 and VIT-L/14, containing respectively and 24 Transformer blocks with 768 and T024 hidden dimensions. The [MSK] token and linear decoder head are randomly initialized and finetuned together with the entire model. During finetuning, we use Adam (Loshchilov & Hutter (017)) optimizer with a weight decay of 0.05, We employ cosine learning rate schedule without any warmup. Following Bao et al. (2022): He et al. (202T). we use a layer- wise learning rate decay (Clark et al. 2020) of 0.65 for both VIT models. The batch size is 1024 for V and 512 for ViT-L/14, and the learning rate is scaled linearly with the batch size (Ir = base_hr x batchsize /256). We use 16 A A00 GPUs, and the training process of MUST adds only small amount of computation overhead after CLIP pre-training. We observe negligible variance on the results between runs with different random seeds.
The model receives input images of size 224 224. During training, we use RandomResized- Crop+Flip=RRnddAup (Cubuk et al. (2200) to augment input images, while using Resize+ Ratzum--Rnd as the weak augmentation to generate pseudo labels. During test, we simply take a center crop after resizing the shorter edge of the image to 224. For the EMA teacher, we follow Baevski et al. (2022) and linearly ramp-up the parameter decay rate μ from Po to 0.9998 in p., iterations. proovide more details of the hyperparameters used for each downstream task, including the pseudo-label threshold, mask patch size, mask ratio, etc.
followBaevskki
4 EXPERIMENTS
4.1 TRANSFER LEARNING DATASETS
We perform experiments on image classification datasets which span many different domains
including common objects (Deng et al. 2009
Binh 201T). fine-grained animals (Parkhi et al. 2012).
indoor and outdoor scenes (Xiao et al.,
2010) foods (Bossard et al. 2014), traffic signs (Stalkkapp
et al. 2011), natural textures (Cimpoi et al. 2014). and human actions (Soomro et al. 2012). Table
shows the detailed information of each dataset.
4.2 MUST RESULTS
Table shows the unsupervised classification results on 6 tasks. MUST substantially improves upon CLIP on all tasks with average accuracy improvement of +11.0% +8.5% for ViT-B / ViT-L. We also experiment with a widely-adopted supervised learning approach: supervised pre-training on ImageNet (Ik or 21k) followed by supervised finetuning on the downstream tasks. We finetune the ImageNet model with the same number of epochs and the same data augmentation (Cubuk et al. 2020) as MUST. Compared to the label-intensive supervised learning, unsupervised MUST achieves comparable performance
some datasets (e.g., Food101 and UCF101). MUST still lags behind supervised learning on some datasets (e.g., GTSRB and DTD). We hypothesis that CLIP's low accuracy is the major cause. Since CLIP has not seen enough data from those niche domains during pre- trainingg its pseudo-label labels would contain a limited amount of useful information. Despite this, MUST can narrow the performance gap between unsupervised and supervised learning by around 50%. Therefore, MUST a low-cost solution for image classification that opens up vast opportunities for practical scenarios where images abundant but labels scarce.
5

---

Published as a conference paper at ICLR 2023
ImageNet
SUN397
Food101
GTSRB
DTD
UCFI01
Average
CLIP
68.3
64,4
88.7
43.4
44.7
68.8
63.1
ViT-B/16
MUST
77.7 (+9.4)
73.1 (+8.7)
92.7 (+4.0)
65.5 (+22.1)
54.1 (+9.4)
81.1 (+12.3)
74.1 (+11.0)
Supervised
81.8
77.2
90.3
97.1
78.6
82.3
84.5
CLIP
75.5
67.4
92.9
50.6
55.4
77.0
69.8
VIT-L/14
MUST
82.1 (+6.6)
75.6 (+8.0)
95.3 (+2.4)
68.7 (+18.1)
62.6 (+7.2)
85.7 (+8.7)
78.3 (+8.5)
VIT-L/16
Supervised
83.4
81.1
93.1
97.3
79.7
86.4
186.8
Table 2: Image classification results a variety of downstream tasks. MUST significantly improves upon CLIP for unsupervised classification all datasets. Results for CL.IP ohtained using the publicly released models. Supervised denotes the label-intensive common practice of supervised pre-training followed by supervised finetuning on downstream tasks. We use the ImageNet- I model from Touvron et al. (202T) for supervised ViT-B and the ImageNet-21k 2k model from Dosovitskiy et al. (2021) for supervised VIT-L.
CLIP none
CoCoOp
CoOp
MUST unlabeled images
Finetuning data
16 labeled images per class
ImageNet Acc.
68.3
71.0
71.5
77.7
Table 3: Comparison with few-shot CLIP adaptation methods ImageNet. All methods use the same CLIP- pretrained ViT-B/16 model image encoder.
4.3
COMPARISON WITH FEW-SHOT ADAPTATION
In Table 3 we compare MUST with two state-of-the-a few-shot adaptation methods (Zhou et al.
2021 2022) that adapt pre-trained CLIP model on labeled ImageNet images. MUST outperforms 16-shot adaptation by a large margin (+6.2%), demonstrating its advantage as a low-cost method to improve classification performance using unlabeled data.
4.4
UNSUPERVISED TRANSFER ACROSS DOMAINS
Since the [MSK] token and linear decoder are trained from scratch. the potential of MUST may not be fully exploited for downstream tasks with limited number of unlabeled images. To address this, we pro- pose to warm-up the model by training on similar domain with more unla- beled images. Specifically, we first em- ploy MUST finetune CLIP ViT-B model ImageNet for single epoch, and then continue finetuning the model on two different datasets with limited number of samples (Pets and Caltech101). During warmup, freeze the linear classifier (i.e., CLIP's text embeddings) anchor the normal- ized image emheddings in their original space for easier transfer. As shown in Table 4 warmup on ImageNet further improves the performance of MUST on both datasets.
Pets
Caltech101
CLIP
88.9
89.8
MUST w/o ImageNet warmup
93.1 (+4.2)
93.1 (+3.3)
MUST w/ ImageNet warmup
94.3 (45.4) 93.7 (+3.9)
Table 4:
Warmup ImageNet improves the performance of MUST on other domains.
4.5 ABLATION STUDY
Effect of the proposed objectives. We study the effect of the three objectives: self self-training, mask image modeling, and global-local feature alignment. The results are shown in Table Removing the alignment loss reduces the average accuracy by 1.5%, and removing MIM further decreases accuracy by 0.7%. The results demonstrate the importance to align the features of [CLS] and [MSK] tokens, which regularizes the model to learn better global features for classification.
Number of [MSK] tokens to align with. Our global-local feature alignment loss aims to align the [CLS] token to all of the [MSK] tokens for image (see equation 5). We relax the alignment strength by only using the 10 [MSK] tokens that are nearest to the [CLS] token in the embedding space. As shown in Table 6 using the nearest-10 10 [MSK] tokens leads to similar performance compared to using all [MSK] tokens, which suggests that MUST is robust to the strength of alignment.

---

Published as a conference paper at ICLR 2023
MUST objectives
ImageNet
SUN397
Food101
GTSRB
DTD
UCFI01
Average
ST+MIM+Align
77.7
73.1
92.7
65.5
54.1
81.1
74.1
ST+MIM
77.1
72.0
92.4
62.5
52.6
79.2
(2.6
ST
76.5
71.4
92.4
59.8
52.4
79.0
71.9 (-2.2)
Table 5: Effect of the proposed training objectives on unsupervised image classification.
Num. [MSK] for Lalign
ImageNet
SUN397
Food101
GTSRB
DTD
UCF101
Average
All
77.7
73.1
92.7
65.5
54.1
81.1
74.1
Nearest- 10
77.6
73.2
92.7
65.7
54.1
80.8
74.1
Table 6: Number of [MSK] tokens used to align with the [CLS] token.
ImageNet
SUN397
Food101
GTSRB
DTD
UCFI01
Average
ST w/ Lreg 76.5
71.4
92.4
59.8
52.4
79.0
71.9
ST w/o freg
74.0
71.0
91.8
50.0
46.8
73.0
67.8 (-4.1)
Table 7: Effect of the fairness regularization loss on self-training performance.
ImageNet
SUN397
GTSRB
UCF101
Figure 3: GradCAM visualization validation images from multiple datasets. We visualize the self-attention from [CLS] to patches last ViT layer. For each triplet, we compare three different models: CLIP (left), ST (middle), MUST (right). make two qualitative observations: (1) MUST generally relies on more patches to make predictions; (2) MUST can better leverage the true correlation between patches and class labels.
Effect of fairness regularization. We propose the fairness regularization loss Lrog counteract the bias in pseudo-labels. In Table 7 we examine its effect on the self-training objective. The regularization improves the performance of self-training on all datasets. It is the least helpful on SUN397, which has a long-tailed class distribution in its training set (Xiao et al. 2010).
4.6 QUALITATIVE ANALYSIS
MUST pays attention to more informative regions. In Figure 3 we visualize the GradCAM (Sel
varaju al. 2017) heatmap for the self-attention from [CLS] to all patches in the last ViT layer. compare three models CLIP, ST, MUST- and make the following qualitative observations: (1) MUST generally relies more patches make predictions. E.g., the fully body of the dog instead of
only the head row column 1; all four corners of the traffic sign in row 3 column I. (2) MUST can better leverage the true correlation between image patches and class labels, instead of the spurious correlations used by CLIP and ST. E.g., the guitar attended for the action "playing guitar" in row column 1, instead of the person or the background.
helpful MIM is not necessarily good at image recovery. In Figure we show examples of recovered images from the MIM output the validation set of different datasets. On some datasets
7

---

Published as a conference paper at ICLR 2023
ImageNet
SUN397
Food101
GTSRB
DTD
UCF101
Figure 4: Examples of mask image modeling on validation images from multiple datasets. For cach triplet, we show the original image (left), the masked image (middle), and the recovered image (right).
ImageNet-C Hendrycks Hendrrcss & Dietterich (019))
CLIP
ST
MUST
MAE+Supervised
He et al. 2021
mCE 4
70.7
55.2
53.4
51.7
Table 8: Robustness evaluation the ImageNet-Corruttio dataset.
ImageNetV2
ImageNet-R
ImageNet-A
CLIP
61.9
77.7
49.9
Inductive MUST on ImageNet Transductive MUST w/o ImageNet warmup
68.9 (+7.0)
68.7 (-9.0)
41.6 (-8.3)
64.7 (+2.8)
87.3 (+9.6)
56.9 (+7.0)
Transductive MUST w/ ImageNet warmup
66.9 (+5.0)
87.6 (49.9)
58.2 (+8.3)
Table 9: Evaluation of MUST under natural distribution shift from ImageNet.
with abundant samples (e.g.. ImageNet), the model can learn to recover masked patches with high quality. On other datasets with fewer number of samples (e.g., GTSRB), the recovered images have lower quality with many artifacts. Despite the low recovery quality, the MIM objective still provides useful supervision that improves the image classification performance.
4.7 ROBUSTNESS TO IMAGE CORRUPTIONS
In Table we study the robustness of MUST image corruptions. We evaluate the performance of & Dietterich) 2019) benchmark, which contains 15 corruption types with various severity. MUST more robust than CLIP and ST, as suggested by lower mean corruption error (mCE). The mCE of MUST is only slightly higher than model that first trained with self-supe MAE (He al. 202I) followed by supervised finetuning on ImageNet.
ImageNet-finetuned models on the
4.8 TRANSDUCTIVE ADAPTATION TO DISTRIBUTION SHIFT
We further investigate the robustness of MUST under natural distribution shifts. We experi- ment with three out-of-distribution variants of ImageNet: ImageNetV2 (Recht et al., 2019) with the same 1000 ImageNet classes, ImageNet-Rendiine Adversarial (Hendrycks et al. 2021b) where each contains a subset 200 classes (see Figure 5 for example images). First, we directly evaluate ViT-B model finetuned on ImageNet using MUST, which has 77.7% accuracy on ImageNet validation set. As shown in Table 9 MUST on ImageNet improves the classification accuracy ImageNetV2, but decreases the performance ImageNet-R and ImageNet-Ar A. similar observation is reported in CLIP (Radford et al. 202T), where supervised adaptation to ImageNet reduces the model's 's average robustness
(Hendrycks et al.) 2021a) and ImageNet-
datasets.
8

---

Published as a conference paper at ICLR 2023
ImageNet
ImageNet-R
ImageNetV2
ImageNet-A
Figure 5: Example images of pomegranate from ImageNet and other datasets of natural distribution shifts.
ImageNet B
SUN397
Food101
GTSRB
DTD
UCF101
Pets
Caltech101
L
B
L
B
L
B
L
B
B
base Ir
2e-5
4e-5
lc-4
2e-4
2e-5
4e-5
2e-5
4e-5
2e-5
4e-5
le-5
2e-5
lc-5
le-5
epoch
30
20
20
20
40
40
20
20
60
60
60
60
60
60
threshold
0.7
0.5
0.4
0.5
0.6
0.7
0.7
mask ratio
0.3
0.3
0.1
0.1
0.1
0.1
0.5
0.3
0.1
0.1
0.1
0.1
0.1
0.1
mask patch size
32
56
32
28
32
56
32
56
32
56
16
14
32
32
Antign
0.2
0.5
0.5
0.1
0.5
0.5
0.5
0.2
0.5
0.5
0.5
0.1
0.1
Xrag
0.2
1
1
EMA init. pio
0.999
0.99
0.999
0.99
0.99
0.99
0.99
0.99
EMA iters. Jan
2000
500
2000
500
100
500
100
100
Table 10: Hyperparameters for MUST on the downstream datasets. B: VIT-B/16; L: ViT-L/14.
To address this, we perform transductive transfer learning (Xian et al. 2017 Rohrbach et al. 2013) with MUST. Transductive learning assume access unlabeled test images from the new distribution. which MUST uses to jointly update model parameters and infer image labels. The results in Table P show that transductive MUST substantially improves the model's accuracy distribution shift. Furthermore, the warmup on ImageNet (Section 4.4) before transductive learning leads to more improvement.
LIMITATIONS
In this paper, we show MUST as a promising domain-specific label-free classification algorithm where different model is trained for each task. Although the adaptation process is efficient, the storage of additional model parameters could be practical limitation if the number of tasks is large. There exists simple way address this concern: gather unlabeled image from all the domains of interest, and perform MUST to learn a single model that can generalize to multiple domains. However, this may sacrifice the model's performance each individual domain.
Due to the use of CLIP pre-trained models, MUST inherits some of the social implications of CLIP that arose from using minimally-fin filteree data crawled on the open internet. Additional analysis on the model is necessary before deploying it for real-world applications.
CONCLUSION
The field of deep learning has been largely driven by simple and effective methods that can scale well. MUST is simple method that can effectively adapt pre-trained zero-shot classifier to downstream tasks in an unsupervised manner. The of MUST is simultancously leverage two different sources of supervision signals obtained from unlabeled data: task-specific supervision from pseudo- labels and task task-agnostic self supervision from raw images. We propose simple alignment objective that bridges the two sources supervision and enables the model to learn better features.
As future work, MUST be naturally extended to other domains such NLP, where mask language modeling could act as the agnostic objective to improve self-training. We hope that this perspective could inspire future methods in unsupervised learning and zero-shot classification.

---

Published as a conference paper at ICLR 2023
REFERENCES
Alexci Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vee: general framework for self-supervised learning speech, vision and language. arXiv preprint arXiv:2202.03555, 2022.
Hangbo Bao, Li Dong, and Furu Wei. Beit: BERT pre-training of image transformers. In ICLR, 2022.
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. In ICLR. 2020.
Nguyen Dang Binh. Online multiple tasks one-shot learning object categories and vision. David Taniar, Eric Pardede, Hlong-Quang Nguyen, J. Wenny Rahayu, and Ismail Khalil (eds.), MoMM, 2011.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative compo- nents with random forests. In David Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), ECCV, 2014.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: pre-training text encoders as discriminators rather than generators. In ICLR, 2020.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In CVPR Workshops, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In CVPR. pp. 248-255, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei A. Efros. Test-time training with masked autoencoders. In NeurIPS, 2022.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint 2021.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In NIPS, 2004.
Bastinn Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach to self-supervised learning. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), NeurlPS, 2020.
Junxian He, Jiatao Gu, Jiajun Shen, and Mare' Aurelio Ranzato. Revisiting self-training for neural sequence generation. In ICLR, 2020a.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020b.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. arXiv preprint arXiv:2111.06377 2021.
10

---

Published as a conference paper at ICLR 2023
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. ICLR, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: critical analysis of 2021a.
generalization. ICCV,
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021b.
R. Devon Hjelm, Alex Fedorov, Samuel Aawoic-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
Dat Huynh and Ehsan Elhamifar. Fine-grained generalized zero-shot learning via dense attribute- based attention. In CVPR, 2020.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- saiin-la representation learning with noisy text supervision. arXiv preprint arXiv:2102.05918. 2021.
Jacob Kahn, Ann Lee, and Awni Y. Hannun. Self-training for end-to-end speech recognition. In ICASSP, pp. 7084-7088, 2020.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICLR, 2017.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In /CML Workshops, 2013.
Junnan Li, Caiming Xiong, and Steven C.H. Hoi. Mopro: Webly supervised learning with momentum prototypes. ICLR, 2021.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-iman -magge pre-training paradigm. In ICLR, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), /CML, 2021.
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), ICML, 2019.
Marcus Rohrbach, Sandra Ebert, and Bernt Schiele. Transfer learning in a transductive setting. In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger (eds.), NeurlPS, 2013.
Attaullah Sahito, Eibe Frank, and Bernhard Pfahringer. Better self-training for image classification through self-supervision. In Guodong Long, Xinghuo Yu, and Sen Wang (eds.), 2022.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das. Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-basce localiza- tion. In ICCV, pp. 618-626, 2017.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi sami-supervised learning with consistency and confidence. In NewrIPS, 2020.

---

Published as a conference paper at ICLR 2023
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 2012.
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: multi-class classification competition. In IJCNN, 2011.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NIPS, pp. 1195-1204 2017.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. ICML, 2021.
Wei Wang, Vincent W. Zheng, Han Yu, and Chunyan Miao. A survey of zero-shot learning: Settings, methods, and applications. TIST, 2019.
Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In CVPR, 2018.
Xudong Wang, Zhirong Wu, Long Lian, and Stella X. Yu. Debiased learning from naturally imbalanced pseudo-labels for zero-shot and semi-supervised learning. In CVPR, 2022.
Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan I.. Yuille, and Fan Yang. Crest: A class-rehalancing self-training framework for imbalanced semi-supervised learning. In CVPR, 2021.
Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning the good, the bad and the ugly. In CVPR, 2017.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. CVPR, 2010.
Qizhe Xie, Minh-Thang Luong, Eduard H. Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. In CVPR, 2020.
Zhenda Xie, Zheng Zhang, Yue Cao. Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. SimMIM: A simple framework for masked image modeling. In CVPR, 2022.
Lewei Yao, Runhui Huang. Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang. Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: fine-grained interactive language-image pre-training. In ICLR, 2022.
Xiaohua Zhai, Xiao Wang. Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In CVPR, 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy. and Ziwei Liu. Learning to prompt for vision- language models. arXiv preprint arXiv:2109.01133 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022.
Barret Zoph, Golnaz Ghiasi, Tsung- Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. Rethinking pre-training and self-training. In NeurlPS, 2020.
12


Reviews:
[{"review_id": "fVOdxiry13Z", "reviewer": "Reviewer_RF4U", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors combine three different self-supervised methods into label-free image classification: Infomax, simsiam(BYOL) and masked encoder. In experiments, the results demonstrate significant improvement in different benchmarks.", "summary_of_the_review": "Because of novelty, I tend to give rejection to this paper. However I am happy to refer to other reviewers as I am only an expert in self-supervised learning and not CLIP related methods.", "strength_and_weaknesses": "1. Strength:\nThe authors propose a very effective method for label-free image classification. The performance is impressive in different benchmarks.\n\n2. weakness:\nHowever, I think novelty is debatable. Frankly, it looks like three different self-supervised methods are being used. EMA + L2 loss is BYOL [1], local-global contrast is deep informax [3] and masked autoencoder [2]. And obviously it losts the [3] in the reference. \n\n\n[1]. Bootstrap your own latent: A new approach to self-supervised Learning\n[2]. Masked Autoencoders Are Scalable Vision Learners.\n[3]. Learning deep representations by mutual information estimation and maximization.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "epGhMOhcQx", "reviewer": "Reviewer_SBFM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Paper proposes a method, MUST, for unsupervised adaptation of a zero shot classifier. A pre-trained open world model CLIP is taken to produce classification embeddings for a vocabulary of words. These are then use to kick-start a training method that adapts a zero-shot classification model by a combination of three objectives. The first one uses a self-training, second uses a masked image modeling objective and the third one enforces global local feature alignment.The proposed method is then evaluated on a variety of datasets.", "summary_of_the_review": "The paper proposes a simple methods (though limited in technical novelty), that leads to significant improvements in zero shot results. I vote accept.", "strength_and_weaknesses": "Strengths\n- Clear and easy to understand writing\n- Simple method.\n- Significant performance improvements over CLIP based baseline approach.\n- Meaningful and informative ablation experiments that study various aspects of the proposed approach.\n\nWeaknesses\n- Limited technical novelty, with the first two terms of the proposed loss already existing and the alignment term being the primary technical contribution.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ng80rj-RCl", "reviewer": "Reviewer_zxPa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes Masked Unsupervised Self-Training (MUST), that uses pseudo labels from CLIP as well as patch masking to train on unlabeled data. The loss consists of three terms a) self training classification loss, b) a masked patch loss, and c) a global-local loss designed to force class level information into the patch embeddings. \n\n\n", "summary_of_the_review": "The paper clearly proposes a method of performing self supervised training using CLIP and a pool of unlabeled data by constructing a loss from classification losses and masked patch losses. A wide variety of experiments are performed over a set of datasets that show the contributions of the various components of MUST. ", "strength_and_weaknesses": "The paper's proposal is mostly clear well-motivated. \n\nExperiments are well thought out and clear and show a clear advantage over direct usage of CLIP across a variety of downstream\ntasks. Many ablations probe the inner working of the loss and analyze what each factor is doing. \n\nA weakness could be the global-local loss, which I felt was the least grounded in any kind of motivation as it is the \"average squared distance between the normalize embeddings of the [CLS] token and all [MSK] tokens for each image\". However, this seems like this decision could be a bias towards datasets that contain a single large and well-framed object in the field of view. \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "L4aLKbHCsu", "reviewer": "Reviewer_rbuG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method of adapting a zero-shot model to a dataset such as ImageNet without using any labels from ImageNet. This process does improve model accuracy on ImageNet by a large margin, which is perhaps not too surprising as the target dataset is used for this adaptation. Their adaptation method consists mainly of masked image modeling and self-training from an EMA teacher.", "summary_of_the_review": "Overall I enjoyed this paper, which is thorough and observes good empirical performance. My main concern is in the experimental set-up which takes a clean, balanced data-set then removes the labels.", "strength_and_weaknesses": "Strengths:\n- This paper appears to be a very detailed, thorough, and reproducible empirical investigation and the results are potentially very useful to the community.\n- Extensive ablations are performed on the design decisions. \n\nWeaknesses:\n- I found the main weakness to be the experimental set-up - I believe this would be much more convincing if the paper considered a datasets which was not clean/balances such as LAION or CC12m for running their procedure.\n- Some comparisons seem to be a bit unfair, for instance Table 3 in which MUST (seeing all images) is compared to methods seeing 16 images per class, what if MUST only sees 16 images per class.\n\nQuestions:\n- What happens if you replace the self-training part with the real labels? Are the other losses still required?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CZe8LzGYvfq", "reviewer": "Reviewer_Xc1a", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces MUST (MASKED UNSUPERVISED SELF-TRAINING) to improve the performance of CLIP-like models in zero-shot recognition. Based on a pre-trained CLIP, three objectives are learned together to fine-tune it on a specific dataset in an unsupervised manner. Three proposed objectives are class-level global feature and pixel-level local feature and enforce a regularization between the previous two objectives. A fairness regularization loss is introduced in the class-level global feature objective. \nExperiments are conducted on 8 common datasets and the results clearly demonstrate the superiority of the proposed methods, which surpass baseline CLIP by large margins. The ablation study is solid, and the individual contribution of each component is demonstrated. ", "summary_of_the_review": "The authors addressed all of my concerns during the discussion. I'm leaning toward acceptance.\n", "strength_and_weaknesses": "Strength:\n\n1: Clear motivation. This paper is well-motivated.\n\n2: Good writing and presentation, easy to follow/read. \n\n3: Very solid experiments and ablation study. Multiple datasets are evaluated. Robustness and calibration metrics are reported.\n\n4: Good performance. The proposed method surpasses CLIP by large margins. \n\nMain weaknesses:\n\n1: The best feature of CLIP is the generality, that is, being able to recognize any image without pre-defined/fixed classes. MUST adapt the CLIP model to a specific dataset (which is the main purpose of this paper). A simple solution is claimed by authors in the limitation section: \"There exists a simple way to address this concern: gather unlabeled image from all the domains of interest, and perform MUST to learn a single model that can generalize to multiple domains.\"  It would be really great to have 1-2 such experiments to verify this hypothesis. \n\nMinor weaknesses: \n\n2:  The paper does not reach out to theoretical backup to explain why MUST works. \n\n3: Adding results on ImageNet-Sketch will further strengthen this paper. \n\n4: A related work [1] is worth discussing. \n\n[1]: Test-time training with masked autoencoders, NeurIPS 2022 \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2t1woTwjeg", "reviewer": "Reviewer_8jDS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a new method of using machine learning to classify images without the use of labels. Training consists of 3 parts: \n- The self training part where the teacher model generates pseudo-labels to train the student model to learn global tasks.\n- The masked image modelling part where they train a model to predict missing pixels of an image in order to learn how to represent local pixels information.\n- The part where they connect the knowledge learned from the two models in order to do the classification.\nThey use this method to investigate how well it performs on 8 image classification tasks and compares its efficacy to other models like CLIP that are known to perform really well in this field. MUST outperforms CLIP based models and seems to be using the relevant/defining parts of the image that correlate with the label to make the classification.", "summary_of_the_review": "Overall, I think the paper provides a significant contribution to exploring methods of overcoming the current issues we have with the cost of labelling datasets. The methodology is promising and I would look forward to future work extending it to apply to more diverse real world datasets.", "strength_and_weaknesses": "Strengths\n- The paper is well written and concepts are explained well.\n- Given the high costs of labelling datasets, the authors explore alternative methodologies that try to overcome this barrier.\n\nWeaknesses\n- Given the authors' awareness of the negative social implications of CLIP pretrained models, what steps will/did you take to minimize these effects given that you will possibly share your trained models?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster