MoPro: Webly Supervised Learning with Momentum Prototypes
Authors: Junnan Li
Caiming Xiong
Steven Hoi
Venue: ICLR 2021 Poster (2021)
Keywords: webly-supervised learning
weakly-supervised learning
contrastive learning
representation learning

Abstract: We propose a webly-supervised representation learning method that does not suffer from the annotation unscalability of supervised learning, nor the computation unscalability of self-supervised learning. Most existing works on webly-supervised representation learning adopt a vanilla supervised learning method without accounting for the prevalent noise in the training data, whereas most prior methods in learning with label noise are less effective for real-world large-scale noisy data. We propose momentum prototypes (MoPro), a simple contrastive learning method that achieves online label noise correction, out-of-distribution sample removal, and representation learning. MoPro achieves state-of-the-art performance on WebVision, a weakly-labeled noisy dataset. MoPro also shows superior performance when the pretrained model is transferred to down-stream image classification and detection tasks. It outperforms the ImageNet supervised pretrained model by +10.5 on 1-shot classification on VOC, and outperforms the best self-supervised pretrained model by +17.3 when finetuned on 1% of ImageNet labeled samples. Furthermore, MoPro is more robust to distribution shifts. Code and pretrained models are available at https://github.com/salesforce/MoPro.

Published as a conference paper at ICLR 2021
MOPRO: WEBLY SUPERVISED LEARNING WITH MOMENTUM PROTOTYPES
Junnan Li, Caiming Xiong, Steven C.H. Hoi
Salesforce Research (junnan. 1i, cxiong, shoi es ls force.ccom
ABSTRACT
We propose a webly-supervised representation learning method that does not suf- fer from the annotation unscalability of supervised learning, nor the computa- tion unscalability of self-supervised learning. Most existing works on webly- supervised representation learning adopt a vanilla supervised learning method without accounting for the prevalent noise in the training data, whereas most prior methods in learning with label noise are less effective for real-world large-scale noisy data. We propose momentum prototypes (MoPro), a simple contrastive learning method that achieves online label noise correction, out-of-distribution sample removal, and representation learning. MoPro achieves state-of-the-art per- formance WebVision, a weakly-labeled noisy dataset. MoPro also shows perior performance when the pretrained model is transferred to down-stream im- age classification and detection tasks. It outperforms the ImageNet supervised pretrained model by +10.5 on 1-shot classification on VOC, and outperforms the best self-supervised pretrained model by +17.3 when finetuned on 1% of ImageNet labeled samples. Furthermore, MoPro is more robust to distribution shifts. Code and pretrained models are available at https://githann com/ salesforce/ MMoooo
INTRODUCTION
I Large-scale datasets with human-annotated labels have revolutionized computer vision. Supervised pretraining on ImageNet (Deng et al., 2009) has been the de facto formula of success for almost all state-of-the-art visual perception models. However, it is extremely labor intensive to manually annotate millions of images, which makes it non-scalable solution. One alternative to reduce an- notation cost is self-supervised representation learning, which leverages unlabeled data. However, self-supervised learning methods (Goyal et al.) 2019. He et al. 2019 Chen et al. 2020a Li et al.) 2020b) have yet consistently shown superior performance compared to supervised learning, espe- cially when transferred to downstream tasks with limited labels.
With the help of commercial search engines, photo-sharing websites, and social media platforms, there is near-infinite amount of weakly-labeled images available on the web. Several works have exploited the scalable source of web images and demonstrated promising results with webly- supervised representation learning (Mahajan et al. 20188) Sun et al. 20177 Li al. [017 Kolesnikov al. 2020). However, there exists two competing claims whether weakly-labeled noisy datasets lead to worse generalization performance. One claim argues that the effect of noise can be over- powered by the scale of data, and simply applies standard supervised learning method on web datasets (Mahajan et al.) 2018 Sun et al. [017]] Li et al., 2017: Kolesnikov et al. 2020). The other claim argues that deep models can easily memorize noisy labels, resulting in worse general- ization (Zhang et al. 2017; Ma et al. 2018). In this paper, show that both claims are partially true. While increasing the size of data does improve the model's robustness to noise, our method substantially boost the representation learning performance by addressing noise.
There exists a large body of literature on learning with label noise (Jiang et al. 2018 Han et al. 2018 Guo et al. 2018 Tanaka al. 2018 Arazo al.) 2019 Li et al. (222aa). However, existing methods have several limitations that make them less effective for suberssuer representation learning. First, most methods do not consider out-of-distribution (OOD) samples, which is major

---

Published as a conference paper at ICLR 2021
lemon orange
orange lemon
Momentum prototype "lemon"
orange
Momentum prototype for "orange"
Noise correction
lemon
Corrupted labels
orange
lemon
orange OOD
lemon OOD
Figure 1: Illustration of the normalized embedding space learned with MoPro. Samples from the same class gather around their class prototype, whereas OOD samples are separated from in-distribution samples. Label correction and OOD removal are achieved based on sample's distance with the prototypes.
source of noise in real-world web datasets. Second, many methods perform computation-heavy procedures for noise cleaning (Jiang et al. 2018 Li et al., 2019 2020a), or require access to a set of samples with clean labels (Vahdat) 2017 Veit et 2017 Lee al. 2018), which limit their scalability in practice.
We propose a new method for efficient representation learning from weakly-labeled web images. Our method is inspired by recent developments in contrastive learning for self-supervised learn- ing (He et al., 2019 Chen et al. (2020a] et al.) 2020b) We introduce Momentum Prototypes (Mo- Pro). simple component which is effective in label noise correction, OOD sample removal, and representation learning. visual explanation our method shown Figure We use a deep network to project images into normalized low-dimensional embeddings, and calculate the prototype for a class as the moving-averas embedding for clean samples in that class. We train the network such that embeddings other prototypes. Images with corrupted labels are corrected either as another class or as an OOD) sample based their distance to the momentum prototypes.
pulled closer to their corresponding prototypes, while pushed away from
We experimentally show that:
MoPro achieves state-of-t ofte-offtheeart performance the upstream
learning task.
MoPro substantially improves representation learning performance when the pretrained model is transferred to downstream image classification and object detection tasks. For the first time, we show that weakly-supervised representation learning achieves similar performance as supervised representation learning, under the same data and computation budget. With larger web dataset, MoPro outperforms ImageNet supervised learning by large margin.
MoPro learns more robust and calibrated model that generalizes better to distribution variations.
RELATED WORK
2.1
WEBLY-SUPERVISED REPRESENTATION LEARNING
number prior works exploit large web datasets for visual representation learning (Divvala al. 2he4| & Gupta 2015 Joulin et al.) 2016 Mahajan et al. 2018 Sun et al. 2017 Li et al. 2017
Kolesnikov et al. 2020). These datasets contain a considerable amount of noise. Approximately
20% of the labels in the JMT-300M dataset (Sun al. 2017) are noisy, whereas 34% of images in the WebVision dataset (Li et al. 2017) are considered outliers. Surprisingly, most prior works have chosen to ignore the noise and applied vanilla supervised method, with the claim that the scale of
data can overpower the noise (Mahajan et al.) 2018) Sun et al.) [017)] Li et al., 2017). However, we show that supervised method cannot fully harvest the power of large-scale weakly-labeled datasets.
2

---

Published as a conference paper at ICLR 2021
Our method achieves substantial improvement by addressing noise, and advances the potential of webly-supervised representation learning.
2.2 LEARNING WITH LABEL NOISE
Learning with label noise has been widely studied. Some methods require access to a small set of clean samples (Xiao et al., 2015 Vahdat 2017: Veit et al. 2017, Lee al., 2018 Zhang et al.) 2020), and other methods assume that no clean labels are available. There exist two major types of approaches. The first type performs label correction using predictions from the network (Reed et al.) 2015 Ma et al. 2018 Tanaka al. 2018 Yi & Wu 2019 Yang al.
2020). The second type separates clean samples from corrupted samples, and trains the model on clean samples (Han al, 2018 Arazo et al. 2019 Jiang et al. 2018 Wang al., 2018: Chen al. 2019 Li et al. 2020a) However, existing methods have yet shown promising results for large-scale weakly suseevvissd rep- resentation learning. The main reasons include: (1) most methods do consider OOD samples, due 2020a Jiang et al., 20188 2020), iterative training (Tanaka
which commonly occur in weal-world web datasets; (2) most methods are computational-henn
co-training (Han et al. 2018 Li et al.
ct al. 2018 Yi & Wu 2019 et al.
2018 Chen et al. 2019) or meta-learning (Li et al. 2019
Zhang al. 2019).
Different from existing methods, MoPro achieves both label correction and OOD sample removal on-the-fly fly with single step, based on the similarity between an image embedding and the momen- tum prototypes. MoPro also leverages contrastive learning to learn robust embedding space.
2.3 SELF-SUPERVISED REPRESENTATION LEARNING
Self-supervised methods have been proposed for representation learning using unlabeled data. The recent developments in self-supervised representation learning can be attributed to contrastive learn- ing. Most methods (He al., 2019 Chen et al.) 2020a Oord et al.) 2018 Wu al. 2018 leverage
the task of instance discrimination. where augmented crops from the same source image are enforced to have similar embeddings. Prototypical contrastive learning (PCL) (Li et al. 2020b) performs clustering to find prototypical embeddings, and enforces an image embedding to be similar to its assigned prototypes. Different from PCL, we update prototypes on-the-fly fly setting, where the momentum prototype of class is the moving average of clean samples' embed- dings. Furthermore, we jointly optimize two contrastive losses and a cross-entropy loss.
weakly-supervisee
Current self-supervised representation learning methods are limited in (1) inferior performance in low-shot task adaptation, (2) huge computation cost, and (3) inadequate to harvest larger datasets. We show that weakly-supervised learning with MoPro addresses these limitations.
METHOD
In this section, we delincate the details of our method. First, we introduce the components in our representation learning framework. Then, describe the loss functions. Finally, we explain the noise correction procedure for label correction and OOD sample removal. A pseudo-code of MoPro provided in appendix
3.1 REPRESENTATION LEARNING FRAMEWORK
Our proposed framework consists of the following components. Figure giives an illustration.
A noisy training dataset ((v,, w where æ; is image and yı {1, } is its class label.
A pseudo-label ii for each image 2;, which is its corrected label. Details for generating the pseudo-label is explained in Sec
An encoder network, which maps augmented image ã; to representation vector e; Rdd
We experiment with ResNet-50 (He et al. 2016) as the encoder, where the activations of the final global pooling layer (dg 2048) are used as the representation vector.
A classifier (a cully-coneected layer followed by softmax) which receives the representation v, as input and outputs class predictions p..
3

---

Published as a conference paper at ICLR 2021
softmax
cross-entropy
function
"Chihuahua"
positives
negatives
backbone
low-dim
L2 norm
contrastive
"Chihuahua"
"orange"
128-D
contrastive
"baseball"
2048-D
momentum prototypes
embeddings
Figure 2: Proposed weakly-superrvise learning framework. We jointly optimize a prototypical contrastive loss using momentum prototypes, instance contrastive loss using momentum embeddings, and cross coos-eytroppy loss using pscudo-labels. The pseudo-label for sample generated based on original training label, model's prediction, the sample's 's distance to prototypes.
A projection network, which maps the representation v, into low-dimensional embedding 2; Rd (dp 128). 2; is always normalized to the unit sphere. Following SimCLR (Chen et al, 2020a), we use MLP with one hidden layer as the projection network.
Momentum embeddings z generated by a momentum encoder. The momentum encoder has the same architecture the encoder followed by the projection network, and its parameters the moving- average the encoder's and the projection network's parameters. Same as MoCo (He) et al. 2019), maintain queue of momentum embeddings of past samples.
Momentum prototypes C R4,*K, The momentum prototype of the k-th class, Ck2 is the normalized moving-averes embedding for samples with pscudo-label ÿ = k.
3.2 CONTRASTIVE Loss
illustrated in Figure 1 we aim to learn an embedding space where samples from the same class gather around its class prototype, while samples from different classes are seperated. We achieve it with two contrastive losses: (I) a prototypical contrastive loss Cpoo which increases the similarity between embedding and its corresponding class prototype, (z Cg.), in contrast to other proto- types; (2) an instance contrastive loss Lins which increases the similarity between two embeddings of the same source image, (z,, z/), in contrast to embeddings of other images. Specifically, the contrastive losses are defined as:
exp(zi CO./T)
exp(z: 2(/T) Σ exp(z: #t/T)
Cpr - log Σk exp(z: cu/T)'
Cips log
(I)
where temperature parameter, and Ø the pseudo-label. We use negative momentum embeddings to construct the denominator of the instance contrastive loss.
We train the classifier with cross-entropy loss, using pseudo-labels as targets.
C = log(
We jointly optimize the contrastive losses and the classification loss. The training objective is:
C [ + Apro + Xins
(3)
1=1
For simplicity, we set Apro Ains for all experiments.
3.3 NOISE CORRECTION
We propose a simple yet effective method for online noise correction during training, which cleans label noise and removes OOD) samples. For each sample, generate soft pseudo-label qi by

---

Published as a conference paper at ICLR 2021
combining the classifier's output probability p; with 8,, class probability distribution calculated using the sample's 's similarity w.r the momentum prototypes:
op; + (1 0x)8:;
exp(zi z1 Cx/T)
(4)
E exp(zi Cx/T)
where the combination weight is simply set
0.5 in all experiments.
We convert qi into a hard pseudo-label y; based on the following rules: (1) if the highest score qi above certain threshold T, the class with the highest score the pseudo-label; (2) otherwise, the score for the original label y: is higher than uniform probability, use ys as the pseudo-label, (3) otherwise, label it as OOD sample.
arg max & of
if max; of > T, elseif o 1/K, otherwise.
0:
V4 OOD
(5)
We remove OOD samples from both the cross-entropy loss and the prototypical contrastive loss so that they do not affect class-specific learning, but include them in the instance contrastive loss further separate them from in-distribution samples. Examples of OOD images and corrected pseudo-labels are shown in the appendices.
3.4 MOMENTUM PROTOTYPES
For each class k, we calculate its momentum prototype as a moving-average of the normalized embeddings for samples with pscudo-label k. Specifically, update e₁ by:
C₁ f Normalize(mica + (1 m)z;), Vi E {i ú: k}
(6)
where Normalize(c) = c/ ||c||2- The momentum coefficient m is set 0.999 in our experiments.
4 EXPERIMENTS
4.1 DATASET FOR UPSTREAM TRAINING
We use the WebVision (Li et al. 2017 dataset as the noisy training data. It consists of images automatically crawled from Google and Flickr, using visual concepts from ImageNet as queries. We experiment with three versions of Web with different sizes: (1) WebVision- -1 1..0 contains 2.44m images with the same classes as the ImageNet-Ik kk (ILSVRC 2012) dataset; (2) WisVinion- V0.5 is a randomly sampled subset of Web Vision-V V1.0, which contains the same number of images (1.28m) as ImageNet- Ik; (3) WebVision-Vel V2.00 contains 16m images with 5k classes.
4.2 IMPLEMENTATION DETAILS
We follow standard settings for ImageNet training: batch size is 256; total number of epochs is 90; optimizer is SGD with momentum of 0.9; initial learning rate is 0.1, decayed at 40 and 80 epochs; weight decay is 0.0001. We use ResNet-50 (He et al. 2016) as the encoder. For MoPro- specific hyperparameters,
set 0.1, a 0.5, T 0.8 momentum for both the momentum encoder and momentum prototypes is set as 0.999, The queue store momentum embeddings has size of 8192. We apply standard data augmentation (crop and horizontal flip) to the encoder's input, and stronger data augmentation (color changes in MoCo (He et al. 2019)) to the momentum encoder's input. We warm-up the model for 10 epochs by training on all samples with original labels, before applying noise correction.
0.6 for WebVision-V2..0) The
4.3 UPSTREAM TASK PERFORMANCE
In Table I we compare MoPro with existing weakly-supervised learning methods trained on Visionn V1.0, where MoPro achieves state-of-the-art performance. Since the training dataset

---

Published as a conference paper at ICLR 2021
WebVision
ImageNet
Method
Architecture
top-1
top-5
top-1
top-5
Cross-Entropy (Tu al. 2020
ResNet-50
66.4
83.4
57.7
78.4
MentorNet Jiang 2018)
InceptionResNet-Y V2
70.8
88.0
62.5
83.0
CurrieulumNet (uuo al
2018
Inception- V2
72.1
89.1
64.8
84,9
CleanNet (Lee al. 2018
ResNet-50
70.3
87.8
63.4
84.6
CurriculumiNer (Guo of al.
2018
Tu et al.
2020
ResNet-50
70.7
88.6
62.7
83,4
SOM (Tu et al. 2020)
ResNet-50
72.2
89.5
65.0
85.1
Distill Zhang
2020)
ResNet-50
65.8
85.8
Cross- (decoupled)
ResNet-50
72.4
89.0
65.7
85.1
MoPro (ours)
ResNet-50
73.9
90.0
67.8
87.0
Table I: Comparison with state-of-the-arr methods on WebVision-VI Numbers denote accuracy (%) on the clean WebVision-Ve V1.0 validation set and the ILSVRC 2012 validation set. CleanNet (Lee et al. 2018 and Distill (Zhagg al. 2020 require data with clean annotations.
has imbalanced number of samples per-class, inspired by Kang et al. (2020), we perform the fol- lowing decoupled training steps re-balance the classifier: (I) pretrain the model with MoPro; (2) perform noise correction on the training data using the pretrained model, following the method in Section 3.3 (3) keep the pretrained encoder fixed and finetune the classifier on the cleaned dataset, using square-root data sampling (Mahajan et al. 2018) which balances the classes. We retrain the classifier for 15 epochs, using learning rate 0.01 which is decayed at 5 and 10 epochs. Surpris- ingly, we also find that vanilla cross-entropy method with decoupled classifier re-balancing can also achieve competitive performance, outperforming most existing baselines.
5 TRANSFER LEARNING
In this section, we transfer We show that MoPro yields superior performance in image classification, object detection, instance segmentation, and obtains better robustness to domain shifts. Implementation details for the transfer learning experiments described in appendix C
learned models to a variety of downstream tasks.
5.1 LOW-SHOT IMAGE CLASSIFICATION ON FIXED REPRESENTATION
First, we transfer the learned representation to downstream tasks with few training samples. We perform low-shot classification two datasets: PASCAL VOC2007 (Everingham et al. 2010) for object classification and Places205 (Zhou et al. 2014) for scene recognition. Following the setup by Goyal et al. (2019); Li et al. (2020b), train linear SVMs using fixed representations from pretrained models. We vary the number k of samples per-class and report the average result
VOCO7
Places205
Method
Pretrain dataset
k=1
k=2
k=4
k=8
k=16
k=
k=2
k=4
k=8
k=16
MoCo v2*
ImageNet ImageNet
46.3
58.4
64.9
72.5
76.1
11.9
17.0
22.6
28.1
32.4
PCL v2"
47.9
59.6
66.2
74.5
78.3
12.5
17.5
23.2
28.1
32.3
CE (Sup.)
ImageNet
54.3
67.8
73.9
79.6
82.3
14.9
21.0
26.9
32.1
36.0
CE
63.9
69.9
76.1
79.2
13.5
19.3
24.7
29.5
33.8
Wib V0.5
MoPro (ours)
54.3
67.8
73.5
79.2
81.8
15.0
21.2
26.6
31.8
36.0
CE
54.5
67.1
72.8
78.4
81.4
15.1
21.5
27.2
32.1
36.4
WebVision V1.0
MoPro (ours)
59.5
71.3
76.5
81.4
83.7
16.9
23.2
29.2
34.5
38.7
CE
73.8
78.7
83.0
85.4
21.8
28.6
35.1
40.0
43.6
WebVision V2.0
MoPro (ours)
64.8
74.8
79.9
83.9
86.1
22.2
29.2
35.6
40.9
44.4
Table 2: Low-shot image classification VOCO7 and Places205 using linear SVMs trained fixed repre- sentations. We vary the number labeled examples per-class (k)), and report average mAP (for VOC) and accuracy (for Places) across 5 independent runs. Visionni. has the same number of training samples as ImageNet. The self-supervised learning methods" are trained for 200 epochs, while other methods are trained for 90 epochs. MoPro outperforms vanilla CE pretrained Web datasets, well supf-supe leamiig and supervised learning methods pretrained on ImageNet.

---

Published as a conference paper at ICLR 2021
across 5 independent runs. Table shows the results. When pretrained weakly-labeled datasets, MoPro consistently outperforms the vanilla CE method. The improvement MoPro becomes less significant when the number of web images increases from 2.4m 16m, suggesting that increasing dataset size is a viable solution to combat noise.
When compared with ImageNet pretrained models, MoPro substantially outperforms self-supervised learning (MoCo v2 (Chen et al. 2020b) and PCL v2 (Li et al. 2020b)), and achieves comparable performance with supervised learning when the same amount of web images (i.e. Web V0.55 is used. Our results for the first time show that weakly-supervised representation learning can be as powerful as supervised representation learning under the same data and computation budget.
5.2 RES-UE TRANSFER WITH FINETUNING
Next, we perform experiment to evaluate whether the pretrained model provides a good basis for finetuning when the downstream task has limited training data. Following the setup by Chen et al. (020a),, we finetune the pretrained model 1% or 10% of ImageNet training samples. Table shows the results, MoPro consistently outperforms CE when pretrained on Web datasets. Compared to self-supervised learning methods pretrained ImageNet, weakly-supervised learning achieves significantly better performance with fewer number of epochs.
Surprisingly, pretraining the larger WebVision- V2 leads to worse performance compared to V0.5 and V1.0. This is because Web Vision V0.5 and V1.0 contain the same 1k class as ImageNet, whereas V2 also contains 4k extra classes. Hence, the representations learned from V2 less task-specific and more difficult to adapt to ImageNet, especially with only 1% of samples for fine- tuning. This suggests that if the classes for downstream task are known priori, is more effective to curate a task-specific weakly-labeled dataset with the same classes.
#Pretrain epochs
Top-1
Top-5
Pretrain Method
Pretrain dataset
1%
10%
1%
10%
Random init.
None
None
None
25.4
56.4
48.4
80.4
PCL
200
62.9
75.3
85.6
SimCLR
1000
48.3
65.6
75.5
87.8
Self-supervised
ImageNet
BYOL
1000
53.2
68.8
78.4
89.0
SwAV
800
53.9
70.2
78.5
89.9
CE
65.9
72.4
87.0
90.9
WebVision V0.5
90
MoPro (ours)
69.3
73.3
89.1
91.7
CE
67.6
73.5
91.7
Weakly-supervised
WebVision-V V1.0
90
MoPro (ours)
71.2
74.8
90.5
92.4
CE
62.1
72.9
86.9
91.4
WebVision- V2.0
MoPro (ours)
65.3
73.7
88.2
92.1
Table 3: Low-resource finetuning on ImageNet. A pretrained model is finetuned with 1% 10% of Ima- geNet training data. Weakly-superrised learning with MoPro substantially outperforms self-supervised learning methods: PCL (Li et al.] (2020b). SimCLR (Chen et al. 2020a), BYOL (Grill al. (020), and SWAV (Caron et al. 2020). Result for random init. is from Zhai et al.
2019)
5.3 OBJECT DETECTION AND INSTANCE SEGMENTATION
We further transfer the pretrained model to object detection and instance segmentation tasks on COCO (Lin et al. 2014). Following the setup by He et al. (2019), we use the pretrained ResNet-50 as the backbone for Mask-RCNN (He al. (017)) with FPN (in al.) (001)). We finetune all layers end-to-end, including BN. The schedule is the default 1 or 2x in Girshick et al. (2018) Table shows the results. Weakly-supervised learning with MoPro outperforms both supervised learning ImageNet and self-supervised learning on one billion Instagram images.
5.4 ROBUSTNESS
has been shown that deep models trained on ImageNet lack robustness to out-of- -iiitii-iitiiittioonn sam- ples, often falsely producing over-confident predictions. Hendricks et al. have curated two bench- mark datasets to evaluate models' robustness to real-world distribution variation: (1) ImageNet- (Hendrycks al. 2020) which contains various artistic renditions of object classes from the

---

Published as a conference paper at ICLR 2021
Method
Pretrain dataset
Aphb
AP
AP75
Apmk
AP50
AP75
random
None
31.0
49.5
33.2
28.5
46.8
30.4
(Sup.)
ImageNet
38.9
59.6
42.7
35.4
56.5
38.1
MoCo
Instagram- IB
38.9
59.4
42.3
35.4
56.5
37.9
CE
39.2
60.0
42.9
35.6
56,8
38.0
Web V1.0
MoPro
39.7
(+0.8)
60.9 (+1.3)
43.1 (+0.4)
36.1 (+0.7)
57.5 (+1.0)
38.6 (40.5)
MoPro
Web Visionn V2.0
40.7
(+1.8)
61.7 (+2.1)
44.5 (+1.8)
36.8 (+1.4)
58.4 (+19)
39.6
(a) 1x schedule
Method
Pretrain dataset
Aph
AP
AP
Ap
APm
AP75
random
None
36.7
56.7
40.0
33.7
53.8
35.9
(Sup.)
ImageNet
40.6
61.3
44.4
36.8
58.1
39.5
MoCo
Instagram- IB
41.1
61.8
45.1
37.4
59.1
40.2
CE
40.9
61.6
44.7
37.2
58.7
40.1
Weh VisionnVV
MoPro
41.2 (+0.6)
62.2 (+0.9)
45.0 (+0.6)
37.4 (+0.6)
58.9 (+0.8)
40.3 (40.8)
MoPro
WebVision-V2.0
41.8 (+1.2)
62.6 (+1.3)
45.6 (+1.2)
37.8 (+1.0)
59.5 (+1.4)
40.6 (+1.1)
(a) schedule
Table 4: Object detection instance segmentation using Mask-RCNN with R50-FPN fine-tuned on COCO train2017. We evaluate bounding-bow AP (AP")) and mask AP (AP")) on va12017. Weakly- supervised learning with MoPro outperforms both supervised learning on ImageNet and self self-supervised learn- ing (MoCo (He et al. 2019h) on billion Instagram images.
ImageNet-R
ImageNet-A
Method
Pretrain dataset
Accuracy (t)
Calib, Error (4)
Accuracy (t)
Calib. Error (4)
CE (Sup.)
ImageNet
36. 14
19.66
0.03
62.50
CE
49.56
10.05
10.24
37.84
WebVision-V V1.0
MoPro
54.87
5.73
11.93
35.85
Table 5: Evaluation of model robustness on images with artistic and natural distribution shifts. Weakly supervised learning with MoPro leads to more robust and well-calibrated model.
original ImageNet dataset, and (2) ImageNet-A (Hendrycks et al. 2019) which contains natural im- ages where ImageNet-pee -erett-t--ee models consistently fail due variations in background elements, color, or texture. Both datasets contain 200 classes, a subset of ImageNet's s 1,000 classes.
trained models these two robustness benchmarks. We report both accuracy and the & calibration error (Kumar et al.) 2019). The calibration error measures the misalignment between model's confidence and its accuracy. Concretely, well-calibrated classifier which give examples 80% confidence should be correct 80% of the time. Results are shown in Table 5 Webly-supervised learning show significantly higher accuracy and lower calibration error. The robustness to distribution shift could come from the higher diversity of samples in Web images. Compared to vanilla CE, MoPro further improves the model's robustness on both datasets. Note that we made sure that the training data of Web Vision does not overlap with the test data.
We evaluate
ABLATION STUDY
We perform ablation study verify the effectiveness of three important components in MoPro: (1) prototypical contrastive loss Lpro. (2) instance contrastive loss Cins (3) prototypical similarity 8, used for noise correction (equation 4). We choose low-resource finetuning on 1% ImageNet training data as the benchmark, and report the top-1 accuracy for models pretrained on Web Vision- v0.5. As shown in Table 6, all of the three components contribute to the efficacy of MoPro.
7 CONCLUSION
This paper introduces a new contrastive learning framework for webly-supervised representation learning. We propose momentum prototypes, simple component that is effective in label noise
8

---

Published as a conference paper at ICLR 2021
MoPro
w/o Ler
w/o Linst
w/o 8, (f.e. a
w/o Lero Linst &s.
CE
ImageNet acc.
69.3
68.0
68.2
68.4
66.9
65.9
Table 6: Ablation study where different components are removed from MoPro. Models are pre- wre-trained on Web Visionno and finetuned on 1% of ImageNet data.
conrection, OOD sample removal, and representation learning. MoPro achieves sate-of-the-an per- formance on the upstream task of learning from real-world noisy data, and superior representation learning performance multiple down-stream tasks. Webly-superrised learning with MoPro does not require the expensive annotation cost in supervised learning. nor the huge computation budget in self-supervised learning. For future work, MoPro could be extended to utilize other sources of free Web data, such as weakly-labeled videos, for representation learning in other domains.
REFERENCES
Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In ICML, pp. 312-321, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.099: 09882, 2020.
Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In ICML, pp. 1062-1070, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICMI,, 2020a.
Xinlei Chen and Abhinav Gupta. Webly supervised learning of convolutional networks. In ICCV, pp. 1431-1439, 2015.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint MiXx::0033,02297,, 2020b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248 200,-255,222 2009.
Santosh Kumar Divvala, Ali Farhadi, and Carlos Guestrin. Learning everything about anything: Webly-supervised visual concept learning. In CVPR, pp. 2270-3277, 2014.
Mark Everingham, Lue Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zis- serman. The pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2):303-338, 2010.
Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. LIBLINEAR: A library for large linear classification. JMLR, 9:1871-1874, 2008
Ross Girshick, Ilija Radosavovic, Georgia Gkioxari, Piotr Dollár, and Kaiming He. Detectron. 2018
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self- supervised visual representation learning. In ICCV, pp. 6391-6400, 2019.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning, arXiv preprint arXiv:2006.07733 2020.
Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong. Matthew R. Scott, and Dinglong Huang. Curriculumnet: Weakly supervised learning from large-scale web images. In ECCV, pp. 139-154, 2018.

---

Published as a conference paper at ICLR 2021
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurlPS, pp. 1336-856,, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, pp- 2980-2988, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning, arXiv preprint arXiv:1911.05722 2019.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint arXiv:1907.07174. 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of
generalization. arXiv preprint arXiv:2006.16241 2020.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In ICML, pp. 2309-2318 2018.
Lu Jiang, Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. In ICML, 2020.
Armand Joulin, Laurens van der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual fea- tures from large weakly supervised data. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), ECCV, 2016.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In ICLR, 2020.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Large scale learning general visual representations for transfer. In ECCV, 2020.
Ananya Kumar, Percy Liang, and Tengyu Ma. Verified uncertainty calibration. In Hanna M. Wal- lach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), NeurIPS, pp. 3787-3798, 2019.
Kuang-Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Cleannet: Transfer learning for scalable image classifier training with label noise. In CVPR, pp. 5447-5456, 2018
Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S. Kankanhalli. Learning to learn from noisy labeled data. In CVPR, pp. 5051-5059, 2019.
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In ICLR, 2020a.
Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven C.H. Hoi. Prototypical contrastive learning of unsupervised representations, arXiv preprint 200Vv:20005049666 2020b.
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Lue Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint /000v.0770.000000 02866, 2017.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, pp. 740-755, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge Be- longie. Feature pyramid networks for object detection. In CVPR, pp. 2017.
10

---

Published as a conference paper at ICLR 2021
Xingjun Ma, Yisen Wang, Michael E. Houle, Shuo Zhou, Sarah M. Erfani, Shu-Tao Xia, Sudanthi N. R. Wijewickrema, and James Bailey. Dimensionality-drieen learning with noisy labels. In ICML, pp. 3361-3370, 2018.
Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, pp. 185-201,2011
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv:1807.037444 2018.
Scott E. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. In ICLR, 2015.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data deep learning era. In ICCV, pp. 843-852, 2017.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization frame- work for learning with noisy labels. In CVPR, pp. 5552-5560, 2018.
Yi Tu, Li Niu, Dawei Cheng, and Liqing Zhang. Protonet: Learning from web data with memory. In CVPR, 2020.
Arash Vahdat. Toward robustness against label noise in training deep discriminative neural networks. In NIPS, pp. 5601-5610, 2017.
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge Belongie. Learn- ing from noisy large-scale datasets with minimal supervision. In CVPR, pp.
2017.
Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. In CVPR, pp.
2018.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In CVPR, pp. 3733-3742, 2018
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, pp. 1699, 2015.
Jingkang Yang. Litong Feng, Weirong Chen, Xiaopeng Yan, Huabin Zheng, Ping Luo, and Wayne Zhang. Webly supervised image classification with self-contained confidence. In ECCV, 2020.
Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In CVPR, 2019.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41: Self-supervised semi- supervised learning. In ICCV, pp. 1476-1485, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2017.
Weihe Zhang, Yali Wang, and Yu Qiao. Metacleaner: Learning to hallucinate clean representations for labsedd visual recognition. In CVPR, 2019.
Zizhao Zhang, Han Zhang, Sercan Ömer Arik, Honglak Lee, and Tomas Pfister. Distilling effective supervision from severe label noise. In CVPR, pp. 9291-9300, 2020.
Bolei Zhou, Ágata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. In NIPS, pp. 487-495, 2014.
11

---

Published as a conference paper at ICLR 2021
APPENDIX A NOISY SAMPLE VISUALIZATION
In Figure show example images randomly chosen from the out by our method. In Figure [ we show random examples where their pseudo-labels are different from the original training labels. By visual examination, observe that method remove OOD samples and correct noisy labels at high success rate.
samples filtered
THE AIRLINER
airliner
JUST NBO
basketball
goldfish, carassius auratus
iPod
Magpie Makeru
magpie
IORANGFA.
orange
wine bottle
Figure 3: Examples of randomly selected out-of-distribution samples filtered out by our method. The original training labels are shown below the images.
12

---

Published as a conference paper at ICLR 2021
airship, dirigible
American eagle
container ship
dining table
dockage
airliner
accordion
barn spider
panda
gray whale
jean
half track
laptop
mail bag
backpack
airship, dirigible
binoculars
envelope
hartebeest
hourglass
minibus
swing
iSto
balloon
fig
grocery store
jackfruit
packet
pin wheel
bahaha
garbage truck
church
motor scooter
remote control
school
traffic light
cab, taxi
cup
expresso
expresso maker
Arctic fox, white fox
gray fox miniskirt
mink
polecat
timber wolf
fur coat
aircraft
amphibian
assault rifle
mask
gas mask
Figure 4: Examples randomly selected samples with noisy labels corrected by our method. The original training labels are shown in red and corrected pseudo-labels are shown in green.
13

---

Published as a conference paper at ICLR 2021
APPENDIX B PSEUDO-CODE OF MOPRO
Algorithm summarizes the proposed method.
Algorithm 1: MoPro's main algorithm.
Input: number of classes K, temperature v, threshold momentum m, encoder network f(-), projection network 9(-), classifier h(.), momentum encoder (+))..)). for {(:; y/ 16 in loader do
// load a minibatch of noisy training data
for (1....,)) b) do
z = weak aug(a;)
// weak augmentation
/ strong aug(a:))
// strong augmentat
0, ((;;)
// representation
2, g(v))
normalized Low-dimensional embodding
2. g' d(((())))
// momentum embedding // class prediction
P: h(v,)
{ 1==1
ex/(2
// proootyypccall score
Σe exp(ar
// ncise correction
9. (p: 8,1/2
// soft pseudo-label
if maxx d T then arg max; o.
else if ahd > 1/K then
15
V1 = 4/
else
17
J. = OOD
end
// calculate losses
19
Lins log - exp(=
exp(x.x//T)
// instance contrastive loss
20
úM not OOD then
Lpr log Er exp(zx0./T)
// prototypical contrastive loss // cross entropy loss
L: = log(p"*) 41
22
23
else
Cpro fic =
end
// update momentum prototypes
Co: Normalize(mec,, + m)
end
28
(Et + fi + fine) L= Ei
// total loss
29
update networks f.g. to minimize C.
30
end
APPENDIX C TRANSFER LEARNING IMPLEMENTATION DETAILS
For low-shot image classification on Places and VOC, we follow the procedure in Li et al. (2020b) and train linear SVMs on the global average pooling features of ResNet-50. We preprocess all images by resizing to 256 pixels along the shorter side and taking 224 x 224 center crop. The SVMs are implemented in the LIBLINEAR (Fan al. 2008) package.
For low-resource finetuning on ImageNet, we adopt different finetuning strategy for different ver- sions of WebVision pretrained models. For Web VO.5 and V1.0, since they contain the same 1000 classes as ImageNet, we finetune the entire model including the classification layer. We train with SGD, using batch size of 256, momentum of 0.9, weight decay of 0, and learning rate of 0.005. We train for 40 epochs, and drop the learning rate by 0.2 at 15 and 30 epochs. For Web Vision 2.0, since it contains 5000 classes, randomly initialize new classification layer with 1000 output
14

---

Published as a conference paper at ICLR 2021
dimension, and finetune the model end-to-end. We train for 50 epochs, using learning rate of 0.01, which is dropped by 0.1 at 20 and 40 epochs.
For object detection and instance segmentation COCO, adopt the same setup in MoCo (He ct al., 2019), using Detectro ron2 (Girshick et al., 2018) codebase. The image scale is in [640, 800] pixels during training and is 800 at inference. We fine-tune all layers end-to-end. We finetune on the train2017 set (~~1 18k images) and evaluate on va12017.
APPENDIX D STANDARD DEVIATION FOR LOW-SHOT CLASSIFICATION
Table 7
reports the standard deviation for the low-shot image classification experiment Section 5.1
VOCO7
Places205
Method
Pretrain dataset
A= 1
h=2
lm4
k=8
l=
l==2
km4
k=8
CE (Sup.) ImageNet
5..3+4488
67.8
73,9+0.9
79.6+0.8
14,9+1. 21,0+0.3
26.9-0.6
32.1-0.44
MoPro
WebVision V1.0
59,5
71,3+2.2
16.
81.4+0.6
23.2+0.3 29.2:0.66 34.5=0.3 16.9
MoPro
WebVision- V2.0
64.816.7 74.8:1..6 19.9 83.9+1.0 22.2+1.3 29.2:0.5 35.6=0.77 40.9
exprrinnnaioooopeeeeenntttt Table 7: Low-shot image classification experiments. Mean and standard deviation are calculated across runs.
15


Reviews:
[{"review_id": "Xgns85xKmuC", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "To train a model with a noisy weakly supervised training set, this paper proposed a momentum prototypes method for label noise correction and OOD sample removal. Noise correction is done by a heuristic rule, that if the prediction is confident enough or the prediction on original label is higher than uniform probability, the label will be kept otherwise it is considered as OOD sample. For training the model, this paper jointly optimizes cross entropy loss on the corrected labels, as well as contrastive loss using prototypical examples and instances. \n\nExtensive experiments are performed on diverse tasks, including classification, transfer, robustness, detection and segmentation. \n\nMy main concern about this paper is on the results. MoPro is a method proposed to handle the noisy training labels, while only the cross entropy baselines are compared. When I checked a few publications, and found that the reported results are significantly lower than the existing methods. For example, learning from noisy labels paper DivideMix [1] reports 77.32% top-1 accuracy  on WebVision and 75.20% top-1 accuracy on ImageNet. Based on the results, it's unclear to me why the proposed method is superior to the existing methods that learn from noisy labels. \n[1] DivideMix: Learning with Noisy Labels as Semi-supervised Learning.\n\nIn Table 3, comparison to self-supervised learning is OK but not very meaningful. It would be nice to include other weakly-supervised learning results [Mahajan 2018, Kolesnikov 2020] for fair comparison purposes.\n\n=====================\n\nPost Rebuttal: I would like to thank the authors for the new results on WebVision-mini and ImageNet-mini, this has partially addressed my concerns as Reviewer3 raised similar issues on the SoTA claim. Overall, I think this paper is well presented and the results are solid, thus updated my rating to reflect this.", "extra_scores": {}, "extra_text": {}}, {"review_id": "VyiyUaSCitJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to modify the recent proposed prototypical contrastive learning work (which is in turn based on improved momentum contrastive learning work) and apply it to noisy web images with detailed insights on how to combat noise. It is shown that the representation learned in this way not only achieves good results on classifying web images, but also shows great transfer performance.\n\n+ I think it is a nice application of the recent unsupervised learning methods to settings that are more realistic (it is not practical to assume that there is absolutely no chance of getting a single label, but instead some label with noise). The proposed approach is reasonable, the motivation is strong, and the solution is (quite) simple and effective.\n+ Quite a nice set of experiments are done, expanding not only the within the webvision benchmark, but also to other down-stream tasks like COCO object detection.\n+ I like the robustness study showing that noisy images are good to train representations that generalize beyond normal ImageNet, but to other corrupted versions.\n\n- In Table 1, it mentions that CE is already achieving really good results and beats most of the state-of-the-art. What might be the reason? Is it possible that stronger data augmentations are used? I am asking because most of the contrastive method these days are NOT using the same set of basic augmentations that ImageNet supervised methods are using. This makes the comparison not too fair to other methods. If possible, please consider report experiments that follow the same augmentation protocol of other compared methods.\n- I noticed that different augmentations are used for the base encoder and the momentum encoder (Sec 4.2), why this is the case?\n- Missing reference: (the first work that explores representation learning with web images) Chen, Xinlei, and Abhinav Gupta. \"Webly supervised learning of convolutional networks.\" Proceedings of the IEEE International Conference on Computer Vision. 2015.  \n(the work that invented the word \"webly\"): Divvala, Santosh K., Ali Farhadi, and Carlos Guestrin. \"Learning everything about anything: Webly-supervised visual concept learning.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014.\nBoth work actually explored some mechanism/design to de-noise web images.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "U3AoWs21-dB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a weakly-supervised learning method based on the self-supervised contrastive loss to combine supervised learning and unsupervised contrastive learning. A label correction mechanism is proposed by utilizing distance in class prototypes. The overall combination is reasonable and convincing.\n\nProns: \n\n- As far as I know, this is the first work that combines weakly supervised learning and self-supervised learning.\n- The performance on Webvision dataset is nice\n- Writing it high-quality and easy to read.\n\nCons:\n\n- Novelty: This method is greatly relying on an previous method, prototypical contrastive learning (PCL).  The key technical contribution is Eq (4) and (5), however, they are relatively naive and straightforward design choices. \n- Experiments are a bit distracting and insufficient. I am not convinced by the larger amount of experiment focusing on tasks not actually related to weakly supervised learning (including detection, instance segmentation). \n- Since it is a weakly-supervised learning method, only one table evaluates this task.\n- Compared with unsupervised learning methods on downstreaming tasks are not that meaningful from my view, because the method accesses weak labels. Moreover, the noisy ratio of webvision dataset isn't that high, which can be inferred by the results in Table 6 (w/o L_pro & L_inst). \n- What is the motivation of validating the methods on so many downstream tasks?\n- The method replies on a temperature T, which is tuned based on datasets. The paper uses different values for webvision versions. Can it be generalized in practice? \n- More weak labeled datasets more meaningful, such as Clothing1M, Food101, and real world datasets presented by this paper \"Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels\" (ICML2020)\n- Distilling Effective Supervision from Severe Label Noise (CVPR2020) shows better Webvision results. So the SoTA claim in the abstract might not be less accurate. Proper discussions of related papers are encouraged.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "35N6re0c_rc", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Caveat: I am not very familiar with the weak supervision literature.\n\nPros:\n+ The method is fairly simple, a combination of three loss terms which have all been explored pretty comprehensively. The fact that a simple combination gives such solid performance improvements is a novel contribution.\n+ The results are convincing and significant. I especially appreciate the low-shot transfer results: I think they are the true test of a trained feature representation.\n+ The ablation and robustness studies are appreciated.\n\nCons:\n- While experimentally the paper is on solid ground, there isn't much intuition presented as to preccisely why this combination of losses is the right thing. For example, the technique proposed in section 3.3 can also be seen as trying to use the consensus between two classifiers, one based on prototypes and the other parametric. Is this the right intuition? If so, does one need to use a prototype based classifier? Or would a nearest neighbor work fine?\n- This paper inherits some of the mysteries of self supervised techniques: what function does the projector provide? what other architectural choices are important?\n- I am not that familiar with the WebVision dataset, but this dataset has been collected by using Google Image Search ccirca 2017: Google might have been using internal convnets to rank images by that point, in which case it is possible that all we are doing here is replicating/slightly improving their internal network. A discussion of this would be good.\n\n\nIn general, I think this is a good paper, but would still like to see the authors' response to the points above.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides an approach for weakly supervised learning by label noise correction and OOD sample removal. Overall, all reviewers agree paper is simple and approach makes sense. The experiments are solid with results on Webvision and ImageNet Mini (there were initial concerns but rebuttal handled some of those concerns). AC agrees with reviewers and recommends acceptance.

Decision: Accept (Poster)