MoPro: Webly Supervised Learning with Momentum Prototypes

1 INTRODUCTION . Large-scale datasets with human-annotated labels have revolutionized computer vision . Supervised pretraining on ImageNet ( Deng et al. , 2009 ) has been the de facto formula of success for almost all state-of-the-art visual perception models . However , it is extremely labor intensive to manually annotate millions of images , which makes it a non-scalable solution . One alternative to reduce annotation cost is self-supervised representation learning , which leverages unlabeled data . However , self-supervised learning methods ( Goyal et al. , 2019 ; He et al. , 2019 ; Chen et al. , 2020a ; Li et al. , 2020b ) have yet consistently shown superior performance compared to supervised learning , especially when transferred to downstream tasks with limited labels . With the help of commercial search engines , photo-sharing websites , and social media platforms , there is near-infinite amount of weakly-labeled images available on the web . Several works have exploited the scalable source of web images and demonstrated promising results with weblysupervised representation learning ( Mahajan et al. , 2018 ; Sun et al. , 2017 ; Li et al. , 2017 ; Kolesnikov et al. , 2020 ) . However , there exists two competing claims on whether weakly-labeled noisy datasets lead to worse generalization performance . One claim argues that the effect of noise can be overpowered by the scale of data , and simply applies standard supervised learning method on web datasets ( Mahajan et al. , 2018 ; Sun et al. , 2017 ; Li et al. , 2017 ; Kolesnikov et al. , 2020 ) . The other claim argues that deep models can easily memorize noisy labels , resulting in worse generalization ( Zhang et al. , 2017 ; Ma et al. , 2018 ) . In this paper , we show that both claims are partially true . While increasing the size of data does improve the model ’ s robustness to noise , our method can substantially boost the representation learning performance by addressing noise . There exists a large body of literature on learning with label noise ( Jiang et al. , 2018 ; Han et al. , 2018 ; Guo et al. , 2018 ; Tanaka et al. , 2018 ; Arazo et al. , 2019 ; Li et al. , 2020a ) . However , existing methods have several limitations that make them less effective for webly-supervised representation learning . First , most methods do not consider out-of-distribution ( OOD ) samples , which is a major source of noise in real-world web datasets . Second , many methods perform computation-heavy procedures for noise cleaning ( Jiang et al. , 2018 ; Li et al. , 2019 ; 2020a ) , or require access to a set of samples with clean labels ( Vahdat , 2017 ; Veit et al. , 2017 ; Lee et al. , 2018 ) , which limit their scalability in practice . We propose a new method for efficient representation learning from weakly-labeled web images . Our method is inspired by recent developments in contrastive learning for self-supervised learning ( He et al. , 2019 ; Chen et al. , 2020a ; Li et al. , 2020b ) We introduce Momentum Prototypes ( MoPro ) , a simple component which is effective in label noise correction , OOD sample removal , and representation learning . A visual explanation of our method is shown in Figure 1 . We use a deep network to project images into normalized low-dimensional embeddings , and calculate the prototype for a class as the moving-average embedding for clean samples in that class . We train the network such that embeddings are pulled closer to their corresponding prototypes , while pushed away from other prototypes . Images with corrupted labels are corrected either as another class or as an OOD sample based on their distance to the momentum prototypes . We experimentally show that : • MoPro achieves state-of-the-art performance on the upstream weakly-supervised learning task . • MoPro substantially improves representation learning performance when the pretrained model is transferred to downstream image classification and object detection tasks . For the first time , we show that weakly-supervised representation learning achieves similar performance as supervised representation learning , under the same data and computation budget . With a larger web dataset , MoPro outperforms ImageNet supervised learning by a large margin . • MoPro learns a more robust and calibrated model that generalizes better to distribution variations . 2 RELATED WORK . 2.1 WEBLY-SUPERVISED REPRESENTATION LEARNING . A number of prior works exploit large web datasets for visual representation learning ( Divvala et al. , 2014 ; Chen & Gupta , 2015 ; Joulin et al. , 2016 ; Mahajan et al. , 2018 ; Sun et al. , 2017 ; Li et al. , 2017 ; Kolesnikov et al. , 2020 ) . These datasets contain a considerable amount of noise . Approximately 20 % of the labels in the JMT-300M dataset ( Sun et al. , 2017 ) are noisy , whereas 34 % of images in the WebVision dataset ( Li et al. , 2017 ) are considered outliers . Surprisingly , most prior works have chosen to ignore the noise and applied vanilla supervised method , with the claim that the scale of data can overpower the noise ( Mahajan et al. , 2018 ; Sun et al. , 2017 ; Li et al. , 2017 ) . However , we show that supervised method can not fully harvest the power of large-scale weakly-labeled datasets . Our method achieves substantial improvement by addressing noise , and advances the potential of webly-supervised representation learning . 2.2 LEARNING WITH LABEL NOISE . Learning with label noise has been widely studied . Some methods require access to a small set of clean samples ( Xiao et al. , 2015 ; Vahdat , 2017 ; Veit et al. , 2017 ; Lee et al. , 2018 ; Zhang et al. , 2020 ) , and other methods assume that no clean labels are available . There exist two major types of approaches . The first type performs label correction using predictions from the network ( Reed et al. , 2015 ; Ma et al. , 2018 ; Tanaka et al. , 2018 ; Yi & Wu , 2019 ; Yang et al. , 2020 ) . The second type separates clean samples from corrupted samples , and trains the model on clean samples ( Han et al. , 2018 ; Arazo et al. , 2019 ; Jiang et al. , 2018 ; Wang et al. , 2018 ; Chen et al. , 2019 ; Li et al. , 2020a ) . However , existing methods have yet shown promising results for large-scale weakly-supervised representation learning . The main reasons include : ( 1 ) most methods do not consider OOD samples , which commonly occur in real-world web datasets ; ( 2 ) most methods are computational-heavy due to co-training ( Han et al. , 2018 ; Li et al. , 2020a ; Jiang et al. , 2018 ; 2020 ) , iterative training ( Tanaka et al. , 2018 ; Yi & Wu , 2019 ; Wang et al. , 2018 ; Chen et al. , 2019 ) , or meta-learning ( Li et al. , 2019 ; Zhang et al. , 2019 ) . Different from existing methods , MoPro achieves both label correction and OOD sample removal on-the-fly with a single step , based on the similarity between an image embedding and the momentum prototypes . MoPro also leverages contrastive learning to learn a robust embedding space . 2.3 SELF-SUPERVISED REPRESENTATION LEARNING . Self-supervised methods have been proposed for representation learning using unlabeled data . The recent developments in self-supervised representation learning can be attributed to contrastive learning . Most methods ( He et al. , 2019 ; Chen et al. , 2020a ; Oord et al. , 2018 ; Wu et al. , 2018 ) leverage the task of instance discrimination , where augmented crops from the same source image are enforced to have similar embeddings . Prototypical contrastive learning ( PCL ) ( Li et al. , 2020b ) performs clustering to find prototypical embeddings , and enforces an image embedding to be similar to its assigned prototypes . Different from PCL , we update prototypes on-the-fly in a weakly-supervised setting , where the momentum prototype of a class is the moving average of clean samples ’ embeddings . Furthermore , we jointly optimize two contrastive losses and a cross-entropy loss . Current self-supervised representation learning methods are limited in ( 1 ) inferior performance in low-shot task adaptation , ( 2 ) huge computation cost , and ( 3 ) inadequate to harvest larger datasets . We show that weakly-supervised learning with MoPro addresses these limitations . 3 METHOD . In this section , we delineate the details of our method . First , we introduce the components in our representation learning framework . Then , we describe the loss functions . Finally , we explain the noise correction procedure for label correction and OOD sample removal . A pseudo-code of MoPro is provided in appendix B . 3.1 REPRESENTATION LEARNING FRAMEWORK . Our proposed framework consists of the following components . Figure 2 gives an illustration . • A noisy training dataset { ( xi , yi ) } ni=1 , where xi is an image and yi ∈ { 1 , ... , K } is its class label . • A pseudo-label ŷi for each image xi , which is its corrected label . Details for generating the pseudo-label is explained in Sec 3.3 . • An encoder network , which maps an augmented image x̃i to a representation vector vi ∈ Rde . We experiment with ResNet-50 ( He et al. , 2016 ) as the encoder , where the activations of the final global pooling layer ( de = 2048 ) are used as the representation vector . • A classifier ( a fully-connected layer followed by softmax ) which receives the representation vi as input and outputs class predictions pi . • A projection network , which maps the representation vi into a low-dimensional embedding zi ∈ Rdp ( dp = 128 ) . zi is always normalized to the unit sphere . Following SimCLR ( Chen et al. , 2020a ) , we use a MLP with one hidden layer as the projection network . • Momentum embeddings z′i generated by a momentum encoder . The momentum encoder has the same architecture as the encoder followed by the projection network , and its parameters are the moving-average of the encoder ’ s and the projection network ’ s parameters . Same as in MoCo ( He et al. , 2019 ) , we maintain a queue of momentum embeddings of past samples . • Momentum prototypes C ∈ Rdp×K . The momentum prototype of the k-th class , ck , is the normalized moving-average embedding for samples with pseudo-label ŷi = k . 3.2 CONTRASTIVE LOSS . As illustrated in Figure 1 , we aim to learn an embedding space where samples from the same class gather around its class prototype , while samples from different classes are seperated . We achieve it with two contrastive losses : ( 1 ) a prototypical contrastive loss Lpro which increases the similarity between an embedding and its corresponding class prototype , ( zi , cŷi ) , in contrast to other prototypes ; ( 2 ) an instance contrastive loss Lins which increases the similarity between two embeddings of the same source image , ( zi , z′i ) , in contrast to embeddings of other images . Specifically , the contrastive losses are defined as : Lipro = − log exp ( zi · cŷi/τ ) ∑K k=1 exp ( zi · ck/τ ) , Liins = − log exp ( zi · z′i/τ ) ∑R r=0 exp ( zi · z′r/τ ) , ( 1 ) where τ is a temperature parameter , and ŷi is the pseudo-label . We use R negative momentum embeddings to construct the denominator of the instance contrastive loss . We train the classifier with cross-entropy loss , using pseudo-labels as targets . Lice = − log ( p ŷi i ) ( 2 ) We jointly optimize the contrastive losses and the classification loss . The training objective is : L = n∑ i=1 ( Lice + λproLipro + λinsLiins ) ( 3 ) For simplicity , we set λpro = λins = 1 for all experiments .

Summary: To train a model with a noisy weakly supervised training set, this paper proposed a momentum prototypes method for label noise correction and OOD sample removal. Noise correction is done by a heuristic rule, that if the prediction is confident enough or the prediction on original label is higher than uniform probability, the label will be kept otherwise it is considered as OOD sample. For training the model, this paper jointly optimizes cross entropy loss on the corrected labels, as well as contrastive loss using prototypical examples and instances.