Learning from Noisy Data with Robust Representation Learning

1 INTRODUCTION . Data in real life is noisy . However , deep models with remarkable performance are mostly trained on clean datasets with high-quality human annotations . Manual data cleaning and labeling is an expensive process that is difficult to scale . On the other hand , there exists almost infinite amount of noisy data online . It is crucial that deep neural networks ( DNNs ) could harvest noisy training data . However , it has been shown that DNNs are susceptible to overfitting to noise ( Zhang et al. , 2017 ) . As shown in Figure 1 , a real-world noisy image dataset often consists of multiple types of noise . Label noise refers to samples that are wrongly labeled as another class ( e.g . flower labeled as orange ) . Out-of-distribution input refers to samples that do not belong to any known classes . Input corruption refers to image-level distortion ( e.g . low brightness ) that causes data shift between training and test . Most of the methods in literature focus on addressing the more detrimental label noise . Two dominant approaches include : ( 1 ) find clean samples as those with smaller loss and assign larger weights to them ( Han et al. , 2018 ; Yu et al. , 2019 ; Shen & Sanghavi , 2019 ; Arazo et al. , 2019 ) ; ( 2 ) relabel noisy samples using model ’ s predictions ( Reed et al. , 2015 ; Ma et al. , 2018 ; Tanaka et al. , 2018 ; Yi & Wu , 2019 ) . The recently proposed DivideMix ( Li et al. , 2020a ) integrates both approaches in a co-training framework , but it also increases computation cost . Previous methods that focus on addressing label noise do not consider out-of-distribution input or input corruption , which limits their performance in real-world scenarios . Furthermore , using a model ’ s own prediction to relabel samples could cause confirmation bias , where the prediction error accumulates and harms performance . We propose a new direction for effective learning from noisy data . Our method embeds images into noise-robust low-dimensional representations , and regularizes the geometric structure of the representations with contrastive learning . Specifically , our algorithmic contributions include : • We propose noise-robust contrastive learning , which introduces two contrastive losses . The first is an unsupervised consistency contrastive loss . It enforces inputs with perturbations to have similar normalized embeddings , which helps learn robust and discriminative representation . • Our second contrastive loss is a weakly-supervised mixup prototypical loss . We compute class prototypes as normalized mean embeddings , and enforces each sample ’ s embedding to be closer to 1Code is in the supplementary material its class prototype . Inspired by Mixup ( Zhang et al. , 2018 ) , we construct virtual training samples as linear interpolation of inputs , and encourage the same linear relationship w.r.t the class prototypes . • We train a linear autoencoder to reconstruct the high-dimensional features using low-dimensional embeddings . The autoendoer enables the high-dimensional features to maximally preserve the robustness of the low-dimensional embeddings , thus regularizing the classifier . • We propose a new noise cleaning method which exploits the structure of the learned representations . For each sample , we aggregate information from its top-k neighbors to create a pseudo-label . A subset of training samples with confident pseudo-labels are selected to compute the weaklysupervised losses . This process can effectively clean both label noise and out-of-distribution ( OOD ) noise . Our experimental contributions include : • We experimentally show that our method is robust to label noise , OOD input , and input corruption . Experiments are performed on multiple datasets with controlled noise and real-world noise , where our method achieves state-of-the-art performance . • We demonstrate that the proposed noise cleaning method can effectively clean a majority of label noise . It also learns a curriculum that gradually leverages more samples to compute the weakly-supervised losses as the pseudo-labels become more accurate . • We validate the robustness of the learned low-dimensional representation by showing ( 1 ) k-nearest neighbor classification outperforms the softmax classifier . ( 2 ) OOD samples can be separated from in-distribution samples . The efficacy of the proposed autoencoder is also verified . 2 RELATED WORK . Label noise learning . Learning from noisy labels have been extensively studied in the literature . While some methods require access to a small set of clean samples ( Xiao et al. , 2015 ; Vahdat , 2017 ; Veit et al. , 2017 ; Lee et al. , 2018 ; Hendrycks et al. , 2018 ) , most methods focus on the more challenging scenario where no clean labels are available . These methods can be categorized into two major types . The first type performs label correction using predictions from the network ( Reed et al. , 2015 ; Ma et al. , 2018 ; Tanaka et al. , 2018 ; Yi & Wu , 2019 ) . The second type tries to separate clean samples from corrupted samples , and trains the model on clean samples ( Han et al. , 2018 ; Arazo et al. , 2019 ; Jiang et al. , 2018 ; 2020 ; Wang et al. , 2018 ; Chen et al. , 2019 ; Lyu & Tsang , 2020 ) . The recently proposed DivideMix ( Li et al. , 2020a ) effectively combines label correction and sample selection with the Mixup ( Zhang et al. , 2018 ) data augmentation under a co-training framework . However , it cost 2⇥ the computational resource of our method . Different from existing methods , our method combats noise by learning noise-robust low-dimensional representations . We propose a more effective noise cleaning method by leveraging the structure of the learned representations . Furthermore , our model is robust not only to label noise , but also to out-of-distribution and corrupted input . A previous work has studied open-set noisy labels ( Wang et al. , 2018 ) , but their method does not enjoy the same level of robustness as ours . Contrastive learning . Contrastive learning is at the core of recent self-supervised representation learning methods ( Chen et al. , 2020 ; He et al. , 2019 ; Oord et al. , 2018 ; Wu et al. , 2018 ) . In selfsupervised contrastive learning , two randomly augmented images are generated for each input image . Then a contrastive loss is applied to pull embeddings from the same source image closer , while pushing embeddings from different source images apart . Recently , prototypical contrastive learning ( PCL ) ( Li et al. , 2020b ) has been proposed , which uses cluster centroids as prototypes , and trains the network by pulling an image embedding closer to its assigned prototypes . Different from previous methods , our method performs contrastive learning in the principal subspace of the high-dimensional feature space , by training a linear autoencoder . Furthermore , our supervised contrastive loss improves PCL ( Li et al. , 2020b ) with Mixup ( Zhang et al. , 2018 ) . Different from the original Mixup where learning happens at the classification layer , our learning takes places in the low-dimensional subspace . 3 METHOD . Given a noisy training dataset D = { ( xi , yi ) } ni=1 , where xi is an image and yi 2 { 1 , ... , C } is its class label . We aim to train a network that is robust to the noise in training data ( i.e . label noise , OOD input , input corruption ) and achieves high accuracy on a clean test set . The proposed network consists of three components : ( 1 ) a deep encoder ( a convolutional neural network ) that encodes an image xi to a high-dimensional feature vi ; ( 2 ) a classifier ( a fully-connected layer followed by softmax ) that receives vi as input and outputs class predictions ; ( 3 ) a linear autoencoder that projects vi into a low-dimensional embedding zi 2 Rd . We show an illustration of our method in Figure 2 , and a pseudo-code in appendix B . Next , we delineate its details . 3.1 CONTRASTIVE LEARNING IN ROBUST LOW-DIMENSIONAL SUBSPACE . Let zi = Wevi be the linear projection from high-dimensional features to low-dimensional embeddings , and ẑi = zi/ kzik2 be the normalized embeddings . We aim to learn robust embeddings with two contrastive losses : unsupervised consistency loss and weakly-supervised mixup prototypical loss . Unsupervised consistency contrastive loss . Following the NT-Xent ( Chen et al. , 2020 ) loss for selfsupervised representation learning , our consistency contrastive loss enforces images with semanticpreserving perturbations to have similar embeddings . Specifically , given a minibatch of b images , we apply weak-augmentation and strong-augmentation to each image , and obtain 2b inputs { xi } 2bi=1 . Weak augmentation is a standard flip-and-shift augmentation strategy , while strong augmentation consists of color and brightness changes with details given in Section 4.1 . We project the inputs into the low-dimensional space to obtain their normalized embeddings { ẑi } 2bi=1 . Let i 2 { 1 , ... , b } be the index of a weakly-augmented input , and j ( i ) be the index of the strong- augmented input from the same source image , the consistency contrastive loss is defined as : Lcc = bX i=1 log exp ( ẑi · ẑj ( i ) /⌧ ) P2b k=1 i 6=k exp ( ẑi · ẑk/⌧ ) , ( 1 ) where ⌧ is a scalar temperature parameter . The consistency contrastive loss maximizes the inner product between the pair of positive embeddings ẑi and ẑj ( i ) , while minimizing the inner product between 2 ( b 1 ) pairs of negative embeddings . By mapping different views ( augmentations ) of the same image to neighboring embeddings , the consistency contrastive loss encourages the network to learn discriminative representation that is robust to low-level image corruption . Weakly-supervised mixup prototypical contrastive loss . Our second contrastive loss injects structural knowledge of classes into the embedding space . Let Ic denote indices for the subset of images in D labeled with class c , we calculate the class prototype as the normalized mean embedding : zc = 1 |Ic| X i2Ic ẑi , ẑ c = zc kzck2 , ( 2 ) where ẑi is the embedding of a center-cropped image , and the class prototypes are calculated at the beginning of each epoch . The prototypical contrastive loss enforces an image embedding ẑi to be more similar to its corresponding class prototype ẑyi , in contrast to other class prototypes : Lpc ( ẑi , yi ) = log exp ( ẑi · ẑyi/⌧ ) PC c=1 exp ( ẑi · ẑc/⌧ ) . ( 3 ) Since the label yi is noisy , we would like to regularize the encoder from memorizing training labels . Mixup ( Zhang et al. , 2018 ) has been shown to be an effective method against label noise ( Arazo et al. , 2019 ; Li et al. , 2020a ) . Inspired by it , we create virtual training samples by linearly interpolating a sample ( indexed by i ) with another sample ( indexed by m ( i ) ) randomly chosen from the same minibatch : xmi = xi + ( 1 ) xm ( i ) , ( 4 ) where ⇠ Beta ( ↵ , ↵ ) . Let ẑmi be the normalized embedding for xmi , the mixup version of the prototypical contrastive loss is defined as a weighted combination of the two Lpc w.r.t class yi and ym ( i ) . It enforces the embedding for the interpolated input to have the same linear relationship w.r.t . the class prototypes . Lpc mix = 2bX i=1 Lpc ( ẑmi , yi ) + ( 1 ) Lpc ( ẑmi , ym ( i ) ) . ( 5 ) Reconstruction loss . We also train a linear decoder Wd to reconstruct the high-dimensional feature vi based on zi . The reconstruction loss is defined as : Lrecon = 2bX i=1 kvi Wdzik22 . ( 6 ) There are several benefits for training the autoencoder . First , with an optimal linear autoencoder , We will project vi into its low-dimensional principal subspace and can be understood as applying PCA ( Baldi & Hornik , 1989 ) . Thus the low-dimensional representation zi is intrinsically robust to input noise . Second , minimizing the reconstruction error is maximizing a lower bound of the mutual information between vi and zi ( Vincent et al. , 2010 ) . Therefore , knowledge learned from the proposed contrastive losses can be maximally preserved in the high-dimensional representation , which helps regularize the classifier . Classification loss . Given the softmax output from the classifier , p ( y ; xi ) , we define the classification loss as the cross-entropy loss . Note that it is only applied to the weakly-augmented inputs . Lce = bX i=1 log p ( yi ; xi ) . ( 7 ) The overall training objective is to minimize a weighted sum of all losses : L = Lce + ! ccLcc + ! pcLpc mix + ! reconLrecon ( 8 ) For all experiments , we fix ! cc = 1 , ! recon = 1 , and change ! pc only across datasets .

Summary: The authors of the paper propose to use the contrastive loss, the mixup prototypical loss, and a reconstruction loss to regularize the learned representation in order to achieve robustness under various kinds of noise like label noise, out-of-distribution input, and input corruption. A noise-cleaning process based on the learned representation is also introduced to further enhance the results. Extensive experiments were conducted to demonstrate the effectiveness of the method.