Year: 2020

Abstract:
Learning from noisy data has attracted much attention, where most methods focus on label noise. In this work, we propose a new framework which simultaneously addresses three types of noise commonly seen in real-world data: label noise, outof-distribution input, and input corruption. In contrast to most existing methods, we combat noise by learning robust representation. Specifically, we embed images into a low-dimensional subspace by training an autoencoder on the deep features. We regularize the geometric structure of the subspace with robust contrastive learning, which includes an unsupervised consistency loss and a supervised mixup prototypical loss. Furthermore, we leverage the structure of the learned subspace for noise cleaning, by aggregating information from neighboring samples. Experiments on multiple benchmarks demonstrate state-of-the-art performance of our method and robustness of the learned representation. Our code will be released1.

heading: 1 INTRODUCTION
text: Data in real life is noisy. However, deep models with remarkable performance are mostly trained on clean datasets with high-quality human annotations. Manual data cleaning and labeling is an expensive process that is difficult to scale. On the other hand, there exists almost infinite amount of noisy data online. It is crucial that deep neural networks (DNNs) could harvest noisy training data. However, it has been shown that DNNs are susceptible to overfitting to noise (Zhang et al., 2017).
As shown in Figure 1, a real-world noisy image dataset often consists of multiple types of noise. Label noise refers to samples that are wrongly labeled as another class (e.g. flower labeled as orange). Out-of-distribution input refers to samples that do not belong to any known classes. Input corruption refers to image-level distortion (e.g. low brightness) that causes data shift between training and test.
Most of the methods in literature focus on addressing the more detrimental label noise. Two dominant approaches include: (1) find clean samples as those with smaller loss and assign larger weights to them (Han et al., 2018; Yu et al., 2019; Shen & Sanghavi, 2019; Arazo et al., 2019); (2) relabel noisy samples using model’s predictions (Reed et al., 2015; Ma et al., 2018; Tanaka et al., 2018; Yi & Wu, 2019). The recently proposed DivideMix (Li et al., 2020a) integrates both approaches in a co-training framework, but it also increases computation cost. Previous methods that focus on addressing label noise do not consider out-of-distribution input or input corruption, which limits their performance in real-world scenarios. Furthermore, using a model’s own prediction to relabel samples could cause confirmation bias, where the prediction error accumulates and harms performance.
We propose a new direction for effective learning from noisy data. Our method embeds images into noise-robust low-dimensional representations, and regularizes the geometric structure of the representations with contrastive learning. Specifically, our algorithmic contributions include: • We propose noise-robust contrastive learning, which introduces two contrastive losses. The first is
an unsupervised consistency contrastive loss. It enforces inputs with perturbations to have similar normalized embeddings, which helps learn robust and discriminative representation.
• Our second contrastive loss is a weakly-supervised mixup prototypical loss. We compute class prototypes as normalized mean embeddings, and enforces each sample’s embedding to be closer to
1Code is in the supplementary material
its class prototype. Inspired by Mixup (Zhang et al., 2018), we construct virtual training samples as linear interpolation of inputs, and encourage the same linear relationship w.r.t the class prototypes.
• We train a linear autoencoder to reconstruct the high-dimensional features using low-dimensional embeddings. The autoendoer enables the high-dimensional features to maximally preserve the robustness of the low-dimensional embeddings, thus regularizing the classifier.
• We propose a new noise cleaning method which exploits the structure of the learned representations. For each sample, we aggregate information from its top-k neighbors to create a pseudo-label. A subset of training samples with confident pseudo-labels are selected to compute the weaklysupervised losses. This process can effectively clean both label noise and out-of-distribution (OOD) noise.
Our experimental contributions include: • We experimentally show that our method is robust to label noise, OOD input, and input corruption.
Experiments are performed on multiple datasets with controlled noise and real-world noise, where our method achieves state-of-the-art performance.
• We demonstrate that the proposed noise cleaning method can effectively clean a majority of label noise. It also learns a curriculum that gradually leverages more samples to compute the weakly-supervised losses as the pseudo-labels become more accurate.
• We validate the robustness of the learned low-dimensional representation by showing (1) k-nearest neighbor classification outperforms the softmax classifier. (2) OOD samples can be separated from in-distribution samples. The efficacy of the proposed autoencoder is also verified.

heading: 2 RELATED WORK
text: Label noise learning. Learning from noisy labels have been extensively studied in the literature. While some methods require access to a small set of clean samples (Xiao et al., 2015; Vahdat, 2017; Veit et al., 2017; Lee et al., 2018; Hendrycks et al., 2018), most methods focus on the more challenging scenario where no clean labels are available. These methods can be categorized into two major types. The first type performs label correction using predictions from the network (Reed et al., 2015; Ma et al., 2018; Tanaka et al., 2018; Yi & Wu, 2019). The second type tries to separate clean samples from corrupted samples, and trains the model on clean samples (Han et al., 2018; Arazo et al., 2019; Jiang et al., 2018; 2020; Wang et al., 2018; Chen et al., 2019; Lyu & Tsang, 2020). The recently proposed DivideMix (Li et al., 2020a) effectively combines label correction and sample selection with the Mixup (Zhang et al., 2018) data augmentation under a co-training framework. However, it cost 2⇥ the computational resource of our method. Different from existing methods, our method combats noise by learning noise-robust low-dimensional representations. We propose a more effective noise cleaning method by leveraging the structure of the learned representations. Furthermore, our model is robust not only to label noise, but also to out-of-distribution and corrupted input. A previous work has studied open-set noisy labels (Wang et al., 2018), but their method does not enjoy the same level of robustness as ours.
Contrastive learning. Contrastive learning is at the core of recent self-supervised representation learning methods (Chen et al., 2020; He et al., 2019; Oord et al., 2018; Wu et al., 2018). In selfsupervised contrastive learning, two randomly augmented images are generated for each input image. Then a contrastive loss is applied to pull embeddings from the same source image closer, while pushing embeddings from different source images apart. Recently, prototypical contrastive learning (PCL) (Li et al., 2020b) has been proposed, which uses cluster centroids as prototypes, and trains the network by pulling an image embedding closer to its assigned prototypes.
Different from previous methods, our method performs contrastive learning in the principal subspace of the high-dimensional feature space, by training a linear autoencoder. Furthermore, our supervised contrastive loss improves PCL (Li et al., 2020b) with Mixup (Zhang et al., 2018). Different from the original Mixup where learning happens at the classification layer, our learning takes places in the low-dimensional subspace.

heading: 3 METHOD
text: Given a noisy training dataset D = {(xi, yi)}ni=1, where xi is an image and yi 2 {1, ..., C} is its class label. We aim to train a network that is robust to the noise in training data (i.e. label noise, OOD input, input corruption) and achieves high accuracy on a clean test set. The proposed network consists of three components: (1) a deep encoder (a convolutional neural network) that encodes an image xi to a high-dimensional feature vi; (2) a classifier (a fully-connected layer followed by softmax) that receives vi as input and outputs class predictions; (3) a linear autoencoder that projects vi into a low-dimensional embedding zi 2 Rd. We show an illustration of our method in Figure 2, and a pseudo-code in appendix B. Next, we delineate its details.

heading: 3.1 CONTRASTIVE LEARNING IN ROBUST LOW-DIMENSIONAL SUBSPACE
text: Let zi = Wevi be the linear projection from high-dimensional features to low-dimensional embeddings, and ẑi = zi/ kzik2 be the normalized embeddings. We aim to learn robust embeddings with two contrastive losses: unsupervised consistency loss and weakly-supervised mixup prototypical loss.
Unsupervised consistency contrastive loss. Following the NT-Xent (Chen et al., 2020) loss for selfsupervised representation learning, our consistency contrastive loss enforces images with semanticpreserving perturbations to have similar embeddings. Specifically, given a minibatch of b images, we apply weak-augmentation and strong-augmentation to each image, and obtain 2b inputs {xi}2bi=1. Weak augmentation is a standard flip-and-shift augmentation strategy, while strong augmentation consists of color and brightness changes with details given in Section 4.1.
We project the inputs into the low-dimensional space to obtain their normalized embeddings {ẑi}2bi=1. Let i 2 {1, ..., b} be the index of a weakly-augmented input, and j(i) be the index of the strong-
augmented input from the same source image, the consistency contrastive loss is defined as:
Lcc = bX
i=1
log exp(ẑi · ẑj(i)/⌧)
P2b k=1 i 6=k exp(ẑi · ẑk/⌧) , (1)
where ⌧ is a scalar temperature parameter. The consistency contrastive loss maximizes the inner product between the pair of positive embeddings ẑi and ẑj(i), while minimizing the inner product between 2(b 1) pairs of negative embeddings. By mapping different views (augmentations) of the same image to neighboring embeddings, the consistency contrastive loss encourages the network to learn discriminative representation that is robust to low-level image corruption.
Weakly-supervised mixup prototypical contrastive loss. Our second contrastive loss injects structural knowledge of classes into the embedding space. Let Ic denote indices for the subset of images in D labeled with class c, we calculate the class prototype as the normalized mean embedding:
zc = 1 |Ic| X
i2Ic
ẑi, ẑ c =
zc
kzck2 , (2)
where ẑi is the embedding of a center-cropped image, and the class prototypes are calculated at the beginning of each epoch.
The prototypical contrastive loss enforces an image embedding ẑi to be more similar to its corresponding class prototype ẑyi , in contrast to other class prototypes:
Lpc(ẑi, yi) = log exp(ẑi · ẑyi/⌧)PC c=1 exp(ẑi · ẑc/⌧) . (3)
Since the label yi is noisy, we would like to regularize the encoder from memorizing training labels. Mixup (Zhang et al., 2018) has been shown to be an effective method against label noise (Arazo et al., 2019; Li et al., 2020a). Inspired by it, we create virtual training samples by linearly interpolating a sample (indexed by i) with another sample (indexed by m(i)) randomly chosen from the same minibatch: xmi = xi + (1 )xm(i), (4) where ⇠ Beta(↵,↵). Let ẑmi be the normalized embedding for xmi , the mixup version of the prototypical contrastive loss is defined as a weighted combination of the two Lpc w.r.t class yi and ym(i). It enforces the embedding for the interpolated input to have the same linear relationship w.r.t. the class prototypes.
Lpc mix = 2bX
i=1
Lpc(ẑmi , yi) + (1 )Lpc(ẑmi , ym(i)). (5)
Reconstruction loss. We also train a linear decoder Wd to reconstruct the high-dimensional feature vi based on zi. The reconstruction loss is defined as:
Lrecon = 2bX
i=1
kvi Wdzik22 . (6)
There are several benefits for training the autoencoder. First, with an optimal linear autoencoder, We will project vi into its low-dimensional principal subspace and can be understood as applying PCA (Baldi & Hornik, 1989). Thus the low-dimensional representation zi is intrinsically robust to input noise. Second, minimizing the reconstruction error is maximizing a lower bound of the mutual information between vi and zi (Vincent et al., 2010). Therefore, knowledge learned from the proposed contrastive losses can be maximally preserved in the high-dimensional representation, which helps regularize the classifier.
Classification loss. Given the softmax output from the classifier, p(y;xi), we define the classification loss as the cross-entropy loss. Note that it is only applied to the weakly-augmented inputs.
Lce = bX
i=1
log p(yi;xi). (7)
The overall training objective is to minimize a weighted sum of all losses:
L = Lce + !ccLcc + !pcLpc mix + !reconLrecon (8)
For all experiments, we fix !cc = 1, !recon = 1, and change !pc only across datasets.

heading: 3.2 NOISE CLEANING WITH SMOOTH NEIGHBORS
text: After warming-up the model by training with the noisy labels {yi}ni=1 for t0 epochs, we aim to clean the noise by generating a soft pseudo-label qi for each training sample. Different from previous methods that perform label correction purely using the model’s softmax prediction, our method exploits the structure of the low-dimensional subspace by aggregating information from top-k neighboring samples, which helps alleviate the confirmation bias problem.
At the t-th epoch, for each sample xi, let pti be the classifier’s softmax prediction, let q t 1 i be its soft label from the previous epoch, we calculate the soft label for the current epoch as:
qti = 1
2 pti +
1
2
kX
j=1
wtijq t 1 j , (9)
where wtij represents the normalized affinity between a sample and its neighbor and is defined as wtij = exp(ẑti ·ẑ t j/⌧)Pk
j=1 exp(ẑ t i ·ẑtj/⌧)
. We set k = 200 in all experiments.
The soft label defined by eqn.(9) is the minimizer of the following quadratic loss function:
J(qti) = kX
j=1
wtij qti qt 1j 2 2 + qti pti 2 2 . (10)
The first term is a smoothness constraint which encourages the soft label to take a similar value as its neighbors’ labels, whereas the second term attempts to maintain the model’s class prediction.
We construct a weakly-supervised subset which contains (1) clean sample whose soft label score for the original class yi is higher than a threshold ⌘0, (2) pseudo-labeled sample whose maximum soft label score exceeds a threshold ⌘1. For pseudo-labeled samples, we convert their soft labels into hard labels by taking the class with the maximum score.
Dtsup = {xi, yi | qti(yi) > ⌘0}[ {xi, ŷti = argmax c qti(c) | 8maxc q t i(c) > ⌘1, c 2 {1, .., C}} (11)
Given the weakly-supervised subset, we modify the classification loss Lce, the mixup prototypical contrastive loss Lpc mix, and the calculation of prototypes ẑc, such that they only use samples from Dtsup. The unsupervised losses (i.e. Lcc and Lrecon) still operate on all training samples. Learning curriculum. Our iterative noise cleaning method learns an effective training curriculum, which gradually increases the size of Dtsup as the pseudo-labels become more accurate. To demonstrate
such curriculum, we analyse the noise cleaning statistics for training our model on CIFAR-10 and CIFAR-100 datasets with 50% label noise (experimental details explained in the next section). In Figure 3 (a), we show the accuracy of the soft pseudo-labels w.r.t to clean training labels (only used for analysis purpose). Our method can significantly reduce the ratio of label noise from 50% to 5% (for CIFAR-10) and 17% (for CIFAR-100). Figure 3 (b) shows the size of Dtsup as a percentage of the total number of training samples, and Figure 3 (c) shows the effective label noise ratio within the weakly-supervised subset Dtsup. Our method maintains a low noise ratio in the weakly-supervised subset, while gradually increasing its size to utilize more samples for the weakly-supervised losses.

heading: 4 EXPERIMENT
text: In this section, we validate the proposed method on multiple benchmarks with controlled noise and real-world noise. Our method achieves state-of-the-art performance across all benchmarks. For fair comparison, we compare with DivideMix (Li et al., 2020a) without ensemble. In appendix A, we report the result of our method with co-training and ensemble, which further improves performance.

heading: 4.1 EXPERIMENTS ON CONTROLLED NOISY LABELS
text: Dataset. Following Tanaka et al. (2018); Li et al. (2020a), we corrupt the training data of CIFAR10 and CIFAR-100 (Krizhevsky & Hinton, 2009) with two types of label noise: symmetric and asymmetric. Symmetric noise is injected by randomly selecting a percentage of samples and changing their labels to random labels. Asymmetric noise is class-dependant, where labels are only changed to similar classes (e.g. dog$cat, deer!horse). We experiment with multiple noise ratios: sym 20%, sym 50%, and asym 40% (see results for sym 80% and 90% in appendix A). Note that asymmetric noise ratio cannot exceed 50% because certain classes would become theoretically indistinguishable.
Implementation details. Same as previous works (Arazo et al., 2019; Li et al., 2020a), we use PreAct ResNet-18 (He et al., 2016) as our encoder model. We set the dimensionality of the bottleneck layer as d = 50. Our model is trained using SGD with a momentum of 0.9, a weight decay of 0.0005, and a batch size of 128. The network is trained for 200 epochs. We set the initial learning rate as 0.02 and use a cosine decay schedule. We apply standard crop and horizontal flip as the weak augmentation. For strong augmentation, we use AugMix (Hendrycks et al., 2020), though other methods (e.g. SimAug (Chen et al., 2020)) work equally well. For all CIFAR experiments, we fix the hyper-parameters as !cc = 1,!pc = 5,!recon = 1, ⌧ = 0.3,↵ = 8, ⌘1 = 0.9. For CIFAR-10, we activate noise cleaning at epoch t0 = 5, and set ⌘0 = 0.1 (sym.) or 0.4 (asym.). For CIFAR-100, we activate noise cleaning at epoch t0 = 15, and set ⌘0 = 0.02. We use faiss-gpu (Johnson et al., 2017) for efficient knn search in the low-dimensional subspace, which finishes within 1 second.
Results. Table 1 shows the comparison with existing methods. Our method outperforms previous methods across all label noise settings. On the more challenging CIFAR-100, we achieve 3-4% accuracy improvement compared to the second-best method DivideMix. Moreover, our method is more computational efficient than DivideMix, which needs co-training for noise filtering.
In order to demonstrate the advantage of the proposed low-dimensional embeddings, we perform knearest neighbor (knn) classification (k = 200), by projecting test images into normalized embeddings. Compared to the trained classifier, knn achieves higher accuracy, which verifies the robustness of the learned low-dimensional representations.

heading: 4.2 EXPERIMENTS ON CONTROLLED NOISY LABELS WITH NOISY IMAGES
text: Dataset. We further corrupt a noisy CIFAR-10 dataset (sym. 50%) by injecting two types of input noise: out-of-distribution (OOD) images and input corruption. For OOD noise, we follow Wang et al. (2018) and add 20k images from either one of the two other datasets: CIFAR-100 and SVHN (Netzer et al., 2011), enlarging the training set to 70k. A random CIFAR-10 label is assigned to each OOD image. For input corruption, we follow Hendrycks & Dietterich (2019) and corrupt each image in CIFAR-10 with a noise randomly chosen from the following four types: Fog, Snow, Motion blur and Gaussian noise. Examples of both types of input noise are shown in Figure 4. We follow the same implementation details as the CIFAR-10 experiments described in Section 4.1.
CIFAR-100 Gaussian NoiseFog Snow Motion BlurSVHN
Out-of-distribution Images Input Corruption
Figure 4: Examples of input noise injected to CIFAR-10.
Results. Table 2 shows the results, where our method consistently outperforms existing methods by a substantial margin. We observe that OOD images from a similar domain (CIFAR-100) are more harmful than OOD images from a more different domain (SVHN). This is because noisy images that are closer to the test data distribution are more likely to distort the decision boundary in a way that negatively affects test performance. Nevertheless, performing knn classification using the learned embeddings demonstrates high robustness to input noise.
In Figure 5, we show the t-SNE (Maaten & Hinton, 2008) visualization of the low-dimensional embeddings for all training samples. As training progresses, our model learns to separate OOD samples (represented as gray points) from in-distribution samples, and cluster samples of the same class together despite their noisy labels.

heading: 4.3 EXPERIMENTS ON REAL-WORLD NOISY DATA
text: Dataset and implementation details. We verify our method on two real-word noisy datasets: WebVision (Li et al., 2017) and Clothing1M (Xiao et al., 2015). Webvision contains images crawled from the web using the same concepts from ImageNet ILSVRC12 (Deng et al., 2009). Following previous works (Chen et al., 2019; Li et al., 2020a), we perform experiments on the first 50 classes of the Google image subset. Clothing1M consists of images collected from online shopping websites where labels were generated from surrounding texts. Note that we do not use the additional clean set for training. For both experiments, we use the same model architecture as previous methods. More implementation details are given in the appendix.
Results. We report the results for WebVision in Table 3 and Clothing1M in Table 4, where we achieve state-of-the-art performance on both datasets. Our method achieves competitive performance on WebVision even without performing noise cleaning, which demonstrates the robustness of the learned representation. Appendix D shows examples of noisy images that are cleaned by our method.

heading: 4.4 ABLATION STUDY
text: Effect of the proposed components. In order to study the effect of the proposed components, we remove each of them and report accuracy of the classifier (knn) across four benchmarks. As shown in Table 5, the mixup prototypical contrastive loss (Lpc mix) is most crucial to the model’s performance. The consistency contrastive loss (Lcc) has a stronger effect with corrupted input or larger number of classes. We also experiment with removing mixup and using the standard prototypical contrastive loss, and using standard data augmentation (crop and horizontal flip) instead of AugMix. The proposed method still achieves state-of-the-art result with standard data augmentation.
Effect of bottleneck dimension. We vary the dimensionality of the bottleneck layer, d, and examine the performance change in Table 6. Our model is in general not very sensitive to the change of d.

heading: 5 CONCLUSION
text: This paper proposes noise-robust contrastive learning, a new method to combat noise in training data by learning robust representation. We demonstrate our model’s state-of-the-art performance with extensive experiments on multiple noisy datasets. For future work, we are interested in adapting our method to other domains such as NLP or speech. We would also like to explore the potential of our method for learning transferable representations that could be useful for down-stream tasks.

Review summaries:
The authors of the paper propose to use the contrastive loss, the mixup prototypical loss, and a reconstruction loss to regularize the learned representation in order to achieve robustness under various kinds of noise like label noise, out-of-distribution input, and input corruption. A noise-cleaning process based on the learned representation is also introduced to further enhance the results. Extensive experiments were conducted to demonstrate the effectiveness of the method. 

The paper proposes noise-robust contrastive learning to combat label noise, out-of-distribution input and input corruption simultaneously. In particular, this paper embeds images into low-dimensional representations by training an autoencoder, and regularizes the geometric structure of the representations by contrastive learning. Furthermore, this paper introduces a new noise cleaning method based on the structure of the representations. Training samples with confident pseudo-labels are selected for supervised learning to clean both label noise and out-of-distribution noise. The effectiveness of the proposed method has been evaluated on multiple simulated and real-world noisy datasets.