Learning from Noisy Data with Robust Representation Learning
Authors: Junnan Li
Caiming Xiong
Steven Hoi
Venue:  (2021)
Keywords: label noise
out-of-distribution noise
contrastive learning

Abstract: Learning from noisy data has attracted much attention, where most methods focus on label noise. In this work, we propose a new framework which simultaneously addresses three types of noise commonly seen in real-world data: label noise, out-of-distribution input, and input corruption. In contrast to most existing methods, we combat noise by learning robust representation. Specifically, we embed images into a low-dimensional subspace by training an autoencoder on the deep features. We regularize the geometric structure of the subspace with robust contrastive learning, which includes an unsupervised consistency loss and a supervised mixup prototypical loss. Furthermore, we leverage the structure of the learned subspace for noise cleaning, by aggregating information from neighboring samples. Experiments on multiple benchmarks demonstrate state-of-the-art performance of our method and robustness of the learned representation. Our code will be released.

Under review as a conference paper at ICLR 2021
LEARNING FROM NOISY DATA WITH ROBUST REPRE- SENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Learning from noisy data has attracted much attention, where methods focus on label noise. In this work, we propose a new framework which simultaneously addresses three types of noise commonly seen in real-world data: label noise, out- of-distribution input, and input corruption. In contrast to most existing methods, we combat noise by learning robust representation. Specifically, we embed images into oiw-dimensional subspace by training an autoencoder on the deep features. We regularize the geometric structure of the subspace with robust contrastive learning, which includes unsupervised consistency loss and supervised mixup prototypical loss. Furthermore, we leverage the structure of the learned subspace for noise cleaning, by aggregating information from neighboring samples. Experiments on multiple benchmarks demonstrate statc-of-the of-thh--rttttttt performance of our method and robustness of the learned representation. Our code will be released
INTRODUCTION
Data in real life is noisy. However, deep models with remarkable performance are mostly trained on clean datasets with high-quality human annotations. Manual data cleaning and labeling is an expensive process that is difficult to scale. On the other hand, there exists almost infinite amount of noisy data online. It is crucial that deep neural networks (DNNs) could harvest noisy training data. However, it has been shown that DNNs are susceptible to overfitting to noise (Zang et al. 2017).
As shown in Figure I real-world noisy image dataset often consists of multiple types of noise. Label noise refers to samples that are wrongly labeled as another class (e.g. flower laheled as orange). outtof-distributton input refers to samples that do not belong to any known classes. Input corruption refers to image-level distortion (e.g. low brightness) that causes data shift between training and test.
Most of the methods in literature focus addressing the more detrimental label noise. Two dominant approaches include: (1) find clean samples as those with smaller loss and assign larger weights to them (Han et al. 2018 Yu et al. 2019 Shen & Sanghavi 2019
Arazo et al. 2019:; relabel noisy
samples using model's predictions (Reed et al.] [2033] Ma et al. 2018: Tanaka et al. 2018 Yi & Wu
2019). The recently proposed DivideMix (Li et al. 2020a) integrates both approaches a co-training framework, but it also increases computation cost. Previous methods that focus on addressing label noise do not consider real-world scenarios. Furthermore, using model's 's own prediction to relabel samples could cause confirmation bias, where the prediction error accumulates and harms performance.
di-tiitritttion input or input corruption, which limits their performance in
We propose new direction for effective learning from noisy data. Our method embeds images into noise-robust fow-dimensional representations, and regularizes the geometric structure of the representations with contrastive learning. Specifically, our algorithmic contributions include:
We propose noise-robust contrastive learning, which introduces two contrastive losses. The first is unsupervised consistency contrastive loss. It enforces inputs with perturbations to have similar normalized embeddings, which helps learn robust and discriminative representation.
Our second contrastive loss is reakly-supervisee mixup prototypical loss. We compute class prototypes as normalized mean embeddings, and enforces each sample's embedding to be closer to
CCode in the supplementary material

---

Under review as a conference paper at ICLR 2021
IORANGEL BLACK
RETTO
Label Noise
Out-of-distributioo Input
Input Corruption
Figure 1:
Google search images from WebVision (Li et al. 2017 dataset with keyword "orange".
its class prototype. Inspired by Mixup (Zang et al. 2018), we construct virtual training samples as linear interpolation of inputs, and encourage the same linear relationship wet the class prototypes.
We train a linear autoencoder to reconstruct the high-dimensional features using low-dimensional embeddings. The autoendoer enables the high-dimensional features maximally preserve the robustness of the -indiimnsional embeddings, thus regularizing the classifier.
We propose a new noise cleaning method which exploits the structure of the learned representations. For each sample, we aggregate information from its top-k neighbors to create pseudo-label. A subset of training samples with confident pseudo-labels are selected to compute the weakly- supervised losses. This process can effectively clean both label noise and out- auttof-diitribution (OOD) noise.
Our experimental contributions include:
We experimentally show that our method robust to label noise, OOD input, and input corruption. Experiments are performed on multiple datasets with controlled noise and real-world noise, where our method achieves state-of-the-an performance.
We demonstrate that the proposed noise cleaning method can effectively clean majority of label noise. also learns curriculum that gradually leverages more samples to compute the weakly-supervised losses as the pseudo-labels become more accurate.
We validate the robustness of the learned www-diensional representation by showing (1) k-nearest neighbor classification outperforms the softmax classifier. (2) OOD samples can be separated from in-distribution samples. The efficacy of the proposed autoencoder is also verified.
2
RELATED WORK
Label noise learning. Learning from noisy labels have been extensively studied in the literature. While some methods require access to small set of clean samples (Xiao et al.) 2015 Vahdat 2017
Veit et al. 2017 Lee et al. 2018 Hendrycks et al. 2018), most methods focus on the more challenging scenario where no clean labels are available. These methods can be categorized into two
major types. The first type performs label correction using predictions from the network (Reed et al.
2015 Ma et al. 2018 Tanaka et al. 2018 Yi & Wu 2019). The second type tries to separate clean
samples les from corrupted samples, and trains the model on clean samples (Han et al. [018 Arazo
al. 2019 Tiang et al. 2018 2020 Wang al.
2018 Chen et al. 2019 Lyu & Tsang 2020p. The
recently proposed Dividc.ix (Li et al. 2020a) effectively combines label correction and sample
selection with the Mixup Zhang al. 2018) data augmentation under a co-training framework.
However, it cost 2x the computational resource of our method.
Different from existing methods, our method combats noise by learning noise-robust low-dimensional representations. We propose a more effective noise cleaning method by leveraging the structure of the learned representations. Furthermore, our model is robust not only to label noise, but also to out-of cit-of-distribution and corrupted input. A previous work has studied open-set noisy labels (Wang) et al. 2018), but their method does not enjoy the same level of robustness as ours.
Contrastive learning. Contrastive learning is at the core of recent sulf-supe representation learning methods (Chen et al. 2020 He et al. 2019 Oord et al. 2018 Wu et al. 2018). In self- supervised contrastive learning, two randomly augmented images are generated for each input image. Then a contrastive loss is applied to pull embeddings from the same source image closer, while pushing embeddings from different source images apart. Recently, prototypical contrastive learning (PCL) (Li et al. 2020b) has been proposed, which uses cluster centroids as prototypes, and trains the network by pulling an image embedding closer to its assigned prototypes.
2

---

Under review as a conference paper at ICLR 2021
function
Softmax
Lee
Shared weights
"Walf"
Subspace projection
Weakly- augmented
Interpolated embedding
Class prototype
Low-D
High-D
normalize
CNN
Freeon
Lpe mix
Lee
Strongly-
augmented
Low-D
High-D
normalize
CNN
Arecon
Interpolated Input
0.2x +0.8x
Our proposed framework for noise-robust contrastive learning. We project images into a low- dimensional subspace, and regularize the geometric structure of the subspace with (1)Cce consistency trastive loss which enforces images with perturbations have similar embeddings: (2) a prototypical contrastive loss augmented with mixup, which encourages the embedding for a linearly-interpoke have the same linear relationship w.rl the class prototypes. The low www-diensional embeddings are also trained to reconstruct the high-dimensional features, which preserves the learned information and regularizes the classifier.
Figure 2:
input to
Different from previous methods, our method performs contrastive learning in the principal subspace of the nigh-dimensional feature space, by training linear autoencoder. Furthermore, supervised contrastive loss improves PCL eLi al. (020b) with Mixup (Zhang et al. 2018). Different from the original Mixup where learning happens the classification layer, our learning takes places in the low-dimensional subspace.
METHOD
Given a noisy training dataset D = {x; ). where æ; is an image and y/ E [1....CC C} is its class label. We aim to train a network that is robust to the noise in training data (i.e. label noise, OOD input, input corruption) and achieves high accuracy on clean test set. The proposed network consists of three components: (1) a deep encoder (a convolutional neural network) that encodes an image æ; to high-dimensional feature vy (2) a classifier (a fully-connected layer followed by softmax) that receives v; as input and outputs class predictions; (3) linear autoencoder that projects v; into a low-dimensional embedding E RRd We show an illustration of our method in Figure 2 and pseudo-code in appendix Next, we delineate details.
3.1 CONTRASTIVE LEARNING IN ROBUST LOW-DIMENSIONAL SUBSPACE
Let 2; W. be the linear projection from high- diggnniiensio features to low-dimensional embed- the normalized embeddings. aim to learn robust embeddings with two contrastive losses: unsupervised consistency loss and weakly-supervised mixup prototypical loss.
dings, and 2; = z ||=:|| 2
Unsupervised consistency contrastive loss. Following the NT-Xent (Chen al. 2020) loss for self- supervised representation learning, consistency contrastive loss enforces images with semantic- preserving perturbations to have similar embeddings. Specifically, given a minibatch of b images, we apply weak- auaaeatgmentttioon and strong-augmentation Weak augmentation is a standard flip-and-shiff augmentation strategy, while strong augmentation consists of color and brightness changes with details given in Section
each image, and obtain 2b inputs (æ, )
We project the inputs into the low-dimensional space to obtain their normalized embeddings { Let E b1 be the index of weakly-augmented input, and j() be the index of the strong-
3

---

Under review as a conference paper at ICLR 2021
augmented input from the same source image, the consistency contrastive loss is defined as:
exp(2; 23(0)/T)
Le
log 26 ΣB=1 Lizs exp(2; 2k/T)"
(I)
where т is a scalar temperature parameter. The consistency contrastive loss maximizes the inner product between the pair of positive embeddings &; and &)((), while minimizing the inner product between 2(6 1) pairs of negative embeddings. By mapping different views (augmentations) of the same image to neighboring embeddings, the consistency contrastive loss encourages the network to learn discriminative representation that robust low-level image corruption.
Weakky-superrised mixup prototypical contrastive loss. Our second contrastive loss injects struc- tural knowledge of classes into the embedding space. Let Ie denote indices for the subset of images in D labeled with class c, we calculate the class prototype as the normalized mean embedding:
(2)
=
I
where & is the embedding of center-cropped image, and the class prototypes are calculated at the beginning of each epoch.
The prototypical contrastive loss enforces an image embedding 2; to more similar to its corre- sponding class prototype 20; in contrast to other class prototypes:
exp(2;
2 /T)
Cpe(En U.)
log
(3)
Ec=1 exp(2; /T)T)
Since the label y: is noisy, would like to regularize the encoder from memorizing training labels. Mixup Zhang et al. 2018) has been shown to be an effective method against label noise (Arazo et al. et al. 2020ap. Inspired by it, create virtual training samples by linearly interpolating a sample (indexed by i) with another sample (indexed by m(i)) randomly chosen from the same minibatch:
2019
x = Xx;; + (1 A)
where À ~ Beta(a, a).
Let 2m be the normalized embedding for æt" the mixup version of the prototypical contrastive loss is defined as a weighted combination of the two Lpe w.e. class ys and Ym(()- It enforces the embedding for the interpolated input to have the same linear relationship w.n.t. the class prototypes.
2b Σ
Lpc.iix
(2."
,3) + (1 (2 (mm(s))-
(5)
Reconstruction loss. We also train a linear decoder Wa to reconstruct the high-dimensional feature v; based on 2;. The reconstruction loss is defined as:
2b
L Lrecon
|00
W
(6)
There are several benefits for training the autoencoder. First, with an optimal linear autoencoder, W. will project v; into its low-dimensional principal subspace and can be understood as applying PCA (aldi & Hornik 1989). Thus the low-dimensional representation z; is intrinsically robust to input noise. Second, minimizing the reconstruction error is maximizing a lower bound of the mutual information between v, and (Vincent et al.) 2010). Therefore, knowledge learned from the proposed contrastive losses can be maximally preserved in the high-dimensional representation, which helps regularize the classifier.
Classification loss. Given the softmax output from the classifier, p(y: æz), we define the classification loss the cross-entropy loss. Note that it is only applied to the weakly-augmented inputs.
b
Lee =
log p(y: x.))
(7)
4

---

Under review as a conference paper at ICLR 2021
100
Poculo Label Accuracy (N6)
senervised Naise Ratio (%)
CIFAR
Num. of Sup. Samples (%)
CIFAR-100
90
90-
10
85-
80
70
Engervisen
75
CIFAR-10
60
CIFAR-10
70
CIFAR-100
CIFAR-100
0
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
Epoch
Epoch
Epoch
(c)
Figure 3: Curricutum learned by the proposed label correction method for training CIFAR datascts with 50% sym. noise. (a) Accuracy of pscudo-labels w.z/ clean training labels. (b) Number of samples the weakly-supervised subset Diup. (c) Label noise ratio the weakly-supervisee subset.
The overall training objective is minimize a weighted sum of all losses:
L Lee + Weehee + WpeCpe mix Wrecon Crecon
(8)
For all experiments, we fix Wee
1, Wrezon = 1, and change wpe only across datasets.
3.2 NOISE CLEANING WITH SMOOTH NEIGHBORS
After warming-up the model by training with the noisy labels {y: for to epochs, we aim to clean the noise by generating soft pscudo-label q: for each training sample. Different from previous methods that perform label correction purely using the model's softmax prediction, our method exploits the structure of the low-dimensional subspace by aggregating information from top-k neighboring samples, which helps alleviate the confirmation bias problem.
At the I-th epoch, for each sample æ; let p be the classifier's softmax prediction, let qi- be its soft label from the previous epoch, we calculate the soft label for the current epoch as:
d p
w. 9
2
where wly represents the normalized affinity between sample and its neighbor and is defined as exp(2 2 /T)
w/y = Σ -xp(: We set k 200 in experiments.
The soft label defined by equ. P is the minimizer of the following quadratic loss function:
((gh)
Σ wh | -11 + l/ pill2
(10)
The first term is smoothness constraint which encourages the soft label take similar value its neighbors' labels, whereas the second term attempts to maintain the model's 's class prediction.
We construct weakly-supervise subset which contains (1) clean sample whose soft label score for the original class yi is higher than a threshold foo, (2) pseudo-labeled sample whose maximum soft label score exceeds threshold 7/ For pseudo-labeled samples, we convert their soft labels into hard labels by taking the class with the maximum score.
Diup {a,, Vi (3.)) > Tot (0: Wf arg max (e) max (c) > m, E (I,,,C)) (11)
Given the weakly-supervised subset, we modify the classification loss Lee, the mixup prototypical contrastive loss Lpe mix, and the calculation of prototypes 2", such that they only use samples from Di.pp The unsupervised losses (i.e. Loe and Lrecon) still operate on all training samples.
Learning curriculum. Our iterative noise cleaning method learns an effective training curriculum, which gradually increases the size of Diup the pseudo-label, become more accurate. To demonstrate
5

---

Under review as a conference paper at ICLR 2021
Dataset Noise type
CIFAR-10 Sym 50%
CIFAR- 100
Sym 20%
Asym 40%
Sym 20%
Sym 50%
Cross-Entropy (Li et al.
2020a
82.7
57.9
72.3
61.8
37.3
Forward (Patrini et al. 7017)
83.1
59.4
83.1
61 .4
37.3
Co-teaching+ Yu
2019
88.2
84.1
64.1
45.3
Mixup Zhang at.] 20
92.3
77.6
66.0
46.6
P-correction
Wu
2019
92.0
88.7
88.1
68.1
56.4
MLNT (Li et at] 2019
92.0
88.8
88.6
67.7
58.0
M-correctron Arazo al.
2019
93.8
91.9
86.3
73.4
65.4
DivideMix (Li et al. 2020a)
95.0
91.4
74.8
72.1
DivideMix treproduced)
95.1110.1
93.6:10.2
91.3:10.8
75.1
72.110.3
Ours (classifier)
95.8+=0.1
94.3+0.2
91.9+
79.1+0.1
74.8+t0.4
Ours (knn)
95.9.10.1
94.5.10.1
92.4110.9
79.410.1
75.0.10.4
Table I: Comparison with state-of-the of-thh-art methods on CIFAR datasets with label noise. Numbers indicate average test accuracy (%) over last 10 epochs. We report results over 3 independent runs with randomly- label noise. Results for previous methods copied from Arazo et al. (2019): [L et al. (2020a) We re-run DivideMix (without ensemble) using the publicly available code on the same noisy data as ours.
such curriculum, we analyse the noise cleaning statistics for training our model on CIFAR- 10 and CIFAR-100 100 datasets with 50% label noise (experimental details explained in the next section). In Figure 3 (a), we show the accuracy of the soft pseudo-labels w. to clean training labels (only used for analysis purpose). Our method can significantly reduce the ratio of label noise from 50% to 5% (for CIFAR-10) 10) and 17% (for CIFAR-100 100). Figure (b) shows the size of D
percentage of the total number of training samples, and Figure 3 (c) shows the effective label noise ratio within the
weakly-supervised subset Duup- Our method maintains low noise ratio in the weakly-supervised subset, while gradually increasing its size to utilize more samples for the weakly-supervised losses.
4 EXPERIMENT
In this section, we validate the proposed method on multiple benchmarks with controlled noise and real-world noise. Our method achieves state-of-the-ar performance across all benchmarks. For fair comparison, compare with DivideMix (Li et al., 2020a) without ensemble. In appendix A report the result of our method with co-training and ensemble, which further improves performance.
4.1 EXPERIMENTS ON CONTROLLED NOISY LABELS
Dataset. Following Tanaka et al. (2018) et al. (2000a) we corrupt the training data of CIFAR- 10 and CIFAR- KKrizhessky & Hinton 2009) with two types of label noise: symmetric and asymmetric. Symmetric noise is injected by randomly selecting percentage of samples and changing their labels to random labels. Asymmetric noise is class-dependant, where labels are only changed to similar classes (e.g. dog+>cat, deer->horse), We experiment with multiple noise ratios: sym 20%, sym 50%, and asym 40% (see results for sym 80% and 90% in appendix A). Note that asymmetric noise ratio cannot exceed 50% because certain classes would become theoretically indistinguishable.
Implementation details. Same as previous works (Arazo et al. 2019 Li et al. 2020a). we use PreAct R8 (He al. 2016) our encoder model. We set the dimensionality of the bottleneck layer as d = 50. Our model is trained using SGD with momentum of 0.9, a weight decay of 0.0005, and a batch size of 128. The network is trained for 200 epochs. We set the initial learning rate as 0.02 and use cosine decay schedule. We apply standard crop and horizontal flip as the weak augmentation. For strong augmentation, we use AugMix (Hendrycks et al. (020), though other methods (e.g. SimAug (Chen et al., 2020)) work equally well. For all CIFAR experiments, we fix the hyper-parameters as Wes = 1, wip = 5, Wrezon = I, 0.3, a 8, 7/1 = 0.9. For CIFAR-I0, 10, we activate noise cleaning epoch to 5, and set To 0.1 (sym.) 0.4 (asym.). For CIFAR-100, activate noise cleaning at epoch to = 15. and set 7/o = 0.02. We use faiss-gpu (Johnson et al. 2017) for efficient knn search in the low-dimensional subspace, which finishes within I second.
Results. Table shows the comparison with existing methods. Our method outperforms previous methods across all label noise settings. On the more challenging CIFAR- 100, achieve 3-4% accuracy improvement compared to the second-ber best method DivideMix. Moreover, our method is more computational efficient than Mixi which needs co-training for noise filtering.

---

Under review as a conference paper at ICLR 2021
CIFAR-10
CE
Iterative
GCE
DivideMix
Ours
Ours
50% sym, noise
Wang et al. 2018
Zhang & Sabuncu 2018
Li et al. 2020a
(cls.)
(knn)
CIFAR-100 20k
53.6
87.2
87.3
89.0
91.5
93.1
+ SVHN 20k
58.1
88.6
88.8
91.9
93.3
93.9-+022
+ Image Cormuption
53.8
87.7
87.9
89.8
91.4
91.6-+0.2
Table 2:
Comparison with state-of-the-art methods datasets with lahel noise and input noise. Numbers indicate average test accuracy (%) over last 10 epochs. We report results over independent runs with randomly- generated noise. We re-run previous methods using publicly available code with the same noisy data and model architecture as ours.
In order to demonstrate the advantage of the proposed low-dimensional embeddings, we perform k- nearest neighbor (knn) classification (k 200), by projecting test images into normalized embeddings. Compared the trained classifier, knn achieves higher accuracy, which verifies the robustness the learned low-dimensional representations.
4.2 EXPERIMENTS ON CONTROLLED NOISY LABELS WITH NOISY IMAGES
Dataset. We further corrupt noisy CIFAR-10 10 dataset (sym. 50%) by injecting two types of input noise: out-of-distribution (OOD) images and input corruption. For OOD noise, we follow Wang et al. (2018) and add 20k images from either one of the two other datasets: CIFAR-100 and SVHN Netzer et al. 10 enlarging the training set to 70k. A random CIFAR-I0 10 label is assigned to each OOD image. For input corruption, we follow Hendrycks & Dietterich (019) and corrupt each image in CIFAR-10 with noise randomly chosen from the following four types: Fog, Snow, Motion blur and Gaussian noise. Examples of both types of input noise are shown Figure 4
We follow the same
implementation details as the CIFAR- 10 experiments described in Section
dittof-distributiinn Images
Input Corruption
2
CIFAR-100
SVHN
Fog
Snow
Motion Blur Gaussian Noise
Figure 4: Examples of input noise injected CIFAR-IO.
Results. Table 2 shows the results, where our method consistently outperforms existing methods by a substantial margin. We observe that OOD images from a similar domain (CIFAR-100) are more harmful than OOD images from are closer to the test data distribution are more likely to distort the decision boundary in a way that negatively affects test performance. Nevertheless, performing knn classification using the learned embeddings demonstrates high robustness to input noise.
different domain (SVHN). This is because noisy images that
In Figure show the t-SNE (Maaten & Hinton 2008) visualization of the low-dimensional embeddings for all training samples. As training progresses, our model learns to separate OOD samples (represented as gray points) from in-distribution samples, and cluster samples of the same class together despite their noisy labels.
-CIFAR-100 Epoch 10
+CIFAR-100 Epoch 200
+SVHN Epach 10
+SVHN Epach 200
Figure 5: 1-SNE visualization of low-dimensional embeddings for CIFAR-10 images (color represents the true class) + OOD images (gray points) from CIFAR-100 or SVHN. The model is trained on noisy CIFAR-10 (50k: images with 50% label noise) and 20k OOD images with random labels. Our method effectively learn (1) cluster CIFAR-I0 images according to their class, despite their noisy labels; (2) separate OOD samples from in-distribution samples, such that their harm is reduced.
7

---

Under review as a conference paper at ICLR 2021
Test dataset
WebVision
ILSVRC12
Accuracy (%)
topl
top5
top1
top5
Forward PPatrini et al.
2017
61.1
82.7
57.4
82.4
Decoupling (Malach Shalev Shwartz
2017
62.5
84.7
58.3
82.3
D2L Ma al. 2018))
62.7
84.0
57.8
MentorNet Unang ot al.
2018
63.0
81.4
57.8
79,9
Co-teaching (Han et al.
2018
63.6
85.2
61.5
84.7
INCV (Chen aL. 2019
63.2
85.3
61.0
85.0
DivideMix ct 202
75.9
90.1
73.3
89.2
Ours (w/o noise cleaning)
75.5
90.2
72.0
90.0
Ours (classifier)
76.3
73.3
Ours (knn)
77.8
91.3
74.4
90.9
Table 3: Comparison with state-of-the-arr methods trained on WebVision (mini).
Method
CE
Forward Joint-Opt
MLNT
MentorMix
SL
DivideMix
Ours (cls.)
Ours (knn)
Accuracy 69.21
69.84
72.16
73.47
74.30
74.45
74.48
74.84
74.97
Table 4: Comparison with state-of-the-art methods on Clothing IM dataset.
4.3 EXPERIMENTS ON REAL-WORLD NOISY DATA
Dataset and implementation details. We verify our method on two real-word noisy datasets:
WebVision (i et al., 2017) and ClothingIM (Xiao et al. 2013). Webvision contains images crawled from the web using the same concepts from ImageNet ILSVRC12 (Deng al. 2009). Following previous works (Chen et al., 2019 Li et al. 2020a), we perform experiments on the first 50 classes of the Google image subset. Clothing M consists of images collected from online shopping websites where labels were generated from surrounding texts. Note that we do not use the additional clean set for training. For both experiments, we use the same model architecture previous methods. More implementation details are given in the appendix.
Results. We report the results for WebVision in Table and Clothing IM in Table where achieve state-of-the-ans performance on both datasets. Our method achieves competitive performance on WebVision even without performing noise cleaning, which demonstrates the robustmess of the learned representation. Appendix shows examples of noisy images that are cleaned by our method.
4.4 ABLATION STUDY
Effect of the proposed components. In order to study the effect of the proposed components, we remove each of them and report accuracy of the classifier (knn) across four benchmarks. As shown in Table the mixup prototypical contrastive loss (Cpe mix) is most crucial to the model's performance. The consistency contrastive loss (Lre) has a stronger effect with corrupted input or larger number of classes. We also experiment with removing mixup and using the standard prototypical contrastive loss, and using standard data augmentation (crop and horizontal flip) instead AugMix. The proposed method still achieves state-of-te tet-hhh-rt art result with standard data augmentation.
CIFAR-10 Sym 50%
CIFAR- 100 20k
Image Corruption
CIFAR-100 Sym 50%
w/o Eppemix
85.9 (86.1)
79.7 (81.5)
81.6 (81.7)
65.6 (65.9)
w/o Los
93.7 (93.8)
91.3 (91.5)
89.4 (89.5)
71.9 (71.8)
w/o Lrecon
93.3 (94.0)
90.7 (92.9)
90.2 (91.0)
73.2 (73.9)
w/o mixup
89.5 (89.9)
85.4 (87.0)
84.7 (84.9)
69.3 (69.7)
w/ standard aug.
94.1 (94.3)
90.8 (92.9)
90.5 (90.7)
74.5 (75.0)
DivideMix
93.6
89.0
89.8
72.1
Ours
94.3 (94.5)
91.5 (93.1)
91.4 (91.6)
74.8 (75.0)
Table 5: Effect of the proposed components. We show the accuracy of the classifier (knn) four benchmarks with different noise. Note that DivideMix (Li et 2020a) also performs mixup.
8

---

Under review as a conference paper at ICLR 2021
Effect of bottleneck dimension. We vary the dimensionality of the bottleneck layer, d, and examine the performance change in Table o
Our model in general not very sensitive to the change of d.
bottleneck dimension
d 25
50
100
d 200
CIFAR-I0 Sym 50%
93.4
94.3
94.2
93.7
CIFAR- Sym 50%
73.8
74.8
74.4
73.8
Table 6: Classifier's test accuracy (%) with different low-dimensions.
5 CONCLUSION
This paper proposes noise-robust contrastive learning, new method to combat noise in training data by learning robust representation. We demonstrate our model's ' state-of of of-the-ard art performance with extensive experiments multiple noisy datasets. For future work, method to other domains such as NI.P or speech. We would also like to explore the potential of our method for learning transferable representations that could be useful for down-stream tasks.
are interested in adapting
REFERENCES
Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In JCML, pp. 312-321, 2019.
Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks, 2(1):533555 58, 1989.
Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In ICML, pp. 1062-1070, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. CVPR, pp. 248-255, 2009.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurlPS, pp. 8536-8546, 8546, 2018.
Jiangfan Han, Ping Luo, and Xiaogang Wang. Deep self-learning from noisy labels. ICCV, pp. 5137-5146, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, pp. 630-645, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR. 2019.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data train deep networks on labels corrupted by severe noise. In NewrIPS, pp. 10477-10486, 2018.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. In ICLR, 2020.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks corrupted labels. In ICML, pp. 2318-2318. 2018.
Lu Jiang, Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. In ICML, 2020.

---

Under review as a conference paper at ICLR 2021
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. arXiv preprint arXiv: 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Mater's thesis, University of Toronto, 2009.
Kuang-Huse Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Cleannet: Transfer learning for scalable image classifier training with label noise. In CVPR, pp. 5447-5456, 2018.
Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S. Kankanhalli. Learning to learn from noisy labeled data. CVPR, pp. 5051-5059, 2019.
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In ICLR, 2020a.
Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven C.H. Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04900 2020b.
Wen Li, Limin Wang. Wei Li, Eirikur Agustsson, and Lue Van Gool. Webvision database: Visual learning and understanding from web data. arXiv:/708.02880 2017.
Yueming Lyu and Ivor W. Tsang. Curriculum loss: Robust learning and generalization against label
corruption. In ICLR, 2020.
Xingjun Ma, Yisen Wang, Michael E. Houle, Shuo Zhou, Sarah M. Erfani, Shu-Tao Xia, Sudanthi N. R. Wijewiekrema, and James Bailey. Dimensionality-drive learning with noisy labels. In ICML, pp. 3361-3370, 2018.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2605, 2008
Eran Malach and Shai Shalev-Shwartz. Decoupling "when to update" from "how to update". In NIPS, pp. 960-970, 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS- WS, 2011.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint / 03748, 2018.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, pp. 2241,, 2017.
Scott F. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. In ICLR, 2015.
Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss minimization. In ICMI., pp. 5739-5748, 2019.
Christian Szegedy, Sergey loffe. Vincent Vanhoucke, and Alexander A. Alemi. Inception- v4, inception-resnet and the impact of residual connections on learning. In AAAL, pp. 4278 4284, 2017.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In CVPR, pp. 5552-5560, 2018.
Arash Vahdat. Toward robustness against label noise in training deep discriminative neural networks. In NIPS, pp. 5601-5610, 2017.
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge J. Belongie. Learning from noisy large-scale datasets with minimal supervision. In CVPR, pp. 6575-6583 2017.
10

---

Under review as a conference paper at ICLR 2021
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations a deep network with local denoising criterion. J. Mach. Learn. Res., 11:3371 1-448,,, 2010.
Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. In CVPR, pp. 8688-8696, 2018.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, pp. 3733-3742, 2018.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, pp. 2691-2699, 2015.
Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In CVPR, 2019.
Xingrui Yu, Bo Han, Jiangehao Yao, Gang Niu, Ivor W. Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? ICML, pp. 7164-7173, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Zhilu Zhang and Mert R. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. NeurlPS, pp. 8792-8802, 2018


Reviews:
[{"review_id": "qm79EGHvQ2_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new methodology for noisy-robust learning by augmenting the simlr methodology with a Mixup-style augmentation and noise-cleaning.\n\nThe paper proposes a new methodology for training models in the presence of label noise. The method is a combination of standard SimCLR: using standard image augmentations and enforcing a contrastive consistence loss between the emeddings of the weakly augmented and strongly augmented versions of the same image. In addition, to the standard augmentation, the method also uses a Mixup style augmentation for the strongly augmented images: that is the projection of a randomly selected image is combined (using a convex combination) with the original image and then another contrastive loss is enforced for the class prototyle.\nThe high dimensional embedding of the weakly augmented image is trained with the (noisy) label using a cross-entropy loss.\nIn addition, the high dimensional embedding is also reconstructed from the low-dimensional projection.\n\nThe final evaluation also contains a noise-cleaning step by generating pseudo-labels from the smooth-neighborhoods using the above embeddings.\n\nThe quality of the writing is high and the paper presents a plausible combination of several strong methods. The paper is nicely written, well motivated and can be followed easily.\n\nThe paper also presents significant improvement of artificially noisy versions of CIFAR-10 and CIFAR-100. Also it is also measured on versions on which the images are noised by using extra datasets (SVHN) to create more interesting augmentations of the datasets.\n\nThe weakness of the paper is that it is only measured on cifar-10/100 and these datasest are often not very representative of real noisy datasets with label noise of complicated structure.\n\nAnother weakness is that this paper presents a relatively complex approach composed of 4 different methods, each of them are well studied and with very limited ablation analyses. It is plausible that the combined approach should do well, still it has limited novelty as it is the straightforward combination of four known approaches also the paper does not give a disciplined overview and study of the contributions of the different components.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "cnSpEnOcavL", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors of the paper propose to use the contrastive loss, the mixup prototypical loss, and a reconstruction loss to regularize the learned representation in order to achieve robustness under various kinds of noise like label noise, out-of-distribution input, and input corruption. A noise-cleaning process based on the learned representation is also introduced to further enhance the results. Extensive experiments were conducted to demonstrate the effectiveness of the method. \n\nAll in all, the paper is well written and easy to understand. The proposed method is well justified and seems to be technically sound. I also find the idea of using neighboring samples to perform noise-cleaning intuitive and interesting. A major complaint I have regarding this submission is the lack of novelty. While the empirical results are encouraging, the proposed method is merely a combination of various previously proposed methods. Moreover, the final objective used (Equation 8) also involves 3 hyper-parameters that can be hard to tune. How sensitive are the results when different weights are used? Why is the $w_{pc}$ changed across different datasets? Due to the limitation addressed above, I think the paper a very borderline submission.\n\nQuestions: \n1. It was demonstrated in the appendix that the prototypical loss is the most critical of all the losses used in this paper. However, no ablation study was done with respect to the effect of the mixup on the prototypical loss. How would regular prototypical loss perform without mixup in the context of these experiments?\n2. It was stated that the labels for the pseudo-labeled samples were converted into hard labels. Does this lead to better performance than if soft labels are used? \n----------------------------------------------------------------------\nAfter Rebuttal: \n\nI would like to thank the authors for answering my questions and addressing my concerns on hyperparameters. I also appreciate the authors' efforts in the additional ablation study conducted. While I still do think that the novelty of the paper is a little bit lacking, I think the experiments are carefully conducted and the empirical results seem to be encouraging. As such, I have raised my score to 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "SxFvTUl4uRx", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors have conducted a range of experiments to validate the performance of the proposed method. And according to the results, it is appealing that the proposed method achieves relative improvement compared to current SOTAs . However, there are some concerns as follows.\n\n1) It is quite ad-hoc about the proposed method, which consists of multiple losses and two phases in the training progresses. The potential hyperparameter space is relative black-box to understand where is the bottleneck of learning with noisy labels in this method. It also requires careful hyperparameter-setting as shown in the implementation details.\n\n2) Some contributions are over-claimed since many components have also used in previous works for learning with noisy labels, e.g., the idea of consistency loss and autoencoder in Bootstrapping [1].  Besides, the title of the paper is also misleading since it is about learning with noisy labels instead of contrastive learning.\n\n\n[1] S. Reed et al. Training deep neural networks on noisy labels with bootstrapping. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "HwPVdjHOvMc", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#######################################################################\n\nSummary:\n \nThe paper proposes noise-robust contrastive learning to combat label noise, out-of-distribution input and input corruption simultaneously. In particular, this paper embeds images into low-dimensional representations by training an autoencoder, and regularizes the geometric structure of the representations by contrastive learning. Furthermore, this paper introduces a new noise cleaning method based on the structure of the representations. Training samples with confident pseudo-labels are selected for supervised learning to clean both label noise and out-of-distribution noise. The effectiveness of the proposed method has been evaluated on multiple simulated and real-world noisy datasets. \n\n#######################################################################\n\nReasons for score: \n \nOverall, I vote for a weak acceptance. The proposed noise-robust contrastive learning introduces two contrastive losses: unsupervised consistency loss and supervised mixup prototypical loss. My major concern is about the clarity of the paper and some additional issues (see cons below). Hopefully the authors can address my concern in the rebuttal period. \n \n#######################################################################\n\nPros: \n \n1. The paper takes one import issue in deep learning: learning from noisy data.  \n \n2. For me, the proposed supervised mixup prototypical contrastive loss is novel for learning with noisy data. Specifically, it injects structure knowledge of classes into embedding space by combining the mixup technique and prototypical contrastive loss. The design is reasonable and interesting. \n \n3. This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed framework. In particular, the proposed method outperforms several state-of-the-art robust learning methods in learning with label noise, out-of-distribution input and input corruption. \n\n#######################################################################\n\nCons: \n  \n1. For the motivation, it would be better to provide more details about it, which seems not very clear to me. Particularly, it is unclear why the contrastive loss is duly used in the paper. Will the functionality of the unsupervised contrastive loss be achieved in the supervised prototypical loss? Additionally, the prototypical contrastive loss in equation (1) is an InfoNCE with normalized mean embeddings as the prototypes, which seems different from the ProtoNCE in the original paper [1]. It is better to clarify the the differences of the formulation and training strategy, and the reason of design of supervised prototypical loss in this paper. \n\n[1] Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven C.H. Hoi. Prototypical Contrastive Learning of Unsupervised Representations, In ICLR, 2020. \n\n2. As the key contribution of this paper, mixup prototypical contrastive loss combines mixup technique and prototypical contrastive loss. In the appendix, the authors have provided ablation study to show the effect of proposed losses, and it shows that the mixup prototypical loss is most crucial to the model’s performance. Here, the authors utilizes mixup in two ways: first is to create virtual training samples, and the other is to define the mixup version of prototypical contrastive loss as an weighted combination of two prototypical contrastive loss with respect to true class and virtual class. However, the effect of mixup augmentation, prototypical contrastive loss and mixup prototypical contrastive loss are unclear.  Since mixup has been shown to be an effective method against label noise, it would be more convincing if the authors can study the individual effect of each components of the proposed loss in the rebuttal period. \n \n3. The proposed method uses many data augmentations, e.g., standard crop and horizontal flip as weak augmentation and AugMix as strong augmentation in the unsupervised consistency contrastive loss, and mixup technique in the supervised prototypical contrastive loss. I am concerning about the fairness in the experimental comparison. It is unclear to me if the authors have applied the same data augmentation to all the compared methods. \n \n#######################################################################\n\nQuestions during rebuttal period: \n \nPlease address and clarify the cons above. \n \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a framework to train a discriminative model robust against (i) label noise, (ii) out-of-distribution input, and (iii) input corruption. To tackle these problems, a complex model is proposed that combines several existing models including InfoNCE-style contrastive learning, prototypical contrastive loss, Mixup, and reconstruction loss. Noisy training labels are cleaned using a temporally consistent label smoothing mechanism, combined with a curriculum learning algorithm. 

Originally, the reviewers raised concerns regarding the limited ablation experiments and the lack of studies on real-world noisy labels. The additional experiments in the revised version addressed some of these concerns. Thus, the reviewers increased their rating slightly.

However, the reviewers in the discussion phase agree that the proposed method has a limited novelty, is complex, and involves many moving parts that require a careful design and hyperparameter tuning, and they do not recommend accepting the submission. I agree with the reviewers and recommend rejection. 
Decision: Reject