Robust Learning via Golden Symmetric Loss of (un)Trusted Labels
Authors: Amirmasoud Ghiassi
Robert Birke
Lydia Y. Chen
Venue:  (2021)

Abstract: Learning robust deep models against noisy labels becomes ever critical when today's data is commonly collected from open platforms and subject to adversarial corruption. The information on the label corruption process, i.e., corruption matrix, can greatly enhance the robustness of deep models but still fall behind in combating hard classes. In this paper, we propose to construct a golden symmetric loss (GSL) based on the estimated confusion matrix as to avoid overfitting to noisy labels and learn effectively from hard classes. GSL is the weighted sum of the corrected regular cross entropy and reverse cross entropy. By leveraging a small fraction of trusted clean data, we estimate the corruption matrix and use it to correct the loss as well as to determine the weights of GSL. We theoretically prove the robustness of the proposed loss function in the presence of dirty labels. We provide a heuristics to adaptively tune the loss weights of GSL according to the noise rate and diversity measured from the dataset. We evaluate our proposed golden symmetric loss on both vision and natural language deep models subject to different types of label noise patterns. Empirical results show that GSL can significantly outperform the existing robust training methods on different noise patterns, showing accuracy improvement up to 18% on CIFAR-100 and 1% on real world noisy dataset of Clothing1M. 

Under review as a conference paper at ICLR 2021
ROBUST LEARNING VIA GOLDEN SYMMETRIC Loss OF (UN)TRUSTED LABELS
Anonymous authors Paper under double-blind review
ABSTRACT
Learning robust deep models against noisy labels becomes ever critical when today's data is commonly collected from open platforms and subject to adversarial corruption. The information on the label corruption process, i.c., corruption matrix, can greatly enhance the robustness of deep models but still fall behind in combating hard classes. In this paper, we propose to construct a golden symmetric loss (GSL) based on the estimated confusion matrix to avoid overfitting to noisy labels and learn effectively from hard classes. GSL is the weighted sum of the corrected regular cross entropy and reverse cross entropy. By leveraging a small fraction of trusted clean data, we estimate the corruption matrix and use it to correct the loss as well to determine the weights of GSL. We theoretically prove the robustness of the proposed loss function in the presence of dirty labels. We provide a heuristics to adaptively tune the loss weights of GSL according to the noise rate and diversity measured from the dataset. We evaluate both vision and natural language deep models subject to different types of label noise patterns. Empirical results show that GSL can significantly outperform the existing robust training methods on different noise patterns, showing accuracy improvement up to 18% on CIFAR- 100 and 1% on real world noisy dataset of Clothing M.
proposed golden symmetric loss on
INTRODUCTION
Diverse datasets collected from the public domain which power up deep learning models present new challenges - highly noisy labels. It not only time consuming to collect labels but also difficult to ensure consistent label quality due to various annotation errors (Patrini et al., 2017) and adversarial attacks (Goodfellow et al. 2015). The large capacity of deep learning models enables effective learning from complex datasets but also suffers from overfitting to the noise structure in the dataset. The curse of memorization effect (Jiang et al., 2018) can degrade the accuracy of deep learning models in the presence of highly noisy labels. For example, in (Zhang et al., (217)) the accuracy of AlexNet to classify CIFARIO images drops from 77% to 10%, when there are randomly flipped labels.
Designing learning models that can robustly train on noisy labels is thus imperative. To distill the impact of noisy labels, the related work either filters out suspiciously noisy data, derives robust loss functions or tries to proactively correct labels. Symmetric Cross entropy Loss (SCL) is shown effective in combating label noise especially for hard classes by combing the regular with the reverse cross entropy. The former avoids overfitting and the latter is resilient to label noise. Given its promising results, there is yet to have a clear principle on how to weight the regular and reverse cross entropy terms, e.g., at different noise rates and patterns. In contrast, Distilling (Li et al. 2017) and Golden Loss Correction (GI.C) (Hendrycks al.) 2018 advocate use small clean data to improve the estimated corruption matrix. Specifically, GLC trains the deep model on both a clean and noisy set, whose loss is corrected through the corruption matrix. While the clean set is evenly chosen from all classes, the corrupted labels may appear unevenly across classes depending on the noise pattern (Xiao et al. 2015). As the corrected loss of GLC does not differentiate the difficulty of classes, it may not learn those hard classes effectively.
We propose GSL constructing the golden symmetric loss that dynamically weights regular/reverse cross entropy and corrects the label prediction based the estimated corruption matrix. Similar to

---

Under review as a conference paper at ICLR 2021
Noise
GLC
SCL
GSL
(a) Corruption matrix
(b) Confusion matrix
Figure 1: Noise corruption matrix and confusion matrices of predictions for CIFAR-10 with 60%
symmetric label noise.
GLC, GSL leverages clean data estimate the corruption matrix which is used correct labels and decide the weights of the golden symmetric loss. As such, GSL can effectively differentiate the difficulty level of classes by adjusting the weights and mitigate the impact noise overfitting via the golden symmetric cross entropy. Specifically, tune the weights of modified cross entropy and reverse cross entropy. We prove that modified cross entropy by using confusion matrix is noise tolerant same the reverse cross entropy.
the noise rate and noise diversity to adaptively
Motivation example. We demonstrate the advantages and disadvantages of GLC and SCL, and the their combination (the proposed GSL) through the example of learning convolution networks CIFAR-10 10 injected with 60% symmetric noise. The experimental setup is detailed in 6 Figure [] shows the corruption matrix of the injected noise and the confusion matrices from the predictions of SCL,, GLC, and GSL Even if the injected noise is symmetric across all classes (see Figure Ta). prediction errors are distributed asymmetrically across the classes (see Figure lb). Though GLC can achieve a lower average eтrоr rate than SCL (reflected in darker diagonal elements on average), it performs worse hard classes, e.g., class (cat) and class (dog) (difference blue shades across the diagonal elements). By setting up proper weights for two types of cross entropy, GSL is able to achieve both superior average and per class accuracy.
2 RELATED WORK
Enhancing the robustness of deep models against noisy labels is an active research area. The massive datasets needed to train deep models are commonly found corrupted, (Wang et al. 2018). severely degrading the achievable accuracy, (Zhang et 2017) The impact of label noise on deep neural networks is first characterized by the theoretical testing accuracy over a limited set of noise patterns (Chen et al. 2019). (Vahdat 2017) suggest an undirected graph model for modeling label noise in deep neural networks and indicate symmetric noise to be more challenging than asymmetric. Solutions the prior art can be categorized into three directions: (i) filtering out noisy labels: (Malach) & Shalev-Shwartz, 2017 HHn et al. 2018h Yu et al. 2019 Wang et al. 2018); (ii) correcting noisy labels: (Patrini et al. 2017 Hendrycks et al. 2018 Li et al. 2017); and (iii) deriving noise resilient
loss functions: (Ma al. 20188 Konstantinov Lampert] 2019).
Noise Resilient Loss Function. The loss function is modified to enhance the robustness to label noise by introducing new loss functions, (Ghosh al., [017 Wann al., 2019), adjusting the weights of noisy data instances, (Ren et al. 2018b] Konstantinov & Lampert] 2019 Ma et al. 2018). Mean Absolute Error (MAE) Ghosh et al. 2017 Zhang & Sabuncu 2018) and General Cross Entropy loss Zhang & Sabuncu 2018) are proposed as a noise resilient alternative but at the cost of slow convergence. To avoid overfitting noise, D2I. (Ma al. (008)) uses the subspace dimensionality to assign weights to each data point, whereas Konstantinov (Konstantinov & Lampert 2019) determines the loss weights based on the trustworthiness level of data sources. (Wang et al. (201)) propose symmetric cross-entropy loss that combines new term of reverse cross entropy with traditional cross entropy via constant weights on both terms. Meta- Weight-Net (Shu et al. 2019) re-weights samples during optimizing loss function in the training process by using multi layer perceptron to predict the weight of each sample. With the same perspective, (Ren et al. 200 uses the similarity of samples the clean instances in the validation set for re-weighting them in loss function.
2

---

Under review as a conference paper at ICLR 2021
Label correction. To avoid the data reduction caused by filtering, label correction methods adjust the
predicted/given labels by using only noisy labels (Patrini et al.) 2017 Tanaka et al. 2018) or jointly with small fraction of trusted data (Veit et al.) 2017 Han et al.) 2018a Li et al. 2017: Hendrreyy et at.l. 2018). Reed et al. train the classifier by the "new" labels combining the raw and predicted labels without access to label ground truth. (Patrini et al., (017) estimate the noise confusion matrix by first training a classifier on the noisy labels and then using the softmax probabilities. (Veit et al.) 2017)) acquire numan-verifiee labels to train a cleaning network for correcting noisy labels of multi-label classification problems. (Han et al. 2018a) estimate the noise transition probability by incorporating human assistance. (Li et al.] 2017) and (Hendrycks et al.) 2018 leverage a small set of clean data to estimate noise corruption matrix from the clean and noisy sets, respectively. DivideMix (Li et al.
2020) semi-supervised method, including two networks and Gaussian Mixture Model for sample selection.
The proposed GSL combines resilient loss function and label correction by curating small fraction of trusted data. We solicit subset of informative data instances to estimate the confusion matrix and provide a minimum supervision on noisy labels. We also provide a heuristic to adaptively tune the weights of golden symmetric loss according to the noise characteristies of the dataset.
GOLDEN SYMMETRIC LOsS
Consider the classification problem having dataset D {( Fo bn)) where Hn, E X' C R denotes the th observed sample, and Un E ) := [1., K} the corresponding given label over K classes. Hereon is ignore for the simplicity. is affected by label noise. The label corruption process is characterised by a corruption matrix Ci₃- = P(ū jly i) for = 1,..., K and j = 1, 1,..., K where y is the true label. Synthetic noise patterns are expressed plus a noise label distribution. Let g(-, 0) denote a neural network-bass classifier parameterized by 0. For each data point æ, f(. 0) predicts the probability for each class label k: p(k(æx Σド 67
label corruption probability
where are the logits. Symmetric Cross Entropy. Let ((k(ar) denote the ground truth probability distribution over the K class labels where q(k[z) = 1 for & equal to the true class y and (kk(z)) ) = 0 for all k # y. The cross entropy loss (Lee) and reverse cross entropy loss (Croe) for sample æ are:
lee Σ (klz) logp(k(2)) free = Σ (lgg)))
(Wang al. 2019) combine cross entropy and reverse cross entropy into the symmetric cross entropy:
1at Loe + B Cree.
where a and 33 are hyperparameters. On the one hand cross entropy loss is not robust to noise (Ghosh cet al.) 2017) but achieves good convergence (Zhang & Sabuncu 2018). On the other hand reverse cross entropy is tolerant to noise (Wang et al. 2019).
We estimate the noise corruption matrix in (Hendrycks) et al. 2018). The method fosters training a first classifier g(-, Q) on noisy data to approximate the elements Ct of the noise corruption matrix via a small fraction of trusted data D with known true label y. Practically given A, the subset of trusted data with label of class {A; CD: = i}. the elements of C' can approximated by:
Estimating Noise Corruption Matrix.
C₁2 P(ū = -ly
g j|= () ZEA,
Σ
(3)
[A
where و(ū = = xr θ) denotes predicted probability of æ having class label j. That is Čiş is computed as the mean predicted probability class j for all trusted data points having true label of class i.
Training with Corrected Labels. Let C' be the estimated noise corruption matrix. Using the method in (Patrini et al. 2017), we increase the noise resilience by correcting the predictions of the
To avoid problems with the logarithm, zero values of g are replaced by a small positive value, i.c. 10-4.
3

---

Under review as a conference paper at ICLR 2021
100
100
Test accuracy
Sym: ce only
Test
Sym rce only
Sym: both
Symrio, m1110.1.0.1)
Bimodal: only
#1-16.0.6.0)
Bimodal: rce only
Bimocial (o. 0))(110,0.11
Bimodal: both
Bimodilita, ().(610)..0))
Epoch
Epoch
(a) Different loss corrections (s 70%)
(b) a.S-tuning: best/worst (c 80%)
Figure 2: Impact of loss correction and a,B-tuning on a 2-layer FC network trained on Twitter data.
classifier using C. Let be the conrected predicted probabilities ÖT p. i.c. for data point ((k])z) = Σi Čup((læ) for ,,..., K. We enhance the regular cross entropy term. Applying the prediction correction to both terms holds lower benefits. We evaluate this empirically with extensive experiments on datasets of text, i.e. Twitter in Figure. 2a and images, i.e. CIFARIO and CIFAR 100 in Appendix Experiment details can be found in 6 We consider different datasets, noise rates, noise types and fractions of trusted data. We see that all cases, except one with a difference < 0.3%, correcting only the cross entropy (ce-only) holds better results than correcting only the reverse cross entropy (rce-only) or correcting both. Focusing on Figure. [2a] ce-only improves accuracy by up to 5% and 8% for bimodal and symmetric noise, respectively. In case of CIFAR-10 and CIFAR-100 datasets the improvements are more pronounced with up to 11% and 50% respectively.
Golden Symmetric Loss. Towards more effective and robust learning we propose to leverage the estimated noise corruption matrix C' to tune the two loss terms based on the observed noise pattern. a and 33 can significantly impact the final model accuracy. Tuning these parameters is no mean feat as different datasets affected by different noise patterns benefit from different optimal values (Wang) iet al. 2019). Again we show this behavior by training a 2-layer FC neural network on the Twitter dataset under eleven different (o, ()) combinations and two noise patterns with 80% noise. Figure. 2b reports for each noise pattern the evolution over the training epochs of the test accuracy for the best and worst (a. ()-pair. For bimodal noise even with a small number of trials, the impact of (a, 3) ranges from an accuracy close to 60% all the way down to almost 0%% Moreover only few (two out of eleven) (a, 3)-ppirrs hold accuracy close to 60%. For symmetric noise the tuning impact lower (limited between 70% and 80%) but the best and worst (a, 3)-pair differ from the bimodal noise case. This underlines both the importance and difficulty of tuning (a, 3). Motivated by the high impact of and 8, we propose to dynamically weight the regular and reverse cross entropy terms. Let A() and B() be weighting functions mapping Ć + RR we define new loss function:
LaSt A(C) loe + B(C) froe
(4)
We call this new loss function golden symmetric loss. A() and B() should capture not only the intensity of the noise pattern, but also the diversity of the noise pattern (see Figure. 2b
Determining Weights of Golden Symmetric Loss (A() and B(). In general the more intense and asymmetric the noise pattern, the lower the weight values should be. Since the final loss function learns from both dirty and clean data (see next paragraph), lower values of a and 3 reduce the influence of dirty data over the one of clean data. Hence, we design A() and B() to capture both noise intensity and diversity. The intensity is given by the noise rate E E [o..... 11, i.e. one minus the average of the diagonal elements of Ó. The diversity is measured via Jain's 's faimess index J [x .r, ) - (Ei.. 21) // 2². We choose / because it bounds the diversity on a similar scale as E between 1 (all equal, full symmetry) down to I/n (highest asymmetry). We apply J on all the K(K 1) noise, i.c. off the diagonal, elements of Ć:
K EK Ej= CitY
J K(K - K E C3.
(5)
4

---

Under review as a conference paper at ICLR 2021
First step
Second step
Untrusted
lee (f()) fest Dost(C"((aaa [()),
g(x)) = p00
Trusted data
Estimate confusion matrix
Train network
Figure 3: Training process of GSL divided into two steps.
For symmetric noise J 1, the more asymmetric the smaller J. Final weights proportional J, s. Putting It All Together. As a final step, to maximize the utility of the trusted data, we foster as additional trusted training data for f(). Since D contains the true labels y no prediction correction applied. Hence, the overall loss function for data points from both and D is:
- A(C) (k=1((k(=))))) log p(k(a),
log(E Cup(ilz) - B(C) E llg x 6 D
D.
Figure. summarises visually the training process divided into two main steps: (i) estimating noise corruption matrix through the first network y trained on untrusted dataset D and (ii) training classifier I on both untrusted D and trusted D through the golden symmetric loss.
4 THEORETICAL ANALYSIS
We prove that the cross entropy loss with label correction is noise tolerant under the definition put forth by (Ghosh et al., 2017 Manwani & Sastry] (013)) and extending prior art results. Let the risk of classifier f and loss function Loe under clean labels be R(f) = Ee Lee (f(x) y) and the risk under noise rate be Re)
(e (xx), 0). E indicates the expectation taken over the random variables indicated as its subscripts. With prediction correction via C, the risk becomes ((=
Re(J, C) Eer.jfoe (C" [(x), ),), 77) Let f* and fa be the global minimizers of R(f) and Re),),
respectively, and C* = p(il)) and Ć' be the true and estimated noise confusion matrices, respectively.
Theorem 4.1. In aulliiclaas. classification problem, tce with prediction correction is noise tolerant under symmetric label noise if the noise rate K- SI. where AA ΣK (C f()) k)
K-1
ΣK lee f(x), k). and AR is the difference of risk minimization between optimal classifier and f. And Ene with prediction correction is also noise tolerant under flip noise when noise rate AW,
where En and Eyk are the correct and flipped class probabilities, y S (1+ AW, (1 AWw) respectively.
The proof is based on the risk minimization framework aiming to show under which condition (e" C²) - Re(J, C) < 0, i.e. the loss function is robust to noise. The detailed steps of the
proof can be found in Appendix C The condition < K-1
generalization of the previous
к &R
2017). Without label correction A,4 = 0 which corresponds to the previous result. Label correction improves the robustness by allowing higher noise rates, i.e. with
bound E < K-1 by (Ghosh et al.
K
label correction A44 2 and hence
K- Similar observations hold for flip noise bound.
AR
K
5

---

Under review as a conference paper at ICLR 2021
5 EXPERIMENTAL SETUP
Dataset, Architecture and Parameters, We consider two types of datasets: vision and text analysis. For vision, we use convolution neural networks (CNN) to classify CIFAR-10 and CIFAR-100 with injected label noise and Clothing IM as real world noisy dataset. For text, we use fully connected neural networks classify noisy Twitter and Stanford Sentiment Treebank (SST). In principle, use the same network architecture on all comparative approaches across different noise resilience techniques. In addition, we test the original network from the respective papers too and report the best results among the two.
CIFAR-10 (riizevsky al. 2009). It contains 60K images classified into 10 classes: 50K training set and TOK as validation set. We use the architecture of Wide-ResNet by (Zagoruyko & Komodakis 2016) with depth 28 and a widening factor 10 and train it with SGD with Nesterov momentum and cosine learning rate schedule (Loshchilov & Hutter). For GSL, we first train f for 75 epochs to obtain the noise corruption matrix. Then we train g for 120 epochs.
CIFAR-100 (Krizhevsky et al. 2009). It contains 60K images classified into 100 classes: 50K as training set and 10K as the validation set. We use the same Wide-ResNe. architecture used for CIFAR-10. For GSL,, we train the f and g networks for 75 and 200 epochs, respectively.
ClothingIM (Xiao et al. 2015). This is a real world dataset with label noise. It includes images scrapped from the Interner classified into 14 categories. We resize and crop each image to 224 224 pixels. This dataset contains 47K and 10K images for training and testing, respectively. These two sets have both given (scrapped) and true (human-checked) labels. We use ResNet-50 pretrained with ImageNet and further train for 10 epochs with batch size 32, SGD optimizer, momentum 0.9, weight decay 10-3 and learning rate 10 3 which is divided by 10 after 5 epochs.
Twitter (Gimpel et al. 2011). The Twitter dataset includes 1,827 tweets annotated with 25 POS tags split in 1000 tweets as training set, 327 tweets as development set and 500 tweets as test set. We add development set to training set, and consider it as training set. We use 2-layer fully connected network with 256 hidden neurons each and GELU nonlinearity as activation function. We train with Adam for 15 epochs with batch size 64 and learning rate of 0.001. We train g for 25 epochs. To regularize all linear output layer, we use &g weight decay with À = 5 x 10-5
Stanford Sentiment Treebank (Socher et al. 2013). The SST dataset includes single sentence movie reviews. We use the 2-class version, including 6911 reviews in the training set, develop- ment set with 872 reviews, and 1821 reviews in the test set. We augment the training set by using development set. We learn 100-dimensional word vectors from scratch for a vocab size of 10000. We train a word-averagin model with an affine output layer using Adam optimizer for 5 epochs for network f and 10 epochs for network g. The batch size and learning rate are 50 and 0.001, respectively. To regularize all linear output layer, we use l weight decay with À x 10 10-1
Noise Corruption. We consider symmetric noise and two different asymmetric noises, namely flip and bimodal. Symmetric noise corrupts the true label into a random other labels with equal probability based on the noise rate. The flip noise is generated by flipping the original label to a paired other class with specific probability. The bimodal noise imitates targeted adversarial attacks (Goodfellow et al.) 2015). Specifically, the true labels are corrupted into two neighborhoods centered on two targeted classes, each of which follows truncated normal distribution, A (i, a, a, b). μ specifies the target and a controls the spread. a and b simply define the class label boundaries. For CIFAR-10 10 we target class and for CIFAR-100 class 30 and 70, for Twitter class 6 and 18, and for SST class and 1. Instead, Clothing MM is already affected by real world label noise and left untouched.
6 EVALUATION
In this section, we empirically compare GSL against state of the art noise resilient networks on noisy vision and text data. We aim to show the effectiveness of GSL via testing accuracy on diverse and challenging noise patterns. Our target evaluation metric is the accuracy achieved on the clean testing set, i.e. not affected by noise.

---

Under review as a conference paper at ICLR 2021
Table Vision analysis: test accuracy(%) of real-world noisy Clothing IM, and CIFAR ILFAFRO/CIFARIO corrupted with 30% and 60% noise for different noise resilient networks. Best results in bold.
CIFAR-10
Noise Rate
Noise Pattern
GSL
GLC
SCL
FORWARD
BOOTSTRAP
SGFORWARD
CO-TEACHING+
30%
Sym.
92.75
89.60
83.56
74.06
75.78
91.03
76.53
30%
Bimodal
92.89
89.20
82.83
73.84
75.74
91.24
74.97
30%
Flip
90.66
91.09
81.32
78.55
78.90
90.81
79.75
60%
Sym.
88.94
81.60
73.67
54.37
59.15
88.10
63.84
60%
Bimodal
87.85
84,80
59.84
46.70
47.10
86.62
58.09
60%
Flip
84.97
80.33
56.65
59.45
59.29
82.44
65.08
CIFAR- 100
Noise Rate
Noise Pattern
GSL
GLC
SCL
FORWARD
BOOTSTRAP
SGFORWARD
CO-TEACHING4
30%
Sym.
75.80
61.80
58.66
40.80
43.30
72.63
53.94
30%
Bimodal
76.38
61.40
47.62
45.22
42.46
73.87
55.22
30%
Tlip
75.57
77..23
55.23
34.70
55.008
76.03
58.86
60%
Sym.
68.31
51.30
29.63
19.83
16.90
67.01
38.07
60%
Bimodal
65.74
48.90
30.01
19.57
10.47
63.39
34.09
60%
Flip
69.21
66.96
40.61
38.63
38.14
67.93
40.99
Clothing IM
Noise Real World
GSL
GLC
SCL
FORWARD
BOOTSTRAP 67.87
SGFORWARD 73.96
CO-TEACHING+
74.86
73.91
70.78
70.04
70.33
6.1
VISION ANALYSIS
We compare GSL against four noise resilient networks from the state of the art: GLC (Hendryeks) et al. 2018). SCL (Wang et al. 2019), FORWARD (Patrini et al. 2017). BOOTSTRAP (Reed et al.) and CO-TEACHING+ Yu et al. 2019). In addition, we extend FORWARD by adding reverse cross-entropy based on SCL and loss correction through the confusion matrix same GLC, called SGFORWARD For training GSL, CO-TEACHING+, SGFORWARD and GLC, we use PyTorch v1.4.0. For all other methods, we use Keras v2.2.4 and Tensorflow 11.13.0.W. We assume 10% of trusted data is available for GSI,, GL.C and SGFORWARD. Table summarizes the testing accuracy for all combinations of noise patterns and comparative approaches.
For CIFAR-1O, GSL achieves the highest accuracy among all resilient networks except for flip noise with 30% noise rate. SGFORWARD is the closest rival to GSL because both use the same mechanism the loss function. Besides, GSL has 2 to 8% higher accuracy than GLC, demonstrating the benefit of introducing symmetric cross cross-entropy, especially in high noise rates. In terms of comparison between GSL and SCL, the accuracy difference is even more visible, implying the benefit of using corruption matrix assign weights two terms in symmetric cross-entropy. We note that SCL uses an 8-layer CNN with 6 convolutional layers followed by 2 fully connected layers instead of a Wide ResNet because of the superior results. SCL performs particularly worse in 60% bimodal noise because this is a more challenging pattern and has no access to the corruption matrix. Moreover, our method can still obtain 11 to 30% higher test accuracy than CO-TEACHING+ that uses two deep networks concurrently.
CIFAR-108 more challenging than CIFAR- 10 due to the larger number of classes. GSL achieves the highest accuracy except for flip noise with 30% rate, and SGFORWARD is the second best result among other competitors. Although for flip noise with 30% rate SGFORWARD performs better than GSL, the improvement of GSL is more significant than SGFORWARD compared to the CIFAR-10 dataset. The largest difference (more than 2%) in accuracy between the GSL and SGFORWARD methods is with bimodal noise. In case of 60% symmetric noise, GSL achieves the accuracy of 68%, whereas GLC and SCL trail far behind. Moreover, given the difficulty training a robust classifier for CIFAR-100 100 with 60% label noise, it is worth mentioning that SCL can achieve similar performance as GLC that is given 10% of trusted data in case of 30% symmetric noise. This also indicates the effectiveness of symmetric cross entropy in learning hard classes even without trusted

---

Under review as a conference paper at ICLR 2021
data. However, when facing extremely noisy labels and patterns, the small amount of trusted data greatly improve the robustness of the classifier but not necessarily the symmetric cross entropy.
Seen from the high accuracy compared GLC and SCL, GSL effectively correct symmetric cross entropy loss and improve the learning on the hard classes. GSL performs slightly better with symmetric noise than with bimodal and flip noise that is more challenging for CIFAR-10 10. In the CIFAR-100, GSL works better on the asymmetrie noise rather than symmetric.
the trusted data to
For Clothing M dataset, shown in Table T GSL obtains the highest test accuracy compared to other methods. Same as CIFAR-10 and CIFAR-100, SGFORWARD achieves a relatively good performance. The difference between GSL and SCL comes from the effectiveness of corruption matrix that makes the regular cross entropy robust.
Table Text analysis: average accuracy (%) of variants combining loss correction and symmetric cross entropy. Results averaged across entire range of noise rates [0, 100). Best accuracy in bold.
Noise
Percent Trusted
GSL
GLC
GFORWARD
TMATRIX
SGFORWARD
STMATRIX
Pattern
Sym.
79.27
66.46
53.36
76.69
78.76
78.95
Sym.
5
81.85
77.18
59.47
79.86
81.24
81.30
Twitter
Bimodal
7
75.74
67.05
52.72
77.92
"75.64
76.64
Bimodal
84.14
78.39
60.57
80.87
80.30
80.39
Flip
1
75.52
83.21
39.37
86.21
73.83""
73.30
Flip
80.35
85.81
48.64
86.34
79.79
80.07
Sym.
0.1
74.38
73.21
72.20
73.73
72.20
73.84
Sym.
75.89
72.88
73.40
75.39
72.89
75.12
SST
Bimodal
0.r
74.81
74.80
72.77
74.07
77277
74.06
Bimodal
74.66
74.61
72.07
74.24
71.76
73.90
Flip
0.r
75.31
74.13
29.48
74.88
-49.77
74.95
Flip
76.44
74.59
50.29
76.27
49.81
75.76
6.1.1 TEXT ANALYSIS
We evaluate GSL on text datasets of Twitter and SST, against resilient networks that leverage corruption matrix, namely GLC and FORWARD. Both GSL and GLC use the trusted data for estimating the corruption matrix, wheres the original FORWARD (Patrini et al., 2017) relies solely on the noisy data. As the proposed loss of golden symmetric cross entropy general and can be combined with different resilient networks, we hence use following four variations of loss correction and symmetric
entropy the existing work:
Forward gold (GFORWARD): we replace the estimation of the corruption matrix by the identity matrix on trusted samples and apply loss correction through the confusion matrix.
True corruption matrix (TMATRIX): we directly use the true corruption matrix and apply loss correction through it.
Forward gold with symmetric cross entropy (SGFORWARD): extend the corrected loss of GFORWARD to the corrected symmetric cross entropy as the GSL.
True coruptiion with symmetric entropy and the true corruption matrix instead of the estimated matrix.
entropy (STMATRIX) we apply golden symmetric cross
We extensively evaluate GSL, GLC GFORWARD, TMATRIX, SGFORWARD, and STMATRIX on Twitter and SST, with label corruption ranging from 0% to 100%. We also vary the percentage of trusted data among 1% and 5%. We summarize the average accuracy across noise rates Table 2
Twitter. GSL consistently achieves the highest average accuracy in most cases. Compared to GLC, GSL has significant higher accuracy for Twitter corrupted with symmetric and bimodal noises, but the difference diminishes with increasing amounts of trusted data. When the percent of trusted data is low, say, 1,, GLC unable to estimate the corruption matrix accurately nor to correct the loss, seen by the difference between GLC and TMATRIX.
SST. Here, the classification involves only two classes and turns out to be less challenging than the Twitter case. The difference among the different comparative approaches considered is smaller than
8

---

Under review as a conference paper at ICLR 2021
for Twitter. For instance, though GSL consistently achieves the best average accuracy in almost all cases, the difference between GSL and GLC is around 1-3%. Again, we see that GSL visibly outperforms GLC on low amounts of trusted data because of using entropy and the difference among them becomes limited. We note that TMATRIX and GFORWARD collapse under Flip noise.
7 CONCLUSION
To enhance the robustness deep models against by label noise, we propose GSL that features on correcting the symmetric cross entropy loss by the noise corruption matrix. GSL uses small fraction of trusted data to accurately estimate the corruption matrix, and further determine the weights applied on regular and reverse cross entropy. GSL learns deep networks from trusted samples through regular cross entropy and from untrusted noisy samples through golden symmetric cross entropy. We prove that the cross entropy corrected by the corruption matrix noise robust. To adapt noise patterns of dataset, we heuristically set the weights of golden symmetric loss based on the corruption matrix. We extensively evaluate GSL on vision and text analysis under diversified noise rates and patterns. Evaluation results show that GSL can achieve a remarkable accuracy improvement, i.c., from 2 to 18% on CIFAR benchmarks and real world noisy data, compared to methods that either correct loss or leverage symmetric cross entropy.
REFERENCES
Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In ICML, pp. 1062-1070, 2019.
Aritra Ghosh, Himanshu Kumar, and Sastry. Robust loss functions under label noise for deep neural networks. In AAAI, pp. 1919-1925, 2017.
Kevin Gimpel, Nathan Schneider, Brendan O'Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A Smith. Part-of-speech tagging for twitter: Annotation, features, and experiments. In ACL. pp. 42-47, 2011.
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor W. Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy supervision. In NeurlPs, pp. 5841-5851, 2018a.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurlPS, pp. 8527-8537, 2018b.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. In NeurIPS, pp. 10456-10465 10465, 2018.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks corrupted labels. In ICML, pp. 2309-2318, 2018.
Nikola Konstantinov and Christoph Lampert. Robust learning from untrusted sources. In ICML, volume 97, pp. 3488 3498, 2019.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 10/100 (Canadian Institute for Advanced Research). 2009. URL http::/ww/ www.stooootoooooooooooooo edw. -kriz/ sariz/cifar cifar. htmi
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In ICLR, 2020.
Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In IEEE International Conference on Computer Vision ICCV, pp. 1928-1936, 2017.
Iyy Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In ICLR.

---

Under review as a conference paper at ICLR 2021
Xingjun Ma, Yisen Wang, Michael E. Houle, Shuo Zhou, Sarah M. Erfani, Shu-Tao Xia, Sudanthi N. R. Wijewickrema, and James Bailey. Dimensionality-drien learning with noisy labels. In ICML, pp. 3361-3370, 2018.
Eran Malach and Shai SawarrS Decoupling" when to update" from" how to update". In NeurlPS, pp. 960-970, 2017.
N. Manwani and P. S. Sastry. Noise tolerance under risk minimization. IEEE Transactions on Cybernetics, 43(3): 1e0:1116-11155 2013. doi: 10. ww.1100TTSMBB22012222444
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In IEEE CVPR, pp. 1944-1952, 2017.
Scott E. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. In ICLR 2015, Workshop Track Proceedings.
Mengye Ren, Wenyuan Zeng. Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. ICML, 2018a.
Mengye Ren, Wenyuan Zeng. Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In ICML, pp. 1331-43401 2018b.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meighhe net: Learning explicit mapping for sample weighting. In NIPS, pp. 1919-1930, 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. conference on empirical methods in natural language processing, pp. 1631-1642. 2013.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In JEEE CVPR, pp. 5552-5560, 2018.
Arash Vahdat. Toward robustness against label noise training deep discriminative neural networks. In NeurlPS, pp. 5596-5605, 2017.
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge J. Belongie. Learning from noisy large-scale datasets with minimal supervision. In CVPR, pp. 6575-6583, 2017.
Yisen Wang, Weiyang Liu, Xingjun Ma. James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. In JEEE CVPR, pp. 8688-8696, 2018
Yisen Wang, Xingjun Ma. Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In IEEE ICCV, pp. 322-330, 2019.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In JEEE CVPR, pp. 2691-2699, 2015. 2015.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor W Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? ICML, pp. 7164-7173, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK September 19-22, 2016, 2016.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization.
ICLRR 2017.
Zhilu Zhang and Mert R. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurlPS, pp. 8792-8802, 2018
10

---

Under review as a conference paper at ICLR 2021
Table Accuracy (%) of different gold fraction on CIFAR-10
Noise
30%
Bimodal
Symmetric
Flip
Label correction
5%
10%
15%
5%
10%
15%
5%
10%
15%
ce only
90.06
91.50
92.53
90.17
91.70
92.50
89.96
91.27
92.58
rce only
84.67
88.46
89.30
84.82
88.11
89.98
81.70
86.79
88.40
both
83.25
87.45
89.30
83.43
87.70
89.46
78.00
85.42
88.21
Noise rate 60%
Bimodal
Symmetric
Flip
Label correction
5%
10%
15%
5%
10%
15%
5%
10%
15%
only
83.06
88.43
90.52
85.79
89.19
90.19
80.03
82.64
85.80
rce only
81.73
86.86
89.07
81.54
86.63
88.98
68.29
80.22
84.35
both
79.83
85.79
88.63
80.27
86.22
88.53
63.63
82.19
83.18
Table 4: Accuracy (%) of different gold fraction on CIFAR-100
Noise rate
30%
Bimodal
Symmetric
Flip
Label correction
5%
10%
15%
5%
10%
15%
5%
10%
15%
only
71.66
73.90
75.20
71.69
74.24
75.11
74.78
75.46
77.01
rce only
25.90
61.44
67.15
26.15
61.19
67.14
23.75
59.68
65.33
both
23.37
57.52
64.30
23.58
57.36
63.76
19.74
54.50
61.20
Noise rate
60%
Bimodal
Symmetric
Flip
Label correction
5%
10%
15%
5%
10%
15%
5%
10%
15%
ce only
58.42
66.96
69.41
55.22
66.87
69.46
65.20
68.33
70.41
rce only
54.35
67.24
69.41
24.90
58.72
64.39
14.43
46.09
68.75
both 24.35
55.18
61.19
25.62
55.16
61.46
12.46
36.76
50.83
APPENDIX
A TRAINING WITH CORRECTED LABELS: VISION RESULTS
Here we present the extensive results of our empirical evaluation on training with corrected labels for the vision datasets. This complements the results presented in } We compare the impact of correcting labels only on the cross entropy term (ce only), only on the reverse cross entropy term (rce only), both. Table and Table show the achieved accuracy for CIFAR-I0 and CIFAR- respectively, under two noise rates, 30% and 60%, three different noise types, symmetric, bimodal and flip, and three fractions of trusted data, 5%. 10% and 15%. For each noise scenario the best case highlighted in bold. ce only achieves the highest accuracy in all cases except one. Under 60% bimodal noise on CIFAR- 100 with 10% trusted data rce only is slightly better by 0.28 percent points. More in general, rce only typically performs second best and both achieves the worst accuracy. Focusing on ce only over the other two, the gain tends to increase with the difficulty of the noise scenarios, i.e. with higher number of classes, higher noise rates and less trusted data. only outperforms the other two by up to 11.74 percent points for CIFAR- 10 and up to 51.03 percent points for CIFAR-100.
B TEXT ANALYSIS ON TWITTER AND SST DATASETS WITH VARYING NOISE RATES
Figure shows how the accuracy changes with respect to different noise rates on the Twitter and SST datasets. GSL and GLC are provided with one percent trusted data. contrast, GSL can effectively use the symmetric cross entropy to overcome the limitation of low trusted data. This also explains why STRIX always trails closely behind GSL by using the true confusion matrix and symmetric entropy loss. One may further improve STMATRIX by using the optimal weights of A
11

---

Under review as a conference paper at ICLR 2021
and B according to the true corruption matrix, instead of estimated corruption matrix of GSL. The Twitter dataset highlights well the differences (see Figure/4ar 4ap. The SST dataset is an easier problem with only two classes and all methods are able to perform equally well (see Figure4b).
100
100
80
80
Test accuracy
60
60
6%
GSL
Test
40
GLC
40
grorward
gforward
20
TMatrix
20-
TMatrix
sgForward
sgForwand
sTMabrix
STMatrix
20
60
80
100
20
60
80
100
Noise ratio
Noise ratio
(a) Twitter
(b) SST
Figure 4: Testing accuracy on text datasets with varying noise rates (1% trusted data).
C PROOF OF ROBUSTNESS OF CROSS ENTROPY Loss WITH LABEL. CORRECTION THEOREM 4.1
Theorem 4.1. In a multi-class classification problem, tee with prediction correction is noise tolerant under symmetric label noise if the noise rale K-26
K-1 2x. where AA 1 Lee (C="""( f(x),
EK= lee fxx,)) k). and AR is the difference of risk minimization between optimal classifier and And tee with prediction correction is also noise tolerant under flip noise when noise rate AW
Eyk: $ (1+ AW, E1 + AW,) where Ek and Eyk are the correct and flipped class probabilities, respectively.
Proof. For symmetric noise:
(e, = En.iloo((" f(x), 1) Byll f(x),
- (C f(x), K
kgy
= =E)y(( f(x)))) + E
Loo (CTT) - (6 f(x)))))))
K
= = C)
Σ
lea ( f(x),k)
R(S. R(J,C))
K
k=1
= =((
εK
E
C k()),)))
K
K
(7)
Let A(E f(z),1) ΣK ( f()). Then rewrite 7 Re(f,C (1
R(S,)( + ACT thus:
gK
Re(S", C*) R,(S,C)
(R(S* - -RJ
K
AR
(
+
(A(C f*(x).)) - A(O f(x),y))
K
AA
12

---

Under review as a conference paper at ICLR 2021
where AR < O, because f is the global minimizer for R. and C* the optimal noise confusion matrix.
Similarly, AA & 0 because for the optimal case say A(C** f y) 0. tee with label correction is robust noise when Re( ; Re Ō) s 0. This is true when:
=K
R, Re(f,C)
AAR
AA
K
K-1
=AR
gK
AR
+
AASO
An<o
K-1
sK
8 AA
20=
(9)
K-1
K
-AR
12 8
AA
(K
K-1
AR
K-1
33
K-
AA
AR
With no label correction, C' missing in A and the two terms become equal, i.c. AA = 0. this condition the bound becomes < K- found by (Ghosh et al. 2017) for cross-entropy without label correction. Since AR 2 0, K-1 K- d1
the new bound can be seen as generalization of AR
AA the previous bound. Alt should also be less than one to ensure a meaningful bound on E, avoiding scenarios of noise rate greater than 1.
For asymmetric flip noise, 1 1 E, is the probability of a label being correct (i.e., k = 3), and the noise condition Eyk classified correctly as class y, rather than the probability (Eyk) of being classified incorrectly as class + y.
Ey generally states that sample has higher probability (1 Ey) of being
(e(f.) = (C" = BeEyjaBBjaa.ffee f(z),
= EE.,v(111 - Ey Eoe (C () + Σ Syhhee (C f(x),,))]
kzy
K
-Em.l(1-Ey))
(C f(x),))
Σ
lee (CT) + Exyhfco(C* f(x),, k)]
k=1
kęy
-Eay((1 - Ey)
Σ Loo (C" f()),k) +
Σ (1 By Eyh (()))
(10)
Similar to the symmetric case we require that Rg ( CC*) - - noise:
-Re(, S 0 for the loss to be robust
K
- RR(f. = Ex.g((1 (xy) 2 ((f,'C) -
Loe (C (())) (CT ((x)))
A=1
AW,
Σ (1 En Exk) loe (c f*(),)))) lae laclCC ( f(x),
50
AW2
(1I)
13

---

Under review as a conference paper at ICLR 2021
where AW2 and AW2 I 0 because C is the optimal noise confusion matrix. Rewriting
K
Bgg.y
SyW
Σ
[ynAWa] 50=
køy
Σ (1 JWW, + E( Ey - EYA) 50=
kēy
(12)
AW By AWW S -AW EyW + ER Aw
AW Ey
AWY 2 -1+ + +y
AW
AW
AW E
AW
+ +1 2 Eyk
AW AWk
According to 12 the bound is Eyk I (1 + AWL) - cy + AW.) ). With no label correction
AW,, = and the bound becomes Eyg < - Ey as found by prior art.
14


Reviews:
[{"review_id": "zUKJ2oQwgcR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors of the paper propose a novel loss function termed the golden symmetric loss to tackle the important problem of learning with noisy labels. The proposed loss function, in essence, involves the use of a corruption matrix to correct the regular cross-entropy loss and, at the same time, to estimate the relative weighting of the corrected cross-entropy loss and that of reverse cross-entropy loss.  A series of empirical experiments were conducted to demonstrate the effectiveness of the proposed method. \n\nIn general, the paper is well written and the proposed method is technically sound. Extensive experiments across different domains and applications were also conducted to demonstrate the effectiveness of the proposed method. The idea to use a confusion matrix to automatically tune for the relative weighting between cross-entropy and reverse cross-entropy is also interesting. One significant weakness of the paper is the lack of novelty. To the best of my understanding, the paper is, in most part, a mere combination of two of the previous approaches (loss correction with corruption matrix and the Symmetric Cross-Entropy). As such, it is not surprising that it works better than these previously proposed strategies. In addition, while the idea of automatically determining the relative weighting is interesting, I am not fully clear about how the function $A(\\hat{C})$ and $A(\\hat{C})$ are determined. It was stated that the \"final weights proportional to $J$, $\\epsilon$\". Can you explicitly define the functions  $A(\\hat{C})$ and $A(\\hat{C})$? Does this mean that the exact weighting still needs to be tuned? If so, what is the point of using $A(\\hat{C})$ and $A(\\hat{C})$? Can't we just directly tune the weighting explicitly? I think further clarification and elaboration can be helpful. Due to the reasons mentioned above, I recommend weakly rejecting the paper. \n\nQuestions: \n1. How important is the clean dataset for the algorithm? Since the corruption matrix can be estimated without a clean dataset, it would be interesting to see experiments without a clean dataset. \n2. Why is the corruption matrix applied only to regular cross-entropy but not reverse cross-entropy?", "extra_scores": {}, "extra_text": {}}, {"review_id": "xcGqGNUGlwd", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##### Summary\n\nThe paper proposes a golden symmetric loss method that combines cross entropy loss and reverse cross entropy loss, and at the same time performs forward loss correction for cross entropy loss, under the problem where we have noisy training labels and clean training labels. It shows theoretical results showing the robustness of cross entropy loss under forward loss correction, and shows that it can be regarded as a generalization of Ghosh et al. 2017's analysis. Empirically, it shows how loss correction on reversed cross entropy or the symmetric cross entropy performs worse compared with only cross entropy, and propose to use forward loss correction only on cross entropy term.\n\n##### Strengths \n\nTheoretical results show the robustness of cross entropy loss under forward loss correction and compares with Ghosh et al. 2017's analysis.\n\nExperiments show that the proposed method is better than baselines in vision and text datasets.\n\n##### Weaknesses\n\nThe theoretical analysis gives new insights to the properties of cross entropy loss with forward label correction, but it is only a special case of the proposed loss function in the paper when $B(\\hat{C})$ is 0.\n\nIt would be nice to have the same baselines for vision and text. For example, there is STMatrix in text but not vision, there is SCL in vision but not in text.\n\nI think I am a bit confused to why STMatrix performs worse than GSL in Table 2. Shouldn't STMatrix perform better than GSL if the only difference is whether the corruption matrix is true or estimated? Or does GSL have access to trusted data while STMatrix doesn't? Does STMatrix use the true matrix to derive the balance between the two losses?\n\n##### Comments and suggestions\n\nI'm guessing that a naive method would be to regard the balance between two losses in Eq.4 as an hyperparameter that can be tuned with validation data. How will that perform compared with determining the weight by Jain's fairness index and noise rate? This experiment might help to understand the additional value of the proposed weight determining procedures.\n\nIt looks like all hyper-parameters were fixed and experiments were only conducted once. It would be nice to have at least a few trials and report mean and standard deviation.\n\nTable 2 seems to be an average over different noise rates. Did each noise rate have a similar tendency to the average for the different methods? This is reported for Twitter dataset in Fig.4 in the Appendix but would be good to see the same figure for SST.\n\n\n##### AFTER RESPONSE\n\nThank you for answering. My concern on theory is resolved, but I still think an additional baseline of regarding it as a hyper-parameter would be helpful as a comparison, having at least a few trials with mean and standard deviation for the experiments would make the conclusions stronger, and unifying the baselines is important.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GKVsemf5Irs", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a golden symmetric loss (GSL) for the noisy label learning problems. Compared to previous weighted symmetric cross-entropy losses, the proposed loss estimates the trade-off parameters using the transition matrix. Empirical studies demonstrate that the GSL method is better than some baselines.\n\nComments:\n1. The proposed GLS is simple. My main problem is that why should we estimate $A(\\hat{C})$ and $B(\\hat{C})$ simultaneously? As they are trade-off parameters, estimating one leads to the same results, but the trainable parameters can be reduced. Moreover, it seems that GSL loss only works when the noise data are generated via a transition matrix. Would it be better if we consider the instance-dependent case and relate the trade-off parameters to the feature $x$?\n2. There are some flaws and informal presentations in theoretical proofs.\n- Eq (7) basically follows Proof 2 in (Ghosh et. al. 2017). But, in the first equation, the law of total expectation should be $E_{x}E_{y|x}E_{\\tilde{y}|y,x}$ instead of $E_{x}E_{y|x}E_{y|\\tilde{y},x}$.\n- In the fourth equation, the expection should be equipped to $l_{ce}(\\cdot)$, since there are $f(x)$ in $l_{ce}(\\cdot)$. Hence, it is also desirable to take expection to define $\\mathcal{A}(\\hat{C}^Tf(x),y)$. Otherwise, $\\mathcal{A}(\\cdot)$ depends deeply on the instance $x$. \n- The biggest problem is $\\Delta\\mathcal{R}$ need not be non-positive. The reason is that $f*$ is the optimal solution to $\\arg\\min R(\\cdot,C*)$ instead of $\\arg\\min R(\\cdot,\\cdot)$. The non-positivity requires a detailed discussion, and I think this property does not hold in some conditions. \n- The lower bound of $\\Delta A / \\Delta R$ is not discussed. If $\\Delta A / \\Delta R < 1$, then the algorithm is able to learn from 100% noise data, which is not realizable.\n- What is the definition of noise-tolerant? The authors need to give a mathematical definition.\n- It is weird that Theorem 4.1 merely analyzed the vanilla cross-entropy loss. The authors need to bridge the results of Theorem 4.1 to GLS loss.\n\nMinor comments:\n1. The references are informal. Please put the author's names into the brackets.\n2. The experimental results seem to be far away from state-of-the-art noisy label learning methods, such as DivideMix [1]. While the SOTA performance is not the most essential issue for me to judge a paper, I highly recommend the authors to explore better model architectures that provide competitive results. \n3. The related works are not thorough for me. Some state-of-the-art noisy label learning methods (in 2019/2020), e.g. DivideMix, are not discussed. \n\n[1] Li J, Socher R, Hoi S C H. DivideMix: Learning with Noisy Labels as Semi-supervised Learning[C]//International Conference on Learning Representations. 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "FApfIBcaDeb", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "###############\n\nSummary:\n\nThis paper proposes a robust loss function that combines both cross-entropy loss and reverse cross-entropy loss.\n\n\n###############\n\nComments:\n\n1.\tThe authors are motivated by observing that GLC cannot handle hard classes effectively. However, the why GSL can handle that is not clear to me. Is that because GSL loss function contains the reverse cross-entropy loss?\n2.\tThe contribution of this work is also not quite clear to me. As mentioned in Section 3, GSL borrows the idea from Symmetric Cross-Entropy loss from Wang et al (2019). Hence the major contribution of this work seems only proposing weights depending on the Noise Transition Matrix on the cross-entropy and reverse cross-entropy loss functions.\n3.\tFigure 1 is somehow hard to read. Figure 1 a) seems wrong to me. As the corruption matrix 1a) illustrates that there is no corruption. It could be better if there are more explanations for them.\n4.    A() and B() -- \" Final weights proportional to \\epsilon and J\". It is still not clear to me how to compute A and B, and why they have to be proportional to  \\epsilon and J. In addition, it is not clear to me how to obtain the value of \\epsilon. It is not clear to me when to estimate weights during the training process in Figure 3. \n5.    The proof on page 12 seems wrong to me. To see this, the first expectation in the first line of the proof is taken over x and \\tilde{y}, while the second expectation is conditional on \\tilde{y}. In fact, we have p(x,\\tilde{y}) = \\sum_y p(\\tilde{y}|y)p(y|x)p(x) (see Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach, Section 4, 2017 CVPR).\n6.    Still in the proof, it is not clear to me why R_\\epsilon (f^*-C^*) - R_\\epsilon (f-\\hat{C}) \\le 0 indicates that \\ell_ce with label\ncorrection is robust to noise.\n\n\n###############\n\nAdditional questions: \n\n1.\t“\\alpha and \\beta can significantly impact the final model accuracy”. I agree but the figure 2b) is somehow confusing to me. The values of \\alpha and \\beta for the orange curve are 60 times as large as those for the blue curves. And the orange curve is better. However, the corresponding optimization problems are the same, since the values of \\alpha and \\beta can be absorbed into the learning rate. If we use the blue curve setting and change the learning rate to 60 times as large as before, the results could be the same. The whole improvement could be resulting from changing the learning rate.\n2.\tThe clean data only used to estimate the Noise Transition Matrix proposed by GLC method (Hendrycks et al. (2018)). Then they are added to the final loss as the clean cross-entropy loss. I was wondering if there are any other benefits of clean data that we can leverage.\n\n###############\n\nThis paper proposes a noise-robust loss function where the contribution is leveraging the Noise Transition Matrix to estimate the weights on cross-entropy loss and reverse cross-entropy loss. It is interesting to introduce the reverse cross-entropy loss. But the novelty of this work does not meet the standard for an ICLR publication.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies robust learning in the presence of noisy labels and proposes a new loss function called the golden symmetric loss (GSL) combining both regular cross-entropy and reverse cross entropy and leveraging an estimate of the corruption matrix. The paper appears to be well-written.

Pros:
- Good range of application domains (both vision and text).
- Learning with noisy labels is an important practical problem.
- Theoretical guarantees for the procedure using framework from recent work.

Cons:
- Limited novelty as the method appears to be a weighted combination of two existing ideas. 
- Concerns about the baselines used: why are the same baselines not being used throughout?
- Having at least a few trials with mean and standard deviation for the experiments would make the conclusions stronger.

Overall, the limited novelty combined with the concerns about the empirical analysis was a key reason for rejection. 
Decision: Reject