Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels
Authors: Erik Englesson
Hossein Azizpour
Venue: NeurIPS 2021 Poster (2021)
Keywords: noisy labels
label noise
robust loss functions
Jensen-Shannon divergence

Abstract: Prior works have found it beneficial to combine provably noise-robust loss functions e.g., mean absolute error (MAE) with standard categorical loss function e.g. cross entropy (CE) to improve their learnability. Here, we propose to use Jensen-Shannon divergence as a noise-robust loss function and show that it interestingly interpolate between CE and MAE with a controllable mixing parameter. Furthermore, we make a crucial observation that CE exhibit lower consistency around noisy data points. Based on this observation, we adopt a generalized version of the Jensen-Shannon divergence for multiple distributions to encourage consistency around data points. Using this loss function, we show state-of-the-art results on both synthetic (CIFAR), and real-world (e.g., WebVision) noise with varying noise rates.

Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels
Erik Englesson KTH
Hossein Azizpour KTH Stockholm, Sweden azizpour
Stockholm, Sweden engless@kth.e.
Abstract
Prior works have found beneficial to combine provably noise-robust loss functions e.g., mean absolute error (MAE) with standard categorical loss function e.g. cross entropy (CE) to improve their Iearnability. Here, we propose to use Sansen-Shannon divergence as noise-robust loss function and show that it interestingly interpolate between CE and MAE with a controllable mixing parameter. Furthermore, we make crucial observation that CE exhibits lower consistency around noisy data points. Based on this observation, we adopt a generalized version the Jensen- Shannon divergence for multiple distributions to encourage consistency around data points. Using this loss function, we show state-of-the-ard results on both synthetic (CIFAR), and real-world (e.g. Web noise with varying noise rates.
Introduction
Labeled datasets, even the systematically annotated ones, contain noisy labels [. Therefore,
designing noise-robust learning algorithms crucial for the real-world tasks. An important avenue to tackle noisy labels to devise robse-robust loss functions [2 5) Similarly, in this work, we propose two noise-robust loss functions based two central observations as follows.
Observation I: Provably-robust loss functions can underfit the training data [2] Observation II: Standard networks show low consistency around noisy data points see Figure
45).
We first propose use Sensen-Shanmer divergence (JS) as a loss function, which we erucially show interpolates between the noise-robust mean absolute eттor (MAE) and the cross entropy (CE) that better fits the data through faster convergence. Figure illustrates the CE-MAE interpolation.
Regarding Observation II, we adopt the generalized version of Tonsen-Shannor divergence (GIS) to encourage predictions on perturbed inputs to be consistent, see Figure Notably, Sensen-Shannon divergence has previously shown promise for test-time robustness to domain shift [6], here we further argue for its training-tine robustness to label noise. The key contributions of this work are:
We make a novel observation that a network predictions' consistency is reduced for noisy- labeled data when overfitting noise, which motivates the use of consistency regularization.
We propose using Sensen-Shannor divergence (JS) and its multi-distribution generaliza- tion (GJS) as loss functions for learning with noisy labels. We relate JS to loss functions that are based on the noise-robustness theory of Ghosh et al. (2] In particular, we prove that JS generalizes CE and MAE. Furthermore, we prove that GJS generalizes JS by incorporating consistency regularization in single principled loss function.
We provide an extensive set of empirical evidences on several datasets, noise types and rates. They show state-of. of-te-----h-rrr results and give in-depth studies of the proposed losses.
we call network consistent around sample (œ) if predicts the class for and its perturbations (æ).
iimplementatinn available at https: htaps:://githute com/ Er
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Validation Accuracy(13
1.00
100
1000
80
Consisterccy(
-00
Consistencyy
80
60
60
60
40
07%
40%
175
40%
20%
52%
100
200
300
400
1000
200
500
400
100
200
300
400
Epochs
Epochs
Epochs
(a) Validation Accuracy
(b) Consistency Clean
(c) Consistency Noisy
Figure I: Evolution a trained network's consistency as overfits noise using CE loss. Here plot the evolution of the validation accuracy (a) and network's consistency (as measured by GIS) on clean (b) and noisy (c) examples of the training set of CIFAR- 100 for varying symmetric noise rates when learning with the cross-entropy loss. The consistency of the learnt function and the accuracy closely correlate. This suggests that enforcing consistency may help avoid fitting to noise. Furthermore, the consistency is degraded more significantly for the noisy data points.
2 Generalized Jensen-Shannon Divergence
We propose two loss functions, the Jensen-Shannoe divergence (JS) and its multi-distribution generalization (GJS), In this section, we first provide background and two observations that motivate our proposed loss functions. This is followed by definition of the losses, and then we show that JS generalizes CE and MAE similarly to other robust loss functions. Finally, we show how GJS generalizes JS to incorporate consistency regularization into a single principled loss function. We provide proofs all theorems, propositions, and remarks this section in Appendix
2.1 Background & Motivation
Supervised Classification. Assume a general function class F where each f IF maps an input x E to the probability simplex A-1, i.e. to a categorical distribution over classes y 21, K}. We seek f E that minimizes a risk Rc(f) Ep Ep[[(e(w), fx))) for some loss function L and joint distribution D over X x Y. where ( is K-vector with one index and zero elsewhere. In practice, D is unknown and, instead, we use S = { 8:) which are independently sampled from D to minimize empirical risk N Σx=1 L(e(v)) ((e())) ((::))
Learning with Noisy Labels. In this work, the goal to learn from noisy training distribution D, where the labels are changed, with probability y) from their true distribution D. The noise called instance-dependent label, and symmetric if it is independent of both and Let fú be the optimizer of the noisy distribution risk (f). A loss function L is then called robust if fū also minimizes Rg. The MAE loss (CmAr(e(w), f())) = (fe(v) f(x)]]
depends on the input, asymmetric if it dependents on the true
robust but not CE 22]
Issue of Underfitting. Several works propose such robust loss functions and demonstrate their efficacy in preventing noise fitting 2 3 5). However, all those works have observed slow
convergence of such robust loss functions leading to underfitting. This can be contrasted with CE that has fast convergence but overfits to noise. Ghosh er al. [2] mentions slow convergence of MAE and GCE (3) extensively analyzes the undefitting thereof. SCE [4] reports similar problems for the reverse cross entropy and proposes a linear combination with CE. Finally, Ma et observe the same problem and consider combination of "active" and "passive" loss functions.
Consistency Regularization. This encourages a network to have consistent predictions for different perturbations of the same image, which has mainly been used for semi-supervised learning 77].
Motivation. In Figure I we show the validation accuracy and measure of consistency during training with the CE loss for varying amounts of noise. First, we note that training with CE loss eventually overfits to noisy labels. Figure Ia indicates that the higher the noise rate, the more accuracy drop when starts to overfit to noise. Figure shows the consistency of predictions for correct and noisy labeled examples of the training set, with the consistency measured as the ratio of examples that have the same class prediction for two perturbations of the same image, see
e.g. softmax neural network classifiers in this work

---

B. 0:01
5.3
Loce volue
m.
15. m- 0.592099
NAE
e
* p
e/il
y
* p
8.0
Probability given
2.08
0.0
1.15
Figure 2: JS loss generalizes CE and MAE. The Censen-Shannow loss (Cas) for different values of the hyperparameter 71. The JS loss interpolates between CE and MAE. For low values of T1, Las behaves like CE and for increasing values of wi it behaves more like the noise robust MAE loss.
Figure 3: GJS Dissection for M=K=3: The decomposition of CGJS (left) into JS term (middle) and a consistency term (right) from Proposition 2 Each point in the simplex correspond to pl)) A?, where the color represents the value of the loss that point. It can be seen that there are two ways minimize LGas. either by making the predictions similar to the label (middle) or similar to the other predic- tions (right) to increase consistency. To better highlight the variations of the losses, each loss has its own range values.
Appendix B.6 for more details. A clear correlation is observed between the accuracy and consistency of the noisy examples. This suggests that maximizing consistency of predictions may improve the robustness to noise. Next, we define simple loss functions that (i) encourage consistency around data points and (ii) alleviate the "issue of underfitting" by interpolating between CE and MAF.
2.2 Definitions
DJs. Let ( 2(2) E A K- have corresponding weights π [T T22 T E Then, the Jensen- Shannon divergence between p(1) and p(2)
Dss (p . ()) (m)) (())))))))))))))))))))) TT2 T22((2(2) DxI ([)) ( (1) with И the Shannon entropy, and m = x1
(1) + T2pp?) Unlike Bellback-Leiblee diver- gence (DKL (p() (p(())) or cross entropy (CE), is symmetric, bounded, does not require absolute continuity, and has crucial weighting mechanism (T)) as we will see later.
DGJS. Similar to DKL Dys satisfies DJs (p("),(()) 2 0, with equality iff ( = pp)), For Das+ this is derived from Jensen's inequality for the concave Shannon entropy. This property holds for finite number of distributions and motivates a generalization of Das to multiple distributions [8]
M
M
DGJS, (L) ((., = H E
Σ
Ti H
(1)
Σ
क. DKL (p())
Σ
kTpp()) (2)
T 1=1
where M is the number of distributions, and [ E 1-1
Loss functions. We aim to Das and DGJS divergences, to measure deviation of the predictive distribution(s), f(x), from the target distribution, e(#), Without loss of generality, hereafter, dedi- cate ()) to denote the target distribution. JS loss, therefore, take the form of Das, ( f(x)).
Generalized JS loss is less straight- construction since DGJS can accommodate more predictive distributions. While various choices can be made for these distributions, in this work, we consider predictions associated with different random perturbations of a sample, denoted by A(x). This choice, shown later, implies interesting analogy to consistency regularization. The choice, also entails no distinction between the M 1 predictive distributions. Therefore, we consider Tы - in all our experiments. Finally, we scale the loss functions by constant factor Z = - -1 क log(1 (1- T1). As will see later, the role of this scaling is merely strengthen the already existing and desirable behaviors of these losses as π approaches zero and one. Formally, we have and GJS losses:
22
Das. (e(v), f()))
Leas(V. f, DGJS, (e(v), f(((()) f(E (((()))
Lis 1 x))
(3)
Z
Z
with (() A(x). Next, we study the connection between JS and losses which are based on robustness theory of Ghosh et al. (2].

---

2.3 JS's Connection to Robust Losses
Cross Entropy (CE) is the prevalent loss function for deep classifiers with remarkable successes. However, CE is prone to fitting noise 19]. On the other hand. Mean Absolute Error (MAE) is theoretically noise-robust (2). Evidently. standard optimization algorithms struggle minimize MAE, especially for more challenging datasets e.g. CIFAR-100 3 S.
Therefore, there have been several proposals that combine CE and MAF, such as Generalized CE (GCE) [.. Symmetric CE (SCE) [4) and Normalized CE (NCE+MAE) (5) The rationale is for CE to help with the learning dynamies of MAE. Next, we show JS has CE and MAE as asymptotes w.r.t. T.
Proposition 1. Letp E A K then
lim Las(e( .p) H(elW),)) X1-U
lim Cus(e( pp= > -pl11 m1
where H(elw), P)) is the cross entropy of e/v) relative to p.
Figure 2 depicts how JS interpolates between CE and MAE for क E (0. The proposition reveals an interesting connection state-of-the-art robust loss functions, however, there important differences. SCE is not bounded (so it cannot be used in Theorem T). and GCE is not symmetrie, while JS and MAE are both symmetric and bounded. In Appendix B.3] we perform dissection to better understand how these properties affect learning with noisy labels. GCE is most similar to JS and is compared further in Appendix B.4
crucial difference to these other losses is that JS naturally extends to multiple predictive distribu- tions (GJS). Next, we show how GJJS generalizes JS by incorporating consistency regularization.
2.4 GJS's Connection to Consistency Regularization
In Figure D, it was shown how the consistency of the noisy labeled examples was reduced when the network overfitted to noise. The following proposition shows how GJS naturally encourages consistency in single principled loss function.
Proposition 2. Let (2), ....,..... (M) E EΛK-1 with M 2 and P>1
EVe T,p&9) 1-=1
Ihen
La.s(e (v) (2) , (2)) = LIs (elv) P>1) + (1 - π1) L ( (M0)
where n = [ T1)² and n" [
(1-#1)
Importantly, Proposition 2 shows that GJS can be decomposed into two terms: 1) JS term between the label and the mean prediction p>1, and 2) GJS term, but without the label. Figure illustrates the effect of this decomposition. The first term, similarly to the standard JS loss, encourages the predictions' mean be closer the label (Figure3 However, the second term encourages all predictions to be similar, that is, consistency regularization (Figure 3 right).
2.5 Noise Robustness
Here, the robustness properties of JS and GJS are analyzed in terms of lower (BL) and upper bounds (Bτ) for the following theorem, which generalizes the results by Zhang et al. ( any bounded loss function, even with multiple predictive distributions.
Theorem 1. Under symmetric noise with v < K-1 if BL S EK ((el0), f) S By, Vo, is satisfied for a loss L, then
K
0 S RE -R (2 Sn BU Bt K
(BB K K-1-nk <
and
S 0,
A tighter bound By- - BL, implies a smaller worst case risk difference of the optimal classifiers (robust when B, = B2). Importantly, while ((e(*), x, = L)) f(x) usually, this subtle distinction is useful for losses with multiple predictive distributions,
Equation 3 In Theorem 2 in Appendix we further prove the robustness of the proposed losses to asymmetric noise.
C.3
For losses with multiple predictive distributions, the bounds in Theorem T and must hold for any a and f, i.e., for any combination of M categorical distributions on K classes. Proposition provides such bounds for GJS.

---

Proposition 3. GJS loss with M S K+1 satisfies B,
S Cous(e p(.))) s Bu
.?)..,,,
for all pl2) p plM) AK-1, with the following bounds
K
K
B1 = Lous(e!h), .u....,u) u)
Bu =
Lais(e ee0) .(..,,((M-1))
where u E A-1 is the uniform distribution. Note the bounds for the JS loss is special case of Proposition 3 for M = 2.
Remark 1. Las and CGJs are robust (B2= = By) in the limit of πΙ +
Remark iis intuitive from Section 2.3 which showed that Lis is equivalent to the robust MAE in this limit and that the consistency term in Proposition 2 vanishes.
In Proposition 3 the lower bound (B₁,) is the same for JS and GJS. However, the upper bound (Brr increases for more distributions, which makes JS have a tighter bound than GJS in Theorem [] and 2 In Proposition
4 we show that JS and GJS have the same bound for the risk difference, given an
assumption based on Figure that the optimal classifier on clean data (f*) is at least as consistent as the optimal classifier on noisy data (fr))
Proposition 4. Las and LGJS have the same risk bounds in Theorem 7 and if Ex L'Jss... (p!²),..., ,..... ...)) - ExIcais (pl2),
,...,, ....,, (....,(M))) where Cess
MM) ) is
the consistency term from Proposition 2
3 Related Works
Interleaved the previous sections, we covered most-related works to us, Le the avenue of identifica- These works, similar to this paper, follow the theoretical construction Ghosh et al. [2]. Furthermore, Liu&Guo [10] use "peer prediction" to propose new family of robust loss functions. Different to these works, here, we propose loss functions based on DJs which holds various desirable properties of those prior works while exhibiting novel ties to consistency regularization; a recent important regularization technique.
tion construction of theoretically-motiittee robust loss functions 2 3
Next, we briefly cover other lines of work. A more thorough version can be found in Appendix D
A direction, that similar to us does not alter training, reweights a loss function by confusion matrix
|||
12 13 14 13). Assuming class-conditional noise model, loss correction is theoretically motivated and perfectly orthogonal to noise-robust losses.
Consistency regularization is a recent technique that imposes smoothness in the learnt function for semi-supervised learning 7 and recently for noisy data [16]. These works use different complex pipelines for such regularization. GJS encourages consistency in simple way that exhibits other desirable properties for learning with noisy labels. Importantly, bensen-Shannon-bar loss functions have been used improve test-time robustness image corruptions (6] and adversarial examples [17], which further verifies the general usefulness of GJS. In this work, we study such loss functions for a different goal: training-tinm -ime label-noise robustness. this context, our thorough analytical and empirical results are, to the best of our knowledge, novel.
consistency
Recently, loss functions with information-lee -heorretic motivations have been proposed [8] 19. JS, with an apparent iitommttion-thhoree interpretation, has a strong connection to those. Especially, the latter a close concurrent work studying JS and other divergences from the family of f-divergences (20). However, in this work, we consider a generalization to more than two distributions and study the role of TT which they treat constant 1 = = These differences lead to improved performance and novel theoretical results, e.g., Proposition [ [ Lastly, another generalization of JS was recently presented by Nielsen (21), where the arithmetic mean is generalized to abstract means.
Experiments
This section, first, empirically investigates the effectiveness of the proposed losses for learning with noisy labels on synthetic (Section 4.1 and real-world noise (Section (.2). This is followed by several experiments and ablation studies (Section 4.3) to shed light on the properties of JS and GJS through empirical substantiation of the theories and claims provided in Section 2 All these additional experiments are done the more challenging CIFAR- 100 dataset.
5

---

Table 1: Synthetic Noise Benchmark CIFAR. We reimplement other noise-robust loss functions into the same learning setup and ResNet-34, including label smoothing (LS), Bootstrap (BS), Symmetric CE (SCE), Generalized CE (GCE), and Normalized CE (NCE+RCE). We used same hyperpurameler oplimization budgel and mechanism for all the prior works and ours. Mean test accuracy and standard deviation are reported from five runs and the statisticallysiinnifiaat top performers are boldfaced. The thorough analysis is evident from the higher performance of CE in our setup compared to prior works. GJS achieves tate-of-thee-as art results for different noise rates, types, and datasets. Generally, GJS's efficacy is more evident for the more challenging CIFAR- 100 dataset.
No Noise
Symmetic Noise Rate
Asymmetric Noise
Dataset
Method
40%
20%
CE
95.77 +: 0.11
91,63 +: 0.27
$7.74
0.46
81.99 :: 0.56
66.51 8: 1.49
92.77 :: 0.24
87.12 :: 1.21
BS
94.58
0.25
91.68 1: 0.33
89.23
0.16
82.65 +: 0.57
16.97
6.36
93.06 = 0.25
83.87 : 1.06
L.S
95.64
99.51 + 0.20
89.90
83.96 + 0:58
67.35
2.71
92,94 0.17
88.10 0.50
CIEAR-10
SCE
95.75
0.16
94.29 + 0.34
92.72 0.25
89.26 + 0.37
80.68
0.42
93.48 +: 0.31
$4.98
+ 0.26
GCE
95.75 + 0.14
94.24 +: 0.18
92.82 0.11
89:37 + 0.27
79.19
2.04
92.83 - 0.36
87.00 :: 0.99
NCE-RCE
0.18
92.03
71.89
0.84
IS
95,89 1.10
94,52 +: 0.21
93.01
0.22
89.64 + 0.15
76.06
0.85
92.18 0.31
87.99 + 0.55
GIS
95.91 + 0.89
95.33 +: 0.18
93.57 0.16
91.64 + 0.22
79.11 #b 0.31
93.94 = 0.25
89.65 +: 0.37
CE
77.60 0.17
65.74 4: 0.22
55.77 0.83
44.42 + 0.84
10.74 4.08
66.85 0.32
49.45 4: 0.37
BS
77.65 1 0.29
72.92 +: 0.50
68.52
0,54
53.50 + 1.76
13.83
1: 4.41
73.79 = 0.43
64.67 1: 0.69
1.5
78.00
74.88 + 0.15
68.41 0.20
54.58 + 0.47
26.98 1.07
73.17 = 0.46
$7.20 + 0.85
CIFAR-100 100
SCE
76:20 0.74
0.37
68.33
0:58
26
$1.12 0.37
GCE
77.66 + 0.17
75,02 + 0.34
71.54 0.39
65.21 + 0.16
49.68 0.84
72.13 = 0.39
51.50 + 0.71
NCE-RCE
74,66 0.21
72.39
1: 0.24
68.79
0.29
62.18 +: 0.35
31.63 1: 3.59
71.35 4: 0.16
57.50 4: 0.52
77.95
0.28
71.12
0.34
45.05
71.70
0.25
GJS
79.27 0.29
78.05 1: 0.25
75.71
0.25
70.15 1: 0.30
$6.49 1: 0.53
74,60 0.47
63.70 4: 0.22
Experimental Setup. We use ResNet 34 and 50 for experiments on CIFAR and Wisvin datasets respectively and optimize them using SGD with momentum. The complete details of the training setup can be found in Appendix A Most importantly, we take three main measures to ensure fair and reliable comparison throughout the experiments: 1) we reimplement all the loss functions we compare with in single shared learning setup, 2) we use the same hyperparameter optimization budget and mechanism for all the prior works and ours, and 3) we train and evaluate five networks for individual results, where in each run the synthetic noise, network initialization, and data-order differently randomized. The thorough analysis is evident from the higher performance of CE in our setup compared to prior works. Where possible, we report mean and standard deviation and denote the matistically-ssgniffi top performers with student t-test.
4.1 Synthetic Noise Benchmarks: CIFAR
Here, evaluate the proposed loss functions the CIFAR datasets with two types of synthetic noise: symmetric and asymmetric. For symmetric noise, the labels are, with probability 7, re- sampled from a uniform distribution over all labels. For asymmetric noise, follow the standard setup of Patrini et al. (22]. For CIFAR-10, the labels are modified, with probability 17, as follows: truck + automobile, bird airplane, cat <> dog, and deer horse. For CIFAR-100 100, labels are, with probability 7), cycled to the next sub-class of the same "super-class". egg. the labels of super-class "vehicles I" are modified as follows: bicycle bus motorcycle - pickup truck + train - bicycle.
We compare with other noise-robust loss functions such as label smoothing (LS)
23 Boot- strap (BS) 24. Symmetric Cross-Entropy (SCF) 4. Generalized Cross-Entropp (GCE) 3 and the
NCE+RCE loss of Ma et al. [5]. Here, we do not compare to methods that propose a full pipeline since, first, conclusive comparison would require re-implementation and individual evaluation of several components and second, robust loss functions can be considered orthogonal to them.
Results. Table 1 shows the results for symmetric and asymmetric noise on CIFAR-10 10 and CIFAR- 100. GJS performs similarly or better than other methods for different noise rates, noise types, and data sets. Generally, GJS'' efficacy is more evident for the more challenging CIFAR- 100 dataset. For example, on 60% uniform noise on CIFAR-100 100, the difference between GJS and the second best (GCE) is 4.94 percentage points, while our results on 80% noise is lower than GCE. We attribute this to the high sensitivity of the results to the hyperparameter settings in such a high-noise rate which are also generally unrealistic (Web Vision has ~20%). The performance of JS is consistently

---

Table Real-world Noise Benchmark on WebVision Mean test accuracy and standard deviation from five runs are reported for the validation sets of (mini) Web Vision and ILSVRCI2. GJS with two networks correspond to the mean prediction of two independently trained GJS networks with different seeds for data augmentation and weight initialization. Here, GJS uses Z 1. Results marked with f are from Zheltonozhski et al. 126]
WebVision
ILSVRCI2
Method
Architecture
Augmenlation Networks
Top
Top
Top
ELR+ Pa
Inception-Reshie ResNet-V2
Mixup
71.78
91.68
70.29
89.76
DivideMix TC
Croption-RoNNee--
Mixup
2
77.32
91.64
75.20
90.84
DivideMix
16
ResNet-50
Mixup
2
76.32 1: 0.36
90.65 = 0.16
74.42 1: 0.29
91.21 + 0.12
ResNet-50
ColorJitte
88.64 0.17
ResNet-50
ColorJitter
74.56
91.09 0.08
70.36
GJS
ResNet-50
ColorJiterr
1
77.99 + 0.35
90.62 = 0.28
74.33 1: 0.46
90.33 1 0.20
GJS
ResNet-50
ColorJiterr
2
79.28 + 0.24
91.22 = 0.30
75.50 + 0.17 91.27 + 0.26
similar the top performance of the prior works across different noise rates, types and datasets. In Section 4.3) we substantiate the importance the consistency term, identified in Proposition] 2 when going from JS to GJS that helps with the learning dynamics and reduce the susceptibility to noise. In Appendix [.1]] we provide results for GJS on instance destance-dependett synthetic noise (25). Next, we test the proposed losses naturally-nois dataset to see their efficacy in real- weal-world scenario.
4.2 Real- World Noise Benchmark: WisVision
WebVision v1 is a large-scale image dataset collected by crawling Flickr and Google, which resulted in an estimated 20% of noisy labels [28). There are 2.4 million images of the same thousand classes ILSVRC12. Here, 50 classes of the Google subset. We compare CE, JS, and GJS on WebVision following the same rigorous procedure as for the synthetic noise. However, upon request by the reviewers, we also compare with the reported results of some state-of-the of-the-art claborate techniques. This comparison deviates from our otherwise systematic analysis.
smaller version called mini WebVision [29] consisting of the first
Results. Table 2 as the common practice, reports the performances on the validation sets of WebVision and ILSVRC12 (first classes). Both JS and GJJS exhibit large margins with standard CE, especially for top-1 accuracy. Top-5 accuracy, due to its admissibility of wrong top predictions, can obscure the susceptibility noise-fitting and thus indicates smaller but still significant improvements.
The two state-of-the-ar methods on this dataset were DivideMix [66] and ELR+ (27]. Compared to setup, both these methods use stronger network (Inception (Inception-ReeNee--22 V2 ResNet-50), stronger augmentations (Mixup vs color jittering) and co-train two networks. Furthermore, ELR+ uses an exponential moving average of weights and DivideMix treats clean and noisy labeled examples differently after separating them using Gaussian mixture models. Despite these differences, GJSS performs good or better in terms of t accuracy Web Vision and significantly outperforms ELR+ on ILSVRC12 (70.29 vs 74.33). The importance of these differences becomes apparent as 1) the t accuracy for DivideMix degrades when using ResNet-50, and 2) the performance of GJJS improves by adding one of their components, i.e. the use of two networks. We train an ensemble of two independent networks with the GJS loss and average their predictions (last row of Table 2 This simple extension, which requires no change in the training code, gives significant improvements. To the best of our knowledge, this the highest reported top-1 accuracy on WebVision and ILSVRC12 when no pre-training is used.
we show state-of-the-art results when using GJS on two other real-world noisy datasets: ANIMAL-1ON 30) and Food-101N 31].
In Appendix B.2
So far, the experiments demonstrated the robustness of the proposed loss function (regarding Propo- sition 3) via the significant improvement of the final accuracy on noisy datasets. While this was central and informative, it is also important to investigate whether this improvement comes from the theoretical properties that were argued for JS and GJS. In what follows, we devise several such experiments, in effort to substantiate the theoretical claims and conjectures.
7

---

80.0
FL
77.5
75.0
Eprchs
Epocns
72.5
JS, 17 20%
JS, 7/ 60%
70.0
0%
40%
67.5
10
65.0
1.1
30
0.
0.
13
1.5
62.5
1000
200
700
400
202
303
303
403
60.0
(c) GJS. 7) 20%
(d) GJS, 60%
2
Number (istributions(M)
Figure 4: Effect of TTI Validation accuracy of JS and GJS during training with symmetric noise CIFARI00. From Proposition I JS be- haves like CE and MAE for low and high values of TT respectively. The signs of noise-fitting for T1 = 0.1 on 60% noise (b), and slow learning of Ti 0.9 (a-b), show this in practice. The GJJS loss does not exhibit overfitting for low values и and learns quickly for large values of क (c-d).
Figure 5: Effect of M. Validation accuracy for increasing number distributions (M) and dif- ferent symmetric noise rates on CIFAR- 100 with Ti = 1/4 For all noise rates, using three instead of two distributions results in higher accuracy. Going beyond three distributions is only help- ful for lower noise rates. For simplicity
M = 3 (corresponding to two augmentations) for all of experiments.
4.3 Towards a Better Understanding of the Jensen-Shannon-bann Loss Functions
Here, we study the behavior of the losses for different distribution weights TT number of distributions M, and epochs. We also provide insights on why GJS performs better than JS.
How does 1 control the trade-off of robustness and learnability? In Figure we plot the validation accuracy during training for both JS and GJS different values of क and noise rates r). From Proposition]] [ expect JS to behave as CE for low values of क and as MAE for larger values of mi Figure 4 (a-b) confirms this. Specifically, mi noise but overfits for 1 0.6 (characteristic of non-robust CE), on the other hand, slowly but robust to high noise rates (characteristic of noise-robust MAE).
learns quickly and performs well for low
0.9 learns
In Figure (c-d), we observe three qualitative improvements of GJS over JS: 1) no signs of overfitting noise for large noise rates with low values of TT 2) better learning dynamics for large values of T that otherwise learns slowly, and 3) converges to higher validation accuracy.
How many distributions to use? Figure depicts validation accuracy for varying number of distributions M. For all noise rates, we observe performance increase going from M = 2 to M = 3. However, the performance of M depends the noise rate. For lower noise rates, having more than three distributions can improve the performance. For higher noise rates egg. 60%, having M > 3 degrades the performance. We hypothesise this is due to: 1) high noise rates, there correctly labeled examples that can help guide the learning, and 2) going from M = 2 to M = adds consistency term, while M increases the importance of the consistency term in Proposition 2 Therefore, for a large enough M, the loss will find it easier to keep the consistency term low (keep predictions close uniform as the initialization), instead of generalizing based on the few clean examples. For simplicity, we have used M 3 for all experiments with GJS.
only few
Is the improvements of GJS over JS due decomposed GJS into a JS term with a mean prediction (f>1) and a consistency term operating on all distributions but the target. In Table we compare the performance of JS and GJS to GJS without the consistency term, i.e., LIs_. (v) P>1). The results suggest that the improvement of GJS over JS can be attributed to the consistency term.
prediction consistency? Proposition
Figure (a-b) showed that JS improves the learning dynamics of MAE by blending it with CE, controlled by Nं. Similarly, we see here that the consistency term also improves the learning dynamics (underfitting and convergence speed) of MAE. Interestingly, Figure (c-d), shows the higher values of का (closer to MAE) work best for GJS, hinting that, the consistency term improves the learning dynamics of MAE much so that the role of CE becomes less important.

---

Table 3: Effect of Consistency. Validation ac- curacy for JS. GJS w/o the consistency term in Proposition and GJS for 40% noise on the CIFAR- 100 dataset. Using the mean two pre- dictions in the JS loss does not improve perfor- mance. On the other hand, adding the consis- tency term significantly helps.
Table 4: Effect of GJS. Validation aceuracy when using different loss functions for clean and noisy examples of the ZIFAR-100 training set with 40% symmetric noise. Noisy examples ben- efit significantly more from GJS than clean ex- amples (74.1 vs 72.9).
Method
R1
Method
Accuracy
Noisy
0.5
Cas(e!M), (/31) Lis (elar(e()), -Bsi) ( D(2) (01)
71.0
15
70.0
71.5
55.3
68.7
GIS
72.6
72.9
70.2
74,3
GJS
71.0
74.1
G38
74.7
Is GJS mostly helping the clean or noisy examples? To better understand the improvements of GJS over JS, we perform an ablation with different losses for clean and noisy examples, see Table We observe that using GJS instead of JS improves performance in all cases. Importantly, using GJS only for the noisy examples performs significantly better than only using for the clean examples (74.1 vs 72.9). The best result is achieved when using GJS for both clean and noisy examples but still close to the noisy-only case (74.7 vs 74.1).
How is different choices of perturbations affecting GJS? In this work, we use stochastic augmen- tations for A, see Appendix for details. Table validation results on 40% symmetric and asymmetric noise on CIFAR-TOO for varying types of augmentation. We observe that all methods improve their performance with stronger augmentation and that GJS achieves the best results in all cases. Also, note that we use weak augmentation for all naturally-noisy datasets (Web Vision, ANIMAL-ION, and Food- 101d and still get state-of-the-an results.
How fast is the convergence? We found that some baselines (especially the robust NCE+RCE) had slow convergence. Therefore, we used 400 epochs for all methods make sure all had time to converge properly. T shows results on 40% symmetric and asymmetric noise on CIFAR-100 when the number of epochs has been reduced by half.
Is training with the proposed losses leading to more consistent networks? Our motivation for investigating losses based on lensen-Shannon divergence was partly due to the observation in Figure] T that consistency and accuracy correlate when learning with CE loss. In Figure[6] we compare CE, JS, and GJS losses in terms of validation accuracy and consistency during training 40% symmetric noise. We find that the networks trained with JS and GJS losses are more consistent and has higher accuracy. In Appendix B.7 we report the consistency of the networks in Table
CIFAR-100 with
Summary of experiments in the appendix. Due to space limitations, we report several important experiments in the appendix. We evaluate the effectiveness of GJJ on 1) instance- deeeedetd synthetic noise (Section I..I), and 2) real-world noisy datasets ANIMAL- 1ON and Food- 101N (Section (B.2). We also investigate the importance of 1) losses being symmetric and bounded for learning with noisy labels (Section B.3). 3. and 2) a clean vs noisy validation set for hyperparameter selection and the effect of single of parameters for all noise rates (Section B.5)
Table 5: Effeet of Augmentation Strategy. Val- idation accuracy for training w/o CutOut(-COO) or w/o RandAAu(-RAA) or w/o both(weak) on 40% symmetric and asymmetric noise on CIFAR-100. All methods improves by stronger augmentations.
Table 6: Effect of Number of Epochs. Vali- dation accuracy for training with 200 and 400 epochs for 40% symmetric and asymmetric noise on CIFAR- 100. GJS still outperforms the baselines and NCE+RCE's performance is reduced heavily by the decrease in epochs.
GJS performs best for all types of augmentations.
Symmetric
Asymmetric
Symmettic
Asymmetric
Method
Method
Full
Weak
-CO
-RA
GCB
70,8
64.2
64.1
58.0
51.7
44.9
46.6
42.9
GCE
70.3
70.8
39.1
51.7
NCE-RCE 68.5
57.5
52.1
49.5
44.4
NCE+RCE
60.0
68.5
35.0
57,5
GJS
74.8
71.3
70.6
66.5
62.6
56.8
52.2
44.9
72.9
74.8
43.2
62.6

---

100
100
100
Val Acouracy(35)
80
Consistency(99)
80
Consietency(%)
80
60
60
60
40
40
40
20
20
20
0
0
0
100
200
300
400
100
200
300
400
100
200
300
400
Epochs
Epochs
Epochs
(a) Validation Accuracy
(b) Consistency Clean
(c) Consistency Noisy
Figure 6: Evolution of a trained network's consistency for the CE, JS, and GJS losses. We plot the evolution of the validation accuracy (a) and network's ' consistency on clean (b) and noisy (c) examples of the training set of CIFAR- 100 when learning with 40% symmetrie noise. All losses use the same learning rate and weight decay and both JS and GJS use = 0.5. The consistency of the learnt function and the accuracy closely correlate. The accuracy and consistency of JS and GJS improve during training, while both degrade when learning with CE loss.
Limitations & Future Directions
We empirically showed that the consistency of the network around noisy data degrades as fits noise and accordingly proposed a loss based on generalized Jensen-Shannon divergence (GJS), While we empirically verified the significant role consistency regularization in robustness to noise, we only theoretically showed the robustness (B,, Byy) of GJS at its limit (IT - 1) where the consistency term gradually vanishes. Therefore, the main limitation is the lack a theoretical proof of the robustness of the consistency term in Proposition 2 area, also for the literature self- or semi- cani-supervised learning and thus of utmost importance for future works.
This is, in general, an important but understudied
Secondly, we had an important observation that GJS with M > 3 might not perform well under high noise rates. While we have some initial conjectures, this phenomenon deserves systematic analysis both empirically and theoretically.
Finally, a minor practical limitation the added computations for GJS forward passes, however this applies training time only and in all experiments, we only use extra prediction (M 3).
6 Final Remarks
We first made two central observations that (i) robust loss functions have underfitting issue and (ii) consistency of foise-fitting networks significantly lower around noisy data points. Correspondingly, we proposed two loss functions, JS and GJS,, based on Jensen-Shannn divergence that (i) interpolates between woise-robuse MAE and conteronng CE, and (ii) encourages consistency around training data points. This simple proposal led to sfte-of-the-ait performance on both synthetic and real- world noise datasets even when compared to the more elaborate pipelines such as DivideMix or ELR+. Furthermore, discussed their robustness within the theoretical construction of Ghosh al. (2). By drawing further connections to other seminal loss functions such as CE, MAE, GCE, and consistency regularization, uncovered other desirable informative properties. We further empirically studied different aspects of the losses that corroborate various theoretical properties.
Overall, we believe the paper provides informative theoretical and empirical evidence for the useful- ness of two simple and novel divergence- loss functions for learning under noisy data that achieve state-of-the-ee results. At the same time, it opens interesting future directions.
Ethical Considerations. Considerable resources are needed to create labeled data sets due to the burden of manual labeling process. Thus, the creators of large annotated datasets are mostly limited well-funded companies and academic institutions. In that sense, developing robust methods against label noise enables less affluent organizations or individuals to benefit from labeled datasets since imperfect automatic labeling can be used instead. On the other hand, proliferation of such harvested datasets can increase privacy concerns arising from redistribution and malicious use.
This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.
Acknowledgement.
10

---

References
[1] Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07/59, 2020.
[2] Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pages 1919-1925, 2017.
[3] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems, pages 8778-8788, 2018.
[4] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE International Conference on Computer Vision, pages 322-330, 2019.
[5] Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Normalized loss functions for deep learning with noisy labels, 2020.
[6] Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. In International Conference on Learning Representation, 2020.
[7] Avital Oliver, Augustus Odena, Colin Raffel, Ekin D Cubuk, and Ian J Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. arXiv preprint arXiv:/804.09) 2018.
[8] Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145-151, 1991.
[9] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization, 2017.
[10] Yang Liu and Hongyi Guo. Peer loss functions:
Learning from noisy labels without knowing noise rates. In Hal DaumÅO III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6226-62288 PMLR, 13-18 Jul 2020.
[11] Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in neural information processing systems, pages 1 1196-1204, 2013.
[12] Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. In Proceedings of the international conference on learning representation, 2015.
[13] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the JEEE Conference on Computer Vision and Pattern Recognition, pages 1944-195 2017.
[14] Bo Han, Jiangehao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy supervision. In Advances in Neural Information Processing Systems, pages 5836-5846, 2018
[15] Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In Advances in Neural Information Processing Systems, pages 6838-688 08889,89, 2019.
[16] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In International Conference on Learning Representation, 2020.
[17] Jihoon Tack, Sihyun Yu, Jongheon Jeong, Minseon Kim, Sung Hwang, and Jinwoo Shin. Consistency regularization for adversarial robustness, 2021.
11

---

[18] Yilun Xu, Peng Cao. Yuqing Kong, and Yizhou Wang. L_dmi: A novel information - loss function for training deep nets robust to label noise. In Advances in Neural Information Processing Systems, pages 6225-6236 2019.
[19] Jiaheng Wei and Yang Liu. When optimizing f-divergence is robust with label noise. In International Conference on Learning Representation, 2021.
[20] I. CSISZAR. Information-tyee -tppe measures of difference of probability distributions and indirect observation. Studia Scientiarum Mathematicarum Hungarica, 2:229-318, 1967.
[21] Frank Nielsen. On the jensen-shannol symmetrization of distances relying on abstract means. Entropy, 21(5), 2019.
[22] Giorgio Patrini, Alessandro Rozza, Aditya Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: loss correction approach, 2017.
[23] Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In International Conference on Machine Learning, 2020.
[24] Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.659% 2014.
[25] Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen. Learning with feature-dependent label noise: A progressive approach, 2021.
[26] Evgenii Zheltonozhskii, Chaim Baskin, Avi Mendelson, Alex M. Bronstein, and Or Litany. Contrast divide: Self-supervised pre-training for learning with noisy labels, 2021.
[27] Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Early- learning regularization prevents memorization of noisy labels, 2020.
[28] Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data, 2017.
[29] Lu Jiang, Zhenyuan Zhou, Thomas Leung, Jia Li, and Fei-Fei Li. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In ICML, 2018.
[30] Hwanjun Song, Minseok Kim, and Jae-Gil Lee. SELFIE: Refurbishing unclean samples for robust deep learning. In ICML, 2019.
[31] Kuang-Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Cleannet: Transfer learning for scalable image classifier training with label noise. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[32] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V. Le. Randaugment: Practical automated data augmentation with reduced search space, 2019.
[33] Terrance DeVries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout, 2017.
[34] Junnan Li, Yongkang Wong, Qi Zhao, and Mohan Kankanhalli. Learning to learn from noisy labeled data, 2019.
[35] Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. Self: Learning to filter noisy labels with self- ensembling. In International Conference Learning Representation, 2019.
[36] Curtis G Northeutt, Tailin Wu, and Isaac I. Chuang. Learning with confident examples: Rank pruning for robust classification with noisy labels. arXiv preprint arXiv:/705.0m 1705.0/936 2017.
[37] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5552-5560, 2018
12

---

[38] Arash Vahdat. Toward robustness against label noise in training deep discriminative neural networks. Advances in Neural Information Processing Systems, pages 5596-5605, 2017.
[39] Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, Ondrej Chum, and Cordelia Schmid. Graph convolutional networks for learning with few clean and many noisy labels. In Proceedings of the European Conference on Computer Vision, 2020.
[40] Paul Hongsuck Seo, Geeho Kim, and Bohyung Han. Combinatorial inference against label noise. In Advances in Neural Information Processing Systems, pages 1173 1183, 2019.
[41] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zhigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 281 2818-2826, 2016.
[42] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii Virtual adversarial training: a regularization method for supervised and cuperssuer learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979-199) 2018.
[43] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems, pages 5049-5059, 2019.
[44] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in neural information processing systems, pages 1195- 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Section C in the Appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (cither in the supplemental material or as a URL)? [Yes] See footnote on the first page.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section A in the Appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See main experiments in Table 2
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section A
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
13

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "r9PipW6Zx6P", "reviewer": "Reviewer_xNPb", "summary": "The paper proposed to use JS divergence as a loss function (JS) to address the underfit problem of provably-robust loss functions. Further, the authors generalize the JS loss function (GJS) to include multiple distributions to encourage consistency during learning with noisy labels. They demonstrate that JS interpolates MAS and CE loss, and they proved the noisy robustness property of the JS and GJS. Extensive experiments are provided to show the effectiveness of the proposed loss functions. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and clearly conveys the authors' thesis.\n\npros:\n1. observing the significant consistency reduction around noisy data samples when the model overfits noise is of theoretical interest and practical importance.\n2. the authors proved the noise robustness properties of the proposed novel loss functions, JS and GJS. They further provided a detailed investigation and ablation study to support their theoretical claims and conjectures.\n4. they carried out extensive experiments and demonstrated the proposed loss functions could achieve state-of-the-art performance under symmetric and asymmetric label noise.\n\ncons/questions:\n1. the consistency regularizer may lead the network to learn a more smooth prediction function. This is similar to the effects of some adversarial defense methods. I wonder if the GJS will lead to better adversarial robustness? \n2. the decrease of consistency when overfitting to noise is an important phenomenon. One major hypothesis of this work is that there is a causal relationship between clean validation accuracy and consistency. It would be better to provide consistency plots for JS and GJS in addition to those validation plots. \n3. In Figure 5, we see the validation accuracy decrease when M>3 for 60% noise rate case. This observation has not been adequately discussed. Adding more distributions would put more emphasis on consistency. Would it be the case that under heavy noise rate, there are not enough anchors to guide the model and balance the force of encouraging consistency? \n4. it will be better to include experiments on instance-based label noise. In that case, the label noise is related to the feature and may introduce \"fake consistency\". It is of interest to see the performance of GJS under such a setting.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact have been addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "pN1IO1Zdsf3", "reviewer": "Reviewer_6kDX", "summary": "$\\textbf{Summary and main contributions:}$\n\n(1) The authors propose to use Jensen-Shannon divergence (JS) to learn with noisy labels, which is shown to interpolate between Cross-Entropy and MAE loss through a hyper-parameter.\n\n(2) Based on the new observation that learning with noisy labels reduces the classifier's prediction consistency for noisy-labeled sets. To overcome this issue, the authors introduce multi-distribution generalization (GJS) which is shown to generalize JS with a consistency regularizer.\n\n(3) Experiment on synthetic noisy labeled datasets and a real-world noisy dataset (WebVision) demonstrate the effectiveness of proposed methods.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "$\\textbf{Originality:}$ the authors introduce two novel robust methods to deal with label noise and alleviate the issue that \"a network's prediction consistency got reduced for noisy-labeled data\".\n\n$\\textbf{Quality:}$ the paper is overall well-written, although there are some additional issues, for example, \n\n$\\textbf{Clarity:}$ some notations and technical terms are not well-defined, also some experiment details are missed.\n\n$\\textbf{Additional concerns/suggestions:}$\n\n(1) Some technical terms require additional explanations, for example, in Line 16-17, readers may not be familiar with \"provably-robust loss functions\" and \"consistency\". The presentation would be much better if the authors explain a bit more about these two terms since these two observations convey some important information about this work.\n\n(2) Line 26, a typo: \"labelled\" -> \"labeled\".\n\n(3) In the caption of Figure 1, it would be more accurate if the authors write \"... as it overfits to label noise using CE loss\". Besides, in Figure 1 (a), is the validation accuracy evaluated w.r.t. a clean or noisy validation set. I assumed that under these four levels of label noise, the learning rate settings are the same. For the plot w.r.t. relative large noise rate such as 60%, will a lower learning rate (after 200 epochs) avoids the corresponding classifier degrades so much? Although I believe that the order of performance will not change, the gap might not be so large?\n\n(4) In Line 43, the notation of simplex $\\Delta^{K-1}$ is not defined.\n\n(5) In Line 49, it would be better if the authors firstly introduce a common real-world noise model where $\\eta$ is usually instance-dependent, e.g., $\\eta(x, y)$ for each instance/feature $x$ and class $y$. Then, state that in this work, the authors are interested in the instance-independent noise model where $\\eta(x)=\\eta$ and $\\eta$ is the same for each class for a theoretical perspective. \n\n(6) The explanation w.r.t. Figure 1 is delayed to Line 63, it might be better if the authors move this in the introduction section?\n\n(7) In Line 77, a formal statement of the properties w.r.t. Jenson Shannon divergence is suggested.\n\n(8) I suggest the authors explain more about the importance of the property \"bounded\" and \"symmetric\".\n\n(9) The $m_{>1}$ in Proposition 2 feels like a weird notation since readers may find it hard to get the lingual relationship between $m$ and the weighted $\\mathbf{p}^{(j)}$. $\\bar{\\mathbf{p}}_{>1}$ or another notation might make the presentation better?\n\n(10) In Theorem 1, I feel like the requirement of $\\forall x, f$ will somehow make $B_U-B_L$ much larger. Do authors think this Theorem 1 can be stated in this view: with probability at least $1-\\delta$, we have $\\mathbb{P}(B_L\\leq \\sum_{i=1}^K \\mathcal{L}(e^{(i)}, x, f)\\leq B_U)\\geq 1-\\epsilon.$ \nFiltering out extreme values in this view, the gap between $B_L$ and $B_U$ appear not so large. I think the proofs are not influenced with probability at least $1-\\delta$.\n\n(11) I didn't find any definition of $\\mathbf{u}$ in Equ. (6). \n\n(12)  In [1], the authors showed the robustness of the variation-form f-div functions w.r.t. two properly defined distributions (joint distribution and the product of marginal distributions via model prediction and label), where Jensen Shannon is a special case. It would be better if authors could discuss more relationships between these two methods (JS in this work and [1]) in the main paper, or with the experiment comparison.\n\n(13) Experiment details: do authors use the same synthetic noise dataset for all methods? Or generate the noisy labels with a random seed for each training. If it is the latter case, do authors fix the random seed?\n\n(14) In Line 220, the comparison with DivideMix [2] and ELR+ [3] are not reasonable. Although [2] and [3] adopt a stronger network, different architectures may result in different performances. For example, my previous observations show that Pre-Resnet 18 outperforms Resnet 34 in certain noisy label synthetic tasks, while the latter one has a stronger network.\n\n$\\textbf{Reference:}$\n\n[1] Wei, Jiaheng, and Yang Liu. \"When Optimizing f-Divergence is Robust with Label Noise.\" In International Conference on Learning Representations. 2020.\n\n[2] Li, Junnan, Richard Socher, and Steven CH Hoi. \"DivideMix: Learning with Noisy Labels as Semi-supervised Learning.\" In International Conference on Learning Representations. 2019.\n\n[3] Liu, Sheng, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. \"Early-Learning Regularization Prevents Memorization of Noisy Labels.\" Advances in Neural Information Processing Systems 33 (2020).\n\n##############################################################\n\n$\\textbf{Post-Rebuttal}$\n\nThank authors for addressing all my concerns. And after the rebuttal, I decide to raise the rating from 6 to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have discussed their limitations in Section 5. I listed constructive suggestions above \"Main Review\".", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "> 12 hours"}, "extra_text": {}}, {"review_id": "SP8LN1U97by", "reviewer": "Reviewer_HdUW", "summary": "The paper targets at learning with noisy labels and propose two loss functions: JS and GJS. To justify the loss functions, robustness and consistency properties are discussed.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros**\n\n-\tThe theoretical analysis on JS and GJS are clear. The justifications on robustness and consistency make sense.\n-\tThere are ablation studies showing the effect of each component.\n\n\n**Cons**\n\n*Mythology*\n\n-\tThe contribution is minor compared with previous works. 1) For the main claims of robustness and underfitting, JS shares the same properties as GCE. 2) The superior of GJS over JS is the consistency property. GJS is equivalent to CE plus a regularization term that encourages low KL between predictions on perturbed inputs. Such a regularization term is widely known in the community: it has long been used in semi-supervised learning, noisy label learning and adversial training. We should be aware that directly introducing such a consistency regularization on any existing methods may improve the performance in learning with noisy labels by a large margin (I have verified this in experiments and the author could conduct experiments if interested). Considering that the consistency regularization is known, the contribution of this paper is minor.\n-\tThe observation on network predictions’ consistency is not novel as claimed in the main contributions. It is straightforward for random noise because the labels are flipped to different classes for similar samples (neighbor samples in some sense) and the input spaces are divided into much more decision regions compared with training on clean. Actually, this has been demonstrated in [Aprit et al. ICML 2017] (please refer to the Critical Sample Ratio which is an estimation of the density of decision boundaries). Naturally, the observation on consistency should be less significant for class-dependent noise and more realistic instance-dependent noise.\n-\tIn this direction, the line of works that study robustness mostly based on the assumption that the label noise is class-conditional (even random as assumed in Theorem 1 in this paper). However, both intuition and rigorous hypothesis testing [Zhang et al. ICLR 2021] imply that real-world noise should be instance dependent.\n\n*Experiments*\n-\tThe empirical results are not significant on more realistic asymmetric noise and real-world noise. This is consistent with the above concerns that the theoretical justification is based on strong assumptions and the consistency observation is most significant for random noise.\n-\tThe author does not compare to methods that propose a full pipeline and claim that the reason is robust loss functions can be considered complementary to existing methods. However, firstly, there are many strong yet simple baselines (sample selection, label correction methods) that are necessary to compare on large-scale experiments on CIFAR-10 and CIFAR-100. Secondly, the paper should demonstrate an improvement by introducing GJS on existing methods since it is claimed as complementary.\n\n\n[Aprit et al. ICML 2017] A Closer Look at Memorization in Deep Networks.\n[Zhang et al. ICLR 2021] Learning with Feature-Dependent Label Noise: A Progressive Approach.\n\n\n*post rebuttal*\n\nThanks for the detailed feedback. My concerns are well addressed.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "GifS1GtwU2", "reviewer": "Reviewer_ogFS", "summary": "This paper considers learning with noisy labels.  The author showed that the\nJensen-Shannon (JS) divergence sits in between the mean absolute error (MSE) and\nthe cross entropy (CS).  The key proposition is to use the generalized JS (GJS)\nas the loss, which presents a trade-off between robustness with respect to the\nnoisy labels and to avoid under-fitting from which robust loss function (such as\nthe MSE) suffers.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this contribution has some theoretical analysis in section 2,\nas well as experimental results on CIFAR/WebVision showing that the proposed GJS loss\nleads to more accurate classification under the setting of noisy labels.\nBased on these contributions, I tend to vote for acceptance.\n\nEssentially, based on the experimental results, the most useful tool proposed \nby the authors is the GJS (rather than JS) divergence as a robust loss. Based on \nproposition 2, the loss has two terms. Minimizing the first term leads to \nlearnability, as it tries to align the target (noisy) label with the centroid\nof the predictions; the second term leads to robustness, as it measures\nthe deviation of the predictions from their centroid.\n\nThe authors provided the recipe to choose the \\pi factor,\nwhich is important to trade-off between learnability and robustness and to make the experiments reproducible.\n\nTheoretically, the main argument in section 2.5 is that the proposed GJS loss\nis (trivially) bounded on both sides, and any bounded loss satisfies\nthe property stated in theorem 1: the deviation of the empirical risk \nwith respect to the label noise can be bounded. Note the GJS loss is\nonly robust at its extreme point when pi tends to be deterministic,\nas \"robustness\" requires the deviation to be absolute zero.\n\nThese theoretical results can be improved by considering the parametric\nfamily of the perturbation A(\\eta), and specific label noise, which is not considered in this\npaper. Is the GJS loss fitted for any noise labels and noisy perturbations\nof the predictions? How are these two difference noise related to affect the learning?\nCan we define \\epsilon-robustness and bound the robustness\nunder the GJS loss? In general, the presented theoretical results\nlack significance and can be enhanced.\n\nMost of the experimental evaluation is based on the classification accuracy.\nThere should be some consistency measurements as shown in figure 1\n(I trust the GJS should perform well because of its 2nd regularization term).\n\nMinor comments:\n\nL70\n\"Assuming causality, suggests that\" -> please rewrite the sentence\n\nL74: is $\\Delta$ introduced before?\n\nL78: \"... is another crucial difference\" with what?\n\nL102: mention GCE, SCE are shorthand for ...\n\nL107 and Figure 2: {L}_{JS} instead of JS (JS = 0 is $\\pi$ goes to extreme)\n\nL223: come from -> comes from\n\n_____\n**Post Rebuttal remarks**\n\nThe reviewer thanks the authors for the response, and for agreeing on the review and several extensions, and addressing all minor comments. The disagreement on the significance is more on whether this type of contribution on applying existing tools from information theory into current machine learning problems are interesting enough for NeurIPS.  Ideally, the statements can use the structure of the JS divergence, which will improve the significance. For example, section 2.5 only use the fact that the JS divergence is bounded. These aspects can definitely be improved.\n\nBesides the current review, in preparing the next version (whether it is accepted or not), the authors are suggested to include proper reference (I leave the authors to do the literature review) on the following:\n- The definition of GJS and other generalizations of JS;\n- Existing analysis on the limit case of GJS;\n- Existing applications of JS/GJS into machine learning/pattern recognition problems, and as loss functions.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers deemed the paper of interest. I would like to thank the authors for providing focused discussion content for both theoretical and experiments, which contributed to explain further the paper's content and its improvements with respect to its submission history.

In the revised version, the authors should include the generalisation to Theorem 1 proposed at discussion phase to cover asymetric noise and in experiments, find a place to report consistency and instance-based label noise, a short discussion vs Wei + Liu's approach as developed in the discussion (point (12) 6kDX) and complete references to be more extensive. This is extremely important given the context of the paper and the richness of the relevant literature in the past decade or so.

AC.
Decision: Accept (Poster)