Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels
Year: 2021

Abstract:
Prior works have found it beneficial to combine provably noise-robust loss functions e.g., mean absolute error (MAE) with standard categorical loss function e.g. cross entropy (CE) to improve their learnability. Here, we propose to use Jensen-Shannon divergence as a noise-robust loss function and show that it interestingly interpolate between CE and MAE with a controllable mixing parameter. Furthermore, we make a crucial observation that CE exhibits lower consistency around noisy data points. Based on this observation, we adopt a generalized version of the JensenShannon divergence for multiple distributions to encourage consistency around data points. Using this loss function, we show state-of-the-art results on both synthetic (CIFAR), and real-world (e.g. WebVision) noise with varying noise rates.

heading: 1 Introduction
text: Labeled datasets, even the systematically annotated ones, contain noisy labels [1]. Therefore, designing noise-robust learning algorithms are crucial for the real-world tasks. An important avenue to tackle noisy labels is to devise noise-robust loss functions [2, 3, 4, 5]. Similarly, in this work, we propose two new noise-robust loss functions based on two central observations as follows.
Observation I: Provably-robust loss functions can underfit the training data [2, 3, 4, 5]. Observation II: Standard networks show low consistency around noisy data points 1, see Figure 1.
We first propose to use Jensen-Shannon divergence (JS) as a loss function, which we crucially show interpolates between the noise-robust mean absolute error (MAE) and the cross entropy (CE) that better fits the data through faster convergence. Figure 2 illustrates the CE-MAE interpolation. Regarding Observation II, we adopt the generalized version of Jensen-Shannon divergence (GJS) to encourage predictions on perturbed inputs to be consistent, see Figure 3. Notably, Jensen-Shannon divergence has previously shown promise for test-time robustness to domain shift [6], here we further argue for its training-time robustness to label noise. The key contributions of this work2 are:
• We make a novel observation that a network predictions’ consistency is reduced for noisylabeled data when overfitting to noise, which motivates the use of consistency regularization.
• We propose using Jensen-Shannon divergence (JS) and its multi-distribution generalization (GJS) as loss functions for learning with noisy labels. We relate JS to loss functions that are based on the noise-robustness theory of Ghosh et al. [2]. In particular, we prove that JS generalizes CE and MAE. Furthermore, we prove that GJS generalizes JS by incorporating consistency regularization in a single principled loss function.
• We provide an extensive set of empirical evidences on several datasets, noise types and rates. They show state-of-the-art results and give in-depth studies of the proposed losses.
1we call a network consistent around a sample (x) if it predicts the same class for x and its perturbations (x̃). 2implementation available at https://github.com/ErikEnglesson/GJS
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

heading: 2 Generalized Jensen-Shannon Divergence
text: We propose two loss functions, the Jensen-Shannon divergence (JS) and its multi-distribution generalization (GJS). In this section, we first provide background and two observations that motivate our proposed loss functions. This is followed by definition of the losses, and then we show that JS generalizes CE and MAE similarly to other robust loss functions. Finally, we show how GJS generalizes JS to incorporate consistency regularization into a single principled loss function. We provide proofs of all theorems, propositions, and remarks in this section in Appendix C.

heading: 2.1 Background & Motivation
text: Supervised Classification. Assume a general function class3 F where each f ∈ F maps an input x ∈ X to the probability simplex ∆K−1, i.e. to a categorical distribution over K classes y ∈ Y = {1, 2, . . . ,K}. We seek f∗ ∈ F that minimizes a risk RL(f) = ED[L(e(y), f(x))], for some loss function L and joint distribution D over X×Y, where e(y) is a K-vector with one at index y and zero elsewhere. In practice, D is unknown and, instead, we use S = {(xi, yi)}Ni=1 which are independently sampled from D to minimize an empirical risk 1N ∑N i=1 L(e(yi), f(xi)).
Learning with Noisy Labels. In this work, the goal is to learn from a noisy training distribution Dη where the labels are changed, with probability η, from their true distribution D. The noise is called instance-dependent if it depends on the input, asymmetric if it dependents on the true label, and symmetric if it is independent of both x and y. Let f∗η be the optimizer of the noisy distribution risk RηL(f). A loss function L is then called robust if f∗η also minimizes RL. The MAE loss (LMAE(e(y), f(x)) := ‖e(y) − f(x)‖1) is robust but not CE [2]. Issue of Underfitting. Several works propose such robust loss functions and demonstrate their efficacy in preventing noise fitting [2, 3, 4, 5]. However, all those works have observed slow convergence of such robust loss functions leading to underfitting. This can be contrasted with CE that has fast convergence but overfits to noise. Ghosh et al. [2] mentions slow convergence of MAE and GCE [3] extensively analyzes the undefitting thereof. SCE [4] reports similar problems for the reverse cross entropy and proposes a linear combination with CE. Finally, Ma et al. [5] observe the same problem and consider a combination of “active” and “passive” loss functions.
Consistency Regularization. This encourages a network to have consistent predictions for different perturbations of the same image, which has mainly been used for semi-supervised learning [7].
Motivation. In Figure 1, we show the validation accuracy and a measure of consistency during training with the CE loss for varying amounts of noise. First, we note that training with CE loss eventually overfits to noisy labels. Figure 1a, indicates that the higher the noise rate, the more accuracy drop when it starts to overfit to noise. Figure 1(b-c) shows the consistency of predictions for correct and noisy labeled examples of the training set, with the consistency measured as the ratio of examples that have the same class prediction for two perturbations of the same image, see
3e.g. softmax neural network classifiers in this work
Appendix B.6 for more details. A clear correlation is observed between the accuracy and consistency of the noisy examples. This suggests that maximizing consistency of predictions may improve the robustness to noise. Next, we define simple loss functions that (i) encourage consistency around data points and (ii) alleviate the “issue of underfitting” by interpolating between CE and MAE.

heading: 2.2 Definitions
text: DJS. Let p(1),p(2) ∈ ∆K−1 have corresponding weights π = [π1, π2]T ∈ ∆. Then, the JensenShannon divergence between p(1) and p(2) is
DJSπ (p (1),p(2)) := H(m)− π1H(p(1))− π2H(p(2)) = π1DKL(p(1)‖m) + π2DKL(p(2)‖m) (1)
with H the Shannon entropy, and m = π1p(1) + π2p(2). Unlike Kullback–Leibler divergence (DKL(p(1)‖p(2))) or cross entropy (CE), JS is symmetric, bounded, does not require absolute continuity, and has a crucial weighting mechanism (π), as we will see later.
DGJS. Similar to DKL, DJS satisfies DJSπ (p (1),p(2)) ≥ 0, with equality iff p(1) = p(2). For DJS, this is derived from Jensen’s inequality for the concave Shannon entropy. This property holds for finite number of distributions and motivates a generalization of DJS to multiple distributions [8]:
DGJSπ (p (1), . . . ,p(M)) := H ( M∑ i=1 πip (i) ) − M∑ i=1 πiH(p (i)) = M∑ i=1 πiDKL ( p(i) ∥∥∥ M∑ j=1 πjp (j) ) (2)
where M is the number of distributions, and π = [π1, . . . , πM ]T ∈ ∆M−1. Loss functions. We aim to use DJS and DGJS divergences, to measure deviation of the predictive distribution(s), f(x), from the target distribution, e(y). Without loss of generality, hereafter, we dedicate p(1) to denote the target distribution. JS loss, therefore, can take the form of DJSπ (e
(y), f(x)). Generalized JS loss is a less straight-forward construction since DGJS can accommodate more predictive distributions. While various choices can be made for these distributions, in this work, we consider predictions associated with different random perturbations of a sample, denoted by A(x). This choice, as shown later, implies an interesting analogy to consistency regularization. The choice, also entails no distinction between the M − 1 predictive distributions. Therefore, we consider π2 = · · · = πM = 1−π1M−1 in all our experiments. Finally, we scale the loss functions by a constant factor Z = −(1−π1) log(1−π1). As we will see later, the role of this scaling is merely to strengthen the already existing and desirable behaviors of these losses as π1 approaches zero and one. Formally, we have JS and GJS losses:
LJS(y, f,x) := DJSπ (e
(y), f(x̃)) Z , LGJS(y, f,x) := DGJSπ (e (y), f(x̃(2)), . . . , f(x̃(M))) Z (3)
with x̃(i) ∼ A(x). Next, we study the connection between JS and losses which are based on the robustness theory of Ghosh et al. [2].

heading: 2.3 JS’s Connection to Robust Losses
text: Cross Entropy (CE) is the prevalent loss function for deep classifiers with remarkable successes. However, CE is prone to fitting noise [9]. On the other hand, Mean Absolute Error (MAE) is theoretically noise-robust [2]. Evidently, standard optimization algorithms struggle to minimize MAE, especially for more challenging datasets e.g. CIFAR-100 [3, 5]. Therefore, there have been several proposals that combine CE and MAE, such as Generalized CE (GCE) [3], Symmetric CE (SCE) [4], and Normalized CE (NCE+MAE) [5]. The rationale is for CE to help with the learning dynamics of MAE. Next, we show JS has CE and MAE as its asymptotes w.r.t. π1. Proposition 1. Let p ∈ ∆K−1, then
lim π1→0 LJS(e(y),p) = H(e(y),p), lim π1→1
LJS(e(y),p) = 1
2 ‖e(y) − p‖1
where H(e(y),p) is the cross entropy of e(y) relative to p.
Figure 2 depicts how JS interpolates between CE and MAE for π1 ∈ (0, 1). The proposition reveals an interesting connection to state-of-the-art robust loss functions, however, there are important differences. SCE is not bounded (so it cannot be used in Theorem 1), and GCE is not symmetric, while JS and MAE are both symmetric and bounded. In Appendix B.3, we perform a dissection to better understand how these properties affect learning with noisy labels. GCE is most similar to JS and is compared further in Appendix B.4.
A crucial difference to these other losses is that JS naturally extends to multiple predictive distributions (GJS). Next, we show how GJS generalizes JS by incorporating consistency regularization.

heading: 2.4 GJS’s Connection to Consistency Regularization
text: In Figure 1, it was shown how the consistency of the noisy labeled examples was reduced when the network overfitted to noise. The following proposition shows how GJS naturally encourages consistency in a single principled loss function.
Proposition 2. Let p(2), . . . ,p(M) ∈ ∆K−1 with M ≥ 3 and p̄>1 = ∑M j=2 πjp (j)
1−π1 , then
LGJS(e(y),p(2), . . . ,p(M)) = LJSπ′ (e (y), p̄>1) + (1− π1)LGJSπ′′ (p (2), . . . ,p(M))
where π′ = [π1, 1− π1]T and π′′ = [π2,...,πM ] T
(1−π1) .
Importantly, Proposition 2 shows that GJS can be decomposed into two terms: 1) a JS term between the label and the mean prediction p̄>1, and 2) a GJS term, but without the label. Figure 3 illustrates the effect of this decomposition. The first term, similarly to the standard JS loss, encourages the predictions’ mean to be closer to the label (Figure 3 middle). However, the second term encourages all predictions to be similar, that is, consistency regularization (Figure 3 right).

heading: 2.5 Noise Robustness
text: Here, the robustness properties of JS and GJS are analyzed in terms of lower (BL) and upper bounds (BU ) for the following theorem, which generalizes the results by Zhang et al. [3] to any bounded loss function, even with multiple predictive distributions.
Theorem 1. Under symmetric noise with η < K−1K , if BL ≤ ∑K i=1 L(e(i),x, f) ≤ BU , ∀x, f is satisfied for a loss L, then
0 ≤ RηL(f ∗)−RηL(f ∗ η ) ≤ η BU −BL K − 1 , and − η(BU −BL) K − 1− ηK ≤ RL(f∗)−RL(f∗η ) ≤ 0,
A tighter boundBU−BL, implies a smaller worst case risk difference of the optimal classifiers (robust when BU = BL). Importantly, while L(e(i),x, f) = L(e(i), f(x)) usually, this subtle distinction is useful for losses with multiple predictive distributions, see Equation 3. In Theorem 2 in Appendix C.3, we further prove the robustness of the proposed losses to asymmetric noise.
For losses with multiple predictive distributions, the bounds in Theorem 1 and 2 must hold for any x and f , i.e., for any combination of M − 1 categorical distributions on K classes. Proposition 3 provides such bounds for GJS.
Proposition 3. GJS loss withM ≤ K+1 satisfiesBL ≤ ∑K k=1 LGJS(e(k),p(2), . . . ,p(M)) ≤ BU for all p(2), . . . ,p(M) ∈ ∆K−1, with the following bounds
BL = K∑ k=1 LGJS(e(k),u, . . . ,u), BU = K∑ k=1 LGJS(e(k), e(1), . . . , e(M−1))
where u ∈ ∆K−1 is the uniform distribution. Note the bounds for the JS loss is a special case of Proposition 3 for M = 2. Remark 1. LJS and LGJS are robust (BL = BU ) in the limit of π1 → 1.
Remark 1 is intuitive from Section 2.3 which showed that LJS is equivalent to the robust MAE in this limit and that the consistency term in Proposition 2 vanishes.
In Proposition 3, the lower bound (BL) is the same for JS and GJS. However, the upper bound (BU ) increases for more distributions, which makes JS have a tighter bound than GJS in Theorem 1 and 2. In Proposition 4, we show that JS and GJS have the same bound for the risk difference, given an assumption based on Figure 1 that the optimal classifier on clean data (f∗) is at least as consistent as the optimal classifier on noisy data (f∗η ). Proposition 4. LJS and LGJS have the same risk bounds in Theorem 1 and 2 if Ex[Lf ∗
GJSπ′′ (p(2), . . . ,p(M))] ≤ Ex[L f∗η GJSπ′′ (p(2), . . . ,p(M))], where LfGJSπ′′ (p (2), . . . ,p(M)) is
the consistency term from Proposition 2.

heading: 3 Related Works
text: Interleaved in the previous sections, we covered most-related works to us, i.e. the avenue of identification or construction of theoretically-motivated robust loss functions [2, 3, 4, 5]. These works, similar to this paper, follow the theoretical construction of Ghosh et al. [2]. Furthermore, Liu&Guo [10] use “peer prediction” to propose a new family of robust loss functions. Different to these works, here, we propose loss functions based on DJS which holds various desirable properties of those prior works while exhibiting novel ties to consistency regularization; a recent important regularization technique.
Next, we briefly cover other lines of work. A more thorough version can be found in Appendix D.
A direction, that similar to us does not alter training, reweights a loss function by confusion matrix [11, 12, 13, 14, 15]. Assuming a class-conditional noise model, loss correction is theoretically motivated and perfectly orthogonal to noise-robust losses.
Consistency regularization is a recent technique that imposes smoothness in the learnt function for semi-supervised learning [7] and recently for noisy data [16]. These works use different complex pipelines for such regularization. GJS encourages consistency in a simple way that exhibits other desirable properties for learning with noisy labels. Importantly, Jensen-Shannon-based consistency loss functions have been used to improve test-time robustness to image corruptions [6] and adversarial examples [17], which further verifies the general usefulness of GJS. In this work, we study such loss functions for a different goal: training-time label-noise robustness. In this context, our thorough analytical and empirical results are, to the best of our knowledge, novel.
Recently, loss functions with information-theoretic motivations have been proposed [18, 19]. JS, with an apparent information-theoretic interpretation, has a strong connection to those. Especially, the latter is a close concurrent work studying JS and other divergences from the family of f-divergences [20]. However, in this work, we consider a generalization to more than two distributions and study the role of π1, which they treat as a constant (π1 = 12 ). These differences lead to improved performance and novel theoretical results, e.g., Proposition 1 and 2. Lastly, another generalization of JS was recently presented by Nielsen [21], where the arithmetic mean is generalized to abstract means.

heading: 4 Experiments
text: This section, first, empirically investigates the effectiveness of the proposed losses for learning with noisy labels on synthetic (Section 4.1) and real-world noise (Section 4.2). This is followed by several experiments and ablation studies (Section 4.3) to shed light on the properties of JS and GJS through empirical substantiation of the theories and claims provided in Section 2. All these additional experiments are done on the more challenging CIFAR-100 dataset.
Experimental Setup. We use ResNet 34 and 50 for experiments on CIFAR and WebVision datasets respectively and optimize them using SGD with momentum. The complete details of the training setup can be found in Appendix A. Most importantly, we take three main measures to ensure a fair and reliable comparison throughout the experiments: 1) we reimplement all the loss functions we compare with in a single shared learning setup, 2) we use the same hyperparameter optimization budget and mechanism for all the prior works and ours, and 3) we train and evaluate five networks for individual results, where in each run the synthetic noise, network initialization, and data-order are differently randomized. The thorough analysis is evident from the higher performance of CE in our setup compared to prior works. Where possible, we report mean and standard deviation and denote the statistically-significant top performers with student t-test.

heading: 4.1 Synthetic Noise Benchmarks: CIFAR
text: Here, we evaluate the proposed loss functions on the CIFAR datasets with two types of synthetic noise: symmetric and asymmetric. For symmetric noise, the labels are, with probability η, resampled from a uniform distribution over all labels. For asymmetric noise, we follow the standard setup of Patrini et al. [22]. For CIFAR-10, the labels are modified, with probability η, as follows: truck→ automobile, bird→ airplane, cat↔ dog, and deer→ horse. For CIFAR-100, labels are, with probability η, cycled to the next sub-class of the same “super-class”, e.g. the labels of super-class “vehicles 1” are modified as follows: bicycle→ bus→ motorcycle→ pickup truck→ train→ bicycle. We compare with other noise-robust loss functions such as label smoothing (LS) [23], Bootstrap (BS) [24], Symmetric Cross-Entropy (SCE) [4], Generalized Cross-Entropy (GCE) [3], and the NCE+RCE loss of Ma et al. [5]. Here, we do not compare to methods that propose a full pipeline since, first, a conclusive comparison would require re-implementation and individual evaluation of several components and second, robust loss functions can be considered orthogonal to them.
Results. Table 1 shows the results for symmetric and asymmetric noise on CIFAR-10 and CIFAR100. GJS performs similarly or better than other methods for different noise rates, noise types, and data sets. Generally, GJS’s efficacy is more evident for the more challenging CIFAR-100 dataset. For example, on 60% uniform noise on CIFAR-100, the difference between GJS and the second best (GCE) is 4.94 percentage points, while our results on 80% noise is lower than GCE. We attribute this to the high sensitivity of the results to the hyperparameter settings in such a high-noise rate which are also generally unrealistic (WebVision has ∼20%). The performance of JS is consistently
similar to the top performance of the prior works across different noise rates, types and datasets. In Section 4.3, we substantiate the importance of the consistency term, identified in Proposition 2, when going from JS to GJS that helps with the learning dynamics and reduce the susceptibility to noise. In Appendix B.1, we provide results for GJS on instance-dependent synthetic noise [25]. Next, we test the proposed losses on a naturally-noisy dataset to see their efficacy in a real-world scenario.

heading: 4.2 Real-World Noise Benchmark: WebVision
text: WebVision v1 is a large-scale image dataset collected by crawling Flickr and Google, which resulted in an estimated 20% of noisy labels [28]. There are 2.4 million images of the same thousand classes as ILSVRC12. Here, we use a smaller version called mini WebVision [29] consisting of the first 50 classes of the Google subset. We compare CE, JS, and GJS on WebVision following the same rigorous procedure as for the synthetic noise. However, upon request by the reviewers, we also compare with the reported results of some state-of-the-art elaborate techniques. This comparison deviates from our otherwise systematic analysis.
Results. Table 2, as the common practice, reports the performances on the validation sets of WebVision and ILSVRC12 (first 50 classes). Both JS and GJS exhibit large margins with standard CE, especially for top-1 accuracy. Top-5 accuracy, due to its admissibility of wrong top predictions, can obscure the susceptibility to noise-fitting and thus indicates smaller but still significant improvements.
The two state-of-the-art methods on this dataset were DivideMix [16] and ELR+ [27]. Compared to our setup, both these methods use a stronger network (Inception-ResNet-V2 vs ResNet-50), stronger augmentations (Mixup vs color jittering) and co-train two networks. Furthermore, ELR+ uses an exponential moving average of weights and DivideMix treats clean and noisy labeled examples differently after separating them using Gaussian mixture models. Despite these differences, GJS performs as good or better in terms of top-1 accuracy on WebVision and significantly outperforms ELR+ on ILSVRC12 (70.29 vs 74.33). The importance of these differences becomes apparent as 1) the top-1 accuracy for DivideMix degrades when using ResNet-50, and 2) the performance of GJS improves by adding one of their components, i.e. the use of two networks. We train an ensemble of two independent networks with the GJS loss and average their predictions (last row of Table 2). This simple extension, which requires no change in the training code, gives significant improvements. To the best of our knowledge, this is the highest reported top-1 accuracy on WebVision and ILSVRC12 when no pre-training is used.
In Appendix B.2, we show state-of-the-art results when using GJS on two other real-world noisy datasets: ANIMAL-10N [30] and Food-101N [31].
So far, the experiments demonstrated the robustness of the proposed loss function (regarding Proposition 3) via the significant improvement of the final accuracy on noisy datasets. While this was central and informative, it is also important to investigate whether this improvement comes from the theoretical properties that were argued for JS and GJS. In what follows, we devise several such experiments, in an effort to substantiate the theoretical claims and conjectures.

heading: 4.3 Towards a Better Understanding of the Jensen-Shannon-based Loss Functions
text: Here, we study the behavior of the losses for different distribution weights π1, number of distributions M , and epochs. We also provide insights on why GJS performs better than JS.
How does π1 control the trade-off of robustness and learnability? In Figure 4, we plot the validation accuracy during training for both JS and GJS at different values of π1 and noise rates η. From Proposition 1, we expect JS to behave as CE for low values of π1 and as MAE for larger values of π1. Figure 4 (a-b) confirms this. Specifically, π1 = 0.1 learns quickly and performs well for low noise but overfits for η = 0.6 (characteristic of non-robust CE), on the other hand, π1 = 0.9 learns slowly but is robust to high noise rates (characteristic of noise-robust MAE).
In Figure 4 (c-d), we observe three qualitative improvements of GJS over JS: 1) no signs of overfitting to noise for large noise rates with low values of π1, 2) better learning dynamics for large values of π1 that otherwise learns slowly, and 3) converges to a higher validation accuracy.
How many distributions to use? Figure 5 depicts validation accuracy for varying number of distributions M . For all noise rates, we observe a performance increase going from M = 2 to M = 3. However, the performance of M > 3 depends on the noise rate. For lower noise rates, having more than three distributions can improve the performance. For higher noise rates e.g. 60%, having M > 3 degrades the performance. We hypothesise this is due to: 1) at high noise rates, there are only a few correctly labeled examples that can help guide the learning, and 2) going from M = 2 to M = 3 adds a consistency term, while M > 3 increases the importance of the consistency term in Proposition 2. Therefore, for a large enough M, the loss will find it easier to keep the consistency term low (keep predictions close to uniform as at the initialization), instead of generalizing based on the few clean examples. For simplicity, we have used M = 3 for all experiments with GJS.
Is the improvements of GJS over JS due to mean prediction or consistency? Proposition 2 decomposed GJS into a JS term with a mean prediction (p̄>1) and a consistency term operating on all distributions but the target. In Table 3, we compare the performance of JS and GJS to GJS without the consistency term, i.e., LJSπ′ (e
(y), p̄>1). The results suggest that the improvement of GJS over JS can be attributed to the consistency term.
Figure 4 (a-b) showed that JS improves the learning dynamics of MAE by blending it with CE, controlled by π1. Similarly, we see here that the consistency term also improves the learning dynamics (underfitting and convergence speed) of MAE. Interestingly, Figure 4 (c-d), shows the higher values of π1 (closer to MAE) work best for GJS, hinting that, the consistency term improves the learning dynamics of MAE so much so that the role of CE becomes less important.
Is GJS mostly helping the clean or noisy examples? To better understand the improvements of GJS over JS, we perform an ablation with different losses for clean and noisy examples, see Table 4. We observe that using GJS instead of JS improves performance in all cases. Importantly, using GJS only for the noisy examples performs significantly better than only using it for the clean examples (74.1 vs 72.9). The best result is achieved when using GJS for both clean and noisy examples but still close to the noisy-only case (74.7 vs 74.1).
How is different choices of perturbations affecting GJS? In this work, we use stochastic augmentations for A, see Appendix A.1 for details. Table 5 reports validation results on 40% symmetric and asymmetric noise on CIFAR-100 for varying types of augmentation. We observe that all methods improve their performance with stronger augmentation and that GJS achieves the best results in all cases. Also, note that we use weak augmentation for all naturally-noisy datasets (WebVision, ANIMAL-10N, and Food-101N) and still get state-of-the-art results.
How fast is the convergence? We found that some baselines (especially the robust NCE+RCE) had slow convergence. Therefore, we used 400 epochs for all methods to make sure all had time to converge properly. Table 6 shows results on 40% symmetric and asymmetric noise on CIFAR-100 when the number of epochs has been reduced by half.
Is training with the proposed losses leading to more consistent networks? Our motivation for investigating losses based on Jensen-Shannon divergence was partly due to the observation in Figure 1 that consistency and accuracy correlate when learning with CE loss. In Figure 6, we compare CE, JS, and GJS losses in terms of validation accuracy and consistency during training on CIFAR-100 with 40% symmetric noise. We find that the networks trained with JS and GJS losses are more consistent and has higher accuracy. In Appendix B.7, we report the consistency of the networks in Table 1.
Summary of experiments in the appendix. Due to space limitations, we report several important experiments in the appendix. We evaluate the effectiveness of GJS on 1) instance-dependent synthetic noise (Section B.1), and 2) real-world noisy datasets ANIMAL-10N and Food-101N (Section B.2). We also investigate the importance of 1) losses being symmetric and bounded for learning with noisy labels (Section B.3), and 2) a clean vs noisy validation set for hyperparameter selection and the effect of a single set of parameters for all noise rates (Section B.5).
Table 6: Effect of Number of Epochs. Validation accuracy for training with 200 and 400 epochs for 40% symmetric and asymmetric noise on CIFAR-100. GJS still outperforms the baselines and NCE+RCE’s performance is reduced heavily by the decrease in epochs.
Method Symmetric Asymmetric
200 400 200 400
GCE 70.3 70.8 39.1 51.7 NCE+RCE 60.0 68.5 35.0 57.5 GJS 72.9 74.8 43.2 62.6

heading: 5 Limitations & Future Directions
text: We empirically showed that the consistency of the network around noisy data degrades as it fits noise and accordingly proposed a loss based on generalized Jensen-Shannon divergence (GJS). While we empirically verified the significant role of consistency regularization in robustness to noise, we only theoretically showed the robustness (BL = BU ) of GJS at its limit (π1 → 1) where the consistency term gradually vanishes. Therefore, the main limitation is the lack of a theoretical proof of the robustness of the consistency term in Proposition 2. This is, in general, an important but understudied area, also for the literature of self- or semi-supervised learning and thus is of utmost importance for future works.
Secondly, we had an important observation that GJS with M > 3 might not perform well under high noise rates. While we have some initial conjectures, this phenomenon deserves a systematic analysis both empirically and theoretically.
Finally, a minor practical limitation is the added computations for GJS forward passes, however this applies to training time only and in all our experiments, we only use one extra prediction (M = 3).

heading: 6 Final Remarks
text: We first made two central observations that (i) robust loss functions have an underfitting issue and (ii) consistency of noise-fitting networks is significantly lower around noisy data points. Correspondingly, we proposed two loss functions, JS and GJS, based on Jensen-Shannon divergence that (i) interpolates between noise-robust MAE and fast-converging CE, and (ii) encourages consistency around training data points. This simple proposal led to state-of-the-art performance on both synthetic and real-world noise datasets even when compared to the more elaborate pipelines such as DivideMix or ELR+. Furthermore, we discussed their robustness within the theoretical construction of Ghosh et al. [2]. By drawing further connections to other seminal loss functions such as CE, MAE, GCE, and consistency regularization, we uncovered other desirable or informative properties. We further empirically studied different aspects of the losses that corroborate various theoretical properties.
Overall, we believe the paper provides informative theoretical and empirical evidence for the usefulness of two simple and novel JS divergence-based loss functions for learning under noisy data that achieve state-of-the-art results. At the same time, it opens interesting future directions.
Ethical Considerations. Considerable resources are needed to create labeled data sets due to the burden of manual labeling process. Thus, the creators of large annotated datasets are mostly limited to well-funded companies and academic institutions. In that sense, developing robust methods against label noise enables less affluent organizations or individuals to benefit from labeled datasets since imperfect or automatic labeling can be used instead. On the other hand, proliferation of such harvested datasets can increase privacy concerns arising from redistribution and malicious use.
Acknowledgement. This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.

Review summaries:
This paper considers learning with noisy labels.  The author showed that the Jensen-Shannon (JS) divergence sits in between the mean absolute error (MSE) and the cross entropy (CS).  The key proposition is to use the generalized JS (GJS) as the loss, which presents a trade-off between robustness with respect to the noisy labels and to avoid under-fitting from which robust loss function (such as the MSE) suffers.

$\textbf{Summary and main contributions:}$  (1) The authors propose to use Jensen-Shannon divergence (JS) to learn with noisy labels, which is shown to interpolate between Cross-Entropy and MAE loss through a hyper-parameter.  (2) Based on the new observation that learning with noisy labels reduces the classifier's prediction consistency for noisy-labeled sets. To overcome this issue, the authors introduce multi-distribution generalization (GJS) which is shown to generalize JS with a consistency regularizer.  (3) Experiment on synthetic noisy labeled datasets and a real-world noisy dataset (WebVision) demonstrate the effectiveness of proposed methods. 

The paper proposed to use JS divergence as a loss function (JS) to address the underfit problem of provably-robust loss functions. Further, the authors generalize the JS loss function (GJS) to include multiple distributions to encourage consistency during learning with noisy labels. They demonstrate that JS interpolates MAS and CE loss, and they proved the noisy robustness property of the JS and GJS. Extensive experiments are provided to show the effectiveness of the proposed loss functions.