Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper considers learning with noisy labels.  The author showed that the
Jensen-Shannon (JS) divergence sits in between the mean absolute error (MSE) and
the cross entropy (CS).  The key proposition is to use the generalized JS (GJS)
as the loss, which presents a trade-off between robustness with respect to the
noisy labels and to avoid under-fitting from which robust loss function (such as
the MSE) suffers.
main_review: Overall, this contribution has some theoretical analysis in section 2,
as well as experimental results on CIFAR/WebVision showing that the proposed GJS loss
leads to more accurate classification under the setting of noisy labels.
Based on these contributions, I tend to vote for acceptance.

Essentially, based on the experimental results, the most useful tool proposed 
by the authors is the GJS (rather than JS) divergence as a robust loss. Based on 
proposition 2, the loss has two terms. Minimizing the first term leads to 
learnability, as it tries to align the target (noisy) label with the centroid
of the predictions; the second term leads to robustness, as it measures
the deviation of the predictions from their centroid.

The authors provided the recipe to choose the \pi factor,
which is important to trade-off between learnability and robustness and to make the experiments reproducible.

Theoretically, the main argument in section 2.5 is that the proposed GJS loss
is (trivially) bounded on both sides, and any bounded loss satisfies
the property stated in theorem 1: the deviation of the empirical risk 
with respect to the label noise can be bounded. Note the GJS loss is
only robust at its extreme point when pi tends to be deterministic,
as "robustness" requires the deviation to be absolute zero.

These theoretical results can be improved by considering the parametric
family of the perturbation A(\eta), and specific label noise, which is not considered in this
paper. Is the GJS loss fitted for any noise labels and noisy perturbations
of the predictions? How are these two difference noise related to affect the learning?
Can we define \epsilon-robustness and bound the robustness
under the GJS loss? In general, the presented theoretical results
lack significance and can be enhanced.

Most of the experimental evaluation is based on the classification accuracy.
There should be some consistency measurements as shown in figure 1
(I trust the GJS should perform well because of its 2nd regularization term).

Minor comments:

L70
"Assuming causality, suggests that" -> please rewrite the sentence

L74: is $\Delta$ introduced before?

L78: "... is another crucial difference" with what?

L102: mention GCE, SCE are shorthand for ...

L107 and Figure 2: {L}_{JS} instead of JS (JS = 0 is $\pi$ goes to extreme)

L223: come from -> comes from

_____
**Post Rebuttal remarks**

The reviewer thanks the authors for the response, and for agreeing on the review and several extensions, and addressing all minor comments. The disagreement on the significance is more on whether this type of contribution on applying existing tools from information theory into current machine learning problems are interesting enough for NeurIPS.  Ideally, the statements can use the structure of the JS divergence, which will improve the significance. For example, section 2.5 only use the fact that the JS divergence is bounded. These aspects can definitely be improved.

Besides the current review, in preparing the next version (whether it is accepted or not), the authors are suggested to include proper reference (I leave the authors to do the literature review) on the following:
- The definition of GJS and other generalizations of JS;
- Existing analysis on the limit case of GJS;
- Existing applications of JS/GJS into machine learning/pattern recognition problems, and as loss functions.
limitations_and_societal_impact: NA
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: $\textbf{Summary and main contributions:}$

(1) The authors propose to use Jensen-Shannon divergence (JS) to learn with noisy labels, which is shown to interpolate between Cross-Entropy and MAE loss through a hyper-parameter.

(2) Based on the new observation that learning with noisy labels reduces the classifier's prediction consistency for noisy-labeled sets. To overcome this issue, the authors introduce multi-distribution generalization (GJS) which is shown to generalize JS with a consistency regularizer.

(3) Experiment on synthetic noisy labeled datasets and a real-world noisy dataset (WebVision) demonstrate the effectiveness of proposed methods.

main_review: $\textbf{Originality:}$ the authors introduce two novel robust methods to deal with label noise and alleviate the issue that "a network's prediction consistency got reduced for noisy-labeled data".

$\textbf{Quality:}$ the paper is overall well-written, although there are some additional issues, for example, 

$\textbf{Clarity:}$ some notations and technical terms are not well-defined, also some experiment details are missed.

$\textbf{Additional concerns/suggestions:}$

(1) Some technical terms require additional explanations, for example, in Line 16-17, readers may not be familiar with "provably-robust loss functions" and "consistency". The presentation would be much better if the authors explain a bit more about these two terms since these two observations convey some important information about this work.

(2) Line 26, a typo: "labelled" -> "labeled".

(3) In the caption of Figure 1, it would be more accurate if the authors write "... as it overfits to label noise using CE loss". Besides, in Figure 1 (a), is the validation accuracy evaluated w.r.t. a clean or noisy validation set. I assumed that under these four levels of label noise, the learning rate settings are the same. For the plot w.r.t. relative large noise rate such as 60%, will a lower learning rate (after 200 epochs) avoids the corresponding classifier degrades so much? Although I believe that the order of performance will not change, the gap might not be so large?

(4) In Line 43, the notation of simplex $\Delta^{K-1}$ is not defined.

(5) In Line 49, it would be better if the authors firstly introduce a common real-world noise model where $\eta$ is usually instance-dependent, e.g., $\eta(x, y)$ for each instance/feature $x$ and class $y$. Then, state that in this work, the authors are interested in the instance-independent noise model where $\eta(x)=\eta$ and $\eta$ is the same for each class for a theoretical perspective. 

(6) The explanation w.r.t. Figure 1 is delayed to Line 63, it might be better if the authors move this in the introduction section?

(7) In Line 77, a formal statement of the properties w.r.t. Jenson Shannon divergence is suggested.

(8) I suggest the authors explain more about the importance of the property "bounded" and "symmetric".

(9) The $m_{>1}$ in Proposition 2 feels like a weird notation since readers may find it hard to get the lingual relationship between $m$ and the weighted $\mathbf{p}^{(j)}$. $\bar{\mathbf{p}}_{>1}$ or another notation might make the presentation better?

(10) In Theorem 1, I feel like the requirement of $\forall x, f$ will somehow make $B_U-B_L$ much larger. Do authors think this Theorem 1 can be stated in this view: with probability at least $1-\delta$, we have $\mathbb{P}(B_L\leq \sum_{i=1}^K \mathcal{L}(e^{(i)}, x, f)\leq B_U)\geq 1-\epsilon.$ 
Filtering out extreme values in this view, the gap between $B_L$ and $B_U$ appear not so large. I think the proofs are not influenced with probability at least $1-\delta$.

(11) I didn't find any definition of $\mathbf{u}$ in Equ. (6). 

(12)  In [1], the authors showed the robustness of the variation-form f-div functions w.r.t. two properly defined distributions (joint distribution and the product of marginal distributions via model prediction and label), where Jensen Shannon is a special case. It would be better if authors could discuss more relationships between these two methods (JS in this work and [1]) in the main paper, or with the experiment comparison.

(13) Experiment details: do authors use the same synthetic noise dataset for all methods? Or generate the noisy labels with a random seed for each training. If it is the latter case, do authors fix the random seed?

(14) In Line 220, the comparison with DivideMix [2] and ELR+ [3] are not reasonable. Although [2] and [3] adopt a stronger network, different architectures may result in different performances. For example, my previous observations show that Pre-Resnet 18 outperforms Resnet 34 in certain noisy label synthetic tasks, while the latter one has a stronger network.

$\textbf{Reference:}$

[1] Wei, Jiaheng, and Yang Liu. "When Optimizing f-Divergence is Robust with Label Noise." In International Conference on Learning Representations. 2020.

[2] Li, Junnan, Richard Socher, and Steven CH Hoi. "DivideMix: Learning with Noisy Labels as Semi-supervised Learning." In International Conference on Learning Representations. 2019.

[3] Liu, Sheng, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. "Early-Learning Regularization Prevents Memorization of Noisy Labels." Advances in Neural Information Processing Systems 33 (2020).

##############################################################

$\textbf{Post-Rebuttal}$

Thank authors for addressing all my concerns. And after the rebuttal, I decide to raise the rating from 6 to 7.
limitations_and_societal_impact: Yes, the authors have discussed their limitations in Section 5. I listed constructive suggestions above "Main Review".
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposed to use JS divergence as a loss function (JS) to address the underfit problem of provably-robust loss functions. Further, the authors generalize the JS loss function (GJS) to include multiple distributions to encourage consistency during learning with noisy labels. They demonstrate that JS interpolates MAS and CE loss, and they proved the noisy robustness property of the JS and GJS. Extensive experiments are provided to show the effectiveness of the proposed loss functions. 
main_review: The paper is well-written and clearly conveys the authors' thesis.

pros:
1. observing the significant consistency reduction around noisy data samples when the model overfits noise is of theoretical interest and practical importance.
2. the authors proved the noise robustness properties of the proposed novel loss functions, JS and GJS. They further provided a detailed investigation and ablation study to support their theoretical claims and conjectures.
4. they carried out extensive experiments and demonstrated the proposed loss functions could achieve state-of-the-art performance under symmetric and asymmetric label noise.

cons/questions:
1. the consistency regularizer may lead the network to learn a more smooth prediction function. This is similar to the effects of some adversarial defense methods. I wonder if the GJS will lead to better adversarial robustness? 
2. the decrease of consistency when overfitting to noise is an important phenomenon. One major hypothesis of this work is that there is a causal relationship between clean validation accuracy and consistency. It would be better to provide consistency plots for JS and GJS in addition to those validation plots. 
3. In Figure 5, we see the validation accuracy decrease when M>3 for 60% noise rate case. This observation has not been adequately discussed. Adding more distributions would put more emphasis on consistency. Would it be the case that under heavy noise rate, there are not enough anchors to guide the model and balance the force of encouraging consistency? 
4. it will be better to include experiments on instance-based label noise. In that case, the label noise is related to the feature and may introduce "fake consistency". It is of interest to see the performance of GJS under such a setting.
limitations_and_societal_impact: Limitations and societal impact have been addressed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper targets at learning with noisy labels and propose two loss functions: JS and GJS. To justify the loss functions, robustness and consistency properties are discussed.
main_review: **Pros**

-	The theoretical analysis on JS and GJS are clear. The justifications on robustness and consistency make sense.
-	There are ablation studies showing the effect of each component.


**Cons**

*Mythology*

-	The contribution is minor compared with previous works. 1) For the main claims of robustness and underfitting, JS shares the same properties as GCE. 2) The superior of GJS over JS is the consistency property. GJS is equivalent to CE plus a regularization term that encourages low KL between predictions on perturbed inputs. Such a regularization term is widely known in the community: it has long been used in semi-supervised learning, noisy label learning and adversial training. We should be aware that directly introducing such a consistency regularization on any existing methods may improve the performance in learning with noisy labels by a large margin (I have verified this in experiments and the author could conduct experiments if interested). Considering that the consistency regularization is known, the contribution of this paper is minor.
-	The observation on network predictions’ consistency is not novel as claimed in the main contributions. It is straightforward for random noise because the labels are flipped to different classes for similar samples (neighbor samples in some sense) and the input spaces are divided into much more decision regions compared with training on clean. Actually, this has been demonstrated in [Aprit et al. ICML 2017] (please refer to the Critical Sample Ratio which is an estimation of the density of decision boundaries). Naturally, the observation on consistency should be less significant for class-dependent noise and more realistic instance-dependent noise.
-	In this direction, the line of works that study robustness mostly based on the assumption that the label noise is class-conditional (even random as assumed in Theorem 1 in this paper). However, both intuition and rigorous hypothesis testing [Zhang et al. ICLR 2021] imply that real-world noise should be instance dependent.

*Experiments*
-	The empirical results are not significant on more realistic asymmetric noise and real-world noise. This is consistent with the above concerns that the theoretical justification is based on strong assumptions and the consistency observation is most significant for random noise.
-	The author does not compare to methods that propose a full pipeline and claim that the reason is robust loss functions can be considered complementary to existing methods. However, firstly, there are many strong yet simple baselines (sample selection, label correction methods) that are necessary to compare on large-scale experiments on CIFAR-10 and CIFAR-100. Secondly, the paper should demonstrate an improvement by introducing GJS on existing methods since it is claimed as complementary.


[Aprit et al. ICML 2017] A Closer Look at Memorization in Deep Networks.
[Zhang et al. ICLR 2021] Learning with Feature-Dependent Label Noise: A Progressive Approach.


*post rebuttal*

Thanks for the detailed feedback. My concerns are well addressed.
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The reviewers deemed the paper of interest. I would like to thank the authors for providing focused discussion content for both theoretical and experiments, which contributed to explain further the paper's content and its improvements with respect to its submission history.

In the revised version, the authors should include the generalisation to Theorem 1 proposed at discussion phase to cover asymetric noise and in experiments, find a place to report consistency and instance-based label noise, a short discussion vs Wei + Liu's approach as developed in the discussion (point (12) 6kDX) and complete references to be more extensive. This is extremely important given the context of the paper and the richness of the relevant literature in the past decade or so.

AC.
Decision: Accept (Poster)