Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies label smoothing when learning with noisy label. It proposed generalized label smoothing (GLS) containing positive label smoothing (PLS) and negative label smoothing (NLS), where NLS allows the smoothing parameter to be negative. The authors found that when the noisy rate is high, NLS is more beneficial than PLS, both theoretically and empirically.
main_review: The authors proposed a very simple yet interesting idea NLS. They theoretically shown NLS can improve the expected model confidence. When label noise rates are high, they shown NLS can be more robust to label noise compared with PLS. Then they demonstrated the theoretical findings by conducting experiments on UCI and CIFAR datasets. The paper is easy to understand and well-organized. It is very good to provide a section about how the proposed GLS connects to other robust methods. However, I have several questions/comments that need the authors to address. 

(a). Since the authors didn't provide any experiments to compare GLS with other loss adjusted methods of learning with noisy label such as [1,2], we have no idea about the practicality of GLS and its practical performance when learning with noisy label. It would be better if the authors could compare GLS with other methods such as [1,2]. Otherwise, its practicality is limited.

(b). Although the authors have shown theoretical results, the unknown parameters (for example, $r^*$ and $e$ in Theorem 7) makes the proposed GLD less practical. That is to say, in general, we don't know how to use GLS in practice.

(c). In the proof of theorem 5 (on page 28), why the inequality (see below) after "Besides, we have ..." holds? It is not obvious to me. 
\begin{equation*}
\mathbb{E}   \left[\log\left(\frac{f^*_r(X)_Y}{1-f^*_r(X)_Y}\right)^{r/2} - \log\left(f^*_r(X)_Y\right)\right] \\
\ge \mathbb{E} \left[\log\left(\frac{f^*_s(X)_Y}{1-f^*_s(X)_Y}\right)^{|r|/2} - \log\left(f^*_s(X)_Y\right)\right] 
\end{equation*}

where $\mathbb{E}[\cdot] := \mathbb{E}_{(X, Y)\sim \mathcal D}[\cdot], r:=r_N, s:=r_P$

(d). Is the quantity model confidence commonly used in the literature? If so, please add a reference.

Reference

1. Symmetric cross entropy for robust learning with noisy labels, ICCV 2019.

2. Early-Learning Regularization Prevents Memorization of Noisy Labels, NeurIPS 2020.

========== comments after rebuttal ============

I would like to thank the authors for their response, which solves my concerns except for the proof of Theorem 5 (of the initial version). The result of original Theorem 5 is important to the contribution of this paper, which unfortunately was removed in current version. In addition, after reading other reviewers' comments, there are still some other problems. Therefore, I will keep my overall score.
summary_of_the_review: This is a theoretical understanding paper for lable smoothing when learning with noisy label. However, its practicality is limited.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper considers generalized label smoothing (GLS) for binary classification tasks where the smoothing coefficient is allowed to be negative (negative label smoothing, NLS). The authors show the connection between GLS and several existing loss functions through elementary algebraic manipulations of the expected loss under (generalized) label smoothing. They also analyze GLS in the noisy label setting where the labels are randomly flipped and show that using negative label smoothing is desirable in the high label noise setting.
main_review: - The major problem with the paper is analyzing the expected loss. Label smoothing is a regularizer which has been shown to improve generalization and model calibration. However, in the infinite sample limit (minimizing the expected loss) it makes no sense to add any regularizer since directly minimizing the expected loss is optimal, e.g., In eq. 4 the authors analyze the expected loss (under the data distribution) subject to a label smoothing penalty. Why does this make sense ?
- The authors claim that negative label smoothing is different from standard label smoothing and present it as a significant result. However, this is trivial. The label-smoothed cross entropy loss can be equivalently written in the constrained form as:
    
    $$\min l(f(x),y) \quad s.t.~ H(U, f(x)) \leq c$$
    
    where U is the discrete uniform distribution and $l(.)$ is the log-likelihood. For negative label smoothing the inequality constrained is reversed. By virtue of the constraints, the minimizer of NLS objective will have greater expected margin (or model confidence as called in the paper) than standard LS which pushes the model predicted class distributions to be closer to uniform.
    
- The claim that negative label smoothing performs well in the high label noise regime is not surprising both from a theoretical and practical standpoint. The point of using label smoothing is to prevent the distribution learned by neural networks to become too peaked (low entropy or overconfident). But when the label noise is high, with labels flipped randomly, then the model trained without any label smoothing is not going to be overconfident (the expected loss of the model is at least the Bayes error and with label noise the Bayes error is already high).
- The paper doesn't position itself clearly with respect to related work. Chen et al 2020 have already studied generalization performance under label smoothing where, unlike this paper, they study the generalization performance of the finite sample minimizer under label smoothing and label noise. While that paper is cited, the results of this paper are not put in context with their results.

## Other comments

- In Theorem 3, what does it mean for the smoothing rate $r_{CL}$ to approach infinity ? In this case the likelihood term is completely ignored.
- High label noise is never clearly defined.
summary_of_the_review: The main problem with the paper is analyzing the effect of label smoothing on the minimizer of the expected loss of a classifier. There is no need for any regularization when minimizing the expected loss. This is a fundamental conceptual problem with the paper. Furthermore, the results are fairly elementary and do not significantly contribute towards theoretical understanding of label smoothing beyond existing literature (c.f. Chen et al 2020).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Label smoothing shows a promising bias-variance tradeoff when training deep neural networks, while it is unclear if label smoothing remains helpful when training labels are noisy. A recent work shows the benefits of using label smoothing. However, the current submission finds contradicting and appealing observations. In Figure 1, the authors presented a convincing set of results that clearly show a phase-transitioning behavior of using label smoothing: while label smoothing seems to work with a low label noise rate, the benefits disappear when the rate goes up high.

The above observation can imply that label smoothing is considered unfavorable with a high label noise rate. The paper then presented a generalized label smoothing scheme where the smoothing constant can instead go negative. Even though using a negative parameter sounds weird, the paper showed that a label smoothing with a negative (but different) smoothing parameter corresponds to several existing solutions to learning with noisy labels! This signals that the negative label smoothing scheme might turn out to be helpful when dealing with high label noise rate. I think this is a pretty interesting observation and connection to uncover.

The two cases are integrated as the generalized label smoothing, where the smoothing constant can be either negative to positive. In section 5, it is shown under what noise rate, a positive or a negative label smoothing is considered a better option. This characterization reveals a clear phase transitioning behavior corresponding to figure 1.
main_review: Strength:

The paper is very well motivated, and revealed new findings for the power of label smoothing. The paper provided empirical instructions for when a negative or a positive smoothing rate should be adopted.

The proposed generalized label smoothing scheme ties closely to several existing learning with noisy labels solutions. This new connection might reveal a deep connection between label smoothing and other regularization techniques in the literature of learning with noisy labels. This seems to provide a set of new understandings to the literature of learning with noisy labels. 

The discovered phase transitioning behavior is supported both theoretically and empirically. This result quantifies the scenarios of using positive and negative label smoothing.

The presented experiment results are thorough and complete. The analysis is well presented. 

Weakness & question:

The discussion in Section 4 on the bias-variance tradeoff seems to disconnect a bit with Section 5. Could the authors clarify its use?

The characterization of the phase transitioning behavior seems to depend on the knowledge of r^*, the optimal smooth rate when data is clean. This knowledge is important to inform when one should adopt a negative smoothing rate and when a positive one. What will happen when only an imperfect r^* is known? 


summary_of_the_review: I tend to accept this paper. It does explain some interesting things, and the experiment is also sufficient.

========== post rebuttal ============

The rebuttal of this article solves my concerns to some extent, but as other reviewers pointed out, some problems still exist, so I changed my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper provides understandings for a generalized notion of label smoothing (GLS) when learning with noisy labels and theoretically show that negative label smoothing improves the expected model confidence over the data distribution. 
main_review: This paper provides understandings for a generalized notion of label smoothing (GLS) when learning with noisy labels and theoretically show that negative label smoothing improves the expected model confidence over the data distribution. The empirical experiments on multiple benchmark datasets demonstrate that with the presence of label noise, negative label smoothing (NLS) becomes competitively robust to label noise.

However, to the best of my knowledge, the following points need to be addressed before it can be published:

1)	My first concern is the originality and novelty of this work. In this work distributed soft label learning is used. To my knowledge, label distribution learning is a related research topic about soft label-based multi-label learning, which should be discussed.
2)	For Figure 1, more detailed descriptions and explanations should be given.
3)	The last paragraph in Section Introduction, please don't cite so many references at the same place. I suggest to delete some unimportant references and then dispersedly cite these references. 
4)	Two UCI datasets are used to evaluate the effectiveness of the proposed algorithm. My question is why the authors choose these datasets. To my knowledge, there are a large number of UCI datasets can be used for evaluating the effectiveness of the proposed algorithm. More datasets may be needed. At least, please explain the reason why these datasets are used in the current experiments.
5)	It is more convinced if the authors can compare the proposed algorithm with some other state-of-the-art label distribution learning algorithms.
6)	With regard to the comparison results, statistical tests are needed in the comparison results.
7)	The presentation can be further polished. For example, some equations have punctuation at the end and some don't have.

summary_of_the_review: This paper provides understandings for a generalized notion of label smoothing (GLS) when learning with noisy labels and theoretically show that negative label smoothing improves the expected model confidence over the data distribution. The empirical experiments on multiple benchmark datasets demonstrate that with the presence of label noise, negative label smoothing (NLS) becomes competitively robust to label noise.

However, to the best of my knowledge, the following points need to be addressed before it can be published:

1)	My first concern is the originality and novelty of this work. In this work distributed soft label learning is used. To my knowledge, label distribution learning is a related research topic about soft label-based multi-label learning, which should be discussed.
2)	For Figure 1, more detailed descriptions and explanations should be given.
3)	The last paragraph in Section Introduction, please don't cite so many references at the same place. I suggest to delete some unimportant references and then dispersedly cite these references. 
4)	Two UCI datasets are used to evaluate the effectiveness of the proposed algorithm. My question is why the authors choose these datasets. To my knowledge, there are a large number of UCI datasets can be used for evaluating the effectiveness of the proposed algorithm. More datasets may be needed. At least, please explain the reason why these datasets are used in the current experiments.
5)	It is more convinced if the authors can compare the proposed algorithm with some other state-of-the-art label distribution learning algorithms.
6)	With regard to the comparison results, statistical tests are needed in the comparison results.
7)	The presentation can be further polished. For example, some equations have punctuation at the end and some don't have.
review_title: null

Meta-review: This paper studies generalized label smoothing (GLS), which unifies positive label smoothing (PLS) and negative label smoothing (NLS), and studies its connections to existing loss functions. It also shows the benefit of NLS in the high noise regime.

Although the reviewers acknowledge that the idea of NLS in this paper is interesting, they also expressed the concerns that: the practicality of GLS is not thoroughly evaluated against prior works; the empirically best setting of parameter r is only verified in a limited number of datasets; the theoretical results' difference with prior works is limited. We encourage the authors to take the reviewers' feedback to strengthen the paper in the next iteration.
Decision: Reject