$\epsilon$-Softmax: Approximating One-Hot Vectors for Mitigating Label Noise

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-02 07:20:53

Abstract: Noisy labels pose a common challenge for training accurate deep neural networks. To mitigate label noise, prior studies have proposed various robust loss functions to achieve noise tolerance in the presence of label noise, particularly symmetric losses. However, they usually suffer from the underfitting issue due to the overly strict symmetric condition. In this work, we propose a simple yet effective approach for relaxing the symmetric condition, namely **$\epsilon$-softmax**, which simply modifies the outputs of the softmax layer to approximate one-hot vectors with a controllable error $\epsilon$. Essentially, ***$\epsilon$-softmax** not only acts as an alternative for the softmax layer, but also implicitly plays the crucial role in modifying the loss function.* We  prove theoretically that **$\epsilon$-softmax** can achieve noise-tolerant learning with controllable excess risk bound for almost any loss function. Recognizing that **$\epsilon$-softmax**-enhanced losses may slightly reduce fitting ability on clean datasets, we further incorporate them with one symmetric loss, thereby achieving a better trade-off between robustness and effective learning. Extensive experiments demonstrate the superiority of our method in mitigating synthetic and real-world label noise.

Main paper: 
  
 Appendix:  


Decision: All reviewers provided positive recommendations (including 3 borderline accepts) for this paper. All reviewer recognized that the proposed method is simple and effective. They also suggested several aspects where the paper needs improvement, including hyperparameter sensitivity analysis, comparison with related works, and some ablation studies. In particular, Reviewer 4jLX pointed out the math should be rigorously defined and explained, which should be improved in the final version. Additionally, it might be better for the authors to add comparison or discussion with those most-related methods that transform the labels, logits or softmax outputs, such as label smoothing [1] and logit clipping [2].  

[1] Lukasik, Michal, et al. "Does label smoothing mitigate label noise?." International Conference on Machine Learning. PMLR, 2020.

[2] Wei, Hongxin, et al. "Mitigating memorization of noisy labels by clipping the model prediction." International Conference on Machine Learning. PMLR, 2023.

Reviews: review: paper_summary: paper_summary: The author proposes the epsilon-softmax technique as a method to address label noise. Epsilon-softmax facilitates peaky predictions by increasing the value of the highest prediction, and it also functions to reduce the magnitude of the gradient when the prediction aligns with the given label. The author introduces the concept of All-k consistency to interpret this paradigm and presents experiments on prominent real-world benchmark datasets in the field of label noise learning, specifically WebVision and Clothing1M.
strength_weakness: strength_weakness: This section addresses two major concerns. For minor concerns, please refer to the "Questions" part.
1. The author mentions in line 40 the necessity for a method that can achieve both effective learning and robustness. While the proposed method offers a different perspective compared to symmetric-like loss methods, it is challenging to assert that it fully meets this necessity. Ironically, to balance the trade-off between effective learning and robustness, the author combines CE_(epsilon) loss and MAE loss. This ability to manage trade-offs is also found in other symmetric-like loss-based methods. In this context, I am interested in understanding why the proposed method might offer a better trade-off and whether it truly provides a better trade-off. I attempted to verify this through experimental comparison, but several issues arose: (1) There are no experiments that allow for a comparison of trade-offs. Experiments demonstrating the trade-off by varying alpha and beta are necessary. (2) The performance of existing methods is reported to be lower. For example, refer to the SCE paper.
2. The proposed method introduces two additional hyperparameters: m and alpha / beta. Unfortunately, based on the recorded experimental results, the proposed method appears to be sensitive to these hyperparameters. If this is not true, providing comprehensive experimental results that show the effects of varying these hyperparameters would enhance the perceived value of the proposed method. 
And if the proposed method is indeed sensitive to changes in hyperparameters, I would like to see evidence that it is not sensitive to hyperparameter variations within the in-distribution domain. I recommend performing validation and test processes to identify optimal hyperparameters (refer to the processes outlined in the GJS and JS papers).
questions: questions: 1. Based on the gradient analysis, it appears that when m is sufficiently large, the sensitivity of performance to m would not be significant. However, as shown in Table 3, the difference in performance is quite notable. Does the author have an explanation for this discrepancy? Additionally, has the author investigated the results when m is infinite, meaning the CE loss is not used when the prediction matches the label?
2. Has the author ever checked the results of using only the CE_(epslion) loss function? Sections 3.1 to 3.3 and Lemma 2 suggest the importance of the single CE_(epslion) loss function.
limitations: limitations: -
paper_summary: paper_summary: The paper introduces “ϵ-softmax,” a method to adjust softmax outputs for better approximation to one-hot vectors, thereby mitigating the impact of label noise in classification tasks. The approach modifies the softmax layer outputs to include a controllable error term ϵ, aiming to improve noise robustness without extensive alteration to the network architecture. The authors provide theoretical backing for the effectiveness of ϵ-softmax in achieving noise-tolerant learning across various loss functions. Extensive experiments with both synthetic and real-world noisy datasets are conducted to validate the claims.
strength_weakness: strength_weakness: This paper should pay attention to the axis labels of its figures. In Figure 1, the x-label is Epoch and y-label is Test Accuracy. In Figures 2 and 3, the axis labels are missing.
questions: questions: 1.This paper seems to focus on classification tasks. Does ε-Softmax also work well for regression tasks?

2.Is ε-Softmax computationally efficient in terms of training time compared with baseline methods?

3.This paper shows the excellent performance of ε-Softmax for label noise. Does ε-Softmax work for input (features) noise as well?

4.I assume that in this paper, for the label noise models, both training and testing labels are noisy. I am curious about the performance when the training labels are clean and the testing labels are noisy.
limitations: limitations: The authors have adequately addressed the limitations.
paper_summary: paper_summary: This manuscript proposes a novel method to approximate the symmetric condition of the loss function, which is necessary for robustness to label noise. Specifically, the proposed method, named \\( \\epsilon \\)-softmax, can adjust the model output to approximate one-hot vector. However, the proposed method alone suffers from underfitting, so the authors combined it with MAE to achieve better performance. The authors evaluated the proposed method on datasets with different noise types and rates.
strength_weakness: strength_weakness: 1. The theoretical discussion with temperature-dependent softmax is missing. As the authors mentioned in L42 to L53, there are other output restriction-based methods in the literature, such as temperature-dependent softmax. Although the authors claim that these methods “lack predictability, fail to achieve a quantitative approximation to one-hot vectors, and exhibit limited effectiveness,” there is no detailed discussion on why the proposed \\( \\epsilon \\)-softmax has superior properties.
2. A direct comparison with sparse regularization [1] is missing. Sparse regularization utilizes temperature- dependent softmax, which this manuscript has already compared, to approximate one-hot vector output. However, sparse regularization also employs an additional regularization term, \\( \\ell_p \\)-norm \\( \\| p(\\cdot | x) \\|^p_p \\) to enhance performance, and this regularization term is equivalent to MAE only if \\( p = 1 \\). It’s necessary to highlight the advantages of the proposed method compared to this highly relevant approach.
3. There is no ablation study on \\( \\alpha \\) and \\( \\beta \\). As the authors mentioned in L218, \\( \\epsilon \\)-softmax alone suffers from a loss in fitting ability, and they combined it with MAE to balance the robustness and effective learning. However, without the relevant ablation study, it’s unclear how this “trade-off” is achieved.
4. The theoretical discussions and experiments regarding instance-dependent label noise are overlooked. In recent years, the instance-dependent label noise has attracted increasing attention [2,3,4]. Experimenting the proposed method on instance-dependent label noise can provide a better understanding of how the proposed method performs with different types of label noise. I encourage the authors to include related discussion in the revised manuscript.

[1] Learning with noisy labels via sparse regularization, ICCV, 2021.

[2] Part-dependent Label Noise: Towards Instance-dependent Label Noise, NeurIPS, 2020.

[3] Learning with Instance-Dependent Label Noise: A Sample Sieve Approach, ICLR, 2021.

[4] Instance-Dependent Label-Noise Learning With Manifold-Regularized Transition Matrix Estimation, CVPR, 2022.
questions: questions: 1. Is \\( \\epsilon \\)-softmax + MAE still All-\\( k \\) calibrated and All-\\( k \\) consistency?
2. Can the proposed method perform better compared to sparse regularization?
limitations: limitations: The authors have acknowledged the limitations and societal impact of their work.
paper_summary: paper_summary: This submission proposes a enhanced softmax layer for label-noise learning, namely $\epsilon$-softmax. By incorporating with the well-known $\epsilon$-relaxation, the proposed $\epsilon$-softmax can regularize the outputs of the model and avoid fitting the label-noise sample. This simple and plug-and-play method theoretically bounds the output logits to be an approximated one-hot vector. Extensive experiments demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: - The ablation studies on the gradient clipping should be conducted and providing experimental results with different backbones would be better.
- It is exhaustive and labor-expensive to find the optimal $m$ for diverse datasets.
questions: questions: - Why choose the MAE as the loss to incorporate with ${CE_\epsilon}$? Following GCE and SCE, the MAE is indeed a practical and evaluated choice. Do there have any alternatives?
- What is the connection or relationship between logit adjustment and $\epsilon$-softmax?
limitations: limitations: No obvious limitations
paper_summary: paper_summary: This paper proposes $\epsilon$-softmax to deal with label noise.  $\epsilon$-softmax modifies the outputs of the softmax layer to approximate one-hot vectors with a controllable error $\epsilon$. Both theoretical and empirical studies show the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. I think the motivation or the underlying reason for the effectiveness needs further explanation.
2. In experiment, the advantage of the proposed method over the competitors is probably not statistically significant.
questions: questions: 1. The term "symmetric condition" in abstract needs further explanation.
2. In the implementation steps in Line 61, is $\mathbf{p}(\cdot)$ the same as ${p}(\cdot)$? Or what is the relationship between these two notations? I think the mathematical notations should be strictly used and defined.
3. The robustness of the proposed $\epsilon$-softmax loss is theoretically justified. However, I'm curious to know the insight, or the underlying reason for its robustness. The authors wrote that "The distinctive attribute of $\epsilon$-softmax lies in its guarantee to possess a controllable approximation error $\epsilon$ to one-hot vectors, thus achieving perfect constraint for the hypothesis class." But, I cannot figure out why controlling approximation error $\epsilon$ to one-hot vectors and achieving perfect constraint for the hypothesis class are useful for handling label noise? What is the direct reason? It would be better if the authors can provide some intuitive explanations.
4. From the experiments, I note that the performance is a bit sensitive to different selection of m. Therefore, is it possible to give some guidance in choosing m for practical use? Besides, will better performance be obtained if we add the $\epsilon$-softmax loss functions with different m?
5. From the experimental results in Table 2, 4, I can see that the improvement of the proposed method over other methods is quite marginal. I guess if we do statistical significance test, maybe such improvement will not be statistically significant.
limitations: limitations: I think this work will not have negative social impacts.

Comments: title: Thanks for your feedback
Thanks for your feedback!
Thanks for your feedback
Thanks for your feedback
Thanks for the rebuttal
Thanks for your feedback
Response to Rebuttal
comment: Dear Reviewer  5cpw,

Thanks for approving our responses! We will  incorporate the additional experimental results in the final version. Your guidance has been instrumental in enhancing the quality of our work.

Best regards,

The authors
Thank you for your additional interpretation and experimentation. My main concerns have been largely addressed. The answer to Q1 provides a good guideline regarding the size of m, and the answer to Q2 presents results that reflect a fundamental extension of the epsilon-softmax method. Integrating these points into the main text will likely enrich the overall content. I will maintain my approval decision.
Dear Reviewers,

The deadline of reviewer-authors discussion is approaching. If you have not done so already, please check the rebuttal and provide your response at your earliest convenience.

Best wishes,

AC
Dear Reviewer MvPF

Thanks for your feedback! We are pleased to address your concerns and greatly appreciate your reviews, which play a crucial role in improving our work.

Best regards,

The authors
Dear Reviewer NkMU

Thanks for your feedback! We are pleased to address your concerns and greatly appreciate your reviews, which play a crucial role in improving our work.

Best regards,

The authors
Thank you for the detailed response. The authors have thoroughly addressed my questions and concerns. I will maintain my original rating.
The authors have addressed my questions and concerns with additional results. I will keep my score.
Thanks for your feedback! We are pleased to address your concerns. The strict definition and detailed explanation of the mathematical symbol are necessary. We will rigorously define the mathematical symbol in the final version. We truly appreciate the pivotal role that reviewers like yourself play in enhancing the quality of our work.
Thanks for providing the additional results. I will increase my score. However, I still think the math should be rigorously defined and *explained* if the paper is finally accepted.
Dear Reviewer d71F,

Thanks for your feedback! We are pleased to address your concerns and greatly appreciate your reviews, which play a crucial role in improving our work.

Best regards,

The authors
Thank you for the response. The authors have addressed my concerns and I do not have other questions. I will keep my rating at 7.