Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper considers classification in the case of noisy training data, where one subset of points suffers from higher label noise then the complementary subset. This paper claims that by increasing the label noise rate on the less noisy subset, it is possible to obtain more accurate models since the learning problem is easier. 

Additionally, in the context of fairness, increasing the noise rate of the advantaged group improves fairness guarantees in the sense that when we are fair on the noisy data, we are also fair on the clean data. 

Lastly, this paper contributes a method to detect which subset suffers from higher noise rates (and how much more is suffers) without access to the ground truth.
main_review: Main Points

1) Observation 2 states that the learning problem becomes easier. In what way? In fact, later on, we establish that we actually have an increase in learning difficulty while overall we reduce the upper bound on the excess risk. 

2) An important point in Theorem 1 seems to be that the noise rates are not known and we use estimated noise rates to correct the loss. This adds some misspecification term in our bound, which is at the core of the tradeoff. This could be stated more upfront in the beginning since the main problem seems to be how to obtain accurate estimates of the noise rates. However, when are the estimated error rates important later on in the experiments? Is this loss correction (2) always applied?  The peer loss seems to specifically remove the limitation of needing the rates. Therefore, the benefit elaborated in Section 3 seems irrelevant here (but is still true for fairness). 

3) Furthermore, I dont see why a lower upper bound actually means that a learning problem is 'easier'. Since we are looking only at upper bounds it is well possible that the excess risk of the balanced noise case is still higher than the excess risk in Theorem 1. I would see this observation more as a motivation to experiment with balanced noise which is indeed emperically advantageous as seen in the experiments. 

4) Overall, I would suggest to communicate the meaning of Section 3 differently. It seems to depend heavily on the loss corretion approach and estimated noise rates, which later on become irrelevant. I think this theoretical observation is nice to motivate the contributed algorithm. After all, Noise+ still improves the baseline Peer loss, even though the peer loss does not need estimates of the noise rates. 

5) Proof of Lemma 1: Why does it hold that P(h(X) \neq Y^hat | Y^hat \neq Y) =  P(h(X) = Y| Y^hat \neq Y) = P(h(X) = Y)? The last step is not trivial, even though we know that the label noise is independent of X.

6) Regarding condition 2 in Theorem 3. How strong is it and what does it imply? We are looking at the optimal classifier on the clean D and the optimal classifier learned with newly generated noise labels. Those two are now flipped and evaluated on the original noisy data. Just intuitively, I would think that h tilde would have a lower loss on the noisy data than h diamond. And if you flip those, you are in the case of condition 2). Is that accurate?

7) It is unclear to me what the competing methods are. What is the difference between MisSL and EstSL? Earlier in the paper, when estimated versions of the noise rates were used, they were called 'model mis-specification error'.

Minor Points: 

9) Line 143: The expected value is written with tildes. Is that correct? I think, here we are talking about the noise balanced case notated with hats. 

10) Notation is sometimes confusing. h with tilde is defined with y hats and  and h hat is defined with y tildes. If you would rename the loss l^hat, you wouldnt need a hat in h, but could put the loss name in the subscript. 

11) Lines 245 to 259 are very condensed and confusing. Is the notation about C_l and C_r established at that point? 

12) What does Lemma 5 tell me? Simply that I need to at least estimate the noise rates right? And with Theorem 6, is this lemma even relevant?

13) For more clarity, it would be good to shortly define what a calibrated loss function is. 

14) The baseline methods, the noise procedure, and the peer loss need more description. This would be a helpful addition to the supplement. 
limitations_and_societal_impact: Regarding limitations, I have addressed this in points 2, 3, and 4, but I would be glad about clarifications from the authors.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper explores ways to improve the generalization performance and fairness guarantees, by increasing a certain class of instances' label noise to balance the rates between different classes. Theorem suggests that when the gap between noise rates is small and when we don't have high confidence in estimating the noise rates, it may be beneficial to increase noise rates. The paper further proposes a method to insert label noise without knowing the ground truth training labels. Experiments show that the rate balancing procedure can be beneficial to increase test performance.
main_review: Pros:
- The theorems shown in the paper are interesting, and provides new insights.
- The paper is well organized, starting from theoretical discussions and then verifying them through experiments. 

Cons:
- Although the theoretical contributions are interesting, some practical limitations remain: the noise gap needs to be small in order for the noise equalization to be beneficial in the equation in line 152~153 and also shown in experiments in Table 2. Is it possible to perform similar large gap experiments with the binary classification setup? Currently, the largest gap seems to be 0.2 in Table 1. It would make the paper stronger if we can see consistent results across different datasets. I was also curious if it is still beneficial under a setting with high noise rates but low noise gap.

Other minor comments:
- It would also be interesting to see experiments without using a pre-trained model for Table 2. Training from scratch with noisy labels may have more significant effects on the trained model.
- The gamma parameter was fixed in the experiments, but how sensitive are the results w.r.t. gamma?
- line 126: denonminator --> denominator
- line 178: FRP --> FPR?
- Reference [27] and [28] are the same

=======================================
After author response: Thank you for the additional experiments and for answering my questions. Since I do not have any further major concerns, I would like to raise my score to 7. Additional minor comment: typo on agreeing" on line 215.
limitations_and_societal_impact: The paper discusses that the method is limited for class-conditional noise. It does not directly discuss societal impacts, but fairness guarantees is discussed throughout the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this work, the authors aim to show that adding label noise in some cases could be beneficial to the test performance and fairness of models. Firstly, they find that increasing-to-balancing can result in an easier learning problem and improves fairness guarantees against label bias. Then they propose a method by inserting label noise to the group of labels with lower noise rates. To estimate the noise rates of various groups, the authors propose a detection method by checking the agreements of noisy labels among local neighbors. Experiments are conducted on three tiny datasets and CIFAR-10 dataset across different levels of noise rates to verify the effectiveness of the proposed algorithm.

main_review: Pros:  

As far as I’m aware, this work is the first to consider model fairness in the setting of label noise. In this work, the authors show that increasing-to-balancing can improve the performance under the setting of asymmetric noisy labels, which is somewhat counter-intuitive from the view of noisy rate.

This paper is well-written and easy-to-follow. The authors also describe the limitation of this work in the aspect of noisy types.


Cons:
In the setting of asymmetric noisy labels, it seems that the gap of noisy rate among classes can be treated as a class-imbalance problem. In the view of class imbalance, the observation would be not surprising because increasing-to-balancing could alleviate the problem of class imbalance. Then the trade-off becomes reducing the noise rate or the class imbalance rate, which reduces the contribution of this work.

The proposed detection method is similar to the KNN method in [1] because the agreement strategy among local neighbors can be seen as a variant of KNN. 

The three datasets used in experiments are very small. Although CIFAR-10 is also used, they only train with the image features extracted by a pre-trained vision transformer. Therefore, the experiments are not enough to show the conclusion from observations and the proposed method can extend to some large datasets trained from scratch, like MNIST, CIFAR-10/100 and Clothing1M (imbalanced).

The improvements from the proposed method are weak, especially on the CIFAR10 dataset. Although improving test accuracy is not the only purpose of this work, the weak improvements are still not convincing enough.


Other comments:

Why don’t try reducing label noise of the groups of labels with higher noise rate by sample selection?  Then we don’t need to consider the tradeoff because we reduce both the noise rate and imbalance rate.

[1] Bahri, Dara, Heinrich Jiang, and Maya Gupta. "Deep k-nn for noisy labels." International Conference on Machine Learning. PMLR, 2020.


***

My concerns have been addressed by the response, thus I decide to improve my score to 6. Although the improvements are still somewhat weak in some cases, this work provides new insights about asymmetric noisy labels for the community, which are more important.
limitations_and_societal_impact: The authors have addressed the limitations in noise types. It would be better if the authors can add more large datasets to show the applicability of the proposed method.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This submission studies the paradigm of injecting label-noise from the perspective of fairness and robustness. The motivation stems from the fact that balancing the label proportions yields an easier learning problem and improves fairness guarantees. The submission derives a bound between the corrected loss with label noise and the population risk, which through the use of standard concentration inequalities, involves the Rademacher complexity. In order to fix one, they propose a method that detects the right choice of labels that suffer from noise, which then is turned into an algorithm for robust and fair label correction.
main_review: The main contributions of the paper consist of two major components: (1) elucidating the trade-off and an in-depth analysis of the error rates in classification along with a generalization bound to explicitly outline this and (2) a detection method that allows you to determine the right group to inject noise for further processing.

The main strength of the submission is indeed (1), which presents an interesting analysis that explains this trade-off and serves an extremely useful purpose to both the practitioner and theoretician working in fairness and robustness. The contribution (2) is more preliminary at this point however will serve useful for future developments that incorporate the results of this paper. 

The weaknesses I can gather include the tightness of the bound presented in (1). Since it involves the Rademacher complexity, it leads me to believe this may be inaccessible to conventional learning settings where we use deep neural networks. I am not entirely familiar with this literature and am not sure whether this as a starting point is substantial enough.

The submission is very clear with its contribution and makes it easy for the reader to understand and follow the story. The notation is mostly easy to follow and uses standard conventions throughout - small things like Rademacher complexity is not formally defined however this is no big deal as it is not the focus of this result. 

Based on my knowledge, it seems the related work is discussed appropriately and the quality of correctness is sufficient throughout the paper as I was not able to find any mistake in the derivation of the results. I believe the topics of dealing at the intersection of robustness, fairness and label-noise is extremely relevant to the NeurIPS community.

Overall, I think this paper meets the bar of inclusion for NeurIPS due to the importance of the problem, elucidation of this trade-off between noise rates and relatively simple method of group-detection for downstream tasks. The results in this work would greatly benefit the machine learning community.

Some questions/comments for the authors:

(1) Why is sgn used in Line 123 in Equation (2)? It seems to me that in the binary classification case it is redundant since Y is either +1 or -1 or perhaps I'm not appreciating some generality here?

(2) It would be useful if you could explain some intuition behind the corrected loss since a non-expert finds it difficult to grasp the weighting parameters.

(2) A minor weakness I pointed out is the Rademacher complexity involvement in the bound - I was wondering if you could have other complexity measures appearing here perhaps if you change the analysis. For example, in the discrete setting, would it be possible to have the VC dimension appearing or even a PAC-Bayesian analysis? I tried to replicate the arguments however it was not so clear to me if the Rademacher complexity can be avoided.

----------------------------------------------------------------------------------------------------------------------------------------
I have read the other reviews and rebuttals. The authors have addressed my questions and concerned and so I will be increasing my score.
limitations_and_societal_impact: Yes they have
review_title: null

Meta-review: Reviewers were unanimously positive about the paper's contribution to a topical problem, namely, learning with label noise and its implications on model fairness. The initial reviews raised some technical questions that were satisfactorily addressed by the authors. The authors are further encouraged to incorporate the other suggestions provided by reviewers for the updated version of the work.
Decision: Accept (Poster)