Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper suggests attention-based method to distinguish mislabeled data and decrease their impact during training. Specifically, the authors add an attention branch at the end of the feature-extraction layer (a.k.a. after the penultimate layer) that outputs a confidence score $\tau$. Then, the attention layer's confidence score $\tau$ is incorporated into the loss function to softly divide samples into clean ones and mislabeled ones; the gradient of clean ones is trained as-is due to their early learning phenomenon, while that of mislabeled ones is not trained because $\tau$ is also trained to decrease the impact of mislabeled ones. The paper supports their idea both theoretically and empirically. 
strengths_and_weaknesses: ## Strength

The strength of this paper can be elaborated as follows:
1. The writing is easy and clear to follow.
2. Incorporation of an attention layer and temporal ensembling-based target estimation is well-motivated and harmonious.
3. Theoretical justification and empirical support are sound.

## Weakness 

1. A strong weakness is the use of two hyperparameters $\lambda$ and $\alpha$. Especially, $\lambda$ should be manually decided for each dataset, which is burdensome and hence becomes a strong hurdle for the suggested method.
2. The baselines in the experiments are inconsistent, possibly due to the copy of experimental results from original papers. 
3. Why accuracy results on some of the two CIFAR datasets, Clothing 1M, and Webvision do not have variance results? They should be consistently reported.
questions: The questions would be about how to deal with the above weakness points.
limitations: No critical limitation is found up to this point.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents a method for training models with noisy labels. The proposed method learns a weight for each training sample based on its noise level (measured by the difference between model prediction and label), using an attention branch. Experimental results on benchmark datasets show that the proposed method was able to improve the model performance compared to several noisy training approaches.
strengths_and_weaknesses: __Strengths__
- The idea is simple and experimental results are promising.

__Weaknesses__
- The theoretical justification is weak. Lemma 4.1 is essentially saying noisy labels hurts the model that trained with cross entropy loss. This is a well-known theory. I don't see how make it as a Lemma helps the discussion in this paper.
questions: 1. All the proposed techniques in this paper are built based upon the observation that noisy samples have higher loss compared to clean samples. How does the proposed method handle samples with different difficulty levels?
2. It is unclear if the noise attention loss is used for the model training alone or in conjunction with the cross entropy loss.
3. Math notation needs to be improved. For example, $j$ is never defined in Sec 3.3 and it is used represent different things in Lemma 4.1 and Theorem 4.1. What is the difference between $t$ and $\hat{t}$ in Algorithm 1?

limitations: No
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors proposed a simple method that trains an attention layer to generate weights for different samples. By using neural networks’ memorization effects, the weights can be learned during the training process. This method simplifies the training process, which does not need to extract confident examples. The authors also give theoretical analyses of their method. The proposed method experimentally shows large improvements compared with baselines.
strengths_and_weaknesses: 1. Training an attention layer, directly outputting a score, is novel. 
2. The proposed method is simple and empirical works well.
3. This paper is well organized and easy to understand.

questions: 1. Although the empirical results are good, could the authors simply explain how this proposed attention branch distinguishes mislabelled samples and some hard examples that are not learned in the early learning phase?
2. For CIFAR-10 with 60% in figure 2, I found some noise is given with high weights, and some clean data is given low weights. Can you explain it?  
3. I found the performance is lower than some state-of-the-art methods, and the authors claim that it does not employ some techniques like MixUp, or two networks. Does the proposed method combine well with these widely used techniques in learning with label noise, for example, MixUp? 

limitations: 1. From my perspective, the word “attention” looks a little ambiguous. It more seems like a confidence score for each sample. 
2. The proposed method builds on semi-technique ensemble predictions. So, I think the authors should compare it with state-of-the-art methods that also use semi-technique e.g., DivideMix, FINE[1], or PES [2], which makes it easy to compare.  
3. The authors only conduct symmetric label noise and asymmetric noise. Some commonly used settings like Pairflip 45% and Instance-dependent label noise [3, 4] are missing. 
4. The hyperparameter lambda is sensitive and has a large range for different datasets. 

[1] FINE Samples for Learning with Noisy Labels
[2] Understanding and Improving Early Stopping for Learning with Noisy Labels 
[3] LEARNING WITH FEATURE-DEPENDENT LABEL NOISE: A PROGRESSIVE APPROACH
[4] Part-dependent Label Noise: Towards Instance-dependent Label Noise Xiaobo
review_title: null

Meta-review: metareview: The work proposes a simple method for training an 'attention layer' that can give weights for different input samples. These weights are learned during the training process. This method appears to the theoretically justified, the method relatively simple and the empirical results seem reasonable. One concern that I share with the reviewers is about the hard (but correctly labeled) samples. The authors provide some explanation and results in the appendix, which mostly allay the concerns. I have to agree that introducing a somewhat sensitive parameter $\lambda$ is not ideal, but overall the empirical and theoretical justifications tip the balance towards acceptance in this case.
confidence: Less certain
Decision: Accept