Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: - the paper proposes a simple idea for robustness to label noise. The idea is to increase the variance of the loss function across training samples. A high variance could generally make sense since ideally the loss remains high for noisy-labelled samples during training while on the other hand the loss for correctly-labelled data is gradually decreased.

- it further derives a form for the derivative of the proposed total loss w.r.t. the model parameters where the proposed loss can be seen as a reweighting of the gradient induced by individual training samples. It proportionally upweights the gradient of those samples with lower losses than the average loss and downweights the gradient of the samples with higher losses than the average loss.

- it applies the method on top of three noise robustness techniques. Two of them are based on noisy label transition matrices, namely Forward and VolMinNet as well as a loss reweighting method.

- the experiments show consistent improvement (although sometimes small) of the modified version of those three methods on different datasets, MNIST, CIFAR, Clothing1M) and noise types and rates (real, synthetic, symmetric, and asymmetric, 30% and 50%).

- some additional studies are done to shed light on the inner workings of the proposed method. Interestingly, it is shown that the proposed method is more robust to erroneous estimation of the noisy label transition matrix.
strength_and_weaknesses: - the paper proposes a simple idea for robustness to label noise

- some results are encouraging compared to the chosen baselines.

- the method seems original in that it enforces high variance on the training data loss. However, quite a few methods that apply some form of selection/reweighting/distribution on the individual sample loss functions can be seen as increasing the variance of training losses. It is important to make such discussion in the paper and highlight the differences of the paper's proposal to such prior works both formally and empirically.

- since the paper's claim on increasing variance for noise robustness is general, either the main experiments should be done on standard loss functions such as CE or the claims and theoretical motivation should be adjusted specifically to the case of noise transition matrices. 

- due to the simplicity of the approach (increasing loss variance) it is natural to assume complications might arise from the confusion of hard but correctly-labelled examples with noisily-labelled examples. Therefore, extensive discussions and experiments are required as to how such variance encouragement is theoretically or empirically unharmful for the difficult examples (e.g., in a scenario where the data is clean but quite complex to model).

- different $\alpha$s are used for different setups, what is the sensitivity of the method to $\alpha$ (in each/some of those setups)? How is  optimized for each setup? Also, it seems like too-early or too-high enforcement of large variances can be detrimental to fitting the cleanly labelled data. Does the method require heavy hyperparameter tuning? If not, how robust is it to hyperparameters? Why is it robust?


clarity,_quality,_novelty_and_reproducibility: *Minor*

- Fig 1: what dataset is the results reported on
- page 4: “noise examples” -> noisy
- page 4: “when work” -> working
- page 5 “log determinate” -> determinant
summary_of_the_review: The paper proposes a simple change of the loss function that encourages variance of the loss across training samples and shows this can bring robustness to training label noise in some scenarios. However, there are some important concerns with the empirical evidence that has been provided in support of the paper's proposal.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper aimed to explore if the variance of losses should be always penalized when learning with noisy labels. The paper suggests adding a variance regularization to the loss, which will encourage the variance of losses. Consequently, when calculating the gradients of the proposed loss, if the loss of an example  is smaller than the expectation of the losses (which the authors assume is true in the case of examples with clean labels), the weight associated with the respective gradient will be increased and the example contributes more to the update of the parameter. If the loss of the example is larger than the expectation of the losses, the weight associated with its gradient is small and the example contributes less to the update of parameter. The assumption is that the latest will happen for noisy examples, because the network tends first to learn from clean examples and increasing the variance of losses would boost this memorization effect and reduce the harmfulness of incorrect labels. 
The authors explain how to add the suggested regularization to several recent methods and show improvement. 
strength_and_weaknesses: Strength:
The overall idea of adjusting the weights, such that noisy samples influence weight update less, sounds like a good idea. The approach suggested to achieve that with addition of variance component to the loss is relatively simple (can be applicable as the authors show to several techniques) and yet relatively effective (improved results compared to the original method -- Tables 1 & 2).

Weakness:
The paper mostly proves the effectiveness of the approach empirically, improving the results by some margin in case of the synthetic noise. The improvement on real noise ( Clothing1M  dataset) is relatively minor (Table 2 -- <0.5% improvement) and since there is no real theoretical proof that the suggested approach should work and under what conditions, the paper overall is not very convincing (in other worlds, that would not be my choice of an approach when dealing with noisy labels).
 In the absence of theory and given that the paper is not aiming to provide a SOTA technique, but discuss the loss variance penalization / encouragement, reorganization of the paper, e.g. :
(1) bringing results from the appendix to the main body (e.g., uniting Table 1 and Table 4&5); (2) writing a paragraph with concrete conclusions and answer to the question posed in the title; (3)  clear conclusion on what types of noise, datasets  (balanced / not balanced) benefit from the approach, can help to strengthen the empirical contribution of the paper.    
clarity,_quality,_novelty_and_reproducibility: Clarity:
The paper is mostly clear, however the experiments are scattered between the main body and appendix, which makes the reading harder.

Quality:
While the empirical validation is relatively thorough, the paper's writing quality can be improved.  

Novelty:
The idea of exploiting early memorization of the network is not novel. Adding loss variance as a regularization to achieve better resilience to noise seems to be novel (to the best of my knowledge).  

Reproducibility:
The authors provide code as a supplementary material. 
summary_of_the_review: I have mixed feelings about this paper. I would suggest the authors reorganize the paper in a way that it provides some insights beyond just illustrating improvements to some methods (and providing the implementation details). 
1) The authors ask the question -- do we always need to finalize variances?  -- I could not find a clear to this question. It seems that the authors actually the opposite:  that the variances should be encouraged. So is the answer - always encourage losses?
2) I am wondering why the authors did not provide the evaluation results of their method on WebVision dataset. Given relatively minor improvement on Clothing1M (< 0.5%), how will the results look like for WebVision? 
3) In my opinion, it's not clear why the authors push the results with little noise and strong noise to the appendix. It will be interesting to see those together with the results of Table 1 and maybe as a graph that shows the improvements (deltas)  as a function of noise. 

I will definitely reconsider my rating if the paper is reorganized and the main learning are clearly stated.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper makes the seemingly counterintuitive claim that the variance of losses should be rewarded, rather than penalized, in problems involving learning with noisy labels. Mathematically, the authors show that rewarding the variance of losses leads to parameter updates so that if the loss of the example is smaller than the average loss over the examples, the magnitude of the update is magnified, and vice-versa. This leads to an intuitive safeguard against the influence of points with noisy labels, which tend to have high loss. The authors present extensive empirical results that show that this type of variance regularization boosts the performance of existing methods for learning with noisy labels.
strength_and_weaknesses: ## Strengths
* Learning with noisy labels is a topic of high importance and interest to the wider ML community.
* The proposed technique is novel and contrary to the traditional advice of penalizing the variance. The authors make a compelling case for the benefit of rewarding the variance in terms of attenuating the influence of noisy labels on the model during training.
* The authors present extensive empirical evidence on a variety of scenarios that demonstrate the improved effectiveness of the method in boosting the performance of existing label-noise learning approaches.
* Sensitivity analysis of the method’s hyperparameter is provided in the supplementary that shows that it can be estimated using a validation data set

## Weaknesses
* Even though there are experiments with VRNL augmenting prior label-noise learning approaches, it would have been very compelling to include results with VRNL alone, without knowledge or reasoning about the transition matrix. For instance, a performance comparison of VRNL vs. no VRNL on a simple CIFAR10 example where, e.g., 10% of the labels are artificially made to be incorrect, would go a long way. 
* The method requires tuning of the hyperparameter $\alpha$ and a wide range of values for $\alpha$ is used for the various experiments on different scenarios in Sec. 4. From the plots in the appendix (Fig. 7), the method’s performance is highly sensitive to the precise value of $\alpha$. The authors did address that a validation data set can be used in a reliable way for CIFAR10, however, it is not clear whether this is always the case.

clarity,_quality,_novelty_and_reproducibility: * Clarity: the paper is well-written with a clear exposition
* Novelty: the proposed approach is contrary to traditional wisdom and the authors make a compelling case for its benefits.
* Quality: the empirical results provide support for the method when used with state-of-the-art label-noise learning approaches.
* Reproducibility: the hyperparameters and the configurations used for the experiments are outlined in the results section (Sec. 4).

summary_of_the_review: This paper addresses a problem that is highly relevant to the ML community. It challenges the traditional wisdom that variance of losses should be penalized, and it instead makes a compelling argument for rewarding the variance. Even though a more direct comparison of VRNL would go a long way (see above), the authors present compelling empirical evidence on the benefits of the proposed method when used in conjunction with prior methods. In light of these considerations, I recommend acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a method for learning with noisy data by increasing the variance of the individual example losses in a batch. It applies the method on top of three noise robustness techniques -- two based on noisy label transition matrices and on a loss reweighting method. Experimental results on MNIST, CIFAR and Clothing1M are presented to support the claims. While the reviewers appreciate the simplicity of the method, they had some outstanding concerns, including, lack of theoretical insight about the method, moderate to low gains over baselines, sensitivity to the variance term hyperparamter. One reviewer noted that parts of the paper overstate the contributions. Simplicity of the method is a strong point of the paper and authors should provide comparisons with other methods that share this property for label noise such as [1,2,3] 
 
[1] Normalized Loss Functions for Deep Learning with Noisy Labels, https://arxiv.org/abs/2006.13554
[2] Constrained Instance and Class Reweighting for Robust Learning under Label Noise, https://arxiv.org/abs/2111.05428
[3] Robust Bi-Tempered Logistic Loss Based on Bregman Divergences, https://arxiv.org/abs/1906.03361


justification_for_why_not_higher_score: Reviewers had some outstanding concerns, including, lack of theoretical insight about the method, moderate to low gains over baselines, sensitivity to the variance term hyperparamter. One reviewer noted that parts of the paper overstate the contributions. Simplicity of the method is a strong point of the paper and authors should provide comparisons with other methods that share this property for label noise such as [1,2,3] 
 
[1] Normalized Loss Functions for Deep Learning with Noisy Labels, https://arxiv.org/abs/2006.13554
[2] Constrained Instance and Class Reweighting for Robust Learning under Label Noise, https://arxiv.org/abs/2111.05428
[3] Robust Bi-Tempered Logistic Loss Based on Bregman Divergences, https://arxiv.org/abs/1906.03361
Decision: Reject