Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: L2B is a learnable loss objective that enables a joint reweighting of instances and labels at once.  L2B dynamically adjusts the per-sample importance weight between the given labels and pseudo-labels in a meta way.
strength_and_weaknesses: Strengths:
1)  L2B dynamically adjusts the per-sample importance weight between the given labels and pseudo-labels.
2) The results are somehow higher than some reweighting methods but not other noisy methods.
3) The experiments are done on different datasets.
4) The convergence is analyzed in the paper.

Weakness:
1) should the \alpha_{t,i} and \beta_{t,i} in Eq.10 and algorithm line 12 be \tilde{\alpha_{t,i}} and \tilde{\beta_{t,i}}?
2) Here is a suggestion: an ablation should be made: \mathcal{L}( \mathcal{F}(x_i,\theta), \alpha y_i^{real}+\beta y_i^{psudo})  \alpha and \beta is learned as in your algorithm. This ablation can better analyze whether the re-weighting between the samples is useful or the re-weighting between the label and psuedo-label is useful.
3) comparison of experiments results: maybe previous works [1][2][3] should be compared. Is there any analysis that the results of L2B are not higher compared to [1][2][3]? Since this work is based on the loss design, can L2B be applicable to the current SOTA noisy methods? Can you please show more results based on these works to better illustrate the effectiveness?
4) typo: 'Eq. equation' 

[1] Li, Junnan, Richard Socher, and Steven CH Hoi. "DivideMix: Learning with Noisy Labels as Semi-supervised Learning." International Conference on Learning Representations. 2019.
[2] Bai, Yingbin, et al. "Understanding and improving early stopping for learning with noisy labels." Advances in Neural Information Processing Systems 34 (2021): 24392-24403.
[3] Liu, Sheng, et al. "Early-learning regularization prevents memorization of noisy labels." Advances in neural information processing systems 33 (2020): 20331-20342.
clarity,_quality,_novelty_and_reproducibility: The clarity is good.
The novelty is not that attractive. It seems more like using meta-learning to learn loss parameters. Maybe a deeper understanding can be shown to improve the novelty.
summary_of_the_review: The parameters of the loss terms are adjusted by meta-learning. However, the novelty seems not that good in this work. Further analysis should be made to give a deeper understanding.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to meta-learn an instance-level weight for real labels and pseudo-labels when learning with noisy labels. 
strength_and_weaknesses: Pros

1. The paper is well-written and easy to follow. 

2. Theoretical guarantee of the convergence of the proposed method is provided. 

Cons

My major concern is that the proposed method is not comparable or compatible with the current SOTA of the methods (of combating noisy labels) which leverage self-supervised learning algorithms and data augmentation [1,2]. 

The author is trying to rule those works out of the scope of this work. In the related work, [1] is categorized as "approaches which separately handle instance reweighting and label reweighting" In Section 4.2, the author validates themselves by saying "Note that we mainly compare with methods which do not use any augmentation techniques, as our L2B does not rely on those."

However, the methods proposed in [1,2,3] have much better performance than the method in this paper. For example, in the setting of CIFAR-10 50% symmetric noise. The proposed method L2B has an accuracy of 88.5 while the method of [1] has an accuracy of 94.5±0.1

I am concerned about the usefulness of L2B in real practice since it does not help improve the SOTA method. 



[1] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semisupervised learning. In ICLR, 2020.
[2] Junnan Li, Caiming Xiong, and Steven C.H. Hoi. Learning from noisy data with robust representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision
(ICCV), pp. 9485–9494, October 2021.
[3] Wang, Zhaoqing, et al. "Exploring set similarity for dense self-supervised representation learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and has nice clarity. The paper proposes a novel technique of weighting losses for combating noisy labels. The significance of the paper is limited since the proposed method's performance is not comparable with the SOTA. 
summary_of_the_review: Overall, considering the clarity, novelty, and quality, the current version of the paper is slightly below the acceptance bar. 

===

After rebuttal, the author shows that the proposed method can improve the performance of the current SOTA. Thus I raise my score from 5 to 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a more generic learnable loss objective which enables a joint reweighting of instances and labels at once, in order to mitigate the effect of noise labels on the model generalization. Specifically, their method dynamically adjusts the per-sample importance weight between the real observed labels and pseudo-labels, where the weights are efficiently determined in a meta process.
strength_and_weaknesses: strength:
1. the motivation of this paper is valid to me, i.e., simultaneously adjust the per-sample loss weight while implicitly relabeling the training samples
2. the results seem to be effective from the main tables on main benchmarks.
3. the paper is written clearly and is easy to understand.

weakness:
1. A small unbiased and clean validation set is required in this method. Is it a strong assumption made in the current framework?
2. One of the popular and relevant baselines is Dividemix[1], which also involves reweighting mechanism in their approach. Have the authors ever compared their method with this baseline?
3. From fig 4, the weight before the samples is very small. It is interesting to explain why this phenomenon happens. Does that affect the learning procedure? Also, if that is indeed the case, why the authors can claim the data distribution is protected since a lot of samples seem to be ``discarded" as well?

[1] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semisupervised learning. In ICLR, 2020.


-----post rebuttal

Thank the authors for the detailed response! I would like to raise my score.
clarity,_quality,_novelty_and_reproducibility: The paper is generally written clearly. The method makes sense to me.


summary_of_the_review: The paper meta-learns two weighting parameters for the real label and the pseudo label for combating the label noise problem. There are some vague explanations/experiments in the paper that need to be clarified. Will consider changing my score after I see some of the other reviewers' opinions and the authors' responses.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:  
This paper proposes a method to learn with noisy labels by jointly reweighting instances and labels at once. To this end, the proposed method (L2B) meta-learns two weighting parameters in the loss for the real label and the pseudo label. In the experiments, L2B is evaluated on standard datasets and shown to outperform some existing methods.

Strengths:  
1. The proposed method, L2B, is well motivated and provides a more general form of the bootstrapping loss. 
2. Theoretical guarantee of the convergence of the proposed method is provided.
3. The paper is generally written clearly. 

Weaknesses:
1. Novelty of the method seems to be in a somewhat narrow context and not ground-breaking.
2. Positioning of the study as well as comparison/compatibility with other noisy label method can be better investigated. 
justification_for_why_not_higher_score: This paper initially received the scores of 5,5,5 ('marginally below the acceptance threshold' x3). One of the reviewers’ main concerns is regarding the novelty of the method since tuning loss parameters (weights) by mete-learning is a relatively well-studied approach, although the reviewers also acknowledge the contribution of the method in the context of the bootstrapping loss. Another concern is about the framing of the study and experimental evaluation. In particular, it was not clear whether the method is comparable or compatible with other noisy learning (i.e., other than meta-learning based ones). The AC also agrees that those concerns are fair.   
During the discussion period, the authors responded with additional results which mitigated some of the concerns, and all reviewer decided to raise the score to 6 (marginally above acceptance threshold).  As a result, the final scores are 6,6,6, which are all slightly positive yet the paper still lacks a strong support. Given the high standard of ICLR and the reviewers' opinions, the paper is still on the borderline but could be accepted depending on the availability of the last rooms.
Decision: Reject