Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose Generalized Data Weighting (GDW) to generalize data weighting from instance level to class level by reweighting gradient flows. This framework can be helpful for better information use when handling label noise and class imbalance.  Specifically, they design a two-stage weight generation scheme embedded in the three-step bi-level optimization framework which can efficiently obtain class-level weights.  This scheme leverages intermediate gradients to update class-level weights via a gradient descent step. Experiments show that GDW achieves remarkable performance improvement in various settings.
main_review: Originality: 
In contrast to the classical instance level weighting, the authors are the first to propose class-level weighting on gradient flows. However, the main techniques this paper leverages have already been well-established in [23] which proposed the MW-Net Learning Algorithm. This is a nice extension of the existing techniques.

Quality: 
The authors conduct extensive experiments in various settings ( Label Noise Setting, Class Imbalance Setting, Real-world Setting) to verify the effectiveness of GDW. 

Clarity: 
This paper is organized well. I can easily understand the main idea and techniques applied

Significance: 
This paper advances state-of-the-art methods to alleviate Label noise and class imbalance by leveraging class-level information. The algorithm proposed in this paper would be helpful to deal with two major issues coexisting in real-world datasets.

limitations_and_societal_impact: Although the proposed algorithm seems to work well empirically, there is no theoretical analysis on the performance. Also, the main techniques this paper leverages have already been well-established in [23]. The technical contribution seems somewhat marginal.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper focuses on a technique to learning per-instance weights in training classification networks. The paper improves over the idea of Meta-Weight-Net [23] which adopts meta-learning to learn per-instance weights. According to the paper, training data labels can be noisy and class labels usually follow long-tailed distributions, therefore learning per-instance weights is expected to better exploit training data for learning a better classification network. The paper's improvement is based on an intuition that class-level weights have not but should have been used for better learning per-instance weights. The paper proposes Generalized Data Weighting (GDW) and claims to *simultaneously* mitigate label noise and class imbalance. However, it is not clear why the proposed method is "generalized", and experiments do not validate the method can *simultaneously* mitigate label noise and class imbalance, because experiments study the two aspects separately. 

main_review: pros:

+ The paper positions itself as using data reweighting to simultaneously address class-imbalanced learning and learning with noisy labels. This problem is interesting (although authors failed in experimentally validating the method on this problem).

+ Learning per-instance weights is a reasonable approach to address either imbalanced learning or learning with noisy labels.


Below are some summarized cons, followed by detailed comments.

- Due to such a position (as simultaneously addressing class-imbalanced learning and learning with noisy labels), the paper failed to justify the method in this experiments. In contrast, the paper studies the two problems separately.

- While class-imbalanced learning can be improved by learning per-instance weights, the paper did not fully survey the literature of class imbalanced learning. For example, [ref1] is missing in the paper that shows a rather simple method greatly improves class-imbalanced learning. Given this, it is natural to question why the paper misses an important baseline -- using [ref1] and data reweighting to simultaneously solve imbalanced learning with noisy labels.  [ref1] Decoupling Representation and Classifier for Long-Tailed Recognition, ICLR 2020


- The paper is incremental in terms of techniques. It improves over Meta-Weight-Net [23] by incorporating class-level weights learning. Perhaps because of this, the method is called *generalized* data reweighting; however, the method is incremental rather than *generalized*.


- There are many parts of the paper hard to follow, as detailed below.

First, as the paper positions itself as "simultaneously mitigate label noise and class imbalance" (Line6), the paper fails to justify whether the proposed method can do it because experiments study the two subproblems separately. Furthermore, it looks quite non-trivial how to exploit class-level weights in front of noisy labels and imbalanced classes. For example, it is not clear how to weight data with noisy labels from rare classes, should the model give a large weight on them because they are from rare classes, or a small weight because they have noisy labels? Authors should delve into this aspect to better support the position of the paper.

Line35: The sentence is confusing "Due to the MLP, MWNet has better scalability on large datasets compared with INSW". It is not clear what INSW does, and why MLP improves MWNet. Authors should improve the writing.

Line37: The sentence is confusing -- "they can not fully utilize class-level information within each instance". What do authors mean by "fully utilize class-level information"? Why "they can not"? What did "they" do such that "they can not"? Authors should clarify. This makes the paper hard to follow.

Line39: It is confusing -- what is the classifier for computing gradients? Does this classifier compute gradients from "not X" in terms of class-X?

Line42: What are the "three kinds of information"? The authors did not define the "three kinds of information". It is hard to understand the paper.

Line43: It is confusing -- "downweighting the "not bird" gradient flow is a waste of information." What do authors mean by "a waste of information"? Can authors clarify?

Figure 1: It is confusing -- "Although the gradient flows for "dog" and "not cat" contain harmful information..." What do authors mean by "harmful information"? Can authors clarify?

Line66: It is unclear what the authors mean by "To the best of our knowledge, we are the first to propose class-level weighting on gradient flows." Because it is not clear what the difference is between gradient reweighting and loss reweighting. Can authors clarify?

Section 2.2: There is one missed citation that is important [ref1]. Authors should carefully survey the literature.
[ref1] Decoupling Representation and Classifier for Long-Tailed Recognition, ICLR 2020

Line87: Although tt is evocative to say "meta-learning methods view instance weights as hyper-parameters and dynamically update them via a meta set to avoid hyper-parameter tuning." Did the authors consider the hyperparameters in the meta-learning? What are the learning rates, momentum, schedules of learning rates and initialization in meta learning? In the paper, authors did not provide such details.

Line94: What is "dynamic regularization"? Can authors define it?

Line127: It is not straightforward to understand how D1 is derived. Necessary derivations are needed. 

Line136: What do authors mean by "retain Softmax-CrossEntropy loss structure after the manipulation"? What is structure? What will happen if one does not "retain" the structure? Authors should improve writing the make the paper readable.

Line140: What do the authors mean by "messes the structure"? What structure and how it "messes the structure"? 

Eq.(4): The authors proposes a zero-mean constraint (Line143), but from what the authors did, it seems that this is factually not a constraint but an assumption with which the authors simply drop the second term. Authors should be careful in writing the statements.

Line213: The authors explain "GLC estimates the label corruption matrix well and thus performs the best", but why GLC does not estimate label corruption matrix well in other noise levels? Authors should provide insights or analysis.

Line229: As authors did not define "non-target weights", "true-target weights" or "non-target", I cannot understand this part. Authors should improve the writing quality.

Line261: The definition of the imbalanced factor is inconsistent. In plain words, it is defined as "the number of instances in the largest class divided by that of the smallest", that said, the factor should be no less than 1. However, authors states the factor is in (0,1). Authors should correct this inconsistency.







----------- post rebuttal --------------

I appreciate the authors' feedback! The answers largely addressed most my confusions, though I still have some below.

First, I think the explanation in the rebuttal is much clearer than the paper! Therefore, I'd upgrade the rating to "5: Marginally below the acceptance threshold". While this is not a recommendation for accepting the current version of the paper, I encourage the authors to incorporate the new results to polish the paper for the next submission.

Importantly, it is still confusing to connect the position of the paper and the proposed methods. Concretely, the position of the paper is "to simultaneously mitigate label noise and class imbalance" (line6 in the abstract) because "Label noise and class imbalance are two major issues coexisting in real-world datasets" (line1 in the abstract). This is a very good problem setup, but the authors only study this setup in the rebuttal.

Because this is a novel problem setup, thoughtful baselines should be compared, such as combining noisy training atop of a state-of-the-art (SOTA) class-imbalanced learning method [ref1] and combining class-imbalanced learning atop of a SOTA noisy training method. I'm not satisfied by the authors argument "it is not our duty to consider it as a baseline for comparison" -- this will hurt the paper. I have some background of long-tailed recognition, as a reader, I would think such a combination as the first method to try out to solve this novel problem, rather than rushing to any complicated methods.

On the other hand, in the rebuttal, the authors make another argument by "Should GDW perform better than this combination, the soundness of the comparison itself would be suspected", implying that the authors are reluctant to compare these straightforward baselines to address the novel problem setup. If the authors really want to sell the method, then authors can change the position of the paper -- instead of proposing a novel problem setup, authors can directly start by methods without emphasizing the problem.

Lastly, I do encourage the authors to make a thoughtful revision of the paper by considering the new experiment results, position of the paper, and comparisons to other baselines.
limitations_and_societal_impact: The paper does not discuss any limitations or societal impacts in the paper. The authors should be aware of the limitations of their methods. Authors should clarify.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a new method for mitigating the problem of label noise and class imbalance. The key idea is to utilize the class-level information to fully exploit the information contained in each training instance. The proposed method is evaluated using the popular dataset and it showed improved performance over the existing methods.
main_review: # Strength
I found the key idea of utilizing class-level information within each instance is novel and interesting. In addition, the paper is basically well written except the part I point out below.  The method is evaluated using popular benchmark datasets and shows comparable or better performance compared to the existing state-of-the-art methods.

# Weakness
My main concerns are two folds.

## 1. The supports for the claims are not sufficient.
### 1.1 The coexistence of label noise and class imbalance and not explicitly handled in the experiment.
The authors repeatedly claim that one of the important problems in the real-world setting is the coexistence of label noise and class imbalance (L1, L24-25), and their GDW simultaneously mitigates these problems (L6, L300-301). However, these two issues are handled separately in the experiment, which is not good for supporting the above-mentioned claim.
### 1.2 The effectiveness of the proposed method in class-imbalance case seems marginal
Looking at the standard deviations, the proposed method is not significantly better than the existing methods. Is there any analysis on why the method works fine in label-nose setting whereas shows relatively lower performance in class-imbalance setting?

### 1.3 Figure 5 and L229-232
The meaning of “target” is not clear as I later point out in more detail. Therefore, it is not clear for me what $w_{nt}^n$ includes. L232 says it represents the importance of the “not bird”, but how about the “dog”? I think decreasing the weight of the “dog” gradient is as important as decreasing the weight of “not cat” in the case of Figure 1.

In addition, it is better to show $w_{tt}^c$, too, for the completeness.
### 1.4 Figure 6
How about the ratio trend of the number of increased $w_8$ on C1-C7, and C8? I think C8 instances should receive lower weights not only in the shown case, but also when C1-C7 instances are used in the training. Similarly, they should receive larger weights when C8 instances are used in the training. How is the same analysis for $w_9$ on C1-C8 and C9? Showing only one example is not sufficient.

### 1.5 L248-249: “We have also tried training without the zero-mean constraint in Section 3.3 and got poor results.”
Just “got poor results” is not enough as an evidence of the importance of zero-mean constraint. Please show it quantitatively.

## 2. Some parts of the paper are not clear enough.
### 2.1 L131-132 `Specifically, GDW tries to downweight the gradient flows for "dog" and "not cat", and upweight the gradient flow for "not bird".’
I understand what the proposed method aims to do. However I do not think the reason why such a thing is possible is clearly explained. Please explain the intuition behind this.
### 2.2 Section 3.3 is not clear
- L136: What does “retain the Softmax-CrossEntropy loss structure” mean? What is the motivation to do so?
- L138: What does “the target (label) position” mean? Especially in noisy label case, does “target” mean a true category or wrongly labeled ground-truth category? For example in Figure 1, what does “target” represent? Without precisely understand what is “target”, I cannot judge the soundness of equation 3 and subsequent discussion. For now, I can only understand equation 3 holds true if $w_t = w_j$.
### 2.3 equation 11
I think it is better to explicitly write down what $g_i$ exactly denotes in the equation.


limitations_and_societal_impact: No.
- It is recommended to discuss the limitation of the method more deeply in the main body of the paper. For example, the proposed method is tested only in label-noise case and in class-imbalance case separately. I think this is one of the limitation of the paper.
- The authors did not discuss any potential negative societal impacts. Therefore, the answer to this question in the checklist should be [No] rather than [N/A]. Any kinds of works can be used in an undesirable manner regardless of the authors’ intention. For example, the method may be used to intentionally increase the bias since it can handle class-level information. It is important to discuss the possible negative aspects (and hopefully how to cope with them).
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In the paper, the authors propose a novel generalized data weighting method to improve the performance of existing label noise and class imbalance methods by manipulating the class-level gradients. Existing instance weighting methods are special cases of the proposed method when the class-level weights are the same. The proposed method is quite simple, easy to be deployed to existing methods, and does not introduce additional computational costs. 
main_review: Comments:
1. In equation (8), performing empirical risk minimization on a small meta dataset may be very likely to overfitting the meta dataset. 
2. For the experiments, how to ensure a fair comparison with traditional methods such as co-teaching [20] where meta dataset is not needed.
3. In Figure 3, some class-level target weights for noisy data are still very large, even larger than that for the clean data. Giving the mislabeled data large weights could greatly harm the performance. I'm not sure whether the Noisy (the blue in legend) in Figure 3 means a group of intact and mislabeled data, or only mislabeled data.
4. Experiments of the paper include label noise (uniform + flip noise + real-world label noise) and class imbalance experiments. It seems that the proposed method largely outperforms other baselines only for the cases of uniform label noise with large noise ratios. For example, for the flip noise setting in label noise, it is hard to conclude the proposed method is better than others. 

Minor comments:
1. The method in [20] is named co-teaching, not co-teach.
2. Class imbalance mainly refers to the setting that both training and test sets are class imbalanced. If the training set is class imbalanced and the test set is class balanced (the setting in the paper I think), better to call it class-prior shift or label shift. 

----------------------------POST REBUTTAL-----------------------------

Thank the authors very much for the rebuttal! After reading the rebuttal and all the reviews, the authors have addressed my concerns properly. Unfortunately, I still tend to reject this paper:

1. The paper mentioned in the abstract about a novel problem setting (simultaneously handle label noise and imbalance) but they didn't present any experiment results under the claimed setting in the main paper. The authors should either add more experiments under the claimed setting (more than the experiments they presented in the rebuttal) or change the position of the paper writing.
2. I still have a concern about the effectiveness of the proposed method in experiments. For example, it seems the proposed method works not that well in flip label noise cases. Also, Figure 3 shows the proposed method cannot well differentiate correct and mislabeled data under the uniform noise. I guess the same figure for the flip label noise could be worse.

limitations_and_societal_impact: No, the authors forgot to address the limitations and potential negative societal impact of their work. I suggest adding some discussions on this point.
review_title: null

Meta-review: The paper studies the problem when data is imbalanced and has label noise. The authors proposed to simultaneously mitigate label noise and class imbalance. The key idea is to exploit the class-level weights, which is very interesting, reasonable, novel, and well-written. Experiments also show the effectiveness of the proposed method. No computational cost will be introduced. However, there are some concerns. (1) Insufficient experiments under the claimed setting to support the proposed method. The authors are expected to either add more experiments under the claimed setting (more than the experiments they presented in the rebuttal) or change the position of the paper writing. (2) The proposed method does not work that well in the cases with flip label noise. More discussions are needed to explain this. We also expect the authors to add experiments on synthesized instance-dependent label noise rather than uniform and pairflip label noise. (3) Thoughtful baselines such as combining some classic/state-of-the-art methods of training with noisy labels and class-imbalanced learning are expected. For example, the paper: Menon et al. Learning from Corrupted Binary Labels via Class-Probability Estimation, ICML 2015, which shows that the balanced error designed for imbalanced data is robust to label noise. This is a very related work which should be discussed as well. In a summary, we think the paper is not ready for publication at this round. It will be a strong paper if the concerns can be well addressed.
Decision: Accept (Poster)