Generalized DataWeighting via Class-Level Gradient Manipulation

Abstract: Can Chen, Shuhao Zheng, Xi Chen, Erqun Dong, Xue (Steve) Liu, Hao Liu, Dejing Dou

Reviews and author responses:

 Dear Reviewer,

Many thanks for your valuable and constructive comments on clarifying, correcting, and improving the materials in this paper!

As you have said,
> To conclude, I improve the rating to 5: Marginally below the acceptance threshold.

We really appreciate your recognition. As the rebuttal DDL is approaching, could you increase the rating in the system? Thanks!

 ### General Reply

Many thanks for your valuable and constructive comments on clarifying, correcting, and improving the materials in this paper! We will carefully revise the paper according to your comments as explained below.

We have conducted experiments on the Places-LT dataset and compare GDW with other meta-learning-based methods.

|  Method    |  L2RW | INSW   | MWNet  |  GDW  |
| :-----:    | :----:| :----: | :----: | :----:|
| Accuracy (%)|  $15.08$|  $17.80$ | $18.08$  | $19.17$ |

For all methods, the weight decay is set to $0.001$ and the batchsize is set to $64$. We adopt $0.01$ initial learning rate and a cosine learning rate decay rule for $10$ epochs. The weight decay is set to $0.001$. The backbone network is ResNet18 and we use the ImageNet pretrained model for initialization.

GDW achieves the best performance among all the comparison methods and outperforms MWNet by $1.09\%$. This improvement is larger than that of CIFAR100. The reason is that GDW can manipulate class-level information and thus performs better on the dataset with a larger number of classes ($365$ in Places-LT and $100$ in CIFAR100). Besides, we can observe that L2RW performs the worst and the reason may be that L2RW suffers from unstable weights [23]. 

#### Does this address your concerns？We are wondering if this rating is your final decision on this paper. Please feel free to ask questions if you have further concerns

 ### General Reply

Many thanks for your valuable and constructive comments on clarifying, correcting, and improving the materials in this paper! We will carefully revise the paper according to your comments as explained below.

> (1) A full comparison is non-trivial, otherwise, the comparison tables are misleading to the readers. Data-reweighting/cost-sensitive method is just one kind of solution for data imbalance or noisy labeling, and comparing with the methods of different kinds (such as [A, B]) is able to give readers a complete understanding of the effectiveness of the proposed method.

Thanks for your suggestion! We will highlight the importance of [A, B] in the final version. Besides, we are designing experiments where [A, B] and GDW can have a fair comparison. We are trying to remove some blocks from the backbone network in [A, B], so that [A, B] and GDW can have the same computational complexity. We will report these results in the final version.

> (2) I appreciated the extra experiments on CIFAR 100 dataset and the results on the large datasets.

Thanks for the suggestion! We are conducting more experiments on CIFAR-100 and other large datasets such as the Places-LT dataset. We hope to report some experimental results before the end of the rebuttal and we will report these results in the final version. 

Additionally, we add a clipping operation on the gradients of the class-level weights (i.e. $\boldsymbol g_i$ in Eq. 11) and find the performance of GDW under the $\mu=0.01$ imbalanced CIFAR-100 setting increases from $37.94$\% $\pm$ $1.58$\% to $38.54$\% $\pm$ $0.42$\% and the performance of GDW under the $\mu=0.1$ imbalanced CIFAR-100 setting increases from  $56.78$\%$\pm$ $0.52$\% to $57.30$\%$\pm$ $0.27$\%. One possible explanation is that the gradient clipping stabilizes the updating process of class-level weights. We will report more results and analysis in the final version. 

Does this address your concerns？We are wondering if this rating is your final decision on this paper. Please feel free to ask questions if you have further concerns

 ### General Reply

Many thanks for your valuable and constructive comments on clarifying, correcting, and improving the materials in this paper! We are carefully revising the paper according to your comments as explained below. 

> To my understanding, manipulating gradients at class level is similar to manipulating the loss or output, as the gradients are computed by class-wise cross-entropy loss.

Thanks for the good question. Manipulating gradients at the class level is quite different from manipulating the loss or output, mainly in three aspects: **interpretability**, **generality**, and **effectiveness**.

## Interpretability
Every gradient flow has some interpretable class-level information. Therefore, reweighting the gradient flows can directly manipulate the class-level information, while manipulating the loss or output is indirect and not interpretable. For example, as shown in Figure 1, every class-level gradient flow has a particular meaning: "not cat", "dog", or "not bird". Thus, reweighting the three gradient flows can directly manipulate the three kinds of information. 

Moreover, in the cross-entropy loss function, there is no individual loss term for every class, so it is impossible to reweight the loss per class directly.

The interpretability benefit has already been discussed in Figure 4-6. But to make it more clear, we conduct one more experiment under the imbalance setting to better verify the interpretability of GDW. As shown below, we report the ratio of the number of increased $\omega_i$ on $C_j$ instances in one epoch, where $C_0$ is the largest class and $C_9$ is the smallest class.

| weight/class | $C_0$ | $C_1$ |$C_2$ | $C_3$ |$C_4$ | $C_5$ |$C_6$ | $C_7$ |$C_8$ | $C_9$ |
| :-----:| :----: | :----: |:-----:| :----: | :----: |:-----:| :----: | :----: |:----: | :----: |
| $\omega_0$ | 0.036 | 0.968 | 0.973 | 0.972 | 0.965 | 0.974 |0.972  |0.976  |0.956  | 0.973 |
| $\omega_1$ | 0.887 | 0.095 | 0.912 | 0.929 | 0.907 |0.927  |0.911  | 0.922 | 0.910 | 0.920 |
| $\omega_2$ | 0.848 | 0.844 | 0.141 | 0.839 | 0.822 |0.845  |0.818  | 0.847 |0.829  | 0.802 |
| $\omega_3$ | 0.585 | 0.608 | 0.552 | 0.405 | 0.569 |0.541  |0.561  |0.559  |0.617  |0.596  |
| $\omega_4$ |0.474  | 0.521 | 0.420 | 0.460 |0.509  |0.455  |0.456  |0.482  |0.467  |0.512  |
| $\omega_5$ |0.291  | 0.261 | 0.288 | 0.252 | 0.309 | 0.701 | 0.303 | 0.267 |0.297  |0.257  |
| $\omega_6$ |0.199  |0.189  |0.169  |0.198  |0.196  |0.222  |0.778  | 0.195 |0.207  |0.182  |
| $\omega_7$ |0.117  |0.117  |0.105  | 0.084 | 0.115 | 0.079 | 0.126 | 0.920 |0.133  |0.090  |
| $\omega_8$ |0.115  |0.124  |0.178  |0.185  |0.184  |0.174  |0.191  |0.181  |0.862  |0.137  |
| $\omega_9$ |0.043  |0.050  | 0.064 | 0.061 |  0.074| 0.062 |0.097  |0.069  |0.040  |0.935  |

Note that $\omega_i$ on $C_i$ contains the "is $C_i$" information in the dataset. As a result, $\omega_i$ on $C_i$ should be large for small classes and small for large classes. As shown above, $\omega_i$ on $C_i$ (the diagonal elements) increases from $0.036$ to $0.935$ as $i$ increases from $0$ to $9$.

On the other hand, $\omega_i$ on $C_j$ ($j \neq i$) contains the "not $C_i$" information in the dataset. If $i$ is a large class, $\omega_i$ on $C_j$ ($j \neq i$) should be large and vice versa. For $\omega_i$ ($i=0, 1, 2, 3, 4$), $\omega_i$ on $C_j$ ($j \neq i$) are generally larger than $0.5$, and for $\omega_i$ ($i=5, 6, 7, 8, 9$), $\omega_i$ on $C_j$ ($j \neq i$) are generally less than $0.5$. These results align with our analysis on the interpretable information of gradient flows.


## Generality

GDW reweights every gradient flow regardless of the design of the loss function. In contrast, manipulating the loss or output often requires manual designs for different loss functions. Since most neural network training adopts SGD and its variants for optimization, GDW has better generality over manipulating the loss or output.

## Effectiveness

As shown in Table 2-4, GDW outperforms methods that manipulate the loss, the output, or the labels by a large margin in most cases. This indicates that directly manipulating class-level gradients are much more effective.

More precisely, GDW outperforms Soft-label [39] and Gen-label [37] in both settings by a large margin, which indicates that GDW is more effective than manipulating the label. Moreover, GDW outperforms GLC [19] in the noise settings only except the 40\% flip noise setting, which indicates that GDW is more effective than manipulating the output. Last but not least, GDW outperforms Focal Loss [21] and Class-Balanced Loss [13] in the imbalance setting, which indicates that GDW is more effective than manipulating the loss.



#### Does this address your concerns？ We are wondering if this rating is your final decision on this paper. Please feel free to ask questions if you have further concerns.


 Thanks for providing the extra experimental results and detailed explanations in the response! The response cancels some concerns, but I still have two main concerns:

(1) A full comparison is non-trivial, otherwise, the comparison tables are misleading to the readers. Data-reweighting/cost-sensitive method is just one kind of solution for data imbalance or noisy labeling, and comparing with the methods of different kinds (such as  [A, B]) is able to give readers a complete understanding of the effectiveness of the proposed method.

(2) I appreciated the extra experiments on CIFAR 100 dataset and the results on the large datasets.

To conclude, I improve the rating to 5: Marginally below the acceptance threshold.

 Thank you for the response! The response eliminates some of my concern on the performance. I would like to improve the rating. But I still think the paper is slightly below the acceptance criteria considering its contribution and novelty. To my understanding, manipulating gradients at class level is similar to manipulating the loss or output, as the gradients are computed by class-wise cross-entropy loss. So, I improve the rating to 5: Marginally below the acceptance threshold.

This paper proposed a new method to tackle the noisy label and imbalanced data problem in classification task. Specifically, the authors proposed to apply class-level weights on the output logits for every instance. The class-level weight predictor is learned on the clean and balanced meta set. Experiments in two settings show that the proposed method has little improvement.   1.	The proposed method is well-motivated and well-illustrated. The Zero-mean Constraint trick (Eq 4) is interesting.
2.	The proposed class-level weighting is similar to soft label methods which reassign the ground-truth label for training. This paper lacks the discussion of the difference. 
3.	I am concerned about learning the weight predictor w(phi) on the clean meta set, which will be used for the noised training set. How can w(phi) learned on clean and balanced data work well on noisy and imbalanced data?
4.	The proposed method has negligible improvements in most experiments. Therefore the effectiveness of the proposed method is limited. 
5.	Suggestion: The term “class-level weighting” is a bit confusing. Perhaps the authors can find more accurate name.
 The authors have told a possible limitation. Societal impact is not applicable.

 ### 0. General Reply

Many thanks for your valuable and constructive comments on clarifying, correcting, and improving the materials in this paper! We will carefully revise the paper according to your comments as explained below.

### 1. Clarity
 1. Thanks for your comments. This is a typo: $l$ should be replaced by $t$, denoting the target label position. We will revise it in the final version. 
 2. Thanks for the good question. The definition of $g_i$ is simply 
    $$
    \frac{1}{m}\sum_{i=1}^m\nabla_{\omega_i} l_{meta}(x_i^v, y_i^v;\boldsymbol \theta_\tau(\boldsymbol \phi_{\tau-1})),
    $$
    which is the intermediate gradient during the back-propagation on meta-set. This gradient is automatically reserved during the back-propagation in PyTorch. We will explicitly write it in the final version.

### Limitations And Societal Impact
Thanks for the suggestion. We will add the two parts in the final version.

 ### 0. General Reply

Many thanks for your valuable and constructive comments on clarifying, correcting, and improving the materials in this paper! We will carefully revise the paper according to your comments as explained below.
### 1. Main Review
 1. Thanks for the comments!
 2. Thanks for the good question. Although similar to the soft label method, GDW is quite different in the following two aspects:
     1. GDW is a generalization of instance weighting while the soft label method has nothing to do with instance weighting. 
     2. GDW controls the gradient flows by multiplying weights while the soft label method only modifies the label term $y$.

    Besides, we compare two meta-learning soft label methods [37, 39] experimentally and the results show that GDW is more effective.

 3. Thanks for the good question. The weight predictor $\omega(\phi)$ leverages the clean and balanced information in the meta-set. Therefore, although the training set is noisy and imbalanced, the weight predictor trained on the meta-set can improve training by applying weights on the gradient flows. The connection between the two kinds of information "noisy imbalanced" and "clean balanced" is explicitly constructed during the bi-level optimization process.
 4. Thanks for your comments. 

    In the uniform noise setting and the real-world setting, GDW outperforms SOTA methods by a large margin, especially under severe noise settings, as illustrated in **Table 2** and **Table 5**.

    In the flip noise setting, except that GLC [19] performs better in the 40% setting, GDW always gets the best performance compared with other methods and gets large performance gain in most settings (e.g. 20%, 40% on CIFAR10, and 40% on CIFAR100). In the 20% setting on CIFAR100, GDW gets comparable results compared to the second-best. GLC [19] estimates a label corruption matrix to correct the prediction. This estimation is quite easy under the 40% flip noise (flip one class to another class w.p. 40%). This is why GLC gets better results. However, under uniform noise scenarios, as shown in **Table 2**, GLC yields bad performance compared to GDW. This is because uniform noise is harder to estimate and correct. Moreover, we stated in Line 211-216 that 40% flip noise setting is rare in real-world datasets. 

    Furthermore, we conduct new experiments to verify the performance of GDW in the mixed setting, i.e. the coexistence of label noise and class imbalance. Specifically, we compared with the mostly second-best method Meta-Weight-Net [23] under the mixed setting of uniform noise and class imbalance on CIFAR10 and CIFAR100. The results are as follows:

    |           Settings              |MW-Net   |GDW   |
|:----:|:----:|:----:|
|CIFAR10 + unif $0.4$ + imb $0.1$| $71.54$    |$76.30$  |
|CIFAR10 + unif $0.6$ + imb $0.1$| $61.62$    |$70.24$  |
|CIFAR10 + unif $0.4$ + imb $0.01$| $48.04$    |$48.53$  |
|CIFAR10 + unif $0.6$ + imb $0.01$| $39.51$    |$40.07$ |
|CIFAR100 + unif $0.4$ + imb $0.1$ |$36.10$    |$38.20$  |
|CIFAR100 + unif $0.6$ + imb $0.1$| $24.80$    |$25.40$  |
|CIFAR100 + unif $0.4$ + imb $0.01$|$21.26$    |$22.07$  |
|CIFAR100 + unif $0.6$ + imb $0.01$|$12.75$   |$14.15$  |

    GDW also shows great performance gain over MW-Net, which means GDW can simultaneously tackle both problems. We will conduct more comprehensive experiments and put the results in the final version of this paper.
 5. Thanks for your suggestion! We will try to find another proper name for "class-level weighting".

 ### 0. General Reply

Many thanks for your valuable and constructive comments on clarifying, correcting, and improving the materials in this paper! We will carefully revise the paper according to your comments as explained below.

### 1. Limitations And Societal Impact

 1. Thanks for the good question. Although similar to the meta-learning method for label correction, GDW is quite different in the following two aspects:
     1. GDW is a generalization of instance weighting while the meta-learning method for label correction has nothing to do with instance weighting. 
     2. GDW controls the gradient flow by multiplying weights while the meta-learning method for label correction only modifies the label term $y$.
 2. Thanks for the good question. In our experiments, we adopted the pre-trained ResNet-18 model as the backbone model and trained it for only limited epochs (5 epochs). If switched to other settings, such as Resnet-50 for 80 epochs, we have confidence that GDW can improve the SOTA performance on Clothing1M.

 ### 0. General Reply

Many thanks for your valuable and constructive comments on clarifying, correcting, and improving the materials in this paper! We will carefully revise the paper according to your comments as explained below.

### 1. Main Review(Cons):
#### 1.1 Method

 1. Thanks for your comment. Exactly, [D] proposes class-level weighting in multi-label imbalance classification, and the weights are directly applied to Logistic Regression loss. However, in our statement, we claim that we are the first to propose class-level weighting on gradient flows, which is different from [D]. We will emphasize *single-label classification* in the final version.
 2. Thanks for the good question. In class imbalance scenarios, the goal for class-level weights is to balance two kinds of information: "is $i$" and "is not $i$" for each class $i$. "is $i$" is contained only in class $i$ instances, while "is not $i$" is contained in all classes $j\neq i$. To balance the two kinds of information, GDW utilizes class-level weights to adjust the overall distribution of classes. We illustrate the motivation in the single-label classification on the imbalanced data in Line 272-281.
 3. Thanks for the good question. We adopt the cross-entropy loss function, which is the default and effective loss function for most single-label classification tasks. In this cross-entropy loss function, there is no individual loss term for every class, so we can not reweight the loss per class directly. Furthermore, since most neural network training adopts SGD and its variants for optimization, applying weights to gradients directly can more clearly reflect the importance of each gradient flow, which corresponds to some interpretable class-level information, as shown in **Figure 1** and **Figure 4**. Moreover, applying weights to gradients is unaware of the loss functions, which means GDW can also be combined with different loss function designs for better performance.

### 2. Experiments are not sufficient

 1. We are running experiments on large-scale datasets like ImageNet, but have not got the final results yet. However, for CIFAR datasets, we conduct extensive experiments on different settings, including the uniform noise setting and the flip noise setting with different noisy ratios, the class imabalnce setting with different imbalance factors and the real-world setting on Clothing1M. If time permits, we will also report the results on large-scale datasets in the final version.
 2. Thanks for your comments. 

    In the flip noise setting, except that GLC [19] performs better in the 40% setting, GDW always gets the best performance compared with other methods and gets large performance gain in most settings (e.g. 20%, 40% on CIFAR10, and 40% on CIFAR100). In the 20% setting on CIFAR100, GDW gets comparable results compared to the second-best. GLC [19] estimates a label corruption matrix to correct the prediction. This estimation is quite easy under the 40% flip noise (flip one class to another class w.p. 40%). This is why GLC gets better results. However, under uniform noise scenarios, as shown in **Table 2**, GLC yields bad performance compared to GDW. This is because uniform noise is harder to estimate. Moreover, we stated in Line 211-216 that 40% flip noise setting is rare in real-world datasets. 

    Futhermore, we conduct new experiments to verify the performance of GDW in the mixed setting, i.e. the coexistence of label noise and class imbalance. Specifically, we compared with the mostly second-best method Meta-Weight-Net [23] under the mixed setting of uniform noise and class imbalance on CIFAR10 and CIFAR100. The results are as follows:

    |           Settings              |MW-Net   |GDW   |
|:----:|:----:|:----:|
|CIFAR10 + unif $0.4$ + imb $0.1$| $71.54$    |$76.30$  |
|CIFAR10 + unif $0.6$ + imb $0.1$| $61.62$    |$70.24$  |
|CIFAR10 + unif $0.4$ + imb $0.01$| $48.04$    |$48.53$  |
|CIFAR10 + unif $0.6$ + imb $0.01$| $39.51$    |$40.07$ |
|CIFAR100 + unif $0.4$ + imb $0.1$ |$36.10$    |$38.20$  |
|CIFAR100 + unif $0.6$ + imb $0.1$| $24.80$    |$25.40$  |
|CIFAR100 + unif $0.4$ + imb $0.01$|$21.26$    |$22.07$  |
|CIFAR100 + unif $0.6$ + imb $0.01$|$12.75$   |$14.15$  |

    GDW also shows great performance gain over MW-Net, which means GDW can simultaneously tackle both problems. We will conduct more comprehensive experiments and put the results in the final version of this paper.
 3. Thanks for the comments. [22,23,24] are published on ICML 2018, NIPS 2019 and NIPS 2019, respectively. These methods are the recent SOTA methods on data reweighting. We will revise the reference in the final version. 

	For [A, B, C], we will add the citations in the final version. However, we have not figured out how to perform a fair comparison between GDW and [A, B], because [A, B] both introduce extra trainable parameters and modify the base network structure. In addition, [C] is designed for multi-label classification while our method is designed for single-label classification.

This work proposed a re-weighting method (GDW), to solve the noisy labeling and long-tailed training data issues in the single-label image classification task. The proposed GDW learns the weights considering both class information and instance information, and the computation of the weight is formulated in a meta-learning setting, using a clean validation dataset. The proposed method is evaluated on Cifar-10, Cifar-100, and Cloth1M datasets, and shows effectiveness in dealing with noisy labeled and imbalanced image classification.
  Pros:

(1) This work proposed a re-weighting method for solving noisy label and imbalanced data issues simultaneously. Formulating these two problems in a uniform solution is novel and interesting. 

(2) The paper is well-written and easy to follow.

(3) Learn to weight, implemented in a meta-learning framework, is reasonable.

Cons:

(I) Method:

(1) The class-wise weighting has been discussed in [D] to deal with the multi-label classification imbalance issue. Thus, this work is not the first work in this direction and the statements in Line 70 may need some change.

(2) For the noisy labeling issue, de-weighting the class of noisy positive class is reasonable, but what is the motivation in single-label classification on imbalanced data?

(3) Why not re-weighting the loss per class? What is the benefit of re-weighting gradient?

(II) Experiments are not sufficient:

(1) For long-tailed image classification, there is a lack of large-scale image classification datasets, such as ImageNet. Just testing on Cifar is not enough due to the small data quantity and a small class number.

(2) From Tab3.and 4, the proposed method does not outperform the competitors with a large margin, and thus it is hard to verify the effectiveness of this work.

(3) In long-tailed image classification, most competitors [22-24] are published on Arixv? Lack of sufficient comparison with the recent SoTa methods, such as [A, B, C].

[A] Long-tailed Recognition by Routing Diverse Distribution-Aware Experts, ICLR21.

[B] Large-Scale Long-Tailed Recognition in an Open World, CVPR19.

[C] Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets, ECCV20.

[D] Single-Label Multi-Class Image Classification by Deep Logistic Regression, AAAI19. 



My main concerns of this work are about the method motivations and experimental comparison. Specifically,

(I) Method:

(1) The class-wise weighting has been discussed in [D] to deal with the multi-label classification imbalance issue. Thus, this work is not the first work in this direction and the statements in Line 70 may need some change.

(2) For the noisy labeling issue, de-weighting the class of noisy positive class is reasonable, but what is the motivation in single-label classification on imbalanced data? 

(3) Why not re-weighting the loss per class? What is the benefit of re-weighting gradient?



(II) Experiments are not sufficient:

(1) For long-tailed image classification, there is a lack of large-scale image classification datasets, such as ImageNet. Just testing on Cifar is not enough due to the small data quantity and a small class number.

(2) From Tab3.and 4, the proposed method does not outperform the competitors with a large margin, and thus it is hard to verify the effectiveness of this work.

(3) In long-tailed image classification, most competitors [22-24] are published on Arixv? Lack of sufficient comparison with the recent SoTa methods, such as [A, B, C].

[A] Long-tailed Recognition by Routing Diverse Distribution-Aware Experts, ICLR21.

[B] Large-Scale Long-Tailed Recognition in an Open World, CVPR19.

[C] Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets, ECCV20.

[D] Single-Label Multi-Class Image Classification by Deep Logistic Regression, AAAI19.


This paper focused on label noise in class-imbalance situation, which is an important but challenge problem in reality. The authors claimed the deficiencies of previous works that adopted meta-learning frameworks, and accordingly proposed the GDW in further utilizing class-level information. This paper conducted extensive experiments on both synthetic and real-world data sets, and the authors claimed the superiority of their proposal.   This paper focused simultaneously on label noise and class imbalance problems, an important but challenging setup in real-world applications. As a canonical methodology, many previous works adopted meta-learning to solve both issues by leveraging a clean and unbiased meta set. However, the authors claimed that these methods only employ the instance-level weights, which cannot fully utilize class-level information within each instance. Accordingly, by class-level gradient manipulation, the authors proposed the Generalized Data Weighting (GDW), which can be viewed as a generalization of the vanilla IW. 

In GDW, the authors adopted a series of techniques to boost the performance of the final algorithm. Specifically, to maintain the loss structure, the authors introduced the zero-mean constraint; for the effectiveness of the learning procedure, the authors improved the MWNet with minimal computational cost. 

In my personal view, I believe this paper has some contributions. The authors proposed a general framework for meta-learning in the literature of label noise learning, which is well-motivated and could address the drawbacks in previous works.  I have a little question to consult the authors. By the discussion below Eq. 6, it seems that the proposal can also be viewed as a meta-learning method for label correction. I think it would be great if the authors could make some clarifications.

I believe this paper is well-motivated and of high quality. However, the experimental results on Clothing1M can hardly beat the performance for many state-of-the-art methods. Could the authors provide some explanation, or is there any space for further improvements?


The paper presents a method called Generalized Data Weighting (GDW) to mitigate label noise and class imbalance problems simultaneously. GDW works by re-weighting instances at the class level and it leverages meta-learning to learn the optimal weights. The main difference of presented approach from existing methods is that it allows to keep useful information contained in some of the logits (such as logits corresponding to non-target classes).  Also, the authors design a two-stage scheme embedded in the bi-level optimization framework for efficient computation of class-level weights without extra cost. Finally, the paper discusses importance of zero-mean constraint on class-level weights for stable training.
The authors conduct extensive experiments on CIFAR10 and CIFAR100 datasets in the label noise and class imbalance settings and compare GDW with other SOTA methods. Also, the model is evaluated on the real-world dataset with label noise, Clothing1M. The results show that GDW outperforms existing methods in almost all experiments. Finally, the authors explain an intuition behind GDW by showing how class-level weights change during training in samples with noisy labels.   Originality: The main novelty of the paper is the proposed method GDW which re-weights data at the class-level during training (instead of instance-level) to cope with label noise and class imbalance problems. The paper contains an extensive literature review of the existing methods. 

Quality: The submission is technically sound and conduct multiple experiments comparing the proposed approach to the SOTA methods. The results suggest that GDW significantly outperforms existing methods while the method does not introduce additional computational cost. 

Clarity: The paper is clearly written and explains all important concepts as well as the intuition behind the proposed method. Although, I have a few questions for the authors: 
1. In equation 4, what is l in the summation? 
2. How exactly gradients g_i on ω_i are computed? Could you please clarify on how ω_i is updated? 

Significance: I believe that the paper is significant as it addresses two important problems occurring in the real-world datasets: label noise and class imbalance. The proposed method is well motivated and outperforms SOTA methods in multiple experiments.  The paper does not address limitations and potential negative societal impact, although I am not sure that it is applicable to the paper. 

Meta-review: There is a strong disagreement with this paper, with scores ranging as 8,7,5, and 4. The reviewer who provided a 4 said she increased her score to 5 but this has not propagated. In this case, the rating would be 6.25 rather than 6. 
While the accept reviews are stronger score wise, the reasons to reject (marginal improvements and novelty) have been far more detailed than the reasons to accept (also centered on improvements and novelty).

My own reading would align with the “accept” reviewers: novelty and improvements are genuine.

How novel is this contribution seems to hinge on a key question: is the proposed method similar to methods that would modify the loss function (see for instance https://arxiv.org/pdf/1811.08400.pdf publishded by one of the reviewers who supports reject) or weight instances ? One can observe that the proposed method does not reduce to a modification of the loss function, but does it add something fundamentally different ? The example in figure 1 is excellent: the authors clearly show that their scheme performs differently than instance weighting. 
Furthermore, I do not think any modification of the loss function could achieve the same results as a gradient-based method.

Improvements are indeed small, but this is what one would expect with a change in the loss function (we are not talking about a new architecture). As they do not incur added computational cost, this is something one could add on top of any system. Basically, the approach does no harm if there is no label noise, but seems to always improve as label noise increases, which is enough for adoption.

My main concern here remains that despite the claim that the method adds no computational complexity, we still do not have results on a larger dataset such as ImageNet (as promised by the authors)


Decision: accept