Generalized DataWeighting via Class-Level Gradient Manipulation
Authors: Can Chen
Shuhao Zheng
Xi Chen
Erqun Dong
Xue Liu
Hao Liu
Dejing Dou
Venue: NeurIPS 2021 Poster (2021)
Keywords: instance weighting
meta-learning
bi-level optimization
classification
chain rule

Abstract: Label noise and class imbalance are two major issues coexisting in real-world datasets. To alleviate the two issues, state-of-the-art methods reweight each instance by leveraging a small amount of clean and unbiased data. Yet, these methods overlook class-level information within each instance, which can be further utilized to improve performance. To this end, in this paper, we propose Generalized Data Weighting (GDW) to simultaneously mitigate label noise and class imbalance by manipulating gradients at the class level. To be specific, GDW unrolls the loss gradient to class-level gradients by the chain rule and reweights the flow of each gradient separately. In this way, GDW achieves remarkable performance improvement on both issues. Aside from the performance gain, GDW efficiently obtains class-level weights without introducing any extra computational cost compared with instance weighting methods. Specifically, GDW performs a gradient descent step on class-level weights, which only relies on intermediate gradients. Extensive experiments in various settings verify the effectiveness of GDW. For example, GDW outperforms state-of-the-art methods by $2.56\%$ under the $60\%$ uniform noise setting in CIFAR10. Our code is available at https://github.com/GGchen1997/GDW-NIPS2021.

Generalized Data Weighting via Class-level Gradient Manipulation
Can Chen'; Shuhao Zheng'; Xi Chen', Erqun Dong', Xue Liu', Hao Liu', Dejing Dou
'McGill University, 'The Hong Kong University of Science and Technology, 'Baidu Research
(can.chen, shuhao zzeng, erquı (rqqundoong)eee dong) @mail ciciiilmmmllll. worgill.co ca
xi.chent wwwmn11mcgill.coom xueliu@cs megill. ca, liuh@ust. hk, doude caadejing@baiine ing@baidu. com
Abstract
Label noise and class imbalance are two major issues coexisting in real-world datasets. To alleviate the two issues, state-of-the-arr methods reweight each instance by leveraging a small amount of clean and unbiased data. Yet, these methods overlook class-level information within each instance, which can be further utilized to improve performance. To this end, in this paper, we propose Generalized Data Weighting (GDW) to simultancously mitigate label noise and class imbalance by manipulating gradients at the class level. To be specific, GDW unrolls the loss gradient to class-level gradients by the chain rule and reweights the flow of each gradient separately. In this way, GDW achieves remarkable performance improvement on both issues. Aside from the performance gain, GDW efficiently obtains class-level weights without introducing any extra com- putational cost compared with instance weighting methods. Specifically, GDW performs a gradient descent step on class-level weights, which only relies on intermediate gradients. Extensive experiments in various settings verify the fectiveness of GDW. For example, GDW outperforms state-of-the-ae methods by 2.56% under the 60% uniform noise setting in CIFARIO. Our code is available at https://ght ////ggttatt thut.uu..cc. com httpc//Ghchee111977/X/--11SSSS5
Introduction
Real-world classification datasets often suffer from two issues, i.c., label noise and class im- balance (2). On the one hand, label noise often results from the limitation of data generation, e.g., sensor errors and mislabeling from crowdsoureing workers [4. Label noise misleads the training process of DNNs and degrades the model performance in various aspects [5 6 7 On the other hand, imbalanced datasets are either naturally long-tailed
[8] or biased from the real-world distribution
due to imperfect data collection [10, I.. Training with imbalanced datasets usually results in poor
classification performance on weakly represented classes 12
14]
Even worse, these two issues
often coexist in real-world datasets (15).
To prevent the model from memorizing noisy information, many important works have been proposed, including label smoothing [16), noise adaptation (17), importance weighting [18), GLC [19], and
Co-teach 200 Meanwhile, [12
21 propose effective methods to tackle class imbalance.
However, these methods inevitably introduce hyper-parameten (e.g., the weighting factor in [3] and the focusing parameter in (2)). compounding real-world deployment.
Inspired by recent advances in meta-learning, some works [22 23 24 25 propose to solve both issues by leveraging clean and unbiased meta These methods treat instance weights hyper-
Equual contribution; Names listed in alphabetical order.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

parameters and dynamically update these weights to circumvent hyper-paramede parameter tuning. Specifically, MWNet [23] adopts an MLP with the instance loss as input and the instance weight as output. Due to the MIP, MWNet has better scalability on large datasets compared with INSW [24 which assigns each instance with a learnable weight. Although these methods can handle label noise and class imbal- ance to some extent, they cannot fully utilize class-level information within each instance, resulting in the potential loss useful information. For example, in a three-class classification task, every instance every logit corresponds to class-level gradient flow which stems from the loss function and back-propagates. These gradient flows represent three kinds of in- formation: "not cat", "dog", and "not bird". Instance weighting methods (23 22
has three logits. As shown in Figure [1
alleviate label noise by downweighting all the gradient flows of the instance, which discards three kinds of information simultancously. Yet. downweighting the "not bird" gradient flow is waste of information. Similarly, in class imbalance scenarios, different gradient flows represent different class-level information. Therefore, it is necessary reweight instances at the class level for better information usage.
To this end, we propose Generalized Data Weighting (GDW) to tackle label noise and class imbalance by class-level gradient manipulation. Firstly, we introduce class-level weights to rep- resent the importance of different gradient flows and manipulate the gradient flows with these class-level weights. Secondly, we impose zero- mean constraint on class-level weights for stable training. Thirdly, to efficiently obtain class-level weights, we develop a two-stage weight gener- ation scheme embedded in bi-level optimization. As a side note, the instance weighting meth- ods 22 23 24 25 can be considered special cases of GDW when class-level weights within any instance are the same. In this way, GDW achieves impressive performance improvement in various settings.
"dog"
Gradient flow info.
"not cat"
"dog"
"not bird"
Instance weighting
0.3
Class-level weighting
0.3
0.3
0.8
Figure 1: noisy instance (e.g. cat mislabeled as "dog"). all gradient flows are downweighted by instance weighting. Although the gradient flows for "dog" and "not cat" contain harmful information, the gra- dient flow for "not bird" is still valuable for train- ing, which should not be downweighted.
Motivation for class- weighting. For
To sum up, our contribution is two-fold:
1. For better information utilization, we propose GDW, generalized data weighting method, which better handles label noise and class imbalance. To the best of our knowledge, we are the first to propose single-label class-level weighting on gradient flows.
2. To obtain class-level weights efficiently, we design a two-stage scheme embedded a bi-level optimization framework, which does not introduce any extra computational cost. To be specific, during the back propppropngttion we store intermediate gradients, with which we update class-level weights via a gradient descent step.
2 Related Works
2.1 Non-Meta-Learrning Methods for Label Noise
Label noise is a common problem in classification tasks 6
To avoid overfitting to label noise, [16] propose label smoothing regularize the model. 17 26 form different models indicate the
5).
18 estimate an importance weight for each instance to represent its value to the model. 220] train two models simultaneously and let them teach each other in every mini-batch. However, without a clean dataset, these methods cannot handle severe noise (22). [19] correct the prediction of the model by estimating the label corruption matrix via clean validation set, but this matrix is the same across all instances. Instead, our method generates dynamic class-level weights for every instance improve training.
relation between noisy instances and clean instances.
2.2 Non-Meta-Learrinn Methods for Class Imbalance
Many important works have been proposed to handle class imbalance 27
29
4
28 29 propose to over-sample the minority class and under-sample the majority class. 27 30

---

Table 1: Related works comparison. "Noise" and "Imbalance" denote whether the method can solve label noise and class imbalance. "Class-level" denotes whether the method utilizes class-level information each instance, and "Scalability" denotes whether the method can scale to large datasets.
Rocal 211
Balancol
Co-teaching
20
GLC
1T
LRWW
22
INSW [4]
MWNN
Soft-label
Gen-label 40
GDW
Imbulance
Class-level
Scalability
learn class-dependent cost matrix obtain robust representations for both majority and minority classes. [12 13, 21 [1] design reweighting scheme to rebalance the loss for each class. These methods are quite effective, whereas they need to manually choose loss functions or
32 B3] manipulate the feature space to handle class imbalance while introducing extra model parameters. (31] decouple representation learning and classifier learning on long-tailed datasets, but with extra hlppe--prametter tuning. In contrast, meta-learning methods view instance weights as hyper-parameters and dynamically update them via a meta set to avoid hyper--aramett tuning.
2.3 Meta-Learning Methods
With recent development in meta-learning 34 35 36
many important methods have been proposed
handle label noise and class imbalance meta set [37 38 22
39
23
24
25 40) [38] propose
MentorNet to provide data-driven curriculum for the base network to focus on correct instances. To distill effective supervision, [1
estimate pseudo labels for noisy instances with meta set. provide
dynamic regularization. 42 40 treat labels as learnable parameters and adapt them to the model's
state. Although these methods can tackle label noise, they introduce huge amounts of learnable parameters and thus cannot scale to large datasets. To alleviate class imbalance, (37] describe method to learn from long-tailed datasets. Specifically, [37 propose to encode meta-knowledge into meta-network and model the tail classes by transfer learning.
Furthermore, many meta- meta-lcarning methods propose to mitigate the issues by reweighting every 25). [43] equip each instance and each class with a learnable parameter to govern their importance. By leveraging a meta set, 22 24 25 learn instance weights and model parameters via li-level optimization to tackle label noise and class imbalance, [22] assign weights to training instances only based on their gradient directions. Furthermore, [24] combine reinforce learning and meta-learning and treats instance weights as rewards for optimization. However, since each instance directly assigned with learnable weight, INSW can not scale to large datasets. adopt weighting network to output weights for instances and use bi-level optimization to jointly update the weighting network parameters and model parameters. Although these methods handle label noise and class imbalance by reweighting instances, a scalar weight for every instance cannot capture class-level information, as shown in Figure [ Therefore, we introduce class-level weights for different gradient flows and adjust them to better utilize class-level information.
instance [22] 43 23 24
Meanwhile, 23 25
We show the differences between GDW and other related methods in Table
Method
3.1 Notations
In most classification tasks, there is a training set Dirain (1 W and assume there is also clean unbiased meta set Dmntt {(x]', M We aim to alleviate label noise and class imbalance Dirain with D Dmata- The model parameters are denoted as 0, and the number classes denoted C.
3.2 Class-level Weighting by Gradient Manipulation
To utilize class-level information, we learn class-level weight for every gradient flow instead of scalar weight for all C gradient flows in 23 Denote L as the loss of any instance. Applying the

---

chain rule, we unroll the gradient of L w.r.t. 0 as
0L aL Øl
VoL
22,
(I)
00
01 00
where E RC represents the predicted logit vector of the instance. We introduce class-level weights w E Rº and denote the jth component of w as wy- To indicate the importance of every gradient flow, we perform an element-wise product fw (+) on D₁ with w. After this manipulation, the gradient becomes
dC al
fw (VoL) =
= (w @ D2 = D/D2.
0l
00
where @ denotes the element-wiss product of two vectors. Note that wy represents the importance of the jth gradient flow. Obviously, instance weighting is special case of GDW when elements of w 46] adopt the Seftmax-Crossaatray CrossEntropy loss. In this case, we have D1 = P where RC denotes the probability vector output by softmax and y E R denotes the one-hot label of the instance (see Appendix for details).
are the same. Most classification tasks 44 45
As shown in Figure D for a noisy instance (e.g., cat mislabeled as "dog"), instance weighting methods assign low scalar weight to all gradient flows of the instance. Instead, GDW assigns class-level weights to different gradient flows by leveraging the meta set. In other words, GDW tries downweight the gradient flows for "dog" and "not cat", and upweight the gradient flow for "not bird". Similarly, in imbalance settings, different gradient flows have different class-level information. Thus GDW can also better handle class imbalance by adjusting the importance of different gradient flows.
3.3 Zero-mean Constraint on Class-level Weights
To retain the Softma-- CrossEntropy loss structure, i.e. the p y form, after the manipulation, we impose a zero-mean constraint on Dí. That is, we analyze the jth element of Dí (see Appendix B.1 for details):
()(P) - y,) y,) -w: (p', yy)
Σ WaPk Wi D'J.
(3)
wi is the weighted probability, and w; denotes the class-level weight at the target where p'} Ex waps
(label) position. We observe that the first term in Eq. (3) satisfies the structure of the gradient of the Sofimax- Oefimar-CCrosEntrey ErossEntropy loss, and thus propose to eliminate the second term which messes the structure. Specifically, we let
DiPeE Wi = we Ejjt WyPp
Pt
where pt is the probability of the target class. Note that E, W,Yj = w,, and thus we have
Ewj(p, - y5) 0.
This restricts the mean of D to be zero. Therefore, we name this constraint as the zero-mean constraint. With this, have
D = w (p' y)-
Eq. (6) indicates that w adjust the gradients in two levels, i.e., instance level and class level. Namely, the sealar wi acts as the instance-level weight previous instance weighting methods [22 23 24 25), and the wy"s 's are the class-level weights manipulating gradient flows by adjusting the probability from p p'.
3.4 Efficient Two-stage Weight Generation Embedded in Bi-level Optimization
In this subsection, we first illustrate the three-step bi-level optimization framework in 23 Further- more, we embed a two-stage scheme in the bi-level optimization framework to efficiently obtain class-level weights, with which manipulate gradient flows and optimize model parameters.

---

91
9w
BP on Dincts
Tle
Store class-level gradients
.90
Tw
R
Figure 2: Two-stage Weight Generation. "BP" denotes the back-propagation in Step 2 of the bi-level optimization framework, denotes the intermediate gradients w.r.t. w. θ denotes the minus operator. Note that w the first-stage (instance-level) weight and w' is the second-stage (class-level) weight.
Three-step Bi-level Optimization. Generally, the goal of classification tasks is to obtain the optimal
model parameters 0 by minimizing the average loss on Drrains denoted as N Σi= train (Te, U:: 0). As an instance weighting method, [23] adopt a three-layer MLP parameterized by Φ as the weighting network and take the loss of the jt instance as input and output scalar weight w;. Then 0* is optimized by minimizing the instance-level weighted training loss:
1
0* (Φ) arg min
wi (er)ler (T Me:
To obtain the optimal W/, they propose to use a meta set meta-lno and minimize the meta-loss to obtain x²:
Φ- arg min
M lont( vi; 0 (Φ)).
Since the optimization for 0 (Φ) and 6 is nested, they adopt online strategy to update 0 and &b with three-step optimization loop for efficiency. Denote the two sets of parameters at the 71th loop as 0, and Φ, respectively, and then the three-step loop is formulated as:
Step Update 0,-1 to 0, (+) via an SGD step on mini-batch training set by Eq. 7
Step 2 With (Φ)) update Φ,-1 to Φ,, via an SGD step on a mini-batee meta set by Eq.
Step With Φ,, update 0 to 0, via an SGD step on the same mini- batch training set by Eq.
Instance weights in Step 3 are better than those in Step 1, and thus used to update 0,-1-
Two-stage Weight Generation. To guarantee scalability, we apply the same weighting network in 23 to obtain weights. To efficiently train & and 0, we also adopt the three-step bi-level optimization framework. Moreover, we propose an efficient two-stage scheme embedded in Step 1-3 generate class-level weights. This process does not introduce any extra computational cost compared to (2 We keep the notations of 0, and Φ,, unchanged.
The first stage is embedded in Step 1. Explicitly, we obtain the first-stage class-level weights wi = w, by cloning the output of the weighting network for C times. Then we leverage the cloned weights W; to manipulate gradients and update 0 with a mini-batch of training instances:
0 (w--1) +- θ,-1 170 Σ fui( 1) (Volrainn (Fs. Mei 0,-1))
where n is the mini- batch size, is the learning rate of 0, and fwr(ts, (-) is the gradient manipula- tion operation defined in Eq. (2).
The second stage is embedded Step 2 and Step 3. Specifically in Step 2, GDW optimizes & with mini-batch meta set:
o + 01-1 - 1 V Vo-- (x) .... vi: (+
(10)
where m is the mini-batch size and no is the learning rate of Φb During the back-propagation in updating o.. GDW generates the second-stage weights using the intermediate gradients gi wi.
5

---

Note that 1 E(=1 V meeaa Cheta(() (2) V.'; (0 and we have
gi
w/ = wi - clip(T
-
c, c),
(1I)
|gl|
where ||g/l denotes the el norm of class-level gradients within minibatch and c 0.2 represents the clip parameter. Then we impose the zero-mean constraint proposed in Eq. (4 on w/ which is later used in Step 3 to update - Note that the two-stage weight generation scheme does not introduce any extra computational cost compared to MWNet because this generation process only utilizes the intermediate gradients during the prood-ppopggtion.. In Step 3, w/, to manipulate gradients and update the model parameters -
0 r 0- - 1) - Σ Swl. (Voltrain (F Yzi 0 -11) 1=1
(12)
The only difference between Step 1 and Step is that the weighting network w; to optimize A. Since we only introduce & as extra learnable parameters, GDW can scale large datasets. We summarize GDW in Algorithm []] Moreover, visualize the two-stage weight generation process in Figure 2 for better demonstration.
w/ instead of the cloned output of
Algorithm 1 Gennrallizd Data Weighting via Class-Level Gradients Manipulation
Input: Training set: Dtrain, Meta set: Dieeta batch size n, # of iterations T Initial model parameters: 0. initial weighting network parameters: Φp
Output: Trained model: Or
for <- to do
2
9e 1 < SampleFrom/ (irain)
{x² li=1= < SampleFrom (Dmeta)
4
Generate w; from L, via the weighting network parameterized by Φ--1
Manipulate gradients by Eq.
(2 and update 0,, by Eq.
Update Φ by Eq. 110):
Update to w' by Eq. 11 and constrain w/ by Eq.
Manipulate gradients with w/ by Eq. (2) and update 0, by Eq. 12
4
Experiments
We conduct extensive experiments classification tasks to examine the performance of GDW. We compare GDW with other methods in the label noise setting and class imbalance setting Section 4. and Section 4.2 respectively. Next, perform experiments the real-world dataset Clothing M 4 Section We conduct further experiments to verify the performance of GDW in the mixed setting, i.e. the coexistence of label noise and class imbalance (see Appendix F for details).
4.1 Label Noise Setting
Setup. Following 23 we study two settings of label noise: Uniform noise: every instance's label uniformly flips to other class labels with probability p, b) Flip noise: each class randomly flips to another class with probability p. Note that the probability represents the noise ratio. We randomly select 100 clean images per class from CIFAR1O [47] as the meta set (1000 images in total).
Similarly, select total of 1000 images from CIFAR 100 its meta set. We use ResNet-32 44 as the classifier model.
Comparison methods. We mainly compare GDW with meta-learning methods: 1) L2RW 22
which assigns weights to instances based on gradient directions; 2) INSW (24), which derives
instance weights adaptively from the meta set; 3) MWNet [33; 4) Soft-label 42), which learns
label smoothing parameter for every instance; 5) Gen-label 40) which generates a meta-soft-lahbe
for every instance. We also compare GDW with some traditional methods: 6) BaseModel, which
trains ResNet-32 on the noisy training set; 7) Fine-tuning, which uses the meta set to fine-tune the trained model in BascModel; 8) Co-teaching [20); 9) GLC [9].

---

Table 2: Test accuracy on CIFARIO and CIFARIO0 with different uniform noise ratios.
CIFARIO
CIFART00
Dataset
0%
40%
60%
0%
40%
60%
BaseModel
92.73 0.37
84.38
0.32
77.92
0.29
70.42
0.54
57.28
0.80
46.86
1.54
Fine-tuning
92.77
0.37
84.73
0.47
78.41
0.31
0.57 70.52
57.38
0.87
47.06 + 1.47
Co-teaching
91.54 0.39
85.26
0.56
78.90
6.64
68.33
59.58
0.83
37.74 +: 2.60
GLC
90.85 + 0.22
86.12 1 0.54
81.55 : 0.60
65.05 0.59
56.99
+ 0.82
41.74 1.98
L2RW
$0.70:10.55
84.60 1.21
79.98 1: 1.18
63.40 1.31
47.06
1: 4.84
36.02 2.17
INSW
92.70
$4.88
0.64
78.77 -
70.52 0.39
57.11:0.1
48.00 +: 1.16
MWNet
92.95 + 0.33
86.46 1 0.31
0.94
70.64 0.31 0.31
58.37
0.33
50.21 +: 2.98
Soft-label
92.63 +: 0.27
86.32 +: 0.10
0.294
70.50 1.500:0: 0.44
57.48
4 0.43
48.18 + 0.89
Gen-label
92.56 + 0.56
84.68
+ 0.57
78.32 + 0.94
70.46 - 0.37
37.86
- 0.50
48.08 + 0.98
GDW
92.94 + 0.15
88.14 + 0.35
84.11 + 0.21
70.65 = 0.52
59.82 + 1.62
53.33 + 3.70
Table 3: Test accuracy on CIFARIO and CIFARIO0 with different flip noise ratios. CIFARIO
CIFARION 100
Dataset
0%
20%
40%
0%
20%
40%
MadeModel
92.73 + 0.37
90.14
0.35
81.20
0.93
70.42
0.54
64.96
0.16
49.83
0.82
Fine-tuning
92.77
0.37
90.15
0.36
81.53
0:33:000 0.96
70.52
0.57
0.22 65.02
50.23
0.71
Co-teaching
91.54 0.39
89.27
0.24
3.97
0.13
62.96
42.54 1.68
GLC
90.85 1 0.22
90.22 0.13
89.74
0.19
65.05
0.59
64.11
63.11
L2RW
89.70
88.21 0.49
82.90 1.27
63.40 1.31
55.27
+ 2.27
45.41 2.53
INSW
92.70 0.57
$0.9011..00
80.09
10.09:2:000
70.52
0.39
65.32 0.27
50.13 + 0.39
MWNet
92.95 + 0.33
89.93 1 0.17
8.85
70.64 0.31
64.72
50.62 + 0.46
Soft-label
92.63 + 0.27
90.17 1217:0000 0.47
$5.52 0.78
70.50 0.44
65.20 0.45
50.97 + 0.41
Gen-label
92.56 + 0.56
90.18 + 0.13
80.93 + 1.29
70.46 = 0.37
64.91 1.91+000 0.53
19.93 + 0.55
GDW
92.94 + 0.15
91.05 + 0.26
87.70 + 0.37
70.65 = 0.52
65.41 + 0.75
52.44 + 0.79
Training. Most of our training settings follow 233 and we use the cosine learning rate decay schedule [49] for a total of 80 epochs for all methods. See Appendix C for details.
Analysis. For all experiments, we report the mean and standard deviation over runs Table 2 and 3 where the best results are in bold and the second- -est results are marked by underlines. First, we can observe that GDW outperforms nearly all the competing methods in all noise settings except for the 40% flip noise setting. Under this setting, GLC estimates the label corruption matrix well and thus performs the best, whereas the flip noise assumption scarcely holds in real- wolld scenarios. Note that GLC also performs much better than MWNet under the 40% flip noise setting as reported in (23]. Besides, under all noise settings, GDW has a consistent performance gain compared with Furthermore, as the ratio increases from 40%
MWNet, which aligns with our motivation in Figure I to 60% in the uniform noise setting, the gap between GDW and MWNet increases from 1.68% to 2.97% in CIFAR10 10 and 1.45% to 3.12% in CIFARIOK 100. Even under 60% uniform noise, GDW still has low test errors in both datasets and achicves more than 3% gain in CIFAR1O 10 and 6% gain in CIFAR100 compared with the second- -bstt method. Last but not least, GDW outperforms Soft-label and Gen-label in all settings. One possible reason is that manipulating gradient flows is a more direct way to capture class-level information than learning labels.
In Figure 3 we show the distribution of class-level target weight (w)) on clean and noisy instances in one epoch under the CIFARI0 40% uniform noise setting. We observe that w; of most clean instances are larger than that of most noisy instances, which indicates that w, can distinguish between clean instances and noisy instances. This is consistent with Eq. (3) that w, serves as the instance weight.
To better understand the changing trend of non-target class-level weights, we visualize the ratio of increased weights in one epoch in Figure 5 under the CIFARI0 40% uniform noise setting. Specifically, there are three categories: norgtarget weights on clean instances (wf)). true target weights on noisy instances (w.)) and non-target (excluding true targets) weights on noisy instances (w")). Formally. "target weight" means the class-level weight on the label position. weight" means the class-level weight on the true label position, which are only applicable for noisy instances. "non-target weight" means the class-level weight except the label position and the true label position. For example, as shown in Figure [weeee a cat is mislabeled as "dog", the corresponding meanings of the notations are as follows: 1) w means Wang ("dog" the target); 2) wit means went ("cat" is the ture target); 3) wit means Whind (" ("bird" is one of the non-targets). For a correctly labeled
target
7

---

Distribution Target Weight Me
17500
:lean
13000
12500
Numbers
"dog"
7500
Gradient flow info.
"aot cut"
"dog"
"not bird"
5300
Initial Weight
2500
MW-Net
0.43
0.50
GDW
0.25
0.40
0.63
Figure 3: Class-level target weight (w,) distri- bution on CIFARIO under 40% uniform noise. Wi of most clean instances are larger than that of most noisy instances, which means we can differentiate between clean and noisy instances.
Figure 4: The change of class-level weights in an iteration for noisy instance (cat mislabeled as "dog"). MWNet downweights all gradient flows. In contrast, GDW upweights the "not bird" gradient flow for better information use.
cat, the corresponding meanings are: 1) wf wit means Went ("cat" is both the target and the ture target); 2) wht means Woog and Whind ("dog" and "bird" are both non-targets).
Note that in Figure D will represents the importance of the "not cat" gradient flow and who represents the importance the "not bird" gradient flow. If the cat image Figure i correctly labeled as "cat", then the two non-target weights w.. used to represent the importance of the "not dog" and the "not bird" gradient flows, respectively. In one epuch, we calculate the ratios of the number of increased wht. wit and wht to the number of all corresponding weights. w and wh, expected to increase since their gradient flows contain valuable information, whereas wit is expected to decrease because the "not cat" gradient flow contains harmful information. Figure Jlaigns perfectly with expectation. Note that the lines of whe and th nearly coincide with each other and fluctuate around 65%. This means non-target weights pattern, i.e., around 65% of wf, and wh; increase. Besides, less than 20% of wit increase and thus more than 80% decrease, which means the gradient flows of wit contain much harmful information.
clean instances and noisy instances share the same changing
In Figure 4 we show the change of class-level weights in an iteration for noisy instance, i.e., a cat image mislabeled "dog". The gradient flows of "not cat" and "dog" contain harmful information and thus are downweighted by GDW. In addition, GDW upweights the valuable "not bird" gradient flow from 0.45 0.63. By contrast, unable capture class-level information, MWNet downweights all gradient flows from 0.45 to 0.43, which leads to information loss on the "not bird" gradient flow.
Training without the zero-mean constraint. We have also tried training without the zero-mean constraint in Section 3.3 and got poor results (see Appendix B.2 for details). Denote the true target as i and one of the non-target labels as nt (nt + it). Note that the gradient can be unrolled as (see Appendix B. for details):
0l, 00
Øl, p/
fw (VoL) = w: Σ (v/) -y,)
+ Σ WIPP wi
Σ
(13)
00
If Σg WAPk: is positive and the learning rate is small enough, (Ex WhPi: - wi) Pte She contributes the decrease of the true target logit lu after a gradient descent step. If negative, (E, - w.) p'se Aha contributes to the increase of the non-target logit lnt. Therefore, without the zero-mean constraint, the second term in Eq. (13) may hurt the performance of the model regardless of the sign of Σn WaPk w, Similarly, training without the constraint results in poor performance in other settings. Hence we omit those results in the following subsections.
20
4.2 Class Imbalance Setting
Setup and comparison methods. The imbalance factor μ E (0,1) of a dataset is defined as the
number of instances in the smallest class divided by that the largest [23]. Long-Taine Tailed CIFAR created by reducing the number of training instances per class according an exponential function m//p/(C-1)) where i is the class index (0-indexed) and n, is the original number of
47

---

Ratio Trend of Increased Weights
Trend increased Gig.
0.35
0.30
0.25
0.20
o/
0.15
0.3
0.10
0.2
0.05
10
25
Epoch
Epoch
Figure 5: Ratio trend of the number of increased W Wit and wht under the CIFARIO 10 40% uni- form noise setting. Around 65% of wy, and wht increase since they contain useful informa- tion. Besides, less than 20% of w increase and thus more than 80% of wit decrease since they contain harmful information.
Figure 6: Ratio trend of the number of increased Wg on C99 instances under the Long- Tailed CI- FARIO μ 0.1 setting. Less than 10% of wa increase and thus more than 90% decrease. A small wg strikes a balance between two kinds of information: "C8" and "not C8", which better handles class imbalance.
Table 4: Test accuracy on the long-tailed CIFARIO and CIFARIO 100 with different imbalance ratios.
CIFARIO
CIFART00
Dataset
/ 0.01
==
| 0.1
|
ModeM
92.73
0.37
85.93 f: 0.57
69.77
1.13
70.42
0.54
37.79
0.82
Fine-tuning
92.77 0.37
82.60
1: 0.49
59.76
70.32
+: 0.57
55.95
37.10
Focal
91.68
0.49
84.57
65.78
4.02
0.38
0.51
37.43 1.00
Balanced
92.80 0.47
0.46 86.05
63.63
3.60
0.56 70.56
55.02
27.60 1.39
L2RW
89.70 0.50
79.11
51.15 7.13
63.4011..
4.52
25.86
25.76
INSW
92.70 0.57
0.28 86.31
70.27 0.24
70.32 + 0.39
55.94
37.67
0.59
MWNet
92.95 0.33
86.17
0.75
62.70
1.76
70.64 0.31
56.49
37.83 1
GDW
92.94 0.15
0.55 86.77
71.31
1.03
0.52 70.65
0.52 56.78
37.94
1.58
training instances. Comparison methods include: 1) L2RW 22]: 2) INSW 24]; 3) MWNet 23
4) BaseModel; 5) Fine-tuning; 6) Balanced 13): 7) Focal 21
Analysis. As shown in Table GDW performs best nearly all settings and exceeds MWNet by 8.6% when the imbalance ratio is 0.01 in CIFARIO. Besides, INSW achieves competitive performance at the cost of introducing huge amount of learnable parameters (equal the training dataset size N). Furthermore, we find that BaseModel achieves competitive performance, but fine- tuning on the meta set hurts the model's performance. We have tried different learning rates from 10-3 to 10-1 for fine-tuning, but the results are similar. One explanation is that the balanced meta set worsens the model learned from the imbalanced training set. These results align with the experimental results in 24 which also deals with class imbalance.
Denote the smallest class as C9 and the second smallest class as C8 in Long-Tailed CIFARIO with = 0.1. Recall that w, denotes the jth class-level weight. For all C9 instances in an epoch, we calculate the ratio of the number of increased wa to the number of all wa, and then visualize the ratio trend in Figure Since C9 is the smallest class, instance weighting methods upweight both Wg and Wy on a C'9 instance. Yet in Figure less than 10% of Wa increase and thus more than 90% decrease. This can be explained as follows. There are two kinds of information in the long-tailed dataset regarded to C8: "C8" and "not C8". Since C8 belongs the minority class, the dataset is biased towards the "not C8" information. Because smaller We weakens the "not C8" information. As result, decreased wa achieves a balance between two kinds of information: "C8" and "not C8", thus better handling class imbalance at the class level. We have conducted further experiments on imbalanced settings verify the effectiveness of GDW and see Appendix D for details.
represents the importance of "not C8".

---

Table Test accuracy on Clothing
Method
BaseModel
Fine-tuning
Co-teaching
GLC
L2RW
INSW
MWNet
Soft-label
Gen-label
GDW
Accuracy()
65.02
67.68
68.13
68.60
68.80
68.25
68.46
68.69
67.64
69.39
4.3 Real-world Setting
Setup and training. The ClothingIM dataset contains one million images from fourteen classes collected from the web [4. Labels are constructed from surrounding texts of images and thus contain some errors. We use the ResNet-18 model pre-trained on ImageNet [50] as the classifier. The comparison methods are the same as those in the label noise setting since the main issue of Clothing IM is label noise 4 All methods are trained for epochs via SGD with a 0.9 momentum, a 10 initial learning rate, a 10 -3 weight decay, and 128 batchsize. See Appendix E for details.
Analysis. As shown in Table 5 GDW achieves the best performance among all the comparison methods and outperforms MW Net by 0.93%. In contrast to unsatisfying results in previous settings, L2RW performs quite well in this setting. One possible explanation is that, compared with INSW and MWNet which update weights iteratively, L2RW obtains instance weights only based on current gradients. As result, L2RW can more quickly adapt to the model's state, but meanwhile suffers from unstable weights [23] In previous settings, we train models from scratch, which need stable weights stabilize training. Therefore, INSW and MWNet generally achieve better performance than L2RW. Whereas in this setting, we use the pre-trained ResNet- 18 model which is already stable enough. Thus L2RW performs better than INSW and MWNet.
5 Conclusion
Many instance weighting methods have recently been proposed to tackle label noise and class imbalance, but they cannot capture class-level information. For better information use when handling the two issues, we propose GDW to generalize data weighting from instance level to class level by reweighting gradient flows. Besides, to efficiently obtain class-level weights, we design two-stage weight generation scheme which is embedded in three-step bi-level optimization framework and leverages intermediate gradients to update class-level weights via gradient descent step. In this way, GDW achieves remarkable performance improvement in various settings.
The limitations of GDW are two-fold. Firstly, the gradient manipulation only applicable to single- label classification tasks. When applied to multi-label tasks, the formulation of gradient manipulation need some modifications. Secondly, GDW does not outperform comparison methods by a large margin in class imbalance settings despite the potential effectiveness analyzed in Section [4.2] One possible explanation is that better information utilization may not result in performance gain which also depends various other factors.
6
Acknowledgement
We thank Prof. Hao Zhang from Tsinghua University for helpful suggestions. This research was supported in part by the MSR-Mila collaboration funding. Besides, this research was empowered in part by the computational support provided by Compute Canada (www.computeeanadaaca))
References
[1] Hwanjun Song, Minseok Kim. Dongmin Park, Yooju Shin, and Jae- Lec. Learning from Noisy Labels with Deep Neural Networks: Survey. arXiv:2007.06 [cs, stat], June 2021.
[2] Haibo He and Edwardo A. Garcia. Learning from Imbalanced Data. IEEE Transactions on Knowledge and Data Engineering, 216))1263-1284, September 2009.
[3] Nancy E. EllHayy and Julien Provost. Systematic Survey on Sensor Failure Detection and Fault- Toleranee in Ambient Assisted Living. Sensors, 18(7):1991, July 2018.
10

---

[4] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2691-2699, Boston, MA, USA, June 2015. IEEE.
[5] Görkem Algan and Ilkay Ulusoy. Label Noise Types and Their Effects Deep Learning. arXiv:2003. 10471 [cs]. March 2020.
[6] Xingquan Zhu and Xindong Wu. Class Noise vs. Attribute Noise: A Quantitative Study. Artificial Intelligence Review, 22(3):177-210, November 2004.
[7] Benoit Frenay and Michel Verleysen. Classification in the Presence of Label Noise: A Survey. IEEE Transactions on Neural Networks and Learning Systems, 25(5):845-869, May 2014.
[8] Qiuye Zhao and Mitch Marcus. Long- Tail Distributions and Unsupervised Learning of Mor- phology. In Proceedings of COLING 2012, pages 3121-3136, Mumbai, India, December 2012. The COLING 2012 Organizing Committee.
[9] Grant Van Horn and Pietro Perona. The Devil is in the Tails: Fine-grained Classification in the Wild. arXiv: 170.01 450 [cs], September 2017.
[10] Reyes Pavón, Rosalía Laza, Miguel Reboiro-Jato, and Florentino Riveroli Assessing the Impact of (lass-Imbllanced Data for Classifying Relevant/Ireleeant Medline Documents, In Miguel P. Rocha, Juan M. Corchado Rodríguez, Florentino Fdez-Riverola, and Alfonso Valencia, editors, 5th International Conference on Practical Applications of Computational Biology & Bioinformatics (PACBB 2011), Advances Intelligent and Soft Computing, pages 345-353, Berlin, Heidelberg, 2011. Springer.
[11] Harshita Patel, Dharmendra Singh Rajput, G Thippa Reddy, Celestine Iwendi, Ali Kashif Bashir, and Ohyun Jo. A review on classification of imbalanced data for wireless sensor networks. International Journal of Distributed Sensor Networks, 1554)115501477209 1400, April 2020.
[12] Qi Dong, Shaogang Gong, and Xiatian Zhu, Class Rectification Hard Mining for Imbalanced Deep Learning. In 2017 IEEE International Conference Computer Vision (ICCV), pages 1879-1878, Venice, October 2017. IEEE
[13] Yin Cui, Menglin Jia, Tsung-Yi Yi Lin, Yang Song, and Serge Belongie. Class-Balanced Loss Based on Effective Number of Samples. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9260,9269, June 2019.
[14] Saptarshi Sinha, Hiroki Ohashi, and Katsuyuki Nakamura. Class- Wise Sefficulty-Balaneee Loss for Solving Class-Imbalance. In Hiroshi Ishikawa, Cheng-Lin Liu, Tomas Pajdla, and Jianbo Shi, editors, Computer Aasion-ACCO 2020, Lecture Notes in Computer Science, pages 549-565. Cham, 2021. Springer International Publishing.
[15] Justin M. Johnson and Taghi M. Khoshgoftaar. Survey on deep learning with class imbalance. Journal of Big Data, 6(1):27, March 2019.
[16] Christian Szegedy, V. Vanhoucke, S. Ioffe, Jonathon Shlens, and Z. Wojna. Rethinking the Inception Architecture for Computer Vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[17] J. Goldberger and E. Ben-Reuven. Training deep In International Conference on Learning Representations, 2017.
using noise adaptation layer.
[18] Tongliang Liu and Dacheng Tao. Classification with Noisy Labels by Importance Reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38, November 2014.
[19] Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[20] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems, volume 31, Curran Associates, Inc.,
2018.

---

[21] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal Loss for Dense Object Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence. 12(2):318-322 327, February 2020.
[22] Mengye Ren, Wenyuan Zeng, Bin Yang. and Raquel Urtasun. Learning to Reweight Ex- amples for Robust Deep Learning. In Proceedings of the 35th International Conference on Machine Learning, pages 4334 4343. PMLR, July 2018.
[23] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-Weight-Ne Learning an Explicit Mapping For Sample Weighting. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[24] Zhiting Hu, Bowen Tan, Russ R Salakhutdinov, Tom M Mitchell, and Eric P Xing. Learning Data Manipulation for Augmentation and Weighting. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[25] Xinyi Wang, Hieu Pham, Paul Michel, Antonios Anastasopoulos, Graham Neubig, and J. Car- bonell. Optimizing Data Usage via Differentiable Rewards. International Conference on Machine Learning, 2020.
[26] Arash Vahdat.
Toward Robustness against Label Noise in Training Deep Discriminative Neural Networks. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[27] Charles Elkan.
The foundations of cost-sensitive learning. In Proceedings of the 17th International Joint Conference on Artificial Intelligence- Volume 2, ICCA101, pages 973-978, San Francisco, CA, USA, August 2001. Morgan Kaufmann Publishers Inc.
[28] Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16(1):321-357,
June 2002.
[29] Ashish Anand, Ganesan Pugalenthi, Gary B. Fogel. and P. N. Suganthan. An approach for classification of highly imbalanced data using weighting and undersampling. Amino Acids, 39(5):1385 13911 November 2010.
[30] Salman H. Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A. Sohel, and Roberto Togneri. Cost-Sensitive Learning of Deep Feature Representations From Imbalanced Data. IEEE Transactions on Neural Networks and Learning Systems, 29(8):3573-3588 August 2018.
[31] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling Representation and Classifier for Long-Tailed Recognition. In International Conference on Learning Representations, September 2019.
[32] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boging Gong, and Stella X. Yu. Large-Scale Long-Taiee Tailed Recognition in an Open World. In 2019 IEEE/CV Conference on Computer Vision and Pattern Recognition (CVPR), pages 2532-2002 2541, Long Beach, CA, USA, June 2019. IEEE.
[33] Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu. Long-tailed Recog- nition by Routing Diverse Distribution-Aware Experts. In International Conference Learning Representations, September 2020.
[34] Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(62666 December 2015.
[35] Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel Programming for Hyperparameter Optimization and Meta-Learning. In Proceedings of the 35th International Conference on Machine Learning, pages 1568-1577. PMLR, July 2018.
[36] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable Architecture Search. In International Conference on Learning Representations, September 2018.
12

---

[37] Yu- Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to Model the Tail. In Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[38] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels. In Proceedings of the 35th International Conference on Machine Learning, pages 2304-2313, PMLR, July 2018.
[39] Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S. Kankanhalli, Learning to Learn From Noisy Labeled Data. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5046-5054, Long Beach, CA, USA, June 2019. IEEE.
[40] G. Algan and I. Ulusoy. Meta Soft Label Generation for Noisy Labels, 2020 25th International Conference on Pattern Recognition (ICPR), 2021.
[41] Zizhao Zhang, Han Zhang, Sercan Ö, Arik, Honglak Lee, and Tomas Pfister. Distilling Effective Supervision From Severe Label Noise. In 2020 IEEE/CVF Conference on Computer Vision and Pattemn Recognition (CVPR), pages 9291 9300, June 2020.
[42] Nidhi Vyas, Shreyas Saxena, and Thomas Voice. Learning Soft Labels via Meta Learning. 00X4:2009.09496 [cs, stat], September 2020.
[43] Shreyas Saxena, Oncel Tuzel, and Dennis DeCoste. Data Parameters: A New Family of Parameters for Learning a Differentiable Curriculum. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[44] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv: 170.04604 [es], April 2017.
[45] Zhenyue Qin, Dongwoo Kim, and Tom Gedeon. Rethinking Softmax with Cross-Entropy: Neural Network Classifier as Mutual Information Estimator. arXiv:1911.1069. [cs. stat]. September 2020.
[46] Shuai Zhao, Liguang Zhou, Wenxiao Wang, Deng Cai, Tin Lun Lam, and Yangsheng Xu. Towards Better Accuracy-efficiency Trade-offs: Divide and Co-training. arXiv:2011.1466 [cs], March 2021.
[47] A. Krizhevsky. Learning Multiple Layers of Features from Tiny Images. MSc Thesis, Univer- sity of Toronto, 2009.
[48] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, June 2016.
[49] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. International Conference on Learning Representations, November 2016.
[50] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, June 2009.
13


Reviews:
[{"review_id": "OHX9QpeGp_", "reviewer": "Reviewer_iJG7", "summary": "The authors propose Generalized Data Weighting (GDW) to generalize data weighting from instance level to class level by reweighting gradient flows. This framework can be helpful for better information use when handling label noise and class imbalance.  Specifically, they design a two-stage weight generation scheme embedded in the three-step bi-level optimization framework which can efficiently obtain class-level weights.  This scheme leverages intermediate gradients to update class-level weights via a gradient descent step. Experiments show that GDW achieves remarkable performance improvement in various settings.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nIn contrast to the classical instance level weighting, the authors are the first to propose class-level weighting on gradient flows. However, the main techniques this paper leverages have already been well-established in [23] which proposed the MW-Net Learning Algorithm. This is a nice extension of the existing techniques.\n\nQuality: \nThe authors conduct extensive experiments in various settings ( Label Noise Setting, Class Imbalance Setting, Real-world Setting) to verify the effectiveness of GDW. \n\nClarity: \nThis paper is organized well. I can easily understand the main idea and techniques applied\n\nSignificance: \nThis paper advances state-of-the-art methods to alleviate Label noise and class imbalance by leveraging class-level information. The algorithm proposed in this paper would be helpful to deal with two major issues coexisting in real-world datasets.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Although the proposed algorithm seems to work well empirically, there is no theoretical analysis on the performance. Also, the main techniques this paper leverages have already been well-established in [23]. The technical contribution seems somewhat marginal.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "EeFpgzT-QC", "reviewer": "Reviewer_tepP", "summary": "This paper proposes a new method for mitigating the problem of label noise and class imbalance. The key idea is to utilize the class-level information to fully exploit the information contained in each training instance. The proposed method is evaluated using the popular dataset and it showed improved performance over the existing methods.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strength\nI found the key idea of utilizing class-level information within each instance is novel and interesting. In addition, the paper is basically well written except the part I point out below.  The method is evaluated using popular benchmark datasets and shows comparable or better performance compared to the existing state-of-the-art methods.\n\n# Weakness\nMy main concerns are two folds.\n\n## 1. The supports for the claims are not sufficient.\n### 1.1 The coexistence of label noise and class imbalance and not explicitly handled in the experiment.\nThe authors repeatedly claim that one of the important problems in the real-world setting is the coexistence of label noise and class imbalance (L1, L24-25), and their GDW simultaneously mitigates these problems (L6, L300-301). However, these two issues are handled separately in the experiment, which is not good for supporting the above-mentioned claim.\n### 1.2 The effectiveness of the proposed method in class-imbalance case seems marginal\nLooking at the standard deviations, the proposed method is not significantly better than the existing methods. Is there any analysis on why the method works fine in label-nose setting whereas shows relatively lower performance in class-imbalance setting?\n\n### 1.3 Figure 5 and L229-232\nThe meaning of “target” is not clear as I later point out in more detail. Therefore, it is not clear for me what $w_{nt}^n$ includes. L232 says it represents the importance of the “not bird”, but how about the “dog”? I think decreasing the weight of the “dog” gradient is as important as decreasing the weight of “not cat” in the case of Figure 1.\n\nIn addition, it is better to show $w_{tt}^c$, too, for the completeness.\n### 1.4 Figure 6\nHow about the ratio trend of the number of increased $w_8$ on C1-C7, and C8? I think C8 instances should receive lower weights not only in the shown case, but also when C1-C7 instances are used in the training. Similarly, they should receive larger weights when C8 instances are used in the training. How is the same analysis for $w_9$ on C1-C8 and C9? Showing only one example is not sufficient.\n\n### 1.5 L248-249: “We have also tried training without the zero-mean constraint in Section 3.3 and got poor results.”\nJust “got poor results” is not enough as an evidence of the importance of zero-mean constraint. Please show it quantitatively.\n\n## 2. Some parts of the paper are not clear enough.\n### 2.1 L131-132 `Specifically, GDW tries to downweight the gradient flows for \"dog\" and \"not cat\", and upweight the gradient flow for \"not bird\".’\nI understand what the proposed method aims to do. However I do not think the reason why such a thing is possible is clearly explained. Please explain the intuition behind this.\n### 2.2 Section 3.3 is not clear\n- L136: What does “retain the Softmax-CrossEntropy loss structure” mean? What is the motivation to do so?\n- L138: What does “the target (label) position” mean? Especially in noisy label case, does “target” mean a true category or wrongly labeled ground-truth category? For example in Figure 1, what does “target” represent? Without precisely understand what is “target”, I cannot judge the soundness of equation 3 and subsequent discussion. For now, I can only understand equation 3 holds true if $w_t = w_j$.\n### 2.3 equation 11\nI think it is better to explicitly write down what $g_i$ exactly denotes in the equation.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No.\n- It is recommended to discuss the limitation of the method more deeply in the main body of the paper. For example, the proposed method is tested only in label-noise case and in class-imbalance case separately. I think this is one of the limitation of the paper.\n- The authors did not discuss any potential negative societal impacts. Therefore, the answer to this question in the checklist should be [No] rather than [N/A]. Any kinds of works can be used in an undesirable manner regardless of the authors’ intention. For example, the method may be used to intentionally increase the bias since it can handle class-level information. It is important to discuss the possible negative aspects (and hopefully how to cope with them).\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}, {"review_id": "BAzC88Mqlsx", "reviewer": "Reviewer_Khcd", "summary": "In the paper, the authors propose a novel generalized data weighting method to improve the performance of existing label noise and class imbalance methods by manipulating the class-level gradients. Existing instance weighting methods are special cases of the proposed method when the class-level weights are the same. The proposed method is quite simple, easy to be deployed to existing methods, and does not introduce additional computational costs. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Comments:\n1. In equation (8), performing empirical risk minimization on a small meta dataset may be very likely to overfitting the meta dataset. \n2. For the experiments, how to ensure a fair comparison with traditional methods such as co-teaching [20] where meta dataset is not needed.\n3. In Figure 3, some class-level target weights for noisy data are still very large, even larger than that for the clean data. Giving the mislabeled data large weights could greatly harm the performance. I'm not sure whether the Noisy (the blue in legend) in Figure 3 means a group of intact and mislabeled data, or only mislabeled data.\n4. Experiments of the paper include label noise (uniform + flip noise + real-world label noise) and class imbalance experiments. It seems that the proposed method largely outperforms other baselines only for the cases of uniform label noise with large noise ratios. For example, for the flip noise setting in label noise, it is hard to conclude the proposed method is better than others. \n\nMinor comments:\n1. The method in [20] is named co-teaching, not co-teach.\n2. Class imbalance mainly refers to the setting that both training and test sets are class imbalanced. If the training set is class imbalanced and the test set is class balanced (the setting in the paper I think), better to call it class-prior shift or label shift. \n\n----------------------------POST REBUTTAL-----------------------------\n\nThank the authors very much for the rebuttal! After reading the rebuttal and all the reviews, the authors have addressed my concerns properly. Unfortunately, I still tend to reject this paper:\n\n1. The paper mentioned in the abstract about a novel problem setting (simultaneously handle label noise and imbalance) but they didn't present any experiment results under the claimed setting in the main paper. The authors should either add more experiments under the claimed setting (more than the experiments they presented in the rebuttal) or change the position of the paper writing.\n2. I still have a concern about the effectiveness of the proposed method in experiments. For example, it seems the proposed method works not that well in flip label noise cases. Also, Figure 3 shows the proposed method cannot well differentiate correct and mislabeled data under the uniform noise. I guess the same figure for the flip label noise could be worse.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No, the authors forgot to address the limitations and potential negative societal impact of their work. I suggest adding some discussions on this point. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "4l-KK7n1_WF", "reviewer": "Reviewer_B8Dp", "summary": "The paper focuses on a technique to learning per-instance weights in training classification networks. The paper improves over the idea of Meta-Weight-Net [23] which adopts meta-learning to learn per-instance weights. According to the paper, training data labels can be noisy and class labels usually follow long-tailed distributions, therefore learning per-instance weights is expected to better exploit training data for learning a better classification network. The paper's improvement is based on an intuition that class-level weights have not but should have been used for better learning per-instance weights. The paper proposes Generalized Data Weighting (GDW) and claims to *simultaneously* mitigate label noise and class imbalance. However, it is not clear why the proposed method is \"generalized\", and experiments do not validate the method can *simultaneously* mitigate label noise and class imbalance, because experiments study the two aspects separately. \n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "pros:\n\n+ The paper positions itself as using data reweighting to simultaneously address class-imbalanced learning and learning with noisy labels. This problem is interesting (although authors failed in experimentally validating the method on this problem).\n\n+ Learning per-instance weights is a reasonable approach to address either imbalanced learning or learning with noisy labels.\n\n\nBelow are some summarized cons, followed by detailed comments.\n\n- Due to such a position (as simultaneously addressing class-imbalanced learning and learning with noisy labels), the paper failed to justify the method in this experiments. In contrast, the paper studies the two problems separately.\n\n- While class-imbalanced learning can be improved by learning per-instance weights, the paper did not fully survey the literature of class imbalanced learning. For example, [ref1] is missing in the paper that shows a rather simple method greatly improves class-imbalanced learning. Given this, it is natural to question why the paper misses an important baseline -- using [ref1] and data reweighting to simultaneously solve imbalanced learning with noisy labels.  [ref1] Decoupling Representation and Classifier for Long-Tailed Recognition, ICLR 2020\n\n\n- The paper is incremental in terms of techniques. It improves over Meta-Weight-Net [23] by incorporating class-level weights learning. Perhaps because of this, the method is called *generalized* data reweighting; however, the method is incremental rather than *generalized*.\n\n\n- There are many parts of the paper hard to follow, as detailed below.\n\nFirst, as the paper positions itself as \"simultaneously mitigate label noise and class imbalance\" (Line6), the paper fails to justify whether the proposed method can do it because experiments study the two subproblems separately. Furthermore, it looks quite non-trivial how to exploit class-level weights in front of noisy labels and imbalanced classes. For example, it is not clear how to weight data with noisy labels from rare classes, should the model give a large weight on them because they are from rare classes, or a small weight because they have noisy labels? Authors should delve into this aspect to better support the position of the paper.\n\nLine35: The sentence is confusing \"Due to the MLP, MWNet has better scalability on large datasets compared with INSW\". It is not clear what INSW does, and why MLP improves MWNet. Authors should improve the writing.\n\nLine37: The sentence is confusing -- \"they can not fully utilize class-level information within each instance\". What do authors mean by \"fully utilize class-level information\"? Why \"they can not\"? What did \"they\" do such that \"they can not\"? Authors should clarify. This makes the paper hard to follow.\n\nLine39: It is confusing -- what is the classifier for computing gradients? Does this classifier compute gradients from \"not X\" in terms of class-X?\n\nLine42: What are the \"three kinds of information\"? The authors did not define the \"three kinds of information\". It is hard to understand the paper.\n\nLine43: It is confusing -- \"downweighting the \"not bird\" gradient flow is a waste of information.\" What do authors mean by \"a waste of information\"? Can authors clarify?\n\nFigure 1: It is confusing -- \"Although the gradient flows for \"dog\" and \"not cat\" contain harmful information...\" What do authors mean by \"harmful information\"? Can authors clarify?\n\nLine66: It is unclear what the authors mean by \"To the best of our knowledge, we are the first to propose class-level weighting on gradient flows.\" Because it is not clear what the difference is between gradient reweighting and loss reweighting. Can authors clarify?\n\nSection 2.2: There is one missed citation that is important [ref1]. Authors should carefully survey the literature.\n[ref1] Decoupling Representation and Classifier for Long-Tailed Recognition, ICLR 2020\n\nLine87: Although tt is evocative to say \"meta-learning methods view instance weights as hyper-parameters and dynamically update them via a meta set to avoid hyper-parameter tuning.\" Did the authors consider the hyperparameters in the meta-learning? What are the learning rates, momentum, schedules of learning rates and initialization in meta learning? In the paper, authors did not provide such details.\n\nLine94: What is \"dynamic regularization\"? Can authors define it?\n\nLine127: It is not straightforward to understand how D1 is derived. Necessary derivations are needed. \n\nLine136: What do authors mean by \"retain Softmax-CrossEntropy loss structure after the manipulation\"? What is structure? What will happen if one does not \"retain\" the structure? Authors should improve writing the make the paper readable.\n\nLine140: What do the authors mean by \"messes the structure\"? What structure and how it \"messes the structure\"? \n\nEq.(4): The authors proposes a zero-mean constraint (Line143), but from what the authors did, it seems that this is factually not a constraint but an assumption with which the authors simply drop the second term. Authors should be careful in writing the statements.\n\nLine213: The authors explain \"GLC estimates the label corruption matrix well and thus performs the best\", but why GLC does not estimate label corruption matrix well in other noise levels? Authors should provide insights or analysis.\n\nLine229: As authors did not define \"non-target weights\", \"true-target weights\" or \"non-target\", I cannot understand this part. Authors should improve the writing quality.\n\nLine261: The definition of the imbalanced factor is inconsistent. In plain words, it is defined as \"the number of instances in the largest class divided by that of the smallest\", that said, the factor should be no less than 1. However, authors states the factor is in (0,1). Authors should correct this inconsistency.\n\n\n\n\n\n\n\n----------- post rebuttal --------------\n\nI appreciate the authors' feedback! The answers largely addressed most my confusions, though I still have some below.\n\nFirst, I think the explanation in the rebuttal is much clearer than the paper! Therefore, I'd upgrade the rating to \"5: Marginally below the acceptance threshold\". While this is not a recommendation for accepting the current version of the paper, I encourage the authors to incorporate the new results to polish the paper for the next submission.\n\nImportantly, it is still confusing to connect the position of the paper and the proposed methods. Concretely, the position of the paper is \"to simultaneously mitigate label noise and class imbalance\" (line6 in the abstract) because \"Label noise and class imbalance are two major issues coexisting in real-world datasets\" (line1 in the abstract). This is a very good problem setup, but the authors only study this setup in the rebuttal.\n\nBecause this is a novel problem setup, thoughtful baselines should be compared, such as combining noisy training atop of a state-of-the-art (SOTA) class-imbalanced learning method [ref1] and combining class-imbalanced learning atop of a SOTA noisy training method. I'm not satisfied by the authors argument \"it is not our duty to consider it as a baseline for comparison\" -- this will hurt the paper. I have some background of long-tailed recognition, as a reader, I would think such a combination as the first method to try out to solve this novel problem, rather than rushing to any complicated methods.\n\nOn the other hand, in the rebuttal, the authors make another argument by \"Should GDW perform better than this combination, the soundness of the comparison itself would be suspected\", implying that the authors are reluctant to compare these straightforward baselines to address the novel problem setup. If the authors really want to sell the method, then authors can change the position of the paper -- instead of proposing a novel problem setup, authors can directly start by methods without emphasizing the problem.\n\nLastly, I do encourage the authors to make a thoughtful revision of the paper by considering the new experiment results, position of the paper, and comparisons to other baselines.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does not discuss any limitations or societal impacts in the paper. The authors should be aware of the limitations of their methods. Authors should clarify.", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": ["Inadequate Data and Algorithm Evaluation", "Privacy and Security (e.g., consent)", "Legal Compliance (e.g., GDPR, copyright, terms of use)"], "time_spent_reviewing": "4 hours"}, "extra_text": {"ethical_concerns": "It is not clear whether the datasets used in this paper have ethical concerns. Especially the Clothing1M dataset has person faces and body, it seems a concern in using the dataset. The authors failed in stating ethical concerns.\n"}}]

Author rebuttal:

Meta-review: The paper studies the problem when data is imbalanced and has label noise. The authors proposed to simultaneously mitigate label noise and class imbalance. The key idea is to exploit the class-level weights, which is very interesting, reasonable, novel, and well-written. Experiments also show the effectiveness of the proposed method. No computational cost will be introduced. However, there are some concerns. (1) Insufficient experiments under the claimed setting to support the proposed method. The authors are expected to either add more experiments under the claimed setting (more than the experiments they presented in the rebuttal) or change the position of the paper writing. (2) The proposed method does not work that well in the cases with flip label noise. More discussions are needed to explain this. We also expect the authors to add experiments on synthesized instance-dependent label noise rather than uniform and pairflip label noise. (3) Thoughtful baselines such as combining some classic/state-of-the-art methods of training with noisy labels and class-imbalanced learning are expected. For example, the paper: Menon et al. Learning from Corrupted Binary Labels via Class-Probability Estimation, ICML 2015, which shows that the balanced error designed for imbalanced data is robust to label noise. This is a very related work which should be discussed as well. In a summary, we think the paper is not ready for publication at this round. It will be a strong paper if the concerns can be well addressed.
Decision: Accept (Poster)