Generalized DataWeighting via Class-Level Gradient Manipulation
Authors: Can Chen
Shuhao Zheng
Xi Chen
Erqun Dong
Xue Liu
Hao Liu
Dejing Dou
Venue: NeurIPS 2021 Poster (2021)
Keywords: instance weighting
meta-learning
bi-level optimization
classification
chain rule

Abstract: Label noise and class imbalance are two major issues coexisting in real-world datasets. To alleviate the two issues, state-of-the-art methods reweight each instance by leveraging a small amount of clean and unbiased data. Yet, these methods overlook class-level information within each instance, which can be further utilized to improve performance. To this end, in this paper, we propose Generalized Data Weighting (GDW) to simultaneously mitigate label noise and class imbalance by manipulating gradients at the class level. To be specific, GDW unrolls the loss gradient to class-level gradients by the chain rule and reweights the flow of each gradient separately. In this way, GDW achieves remarkable performance improvement on both issues. Aside from the performance gain, GDW efficiently obtains class-level weights without introducing any extra computational cost compared with instance weighting methods. Specifically, GDW performs a gradient descent step on class-level weights, which only relies on intermediate gradients. Extensive experiments in various settings verify the effectiveness of GDW. For example, GDW outperforms state-of-the-art methods by $2.56\%$ under the $60\%$ uniform noise setting in CIFAR10. Our code is available at https://github.com/GGchen1997/GDW-NIPS2021.

Generalized Data Weighting via Class-level Gradient Manipulation
Can Chen'; Shuhao Zheng'; Xi Chen', Erqun Dong', Xue Liu', Hao Liu', Dejing Dou
'McGill University, 'The Hong Kong University of Science and Technology, 'Baidu Research
(can.chen, shuhao zzeng, erqun (rqqundoong)eee dong) @mail ciciiilmmmllll. worgill.co ca
xi.chent wwwmn11mcgill.coom xueliu@cs megill. ca, liuh@ust. hk, doude saadejing@baiine ing@baidu. com
Abstract
Label noise and class imbalance are two major issues coexisting in real-world datasets. To alleviate the two issues, state-of-the-arr methods reweight each instance by leveraging a small amount of clean and unbiased data. Yet, these methods overlook class-level information within each instance, which can be further utilized to improve performance. To this end, in this paper, we propose Generalized Data Weighting (GDW) to simultancously mitigate label noise and class imbalance by manipulating gradients at the class level. To be specific, GDW unrolls the loss gradient to class-level gradients by the chain rule and reweights the flow of each gradient separately. In this way, GDW achieves remarkable performance improvement on both issues. Aside from the performance gain, GDW efficiently obtains class-level weights without introducing any extra com- putational cost compared with instance weighting methods. Specifically, GDW performs a gradient descent step on class-level weights, which only relies on intermediate gradients. Extensive experiments in various settings verify the fectiveness of GDW. For example, GDW outperforms state-of-the-ae methods by 2.56% under the 60% uniform noise setting in CIFARIO. Our code is available at https://ght ////ggttatt thuthhu.bcc. com httpc//Ghchee111977/X/--11SSSS5
Introduction
Real-world classification datasets often suffer from two issues, i.c., label noise and class im- balance (2). On the one hand, label noise often results from the limitation of data generation, e.g., sensor errors and mislabeling from crowdsoureing workers [4. Label noise misleads the training process of DNNs and degrades the model performance in various aspects [5 6 7 On the other hand, imbalanced datasets are either naturally long-tailed
[8] or biased from the real-world distribution
due to imperfect data collection [10, I.. Training with imbalanced datasets usually results in poor
classification performance on weakly represented classes 12
14]
Even worse, these two issues
often coexist in real-world datasets (15).
To prevent the model from memorizing noisy information, many important works have been proposed, including label smoothing [16), noise adaptation (17), importance weighting [18), GLC [19], and
Co-teach 200 Meanwhile, [12
21 propose effective methods to tackle class imbalance.
However, these methods inevitably introduce hyper-parameten (e.g., the weighting factor in [3] and the focusing parameter in (2)). compounding real-world deployment.
Inspired by recent advances in meta-learning, some works [22 23 24 25 propose to solve both issues by leveraging clean and unbiased meta These methods treat instance weights hyper-
Equual contribution; Names listed in alphabetical order.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

parameters and dynamically update these weights to circumvent hyper-paramede parameter tuning. Specifically, MWNet [23] adopts an MLP with the instance loss as input and the instance weight as output. Due to the MIP, MWNet has better scalability on large datasets compared with INSW [24 which assigns each instance with a learnable weight. Although these methods can handle label noise and class imbal- ance to some extent, they cannot fully utilize class-level information within each instance, resulting in the potential loss useful information. For example, in a three-class classification task, every instance every logit corresponds to class-level gradient flow which stems from the loss function and back-propagates. These gradient flows represent three kinds of in- formation: "not cat", "dog", and "not bird". Instance weighting methods (23 22
has three logits. As shown in Figure [1
alleviate label noise by downweighting all the gradient flows of the instance, which discards three kinds of information simultancously. Yet. downweighting the "not bird" gradient flow is waste of information. Similarly, in class imbalance scenarios, different gradient flows represent different class-level information. Therefore, it is necessary reweight instances at the class level for better information usage.
To this end, we propose Generalized Data Weighting (GDW) to tackle label noise and class imbalance by class-level gradient manipulation. Firstly, we introduce class-level weights to rep- resent the importance of different gradient flows and manipulate the gradient flows with these class-level weights. Secondly, we impose zero- mean constraint on class-level weights for stable training. Thirdly, to efficiently obtain class-level weights, we develop a two-stage weight gener- ation scheme embedded in bi-level optimization. As a side note, the instance weighting meth- ods 22 23 24 25 can be considered special cases of GDW when class-level weights within any instance are the same. In this way, GDW achieves impressive performance improvement in various settings.
"dog"
Gradient flow info.
"not cat"
"dog"
"not bird"
Instance weighting
0.3
Class-level weighting
0.3
0.3
0.8
Figure 1: noisy instance (e.g. cat mislabeled as "dog"). all gradient flows are downweighted by instance weighting. Although the gradient flows for "dog" and "not cat" contain harmful information, the gra- dient flow for "not bird" is still valuable for train- ing, which should not be downweighted.
Motivation for class- weighting. For
To sum up, our contribution is two-fold:
1. For better information utilization, we propose GDW, generalized data weighting method, which better handles label noise and class imbalance. To the best of our knowledge, we are the first to propose single-label class-level weighting on gradient flows.
2. To obtain class-level weights efficiently, we design a two-stage scheme embedded a bi-level optimization framework, which does not introduce any extra computational cost. To be specific, during the back propppropngttion we store intermediate gradients, with which we update class-level weights via a gradient descent step.
2 Related Works
2.1 Non-Meta-Learrning Methods for Label Noise
Label noise is a common problem in classification tasks 6
To avoid overfitting to label noise, [16] propose label smoothing regularize the model. 17 26 form different models indicate the
5).
18 estimate an importance weight for each instance to represent its value to the model. 220] train two models simultaneously and let them teach each other in every mini-batch. However, without a clean dataset, these methods cannot handle severe noise (22). [19] correct the prediction of the model by estimating the label corruption matrix via clean validation set, but this matrix is the same across all instances. Instead, our method generates dynamic class-level weights for every instance improve training.
relation between noisy instances and clean instances.
2.2 Non-Meta-Learrine Methods for Class Imbalance
Many important works have been proposed to handle class imbalance 27
29
4
28 29 propose to over-sample the minority class and under-sample the majority class. 27 30

---

Table 1: Related works comparison. "Noise" and "Imbalance" denote whether the method can solve label noise and class imbalance. "Class-level" denotes whether the method utilizes class-level information each instance, and "Scalability" denotes whether the method can scale to large datasets.
Rocal 211
Balancol
Co-teaching
20
GLC
1T
LRWW
22
INSW [4]
MWNN
Soft-label
Gen-label 40
GDW
Imbulance
Class-level
Scalability
learn class-dependent cost matrix obtain robust representations for both majority and minority classes. [12 13, 21 [1] design reweighting scheme to rebalance the loss for each class. These methods are quite effective, whereas they need to manually choose loss functions or
32 B3] manipulate the feature space to handle class imbalance while introducing extra model parameters. (31] decouple representation learning and classifier learning on long-tailed datasets, but with extra hlppe--prametter tuning. In contrast, meta-learning methods view instance weights as hyper-parameters and dynamically update them via a meta set to avoid hyper--aramett tuning.
2.3 Meta-Learning Methods
With recent development in meta-learning 34 35 36
many important methods have been proposed
handle label noise and class imbalance meta set [37 38 22
39
23
24
25 40) [38] propose
MentorNet to provide data-driven curriculum for the base network to focus on correct instances. To distill effective supervision, [1
estimate pseudo labels for noisy instances with meta set. provide
dynamic regularization. 42 40 treat labels as learnable parameters and adapt them to the model's
state. Although these methods can tackle label noise, they introduce huge amounts of learnable parameters and thus cannot scale to large datasets. To alleviate class imbalance, (37] describe method to learn from long-tailed datasets. Specifically, [37 propose to encode meta-knowledge into meta-network and model the tail classes by transfer learning.
Furthermore, many meta- meta-lcarning methods propose to mitigate the issues by reweighting every 25). [43] equip each instance and each class with a learnable parameter to govern their importance. By leveraging a meta set, 22 24 25 learn instance weights and model parameters via li-level optimization to tackle label noise and class imbalance, [22] assign weights to training instances only based on their gradient directions. Furthermore, [24] combine reinforce learning and meta-leamnan meta-learning, and treats instance weights as rewards for optimization. However, since each instance directly assigned with learnable weight, INSW can not scale to large datasets. adopt weighting network to output weights for instances and use bi-level optimization to jointly update the weighting network parameters and model parameters. Although these methods handle label noise and class imbalance by reweighting instances, a scalar weight for every instance cannot capture class-level information, as shown in Figure [ Therefore, we introduce class-level weights for different gradient flows and adjust them to better utilize class-level information.
instance [22] 43 23 24
Meanwhile, 23 25
We show the differences between GDW and other related methods in Table
Method
3.1 Notations
In most classification tasks, there is a training set Dirain (1 W and assume there is also clean unbiased meta set Dmntt {(x]', M We aim to alleviate label noise and class imbalance Dirain with D Dmata- The model parameters are denoted as 0, and the number classes denoted C.
3.2 Class-level Weighting by Gradient Manipulation
To utilize class-level information, we learn class-level weight for every gradient flow instead of scalar weight for all C gradient flows in 23 Denote L as the loss of any instance. Applying the

---

chain rule, we unroll the gradient of L w.r.t. 0 as
0C aL Øl
VoL
22,
(I)
00
01 00
where E RC represents the predicted logit vector of the instance. We introduce class-level weights w E Rº and denote the jth component of w as wy- To indicate the importance of every gradient flow, we perform an element-wise product fw (+) on D₁ with w. After this manipulation, the gradient becomes
dC al
fw (VoL) =
= (w @ D2 = D/D2.
0l
00
where @ denotes the element-wiss product of two vectors. Note that wy represents the importance of the jth gradient flow. Obviously, instance weighting is special case of GDW when elements of w 46] adopt the Seftmax-Crossaatrayy CrossEntropy loss. In this case, we have D1 = P where RC denotes the probability vector output by softmax and y E R denotes the one-hot label of the instance (see Appendix for details).
are the same. Most classification tasks 44 45
As shown in Figure D for a noisy instance (e.g., cat mislabeled as "dog"), instance weighting methods assign low scalar weight to all gradient flows of the instance. Instead, GDW assigns class-level weights to different gradient flows by leveraging the meta set. In other words, GDW tries downweight the gradient flows for "dog" and "not cat", and upweight the gradient flow for "not bird". Similarly, in imbalance settings, different gradient flows have different class-level information. Thus GDW can also better handle class imbalance by adjusting the importance of different gradient flows.
3.3 Zero-mean Constraint on Class-level Weights
To retain the Softma-- CrossEntropy loss structure, i.e. the p y form, after the manipulation, we impose a zero-mean constraint on Dí. That is, we analyze the jth element of Dí (see Appendix B.1 for details):
()(P) - y,) y,) -w: (p', yy)
Σ WaPk Wi D'J.
(3)
wi is the weighted probability, and w; denotes the class-level weight at the target where p'} Ex waps
(label) position. We observe that the first term in Eq. (3) satisfies the structure of the gradient of the Sofimax- Oefimar-CCrosEntrey ErossEntropy loss, and thus propose to eliminate the second term which messes the structure. Specifically, we let
DiPeE Wi = we Ejjt WyPp
Pt
where pt is the probability of the target class. Note that E, W,Yj = w,, and thus we have
Ewj(p, - y5) 0.
This restricts the mean of D to be zero. Therefore, we name this constraint as the zero-mean constraint. With this, have
D = w (p' y)-
Eq. (6) indicates that w adjust the gradients in two levels, i.e., instance level and class level. Namely, the sealar wi acts as the instance-level weight previous instance weighting methods [22 23 24 25), and the wy"s 's are the class-level weights manipulating gradient flows by adjusting the probability from p p'.
3.4 Efficient Two-stage Weight Generation Embedded in Bi-level Optimization
In this subsection, we first illustrate the three-step bi-level optimization framework in 23 Further- more, we embed a two-stage scheme in the bi-level optimization framework to efficiently obtain class-level weights, with which manipulate gradient flows and optimize model parameters.

---

91
9w
BP on Dincts
Tle
Store class-level gradients
.90
Tw
R
Figure 2: Two-stage Weight Generation. "BP" denotes the back-propagation in Step 2 of the bi-level optimization framework, denotes the intermediate gradients w.r.t. w. θ denotes the minus operator. Note that w the first-stage (instance-level) weight and w' is the second-stage (class-level) weight.
Three-step Bi-level Optimization. Generally, the goal of classification tasks is to obtain the optimal
model parameters 0 by minimizing the average loss on Drrains denoted as N Σi= train (Te, U:: 0). As an instance weighting method, [23] adopt a three-layer MLP parameterized by Φ as the weighting network and take the loss of the jt instance as input and output scalar weight w;. Then 0* is optimized by minimizing the instance-level weighted training loss:
1
0* (Φ) arg min
wi (er)ler (T Me:
To obtain the optimal W/, they propose to use a meta set meta-lno and minimize the meta-loss to obtain x²:
Φ- arg min
M lont( vi; 0 (Φ)).
Since the optimization for 0 (Φ) and 6 is nested, they adopt online strategy to update 0 and &b with three-step optimization loop for efficiency. Denote the two sets of parameters at the 71th loop as 0, and Φ, respectively, and then the three-step loop is formulated as:
Step Update 0,-1 to 0, (+) via an SGD step on mini-batch training set by Eq. 7
Step 2 With (Φ)) update Φ,-1 to Φ,, via an SGD step on a mini-batee meta set by Eq.
Step With Φ,, update 0 to 0, via an SGD step on the same mini- batch training set by Eq.
Instance weights in Step 3 are better than those in Step 1, and thus used to update 0,-1-
Two-stage Weight Generation. To guarantee scalability, we apply the same weighting network in 23 to obtain weights. To efficiently train & and 0, we also adopt the three-step bi-level optimization framework. Moreover, we propose an efficient two-stage scheme embedded in Step 1-3 generate class-level weights. This process does not introduce any extra computational cost compared to (2 We keep the notations of 0, and Φ,, unchanged.
The first stage is embedded in Step 1. Explicitly, we obtain the first-stage class-level weights wi = w, by cloning the output of the weighting network for C times. Then we leverage the cloned weights W; to manipulate gradients and update 0 with a mini-batch of training instances:
0 (w--1) +- θ,-1 170 Σ fui( 1) (Volrainn (Fs. Mei 0,-1))
where n is the mini- batch size, is the learning rate of 0, and fwr(ts, (-) is the gradient manipula- tion operation defined in Eq. (2).
The second stage is embedded Step 2 and Step 3. Specifically in Step 2, GDW optimizes & with mini-batch meta set:
o + 01-1 - 1
V Vo-- (x) .... vi: (+
(10)
where m is the mini-batch size and no is the learning rate of Φb During the back-propagation in updating o.. GDW generates the second-stage weights using the intermediate gradients gi wi.
5

---

Note that 1 E(=1 V meeaa Cheta(a) (2) V.'; (0 and we have
gi
w/ = wi - clip(T
-
c, c),
(1I)
|gl|
where ||g/l denotes the el norm of class-level gradients within minibatch and c 0.2 represents the clip parameter. Then we impose the zero-mean constraint proposed in Eq. (4 on w/ which is later used in Step 3 to update - Note that the two-stage weight generation scheme does not introduce any extra computational cost compared to MWNet because this generation process only utilizes the intermediate gradients during the pack-propagation. In Step 3, w/, to manipulate gradients and update the model parameters -
0 r 0- - 1) - Σ Swl. (Voltrain (F Yzi 0 -11) 1=1
(12)
The only difference between Step 1 and Step is that the weighting network w; to optimize A. Since we only introduce & as extra learnable parameters, GDW can scale large datasets. We summarize GDW in Algorithm []] Moreover, visualize the two-stage weight generation process in Figure 2 for better demonstration.
w/ instead of the cloned output of
Algorithm 1 Gennrallizd Data Weighting via Class-Level Gradients Manipulation
Input: Training set: Dtrain, Meta set: Dieeta batch size n, # of iterations T Initial model parameters: 0o initial weighting network parameters: Φp
Output: Trained model: Or
for <- to do
2
9e 1 < SampleFrom/ (irain)
{x² li=1= < SampleFrom (Dmeta)
4
Generate w; from L, via the weighting network parameterized by Φ--1
Manipulate gradients by Eq.
(2 and update 0,, by Eq.
Update Φ by Eq. 110):
Update to w' by Eq. 11 and constrain w/ by Eq.
Manipulate gradients with w/ by Eq. (2) and update 0, by Eq. 12
4
Experiments
We conduct extensive experiments classification tasks to examine the performance of GDW. We compare GDW with other methods in the label noise setting and class imbalance setting Section 4. and Section 4.2 respectively. Next, perform experiments the real-world dataset Clothing M 4 Section We conduct further experiments to verify the performance of GDW in the mixed setting, i.e. the coexistence of label noise and class imbalance (see Appendix F for details).
4.1 Label Noise Setting
Setup. Following 23 we study two settings of label noise: Uniform noise: every instance's label uniformly flips to other class labels with probability p, b) Flip noise: each class randomly flips to another class with probability p. Note that the probability represents the noise ratio. We randomly select 100 clean images per class from CIFAR1O [47] as the meta set (1000 images in total).
Similarly, select total of 1000 images from CIFAR 100 its meta set. We use ResNet-32 44 as the classifier model.
Comparison methods. We mainly compare GDW with meta-learning methods: 1) L2RW 22
which assigns weights to instances based on gradient directions; 2) INSW (24), which derives
instance weights adaptively from the meta set; 3) MWNet [33; 4) Soft-label 42), which learns
label smoothing parameter for every instance; 5) Gen-label 40) which generates a meta-soft-lahbe
for every instance. We also compare GDW with some traditional methods: 6) BaseModel, which
trains ResNet-32 on the noisy training set; 7) Fine-tuning, which uses the meta set to fine-tune the trained model in BascModel; 8) Co-teaching [20); 9) GLC [9].

---

Table 2: Test accuracy on CIFARIO and CIFARIO0 with different uniform noise ratios.
CIFARIO
CIFART00
Dataset
0%
40%
60%
0%
40%
60%
BaseModel
92.73 0.37
84.38
0.32
77.92
0.29
70.42
0.54
57.28
0.80
46.86
1.54
Fine-tuning
92.77
0.37
84.73
0.47
78.41
0.31
0.57 70.52
57.38
0.87
47.06 + 1.47
Co-teaching
91.54 0.39
85.26
0.56
78.90
6.64
68.33
59.58
0.83
37.74 +: 2.60
GLC
90.85 + 0.22
86.12 1 0.54
81.55 : 0.60
65.05 0.59
56.99
+ 0.82
41.74 1.98
L2RW
$0.70:10.55
84.60 1.21
79.98 1: 1.18
63.40 1.31
47.06
1: 4.84
36.02 2.17
INSW
92.70
$4.88
0.64
78.77 -
70.52 0.39
57.11:0.
48.00 +: 1.16
MWNet
92.95 + 0.33
86.46 1 0.31
0.94
70.64 0.31 0.31
58.37
0.33
50.21 +: 2.98
Soft-label
92.63 +: 0.27
86.32 +: 0.10
0.294
70.50 1.50=:0: 0.44
57.48
4 0.43
48.18 + 0.89
Gen-label
92.56 + 0.56
84.68
+ 0.57
78.32 + 0.94
70.46 - 0.37
37.86
- 0.50
48.08 + 0.98
GDW
92.94 + 0.15
88.14 + 0.35
84.11 + 0.21
70.65 = 0.52
59.82 + 1.62
53.33 + 3.70
Table 3: Test accuracy on CIFARIO and CIFARIO0 with different flip noise ratios. CIFARIO
CIFARION 100
Dataset
0%
20%
40%
0%
20%
40%
MadeModel
92.73 + 0.37
90.14
0.35
81.20
0.93
70.42
0.54
64.96
0.16
49.83
0.82
Fine-tuning
92.77
0.37
90.15
0.36
81.53
0:33:000 0.96
70.52
0.57
0.22 65.02
50.23
0.71
Co-teaching
91.54 0.39
89.27
0.24
3.97
0.13
62.96
42.54 1.68
GLC
90.85 1 0.22
90.22 0.13
89.74
0.19
65.05
0.59
64.11
63.11
L2RW
89.70
88.21 0.49
82.90 1.27
63.40 1.31
55.27
+ 2.27
45.41 2.53
INSW
92.70 0.57
$0.9011..00
80.09
10.09:2:000
70.52
0.39
65.32 0.27
50.13 + 0.39
MWNet
92.95 + 0.33
89.93 1 0.17
8.85
70.64 0.31
64.72
50.62 + 0.46
Soft-label
92.63 + 0.27
90.17 1217:0000 0.47
$5.52 0.78
70.50 0.44
65.20 0.45
50.97 + 0.41
Gen-label
92.56 + 0.56
90.18 + 0.13
80.93 + 1.29
70.46 = 0.37
64.91 1.91+000 0.53
19.93 + 0.55
GDW
92.94 + 0.15
91.05 + 0.26
87.70 + 0.37
70.65 = 0.52
65.41 + 0.75
52.44 + 0.79
Training. Most of our training settings follow 233 and we use the cosine learning rate decay schedule [49] for a total of 80 epochs for all methods. See Appendix C for details.
Analysis. For all experiments, we report the mean and standard deviation over runs Table 2 and 3 where the best results are in bold and the second- -est results are marked by underlines. First, we can observe that GDW outperforms nearly all the competing methods in all noise settings except for the 40% flip noise setting. Under this setting, GLC estimates the label corruption matrix well and thus performs the best, whereas the flip noise assumption scarcely holds in real- wolld scenarios. Note that GLC also performs much better than MWNet under the 40% flip noise setting as reported in (23]. Besides, under all noise settings, GDW has a consistent performance gain compared with Furthermore, as the ratio increases from 40%
MWNet, which aligns with our motivation in Figure I to 60% in the uniform noise setting, the gap between GDW and MWNet increases from 1.68% to 2.97% in CIFAR10 10 and 1.45% to 3.12% in CIFARIOK 100. Even under 60% uniform noise, GDW still has low test errors in both datasets and achicves more than 3% gain in CIFAR1O 10 and 6% gain in CIFAR100 compared with the second- -bstt method. Last but not least, GDW outperforms Soft-label and Gen-label in all settings. One possible reason is that manipulating gradient flows is a more direct way to capture class-level information than learning labels.
In Figure 3 we show the distribution of class-level target weight (w)) on clean and noisy instances in one epoch under the CIFARI0 40% uniform noise setting. We observe that w; of most clean instances are larger than that of most noisy instances, which indicates that w, can distinguish between clean instances and noisy instances. This is consistent with Eq. (3) that w, serves as the instance weight.
To better understand the changing trend of non-target class-level weights, we visualize the ratio of increased weights in one epoch in Figure 5 under the CIFARI0 40% uniform noise setting. Specifically, there are three categories: norgtarget weights on clean instances (wf)). true target weights on noisy instances (w.)) and non-target (excluding true targets) weights on noisy instances (w")). Formally. "target weight" means the class-level weight on the label position. weight" means the class-level weight on the true label position, which are only applicable for noisy instances. "non-target weight" means the class-level weight except the label position and the true label position. For example, as shown in Figure [weeee a cat is mislabeled as "dog", the corresponding meanings of the notations are as follows: 1) w means Wang ("dog" the target); 2) wit means went ("cat" is the ture target); 3) wit means Whind (" ("bird" is one of the non-targets). For a correctly labeled
target
7

---

Distribution Target Weight Me
17500
:lean
13000
12500
Numbers
"dog"
7500
Gradient flow info.
"aot cut"
"dog"
"not bird"
5300
Initial Weight
2500
MW-Net
0.43
0.50
GDW
0.25
0.40
0.63
Figure 3: Class-level target weight (w,) distri- bution on CIFARIO under 40% uniform noise. Wi of most clean instances are larger than that of most noisy instances, which means we can differentiate between clean and noisy instances.
Figure 4: The change of class-level weights in an iteration for noisy instance (cat mislabeled as "dog"). MWNet downweights all gradient flows. In contrast, GDW upweights the "not bird" gradient flow for better information use.
cat, the corresponding meanings are: 1) wf wit means Went ("cat" is both the target and the ture target); 2) wht means Woog and Whind ("dog" and "bird" are both non-targets).
Note that in Figure D will represents the importance of the "not cat" gradient flow and who represents the importance the "not bird" gradient flow. If the cat image Figure i correctly labeled as "cat", then the two non-target weights w.. used to represent the importance of the "not dog" and the "not bird" gradient flows, respectively. In one epuch, we calculate the ratios of the number of increased wht. wit and wht to the number of all corresponding weights. w and wh, expected to increase since their gradient flows contain valuable information, whereas wit is expected to decrease because the "not cat" gradient flow contains harmful information. Figure Jlaigns perfectly with expectation. Note that the lines of whe and th nearly coincide with each other and fluctuate around 65%. This means non-target weights pattern, i.e., around 65% of wf, and wh; increase. Besides, less than 20% of wit increase and thus more than 80% decrease, which means the gradient flows of wit contain much harmful information.
clean instances and noisy instances share the same changing
In Figure 4 we show the change of class-level weights in an iteration for noisy instance, i.e., a cat image mislabeled "dog". The gradient flows of "not cat" and "dog" contain harmful information and thus are downweighted by GDW. In addition, GDW upweights the valuable "not bird" gradient flow from 0.45 0.63. By contrast, unable capture class-level information, MWNet downweights all gradient flows from 0.45 to 0.43, which leads to information loss on the "not bird" gradient flow.
Training without the zero-mean constraint. We have also tried training without the zero-mean constraint in Section 3.3 and got poor results (see Appendix B.2 for details). Denote the true target as i and one of the non-target labels as nt (nt + it). Note that the gradient can be unrolled as (see Appendix B. for details):
0l, 00
Øl, p/
fw (VoL) = w: Σ (v/) -y,)
+ Σ WIPP wi
Σ
(13)
00
If Σg WAPk: is positive and the learning rate is small enough, (Ex WhPi: - wi) Pte She contributes the decrease of the true target logit lu after a gradient descent step. If negative, (E, - w.) p'se Aha contributes to the increase of the non-target logit lnt. Therefore, without the zero-mean constraint, the second term in Eq. (13) may hurt the performance of the model regardless of the sign of Σn WaPk w, Similarly, training without the constraint results in poor performance in other settings. Hence we omit those results in the following subsections.
20
4.2 Class Imbalance Setting
Setup and comparison methods. The imbalance factor μ E (0,1) of a dataset is defined as the
number of instances in the smallest class divided by that the largest [23]. Long-Taine Tailed CIFAR created by reducing the number of training instances per class according an exponential function m//p/(C-1)) where i is the class index (0-indexed) and n, is the original number of
47

---

Ratio Trend of Increased Weights
Trend increased Gig.
0.35
0.30
0.25
0.20
o/
0.15
0.3
0.10
0.2
0.05
10
25
Epoch
Epoch
Figure 5: Ratio trend of the number of increased W Wit and wht under the CIFARIO 10 40% uni- form noise setting. Around 65% of wy, and wht increase since they contain useful informa- tion. Besides, less than 20% of w increase and thus more than 80% of wit decrease since they contain harmful information.
Figure 6: Ratio trend of the number of increased Wg on C99 instances under the Long- Tailed CI- FARIO μ 0.1 setting. Less than 10% of wa increase and thus more than 90% decrease. A small wg strikes a balance between two kinds of information: "C8" and "not C8", which better handles class imbalance.
Table 4: Test accuracy on the long-tailed CIFARIO and CIFARIO 100 with different imbalance ratios.
CIFARIO
CIFART00
Dataset
/ 0.01
==
| 0.1
|
ModeM
92.73
0.37
85.93 f: 0.57
69.77
1.13
70.42
0.54
37.79
0.82
Fine-tuning
92.77 0.37
82.60
1: 0.49
59.76
70.32
+: 0.57
55.95
37.10
Focal
91.68
0.49
84.57
0.83
65.78 4.02
0.38
0.51
37.43 1.00
Balanced
92.80 0.47
0.46 86.05
63.63
3.60
0.56 70.56
55.02
27.60 1.39
L2RW
89.70 0.50
79.11
51.15 7.13
63.4011..
4.52
25.86
25.76
INSW
92.70 0.57
0.28 86.31
70.27 0.24
70.32 + 0.39
55.94
37.67
0.59
MWNet
92.95 0.33
86.17
0.75
62.70
1.76
70.64 0.31
56.49
37.83 1
GDW
92.94 0.15
0.55 86.77
71.31
1.03
0.52 70.65
0.52 56.78
37.94
1.58
training instances. Comparison methods include: 1) L2RW 22]: 2) INSW 24]; 3) MWNet 23
4) BaseModel; 5) Fine-tuning; 6) Balanced 13): 7) Focal 21
Analysis. As shown in Table GDW performs best nearly all settings and exceeds MWNet by 8.6% when the imbalance ratio is 0.01 in CIFARIO. Besides, INSW achieves competitive performance at the cost of introducing huge amount of learnable parameters (equal the training dataset size N). Furthermore, we find that BaseModel achieves competitive performance, but fine- tuning on the meta set hurts the model's performance. We have tried different learning rates from 10-3 to 10-1 for fine-tuning, but the results are similar. One explanation is that the balanced meta set worsens the model learned from the imbalanced training set. These results align with the experimental results in 24 which also deals with class imbalance.
Denote the smallest class as C9 and the second smallest class as C8 in Long-Tailed CIFARIO with = 0.1. Recall that w, denotes the jth class-level weight. For all C9 instances in an epoch, we calculate the ratio of the number of increased wa to the number of all wa, and then visualize the ratio trend in Figure Since C9 is the smallest class, instance weighting methods upweight both Wg and Wy on a C'9 instance. Yet in Figure less than 10% of Wa increase and thus more than 90% decrease. This can be explained as follows. There are two kinds of information in the long-tailed dataset regarded to C8: "C8" and "not C8". Since C8 belongs the minority class, the dataset is biased towards the "not C8" information. Because smaller We weakens the "not C8" information. As result, decreased wa achieves a balance between two kinds of information: "C8" and "not C8", thus better handling class imbalance at the class level. We have conducted further experiments on imbalanced settings verify the effectiveness of GDW and see Appendix D for details.
represents the importance of "not C8".

---

Table Test accuracy on Clothing
Method
BaseModel
Fine-tuning
Co-teaching
GLC
L2RW
INSW
MWNet
Soft-label
Gen-label
GDW
Accuracy()
65.02
67.68
68.13
68.60
68.80
68.25
68.46
68.69
67.64
69.39
4.3 Real-world Setting
Setup and training. The ClothingIM dataset contains one million images from fourteen classes collected from the web [4. Labels are constructed from surrounding texts of images and thus contain some errors. We use the ResNet-18 model pre-trained on ImageNet [50] as the classifier. The comparison methods are the same as those in the label noise setting since the main issue of Clothing IM is label noise 4 All methods are trained for epochs via SGD with a 0.9 momentum, a 10 initial learning rate, a 10 -3 weight decay, and 128 batchsize. See Appendix E for details.
Analysis. As shown in Table 5 GDW achieves the best performance among all the comparison methods and outperforms MW Net by 0.93%. In contrast to unsatisfying results in previous settings, L2RW performs quite well in this setting. One possible explanation is that, compared with INSW and MWNet which update weights iteratively, L2RW obtains instance weights only based on current gradients. As result, L2RW can more quickly adapt to the model's state, but meanwhile suffers from unstable weights [23] In previous settings, we train models from scratch, which need stable weights stabilize training. Therefore, INSW and MWNet generally achieve better performance than L2RW. Whereas in this setting, we use the pre-trained ResNet- 18 model which is already stable enough. Thus L2RW performs better than INSW and MWNet.
5 Conclusion
Many instance weighting methods have recently been proposed to tackle label noise and class imbalance, but they cannot capture class-level information. For better information use when handling the two issues, we propose GDW to generalize data weighting from instance level to class level by reweighting gradient flows. Besides, to efficiently obtain class-level weights, we design two-stage weight generation scheme which is embedded in three-step bi-level optimization framework and leverages intermediate gradients to update class-level weights via gradient descent step. In this way, GDW achieves remarkable performance improvement in various settings.
The limitations of GDW are two-fold. Firstly, the gradient manipulation only applicable to single- label classification tasks. When applied to multi-label tasks, the formulation of gradient manipulation need some modifications. Secondly, GDW does not outperform comparison methods by a large margin in class imbalance settings despite the potential effectiveness analyzed in Section [4.2] One possible explanation is that better information utilization may not result in performance gain which also depends various other factors.
6
Acknowledgement
We thank Prof. Hao Zhang from Tsinghua University for helpful suggestions. This research was supported in part by the MSR-Mila collaboration funding. Besides, this research was empowered in part by the computational support provided by Compute Canada (www.computeeanadaaca))
References
[1] Hwanjun Song, Minseok Kim. Dongmin Park, Yooju Shin, and Jae- Lec. Learning from Noisy Labels with Deep Neural Networks: Survey. arXiv:2007.06 [cs, stat], June 2021.
[2] Haibo He and Edwardo A. Garcia. Learning from Imbalanced Data. IEEE Transactions on Knowledge and Data Engineering, 216))1263-1284, September 2009.
[3] Nancy E. EllHayy and Julien Provost. Systematic Survey on Sensor Failure Detection and Fault- Toleranee in Ambient Assisted Living. Sensors, 18(7):1991, July 2018.
10

---

[4] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2691-2699, Boston, MA, USA, June 2015. IEEE.
[5] Görkem Algan and Ilkay Ulusoy. Label Noise Types and Their Effects Deep Learning. arXiv:2003. 10471 [cs]. March 2020.
[6] Xingquan Zhu and Xindong Wu. Class Noise vs. Attribute Noise: A Quantitative Study. Artificial Intelligence Review, 22(3):177-210, November 2004.
[7] Benoit Frenay and Michel Verleysen. Classification in the Presence of Label Noise: A Survey.
IEEE Transactions on Neural Networks and Learning Systems, 25(5):845-869, May 2014.
[8] Qiuye Zhao and Mitch Marcus. Long- Tail Distributions and Unsupervised Learning of Mor- phology. In Proceedings of COLING 2012, pages 3121-3136, Mumbai, India, December 2012. The COLING 2012 Organizing Committee.
[9] Grant Van Horn and Pietro Perona. The Devil is in the Tails: Fine-grained Classification in the Wild. arXiv: 170.011 450 [cs], September 2017.
[10] Reyes Pavón, Rosalía Laza, Miguel Reboiro-Jato, and Florentino Fdez-Riverola Assessing the Impact of (lass-Imbllanced Data for Classifying Relevant/Ireleeant Medline Documents, In Miguel P. Rocha, Juan M. Corchado Rodríguez, Florentino Fdez-Riverola, and Alfonso Valencia, editors, 5th International Conference on Practical Applications of Computational Biology & Bioinformatics (PACBB 2011), Advances Intelligent and Soft Computing, pages 345-353, Berlin, Heidelberg, 2011. Springer.
[11] Harshita Patel, Dharmendra Singh Rajput, G Thippa Reddy, Celestine Iwendi, Ali Kashif Bashir, and Ohyun Jo. A review on classification of imbalanced data for wireless sensor networks. International Journal of Distributed Sensor Networks, 1554)115501477209 1404, April 2020.
[12] Qi Dong, Shaogang Gong, and Xiatian Zhu, Class Rectification Hard Mining for Imbalanced Deep Learning. In 2017 IEEE International Conference Computer Vision (ICCV), pages 1879-1878, Venice, October 2017. IEEE
[13] Yin Cui, Menglin Jia, Tsung-Yi Yi Lin, Yang Song, and Serge Belongie. Class-Balanced Loss Based on Effective Number of Samples. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9260,9269, June 2019.
[14] Saptarshi Sinha, Hiroki Ohashi, and Katsuyuki Nakamura. Class- Wise Sefficulty-Balaneee Loss for Solving Class-Imbalance. In Hiroshi Ishikawa, Cheng-Lin Liu, Tomas Pajdla, and Jianbo Shi, editors, Computer Aasion-ACCO 2020, Lecture Notes in Computer Science, pages 549-565. Cham, 2021. Springer International Publishing.
[15] Justin M. Johnson and Taghi M. Khoshgoftaar. Survey on deep learning with class imbalance. Journal of Big Data, 6(1):27, March 2019.
[16] Christian Szegedy, V. Vanhoucke, S. Ioffe, Jonathon Shlens, and Z. Wojna. Rethinking the Inception Architecture for Computer Vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[17] J. Goldberger and E. Ben-Reuven. Training deep In International Conference on Learning Representations, 2017.
using noise adaptation layer.
[18] Tongliang Liu and Dacheng Tao. Classification with Noisy Labels by Importance Reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38, November 2014.
[19] Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[20] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems, volume 31, Curran Associates, Inc.,
2018.

---

[21] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal Loss for Dense Object Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence. 12(2):318-322 327, February 2020.
[22] Mengye Ren, Wenyuan Zeng, Bin Yang. and Raquel Urtasun. Learning to Reweight Ex- amples for Robust Deep Learning. In Proceedings of the 35th International Conference on Machine Learning, pages 4334 4343. PMLR, July 2018.
[23] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-Weight-Ne Learning an Explicit Mapping For Sample Weighting. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[24] Zhiting Hu, Bowen Tan, Russ R Salakhutdinov, Tom M Mitchell, and Eric P Xing. Learning Data Manipulation for Augmentation and Weighting. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[25] Xinyi Wang, Hieu Pham, Paul Michel, Antonios Anastasopoulos, Graham Neubig, and J. Car- bonell. Optimizing Data Usage via Differentiable Rewards. International Conference on Machine Learning, 2020.
[26] Arash Vahdat.
Toward Robustness against Label Noise in Training Deep Discriminative Neural Networks. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[27] Charles Elkan.
The foundations of cost-sensitive learning. In Proceedings of the 17th International Joint Conference on Artificial Intelligence- Volume 2, ICCA101, pages 973-978, San Francisco, CA, USA, August 2001. Morgan Kaufmann Publishers Inc.
[28] Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16(1):321-357,
June 2002.
[29] Ashish Anand, Ganesan Pugalenthi, Gary B. Fogel. and P. N. Suganthan. An approach for classification of highly imbalanced data using weighting and undersampling. Amino Acids, 39(5):1385 13911 November 2010.
[30] Salman H. Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A. Sohel, and Roberto Togneri. Cost-Sensitive Learning of Deep Feature Representations From Imbalanced Data. IEEE Transactions on Neural Networks and Learning Systems, 29(8):3573-3588 August 2018.
[31] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling Representation and Classifier for Long-Tailed Recognition. In International Conference on Learning Representations, September 2019.
[32] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boging Gong, and Stella X. Yu. Large-Scale Long-Taiee Tailed Recognition in an Open World. In 2019 IEEE/CV Conference on Computer Vision and Pattern Recognition (CVPR), pages 2532-2002 2541, Long Beach, CA, USA, June 2019. IEEE.
[33] Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu. Long-tailed Recog- nition by Routing Diverse Distribution-Aware Experts. In International Conference Learning Representations, September 2020.
[34] Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(62666 December 2015.
[35] Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel Programming for Hyperparameter Optimization and Meta-Learning. In Proceedings of the 35th International Conference on Machine Learning, pages 1568-1577. PMLR, July 2018.
[36] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable Architecture Search. In International Conference on Learning Representations, September 2018.
12

---

[37] Yu- Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to Model the Tail. In Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[38] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels. In Proceedings of the 35th International Conference on Machine Learning, pages 2304-2313, PMLR, July 2018.
[39] Jumnan Li, Yongkang Wong, Qi Zhao, and Mohan S. Kankanhalli, Learning to Learn From Noisy Labeled Data. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5046-5054, Long Beach, CA, USA, June 2019. IEEE.
[40] G. Algan and I. Ulusoy. Meta Soft Label Generation for Noisy Labels, 2020 25th International Conference on Pattern Recognition (ICPR), 2021.
[41] Zizhao Zhang, Han Zhang, Sercan Ö, Arik, Honglak Lee, and Tomas Pfister. Distilling Effective Supervision From Severe Label Noise. In 2020 IEEE/CVF Conference on Computer Vision and Pattemn Recognition (CVPR), pages 9291 9300, June 2020.
[42] Nidhi Vyas, Shreyas Saxena, and Thomas Voice. Learning Soft Labels via Meta Learning. 00X4:2009.09496 [cs, stat], September 2020.
[43] Shreyas Saxena, Oncel Tuzel, and Dennis DeCoste. Data Parameters: A New Family of Parameters for Learning a Differentiable Curriculum. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[44] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv: 170.046044 [es], April 2017.
[45] Zhenyue Qin, Dongwoo Kim, and Tom Gedeon. Rethinking Softmax with Cross-Entropy: Neural Network Classifier as Mutual Information Estimator. arXiv:1911.1668 [cs. stat]. September 2020.
[46] Shuai Zhao, Liguang Zhou, Wenxiao Wang, Deng Cai, Tin Lun Lam, and Yangsheng Xu. Towards Better Accuracy-efficiency Trade-offs: Divide and Co-training. arXiv:2011.1466 [cs], March 2021.
[47] A. Krizhevsky. Learning Multiple Layers of Features from Tiny Images. MSc Thesis, Univer- sity of Toronto, 2009.
[48] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, June 2016.
[49] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. International Conference on Learning Representations, November 2016.
[50] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, June 2009.
13


Reviews:
[{"review_id": "PJ0LroomLN", "reviewer": "Reviewer_WWma", "summary": "The paper presents a method called Generalized Data Weighting (GDW) to mitigate label noise and class imbalance problems simultaneously. GDW works by re-weighting instances at the class level and it leverages meta-learning to learn the optimal weights. The main difference of presented approach from existing methods is that it allows to keep useful information contained in some of the logits (such as logits corresponding to non-target classes).  Also, the authors design a two-stage scheme embedded in the bi-level optimization framework for efficient computation of class-level weights without extra cost. Finally, the paper discusses importance of zero-mean constraint on class-level weights for stable training.\nThe authors conduct extensive experiments on CIFAR10 and CIFAR100 datasets in the label noise and class imbalance settings and compare GDW with other SOTA methods. Also, the model is evaluated on the real-world dataset with label noise, Clothing1M. The results show that GDW outperforms existing methods in almost all experiments. Finally, the authors explain an intuition behind GDW by showing how class-level weights change during training in samples with noisy labels. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The main novelty of the paper is the proposed method GDW which re-weights data at the class-level during training (instead of instance-level) to cope with label noise and class imbalance problems. The paper contains an extensive literature review of the existing methods. \n\nQuality: The submission is technically sound and conduct multiple experiments comparing the proposed approach to the SOTA methods. The results suggest that GDW significantly outperforms existing methods while the method does not introduce additional computational cost. \n\nClarity: The paper is clearly written and explains all important concepts as well as the intuition behind the proposed method. Although, I have a few questions for the authors: \n1. In equation 4, what is l in the summation? \n2. How exactly gradients g_i on ω_i are computed? Could you please clarify on how ω_i is updated? \n\nSignificance: I believe that the paper is significant as it addresses two important problems occurring in the real-world datasets: label noise and class imbalance. The proposed method is well motivated and outperforms SOTA methods in multiple experiments. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does not address limitations and potential negative societal impact, although I am not sure that it is applicable to the paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "K13cDk-lSbn", "reviewer": "Reviewer_NjUA", "summary": "This paper focused on label noise in class-imbalance situation, which is an important but challenge problem in reality. The authors claimed the deficiencies of previous works that adopted meta-learning frameworks, and accordingly proposed the GDW in further utilizing class-level information. This paper conducted extensive experiments on both synthetic and real-world data sets, and the authors claimed the superiority of their proposal. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper focused simultaneously on label noise and class imbalance problems, an important but challenging setup in real-world applications. As a canonical methodology, many previous works adopted meta-learning to solve both issues by leveraging a clean and unbiased meta set. However, the authors claimed that these methods only employ the instance-level weights, which cannot fully utilize class-level information within each instance. Accordingly, by class-level gradient manipulation, the authors proposed the Generalized Data Weighting (GDW), which can be viewed as a generalization of the vanilla IW. \n\nIn GDW, the authors adopted a series of techniques to boost the performance of the final algorithm. Specifically, to maintain the loss structure, the authors introduced the zero-mean constraint; for the effectiveness of the learning procedure, the authors improved the MWNet with minimal computational cost. \n\nIn my personal view, I believe this paper has some contributions. The authors proposed a general framework for meta-learning in the literature of label noise learning, which is well-motivated and could address the drawbacks in previous works. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have a little question to consult the authors. By the discussion below Eq. 6, it seems that the proposal can also be viewed as a meta-learning method for label correction. I think it would be great if the authors could make some clarifications.\n\nI believe this paper is well-motivated and of high quality. However, the experimental results on Clothing1M can hardly beat the performance for many state-of-the-art methods. Could the authors provide some explanation, or is there any space for further improvements?\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2h"}, "extra_text": {}}, {"review_id": "4J2Y8NBWzkp", "reviewer": "Reviewer_AiLN", "summary": "This work proposed a re-weighting method (GDW), to solve the noisy labeling and long-tailed training data issues in the single-label image classification task. The proposed GDW learns the weights considering both class information and instance information, and the computation of the weight is formulated in a meta-learning setting, using a clean validation dataset. The proposed method is evaluated on Cifar-10, Cifar-100, and Cloth1M datasets, and shows effectiveness in dealing with noisy labeled and imbalanced image classification.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n(1) This work proposed a re-weighting method for solving noisy label and imbalanced data issues simultaneously. Formulating these two problems in a uniform solution is novel and interesting. \n\n(2) The paper is well-written and easy to follow.\n\n(3) Learn to weight, implemented in a meta-learning framework, is reasonable.\n\nCons:\n\n(I) Method:\n\n(1) The class-wise weighting has been discussed in [D] to deal with the multi-label classification imbalance issue. Thus, this work is not the first work in this direction and the statements in Line 70 may need some change.\n\n(2) For the noisy labeling issue, de-weighting the class of noisy positive class is reasonable, but what is the motivation in single-label classification on imbalanced data?\n\n(3) Why not re-weighting the loss per class? What is the benefit of re-weighting gradient?\n\n(II) Experiments are not sufficient:\n\n(1) For long-tailed image classification, there is a lack of large-scale image classification datasets, such as ImageNet. Just testing on Cifar is not enough due to the small data quantity and a small class number.\n\n(2) From Tab3.and 4, the proposed method does not outperform the competitors with a large margin, and thus it is hard to verify the effectiveness of this work.\n\n(3) In long-tailed image classification, most competitors [22-24] are published on Arixv? Lack of sufficient comparison with the recent SoTa methods, such as [A, B, C].\n\n[A] Long-tailed Recognition by Routing Diverse Distribution-Aware Experts, ICLR21.\n\n[B] Large-Scale Long-Tailed Recognition in an Open World, CVPR19.\n\n[C] Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets, ECCV20.\n\n[D] Single-Label Multi-Class Image Classification by Deep Logistic Regression, AAAI19.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\n\n\n\nMy main concerns of this work are about the method motivations and experimental comparison. Specifically,\n\n(I) Method:\n\n(1) The class-wise weighting has been discussed in [D] to deal with the multi-label classification imbalance issue. Thus, this work is not the first work in this direction and the statements in Line 70 may need some change.\n\n(2) For the noisy labeling issue, de-weighting the class of noisy positive class is reasonable, but what is the motivation in single-label classification on imbalanced data? \n\n(3) Why not re-weighting the loss per class? What is the benefit of re-weighting gradient?\n\n\n\n(II) Experiments are not sufficient:\n\n(1) For long-tailed image classification, there is a lack of large-scale image classification datasets, such as ImageNet. Just testing on Cifar is not enough due to the small data quantity and a small class number.\n\n(2) From Tab3.and 4, the proposed method does not outperform the competitors with a large margin, and thus it is hard to verify the effectiveness of this work.\n\n(3) In long-tailed image classification, most competitors [22-24] are published on Arixv? Lack of sufficient comparison with the recent SoTa methods, such as [A, B, C].\n\n[A] Long-tailed Recognition by Routing Diverse Distribution-Aware Experts, ICLR21.\n\n[B] Large-Scale Long-Tailed Recognition in an Open World, CVPR19.\n\n[C] Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets, ECCV20.\n\n[D] Single-Label Multi-Class Image Classification by Deep Logistic Regression, AAAI19.\n", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "1vzCgiAFoM5", "reviewer": "Reviewer_jnwG", "summary": "This paper proposed a new method to tackle the noisy label and imbalanced data problem in classification task. Specifically, the authors proposed to apply class-level weights on the output logits for every instance. The class-level weight predictor is learned on the clean and balanced meta set. Experiments in two settings show that the proposed method has little improvement. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1.\tThe proposed method is well-motivated and well-illustrated. The Zero-mean Constraint trick (Eq 4) is interesting.\n2.\tThe proposed class-level weighting is similar to soft label methods which reassign the ground-truth label for training. This paper lacks the discussion of the difference. \n3.\tI am concerned about learning the weight predictor w(phi) on the clean meta set, which will be used for the noised training set. How can w(phi) learned on clean and balanced data work well on noisy and imbalanced data?\n4.\tThe proposed method has negligible improvements in most experiments. Therefore the effectiveness of the proposed method is limited. \n5.\tSuggestion: The term “class-level weighting” is a bit confusing. Perhaps the authors can find more accurate name.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have told a possible limitation. Societal impact is not applicable.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is a strong disagreement with this paper, with scores ranging as 8,7,5, and 4. The reviewer who provided a 4 said she increased her score to 5 but this has not propagated. In this case, the rating would be 6.25 rather than 6. 
While the accept reviews are stronger score wise, the reasons to reject (marginal improvements and novelty) have been far more detailed than the reasons to accept (also centered on improvements and novelty).

My own reading would align with the “accept” reviewers: novelty and improvements are genuine.

How novel is this contribution seems to hinge on a key question: is the proposed method similar to methods that would modify the loss function (see for instance https://arxiv.org/pdf/1811.08400.pdf publishded by one of the reviewers who supports reject) or weight instances ? One can observe that the proposed method does not reduce to a modification of the loss function, but does it add something fundamentally different ? The example in figure 1 is excellent: the authors clearly show that their scheme performs differently than instance weighting. 
Furthermore, I do not think any modification of the loss function could achieve the same results as a gradient-based method.

Improvements are indeed small, but this is what one would expect with a change in the loss function (we are not talking about a new architecture). As they do not incur added computational cost, this is something one could add on top of any system. Basically, the approach does no harm if there is no label noise, but seems to always improve as label noise increases, which is enough for adoption.

My main concern here remains that despite the claim that the method adds no computational complexity, we still do not have results on a larger dataset such as ImageNet (as promised by the authors)

Decision: Accept (Poster)