Noise Attention Learning: Enhancing Noise Robustness by Gradient Scaling
Authors: Yangdi Lu
Yang Bo
Wenbo He
Venue: NeurIPS 2022 Accept (2022)
Keywords: Machine Learning
Label Noise
Robustness

Abstract: Machine learning has been highly successful in data-driven applications but is often hampered when the data contains noise, especially label noise. When trained on noisy labels, deep neural networks tend to fit all noisy labels, resulting in poor generalization. To handle this problem, a common idea is to force the model to fit only clean samples rather than mislabeled ones. In this paper, we propose a simple yet effective method that automatically distinguishes the mislabeled samples and prevents the model from memorizing them, named Noise Attention Learning. In our method, we introduce an attention branch to produce attention weights based on representations of samples. This attention branch is learned to divide the samples according to the predictive power in their representations. We design the corresponding loss function that incorporates the attention weights for training the model without affecting the original learning direction. Empirical results show that most of the mislabeled samples yield significantly lower weights than the clean ones. Furthermore, our theoretical analysis shows that the gradients of training samples are dynamically scaled by the attention weights, implicitly preventing memorization of the mislabeled samples. Experimental results on two benchmarks (CIFAR-10 and CIFAR-100) with simulated label noise and three real-world noisy datasets (ANIMAL-10N, Clothing1M and Webvision) demonstrate that our approach outperforms state-of-the-art methods.
	 

Noise Attention Learning: Enhancing Noise Robustness by Gradient Scaling
Yangdi Lu Department of Computing and Software
Yang Bo
Department of Computing and Software McMaster University
McMaster University luy1 000mmmaster..ccaa
boy2@mcmaster.cm
Wenbo He
Department of Computing and Software
McMaster University
hew 11 @mcmas ter.
Abstract
Machine learning has been highly successful in data-driven applications but is often hampered when the data contains noise, especially label noise. When trained on noisy labels, deep neural networks tend to all noisy labels, resulting in poor generalization. To handle this problem, common idea is to force the model to only clean samples rather than mislabeled ones. In this paper, we propose a simple yet effective method that automatically distinguishes the mislabeled samples and prevents the model from memorizing them, named Noise Attention Learning. attention branch to produce attention weights
In our method, we introduce based on representations of samples. This attention branch learned to divide the samples according to the predictive power in their representations. We design the corresponding loss function that incorporates the attention weights for training the model without affecting the original learning direction. Empirical results show that most of the mislabeled samples yield significantly lower weights than the clean ones. Furthermore, training samples are dynamically scaled by the attention weights, implicitly preventing memorization of the mislabeled samples. Experimental results benchmarks (CIFAR-IP 10 and CIFAR- 100) with simulated label noise and three real- world noisy datasets (ANIMAL- 1ON, Clothing M and Webvision) demonstrate that our approach outperforms state-of-the-art methods.
theoretical analysis shows that the gradients
1 Introduction
With the emergence of highly-curater datasets such as ImageNet [1] and CIFAR [2), deep neural networks (DNNs) have achieved remarkable performance many classification tasks. However, in real-world applications, it is extremely time-consuming and expensive, sometimes even impossible large-scale dataset containing fully correct annotations. To alleviate this problem, one may obtain the data with lower quality annotations efficiently through online keywords queries [3] crowdsourcing [4]. but noisy labels (c.g. cat image is mislabeled as dog) are inevitably introduced consequently. Previous study [5] has empirically demonstrated that noisy labels in training data are problematic for deep neural networks, resulting in overfitting and performance degradation. Therefore, designing robust algorithms against noisy labels is of great practical importance.
to label
Given the training data consists of clean (correctly labeled) samples and mislabeled samples, DNNs have been observed to correctly predict the true labels for all training samples during a early learning
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Fracion of samgles
1.0
Clean samples
y 1:0
Mistabeled samples
g
Clean camples
1100-
Mislabeled camples
0.8
0.8
0.8
0.8
0.6
correct
8-
0.6
correct
incorrect
0.6
corfect incorrect
Fractio of
0.6
correct Incorrect
0.4
icorrect
0.4
memortiae loud
0.4
0.4
memror 2ed
0.2
0.2
0.2
0.2
100
200
300
400
0.0
300
400 500
0.0
100
500
100
Epoch
Epach
Epoch
(a) CE
(b) CE
(c) NAL (ours)
(d) NAL (ours)
The results of training a ResNet34 [14] with cross entropy (CE) loss and our proposed method on the CIFAR-10 with 40% symmetric noise. Plots (a) and (c) show the fraction of clean samples that are predicted correctly (green) and incorrectly (red). In contrast, plots (b) and (d) show the fraction of mislabeled samples that are predicted correctly (green), memorized (i.e. the prediction equals the wrong label, shown in blue). and incorrectly predicted as neither the true nor the labeled class (red). For clean samples, both models predict them correctly with the increasing of epochs. However, for mislabeled samples in plot (b), the model trained with CE loss first predicts the true labels correctly, but eventually memorizes all mislabeled samples. In contrast, our approach prevents memorization of mislabeled samples, allowing the model to continue learning the clean samples to attain high accuracy on all samples.
Figure 1:
stage, and then start to make incorrect predictions as it gradually memorizes the mislabeled samples [6, 7] (see Figure 1(a) and 1(b)). Based on this empirical finding, many existing approaches [8-11] use the training loss or prediction confidence to identify mislabeled samples in the early learning stage, i.e., the samples with small loss or high confidence are more likely to be clean samples. After dividing the samples into clean and mislabeled sets, one can simply train the model only based on clean set [8. 11]. or correct these mislabeled samples by pseudo-labels [9, 12), or treat them as unlabeled samples and train the model using semi-supervised learning algorithms [10]. However, these methods typically involve eophestioateee sophisticated training procedure modifications and sometimes require
high computational costs.
In this paper, we propose Noise Attention Learning (NAL) which enjoys simplicity and universality from a new perspective. The core idea behind NAL is to let the model automatically differentiate the mislabeled samples from clean samples, and prevent the memorization of mislabeled samples in training. Since the memorization of DNNs has a preference for easy (clean) samples, the predictive power of sample's 's representation aligns with its label cleanliness [6]. To transform this qualitative observation into a quantitive measure, we introduce an attention branch that outputs scalar attention weights to indicate the predictive power of representations. We incorporate the attention weights in our loss function, so that the attention branch can be learned to divide samples, i.e., clean samples are learned to be associated with large weights, while mislabeled ones with small weights. In this way, the proposed method does not require extra steps (c.g. modeling loss distribution [9, 10]. selecting small-loss samples [8]) to separate the training set, effectively simplifying the training procedure. Theoretically. demonstrate that NAL neutralizes the effect of the mislabeled samples on the gradient and ensures the contribution of clean samples to the whole gradient remains dominant, thereby preventing the deep neural networks from overfitting mislabeled samples.
The proposed method has a similar effect on gradients as an existing regularization method ELR [7]. In general, the performance of pure regularization methods cannot be compared to complex methods. Despite that the regularization methods can effectively hinder the memorization of mislabeled samples, the limited number of clean samples makes the model not rich enough to generalize effectively to the held-out data. To solve this problem, ELR uses temporal ensembling [13] to estimate the targets in its loss function to correctly guide the model for improving performance. Similarly, instead of directly using noisy labels as target distribution in our loss function, we adopt SELC [12] to estimate the targets, allowing the generation of more clean labels for learning sufficiency.
Empirically, we show that the proposed approach achieves better robustness than the state-of-the-ar methods on two benchmarks (CIFAR- 10 and CIFAR- 100 [2]) with simulated label noise and three challenging datasets in noisy label literature (ANIMAL-ION ION [15), Clothing MM [16] and Webvision [3]) with real-world label noise. To better understand our method, we conduct many empirical analyses, including memorization analysis, gradient analysis, attention weights, quality of targets, hyperparameter sensitivity and ablation study, to verify our theoretical explanations and design goals.
2

---

2 Preliminaries
Classification with Noisy Labels. Consider the K-class classification problem in noisy-label sce- nario, the ground truth label y is unobservable. We only have noisy training set D [(x ) where 2!(6 is an input and أار E y (1,.... K} is the corresponding noisy label. We denote gl! E {0, 1}K as one-hot vector of noisy label g|و A DNN N₂ maps an input 2! a K- dimensional logits and then feeds the logits softmax function obtain pl of the conditional probability of each class. denotes the parameters of the DNN and 2 E IR Kx1 denotes the logits (i.e. pre-softmax output). We refer to it prediction branch in this work. We have 2 (=! and p boftmax(21l the typical cross-entropy (CE) loss is often used as empirical risk to measure how well the model fits the training set D as follows:
Without knowing the ground truth joint probability distribution P(x, y).
N Σ 1=1
N
Lee
1
.p = N
(0 T log(p!"). E0
N
Early Learning Phenomenon When optimizing Lee by stochastic gradient descent (SGD), it has been observed that DNNS fit the easy (clean) samples first before memorizing of hard (mislabeled) samples [6]. Figure ((a) and 1(b) shows this qualitative behavior. The model begins by learning predict the true labels, even for majority of mislabeled samples, but eventually learns to predicts wrong labels due to memorization of mislabeled samples.
Label Noise Model. The generation of real-world label noise is unpredictable, a typical methodology to dealing with noisy labels is to posit noise model and develop robust algorithms under the hypothetical noise model. The algorithms are then tested on the real-world noisy datasets to see how effective they are. A widely-accepted noise model is class-conditional noise [17, 18], where label noise is independent of input features and true labels corrupted by symmetric or asymmetric noise transition matrix (simulation details are in Appendix B.2). instance-dependen noise [19, 20] is another noise model in which the noise is affected not only by the label but also by the input feature.
3 Noise Attention Learning
In this section, we present our approach NAL for learning with noisy labels. NAL consists of three key elements: (I) An attention branch based on learned representations produce the attention weight for each training sample; (2) A noise attention loss is specifically designed to learn the attention weights; (3) target estimation strategy to help generate more clean labels for sufficient learning.
3.1 Attention Branch based on Learned Representations
To enable the deep neural networks to produce an attention weight that reflects the quality of learned representation, we introduce an attention branch just after the penultimate layer of the original model. We denote M/ -i-dimensional penultimate layer (representation) of input 2 as H H is shared in both prediction and attention branches. For each input, the prediction branch outputs the softmax prediction pl as usual. The attention branch contains one fully connected layer to produce a single scalar value h, and s sigmoid function is applied scale it between to Specifically, we have h|4l W H( + b, where W E RIxM denotes the weights and IR denotes the bias in penultimate layer of the attention branch. We have the attention weight -| for each sample 2!!!
sigmoid(h!!),
g|( (0,
(2)
Note that W and b are extra network parameters need to be learned to produce meaningful r's. Therefore, attention weights are not fixed but dynamic during the training.
3.2 Noise Attention Loss
The early learning phenomenon reveals that the DNNs memorize the clean samples before the mislaheled samples. Therefore, clean samples are more likely to have better learned representations than mislabeled samples in the carly learning phase. To enable the attention weights to automatically
3

---

CIFAR-10 with 0% noise
CIFAR-10 with
CIFAR-10 40% naise
CIFAR-10 60% noise
200
madeteled samplexs
Clean samples
60
camples ribiboledd samples
30
mibAbblldd samples
camplee
20
ribAbblled scmplees
150
Dentry
Censtte
40
Demetty
20
Denety
15
100
0
0.4
0.5
08
1.0
0.4
0.8
0.2
0.8
Weght
Weigt
Weglt
1IFAR-100 with 0% noise
CIFAR-100 with 20% noise
CIFAR-100 with 40% noise
CIFAR-100 with 60% noise
50
clean samples
dlean samples
400
cloon sumples
dlean samples
madebeled surplles
400
m malateled sampliea
medateled surrples
modlbbled samples
300
Deniltt
300
Dennsite
20
200
200
10
100
100
0.4
0.6
0.8
1.0
0.65
0.90
0.95
1.00
0.90
0.95
1.00
0.90
0.06
1.00
Weight
Weigh T
Weigh:
Weight
Figure 2: The distribution of attention weights on the CIFAR-10/CIFAR-000 with different ratios of symmetric label noise using ResNet34.
capture the difference of representations as well as learn the classification task, we propose a noise attention loss consisting of two terms: attention term L, and boost term Lb.
N Σ (1) T llo (7 (79 |1
LNAL= = La + AL
log(?!!),
(3)
N
N
perceptual shortcut
where À is the hyperparameter. Intuitively, the attention term La can be interpreted dynamic expansion of the CE loss, where the prediction is modified to 'perceptual shortcut' T(p ÿ) + ÿ (Note that many existing methods [9, 21, 22] use this linear interpolation formula to infer the label distribution, while the proposed NAL applies it to prediction distribution for learning the meaningful attention weights). Assuming no representations are corrupted and all weights 7 - 1, the attention term La reduces to exactly Lee in Eq. (1). On the contrary, when all T + 0, the attention term La becomes "lazy" as it produces many zero gradients.
The mechanism of attention term can be explained as follows: (1) For mislabeled samples, the model does not memorize them in the early learning phase. Therefore, their representations under learned compared to clean samples and (p ÿ) remains large. By minimizing La, it forces of mislabeled samples toward O, which is treated as 'shortcut'. (2) For clean samples, the model memorizes them first, resulting in (p- 9) - O. It makes T have no influence on minimizing La as the prediction distribution is already equal to ÿ. As result, by solely minimizing Ca, we may obtain a trivial optimization result that the model always produces T - O for any inputs. To avoid this 'lazy' learning cireumstance, we have another boost term L.. It can be interpreted as a binary CE loss, where the target of T is always I for all inputs. By adding the boost term, the attention weights of clean samples are pushed to 1.0, and weights of mislabeled samples are still close to 0 as expected.
To verify the effectiveness of EALL in discriminating the mislabeled samples from clean samples, we empirically analyze the weight distribution of clean and mislabeled samples with simulated symmetric noise. Figure 2 shows results on 10 and CIFAR- with different ratios of symmetric noise. In 0% noise cases, we observe that all samples have weights close 1.0, which indicates that LNAL reduced to exactly fee- Thus, the proposed loss would not affect the performance when training with clean data. At other ratios of label noise, there is a clear separation between attention weights of clean and mislabeled samples. The weights of clean samples form spike close to 1.0, whereas the weights of most mislabeled samples are plainly less than 1.0, which satisfies our design goal.
3.3 Target Estimation
For regularization methods, directly using the noisy labels as target distribution in loss function is less effective as the limited number of clean samples makes the model not rich enough to generalize effectively to the held-out data, especially at the high noise level. To yield better performance, ELR [7]

---

Algorithm Noise Attention Learning (NAL) pseudocode
1: Input: DNN No; Training set D {(x!1) / Hyperparameters À and momentum a;
Total epoch F..
2: for epoch e to E do
3: for each minibatch B do for i in B do
4:
Obtain 2 A No (x
5:
6:
Obtain p 7 sof softmax( (2] sigmoid(h(e))
Update [ 8: end for
7:
[e) by Eq. (4).
9: Calculate .NAL- = D E! (1))) TTog log (1l0 - 1/21 + +/)))
[3 El log(r
10: Update 0 using stochastic gradient descent.
11: end for
12:
end for
13:
Output: 0.
the temporal ensembling [13] to estimate targets in its regularizer and steer the model towards these new targets. The ensemble predictions are formed by an exponential moving average of historical outputs. Specifically, in the m-th epoch, it can be expressed Pinn) D where 0 < < 1 is the momentum and Pijl represents the output of j-th epoch. In this paper, use the ensemble strategy in SELC [12] to estimate our targets t for the mnth epoch as follows:
th (1 a)a" a Pbb
f(m]
a U
aaaa PLi
(4)
initial term
ensemble torm
The initial term preserves the original noisy labels with exponential decaying weights a the early learning phase (small m), the overall target relies on noisy labels to learn the model. As the learning continues (larger ra), am approaches 0. The overall target eventually depends on the ensemble term Pim) and penalizes model predictions that are inconsistent with this target. In our experiment, we will demonstrate that estimated targets are more accurate than original noisy labels. We simply replace noisy label with ensemble target in our loss. Algorithm 1 shows pseudocode of our method.
4 Theoretical Justification
In this section, we demonstrate the noise robustness of LNLL by analyzing how it scales the gradients accordingly to prevent memorization of mislabeled samples. The omitted proofs in this section
in Appendix A. For clarity explanation, we denote the true label of sample æ as y E {L, K....K K}.
x)) = 1. Consider The ground tround-truth distribution over labels for sample æ is ())22)) and Σ q(k the case of a single ground-truth label y. then ((y(x)) = and ((k(2z)) = for all h: # y. We denote the prediction probability as p(k|az) (x) and Σk pkk]zz) = 1. For notation simplicity, we denote Pk= Qso Pys qn Pj. Qj abbreviations for ((k)))) q(k[az), p(y) (z), q(yl=)), p(j x)) and q(ilz)), where represents j-th entry. First, we show how the CE loss fails in noisy-label scenario.
Lemma Given the cross-entropy loss Lce in Eq. (1), we rewrite the sample-wise loss tee =
-Ek
K 1= Th log Pk- Its gradient with respect to 2j is
ol
Pj -150,
f qu (j is the true class for xx),
(5)
θzj
Pj 0, 9 (j is not the true class for x)),
where zj is the j-th entry of logits z. 0j 4y = means j equals the true class y.
clean-label scenario, Lemma ensures that, during SGD, learning direction of CE loss continues towards the true class as the corresponding gradient remains negative. However, in noisy-label scenario, the learning direction fluctuates. Suppose j is true class and equals y. but g due to the label noise, then the contribution of mislabeled sample to the gradient is reversed (i.c. gradient
5

---

1.0
samples
mis lbeled samples
Gradient
0.5
Gradient
0.2
0.0
0.0
-0.5
clean samples mislabeled samples
-0.2
-1.0
-0.4
0 100 200 300 400 500 Epoch
100 200 300 400 500 Epoch
(a) Gradient of Cee
(b) Gradient of CNAI.
Figure 3: The results of gradient on CIFAR-I0 with 40% symmetric label noise using ResNet34. We observe that in plot (a), the gradient of clean samples dominates in early learning stage, but afterwards it vanishes (close to 0) and the gradient of mislabeled samples dominates. In plot (b), NAL effectively keeps the gradient of clean samples dominant and diminishes the gradient of mislabeled samples.
should be negative but get positive instead). The entry corresponding to the impostor class j', is also reversed because q,- 1, causing the gradient of mislabeled samples dominates (in Figure 3(a)). Consequently, training with CE loss results in memorization of the mislabeled samples.
Theorem Given the noise attention loss LNAL Eq. (3), we rewrite the sample-wise loss eNAL = -EE log(T(pE - 4k) + q1)) - X log r. Its gradient with respect to the logits 2j can be
derived
Py
(p.j - 0,
qj (j the true class for x),
(6a)
dl
Py
1+ 1/7
0zj
Py 1 1/7 P.
Py
20. 0,
qj = 0 ij not the true class for x)),
(6b)
P7 where the multiplier
Pu 14-17 E (0,
Theorem I ensures that learning true class persists when training with noise attention loss. In addition, compared to the gradient of lee, the gradient ÉNAL scaled by an positive multiplier term
Py Py 14/7 Let's denote ;p Ps - ip is monotonically increasing on We have lim,-11 &p 1, and lim, o O. For the samples with the class Eq. (6a), the cross-entropy gradient term Pi 1 of clean samples tends to vanish after early learning stage, causing mislabeled samples to dominate the gradient. However, by multiplying o (note that y - for mislabeled samples and - for clean samples due to attention weight distribution). it counteracts the effect of gradient dominating by mislabeled samples. For the samples that j is not the true class Eq. (6b), the gradient term pj is positive. Multiplying yp < I effectively dampens the magnitudes of coefficients on these mislabeled samples, thereby diminishing their effect on the gradient. Figure 3(b) empirically shows the gradient when training with LNAL- It can be observed that NAL keeps the gradient of clean samples dominant and diminishes the gradient of mislabeled samples when epoch increases, effectively preventing memorization of mislabeled samples.
Py
5 Experiments
In this section, we first compare NAL with the existing methods on five widely used datasets. Then provide several empirical results towards a better understanding of NAL. We also provide an ablation study to evaluate the influence of each component and show the results when integrated with another noise robust method SCE [23]. Finally, discuss the scenarios when NAL encounters hard samples. All experiments are implemented in Py Torch and run in a single NIVDIA A100 GPU.
5.1 Comparison with Existing Methods
Datasets and Setups. We evaluate our approach two benchmarks CIFAR-10 and CIFAR-100
[2] with simulated label noise, and three real-worls datasets, ANIMAL--ION [15], Clothing M [16] and WisVision [3]. Since CIFAR-10 10 and CIFAR-100 are initially clean, we follow [18] to inject

---

Table 1: Test Accuracy (%) CIFAR-10 and CIFAR-100 with various levels of label noise injected to the training set. We compare with previous works under the same backbone ResNet34. The results averaged over 3 trials. Results are taken from their original papers. The best results are in bold.
Dhtract
CIFAR-10
CIFAR-IC0
Noise typ
1ymm
syym
kymm
seymm
Nosse ratio
21%
677
60%
807
80%
20%
40%
0MP6
30%
40%
Crosc Estropy
Forward [18]
74.96
Bostiing [23)
73.12
47.94
NI
04.25
92.43
71.52
$6.51
45.70
SCE(25) (25)
$7.13
62.51
081
25.11 0:44
69.32,
DMC
90,71
74.34
13.55
57,17
(29)
96.14
92.64
89.15
78.58
75.77
71,08
62.69
58.72
SELC
99.09 0.07
91.18
0.05
K7.25
94.13
0.14
91.05
0.11
73.83
0.07
68.46
5941
1:11::0000
31.33-
2082 0.00
92.12:+00:55
91.43 021
68.47
90.50
0.57
55
74.08
0:311
68.43 +
6005 + 0.78
90.2 +
75.75
NAL (ours)
94.37 x: 0.04
93.49 6.07
90.56
0.07
80.98
0.27
92.09 : 0.12
77.79 0.28
74.65 : 0,09
68.45 a 0.16
36.77 0.77
7..75
Table 2: Test accuracy (%) on CIFAR under different types of PMD noise [20]. All methods use PreActResNet34 architecture.
Dataset
Noise
Cross Entropy
SCE [23]
LRT [29]
PLC [20)
SELC (12)
NAL (ours)
Type-1 (35%)
78.11 + 0.74
79.76 + 0.72
80.98 + 0.80
82.80 + 0.27
86.97 + 0.15
88.81
0.13
CIFAR-10
Type-II (35%)
76.65 + 0.57
77.92 = 0.89
80.74 +: 0.25
81.54 0.47
87.06
0,20
87.66 0.23
Type-III (35%)
76.89 0.79
78.81 0.29
81.08 0.35
81.50 0.50
87.31
88.57 0.16
Type-I (35%)
$7.68 + 0.29
55.20 = 0.33
56.74 + 0.34
60.01 = 0.43
65.72 + 0.17
66.55
0.16
CIFAR-100
Type-II (35%)
57.83 + 0,25
56,10 = 0.73
57.25 + 0.68
63.68 = 0.29
66.79 + 0.18
67.15 0.11
Type-Ill (35%)
56.07 0.79
56.04 0.74
56.57
0.30
63.68
0.29
66.59 0.40
Table 3: The accuracy (%) results ANIMAL-ION. All methods VGG-19 architecture. Results of other methods are taken from original papers.
Cross Entropy
Nested [30]
SELFIE
PLC
SELC
Nested Co-teuching [30]
NAL (ours)
79.40 0.14 81.30 +: 0.60
81.80 + 0.09
83.40 +:2 0.43
83.73 +:2 0.06
84.10 0.10
84.18 0.19
Table 4: The accuracy (%) results Clothing IM. All compared methods use the ResNet-50 pretrained on ImageNet. The marker & denotes the model is trained from scratch.
Cross Entropy
SCE [23]
DMI [31]
ODNL [32]
ELR [7]
FINE [33]
Nested [30]
HOC [34]
NALÉ (ours)
69.21
71.02
72.46
72.47
72.87
72.91
73.10
73.39
73.58
Table 5: The accuracy (%) results on (mini) Webvision. Results of other methods are taken from their original papers. All methods use InceptionnesNetV2 architecture.
Co-teaching [8]
Iterative-CV [35]
CRUST [36]
ODD [37]
[38]
ELR (ours)
WebVision
top1 top5
63.58
65.24
74.60
75.16
76.26
77.41
85,20
85,34
89.56
90.60
90.77
91.26
92.25
topl
61.48
61.60
67.36
57.80
71.73
68.71
74.09
ILSVRC12
top5
84,70
84,98
87.84
86.30
91.61
87.84
92.09
class-come coasssioniiional label noise: symmetric and asymmetric label noises. Symmetric noise is generated by uniformly flipping the label one the other class label. Asymmetric noise is simulation of fine-grained classification, where the label flipping only occurs within very similar classes (e.g. dog ++ cat). Following PL. [20], we also evaluate NAL. on Polynomial Margin Diminishing (PMD) noise, an instance-dependeent noise that allows arbitrary noise strength in wide buffer near the decision
7

---

boundary. ANIMAL-ION 1ON contains human-labeled online images for 10 animals with confusing appearance. Its estimated noise rate is 8%. Clothing IM consists of 1 million images collected from online shopping websites with labels generated from surrounding texts. Its estimated noise rate is 38.5%. WehVision contains 2.4 million images crawled from the web using the 1000 concepts in ImageNet ILSVRCI2. Its estimated noise rate is 20%. For CIFAR with class-conditionss noise, we use a ResNet-34 [39] and train it using SGD with a batch size of 64. For CIFAR with instance- dependent noise, we use a PreActResNe134 [14] and train it using SGD with batch size of 128. For ANIMAL-ION, we use VGG-19 [40] with batch normalization and train it using SGD with a batch size of 128. For Clothing M,, train a ResNet-50 [39] using SGD with a batch size of 64. For Webvision, we train a InceptionRes details can be found in Appendix B.
[41] using SGD with a batch size of 32. More training
Table shows the results on CIFAR-10 10 and CIFAR- with different levels of symmetric and asymmetric label noise. We compare NAL to the best performing approaches that only modify the training loss. All of these methods use the same backbone (ResNet34). NAL shows substantial improvements over other methods and obtains the highest accuracy in most cases.
Table 2 shows the results on three types of instance-dependent (PMD) noise from PLC [20]. We observe that the proposed method outperforms baselines across different noise settings.
Table 3 shows the results on ANIMAL- ION. NAL achieves state-of-the-ee performance, even better than Nested which uses Co-teaching to boost performance.
Table 4 shows the results on the Clothing I M. All compared methods are trained using ResNet50 pretrained on ImageNet. However, as NAL needs to learn the parameters W and b in the attention branch, we directly train our model from scratch. As we can observe from Table 4, NAL still slightly outperforms other methods even without using an ImageNet pretrained model.
Table 5 shows the results on the (mini) WebVision following [35]. All the compared methods are evaluated on WebVision and ImageNet ILSVRCI2 validation sets. We observe that NAL consistently outperforms state-of tftte-of-thearrd
the-art art methods on both validation sets in terms of top1 and top5 accuracy.
Note that do not compare with some state-of of-the-art methods like RoCL [42] and DivideMix [10] as baselines, because these methods are aggregations of multiple techniques (e.g. mixup [43], multiple networks [8]. and complex augmentation [44]), while this paper only focuses one, so the comparison seems unfair.
5.2 Empirical Analysis of NAL
Memorization Procedure of NAL. Figure 1(c) and 1(d) show the memorization procedure of NAL on clean and mislabeled samples, respectively. Compared to CE loss in Figure ((a) and 1(b), NAL effectively prevents memorization of mislabeled samples, allowing the model to continue learning on the clean samples to attain high classification accuracy.
Gradient of Lee and LNAL. Figure 3(a) and 3(b) show the gradients of CE loss and noise attention loss CIFAR-10 with 40% symmetric noise using ResNet34 [39] respectively. In comparison to CE loss, the proposed method keeps the gradient of clean samples dominant and dampens the gradient of mislabeled samples, resulting the whole gradient being guided by clean samples.
Attention Weights. When trained on noisy labels, we have observed that the mislabeled samples have smaller weights than the clean samples in Figure 2. Here, we report the average weights of samples in Figure 4(a). The (-th block represents the average weights of samples with clean label i and noisy label j. We observe that the attention weights the diagonal blocks are higher than those on non-diagonal blocks, which indicates NAL does properly down-weight the mislabeled samples.
Quality of Estimated Targets. We evaluate the quality of new targets t by correction accuracy defined via 1 E. 1(argmax, // argmax, t!:l), where y! is the true label of input [ Figure N' 4(d) shows the correction accuracy vs. epochs on CIFAR- with different levels of label noise. The target estimation in Section 3.3 steadily improves the quality of new targets. Figure 4(b) and Figure 4(c) shows the confusion matrix of noisy labels and corrected labels (i.e. the hard version of new targets) w.r.t the clean labels on CIFAR-IO with 60% symmetric noise respectively. We observe that target estimation corrects the noisy labels impressively well for all classes.
8

---

airp. I0
6:17
6.5
airp.
2.05
1.0
0.01
0.00
auf:
0.45
but. -0.00 8:45 0.27
0.05
aut 02
0.000.00
bird
855
0.00
0.060
06
bird 0
6:37..00
000.00
cat
6:666557
cat 0.00 0.00
06
0.00
cat
00:5000000000 00000
5:01 000.00
0.00 0.000.0000007
0.000
0:23
0010000
0.000.000.00
dog 2.5L 9.4.0.69:11
0.7
dog 4.05 0.800
0:660.00
dog
0-:3 00 0.060
02
0.000000000
frog
4.00 0..66.006 0.060
-0.1
frog
horse
0.500.30 52 0656 5454
horse 4:06 06
0:06:0:06
(:05 0 0.066.0
horse: 1:40 00 0.80 6:220:00 0.00
000.000
ship-188
ship 1p-000
8.ce
ship
truck 3.5 0.50 40555
8.47
truck-8.00 2...8... 0. Beee
0e 0.00 3:47
truck 9.01 1210000 22c.0
8.00 0.00 2.20
dess
froe
saype out.
cat
dees
dog
horree
mace
airp
aot
Cae
deas
free
trovc
(a) Average Attention Weights
(b) Given Noisy Labels
(c) Estimated Targets
Correction accuracy (%)
100
100
95
80
Accuracy (%)
80
Accuracy (%)
85
60
60
40
40
75
symmetticc nolse
sommatttc nate
20
40% esymmetric nosse
0
100 200 300 400 500 Epoch
0.01 0.05 0.1 Hyperparameter
6,70
Momentum a
(d) Correction Accuracy
(e) Sensitivity
(f) Sensitivity of cx
Figure 4: (a). Average attention weights of noisy labels w.r.t clean labels on CIFAR-10 with 60%
symmetric noise. (b). Confusion matrix of noisy labels w.r. true labels on CIFAR-10 with 60%
symmetric noise. (c). Confusion matrix of corrected labels w.r.t true labels on CIFAR-I0 with 60%
symmetric noise. (d). Label correction accuracy vs. epochs on CIFAR-10 with different levels of
noise. Plots (c) and (f) show the sensitivity of À and a on CIFAR-10 10 with 60% symmetric noise.
Sensitivity of Hyperparameters À and NAL contains two hyperparameters: À is the coefficient for boost term Lo. a is the momentum in target estimation. Figure 4(e) and 4(f) show their sensitivity to the performance. We observe that the sensitivity to is quite mild as long as it is set close to 1. need to be tuned according to the complexity of dataset. It cannot be set to be very large or very small, resulting in neglecting attention term falling into the 'lazy' learning circumstance. In our experiments, we fixed a = 0.9 for all datasets. For CIFAR-10 10 and ANIMAL-ION, we set À = 0.5. For 100, we set À 10. For Webvision and Clothing IM, we set X = 50.
5.3 Ablation Study
Table 6 reports the influence of the two components in NAL: target estimation and attention branch. Removing the target estimation leads to significant performance drop. This suggests that using target estimation is crucial as it generates more clean samples for the model to learn. To validate the effect of the attention branch, we conduct another way to caleulate the attention weights: using the confidence score (highest probability in output) as the weight, i.e., max, j E [1, K]. We observe that the model does not converge in hard cases. We conjecture that using the confidence from model output does interfere with the original prediction branch, while adding the attention branch to get attention weight effectively solve this problem. We also explore the possibility of integrating NAL with other methods. Here we adopt the idea from symmetric cross entropy (SCE) [23], which adds the reverse term achieve noise robustness. As shown in Table 6, adding RNAL affects the performance under CIFAR- 10, but enjoys performance boost under CIFAR- without extra cost.
5.4 Discussion on Hard Samples
We assume the training set consists of mislabeled samples, easy (clean) samples and hard (clean) samples. In this work, we do not design a specific technique to distinguish hard samples but the proposed NAL can handle hard samples as well. the early learning phase, both mislabeled samples and hard samples have lower weights compared to the easy samples, which forces the model to learn only from easy samples due to gradient scaling (Theorem 1). After the model has learned from easy samples for while, the model then gradually learns from hard samples rather than the mislabeled

---

Table 6: Ablation study results in terms of test accuracy (%) on CIFAR-I0 10 and CIFAR-100. C means the model fails to converge. + RNAL means adding the reverse term of noise attention loss.
Datasel
CIFAR-10
CIFAR-100
Noise type
symm
asymm
symm
asymm
Noise ratio
40%
80%
40%
40%
80%
40%
NAL
93,49 = 0.07
80.98
0.27
92.09 = 0.12
74,65 + 0.09
36.77
0.71
74.73 0.12
NAL w/o target estimation
89.47 0.50
0.22 76.91
88.23 + 0.22
69.91 0.21
31.33
+ 0.38
55.68 0.17
NAL w/o attention brunch
90.94 = 0.28
0
91.55 + 0.07
NAL+ + RNAL,
92.92 + 0.29
80.20 + 0.42
88. + 0.63
75.38 + 0.08
38.24 + 0.55
74.89 + 0.20
samples. This is because the hard samples are more connected to the easy samples than the mislabeled samples, as the former two share some common features [45, 46]. Therefore, the proposed method can continuously learn from clean samples (both easy and hard) and prevent the memorization of mislabeled samples. In Figure 2, we can observe less overlap between clean and mislabeled samples. We also observe that some clean samples are given low weights, and some mislabeled samples are given high weights. The clean samples given low weights actually hard clean samples. There are also mislabeled samples given high weights. These mislabeled samples are hard samples with "proper" wrong labels. For example, given hard sample: a image that looks like a dog due to blurry resolution (e.g. 32 x 32). When generating simulated label noise, it is likely to assign a dog label for this sample. Compared to regular mislabeled samples, this kind of mislabeled samples can be learned easily by the classifier, resulting in having high weights.
Related Work
We briefly discuss the existing noise-robust methods that do not require set of clean training data (as opposed to [16, 47, 48]). Loss Correction. Many approaches focus on correcting the loss function explicitly by estimating the noise transition matrix [49, 18, 50]. Robust Loss Functions. These studies develop loss functions that are robust to label noise, including LDMI [31]. MAE [51], GCE [24], SCE [23], NCE [52]. TCE [53] and GJS [54]. Above two categories of methods do not utilize the early learning phenomenon. Sample Selection During the early learning stage, the samples with smaller loss values are more likely to be the clean samples. Based on this observation, MentorNet [55] pre-trains mentor network for selecting small-loss samples to guide the training of the student network. Co-teaching related methods [8. 56-58] maintain two networks, and each network is trained on the small-loss samples selected by its peer network. Label Correction Joint Opt [25] and PENCIL [59] replace the noisy labels with soft (i.e. model probability) or hard (i.e to one-hot vector) pseudo-labels. [9] weigh the clean and mislabeled samples by fitting two-component Beta mixture model to loss values, and corrects the labels via mixup combination. Similarly, DivideMix [10] trains two networks to separate the clean and mislabeled samples via a two-component Gaussian mixture model, and further uses MixMatch [60] to enhance the performance. Regularization [61] prove the gradient descent with early stopping is an effective regularization to achieve robustness to label noise. [62] explicitly add the regularizer based on neural tangent kernel [63] to limit the distance between the model parameters to initialization. ELR [7] estimates the target by temporal ensembling [13] and adds a regularization term to cross-entropy loss to avoid memorization. Other techniques, such as mixup augmentation [43], label smoothing [41] and weight averaging [64], can effectively improve the performance under label noise.
Conclusion
In this work, we propose NAL for learning with noisy labels. Our method leverages attention branch and a noise attention loss to learn the attention weights for distinguishing the mislabeled samples from clean samples. NAL can effectively diminish the gradient of mislabeled samples, mitigating the effect of noisy labels. We also provide extensive empirical analyses and evaluate its effectiveness across multiple datasets with different types and ratios of label noises.
There are still multiple open problems for future research. Currently, label noise has been extensively studied in image classification task. Other research areas, such as graph learning and federated learning, remain to be explored. On the methodological front, we hope that our work will trigger interest in designing novel network architectures that inherently provide robustness to label noise.
10

---

References
[1] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Féi-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Iece, 2009.
[2] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
[3] Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. CoRR, 2017.
[4] Xiyu Yu, Tongliang Liu, Mingming Gong, and Dacheng Tao. Learning with biased complemen- tary labels. Proceedings of the European Conference on Computer Vision (ECCV), pages 68 83, 2018.
[5] C Zhang, S Bengio, M Hardt, B Recht, and O Vinyals. Understanding deep learning requires rethinking generalization, 2018.
[6] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, al. closer look at memorization in deep networks. In Proceedings of the 34th International Conference Machine Mharning-Volumte 70, pages 233-242, JMLR. org, 2017.
[7] Sheng Liu, Jonathan Niles Weed, Narges Razavian, and Carlos Fernnndez-Granda. Early- learning regularization prevents memorization of noisy labels. Advances in Neural Information Processing Systems, 33, 2020.
[8] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pages 8527-8537, 2018.
[9] Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin Mcguinness. Unsupervised label noise modeling and loss correction. In Proceedings of the 36th International Conference on Machine Learning, pages 312-321, 2019.
[10] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. International Conference on Learning Representations, 2020.
[11] Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems, 15:17044417055 2020.
[12] Yangdi Lu and Wenbo He. Selc: Self-ensemble label correction improves learning with noisy labels. I/CA/ 2022, 2022.
[13] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning, arXiv preprint 1rX1v:1610.02242 2016.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. European conference on computer vision, pages 630-643 645. Springer, 2016.
[15] Hwanjun Song, Minseok Kim, and Jae-Gil Lee. Selfie: Refurbishing unclean samples for robust deep learning. In International Conference on Machine Learning, pages 5907-5915, PMLR, 2019.
[16] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2691-2699, 2015.
[17] Nagarajan Natarajan, Inderjit $ Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. NIP,, 26:1 1996-1104, 2013.
[18] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: loss correction approach. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 1944-1952 2017.
11

---

[19] Jiacheng Cheng, Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance and labelddependent label noise. In International Conference on Machine Learning, pages 1789-1799. PMIR, 2020.
[20] Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen. Learning with feature-dependent label noise: A progressive approach. In International Conference on Learning Representations, 2021.
[21] Scott E Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. In ICLR (Workshop). 2015.
[22] Xingjun Ma, Yisen Wang, Michael E Houle, Shuo Zhou, Sarah Erfani, Shutao Xia, Sudan- thi Wijewickrema, and James Bailey. Dimensionality-drive learning with noisy labels. In International Conference on Machine Learning, pages 3355-3364, PMLR, 2018.
[23] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey, Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 322-330, 2019.
[24]
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems, pages 8778 -7888, 2018.
[25] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 5552-5560, 2018.
[26] Youngdong Kim, Junho Yim, Juscung Yun, and Junmo Kim. Ninl: Negative learning for noisy labels. In Proceedings of the IEEE International Conference Computer Vision, pages 101-110, 2019.
[27] Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal Mohd- Yusof. Combating label noise in deep learning using abstention. In International Conference Machine Learning, pages 6234-6243. PMLR, 2019.
[28] Lang Huang, Chao Zhang, and Hongyang Zhang. Self-adaptive training: beyond empirical risk minimization. NeurIPS, 2020.
[29] Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In ICML. pages 11447-11457. PMLR, 2020.
[30] Yingyi Chen, Xi Shen, Shell Xu Hu, and Johan AK Suykens. Boosting co-teaching with compression regularization for label noise. In CVPR, pages 2688-2692, 2021.
[31] Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L_dmi: A novel information-thheortic loss function for training deep nets robust to label noise. In NeurlPS, pages 6222-6233, 2019.
[32] Hongxin Wei, Lue Tao, Renchunzi Xie, and Bo An. Open-set label noise can improve robustness against inherent label noise. NIPS, 2021.
[33] Tachyeon Kim, Jongwoo Ko, Sangwook Cho, Jinhwan Choi, and Se- Young Yun. Fine samples for learning with noisy labels. NIPS, 2021.
[34] Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability as an alternative to anchor points when learning with noisy labels. In International Conference on Machine Learning, pages 12912-12923. PMLR, 2021.
[35] Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In International Conference Machine Learning, pages 1062-1070, 2019.
[36] Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovee. Coresets for robust training of neural networks against noisy labels. NIPS, 2020.
12

---

[37] Jiaming Song. Yann Dauphin, Michael Auli, and Tengyu Ma. Robust and on-the-fly fhye-fl dataset denoising for image classification. European Conference on Computer Vision, pages 556-572. Springer, 2020.
[38] Fahad Sarfraz, Elahe Arani, and Bahram Zonooz. Noisy concurrent training for efficient learning under label noise. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3159-3168 2021.
[39] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[40] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
[41] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
[42] Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Robust curriculum learning: From clean label de- tection to noisy label self-correction. International Conference on Learning Representations, 2020.
[43] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018b.
[44] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
[45] Yingbin Bai and Tongliang Liu. Me-momentum: Extracting hard confident examples from noisily labeled data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9312-9321, 2021.
[46] Filipe R Cordeiro, Vasileios Belagiannis, Ian Reid, and Gustavo Carneiro. Propmix: Hard sample filtering and proportional mixup for learning with noisy labels. BMVC, 2021.
[47] Arash Vahdat. Toward robustness against label noise in training deep discriminative neural networks. Advances in Neural Information Processing Systems, pages $596-5605 2017.
[48] Dan Hendrycks, Mantas Mazcika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. In NeurIPS, 2018.
[49] Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using noise adapta- tion layer. 2016.
[50] Ryutaro Tanno, Ardavan Saeedi, Swami Sankaranarayanan, Daniel C Alexander, and Nathan Silberman. Learning from noisy labels by regularized estimation of annotator confusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11244- 11253, 2019.
[51] Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
[52] Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Normalized loss functions for deep learning with noisy labels. In International Conference on Machine Learning, pages
PMLR, 2020.
[53] Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li Li, and Bo An. Can cross entropy loss be robust to label noise. In Proceedings of the 29th International Joint Conferences on Artificial Intelligence, pages
2020.
[54] Erik Englesson and Hossein Azizpour. Generalized jensen-shannon divergence loss for learning with noisy labels. Advances in Neural Information Processing Systems, 34, 2021.
13

---

[55] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentorne:: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference Machine Learning, pages 2304-2313, PMLR, 2018.
[56] Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learning, pages 7164-7173. PMLR. 2019.
[57] Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13726-13735, 2020.
[58] Yangdi Lu, Yang Bo, and Wenbo He. An ensemble model for combating label noise. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pages 608 617, 2022.
[59] Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 7017-7025, 2019.
[60] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach semi-supervised learning. In Advances in Neural Information Processing Systems, pages 5050-5060, 2019.
[61] Mingehen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping provably robust to label noise for overparameterized neural networks. International Conference on Artificial Intelligence and Statistics, pages 4313-4324, PMLR, 2020.
[62] Wei Hu, Zhiyuan Li, and Dingli Yu. Simple and effective regularization methods for training on noisily labeled data with generalization guarantee. In International Conference on Learning Representations, 2019.
[63] Arthur Jacot, Franck Gabriel, and Clément Hongler. generalization in neural networks. Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 8580-85555 2018
Neural tangent kerne:: convergence and
[64] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Proceedings of the 3 International Conference on Neural Information Processing Systems, pages 1195-1204, 2017.
[65] Pengfei Chen, Junjie Ye, Guangyong Chen, Jingwei Zhao, and Pheng-Ann Heng. Robustness of accuracy metric and its inspirations in learning with noisy labels. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1151-11461, 2021.
14


Reviews:
[{"review_id": "cJM2yeFYHFB", "reviewer": "Reviewer_BcSR", "summary": "The authors proposed a simple method that trains an attention layer to generate weights for different samples. By using neural networks’ memorization effects, the weights can be learned during the training process. This method simplifies the training process, which does not need to extract confident examples. The authors also give theoretical analyses of their method. The proposed method experimentally shows large improvements compared with baselines.", "questions": "1. Although the empirical results are good, could the authors simply explain how this proposed attention branch distinguishes mislabelled samples and some hard examples that are not learned in the early learning phase?\n2. For CIFAR-10 with 60% in figure 2, I found some noise is given with high weights, and some clean data is given low weights. Can you explain it?  \n3. I found the performance is lower than some state-of-the-art methods, and the authors claim that it does not employ some techniques like MixUp, or two networks. Does the proposed method combine well with these widely used techniques in learning with label noise, for example, MixUp? \n", "limitations": "1. From my perspective, the word “attention” looks a little ambiguous. It more seems like a confidence score for each sample. \n2. The proposed method builds on semi-technique ensemble predictions. So, I think the authors should compare it with state-of-the-art methods that also use semi-technique e.g., DivideMix, FINE[1], or PES [2], which makes it easy to compare.  \n3. The authors only conduct symmetric label noise and asymmetric noise. Some commonly used settings like Pairflip 45% and Instance-dependent label noise [3, 4] are missing. \n4. The hyperparameter lambda is sensitive and has a large range for different datasets. \n\n[1] FINE Samples for Learning with Noisy Labels\n[2] Understanding and Improving Early Stopping for Learning with Noisy Labels \n[3] LEARNING WITH FEATURE-DEPENDENT LABEL NOISE: A PROGRESSIVE APPROACH\n[4] Part-dependent Label Noise: Towards Instance-dependent Label Noise Xiaobo\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Training an attention layer, directly outputting a score, is novel. \n2. The proposed method is simple and empirical works well.\n3. This paper is well organized and easy to understand.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SXUFbYjdGAU", "reviewer": "Reviewer_umW7", "summary": "This paper presents a method for training models with noisy labels. The proposed method learns a weight for each training sample based on its noise level (measured by the difference between model prediction and label), using an attention branch. Experimental results on benchmark datasets show that the proposed method was able to improve the model performance compared to several noisy training approaches.", "questions": "1. All the proposed techniques in this paper are built based upon the observation that noisy samples have higher loss compared to clean samples. How does the proposed method handle samples with different difficulty levels?\n2. It is unclear if the noise attention loss is used for the model training alone or in conjunction with the cross entropy loss.\n3. Math notation needs to be improved. For example, $j$ is never defined in Sec 3.3 and it is used represent different things in Lemma 4.1 and Theorem 4.1. What is the difference between $t$ and $\\hat{t}$ in Algorithm 1?\n", "limitations": "No", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths__\n- The idea is simple and experimental results are promising.\n\n__Weaknesses__\n- The theoretical justification is weak. Lemma 4.1 is essentially saying noisy labels hurts the model that trained with cross entropy loss. This is a well-known theory. I don't see how make it as a Lemma helps the discussion in this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "0ssD9X9bx-", "reviewer": "Reviewer_fZY5", "summary": "This paper suggests attention-based method to distinguish mislabeled data and decrease their impact during training. Specifically, the authors add an attention branch at the end of the feature-extraction layer (a.k.a. after the penultimate layer) that outputs a confidence score $\\tau$. Then, the attention layer's confidence score $\\tau$ is incorporated into the loss function to softly divide samples into clean ones and mislabeled ones; the gradient of clean ones is trained as-is due to their early learning phenomenon, while that of mislabeled ones is not trained because $\\tau$ is also trained to decrease the impact of mislabeled ones. The paper supports their idea both theoretically and empirically. ", "questions": "The questions would be about how to deal with the above weakness points.", "limitations": "No critical limitation is found up to this point.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\nThe strength of this paper can be elaborated as follows:\n1. The writing is easy and clear to follow.\n2. Incorporation of an attention layer and temporal ensembling-based target estimation is well-motivated and harmonious.\n3. Theoretical justification and empirical support are sound.\n\n## Weakness \n\n1. A strong weakness is the use of two hyperparameters $\\lambda$ and $\\alpha$. Especially, $\\lambda$ should be manually decided for each dataset, which is burdensome and hence becomes a strong hurdle for the suggested method.\n2. The baselines in the experiments are inconsistent, possibly due to the copy of experimental results from original papers. \n3. Why accuracy results on some of the two CIFAR datasets, Clothing 1M, and Webvision do not have variance results? They should be consistently reported.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept