ROBUST DISCRIMINATIVE REPRESENTATION LEARNING VIA GRADIENT RESCALING: AN EMPHASIS REGULARISATION PERSPECTIVE
Authors: Xinshao Wang
Yang Hua
Elyor Kodirov
Neil M. Robertson
Venue:  (2020)
Keywords: examples weighting
emphasis regularisation
gradient scaling
abnormal training examples

Abstract: It is fundamental and challenging to train robust and accurate Deep Neural Networks (DNNs) when semantically abnormal examples exist. Although great progress has been made, there is still one crucial research question which is not thoroughly explored yet: What training examples should be focused and how much more should they be emphasised to achieve robust learning? In this work, we study this question and propose gradient rescaling (GR) to solve it. GR modifies the magnitude of logit vector’s gradient to emphasise on relatively easier training data points when noise becomes more severe, which functions as explicit emphasis regularisation to improve the generalisation performance of DNNs. Apart from regularisation, we connect GR to examples weighting and designing robust loss functions. We empirically demonstrate that GR is highly anomaly-robust and outperforms the state-of-the-art by a large margin, e.g., increasing 7% on CIFAR100 with 40% noisy labels. It is also significantly superior to standard regularisers in both clean and abnormal settings. Furthermore, we present comprehensive ablation studies to explore the behaviours of GR under different cases, which is informative for applying GR in real-world scenarios.

Under review as a conference paper at ICLR 2020
ROBUST DISCRIMINATIVE REPRESENTATION LEARN- ING VIA GRADIENT RESCALING: AN EMPHASIS REG- ULARISATION PERSPECTIVE
Anonymous authors Paper under double-blind review
ABSTRACT
It is fundamental and challenging to train robust and accurate Deep Neural Net- works (DNNs) when semantically abnormal examples exist. Although great progress has been made, there is still one crucial research question which is not thoroughly explored yet: What training examples should be focused on and how much more should they be emphasised to achieve robust learning? ? In this work, we study this question and propose gradient rescaling (GR) solve it. GR modi- fies the magnitude of logit vector's s gradient to emphasise on relatively easier train- ing data points when noise becomes more severe, which functions as explicit em- phasis regularisation to improve the generalisation performance of DNNs. Apart from regularisation, we connect GR to examples weighting and designing robust loss functions. We empirically demonstrate that GR is highly anomaly-robus and outperforms the state-of-the-arr by a large margin, e.g., increasing 7% on CIFAR- 100 with 40% noisy labels. It is also significantly superior to standard regularisers in both clean and abnormal settings. Furthermore, we present comprehensive ab- lation studies to explore the behaviours of GR under different cases, which is informative for applying GR in real-world scenarios.
INTRODUCTION
DNNs have been successfully applied in diverse applications (Socher et al. 2011 Krizhevsky al., 2012
LeCun et al. 2013). However, their success is heavily reliant on the quality of training data, especially accurate semantic labels for learning supervision. Unfortunately, on the one hand, main- taining the quality of semantic labels as the scale of training data increases is expensive and almost impossible when the scale becomes excessively large. On the other hand, it has been demonstrated that DNNs are capable of memorising the whole training data even when all training labels are ran- dom (Zhang et al. 2017). Therefore, DNNs struggle to discern meaningful data patterns and ignore semantically abnormal examples] simultancously (Krueger et al., 2017 Arpit et al. 2017). Conse- quently, becomes an inevitable demand for DNNs to hold robustness when training data contains anomalies (Larsen et al.
1998 Natarajan et al. 2013 Sukhbaatar & Fergus 2014 Xiao al. 2015
Patrini et al. 2017: Vahdat
Veit et al. 2017 et 2017).
Recently, great progress has been made towards robustness against anomalies when training DNNs (Krueger al., 2017). There are three appealing perspectives in terms of their simplicity and effectiveness: 1) Examples weighting. For example, knowledge distilling from auxiliary mod- els is popular for heuristically designing weighting schemes. However, it is challenging to select and train reliable auxiliary models in practice (Li et al. 2017: Malach & Shalev-Shwarte 2017 2018b). 2) Robust loss functions (Van Rooyen et al, Jiang et al. 2018 Ren et al. 2018 Han et al. 2013 Ghosh al. [0177] Zhang& & Sabuncul 2018 Wang et al.) 2019b); 3) Explicit regularisation
Techniques (Arpit et al.] 2017; Zhang et al.
2018a). Although designing robust losses or explicit regularisation is easier and more flexible in practice, the performance
the optimal yet.
'One training example composed of input and its corresponding label. semantically abnormal example means the input is semantically unrelated to its label, which may come from corrupted input or label. For example, in Figure 3 in the supplementary material: 1) Out-of distribbbt anomalies: An image may contain only background or object which does not helong to any training class; 2) In-distribution anomalies: An image of class may he annotated class or an image may contain more than one semantic object.

---

Under review as a conference paper at ICLR 2020
Regarding examples weighting, there is a core research question which is not well answered yet: What training examples should be focused on and how large the emphasis spread should be? In this work, we present a thorough study of this practical question under different settings. For better analysis, we propose two basic and necessary concepts: emphasis focus and spread with explicit definition in Sec. 3.2 They are conceptually introduced as follows:
Emphasis focus. It is a common practice to focus on harder instances when training DNNs (Shri vastava et al., 2016; Lin et al. 2017). When dataset is clean, it achieves faster convergence and better performance to emphasise on harder examples because they own larger gradient magnitude, which means more information and a larger update step for model's parameters. However, when severe noise exists, as demonstrated in (Krueger et al., 2017: Arpit et al. 2017). DNNs learn simple meaningful patterns first before memorising abnormal ones. In other words, anomalies are harder to fit and own larger gradient magnitude in the later stage. Consequently, if we use the default sample weighting in categorical cross entropy (CCE) where harder samples obtain higher weights, anoma- lies tend to be fitted well especially when a network has large enough capacity. That is why we need to move the emphasis focus lowards relatively easier ones, which serves as emphasis regularisation.
Emphasis spread. We term the weighting variance of training examples emphasis spread. The key concept is that we should not treat all examples equally, neither should we let only a few be emphasised and contribute to the training. Therefore, when emphasis focus changes, the emphasis spread should be adjusted accordingly.
We integrate emphasis focus and spread into unified example weighting framework. Emphasis focus defines what training examples own higher weights while emphasis spread indicates how large variance over their weights. Specifically, we propose gradient rescaling (GR), which modifies the magnitude of logit vector's gradient. The logit vector is the output of the last fully connected (FC) layer of a network. We remark that do not design the weighting scheme heuristically from scratch. Instead, it is naturally motivated by the gradient analysis of several loss functions.
Interestingly, GR can be naturally connected to examples weighting, robust losses, explicit regu- larisation: 1) The gradient magnitude of logit vector can be regarded as weight assignment that built-in in loss functions (Gopal) |016] Alaiin et al., 2016) ZZang et al., (2018b). Therefore, rescaling the gradient magnitude equals to adjusting the weights of examples; 2) A specific loss function owns a fixed gradient derivation. Adjusting the gradient can be treated as more direct and flexible way of modifying optimisation objectives; 3) Instead of focusing on harder examples by default, we can adjust emphasis focus to relative easier ones when noise is severe. GR serves as emphasis reg- ularisation and is different from standard regularisers, e.g., 1.2 weight decay constraints on weight parameters and Dropout samples neural units randomly (Srivastava et al., (014));
GR is simple yet effective. We demonstrate its effectiveness on diverse computer vision tasks using different net architectures: 1) Image classification with clean training data; 2) Image classification with synthetic symmetric label noise, which is more challenging than asymmetric noise evaluated by [Vahdat 2017; Ma et al.) 2018); 3) Image classification with world unknown anomalies, which may contain open-set noise (Wang et al. 2018), e.g., images with only background, or outliers, etc.; 4) Video person re-identification, video retrieval task containing diverse anomalies. Beyond, we show that GR is notably better than other standard regularisers, e.g., L.2 weight decay and dropout. Besides, to comprehensively understand GR's s behaviours, we present extensive ablation studies.
Main contribution. Intuitively and principally, we claim that two basic factors, emphasis focus and spread, should be babysat simultaneously when it comes to examples weighting. To the best of our knowledge, we are the first to thoroughly study and analyse them together in a unified framework.
2 RELATED WORK
Aside from examples weighting, robust losses minimisation and explicit regularisation techniques, there are another two main perspectives for training robust and accurate DNNs when anomalies exist:
2016 Loshchilov & Hutter 2016 Hinton 2007), ) gradient magnitude (Gopal 2016 Alain et al., 2016). or input-to-labe relevance score (Lee) et al. 20188. The input-to-label relevance score means the probability of an input belonging to its labelled class predicted by current model. The difficulty of an example may change as the model learns. In summary, higher difficulty, larger loss, larger gradient magnitude, and lower input-to-label relevance score are equal concepts.
An example's difficulty can be indicated by its loss Shrivastava et al.
2

---

Under review as a conference paper at ICLR 2020
1) Robust training strategies (Miyato et al. 2018 Guo et al.
2018
2019: Thulasidasan et al. Li et
2019); 2) Noise-aware modelling, and alternative label and parameter optimisation are popular when only label noise exists. Some methods focus noise-aware modelling for correcting noisy labels
or empirical losses Larsen et al. 1998 Natarajan et al. 2013, Sukhbaatar & Fergus 2014 Xiao
al. 2015: Vahdat 2017] Veit al. 2017 Goldberger & Ben-Reuven 2017 Han al.] 201 8a).
However, it is non- trivial and time-consuming to learn a noise-aware model, which also requires
prior extra information or some specific assumptions. For example, Masking (Han al. (018 8ap is assisted by human cognition to speculate the noise structure of noise-aware matrix while (Veit et al.,
2017 Li al. 2017, Lee al. 2018 Hendryeks et al.) 2018) exploit an extra clean dataset, which hyper-factor and hard to control in practice. Some other algorithms iteratively train the model and infer latent true labels (Wang et al., 2018 Tanaka et al. 2018). Those methods have made great progress on label noise. But they are not directly applicable to unknown diverse semantic anomalies, which covers both out-of-distribution and in-distribution cases.
2.1 REMARKS ON ROBUSTNESS THEOREMS CONDITIONED ON SYMMETRIC LOSSES
We note that (Ghosh al. 2017) proposed some theorems showing that empirical risk minimization is robust when the loss function is symmetric and the noise type is label noise. However, they are not applicable for deep learning under arbitrary unknown noise: 1) We remark that we target at the problem of diverse or arbitrary abnormal examples, where input may be out-of-distribtiinn, ie., not belonging to any training class. As result, the symmetric losses custom-designed for label noise are not applicable. 2) GR is independent of empirical loss expressions in Table [] Therefore, one specific loss is merely an indicator of how far we are away from a specific minimisation objective. It has impact the robustness of the learning process since has no direct influence on the gradient back-propagation. Similar to the prior work of rethinking generalisation (Zhag al., 2017). we need rethink robust training under diverse anomalies, where the robustness theorems conditioned on symmetric losses and label noise are not directly applicable.
presented
EMPHASIS REGULARISATION BY GRADIENT RESCALING
Notation. We are given N training examples X = {(x,, y)) where (x,, y1) denotes i-th the number of classes. Let's consider
sample with input x; R and label y/1 E f1, 2,.,, ..., C}. a deep neural network z composed of an embedding network f(-): : R² - I&K and linear classifier g(.) : R.K + RC, i.e., 2, = =(x,) = (xx()x)) RDD + R Generally, the linear classifier the last FC layer which produces the final output of z, i.e., logit vector z E RC To obtain probabilities of a sample belonging to different classes, logit vector is normalised by a softmax function:
p(ix,)) = exp(zi))/ m=1 exp(Zim).
p(j(x;) )x)) is the probability of belonging to class j. A sample's input-to-label relevance score defined by p; (x)) In what follows, we will uncover the sample weighting in popu- lar losses: CCE, Mean Absolute Error (MAE) and Generalised Cross Entropy (GCE) (Zhang & Sabuncu 2018).
3.1 ANALYSING INTRINSIC SAMPLE WEIGHTING IN CCE, MAE AND GCE
CCE, The CCE loss with respect to (x,, y:), and its gradient with respect to Zij are defined as:
LCCE y.) = log p(y x.) and ôLCCE [p(vilx)) 1,
j=
(2)
dziz (p(j)x;),
Therefore, we have || 0LoCE || 2(1 - ((1=p((x(x,)) = 2(1-p;). p.). Here we choose L1 norm to measure
dh:
the magnitude of gradient because of its simpler statistics and computation.
Since we back-propagatt åLcce/z, to update the model's parameters, an example's gradient mag-
nitude determines how much impact it has, i... its weight w = &Leek CCE, more difficult examples with smaller ps get higher weight.
2(1 - p.). In
Uz.
MAE. When it comes to MAE, the loss of (xi, vi) and gradient with respect Zim are:
3

---

Under review as a conference paper at ICLR 2020
Table 1: Comparison between GR and other learning supervisions. 0~0.5 and 0~1 indicate the em- phasis focus is adjustable and ranges from 0 to 0.5 and 0 to 1, respectively. Note that GR manipulates the gradients and is independent of specific losses, e.g., CCE, MAE and GCE.
Supervision
Empirical loss
Gradient rescaling
Emphasis focus
Adjustable emphasis spread
CCE
CCE
No
0
No
MAE
MAE
No
0.5
No
GCE
GCE
No
0~00.5
No
GR
CCE/MAE/GCE
Yes
0~1
Yes
GR-212
GR-10-0
Sample's weight
MAE OFF
Sample's meiggn
100
8 8
10.3
Sympar werdi
8 8
0.2 0.6 0.6
0.0
Input-to-laoel relovanca Scord
Inpet-to-labll relevance score
Inout aaoe relevance score
(a) GR, CCE, MAE, GCE, We show 3 set- tings of GR: (3 = 2 0), (8 8 =
(b) GR when fixing 0.5 (emphasis focus is within
(c) GR when fixing / 8. When À increases, the empha- sis facus moves towards and emphasis spread drops.
0.5) and (8 = 12, A = 1). Their correspond- ing emphasis focuses are O, 0~00.5 and 0.5.
0~~0.5) or À = 2 (emphasis focus is within 0,5~11)
Figure| I: A sample's weight w; along with its input-to-label relevance score pt. GR is a unified sample reweighting framework from the perspective of gradient rescaling, where the emphasis focus and spread can be adjusted by choosing proper À and / in practice. Better viewed in colour.
OLMAE (2plys (x; (p(ui)xx) 1), Oz.j
j j= y/
LMAE(X; y:) = 2(1 -pyppllxx (x.)
and
(3)
j * y:
Therefore, uMMA= || |لMااه 1|1 4p(v 4p(v)/x;;) p(ul(x,))
4p,(1 p.). In MAE, those images whose input-to-label relevance scores are 0.5 become the emphasis focus. focus.
GCE. In GCE, the loss calculation (x,, and gradient with respect to logit vector z, are:
LGCE(X,, y.) 1 - p(u (x,"
and θLGCE [p(U xx (x))) (p(ss())) - 1), Ōz.j
Vi (4)
q
(p(s x; xx) "p(((xi);
x ys
where q E [0, Therefore, w GCE = case, the emphasis focus can be adjusted from 0 to 0.5 when g ranges from 0 to 1. However, in their
&Lece ||1 = 2p((y)/,)) (1-p(u = 2p In this thes
practice (Zhang & Sabuncu 2018), instead of using this naive version, truncated one is applied:
Lg(ps). LGCE (x, y:) (0.5).
Pt > 0.5
and L,(7) = (1-
(5)
p: S 0.5
The loss of an example with P. I 0.5 is constant that its gradient is zero, which means it is dropped and does not contribute to the training. The main drawback is that at the initial stage, the model is not well learned that the predicted p, of most samples are smaller than 0.3. To address it, alternative convex search is exploited for iterative data pruning and parameters optimisation, making it quite complex and less appealing in practice.
The derivation details of Eq.
are presented in Section
B of the supplementary material.
3.2 GRADIENT RESCALING FOR EMPHASIS REGULARISATION
A loss function provides supervision information by its derivative with respect to a network's 's output. Therefore, there are two perspectives for improving the supervision information: 1) Modifying the loss format to improve its corresponding derivative; 2) Manipulating the gradient straightforwardly. In this work, choose to control the gradient, which is more direct and flexible.
4

---

Under review as a conference paper at ICLR 2020
the gradients of CCE, MAE and GCE share the same direction. Our proposal GR unifies them from the gradient perspective. Being independent of loss formulas, a sample's gradient is rescaled linearly so that its weight is wfR.
According to Eq. 2 3
aL OLCCE
w wR
ŌLMAE oR ÖLGCE nh Oz. GCE
GRl wi = g(3pp p.)) => oz. Oz. OCE
(6)
Oz:
w MAF
where X, are hyper-parameters for controlling the emphasis focus and spread, respectively. By choosing a larger when more anomalies exist, GR regularises examples weighting by moving emphasis focus toward relatively casier training data points, thus embracing noisc-robustness.
For clarification, explicitly define the emphasis focus and spread over training examples: Definition (Emphasis Focus 16). The emphasis focus refers to those examples that own the largest weight. Since an example's 's weight is determined by its iaput-to-labed relevance score pi for sim- plicity, we define the emphasis focus to be an input-to-label score to which the largest weight is assigned, i.e., vo arg max u. [0,1).
pt
Definition 2 (Emphasis Spread σ). The emphasis spread is the weight variance over all training instances in a mini-batch, Le., a = E(( E(( where E(.) denotes the expectation value of a variable.
With these definitions, we differentiate GR with other methods in Table [ We show the sample weighting curves of GR with different settings in Figure [ As shown in Figure [T] the emphasis spread declines as increases. Therefore, we choose larger 33 values when À is larger in Sec. 4.2. Principally, transformation could be designed as any monotonically increasing function. Because the non-linear exponential mapping can change the overall weights' variance and relative weights between any two examples, we choose g(-) = exp(-). ), which works well in our practice. By integral, the exact loss format is an error function (non-elementary). We summarise several existing cases as follows (the ellipsis refers to other potential options which can be explored in the future):
w
CCE MAB
B 2, 0, 9 identity
8 = 4, A = 1, identity ity
GR Wi
w. GCE
8 = 1,1 A 2 0.g identity
exp(Bp/ pt)),
8> 0, 0, g exp
3.3 WHY DOES GR CONTRIBUTE TO ROBUST LEARNING?
Let's regard a deep network = as a black box, which produces C. logits. C' is the class number. Then during gradient back-propagation, an example's 's impact on the update of is determined by its gradient w.r.t. the logit vector. The impact can be decomposed into two factors, i.e., gradient direction and magnitude. To reduce the impact of noisy sample, we can either reduce its gradient magnitude or amend its gradient direction. In this work, inspired by the analysis of CCE, MAE and GCE, which only differ in the gradient magnitude while perform quite differently, leading natural motivation that gradient magnitude matters. That is why we explore rescaling the gradient magnitude as illustrated in Figure It is worth studying amending gradient directions in the future.
4 EXPERIMENTS
4.1 IMAGE CLASSIHCATION WITH CLEAN TRAINING DATA
Datasets. We test on CIFAR-IO and CIFAR-100 (rizhevssy 2009), which contain 10 and 100 classes, respectively. In CIFAR-10, 10, the training data contains 5k images per class while the test set includes 1k images per class. In CIFAR-100, there are 500 images per class for training and 100 images per class for testing.
Implementation details. On CIFAR-10, following (He et al. 2016), we adopt ResNet-20 and ResNet-56 as backbones so that we can compare fairly with their reported results. On CIFAR- 100, we follow D2L MM et al. 2018)) to choose ResNet-44 and compare with its reported results. We also use an SGD optimiser with momentum 0.9 and weight decay 10-4, The learning rate is initialised with 0.1, and multiplied with 0.1 every 5k iterations. We apply the standard data augmentation as in (He et al.) 2016 Ma et al. 2018): The original images are padded with 4 pixels on every side, followed by random crop of 32 32 and horizontal flip. The batch size is 128.
5

---

Under review as a conference paper at ICLR 2020
Table Classification accuracies (%) of CCE, and GR on clean CIFAR-IO and CIFAR-100, means the emphasis focus is O where we fix / = 2. 3 = 0 means all examples are treated equally.
Dataset
Backbone
CCE
GR (A = 0)
GR (8 0)
ResNet-20
91.8
91.8
91.0
CIFAR-10
ResNet-56
92.4
92.5
91.9
CIFAR-100
ResNet-44
68.1
68.4
66.4
Results. Our purpose is to show GR can achieve competitive performance with CCE under clean data to demonstrate its general applicability. As reported in D2L, all noise-tolerant proposals (Patrini cct al. 2017 Reed et al. 2015 Ma et al. (018)) perform similarly with CCE when training labels are clean. Therefore we do not present other related competitors here. Our reimplemented results shown in Table 2
For reference, the reported results in (He et al. 2016) on CIFAR- 10 with CCE are 91.3% for ResNet-20 and 93.0% for ResNet-56. In D2L., the result CIFAR-100 with ResNet-44 is 68.2%. Our reimplemented performance of CCE is only slightly different. For GR, we observe the best performance when emphasis focus is O, i... À= 0. Furthermore, it is insensitive range of emphasis spreads according to our observations in Figure in the supplementary material.
wide
Treating training examples equally. As shown in Table 2 we obtain competitive performance by treating all training examples equally when 3 = 0. This is quite interesting and motivates us that sample differentiation and reweighting work much better only when noise exists.
4.2 IMAGE CLASSIFICATION WITH SYNTHETIC SYMMETRIC LABEL NOISE
Symmetric noise generation. Given a probability r, the original label of image is changed to one of the other class labels uniformly following (Tanaka et al. 2018: Ma et al.] 2018). r denotes the noise rate. Symmetric label noise generally exists in large-scale real-world applications where the dataset scale is so large that label quality is hard to guarantee. It is also demonstrated in (Vahdat) 2017) that it more challenging than asymmetric noisy labels (Reed et al. 2013 Patrini et al. 2017). which assume that label errors only exist within predefined set of similar classes. All augmented training examples share the same label as the original one.
4.2.1 EMPIRICAL ANALYSIS OF GR ON CIFAR-10
To understand GR well empirically, we explore the behaviours of GR on CIFAR-10 10 with r 20%, 40%. 60%, 80%, respectively. We use ResNet-56 which has larger capacity than ResNet-20.
Design choices. We mainly analyse the impact of different emphasis focuses for different noise rates. We explore 5 emphasis focuses by setting / = 0 or different X: 1) None: / = 0. There is no emphasis focus since all examples are treated equally: 2) 0: 0: 3) 0~0.5:
0.5; 4) 0.5: A = 1; 5) 0.5~1: X 2. We remark that when X is larger, the emphasis focus is higher, leading to relatively easier training data points are emphasised. As shown in Figure [ when emphasis focus changes, emphasis spread changes accordingly. Therefore, to set proper spread for each emphasis focus, we try 4 emphasis spread and choose the best one to compare the impact of emphasis focus.
Results analysis. We show the results in Table 3 The intact training set serves and we observe that its accuracy is always consistent with the final test accuracy. This motivates us that we can choose model's hyper-parameters À via a validation set in practice. We display the training dynamics in Figure We summarise our observations as follows:
validation set
Fitting and generalisation. We observe that CCE always achieves the best accuracy on corrupted training sets, which indicates that CCE has a strong data fitting ability even if there is severe noise (Zhang et al. 2017). As a result, CCE has much worse final test accuracy than most models.
Emphasising on harder examples. When there exist abnormal training examples, we obtain the worst final test accuracy if emphasis focus O, i.e., CCE and GR with À 0. This unveils that in applications where we have to learn from noisy training data, it will hurt the model's generalisation dramatically we use CCE or simply focus on harder training data points.
Emphasis focus. When noise rate is 0, 20%, 40%, 60%, and 80%, we obtain the best final test accuracy when x 0, X 0.5, X 1, X = 2, and X = 2, respectively. This demonstrates that
SSinee there is a large interval between different /3 in four trials, we deduce that the chosen one is not the optimal. The focus of this work is not optimize the hyper-parameters.

---

Under review as a conference paper at ICLR 2020
Table Results of CCE, GR on CIFAR-I0 with noisy labels. For every model, we show its best test accuracy during training and the final test accuracy when training terminates, which are indicated by 'Best' and 'Final', respectively. We also present the results on corrupted training sets and original intact one. The overlap rate between corrupted and intact sets is (1 r). Therefore, we can regard intact training set as a validation set. When À is larger, should be larger as shown Figure Ic
Testing
Accuracy on Training Sets (%)
Noise
Emphasis Focus
Model
Accuracy (%)
Rate
Best
Final
Corrupted/Fiting
Intact/Vliidation
CCE
86.5
95.7
80.6
None
GR (f=0)
83.5
58.1
50.6
60.2
0 0)
GR (8 2)
84.9
76.4
85.3
80.5
20%
0~00.5 (A 0.5)
(B = 12)
89.4
95.0
0.5 (A = 1)
GR (3 16)
87.3
86.7
78.4
93.8
0.5~1 (A = 2)
GR (B = 24)
85.8
85.5
76.0
91.4
0 None
CCE
82.8
60.9
83.0
64.4
GR (3=0)
71.8
44.9
31.3
45.8
0 (A = 0)
GR (3 1)
78.4
65.6
63.3
66.6
40%
0~0.55 (A = 0.5)
GR (55 = 12)
85.1
79.9
67.7
85.7
0.5 (A = 1)
GR (3 16)
84.7
83.3
60.3
88.9
0.5m1 2)
GR (B 20)
52.7
52.7
35.4
53.6
CCE
69.5
37.2
84.1
40.5
None
GR (8=0)
69.9
57.9
40.1
58.6
(A 0)
GR (3 0.5)
72.3
53.9
42.1
55.1
60%
0~0055 0.5)
GR 12)
77.5
58.5
55.5
62.6
= 1)
12)
41.0
73.9
0.5m1 (A 2)
GR (3 = 12)
80.2
72.5
44.9
75.4
CCE
36.1
16.1
54.3
18.4
None
(B=0)
44.4
28.2
20.6
28.8
0 () = 0) 0~0.55 (A = 0.5)
GR (3 0.5)
46.2
21.3
27.8
23.1
80%
GR (3 = 8)
51.6
22.4
46.1
24.4
0.5 (A= = 1)
GR (5 = 8)
35.5
31.5
19.8
32,3
0.5w1 (A
GR (B 12)
33.0
32.8
14.2
32.6
when noise rate higher, we can improve a model's robustness by moving emphasis focus towards relatively less difficult examples with larger X, which is informative in practice.
Emphasis spread. As displayed in Table Jnd Figures 7 in in the supplementary material, emphasis spread also matters lot when fixing emphasis focus, i.e., fixing X. For example in Table 3 when
= O, although focusing on harder examples similarly with CCE, GR can outperform CCE by modifying the emphasis spread. As shown Figures 10 some models even collapse and cannot converge if the emphasis spread is not rational.
4.2.2 COMPETING WITH THE STATE- OFL---F--ITT THE ART ON CIFAR-10
Implementation details. We follow the same settings as MentorNet (Jiang et al. 2018 8 to compare fairly with its reported results. Optimiser and data augmentation are described in Section 4.1
FullModel is the standard CCE trained using L2 weight decay and dropout Srriaastaaal ct al. 2014). Forgetting (Arpit et al. 2017) searches the dropout parameter in the range of (0.2-0.9).
Competitors.
Self-paced (Kumar al. (0TO)) Focal Loss (Lin al., 2017), and MentorNet (Jiang al. 2018
are representatives of example reweighting algorithms. Reed Soft (Reed et al. (015) a weakly-
supervised learning method. All methods use GoogLeNet V1 (Szegedy al.) 2015).
Results. We compare the results under different noise rates in Table GR with fixed hyper-
parameters - 8, À 0.5 outperforms the state-of-the-art GCE by large margin, especially when label noise becomes severe. Better results can be expected when optimising the hyper-parameters for each case. We remark that FullModel (naive CCE) (Jiang al. 2018) was trained with L2 weight decay and dropout. However, GR's regularization effect is much better in both clean and noisy cases.

---

Under review as a conference paper at ICLR 2020
0.8
Train accuracies
0.8
Train accuracies
0.5
0.6
0.6
0.4
9.4
0.2.
Lift Nores
GR:
GR. C
.....
0.5-1
10
Iterations
Iterations
Iterations
Text accurace
Test sccurciss
accurce e5
0.5
0.6
0.6
0.4
0.2
0
10
Itorations
110*
Rerations
-10
Iterations
100
(a) 20%.
(b) 40%.
(c) 60%.
The learning dynamics of ResNet-56 on CIFAR-10, 10, i.e., training and testing accuracies along with training iterations. The legend in the top left is shared by all subfigures. 'xxx: yyy' means 'method: emphasis focus'. The results of r = 80% are shown in Figure 6 in the supplementary material. We have two key observations: 1) When noise rate increases, better generalisation is obtained with higher emphasis focus, i.e., focusing on relatively easier examples; 2) Both overfitting and underfitting lead to bad generalisation. For example, 'CCE: O' fits training data much better than the others while 'GR: None' generally fits it unstably or lot worse. Better viewed in colour.
Figure 2:
4.2.3 COMPETING WITH THE OFATE-OF- ON CIFAR-100
Implementation details. Most basclines have been reimplemented in (Ma et al., 2018 with the same settings. Therefore, for direct comparison, we follow exactly their experimental configurations and use 4esNet-44 (He et al. 2016 Optimiser and data augmentation are described in Section 4.1 We repeat training and evaluation 5 times where different random seeds are used for generating noisy labels and model's initialisation. The mean test accuracy and standard deviation are reported.
Competitors. We compare with D2L (Ma et al. 2018). GCE (Zhang & Sabuncu 2018), and other baselines reimplemented in D2L: 1) Standard CCE (Ma et al., 2018); 2) Forward (Patrini et al.) 2017) uses noise-trane transitionn matrix to multiply the network's 's predictions for label correction; 3) Backward (Patrini al. 2017) applies the noise-transition matrix to multiply the CCE losses for loss correction; 4) Bootstrapping (Reed et al., 2013) trains models with new labels generated by convex combination of the original ones and their predictions. The convex combination can be soft (Boot-soft) or hard (Boot-hard); 5) D2L (Ma et al. 2018) achieves noise-robustness from novel perspective restricting the dimensionality expansion of learned subspaces during training and is the state-of-the-arr, 6) Since GCE outperforms MAE (Zhang & Sabuncu (018)). we only reimplement GCE for comparison; 7) SL. (Wang et al. 2019c) boosts CCE symmetrically with noise-robust counterpart, i.e., reverse cross entropy.
Results. We compare the results of GR and other algorithms Table 5 GR outperforms other competitors by large margin, especially when label noise is severe, e.g.. 40% and 60%. More importantly, we highlight that GR much simpler without any extra information. Compared with Forward and Backward, GR does not need any prior knowledge about the noise-transition matrix. Bootstrapping targets at label correction and time-consuming. D2L estimates the local intrinsic dimensionality every b mini-batches and checks the turning point for dimensionality expansion every epochs. However, b and e are difficult to choose and iterative monitoring is time-consuming.
4.3 IMAGE CLASSIHICATION WITH REAL-WORLD UNKNOWN NOISE
Dataset. Clothing IM (Xiao et al. 2015) contains million images. is an industrial- dataset and its noise structure is agnostic. According (Xiao et al.) 2015), around 61.54% training labels are reliable, i.c., the noise rate is about 38.46%. There are 14 classes from several online shopping websites. In addition, there are 50k, 14k, and 10k images with clean labels for training, validation,
8

---

Under review as a conference paper at ICLR 2020
Table 4: The results of GR and other noise-robust approaches on CIFAR-10 using GoogLeNet VI.
Noise rate r
FullModel (naive CCE)
Self- paced
Focal Loss
Reed Soft
MentorNet PD
Mentor DD
GR
Forgetting
GCE
(3 8,A = 0.5)
0.81
0.83
0.85
20%
0.76
0.76
0.80
0.77
0.78
0.79
0.79
0.81
0.83
40%
0.73
0.71
0.74
0.74
0.73
0.74
0.76
0.78
0.79
80%
0.42
0.44
0.33
0.40
0.39
0.44
0.46
0.50
0,57
Table 5: The accuracies (%) of GR and recent approaches on CIFAR-101 The results of fixed parameters (3 8, == 0.5) are shown in the second last column. With a little effort for optimising and X, the results and corresponding parameters are presented in the last column. The trend consistent with Table When r raises, we can increase B8 À for better robustness. The increasing scale is much smaller. This is because CIFAR-100 100 has 100 classes so that its distribution of pa (input-to-label relevance score) is different from CIFAR-10 10 after softmax normalisation.
GR (B 18=3 0.5)
Noise rate
CCE
GCE
Forward Backward
Boot-
Boot- soft
D2L
SL
GR (3.A)
hard
40% 42.9=0.2 60.30.02 08.7--228880000 38.7++0.3 58.5+0..4 5..3 02.2 60.00..22......0000
20%
64.1 0.2 (6, 0.3)
45.4+00.2
59,3+0.2 60.0 (6, 0.4) 44.4=0.11 01.9 52.0+033 53.7000
60% 41.00222...2022212.
34.510.2 36.7.1.33 32.310.11 42.310.2 41.5.0.00 49.9 49.9 (8, 0.5)
and testing, respectively. Here, follow and compare with existing methods that only learn from noisy training data since we would like to avoid exploiting auxiliary information.
Implementation details. We train ResNet-50 0 (He et al. 2016) and follow exactly the same settings as (Patrini et al.) 2017] Tanaka et al. 2018): 1) Initialisation: ResNet-50 is initialised by publicly available model pretrained on ImageNet (Russakovsky et al.
2015); 2) Optimisation: A SGD opti- miser with a momentum of 0.9 and weight decay of 10 is applied. The learning rate starts at 10-3 and is divided by 10 after 5 epochs. Training terminates 10 epochs; 3) Standard data aug- mentation: We first resize a raw input image to 256 x 256, and then crop it randomly at 224 x 224 followed by random horizontal flipping. The batch size is 64 due to memory limitation. Since the noise rate is around 38.46 46%, we simply set À = 1, = 16 following Table when noise rate is 40%.
Competitors. We compare with other cobbstt algorithms that have been evaluated on Cloth- ing IM with similar settings: 1) Standard CCE (Patrini et al. (017): 2) Since Forward outper- forms Backward on Clothing 1M (Patrini et al. 2017). we only present the result of Forward; 3) S S-adaptation applies an additional softmax layer estimate the noise-transition matrix (Goldberger & Ben-Reuven] 2017); 4) Masking a human-
approach that conveys human cognition
to speculate the structure of the noise-transition matrix (Han et al., 2018a). 5) Label optimisation
Tanaka et al., (2018) learns latent true labels and model's 's parameters iteratively. Two regularisation
terms added for label optimisation and adjusted in practice.
The results are compared in Table Under real-world agnostic noise, GR also outperforms the state-of-the-art. It is worth mentioning that the burden of noise-transition matrix estimation in Forward and S-adaptation is heavy due to alternative optimisation steps, and such estimation is non-trivial without big enough data. Masking exploits human cognition of structure prior and reduces the burden of estimation, nonetheless its performance is not competitive. Similarly, Label Optimisation requires alternative optimisation steps and is time-consuming.
Results.
4.4 VIDEO RETRIEVAL WITH DIVERSE ANOMALIES
Dataset and evaluation settings. MARS contains 20,715 videos of 1,261 persons ZZeng et al.) 2016). There are 1,067,516 frames in total. Because person videos are collected by tracking and de- tection algorithms, abnormal examples exist as shown in Figure the supplementary material. We remark that there are some anomalies containing only background or an dfist---iittitttio person. Exact noise type and rate unknown. Following standard settings, use 8,298 videos of 625 persons for training and 180 videos of the other 636 persons for testing. We report the cumulated matching characteristics (CMC) and mean average precision (mAP) results.

---

Under review as a conference paper at ICLR 2020
The classification accuracy (%) on Clothing IM with ResNet-50. CCE and GCE were reported in (Patrini et al. 2017) and (Wang et al., 2019c), respectively. CCE and GCE* are our reproduced results using the Caffe framework (Jia et al. 2014).
Table 6:
Boot Forward Optimisation S-adaptation Masking Optimisation -soft
Bilevel
Joint
CCE CCE* GCE GCE* SL
GR
69.1
69.8
69.9
70.4
71.1
72.2
68.9
71.7 69.8
72.5 71.0 73.2
Table 7: The video retrieval results on MARS. For fair comparison, all other methods use GoogLeNet V2 except DRSA and CAE using more complex ResNet-50.
Metric
CCE
MAE
GCE
DRSA
CAE
OSM+CAA
GR
MAP (%)
58.1
12.0
31.6
65.8
67.5
72.4
72.8
CMC-1 (%)
73.8
26.0
51.5
82.3
82.4
84.7
84.3
Implementation details. Following (Liu et al. 2017 Wang et al. 2019a), we train GoogleNet V2 (Toffe & Szegedy 2015) and treat a video as an image set, which means we use only appearance information without exploiting latent temporal information. A video's s representation is simply the average fusion its frames' representations. The learning rate starts from 0.01 and divided by every 10k iterations. We stop training at 50k iterations. We apply an SGD optimiser with a weight decay of 0,0005 and a momentum of 0.9. The batch size is 180. We use standard data augmentation: 227 x 227 crop is randomly sampled and flipped after resizing Training settings are the same for each method. We implement GCE with its reported best settings. At testing, following (Wang al. 2019a| Movshovitz-Attie Attias al. 2017 Law et al.) 2017). we first L2 normalise videos' features and then calculate the cosine similanity between every two of them.
original image 256 x 256.
The results are displayed in Table Although DRSA eL 2018) and CAE (Chen al. 2018) exploit extra temporal information by incorporating attention mechanisms, GR is superior to them in terms of both effectiveness and simplicity. OSM+CAA (Wang et al., 2019a is the only comparable method. However, OSM+CAA combines CCE and weighted contrastive loss to address anomalies, thus being more complex than GR. In addition, we highlight that one query may have multiple matching instances in the MARS benchmark. Consequently, mAP is a more reliable and accurate performance assessment. GR the best terms of mAP.
Results.
4.5 BEATING STANDARD REGULARISERS UNDER LABEL NOISE
In Table we compare our proposed regulariser GR with other standard ones, i.e., L2 weight decay and Dropout (Srivastava et al. 2014). We set the dropout rate to 0.2 and L2 weight decay rate to 10-4. For GR, as mentioned in Section 4.2.3 we fix / 8, X 0.5. Interestingly. Dropout+1.2 achieves 52.8% accuracy, which is even better than the state-of-the-ar in Table [] i.e., D2L with 52.0% accuracy. However, GR is better than those standard regularisers and their combinations significantly. GR works best when is together with L2 weight decay.
Table 8: Results of GR and other standard regularisers on 100. We set r 40%, i.e., the label noise is severe but not belongs the majority. We train ResNet-44. We report the average test accuracy and standard deviation (%) over 5 trials. Baseline means CCE without regularisation.
Baseline
L.2
Dropout Dropout+1.2
GR
GR+L2
GR+Dropout GR+L.2+
44.7+0.11 51.5+
46.7+0.5 52.8+0.44
55.7++0.3
59.
54.3+0.4
58.3+0.3
5 CONCLUSION
In this work, we present three main contributions: 1) We analyse and answer a core research ques- tion: What training examples should focused on and how large the emphasis spread should be? 2) We uncover and analyse that two basic factors, emphasis focus and spread, should be babysat simultaneously when it comes to examples weighting. Consequently, we propose a simple yet ef- fective gradient rescaling framework serving emphasis regularisation. 3) Extensive experiments on different tasks using different network architectures are reported for better understanding and demonstration of GR's effectiveness, which also valuable for applying GR in practice.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Guillaume Alain, Alex Lamb, Chinnadhurai Sankar, Aaron Courville, and Yoshua Bengio. Variance reduction in sgd by distributed importance sampling. In ICLR Workshop, 2016.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. closer look at memorization deep networks. ICMI, 2017.
Dapeng Chen, Hongsheng Li, Tong Xiao, Shuai Yi, and Xiaogang Wang. Video person re- identification with competitive simppe-ss aggregation and co-attentive snippet embedding. In CVPR, 2018.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. In AAAI, 2017.
Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using noise adaptation layer. In ICLR, 2017.
Siddharth Gopal. Adaptive sampling for sgd by exploiting side information. In ICMI,, 2016.
Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R Scott, and Dinglong Huang. Curriculumnet: Weakly supervised learning from large-scale web images. In ECCV, 2018.
Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy supervision. In NeurIPS, 201 8a.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurlPS, 2018b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks labels corrupted by severe noise. In NewrIPS, 2018.
Geoffrey E Hinton. To recognize shapes, first learn generate images. Progress in brain research, pp. 535-547, 2007.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In JCML, 2015.
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Ser- gio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embed- ding. ACMMM, 2014.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks corrupted labels. In ICML, 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In NewrIPS, 2012.
David Krueger, Nicolas Ballas, Stanislaw Jastrzebski, Devansh Arpit, Maxinder S Kanwal, Tegan Maharaj, Emmanuel Bengio, Asja Fischer, and Aaron Courville. Deep nets don't learn via mem- orization. In ICLR Workshop, 2017.
M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In NewrIPS, 2010.
Jan Larsen, L Nonboe, Mads Hintz-Madsen and Lars Kai Hansen. Design of robust neural network classifiers. In ICASSP, 1998.
11

---

Under review as a conference paper at ICLR 2020
Mare T Law, Raquel Urtasun, and Richard S Zemel. Deep spectral clustering learning. In ICML, 2017.
Yann LeCun, Yoshua Bengio, and Geoftrey Hinton. Deep learning. Nature, pp. 436, 2015.
Kuang-Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Cleannet: Transfer learning for scalable image classifier training with label noise. In CVPR, 2018.
Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S Kankanhalli. Learning to learn from noisy labeled data. CVPR, 2019.
Shuang Li. Slawomir Bak, Peter Carr, and Xiaogang Wang. Diversity regularized spatiotemporal attention for video-based person re-identification. In CVPR, 2018.
Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In ICCV, 2017.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection, In ICCV, 2017.
Yu Liu, Junjie Yan, and Wanli Ouyang. Quality aware network for set recognition. In CVPR, 2017.
Ilya Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks. In ICLR Workshop, 2016.
Xingjun Ma, Yisen Wang, Michael Houle, Shuo Zhou, Sarah M Erfani, Shu-Tao Xia, Sudanthi Wijewickrema, and James Bailey. Dimensionality-drienn learning with noisy labels. In JCMI., 2018.
Eran Malach and Shai Shalev-Shwartz. Decoupling "when to update" from "how update". In NewrIPS, 2017.
Takeru Miyato, Shin-ichi Maeda, Shin Ishii, and Masanori Koyama. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1979 1993, 2018.
Yair Movshovitz-Arin Attias, Alexander Toshev, Thomas K Leung, Sergey Ioffe, and Saurabh Singh. No fuss distance metric learning using proxies. In ICCV, 2017.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In NeurIPS, 2013.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, 2017.
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Ra- binovich. Training deep neural networks on noisy labels with bootstrapping. In ICLR Workshop, 2015.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In ICML, 2018.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, pp. 211-252, 2015.
Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region- based object detectors with online hard example mining. In CVPR, 2016.
Richard Socher, Cliff C Lin, Chris Manning, and Andrew Ng. Parsing natural scenes and natural language with recursive neural networks, In ICML, 2011.
12

---

Under review as a conference paper at ICLR 2020
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, 1lya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, pp. 1958, 2014.
Sainbayar Sukhbaatar and Rob Fergus. Learning from noisy labels with deep neural networks. arXiv preprint arXiv:1406.2080/ 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization frame- work for learning with noisy labels. In CVPR, 2018.
Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal Mohd- Yusof. Combating label noise in deep learning using abstention. In ICML, 2019.
Arash Vahdat. Toward robustness against label noise in training deep discriminative neural networks. In NeurlPS, 2017.
Brendan Van Rooyen, Aditya Menon, and Robert C Williamson. Learning with symmetric label noise: The importance of being unhinged. In NeurIPS. 2015.
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge Belongie. Learning from noisy large-scale datasets with minimal supervision. In CVPR, 2017.
Xinshao Wang, Yang Hua, Elyor Kodirov, Guosheng Hu, and Neil M. Robertson. Deep metric learning by online soft mining and class-aware attention. In AAAI, 2019a.
Xinshao Wang, Elyor Kodirov, Yang Hua, and Neil M Robertson. Improving MAE against CCE under label noise. arXiv preprint arXiv:1903.1 121.11193,,, 2019b.
Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. In CVPR, 2018
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In ICCV, 2019c.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, 2015.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLRR, 2017.
Hongyi Zhang. Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ICLR, 2018a.
Xu Zhang, Felix Xinnan Yu, Svebor Karaman, Wei Zhang, and Shih-Fu Chang. Heated-up softmax embedding. arXiv preprint 18X9 2018b.
Zhilu Zhang and Mert R Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurlPS, 2018.
Liang Zheng, Zhi Bie, Yifan Sun. Jingdong Wang. Chi Su. Shengjin Wang, and Qi Tian. Mars: A video benchmark for large-scale person re-identification. In ECCV, 2016.
13

---

Under review as a conference paper at ICLR 2020
Supplementary Material for Robust Discriminative Representations Learning via Gradient Rescaling: An Emphasis Regularisation Perspective
A DISPLAY OF SEMANTICALLY ABNORMAL TRAINING EXAMPLES
Horse class: The first three images are deer semantically.
This video labelled the person wearing black skirt.
This video is labelled as the person wearing green shirt.
Figure 3: Diverse semantically abnormal training examples highlighted by red boxes. The 1st row shows synthetic abnormal examples from corrupted CIFAR-10 (Krizhevsky) (009). The 2nd and 3rd rows present realistic abnormal examples from video person re-identification benchmark MARS (Zheng et al. (2166).
distri-distribution anomalies: The first image in the 3rd row contains only background and no semantic information at all. 2) The 2nd first image or the last one in the 3rd row may contain person that does not belong to any person in the training set.
In-distribution anomalies: 1) Some images of deer class are wrongly annotated to horse class. 2) We cannot decide the object of interest without any prior when an image contains more than one object, e.g., some images contain two persons in the 2nd row.
B DERIVATION DETAILS OF SOFTMAX, CCE, MAE AND GCE
B.1 DERIVATION OF SOFTMAX NORMALISATION
Based on Eq. (T). we have
p()[,;;) 1+
exp(zij Zig. )-
(8)
For left and right sides of Eq.
we calculate their derivatives w.r.t. z, simultaneously.
j = Vi-
Op(un(x;)
exp(zij Zin.)
p(u: 1x
Zigs
Öp(ui (xx)
p(ys/x;;(1 -1x(l(((()))))))
Zig.
If # yi.
8p(u6(x)) exp(zz; - Zin
pl x2)
Zij
(10)
Op(y: (x:) -p(()(xs)p((xxx))
=>
Zij
14

---

Under review as a conference paper at ICLR 2020
In summary, the derivation of softmax layer is:
Op(y. x.)
p(ve x:) - p(ui(x))).
= Vi
(1I)
Ōz
-p((y ((lu(xx)p((xxx))
+ yA
B.2 DERIVATION CCE
According to Eq. 2 we have
LOCE(x;; fe. W) = - p xx).
(12)
Therefore, we obtain (the parameters are omitted for brevity).
aLCCE Op(j|x;) 10.
-p(Us(x;)-
j == yi
(13)
j 5 yi
B.3
DERIVATION OF MAE
According to Eq. 1 we have
LMAB(X;; fo. W) = 2(1 - (p(()(x;))
(14)
Therefore, we obtain
OLMAE Op([(x;)
( -2
j j= yi
(15)
0,
1 3 y1
B.4 DERIVATION OF GCE
According to Eq. (4. we have
p(u: xx:)
LeCE(x; fe, W)
(16)
Therefore, obtain
aLGCE
( (-((xi)x)) y/i
(17)
0p(jlx:) 0,
+ y/4
B.5 DERIVATIVES W.R.T. LOGITS
B.5.1 OLccr/O2;
The calculation is based on Eq. 13 and Eq.
11
j = Vi. we have:
C oLece Op(y)
ôLocE Σ
Oziy.
Op(ilx:)
Zij
(18)
1=1
p(u((x)) -.
If j * yi, it becomes:
θlcCE C OLccR Op(ui(x))
dzij
Op(ilx;) Zij
(19)
p(j(xs).
In summary, OLocr/Oz, can be represented as:
aLocE
1 V/ Ozij (p(i(x.).
[p(v](x:)
(20)
j # y
15

---

Under review as a conference paper at ICLR 2020
B.5.2 0LMAE/OZ.
The calculation is analogous with that of ÖLoce /d..
According to Eq. 15 and Eq.
if y4:
θ.MAE
θLMAR @p(vilx;)
E
Özig.
0p(ilx)) Zij
(21)
--2p(ul(s)(( - p(us(x))).
otherwise (j yy):
C aLMAE Op(uilxi) E
ILMAE
dzij
8p(ilxs) 2ij
(22)
j=1
2p( x xxxx))
In summary, ÔLMAE/OZ, is:
θLMAE
[2plui x; (p(())))) 1), (()((xx)))
==
(23)
dz.s
B.5.3 ÖLace/Oz,
The calculation is based on Eq.
17 and Eq.
11
If j Vi. have:
aLocE
aLace @p(v)(x)) Op(jix;)
Uziy.
Zij
(24)
= (p(vlxx))(((((xxx))) 1).
If j * yr. it becomes:
òLGCE C ÒLGCE Op(ui(x;)
Oz.j
Op(iix)) Vij
(25)
1=1
=(,,, " p(j(>x;).
In summary, ÖLGCE/0e /0z; can be represented as:
ØLGCE
[p(ui (p(unlx))))) - 1), (p(lss)
j = yi
(26)
Özii
j * yi
C SMALL-SCALE FINE-GRAINER VISUAL CATEGORISATION OF VEHICLES
How does GR perform small datasets, for example, the number of data points is than 5,000? We have tested GR on CIFAR-10 and CIFAR-100 in the main paper. However, both of them contain training of 50,000 images.
For this question, we answer it from different perspectives as follows:
1. The problem label noise study on CIFAR-I0 and CIFAR-100 in Section 4.2 is of similar scale. For example:
In Table 4 when noise rate 80% on CIFAR-10, the number of clean training examples is around 50, 000 x 20% 5. 000 x 2. Therefore, this clean set is only two times large as 5,000. Beyond, the learning process may be interrupted by other noisy data points.
In Table when noise rate is 60% on CIFAR-100 100, the number of clean training data points is about 50, 000
40% 5, 000
i.e., four times large as 5,000.
16

---

Under review as a conference paper at ICLR 2020
2. We compare GR with other standard regularisers small-scale fine- fine-grained visual categori- sation problem in Table
Vehicles-10 10 Dataset. In CIFAR-100 Krizhevsky 2009., there 20 coarse classes, including hicles and 2. Vehicles I contains 5 fine classes: bicycle, bus, motorcycle, pickup truck, and train. Vehicles 2 includes another 5 fine classes: lawn-mower, rocket, streetcar, tank, and tractor. We build a small-scale vehicles classification dataset composed of these 10 vehicles from CIFAR-100. Specif- ically, the training contains 500 images per vehicle class while the testing set has 100 images per class. Therefore, the number of training data points is 5,000 in total.
Table 9: The test accuracy (%) of GR and other standard regularisers on Vehicles-10. We train ResNet-44. Baseline means CCE without regularisation. We test two cases: with symmetric label noise/ 40% and without symmetric label noise 0.
r
Baseline
1,2
Dropout
Dropout+1.2
GR
GR+L2
GR+Dropout
GR+L2+Dropout
0
75.4
76.4
77.9
78.7
83.8
84.4
84.5
84.7
40%
42.3
44.8
41.6
47.4
45.8
55.7
48.8
58.1
D TRAINING UNDER ASYMMETRIC LABEL NOISE
We evaluate on CIFAR-100, whose 100 classes are grouped into 20 coarse classes. Every coarse class has fine classes. Within each coarse class, an image's label is flipped to of the other four labels uniformly with a probability r. represents the noise rate. We set 0.2. The results are displayed in Table TO When GR is used, the performance is better than its counterparts without GR.
Table 10: The test accuracy (%) GR and other standard regularisers trained under asymmetric label noise. We train ResNet-44, Baseline means CCE without regularisation. We simply fix 8 8, A 0.5 when GR is used. Better results can be expected B, are optimised for each case.
Baseline
1.2
Dropout
Dropout+1.2
GR
GR+1.2
GR+Dropout
GRR-122Dropouut
55.1
59.4
57.1
63.7
59.4
61.4
E
THE EFFECTIVENESS OF LABEL CORRECTION
The results are shown Table
Table 11: How much fitting of the clean training subset and how much fitting of the noisy training subset? Is it plausible to correct the labels of training data? Our results demonstrate the effectiveness of label correction using DNNs trained by GR. When retraining from scratch on the relabelled training data, we do not adjust the and X. Therefore, the reported results of retraining relabelled datasets not the optimal.
Testing Accuracy (%)
Accuracy on
Fitting degree of subsets (%)
Retrain
Noise Emphasis Focus
Model
Training Sets (%)
after
Rate r
label correction
Best
Final
Noisy
Intact
Clean
Noisy
CCE
86.5
76.8
95.7
80.6
99.0
85,9
20%
0~0.5
GR 12)
89.4
87.8
81.5
95.0
98.8
11.7
89.3 (+1.5)
(A = 0.5)
CCE
82.8
60.9
83.0
64.4
97.0
81.1
40%
0.5
GR (8 16)
84.7
83.3
60.3
88.9
94.8
7.5
85.3 (+2)
(A 1)
17

---

Under review as a conference paper at ICLR 2020
F MORE EMPIRICAL RESULTS
F.1 REVIEW
Question:
What training examples should be focused on and how much more should they be em- phasised when training DNNs under label noise?
Proposal: Gradient rescaling incorporates emphasis focus (centre/focal point) and emphasis spread, and serves as explicit regularisation in terms of sample reweighinng/emhhasi.
Finding: When noise rate higher, we can improve model's robustness by moving emphasis focus towards relatively less difficult examples.
F.2 DETAILED RESULTS ON CIFAR-100
The more detailed results on CIFAR-100 are shown in Table 12 which is the supplementary of Table in in the main text.
Table Exploration of GR with different emphasis focuses (centres) and spreads on CIFAR-100 when 20%, 40%. 60%, respectively. This table presents detailed information of optimising X, B mentioned in Table in the paper. Specifically, for each X, we try B values from {2, 4, 6, 8, 10) and select the best as the final result of the X. We report the mean test accuracy Our key finding is demonstrated again: When r raises, we can increase 3, X for better robustness. The increasing scale is much smaller than CIFAR-IO. This is because CIFAR- 1-00 has 100 classes so that its distribution of p; (input-to-label relevance score) is different from CIFAR-10 after softmax normalisation.
5 repetitions.
Noise
0
Testing accuracy (%)
0.1
0.2
63.3
0.3
6
64.1
20%
0.4
6
63.6
0.5
62.6
62.5
0.1
4
55.5
0.2
4
58.2
0.3
6
59.1
40%
0.4
60.0
0.5
8
59.3
0.6
8
58.5
0.1
4
44,9
0.2
4
47.5
0.3
60%
0.4
6
49.9
0.5
8
49,9
0.6
8
47.3
F.3 DETAILED TRAINING DYNAMICS
There are more detailed training dynamics displayed in the Figures 10
18

---

Under review as a conference paper at ICLR 2020
1
0.9
0.9
Train accuracies
0.8
0.8
0.7
0.7
0.6
CCE
fest [Cuur
0.6
GR-30.55-10
0.5
0.5
GR-120.5-X0
GR-31-X0
0.4
GR-02-X0
0.4
GR-011X0
0.3
GR-34-X0
0.3
GR-02-X0
GR-044X0
0.2
GR-30
0.2
GR-20
0.1
0.1
5
10
5
10
Iterations
x104
Iterations
x x10
Figure 4: The training and test accuracies on clean CIFAR-10 along with training iterations. The training labels are clean. We fix À focus on harder examples while changing emphasis spread controller B. The backbone is ResNet-20. The results of ResNet-56 are shown in Figure 5 Better viewed colour.
0.9
0.9
Train acc accuracies
0.8
0 0.8
0.7
0.7
CCE
0.6
GR-10.5-X0
GR-90.5-110
0.5
GR-01-10
G.-9---0-0000
0.4
GR-(22-10
0.4
GR-12-10
0.3
GR-34-10
0.3
GR-04-10
0.2
GR-30
0.2
GR-90
0.1
Iterations
x10
Iterations
x10*
Figure 5: The training and test accuracies on clean CIFAR-10 along with training iterations. The training labels are clean. We fix X = 0 to focus on more difficult examples while changing emphasis spread controller The backbone is ResNer-56. The results ResNet-20 are shown Figure Better viewed in colour.
19

---

Under review as a conference paper at ICLR 2020
W
GR:
Train accuracies
GR.
0.5-1
Test accuracies
0.4
0.4
m
0.2
0.2
2
6
8
10
0
2
4
6
8 10
Iterations
x10*
Iterations
x10*
Figure 6: The learning dynamics CIFAR-10 80%) with ResNet-56, i.e., training and testing accuracies along with training iterations. The legend in the top left shared by two subfigures. 'xxx: yyy" means 'method: emphasis focus'. The results of 20%, 40%, 60% are shown in Figure 2 in the paper.
We have two key observations: 1) When noise rate increases, better generalisation is obtained with higher emphasis focus, i.e., focusing on relatively easier examples; 2) Both overfitting and underfit- ting lead bad generalisation. For example, 'CCE: O' fits training data much better than the others while 'GR: None" generally fits it unstably or lot worse. Better viewed in colour.
Train accuraces
0.0
0.0
0.8
0.6
0.6
0.6
0.6
0.4
0.1
10
Itorations
-10
"10
x204
GR--8-11 10
Tee accuaries
0.8
000
0.2
GR-112-03
0.
GBB-16-A1
9.3
10
0.1
Iterations
100
"10
Figure 7: ResNet-56 on CIFAR-10 (r 20%). From left to right, the results four emphasis
focuses 0, 0~0.5, 0.5, 0.5~1 with different emphasis spreads are displayed in each column respec- tively. When À larger, should be larger as displayed in Figure Icin the paper. Specifically
1) when X = 0): we tried B = 0.5, 1, 2, 4;
2) when 0.5: we tried 3 4, 8, 12, 16;
3) when x 1: we tried 8 8, 12, 16, 20;
4) when = 2: we tried 3 16, 20, 24.
20

---

Under review as a conference paper at ICLR 2020
0.9
0.9
0.9
Trein accuraces
0.7
6.7
0.7
0.7
0.3
0.3
0.3
0.1
0.3
lereltinns
-10*
-10
x30
Toor accuractes
0.8
GR
0.8
CCE
GR.
0.8
GRo-8-1 S0N.
12G-43
U3D-L CR-22012
0.6
0.6
0.6
0.4
G2-1:2-193
0.1
-- GR.00
0.2
6.1
0.3
10
Iteretions
x10*
-20
Figure 8: ResNet-56 on CIFAR-10 (r = 40%). From left to right, the results of four emphasis
focuses 0, 0~0.5, 0.5, 0.5~1 with different emphasis spreads are displayed in each column respec- tively. When À is larger, 3 should be larger as displayed in Figure Icin the paper. Specifically
1) when = 0:: we tried B = 0.5, 1, 1,2,45 4;
2) when À 0.5:
tried 8 8, 12, 16;
3) when À 1: we tried 8 8, 12, 16, 20;
when À 2: tried B 12, 20, 24.
0.9
0.9
0.9
Train accuracies
0.7
0.3
0.1
0.
10
Iteretions
-10'
>10
20
100
CCE
0.9
-011
0.9
soueght 24-10
0.7
0.7
0.
0.3
0.3
0.3
0.0
10
6.1
10
Rerations
x301
30
Figure 9: ResNet-56 on CIFAR-10
60%). From left to right, the results four emphasis focuses 0, 0~0.5, 0.5, 0.5~1 with different emphasis spreads are displayed in each column respec- tively. When À larger, should be larger as displayed Figure Teln the paper. Specifically
1) when X = 0): we tried B = 0.5, 1, 4;
2) when = 0.5: we tried 3 = 4,8, 12, 16;
3) when À I: we tried 8 12, 16, 20;
4) when = 2: we tried 8 = 16, 20, 24.
21

---

Under review as a conference paper at ICLR 2020
0.6
0.0
Train accuratie
0.5
0.4
0.3
0.3
0.2
0.2
0.1
0.
0
0
>10
<10
10
x10
10
Teet accuaciss
CCE
GR .a
GR--0-10
0.5
0.
06
CD 11--1 -
.22-2 CB-00-12
0.4
0.4
04
0.4
0.3
0.5
0.3
0.2
0.2
02
02
0.1
0.3
0
5
10
-10
>10*
10
+10
Figure 10:
ResNet-56 on CIFAR-10 (r- 80%). From left to right, the results of four emphasis focuses 0, 0~0.5, 0.5, 0.5~1 I with different emphasis spreads displayed in each column respec- tively. When À is larger, 3 should be larger as displayed in Figure in the paper. Specifically
1) when 0: we tried
0.5, 1, 2, 4:
2) when À = 0.5: we tried 8 = 4,8,15 12, 16:
3) when À 1: we tried 8, 12, 16, 20;
4) when A = 2: we tried B = 12, 16, 20, 24.
22


Reviews:
[{"review_id": "rJgz97JRYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes a method for noise robustness based on scaling gradients of examples. By choosing the proper scaling parameters (alpha and beta), the method recovers standard losses such as CCE, MAE, and GCE, while also recovering other losses. The method is strongly related to reweighting training examples, where alpha and beta define the shape of this weighting as a function of the model's prediction (i.e., p_i). Experiments show that the proposed method achieves competitive results on several standard benchmarks for noisy-labelled data.\n\nComments:\n- The main drawback of the proposed approach is that there is no clear way of choosing alpha and beta, other than a hyper-parameter search, which is not very practical and can lead to overfitting the test set.\n- The paper in general is easy to follow, but the paper is not very rigorous or clear on some important concepts. For example: \n     * No clear mathematical definition of emphasis focus and spread\n     * The term \"semantically abnormal examples\" should be defined in the main text.\n     * It is not so clear what it means to \"babyset\" emphasis focus and spread.\n     * I don't understand what Eq. 6 is supposed to tell.\n     * What are the \\dots in equation \n- The experiments are very thorough and the results are very good, but I have few clarifying questions:\n     * The procedure for choosing beta/gamma is not clear, and I see that for every experiment those values change.\n     * It would be nice if the CIFAR-10 Table 4 results are performed using the exact same setting to prior work to make sure the comparison is fair. For example, GCE results in (Zhang & Sabuncu 2018) are much that the reported ones. While it seems that you're using GoogLeNet V1 architecture similar to Jiang et al. 2018, it's not clear which experimental setting you are comparing against.\n     * Can you be more specific what do you mean by \"with a little effort for optimizing beta and gamma\" in caption of Table 5?\n\nMinor:\n     * Grammer mistake: \"what training examples...focused *on*...\"\n     * Citations should be done with parentheses\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xJEl_Z9r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThis paper presents Gradient Rescaling (GR) for robust learning to combat label noise. They propose to treat each data sample with different significance scores: some samples are important to learning, and some examples are insignificant (or even detrimental) to learning. So they desire to weight each samples according to their significance. They propose the notion of emphasis focus (When learning, whether we should put emphasis on learning “hard” examples or “easy” examples) and emphasis spread (the variance of these significance weights). The authors propose that this “difficulty” of samples are proportional to their network output logit values.\nThe authors examine the analytical forms of the gradients of popular loss functions such as Categorical Cross Entropy, Mean Absolute Error and Generalized Cross Entropy. They find that the formulas for the gradient are of similar family with varying hyperparameters. Authors claim that tweaking these hyperparameters result in tuning the emphasis focus and spread.\nThe authors conduct Experiments on CIFAR10, CIFAR100 with simulated symmetric noise. Also, they conduct experiments on real-world noisy datasets: Clothing 1M dataset and MARS video dataset. The authors claim that the performance of GR exceeds various baselines.\n\n\nSignificance/Novelty/Clarity\nSignificance: Low-Medium. The performance increase exhibited in the experiments are a bit underwhelming (when considering the fact that benchmarks of most recent noise-robustness algos such as <Lee et al. 2019 ICML> are missing).\nNovelty: Medium. The paper is interesting in the sense that the authors integrated (and allegedly generalized) the gradient formulas for several losses into one family, and tried to integrate and tweak their postulation of  “Emphasis focus” and “emphasis spread” into the framework. However, the theoretical ground and convincing reasoning for their claim seems a bit lackluster.\nClarity: Low. The overall flow of the paper is a bit fuzzy - exhibiting a stream-of-consciousness style flow.\n\n\nPros and Cons in Detail\nPros:\n1.The authors try to unify the analytical forms of the gradients of various loss functions into a single family equipped with hyperparameters that control emphasis focus and spread.\n2.Conducted experiments show that GR achieved increased performance when compared to the baselines.\nCons:\nMy major concern is about tuning newly introduced hyperparameters in practical settings. How can we guarantee to have intact validation set?  Can we get any improvement via GR even with corrupted validation set for tuning hyperparameters? \n1. The arguments of the authors are grounded in the premise that “difficult” samples will exhibit small logit values, and “easy” samples high logit values.\n2. No justifications (both theoretical and experimental) are provided on the claim that controlling emphasis focus/spread will result in more robust learning. \n3. This algorithm introduces 2 additional hyperparameters that are correlated with each other. This introduces additional labor.\n4. By changing the loss function, the outputs of the network might lose its interpretation as a probability distribution.\n5. No confidence intervals are shown except for the CIFAR-100 experiment.\n6. Experiments are only conducted on vision tasks.\n7. The baseline menagerie also changes when the authors change the target dataset.\n8. Additional benchmarks of most recent noise-robustness algos such as <Lee et al. 2019 ICML> are required.\n\n\nQuestions\n1. Is it always the case that “difficult” samples exhibit small logit values, and “easy” samples high logit values?\n2. If not, GR’s emphasis manipulation might result in neglecting samples containing valuable information.\n3. Can GR be used simultaneously with other noise-robust learning methods to further boost the performance?\n4. Technically, GR aims to rescale the gradients of the logits. How will it interact with optimizers  other than SGD such as Adam?\n5. Does GR still work well on small datasets(#points < 5000)?\n\n\nMisc. Comments\nPage 3-> inside L1 norm, no differentiation sign in the denominator.\nAround eq 2 and 4: missing derivative symbol w.r.t. z", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xnoAwm9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors first analyze and answer the question: What training examples should be focused and how large the emphasis spread should be? Then, they proposed the gradient rescaling framework serving as emphasis regularization. \n\nStrengths:\n1. The paper is well organized except the reference citation (read difficultly)\n2. The proposed method is very simple and effective. \n3. Experiments show the improvements over SOTA. \n\nWeakness:\n1. The experiments lack the recent important baseline \"symmetric cross entropy for robust learning with noisy labels, ICCV2019\", which are the current SOTA. Maybe the author should check the above paper and show the results.  \n2. The experiments are only conducted on symmetric noise. Actually, asymmetric noise is also important. The author should conduct at least some experiments on asymmetric noise. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a gradient rescaling method to make deep neural network training more robust to label noise. The intuition of focusing more on easier examples is not particularly new, but empirical results are promising. On the weak side, no theoretical justification is provided, and the method introduces extra hyperparameters that need to be tuned. Finally, more discussions on recent SOTA methods (e.g., Lee et al. 2019) as well as further comprehensive evaluations on various cases, such as asymmetric label noise, semantic label noise, and open-set label noise, would be needed to justify and demonstrate the effectiveness of the proposed method. 
Decision: Reject