Searching to Exploit Memorization Effect in Learning from Corrupted Labels
Authors: Hansi Yang
Quanming Yao
Bo Han
Gang Niu
Venue:  (2020)
Keywords: Noisy Label
Deep Learning
Automated Machine Learning

Abstract: Sample-selection approaches, which attempt to pick up clean instances from the training data set, have become one promising direction to robust learning from corrupted labels. These methods all build on the memorization effect, which means deep networks learn easy patterns first and then gradually over-fit the training data set. In this paper, we show how to properly select instances so that the training process can benefit the most from the memorization effect is a hard problem. Specifically, memorization can heavily depend on many factors, e.g., data set and network architecture. Nonetheless, there still exists general patterns of how memorization can occur. These facts motivate us to exploit memorization by automated machine learning (AutoML) techniques. First, we designed an expressive but compact search space based on observed general patterns. Then, we propose to use the natural gradient-based search algorithm to efficiently search through space. Finally, extensive experiments on both synthetic data sets and benchmark data sets demonstrate that the proposed method can not only be much efficient than existing AutoML algorithms but can also achieve much better performance than the state-of-the-art approaches for learning from corrupted labels.

Under review as a conference paper at ICLR 2020
SEARCHING TO EXPLOIT MEMORIZATION EFFECT IN LEARNING FROM NOISY LABELS
Anonymous authors Paper under double-blind review
ABSTRACT
Sample-selection approaches, which attempt to pick up clean instances from the noisy training data set, have become one promising direction to robust learning from noisy labels. These methods all build on the memorization effect, which means deep networks learn easy patterns first and then gradually over-fit the train- ing data set. In this paper, show how to properly select instances that the training process can benefit the most from the memorization effect is a hard prob- lem. Specifically, memorization can heavily depend many factors, e.g., data set and network architecture. Nonetheless, there still exists general patterns of how memorization can occur. These facts motivate to exploit memorization by au- tomated machine learning (AutoML) techniques. First, we design an expressive but compact search space based on observed general patterns. Then, we propose to use the natural gradient-base based search algorithm to efficiently search through space, Finally, extensive experiments on both synthetic data sets and benchmark data sets demonstrate that the proposed method can not only be much efficient than existing AutoML. algorithms but can also achieve much better performance than the state-of-the-arr approaches for learning from noisy labels.
INTRODUCTION
Learning with deep neural networks has enjoyed huge empirical success in recent years across a wide variety of tasks, from image processing to speech recognition, and from language modeling to recommender system (Goodfellow et al., 2016). However, their success highly counts on the availability of well-annotated and big data, which barely available for real-world applications. Instead, what we are facing with in practice are large data sets which are collected from crowd- sourcing platforms or crawled from the Internet, thus containing many noisy labels (Li et al., 2017b; Patrini et al., 2017). Besides, due to the vast learning capacity of deep networks, they will eventually over-fit fit on these noisy labels, leading to poor predicting performance, which can be worse than that obtained from simple models (Zhang et al., 2016; Arpit et al., 2017).
To reduce negative effects from noisy labels, many methods have been proposed (Sukhbaatar et al., 2015; Reed al., 2015; Patrini et al., 2017: Ghosh al., 2017: Malach & Shalev-Shwartz, 2017) Recently, a promising direction is training networks only on selected instances that are more likely be clean (Jiang et al., 2018; Han et al., 2018b; Ma al., 2018: Yu al., 2019; Wang al., 2019). Intuitively, as the training data becomes less noisy, better performance can be obtained. Among those works, the representative methods are MentorNet (Jiang et al., 2018) and Co-teaching (Han et al., 2018b; Yu et al., 2019), they take small-loss samples in each mini-bated as clean instances. Specifically. MentorNet pre-trains an extra network, and then uses the extra network for selecting clean instances to guide the training. When the clean validation data is not available, MentorNet has to use a predefined curriculum (Bengio et al., 2009). Co-teaching is an improvement over MentorNet, it simultancously maintains two networks which have identical architectures during the training process. And in each mini-batch of data, each network is updated using the other network's 's small-loss instances.
To the success of these sample-selece et al., 2016; Arpit et al., 2017) is the crux. Memorization happens widely in various architectures of deep network, e.g., multilayer perceptron (MLP) and convolutional neural network (CNN). Specif- ically, it means that deep networks tend to learn easy and correct patterns first and then over-fit on
methods, the memorization effect of deep networks (Zhang

---

Under review as a conference paper at ICLR 2020
(possibly noisy) training data set (see Fig.1(a)-(b)) Thus, when learning with noisy labels, while the validation loss will first increase and then significantly decrease, the training loss will continuously get smaller with more training epochs.
Due to such effect, samplc-selection methods can learn correct patterns at early stage and then use the obtained discriminative ability to filter out corrupted instances in subsequent training epochs (Jiang et al., 2018; Han et al., 201 8b; Chen et al., 2019). While the memorization effect is critical the success of sample-selection methods, however, how to properly exploit it is not addressed in the literature. And trivial attempts can easily lead to even worse performance than standard deep networks (Han et al., 2018b). Some recent endeavors seek to evade from this problems by integrating with other auxiliary information, e.g., small clean subset is used in (Ren et al., 2018), and knowledge graphs are utilized (Li al., 2017b).
In this paper, motivated by the success of automated machine learning (AutoML) on designing data-depnndent models (Hutter al., 2018), and the fact that memorization heavily depends on many factors (Zhang et al., 2016; Arpit et al., 2017), we propose to exploit memorization effects automatically using AutoML techniques. Contributions are summarized as follows:
First, to have an in-depth understanding of why it is difficult to tune sample-selectinn selection methods with good performance. We examine behaviors of memorization effect from multiple perspec- tives. We find that, while there exist general patterns in how memorization occurs with the train- ing process (see Fig.)())(b)), it is hard to quantize to which extend such effect can happen (see Fig. ((b) Especially, memorization can be affected by many factors, e.g., data sets, network architectures, and the choice of the optimizers. It is exactly such complex dependency make the design of proper sample-selection rules a hard problem, which motivates us to solve the problem by AutoML techniques.
To make good use of AutoML techniques, then derive an expressive search space for exploiting memorization, which is from the above observations, i.e., the curvature of how many instances need to be sampled during iterating should be similar with the inverse of the learning curve on the validation set. Such space is not too huge since it has only few variables, thus allows subsequent algorithms converging fast to promising candidates.
Then, to design an efficient algorithm, we show the failure of gradient- methods and the inefficiency of derivative-free methods. These motivate to take a probabilistic view of the search problem and adopt natural gradient descent (Amari, 1998; Pascanu & Bengio, 2013) for optimization. The designed algorithm can effectively address above problems and is significantly faster than other popular search algorithms.
Finally, we conduct extensive experiments on both synthetic. benchmark, and real data sets, under various settings using different network architectures. These experiments demonstrate that the proposed method can not only be much more efficient than existing AutoMI. algorithms, but also can achieve much better performance than the state-of-the-ar sample-selection approaches designed by humans. Besides, we further visualize and explain the searched functions, which can also help design better rules to control memorization effects in the future.
2 RELATED WORK
2.1 LEARNING FROM NOISY LABELS
The mainstream research focuses on class-conditional noise (CCN) (Angluin & Laird, 1988), where the label corruption independent of features. Generally, recent methods for handling CCN model can be classified into three categories. The first one is based on the estimation of transition matrix, which tries to capture how correct labels flip into wrong ones (Sukhbaatar et al., 2015; Reed al., 2015; Patrini et al., 2017; Ghosh et 2017). These methods then use the estimated matrix to correct gradients or loss during training. However, they are fragile to heavy noise and unable to handle many classes (Han et al., 2018b). The second type is the regularization approach (Miyato et al., 2016; Laine & Aila, 2017: Tarvainen & Valpola, 2017). Although regularization approach can achieve a satisfying performance, it is still incomplete approach since (Jiang et al., 2018) shows that it can only delay the overfitting progress rather than avoid it, i.c. given enough training time, it can still fit the noisy data completely. Thus, it requires much domain knowledge to determine the appropriate
2

---

Under review as a conference paper at ICLR 2020
number of training epochs in order to prevent overfitting. The last one is sample-selection approach, which attempts to reduce negative effects from noisy labels by selecting clean instances during training. The recent state-of-the-an method is also built on sample-selection approach (Jiang al., 2018; Han et al., 201 8b; Malach & Shalev-Shwartz, 2017; Yu et al., 2019).
Active learning (Settles, 1994) is closely related method, which iteratively selects unlabeled sam- ples with high high-confident predictions into the training data set. Thus, to do active learning, we need obtain classifier of which the performance is good enough. As result, active learning is not applicable for directly learning from noisy labels here.
A promising criteria select "clean instances" is pick up instances that has relatively small losses in each mini-batch (Jiang et al., 2018; Han et al., 2018b). The fundamental property behind these methods is the memorization effect of deep networks (Zhang et al., 2016; Arpit al., 2017), which means deep networks can learn simple patterns first and then start to over-fit. Such effect helps classifiers set up discriminate ability in the early stage, then make clean instances more likely to have smaller loss that those noisy ones. The general framework of sample-selection approach is in Alg.I. Specifically, some small-loss instances f are selected from the mini-batch D in step 5. These "clean" instances are then used to update network parameters in step 6. The R(()) in step 8, which controls how many instances to be kept in each epoch, is the most important hyper-pane parameter as it explicitly exploits the memorization effect (Han et al., 2018b; Jiang et al., 2018; Yu et al., 2019).
Algorithm Framework of the sample-selection approach (Jiang et al., 2018; Han et al., 2018b).
1: for = 1,... .'' do
2: shuffle training set DD
3: for 1,... N do
4: draw a mini-batch D from D;
5: select D,, i.e., R(()) small-loss instances from D based network's predictions;
6: update the network's parameter using gradient from Dy;
7: end for
8: exploit memorization effects using R(t) (estimation on percentage of clean instances);
9: end for
However, it is hard to exactly determine how much proportion of small-loss samples should be selected in each epoch (Jiang et al., 2018; Ren et al., 2018). As will be discussed in Sec.3. I, due to various practical usages issues, to which extend memorization effect can happen is hard to quantize. Thus, performance obtained from existing solutions are far from desired, and we are motivated to solve this issue by AutoML.
2.2 AUTOMATED MACHINE LEARNING (AUTOML)
Automated machine learning (AutoML) (Hutter et al., 2018) has recently exhibited its power in eas- ing the usage of and designing better machine learning models. Basically, AutoML can be regarded as a black-box optimization problem where we need to efficiently and effectively search for hyper- parameters or designs for the underlying learning models evaluated by the validation set. Regarding the success of AutoMI, there are two important perspectives (Feurer et al., 2015; Zoph & Le, 2017; Xic & Yuille, 2017; Bender et al., 2018):
Search space: First, it needs be general enough, which means it should cover existing models as special cases. This also helps experts better understand limitations of existing models and thus facilitate future researches. However, the space cannot be too general, otherwise searching in such a space will be too expensive.
Search algorithm: Optimization problems in AutoML are usually black-box. Unlike convex op- timization, there is universal and efficient optimization tools. Once the search space is deter- mined, domain knowledge should also be explored in the design of search algorithm so that good candidates in the space can be identified efficiently.
Search space is domain-specine and needs to specially designed for every AutoML problem. There are two types of search algorithms popularly used. The first one is derivative-fie optimization methods, it is usually used for searching in general search space, e.g., reinforcement learning
3

---

Under review as a conference paper at ICLR 2020
(Zoph & Le. 2017: Baker et al., 2017), genetic programming (Escalante et al., 2009; Xie & Yuille, 2017), and Bayes optimization (Feurer et al., 2015; Snock et al., 2012). More recently, gradient- based methods, which alternatively update parameters and hyper-parametent have been developed as more efficient replacements for derivative-free optimization methods on some AutoMI. problems, e.g., neural network architecture search (Liu al., 2019; Akimoto al., 2019; Xie et al., 2018).
However, existing AutoML techniques cannot be directly used in exploiting memorization here. First, we need to carefully define domain-specific space. Besides, we will also show existing algorithms are neither not applicable nor too slow. This motivates us to propose a new algorithm based on natural gradient.
THE PROPOSED METHOD
Here, we first give closer look at why it is difficult to exploit the memorization effect (Sec.3.1) I.). This also helps us identify key observations on how memorization
happen. This observation subsequently enables us to design an expressive but compact search space (Sec.3.2), and motivates us natural gradient method (Amari, 1998; Ollivier et al., 2017) that generate gradients in the parameterized space for efficient optimization (Sec.3.3).
3.1 DIFFICULTIES IN EXPLOITING MEMORIZATION
As in Sec.2.2, an important challenge in designing search spaces is to balance the size (or dimension) and the expressive ability of the search space. An overly constrained search space may not contain candidates that have a satisfying performance, whereas too large search spaces will be difficult to effectively search. All these require us have in-depth understanding of the difficult in designing of R(T), which is at step of Alg. and is used to exploit the memorization effect. Thus, we are motivated to look factors which can affect the memorization of deep networks, and to seek patterns from the resultant influences. Specifically, we examine memorization when data sets, architectures, or optimizers are changed. Results are in Fig.1. From these figures, we can observe:
Foot-stone of space design: There exists general pattern among all cases, i.e., all models' test accuracy will first increase, then decrease.
3TS (jE MATIS 50%
XIS
04
(irsk - cymmertr 200
HA87 595 HEVARTC
04-
02-
01-
(a) training accuracy.
(b) testing accuracy.
(c) different architectures.
C.205
sur
31
Eacch
Eaech
Excch
(d) different optimizers.
(e) different batch size (Adam). (f) different learning rate (Adam).
Figure 1: Memorization effects are shown in Fig.(a) and (b), where training accuracy continuously increases while testing accuracy first increases and then significantly decreases due to noisy labels. Fc-.(c-f) show memorization not only heavily depends on data sets, and also many factors (details of experiment are in Appendix A.1).
This general pattern is consistent with that in the literature (Zhang et al., 2016; Arpit et al., 2017; Tanaka et al., 2018; Han et al., 2018b). It is also the key domain knowledge for designing an
4

---

Under review as a conference paper at ICLR 2020
expressive and compact search space, which makes the subsequent search possible. However, the more important observation is:
Need of AutoML: Curvature can be significantly affected by these factors. When the peak will appear (i.e., stop learning from simple patterns and start to over-fit), and to which extend the performance will drop from peak (i.e., over-fit on noisy labels) are hard quantize.
This observation shows great variety in appearances of the memorization effect, which further poses significant need for automated exploiting of such effect. Besides, it is hard to know in advance what learning curve will exactly look like in advance and thus impossible to manually design before learning is performed.
3.2 SEARCH SPACE
Recall that in step 8 of Alg. It(t) controls how many instances are kept in each mini-batch, and we want to exploit the memorization effect through the variation of R(t). Based on the first empirical observation Sec.3.1, our design of R(t) first should satisfy:
Curvature: R(t) should be similar with the inverse of the learning curve. other words, R(t) should first drop, then (possibly) rise.
The reason behind this constraint is as follows: Since the learning curve represents the model's accuracy, when it rises, should drop large-loss samples the large loss is more likely the result of noisy labels than model's misclassification. And when the learning curve falls, we should drop less to help the model learn more.
Range: R(t) E [0, 1] for t {L..... with R(L) = 1.
Since R(t) denotes the proportion of selected instances, it is naturally in [0, 1]. Besides, at the beginning, dropping small-loss samples will be same as dropping samples randomly, and we need to pass sufficient number of instances such that the model can learn from patterns.
10
Target
functions
definition
decreasing
power (pow-d)
1/(1+0))
log
exponential (exp-d)
e 62"
06-
logarithmic (log-d)
loR(b//lg(att0)
increasing
power (pow-i)
b(t/z)²
04
logarithmic (log-i)
1og(1+2""
[og[1+79]
25
50
75
100
125
150
175
200
Figure 2:
Construction elements for fi (cf. left table) and some approximated curves as examples (cf. right figure). Target is an example of the ground-truth Ri()) to approximate.
As R(t) itself can be seen as function which takes t as input and outputs scalar, we can use some simple fü as basis functions approximate the complicate R(t). We list some simple monotonously decreasing or increasing functions in Fig.2 2 and express R(t) as a linear combination of them:
R(I)= = Σx
1= [1(4,))) s.t. a aEA.
where the or is the Syper--arameterss controlling each term (for example and b in Fig.2).
Let the clean validation set be Goat, F be Alg.1 with network parameter ur and sampling rule a (R(t) parameterized by (q). M(w, Goal) measures the validation performance with the model parameter w. Thus, memorization effect can be automatically exploited by solving the following problem
a = arg max M (a), gral), s.t. w" (a) arg min F(w: a))
(2)
a
Then, the optimal curvature is derived from R(I) (()==>=(i fi(1;07).
5

---

Under review as a conference paper at ICLR 2020
Remark 3.1. In Co-teaching algorithm (Han al., 2018b), R(t) is determined follows R(t) T- min ((/)2)),1);
(3)
which contains three hoper-parameters: T, C. and lk- Thus, we can formulate a search space as: 1),c E (0,+0) and la {1,... ,T). The expressivity of such R(t) is not enough as (q). here (since Eqa.(3) 3) is covered as special case).
3.3 SEARCH ALGORITHM
Here, we first discuss problems of using gradient-based methods here (Sec.3.3.1). These problems motivate us design an efficient search algorithm based natural gradient algorithm (Sec.3.3.2).
3.3.1 ISSUES OF EXISTING ALGORITHMM
Gradient-based methods need the chain rule to obtain the gradient w.r.t hyper-parameter from the gradient w.r.t network weights, i.e., V Va B.M/ow- Ow/aa. This condition does not hold for our problem since our hyper-parameterss control how many samples will be used to update the weights. Thus, is hard compute V here. Besides, the learned parameters w cannot be shared among different a. In previous methods, hyper-parameters are not coupled with the training process, e.g,, network architectures (Liu al., 2019; Akimoto et al., 2019) and regularization (Luketina et al., 2016). However, R(t) here will heavily influence the training process, which is also shown in our 1ig.1. Thus, gradient- based methods cannot be applied here.
3.3.2 PROPOSED ALGORITHM
The analysis above demonstrates that only derivative-free methods are applicable our problem, which can be slow. Here, we discover that natural gradient (NG) (Amari, 1998; Pascanu & Bengio, 2013; Ollivier al., 2017), can be used. The most interesting point here is that NG can still benefit from gradient descent, but without the computation of aM/aa or sharing of w.
The basic idea of NG is summarized as follows: instead of directly optimizing w.r.t a, we consider a random distribution po (a) over a parametrized by 0, and maximize the expected value of our validation performance M w.r. 6, i.e.,
max (()= = . (w" (ax,, (vat) po(a) da, s.t. w= = arg min F(w; or). 0
(4)
aEA
To optimize a(0) w.r.t 0, NG updates by
mm+ = TT + pH (
(5)
where p is the step-size, H is the Fisher information matrix at 0m and
VoJ / Mw (w" (ax), Grat) Vopo(a))de )a -Epo(a) [M (w" (a), Geat) Vo log po (o:)). JAEA
(6)
By sampling candidate R(t)s from the given distribution Pam in each iteration, (6) just needs validation's performance (without computation @M/aa)). Besides, such gradient has proved to give the steepest ascend direction in the probabilistic space (Theorem in (Angluin & Laird, 1988)).
The last question is how to choose Pa. which may significantly influence the algorithm's con- vergence behavior. Fortunately, NG exhibits strong robustness against different choices The reason is that Fisher matrix, which encodes the second order approximation to the Kullback- Leibler divergence, can accurately capture curvatures introduced by various pg. Thus, NG descent is aarameterization-invaaant has good generalization ability, and moreover can be regarded as second-order method in the space of (Ollivier et al.. 2017). The proposed search algorithm is in Alg.2 (details are in Appendix A.2). As will be shown in experiments, all these properties make NG an ideal search algorithm here, and can be faster than other popular AutoML approaches.
pa-
EXPERIMENTS
We implement all experiments using PyTorch 0.4.1 GTX 1080 Ti GPU.

---

Under review as a conference paper at ICLR 2020
Algorithm 2 Proposed search algorithm (based on natural gradient).
while not converged do
2: for n = 1, .... NN do
3: draw an a from the current distribution po( || approximate gradients
4: run Alg.1 using R(t) described by Eqa.(1): // no weight-sharing
5: end for
6: use samples in step 2-5 to compute Fisher information matrix;
7: update by (5). and (6);
8:
end while
4.1 EXPERIMENTS ON SYNTHETIC DATA
In this section, we demonstrate the superiority of the proposed search space and search algorithm on the synthetic data. The ground-truth R(t) is shown in Fig.3(a), which is an example curvature satisfying two requirements in Sec.3.2.
4.1.1 SEARCH SPACE COMPARISON
The goal here is to approximate R(t) in Fig.3(a). Let the estimated function be R(t) where t Three different search spaces compared: 1). Full space: (1) (el, e2 i.e., there is no constraint in the space, and estimation for R(t) at each time stamp is performed independently; 2). Co-teaching's space in (3,. i.e., (t)
1, 200. The target is minimize RMSE, i.e. f(R) [1200 2020 (R=)( R(t)){)/
T min(((/)))" 1), and {T, c,t ) needs to be estimated; and 3). The proposed space in (1), which encodes our observations in Scc.3.1. Random search (Bergstra & Bengio, 2012) performed in all three spaces.
Synthekk
Syntheb
072
kg
Toyes
0.00
032
Swerthed kMN
02
181
255
100
175
8 06-
(a) Grount-truth R(().
34-
02
(b) Search space.
(c) Scarch algorithm.
Figure 3: Comparison of the search space (ef. the middle figure) and search algorithm (cf. the right figure) the synthetic data.
Results are in Fig.3(b). We can that constraints on the search space will lead to disastrous per- formance and extremely slow convergence rate. And compare our space with previous Co-teaching's 's space, we that our space can approximate the target better due its larger degree of freedom.
4.1.2 SEARCH ALGORITIIM COMPARISON
We first test our proposed method's 's efficiency over other cepperpparametr optimization (HPO) algo- rithms on a synthetic problem as follows: Given a pre-defined R(t) as target, we try to search for R(t) that has the lowest squared loss to the target. We compare proposed method with random search (Bergstra & Bengio, 2012) and Bayesian optimization (Kandasamy et al., 2019), which are two popular methods for hyper-parameter optimization. Results are in Fig.3(c). The results demon- strate that our proposed natural gradient approach has the fastest convergence rate. And it can find the best R(() with the least RMSE to the target.
7

---

Under review as a conference paper at ICLR 2020
4.2 EXPERIMENTS ON BENCHMARK DATA SETS
We verify the efficiency of our approach on three benchmark data sets, i.e., MNIST, CIFAR- 10 and CIFAR- 100 (details are in Appendix A.3). These data sets are popularly used for the evaluation of learning with noisy labels in the literature (Zhang et al., 2016; Arpit al., 2017: Jiang et al., 2018; Han et al., 2018b). Following (Patrini et al., 2017; Han al., 2018b; Chen al., 201 1019), we corrupt these data sets manually by two types of noise. (I) Symmetry flipping (with 20% and 50% noise level) and (2) Pair flipping (with 45% noise level).
All these noise patterns correspond to real-world scenarios. For example, on the macro-level, class cat flipping to the class dog makes sense, while class dog flipping to class cat also makes sense. Such (Patrini et al., 2017). On the micro-level, for
flipping yields a noise pattern called symmetric- dogs, class Norfolk terrier flipping to class Norwich terrier makes sense, while class Norfolk terrier flipping class Australian terrier not. This flipping yields noise pattern called pair-flip (Han al., 201 8a), which depicts the fine-grained classification case.
We the network architecture the same in (Yu al., 2019) (Appendix A.4). To measure the performance, same as (Patrini et al., 2017; Han et al., 2018b; Chen et al., 2019), we use the test accuracy, i.e., lest accuracy (#of correct predictions) Hoof lest datasel). Specifically, the full learning curve, i.e., testing accuracy v.s. epochs, is reported. Ideally, if method is robust to noisy labels, its performance will increase with more training epochs. Thus, if the learning curve of one method quick falls down after reaching the maximum, then it means the method is not robust intrinsically (Zhang et al., 2016; Arpit al., 2017).
Standard
Decoupling
F-correction
MentorNet
Co-teaching
Co Co-teaching+
Proposed
mnh
120
Symmetric 20%.
(b) Symmetric 50%.
Pair 45%.
Figure 4: Comparison testing accuracy between the proposed and other human-designed meth- ods. Top to bottom: MNIST, CIFAR-10 10 and CIFAR-100. Best testing accuracy of each method is reported in Appendix B.
4.2.1 COMPARISON ON LEARNING PERFORMANCE
To show better accuracy be achieved by automatically exploiting the memorization effect, compared the proposed method with 1). MentorNer (Jiang et al., 2018); 2). Co-teaching (Han al., 20180 10);;); 3). Co-teaching+ (Yu al., 2019) (enhancing Co-teaching by disagreements
8

---

Under review as a conference paper at ICLR 2020
predictions); 4). Decoupling (Malach & Shalev-Shwartz, 2017); 5). F-correction (Patrini et al., 2017); 6). As a simple baseline, we also compare with the standard deep network that directly trains on noisy datasets (abbreviated as Standard). As example usage, the proposed method is combined with Co-teaching, i.e., Co-teaching is run with search R(t). Fig.4 shows the comparison with various human-designed methods. We can that the proposed method significantly outperforms existing methods by a large margin, especially on the more noisy cases (i.e., symmetric-500 and pair-45%).
Besides, the proposed method not only beats Co-teaching due better exploiting of the memoriza- tion effect, but also wins Co-teaching+, which further filter small-loss instances in Co-teaching by checking disagreements on predictions of labels. These demonstrates the importance and benefits of searching proper R(t).
4.2.2 CASE STUDY ON SAMPLING RATE
To understand why the proposed method can obtain much higher testing accuracy, we plot the searched R(t) in Fig.5. We can see that all methods finally drop more large-loss instances than actual noise level. The reason is very intuitive, a large-loss instance usually also has larger gradient, and it can have much more influences than several clean small-loss instances if its label is wrong. Thus, we may want to drop more samples eventually. However, this is not an easy task, as in Tab.8 of (Han et al., 2018b), simply making larger sometimes leads decrease in testing accuracy; and in the third row of Fig.5, we may also need to keep more samples during the training. A more evolved examination on precision of clean labels are in A.p.
Standard
Co- Co-teaching/ Co-teaching+
Proposed
0.5
03
0.8
04-
0.2
04
300
12
150
115
200
50
130
153
375
200
25
50
75
130
15)
175
20
CIAA-30 Symmetnc 20%
0.2
32-
0.2
300
150
100
25
300
150
200
25
300
300
CPAt 200, Symmosis
CIAA1-106, Symmetre
CITAB-206, Partto 45%
0.-
0166-000
3.
3.-
05
300
Earch
(a) Symmetric 20%.
(b) Symmetric 50%.
(c) Pair 45%.
Figure 5: Comparison on R(t) between the proposed (automatically searched from the data) and Co-teaching (note that Co-teaching+ uses the same R(t) as Co-teaching). Top bottom: MNIST, CIFAR- 10 and CIFAR-100.
4.2.3 COMPARISON WITH HPO METHODS
Finally, in this section, compare the proposed natural gradient (NG) algorithm with 1). random search (Bergstra & Bengio, 2012) and 2). Hyperband (Li et al., 2017a). Note that Bayesian op- timization is slower than Hyperband, thus not compared (Hyperband cannot be used in Sec. 1.2

---

Under review as a conference paper at ICLR 2020
due to no inner loops). Besides, reinforcement learning (RL) (Zoph & Le, 2017) is not compared as the searching problem is not a multi-step one. Genetic programming (Xie Yuille, 2017) is not considered neither, the search space is a continuous one. Fig.6 compares the proposed method with random search and Hyperband. From the figure, we can see that natural gradient converges faster than other HPO methods in this problem. Our proposed method can also find better R(t)s under different data sets and noise settings.
CINN-1 Symerete: 30%
CFAR 10 Symmerrc C50
CIRAR-16, Parto 47%
0.38
0313
0550
036
Sum3u Prppardd
Randen
Rander
0400
popros
Pooces
(a) Symmetric 20%.
(b) Symmetric 50%.
(c) Pair 45%.
Figure 6: Comparison between the proposed search and other HPO algorithm. CIFAR-10 is used.
4.3 EXPERIMENTS ON REAL FACE DATA SET
Finally, we the validity of our approach on real data sets. We conduct experiments on the applica- tion of deep face recogmition (Parkhi et al., 2015). Following (Wang et al., 2019), we use VggFace2- (Cao al., 2018) data set, which training, then tune param--arraettrs and report testing accuracy on four small and clean image data sets, i.c., CALFW (Zheng et al., 2017), CPLFW (Zheng Deng, 2018), AgeDB (Moschoglou et al., 2017), and CFP (Sengupta al., 2016) (details are in Appendix A.3).
noisy data set collected from Google image search, for
We compare the performance of our proposed method with those methods used in Sec.4.2.1 and Co-mining (Wang et al., 2019). Testing accuracy is reported in T. We can see that, our pro- posed method consistently achieves the best performance. Thus, our method is not only useful with synthetic noise but also work well on real applications.
Table 1: Experiments deep face recognition with real data sets. All models trained with noisy VggFace2-R data set, and evaluate on the four clean test data sets. Except the proposed method, all performance copied from (Wang al., 2019); F-correction is not reported as its performance is not available in (Wang et al., 2019).
CALFW
CPLFW
AgeDB
CFP
average
Standard 90.11
86.30
92.81
95.50
91.18
Decoupling
90.23
86. 14
93.90
95.85
91.53
MentorNet
90.14
85.41
92.70
95.20
90.86
Co-teaching
89.90
85.05
92.05
95.05
90.62
Co-teaching+
89.43
85.23
92.50
95.41
90.64
Co- Co-Mining
91.06
87.31
94.05
95.87
92.07
Proposed
92.04
89.43
95.22
96.16
93.20
5 CONCLUSION
In this paper, motivated by the main difficulty that to what extent the memorization effect of deep networks can happen, we propose to exploit memorization by automated machine learning (AutoM- L) techniques. This is done by first designing expressive but compact search space, which is based on observed general patterns for memorization, and designing a natural gradient- based search algo- rithm, which overcomes the problem of non- con-differential and failure of parameter-sharing. Extensive experiments on both synthetic data sets and benchmark data sets demonstrate that the proposed method not only be much efficient than existing AutoML algorithms, but also achieve much better performance than the state-of-the-ee sample-selection approach.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Y. Akimoto, S. Shirakawa, N. Yoshinari, K. Uchida, S. Saito, and K. Nishida. Adaptive stochastic natural gradient method for one-shot neural architecture search. In ICML, pp. 171-180, 2019.
Amari. Natural gradient works efficiently learning. Neural Computation, 10(2):251-276, 1998.
D. Angluin and P. Laird. Learning from noisy examples. Machine Learning, 2(4):343-370) 1988.
D. Arpit, S. Jastrzkbski, N. Ballas, D. Krueger, E. Bengio, M. Kanwal, T. Maharaj. A. Fischer, A. Courville, and Y. Bengio. A closer look at memorization in deep networks. In ICML, pp- 233-242 242. JMLR. org, 2017.
B. Baker, O. Gupta, N. Naik, and R. Raskar. Designing neural network architectures using rein- forcement learning. In ICLR, 2017.
G. Bender, P.-J. Kinderm, B. Zoph. V. Vasudevan, and Q. Le. Understanding and simplifying one- shot architecture search. In ICML, pp. 549-558, 2018.
Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. ICML, pp. 41-48. ACM, 2009.
Bergstra and Y. Bengio. Random search for hyper-parameter optimization. JMLR, (3 305, 2012.
Q. Cao, L. Shen, W. Xie, O. M Parkhi, and A. Zisserman. Vggface2: A dataset for recognising faces across pose and age. FG, pp. 67-74. IEEE, 2018.
Chen, B. Liao, G. Chen, and S. Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In ICML, pp. 1062-1070, 2019.
Escalante, M. Montes, and Luis E. Sucar. Particle swarm model selection. JMLR, 10(Feb):405- 440, 2009.
M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and F. Hutter. Efficient and robust automated machine learning. NeurlPS, pp. 2962-2970, 2015.
A. Ghosh, H. Kumar, and P. Sastry. Robust loss functions under label noise for deep neural networks. In AAAI, pp. 1919-1925, 2017.
Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT press, 2016.
B. Han, J. Yao, G. Niu, M. Zhou, I. Tsang, Y. Zhang, and M. Sugiyama. Masking: A new perspective of noisy supervision. In NeurlPS, pp. 5836-5846 2018a.
B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang. and M. Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurlPS, pp. 8527-8537, 2018b.
F. Hutter, L. Kotthoff, and J. Vanschoren (eds.). Automated Machine Learning: Methods, Systems, Challenges. Springer, 2018. In press, available at http://automllorg/bbol
L. Jiang, Z. Zhou, T. Leung, J. Li, and F.-F. Li. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In ICML, pp-
2018.
K. Kandasamy, K. Vysyaraju, W. Neiswanger, B. Paria, C. Collins, Schneider, B. Poczos, and E. Xing, Tuning hyperparameters without grad students: Scalable and robust bayesian optimisa- tion with Dragonfly. Technical report, arXiv preprint arXiv:1903.06694. 2019.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Laine and Aila. Temporal ensembling for semi-supervised learning. In ICLR, 2017.
L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: novel bandit- based approach to hyperparameter optimization. JMLR, 18(1):775-66866 2017a.

---

Under review as a conference paper at ICLR 2020
Y. J. J. Yang, Y. Song, L. Cao, J. Luo, and L.-J. Li. Learning from noisy labels with distillation. In ICCV, pp. 1918,1918, 2017b.
H. Liu, K. Simonyan, and Y. Yang. DARTS: Differentiable architecture search. In ICLR, 2019.
Luketina, M. Berglund, K. Greff, and T. Raiko. Scalable gradient-based tuning of continuous regularization hyperparameters. In ICML, pp. 2952-2960, 2016.
X. Ma. Y. Wang. M. Houle, S. Zhou, S. Erfani, S.-T. Xia, S. Wijewickrema, and J. Bailey. Dimensionality-drive learning with noisy labels. In ICML, pp. 3361-3370, 2018.
E. Malach and S. Shalev-Shwartz. Decoupling" when to update" from" how to update". In NIPS, pp. 960-970, 2017.
T. Miyato, S. Maeda, M. Koyama, and S. Ishii. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. In ICLR, 2016.
Moschoglou, A. Papaioannou, C. Sagonas, J. Deng. I. Kotsia, and Zafeiriou. Agedb: the first manually collected, in-the-wild age datahase. In CVPRW, 2017.
Y. Ollivier, L. Amold, A. Auger, and Hansen. Informatiin-geometri optimization algorithms: A unifying picture via invariance principles. JMLR, 18(1):564-66) 628, 2017.
O. Parkhi, A. Vedaldi, and A. Zisserman. Deep face recognition. Technical report, University of Oxford, 2015.
R. Pascanu and Y. Bengio. Revisiting natural gradient for deep networks. Technical report, arXiv preprint
2013.
G. Patrini, A. Rozza, A. Menon, R. Nock, and L. Qu. Making deep neural networks robust to label noise: a loss correction approach. In CVPR, pp. 2233-2241, 2017.
S. Reed, H. Lee, D. Anguelov, C. Szegedy, D. Erhan, and A. Rabinovich. Training deep neural networks on noisy labels with bootstrapping. In ICLR Workshop, 2015.
M. Ren, W. Zeng, B. Yang, and R. Urtasun. Learning to reweight examples for robust deep learning. In ICML, pp. 4331-4340, 2018.
Sengupta, J. C. Chen, C. Castillo, V. M. Patel, R. Chellappa, and D. W. Jacobs. Frontal to profile face verification in the wild. In WACV, 2016.
B. Settles. Active learning literature survey. Machine Learning. 15(2):201-221, 1994.
J. Snoek, H. Larochelle, and R. Adams. Practical Bayesian optimization of machine learning algo- rithms. In NIPS, pp. 2951-2959, 2012.
Sukhbaatar, J. Bruna, M. Paluri, L. Bourdev, and R. Fergus. Training convolutional networks with noisy labels. In ICLR Workshop, 2015.
D. Tanaka, D. Ikami, T. Yamasaki, and K. Aizawa. Joint optimization framework for learning with noisy labels. In CVPR, pp. 5552-5560, 2018.
A. Tarvainen and H. Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi- semi-supervised deep learning results. In NIPS, 2017.
X. Wang, S. Wang, J. Wang. H. Shi, and T. Mei. Co-Mining: Deep face recognition with noisy labels. In ICCV, pp. 9358-9367, 2019.
L. Xie and A. Yuille. Genetic CNN. In ICCV, pp. 1388-1399 2017.
Xie. Zheng, C. Liu, and Lin. SNAS: stochastic neural architecture search. In ICLR, 2018.
X. Yu, B. Han, J. Yao, G. Niu, I. Tsang, and M. Sugiyama. How does disagreement help general- ization against label corruption? In ICMI., pp. 7164-7173, 2019.
12

---

Under review as a conference paper at ICLR 2020
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2016.
T. Zheng and W. Deng. Cross-pose Ifw: A database for studying crosspose face recognition in unconstrained environments. Technical Report 18-01, Beijing University of Posts and Telecom- munications, February 2018.
T. Zheng, W. Deng, and J. Hu. Cross-age Ifw: A database for studying cross-age face recognition in unconstrained environments. Technical report, arXiv preprint arXiv: 1708.08197, 2017.
B. Zoph and Q. Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
13

---

Under review as a conference paper at ICLR 2020
A IMPLEMENTATION DETAILS
A.1 DETAILS FOR FIG.
(a-b) We use the same MNIST/CIFAR-ION 10/100 datasets and their details are in Appendix A.3. The network model model and its details are given in Appendix A.4. The number of training epochs is 200 and the batch size is set to 128. The initial learning rate for the Adam optimizer 0.001 and linearly decayed to zero from the 80th epoch to final.
(c) Details of each model listed in Appendix A.4. Note that model 2 on CIFAR-10 has the output dimension of 10 instead of 100. The number of training epochs, batch size, optimizer and learning rate schedule follows (a) and (b).
(d) The learning rate schedule for Adam follows (a) and (b). The initial learning rate for the SGD optimizer is 0.1, and decay to 0.01 and 0.001 from the 500th and 750th epoch, respectively. And for the RMSPPrp optimizer, the learning rate is set constantly to 0.01. To demonstrate the memorization effects for SGD optimizer, the number of training epochs is 1000 instead of 200. The model is model and the batch size is 128.
(c) The model, number training epochs, optimizer and learning rate schedule the same as (a) and (b). In word, we only change the batch size.
(I) We only change the initial learning rate for the learning rate schedule. Also, demonstrate the memorization effects for small learning rate cases, the number of training epochs is 1000 instead of 200. And the model, batch size, optimizer all follow (a) and (b).
A.2 NATURAL GRADIENT
Since our search space for a is bounded, we use Beta distribution in natural gradients, i.c. pe(A) = I Beta(ay; 0,). Each Beta distribution has two parameters x; y and the probability density function (PDF) is given by
Beta x. v) = Mo++s)//(())) a)"
where ['(x) is the gamma function. The Fisher information matrix
(()) = (p,(a)) Ve log pa (a) log log ()())))
is estimated by samples since explicit formula is available. And the gradient in Fisher V log po (ax) is different for different parameters x and y, as
V log Beta(a; v. = log(a) + (xx + y) - (zx) and Vy log Beta(a; x, y) log(1 a) +0x + v) (v))
where (xx denotes the digamma function.
For synthetic data, run for 10 iterations and in each iteration, we sample 10 samples and evaluate its performance. The learning rate for natural gradient is set to 80.
For benchmark and real noisy data sets, all search algorithms will for 5 iterations in total. In random search and natural gradient, we sample 10 samples for CIFAR-10 and 20 samples for other evaluate its performance in each iteration. The learning rate for natural gradient is
four data set to 20, 500, 80, 30 for MNIST, C1FAR-10, CIFAR-100 and VggFace2-R data set respectively.
To obtain better estimation of the Fisher information matrix, in spite of those samples for evalua- tion, we also sample another 10000 samples to compute the Fisher in each iteration.
A.3
DATA SETS
We obtain MNIST, CIFAR-10, CIFAR- 100 from PyTorch's torchvision package, and VggFace2-R, CALFW, CPLFW, AgeDB, CFP from its original source. Some statistics about those datasets are given below.
14

---

Under review as a conference paper at ICLR 2020
Table 2: Summary of data used in the experiments.
# of train
# of validation
# of test
# of class
MNIST
60,000
5,000
5,000
10
CIFAR-IO
50,000
5,000
5,000
10
CIFAR- 100
50,000
5,000
5,000
100
VggFace2-R
3.31M
(Train only)
(Train only)
9.131
CALFW
(Validation & test only)
6,087
6,087
5,749
CPLFW
(Validation & test only)
5,826
5,826
5,749
AgeDB
(Validation & test only)
8,244
8,244
568
CFP
(Validation & test only)
3,500
3,500
500
A.4 NETWORK STRUCTURE
For VggFace2-R, we use ResNet-50 in (Wang et al., 2019). Adam optimizer (Kingma Ba, 2014) (momentum=0.9) is used with an initial learning rate of 0.001, and the batch size is set to 128 and we run 200 epochs. The learning rate is linearly decayed to zero from 80 200 epochs.
Table 3: MLP and CNN models used in our experiments.
MLP on MNIST
CNN on CIFAR-10
CNN on CIFAR-100
Model 3
(Model 1)
(Model 2)
28 x8828 Gray Image
32x32 RGB Image
32x32 RGB Image
32x32 RGB Image
3x3 Conv, 64 BN, ReLU
3x3 Conv, 128 BN, LReLU
5x5 Conv, ReLU
3x3 Conv, 128 BN, LReLU
3x3 Conv, 64
3x3 Cony, 128 BN, LReLU
2x2 Max-pool
BN, ReLU
2x2 Max-pool, stride 2 Dropout, p=0.25
2x2 Max-pool
3x3 Conv, 128
3x3 Conv, 256 BN, LReLU
5x5 Cony, ReLU 2x2 Max-pool
BN, ReLU
3x3 Conv, 256 BN, LReLU
Dense
3x3 Conv, 128
3x3 Conv, 256 BN, LReLU
28x26-2256 ReLU
BN, ReLU
2x2 Max-pool, stride 2 Dropout, p=0.25
2x2 Max-pool
Dense
3x3 Conv, 196
3x3 Conv, 512 BN, LReLU
16x5x5- 120
BN, ReLU
3x3 Cony, 256 BN, LReLU
ReLU Dense 120-+84
3x3 Conv, 196 BN, ReLU
3x3 Conv, 128 BN, LReLU Avg-pool
ReLU
2x2 Max-pool
Dense
Dense 84-+10
Dense 256
Dense 128
15

---

Under review as a conference paper at ICLR 2020
B MORE RESULTS
B.1 PERFORMANCE COMPARISON WITH EARLY STOPPING
To make a better comparison, we apply early-stop trick to all those methods in Sec.4.2 and report their best accuracy instead of the final accuracy. The accuracy is averaged in 5 runs and also report the standard deviation. Results are shown below:
Table 4: Testing accuracy (in percentage) with early stop on MNIST.
noise
symmetric 20%
symmetric 50%
pairflip 45%
Standard 97.16=0.20
95,40=0.09
68. 5.26
Decoupling
97.09
94.88 0.39
18.03-11.76
F-corecction
97.32+0.13
95.59 0.30
9.50+
MentorNet
97.23=00.11
95.32 0.23
90.47+:2.09
Co-teaching
97.26.10.14
95,42 0.27
90,49:11.52
Co-teaching+
97.88-100 1:010
96.52
0.10
90.00
Proposed
97.89-1.33
96.91 0.25
95.94 0.70
Table 5: Testing accuracy (in percentage) with early stop on CIFAR-10.
noise
symmetric 20%
symmetric 50%
pairflip 45%
Standard 09.64=0.96
52.23=1.32
44.01
Decoupling
56.16-t 1.02
43.57 3.03
45.31+1.00
F-correction
=9.74
53.71= 1.48
10.22+1.49
MentorNet
59.93
51.08=1.00
37.45::2.45
Co-teaching
60.62.1 1.03
53.48 0.86
41.26:10.74
Co-teaching+
59.70 1.07
52.49
1.52
43.66+1.28
Proposed
0..27
54.26 1.55
50.80
Table Testing accuracy (in percentage) with early stop on CIFAR-100.
noise
symmetrie 20%
symmetric 50%
pairflip 45%
Standard
43.98=0.87
34.19_0.52
26.3010.87
Decoupling
2..5
23.18=5.85
17.95+22.86
E-correction
46.80=0.13
39.20=0.30
1.58++0.31
MentorNet
46.82+1.19
38.68=0.39
25.98::0226
Co-teaching
47.69-10.66
40.10 0.54
27.01 +0.69
Co-teaching+
49.03=00.22
39.24 0.64
31.39+0089
Proposed
49.63
42.90 0.40
36.00.10.68
B.2 PERFORMANCE COMPARISON WITH SIMPLE DECAY FUNCTION
Here, we compare the performance obtained (under early stop setup in Sec.B.1) from
the space used in Co-teaching (Han al., 2018b) (see Remark 3.1);
space spanned by a simple decay function (i.e., pow-d Fig.2); and
the proposed space in (q..
Natural gradient proposed in S.c.3.3.2 is used for optimization. Results are shown in Fig.7. We see that performance obtained from the proposed method is much better than that from simple decay function. This again demonstrates the needs of approximating R(T) by a linear combination of some basis functions.
16

---

Under review as a conference paper at ICLR 2020
Table 7: Testing accuracy (in percentage) comparison between the proposed method and that from a simple decay function.
space from
pow-d
noisy
Co-teaching
(Figure 2)
proposed
symmetric 20%
97.83=+0.70
97.67+0003
97.89:10.21
MNIST
symmetric 50%
96.54.10.40
96.5610.01
96.9110..25
pairflip 45%
93.27--0.80
94.99+0.03
+5.94
symmetric 20%
60.72++0.31
60.83+0.10
61.27+0.59
CIFAR-10
symmetric 50%
54.11:00.56
53.1910.50
54.26:1..55
pairflip 45%
1..4
49.1710.38
50.50
symmetric 20%
18.18-11.27
47.93-10.02
49.63 0.74
CIFAR-100
symmetric 50%
40.57:10338
41.67=1.57
42.90110.40
pairflip 45%
27.30+F0.99
32.47+0.05
36.00
B.3 PRECISION OF CLEAN SAMPLES
Here, to understand why the proposed method can lead better performance than Co-teaching, fol- lowing Han et al. (2018b), we also show label precision, i.c., the average ratio (in percentage) of clean samples in selected samples, in each epoch. Results are in Tab.7. As we can see, precision is consistently much higher than that from Co-teaching, This means the training samples used by the proposed method is cleaner than those used by Co-teaching, and thus better performance can be obtained by the proposed method.
Ture mie (WNIST symmeerne 2090
20
02-
06
04-
03
02
Frodsed
Frossed
179
25
130
300
10
rabi (CIPA-1O symmetic 50%)
ratto IC CPA--10 partip 45%1
04
3.-
02
32-
Esida
Esuda
Pure rite FAR 100 2013
Pure nice 127MN-200 symmitic 30%
Tare rabs ICPAN parlip 40%
2.4
12-
03
180 Sperhs
150
(a) Symmetric 20%.
(b) Symmetric 50%.
(c) Pair 45 %.
Figure 7: Comparison on precision of clean samples between the proposed method and Co-teaching. Top to bottom: MNIST, CIFAR-10 and CIFAR-100.
17


Reviews:
[{"review_id": "ryl61yYAtr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of learning from corrupted labels via picking up clean instances from training dataset. The sample selection mainly based on function R(t), which controls how many instances are kept. This paper proposes a unique curvature of R(t) based on intuition and presents how R(t) can be learned via combination of some existing functions. Natural gradient is presented to optimize the parameters in the autoML framework. Experimental results on both synthetic data and real-world data demonstrate the effectiveness of the proposed method. \n\nA few comments on this paper:\n1. The paper is very verbose and hard to follow. It introduces too many basic concepts in autoML.\n2. A key part of the paper is the curvature of R(t), which is based on intuition. Meanwhile, the learned curvature (in Fig 5) doesn't follow the curvature. Does this mean this paper is contradicting its self? The curvature of defined R(t) is not needed?\n3. The major difference between this paper and (Han et al. 2018) is how R(t) is defined and learned. The technical contribution of this paper is limited. \n\nMinor comments:\n1. For all the figures, it is difficult to view the y-axis (or the y-axis is missing). \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Skg9z1HZcH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper develops a method for sample selection that exploits the memorization effect. In essence, the authors adopt the co-teaching  (Han et al. NeurIPS 2018) and MentorNet (Jiang et al., ICML 2018) framework, which selects some fraction of examples per minibatch that are hopefully \"cleaner\" than noisier examples to compute updates from. While in Han et al. the number of instances R selected depends on the number of epochs that have been completed, this paper instead seeks to learn R by approximating it as a linear combination of different types of basis functions  and using natural gradient as the search algorithm. The search space proposed by the authors seem comprehensive: it encompasses the search space of co-teaching, the prior state of the art. Results on synthetic tasks as well as MNIST/CIFAR appear to show the superiority of the proposed method over random search, co-teaching, and other baselines, although the results don't seem conclusive. Overall, I have concerns with some of the contributions, experiments, and presentation, which leaves me at a weak reject.\n\ncomments:\n- Section 3.1 isn't very compelling to me. Experiments done on just CIFAR with two architectures and optimizers are certainly not sufficient to make any broad claims. I don't think this qualifies as a \"contribution\" of the paper. \n- The paper is difficult to understand, and much of this difficulty stems from poor writing / presentation. The plots depicting experimental results are especially hard to follow. \n- I'm a little confused with the setup here. Most practitioners use early stopping to halt training after performance on the validation set drops. As such, why should we care about the held-out curve after the maximum is reached? Shouldn't we care more about the training curve, as at some point during training the noisy labels will also be memorized? Isn't this the definition of the \"memorization effect\"? \n- What about standard baseline methods e.g., active learning to help with this problem? Active learning seems highly relevant yet is not mentioned anywhere in this paper. \n- Are all of the basis functions in Fig 2 necessary for the performance of the proposed method? How were they selected? Why is this motivated by the Taylor expansion?\n- Figure 5 shows a bunch of R(t) curves learned by the proposed approach across a variety of datasets / noise levels. All of the curves look very similar! A reasonable baseline motivated by these results is to just apply a simple decay function to R(t) with a single hyperparameter controlling the rate of decay. I suspect this would also work better than the co-teaching approach, and perhaps render the more complex method here unnecessary. In fact, all of the gains associated with this method could just be due to co-teaching dropping far less examples as training progresses, as its decay rule isn't optimal. \n\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkxT0H5DcB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on the topic of learning from noisy -- or as they call it \"corrupted\" -- labels. Specifically this focuses on an approach where data selection -- ideally of cleaner/less noisy examples --  can help the learn model overcome data noise, akin to the approaches this builds upon (i.e, the Co-Teaching and MentorNet approaches). The specific idea here is to take an AutoML style approach to the problem  in particular to determine how many examples are selected in each mini-batch. The proposed method is based upon natural gradient based updates to the hparams (which was really the only feasible way to tackle this problem given the complex dependence on the hparams and a good choice). The experimental results using synthetic noise corruption are indicative of improved performance compared to the baseline techniques.\n\nOverall while I thought the paper made for a very interesting read and showed some great promise I had some significant concerns as well.\n\nOn the plus side:\n\n+ The empirical results on the simulated noisy data are quite positive/\n+ The proposed method makes sense as does the search algorithm in the hparam space.\n\nMy main concerns with the work stem from the empirical study and choices made there. While I understand that other existing techniques like the Co-teaching and MentorNet approaches have used simulated noise to study the impact of performance of these robustness techniques, at some point I question their validity on real datasets. Noise patterns in real datasets hardly follow some set pattern and thus I hesitate to read much into results derived solely on synthetic datasets. Given that the goal of these techniques is to improve performance when training with real, noisy labeled data why not actually demonstrate performance on such benchmarks? For example, there are numerous datasets from domains like crowdsourcing that allow you to get \"noisy\" ratings for datapoints. Wouldn't a more compelling argument be derived by showing improved performance on such datasets?\n\nThus to summarize: I worry that the results derived solely on simulated noise may not be very indicative of performance in more realistic settings and would request the authors to consider providing evidence on more realistic datasets.\n\nI also wanted to note that the paper exposition is lacking in some aspects and I needed to reread certain sections to make sure I understood them correctly. I think the paper would benefit from a good proofread not just from the grammar/spelling perspective (which there are multiple instances which could be improved) but also from the overall presentation and legibility perspective.\n\nAll this said: I want to clarify that this topic is not my research focus and hence I am uncertain as to how much findings on these simulated noise patterns carry over to real datasets and their associated noise patterns. If there is existing evidence indicating strong correlation, then perhaps my review may have varied.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper develops a method for sample selection that exploits the memorization effect. While the paper has been substantially improved from its original form, the paper still does not meet the quality bar of ICLR in terms of presentation of the results and experimental validation. The paper will benefit from a revision and resubmission to another venue.
Decision: Reject