Searching to Exploit Memorization Effect in Learning from Corrupted Labels

1 INTRODUCTION . Learning with deep neural networks has enjoyed huge empirical success in recent years across a wide variety of tasks , from image processing to speech recognition , and from language modeling to recommender system ( Goodfellow et al. , 2016 ) . However , their success highly counts on the availability of well-annotated and big data , which is barely available for real-world applications . Instead , what we are facing with in practice are large data sets which are collected from crowdsourcing platforms or crawled from the Internet , thus containing many noisy labels ( Li et al. , 2017b ; Patrini et al. , 2017 ) . Besides , due to the vast learning capacity of deep networks , they will eventually over-fit on these noisy labels , leading to poor predicting performance , which can be worse than that obtained from simple models ( Zhang et al. , 2016 ; Arpit et al. , 2017 ) . To reduce negative effects from noisy labels , many methods have been proposed ( Sukhbaatar et al. , 2015 ; Reed et al. , 2015 ; Patrini et al. , 2017 ; Ghosh et al. , 2017 ; Malach & Shalev-Shwartz , 2017 ) . Recently , a promising direction is training networks only on selected instances that are more likely to be clean ( Jiang et al. , 2018 ; Han et al. , 2018b ; Ma et al. , 2018 ; Yu et al. , 2019 ; Wang et al. , 2019 ) . Intuitively , as the training data becomes less noisy , better performance can be obtained . Among those works , the representative methods are MentorNet ( Jiang et al. , 2018 ) and Co-teaching ( Han et al. , 2018b ; Yu et al. , 2019 ) , they take small-loss samples in each mini-batch as clean instances . Specifically , MentorNet pre-trains an extra network , and then uses the extra network for selecting clean instances to guide the training . When the clean validation data is not available , MentorNet has to use a predefined curriculum ( Bengio et al. , 2009 ) . Co-teaching is an improvement over MentorNet , it simultaneously maintains two networks which have identical architectures during the training process . And in each mini-batch of data , each network is updated using the other network ’ s small-loss instances . To the success of these sample-selection methods , the memorization effect of deep networks ( Zhang et al. , 2016 ; Arpit et al. , 2017 ) is the crux . Memorization happens widely in various architectures of deep network , e.g. , multilayer perceptron ( MLP ) and convolutional neural network ( CNN ) . Specifically , it means that deep networks tend to learn easy and correct patterns first and then over-fit on ( possibly noisy ) training data set ( see Fig.1 ( a ) - ( b ) ) . Thus , when learning with noisy labels , while the validation loss will first increase and then significantly decrease , the training loss will continuously get smaller with more training epochs . Due to such effect , sample-selection methods can learn correct patterns at early stage and then use the obtained discriminative ability to filter out corrupted instances in subsequent training epochs ( Jiang et al. , 2018 ; Han et al. , 2018b ; Chen et al. , 2019 ) . While the memorization effect is critical to the success of sample-selection methods , however , how to properly exploit it is not addressed in the literature . And trivial attempts can easily lead to even worse performance than standard deep networks ( Han et al. , 2018b ) . Some recent endeavors seek to evade from this problems by integrating with other auxiliary information , e.g. , a small clean subset is used in ( Ren et al. , 2018 ) , and knowledge graphs are utilized in ( Li et al. , 2017b ) . In this paper , motivated by the success of automated machine learning ( AutoML ) on designing data-dependent models ( Hutter et al. , 2018 ) , and the fact that memorization heavily depends on many factors ( Zhang et al. , 2016 ; Arpit et al. , 2017 ) , we propose to exploit memorization effects automatically using AutoML techniques . Contributions are summarized as follows : • First , to have an in-depth understanding of why it is difficult to tune sample-selection methods with good performance . We examine behaviors of memorization effect from multiple perspectives . We find that , while there exist general patterns in how memorization occurs with the training process ( see Fig.1 ( a ) - ( b ) ) , it is hard to quantize to which extend such effect can happen ( see Fig.1 ( b ) - ( f ) ) . Especially , memorization can be affected by many factors , e.g. , data sets , network architectures , and the choice of the optimizers . It is exactly such complex dependency make the design of proper sample-selection rules a hard problem , which motivates us to solve the problem by AutoML techniques . • To make good use of AutoML techniques , we then derive an expressive search space for exploiting memorization , which is from the above observations , i.e. , the curvature of how many instances need to be sampled during iterating should be similar with the inverse of the learning curve on the validation set . Such a space is not too huge since it has only a few variables , thus allows subsequent algorithms converging fast to promising candidates . • Then , to design an efficient algorithm , we show the failure of gradient-based methods and the inefficiency of derivative-free methods . These motivate us to take a probabilistic view of the search problem and adopt natural gradient descent ( Amari , 1998 ; Pascanu & Bengio , 2013 ) for optimization . The designed algorithm can effectively address above problems and is significantly faster than other popular search algorithms . • Finally , we conduct extensive experiments on both synthetic , benchmark , and real data sets , under various settings using different network architectures . These experiments demonstrate that the proposed method can not only be much more efficient than existing AutoML algorithms , but also can achieve much better performance than the state-of-the-art sample-selection approaches designed by humans . Besides , we further visualize and explain the searched functions , which can also help design better rules to control memorization effects in the future . 2 RELATED WORK . 2.1 LEARNING FROM NOISY LABELS . The mainstream research focuses on class-conditional noise ( CCN ) ( Angluin & Laird , 1988 ) , where the label corruption is independent of features . Generally , recent methods for handling CCN model can be classified into three categories . The first one is based on the estimation of transition matrix , which tries to capture how correct labels flip into wrong ones ( Sukhbaatar et al. , 2015 ; Reed et al. , 2015 ; Patrini et al. , 2017 ; Ghosh et al. , 2017 ) . These methods then use the estimated matrix to correct gradients or loss during training . However , they are fragile to heavy noise and unable to handle many classes ( Han et al. , 2018b ) . The second type is the regularization approach ( Miyato et al. , 2016 ; Laine & Aila , 2017 ; Tarvainen & Valpola , 2017 ) . Although regularization approach can achieve a satisfying performance , it is still an incomplete approach since ( Jiang et al. , 2018 ) shows that it can only delay the overfitting progress rather than avoid it , i.e . given enough training time , it can still fit the noisy data completely . Thus , it requires much domain knowledge to determine the appropriate number of training epochs in order to prevent overfitting . The last one is sample-selection approach , which attempts to reduce negative effects from noisy labels by selecting clean instances during training . The recent state-of-the-art method is also built on sample-selection approach ( Jiang et al. , 2018 ; Han et al. , 2018b ; Malach & Shalev-Shwartz , 2017 ; Yu et al. , 2019 ) . Active learning ( Settles , 1994 ) is a closely related method , which iteratively selects unlabeled samples with high high-confident predictions into the training data set . Thus , to do active learning , we need to obtain a classifier of which the performance is good enough . As a result , active learning is not applicable for directly learning from noisy labels here . A promising criteria to select “ clean instances ” is to pick up instances that has relatively small losses in each mini-batch ( Jiang et al. , 2018 ; Han et al. , 2018b ) . The fundamental property behind these methods is the memorization effect of deep networks ( Zhang et al. , 2016 ; Arpit et al. , 2017 ) , which means deep networks can learn simple patterns first and then start to over-fit . Such effect helps classifiers set up discriminate ability in the early stage , then make clean instances more likely to have smaller loss that those noisy ones . The general framework of sample-selection approach is in Alg.1 . Specifically , some small-loss instances D̄f are selected from the mini-batch D̄ in step 5 . These “ clean ” instances are then used to update network parameters in step 6 . The R ( t ) in step 8 , which controls how many instances to be kept in each epoch , is the most important hyper-parameter as it explicitly exploits the memorization effect ( Han et al. , 2018b ; Jiang et al. , 2018 ; Yu et al. , 2019 ) . Algorithm 1 Framework of the sample-selection approach ( Jiang et al. , 2018 ; Han et al. , 2018b ) . 1 : for t = 1 , · · · , T do 2 : shuffle training set D ; 3 : for n = 1 , · · · , N do 4 : draw a mini-batch D̄ from D ; 5 : select D̄f , i.e. , R ( t ) small-loss instances from D̄ based on network ’ s predictions ; 6 : update the network ’ s parameter using gradient from D̄f ; 7 : end for 8 : exploit memorization effects using R ( t ) ( estimation on percentage of clean instances ) ; 9 : end for However , it is hard to exactly determine how much proportion of small-loss samples should be selected in each epoch ( Jiang et al. , 2018 ; Ren et al. , 2018 ) . As will be discussed in Sec.3.1 , due to various practical usages issues , to which extend memorization effect can happen is hard to quantize . Thus , performance obtained from existing solutions are far from desired , and we are motivated to solve this issue by AutoML . 2.2 AUTOMATED MACHINE LEARNING ( AUTOML ) . Automated machine learning ( AutoML ) ( Hutter et al. , 2018 ) has recently exhibited its power in easing the usage of and designing better machine learning models . Basically , AutoML can be regarded as a black-box optimization problem where we need to efficiently and effectively search for hyperparameters or designs for the underlying learning models evaluated by the validation set . Regarding the success of AutoML , there are two important perspectives ( Feurer et al. , 2015 ; Zoph & Le , 2017 ; Xie & Yuille , 2017 ; Bender et al. , 2018 ) : • Search space : First , it needs to be general enough , which means it should cover existing models as special cases . This also helps experts better understand limitations of existing models and thus facilitate future researches . However , the space can not be too general , otherwise searching in such a space will be too expensive . • Search algorithm : Optimization problems in AutoML are usually black-box . Unlike convex optimization , there is no universal and efficient optimization tools . Once the search space is determined , domain knowledge should also be explored in the design of search algorithm so that good candidates in the space can be identified efficiently . Search space is domain-specific and needs to specially designed for every AutoML problem . There are two types of search algorithms popularly used . The first one is derivative-free optimization methods , it is usually used for searching in a general search space , e.g. , reinforcement learning ( Zoph & Le , 2017 ; Baker et al. , 2017 ) , genetic programming ( Escalante et al. , 2009 ; Xie & Yuille , 2017 ) , and Bayes optimization ( Feurer et al. , 2015 ; Snoek et al. , 2012 ) . More recently , gradientbased methods , which alternatively update parameters and hyper-parameters , have been developed as more efficient replacements for derivative-free optimization methods on some AutoML problems , e.g. , neural network architecture search ( Liu et al. , 2019 ; Akimoto et al. , 2019 ; Xie et al. , 2018 ) . However , existing AutoML techniques can not be directly used in exploiting memorization here . First , we need to carefully define a domain-specific space . Besides , we will also show existing algorithms are neither not applicable nor too slow . This motivates us to propose a new algorithm based on natural gradient .

Summary: This paper focuses on the topic of learning from noisy -- or as they call it "corrupted" -- labels. Specifically this focuses on an approach where data selection -- ideally of cleaner/less noisy examples --  can help the learn model overcome data noise, akin to the approaches this builds upon (i.e, the Co-Teaching and MentorNet approaches). The specific idea here is to take an AutoML style approach to the problem  in particular to determine how many examples are selected in each mini-batch. The proposed method is based upon natural gradient based updates to the hparams (which was really the only feasible way to tackle this problem given the complex dependence on the hparams and a good choice). The experimental results using synthetic noise corruption are indicative of improved performance compared to the baseline techniques.