Iteratively Learning from the Best

We study a simple generic framework to address the issue of bad training data ; both bad labels in supervised problems , and bad samples in unsupervised ones . Our approach starts by fitting a model to the whole training dataset , but then iteratively improves it by alternating between ( a ) revisiting the training data to select samples with lowest current loss , and ( b ) re-training the model on only these selected samples . It can be applied to any existing model training setting which provides a loss measure for samples , and a way to refit on new ones . We show the merit of this approach in both theory and practice . We first prove statistical consistency , and linear convergence to the ground truth and global optimum , for two simpler model settings : mixed linear regression , and gaussian mixture models . We then demonstrate its success empirically in ( a ) saving the accuracy of existing deep image classifiers when there are errors in the labels of training images , and ( b ) improving the quality of samples generated by existing DC-GAN models , when it is given training data that contains a fraction of the images from a different and unintended dataset . The experimental results show significant improvement over the baseline methods that ignore the existence of bad labels/samples . 1 INTRODUCTION . This paper is motivated by the problem that training large machine learning models requires wellcurated training sets with lots of samples . This is an issue in both supervised and unsupervised settings . For supervised problems ( e.g . multi-label classification ) we need very many properly labeled examples ; generating these requires a lot of human effort . Errors in labels can significantly affect the accuracy of naive training . For example , a faulty CIFAR-10 dataset with 20 % of automobile images mis-labeled as “ airplane ” ( and so on for the other classes ) leads to the accuracy of a neural architecture like WRN of Zagoruyko & Komodakis ( 2016 ) to go from over 90 % to about 70 % . Even in unsupervised problems , where there are no labels , spurious samples are a nuisance . Consider for example the task of training a generative adversarial network ( GAN ) Goodfellow et al . ( 2014 ) to produce realistic images of faces . While this has been shown to work well using a dataset of face images 1 , like Celeb-A Liu et al . ( 2015 ) , it degrades quickly if the training dataset is corrupted with non-face images . Our experiments illustrate these issues . Main idea We study a simple framework that addresses the problem of spurious data in both supervised and unsupervised settings . Note that if we knew which samples were clean , we could learn a model ; conversely , if we had a good model , we could find the clean samples . But we do not have either ; instead we have a chicken-and-egg problem : finding the best subset of samples needs a model , but finding a model needs a subset of samples . In this paper , we take the natural approach to this problem by ( 0 ) fitting an initial model on all samples , and then alternating between ( 1 ) finding the best subset of samples given a model , and ( 2 ) fitting a model given a subset . 2 Our approach is most related to the works of Bhatia et al . ( 2015 ) and Vainsencher et al . ( 2017 ) , and we contrast to these in the related work section . 1https : //github.com/carpedm20/DCGAN-tensorflow 2Our framework sounds initially similar to EM-style algorithms like k-means . Note however that EM needs to postulate a model for all the data points , while we search over a subset and do not worry about the loss on corrupted points . We are alternating between a simple search over subsets and a fitting problem on only the selected subset ; this is not an instance of EM . Our approach can be applied to any existing model that provides ( a ) confidence scores for samples , and ( b ) a way to train the model for a set of samples . Indeed in our experiments we use existing models and training code , but with iterative ( re ) selections of samples as above . In supervised settings , any model that maps features to responses typically associates a confidence to every sample , e.g. , in linear regression , given a model θ , the confidence in a sample ( xi , yi ) increases as its squared error ( yi − x > i θ ) 2 decreases . For a neural network classifier , on the other hand , the confidence can be measured by the cross entropy loss between the output probability distribution and the true distribution . In unsupervised settings , the confidence of a sample is given by its “ likelihood ” under the model . If we are lucky , e.g. , in Gaussian mixture models , this likelihood is explicit and can be evaluated . However , if there is no explicit likelihood , we need to rely on a surrogate . E.g. , in our GAN setting below , we rely on the output of the discriminator as a confidence measure . We also note an important caveat : this paper is not focused on ( and may not be effective for ) the settings of adversarial examples , or of gross corruptions of features . Rather , our focus is to address the issue of bad labeling or curation of data , as motivated above . OUR RESULTS AND PAPER OUTLINE . We show that our framework ( outlined in Section 3 ) can address the spurious sample problem , both theoretically and empirically and for both supervised and unsupervised problems : ( 1 ) Mixed linear regression ( Section 4 ) and Gaussian mixture models ( Section 5 ) : For each of these mixture problem settings , under the standard statistical assumptions , we establish both statistical consistency and linear convergence of our iteratively updated model parameters . This holds true from any initial point ; the convergence is to the mixture component that is closest to the initial point . ( 2 ) Neural network classifiers ( Section 6 ) : We investigate image classification in CIFAR-100 , CIFAR-10 and MNIST datasets ( each with a different and appropriate existing neural architecture ) in the setting where there are systematic errors in the labels – i.e . every bad example in one label class comes from the same other label class . We show that our framework can significantly improve the classification accuracy over the baseline method that ignores the existence of bad samples . ( 3 ) Deep generative models ( Section 7 ) : We show both quantitatively and qualitatively the improvement of using our algorithm for image generation when the dataset consists of two types of images . Specifically , investigate training a GAN to ( a ) generate face images as in the “ intended ” Celeb-A dataset , but when the given training data has a fraction of the samples being the “ spurious ” CIFAR-10 images , and ( b ) generate intended MNIST digits when training data contains a fraction of samples from the spurious Fashion-MNIST dataset . A crucial innovation here is that the output of the discriminator network in the GAN can be used as a confidence metric for the training samples . In Section 8 , we discuss the positives and negatives of our results , underlying intuition and also some future directions . All of our experiments were done on clearly identified publicly available neural network architectures , datasets , and training software . We also clearly describe our error and noise effects ; we expect them to be easily reproducible . 2 RELATED WORK . Alternating minimization for robustness Two works are most related to ours . Bhatia et al . ( 2015 ) proposes an iterative hard thresholding algorithm for adversarial noise in linear regression responses ; they establish statistical guarantees and linear convergence under this harder noise model . We focus on multiple mixed linear regression , a different setting that results in an easier noise model , and we provide convergence guarantee for recovering any single mixture component starting from a local region . In addition we also prove these for Gaussian mixture models , which is not studied in their work . Vainsencher et al . ( 2017 ) proposes what is essentially a soft version of our method , and proves local convergence and generalization . However they do not have any initialization , and hence no global or consistency guarantees . Neither of these empirically explore the overall approach for more complex / neural network models . Robust regression Recent work on robust regression consider strong robustness in terms of both the inputs and the outputs and provide guarantees for constant corruption ratio Diakonikolas et al . ( 2018 ) ; Prasad et al . ( 2018 ) ; Klivans et al . ( 2018 ) . Chen et al . ( 2013 ) ; Balakrishnan et al . ( 2017a ) ; Liu et al . ( 2018 ) study the high dimensional setting under this regime . These algorithms usually require much more computation , compared with methods for dealing with noisy outputs , e.g. , Bhatia et al . ( 2015 ) . Another type of robustness considers heteroscedastic noise for every sample . From the learning perspective , Anava & Mannor ( 2016 ) ; Chaudhuri et al . ( 2017 ) both require strong constraints on the noise model which largely reduces the degree of freedom . Anava & Mannor ( 2016 ) considers the online learning setting , while Chaudhuri et al . ( 2017 ) considers the active learning setting . Mixed linear regression Alternating minimization type algorithms are used for mixed linear regression with convergence guarantee Yi et al . ( 2014 ) . Chen et al . ( 2014 ) provides a convex formulation for the problem . Balakrishnan et al . ( 2017b ) shows expectation-maximization ( EM ) algorithm is provably effective for a set of tasks including the basic setting of mixed linear regression and Gaussian mixture model . Sedghi et al . ( 2016 ) extends to the multiple component setting , where provable tensor decomposition methods are used . More recently , Li & Liang ( 2018 ) gives a more general result for learning mixtures of linear regression . On the other hand , Ray et al . ( 2018 ) studies the problem of finding a single component in mixed linear regression problem using side information . Noisy labels Classification tasks with noisy labels are also of wide interest . Frénay et al . ( 2014 ) gives an overview of the related methods . Theoretical guarantee for noisy binary classification has been studied under different settings Scott et al . ( 2013 ) ; Natarajan et al . ( 2013 ) ; Menon et al . ( 2016 ) . More recently , noisy label problems have been studied for deep neural network based models . Reed et al . ( 2014 ) and Malach & Shalev-Shwartz ( 2017 ) develop the idea of bootstrapping and query-bycommittee into neural network based models . On the other hand , Khetan et al . ( 2018 ) and Zhang & Sabuncu ( 2018 ) provide new losses for training under the noise . Sukhbaatar & Fergus ( 2014 ) adds a noise layer into the training process , while Ren et al . ( 2018b ) provides a meta-algorithm for learning the weights of all samples by referencing to a clean subset of validation data during training . Ren et al . ( 2018a ) consider building robust classifiers for the positive-unlabeled classification problem using l0 norm penalization . Others Bora et al . ( 2018 ) studies the problem of noisy images for GANs . In their setting , all images are of low quality under some measurements , while in our problem , we consider the image dataset consists of ( possibly more than ) two types of images , which is different . There are also studies that consider noisy samples more generally . The classical RANSAC method Fischler & Bolles ( 1981 ) provides a paradigm for fitting a model to experimental data , and can be potentially used for the noisy data setting . However , the algorithm requires multiple times of random sampling to find a valid consensus score , which is conceptually different from our idea of doing iterative filtering . EM algorithm Moon ( 1996 ) can also be used for detecting noisy samples , e.g. , in the setting of mixture models Balakrishnan et al . ( 2017b ) . However , our algorithm does not need to know the likelihood for the bad samples for the iterative updates .

Summary: The authors propose an iterative method for discarding outlying training data: first, learn a model on the entire training dataset; second, identify the training examples that have high loss under the learned model; and then alternate between re-learning the model on the training examples that do not have high loss, and re-identifying the training examples with high loss under the new model. This method works for both supervised and unsupervised learning, and the authors show that in theory, their method has some convergence properties in the mixed linear regression and Gaussian mixture model settings. The authors also run some experiments on neural networks and datasets with synthetic noise to show the benefits of their proposed method.