Iteratively Learning from the Best

We study a simple generic framework to address the issue of bad training data ; both bad labels in supervised problems , and bad samples in unsupervised ones . Our approach starts by fitting a model to the whole training dataset , but then iteratively improves it by alternating between ( a ) revisiting the training data to select samples with lowest current loss , and ( b ) re-training the model on only these selected samples . It can be applied to any existing model training setting which provides a loss measure for samples , and a way to refit on new ones . We show the merit of this approach in both theory and practice . We first prove statistical consistency , and linear convergence to the ground truth and global optimum , for two simpler model settings : mixed linear regression , and gaussian mixture models . We then demonstrate its success empirically in ( a ) saving the accuracy of existing deep image classifiers when there are errors in the labels of training images , and ( b ) improving the quality of samples generated by existing DC-GAN models , when it is given training data that contains a fraction of the images from a different and unintended dataset . The experimental results show significant improvement over the baseline methods that ignore the existence of bad labels/samples . 1 INTRODUCTION . This paper is motivated by the problem that training large machine learning models requires wellcurated training sets with lots of samples . This is an issue in both supervised and unsupervised settings . For supervised problems ( e.g . multi-label classification ) we need very many properly labeled examples ; generating these requires a lot of human effort . Errors in labels can significantly affect the accuracy of naive training . For example , a faulty CIFAR-10 dataset with 20 % of automobile images mis-labeled as “ airplane ” ( and so on for the other classes ) leads to the accuracy of a neural architecture like WRN of Zagoruyko & Komodakis ( 2016 ) to go from over 90 % to about 70 % . Even in unsupervised problems , where there are no labels , spurious samples are a nuisance . Consider for example the task of training a generative adversarial network ( GAN ) Goodfellow et al . ( 2014 ) to produce realistic images of faces . While this has been shown to work well using a dataset of face images 1 , like Celeb-A Liu et al . ( 2015 ) , it degrades quickly if the training dataset is corrupted with non-face images . Our experiments illustrate these issues . Main idea We study a simple framework that addresses the problem of spurious data in both supervised and unsupervised settings . Note that if we knew which samples were clean , we could learn a model ; conversely , if we had a good model , we could find the clean samples . But we do not have either ; instead we have a chicken-and-egg problem : finding the best subset of samples needs a model , but finding a model needs a subset of samples . In this paper , we take the natural approach to this problem by ( 0 ) fitting an initial model on all samples , and then alternating between ( 1 ) finding the best subset of samples given a model , and ( 2 ) fitting a model given a subset . 2 Our approach is most related to the works of Bhatia et al . ( 2015 ) and Vainsencher et al . ( 2017 ) , and we contrast to these in the related work section . 1https : //github.com/carpedm20/DCGAN-tensorflow 2Our framework sounds initially similar to EM-style algorithms like k-means . Note however that EM needs to postulate a model for all the data points , while we search over a subset and do not worry about the loss on corrupted points . We are alternating between a simple search over subsets and a fitting problem on only the selected subset ; this is not an instance of EM . Our approach can be applied to any existing model that provides ( a ) confidence scores for samples , and ( b ) a way to train the model for a set of samples . Indeed in our experiments we use existing models and training code , but with iterative ( re ) selections of samples as above . In supervised settings , any model that maps features to responses typically associates a confidence to every sample , e.g. , in linear regression , given a model θ , the confidence in a sample ( xi , yi ) increases as its squared error ( yi − x > i θ ) 2 decreases . For a neural network classifier , on the other hand , the confidence can be measured by the cross entropy loss between the output probability distribution and the true distribution . In unsupervised settings , the confidence of a sample is given by its “ likelihood ” under the model . If we are lucky , e.g. , in Gaussian mixture models , this likelihood is explicit and can be evaluated . However , if there is no explicit likelihood , we need to rely on a surrogate . E.g. , in our GAN setting below , we rely on the output of the discriminator as a confidence measure . We also note an important caveat : this paper is not focused on ( and may not be effective for ) the settings of adversarial examples , or of gross corruptions of features . Rather , our focus is to address the issue of bad labeling or curation of data , as motivated above . OUR RESULTS AND PAPER OUTLINE . We show that our framework ( outlined in Section 3 ) can address the spurious sample problem , both theoretically and empirically and for both supervised and unsupervised problems : ( 1 ) Mixed linear regression ( Section 4 ) and Gaussian mixture models ( Section 5 ) : For each of these mixture problem settings , under the standard statistical assumptions , we establish both statistical consistency and linear convergence of our iteratively updated model parameters . This holds true from any initial point ; the convergence is to the mixture component that is closest to the initial point . ( 2 ) Neural network classifiers ( Section 6 ) : We investigate image classification in CIFAR-100 , CIFAR-10 and MNIST datasets ( each with a different and appropriate existing neural architecture ) in the setting where there are systematic errors in the labels – i.e . every bad example in one label class comes from the same other label class . We show that our framework can significantly improve the classification accuracy over the baseline method that ignores the existence of bad samples . ( 3 ) Deep generative models ( Section 7 ) : We show both quantitatively and qualitatively the improvement of using our algorithm for image generation when the dataset consists of two types of images . Specifically , investigate training a GAN to ( a ) generate face images as in the “ intended ” Celeb-A dataset , but when the given training data has a fraction of the samples being the “ spurious ” CIFAR-10 images , and ( b ) generate intended MNIST digits when training data contains a fraction of samples from the spurious Fashion-MNIST dataset . A crucial innovation here is that the output of the discriminator network in the GAN can be used as a confidence metric for the training samples . In Section 8 , we discuss the positives and negatives of our results , underlying intuition and also some future directions . All of our experiments were done on clearly identified publicly available neural network architectures , datasets , and training software . We also clearly describe our error and noise effects ; we expect them to be easily reproducible . 2 RELATED WORK . Alternating minimization for robustness Two works are most related to ours . Bhatia et al . ( 2015 ) proposes an iterative hard thresholding algorithm for adversarial noise in linear regression responses ; they establish statistical guarantees and linear convergence under this harder noise model . We focus on multiple mixed linear regression , a different setting that results in an easier noise model , and we provide convergence guarantee for recovering any single mixture component starting from a local region . In addition we also prove these for Gaussian mixture models , which is not studied in their work . Vainsencher et al . ( 2017 ) proposes what is essentially a soft version of our method , and proves local convergence and generalization . However they do not have any initialization , and hence no global or consistency guarantees . Neither of these empirically explore the overall approach for more complex / neural network models . Robust regression Recent work on robust regression consider strong robustness in terms of both the inputs and the outputs and provide guarantees for constant corruption ratio Diakonikolas et al . ( 2018 ) ; Prasad et al . ( 2018 ) ; Klivans et al . ( 2018 ) . Chen et al . ( 2013 ) ; Balakrishnan et al . ( 2017a ) ; Liu et al . ( 2018 ) study the high dimensional setting under this regime . These algorithms usually require much more computation , compared with methods for dealing with noisy outputs , e.g. , Bhatia et al . ( 2015 ) . Another type of robustness considers heteroscedastic noise for every sample . From the learning perspective , Anava & Mannor ( 2016 ) ; Chaudhuri et al . ( 2017 ) both require strong constraints on the noise model which largely reduces the degree of freedom . Anava & Mannor ( 2016 ) considers the online learning setting , while Chaudhuri et al . ( 2017 ) considers the active learning setting . Mixed linear regression Alternating minimization type algorithms are used for mixed linear regression with convergence guarantee Yi et al . ( 2014 ) . Chen et al . ( 2014 ) provides a convex formulation for the problem . Balakrishnan et al . ( 2017b ) shows expectation-maximization ( EM ) algorithm is provably effective for a set of tasks including the basic setting of mixed linear regression and Gaussian mixture model . Sedghi et al . ( 2016 ) extends to the multiple component setting , where provable tensor decomposition methods are used . More recently , Li & Liang ( 2018 ) gives a more general result for learning mixtures of linear regression . On the other hand , Ray et al . ( 2018 ) studies the problem of finding a single component in mixed linear regression problem using side information . Noisy labels Classification tasks with noisy labels are also of wide interest . Frénay et al . ( 2014 ) gives an overview of the related methods . Theoretical guarantee for noisy binary classification has been studied under different settings Scott et al . ( 2013 ) ; Natarajan et al . ( 2013 ) ; Menon et al . ( 2016 ) . More recently , noisy label problems have been studied for deep neural network based models . Reed et al . ( 2014 ) and Malach & Shalev-Shwartz ( 2017 ) develop the idea of bootstrapping and query-bycommittee into neural network based models . On the other hand , Khetan et al . ( 2018 ) and Zhang & Sabuncu ( 2018 ) provide new losses for training under the noise . Sukhbaatar & Fergus ( 2014 ) adds a noise layer into the training process , while Ren et al . ( 2018b ) provides a meta-algorithm for learning the weights of all samples by referencing to a clean subset of validation data during training . Ren et al . ( 2018a ) consider building robust classifiers for the positive-unlabeled classification problem using l0 norm penalization . Others Bora et al . ( 2018 ) studies the problem of noisy images for GANs . In their setting , all images are of low quality under some measurements , while in our problem , we consider the image dataset consists of ( possibly more than ) two types of images , which is different . There are also studies that consider noisy samples more generally . The classical RANSAC method Fischler & Bolles ( 1981 ) provides a paradigm for fitting a model to experimental data , and can be potentially used for the noisy data setting . However , the algorithm requires multiple times of random sampling to find a valid consensus score , which is conceptually different from our idea of doing iterative filtering . EM algorithm Moon ( 1996 ) can also be used for detecting noisy samples , e.g. , in the setting of mixture models Balakrishnan et al . ( 2017b ) . However , our algorithm does not need to know the likelihood for the bad samples for the iterative updates .

Summary: This paper introduces a framework for situation when the training samples are not pure. The idea is a simple approach by training a model and removing a portion of examples from the training set based on the loss of the model. The authors provide some theoretical study on two models: linear regression and Gaussian mixture model and utilize deep neural network to show their framework performs well experimentally.