ChoiceNet: Robust Learning by  Revealing Output Correlations

1 INTRODUCTION . Training a deep neural network requires immense amounts of training data which are often collected using crowdsourcing methods , such as Amazon ’ s Mechanical Turk ( AMT ) . However , in practice , the crowd-sourced labels are often noisy ( Bi et al. , 2014 ) . Furthermore , deep neural networks are vulnerable to over-fitting given the noisy training data in that they are capable of memorizing the entire dataset even with inconsistent labels , leading to a poor generalization performance ( Zhang et al. , 2016 ) . Assuming that a training dataset is generated from a mixture of a target distribution and other distributions , we address this problem through the principled idea of revealing the correlation between the target distribution and the other distributions . We present a framework for robust learning which is applicable to arbitrary neural network architectures such as convolutional neural networks ( He et al. , 2016a ) or recurrent neural networks ( Chung et al. , 2014 ) . We call this framework ChoiceNet . Throughout this paper , we aim to address the following questions : 1 . How can we measure the quality of training data in a principled manner ? 2 . In the presence of inconsistent outputs , how can we infer the target distribution in a scalable manner ? Traditionally , noisy outputs are handled by modeling additive random distributions , often leading to robust loss functions ( Hampel et al. , 2011 ) . However , we argue that these approaches are too restrictive when handling severe outliers or inconsistencies in the datasets . To address the first question , we leverage the concept of a correlation . Precisely , we measure the quality of training data using the correlation between the target distribution and the data generating distribution . However , estimating the correct correlation requires an access to a target distribution , whereas learning the correct target distribution requires knowing the correlation between the distributions to be known , making it a chicken-and-egg problem . To address the second question , we simultaneously estimate the target distribution as well as the correlation in an end-to-end-manner using stochastic gradient decent methods , in this case Adam ( Kingma & Ba , 2014 ) , to achieve scalability . The cornerstone of the proposed method is a mixture of correlated density network ( MCDN ) block . First , we present a Cholesky transform method for sampling the weights of a neural network that enables us to model correlated outputs . We also present an effective regularizer to train ChoiceNet . To the best of our knowledge , this represents the first approach simultaneously to infer the target distribution and the output correlations using a neural network in an end-to-end manner . Revealing the output correlations was proposed in earlier work ( Bonilla et al. , 2008 ) , in which a multi-task Gaussian process prediction ( MTGPP ) model is proposed . In particular , MTGPP used correlated Gaussian processes to model multiple tasks by learning a free-form cross-covariance matrix . However , due to the multi-task learning setting , it is not suitable for learning a single target function . In other work ( Choi et al. , 2016 ) , a leverage optimization method which optimizes the leverage of each demonstrations is proposed . Unlike a former study ( Bonilla et al. , 2008 ) , the latter ( Choi et al. , 2016 ) focused on inferring a single expert policy by incorporating a sparsity constraint by assuming that the most demonstrations are collected from a skillful consistent expert . However , these methods suffer from the scalability issues when applying on large-scale tasks . ChoiceNet is initially applied to a synthetic regression task , where we demonstrate its robustness to extreme outliers and ability to distinguish the target distribution and noise distributions . We then apply it to a behavior cloning scenario where the demonstrations are collected from both an expert and an adversarial policies . Subsequently , we move on to the classification tasks using the MNIST and CIFAR-10 datasets . We show that the proposed method outperforms existing baseline methods in terms of robustness with regard to the handling different types of noisy labels . 2 RELATED WORK . Recently , robustness in deep learning has been actively studied ( Fawzi et al. , 2017 ) as deep neural networks are being applied to diverse tasks involving real-world applications such as autonomous driving ( Paden et al. , 2016 ) or medical diagnosis ( Gulshan et al. , 2016 ) where a simple malfunction can have catastrophic results ( AP & REUTERS , 2016 ) . Existing work for handing noisy training data can be categorized into four groups : small-loss tricks Jiang et al . ( 2017 ) ; Ren et al . ( 2018 ) ; Han et al . ( 2018 ) ; Malach & Shalev-Shwartz ( 2017 ) , estimating label corruptions Patrini et al . ( 2017 ) ; Goldberger & Ben-Reuven ( 2017 ) ; Sukhbaatar et al . ( 2014 ) ; Bekker & Goldberger ( 2016 ) ; Hendrycks et al . ( 2018 ) ; Veit et al . ( 2017 ) , using robust loss functions Natarajan et al . ( 2013 ) ; Belagiannis et al . ( 2015 ) , and explicit and implicit regularization methods Reed et al . ( 2014 ) ; Lee ( 2013 ) ; Goodfellow et al . ( 2016 ) ; Xie et al . ( 2016 ) ; Tokozume et al . ( 2018 ) ; Zhang et al . ( 2017 ) ; Miyato et al . ( 2018 ) ; Tarvainen & Valpola ( 2017 ) ; Laine & Aila ( 2017 ) . Our proposed method is mostly related to the robust loss function approach but can not fully categorized into this group in that we present a novel architecture , a mixture of correlated density network block , for achieving robustness based on the correlation estimation . First of all , the small-loss tricks selectively focus on training instances based on a certain criterion such as having small cost values Han et al . ( 2018 ) . ( Malach & Shalev-Shwartz , 2017 ) proposed a meta-algorithm for tackling the noisy label problem by training two networks only when the predictions of the two networks disagree , where selecting a proper network from among the two networks can be done using an additional clean dataset . ( Ren et al. , 2018 ) reweighs the weight of each training instance using a small amount of clean validation data . MentorNet Jiang et al . ( 2017 ) concentrated on the training of an additional neural network , which assigns a weight to each instance of training data to supervise the training of a base network , termed StudentNet , to overcome the over-fitting of corrupt training data . Recent work Han et al . ( 2018 ) presented Co-teaching by maintaining two separate networks where each network is trained with small-loss instances selected from its peer network . The second group of estimating label corruption information is mainly presented for classification tasks where training labels are assumed to be corrupted with a possibly unknown corruption matrix . An earlier study in Bekker & Goldberger ( 2016 ) proposed an extra layer for the modeling of output noises . Jindal et al . ( 2016 ) extended the aforementioned approach by adding an additional noise adaptation layer with aggressive dropout regularization . A similar method was then proposed in Patrini et al . ( 2017 ) which initially estimated the label corruption matrix with a learned classifier and used the corruption matrix to fine-tune the classifier . Other researchers ( Goldberger & Ben-Reuven , 2017 ) presented a robust training method that mimics the EM algorithm to train a neural network , with the label noise modeled as an additional softmax layer , similar to earlier work ( Jindal et al. , 2016 ) . A self-error-correcting network was also presented ( Liu et al. , 2017 ) . It switches the training labels based on the learned model at the beginning stages by assuming that the deep model is more accurate during the earlier stage of training . Researchers have also focussed on using robust loss functions ; ( Natarajan et al. , 2013 ) studied problem of binary classification in the presence of random labels and presented a robust surrogate loss function for handling noisy labels . Existing loss functions for classification were studied ( Ghosh et al. , 2017 ) , with the results showing that the mean absolute value of error is inherently robust to label noise . In other work ( Belagiannis et al. , 2015 ) , a robust loss function for deep regression tasks were proposed using Tukey ’ s biweight function with median absolute deviation of the residuals . The last group focusses on using implicit or explicit regularization methods while training . Adding small label noises while training is known to be beneficial to training , as it can be regarded as an effective regularization method ( Lee , 2013 ; Goodfellow et al. , 2016 ) . Similar methods have been proposed to tackle noisy outputs . A bootstrapping method ( Reed et al. , 2014 ) which train a neural network with a convex combination of the output of the current network and the noisy target was proposed . Xie et al . ( 2016 ) proposed DisturbLabel , a simple method which randomly replaces a percentage of the labels with incorrect values for each iteration . Mixing both input and output data was also proposed ( Tokozume et al. , 2018 ; Zhang et al. , 2017 ) . One study ( Zhang et al. , 2017 ) considered the image recognition problem under label noise and the other ( Tokozume et al. , 2018 ) focused on a sound recognition problem . Temporal ensemble was proposed in Laine & Aila ( 2017 ) where an unsupervised loss term of fitting the output of of an augmented input to the augmented target updated with an exponential moving average . ( Tarvainen & Valpola , 2017 ) extends the temporal ensemble in Laine & Aila ( 2017 ) by introducing a consistency cost function that minimizes the distance between the weights of the student model and the teacher model . ( Miyato et al. , 2018 ) presented new regularization method based on virtual adversarial loss which measures the smoothness of conditional label distribution given input . Minimizing the virtual adversarial loss has a regularizing effect in that it makes the model smooth at each data point . The foundation of the proposed method is the mixture of correlated density network ( MCDN ) block where the output distribution is modeled using a mixture of correlated distributions . Modeling correlations of output training data has been actively studied in light of Gaussian processes ( Rasmussen , 2006 ) . MTGPP ( Bonilla et al. , 2008 ) that models the correlations of multiple tasks via Gaussian process regression was proposed in the context of multi-task setting . Choi et al . ( 2016 ) proposed a robust learning from demonstration method using a sparse constrained leverage optimization method which estimates the correlation between training outputs and showed its robustness compared to several baselines . On the other hand , Platanios et al . ( 2016 ) presented a graphical model that can model the structure of the error rates of learned classifiers using non-parametric Bayesian methods . We would like to note that while our problem setting is similar to the latter study ( Choi et al. , 2016 ) , we propose end-to-end learning of both the target distribution and the correlation of each training data , thus offering , a clear advantage in terms of scalability .

Summary: This paper presents an apparently original method targeted toward models training in the presence of low-quality or corrupted data. To accomplish this they introduce a "mixture of correlated density network" (MCDN), which processes representations from a backbone network, and the MCDN models the corrupted data generating process. Evaluation is on a regression problem with an analytic function, two MuJoCo problems, MNIST, and CIFAR-10.