NADS: Neural Architecture Distribution Search for Uncertainty Awareness

Machine learning systems often encounter Out-of-Distribution ( OoD ) errors when dealing with testing data coming from a different distribution from the one used for training . With their growing use in critical applications , it becomes important to develop systems that are able to accurately quantify its predictive uncertainty and screen out these anomalous inputs . However , unlike standard learning tasks , there is currently no well established guiding principle for designing architectures that can accurately quantify uncertainty . Moreover , commonly used OoD detection approaches are prone to errors and even sometimes assign higher likelihoods to OoD samples . To address these problems , we first seek to identify guiding principles for designing uncertainty-aware architectures , by proposing Neural Architecture Distribution Search ( NADS ) . Unlike standard neural architecture search methods which seek for a single best performing architecture , NADS searches for a distribution of architectures that perform well on a given task , allowing us to identify building blocks common among all uncertainty aware architectures . With this formulation , we are able to optimize a stochastic outlier detection objective and construct an ensemble of models to perform OoD detection . We perform multiple OoD detection experiments and observe that our NADS performs favorably compared to state-of-the-art OoD detection methods . 1 INTRODUCTION . Detecting anomalous data is crucial for safely applying machine learning in autonomous systems for critical applications and for AI safety ( Amodei et al. , 2016 ) . Such anomalous data can come in settings such as in autonomous driving ( Kendall & Gal , 2017 ; NHTSA , 2017 ) , disease monitoring ( Hendrycks & Gimpel , 2016 ) , and fault detection ( Hendrycks et al. , 2019b ) . In these situations , it is important for these systems to reliably detect abnormal inputs so that their occurrence can be overseen by a human , or the system can proceed using a more conservative policy . The widespread use of deep learning models within these autonomous systems have aggravated this issue . Despite having high performance in many predictive tasks , deep networks tend to give high confidence predictions on Out-of-Distribution ( OoD ) data ( Goodfellow et al. , 2015 ; Nguyen et al. , 2015 ) . Moreover , commonly used OoD detection approaches are prone to errors and even assign higher likelihoods to samples from other datasets ( Lee et al. , 2018 ; Hendrycks & Gimpel , 2016 ) . Unlike common machine learning tasks such as image classification , segmentation , and speech recognition , there are currently no well established guidelines for designing architectures that can accurately screen out OoD data and quantify its uncertainty . Such a gap in our knowledge makes Neural Architecture Search ( NAS ) a promising option to explore the better design of uncertaintyaware models ( Elsken et al. , 2018 ) . NAS algorithms attempt to find an optimal neural network architecture for a specific task . Existing efforts have primarily focused on searching for architectures that perform well on image classification or segmentation . However , it is unclear whether architecture components that are beneficial for image classification and segmentation models would also lead to better uncertainty quantification and thereafter be effective for OoD detection . Moreover , previous work on deep uncertainty quantification shows that ensembles can help calibrate OoD classifier based methods , as well as improve OoD detection performance of likelihood estimation models ( Lakshminarayanan et al. , 2017 ; Choi & Jang , 2018 ) . Because of this , instead of a single best performing architecture for uncertainty awareness , one might consider a distribution of wellperforming architectures . Along this direction , designing an optimization objective which leads to uncertainty-aware models is also not straightforward . With no access to labels , unsupervised/self-supervised generative models which maximize the likelihood of in-distribution data become the primary tools for uncertainty quantification ( Hendrycks et al. , 2019a ) . However , these models counter-intuitively assign high likelihoods to OoD data ( Nalisnick et al. , 2019a ; Choi & Jang , 2018 ; Hendrycks et al. , 2019a ; Shafaei et al. ) . Because of this , maximizing the log-likelihood is inadequate for OoD detection . On the other hand , Choi & Jang ( 2018 ) proposed using the Widely Applicable Information Criterion ( WAIC ) ( Watanabe , 2013 ) , a penalized log-likelihood score , as the OoD detection criterion . However , the score was approximated using an ensemble of models that was trained on maximizing the likelihood and did not directly optimize the WAIC score . To this end , we propose a novel Neural Architecture Distribution Search ( NADS ) framework to identify common building blocks that naturally incorporate model uncertainty quantification and compose good OoD detection models . NADS is an architecture search method designed to search for a distribution of well-performing architectures , instead of a single best architecture by formulating the architecture search problem as a stochastic optimization problem . Using NADS , we optimize the WAIC score of the architecture distribution , a score that was shown to be robust towards model uncertainty . Such an optimization problem with a stochastic objective over a probability distribution of architectures is unamenable to traditional NAS optimization strategies . We make this optimization problem tractable by taking advantage of weight sharing between different architectures , as well as through a parameterization of the architecture distribution , which allows for a continuous relaxation of the discrete search problem . Using the learned posterior architecture distribution , we construct a Bayesian ensemble of deep models to perform OoD detection . Finally , we perform multiple OoD detection experiments to show the efficacy of our proposed method . 2 BACKGROUND . 2.1 NEURAL ARCHITECTURE SEARCH . Neural Architecture Search ( NAS ) algorithms aim to automatically discover an optimal neural network architecture instead of using a hand-crafted one for a specific task . Previous work on NAS has achieved successes in image classification ( Pham et al. , 2018 ) , image segmentation ( Liu et al. , 2019 ) , object detection ( Ghiasi et al. , 2019 ) , structured prediction ( Chen et al. , 2018 ) , and generative adversarial networks ( Gong et al. , 2019 ) . However , there has been no NAS algorithm developed for uncertainty quantificaton and OoD detection . NAS consists of three components : the proxy task , the search space , and the optimization algorithm . Prior work in specifying the search space either searches for an entire architecture directly , or searches for small cells and arrange them in a pre-defined way . Optimization algorithms that have been used for NAS include reinforcement learning ( Baker et al. , 2017 ; Zoph et al. , 2018 ; Zhong et al. , 2018 ; Zoph & Le , 2016 ) , Bayesian optimization ( Jin et al. , 2018 ) , random search ( Chen et al. , 2018 ) , Monte Carlo tree search ( Negrinho & Gordon , 2017 ) , and gradient-based optimization methods ( Liu et al. , 2018b ; Ahmed & Torresani , 2018 ) . To efficiently evaluate the performance of discovered architectures and guide the search , the design of the proxy task is critical . Existing proxy tasks include leveraging shared parameters ( Pham et al. , 2018 ) , predicting performance using a surrogate model ( Liu et al. , 2018a ) , and early stopping ( Zoph et al. , 2018 ; Chen et al. , 2018 ) . To our best knowledge , all existing NAS algorithms seek a single best performing architecture . In comparison , searching for a distribution of architectures allows us to analyze the common building blocks that all of the candidate architectures have . Moreover , this technique can also complement ensemble methods by creating a more diverse set of models for the ensemble decision , an important ingredient for deep uncertainty quantification ( Lakshminarayanan et al. , 2017 ) . 2.2 UNCERTAINTY QUANTIFICATION AND OUT-OF-DISTRIBUTION DETECTION . Prior work on uncertainty quantification and OoD detection for deep models can be divided into model-dependent ( Lakshminarayanan et al. , 2017 ; Gal & Ghahramani , 2016 ; Liang et al. , 2017 ) , and model-independent techniques ( Dinh et al. , 2016 ; Germain et al. , 2015 ; Oord et al. , 2016 ) . Model-dependent techniques aim to yield confidence measures p ( y|x ) for a model ’ s prediction y when given input data x . However , a limitation of model-dependent OoD detection is that they may discard information regarding the data distribution p ( x ) when learning the task specific model p ( y|x ) . This could happen when certain features of the data are irrelevant for the predictive task , causing information loss regarding the data distribution p ( x ) . Moreover , existing methods to calibrate model uncertainty estimates assume access to OoD data during training ( Lee et al. , 2018 ; Hendrycks et al. , 2019b ) . Although the OoD data may not come from the testing distribution , this assumes that the structure of OoD data is known ahead of time , which can be incorrect in settings such as active/online learning where new training distributions are regularly encountered . On the other hand , model-independent techniques seek to estimate the likelihood of the data distribution p ( x ) . These techniques include Variational Autoencoders ( VAEs ) ( Kingma & Welling , 2013 ) , generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) , autoregressive models ( Germain et al. , 2015 ; Oord et al. , 2016 ) , and invertible flow-based models ( Dinh et al. , 2016 ; Kingma & Dhariwal , 2018 ) . Among these techniques , invertible models offer exact computation of the data likelihood , making them attractive for likelihood estimation . Moreover , they do not require OoD samples during training , making them applicable to any OoD detection scenario . Thus in this paper , we focus on searching for invertible flow-based architectures , though the presented techniques are also applicable to other likelihood estimation models . Along this direction , recent work has discovered that likelihood-based models can assign higher likelihoods to OoD data compared to in-distribution data ( Nalisnick et al. , 2019a ; Choi & Jang , 2018 ) ( see Figure 13 for an example ) . One hypothesis for such a phenomenon is that most data points lie within the typical set of a distribution , instead of the region of high likelihood ( Nalisnick et al. , 2019b ) . Thus , Nalisnick et al . ( 2019b ) recommend to estimate the entropy using multiple data samples to screen out OoD data instead of using the likelihood . Other uncertainty quantification formulations can also be related to entropy estimation ( Choi & Jang , 2018 ; Lakshminarayanan et al. , 2017 ) . However , it is not always realistic to test multiple data points in practical data streams , as testing data often come one sample at a time and are never well-organized into in-distribution or out-of-distribution groups . With this in mind , model ensembling becomes a natural consideration to formulate entropy estimation . Instead of averaging the entropy over multiple data points , model ensembles produce multiple estimates of the data likelihood , thus “ augmenting ” one data point into as many data points as needed to reliably estimate the entropy . However , care must be taken to ensure that the model ensemble produces likelihood estimates that agree with one another on in-distribution data , while also being diverse enough to discriminate OoD data likelihoods . In what follows , we propose NADS as a method that can identify distributions of architectures for uncertainty quantification . Using a loss function that accounts for the diversity of architectures within the distribution , NADS allows us to construct an ensemble of models that can reliably detect OoD data .

Summary: This paper considers the neural architecture search (NAS) problem under the out-of-distribution (OoD) environment. As the OoD problem is not visited in the current NAS literature, this paper proposes replacements for each of the three standard components in NAS, i.e., the proxy task, the search space, and the optimization algorithm; and each replacement is built upon an ensemble of existing techniques. Experiments are further verified on CelebA, CIFAR-10, SVHN, and MNIST.