NADS: Neural Architecture Distribution Search for Uncertainty Awareness
Authors: Randy Ardywibowo
Shahin Boluki
Xinyu Gong
Zhangyang Wang
Xiaoning Qian
Venue:  (2020)
Keywords: Neural Architecture Search
Bayesian ensembling
out-of-distribution detection
uncertainty quantification
density estimation

Abstract: Machine learning systems often encounter Out-of-Distribution (OoD) errors when dealing with testing data coming from a different distribution from the one used for training. With their growing use in critical applications, it becomes important to develop systems that are able to accurately quantify its predictive uncertainty and screen out these anomalous inputs. However, unlike standard learning tasks, there is currently no well established guiding principle for designing architectures that can accurately quantify uncertainty. Moreover, commonly used OoD detection approaches are prone to errors and even sometimes assign higher likelihoods to OoD samples. To address these problems, we first seek to identify guiding principles for designing uncertainty-aware architectures, by proposing Neural Architecture Distribution Search (NADS). Unlike standard neural architecture search methods which seek for a single best performing architecture, NADS searches for a distribution of architectures that perform well on a given task, allowing us to identify building blocks common among all uncertainty aware architectures. With this formulation, we are able to optimize a stochastic outlier detection objective and construct an ensemble of models to perform OoD detection. We perform multiple OoD detection experiments and observe that our NADS performs favorably compared to state-of-the-art OoD detection methods.

Under review as a conference paper at ICLR 2020
NADS: NEURAL ARCHITECTURE DISTRIBUTION SEARCH FOR UNCERTAINTY AWARENESS
Anonymous authors Paper under double-blind review
ABSTRACT
Machine learning systems often encounter Out-of-Distribution (OoD) errors when dealing with testing data coming from different distribution from the used for training. With their growing use in critical applications, it becomes important develop systems that are able accurately quantify its predictive uncertainty and screen out these anomalous inputs. However, unlike standard learning tasks, there is currently no well established guiding principle for designing architectures that can accurately quantify uncertainty. Moreover, commonly used OoD detec- tion approaches are prone to errors and even sometimes assign higher likelihoods to Ool samples. To address these problems, we first seek to identify guiding prin- architectures, by proposing Neural Archi-
ciples for designing uncertainty- tecture Distribution Search (NADS). Unlike standard neural architecture search methods which seek for a single best performing architecture, NADS searches for distribution of architectures that perform well given task, allowing identify building blocks common among all uncertainty aware architectures, With able to optimize stochastic outlier detection objective
this formulation, and construct an ensemble of models to perform OoD detection. We perform mul- tiple OoD detection experiments and observe that our NADS performs favorably compared to state-of-the-art OoD detection methods.
INTRODUCTION
Detecting anomalous data crucial for safely applying machine learning in autonomous systems for critical applications and for Al safety (Amodei et al. 2016). Such anomalous data can come in settings such as in autonomous driving (Kendall Gal 2017 NHTSA 2017). disease monitor- ing (Hendrycks & Gimpel| 2016), and fault detection (Hendrycks et al. 2019b). In these situations, is important for these systems to reliably detect abnormal inputs so that their occurrence can be overseen by human, the system proceed using more conservative policy.
The widespread use of deep learning models within these autonomous systems have aggravated this issue. Despite having high performance in many predictive tasks, deep networks tend to give high confidence predictions on Out-of-Distribution (OoD) data (Goodfflloow et al.) 2015) Nguyen et al.] 2015). Moreover, commonly used OoD detection approaches are prone errors and even assign higher likelihoods to samples from other datasets (Lee et al. 2018 Hendrycks & Gimpel 2016).
Unlike common machine learning tasks such image classification, segmentation, and speech recognition, there are currently no well established guidelines for designing architectures that can accurately screen out OoD data and quantify uncertainty. Such a gap our knowledge makes Neural Architecture Search (NAS) a promising option to explore the better design of uncertainty- aware models (Elsken et al. 2018). NAS algorithms attempt to find an optimal neural network architecture for specific task. Existing efforts have primarily focused on searching for architec- tures that perform well on image classification or segmentation. However, it is unclear whether architecture components that are beneficial for image classification and segmentation models would also lead to better uncertainty quantification and thereafter be effective for OoD detection. More- over, previous work on deep uncertainty quantification shows that ensembles can help calibrate OoD classifier based methods, as well as improve OoD detection performance likelihood estimation models (Laaksmminarayanan al. 2017 Choi & Jang 2018). Because of this, instead of a single

---

Under review as a conference paper at ICLR 2020
best performing architecture for uncertainty awareness, one might consider distribution of well- performing architectures.
Along this direction, designing an optimization objective which leads to uncertainty-aware mod- els is also not straightforward. With no access to labels, seapperviied/ssll-suuuriiisieeei generative models which maximize the likelihood of in-distribution data become the primary tools for uncer- tainty quantification (Hendrycks et al., (2019)). However, these models counter-intuitively assign high likelihoods OoD data (Nalisnick et al. 2019a Choi & Jang 2018 Hendryyks et al.) 2019a] Shafaei et al.). Because of this, maximizing the log-likelihodd is inadequate for OoD detection. On
the other hand, Choi & Jang (2018) proposed using the Widely Applicable Information Criterion (WAIC) [Watanabel ) a penalized log-likelihood score, as the OoD detection criterion. How- ever, the score was approximated using an ensemble of models that was trained on maximizing the likelihood and did not directly optimize the WAIC score.
To this end, we propose novel Neural Architecture Distribution Search (NADS) framework to identify common building blocks that naturally incorporate model uncertainty quantification and compose good OoD detection models. NADS is an architecture search method designed to search for distribution of well-performing architectures, instead of single best architecture by formulating the architecture search problem as a stochastic optimization problem. Using NADS, we optimize the WAIC score of the architecture distribution, score that was shown to be robust towards model uncertainty. Such an optimization problem with a stochastic objective over a probability distribution of architeccurrs is unamenable to traditional NAS optimization strategies. We make this optimization problem tractable by taking advantage of weight sharing between different architectures, as well as through a parameterization of the architecture distribution, which allows for a continuous relaxation of the discrete search problem. Using the learned posterior architecture distribution, we construct a Bayesian ensemble of deep models to perform OoD detection. Finally, we perform multiple OoD detection experiments to show the efficacy of our proposed method.
BACKGROUND
2.1
NEURAL ARCHITECTURE SEARCH
Neural Architecture Search (NAS) algorithms aim to automatically discover an optimal neural net- work architecture instead of using a hand-crafted one for a specific task. Previous work on NAS has achieved successes in image classification (Pham et al. 2018), image segmentation (Liu et al.) 2019), object detection (Ghiasi et al. 2019), structured prediction (Chen et al. 2018), and generative adversarial networks (Gong et al. 2019). However, there has been no NAS algorithm developed for uncertainty quantificaton and OoD detection.
NAS consists of three components: the proxy task, the search space, and the optimization algo- rithm. Prior work in specifying the search space either searches for an entire architecture directly, or searches for small cells and arrange them in pre-defined way. Optimization algorithms that have been used for NAS include reinforcement learning (Baker et al., 2017 Zoph et al. 2018 Zhong 2018), random search (Chen et al. 2018). Monte Carlo tree search (Negrinho & Gordon 2017), and gradient- based optimization
al. 2018 Zoph & Le 2016). Bayesian optimization (Jin
methods (Liu et al.] 2018b] Ahmed & Torresani] [0188). To efficiently evaluate the performance of discovered architectures and guide the search, the design of the proxy task is critical. Existing proxy tasks include leveraging shared parameters (Pham et al. 2018). predicting performance using surrogate model (Liu et al. 2018a), and early stopping (Zoph et al. 2018 Chen et al. 2018).
To our best knowledge, all existing NAS algorithms seek a single best performing architecture. In comparison, searching for distribution of architectures allows to analyze the common building blocks that all of the candidate architectures have. Moreover, this technique can also complement ensemble methods by creating a more diverse set of models for the ensemble decision, important ingredient for deep uncertainty quantification Lakshminarayanan et al. 2017).
2.2 UNCERTAINTY QUANTIFICATION AND OUT-OF COT-OF-DISTRIBUTION DETECTION
Prior work on uncertainty quantification and OoD detection for deep models can be divided into model-dependent (Lakshminarayanan al. 2017 Gal & Ghahramani 2016 Liang et al. 2017).
2

---

Under review as a conference paper at ICLR 2020
and model-indeependen techniques (Dinh al. 2016: Germain al. 2015 Oord al. 2016
Model-dependent techniques aim to yield confidence measures p(y for a model's 's prediction when given input data However, limitation of model-dependent OoD detection is that they may discard information regarding the data distribution p(æ) when learning the task specific model p(ylaz). This could happen when certain features of the data are irrelevant for the predictive task, causing information loss regarding the data distribution p(x). Moreover, existing methods to cal- ibrate model uncertainty estimates assume access to OoD data during training (Lee al., 2018 Hendrycks et al. 2019b). Although the OoD data may not come from the testing distribution, this assumes that the structure of OoD data is known ahead of time, which can be incorrect in settings such as active/online learning where new training distributions are regularly encountered.
On the other hand, inneel-indeeendentt techniques seek to estimate the likelihood of the data distribu- tion p(x:). These techniques include Variational Autoencoders (VAEs) (Kingma & Welling 2013), generative adversarial networks (GANs) (Goodfellow et al.) 2014), autoregressive models (Germain et al., 2015 Oord al. 6 and invertible flow-based models (Dinh et al. 2016 Kingma & the data likelihood, making them attractive for likelihood estimation. Moreover, they do not require OoD samples during training, making them applicable to any OoD detection scenario. Thus in this paper, focus searching for invertible flow-based architectures, though the presented techniques also applicable to other likelihood estimation models.
Dhariwal 2018). Among these techniques, invertible models offer exact computation
Along this direction, recent work has discovered that likelihood-based models assign higher likelihoods to OoD data compared to in-distribution data (Nalisnick et al., 2019a Choi & Jang. 2018 (see Figure 13 for example). One hypothesis for such a phenomenon points lie within the typical set of distribution, instead of the region of high likelihood (Nalisnick) et al. (2019b) recommend to estimate the entropy using multiple data samples to screen out OoD data instead of using the likelihood. Other uncertainty quantification formulations can also be related to entropy estimation (Choi & Jang 2018 Lakshminarayanal et al.) 2017). However, it is not always realistic to test multiple data points in practical data streams, as testing data often come one sample at a time and are never well-organized into in-distribution or out-of-distribution groups.
that most data
ct al. 2019b). Thus,
Thus,NNalisnick Nalisnick
With this in mind, model ensembling becomes a natural consideration to formulate entropy estima- tion. Instead of averaging the entropy multiple data points, model ensembles produce multiple estimates of the data likelihood, thus "augmenting" one data point into as many data points as needed to reliably estimate the entropy. However, care must be taken to ensure that the model en- semble produces likelihood estimates that agree with one another on in-distribution data, while also being diverse enough to discriminate OoD data likelihoods. In what follows, propose NADS as a method that can identify distributions of architectures for uncertainty quantification. Using a loss function that accounts for the diversity of architectures within the distribution, NADS allows to construct an ensemble of models that can reliably detect OoD data.
3 NEURAL ARCHITECTURE DISTRIBUTION SEARCH (NADS)
Putting Neural Architecture Distribution Search (NADS) under common NAS framework (Elsken) jet al. 2018), we break down our search formulation into three main components: the proxy task, the search space, and the optimization method. Specifying these components for NADS with the ulti- mate goal of uncertainty quantification for OoD detection is not immediately obvious. For example, naively using data likelihood maximization Nalisnick et al. (2019)), with models assigning higher likelihoods to OoD data. On the other hand, the search space needs be large enough to include a diverse range of architectures, yet still allow- ing a search algorithm to traverse it efficiently. In the following sections, we motivate our decision on these three choices and describe these components for NADS in detail.
proxy task would run into the issue pointed out by
3.1 PROXY TASK
The first component of NADS is the training objective that guides the neural architecture search. Different from existing NAS methods, our aim is to derive an ensemble of deep models to im- prove model uncertainty quantification and OoD detection. To this end, instead of searching for
3

---

Under review as a conference paper at ICLR 2020
Affine coupling
Concalenntee
step flo
Invettbble
squasze
aconomm
split
Op
Op
Step flow
x U
ReuU Come
ReLU Core-
squetze
Figure 1: Search space of a single block in the architecture
architectures which maximize the likelihood of in-distribution data, which may cause our model to incorrectly assign high likelihoods to OoD data, we instead seek architectures that can perform en- tropy estimation by maximizing the Widely Applicable Information Criteria (WAIC) of the training data. The WAIC score is Bayesian adjusted metric to calculate the marginal likelihood (Watanabe, 2013). This metric has been shown by Choi Jang (2018) to be robust towards the pitfall causing likelihood estimation models to assign high likelihoods to OoD data. The score is defined as follows:
WAIC(x) Ba-p(o) P.. Va-p(o) Po Pa (z)]
(1)
Here, B|:] and V[-] denote expectation and variance respectively, which are taken over all architec- tures sampled from the posterior architecture distribution p(a). Such strategy captures model uncertainty in a Bayesian fashion, improving OoD detection. Intuitively, minimizing the variance of training data likelihoods allows its likelihood distribution to remain tight which, by proxy, minimizes the overlap of in-distribution and out-of-distribution likelihoods, thus making them separable.
Under this objective function, we search for optimal distribution of network architectures p(a) by deriving the corresponding parameters that characterize p(ax). Because the score requires aggregat- ing the results from multiple architectures a, optimizing such a score using existing search methods can be intractable, as they typically only consider a single architecture at time. Later, we will show how to circumvent this problem our optimization formulation.
3.2 SEARCH SPACE
NADS constructs layer-wise search space with a pre-defined macro-architecture, where each layer can have different architecture component. Such a search space has been studied by (Zoph & Lej 2016 Liu et al. 2018b Real et al. 2219), where it shows be both expressive and scalable/efficient.
The macro-architecture closely follows the Glow architecture presented Kingma & Dhariwal 2018). Here, each layer consists of an actnorm, invertible 1 x 1 convolution, and affine coupling layer. Instead of pre-defining the affine coupling layer, we allow it to be optimized by our architecture search. The search space can be viewed in Figure [I] Here, each operational block of the affine coupling layer is selected from a list of candidate operations that include 3 x 3 average pooling, 3 max pooling, skip-connections, 3 and 5 separable convolutions, 3 x 3 and 5 x 5 dilated convolutions, identity, and zero. We choose this search space to answer the following questions towards better architectures for OoD detection:
What topology of connections between layers is best for uncertainty quantification? Traditional likelihood estimation architectures focus only feedforward connections without adding any skip-connection structures. However, adding skip-connections may improve optimization speed and stability.
Are more features/filters better for OoD detection? More feature outputs of each layer should lead to more expressive model. However, if many of those features are redundant, it may slow down learning, overfitting nuisances and resulting in sub-optimal models.
Which operations are best for OoD detection? Intuitively, operations such max/average pool- ing should not be preferred, as they discard information of the original data point "too aggres- sively". However, this intuition remains be confirmed.
4

---

Under review as a conference paper at ICLR 2020
3.3 OPTIMIZATION
Having specified our proxy task and search space, we now describe our optimization method for NADS. Several difficulties arise when attempting to optimize this setup. First, optimizing p(a), distribution over high-dimensional discrete random variables a, jointly with the network parameters intractable as, worst, each network's optimal parameters would need be individually identi- fied. Second, even if we relax the discrete search space, the objective function involves computing an expectation and variance over all possible discrete architectures. To alleviate these problems, we first introduce continuous relaxation for the discrete search space, allowing us to approximately optimize the discrete architectures through backpropagation and weight sharing between common architecture blocks. We then approximate the stochastic objective by using Monte Carlo samples to estimate the expectation and variance.
Specifically, let A denote our discrete architecture search space and a A be an architecture in this space. Let lg- (a)) be the loss function of architecture a with its parameters set to 0* such that satisfies 0 arg min, (0) a:) for some loss function ((-). We are interested in finding distribution (a(a)) parameterized by ø that minimizes the expected loss of an architecture a sampled from it. We denote this loss function L(Ø) Ea-ps(a) [-- (a)]. For our NADS, this loss function is the negative WAIC score of in-distribution data [( = Σ WAIC(=;).
Solving L(d) for arbitrary parameterizations of (a(a)) can be intractable, as the inner loss function le- (a) involves searching for the optimal parameters 6" of a neural network architecture a. More- over, the outer expectation causes backpropagation to be inapplicable due to the discrete random architecture variable We adopt tractable optimization paradigm to circumvent this problem through a specific reparameterization of the architecture distribution (a()), allowing us to back- propagate through the outer expectation and jointly optimize & and 6.
For clarity of exposition, we first focus on sampling an architecture with a single hidden layer. In this setting, we intend to find probability vector i Øx] with which we randomly pick a single operation from a list of K different operations [or.... (o1..... ox] Let = jbt.....br by] denote the random categorical indicator vector sampled from Φb, where is if the 7th operation is chosen, and zero otherwise. Note that b is equivalent to the discrete architecture variable a in this setting. With this, we write the random output y of the hidden layer given input
y
by 0z (xx)
To make optimization tractable, we relax the discrete mask to be a continuous random variable using the Gumbeel-Softmax reparameterization (Gumbel 1954, MMddison et al. 2014) follows:
b. cxp((log(0)) + 9:)T)
for i i= 1,..., K.
(3)
Σ ==1 exp((1og(0)) + 9:)/r)
Here, 91 ....... ~ ~~- log(- log(u)) where u ~ Unif(O, 1), and 7 is temperature parameter. For low values of r, b approaches sample of a categorical random variable, recovering the original discrete problem. While for high values, b will equally weigh the K operations (Jang et al. (0166). Using this, we can compute backpropagation by approximating the gradient of the discrete architecture a with the gradient of the continuously relaxed categorical random variable b, as Va,aa = Vs,eb R Ve.sb. With this backpropagation gradient defined, generalizing the above setting to architectures with multiple layers simply involves recursively applying the above gradient relaxation to each layer.
With this formulation, can gradually remove the continuous relaxation and sample discrete archi- tectures by annealing the temperature parameter т. With this, we are able to optimize the architecture distribution (a(a) and sample candidate architectures for further retraining, finetuning, or evalua- tion. By sampling M architectures from the distribution, we are able to approximate the WAIC score expectation and variance terms as:
A5 (log Pa, (xx))) (2
M
-L(d) wAIC( Σ
Σ log Pa, (x,)
(E
log Pay, (2::)
5

---

Under review as a conference paper at ICLR 2020
Concatenate
Concatenate
Concatenate
Concatenate
Aug
Avg
Aug Pool
Sid Conw
5x5 D1 Sorv
Si5 DI Cone
Sd Can
RCB
RCB
RCB
RCB
RCB
RCB
Block
Block 2
Block
Block 4
Summary of our architecture search findings: the most likely architecture structure for each block K found by NADS.
Figure 2:
3.4 SEARCH RESULTS
We applied our architecture search on five datasets: CelebA (Liu et al.), CIFAR-10, CIFAR- 100, (Krizhevsky et al. 2009). SVHN (Netzer et al.) 2011). and MNIST ([eCun)) In all experiments, we used the Adam optimizer with a fixed learning rate of 10-5 with batch size of for 10000 iterations. We approximate the WAIC score using M 4 architecture samples, and set the temperature parameter T 1.5 The number of layers and latent dimensions is the same as in the original Glow architecture (Kingma & Dhariwal 2018), with 4 blocks and 32 flows per block. Images were resized to 64 x 64 as inputs to the model. With this setup, we found that we are able to identify neural architectures in less than GPU day.
Our findings are summarized in Figure while more samples from our architecture search can be seen in Appendix [ Observing the most likely architecture components found on all of the datasets, a number of notable observations can be made:
The first few layers have simple feedforward structure, with either only few convolutional operations or average pooling operations. On the other hand, more complicated structures with skip connections are preferred in the deeper layers of the network. We hypothesize that in the first few layers, simple feature extractors are sufficient to represent the data well. Indeed, recent work analyzing neural networks for image data have shown that the first few layers have filters that are very similar to SIFT features or wavelet bases (Zeiler & Fergus 2014; Lowel 1999).
The max pooling operation is almost never selected by the architecture search. This confirms hypothesis that operations that discard information about the data is unsuitable for Ool) detection. However, to our surprise, average pooling is preferred in the first layers of the network. We hypothesize that average pooling has less severe effect in discarding information, as it can be thought of as convolutional filter with uniform weights.
The deeper layers prefer a more complicated structure, with some components recovering the skip connection structure of ResNets (He et al. 2016). We hypothesize that deeper layers may require more skip connections in order feed strong signal for the first few layers. This increases the speed and stability of training. Moreover, larger number of features can be extracted using the more complicated architecture.
Interestingly enough, we found that the architectures that we sample from our NADS perform well image generation without further retraining, as shown in Appendix D
4 BAYESIAN MODEL ENSEMBLE OF NEURAL ARCHITECTURES
4.1 MODEL. ENSEMBLE FORMULATION
Using the architectures sampled from search, create Bayesian ensemble of models to timate the WAIC score. Each model of our ensemble is weighted according to its probability. as in Hoeting al. 1999). The log-likelihood estimate as well as the variance of this model ensemble

---

Under review as a conference paper at ICLR 2020
MAIST
MUST
KORMET
PHNST
40000
KNRKET PARBST NAINMEST
35002
ROWNNGT
3000
LSUN
20002
LSUB
2005
2000
2000
1000
5025
5002
0.5
-00
#4C
WATC
MANC
M=1
M=5
Mm1
M
(a) MNIST
(b) SVHN
20000
-0.30
33063
6300
5030
23:0
112
(c) CIFAR-10
Figure 3: Effect of ensemble size to the distribution of WAIC scores estimated by model ensembles
trained on different datasets. Larger ensemble sizes causes the WAIC score likelihood estimate of OoD data to be lower. Additional histograms for different ensemble sizes in Appendix F are with higher resolution.
1:4
1.0
7.2
H=1
2.6
0.4
3.4
00R
COIV
0249
COR Iddrxx
Baselio
Fass Nobbr Bats (FTR)
Fabe Follee fats (11)
Patoe Toa falu (171)
Fabs Poobas fale (FF))
1.0
3.6
CDN
OCR
0214
022
3.0
3.0
Recal
local
Roca
(a) CIFARIOO-CIFARI (b) CIFARIO-CIFAR100
(c) SVHN-CIFARE CIFARI0
(d) MNIST- KMNIST
Figure 4: ROC and PR curve comparison of the most challenging evaluation setups for our NADS ensemble. Here, 'Baseline' denotes the method proposed by Hendrycks & Gimpel (0016). Subcap- tions denote training-testing set pairs. Additional figures are provided in Appendix G
given as follows:
po(as) M
E (oggs(())oog((r))
Σ logpa aEd
log (x)
(5)
ビ== Po(aa)
M
(o(as) 21=1 (ay)
V logg((x) 2
(Vloogo, (x)] + (logPa, (2))
(6)
Intuitively, we are weighing each member of the ensemble by their posterior architecture distribution Pa(a), measure of how likely each architecture is in optimizing the WAIC score. We note that for our setup. V[log Pa. (z)] is zero for each model in our ensemble; however, for models which do have
variance estimates, such as models that incorporate variational dropout (Gal et al.) 2017;; Kingma al. 2015 Gal & Ghahramani 2016), this term may be nonzero. Using these estimates, we are able to approximate the WAIC score in equation
4.2 ENSEMBLE RESULTS
We trained our proposed method on datasets: CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009). SVHN (Netzer al. 2011), and MNIST (LeCun). In all experiments, randomly sampled an

---

Under review as a conference paper at ICLR 2020
Table 1: OoD detection results various training and testing experiments. We compared
method with MSP (Hendrycks & Gimpel| 2016). and Outlier Exposure (OE) (Hendrycks et al.
2019b).
FPR% at TPR 95%
AUROC%
AUPR% %
Din
Dout
Base
OE
Ours
Base
OE
Ours
Base
OE
Ours
SVHN MNIST
DOL-MNIST
10.3
0.25
0.00
97.2
99.86
100
97.4
99.86
100
F-MNIST
61.1
0.99
0.00
88.8
99.83
100
90.8
99.83
100
K-MNIST
29.6
0.03
0.76
93.6
97.60
99.80
94.3
97.05
99.84
Texture
33.9
1.04
0.07
89.3
99.75
99.26
86.8
99.09
97.75
Places365
22.2
0.02
0.00
92.8
99.99
99.99
99.7
99.99
99.99
LSUN
26.8
0.05
0.02
88.2
99.98
99.99
90.4
99.95
99.99
CIFARIO
23.2
3.11
0.37
91.1
99.26
99.92
91.9
97.88
99.83
SVHN
30.5
8.41
17.05
89.5
98.20
97.65
94.9
97.97
99.07
Texture
39.8
14.9
0.25
87.7
96.7
99.81
79.8
94.39
99.86
CIFARIO
Places365
36.0
19.07
0.00
88.1
95.41
100
99.5
95.32
100
LSSUN
14.6
15.20
0.44
95.4
96.43
99.83
96.1
96.01
99.89
CIFARION 100
33.1
26.59
36.36
88.7
92.93
91.23
87.7
92.13
91.60
Gaussian
6.3
0.7
0.00
97.7
99.6
100
93.6
94.3
100
Rademacher
6.9
0.5
0.00
96.9
99.8
100
89.7
97.4
100
SVHN
46.2
42.9
45.92
82.7
86.9
94.35
91.3
80.21
96.01
Texture
74.3
55.97
0.42
72.6
84.23
99.76
75.76
99.81
CIFAR 100
Places365
63.2
57.77
0.012
76.2
82.65
99.99
98.9
81.47
99.99
LSUN
69.4
57.5
38.85
83.7
83.4
90.65
70.1
77.85
90.61
CIFARIO
62.5
59.96
34.41
75.8
77.53
92.83
74.0
72.82
91.93
Gaussian
29.3
12.1
0.00
86.5
95.7
100
66.1
71.1
100
Rademacher 59.4
17.1
0.00
51.7
93.0
100
32.7
56.9
100
ensemble of M models from the posterior architecture distribution (a) found by NADS. Although these models can sufficiently perform image synthesis without retraining as shown in Appendix[D we observed that further retraining these architectures led to a significant improvement OoD detection. Because of this, we retrained each architecture on data likelihood maximization for 150000 iterations using Adam with a learning rate of 1 x 10
We first show the effects increasing the ensemble size in Figure and Appendix [F] Here, we can see that increasing the ensemble size causes the OoD scores scores to decrease their corresponding histograms shift away from the training data WAIC scores, thus improving OoD detection perfor- mance. Next, we compare our ensemble search method against traditional ensembling method that uses a single Glow architecture trained with multiple random initializations. As shown in Table 2 we find that method is superior compared to the traditional ensembling method when compared on OoD detection using CIFAR-10 as the training distribution.
We then compared NADS ensemble OoD detection method for screening out samples from datasets that the original model was not trained on. For SVHN, we used the Texture, Places, LSUN, and CIFAR-10 the OoD dataset. For CIFAR-10 and CIFAR-100, Texture, Places, LSUN, CIFAR-100 (CIFAR-10 for CIFAR-100) datasets, as well as the Gaussian and Rademacher distributions as the OoD dataset. Finally, for MNIST, we used the not-MNIST, F-MNIST, and K-MNIST datasets. We compared our method against a baseline method that uses maximum softmax probability (MSP) (Hendrycks & Gimpel 2016), well as two popular OoD detection methods: ODIN (Liang et al. 2017) and Outlier Exposure (OE) (Hendrycks et al. 2019b). ODIN attempts calibrate the uncertainty estimates of existing model by reweighing its output softmax score using temperature parameter and through random perturbations of the input data. For this, we use DenseNet as the base model as described in (Liang et al. (017). On the other hand, OE models are trained to minimize loss regularized by outlier exposure loss term, a loss term that requires access to OoD samples.
used the SVHN,
As shown in Table I and Table method outperforms the baseline MSP and ODIN significantly while performing better or comparably with OE, which requires OoD data during training, albeit not from the testing distribution. We plot Receiver Operating Characteristic (ROC) and Precision-Recs Recall (PR) curves in Figure 4 and Appendix G for more comprehensive comparison. In particular, our method consistently achieves high under PR curve (AUPR%), showing that especially
8

---

Under review as a conference paper at ICLR 2020
capable of screening out OoD data in settings where their occurrence is rare. Such feature is important situations where anomalies are sparse, yet have disastrous consequences. Notably, ODIN underperforms in screening out many OoD datasets, despite being able reach the original reported performance when testing on LSUN using a CIFAR1O 10 trained model. This suggests that ODIN may not be stable for on different anomalous distributions.
CONCLUSION
Unlike NAS for common learning tasks, specifying a model and an objective to optimize for un- certainty estimation and outlier detection is not straightforward. Moreover, using a single model may not sufficient to accurately quantify uncertainty and successfully screen out OoD data. We developed a novel neural architecture distribution search (NADS) formulation to identify a random ensemble of architectures that perform well on a given task. Instead of sceking to maximize the likelihood of in-distribution data which may cause OoD samples to be mistakenly given higher likelihood, we developed a search algorithm to optimize the WAIC score, Bayesian adjusted es- have identified several key features that
timation of the data entropy. Using this formulation, make up good uncertainty quantification architectures, namely a simple structure in the shallower layers, use of information preserving operations, and larger, more expressive structure with skip connections for deeper layers to ensure optimization stability. Using the architecture distribution learned by NADS, we then constructed an ensemble of models estimate the data entropy using the WAIC score. We demonstrated the superiority of our method to existing OoD detection methods and showed that our method has highly competitive performance without requiring access to Ool) samples. Overall, NADS new uncertainty-awer certainty quantification that critical for more robust and generalizable deep learning, a crucial step in safely applying deep learning to healthcare, autonomous driving, and disaster response.
architecture search strategy enables model un-
REFERENCES
Karim Ahmed and Lorenzo Torresani. Maskconnect: Connectivity learning by gradient descent. In European Conference on Computer Vision, pp. 362-378. Springer, 2018.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in Al safety. arXiv preprint arXiv:1606.06500 2016.
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network archi- tectures using reinforcement learning. In International Conference on Learning Representations, 2017.
Liang-Chieh Chen, Maxwell Collins, Yukun Zhu, George Papandreou, Barret Zoph, Florian Schroff, Hartwig Adam, and Jon Shlens. Searching for efficient multi-scale architectures for dense image prediction. In Advances in Neural Information Processing Systems, pp. 8699-8710, 2018
Hyunsun Choi and Eric Jang. Generative ensembles for robust anomaly detection. arXiv preprint arXiv:1810.01399 2018
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint /605.
2016.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey, arXiv preprint a8Xiv:1808.0 003887 2018.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050~1059, 2016.
Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In Advances in Neural Information Processing Systems, pp. 3581-3590, 2017.
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. In International Conference on Machine Learning, pp. 881-889, 2015.

---

Under review as a conference paper at ICLR 2020
Golnaz Ghiasi, Tsung- Yi Lin, and Quoe V Le. Nas-fpn: Learning scalable feature pyramid architec- ture for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7036-7045, 2019.
Xinyu Gong, Shiyu Chang. Yifan Jiang, and Zhangyang Wang. Autogan: Neural architecture search for generative adversarial networks. In The IEEE International Conference on Computer Vision (ICCV), Oct 2019.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
Emil Julius Gumbel. Statistical theory of extreme values and some practical applications. NBS
Applied Mathematics Series, 33, 1954.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. A bascline for detecting misclassified and cat-of-distributtion examples in neural networks. In International Conference Learning Representations, 2016.
Dan Hendrycks. Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. arXiv preprint arXiv: 099600 2019a.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019b.
Jennifer A Hoeting, David Madigan, Adrian E Raftery, and Chris T Volinsky. Bayesian model averaging: A tutorial. Statistical science, pp. 382 882-401, 1999.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with Gumbel-Softmax. arXiv preprint arXiv:1611.01144 2016.
Haifeng Jin, Qingquan Song, and Xia Hu. Auto-keras: Efficient neural architecture search with
network morphism. arXiv preprint arXiv:1806. 10282, 2018.
Alex Kendall and Yarin Gal. What uncertainties do we need in Bayesian deep learning for computer vision? Advances in neural information processing systems, pp. 5574-5584, 2017.
Diederik P Kingma and Max Welling.
Auto-encoding variational
Bayes.
arXiv preprint
1312.66 14, 2013.
Durk P Kingma and Prafulla Dhariwal. Glow:
Generative flow with invertible Ixl convolutions. In
Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteri- zation trick. In Advances in Neural Information Processing Systems, pp. 2575-2583, 2015.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell, Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp. 6402-6413, 2017.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/nmist//
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrice classifiers for detecting outtof--iitributiio samples. In International Conference on Learning Representations, 2018.
10

---

Under review as a conference paper at ICLR 2020
Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distributinn image detec- tion in neural networks. arXiv preprint arXiv: 700.02269. 2017.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In European Conference on Computer Vision, pp. 19-35. Springer, 2018a.
Chenxi Liu, Liang-Chieh Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei- Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 82-92, 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv: 180.:1800.00005,,, 2018b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset.
David G Lowe. Object recognition from local scale-invariant features. In Proceedings of the Seventh IEEE International Conference on Computer Vision, volume 2, pp. 1150-1157, 1999.
Chris Maddison, Daniel Tarlow, and Tom Minka. A sampling. In Advances in Neural Information Processing Systems, pp. 3086-3094, 2014.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan, Do deep generative models know what they don't know? In International Conference on Learning Representations, 2019a.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting out-of-distribution inputs deep generative models using test for typicality. arXiv preprint 1rXiv:1906.0299) 2019b.
Renato Negrinho and Geoff Gordon. Deeparchitect: Automatically designing and training deep architectures. arXiv preprint arXiv:/70.08899,, 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
NHTSA. Tesla crash preliminary evaluation report. Technical report, U.S. Department of Trans- portation, National Highway Traffic Safety Administration, Jan 2017.
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: 1609.03499 2016.
Hieu Pham, Melody Guan, Barret Quoc Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In International Conference on Machine Learning, pp. 4092-4101, 2018.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4780 4789, 2019.
Alireza Shafaci. Mark Schmidt, and James Little. A less biased evaluation of out-of-distribution sample detectors.
Sumio Watanabe. A widely applicable Bayesian information criterion. Journal of Machine Learning Research, 14(Mar).67-887, 2013.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In
European conference computer vision, pp. $18-833. Springer, 2014.

---

Under review as a conference paper at ICLR 2020
Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng- Liu. Practical block-wiss neural network architecture generation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2423 2232.. IEEE, 2018.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint 1..057,6, 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8697-8710. IEEE, 2018.
12

---

Under review as a conference paper at ICLR 2020
A FIXED MODEL ABLATION STUDY
Table Ool detection results on various training and testing experiments comparing our method with bascline ensembling method that uses fixed architecture trained multiple times with different random initializations.
FPR% at TPR 95%
AUROC%
AUPR%
Din. Doat
Base Ensemble
Ours
Base Ensemble
Ours
Base Ensemble
Ours
SVHN
50.07
17.05
93,48
97.65
95.98
99.07
Texture
6.22
0.25
97.68
99.81
97.44
99.86
CIFARIO
Places365
1.03
0.00
99.59
100
99.97
100
LSUN
34.35
0.44
91.55
99.83
92.15
99.89
CIFARI00
65.13
36.36
78.44
91.23
79.44
91.60
Gaussian
0.00
0.00
100
100
100
100
Rademacher
0.00
0.00
100
100
100
100
B OOD DETECTION PERFORMANCE COMPARISON WITH ODIN
Table 3: OoD detection results on various training and testing experiments comparing our method
with ODIN (Liang et al. 2017).
FPR% at TPR 95%
AUROC%
AUPR%
Din
Dout
ODIN
Ours
ODIN
Ours
ODIN
Ours
MNIST
nOI-MNIST
8.7
0.00
98.2
100
98.0
100
F-MNIST
65
0.00
88.6
90.5
K-MNIST
36.5
0.76
94.0
99.80
94.6
99.84
Texture
33.9
0.07
92.4
99.26
88.2
97.75
SVHN
Places365
22.2
0.00
94.9
99.99
99.8
99.99
LSUN
26.8
0.02
93.5
99.99
93.1
99.99
CIFAR10
21.6
0.37
94.8
99.92
94.4
99.83
SVHN
36.5
17.05
89.7
97.65
95.6
99.07
Texture
76.2
0.25
81.4
99.81
76.7
99.86
CIFARIO
Places365
44.0
0.00
89.0
100
99.6
100
LSUN
3.9
0.44
99.2
99.83
99.2
99.89
CIFARI00
45.4
36.36
88.3
91.23
88.5
91.60
Gaussian
0.1
0.00
100
100
99.9
100
Rademacher
0.3
0.00
99.9
100
99.8
100
SVHN
32.8
45.92
90.3
94.35
95.3
96.01
Texture
78.9
0.42
75.7
99.76
64.5
99.81
CIFAR 100
Places365
63.3
0.012
79.0
99.99
99.1
99.99
LSUN
17.6
38.85
96.8
90.65
96.5
90.61
CIFAR10
78.2
34.41
70.6
92.83
69.7
91.93
Gaussian
1.3
0.00
99.5
100
97.8
100
Rademacher
13.8
0.00
92.7
100
75.0
100
13

---

Under review as a conference paper at ICLR 2020
C ADDITIONAL SAMPLE ARCHITECTURES
I- I-aepueom 5x3
ReC BN
ReL. Conr UN_]
2-diamm Jc1
concotenzze
ReLU Cony BN_
4 dil Loom 55a
add
Roll BN
Rel.I-Com-BN
2 ring por 3o5
HUL
concutenzze
sep
(a) Block
(b) Block
(c) Block
(d) Block 4
Figure 5: Maximum likelihood architectures inferred by our search algorithm on CelebA. Shown samples taken from each block.
RellI-Com-BN.
I-mpp pood 3c
up
sp 3otv
add_2
Relll-Cooo-RN
pool
1-mep.pool_p
Rell Rel.l-Coom-NN
concatcnaic
Rel.t-Com
sep_ooav Si5
(a) Block
(b) Block
(c) Block 3
(d) Block 4
Figure 6: Maximum likelihood architectures inferred by our search algorithm on MNIST. Shown two samples taken from each block.
14

---

Under review as a conference paper at ICLR 2020
BN_
I- 565
2-
4- app comyy
ReLL BN_
sep
akll
concotonote
2-wepusom
(a) Block
(h) Block
(c) Block
(d) Block 4
Figure 7: Maximum likelihood architectures inferred by our search algorithm on SVHN. Shown are two samples taken from each block.
sqy Lin 5K3
AMLI
J-seg.oar. 3x3
BN.
Relli-CanBBN
KeLu Laeiii-LEn-NN. LN
ReLL Commn-B-N
A
I- 1-ppppooarr...m 555
RollT-CCess RN
2- sip comu Ses
WiLL
Ril Co
done Sct
(a) Block
(b) Block 2
Rel.U-CCon-BiN
x-p 5a5
2 cottvo3
+SL
concutenate
BeL.ICCoe.BN.11
adi
3 sopppoomr__dd
Bclli-CuvvBB_J
concatenate
1. sep_copv_t $35
WALl
2-sep_oamr_a
Bol.i-CCr.
Bi Miltewihh IN
(c) Block
(d) Block
Figure 8: Maximum likelihood architectures inferred by our search algorithm on CIFAR-10. Shown are two samples taken from each block.
15

---

Under review as a conference paper at ICLR 2020
D IMAGE GENERATION SAMPLES
Figure 9: Samples taken from randomly sampled NADS architectures searched on CelebA. Images were not cherry-picked and the architectures were sampled without further retraining.
Q
7
2
9
3
7
9
9
7
6
8
6
3
9
0
2
9
8
5
8
9
6
9
7
b
]
2
8
1
7
8
1
0
9
Figure 10: Samples taken from randomly sampled NADS architectures searched MNIST. Images were not cherry-picked and the architectures were sampled without further retraining.
8
Figure 11: Samples taken from randomly sampled NADS architectures searched on SVHN, Images were not cherry-picked and the architectures were sampled without further retraining.
16

---

Under review as a conference paper at ICLR 2020
Figure 12:
Samples taken from randomly sampled NADS architectures searched on CIFAR-10.
Images not cherry-picked and the architectures sampled without further retraining.
E LIKELIHOOD ESTIMATION MODELS ASSIGN HIGHER LIKELIHOOD TO OoD DATA
3000
cifar10
celeba svhn
2500
2000
1500
1000
3.00
3.25
3.50
3.75
4.00
4.25
4.50
4.75
Log-likelihood
Figure 13: Likelihood distributions of different datasets evaluated on Glow model trained on CelebA. The model assigns higher likelihood to OoD samples from CIFAR- 10 and SVHN.
17

---

Under review as a conference paper at ICLR 2020
F
EFFECT OF ENSEMBLE SIZE
MNIST
MNIST KMNIST
4000
KMNIST
FMNIST
FMNIST
3000
NotMNIST
3000
NotMNIST
2000
2000
1000
1000
0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
WAIC
WAIC
=
M=2
MNIST
MNIST
4000
KMNIST
4000
KMNIST
FMNIST
FMNIST
3000
NotMNIST
3000
NotMNIST
2000
1000
1000
0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
WAIC
WAIC
M=3
M=4
MNIST
4000
KMNIST
FMNIST
3000
NotMNIST
2000
1000
0
0.0
0.5
1.0
1.5
WAIC
M=5
Figure 14: Effect of ensemble size to the distribution of WAIC scores estimated by model ensembles trained on MNIST.
18

---

Under review as a conference paper at ICLR 2020
CIFAR-10
CIFAR-10
SVHN
15000
SVHN
15000
Texture
Texture
Places
Places
10000
LSUN
10000
LSUN
5000
5000
0
-30
-20
-10 10
0
30
-20
-10
0
WAIC
WAIC
M=1
M=2
CIFAR-10
CIFAR-10
15000
SVHN
15000
SVHN
Texture
Texture
Places
Places
10000
LSUN
10000
LSUN
5000
5000
0
-30
-20
-10
0
-30
-20
-10
0
WAIC
WAIC
M=3
M=4
CIFAR-10
15000
SVHN
Texture
Places
10000
LSUN
5000
0
-30
-20
-10
0
WAIC
M=5
Figure 15: Effect ensemble size to the distribution of WAIC scores estimated by model ensembles trained on SVHN.
19

---

Under review as a conference paper at ICLR 2020
CIFAR10
CIFARI0
12500
SVHN
12500
SVHN
10000
Texture
10000
Texture
Places365
Places365
7500
LSUN
7500
LSUN
5000
5000
2500
2500
0
-100
-80
-60
-40
-20
-100
-80
-60
-40
-20
0
WAIC
WAIC
M=1
M=2
10000
10000
CIFAR10
CIFAR10
SVHN
SVHN
8000
Texture
8000
Texture
Places365
Places365
6000
LSUN
6000
LSUN
4000
4000
2000
2000
0
-100
-80
-60
40
-20
-100
-80
-60
-40
-20
0
WAIC
WAIC
M=3
M=4
10000
CIFAR10
SVHN
8000
Texture
Places365
6000
LSUN
4000
2000
100
-80
-60
-40
-20
0
WAIC
M=5
Figure 16: Effect of ensemble size to the distribution of WAIC scores estimated by model ensembles trained on CIFAR-10.
20

---

Under review as a conference paper at ICLR 2020
G ADDITIONAL ROC AND PRECISION-RECALL CURVES
3.6
KABS Ereende jour
WA25 Incentle (burn
Tue 2.4
RAb5 Ensemble
oow
OPN
00N
Baod no
Banelie
3.0
Saference
Birkmence
Raference
ce Reothee kate (FP6)
Falte Fottive Rose (1P8)
Paios Foe the katt (1P4)
Fale Poctiun Rate (FP63
1.0
1.c
1.00
0.30
3.6
RApS Ensemble shurt)
ROSS Frcemale (Darg)
MAis Fnceriile (Oun)
3.80
CON
ocn
0340
00.
25
Duacitc
Bandrn
Faceine
Reaal
Recal
Recal
Recal
(a) LSUN
(b) Texture
(c) Places
(d) SVHN
Figure 17: ROC and curve comparison of methods trained on CIFAR-10
1.0-
1.0
Positoo (T9R)
True Poelttee Bate (TAR)
0.8
0.0-
0.8
Postivv
0.4
NADS Ensemble (Ours)
0.4-
NADS Ensemble (Ours)
0.4
NADS Ensemble (Durs)
ODIN
022
Baseline
0.2
Baseline
0.2
Baseline
Reference
Reference
Reference
0.4
0'6
0.2
0.4
0.6
0.0
0.2
1.0
False Positive (FPR)
False Positive (FPR)
False Positive Rate (FPR)
1.000
0.8
0.8
0.985
Preddion
0.6
0.6
0.2
NADS Ensemble (0urs)
NADS Ensemble (Oung)
DGIN
NADS Ensemble (Ours)
0.0
Baseline
0.2
Baseline
0.975
Baseinc
(a) LSUN
(b) Texture
(c) Places
Figure 18: ROC and PR curve comparison of methods trained on SVHN
21

---

Under review as a conference paper at ICLR 2020
1.0
1.0
True Positive Rate (TPR)
True Positive Rate (TPR)
0.8
0.8
M
0.6
0.6
M
M = 2
M
M 3
ODIN
0.2
M
0.2
Baseline Reference
Reference
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate (FPR)
False Positive Rate (FPR)
0.8
0.8
Precision
0.6
Precision
0.6
M =
M
0.4
M
0.4
M
0.2
0.2
Baseline
0.4
0.6
0.0
0.2
1.0
Recall
Recall
(a) LSUN
(h) SVHN
Figure 19: ROC and PR curve comparison methods trained on CIFAR-100
1.0
True Positive Rate (TPR)
True Positive Rate (TPR)
0.8
0.8
0.6
M = 1
0.6
M 1
M 2
M
0.4
M 3
0.4
M
M
0.2
0.2
Reference
Reference
0.0
0.0
0.6 False Positive Rate (FPR)
1.0
False Positive Rate (FPR)
1.00
M
1.04
M
0.99
1.02
Precision
Precision
0.98
1.00
M
M
0.97
M
0.98
M
0.96
0.96
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Recall
Recall
(a) NotMNIST
(b) FMNIST
Figure 20: ROC and PR curve comparison of methods trained on MNIST
22


Reviews:
[{"review_id": "r1x0cOCwYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper needs crucial, quick fixes.\n\nThis paper throws neural architecture search at the problem of out-of-distribution detection. Rather than searching over multi-class classifier architectures, and using the result for OOD detection, they instead search for generative model architectures.\nThe approach appears to work on some of their cherry-picked OOD datasets.\n\nI give this paper a 3 because they are quite possibly only showing their strongest results and not giving a complete picture, and there are numerous small errors throughout the paper.\nAfter a more thorough evaluation, the technique will likely not look strong on datasets such as CIFAR-10 vs CIFAR-100.\nThis is acceptable since they are comparing density estimators vs multi-class OOD detectors, the latter of which has been vastly superior for many years.\nTheir technique brings density estimators within striking distance of multi-class classifier perofrmance, but the paper must give a complete picture. If these issues are fixed, the paper is easily a 6 or an 8, depending on the results of currently unshown OOD datasets. It is acceptable if their technique gets and 55% AUROC for CIFAR-10 vs CIFAR-100 while multi-class classifiers with Outlier Exposure get 96%. It is OK because this technique appears superior to other generative models, which have lagged far behind. Currently the paper leaves the impression that this is not only leapfrogging past previous density estimators, but that it also is beating multi-class classifiers. This likely isn't true. We need to see performance on other OOD datasets.\nI am willing to increase my score from a 6 to an 8 even if the results are negative.\n\nThe CIFAR-10 model should have to detect OOD samples from CIFAR-100, Rademacher/Bernoulli noise, and Gaussian noise. In addition, they should train a model on CIFAR-100 since many OOD detection techniques exhibit much worse behavior on CIFAR-100, compared to CIFAR-10 or MNIST or SVHN.\n\nIn summary, I give this a 3 due to critical flaws, but if these are rectified, the paper will likely deserve a 6 or 8.\n\nMiscellaneous Points:\n\nPlease include CIFAR-10/100 code. Currently the code is for MNIST.\n\nThere are several errors in discussing related work.\n\n> Moreover, previous work on deep uncertainty quantification shows that a single model may not suffice to quantify uncertainty and detect OoD samples (Lakshminarayanan et al., 2017; Choi & Jang, 2018)\n\nEnsembles do not perform appreciably better at OOD detection under systematic OOD benchmarking when using multi-class classifiers. For generative models, ensembles can help. Ensembles mainly help multi-class classifier _calibration_ on in-distribution in-class data, but they have miniscule to nonexistent utility in classifier OOD detection. Please add appropriate qualifiers.\n\n> \"With no access to OoD data, unsupervised/self-supervised generative models which maximize the likelihood of in-distribution data become the primary tools for uncertainty quantification.\"\nI think they mean \"with access to labels.\" Multi-class classifiers are the most performant tool for OOD detection, and unsupervised generative models are around chance-levels. _Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty_ (NeurIPS) uses self-supervised learning for OOD detection and achieves good performance, but this work is not cited.\n\n> \"However, these models counter-intuitively assign high likelihoods to OoD data (Nalisnick et al., 2019a; Choi & Jang, 2018)\"\nThis was shown in previous work, such as Shafaei et al. 2019 and Hendrycks et al. 2019, and hence deserve mention.\n\n> \"Moreover, existing methods to calibrate model uncertainty estimates assume access to OoD data during training (Lee et al., 2018; Hendrycks et al., 2019). This is flawed when anomalous data is rare or not known ahead of time\"\nLee et al. does not assume access to OOD data during training; they synthesize their own. Hendrycks et al. does assume access to OOD data, but not OOD data seen during evaluation; in this way, they do not assume data is \"known ahead of time,\" which they reiterate throughout their paper. This sentence is painting with too broad a brush.\n\nThere is a smaller experimental problem. They compare to ODIN, but ODIN assumes access to OOD data from the test distribution. While this assumption is clearly questionable, their evaluation does not use the assumption they did, so their technique is not actually ODIN. I suggest just comparing against the Maximum Softmax Probability Baseline in Table 1 since the rest of this paper assumes OOD examples are not known ahead of time.\n\nUpdate: I have changed my score to an 8.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Skg5TJhhKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the neural architecture search (NAS) problem under the out-of-distribution (OoD) environment. As the OoD problem is not visited in the current NAS literature, this paper proposes replacements for each of the three standard components in NAS, i.e., the proxy task, the search space, and the optimization algorithm; and each replacement is built upon an ensemble of existing techniques. Experiments are further verified on CelebA, CIFAR-10, SVHN, and MNIST.\n\nOverall, the novelty in this paper is limited and experiments are not very convincing. Please see the questions below:\n\nQ1. In the introduction, the authors write \"Machine learning systems often encounter OoD errors when dealing with testing data coming from a different distribution from the one used for training\".\n- What is the validation set used in this paper?\n- For the NAS problem, the architectural parameters must be guided by the validation set. So, does the validation set follows the same distribution as the training data set or the testing data set?\n- If the validation set and the testing set have the same distribution, is it still a meaningful OoD problem?\n\nQ2. \"For example, naively using data likelihood maximization as a proxy task would run into the issue pointed out by Nalisnick et al. (2019a), with models assigning higher likelihoods to OoD data\".\n- How can I see this point from the given experiments?\n- Is it better adding this into an ablation study?\n\nQ3. Except for WAIC, what other metrics can we consider? The authors should have a more comprehensive related work section, which includes discussion on this part.\n- Is it more meaningful to search a better metric than search architectures (which is just a standard applicaiton)?\n\nQ4. In Section 3.4:  \"CelebA (Liu et al.), CIFAR-10 (Krizhevsky et al., 2009), SVHN (Netzer et al., 2011), and MNIST (LeCun)\" are used.\n- Could the authors explain more tasks and details on these data sets? Specifically, why they are OoD problems.\n- Based on the descriptions from the authors, these data sets seem to be standard ones.\n\nQ5. Variation is already considered in (1), I mean the second term there. \n- Why should we still consider an ensemble of models?\n- Is it better adding an ablation study on M about \" ROC and PR curve\"? (not Figure 13, 14 in the appendix)\n\nQ6. Is it better to have a comparison with standard NAS in the experiments?\n- While authors argue they are not applicable here, it is still good to demonstrate how not applicable they are.\n- If the validation set follows the same distribution as the testing set, gradient signals on architectural parameters perhaps can still be helpful.\n\nQ7. Can natural gradient descent be applied to (4)?\n- \"First, optimizing p(α), a probability over ..., each network’s optimal parameters would need to be individually\". The first problem is not a really challenging problem please have a check at \"Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search\".\n\nQ8. What is the searching time of the proposed method? How's it compared with recent NAS methods? e.g. DARTS (DARTS: Differentiable Architecture Search). ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlXU1DsFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a neural architecture search (NAS) method to construct a Bayesian ensemble of deep learning models. This ensemble is then employed to detect out-of-distribution examples.\n\n\nThe authors propose to use a differentiable architecture search method which model the architectural parameters using a concrete distribution. This idea was originally proposed by Xie et al. (2019) but this work was not discussed. Similarly, the work by Chang et al. (2019) is not discussed. In my opinion the novelty with respect to NAS is the WAIC objective function and its application to out-of-distribution detection.\n\n\nThe idea of using ensemble to detect out-of-distribution examples is not new. The authors already refer to the works by Choi & Jang (2018) and Lakshminarayanan et al. (2017). I'd like to add MC-Dropout (Gal et al., 2016) to this list which was used e.g. to detect adversarial examples.\n\n\nThe experimental section is well-written and the proposed method is able to outperform the chosen baselines. Obvious baselines are missing. There is no experiment that proof that this way of searching architectures finds better suited ensembles. How about maximizing the cross-entropy and train the discovered architecture multiple times from scratch and use these models in an ensemble to detect out-of-distribution examples? How about any ensemble-based method mentioned in the previous paragraph?\n\n\nConcluding, the idea is nice but based on the current state of the paper it seems incremental. Experiments to back the usefulness of the described method are missing.\n\n\nSirui Xie, Hehui Zheng, Chunxiao Liu, Liang Lin: SNAS: stochastic neural architecture search. ICLR 2019\nJianlong Chang, Xinbang Zhang, Yiwen Guo, Gaofeng Meng, Shiming Xiang, Chunhong Pan: Differentiable Architecture Search with Ensemble Gumbel-Softmax. arXiv (2019)\nYarin Gal, Zoubin Ghahramani: Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. ICML 2016: 1050-1059", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a neural architecture search method that is geared towards yielding good uncertainty estimates for out-of-distribution (OOD) samples.

The reviewers found that the OOD prediction results are strong, but criticized various points, including the presentation of the OOD results, novelty as a NAS paper, missing citations to some recent papers, and a lack of baselines with simpler ensembles.
The authors improved the presentation of their OOD results and provided new experiments, which causes one reviewer to increase his/her score from a weak reject to an accept. The other reviewers appreciated the rebuttal, but preferred not to change their scores from a weak reject and a reject, mostly due to lack of novelty as a NAS paper.

I also read the paper, and my personal opinion is that it would definitely be very novel to have a good neural architecture search for handling uncertainty in deep learning; it is by no means the case that "NAS for X" is not interesting just because there are now a few papers for "NAS for Y". As long as X is relevant (which uncertainty in deep learning definitely is), and NAS finds a new state-of-the-art, I think this is great. For such an "application" paper of the NAS methodology, I do not find it necessary to introduce a novel NAS method, but just applying an existing one would be fine. The problem is more that the paper claims to introduce a new method, but that that method is too similar to existing ones, without a comparison; actually just using an existing NAS method would therefore make the contribution and the emphasis on the application domain clearer. 
I have one small question to the authors about a part that I did not understand: to optimize WAIC (Eq 1), why is it not optimal to just set the parameterization \phi such that the variance is minimized, i.e., return a delta distribution p_\phi that always returns the same architecture (one with a strong prediction)? Surely, that's not what the authors want, but wouldn't that minimize WAIC? I hope the authors will clarify this in a future version.

In the private discussion of reviewers and AC, the most positive reviewer emphasized that the OOD results are strong, but admitted that the mixed sentiment is understandable since people who do not follow OOD detection could miss the importance and context of the results, and that the paper could definitely improve its messaging. The other reviewers' scores remained at 1 and 3, but the reviewers indicated that they would be positive about a future version of the paper that fixed the identified issues. My recommendation is to reject the paper and encourage the authors to continue this work and resubmit an improved version to a future venue.
Decision: Reject