Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
Authors: Sheheryar Zaidi
Arber Zela
Thomas Elsken
Christopher C. Holmes
Frank Hutter
Yee Whye Teh
Venue: NeurIPS 2021 Poster (2021)
Keywords: uncertainty estimation
deep ensemble
dataset shift
robustness
uncertainty calibration
Bayesian neural networks

Abstract: Ensembles of neural networks achieve superior performance compared to standalone networks in terms of accuracy, uncertainty calibration and robustness to dataset shift. Deep ensembles, a state-of-the-art method for uncertainty estimation, only ensemble random initializations of a fixed architecture. Instead, we propose two methods for automatically constructing ensembles with varying architectures, which implicitly trade-off individual architectures’ strengths against the ensemble’s diversity and exploit architectural variation as a source of diversity. On a variety of classification tasks and modern architecture search spaces, we show that the resulting ensembles outperform deep ensembles not only in terms of accuracy but also uncertainty calibration and robustness to dataset shift. Our further analysis and ablation studies provide evidence of higher ensemble diversity due to architectural variation, resulting in ensembles that can outperform deep ensembles, even when having weaker average base learners. To foster reproducibility, our code is available: https://github.com/automl/nes

Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
Sheheryar Zaidi
Arber Zela?* Thomas Elsken
Chris Holmes!
Frank Hutter Yee Whye Teh
University Oxford, Univerritty of Freiburg, Bossh Center for Artificial Intelligence {szaidi, cholmes, y.w. w.tthhSstttee o ac.uk, uk, {zelaa, fh} uni-freiburg.de thomas. elsken@de. bosch. com
Abstract
Ensembles neural networks achieve superior performance compared to stand- alone networks in terms of accuracy, uncertainty calibration and robustness dataset shift. Deep ensembles, a state-of-the- method for uncertainty estimation, only ensemble random initializations of fixed architecture. Instead, we propose two methods for automatically constructing ensembles with varying architectures, which implicitly trade-off individual architectures' strengths against the ensemble's diversity and exploit architectural variation source of diversity. On variety of classification tasks and modern architecture search spaces, we show that the resulting ensembles outperform deep ensembles not only in terms of accuracy but also uncertainty calibration and robustness dataset shift. Our further analysis and ablation studies provide evidence of higher ensemble diversity due to architectural variation, resulting ensembles that can outperform deep ensembles, even when having weaker average base learners. To foster reproducibility, our code is available: hnttps://githin com/ aat//m/tommm/eee
Introduction
Some applications of deep learning rely only on point estimate predictions made by neural network. However, many critical applications also require reliable predictive uncertainty estimates and robust- ness under the presence dataset shift, that is, when the observed data distribution at deployment differs from the training data distribution. Examples include medical imaging [15] and self-driving
cars S Unfortunately, several studies have shown that neural networks are not always robust to dataset shift [46] 26), nor do they exhibit calibrated predictive uncertainty, resulting in incorrect predictions made with high confidence [21].
33 achieve state-of-the-art results for predictive uncertainty calibration and robust- ness to dataset shift. Notably, they have been shown to outperform various approximate Bayesian neural networks (33 46 22 Deep ensembles are constructed by training a fixed architecture mul- tiple times with different random initializations. Due to the multi-modal loss landscape [8 [4]. randomization by different initializations induces diversity among the base learners to yield a model with better uncertainty estimates than any of the individual base learners (i.e. ensemble members).
Deep ensembles
Our work focuses on automatically selecting varying base learner architectures in the ensemble, exploiting architectural variation a beneficial source of diversity missing in deep ensembles due to their fixed architecture. Such architecture selection during ensemble construction allows a more "ensemble-aware" choice of architectures and based data rather than manual biases. As
Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

discussed in Section 2 while ensembles with varying architectures has already been explored in the literature, variation in architectures is typically limited to just varying depth and/or width, in contrast to more complex variations, such as changes in the topology of the connections and operations used, as considered our work. More generally, automatic ensemble construction well-explored
in AutoMI. [17, 36 45 43 Our work builds on this by demonstrating that, in the context of uncertainty estimation, automatically constructed ensembles with varying architectures outperform deep ensembles that use sate-of-the-art, or even optimal, architectures (Figure []. Studied under controlled settings, we assess the ensembles by various measures, including predictive performance, uncertainty estimation and calibration, base learner performance and two ensemble diversity metrics, showing that architectural variation is beneficial in ensembles.
Note that, a priori, it is not obvious how to find a set of diverse architectures that work well as an en- semble. On the one hand, optimizing the base learn- ers' architectures in isolation may yield multiple base learners with similar architectures (like deep ensemble). On the other hand, selecting the archi- tectures randomly may yield numerous base learn- ers with poor architectures harming the ensemble. Moreover, in neural architecture search (NAS), we face the challenge of needing to traverse vast architectural search spaces. We address these chal- lenges in the problem of Neural Ensemble Search (NES), an extension of NAS that aims to find set of complementary architectures that together form strong ensemble. In summary, our contributions are as follows:
Error
foral 66 metrics smaller (ue cloase cencer)
Envembble NLL
ECE
FG2
Aug. base leamer NLL
Freed Disugr
Oracle NLL
NES-RS -
NES-RE
DeopEns (best arch.)
Figure 1: A comparison of deep ensem- ble with the best architecture (out of 15,625 possible architectures) and ensembles con- structed by our method NES on ImageNet- 16- 120 over NAS-Bench-201. Performance metrics (smaller is better): error, negative log- likelihood (NLL.) and expected calibration error (ECH). We also measure average base learner NLL and two metries for ensemble diversity (see Section 3): oracle NLL and [1 [1-predictive disagreement]; small means more diversity for both metrics. NES ensembles outperform the deep ensemble, despite the latter having sig- nificantly stronger average base learner.
1. We present two NES algorithms for automati- cally constructing ensembles with varying base learner architectures. As first step, we present NES with random search (NES-RS), which is simple and easily parallelizable. We further pro- pose NES-RE inspired by regularized evolution [48. which evolves population of architectures yielding performant and robust ensembles.
2. This work is the first to apply automatic ensem- ble construction over architectures to complex. state-of-the-art neural architecture search spaces.
3. In the context of uncertainty estimation and ro- bustness to dataset shift, we demonstrate that ensembles constructed by NES improve upon tfat-oofthhe-art deep ensembles. We validate our findings over five datasets and two architecture search spaces.
2 Related Work
Ensembles and uncertainty estimation. Ensembles of neural networks [23 32
are commonly
used to boost performance. In practice, strategies for building ensembles include independently training multiple initializations of the same network, i.c. deep ensembles [33], training base learners on different bootstrap samples of the data (62). training with diversity-encouragee
losses [40] 35
60 5 47 and using checkpoints during the training trajectory of a network [T]. Despite
variety of approaches, Ashukha et al. I found many sophisticated ensembling techniques to be equivalent to a small-sized deep ensemble by test performance.
Much recent interest in ensembles has been due to their state-of ofee--ff-thett- predictive uncertainty estimates, with extensive empirical studies [46]] 22 observing that deep ensembles outperform other approaches for uncertainty estimation, notably including Bayesian neural networks [4, 19 52 and
post-hoe calibration 2T). Although deep ensembles are not, technically speaking, equivalent to Bayesian neural networks and the relationship between the two is not well understood, diversity

---

among base learners in a deep ensemble yields model which is arguably closer to exact Bayesian model averaging than other approximate Bayesian methods that only capture a single posterior mode in multi-modal landscape [54 18] Also, He el al. [24] draw rigorous link between Bayesian methods and deep ensembles for wide networks, and Pearce et al. [47] propose technique for approximately Bayesian ensembling. Our primary baseline is deep ensembles as they provide
state-of-the-art results in uncertainty estimation.
AutoML and ensembles of varying architectures. Automatic ensemble construction is commonly used in AutoML [7 28 Prior work includes use of Bayesian optimization to tune -onaarchitettual hyperparameters of an ensemble's 's base learners [36], posthoc ensembling of fully-connected networks evaluated by Bayesian optimization [43 and building ensembles by iteratively adding (ub-))newworks
to improve ensemble performance TO 42 Various upproaches, including ours, rely on ensemble
selection 77. We also note that Simonyan Zisserman [49], He et al. (25] employ ensembles with varying architectures but without automatic construction. Importantly, in contrast to our work, all aforementioned works limit architectural variation to only changing width/depth or fully-connected networks. Moreover, such ensembles have not been considered before in terms of uncertainty estimation. Another important part of AutoML is neural architecture search (NAS), the process of automatically designing single model architectures 14, using strategies such as reinforcement learning [63], evolutionary algorithms
[48] and gradient- based methods [39]. We use the search spaces defined by Liu et al. [39] and Dong & Yang [33, two of the most commonly used ones in recent literature.
Concurrent to our work, Wenzel el al. 53 consider ensembles with base learners having varying hyperparameters using an approach similar to NES-RS. However, they focus on non-architectural hyperparameters such as L2 regularization strength and dropout rates, keeping the architecture fixed. As in our work, they also consider predictive uncertainty calibration and robustness to shift, finding similar improvements over deep ensembles.
Visualizing Ensembles of Varying Architectures
In this section, we discuss diversity in ensembles with varying architectures and visualize base learner predictions to add empirical evidence to the intuition that architectural variation results in more diversity. We also define two metries for measuring diversity used later in Section 5
3.1 Definitions and Set-up
1, 1,..., N} be the training dataset, where the input æ; E RDD and, assuming classification task, the output y; E {1....,C') C),. We use Deal and Dust for the validation and test datasets, respectively. Denote by fo a neural network with weights 0, so fo(x) E Eº the predicted probability vector over the classes for input x. Let ((fe(x)) y) be the neural network's s loss for data point (ax, y). Given M networks for. . Sors we construct the ensemble F of these networks by averaging the outputs, yielding F(x)- = 1 EM (z.)
Let Dirain {ar,, yi)
In addition to the ensemble's 's loss ((F(x)) y), we will also consider the average buse learner loss and the oracle ensemble's 's loss. The average base learner loss is simply defined as J E {(fe (w), y): we use this to measure the average base learner strength later. Similar to prior work [35 60), the oracle ensemble FoR composed of base learners Jer..... Loy is defined to be the function which, given an input x, returns the prediction of the base learner with the smallest loss for (xx v), that is,
FxR = for (z), where 1 E argmin {(fa, (z).
The oracle ensemble can only be constructed if the true class y is known. We use the oracle ensemble loss as one the measures of diversity in base learner predictions. Intuitively, if base learners make diverse predictions for æ. the oracle ensemble is more likely to find some base learner with small loss, whereas if all base learners make identical predictions, the oracle ensemble yields the same output as any (and all) base learners. Therefore, as rule of thumb, all else being equal, smaller oracle ensemble loss indicates more diverse base learner predictions.
Proposition 3.1. Suppose é is negative log-likelihood (NLL). Then, the oracle ensemble loss, ensemble loss, and average base learner loss satisfy the following inequality:
M
{(For(*). S ((F()),)) S
((fo
M
3

---

We refer to Appendix A for a proof. Proposition 3.1 suggests that it can be beneficial for ensembles to not only have strong average base learners (smaller upper bound), but also more diversity in their predictions (smaller lower bound). There
ME MA
is extensive theoretical work relating strong base learner per- formance and diversity with the generalization properties of
I:SNE dimersion
ensembles [23 61 6 [0 20). In Section [ the two metrics
we use for measuring diversity are oracle ensemble loss and (normalized) predictive disagreement, defined as the average pairwise predictive disagreement amongst the base learners, normalized by their average error [8]
3.2 Visualizing Similarity in Base Learner Predictions
t-SNE dimension
The fixed architecture used to build deep ensembles is typ- ically chosen to be a strong stand-alone architecture, either hand-crafted or found by NAS. However, optimizing the base learner's 's architecture and then constructing a deep ensemble neglect diversity in favor of strong base learner perfor- mance. Having base learner architectures vary allows more diversity in their predictions. We provide empirical evidence for this intuition by visualizing the base learners' predictions. Fort et al. [18 found that base learners in deep ensemble explore different parts of the function space by means of apply- ing dimensionality reduction to their predictions. Building on this, we uniformly sample five architectures from the DARTS search space [39], train 20 initializations of each architecture on CIFAR- 10 and visualize the similarity among the networks' predictions on the test dataset using t-SNE [50]. Experiment details are available in Section 5 and Appendix
(a) Five different architectures, each trained with 20 diffcrent initializations.
Vayaa
I:SNE dimension
USNE dimension
(b) Predictions of base learners in two ensembles. with fixed architecture and one with varying architectures.
As shown in Figure 2a observe clustering of predictions fixed architecture, sug-
made by different initializations gesting that base learners with varying architectures explore different parts of the function space. Moreover, we also visu- alize the predictions of base learners of two ensembles, each of size M = 30, where one is a deep ensemble and the other has varying architectures (found by NES-RO ESS-RSeoo as presented in Section (.. Figure [bb shows more diversity in the ensemble with varying architectures than in the deep ensemble. These qualitative findings can be quantified by measuring diversity: for the two ensembles shown in Figure 2b) we find the predictive disagreement be 94.6% for the ensemble constructed by NES and 76. 76.7% for the deep ensemble (this is consistent across independent runs). This indicates higher predictive diversity in the ensemble with varying architectures, in line with the t-SNE results.
t-SNE visualization of base learner predictions. Each point corresponds to one network's (dimension-reduuce)) predictions.
Figure 2:
4 Neural Ensemble Search
In this section, we define neural ensemble search (NES). In summary, a NES algorithm optimizes the architectures of base learners in an ensemble to minimize ensemble loss.
Given a network f IR - R2C let L(F = Eley)e {(f(x), v)) be the loss of ff over dataset D. Given a set of base learners i J. let Ensemble be the function which maps .f fM the notation fe.a to denote a network with architecture a E A and weights 8, where AA is a search space (SS) of architectures. A NES algorithm aims to solve the following optimization problem:
M to the ensemble F MM Σ fi as defined in Section 3 To emphasize the architecture,
min E.A
fos Dval)
(I)
S.L. 0, E argmin L(fe.a,, Dusin) for i
M
6
Eq. [ is difficult to solve for at least two reasons. First, we are optimizing over M architectures, so the search space is effectively A compared to it being A in typical NAS, making it more difficult

---

Select ensemble
Sample parent
Update population
Mutate
J
Figure 3: Illustration of one iteration of NES-RE. Network architectures are represented as colored bars of different lengths illustrating different layers and widths. Starting with the current population. ensemble selection is applied to select parent candidates, among which one is sampled as the parent. mutated copy of the parent is added the population, and the oldest member is removed.
to explore fully. Second, a larger search space also increases the risk of overfitting the ensemble loss D A possible approach here consider the ensemble as a single large network to which we apply NAS, but joint training of an ensemble through single loss has been empirically observed to underperform training base learners independently, specially for large networks [5]] Instead, our general approach to solve Eq. [ consists of two steps:
1. Pool building: build pool P = {fo......... ........ fex.ax of size K consisting of potential base learners, where each fe,,a is network trained independently on Durain
2. Ensemble selection: select M base learners (without replacement as discussed below) .e...i..... ................................ fejr.aj from P to form an ensemble which minimizes loss on Vval. (We set K 2 M.)
Step I reduces the options for the base learner architectures, with the intention to make the search more feasible and focus strong architectures; step 2 then selects performant ensemble. This procedure also ensures that the ensemble's 's base learners are trained independently. We use ensemble selection without replacement 77] for step 2. More specifically, this is forward step-wise selection; that is, given the set of networks P. we start with an empty ensemble and add the network from P which minimizes ensemble loss on D We repeat this without replacement until the ensemble is of size M. ForsardSelect (P, Deat, .M) denotes the resulting set of M base learners selected from P.
Note that selecting the ensemble from P is combinatorial optimization problem; a greedy approach such as ForvardSelect nevertheless effective as shown by Caruana et al. (7), while keeping computational overhead low, given the predictions of the networks on D We also experimented with various other ensemble selection algorithms,, including weighted averaging, as discussed in Section 5.2 and Appendix C.8 finding ForwardSelect to perform best.
We have not yet discussed the algorithm for building the pool in step I; we present two approaches, NES-RS (Section[4.1) and NES-RE (Section[4.2) 4.2). NES-RS RS is simple random search based algorithm, while WES-RE is based on regularized evolution [48] state-of-the art NAS algorithm. Note that while gradient-basse NAS methods have recently become popular, they are not naively applicable in our setting as the base learner selection component ForwardSelect typically non-differentiable.
4.1 NES with Random Search
In NAS, random search (RS) is a competitive baseline carefully designed architecture search spaces [37 57 [88. Motivated by its success and simplicity, NES with random search (NES-RS) builds the pool P by independently sampling architectures uniformly with replacement from the search space A (and training them). Since the architectures of networks P vary, applying ensemble selection is simple way exploit diversity, yielding performant ensemble. Importantly, NES-RS is easy to parallelize, exactly like deep ensembles. See Algorithm in Appendix B.3 for pseudocode.
4.2 NES with Regularized Evolution
While RS has the benefit of simplicity by sampling architectures uniformly, the resulting pool might contain many weak architectures, leaving few strong architectures for ForvardSelect to choose between. RE is an evolutionary algorithm used on NAS spaces. It explores the search space by evolving (via mutations) population of architectures. We first briefly describe RE as background before NES-RE. RE starts with randomly initialized fixed-size population of architectures. At each iteration, subset of size m of the population is sampled, from which the best network by validation loss is selected as the parent. A mutated copy (c.g. changing an operation in the network, see Appendix
more guided approach for building the pool P is using regularized evolution (RE) 48
5

---

Algorithm 1: NES with Regularized Evolution
Data: Search space A; ensemble size M: comp. budget K; Drrin: Deat: population size P: number of parent candidates m.
1 Sample P' architectures a1....., ap independently and uniformly from A.
2 Train each architecture a using Drain and initialize p = P = (fe f0p.0r..
3 while |P| < K do
4 Select m parent candidates "
} = ForwardSelect(p Dvat, m).
Sample uniformly parent architecture from ( , am
// stays in p.
6 Apply mutation to a, yielding child architecture 3.
Train 3 using rainn and add the trained network fe.g to p and P. // as done in RE [48]
8
Remove the oldest member in p.
9 Select base learners {fa;.of;.... for, o3g = ForwardSelect Deal, M) by forward step-wise selection without replacement.
10 return ensemble Ensemble (foj ......................... a3))
B.4 for examples of mutations) of the parent architecture, called the child, is trained and added to the population, and the oldest member of the population removed, preserving the population size. This is iterated until the computational budget is reached, returning the history, i.e. all the networks evaluated during the search, from which the best model is chosen by validation loss.
Building on RE for NAS, we propose NES-RE to build the pool of potential base learners. NES- RE starts by randomly initializing population p of size P. At each iteration, we first apply ForwardSelect to the population to select an ensemble of size m, then we uniformly sample one base learner from this ensemble be the parent. A mutated copy of the parent is added to p and the oldest network is removed, as in usual regularized evolution. This process is repeated until the computational budget is reached, and the history is returned the pool P. See Algorithm for pseudocode and Figure for an illustration.
Also, note the distinction between the population and the pool in JES-RE: the population is evolved, whereas the pool is the set of all networks evaluated during evolution (i.e., the history) and used post-hoe for selecting the ensemble. Moreover, ForwardSelect is used both for selecting m parent candidates (line 4 in NES-RE) and choosing the final ensemble of size M (line 9 in NES-RE). In general, m M.
4.3 Ensemble Adaptation to Dataset Shift
Deep ensembles offer (some) robustness to dataset shift relative to training data. In general, one may not know the type of shift that occurs test time. By using an ensemble, diversity base learner predictions prevents the model from relying one base learner's predictions which may not only be incorrect but also overconfident.
We assume that one does not have access to data points with test-time shift at training time, but one does have access to some validation data Delift with a validation shift, which encapsulates one's belief about test-time shift. Crucially, test and validation shifts are disjoint. To adapt NES-RS and NES-RE return ensembles robust to shift, we propose using Dshift instead of D..i whenever applying ForwardSelect to select the final ensemble. In Algorithms [ and [ this is in lines 9 and 3, respectively. Our experiments show that this highly effective against shift.
Note that in line 4 of Algorithm I we can also replace Deal with Dshill when expecting test-time shift; we simply sample one Dvat; Dehint uniformly at each iteration, order to promote exploration of architectures that work well both in-distribution and during shift and reduce cost by avoiding running NES-RE RE once for each of Dvai, Dshift See Appendices C.1.3 and B.4 for further discussion.
5
Experiments
5.1 Comparison to Baselines: Uncertainty Estimation & Robustness to Dataset Shift
We compare NES to deep ensembles on different choices of architecture search space (DARTS [39 and NAS-Bench-201 [13] search spaces) and dataset (FaShion-MNISTT CIFAR-10. 100.

---

CIPAR
OFAR:
OFAR-10
CIFRR 100
Thy ImageNet
0.30
0.30
Desalira (nternaterr
Enoordic NLL
Emergtee KLL
1..0
Enwaybee NLL
2.00
0.24
Emermbe
2.70
3.50
0.28
1.20
2:40
3:30
Ensembre toe (H]
Cmnertae size (4)
Ensemble in doe (HI
Ensamble dize (M]
Ensemit kize (N)
Ennembte (ME
(a) No data shift
(h) Dataset shift: severity 5 (out of 5)
Figure 4: NLL vs. ensemble sizes on CIFAR-IO, 10, CIFAR-ION and Tiny ImageNet with and without dataset shifts [26] over the DARTS search space. Mean NLL shown with 95% confidence intervals.
ImageNet- 16- 120 and Tiny ImageNet). The search spaces are cell-based, containing a rich variety of convolutional architectures with differing cell topologies, number of connections and operations (see Appendix(B..j for for visualizations). For CIFAR- 1FFAR-10/10 10/100 and Tiny ImageNet, we also consider dataset shifts proposed by Hendrycks & Dietterich [26]. We use NLL, classification error and expected calibration error (ECE) [21 44 our metrics, for which small values are better. Note that NLL and ECE evaluate predictive uncertainty. Experimenna/immleeentatioo details are in Appendix B All evaluations on the test dataset. Each paragraph
additional experiments are in Appendix C below highlights one of our main findings.
One of our main baselines is deep ensembles with fixed, optimized architectures. We consider various optimized architectures, indicated as "DeepEns (arch)":
Baselines.
On the DARTS search space, we consider the architectures found by:
the DARTS algorithm (DeepEns (DARTS)),
2. regularized evolution (DeepEns (AmoebaNet)),
3. random search, with the same number of networks trained as NES algorithms (DeepEns (RS)).
On the NAS-Bench-201 search space, in addition to DeepEns (RS), we consider:
1. the architecture found by GDAS (DeepEns (GDAS)),
2. the best architecture in the search space by validation loss (DeepEns (best arch ) F
We also compare to anchored ensembles 47 a recent technique for approximately Bayesian en- sembles, which explicitly regularizes each base learner towards fresh initialization sample and aims to induce more ensemble diversity. We use the DARTS architecture, and our implementation is described in Appendix B.5
NES shows improved predictive uncertainty (NLL) and robusiness to dataset shift (Figure 4). Figure 4a shows the NLL achieved by NES-RS, NES-RE and the baselines as functions of the ensemble size M without dataset shift. We find that NES algorithms consistently outperform deep ensembles, with NES-RE usually outperforming NES-RS. Next, we evaluate the robustness of the ensembles to dataset shift in Figure 4b In our setup, all base learners are trained on Dminn without data augmentation of shifted examples. However, as explained in Section 4.3 we use shifted validation dataset, Delun, and evaluate on a shifted test dataset, Diauft The types of shifts appearing in val are disjoint from those in Dshift
test The severity of the shift varies between 1-5. We refer to Appendix B and Hendrycks & Dietterich (66 for details. The fixed architecture used in the baseline DeepEns (RS) is selected based on its loss over Dehift
val but the DARTS and AmoebaNet are architectures from the literature. As shown in Figure 4b] ensembles picked by NES-RS and NES-RE are significantly more robust to dataset shift than the baselines, highlighting the effectiveness of applying ForwardSelect with Dshift. Unsurprisingly, AnchoredEns (DARTS) and DeepEns poorly compared to the other methods, as they are not optimized to deal with dataset shift here. The results of our experiments on Mithion-MNIST are in Appendix C.I.I In line with the results in this section, both NES algorithms outperform deep ensembles with NES-RE performing best.
perform
Better uncertainty calibration versus dataset shift and classification error (Figure Table 1. We also assess the ensembles by error and ECE, In short, ECE measures the mismatch between the model's 'ss confidence and accuracy. Figure 5 shows the ECE achieved by the ensembles at
We did not consider the DARTS algorithm on NAS-Bench-200 since returns degenerate architectures with poor performance on this space [3. Whereas, GDAS yields state-of-the-art performance on this space. This feasible, because all architectures in this search space were evaluated and available.
7

---

CIPAR
OFAR:
Iragehies
Trogglee:: 16:120
2.55
maaeelan
0.29
0.40
Devalina Marte (eceeeeeer
Ensrmble SCE
Eroe Drperade ECE
3.250
2.65
0.00
6:35
Crgentk
3.55
1200
6:24
2.4
3.5
2.125
St ceurriyy
0129 soveriy by
5h. sevedity
Murber rida undaedd
Namtorr modo
Figure 5: ECE vs. dataset shift severity on CIFAR-10, CIFAR-100 and Tiny ImageNet over the DARTS search space. No dataset shift is indi- cated as severity 0. Ensemble size is M = 10.
Figure 6: Ensemble, average base learner and ora- cle ensemble NLL versus budget K on ImageNet- 16-120 over the MAS-Bench-201 search space. Ensemble size is M = 3.
varying dataset shift severities, noting that uncertainty calibration is especially important when models are used during dataset shift. Ensembles found with NES tend to exhibit better uncertainty calibration and are either competitive with or outperform anchored and deep ensembles for most shift severities. Notably, on CIFAR-10. 10, ECE is reduced by up to 40% relative to the baselines. In terms of classification error, we find that ensembles constructed by NES outperform deep ensembles, with reductions of up to percentage points in error, shown in Table T outperforms NES-RS.
As with NLL, NES-RE tends to
Ensembles found by NES tend to be more di- verse (T. We measure ensemble diversity using the two metrics predictive disagreement (larger means more diverse) and oracle ensemble NLL (smaller means more diverse) as defined in Section 3 and average base learner NLL. Table 3 contrasts NES with the baselines in terms of these metrics. In terms of both diversity met- rics, ensembles constructed by NES tend to be more diverse than anchored and deep ensembles. Ranking of the methods is largely consistent for
Table I: Classification error of ensembles for dif- ferent shift severities. Best values and all values within 95% confidence interval are bold faced. Note that tecture trained with three random initializations; therefore we set M = 3 in that case.
comes with each archi-
Skn Seuriy
Deartao (85)
Decplia (Ancec))
Docpliza (DARIS)
Andacudiiza (UAKISS)
NES RS
NES. RE
10 10.8
0.7
16.0
10.2
8:4
8:4
CHARR110
25 2.
31.6
16.3
18.6
13:3
different ensemble sizes M (see Appendix C). Unsurprisingly, note that the average base learner in NES is not always the best: by optimizing the fixed architecture first and then ensembling it, deep ensembles end up with a strong average base learner at the expense of less ensemble di- versity. Despite higher average base learner NLL, NES ensembles perform better (recall Figure 4 highlighting once again the importance of diver-
CIRAR-10 100
1022...
34.2
1530
3533
15.1
3033
02.2
09.
49 Jass
tragehee
Tixs
30.5
71.
13.4
14.0
56.7
330
00.0
(a) M 10; DARTS search space.
Dastawt
Shint Severity
Daspltes (GIAS)
(zex. arch))
Desplins
(55,
Doopliss (RS)
NAS Bel--Benchhh 2011
RES-RS
NSS RE
space
7,5aro
7:7
T.uue
CIEAR.10 10
58
22.
12.
41.8
46.8
$7.1
32.8
R3.0esa
25.1
28 333
23.3
23.
sity.
CIFAR 100
57 57.0
46.0
NES outperforms the deep ensemble of the best architecture in the NAS-Bench-201 search space (Figure 6). Next, we compare NES to deep ensembles over the NAS-Bench-
ImageNis-16-11 16-17
0
1991
45.9
50.5
47. 47.9
(b) M= 3; NAS-Bench-201 search space.
201 search space, which has two benefits: demonstrate that our findings not specific to the DARTS search space, and MuSS-Benc--2a is an exhaustively evaluated search space for which all architectures' trained weights are available (three initializations per architecture), allowing us to compare NES to the deep ensemble of the best architecture by validation loss. Results shown in F compare the losses of the ensemble, average base learner and oracle ensemble versus the number of networks evaluated K. Interestingly, although DeepEns (best arch.) has a significantly stronger average base learner than the other methods, its lack of diversity, as indicated by higher oracle ensemble loss (Figure (6) and lower predictive disagreement (Figure []. yields a weaker ensemble than both NES algorithms. Also, NES-RE outperforms NES-RS with a 6.6x speedup as shown in Figure left.
5.2 Analysis and Ablations
Why does NES work? What if deep ensembles use ensemble selection over initializations? (Figure 7). NES algorithms differ from deep ensembles in two important ways: the ensembles use
8

---

Table 3: Diversity and base learner strength. Pre- dictive disagreement (larger means more diverse) and oracle ensemble NLL (smaller means more diverse) are defined in Section 3 Despite the stronger base learners, deep ensembles tend be less diverse than ensembles constructed by NES. The results are consistent across datasets and shift severities (Appendix C. Best values and all values within 95% confidence interval are bold faced.
Mathod
setie trainee
Exscasbie
DeepEas (DARIS)
32
10
Desplins ES (DARTS)
52
2:0
Despiax (Amochulict)
Deepliis (AmoeebNen)
25300
Deopliis B8 (85)
NES 8S 2003
200
NES RS (X: 4003
400
NESS-RR 100)
180
NES RE
430
29
Figure 7: Comparison of NES to deep ensem- bles with ensemble selection on Tiny ImageNet. LEFT: NLL vs ensemble size. RIGHT: Cost for Tiny ImageNet experiments reported in terms of the number of networks trained when M = 10. The "arch" column indicates the number of net- works trained to first select an architecture, and the "ensemble" column contains the number of networks trained to build the ensemble.
Dustant Metrie
Method Swith M
Pi)
Deppins
Dreplize Dipplinc Anchvnofiis
NES-RS
NBS
04S0
(Amre))
Prod Disag
CUSSL vo.
0.987
01:37
0.513
CHIRR-I Ocadle NLL
0,120
0.108
0.0013
0,118
0.086.s.x 0 0.08
Prid Daagg
0.53 ver 0.966
01003
0.838
0.901L we 0.3
CIRAR-100 Coadl NLL
6483 sa 0.302
01:00
0.385
1 wn O 3877
Ang. 14l NL.
1,40,
1.450
1,200
1.47700x
ImggNNet
Thy
Onudio NLL
Prod. Doagt 0.742-4.00
015Mic wo 0U5R7
3.757
0.749
1
0.002
1.305
0.773 an 0.768amcm
0, wn 0.M00gaea
Ang, 2el. NLL 1,768-4
1.784
1813
1,81
1,756-a 1.728
varying architectures and NES utilizes ensemble selection (Forwardšelect) to pick the base learners. On Tiny ImageNet over the DARTS search space, we conduct an experiment to explore whether the improvement offered by NES over deep ensembles is only due to ensemble selection. The baselines "DeepEns+ + ES" operate follows: we optimize a fixed architecture for the base learners, train K' random initializations of it to form a pool and apply ForwardSelect to select an ensemble of size M. Figure left shows that both NES algorithms (each shown for two computational budgets K = 200, 400) outperform all DeepEns + ES baselines. Figure right contains the cost of each method in terms of the number of networks trained. Note that DeepEns ES (RS) is the most competitive of the deep ensemble baselines, and, at an equal budget of 400, it is outperformed by both NES algorithms. However, even at half the cost (200), NES-RE outperforms DeepEns ES (RS) while NES-RS performs competitively. As expected, deep ensembles with ensemble selection consistently perform better than without ensemble selection the expense of higher computational cost, but do not close the gap with NES algorithms. We also re-emphasize that comparisons between NES and deep ensembles in our experiments always fix the base learner training routine and method for composing the ensemble, so variations in ensemble performance are only due to the architecture choices. Therefore, to summarize, combined with the finding that NES outperforms deep ensembles even when NES' average base learner is weaker (as in e.g. Figure and Table we find architectural variation in ensembles to be important for NES' performance gains.
Tiny ImageNet
Computational cost and parallelizability. The primary computational cost involved in NES is training K networks to build the pool P. In our experiments on the DARTS search space, we set K to be 400 for CIFAR- 10/100 and 200 for Tiny ImageNet (except Figure 7 which additionally considers K = 400). Figure right gives an example of costs for each method. The cost of a deep ensemble with an optimized architecture stems from the initial architecture search and the subsequent training of M random initializations to build ensemble. We refer to Appendix [B for details and discussion of computation cost, including training times. We note that apart from DeepEns (DARTS) and AnchoredEns (DARTS), NES has a lower computational cost than the baselines in our experiments. Similar to deep ensembles, training the pool for NES-RS is embarrass- ingly parallel. Our implementation of NES-RE is also parallelized as described in Appendix
3.50
NES-RE
NES
Ensemble NLL
3.00
2.50
2.00
1.50
10a
10a
10
Validation size
Figure 8: Test perfor- mance of NES algo- rithms with varying val- idation data sizes. Each curve corresponds to one particular dataset shift severity (0-5 with 0 being no shift). The more trans- parent curves correspond higher shift severities.
Comparison of different ensemble selection algorithms. Both NES algorithms utilize ForwardSelect as the ensemble selection algorithm (ESA). We experimented with various other choices of ESAs, including weighted averaging and explicit diversity regularization, as shown in Figure 9 A detailed description of each ESA is in Appendix C.8 In summary, our choice of ForvardSelect performs better than or at par with all ESAS considered. Moreover, weighted averaging using stacking and Bayesian model averaging has
9

---

0.33
(EsterrSeedd RCS (Tip Mi
0.165
0.160
0.31
HSS if
HSS (irneigttdd seding
0.155
42-45 cowesciion
0.150
0.145
0.32
0.125
0.50
0.55
Ensemble (M)
Ensemble
I-NNE dimension
Ensemble selection
(b) Ensemble selection with explicit diversity regularization.
Figure 10: LEFT: t-SNE of the test predictions of 20 single random mutations (circles) from parent architecture (stars). RIGHT: NLL and error achieved by these architectures.
algorithms.
Figure 9: NES-RS on CIFAR-10.
very minor impact since the weights end up being close to uniform. Explicit diversity regularization, as shown in Figure 99) appears to slightly improve performance in some cases, provided that the diversity regularization strength hyperparameter is appropriately tuned.
We study the sensitivity of NES to the size of D Specifically, we measure test loss of the ensembles selected using Dval of different sizes (with as few as 10 validation samples). The results in Figure [S] indicate that NES is insensitive to validation size for different levels of dataset shift severity, achieving almost the same loss when using 10x fewer validation data. We provide more details in Appendix C.5 and we also discuss why overfitting to Daal is averted during ensemble selection.
NES is insensitive to the size of validation dataset Dval-
Mutations of an architecture are similar in function space and by performance. To explore how NES-RE traverses the search space, we analyzed how mutations affect an architecture as follows. We sampled five random architectures from the DARTS space, and for each one, we applied a single random mutation twenty times, yielding "family" of parent-children architectures, which are then trained on CIFAR- 10. Figure Ieft shows the result of t-SNE applied to the test predictions of these architectures, where each color corresponds to one "family", of which the "star" is the parent architecture and the circles are the children architectures. The clustering demonstrates that architectures which differ by only a single mutation similar in function space after training. Figure right shows the NLL and error achieved by these architectures. Again, similar clustering shows that architectures differing by single mutation also perform similarly w.r.t. NLL and error. This confirms that mutations allow for locally exploring the search space.
Further experiments. We also provide additional experiments in the Appendix. This includes comparison of ensembles built by averaging logits vs. probabilities (Appendix C.7). a comparison of NES to ensembles with other hyperparameters being varied (either width/depth training hyperpa- rameters similar to Wenzel et al. (531) (Appendix C.4) and using a weight-sharing model [2] as a proxy to accelerate the search phase in NES (Appendix C.9
Conclusion, Limitations & Broader Impact
We presented Neural Ensemble Search for automatically constructing ensembles with varying archi- tectures and demonstrated that the resulting ensembles outperform state-of-the-art deep ensembles in terms of uncertainty estimation and robustness to dataset shift. Our work highlights the benefit of ensembling varying architectures. In future work, aim to address the limitation posed by the computational cost of NES due to building the pool P. An interesting approach in this direction could be the use of differentiable NAS methods simultaneously optimize base learner architectures within a one-shot model and reduce cost other hyperparameters can be varied to improve ensemble performance and how best select them.
33 56 g More generally, we also hope to explore what
Our work can readily be applied to many existing ensemble-based deep learning systems to improve their predictive performance. This work also focuses on improving uncertainty estimation in neural networks, which is key problem of growing importance with implications for safe deployment of such systems. We are not aware of any direct negative societal impacts of our work, since NES is task-agnostic and its impact depends on its applications.
10

---

Acknowledgments and Disclosure of Funding
AZ, TE and FH acknowledge support by the European Research Council (ERC) under the European Union Horizon 2020 research and innovation programme through grant no. 716721, and by BMBF grant DeL SZ acknowledges support from Aker Scholarship. CH wishes to acknowledge support from The Alan Turing Institute, The Medical Research Council UK, and the EPSRC Bayes4Health grant. The authors acknowledge support from ELLIS. We also thank Julien Siems for providing parallel implementation of regularized evolution and Bobby He for useful comments on the manuscript.
References
[1] Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in in-domain uncertainty estimation and ensembling in deep learning. In International Conference on Learning Repre- sentations (ICIR)) 2020.
[2] Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quac Le. Understanding and simplifying one-shot architecture search. In International Conference on Machine Learning, 2018.
[3] Yijun Bian and Huanhuan Chen, When does Diversity Help Generalization in Classification Ensembles? ArXiv, abs/1903.06220 2019.
[4] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In Proceedings of the 32nd International Conference on Machine Learning. volume 37 of Proceedings of Machine Learning Research, pages 1613-1622, Lille, France, 07 07-09 Jul 2015. PMLR.
[5] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Fimer, Beat Flepp. Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End End Learning for Self- Driving Cars. CoRR, obs//160,, 2016.
[6] Lco Breiman. Random forests. In Machine Learning, pages 5-32, 2001.
[7] Rich Caruana, Alexandru Viculescu-Mizit Geoff Crew, and Alex Ksikes. Ensemble Selection from Libraries of Models. In Proceedings of the Twenty-First International Conference Machine Learning. ICML. '04, page 18, New York, NY, USA, 2004. Association for Computing Machinery.
[8] Xiangning Chen, Ruochen Wang, Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsieh. Dr(nas): Dirichlet neural architecture search. International Conference on Learning Representations, 2021.
[9] Patryk Chrabaszez, Nly Loshchilov, and Frank Hutter. A downsampled variant of imagenet alternative to the cifar datasets. arXiv preprint MXKv:1707.08811 2017.
[10] Corinna Cortes, Xavier Gonzalvo, Vitaly Kuznetsov, Mehryar Mohri, and Scott Yang. AdaNet: Adaptive Structural Learning of Artificial Neural Networks. In Proceedings of the 34th International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pages 774, International Convention Centre, Sydney, Australia, 06 11 Aug 2017. PMLR.
[11] Thomas Dietterich. Ensemble Methods in Machine Learning. In Multiple Classifier Systems, pages 1-15, Berlin, Hcidelberg, 2000. Springer Berlin Heidelberg.
[12] Xuanyi Dong and Yi Yang. Searching for a Robust Neural Architecture in Four GPU Hours. In Computer Vision and Pattern Recognition (CVPR), pages 1761-1770, 2019.
[13] Xuanyi Dong and Yi Yang. Nas-hench-201: Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations (ICLR), 2020.
[14] Thomas Blsken, Jan Hendrik Metzen, and Frank Hutter. Neural Architecture Scarch: A Survey. Journal of Machine Learning Research, 20(55):1-21, 2019.
[15] Andre Esteva, Brett Kuprel, Roberto A. Novoa, Justin Ko, Susan M. Swetter, Helen M. Blau, and Sehastian Thrun. Dertatologist-level classification of skin cancer with deep neural networks. Nature. 552:115-, 2017.
[16] Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and Efficient Hyperparameter Optimization at Scale. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1437- 1446, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018 PMLR.

---

[17] Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems 28, pages 2962-2970. Curran Associates, Inc., 2015.
[18] Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep Ensembles: A Loss Landscape Perspective. ArXiv, abs/191 122.02777 2019.
[19] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050-1059, New York. New York, USA, 20-22 Jun 2016. PMLR.
[20] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
[21] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321-1330, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR.
[22] Fredrik K Gustafsson, Martin Danelljan, and Thomas Schön. Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
[23] Lars K. Hansen and Peter Salamon. Neural network ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(10):993-000) 1001, 1990.
[24] Bobby He, Balaji Lakshminarayanan, and Yee Whye Teh. Bayesian deep ensembles via the neural tangent kernel. Advances in Neural Information Processing Systems, volume 33, pages 1010-1022. Curran Associates, Inc., 2020.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedingy of the IEEE Conference on Computer Vision and Puttern Recugnition, pages 770-778, 2016.
[26] Dan Hendrycks and Thomas Dietterich. Benchmarking Neural Network Robustness Common Corrup- tions and Perturbations. International Conference on Learning Representations (ICLR), 2019.
[27] Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John Hopcroft, and Kilian Weinberger. Snapshot Ensembles: Train 1, get m for free. In International Conference on Learning Representations (ICLR), 2017.
[28] F. Hutter, L. Kotthoff, and J. Vanschoren, editors. Automated Machine Learning: Methods, Systems, Challenges. Springer, 2018. In press, available at http://automl.orgbbookk
[29] Siddhartha Jain, Liu, Jonas Mueller, David Giffford. Maximizing overall diversity for improved uncertainty estimates deep ensembles. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):4261-4271 Apr. 2020.
[30] Zhengshen Jiang, Hongzhi Liu, Bin Fu, and Zhonghai Wu. Generalized Ambiguity Decompositions for Classification with Applications in Active Learning and Unsupervised Ensemble Pruning. In AAAI, pages 2073-20 1773-2079.42 AAAI Press, 2017.
[31] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 10 (canadian institute for advanced research), 2009.
[32] Anders Krogh and Jesper Vedelsby. Neural Network Ensembles, Cross Validation, and Active Learning. In Advances in Neural Information Processing Systems 7, pages 231-238. MIT Press, 1995.
[33] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. In Advances Neural Information Processing Systems 30, pages 6402-641 6413. Curran Associates, Inc., 2017.
[34] Y. Le and X. Yang. Tiny imagenet visual recognition challenge, 2015.
[35] Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks, arXiv e-prints, page arXiv: 1.16111111 2015.
12

---

[36] Julien-Charles Lévesque. Christian Gagné, and Robert Sabourin. Bayesian hyperparameter optimization for ensemble learning. In Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence, UAI' 16, page 437-446, Arlington, Virginia, USA, 2016. AUAI Press.
[37] Liam Li and Ameet Talwalkar. Random Search and Reproducibility for Neural Architecture Search. In UAI, page 129. AUAI Press, 2019.
[38] Marius Lindauer and Frank Hutter. Best practices for scientific research neural architecture search. arXiv preprint arXiv: 190,00499 2019.
[39] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable Architecture Search. In International Conference on Learning Representations (ICLR), 2019.
[40] Y. Liu and X. Yao. Ensemble leamning via negative correlation. Neural networks the jourralljuuraall of the International Neural Network Society, 12 111119994444 1999.
[41] Iyy Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. In International Conference on Learning Representations (ICLR), 2017.
[42] Vladimir Macko, Charles Weill, Hanna Mazzawi, and Javier Gonzalvo. Improving Neural Architecture Scarch Image Classifiers via Ensemble Learning. ArXiv, 0bs//99330023366 2019.
[43] Hector Mendoza, Aaron Klein, Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Towards Automatically-Tuned Neural Networks. In ICML 2016 AutoMI. Workshop, 2016.
[44] Mahdi Pakdaman Nacini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining Well Calibrated Prob- abilities using Bayesian Binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI' 15, page 2901-2907. AAAI Press, 2015.
[45] Randal S. Olson and Jason H. Moore. Tpot: tree-hased pipeline optimization tool for automating machine learning. In Workshop on Automatic Machine Learning, volume 64 of Proceedings of Machine Learning Research, pages 66-74, New York, New York, USA, 24 Jun 2016. PMLR.
[46] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems 32, pages 13991- 14002. Curran Associates, Inc., 2019.
[47] Tim Pearee, Felix Leibfried, and Alexandra Brintrup. Uncertainty in neural networks: Approximately bayesian ensembling. In Proceedings of the Twenty Third International Conference Artificial Intelligence and Statistics, volume 108 Proceedings of Machine Learning Research, pages 234 244, Online, 26 28 Aug 2020. PMLR.
[48] Estchan Real, Alok Aggarwal, Yanping Huang, and Quoe V. Lc. Regularized Evolution for Image Classifier Architecture Search. In AAAI, pages 4780-4789. AAAI Press, 2019.
[49] Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. In International Conference on Learning Representations (ICLR), 2015.
[50] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(577-2205)::55 2008
[51] Andrew M. Webb, Charles Reynolds, Dan-Andrei Ilieseu, Henry W. J. Reeve, Mikel Luján, and Gavin Brown, Joint Training of Neural Network Ensembles. CoRR, abs/1902.04422. 2019.
[52] Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML'11, 11, page 681 688, Madison, WI, USA, 2011. Omnipress.
[53] Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles robust- ness and uncertainty quantification. In Advances in Neural Information Processing Systems, volume 33, pages 65 14 6527, Curran Associates, Inc., 2020.
[54] Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of general- ization. In Advances in Neural Information Processing Systems, volume 33, pages 4697-4708. Curran Associates, Inc., 2020.
[55] Han Xiao, Kashif Rasul, and Roland Vollgraf. Machine Learning Algorithms. arXiy e-prints, 2017.
A Novel Image Dataset for Benchmarking
13

---

[56] Yuhui Xu, Lingxi Xie. Xiaopeng Zhang, Xin Chen, Guo-Jun Qi. Qi Tian, and Hongkai Xiong. PC-DARTS: Partial Channel Connections for Demoor-Effficint Architecture Search. In International Conference Learning Reprexentations (ICLR), 2020.
[57] Antoine Yang. Pedro M. Esperança, and Fabio M. Carlucci. NAS evaluation is frustratingly hard. In International Conference on Learning Representations (ICLR), 2020.
[58] Kaicheng Yu, Christian Sciuto, Martin Jaggi. Claudiu Musat, and Mathieu Salzmann. Evaluating the Search Phase Neural Architecture Search. International Conference on Learning Representations (ICLR), 2020.
[59] Arber Zela, Julien Siems, and Frank Hutter. Nas-bench- Ishol:: Benchmarking and dissecting one-shot neural architecture search. In International Conference on Learning Representations, 2020.
[60] Tianyi Zhou, Shengjie Wang, and Jeff' A Bilmes. Diverse Ensemble Evolution: Curriculum Data-Model Marriage. Advances Neural Information Processing Systems 31, pages 5905-5916. Curran Associates, Inc., 2018.
[61] Zhi-Hua Zhou. Ensemble Methods: Foundations and Algorithms. Chapman & Hall, 1st edition, 2012.
[62] Zhi- Zhou, Jianxin Wu, and Tang. Ensembling neural networks: Many could better than all. Artificial Intelligence, 137(1):239 2633, 2002.
[63] Barret Zoph and Quoc Le. Neural Architecture Search with Reinforcement Learning. In International Conference on Learning Representations (ICLR), 2017.
[64] Barret Zoph. Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning Transferable Architectures for Scalable Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 8697-8710, 2018.
14


Reviews:
[{"review_id": "c0kzoSU8OK", "reviewer": "Reviewer_28yA", "summary": "This paper proposes a novel approach for searching ensembles of neural networks to improve the quality of uncertainty estimation and  performance improvement under the dataset shift. The main idea of the paper is to construct a pool of architectures, from which a new ensemble member is selected using a forward greedy algorithm. The paper is empirical, and all claims are strongly supported by thorough experiments. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First, this is a great paper, and it has been my pleasure to read it. I also highly appreciate the authors releasing the code. \n\n\nI have only two points that would be great to discuss (or add additional results if possible) in the camera-ready version:\n\n1. What is the effect of data augmentation? I have understood from the paper that no augmentations have been used. Why is that so? Is there an ablation study in the supplementary that I am missing?\n\n2. It would be important to see the results on selecting an ensemble with NAS when $D_{val}^{shift}$ is not used for model selection. Since the proposed method used the $D_{val}^{shift}$, and Deep Ensembles did not, it does not seem to be a fair comparison. One can easily add a term to DEs model selection where $D_{val}^{shift}$ is used. Please discuss this in the paper.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main limitation of this work is the computational cost of the introduced method. This, however, has been discussed by the authors. Other limitations include the concerns raised in the main review, and may only require textual changes in the manuscript.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "aii0bmOUfod", "reviewer": "Reviewer_dkBq", "summary": "This paper proposes neural network ensembles with varying architectures, arguing these will be produce better uncertainty estimates, and considers various straightforward techniques to aggregate (ie. ensemble) the predictions from a pool of independently trained heterogeneous neural network architectures. The aggregation algorithm in this work is simply Ensemble Selection from Caruana et al., and random search / evolution methods are considered for automatically constructing the neural architectures in the base learner pool, rather than requiring these are predefined.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Two key questions are considered in this paper:\n\nQ1) How to generate a pool of neural networks to ensemble?\nQ2) How to aggregate the predictions from this NN pool into a single prediction?\n\nThere is little theory or conceptual justification for the authors' choices to address each question, so the empirical comparison must be sufficiently convicing to justfy the contribution of this paper. However, I do not find the comparisons sufficently thorough in the current paper.\n\nFurthermore, the novelty of the proposed methods is limited (which would not be a problem if the proposed straightforward methods showed clear superiority over existing state of the art, but this is not clearly demonstrated in my opinion). A standard approach in eg AutoML systems is to do hyperparameter optimization (HPO) followed by ensembling of the models corresponding to best hyperparameter candidates. This work essentially considers the same framework, except the hyperparameter search space now includes neural architectures and rather than Bayesian optimization HPO algorithms, simple NAS algorithms are considered for searching through this space.\n\nRegarding Q2: it is unclear what is specific to the base learners being NN models, and it seems like any arbitrary ensembling algorithm could apply. The authors here advocate for Ensemble Selection (ES) from Caruana et al., which is certainly a reasonable choice, but it would be good to see more comparisons against alternative standard ensembling algorithms. The alternative ensembling algorithms considered by the authors as baselines vs. ES are too limited and non-adaptive unlike many ensembling algorithms which exist today (and where are the empirical results that back up the claim: \" these three performed comparatively or worse\n204 than our choice ForwardSelect\"?). Here are some straightforward alternatives off the top of my head:\n\nA) Train a linear model or neural network to combine the predictions. Ie. stacking, unclear why this left for future work as it is simple to try out, see for example:\n\nAutoGluon-Tabular: Robust and Accurate AutoML for Structured Data.\nErickson et al. 2020.\nhttps://arxiv.org/abs/2003.06505\n\nwhere this has been shown to be highly performant method for ensembling heterogeneous base learners. Also blending is another standard alternative to stacking. AutoGluon also appears to use Bayesian optimization to tune architectural hyperparameters of NN base learners (contrary to what the authors write in the \"AutoML and ensembles of varying architectures\" section).\n\nB) Various voting ensemble schemes that have been previously proposed, eg:\nhttps://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html\n\nFurthermore, it seems promising to exploit the fact that the base learners are all NN models to consider more interesting differentiable aggregators such that more interesting overall strategies can be considered like those of Webb et al. [49]. For example, rather than purely ensembling based on validation score, perhaps the aggregator should also consider some properties of the architectures being ensembled which might lead to better OOD uncertainty estimation, something that is not easily achieved solely thru validation-score ensemble construction.\n\nThe authors should also justify why they aggregate predicted probabilities instead of say logits, and why a simple average of these is considered rather than say a weighted combination. For example: in the baseline \"Select the top M networks by validation performance\", will performance improve if the resulting predictions of the networks weighted by their validation performance in the aggregator (as in Bayesian model averaging)?\n\n\nRegarding Q1:\n\n- As a baseline, why not use standard NAS algorithms (eg. ENAS, ProxylessNAS) to generate the pool? This could still be followed by ensemble-selection just as applied to pools considered in this paper. I suppose one issue is the weight-sharing scheme in standard algorithms like ENAS but it would still be insightful to see how ensembles of these weight-shared candidates perform.\n\n- Does the pool improve if bootstrapping is used to train each NN base learner on a different subsampled dataset? This would presumably increase diversity.\n\n- For efficiently generating more pool candidates, you could also consider ensemble selection applied to all NN checkpoints (eg. every few epochs) rather than only fully trained NN base learners. Such checkpoint-ensembles are known to be superior to individual NN models.\n\n\n- Other comparisons the paper should consider:\n\nWhile briefly cited as a related work, Webb et al. 2019 [49] needs to be discussed in more detail, and conceptually constrasted against the proposed ideas in this paper. You only cite [49] as evidence that joint training of ensemble is worse than indpeendent training of base learners, but [49] does a much deeper analysis and finds some amount of joint training (ie. end-to-end learning) combined with independent training can lead to the best results.\n\nAppendix C.4 should also compare against another version of HyperEns, where similar architectural design search spaces as considered in NES are considered as hyperparameters to be ensembled in HyperEns. This is a trivial modification of HyperEns and is a highly relevant baseline to compare against (probably should even appear in the main paper).\n\n- How were hyperparameters chosen in this work?\n\n- When it comes to epistemic uncertainty (ie. model uncertainty), the typical assumption is that model family (ie. hypothesis class) is correctly specified and we just don't have enough data to properly identify the right model from the class (this is sometimes called the \"M-closed\" assumption in Statistics). A major advantage of ensembling heterogeneous models as done here would be the ability to provide better uncertainty estimates under model misspecification, which is always present in practice (especially with neural network models). I think the authors could mention this connection and cite some related literature in this area. I'm not sure Proposition 3.1 is the best way to motivate having more diverse ensembles, as there are other standard arguments in favor of diversity presented in the ensembling literature (eg. statistical independence of base learner errors).\n\n\n- A remaining concern I have is the following: This paper seems to train networks on D_train and the construct ensemble based on D_val, and finally score the ensemble on D_test. However an alternative standard procedure is to refit all networks on D_train+D_val before evaluating on D_test. I suspect DeepEnsembles would fare better under this alternative than Ensemble-Selection whose outperformance is more contingent on the base learners behaving similarly when fit to just D_train vs D_train+D_val. It would be good to address this point.\n\n\n- There is also much related work in which ensembles of heterogeneous NN architectures have been found useful that could be referenced. Here's just a couple papers I found thru google:\n\nEnsembles of Networks Produced from Neural Architecture Search. \nHerron et al. 2020.\nhttps://link.springer.com/chapter/10.1007/978-3-030-59851-8_14\n\nEnsembling neural networks: Many could be better than all.\nZhou et al. 2002.\nhttps://www.sciencedirect.com/science/article/pii/S000437020200190X?via%3Dihub\n\nLabel Denoising with Large Ensembles of Heterogeneous Neural Networks.\nOstyakov et al. 2019.\nhttps://arxiv.org/abs/1809.04403\n\nAntibody Complementarity Determining Region Design Using High-Capacity Machine Learning.\nLiu et al. 2020.\nhttps://academic.oup.com/bioinformatics/article/36/7/2126/5645171\n\nSolo or Ensemble? Choosing a CNN Architecture for Melanoma Classification. \nPerez et al. 2019.\nhttps://openaccess.thecvf.com/content_CVPRW_2019/html/ISIC/Perez_Solo_or_Ensemble_Choosing_a_CNN_Architecture_for_Melanoma_Classification_CVPRW_2019_paper.html\n\nAn ensemble of neural networks for weather forecasting. \nMaqsood et al. 2004.\nhttps://link.springer.com/article/10.1007/s00521-004-0413-4\n\nNADS: Neural Architecture Distribution Search for Uncertainty Awareness. \nArdywibowo et al. 2020.\nhttps://arxiv.org/abs/2006.06646\n\n\n### Update after reading author response to my original review ###\n\nI am glad to see the authors' additional experiments that addressed my major concerns and have increased my score. I encourage the authors to include all of the points in their response in the revised paper (can squeeze these into appendix as necessary), as I think they will be useful to future readers. And I encourage the authors to tone down any claims of being the first work to ensemble diverse architectures and instead highlight the novelty of this work in terms of discovering the utility of architecture-ensembles for better uncertainty estimation and their new ensemble-aware architecture search algorithm.\n\nRegarding: \"perhaps the aggregator should also consider some properties of the architectures being ensembled which might lead to better OOD uncertainty estimation, something that is not easily achieved solely thru validation-score ensemble construction\"\n\nOne idea could be look at the correlation/variance in the individual ensemble-members predictions as an auxiliary diversity objective (or take a weighted combination of validation-score objective and diversity objective). Some examples are considered in these works:\n\nBrown, G. 2004. Diversity in neural network ensembles. Ph.D. Dissertation, University of Birmingham.\n\nJain et al. 2020. Maximizing Overall Diversity for Improved Uncertainty Estimates in Deep Ensembles. https://arxiv.org/abs/1906.07380 ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I believe the authors have adequately addressed societal impact of their work.\nHowever a key limitation of the proposed ensembling is that inference latency will be slower for deploying a neural ensemble vs a single network, and may not be suitable for real-time applications. It would be insightful if the authors could compare their ensemble against a single large network where both models are restricted to a total number of NN parameters (summed across all ensemble members in the former case) or restricted to the same inference latency.\n", "review_text": "", "extra_scores": {"ethical_concerns": "I don't have any remaining omitted ethical concerns regarding this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "ZuMelffnlY", "reviewer": "Reviewer_xqD9", "summary": "The paper proposes neural ensemble search (NES) for adding architecture variations in forming ensembles. The main observation is that the architecture variations can be more helpful in terms of accuracy and uncertainty estimation, due to their potentially more diverse predictions, compared with constructing ensembles with a fixed architecture and based on random initializations.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Improving uncertainty estimation of neural networks which the paper targets is an important problem. The proposed method builds a pool of architectures, either by random sampling or an evolutionary method, and selects an ensemble of smaller size with a greedy approach. Both the evolutionary method and greedy ensemble selection are based on previous work, without any major modification.\n\nUsing diverse models for predictive tasks is common but the work seems to be the first to systematically focus on and evaluate the effect of varying architectures in ensembles within a NAS space on improving accuracy and uncertainty estimates.\n\nThe paper is written clearly and describes the methods and choices made in the design of the method well. It performs several ablation studies to show the effect of the different parts of the method and details about the experimental setups are provided.\n\nThere are some improvements that can be made in the experiments.\nFor the DARTS dataset shift experiments, to isolate the effect of ensembles based on one architecture vs. multiple, DeepEns can be tested with the best architecture according to error on shifted validation data among architectures trained in the proposed NES methods. \n\nIn Figure 7, showing only the number of networks trained may not be the best metric as some methods are embarrassingly parallel. Also, why the DeepEns (RS) versions are expected to perform better than other DeepEns versions when there is no dataset shift? The ablation study is important in empirically validating the paper’s claim, so it would be good to report similar results for more than one setup and dataset and also to test DeepEns with the best architecture among the ones which NES methods select from.\n\nThe ECE results (Figure 5), and predictive disagreement (Table 2) do not conclusively demonstrate that the proposed methods are performing better in terms of uncertainty and diversity. Experiments on common OOD setups can help the claims about better uncertainty estimation.\n\nOverall, although the methodological contributions of the paper are not very significant, the execution and the results are valuable. \n\n\n--After author response--\n\nThanks to the authors for their replies. Including the additional results and discussion will further improve the paper and support its claims. I remain positive about the paper. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Sox7H5ilhh", "reviewer": "Reviewer_WTyi", "summary": "This paper introduces ‘Neural Ensemble Search’, a method for (automatically) constructing ensembles of NNs which (in contrast with most iterations of deep ensembles) generally do not have the same architecture. The authors show that their method consistently outperforms baseline ensemble methods in accuracy and uncertainty calibration. They hypothesize that the success of this method is mainly due to the varied architectures leading to increased diversity of the predictive distributions of ensemble members, and they provide evidence supporting this claim.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "From a practical standpoint, the method proposed offers modest gains over baselines, particularly considering the high training cost associated with producing the pool of candidate models (though this is less important than test-time cost, which is comparable to other methods, and much of the training can be parallelized). \n\nHowever, the contributions of this paper largely outweigh this limitation. The relationship between diversity and performance has emerged as an important topic in understanding ensembles. This paper takes a significant step in addressing the effect of varied architectures on ensembles, which has not been previously studied in detail to my knowledge. The authors are thorough in their experiments, both to show their method outperforms baselines and to attempt to isolate the effects of architecture.\n\nSome ideas that could potentially strengthen this submission:\n\n* Experimental results for ensemble methods in which architecture varies, but ‘ForwardSelect’ is not used. Some examples are described in lines 198-204. Such results would complement the ablation experiments in which you use ForwardSelect over initializations for Deep Ensembles, and (one hopes) further support the claim that the variation in architecture, rather than the selection algorithm, is the main reason for the success of this method\n\n* More analysis focusing on the architecture in detail. For example, is there a clear relationship between how different two models’ architectures are and how diverse their predictions are? I would be very interested in a t-SNE or UMAP plot, similar to that in Figure 2a, where each dot in a color represents a single mutation from the initial architecture instead of a different initialization of it. As another example, are there characteristics that tend to be shared among high performing baseline learners, and do similar architectures (differing by only a few mutations) have similar performances? Such insights could conceivably be leveraged into even better strategies for producing a pool of models and for model selection.\n\n** Update after author response period ** Thanks to the authors for their replies. I remain positive about the work and keep my original recommendation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends prior work on deep ensembles by introducing methods for constructing ensembles of varying architectures instead of relying on multiple random nationalizations of the same architecture. The methods introduced in the paper represent a novel combination of existing approaches applied specifically to developing architecturally diverse ensembles to improve uncertainty estimation. The writing is clear and the methods are technically correct. The authors present extensive experiments showing that the proposed method outperforms prior approaches. The reviewers had many questions and suggestions for the authors in their initial reviews. Following the discussion, the reviewers were in agreement that their primary questions had been adequately addressed and that the paper should be accepted. The authors need to be sure to include all of the discussed updates in the final version of the paper.

Decision: Accept (Poster)