Neural Ensemble Search for Uncertainty Estimation and Dataset Shift

1 Introduction . Some applications of deep learning rely only on point estimate predictions made by a neural network . However , many critical applications also require reliable predictive uncertainty estimates and robustness under the presence of dataset shift , that is , when the observed data distribution at deployment differs from the training data distribution . Examples include medical imaging [ 15 ] and self-driving cars [ 5 ] . Unfortunately , several studies have shown that neural networks are not always robust to dataset shift [ 46 , 26 ] , nor do they exhibit calibrated predictive uncertainty , resulting in incorrect predictions made with high confidence [ 21 ] . Deep ensembles [ 33 ] achieve state-of-the-art results for predictive uncertainty calibration and robustness to dataset shift . Notably , they have been shown to outperform various approximate Bayesian neural networks [ 33 , 46 , 22 ] . Deep ensembles are constructed by training a fixed architecture multiple times with different random initializations . Due to the multi-modal loss landscape [ 18 , 54 ] , randomization by different initializations induces diversity among the base learners to yield a model with better uncertainty estimates than any of the individual base learners ( i.e . ensemble members ) . Our work focuses on automatically selecting varying base learner architectures in the ensemble , exploiting architectural variation as a beneficial source of diversity missing in deep ensembles due to their fixed architecture . Such architecture selection during ensemble construction allows a more “ ensemble-aware ” choice of architectures and is based on data rather than manual biases . As ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . discussed in Section 2 , while ensembles with varying architectures has already been explored in the literature , variation in architectures is typically limited to just varying depth and/or width , in contrast to more complex variations , such as changes in the topology of the connections and operations used , as considered in our work . More generally , automatic ensemble construction is well-explored in AutoML [ 17 , 36 , 45 , 43 ] . Our work builds on this by demonstrating that , in the context of uncertainty estimation , automatically constructed ensembles with varying architectures outperform deep ensembles that use state-of-the-art , or even optimal , architectures ( Figure 1 ) . Studied under controlled settings , we assess the ensembles by various measures , including predictive performance , uncertainty estimation and calibration , base learner performance and two ensemble diversity metrics , showing that architectural variation is beneficial in ensembles . Note that , a priori , it is not obvious how to find a set of diverse architectures that work well as an ensemble . On the one hand , optimizing the base learners ’ architectures in isolation may yield multiple base learners with similar architectures ( like a deep ensemble ) . On the other hand , selecting the architectures randomly may yield numerous base learners with poor architectures harming the ensemble . Moreover , as in neural architecture search ( NAS ) , we face the challenge of needing to traverse vast architectural search spaces . We address these challenges in the problem of Neural Ensemble Search ( NES ) , an extension of NAS that aims to find a set of complementary architectures that together form a strong ensemble . In summary , our contributions are as follows : 1 . We present two NES algorithms for automatically constructing ensembles with varying base learner architectures . As a first step , we present NES with random search ( NES-RS ) , which is simple and easily parallelizable . We further propose NES-RE inspired by regularized evolution [ 48 ] , which evolves a population of architectures yielding performant and robust ensembles . 2 . This work is the first to apply automatic ensemble construction over architectures to complex , state-of-the-art neural architecture search spaces . 3 . In the context of uncertainty estimation and ro- bustness to dataset shift , we demonstrate that ensembles constructed by NES improve upon state-of-the-art deep ensembles . We validate our findings over five datasets and two architecture search spaces . 2 Related Work . Ensembles and uncertainty estimation . Ensembles of neural networks [ 23 , 32 , 11 ] are commonly used to boost performance . In practice , strategies for building ensembles include independently training multiple initializations of the same network , i.e . deep ensembles [ 33 ] , training base learners on different bootstrap samples of the data [ 62 ] , training with diversity-encouraging losses [ 40 , 35 , 60 , 51 , 29 , 47 ] and using checkpoints during the training trajectory of a network [ 27 , 41 ] . Despite a variety of approaches , Ashukha et al . [ 1 ] found many sophisticated ensembling techniques to be equivalent to a small-sized deep ensemble by test performance . Much recent interest in ensembles has been due to their state-of-the-art predictive uncertainty estimates , with extensive empirical studies [ 46 , 22 ] observing that deep ensembles outperform other approaches for uncertainty estimation , notably including Bayesian neural networks [ 4 , 19 , 52 ] and post-hoc calibration [ 21 ] . Although deep ensembles are not , technically speaking , equivalent to Bayesian neural networks and the relationship between the two is not well understood , diversity among base learners in a deep ensemble yields a model which is arguably closer to exact Bayesian model averaging than other approximate Bayesian methods that only capture a single posterior mode in a multi-modal landscape [ 54 , 18 ] . Also , He et al . [ 24 ] draw a rigorous link between Bayesian methods and deep ensembles for wide networks , and Pearce et al . [ 47 ] propose a technique for approximately Bayesian ensembling . Our primary baseline is deep ensembles as they provide state-of-the-art results in uncertainty estimation . AutoML and ensembles of varying architectures . Automatic ensemble construction is commonly used in AutoML [ 17 , 28 ] . Prior work includes use of Bayesian optimization to tune non-architectural hyperparameters of an ensemble ’ s base learners [ 36 ] , posthoc ensembling of fully-connected networks evaluated by Bayesian optimization [ 43 ] and building ensembles by iteratively adding ( sub- ) networks to improve ensemble performance [ 10 , 42 ] . Various approaches , including ours , rely on ensemble selection [ 7 ] . We also note that Simonyan & Zisserman [ 49 ] , He et al . [ 25 ] employ ensembles with varying architectures but without automatic construction . Importantly , in contrast to our work , all aforementioned works limit architectural variation to only changing width/depth or fully-connected networks . Moreover , such ensembles have not been considered before in terms of uncertainty estimation . Another important part of AutoML is neural architecture search ( NAS ) , the process of automatically designing single model architectures [ 14 ] , using strategies such as reinforcement learning [ 63 ] , evolutionary algorithms [ 48 ] and gradient-based methods [ 39 ] . We use the search spaces defined by Liu et al . [ 39 ] and Dong & Yang [ 13 ] , two of the most commonly used ones in recent literature . Concurrent to our work , Wenzel et al . [ 53 ] consider ensembles with base learners having varying hyperparameters using an approach similar to NES-RS . However , they focus on non-architectural hyperparameters such as L2 regularization strength and dropout rates , keeping the architecture fixed . As in our work , they also consider predictive uncertainty calibration and robustness to shift , finding similar improvements over deep ensembles . 3 Visualizing Ensembles of Varying Architectures . In this section , we discuss diversity in ensembles with varying architectures and visualize base learner predictions to add empirical evidence to the intuition that architectural variation results in more diversity . We also define two metrics for measuring diversity used later in Section 5 . 3.1 Definitions and Set-up . Let Dtrain = { ( xi , yi ) : i = 1 , . . . , N } be the training dataset , where the input xi ∈ RD and , assuming a classification task , the output yi ∈ { 1 , . . . , C } . We use Dval and Dtest for the validation and test datasets , respectively . Denote by fθ a neural network with weights θ , so fθ ( x ) ∈ RC is the predicted probability vector over the classes for input x . Let ` ( fθ ( x ) , y ) be the neural network ’ s loss for data point ( x , y ) . Given M networks fθ1 , . . . , fθM , we construct the ensemble F of these networks by averaging the outputs , yielding F ( x ) = 1M ∑M i=1 fθi ( x ) . In addition to the ensemble ’ s loss ` ( F ( x ) , y ) , we will also consider the average base learner loss and the oracle ensemble ’ s loss . The average base learner loss is simply defined as 1M ∑M i=1 ` ( fθi ( x ) , y ) ; we use this to measure the average base learner strength later . Similar to prior work [ 35 , 60 ] , the oracle ensemble FOE composed of base learners fθ1 , . . . , fθM is defined to be the function which , given an input x , returns the prediction of the base learner with the smallest loss for ( x , y ) , that is , FOE ( x ) = fθk ( x ) , where k ∈ argmin i ` ( fθi ( x ) , y ) . The oracle ensemble can only be constructed if the true class y is known . We use the oracle ensemble loss as one of the measures of diversity in base learner predictions . Intuitively , if base learners make diverse predictions for x , the oracle ensemble is more likely to find some base learner with a small loss , whereas if all base learners make identical predictions , the oracle ensemble yields the same output as any ( and all ) base learners . Therefore , as a rule of thumb , all else being equal , smaller oracle ensemble loss indicates more diverse base learner predictions . Proposition 3.1 . Suppose ` is negative log-likelihood ( NLL ) . Then , the oracle ensemble loss , ensemble loss , and average base learner loss satisfy the following inequality : ` ( FOE ( x ) , y ) ≤ ` ( F ( x ) , y ) ≤ 1 M M∑ i=1 ` ( fθi ( x ) , y ) . We refer to Appendix A for a proof . Proposition 3.1 suggests that it can be beneficial for ensembles to not only have strong average base learners ( smaller upper bound ) , but also more diversity in their predictions ( smaller lower bound ) . There is extensive theoretical work relating strong base learner performance and diversity with the generalization properties of ensembles [ 23 , 61 , 6 , 30 , 3 , 20 ] . In Section 5 , the two metrics we use for measuring diversity are oracle ensemble loss and ( normalized ) predictive disagreement , defined as the average pairwise predictive disagreement amongst the base learners , normalized by their average error [ 18 ] . 3.2 Visualizing Similarity in Base Learner Predictions . The fixed architecture used to build deep ensembles is typically chosen to be a strong stand-alone architecture , either hand-crafted or found by NAS . However , optimizing the base learner ’ s architecture and then constructing a deep ensemble can neglect diversity in favor of strong base learner performance . Having base learner architectures vary allows more diversity in their predictions . We provide empirical evidence for this intuition by visualizing the base learners ’ predictions . Fort et al . [ 18 ] found that base learners in a deep ensemble explore different parts of the function space by means of applying dimensionality reduction to their predictions . Building on this , we uniformly sample five architectures from the DARTS search space [ 39 ] , train 20 initializations of each architecture on CIFAR-10 and visualize the similarity among the networks ’ predictions on the test dataset using t-SNE [ 50 ] . Experiment details are available in Section 5 and Appendix B . As shown in Figure 2a , we observe clustering of predictions made by different initializations of a fixed architecture , suggesting that base learners with varying architectures explore different parts of the function space . Moreover , we also visualize the predictions of base learners of two ensembles , each of size M = 30 , where one is a deep ensemble and the other has varying architectures ( found by NES-RS as presented in Section 4 ) . Figure 2b shows more diversity in the ensemble with varying architectures than in the deep ensemble . These qualitative findings can be quantified by measuring diversity : for the two ensembles shown in Figure 2b , we find the predictive disagreement to be 94.6 % for the ensemble constructed by NES and 76.7 % for the deep ensemble ( this is consistent across independent runs ) . This indicates higher predictive diversity in the ensemble with varying architectures , in line with the t-SNE results .

Summary: This paper proposes neural network ensembles with varying architectures, arguing these will be produce better uncertainty estimates, and considers various straightforward techniques to aggregate (ie. ensemble) the predictions from a pool of independently trained heterogeneous neural network architectures. The aggregation algorithm in this work is simply Ensemble Selection from Caruana et al., and random search / evolution methods are considered for automatically constructing the neural architectures in the base learner pool, rather than requiring these are predefined.