Neural Ensemble Search for Uncertainty Estimation and Dataset Shift

1 Introduction . Some applications of deep learning rely only on point estimate predictions made by a neural network . However , many critical applications also require reliable predictive uncertainty estimates and robustness under the presence of dataset shift , that is , when the observed data distribution at deployment differs from the training data distribution . Examples include medical imaging [ 15 ] and self-driving cars [ 5 ] . Unfortunately , several studies have shown that neural networks are not always robust to dataset shift [ 46 , 26 ] , nor do they exhibit calibrated predictive uncertainty , resulting in incorrect predictions made with high confidence [ 21 ] . Deep ensembles [ 33 ] achieve state-of-the-art results for predictive uncertainty calibration and robustness to dataset shift . Notably , they have been shown to outperform various approximate Bayesian neural networks [ 33 , 46 , 22 ] . Deep ensembles are constructed by training a fixed architecture multiple times with different random initializations . Due to the multi-modal loss landscape [ 18 , 54 ] , randomization by different initializations induces diversity among the base learners to yield a model with better uncertainty estimates than any of the individual base learners ( i.e . ensemble members ) . Our work focuses on automatically selecting varying base learner architectures in the ensemble , exploiting architectural variation as a beneficial source of diversity missing in deep ensembles due to their fixed architecture . Such architecture selection during ensemble construction allows a more “ ensemble-aware ” choice of architectures and is based on data rather than manual biases . As ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . discussed in Section 2 , while ensembles with varying architectures has already been explored in the literature , variation in architectures is typically limited to just varying depth and/or width , in contrast to more complex variations , such as changes in the topology of the connections and operations used , as considered in our work . More generally , automatic ensemble construction is well-explored in AutoML [ 17 , 36 , 45 , 43 ] . Our work builds on this by demonstrating that , in the context of uncertainty estimation , automatically constructed ensembles with varying architectures outperform deep ensembles that use state-of-the-art , or even optimal , architectures ( Figure 1 ) . Studied under controlled settings , we assess the ensembles by various measures , including predictive performance , uncertainty estimation and calibration , base learner performance and two ensemble diversity metrics , showing that architectural variation is beneficial in ensembles . Note that , a priori , it is not obvious how to find a set of diverse architectures that work well as an ensemble . On the one hand , optimizing the base learners ’ architectures in isolation may yield multiple base learners with similar architectures ( like a deep ensemble ) . On the other hand , selecting the architectures randomly may yield numerous base learners with poor architectures harming the ensemble . Moreover , as in neural architecture search ( NAS ) , we face the challenge of needing to traverse vast architectural search spaces . We address these challenges in the problem of Neural Ensemble Search ( NES ) , an extension of NAS that aims to find a set of complementary architectures that together form a strong ensemble . In summary , our contributions are as follows : 1 . We present two NES algorithms for automatically constructing ensembles with varying base learner architectures . As a first step , we present NES with random search ( NES-RS ) , which is simple and easily parallelizable . We further propose NES-RE inspired by regularized evolution [ 48 ] , which evolves a population of architectures yielding performant and robust ensembles . 2 . This work is the first to apply automatic ensemble construction over architectures to complex , state-of-the-art neural architecture search spaces . 3 . In the context of uncertainty estimation and ro- bustness to dataset shift , we demonstrate that ensembles constructed by NES improve upon state-of-the-art deep ensembles . We validate our findings over five datasets and two architecture search spaces . 2 Related Work . Ensembles and uncertainty estimation . Ensembles of neural networks [ 23 , 32 , 11 ] are commonly used to boost performance . In practice , strategies for building ensembles include independently training multiple initializations of the same network , i.e . deep ensembles [ 33 ] , training base learners on different bootstrap samples of the data [ 62 ] , training with diversity-encouraging losses [ 40 , 35 , 60 , 51 , 29 , 47 ] and using checkpoints during the training trajectory of a network [ 27 , 41 ] . Despite a variety of approaches , Ashukha et al . [ 1 ] found many sophisticated ensembling techniques to be equivalent to a small-sized deep ensemble by test performance . Much recent interest in ensembles has been due to their state-of-the-art predictive uncertainty estimates , with extensive empirical studies [ 46 , 22 ] observing that deep ensembles outperform other approaches for uncertainty estimation , notably including Bayesian neural networks [ 4 , 19 , 52 ] and post-hoc calibration [ 21 ] . Although deep ensembles are not , technically speaking , equivalent to Bayesian neural networks and the relationship between the two is not well understood , diversity among base learners in a deep ensemble yields a model which is arguably closer to exact Bayesian model averaging than other approximate Bayesian methods that only capture a single posterior mode in a multi-modal landscape [ 54 , 18 ] . Also , He et al . [ 24 ] draw a rigorous link between Bayesian methods and deep ensembles for wide networks , and Pearce et al . [ 47 ] propose a technique for approximately Bayesian ensembling . Our primary baseline is deep ensembles as they provide state-of-the-art results in uncertainty estimation . AutoML and ensembles of varying architectures . Automatic ensemble construction is commonly used in AutoML [ 17 , 28 ] . Prior work includes use of Bayesian optimization to tune non-architectural hyperparameters of an ensemble ’ s base learners [ 36 ] , posthoc ensembling of fully-connected networks evaluated by Bayesian optimization [ 43 ] and building ensembles by iteratively adding ( sub- ) networks to improve ensemble performance [ 10 , 42 ] . Various approaches , including ours , rely on ensemble selection [ 7 ] . We also note that Simonyan & Zisserman [ 49 ] , He et al . [ 25 ] employ ensembles with varying architectures but without automatic construction . Importantly , in contrast to our work , all aforementioned works limit architectural variation to only changing width/depth or fully-connected networks . Moreover , such ensembles have not been considered before in terms of uncertainty estimation . Another important part of AutoML is neural architecture search ( NAS ) , the process of automatically designing single model architectures [ 14 ] , using strategies such as reinforcement learning [ 63 ] , evolutionary algorithms [ 48 ] and gradient-based methods [ 39 ] . We use the search spaces defined by Liu et al . [ 39 ] and Dong & Yang [ 13 ] , two of the most commonly used ones in recent literature . Concurrent to our work , Wenzel et al . [ 53 ] consider ensembles with base learners having varying hyperparameters using an approach similar to NES-RS . However , they focus on non-architectural hyperparameters such as L2 regularization strength and dropout rates , keeping the architecture fixed . As in our work , they also consider predictive uncertainty calibration and robustness to shift , finding similar improvements over deep ensembles . 3 Visualizing Ensembles of Varying Architectures . In this section , we discuss diversity in ensembles with varying architectures and visualize base learner predictions to add empirical evidence to the intuition that architectural variation results in more diversity . We also define two metrics for measuring diversity used later in Section 5 . 3.1 Definitions and Set-up . Let Dtrain = { ( xi , yi ) : i = 1 , . . . , N } be the training dataset , where the input xi ∈ RD and , assuming a classification task , the output yi ∈ { 1 , . . . , C } . We use Dval and Dtest for the validation and test datasets , respectively . Denote by fθ a neural network with weights θ , so fθ ( x ) ∈ RC is the predicted probability vector over the classes for input x . Let ` ( fθ ( x ) , y ) be the neural network ’ s loss for data point ( x , y ) . Given M networks fθ1 , . . . , fθM , we construct the ensemble F of these networks by averaging the outputs , yielding F ( x ) = 1M ∑M i=1 fθi ( x ) . In addition to the ensemble ’ s loss ` ( F ( x ) , y ) , we will also consider the average base learner loss and the oracle ensemble ’ s loss . The average base learner loss is simply defined as 1M ∑M i=1 ` ( fθi ( x ) , y ) ; we use this to measure the average base learner strength later . Similar to prior work [ 35 , 60 ] , the oracle ensemble FOE composed of base learners fθ1 , . . . , fθM is defined to be the function which , given an input x , returns the prediction of the base learner with the smallest loss for ( x , y ) , that is , FOE ( x ) = fθk ( x ) , where k ∈ argmin i ` ( fθi ( x ) , y ) . The oracle ensemble can only be constructed if the true class y is known . We use the oracle ensemble loss as one of the measures of diversity in base learner predictions . Intuitively , if base learners make diverse predictions for x , the oracle ensemble is more likely to find some base learner with a small loss , whereas if all base learners make identical predictions , the oracle ensemble yields the same output as any ( and all ) base learners . Therefore , as a rule of thumb , all else being equal , smaller oracle ensemble loss indicates more diverse base learner predictions . Proposition 3.1 . Suppose ` is negative log-likelihood ( NLL ) . Then , the oracle ensemble loss , ensemble loss , and average base learner loss satisfy the following inequality : ` ( FOE ( x ) , y ) ≤ ` ( F ( x ) , y ) ≤ 1 M M∑ i=1 ` ( fθi ( x ) , y ) . We refer to Appendix A for a proof . Proposition 3.1 suggests that it can be beneficial for ensembles to not only have strong average base learners ( smaller upper bound ) , but also more diversity in their predictions ( smaller lower bound ) . There is extensive theoretical work relating strong base learner performance and diversity with the generalization properties of ensembles [ 23 , 61 , 6 , 30 , 3 , 20 ] . In Section 5 , the two metrics we use for measuring diversity are oracle ensemble loss and ( normalized ) predictive disagreement , defined as the average pairwise predictive disagreement amongst the base learners , normalized by their average error [ 18 ] . 3.2 Visualizing Similarity in Base Learner Predictions . The fixed architecture used to build deep ensembles is typically chosen to be a strong stand-alone architecture , either hand-crafted or found by NAS . However , optimizing the base learner ’ s architecture and then constructing a deep ensemble can neglect diversity in favor of strong base learner performance . Having base learner architectures vary allows more diversity in their predictions . We provide empirical evidence for this intuition by visualizing the base learners ’ predictions . Fort et al . [ 18 ] found that base learners in a deep ensemble explore different parts of the function space by means of applying dimensionality reduction to their predictions . Building on this , we uniformly sample five architectures from the DARTS search space [ 39 ] , train 20 initializations of each architecture on CIFAR-10 and visualize the similarity among the networks ’ predictions on the test dataset using t-SNE [ 50 ] . Experiment details are available in Section 5 and Appendix B . As shown in Figure 2a , we observe clustering of predictions made by different initializations of a fixed architecture , suggesting that base learners with varying architectures explore different parts of the function space . Moreover , we also visualize the predictions of base learners of two ensembles , each of size M = 30 , where one is a deep ensemble and the other has varying architectures ( found by NES-RS as presented in Section 4 ) . Figure 2b shows more diversity in the ensemble with varying architectures than in the deep ensemble . These qualitative findings can be quantified by measuring diversity : for the two ensembles shown in Figure 2b , we find the predictive disagreement to be 94.6 % for the ensemble constructed by NES and 76.7 % for the deep ensemble ( this is consistent across independent runs ) . This indicates higher predictive diversity in the ensemble with varying architectures , in line with the t-SNE results .

Summary: The paper explores whether one can use Architecture Search to enhance ensemble diversity. They start with the observation that embeddings generated by different architectures (for multiple different initialization per architecture) are well separated from each other. They then try out a couple of architecture search methods to find ensembles with diverse architectures that minimize the loss.