Bayesian Quadrature for Neural Ensemble Search
Authors: Saad Hamid
Xingchen Wan
Martin Jørgensen
Binxin Ru
Michael A Osborne
Venue: Accepted by TMLR ()

Abstract: Ensembling can improve the performance of Neural Networks, but existing approaches struggle when the architecture likelihood surface has dispersed, narrow peaks. Furthermore, existing methods construct equally weighted ensembles, and this is likely to be vulnerable to the failure modes of the weaker architectures. By viewing ensembling as approximately marginalising over architectures we construct ensembles using the tools of Bayesian Quadrature -- tools which are well suited to the exploration of likelihood surfaces with dispersed, narrow peaks. Additionally, the resulting ensembles consist of architectures weighted commensurate with their performance. We show empirically -- in terms of test likelihood, accuracy, and expected calibration error -- that our method outperforms state-of-the-art baselines, and verify via ablation studies that its components do so independently.

Published in Transactions on Machine Learning Research (07/2023)
Bayesian Quadrature for Neural Ensemble Search
Saad Hamid University of Oxford
saad@robols.oz.co
Xingchen Wan University of Oxford
awan@robols.oor.co
Martin Jørgensen University of Oxford
martinj@robols. or.as.....c..........
Binxin Ru University of Ouford
robin@robots.oz.c or.potsssssx......
Michael Osborne University of Oxford
mosb@rooolssor..cccoo
Reviewed on OpenReview: https: // / openreview. net/ forum? 75sXdM03EQ
Abstract
Ensembling can improve the performance of Neural Networks, but existing approaches strug- gle when the architecture likelihood surface has dispersed, narrow peaks. Furthermore, cxisting methods construct cqually weighted ensembles, and this likely to be vulnora- ble to the failure modes of the weaker architectures. By viewing ensembling as approxi- mately marginalising over architectures we construct ensembles using the tools of Bayesian Quadrature tools which are well suited the exploration of likelihood surfaces with dis- persed, narrow peaks. Additionally, the resulting ensembles consist of architectures weighted commensurate with their performance. We show empirically in terms of test likelihood, accuracy, and expected calibration error - that our method outperforms state-of-the-art baselines, and verify via ablation studies that its components do so independently.
1 Introduction
Neural Networks (NNs) are extremely effective function approximators. Their archilect are, however, typically designed by hand, a painstaking process. Therefore, there has been significant interest in the automatic selection of NN architectures. In addition to search strategy, this involves defining scarch space from which to select the architecture non-trivial which is also active area of research. Recent work shows ensembles of networks of different architectures from given search space can outperform the single best architecture or ensembles of networks of the same architecture (Zaidi et al., 2022 Shu et al. 2022). Finding the single best architecture is typically referred to as Neural Architecture Search (NAS) (Zoph Le 2016: Elsken et al., 2019 He et al., (022). Such ensembles improve performance on a range of metrics, including the test set's predictive accuracy, likelihood, and expected calibration error. The latter two metrics measure the quality of the model's uncertainty estimates, which can be poor for single archilectures in some cases (Guo et al., 2017). Performant models in these metrics are crucial for systems which make critical decisions, such as self-driving vehicles. Ensemble selection is an even more difficult problem to tackle manually than selecting a single architecture, as it requires a combinatorial search over the same space. Hence, interest in methods for automatic ensemble construction is growing. This paper targets exactly this problem.
Conceptually, Neural Ensemble Search (NES) algorithms can be split into two stages. The first is the candidate selection stage, which seeks to characterise the posterior distribution, p(a D), given the training

---

Published in Transactions on Machine Learning Research (07/2023)
data D, over architectures from given search space E A. Multiple approaches have been proposed. One such is an evolutionary strategy which seeks the modes of this distribution (aidi et al. 2022 An alternative is training "supernet" and using it to learn the parameters of a variational approximation to this distribution (Shu et al. 2022). This involves evaluating the likelihood of a set of architectures from the search space; an evaluation which requires first training the architect weights. The second stage is ensemble selection, where the ensemble members are selected from the candidate set and each member's weight is chosen. Several approaches have also been suggested for ensemble selection, such as beam search and sampling from the (approximate) posterior over architectures.
In this work, we investigate novel approaches to both stages of NES algorithm. We view ensembling, the averaging over architectures, as marginalisation with respect to a particular distribution over architectures. When this distribution is the posterior over architectures, we are taking the hierarchical Bayesian approach. The key advantage of this approach is the principled accounting of uncertainty, which also improves accuracy by preventing overconfidence in a single architecture. Additionally, this paradigm allows us bring the tools of Bayesian Quadrature to bear upon the problem of Neural Ensemble Search. Specifically, the contributions of this work are follows
We propose using an acquisition function for adaplive Bayesian Quadrature to select the candidate set of architectures to train. It is from this candidate set that the ensemble members are later selected.
We show how recombination of the approximate posterior over architectures can be used to construct a weighted ensemble from the candidate set.
We undertake an empirical comparison of our proposals against stale-of-the-art baselines. Addition- ally, we conduct ablation studies understand the effect of our proposals for each stage of the NES pipeline.
2 Background
2.1 Neural Architecture Search (NAS)
NAS aims to aulomatically discover high-performing NN archilectures and has shown promising performance in various tasks (Real et al. 2017 Zoph et al. 2018 Liu et al., 2019a). It is typically formulated as an optimisation problem, i.c. maximising some measure of performance of over a space of NN architectures A,
(1)
Elsken et al. 2019: identify three conceptual elements of NAS pipeline: search space, search strategy, and a performance estimation strategy.
The first part of a NAS pipeline the search space is the way in which the possible space of NN architectures is defined. In this paper, we require the search space to be such that a Gaussian Process (GP) can be defined upon it. In particular, we focus on the two types of search space most common in the literature. The first is a cell-based search space, which consists of architectures made by swapping out "cells" in a fixed macro-skeleton (Pham al., 2018: Dong et. al.; 2021 Liu al.) 2019b). These cells are represented directed acyclic graphs where each edge corresponds to operation from pre-defined set of operations. Typically, the macro-skeleton will be structured so that repeating copics of the same cell are stacked within the macro-skeleton. This structure allows for the representation of the architecture by the corresponding cell. The second is a macro search space defined by varying structural parameters such as kernel size, number of layers, and layer widths. An example of such a search space is the Slimmable network: (Yu et al. 2019 Yu & Huang 2019) on the MobileNet search space (Sandler et al.) 2018): the largest possible network is trained and all other networks in the search space are given sub-networks or "slices".
A NAS pipeline's second phase is the search strategy. This is procedure for selecting which architectures to query the performance of. All strategies will exhibit an exploratioo-exxploiaatie trade-off, where exploration
1 An implementation of our proposals can be found hutne///sthaalcccmm//aaaaaaaaiia
2

---

Published in Transactions on Machine Learning Research (07/2023)
is covering the search space well, and exploitation is selecting architectures that are similar to the well- performing architectures in the already queried history.
The final element of a NAS pipeline is the performance estimation strategy, which is the method for query- ing the performance of a given architecture. Typically, this is done by training the NN weights, given the architecture, on a training dataset, and evaluating its performance on a validation set. This demands signif- icant computation and practically limits the total number of architecture evaluations available the search strategy. However, for some search spaces where network weights are shared performance estimation is considerably cheaper.
There exists large body of literature devoted to NAS, pursuing a range of strategies such as one-shot NAS
[Liu et al. 2019b Xu el al. 2020 Chen et al. 2019: Yu et al. 2020 Bender el al. 2018). evolutionary
strategies Real et al. (2017): Liang et al. (2018): ; Liu et al. [0021) and Bayesian Optimisation.
2.2
Bayesian Optimisation for Neural Architecture Search
An effective approach to NAS is Bayesian Optimisation (BO) (Kandasamy et al. 2019 White et al. 2020:
Ru et al. 2021 Wan et al. 2022 Shi et al. 2020 Zhou et al. 2023). On a high level, BO models the objective function and sequentially y selects where to query next. based on an acquisítion function, with the goal of finding the optimal value of the objective function in a samplo-effcient manner. Typically, the objective function is modelled using a Gaussian Process (GP) a stochastic process for which all finite subsets of random variables are joint normally distributed (Rasmussen & Williams, 2006).
A GP is defined using mean function m(a) that specifies the prior mean at a, and kernel function k(ax, a') that specifies the prior covariance between f(a) and f'aa). The posterior, conditioned set of observations A = {(aa)) )} and y (((o))....f((x)]]] is also a GP with moments
ma(-) m()) K. A (u m(a))
and
(2)
ka(:-)= K., KAA
where Kxy indicates a matrix generated by evaluating the kernel function between all pairs of points in the sets and Y. The prior mean function m is typically set to zero.
Ru al. (2021) showed that the Weisfeiler-Lehman graph kernel (WL kernel) (Shervashidze 2011] is an appropriate choice for modelling NN performance metrics on cell-based NAS search space with a GP. To apply the WL kernel, a cell first needs to be represented
labelled DAG. Next, feature vector is built up by aggregating labels for progressively wider neighbourhoods of cach node, and building histogram of the resulting aggregated labels. The kernel is then computed as the dot product of the feature vectors for a pair of graphs.
A common acquisition function for BO is Expected Improvement (Garnetl 2021 )
a (a) = Ep((i)) max (f(x) ((a))) o)]
(4)
where â is the best architecture found so far. Using this acquisition function in conjunction with a GP using the WL kernel was shown by Ru et al. (2021) to be effective for NAS.
2.3 Neural Ensemble Search
Neural Ensemble Search ŞZidii aL. 2022 is method for automatically constructing ensembles of a given size, M, from a NAS search space A. First, candidate set of architectures, A C A, is selected using a regularised evolutionary strategy (NES-RE), or random sampling from the search space. The authors propose several ensemble selection methods to subsequently select a subset of M architectures AM C A. Of particular interest in this work are Beam Search (BS) and Weighted Stacking (WS).
BS initially adds the best-performing architecture to the ensemble and greedily adds the architecture from the candidate set (without replacement) that most improves the validation loss of the ensemble. WS optimises
3

---

Published in Transactions on Machine Learning Research (07/2023)
the ensemble weights over the whole candidate set the validation loss (subject to the weights being non- negative and summing to one). The members with the highest M weights are included in the ensemble, and their corresponding weights renormalised. The authors compare BS to WS on the CIFAR-10 dataset, and find performance in terms of the log-likclihood of the test set to be better for BS for small ensembles, but similar for larger ensembles.
Neural Ensemble Search via Bayesian Sampling (Shu al. 2022) approximates the posterior distribution over architectures p(a D) with variational distribution of the form ((a) II q: 0z), where iterates over the connections within cell, is the operation for connection i, and 0, are the variational parameters for q. The form of q: is chosen to be softmax over 0, The ensemble is then selected by using Stein Variational Gradient Descent with Regularised Diversity to select a diverse set of M samples from (a continuous relaxation of) the variational distribution.
DeepEnsembles [Lakshminarayanan et al.] 2017) soek to approximately marginalise over the parameters of given NN archilecture. The archilecture is trained from several random initialisations, and the ensemble makes prediction as equally weighted sum of these. Hyper-deep ensembles (Wenzel et al. 2020 build this idea by training architectures from several randomly selected hyperparameter settings. They then construct an ensemble by using Beam Search, with replacement.
Relatedly, there has been interest in ensembling for improving uncertainty calibration in the context of Rein- forcement Learning (Osband et al. 2018 Dwaracherla et al. 2022). Additionally, methods for marginalising over the parameters of fixed architecture such as (Shui et al. (2018): He et al. (2020); D'Angelo & Fortuin (202)) frequently require constructing ensembles. Such methods orthogonal to our work, which seeks to construct ensembles of different architectures, rather than ensembles of different parameter scttings of the same architecture.
2.4 Bayesian Quadrature
Bayesian Quadrature (BQ) OO'Haaan 1991 Minka 2000 is a probabilistic numerical (Hennig & Osborne, 2022 integration technique that largets the computation of 2 = s()dd(()) based on evaluations of the integrand f (assuming a given prior z). Similar to BO, it maintains a surrogate model over the integrand f, which induces a posterior over the integral value Z. BQ also makes use an acquisition function to iteratively select where next to query the integrand.
The surrogate model for BQ is usually chosen to be GP, and this induces Gaussian posterior over Z N(uz, The moments of this posterior are given by
Pz / ((,x))d(()Kxx) and
(5)
o2 o2= /
X (XX))))))))))
(6)
where X is the set of query points, and f are the corresponding integrand observations. Note that the postcrior mean Pzz takes the form of quadrature rule weighted sum of function evaluations E. w; (xx) where wa are the elements of the vector (()
Frequently, the integrand of interest is non-negative. Important examples of such integrands are likelihood functions (which are integrated with respect to a prior to compute model evidence) and predictive densi- ties (which are integrated with respect to a posterior to compute a posterior predictive density). Warped Bayesian Quadrature (Osborne et al. 2012 Gunter et al. 2014 Chai & Garnett 2019) allows practitioners to incorporate the prior knowledge that the integrand is non- non-negative into the surrogate model. Of particular interest in this work will be the WSABI-L model (Gunter et al.) 2014), which models the square root of the integrand with a GP, /2([(x)- - 3) ~ (P(nn(x), En(a, x')). This induces a (non-central) chi-squared distribution over f, which can be approximated with a GP, with moments
m(x) 3 + 2 _u(())?
(7)
(8)
4

---

Published in Transactions on Machine Learning Research (07/2023)
Gunter et al. (2014) established, empirically, that the uncertainty sampling acquisition function works well for Bayesian Quadrature. This acquisition function targets the variance of the integrand
( = Ep(x, (x((),))
(9)
This naturally trades off between exploration (regions where Ep(x, is high), and exploitation (regions where AAD (2) is high most of the volume under the integrand is concentrated here).
Just as BO natural choice for NAS an expensive black- box optimisation problem so BQ is a natural choice for NES an expensive black-box marginalisation problem. this realisation that inspires our proposals in Section 3
2.5 Recombination
Typically, Neural Ensemble Search requires selecting a subset of the trained architectures to build the ensemble, making predictions with the whole candidate set is too computational ionally burdensome. To achieve this within the Bayesian Quadrature framework will require reducing the support of the quadrature rule. This problem is referred to as recombination (Litterer & Lyons 2012).
Given non-negative measure supported N points {(w,, Fn.) ) where un and Σ Wn and M 1 "test" functions (r it is possible to find subset of M < points fan [2n)m=1 {a J. for which
N
W' oi (Im)
Σ
Wn Wn (x
(10)
for all Φi, with Wm and Σm Wmn
Tchernychova 2015).
For Kernel Quadrature, one use the Nyström approximation of the kernel matrix to obtain set of test
functions (Hayakawa et al. 2022 Adachi et al. 2022). Using subset, S, of M - data points, the kernel can
be approximated x(xxx') = k(x, S),((s)ssss S) "S,, By taking an eigendecomposition, k(S, UAUT,
the approximate kernel can be expressod as
M-1 b(I,x') = (4
(11)
where are the columns of U, and A the diagonal elements of A. We can then use dr(-) u/ k(S, as test functions.
3 Bayesian Quadrature for Neural Ensemble Search
We decompose NES into two sub-problems:
1. The selection of a candidate set of architectures {ai),1 = A C. A for which to train the architecture parameters.
The selection of a set of M members from the candidate set to include in the ensemble, and their weights, w E RM
We take novel approaches to each of these sub-problems, described respectively in the following two subsec- tions. Algorithms and summarise our propositions.
3.1 Building the Candidate Set
An ensemble's prediction is a weighted sum of the predictions of its constituent members, AM. and this can always be viewed as approximating an expectation with respect to distribution, म, over architectures,
En(a) pp(c x, a.) Σ p(c (()(z, acA
Σ p x, ())))a(()).
(12)

---

Published in Transactions on Machine Learning Research (07 (07/202) 2023)
Ensemble predictions
X
likelihood p aa)
Final prediction
W2 x
A
Figure 1: A schematic representation of our proposal. The plot on the left shows a Gaussian Process modelling the likelihood over the space of architectures. The architectures to train and evaluate the likelihood for are selected by maximising a Bayesian Quadrature acquisition function, as described Section [3.1 One of the algorithms described Scction 3.2 then usod seleet the subset of architectures include in the ensemble, along with their weights. The final prediction is then linear combination of the predictions of each ensemble member, as shown by the bar plots the right (where each bar indicates the probability assigned to particular class).
p(c o, ar D) is the predictive probability assigned to class E {I....,C} by the architecture a (condi- tioned on the training data D) given the input z E X. The expectation Equation (12 corresponds to a marginalisation over architectures. The set of architectures As and their weights иM can be seen as a quadrature rule approximate this marginalisation. When x is the posterior over architectures p(a D), are performing hierarchical Bayesian inference, and the result is the posterior predictive distribution,
plc x, D)
Σ p(c x a, D)pla D) aεA
ElEAP(c x, a, p(D p) EoEAP(D p(a)
(13)
By taking the view of ensembling marginalisation, the practitioner has the ability to make their belief over A explicit. As the training data is finite, it is rarely appropriate to concentrate all of the probability mass of н on a single architecture. Arguably, x(a)) = p(a D) is the most appropriate choice distribution implied by the prior and the architectures' ability to explain the observed data. The hierarchical Bayesian framework should offer the most principled accounting of uncertainty in the choice of architecture (a more concentrated π should over-fit, less concentrated should under-fit).
as it the
From Equation 13 we see that, to compute the posterior predictive, need to compute C' sums of products of functions of the architecture and the architecture likelihoods. Intuitively, we expect quadrature scheme that approximates well the sum in the denominator of (13 will also approximate the sum in the numeralor well. Therefore, we propose using a Bayesian Quadrature acquisition function to build up the candidate set, as these architoctures will form the nodes of a query-efficient quadrature scheme for and so good basis for an ensemble.
The likelihood of an architecture p(D a) is not typically available, this would require marginalisation over the NN weights, w, of the architecture. We instead approximate using the MLE, which is equivalent to assuming the prior of the architecture weights is a Dirac delta distribution at the maximiser of the (architecture weights") likelihood function.
p(D aa) / p(D w, a) p(w a) p(D ú a,
(14)
à P(D w. a)p(w) a).
(15)

---

Published in Transactions on Machine Learning Research (07/2023)
Computing p(c | a, requires analogous intractable marginalisation. We approximate it similarly, noting that it depends only indirectly the training data, through the optimisation procedure, i.e.
/ p(c I, a, p) D)dw
plc a)
se p(c x, w.
(16)
Concretely, we place functional prior on the architecture likelihood surface, warped using the square-rool transform, (2(p(D) a) 3) GP, and use uncertainty sampling to make observations of the likelihood al set of architectures {as) ACA.
This provides with an estimate of the model evidence Z = =oE w a) p(a), which we denote 2.
The computation of this estimate requires Monte Carlo sampling to approximate sums of (products of) the WL-kernel over A. Note this is far more feasible than approximating the original sums in Equation (13) with Monte Carlo sampling as Kaa,, A) is far cheaper to evaluate than p(D (Ū, aÿ) or p(c x, w, az) either would require training architecture Ozj-
Algorithm 1
Candidate set selection algorithm using a BQ acquisition function. Returns architectures
A {a and their corresponding likelihoods L {p(D) w o;)) ))
A, L + sample(n, A)
> Initial samples.
0 + argmax.p(L A,0)
Optimise WL kernel.
while do
a argmax
function(a, A, L,
A + a
L + (L, PDD úû a)} 0 + argmax.p(4 [ A,
end while
return A, L
3.2 Selecting the Ensemble
In principle, the ensemble can be constructed using the weights provided by the quadrature scheme, as these weights naturally tradc-off between member diversity and member performance. However, we wish to select subset of the candidate set for the ensemble (as it is assumed that ensemble of the whole candidate set is too costly to be practical for deployment). Concretely, we seek subset Ass C A, along with weights w M such that
N
plc x. D) P(D an)) p(an) x, Ony D) +
(17)
3
UCp(c I. Omm D)
(18)
We expect be small if regions of high likelihood have been well-explored by the acquisition function
the building of the candidate set. To select the weights w and the set Aa we can use any rocombination generate the test. functions, as described in Section [2.
algorithm, using the Nyström approximation and the estimated posterior over architectures as the measure to recombine. We refer to this algorithm Posterior Recombination (PR).
A second approach, which we refer lo as Re-weighted Stacking (RS), is a modification of Weighted Slacking, Similar to WS, we optimise the weights of an ensemble the whole candidate set to minimize the validation loss. The ensemble members are then chosen by selecting the members with the M highest weights. However, rather than renormalising the corresponding weights, as suggested in Zaidi al. (2022), we reallocate the weight assigned to excluded architectures proportionally to the relative covariance between them and the
7

---

Published in Transactions on Machine Learning Research (07/2023)
ensemble members. Concretely, let {(am (m)]m= be the ensemble members and their optimised weights, and (tor,4))i=1 be the excluded architectures and their optimised weights. The weights of the ensemble w E IR are given by
N
k(am, ai)
+
Wt.
(19)
m=l k(am: 04)
Algorithm 2 Posterior recombination.
T
nystrom test functions( K AA; A)
From Eq
11
p(Dla.)pla...
N
2
w, Am < recombination
Algorithm Re-weighted stacking.
w f a wip(e x, an, D), vul))
I + select_top(( w)
Select top M.
for m in I do
Wm reweight L,, k(A,
Eq
19
end for
Our proposals can be combined to yield two possible algorithms. Both share the same candidate selection strategy that uses WSABI-L surrogate model with the uncertainty sampling acquisition function to select the set of architectures to train (Algorithm D. "BQ-R" then uses posterior recombination (Algorithm 2 to select a subset of architectures from the candidate set to include in the ensemble, and choose their corresponding weights. "BQ-S" instead uses re-weighted stacking (Algorithm 3 to select, and weight, the ensemble members from the candidate set. Note that "BQ-R" performs approximate hierarchical Bayesian inference using BQ, but "BQ-S" a hcuristic inspired by BQ. Figure a schematic representation of these algorithms.
4 Experiments
We begin by performing comparisons on the NATS-Bench benchmark (Dong et al. 2021). Specifically, we use the provided topology search space, which consists of cells with 4 nodes, connections, and 5 possible operations (including "zeroise" which is equivalent to removing a connection) in a fixed macro-skeleton. The (a smaller version of
architecture weights are trained for 200 epochs on the CIFAR-100 and ImageNet166 ImageNet with 16 16 pixel input images, and 120 classes) datasets. We will compare ensemble performance mcasured by test accuracy, test likelihood, and expected calibration error the test set for range of ensemble sizes. The logelikel of the test set measures model's 's performance both in terms of accuracy and uncertainty calibration, as placing a very low probability on the true class (i.e. being confidently wrong) is heavily penalised by this metric.
First, we verify that our chosen surrogate model (WSABI-L) performs well. Table 1 shows model perfor- mance, mcasured by root mcan square error (RMSE) and negative log prodictive density (NLPD) on a test set. The lest set is selected by ranking all the architectures in the search space by validation loss, and selecting every 25th architecture. This ensures that the test set contains architectures across the full range of performance. We build on the results of Ru et al. (2021), who showed that GP with WL kernel is able to model the architecture likelihood surface well. Our results show that WSABI-L (with a WL kernel) is a consistently better model than an ordinary GP (with a WL kernel).
Next, we examine the effect of the candidate selection algorithm, shown Table 2 In all cases, we use our variant of weighted stacking, described in Section 3.2 to select and weight the ensemble members. We compare Expected Improvement (EI) with GP surrogate with WT. kernel, Uncertainty Sampling with
8

---

Published in Transactions on Machine Learning Research (07/2023)
CIFAR-100
ImageNet16-122
Model
RMSE
NLPD
RMSE
NLPD
GP
6.165 + 0.116
0.124
0.013
9.610 + 0.626
0.121 + 0.012
WSABI-L
5.797 + 0.043
-2.741 + 0.095
4.078 + 0.058
-3.437 + 0.040
Table 1: The (normalised) RMSE and NLPD of WSABI-L surrogate and a GP surrogate on the test sets.
CTFAR-100
ImageNet16-121
A Algorithm
Accuracy
ECE
LL
Accuracy
RCE
L.
M= 3
RE EI
77.1 + 0.2
0.018 + 0.001
-4385 + 24.89
51.9 + 0.2
0.029 + 0.002
-5595 + 12.15
76.1
0.2
0.024 0.001
4472
29.74
51.4
0.034 1 0.002
-5632 + 11.91
US
76.6 + 0.2
0,021 + 0.001
4417 + 35.85
52.2 + 0.1
0.029 1 0.001
-5443 + 10.87
M =5
RE
78.5
0.2
0.033 + 0.001
-003 + 19.08
53.3 0.2
0.043 1 0.002
-5417 + 12.90
EI
77.4
0.2
0.039 + 0.001
-126
22.25
52.6 0.3
0.053 + 0.003
5479 + 15.55
US
77.8 + 0.2
0.040 + 0.002
4077 + 33.60
53.6 + 0.1
0.050 + 0.002
5380 + 12.31
M = 10
RE
79.4 + 0.1
0.053 1 0.002
3759 + 16.38
54.5 + 0.2
0.065 + 0.002
5280 + 16.85
EI
78.2
0.2
0.055 + 0.002
-3899
23.79
53.4 0.2
0.071 + 0.002
5368 + 19.47
US
78.6 + 0.2
0.059 + 0.001
-3843 + 22.71
54.7 + 0.1
0.072 + 0.001
5262 + 9.964
Table 2: Test accuracy, expected calibration error, and log-likelihood on CIFAR-100 and ImageNet16: 1--112 for our candidate set selection method (US) and baselines. The numbers shown are means and standard error of the mean over 10 repeats. Each candidate selection method is initialised with 10 random architectures, and used to build set of 150 architectures. The ensemble is chosen and weighted using weighted stacking. We see that the RE candidate set performs best for CIFAR-100 and in terms of ECE for ImageNet16-120, The US candidate set performs best in terms of accuracy and LL for TmageNet16-120.
variant of
WSABI-L surrogate using WL kernel (US), and Regularised Evolution (RE). We find that the US candidate set performs best for 1e6geNe in terms of accuracy and LL, but that the RE candidate set performs best for ECE on ImageNet16- 120, and across all metrics for CIFAR-100.
We then move on to comparing the effect of the ensemble selection algorithm, shown Table 3 In all cases, we use uncertainty sampling with a WSABI-L surrogate to build the candidate set. We initialise with 10 architectures randomly selected from a uniform prior over the search space, and use the acquisition function build a set of 150 architectures. We compare beam search (BS), weighted stacking (WS), recombination of the approximate posterior (PR), and re-weighted stacking (RS). We find that the stacking variants consistently perform best (with RS slightly improving upon WS) in terms of accuracy and LL, and PR in terms of ECE for larger datasets.
We then proceed to compare the two variants our algorithm BQQR and BQ-S - with several baselines.
Random The ensemble an evenly weighted combination of M architoctures randomly sampled from the prior p(a) over the search space.
Hyper-DE The candidate set selected by randomly sampling from the prior p(a). The ensemble then chosen using beam search, with replacement.
NES-RE The candidate set is selected using regularised evolution, and the ensemble members are chosen using beam search. The ensemble members are equally weighted.
NES-BS The posterior over architectures p(a| D) is approximated using a variational distribution. The ensemble is constructed by sampling M architectures from the variational distribution using Stein- Variational Gradient Descent. As no implementation is publicly available, provide

---

Published in Transactions on Machine Learning Research (07 (07/2023)
CIFAR-100
ImsgeNeet66-20
Algorithm
Accuracy
ECE
Accuracy
ECE
LL
M 33
BS
75.2 + 0.2
0.030 + 0.002
-4500 + 11.04
52.2 + 0.1
0.036 + 0.002
-5572 + 13.17
WS
76.4 + 0.2
0.021 + 0.001
-4426 + 35.87
52.1 - 0.1
0.029 + 0.001
-5545 + 10.57
PR
71.9 0.8
0.075 + 0.025
-5259
300.9
16.7 +
0.052 + 0.021
-6317 + 480.5
RS
76.6 + 0.2
0.021 + 0.001
-4417 + 35.85
52.2 + 0.1
0.029 + 0.001
-5543 + 10.87
M = 5
BS
76.4 + 0.2
0.048 + 0.002
4233 + 36.48
53.4 + 0.1
0.058 + 0.002
5410 + 10.70
WS
77.7
0.2
0.036 + 0.002
-088 + 34.13
53.6 + 0.1
0.049 + 0.001
5382 + 12.55
PR
73.3 + 0.9
0.040 + 0.004
-1768 + 174.3
50.7 + 0.3
0.028 + 0.004
-5647
+ 50.58
RS
77.8 0.2
0.040 + 0.002
-077
33.60
53.6 0.1
0.050 + 0.002
5380 + 12.31
M 10
BS
76.9 + 0.3
0.063 + 0.001
4079 + 50.29
54.1 + 0.1
0.076 + 0.001
-5307 + 9.795
WS
78.5 + 0.2
0.055 + 0.002
-3888 + 23.96
54.6 + 0.1
0.070 + 0.001
-5264 + 10.11
PR
75.5 0.9
0.037 + 0.002
-4309 + 172.6
52.3 0.3
0.018 + 0.001
-5412 + 22.96
RS
78.6 + 0.2
0.059 + 0.001
3843 + 22.71
54.7 + 0.1
0.072 + 0.001
-5262 + 9.964
Table 3: Test accuracy, expected calibration error, and log-likelihodd on CTFAR-100 and ImageNet for Beam Search (BS), Weighted Stacking (WS), Posterior Recombination (PR), and Re-weighted Stacking (RS). The numbers shown are means and standard error of the mean over 10 repeats. The candidate set selection method is our method Uncertainty Sampling with WSABI-I. surrogate initialised with 10 random architectures, and used to build set of 150 architectures. We see that the stacking variants consistently perform best for accuracy and LL, with RS slightly improving upon WS. For ECE, RS and WS perform well for small ensembles, but PR works best for larger ensembles.
However, our implementation learns the variational parameters by approximating the expected log- likelihood term of the ELBO using a subset of the search space, rather than by backpropagating through a "supernet" as described by Shu et al. (002)). The subset we use is the 150 architectures in the search space with the highest likelihoods on the validation set. (Of course, this only possible when working with a NAS benchmark.) We argue that our approximation suitable as most posterior mass will be concentrated on these architectures, so good variational distribution will concentrate mass them as well. Additionally, approximation is much faster it does not require training a supernet.
Table 4 presents the results on CIFAR-100 and ImageNet16- 16-112 for range of ensemble sizes. Whilst NES- RE matches or does slightly better than our proposals in terms of accuracy and LL on CIFAR-100, we find that both BQ-S and BQ-R often perform better in terms of expected calibration error. BQ-S achieves the best performance on TmageNNet66- 1-11t terms of L.L across all ensemble sizes, joint best with NES-RE terms of accuracy, and often outperforms NES-RE in terms of ECE.
Next, we perform study consisting of 614,625 architectures. Sub-networks or "slices" of this supernet constitute architectures within this search space. The architectures are structured as a chain of blocks, each of which can have up to layers. These sub-networks can be represented a 28-dimensional ordinal space (with 4 options along each dimension). We compare the best-performing variant of our method, BQ-S, and the best-performing bascline, NES-RE, from the smaller NATS-Bench scarch space. We use an RBF kernel with WSABI-L for Uncertainty Sampling with our method BQ-S, and compare it NES-RE. The results are shown in Table We see that BQ-S consistently outperforms NES-RE in terms of the log- likelihood of the test set and, for CIFAR-100, in terms of expected calibration error as well.
larger search space defined by "slimmable network" (Yu et al. 2019
Finally, we perform experiments to examine the robustness to datasel shift. Previous work has provided evidence that ensembling of Neural Networks provides robustness to shifts in the underlying data distribution (Zaidi et al., 2022; Shu et al.) (022). However, these investigations have assumed the availability of a validation set from the shifted distribution, which we argue is unrealistic in practice. Instead, we examine the setting where only the test set is shifted, and the validation set is representative of the training set. We
10

---

Published in Transactions on Machine Learning Research (07/2023)
C-F0R-100
ImageNet16-120
Algorithm
Accuracy
ВCE
LL
Accuracy
ECE
LL
Best Single
69.1
0.088
-5871
45.9
0.062
-6386
M =
Random
69.2 1.5
0.075 + 0.007
-5778 = 291.3
39.7 + 2.2
0.097 + 0.007
-7459 = 309.6
Hyper-DE
76.2 = 0.2
0.002
-4390
51.4 = 0.2
0.040 0.002
-5639 = 24.21
NES-RE
76.6 + 0.2
0.026 + 0.002
-4340 = 19.58
52.0 + 0.2
0.033 + 0.002
-5582 = 8.858
NES-BS
66.2 1.5
0.073 0.009
-6477 = 203.0
45.7 = 0.3
0.058 + 0.003
6403 = 28.04
BQ-R
71.9 = 0.8
0.075 0.025
-5259
300.9
16.7 = 2.3
0.052 0.021
6347 = 480.5
BQ-S
76.6 + 0.2
0.021 + 0.001
-4177 = 35.85
52.2 + 0.1
0.029 + 0.001
5543 = 10.87
=
Random
72.2 0.9
0.111 0.009
5304 180.9
42.7 + 1.5
0.129 + 0.008
-133 = 216.1
Hyper-DE
77.6 = 0.1
0.088 0.002
-099 12.67
52.4 = 0.2
0.061 0.001
-5535 = 22.38
NES-RE
78.2 + 0.1
0.042 + 0.002
4002 = 17.11
53.4 + 0.2
0.051 + 0.001
-5404 = 12.59
NES-BS
65.9 1.5
0.073 0.009
-6181 208.7
45.7 = 0.3
0.058 0.003
-6403 = 28.01
BQ-R
73.3 = 0.9
0.040 + 0.004
4768 = 174.3
50.7 = 0.3
0.028
0.004
-5647 = 50.58
BQ-S
77.8 0.2
0.040 + 0.002
1077 = 33.60
53.6 + 0.1
0.050 + 0.002
5380 = 12.31
M 10
Random
74.7 0.3
0.150 0.010
50188 82.21
45.1 + 0.4
0.159 + 0.008
6916 = 73.21
Hyper-DE
78.6 = 0.1
0.066 0.001
-3862 9.328
+ 0.2
0.075 0.002
-516 = 18.60
NES-RE
79.4 + 0.1
0.060 + 0.001
-3763 15.16
54.5 + 0.2
0.069 + 0.001
-5269 = 17.83
NES-BS
= 0.4
0.085 0.005
-6119 36.31
= 0.3
0.068 0.004
-6422 = 24.47
BQ-R
75.5 = 0.9
0.037 0.002
-4309 = 172.6
52.3 = 0.3
0.018 + 0.001
-5112 = 22.96
BQ-S
78.6 0.2
0.059 0.001
3843 = 22.71
54.7 + 0.1
0.072 0.001
-526 = 9.964
Table Test accuracy, expected calibration error (ECE), and log-likelihood (LL) on CIFAR-100 and ImageNet16- 120 for our proposals (BQ-R and BQ-S) and baselines. For reference, we also include the performance of the best architecture (measured by validation loss) on the test set (labelled Best Single). The numbers shown candidate set selection method is initialised with 10 random architectures and used to build set of 150 architectures. For ImageNet16-122 we see that BQ-S performs best across ensemble sizes in terms of LL, and joint best with NES-RE in terms of accuracy. For CIFAR-100 we find that NES-RE performs best in terms of accuracy and LL. Particularly for larger ensembles, BQ-R performs best in terms of ECE.
means and standard error of the mean 10 repeats. Where applicable, the
CIFAR-10
CIFAR-100
Algorithm
Accuracy
ECE
LL
Accuracy
ECE
LL
M 3 NES-RE BQ-S
93.8
0.029 0.001
1165
5.602
= 0.2
0.072
0.004
-133
61.49
93.7 + 0.1
0.030 + 0.000
1152 + 5.215
74.4 = 0.1
0.063 + 0.002
-5021 : 22.71
M W=5 NES-RE BQ-S
93.8 + 0.0
0.030 0.001
1165 + 5.503
74.3 - 0.2
0.071 + 0.001
-134 + 60.72
93.7 + 0.1
0.032 = 0.000
1113 + 4.123
74.5 = 0.1
0.055 + 0.002
4897 = 25.66
M 10 NES-RE
93.8 + 0.0
0.030 + 0.001
1159 + 5.959
74.3 + 0.2
0.069 + 0.001
5083 + 58.12
BQ-S
93.8
0.031 0.000
1098 3.752
74.7 + 0.1
0.045 0.001
4766 15.89
Table 5: Test accuracy, expected calibration error (ECE), and log-likelihood (LL) on CIFAR-10 and CIFAR- 100 for BQ-S (our proposal) and NES-RE (the strongest bascline) for the "Slimmable Network" search space. We see that BQ-S consistently outperforms NES-RE in terms of ECE and LL, whilst maintaining the same accuracy.
use the benchmark established by Hendrrycks & Dietterich (2019) generate shifted datasets by applying one of 30 corruption types each image for CIFAR-10 and CIFAR-100 Each corruption type has severity level 1 5 scale. Table shows comparison between NES-RE and BQ-S in this setting (on the
11

---

Published in Transactions on Machine Learning Research (07/2023)
Severity Level 1
CIFAR-10
CIFAR-100
Algorithm
Aceuracy
ECE
LL
Accuracy
ECE
LL
NES-RE
86.20
0.04
0.046
0.001
59259.6
596.907
62.36
0.08
169235 + 1632.43166
BQ-S
86.26
0.08
0.036
0.001
-54283.4
642.383
62.52 0.10
0.093 + 0.002
-149022
364.57480
5 NES RE
86.25
0.04
0.046
0.001
-59178.3
851.719
0.09 62.58
0.133
169999 + 1553.96435
BQ-S
86.16
0.06
-52178.6
202.350
0.103
0.004
152466
1249.96873
NES
86.26
0.04
0.043
0.001
-37010.4
722.311
62.46
0.07
164816
975.78975
BQ-S
86.22
0.05
-50504.6 443.984
0.08
0.093 0.002
149022
364.57480
Severity Level 3
CIFAR-10
CIFAR-100
Algorithm
Accuracy
ECE
LL
Accuracy
ECE
LL
M =3 NES-RE
73.16 + 0.08
0.147 + 0.002
-133205 + 1537.15
49.18 = 0.07
0.235 + 0.005
-270710 + 2628, 75-62
BQ.S
73.31 + 0.12
0.131 + 0.002
-123113 + 1498.55
49.45 + 0.12
0.193 + 0.007
-250337 = 3304.95
M NES-RE BQ S
73.18 + 0.09
0.148 + 0.002
113323 + 1904.50
49.20 = 0.09
0.239 + 0.004
-272004 + 2482.09961
73.23 + 0.07
0.125 + 0.001
-18756 + 614.899
49.57 + 0.09
0.175 + 0.005
-241407 = 2438.78
M 10 NES-RE BQ 8
73.23 + 0.08
0.143 + 0.002
-128663 + 1664.07
49.29 = 0.07
0.227 + 0.003
263339 + 1596.01214
73.39 + 0.11
0.120
0.002
-114613 + 1247.44
49.57 + 0.06
0.163 + 0.003
235152 = 881.120
Severity Level
CIFAR-10
CIFAR-100
Algorithu
Accuracy
ECE
LL
Accuracy
ECE
LL
=
NES-RE
55.49
0.08
0.285
0.002
-239927
2187.24
34.04 = 0.06
0.339
0.005
-115182
3710.25
BQ-S
55.67 = 0.14
0.265
0.003
-226433
2523.79
34.24 + 0.11
0.291 = 0.008
-385063 = 5355.88
NES-RE
55.53
0.08
0.286
0.003
-240154
2835.89
34.04 = 0.07
0.344 0.005
-117110
3476.95
BQ-S
55.51 = 0.05
0.260
0.002
-220196
1198.20
34.35 + 0.10
0.270 = 0.006
-371575 = 4083.67
NES-RE
55.54 0.08
0.279
0.002
2334441
2474.23
34.11 = 0.07
0.331 = 0.003
405068 = 2327.88
BQ-S
55.61 = 0.11
0.254 0.003
-214126 2030.20
34.35 + 0.07
0.257 = 0.003
-361876 = 1666.07
Table 6: Test accuracy, expected calibration error (ECE), and ligelikelihood (LL) on CIFAR-10 and CIFAR- 100 for NES-RE (the strongest baseline), and BQ-S (our strongest proposal) using the "Slimmable Network" search space for a range of corruption severities. We see that BQ-S is more robust than NES-RE datasct shift, especially in terms of LL and ECE.
slimmable network search space). We see that, whilst our proposal performs similarly in terms of accuracy, it produces ensembles that perform significantly better in terms of expected calibration error and test set log- likellioooood This trend holds across corruption severity levels.
5 Discussion and Future Work
We proposed method for building ensembles of Neural Networks using the tools provided by Bayesian Quadrature. Specifically, by viewing ensembling approximately performing marginalisation over architec- tures, we used the warped Bayesian Quadrature framework to select a candidate set of architectures to train. We then suggest two methods of constructing the ensemble based upon this candidate set: one based upon recombination of the approximate posterior over architectures (BQ-R), and one based upon optimisation of the ensemble weights (BQ-S) using a validation set. BQ-R approximately performs hierarchical Bayesian inference using BQ, whereas BQ-S is heuristic inspired by BQ. The discrepancy in performance is likely due to the fact that. BQ-R does not make of the validation set, it takes the Bayesian perspective
12

---

Published in Transactions on Machine Learning Research (07/2023)
and performs hicrarchical inference over both architecture weights and architectures using the training set. (In principle, BQ-R can use the union of the training and validation sets to perform hierarchical inference. However, we did not run experiments in this setting as it would obviously allow BQ-R significantly more compute than the alternative methods.) For the same reason, BQ-R is more sensitive to any errors intro- duced by approximaling the architect likelihood using MLE (or any other approximation). BQ-S (and all the baselines), however, make use of separate validation set to select the ensemble weights. We additionally show that BQ-S outperforms state-of-the-art baselines when the search space is large, and on the largest datasets for smaller search spaces. This is likely because it is more exploratory than alternative methods, and so less likely to become stuck near local minima of the architecture likelihood. Lastly, we demonstrated that BQ-S is more robust to dataset shift than state-of-the-art baselines.
A limitation of our proposals is that they do not outperform existing methods in some cases, notably on the CTFAR-100 dataset for the NATS-Bench search space. Additionally, will be challenging to scale our proposals to larger evaluation budgets (greater than 1000) as the computational burden inverting the GP's covariance matrix will become too large. An interesting direction for future work is examine the effect of marginalising over architecture weights well as over architectures.
We introduce a general-purpont method, so its societal impacts will depend on the specific tasks to which it is applied. We find it difficult to anticipate what those tasks will be, and even more difficult. to speculate meaningfully about any societal impacts will be.
Acknowledgments
The authors would like to thank anonymous reviewers for detailed and constructive feedback. S.H. and M.O grateful for funding from the EPSRC and AIMS at the University of Oxford. S.H. is additionally supported by scholarship from Kellogg College. X.W. and B.R. are supported by the Clarendon Scholarship at the University of Oxford. M.J. is supported by the Carlsberg Foundation.
References
Masaki Adachi, Satoshi Hayakawa, Martin Jørgensen, Harald Oberhauser, and Michael A. Osborne. Fast bayesian inference with batch bayesian quadrabure via kernel recombination. Advanced in Neural Infor- malion Processing Systems, 35, 2022.
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Understanding and simplifying onc-shot architecture search. In International conference on machine learning, pp. 550-559. PMLR, 2018.
Henry Chai and Roman Garnett. Improving quadrature for constrained integrands. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, 2019.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Xuanyi Dong, Ta Liu, Katarzyna Musial, and Bogdan Gabrys. NATS-bench: Benchmarking NAS algorithms for architecture topology and size. IEEE Transactions on Pottern Analysis and Machine Intelligence, pp. 1-1, 2021. ISSN 0162-8828, 2160-9292, 1939-3539. doi: 10.1109 10.1109/TPAMI.CC. 2P0....202.3054485
Vikranth Dwaracherla, Zheng Wen, Ian Osband, Xiuyuan Lu, Seyed Mohammad Asghari, and Benjamin Van Roy. Ensembles for uncertainty estimation: Benefits of prior functions and bootstrapping, 2022.
Francesco D' D'Angelo and Vincent Fortuin. Repulsive deep ensembles are bayesian. In Advances in Neural Information Processing Systems, volume 34, 2021.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21 2019.
13

---

Published in Transactions on Machine Learning Research (07/2023)
Roman Garnett. Bayesian Optimization. Cambridge University Press, 2021.
Tom Gunter, Michael A. Osborne, Roman Garnett, Philipp Hennig, and Stephen J. Roberts. Sampling
for inference in probabilistic models with fast bayesian quadrature. In Proceedings of the 28th Annual Conference Newral Information Processing Systems, pp. 2789 2797, 2014.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference of Machine Learning, 2017.
Satoshi Hayakawa, Harald Oberhauser, and Terry Lyons. Positively weighted kernel quadrature via subsam- pling. arviv: 2107.09597, 2022.
Bobby He, Balaji Lakshminarayanan, and Yee Whye Teh. Bayesian deep ensembles via the neural tangent kernel. In Advances in Newral Information Processing Systems, volume 33, 2020.
Xin He, Kaiyong Zhao, and Xiaowen Chu. Automl: A survey of the state-of-the-art. Bhowlldge-BBsele Systems, 212:106622, 2021.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In Proceedings of the International Conference on Learning Representations. arXiv, 2019.
P Hennig and M Osborne. Probabilistic Numerics. Cambridge University Press, 2022.
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric Xing. Neural ar- chitecture search with bayesian optimisation and optimal transport. Advances in Neural Information Processing Systems, 31, 2019.
lan Kivlichan, Zi Lin, Jeremiah Liu, and Lucy Vasserman. Measuring and improving model-moderator collaboration using uncertainty estimation. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), pp. 36-53. Association for Computational Linguistics, 2021.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncer- tainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 30, 2017.
Jason Liang, Elliot Meyerson, and Risto Miikkulainen. Evolutionary architocture search for deep multitask networks. In Proceedings of the genetic and evolutionary computation conference, pp. 466 476,473,440 2018.
C. Litterer and T. Lyons. High order recombination and an application to cubature on wiener space. 22(4). 2012.
Chenxi Liu, Hlang-Chiel Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei-Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 82-92, 2019a.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In Proceedings of the International Conference on Learning Representations, 2019b.
Yuqiao Liu, Yanan Sun, Bing Xue, Mengjie Zhang, Gary G Yen, and Kay Chen Tan. A survey on evolutionary neural architecture search. IEEE transactions neural networks and learning systems, 2021.
Wesley Maddox, Timur Garipov, Pavel Ixmailov, Dmitry Vetrov, and Andrew Gordon Wilson. simple baseline for bayesian uncertainty in deep learning, In Advanced in Neural Information Processing Systems, volume 32. arXiv, 2019.
Thomas Minka. Deriving quadrature rules from gaussian processes. pp. 1 1-21, 2000.
A. O O'Hagan. Bayes-hermite hermite quadrature. Journal of Stalistical Planning and Inference 29(3):245 260, 1991.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning.
Tn Advances in Neural Information Processing Systems, volume 31, 2018.
14

---

Published in Transactions on Machine Learning Research (07/2023)
Michael A Osborne, David Duvenaud, Roman Garnett, Carl E Rasmussen, Stephen J Roberts, and Zoubin Ghahramani. Active learning of model evidence using bayesian quadrature. In Advances in Newral Infor- mation Processing Systems 26, pp. 46 54, 2012.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advanced in Newral Information Processing Systems, volume 32, 2019.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International conference on machine learning, pp. 4095 4104. PMLR, 2018.
Carl Rasmussen and Christopher Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoe V Le, and Alexcy Kurakin. Large-scalc evolution of image classifiers. In International Conference on Machine Learning, pp. 2902 2911. PMLR, 2017.
Jože M. Rožance, Luka Bizjak, Elena Trajkova, Patrik Zajec, Jelle Keizer, Blaž Fortuna, and Dunja Mladenić. Active learning and novel model calibration measurements for aulomated visual inspection manulac- turing. 2023.
Binxin Ru, Xingchen Wan, Xiaowen Dong, and Michael Osborne. Interpretable neural architect search via bayesian optimisation with weisfeiler-lehmnn kernels. In Proceedings of the 9th International Conference on Learning Representations, 2021.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on compuler vision and pattern recognition, pp. 4510-4520, 2018.
Nino Shervashidze. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research, 12:2539 2561, 2011.
Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James Kwok, and Tong Zhang, Bridging the gap between sample- based and one-shot neural architecture search with bonas. Advances in Neural Information Processing Systems, 13:1808-1819, 2020.
Yao Shu, Yizhou Chen, Zhongxiang Dai, and Bryan Kian Hsiang Low. Neural ensemble search via bayesian sampling. In Proceedings of the Thirty-Eightt Conference on Uncertainty in Artificial Intelligence, 2022.
Changjian Shui, Azadeh Sadat Mozafari, Jonathan Marek, lhsen Hedhli, and Christian Gagné. Diversity regularization in deep ensembles. In International Conference on Learning Representations, 2018
Maria Tchernychova. Caratheodory cubature measures, 2015.
Xingchen Wan, Binxin Ru, Pedro M. Esparança, and Fabio Maria Carlucci. Approxinate neural architec- ture search via operation distribution learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2377-230 2386, January 2022.
Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robust- ness and uncertainty quantification. In Advances in Neural Information Processing Systems, volume 33, 2020.
Colin White, Willie Neiswanger, and Yash Savani. BANANAS: Bayesian optimization with neural architec- tures for neural architecture search. Advances in Neural Information Processing Systems, 36, 2020.
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. PC-DARTS: Partial channel connections for memory-efficient architecture search. IEEE Trunsactions on Pattern Anal- 2020.
ysis and Machine Intelligence,
15

---

Published in Transactions on Machine Learning Research (07/2023)
Jiahui Yu and Thomas S Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE/CV intervalional conference computer vision, pp. 1503-1811, 2019.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In
International Conference on Learning Representations. arXiv, 2019.
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural architecture search with big single-stage models. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII 16, pp. 702-717. Springer, 2020.
Sheheryar Zaidi, Arber Zela, Thomas Elsken, Chris Holmes, Frank Hutter, and Yee Whye Tch. Neural ensemble search for uncertainty estimation and dataset shift. In Advances in Neural Information Processing Systems, volume 36, 2022.
Han Zhou, Xingchen Wan, Ivan Vulié, and Anna Korhonen. Autopeft: Automatic configuration search for parameter-officient fino-tuning. arx preprint orXw:2301.12132 2023.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.0157% 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for seal- able image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
A Computational Complexity
The computational complexity of candidate set selection strategies that require GP surrogate (such as our proposals) is dominated by the cost of inverting the kernel matrix, which is, at each iteration, cubic in the size of the candidate set ([AAs)) The complexity of weighttteed stacking is (I log((A)) + |A [[AA] [AM))). The first term is due to the sorting of candidate set, and the second to the computation the relevant covariances. The complexity of posterior recombination is dominated by the eigendecomposition of the kernel matrix, ((44)))
Table compares the run-time of our proposals against the most performant baseline on the NATS-Bende topology search space. Note that all architecture-reltted evaluations are cached (i.e. the trained weights are loaded from the NATS- are loaded from disk). The key reason that NES-RE is slower than proposals is the iterative nature of Beam Search (required at each step of candidate set select and then for ensemble selection). Recall that it greedily builds up the set of parent candidates (resp. final ensemble) by iterating through the whole pool (resp. population). Each itcration requires loading a set of logits from disk which, whilst cheaper than loading and performing a forward pass through the architecture, still incurs significant computational cost. in aggregate.
API, and the logits for the forward pass through the train/validatinn validation set
Note that, for most search spaces, the computational cost of Neural Ensemble Search is dominated by the cost of evaluating the likelihoods of the architectures for the candidate set. For some spaces, such as those defined by a supernet, this cost is lower as the supernet is only trained once. However, this initial training is still computationally intensive. Therefore, in either case, the computation related to training and evaluating architectures is likely to be significantly larger than the computation required for the Neural Ensemble Search method.
B Further Analysis of Results
Our results over the NATS-Bench topology scarch space show that NES-RE performs best for the CIFAR- 100 dataset, and BQ-S performs best for ImageNet16 (see Table 4. The ablation study in Tables 2 and 3 suggest that this is due to the candidate selection strategy. Recall that NES-RE regularised
16

---

Published in Transactions on Machine Learning Research (07/2023)
Algorithm
CIFAR-100
ImageNet166 16-11
NES-RE
24236.3 + 109.551
12020.6 + 98.1485
BQ-R
308.872 + 7.05121
258.444 + 3.76138
BQ-S
348.650 + 11.7402
280.689 + 9.76650
Table 7: The runtime of our proposals, BQ-R and BQ-S, against the most competitive (in terms of perfor- mance) baseline on CIFAR-100 and ImageNet 6-120 over the NATS-Bench topology search space. The total evaluation budget is 150 architectures, and we select ensembles of size We report the means and standard error of the means over runs. All architecture-rrllted evaluations are cached (i.e. the trained weights are loaded from the NATS-Bench API, and the logits for the forward pass through the train/ calinvaaidation set are loaded from disk).
evolution for candidate set selection and beam search for ensemble selection, whereas BQ-S uses uncertainty sampling with a WSABI-L surrogale and re-weighted stacking. Table 2 suggests that (for a fixed ensemble selection strategy) regularised evolution is the best method for CIFAR-IA 100 but that uncertainty sampling with WSABI-L surrogate is the best method for ImageNet16-1200 Table 3 that re-weighted stacking consistently outperforms beam search given fixed candidate set. From these results, we can infer that regularised evolution is a better candidate selection strategy for CIFAR-100 and uncertainly sampling is better for ImageNet16-120, This must be due to the nature of the architecture likelihood surfaces for each datasot. As these surfaces are defined over an input space of architoctures, they are difficult to visualise. One possible method is shown in Figure 2 We consider the 500 architectures with the highest likelihood. We then visualise the covariance matrix between them using the WI. kernel of a trained GP. The architectures are sorted using the GP's estimate of the likelihood (to smooth out noise). We observe larger clusters of architectures along the diagonal of the covariance matrix for CIFAR-100. This means that they covary strongly and have similar likelihoods. Further, note that these clusters covary strongly with each other. This provides some evidence that the peaks of the CIFAR-100 likelihood surface are broader and closer together (based on the metric implied by the WL kernel) than those for the ImageNet 1engeNee6661110 16-120 likelihood surface. On this basis, we suggest that uncertainty sampling with a WSABI-L surrogate is better suited to architecture likelihood surfaces with dispersed, narrow peaks and regularised evolution is better suited architecture likelihood surfaces with wider peaks.
C Additional Uncertainty Metrics
To verify that the quality of the uncertainty is well measured by the expected calibration error we check that
it correlates well with the calibration AUC (Kivlichan et al. 2021 Rožanec et al. 2023). These results are shown in Table g (for the same experimental setup as in Table Note that well-calibrated uncertainty is indicated by high calibration AUC but low expected calibration crror. The correlation between the (means
of the) two metrics is -0.71 for CIFAR-100 and -0.20 for ImageNet16-122 They are, therefore, generally agreement about the quality of model's calibration estimates.
D Architecture Likelihood Approximation
We examine the impact of the approximations suggested in Equations 15 and 16 by comparing their per- formance to an alternative Stochastic Weight Averaging (Gaussian) Maddox et al. 2019). This instcad approximales the posterior over architecture weights using a diagonal Gaussian, whose moments are the empirical mean and variance of several SGD iterates, obtained by continuing training. The results are shown on CIFAR-100 for the NATS-Bench search space in Table D We see significant improvements across all metrics when using the SWAG approximation, suggesting that there much be gained by marginalising over both architecture weights and architectures. This would be promising direction for future work.
17

---

Published in Transactions on Machine Learning Research (07/202) 2022)
(a) CIFAR-100
(b) ImageNet1661200
Figure 2: Visualisation of the (WL) covariance matrix for the 500 architectures with the highest likelihoods in the search space for each dataset, sorted by (a smoothed estimate, using GP, of the) likelihood. The colour scale varies from 1 (yellow) to 0 (blue). We observe larger blocks of architectures within the top 500 that covary strongly for CIFAR-100 than for ImageNct16-1200, which implics that the modes of the architecture likelihood surface are wider for CIFAR-100. This suggests that a more exploratory strategy will do better on ImageNet16-120, and a more exploitative strategy for CIFAR-100.
E Experimental Setup
The codebase uses PyTorrc (Paszke et al. 2019 to handle deep learning and backpropagation. Except where otherwise stated, the experimental results report means and standard error of the mean over 10 repeats. Where applicable, each candidate selection method is initialised with 10 architectures randomly selected from the search space, and allowed to select an additional 140.
Our proposal over the cell-based search space uses the WL kernel, with its level hyperparameter chosen from [1,2] using the GP's marginal likelihood. For the macro-based search space, our proposal uses an ARD RBF kernel, whose hyperparameters the minimum and maximum distances between observations along the relevant dimensions. Architecture likelihoods are normalised so that the maximum observed is before modelling with the GP surrogate. The noise hyperparameter is also selected to optimise the probability density assigned to the observed data under the GP prior. It constrained in the range [10-5, 10-1]. The acquisition function is always optimised using an evolutionary strategy, using pool size of 1024. Per iteration, we allow 128 mutations, of which 16 are modifications of the architecture with the highest acquisition value, and the remainder are selected uniformly at random from the pool.
optimised using LBFGS. The lengthscales
constrained between
18

---

Published in Transactions on Machine Learning Research (07 /023)
CIFAR-100
ImageNet16-120
Algorithm
ECE
C-AUC
ECE
C-AUC
M = 3
NES-RE
0.026 = 0.002
0.873 + 0.002
0.033 + 0.002
0.823 + 0.002
NES-BS
0.073 = 0.009
0.848 + 0.003
0.058 + 0.003
0.809 + 0.003
BQ-R
0.075 = 0.025
0.866 + 0.002
0.052 + 0.021
0.815 0.003
BQ-S
0.021 0.001
0.874 + 0.001
0.029 + 0.001
0.824 0.002
M = 5
NES-RE
0.042 + 0.002
0.875 + 0.001
0.051 +: 0.001
0.826 + 0.002
NES-BS
0.073 = 0.009
0.848 = 0.003
0.058 = 0.003
0.809
0.003
BQ-R
0.040 = 0.004
0.871 + 0.001
0.028 0.004
0.820 0.002
BQ-S
0.040 = 0.002
0.877 + 0.001
0.050 + 0.002
0.825 0.002
M = 10
NES-RE
0.060 + 0.001
0.877 + 0.001
0.069 + 0.001
0.825 0.001
NES-BS
0.085 = 0.005
0.848 + 0.001
0.068 + 0.004
0.808
0.002
BQ-R
0.037 0.002
0.872 = 0.001
0.018 = 0.001
0.819
0.002
BQ-S
0.059 + 0.001
0.879 + 0.001
0.072 + 0.001 0.826 + 0.002
Table 8: Expected calibration error (ECE) and calibration AUC (C-AUC) on CIFAR-100 and ImageNet16- 120 for our proposals (BQ-R and BQ-S) and baselines. Well-calibrated uncertainty is indicated by high calibration AUC but low expected calibration error. We see that the two metrics are generally in agreement about the quality of a model's uncertainty estimates.
CIFAR-100
Algorithm
Accuracy
ECE
LL
M = 3
BQ-R (MLE)
71.9 + 0.8
0.075 + 0.025
-5259 + 300.9
BQ-R (SWAG)
75.1
+ 0.6
0.035 = 0.007
-4516 + 140.2
BQ-S (MLF)
76.6
0.2
0.021
0.001
-4417 + 35.85
BQ-S (SWAG)
77.4
0.3
0.022
0.000
-4345 + 38.19
M = 5
BQ-R (MLF)
73.3
+ 0.9
0.040 + 0.004
-4768 + 174.3
BQ-R (SWAG)
75.0
1.1
0.039 + 0.012
-4335 + 133.4
BQ-S (MLE)
77.8
0.2
0.040 0.002
-4077 + 33.60
BQ-S (SWAG)
78.8
0.4
0.033 0.001
3394 + 25.47
M = 10
BQ-R (MLE)
75.5
+ 0.9
0.037 + 0.002
-4309 + 172.6
BQ-R (SWAG)
77.9
0.2
0.063 0.006
-3944 + 8.734
BQ-S (MLE)
78.6
0.2
0.059 0.001
-343 +\2 22.71
BQ-S (SWAG)
79.9 + 0.1
0.053 0.001
-3600 + 18.40
Table 9: Test accuracy, expected calibration error (ECE), and log- likeliol (LL) on CIFAR-100 for BQ-R and BQ-S with MLE and SWAG approximations for the architecture likelihood. Note that, due to their computation expense, results for the SWAG approximations are means and standard error of the mean over runs. For the MLF approximations, we report the mean and standard error of the mean over 10 runs.
19


Reviews:
[{"review_id": "em0SympTdJ", "reviewer": "Reviewer_bZZQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n**Strengths:**\n\n+ The idea of using Bayesian quadrature for NES is quite sensible and may deserver further investigation.\n\n+ A variant of the BQ approach demonstrates promising empirical performance.\n\n**Weaknesses:**\n\n- The BQ-R method, which is closer to a full implementation of Bayesian quadrature, performs worse than most baselines. Thus, the evidence for the efficacy of BQ is somewhat limited.\n\n- In the implementation, two marginalization operations over the NN parameters are replaced with an MLE approximation.  This is understandable given the increased computational cost of the latter, but it deviates from the BQ method and may be related to the previous point, the worsened performance of BQ-R.  The authors argued that is because the baselines have access to validation samples, but I think there should also be discussion around the MLE approximations, since (in many related settings) a fully Bayesian approach would not be hindered by the lack of validation samples.\n\n(Note that I'm not familiar with the recent empirical results on NAS and NES, and my evaluation of the experiments is purely based on the authors' report.)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes using Bayesian quadrature (BQ) for neural ensemble search (NES), which is a natural counterpart of using Bayesian optimization for neural architecture search (NAS).  The authors propose two approaches for NES based on a Gaussian process model over the architectural DAGs: a more faithful implementation to BQ (BQ-R), and a stacking variant (BQ-S) that reweight the learned ensemble using additional validation samples.  Across several benchmarks constructed out of the CIFAR and ImageNet datasets, the BQ-R method outperforms recent NES baselines in terms of uncertainty-related metrics. \n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "\n- (Enhancement) More discussion about the performance of BQ-R.\n\n- (Enhancement) In Table 3 and 5 the BQ-R method is only competitive in the log likelihood (LL) metric.  The authors argued that the ECE and LL metrics are \"crucial for systems which make critical decisions\", but this is only obviously true for ECE (or other metrics that directly reflect calibration error), so there should be more discussions around the use of the LL metric.\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IIR1HUVGNu", "reviewer": "Reviewer_M93y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n+ **Novelty:** While novelty is not an acceptance criterion for TMLR, this paper does propose a novel and interesting approach to ensemble construction. I suspect that many people would find this work stimulating and that it could lead to further interesting work in the future.\n\n+ **Solid Experiments:** The experimental evaluation seems solid, and most of the claims are well supported. \n\n## Weaknesses\n\n- **Clarity:** I found the paper hard to understand in several places. The main issue is that the authors do not guide the reader through a few key sections. As a result, many statements seem arbitrary and do not follow previous statements, and I struggled to gain insight into some parts of the method. Making connections more explicit and assuming less knowledge/understanding on the reader's part would help greatly. I make concrete suggestions in the next section. \n\n- **Evidence:** There are a small number of key statements for which I do not believe there is sufficient evidence in the paper.\n   1. Existing approaches to ensembling struggle when the architecture likelihood surface has dispersed and narrow peaks.\n   2. Bayesian Quadrature is well suited to exploring such likelihood surfaces. \n   3. The proposed method improves calibration. The paper uses two metrics–LL and ECE–as measures of calibration. However, LL is not an ideal choice because it combines both accuracy and uncertainty calibration, making it difficult to disentangle where the improvement is coming from. Furthermore, in almost all cases, when LL improves, so does the accuracy, which does not help. ECE, on the other hand, is not entangled with accuracy but has several other pathologies. For example, a classifier can get perfect ECE while predicting only the marginal probabilities for each class. Another issue is that it is not obvious how ECE should be applied in a multi-class setting. \n  4. Bayesian Quadrature is being used to construct the ensemble. Or rather, the Bayesian Quadrature used to construct the ensemble is well approximated. My concern is the approximations used in equations (14) and (16). These seem like very strong approximations, making it unclear whether the proposed method is doing something close to proper quadrature or if we should view the method as a well-performing heuristic inspired by quadrature. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper uses Bayesian Quadrature to construct ensembles of neural networks. The key observation is that constructing an ensemble of neural networks can be seen as marginalizing over different neural network architectures and that this marginalization can be approximated using quadrature. This is analogous to the way in which neural architecture search can be seen as an optimization problem over the space of different architectures that can be approximated using Bayesian optimization. The paper views ensemble construction as a two-step process. Firstly, a set of candidate architectures is selected using an acquisition function. Secondly, a (weighted) ensemble is constructed from a subset of the candidate architectures. The first problem is solved with a novel use of Bayesian Quadrature. Two potential methods are proposed for solving the second problem, each of which is better in some settings.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I use [major] and [minor] to indicate the changes that would be crucial to securing my recommendation or would simply strengthen the work, respectively. Although most clarity issues are listed individually as [minor], they represent a more major issue when taken together. \n\n1. [major] Add a citation (or other evidence) for the claim that \"... existing approaches to ensembling struggle when the architecture likelihood surface has dispersed, narrow peaks\".\n2. [major] Add a citation for the claim that Bayesian Quadrature is \"... well suited to exploring likelihood surfaces with dispersed, narrow peaks\".\n3. [minor] Add a discussion of Wenzel et al. (2020), alongside Zaidi et al. (2022) and Shu et al. (2022) when discussing the idea that ensembles of different architectures outperform ensembles of the same architecture. Wenzel et al. (2020) show that ensembles with different hyperparameters (which could include architectural hyperparameters) outperform standard ensembles. \n4. [minor] The claim that single architectures provide poor uncertainty estimates is not well supported. Firstly, Guo et al. (2017) focus on a narrow range of architectures (namely CNNs), and a follow-up study (Minderer et al., 2021) shows that this is not true for other architectures (e.g. ViTs). Secondly, several techniques can be applied to single models to improve calibration. I would change this statement and instead say that ensembles tend to be better calibrated than single models. \n5. [major] Add additional uncertainty metrics. I recommend using Calibration AUC (Kivlichan et al., 2021, Tran et al., 2022), which does not suffer from the same pathologies as ECE. Oracle Collaborative AUC and OOD detection AUC could also be a good choices. I acknowledge that adding these additional metrics to all experiments might require a large amount of work. However, I think that adding just Calibration AUC to only Table 4 would be good enough as if there is a good correlation between ECE and Calibration AUC there, then we can assume that the ECE results are reliable elsewhere. \n6. [minor] Section 2.5 must be better integrated into the rest of the text. Reading the paper for the first time, this section seemed very arbitrary, and I was unsure how this would connect to the rest of the work. Even after a second reading, I felt it would be helpful to make connections between this section and the rest of the paper more explicit. \n7. [minor] Add a legend to figure 1. What do the different colors mean? \n8. [minor] On page 6, 3rd paragraph, the sentence \"Therefore, we propose using a Bayesian Quadrature acquisition function to build the candidate set ...\", did not follow from the previous sentence. Adding additional explanation and intuition would be helpful. \n9. [major] Add an ablation study to show that the approximations in equations (14) and (16) are reasonable. Even in a small-scale setting and using some approximate inference scheme (I would suggest linearised Laplace), it would be useful to compare the performance of the proposed method with and without marginalizing the weights in (14) and (16). Otherwise, it is impossible to know what kind of impact the approximation has and how close the proposed method is in practice.\n10. [minor] I think it would be useful to be more explicit about the different levels of Bayesian inference that are taking place. There is a first level in which we do inference over weights. And a second level where we do inference over architectures. The evidence of the first level is the likelihood of the second level. Chapter 28 of \"Information Theory, Inference, and Learning Algorithms\" by David J.C. MacKay explains this well. \n11. [minor] The statement \"computation of this estimate requires Monte-Carlo sampling to approximate sums of (products of) the WL kernel over A\" needs to be made more explicit by writing out the MC sampling equation.\n12. [minor] A few preliminary results showing the impact of using the union of the train and validation sets for BQ-R would be an interesting and useful addition. \n13. [minor] Add a short discussion of the limitation of the method.\n14. [major] Add an appendix section which describes the experimental setup in detail. Details such as which deep learning framework, how many random seeds for each experiment, and any hyper-parameter sweeps and settings should be included. \n\n\n\n**References**\n\n* Florian Wenzel, Jasper Snoek, Dustin Tran, Rodolphe Jenatton: Hyperparameter Ensembles for Robustness and Uncertainty Quantification. NeurIPS 2020\n\n* Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, Mario Lucic:\nRevisiting the Calibration of Modern Neural Networks. NeurIPS 2021: 15682-15694\n\n* Ian D. Kivlichan, Zi Lin, Jeremiah Z. Liu, Lucy Vasserman:\nMeasuring and Improving Model-Moderator Collaboration using Uncertainty Estimation. CoRR abs/2107.04212 (2021)\n\n* Dustin Tran, Jeremiah Z. Liu, Michael W. Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, Neil Band, Tim G. J. Rudner, Karan Singhal, Zachary Nado, Joost van Amersfoort, Andreas Kirsch, Rodolphe Jenatton, Nithum Thain, Honglin Yuan, Kelly Buchanan, Kevin Murphy, D. Sculley, Yarin Gal, Zoubin Ghahramani, Jasper Snoek, Balaji Lakshminarayanan:\nPlex: Towards Reliability using Pretrained Large Model Extensions. CoRR abs/2207.07411 (2022)", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hG7Fw2NkXH", "reviewer": "Reviewer_56g8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is clearly written.\n- NES is an important problem.\n- Using ideas from BQ is a creative approach and seems to be useful.\n\nWeaknesses:\n- Related work is missing.\n- Similarly, some important baselines are missing.\n- There is little discussion and empirical results on the computational cost / runtimes.\n\nMajor comments:\n- In the Algorithm boxes, you use a lot of shorthand functions (acquisition_function, nystrom_test_function, reweight, etc). I think it would be more useful to write these out in maths, so one can see at one glance what is happening.\n- While the experiments provide useful ablations between the different strategies, I feel like they are missing more common baselines. I would expect to see a standard deep ensemble (with the fixed largest architecture from the search space), a hyper-deep ensemble [1] and maybe some kind of diversity-regularized ensemble [e.g., 2-8].\n- The paper is entirely missing a related work section. What have people done before to do NES? How does it compare to existing NAS approaches? What should at least be mentioned are the very related hyper-deep ensembles [1] and probably some examples of other diversity-regularized ensembles [e.g., 2-8].\n\nMinor comments:\n- In Tab. 2, why do US and RE perform differently on ImageNet and CIFAR? Do they encourage different kinds of architectures or amounts of diversity? It would be interesting to examine this more closely.\n- How do the methods compare in terms of runtime? It would be useful for practitioners to know the tradeoffs, so reporting wallclock or GPU runtimes in all experiments would be helpful.\n\n[1] https://arxiv.org/abs/2006.13570\n\n[2] https://arxiv.org/abs/1806.03335\n\n[3] https://arxiv.org/abs/2206.03633\n\n[4] https://openreview.net/forum?id=BJlahxHYDS\n\n[5] https://proceedings.neurips.cc/paper/2020/hash/0b1ec366924b26fc98fa7b71a9c249cf-Abstract.html\n\n[6] https://arxiv.org/abs/1802.07881\n\n[7] https://arxiv.org/abs/2106.10760\n\n[8] https://proceedings.neurips.cc/paper/2021/hash/1c63926ebcabda26b5cdb31b5cc91efb-Abstract.html", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors consider the problem of Neural Ensemble Search (NES), that is, Neural Architecture Search over the space of neural network ensembles. They phrase this as a Bayesian Quadrature (BQ) problem, in which the selected ensemble members are used to approximate the marginal predictive over all architectures. They show that using acquisition functions from BQ can help with this problem and also propose a new recombination strategy for model weighting.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Add a related work section\n- Add some baselines (see above)\n- Add runtimes to the tables.", "broader_impact_concerns": "No broader impact concers.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel approach for Neural Ensemble Search using Bayesian Quadrature. A number of issues were raised, but the main ones concerning the method itself focused on the approximations involved (BQ-R vs BQ-S)  and the use of log-likelihood to make claims about modelling uncertainty. There was relatively little discussion needed as the author's subsequent draft resolved all of these issues along with many other minor ones. The reviewers unanimously agreed that the paper should be accepted to TMLR.
Decision: Accept (poster)