Ensembles for Uncertainty Estimation: Benefits of Prior Functions and Bootstrapping
Authors: Vikranth Dwaracherla
Zheng Wen
Ian Osband
Xiuyuan Lu
Seyed Mohammad Asghari
Benjamin Van Roy
Venue: Accepted by TMLR ()

Abstract: In machine learning, an agent needs to estimate uncertainty to efficiently explore and adapt and to make effective decisions. A common approach to uncertainty estimation maintains an ensemble of models. In recent years, several approaches have been proposed for training ensembles, and conflicting views prevail with regards to the importance of various ingredients of these approaches. In this paper, we aim to address the benefits of two ingredients -- prior functions and bootstrapping -- which have come into question. We show that prior functions can significantly improve an ensemble agent's joint predictions across inputs and that bootstrapping affords additional benefits if the signal-to-noise ratio varies across inputs.  Our claims are justified by both theoretical and experimental results.

Published in Transactions on Machine Learning Research (04/2023)
Ensembles for Uncertainty Estimation: Benefits of Prior Functions and Bootstrapping
Vikranth Dwaracherla; Zheng Wen; Ian Osband, Xiuyuan Lu, Seyed Mohammad Asghari, Benjamin Van Roy
Efficient Agent Team, DeepMind, Mountain View, CA
Reviewed on OpenReview: https: // openreview. net/ forun? id= IqJyyulDU
Abstract
In machine learning, an agent needs to estimate uncertainty efficiently explore and adapt and to make effective decisions. A common approach to uncertainty estimation maintains an ensemble of models. In recent years, several approaches have been proposed for training ensembles, and conflicting views prevail with regards lo the importance of various ingredients of these approaches. In this paper, we aim to address the benefits of two ingredients prior functions and bootstrapping which have come into question. We show that prior functions can significantly improve an ensemble agent's 'ss joint predictions across inputs and that bootstrapping affords additional benefits if the signal-to-noise ratio varies across inputs. Our claims are justified by both theoretical and experimental results.
1 Introduction
Effective decision making, exploration, and adaption often require an agent to know what it knows and also what it does not know, which in turn relies on the agent's capability to estimate uncertainty, For instance, in reinforcement lcarning (Sutton, 1988) and bandit (Lattimore & Szepesvári, 2020) problems, efficient exploration schemes such as upper-confidence bound (Lattimore & Szepesvári, 2020) and Thompson sampling (Russo et al., 2018) heavily rely on the agent's ability to model uncertainty. In the past few decades, many different approaches have been developed for uncertainty estimation, such as dropout (Gal & Ghahramani, 2016), Bayes by Backprop (Blundell et al., 2015), hypermodels (Dwaracherla et al., 2020), epinets (Osband al., 2021), and stochastic Langevin MCMC (Welling & Teh, 2011; Dwaracherla & Van Roy, 2020). Maintaining an ensemble of models is one of the simplest and most widely used uncertainty estimation method. In particular, several variants of ensemble agents have been developed for uncertainty estimation (Lalshminarayanan et al., 2017; Fort et al., 2019; Osband et al., 2018) and downstream decision problems, such as bandits (Lu & Van Roy, 2017) and reinforcement learning (Osband et al., 2016). Compared to other uncertainty estimation approaches, the simplicity of ensemble agents makes it easier to analyze and implement in many practical problems. Moreover, since most uncertainty modeling methods face similar challenges, we believe that many insights gained from studying ensemble agents also translate to other methods, such hypermodels and epinets.
An ensemble agent represents uncertainty based on a set of different models, but how should we train such different models based on the same training dataset and prior knowledge? Different approaches have been proposed, and it is still highly debated which approaches should be adopted for various practical problems. Some recent. papers (Takshmiiarryyaaa et al., 2017; Fort al., 2019) suggest, that we can train such models just with random parameter initialization and data shuffling. However, this approach does not pass some simple sanity checks; for example, it does not produce diversity across models when applied to linear regression. On the other hand, Osband et al. (2018) and He et al. (2020) propose to train ensemble agents with randomized prior functions and emphasize the importance of incorporating agent's 's prior uncertainty into
"equal contribution

---

Published in Transactions on Machine Learning Research (04/2023)
ensemble agent training. Finally, bootstrapping (Efron & Tibshirani, 1994) is class of widely used techniques to train diverse models. However, a recent paper (Nixon et al., 2020) questions whether bootstrapping plays an essential role and suggests that it can even hurt performance. More broadly, the literature presents conflicting messages with regards to the importance of prior functions and bootstrapping. In this paper, we aim to bring clarity to these issues.
Most work on supervised learning has focused on producing accurate marginal predictions for each input. However, recent papers (Wen et al., 2021; Wang et al., 2021; Osband et al., 2022a) emphasize the importance of accurate joint predictions. In particular, accurate joint predictions are crucial for efficient exploration, adaptation and effective decision making. Taking cue from this line of rescarch, we evaluate performance of ensemble agents based on their joint as well as marginal predictions.
We frame a class of ensemble agents - which includes those with and without prior functions or bootstrapping - through proposing loss function perturbation as a unifying concept. By systematically comparing performance of agents across this class, we elucidate benefits of prior functions and bootstrapping. In particular, we show that prior functions can significantly enhance an ensemble agent's joint predictions. As explained in (Wen et al., 2021), joint predictions drive efficiency of exploration and adaptation. With regards to bootstrapping, we show that further benefits depend on the variation of the signal-to-noise ratio (SNR) across the input space. If the SNR is uniform across the input space and prior functions are appropriately tuned then bootstrapping offers no significant value. On the other hand, if the SNR varies substantially across the input space then bootstrapping can significantly improve performance regardless of how prior functions are tuned.
The remainder of this paper proceeds as follows: we first review literature in Section 2 and formulate the problem in Section 3. Then we justify the main points of this paper, as discussed above, via analysis on Bayesian linear regression (Section 4), numerical experiments on classification using neural networks (Section 5) and bandit problems (Section 6). For experiments on classification we consider the synthetic data generated by Neural Testbed (Osband et al., 2022a), and CIFAR10 (Krizhevsky, 2009), a widely used benchmark image dataset. Finally, Section 7 concludes the paper.
2 Literature review
In recent years, in addition to improving prediction accuracy (Hastie et al., 2009; Dietterich, 2000), ensemble agents have also been built for uncertainty estimation (Lakshminarayanan et al., 2017; Fort et al., 2019; Osband et al., 2018) and decision-making (Lu & Van Roy, 2017). In particular, Lakshminarayanan et al. (2017) proposed to train ensemble agents for uncertainty estimation only based on random parameter initialization and data shuffling. Though this simple approach might work in certain scenarios, it does not pass simple sanity checks as shown in Scction 4. On the other hand, Osband et al. (2018); Hc et al. (2020) proposed to train ensemble agents by effectively perturbing the loss function of each model with a random but fixed additive prior function, which incorporates the agent's prior uncertainty.
There is long line of research on the topic of diversity in deep ensembles. Specifically, Shui et al. (2018) proposes to train deep ensembles with diversity function regularization. Jain et al. (2020) proposes to train ensemble models for uncertainty estimation by maximizing "overall diversity". Nam et al. (2021) studies knowledge distillation (KD) for deep ensembles, and proposes a new approach based on Output Diversified Sampling (ODS). D' A'Angelo& & Fortuin (2021) studies repulsive deep ensembles, which added a kernelized repulsive term in the update rule of the deep ensembles. It also shows that the training dynamics of the proposed approach follow a Wasserstein gradient flow of the KL divergence to the true posterior. Rame & Cord (2021) introduces novel training criterion called DICE, which increases the diversity by reducing spurious correlations among features. Theoretical properties of ensembles with prior functions have previously been discussed in Ciosck et al. (2020). However, we believe that thcoretical insights there suffer from some major limitations: first, Ciosek et al. (2020) assumes that uncertainty is independent across different inputs. Ideally if we expect the uncertainty at one input is high, so should the uncertainty at another input that is close. However, this not captured by the mothod proposed in Ciosck et al. (2020). Second, Ciosck et al. (2020) assumes that uncertainty only depends on the input. Though this assumption is reasonable for the linear setting, it is quite restrictive for more complex models.
2

---

Published in Transactions on Machine Learning Research (04/2023)
Ensemble agents are also widely used in sequential decision problems, such bandits (Lu & Van Roy, 2017) and reinforcement learning (Osband et al., 2016). In particular, ensemble sampling (Tu & Van Roy, 2017; Qin et al., 2022) is particular version of Thompson sampling (Thompson, 1933; Russo et al., 2018) that uses an ensemble of models to represent the uncertainty.
In most existing literature, agents are evaluated based on their marginal predictive distributions. Examples of such metrics include marginal negative log- log-likelihood (NLL), Brier score (Brier et al., 1950), accuracy, and expected calibration error (ECE) (Guo et al., 2017). Recent papers (Wen al., 2021; Wang et al., 2021) discussed the importance of joint predictive distribution for wide range of downstream tasks, such as sequential decision problems. Practical evaluation metrics have also been developed to evaluate the joint predictive distributions (Osband et al., 2022a;b). A recent paper (Nixon et al., 2020) suggests that non-parametric bootstrapping might not be helpful for building better ensemble agents. However, this paper is limited small ensemble sizes, high-SNR deep learning problems, and the agents are evaluated based on their marginal predictive distributions. In addition to ensemble agents, there are other agents developed for uncertainly est estimation, discussed in Section 1. Recently, the concept of epistemic neural network (Osband et al., 2021) has been developed unifying umbrella for these agents.
3 Preliminaries
Consider a sequence of data pairs ((,,, :,,,,,,,,, where each X, is an input vector and Y++1 is the associated output. Each output Y441 is independent of all other data, conditioned on X₁, and distributed according to {())x)) The conditional distribution & is referred to the environment. The environment & is random; and this reflects the agent's uncertainty about how outputs generated given inputs.
Note that P(Y++1 E X,) = E(1)x,)
and P(Y+1 E (1,)) = [+),
agent
random initialization
prior function
bootstrapping
ensemble-N
Yes
No
No
ensemble-P
Yes
Yes
No
ensemble-BP
Yes
Yes
Yes
Table 1: Comparison of ensemble-N, ensemble-P, and ensemble-BP: whether or not random initialization, prior function, and bootstrapping are used for training different models.
To illustrate the main points of this paper, we consider three variants of ensemble agents, referred to as ensemble-N, ensemble-P. and ensemble-BP All these agents include an ensemble of models, and are trained based on Dr = ((X,, Y(+1) 0, ............ 1). However, they differ in if prior functions and bootstrapping are used for training, illustrated in Table 1. In particular, the models in ensemble-N are trained with the same loss function but different random initialization. On the other hand, in addition random initialization, models in ensemble-P are trained with random prior functions, and models in ensemble- BP are trained with both prior functions and bootstrapping. Note that we will not consider an ensemble agent that is trained with bootstrapping alone and without random prior functions. This is because when there is no data (e.g. in cold-start problems)) bootstrapping alone cannot induce any uncertainty, irrespective of the bootstrapping noise level.
For each ensemble agent, denote its m-th model Êm, and hence the agent is represented by θ,- (Éi...., E,....,,,,,, where M is the nunber of models in the ensemble. Note that ensemble agent defines a discrete distribution in the space of environments:
P E -(Or) M EM 1 E.:
where & is the agent's model of the environment. This discrete distribution is an approximation of the true posterior (E E (r), and characterizes the agent's uncertainty about E.
1ap" stands for prior function, "B" bootstrapping, and "N" for neither.
3

---

Published in Transactions on Machine Learning Research (04/2023)
We now briefty discuss how to evaluate an ensemble agent's performance. In supervised learning, natural metric is the expected Kullback-Leibler (KT.) divergence between P E (Dr) and P E - Tn Section 4, we use this metric to evaluate ensemble agents with a large number of models in linear regression problems. However, for more general problems, this metric is computationally intractable and can also be infinite. Thus, following previous papers (Wen et al., 2021; Osband et al., 2022a;b), we evaluate the ensemble agent based on its predictive distributions. Specifically, for any inputs XT.T+7-1 = (Xr...., XT++-1), the ensemble agent determines predictive distribution, which could be used to sample imagined outcomes YT+I-++> = (fr.....,,yu Hence, the agent's 7-th order predictive distribution is given by
PTo---11 = P(P E clOrr, Xr:++r1));
which represents an approximation to the underlying output distribution that would be obtained by condi- tioning on the environment:
Pir++-1 = (YT+1.++7 E (E, XT:
(2)
= 1, this represents a marginal prediction; that is a prediction of a single output. For T > 1, this is a joint prediction over a sequence of T outputs. Recent papers (Wen et al., 2021; Osband al., 2022a)
propose to use the expected KL-divergence between P7:++-1 and Prot+--1 referred refereed to dkr. to evaluate both the marginal prediction and the joint prediction. Osband et al. (2022b) further proposed a variant of dii based on dyadic sampling, referred to as d to evaluate the agents' joint prediction for practical problems with high imput dimension. In Section 5, we follow Osband et al. (2022b) to use dkr, with 7 = 1 to evaluate marginal predictions and dki. with = 10 to evaluate joint predictions. Finally, in bandit problems (Section 6), we use the standard expected cumulative regret to evaluate ensemble agents. We will further discuss the evaluation metrics in subsequent sections and Appendix A.
4 Bayesian linear regression
In this section, we use the classical Bayesian linear regression problem as a didactic example to compare ensemble-N, ensemble-P, and ensemble-BP with large number of models. In particular, we highlight the benefits of bootstrapping ensemble-BP.
Consider a Bayesian linear regression problem with heteroscedastic noises:
Y2+1 = 0%' + Wt+1, t= 0,
(3)
where 0g E Fdd is the coefficient vector and identifies the environment E, and W4+1' are additive noises. We assume that inputs X,'s are i.i.d. sampled from an input distribution Px. Moreover, conditioned on Xr, the additive noise W:+1 independently sampled from N(0, o( where o² Rdd + is the noise variance function, i.e., noise variance depends on imput. The agent does not know 9., but knows that prior over 0. is N(0, 2 and also knows (-). Conditioned on the training data Dr, the posterior over 0, is 0- Dr ~ N (HT, Er), where the posterior mean pr and the posterior covariance En are
Σx L/08 +I= x,xX /o(x0) and =T= = En ETD XXX++1/0((X))
4.1 Ensemble agents with a large number of models
We consider ensemble agents with a large number of models, and each m-th model is identified by a coefficient vector θ, Rd. As standard method for training ensemble agents lincar regression problems (Lu & Van Roy, 2017; Qin et al., 2022), we compute 6, by minimizing the following perturbed loss function:
Om arg min Σ
((x) 2
(0=xx- X. (Ye4) + Ze+1 2+m,,)])
> 110 0ml1? Vm 1, M
Note that this isometric prior assumption is without loss of generality: any Bayesian linear regression problem satisfies this assumption after appropriate coordinate transformation.
4

---

Published in Transactions on Machine Learning Research (04/2023)
where I,- is additive perturbation for data pair (X,, Yi+1) sampled from N (0. (X.)): v, o²: Rd - R are respectively the weight function and the bootstrapping variance function; À 2 0 is the regularization coefficient; and Om is independently sampled from N0, o.1). The magnitude of 0 reflects the agent's prior uncertainty, and hence dn is known as the agent's prior scale. Note that ||0 - 110-6mlle is a randomly perturbed regularizer, and equivalent to prior function for linear regression (Osband al., 2018). is straightforward to show that
6. = (Ei (X,)xxX + + (Et ((A) Z4t1,)) X2 20m)
(5)
Note that Zı-1. and 6m are random variables, and fm Dr ~ ((r; Er), where for and Ey are respectively the mean and covariance of 6m conditioned on Dr (see Appendix B.1 for their analytical solutions). We say an ensemble agent is unbiased if jJT PT.
As discussed Section 1, we consider three variants of ensemble agents: ensemble-N, ensemble-P, and ensemble-BP. Note that for Bayesian linear regression, all these three agents can be trained based on the perturbed loss funetion (4), but with different constraints on loss function parameters reflecting if prior functions and strappigg are enabled. In particular, ensemble-N does not use prior functions bootstrapping, and fix o 0 and (²) O. ensemble-P uses prior functions, but does not use bootstrapping by fixing (-)- = O. ensemble-BP BP uses both prior functions and bootstrapping. Table 2 summarizes the differences wetn ensemble-N, ensemble-P, and ensemble-BP.
agent
À and v(-)
(²(-)
ensemble-N
allowed to tune fixed at 0
fixed at 0
ensemble-P
allowed to tune allowed to tune
fixed at 0
ensemble-BP
Proposition 1 Proposition 1 Proposition
Table 2: Comparison of ensemble-N, ensemble-P, and ensemble-BP for linear regression. The parameters for ensemble-BP are chosen the optimal parameters (Proposition 1).
Note that the discrete distribution represented by an ensemble agent an empirical distribution of M samples from P(ém. E (Dr) (Ar, Moreover, as M - oo, this discrete distribution converges P(0m E Dr)). In this section, we choose to evaluate
B[dkL(P(0,. E (Dr) P(Om E (Dr)))]
(6)
which effectively measures the performance of ensemble agent with large number of models. Further discussion on this metric is provided in Appendix A. The following results show that for Bayesian linear regression, ensemble-N cannot represent any uncertainty, since its models use the same strictly convex loss function. On the other hand, ensemble-BP with infinite models is able to match the exact posterior P(e. E DD) by properly choosing the parameters in the perturbed loss function (4). The proofs are straightforward and are provided Appendix B.2.
Proposition (ensemble-BP) For perturbed loss function (4), if we choose = =))==)(()) v())),)())))))))) 1/02(), 0e oû and À = 1/0 then P(om. E ( P E l
Proposition 2 (ensemble-t)) For perturbed loss function (4), if we choose o²(-) and 03 0, then θ. is deterministic conditioned on Dr.
4.2 Benefits of bootstrapping
We now compare ensemble-P and ensemble-BP, and show the benefits of bootstrapping. Note that Proposi- tion 1 indicates that by appropriately choosing parameters for ensemble-BP, P(ê.,. E | = P (0, E Dr). On the other hand, derive lower bound on the expected KT-divergence for unbiased ensemble-P based
5

---

Published in Transactions on Machine Learning Research (04/2023)
expected signal-to-noise (SNR) ratios along orthogonal directions. Consider the positive semi-definite matrix
r T= xxx /o2
where the expectation is over the input X Let 7i, i= = 1, ., let D, denote the normalized eigenvector associated with %/2 Recall that Wts are nonnegative and 0,'s are orthonormal. Hence, we have X, Ei= v0") (xFv.) and
.....,d, d, denote the d eigenvalues of r, and
v2) = E
V (X7 (0 X
= 4 Te; =
E
o2())
where V[1 denotes the (conditional) variance. Note that V (({ o0 vx) Xx] can be interpreted the magnitude of signal at X, along direction vi, and (Xe) is the magnitude of noise at X,, Thus, eigenvalue " can be interpreted as expected SNR along direction us We have the following lower bound for unbiased ensemble-P based on ''ss:
Theorem For perturbed loss function (4) and ensemble-P agents with (²)-) 0, under any choice of v()), X, and og such that ensemble-P is unbiased, we have
E E [dKI. (P(O.E-IDT) (mm E e.(Dr))
Σ In
1 1+77 1 T7:
(8)
where % 's are d eigenvalues of I and 9 Ei= %
Please refer Appendix B.3 for the proof of Theorem Note that this lower bound is always non-negative due Jensen's inequality, and is zero when "i's are equal. Theorem states that if expected SNRs along different directions are very different, or equivalently if matrix I is "ill-conditioned"" then the expected KL defined in (6) is large for unbiased ensemble-P. Note that I can be ill-conditioned for several different reasons. For instance, I' is likely to be ill-conditioned if ox,) varies significantly across different directions (heteroscodasttcc. or if the covariance matrix of the input distribution Px ill-oonditioned.
Theorem a general lower bound and demonstrates the fundamental limitation of ensemble-P and potential benefits of bootstrapping. However, there are special cases where bootstrapping is not strictly necessary and ensemble-p perform well after appropriate paramcter tuning for cach setting. Please refer to Appendix B.4 for such examples.
5 Classification on Synthetic and Real Data
In this section, we switch the gear to compare ensemble-N, ensemble-P, and ensemble-BP in classification problems. In specific, we look at classification problems on an open-source test suite called Neural Testbed (Osband et al., 2022a) and a widely used benchmark image dataset, CIFAR10 (Krizhevsky, 2009). More details about Neural Testbed and CIFAR10 dataset are provided Appendices and D. Our experiment. results show that many insights developed in Scction 4 also translatc to these classification problems. is worth mentioning that in this section, we consider ensemble agents with limited number of models, and each model is neural network.
In Section 4, we have seen that ensemble-N reduces to single point estimate. This might be bit contrived as the loss function is not convex in many deep learning problems. One would hope that, with neural networks, random initialization sufficient and ensemble-N will perform well, as suggested by Lakshminarayanan et al. (2017). Indeed, in our experiments, we find that ensemble-N improves over the performance single point estimate. However, with prior functions and bootstrapping, the performance of ensemble agents can be further improved significantly.

---

Published in Transactions on Machine Learning Research (04/2023)
Given dataset Dr= (Xe, o the m-th model in ensemble agent for the classification problems aims to minimize the perturbed loss function of form
exp (Jom (X+))
[(0m; DT)
Σ W log
> 1 11
(9)
E exp (fo. (Xx));
where W is the weight corresponding to the data pair (X,, Y241) and the m-th model in the ensemble, fx(x E Rl is the vector of logits for input Xt, and y is the set of possible labels. Note that the weights W are sampled iiid, for each model and data pair (Xe, Y441), at the start and remain fixed for the entire experiment. For both ensemble-N and ensemble-P agents, Wi+1, 1 Vt, m, while for an ensemble-BP agent, W111,m are sampled i.i.d. from 2 x Bernoulli(O.5.
5.1 Dyadic Sampling
We evaluate the agents on both marginal predictions and joint predictions. In specifie, we use dyadic sampling (Osband et al., 2022b) to evaluate agents on joint predictions. We provide a high-level overview of dyadic sampling here, and please refer to Appendix A for more details.
Dyadic sampling is a computationally efficient approach to evaluate joint predictions, which provides additional insights beyond marginal predictions. In a high level, dyadic sampling proceeds as follows: it first samples two random anchor points from the input space, and then resamples t inputs from these two anchor points. The expected KL-divergence under dyadic sampling is referred to as dki- Similar to Osband et al. (2022b), we use dyadic sampling with r =0 10.
Moreover, as discussed in recent papers (Wen et al., 2021; Osband et al., 2022b), expected KL-divergence is equivalent to negative log-likelihood (ull), which is also known as the cross-entropy loss, for agent comparison. Thus, for experiments on the CIFARIO dataset, where the likelihood under the true environment is unknown, we use cross-entropy loss under dyadic sampling to compare agents.
5.2 Experiments on Neural Testbed
We first compare ensemble-N, ensemble-P, and ensemble-BP on Neural Testbed to demonstrate the benefits of prior functions and bootstrapping in different scenarios. When comparing agents on Neural Test the network for the single point estimate is chosen to have the same architecture as the generative model, 2-layer MT.P. We refer to this agent as mlp agent. The ensemble-N agent uses an ensemble of 2-layer MLPs. The ensemble-P agent uses an ensemble of single point estimates combined with additive prior function (Osband et al., 2018) at logits. The additive prior functions for each model of ensemble-P agent are different only through initialization. In particular, we use random MLPs, with the same architecture as the generative model, as the prior functions for experiments on Neural Testbod. The ensemble-BP agent uses the same model as an ensemble-P agent, but uses bootstrapping to train the model. Please find more details about Neural Testbed and agents in Appendix C.
Figure 1 comparcs mlp ensemble-N, ensemble-P, and ensemble-BP agents on Neural Testbed problems with input dimension 100. All the results are normalized w.r.t performance of mlp agent. We consider two ways of tuning the hyperparameters, per problem and global tuning, where we pick a single hyperparameter set for all Neural Testbed problems. We observe that in both types of tuning, all the agents perform similarly in marginal predictions. None of the ensemble agents are statistically distinguishable from an mlp which uses a single point estimate. However, when evaluated by joint predictions, the agents are clearly distinguished. When we pick the best hyperparameter per problem (after averaging over 10 random seeds), we can see that ensemble-N offers some advantage over mlp. This is significantly improved by ensemble-P which uses prior functions. This highlights the benefits of prior functions. With bootstrapping, ensemble-Bi performs very similarly to ensemble-P and doesn't improve the performance further. However, when a single hyperparameter is used over the entire testbed, ensemble- agent performs better than mlp and other ensemble agents. This indicates that one benefit of bootstrapping robustness to parameter tuning.
As presented in Section 4.2, for classification problems with different SNRs across inputs, bootstrapping can further enhance the performance of ensemble agents. To illustrate this, we consider Neural Testbed problems
7

---

Published in Transactions on Machine Learning Research (04/2023)
normalized marginal kl
normalized joint kl
agent
mlp
ensemble-N
ensemble-P
0.6
ensemble-BP
0.4
0.4
global
per ser_setting
global per_setting
tuning
tuning
Figure 1: Marginal predictions do not distinguish agents. Prior functions significantly improve the joint predictions, with per setting tuning. Bootstrapping increases robustness to parameter specification and improves predictions, with global tuning.
with medium temperature 0.1, and with 25% of samples with label 1 in the training dataset randomly flipped label 0 for cach problem. This essentially generates two different SNRs across inputs for cach problem. We average the results over 100 random seeds. Figure shows the performance of ensemble-P and ensemble-BP agents marginal and joint predictions. Best hyperparameters We observe that ensemble-Bi agent doesn't show a statistically significant improvement over ensemble-P in marginal predictions, but does show such an improvement in joint predictions. A more detail breakdown of the performance across different data regimes and results from additional experiments are provided in Appendix C.2.
chosen per setting for both the agents.
0.55
0.30
marginal kl
joint kl
0.28
0.27
0.45
0.26
0.40
0.25
ensemble-P ensemble-BP
ensemble-P ensemble-8P
agent
agent
Figure 2: Marginal and joint KL on Neural Testbed problems with varying SNR.
5.3 Experiments on CIFAR10
In Section 5.2, we have seen results highlighting the benefits of prior functions and bootstrapping Neural Testbed. Neural Testbed offers us simple and transparent setting and helps us do systematic research and gain insights. However, at the end the ensemble agents need to be applied on practical problems. We compare the considered ensemble agents on CIFAR10 dataset and show the benefits prior functions and bootstrapping.
CIFAR10 has been widely known in the deep learning community as benchmark dataset and used test classification algorithms. We look at the average performance across 4 problems, each with different training dataset size from {10, 100, 1000, 50000), where 50000 is the size of full training data set. For these experiments, we use vgg model (Simonyan & Zisserman, 2014) as the single point estimate model. An ensemble-it agent consists of an ensemble of vgg models and ensemble-P agent consists of an ensemblle of models with each model being a vgg model combined with a small randomly initialized convolution network at logits. The ensemble-BP agent uses the same network architecture as ensemble-P agent, but uses loss functions perturbed via bootstrapping. In specific, we choose weights in Equation as WI+1. Bernoull(p)) /p. We consider three variants with p 0.5, p 0.75, and p 0.9, and the respective agents referred to
8

---

Published in Transactions on Machine Learning Research (04/2023)
ensemble-BP(0.3) ensemble-BP(0.7)) and ensemble-BP(0.9) More details about the agents and the CIFAR10 dataset are provided in Appendix D.
Figure 3 compares the vgg, ensemble-N and ensemble-BP agents. The ensemble agents use 100 models, averaged across random seeds, and we normalize performance w..tt vgg. Similar to Figure 1, we observe that both ensemble-N and ensemble-P perform similarly on marginal predictions and offer some improvement over single point estimate, vgg. However, once evaluated by joint predictions, ensemble-P performs much better than ensemble-te M, and both ensemble-N and ensemble-P improve vgg. The agents evaluated by negative log- likelihood, which is equivalent to expected KL divergence for agent comparison (Appendix A.3). A detail breakdown of performance across different problems is provided in Appendix D.2.
marginal nll
0.75
0.75
joint nll
0.50
0.50
0.25
0.25
vgg
ensemble-N agent
ensemble-P
vgg
ensemble-N
ensemble-P
agent
Figure 3: Prior functions significantly improve ensemble agent's joint predictions.
In Figure 3, observe that prior functions significantly improve an ensemble agent's joint predictions. In Figure 4, We compare the ensemble-BP agents along with other ensemble agents and show that bootstrapping doesn't improve the performance over well tuned ensemble-P agent CIFAR1O problems. We use ensemble of size 10 for all the agents, average the results over 5 random seeds, and normalize the performance w.r.t ensemble-N agent. For ensemble-P and ensemble-BP agents prior scale is swept. over (1,3, 10,30) and the best prior scale is chosen per problem. We observe that all the ensemble agents perform pretty similarly on marginal predictions, except for ensemble-BP(0.5). On comparing the agents on joint predictions, we can see that prior functions offer a clear advantage and bootstrapping doesn't offer an advantage over well tuned ensemble-P agent. This is consistent with observations from Appendix B.4. We suspect that the poor performance of ensemble-BP(0.5) on both marginal and joint predictions could be due to the CIFARI0 problems being high-SNR problems and the number of models in each ensemble is small. This is consistent with observations of Fort et al. (2019) that bootstrapping might hurt performance on marginal predictions. Also comparing ensemble-b and ensemble-P in Figure 3 and 4, we can see that difference between the agents increases as we have more models the ensemble.
agent
normalized nll
1
ensemble-N
ensemble-P
0.9
ensemble-BP (0.9)
ensemble-BP
0.8
(0.75)
ensemble-BP
(0.5)
marginal nll
joint nll
Figure 4: Performance of ensemble agents with 10 models across CIFAR10 problems. Bootstrapping doesn't. offer an advantage over well tuned ensemble-P in this setting.
In this section, we also compare ensemble agents on problem with non-uniform SNR across input space, similar to Figure in Section 5.2. We follow a similar procedure as Section 5.2 and flip labels of fraction of data. For this experiment, consider the full CIFAR10 training dataset and flip the labels of 25% of

---

Published in Transactions on Machine Learning Research (04/2023)
randomly picked images corresponding classes 10,1, and assign each of them to random uniformly sampled class from (5, 6, 7, 8,99 per image. This creates different SNRs across different classes. Figure 5 shows the results in such setting. All the ensemble agents use 10 models, results are averaged across random sceds, and the performance normalized w.r.t performance of ensemble-N. Best prior scale is chosen for ensemble-P and ensemble-BP by sweeping over {0, 1,3, 10). The performance of ensemble-P is very close that of ensemble-N, this might be due to prior functions, based on the randomly initialized convolution networks, being ineffective in this setting with heteroskedasticityy We conjecture that with an appropriate prior which reflects this class imbalance, ensemble-P could perform better than ensemble-N and ensemble-BP(0.77 offers clear advantage joint predictions, and
We can see that ensemble-BP(U ensemble-BP(0.9) offers an improvement even on marginal predictions. The results in Figures and 2 show that when SNR non-uniform across input space, ensemble agents could further benefit. from bootstrapping, even the ensemble agents already use prior functions.
1.05
agent
normalized nll
ensemble-N
1
ensemble-P
ensemble-BP
(0.9)
0.95
ensemble-BP
(0.75)
ensemble-BP
(0.5)
0.90
marginal nll
joint nll
Figure 5: Bootstrapping further improves ensemble agent's 's joint predictions.
6 Bandit experiments
One of our motivations for examining variants of ensemble agents is to understand how to train an ensemble agent that drives effective decision making. The results of Sections 4 and show that prior functions and bootstrapping can improve the quality of joint predictions of ensembles on fixed dataset problems. Theoretically, better joint predictions should enable a learning agent. to make better sequential decisions (Wen et al., 2021). We evaluate this empirically using Thompson sampling on heteroscedastic linear bandit problems. Note that this is relatively simple setting and mainly for illustrative purposes. We leave more exlensive experimental work neural bandits and deep reinforcement learning to future work.
The heteroscedastic linear bandit is a straightforward extension of the heteroscedastic linear regression problem considered in Section 4. First, we sample a set of actions X' = {21...., EN i..d. from a d d-dimensional standard normal distribution. We then sample a d-dimensional parameter vector 8" from a prior (0, T), and diagonal matrix Eobs whose entries are sampled i.i.d from Uniform(0, 1). The Dots controls the variance of the observation noise, and for an input X,, the observation noise is given by Wi+1 NN X,, Xo). This leads to different actions in X having difference observation noise variances and hence leading to heteroscedastic behaviour.
At each time I, the agent samples a model from the approximate posterior and acts greedily w.r.t the parameter obtained. The agent selects an action X, E X and observes a reward R Y = 0, X, W Let R2 = EE [R++1l,, X, æ] denote the expected reward of action x conditioned on the environment, and let X, E arggmax m R2 denote the optimal action. We assess agent performance through
regret(T) = Σ E [xx. Rx.],
which measures the shortfall in expected cumulative rewards relative an optimal decision maker. Note that the expectation in the regret definition is taken over all the random instances.
10

---

Published in Transactions on Machine Learning Research (04/2023)
In this section, we empirically examine the performance of Thompson sampling (TS) agents, that use variants of ensembles to approximate the posterior over the environment and drive decisions. At each timestep, a TS agent samples an imaginary enviromment from its approximate posterior and selects an action greedy to the sampled environment (Thompson, 1933; Russo et al., 2018). This simple hcuristic is known to effectively balance the exploration and exploitation trade-ofT, and allows us to evaluate the ability of an ensemble agent to drive decisions. The complete procodure for evaluating agents bandit problems provided Algorithm 1 in Appendix E.
Figure 6 compares the ensemble-N, ensemble-P and ensemble-BP dimensional fcatures, 4 actions N = 4. For all three agents, we consider an infinitely large ensemble, which is equivalent to resolving the optimization problem in (4) with new prior function and data pair weights at each time. This allows us to use the distributions derived in Appendix B.1 for different ensemble agents. We tune parameters of ensemble agents, as outlined Table 2, to minimize the final regret. Even in this simple problem, we can see that ensemble-N performs far worse than other two agents, ensemble-P performs much better than ensemble-N, but still worse than ensemble-BP. This clearly highlights the benefits of prior functions and bootstrapping for ensemble agents.
simple bandit problem with 2-
cumulative regret
1e4
1e3
agent ensemble-N
162
ensemble-P
ensemble-BP
le1
1
5000
10000 15000 20000 25000
t
Figure 6: Cumulative regret of ensemble agents on heteroscedastic bandit
7 Conclusion
We address a long-standing debate on the efficacy of prior functions and bootstrapping, in ensemble agent training, for uncertainty estimation and decision making. We show that prior functions can significantly enhance the joint predictions of ensemble agents. Moreover, we show that bootstrapping can further improve the performance when SNR varies substantially across the input space. We also show that, if the SNR is uniform across the input space and prior functions are appropriately tuned then bootstrapping offers significant benefit. We justify our claims using theoretical and experimental results on linear regression, classification, and bandit problems. We believe that some of our analysis for linear regression can also be extended to neural network regression via neural tangent kernel (NTK) type analysis (Jacot et al., 2018; Lee et al., 2019), but we defer this for future. We also believe that many of our claims extend to reinforcement learning problems. Through this work we hope to improve the community's understanding of how to train better ensemble agents for uncertainty estimation.
It is worth emphasizing that most results in this paper obtained under the assumption that the prior distribution is isometric. As we have discussed, this assumption is without loss of generality since in many problems it can be satisfied after appropriate data pre-processngg In more general settings, we can show that bootstrapping can be helpful in addition to prior functions if SNR varies significantly across the input space, or the covariance matrices of the prior and/or the input distribution are ill-conditioned.
Finally, though this paper has focused on ensemble agents, we believe that the obtained insights on the benefits of prior functions and bootstrapping also extend to other uncertainty estimation approaches, such
11

---

Published in Transactions on Machine Learning Research (04/2023)
hypermodels (Dwaracherla et al., 2020) and epinets (Osband et al., 2021). We believe this is interesting direction for future research.
Acknowledgments
We thank Yee Whye Teh and Brendan O'Donoghue for helpful discussions and feedback. We also thank John Maggs for organization and management of this rescarch effort.
References
Charles Blundell, Julien Cornebise, Koray Kavukeuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pp. 1613 11222. PMTR, 2015.
Glenn W Brier et al. Verification of forecasts expressed in terms of probability. Monthly wealher review, (1):133 1950.
Kamil Ciosek, Vincent Fortuin, Ryota Tomioka, Katja Hofmann, and Richard Turner. Conservative uncertainty estimation by fitting prior networks. In International Conference on Learning Representations, 2020.
Francesco D' D'Angelo and Vincent Fortuin. Repulsive deep ensembles are bayesian. Advances in Neural Information Processing Systems, 34:3451-3465, 2021.
Thomas G Dictterich. Ensemble methods in machine learning. In International workshop on multiple classifier systems, pp. 1 1-15. Springer, 2000.
Vikranth Dwaracherla and Benjamin Van Roy. Langevin DQN. arXiv preprint arXiv:2002.07285 2020.
Vikranth Dwaracherla, Xiuyuan Lu, Morteza Ibrahimi, lan Osband, Zheng Wen, and Benjamin Van Roy. Hypermodels for exploration. In International Conference on Learning Representations, 2020.
Bradley Efron and Robert J Tibshirani. An introduction to the bootstrup. CRC press, 1994.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: loss landscape perspective. arXiv preprint orXi::1912.022
2019.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, 2016.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. Tn International Conference on Machine Learning, pp. 1321 1330. PMLR, 2017.
Trevor Hastie, Robert Tibshirani, Jerome Friedman, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction, volume Springer, 2009.
Bobby He, Balaji Lakshminarayanan, and Yee Whye Teh. Bayesian deep ensembles via the neural tangent kernel. Advances in Neural Information Processing Systems, 33:1010-1022, 2020.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Siddhartha Jain, Ge Liu, Jonas Mueller, and David Gifford. Maximizing overall diversity for improved uncertainty estimates in deep ensembles. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4264-4271, 2020.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp. 6405-6416,
2017.
12

---

Published in Transactions on Machine Learning Research (04/2023)
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohi-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neurol information processing systems, 32, 2019.
Xiuyuan Lu and Benjamin Van Roy. Ensemble sampling. In Advances in Neurol Information Processing Systems, pp. 3260-3268, 2017.
Xiuyuan Lu, Ian Osband, Benjamin Van Roy, and Zheng Wen. Evaluating probabilistic inference in deep learning: Beyond marginal predictions. arXic preprint arXiv:2107.0922 2021.
Giung Nam, Jongmin Yoon, Yoonho Lee, and Juho Lee. Diversity matters when learning from ensembles. Advances in neural information processing systems, 34:8367-888 0377, 2021.
Jeremy Nixon, Balaji Lakshminarayanan,, and Dustin Tran. Why are bootstrapped deep ensembles not better? In "I Can't Believe It's Not Better!" NeurIPS 2020 workshop, 2020.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrappod DQN. In Advances In Neural Information Processing Systems 29, pp. 4026-40344 2016.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for doep reinforcement lcarning.
In Advances in Neural Information Processing Systems 31, pp. 8617 8629. Curran Associates, Inc., 2018.
Ian Osband, Zheng Wen, Mohammad Asghari, Morteza Tbrahimi, Xiyuan Lu, and Benjamin Van Roy. Epistemic neural networks. arXiv preprint arXiv:2107.0890 2021.
lan Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Botao Hao, Morteza Ibrahimi, Dicterích Lawson, Xinyuan Lu, Brendan O'Donoghue, and Benjamin Van Roy. The noural testbed: Evaluating predictive distributions, 2022a.
Tan Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Xiuyuan Lu, and Benjamin Van Roy. Evaluating high-order predictive distributions in deep learning, 2022b.
Chao Qin, Zheng Wen, Xiuyuan Lu, and Benjamin Van Roy. An analysis of ensemble sampling. arXiv preprint ar Xie:2200. 01303, 2022.
Alexandre Rame and Matthieu Cord, Dice: Diversity in deep ensembles via conditional redundancy adversarial estimation. arXiv preprint arXiv:2101. 05544,, 2021.
Daniel J. Russo, Benjamin Van Roy, Abbas Kazcrouni, Ian Osband, and Zheng Wen. A tutorial on thompson sampling. Found. Trends Mach. Learn., 11(1):1-96, July 2018. ISSN 1935-8237.
Changjian Shui, Azadch Sadat Mozafari, Jonathan Marek, lhsen Hedhli, and Christian Gagné. Diversity regularization in deep ensembles. arXie preprinl 0788, 2018
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arxiv preprint arXiv:1409. 1556, 2014.
R.S. Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3, 1988.
William R Thompson. On the likelihood that one unknown probability exceods another in view of the evidence of two samples. Biometrika, (5:35)) 294, 1933.
Chaoqi Wang, Shengyang Sun, and Roger Grossc. Beyond marginal uncertainty: How accurately can bayesian regression models estimate posterior predictive correlations? In International Conference Artificial PMLR, 2021.
Intelligence and Statistics, pp.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681-688. Citeseer, 2011.
Zheng Wen, Ian Osband, Chao Qin, Xiuyuan Lu, Morteza Tbrahimi, Vikranth Dwaracherla, Mohammad Asghari, and Benjamin Van Roy. From predictions to decisions: The importance of joint predictive distributions. Xix e-prints, pp. arXiv-2101 2107, 2021.
13

---

Published in Transactions on Machine Learning Research (04/2023)
Appendices
A Evaluation metrics
Most work on supervised learning has focused producing accurate marginal predictions for each input. However, some recent papers (Wen et al., 2021; Wang et al., 2021; Osband et al., 2022a;b) show that for a broad class of downstream tasks such as combinatorial decision problems and sequential decision problems, accurate marginal predictions are insufficient and accurate joinl predictions are essential. Wen el al. (2021); Osband et al. (2022a) also propose evaluate the predictive distributions based on expected KI. divergence. As defined in Section 3,
PI.t--- P(Y++1.4+ E -.10,,X XI:T+r11
is the ensemble agent's r-th order predictive distribution at inputs Xr and
Pir++-1 = P(Y+41T4 E (T:+Tr1)-1)
is the environment's 7-th order predictive distribution at the same inputs. Osband et al. (2022a) proposes to evaluate
dkL 15 [KL
(10)
where the expectation over all random variables, including environment E, the ensemble agent paramcters θr, inputs X7.T+1-1 and outcomes I:+1T Recall that XT. of the sequence of data pairs ((2,, Y(+1) = 0, 2,.......) In many supervised learning problems, they are sampled i.i.d. from input distribution Px. Also recall that when 1, dku, measures the agent's marginal predictive distribution; when > 1, it measures the agent's joint predictive distribution.
XT+I. Xr+r-1) (x+rr...Xx++. are part
A.1 Metric for Bayesian linear regression
We now discuss the relationship between dig and the evaluation metric (6) used in Section 4. We explain why the evaluation metric (6) is an equivalent metric of dkL, for ensemble agents with a large number of models, as r + co. The explanation here is informal, and we leave rigorous mathematical proof to future work. First, we also define
PT.T+-1 P (YT+IT++ E IDr, XT.T
In sense, PT.T represents the result of perfect inference. We define the expected KL divergence between PTete-I and Pr.T+T-1 as
dir. = dKL. (PT.TTT-I|AATTTTAA))
Note that from Proposition 1 of Lu et al. (2021),
dki. = Fx.. + I[E; Y1+I-T (rr+t11)
where T (E; Yr+IT++ (Dr, XT-T+T-1 is the conditional mutual information between & and Yr+1-T+, condi- tioned on Dy and Xy-T=--- Note that I (E;; YT+1T4 XT:Ttrr) does not depend on the agents, thus diu, and dgt. are equivalent for agent comparison.
Recall that for the Bayesian linear regression problem described in Section 4, Or includes M samples drawn i..d. from P (0m E - N (pr:Er) When M is sufficiently large, we have
P PPrrIT++ E | XT.T+-1) xe P (Vr+1tt7 E -|pr, Er, Xrstr-)).
On the other hand, recall the posterior over d., which indexes the environment E, is N ET). Hence, Pre+r-i can be rewritten
P PYTT41T++ lPT, Er, XT:
14

---

Published in Transactions on Machine Learning Research (04/2023)
Consequently, when M is sufficiently large, we have
dkL E dRL (Yr+IT++ =IuT, Er, XT.T+-I) i)
(IT+IT+r E e-Lir: Lt, Si, Xr.++r))]]
Finally, as is discussed in Section 2.5 of Wen et al. (2021), co, under suitable regularity conditions, we have
ki. E dKL (N(rT, Er) N(prr [ En))
=E =k| (PPO, E (Pr) ((0 E (Dr)))
(11)
In other words, as 7 + oo, diL is approximately equivalent to the metric in (6) for comparing ensemble agents with a large number of models.
A.2 Dyadic sampling
One limitation of dir.. is that the magnitude of required to provide additional insight beyond marginals can become intractably large, especially when the dimension of the input X, is high (Osband et al., 2022b). As a solution, Osband et al. (2022b) proposes to use dyadic sampling to evaluate joint predictive distributions. high level, dyadic sampling changes the joint distribution inputs XT.T+7-1: it first samples two random anchor points from the input space, and then resamples inputs from these two anchor points. The expected KL-divergence under dyadic sampling is referred to dkL
Similar to Osband et al. (2022b), we use dyadic sampling with 7 10 to evaluate ensemble agents' joint predictions in Section 5. Note that dkt with = 10 can be efficiently computed based on Monte-Carlo simulation (see Algorithm 1 of Osband et al. (2022b))-
A.3 Expected KL-divergence and negative log- logelikesssods
As discussed in recent papers (Wen et al., 2021; Osband et al., 2022b), d'L and its variants (e.g. d are equivalent to variants of negative log- lygliiklihood (nll), which is also known as the cross-entropy loss, for agent comparison. In particular, note that the r-th order cross-entropy loss is
de = -E[1og (0
(12)
and it is straightforward to show that
diL dée + Ellog PT:Ttr-(Y1+120
Since Pi does not depend the agent, d&L and deg rank agents identically.
For experiments on the CIFAR10 dataset, since (Y the likelihood under the true environ- ment E, is unknown, we choose to use 7-th order cross-cntropy loss or negative log- ligliiidd (nll) to compare agents.
B Bayesian linear regression
B.1 Perturbed loss function and solution
As is described in Section 4, the perturbed loss function is:
8m E arg min
Σ ((x)) (0' (Ye + Z m.
2 |10 -mm
A + 2
where Ze+1. additive perturbation for data pair (X,, Y441) sampled from N (0.))(X))): v, 0²: अ2 R+ are respectively the weight function and the bootstrapping variance function; À > 0 is the regularization
15

---

Published in Transactions on Machine Learning Research (04/2023)
coefficient; and 8n is independently sampled from N (0), The magnitude of og reflects the agent's prior uncertainly, and hence do is known as the agent's prior scale. Note that ||0 dm ll2 is randomly perturbed regularizer, and is equivalent to prior function for linear regression (Osband et al., 2018). It is straightforward to show that
6m = (Eto v(X))xxX7 + -1 - v(Xi) (Y4+ Ze+1,m) X2 + +10m)
Note that 0m DT N (ir: ) where
pr Σ (XX))xXX+ +
Σ CuXo)anX)))
( EUXXXXIX tNL
Σr
XXX)xx+++M +
Σ (x))) +
B.2 Proofs for Proposition 1 and 2
Proof for Proposition 1
Proof Based on the analytical formula of for and ΣT provided in Appendix 3.1,ife(() = ((()){)) 1/0²(),
o où and À 1/00, have for pr and Σx Er. Thus, P(0m E | (r) PP =P
E E
Proof for Proposition 2
Proof Note that when (-) 0 and og 0, we have
On Σ (Xx)xxxx +11)-1 (Eto v(X.)
which is deterministic conditioned on Dr-
B.3 Proof for Theorem 1
Proof Notice that to ensure that for pr, need v(-) c/o2(.) and x c/aß for constant 0.
Without loss of generality, we choose = 1. Under this choice and o (²( O, we have:
1/1) -1 Σx = Σ (Exx)) X,XT + og
T-1
T-1
-1
(0b/001)
E
Σ 1
X,X7
+ -
1 (x)))
0 2 = m>
(13)
where we define the shorthand notation = 00/00 to simplify the exposition. Then following the KL- divergence formula for multivariate Gaussian distributions, we have
dRL (p.0. (Dr) P (0m .(Dr)) dRL (N Er) || N (per, 1/22..))
1
1
dlog + log det (Er) d + tr
(14)
2
17
16

---

Published in Transactions on Machine Learning Research (04/2023)
Taking an expectation over Dr, have
E dRL (P. -IDr) p (0m E -|Dr))
d llog + det (Ei) d + -tr
1
[2
1/
1
IE
d - log (ETT) - d +
-
Ir [TT]]
11
1
dlog EE [log det (2771) )]
d
+
E
tr
"
(M)
1
2
d'log log det (E [Σ I] +
[
(15)
"
where (a) follows from log det (Er) = log det (Irr") and (b) follows from E [ir [Σ>
tr
and
log det (Σ7') ) - log det (E [r
since log det(-) is a concave function. Note that
x
1 Σ
(x,)
1=
hence
E
1+
Let Ci, Sa denote the eigenvalues of E
Σ
consequently have
1 T%
4
of
where 't' are eigenvalues of r E eigenvalues, we have
xx (x,)
By rewriting log det (E [>= and tr [ [2:7]] using
']]
d
d
deL E-Dr) P (0m E <-Ip:))] 22 d dlog 1j Σ
log C. - d
Σ
By minimizing the right-hand side of the above inequality over 7,, which is equivalent to minimizing over the prior sample variance oz. we obtain the minimizing 7
d 1=1
Plug in the minimizing Th, we have
E dKL (P(0.E E
(Pr) E || (pm E e.(pr))
d log
1 Σ
Ci
log Co
1=1
i=1
Rewrite the above equation using the eigenvalues of Ex, have
E dkL. (P(0. --rr) P (0m E (
1 1+77
1 + Tri
d log
log
ob
of
1+ 1+T7
In
(16)
1 T76
where 7 = -22,
Edm 7c- This concludes the proof.
17

---

Published in Transactions on Machine Learning Research (04/2023)
B.4 When bootstrapping is not strictly necessary
We would like to clarify that bootstrapping is not always strictly necessary for the considered Bayesian linear regression problems, and there are cases in which ensemble-P can work well. The following is one example: assume that X₃ ss are independently drawn from (N0, 7), and the linear regression problem is homoscedastic in the sense that (X,, = o² is a constant. In this case, if ensemble-P chooses (Xxe) = 1/o² for all X, and 1/00; then we have for = PTT,
T-1 Σx = [T/6 + Σ /02 and ΣT = = 6/522/05
(see Appendix B.1). Also note that for sufficiently large T, + ETT X, ~ I due to Law of large numbers. ag00 (o2+To I. In other words, if we choose
I and En Hence, we have Ey o²4To]
agaa
a (1 + To)
then P(0m E (DT) approximately matches P P0. E (DD) under ensemble-P. Notice that this choice of og depends on the training data size T. Also note that for this case, the lower bound in Theorem zero since r T= o=
Another example is high SNR scenario where (X,,) 0 for all input X,, From Proposition 1, the optimal ensemble-Bi agent will choose o² (X,, (X,, x 0 for all X₃. Consequently, under mild technical conditions, an ensemble-P agent can be near-optimal.
C Neural Testbed experiments
Neural Testbed (Osband et al., 2022a) is open-source test suite designed to evaluate the predictive distributions of deep learning algorithms on randomly generated classification problems, based on a neural network generative model. Problem instances are binary classification problems which are sampled from a random 2-layer MLP, with 50 hidden units, across different input dimensions d E {2, 10, 100}, training set sizes T dr with E {1, 10, 100, 1000), and output temperatures (0.01, 0.1, 0.5) which correspond to high, medium, and low SNR. By default, results are averaged over 10 random seeds in each setting, and confidence bars are also reported. We evaluate the agents based their marginal predictions and joint predictions discussed in Appendix A.
C.1 Agent description
The mlp agent uses single point estimate that has the same network architecture the generative model, which is a 2-hidden layer MIP with 50 units in each hidden layer. The ensemble-N agent uses 100 models with each model being 2-layer MLP that only differ in initialization. The ensemble-P agent also uses 100 models with each model being a 2-layer MLP with an additive prior function which is also a random 2-layer MLP (Osband et al., 2018). In specific, m-th model of the ensemble-P model takes the form
Sem. (x) = go... (x:) + Pm (x),
where 90.. Pm (x): X' - R1 are the trainable network and randomly initialized prior function which remains fixed through out the experiment. (m) differs across m only through initialization. The ensemble-BP also uses 100 models that are similar to models of ensemble-P, The main difference between the ensemble-BP and ensemble-P is the loss function. In Equation (9), ensemble-M and ensemble-P chooses constant weights WL+1. = 1, Vm, t, while ensemble-BP used in Neural Testbed experiments samples W111. i.i.d from (ernoulli(0.5).
We modify the code from emn library at https://github com/ http////eemdd//eeeeeee and Neural Testbed library at https::/sse t/git:b com/ hem/ddepmindd/nee wewrralteetteeaaeeeeeeeeeee model our agents and run experiments on Neural Testbed. We run our experiments using 8-core CPU and GB ram instances Google cloud compute. Tn
18

---

Published in Transactions on Machine Learning Research (04/2023)
all the figures, normalized the scores of all agents in each setting with the score of an uniform random agent, which picks each class 0 and 1, with probability of 0.5. This helps us to ensure that problems across different temperature and input dimensions contribute roughly equally to the final score.
For the ensemble agents used for Neural Testbed experiments, we find the best hyperparamters by sweeping over set of hyperparameters. We use similar sweeps as mentioned in Oshand al. (2022b). For mlp and ensemble-N agents, for problem with input dimension D and temperature p, we choose the weight decay term (A in Equation 9) from À E {0.1, 0.3, 1, 10} x d/ For ensemble-P agent, in addition to sweeping over the weight decay term, also sweep the prior scale of the additive prior functions. In specific, for problem with temperature p, we choose values from {0.3/ Vp. 0.3/ 3/
C.2 Additional results
ensemble-P's sensitivity to prior scale:
Figure 7 shows the sensitivity of performance of ensemble-P agent with respect to different prior scales, In particular we choose input dimension 100 and temperature of 0.01. But we observe similar trend across other settings as well. We observe that the performance of ensemble-P agent does depend the prior scale, but the performance is not too sensitive as long prior scale is roughly correct. The dotted line denote the performance of uniform random agent which picks both classes with equal probability. We can see that, especially when the training dataset size is small, the ensemble-N agent (prior scale is 0) performs close to an uniform random agent.
kl estimate (joint)
training dataset size (T)
100
300
1000
0.01
1.0 prior scale
10.0 30.0 100.0
Figure 7: We see that performance does depend on the prior scale,but the performance is not very sensitive to prior scale as long as it is in the correct range. The dotted line indicates the performance of a uniform random agent.
Performance of ensemble-N on varying SNR testbed problems
Figure& is an extension of Figure with the performance of ensemble-II included on the varying SNR problems derived by flipping labels of Neural Testbed problems. We can see that all three ensemble agents perform very similarly marginal predictions. However, when evaluated by joint predictions, ensemble-P performs significantly better than ensemble-N and, with bootstrapping, ensemble-BP improves over ensemble-P.
Figure 9 shows the breakdown of performance of the agents in Figure 8 for input dimension 10. The main improvement of ensemble-BP over ensemble-P arises when the training dataset size T is not too small. When is close to O, prior plays the most prominent role, and after a point, as increases the posterior distribution concentrates and the benefits of bootstrapping start to diminish.
D Cifar10 experiments
CIFAR10 has been widely known in the deep learning community as benchmark dataset and used evaluate classification algorithms. Each data pair in CIFAR10 dataset consists of an image and a label corresponding
19

---

Published in Transactions on Machine Learning Research (04/2023)
marginal kl estimate
0.55
joint kl estimate
0.45
0.50
0.40
0.35
0.30
0.40
0.25
ensembleN
ensembleP agent
ensembleBP
ensembleN
ensembleP agent
ensembleBP
The ensemble agents are distinguished on joint predictions. ensemble-P improves over the performance of ensemble-N agent which is further improved by ensemble-BP.
Figure
marginal estimate
0.7
joint kl estimate
agent ensemble-N
0.5
0.5
ensemble-P
0.3
ensemble-BP
0.3
0.2
10
100
1000
10000
10
100
1000
10000
training set size
training set size
Figure 9: Breakdown of bootstrap results on problems with varying SNR
to of 10 classes which we refer to as {0, 1,,,, ... 9}. We look at different problems constructed from the dataset, by limiting the training set size. In particular, we consider training sets of sizes 10, 100, 1000, and 50000, where 50000 is the size of the full training dataset.
For Figure 5, we flip the labels of 25% of images corresponding to first classes, 10, and assign each of them to a random uniformly sampled class from the other classes, {5, 6, 7, 8, 9} per image. This creates different SNRs across different classes. For this experiment, we consider a single setting, where the agents are trained with 50, 000 training samples (full training data).
D.1 Agent description
For the experiments on CIFAR10 dataset, we use vgg model (Simonyan & Zisserman, 2014) as the single point estimate model. An ensemble-h agent consists of an ensemble of vgg models and ensemble-2 agent consists an ensemble of models with each model being a vgg model with output a small randomly initialized convolution network added to the logits. Each model of ensemble-BP same that of ensemble e-P,, but uses slightly different loss function. In Equation (9), ensemble-I and ensemble-P uses constant weights W 1, Vm, 4, while ensemble-BP variants of ensemble-BP based on the mean probability of the Bernoulli distribution. We use Insemble-BP(1 to denote the ensemble-Be agent with weights WELL. sampled i.i.d from Bernoull(p)/p.
1 sampled i.i.d from Bernoulli distribution. We consider few
The single point estimate agent vgg uses VGG model (Simonyan & Zisserman, 2014). We make use www.///pmmdd///eeeeeeeeeeeeeeeee enn. The VGG model consists of blocks whose output finally passod through an average pool layer and a dense layer to get the logits. Each of the 11 bolcks consists of a convolution layer, batchnorm layer, and relu layer. The 11 blocks have convolution layers with kernel width of (3,3) with number of channels from 1st to 11th
of the VGG model implementation at https://gittn
20

---

Published in Transactions on Machine Learning Research (04/2023)
block (64, 64, 128, 128, 128, 256, 256, 256, 512, 512, 512) and stride length as 1 The ensemble-N agent uses an ensemble of multiple VGG models. The ensemble-P agent uses an ensemble of multiple models, with each model consisting of a VGG model combined with a randomly initialized convolution network at the logits. The convolution prior network consists 3 convolution layers with kernel of size (5, 5) and output channels as 4, 8, 4 in first, second, and third layers respectively. Finally the output of third layer is passed through a dense layer to get logits which are added the logits of VGG model. The ensemble-BP agent uses the same network as ensemble-P.
512)
For experiments CIFAR10, consider 4 different problems, based the training dataset size. Once trained, all the agents in all problems are evaluated on the same (full) test datasct. We choose the weight decay of (traiiiig dataset size) for all the agents. This value was obtained by sweeping weight decay for ensemble-N with 10 VGG models. For ensemble-P agent in Figure 3 and 10, we choose a prior scale of 30. In the experiments for Figure 11, both ensemble-P and ensemble-B3 use a prior scale 3. For all ensemble agents, we train the the models of ensemble separately and combine them during evaluation. Each model is trained on 2x2 TPU with per-device batch size of 32. Each model is trained for 400 epochs. For training, we use an SGD optimizer with learning rate schedule with initial learning rate as {0.0001, 0.001, 0.01, 0.025] for training dataset sizes (10, 100, 1000, 50000) and the learning rate is reducod to one-tenth after 200 epochs, mnehhndredth after 300 epochs, and one-thousandth after 350 epochs.
D.2 Additional results
Figure 3 compares the average performance of vgg, ensemble-N, and ensemble-P across CIFAR10 problems of different training dataset sizes. Error bars are obtained by evaluating the performance of agents across 5 seeds. For each seed, we compute the performance of ensemble-N and ensemble-P agents, by sampling 100 particles with repetition from a set of 100 trained particles of the corresponding agent. This was done avoid training a huge number of particles. Figure 10 shows more detailed breakdown of performance across CIFAR10 problems of different training dataset sizes. We see that marginal predictions, both ensemble-tt and ensemble-P perform similarly, while they are clearly distinguished on joint predictions. We also see that the benefit of the prior functions is higher when the training dataset size is lower. This is what we expect as the prior information matters the most when we have low data.
3
30
marginal nll
agent vgg ensemble-N
joint nll
2-
20
10
ensemble-P
0
10
100
1000
50000
10
100
1000
50000
training dataset size
training dataset size
Figure 10: Performance of ensemble agents with 100 models across CIFAR10 problems of different training dataset sizes.
Figure 11 shows the average performance of ensemble agents CIFAR10 problems across different ensemble sizes. We consider slight variation of the above problem, where 1% of the images have their labels assigned to uniformly random values in 21, 10). This ensures that the problem is not extremely high SNR. We use a fixed prior scale 3 3 for ensemble-P and ensemble-BP on all the problems. Note that this might not be the best prior scale for ensemble-P and ensemble-BP agents; however, this helps in gaining insights into bootstrapping methods. The results are normalized w.r.t performance of ensemble-N. We see that for small ensemble sizes, on marginal predictions, bootstrapping might actually hurt the performance as suggested in Fort et al. (2019). However, as increase the ensemble size, even marginal predictions, bootstrapping
21

---

Published in Transactions on Machine Learning Research (04/2023)
starts to help. On joint predictions, which are actually indicative of downstream tasks (Wen et al., 2021), bootstrapping offers a clear benefit.
normalized marginal nil
normalized joint nill
1.05
agent
normalized
ensemble-N
0.95
ensemble-P
ensemble-BP
(0.9)
ensemble-BP
(0.5)
0.85
10
30 ensemble size
10
Figure 11: Performance of ensemble agents across different ensemble sizes. Bootstrapping helps on larger ensembles when evaluated on joint predictions.
E
Bandit experiments
Section 6 demonstrates the performances of variants of ensemble agents, including ensemble-N, ensemble-P. and ensemble-BP, on randomly generated bandit problems. In particular, all the ensemble agents use Thompson sampling (TS) as the exploration scheme. Algorithum provides the pseudo-code for how we evaluate ensemble agents on bandit problems. In particular, describes how random bandit problems are sampled and how the ensemble agents adaptively choose actions.
Figure 12 is an extension of Figure 6 with additional agent that is variation of ensemble-P which uses different weights for data pairs based on the observation noise variance at that data pair. Typically, when ensemble-N or ensemble-P agents are used, they choose a fixed weight across all data pairs. We can see that using weights based on observation noise variance improves performance to some extent, but ensemble-BP still performs the best. This further justifies the benefits of bootstrapping.
cumulative regret
1e4
agent
1e3
ensemble-N
ensemble-P
1e2
ensemble-P with scaled weights ensemble-BP
le1
1
0
5000
10000
15000 20000 25000
t
Figure 12: Heteroscadastie linear bandit with actions
22

---

Published in Transactions on Machine Learning Research (04/2023)
Algorithm Evaluation of ensemble agents on bandit problems
Require: the following inputs are required
1. distribution over the environment P(E E +), action distribution Px, and the number of actions
2. ensemble agent type (e.g. ensemble-N, ensemble-P, ensemble-BP)
3. number of time steps
4. number of sampled problems J
for j 11,2....,, do
Step 1: sample bandit problem
1. sample environment & P(E
2. sample a set X' of N actions æ1, , i.i.d. from Px
3. obtain the mean rewards corresponding to actions in X'
F = E(Y = æ],, Vr E X
4. compute the optimal expected reward R, m R.z
Step 2: run the ensemble agent
Initialize the data buffer Do = -)
for t 1.2,..., do
1. Update agent parameters O based on Dz-1
2. action selection based on TS:
i. sample model & from the agent belief distribution
& (~P(GEE(0))
ii act greedily based on &
X₂ arg max E = E,, X. x] aEX
iii. generate observation Y41 based on action Xt
Y41 P (Yi+1 E 18, Xi)
3. update the buller D, +- append (D,-1, (X,, Y++)) end for
compute the total regret incurred in T time steps
Regret, (T) ΣT RR Rx.)
end for return average total regret 1 Σ Regret,(1) (T)
23


Reviews:
[{"review_id": "M5unTTZ8nw", "reviewer": "Reviewer_DqPm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "While the paper attempts to answer questions that would be of interest to those working with uncertainty estimation in ML, I am ultimately not convinced that the claims of the paper (benefits of prior functions and bootstrapping) are adequately supported by the evidence. Theoretical analysis is limited to a toy linear regression problem, a setting where the baseline ensemble method \"ensemble-N\" trivially offers no uncertainty estimation benefits and is thus unlikely to be representative of machine learning/deep learning settings where such ensembling would be used. Empirical results consist of a setting with a small 2 layer MLP in NeuralTestbed, CIFAR10, and a linear bandit, which are all fairly toy tasks that are not necessarily meaningfully representative of realistic tasks in machine learning. \n\nSpecifically in comparisons between the baseline ensemble-N and prior functions (ensemble-P), I believe there is limited value in showing that prior functions leads to improved joint uncertainty esitmation in these toy tasks when the Osband 2018 [https://arxiv.org/pdf/1806.03335.pdf] which introduced randomized prior functions already demonstrate improved exploration over regular ensembles when applied in deep RL settings. I believe there would need to be a much more extensive empirical evaluation (larger scale supervised learning tasks, more complex reinforcement learning tasks than the low-dimensional linear bandit considered) for the results to be of interest. Additionally, it is interesting that in Fig 5 where non-uniform label noise is added, ensemble-P performs the same as ensemble-N in joint NLL (despite outperforming it in the clean dataset), a regression that should be investigated further.\n\nFor empirical results showing bootstrapping (ensemble-BP) improves when there is non-uniform SNR across inputs, my concern is that in all settings considered, there is explicit label noise being added to induce the non-uniform SNR. This synthetic non-uniformity may not be representative of realistic settings, and there should be a experiments validating this hypothesis in realistic tasks instead. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission analyzes uncertainty estimation when using prior functions and bootstrapping with ensemble methods, aiming to show when these two ingredients improve over \"vanilla\" ensembles. They analyze a toy Bayesian linear regression example to show theoretical benefits of these two approaches, in particular benefits of bootstrapping in situations with different SNR across inputs.\nThey show improved empirical results in joint uncertainty estimation with small MLPs in the Neural Testbed setting as well as in image classification on CIFAR10, in particular illustrating the benefits of bootstrapping when artificially adding targeted label noise to simulate different SNRs across inputs. \nFinally, they show improved regret when evaluating these techniques in a toy bandit problem (linear bandit with heteroskedatic noise) when using ensembles for Thompson sampling.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See strengths/weakensses section for requested changes around more extensive empirical validation. \n\nMinor Typos:\n \nEnd of page 1: \" In particular, accurate joint predictions are crucial for efficient exploration, adaptation and effective decision\" effective decision -> effective decision making\n\nEnd of section 1: \"Then we justify the main points of this paper, as discussed above, via analaysis on Bayesian linear regression (Section 4),\" analaysis -> analysis\n\n2nd paragraph of sec 5: \"This might be bit contrived as loss function is not convex in many deep learning problems.\" \nbe bit -> be a bit, add \"the\" before \"loss function\". ", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1Pi8VTKYEs", "reviewer": "Reviewer_Vuf7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts:\n- The paper sheds some light on the role of different ensemble techniques, which, from the literature review by the authors, seems to be a matter of open discussion in the community. So, the results should be of interest to researchers and practitioners using ensemble methods in supervised or reinforcement learning.\n- The claims are supported both by theory and experiments, and both appear to be sound and convincing.\n- The paper is generally well written and clear.\n\nWeaknesses:\n1. The different techniques are considered in an incremental fashion: bootstrapping on top of random priors on top of random initialization. This is not fully motivated, and leaves some questions open, mostly what would be the effect of using bootstrapping without random priors. This case is considered neither in the theory nor in the experiments.\n2. Only a randomly-generated linear bandit task is considered. Generating linear bandits in this way can lead to fundamentally easier problems than one would encounter in practice, as observed for instance in \"Leveraging Good Representations in Linear Contextual Bandits\" (Papini et al., 2021, Lemma 8). This is still an interesting and convincing illustrative example, but disappointingly limited when compared with the supervised learning experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper discusses techniques to train ensemble models for uncertainty estimation in supervised and bandit learning. \nThree existing techniques are considered: random initialization, random priors, and bootstrapping. The aim of the paper is to clarify the effectiveness of these methods when used alone or combined.\nThe two main claims of the paper are:\n\nClaim 1: prior functions significantly improve joint predictions w.r.t. just using random initialization\n\nClaim 2: bootstrapping is only useful (on top of the other techniques) when the input noise is significantly non-isotropic \n\nThe claims are substantiated both by theory and experiments. The theoretical results are in the special case of bayesian linear regression:\n- Random initialization alone cannot represent any uncertainty, suggesting that random priors are necessary (Claim 1)\n- Random priors + bootstrapping are sufficient to reconstruct the posterior distribution (Claim 1 and 2)\n- When the input noise is significantly non-isotropic, random priors without bootstrapping do not allow to reconstruct the posterior distribution, suggesting that bootstrapping is necessary in general (Claim 2)\n\nExperiments are conducted with ensembles of neural networks. In classification problems (Neural Testbed, CIFAR 10) marginal predictions are evaluated using the KL divergence and joint predictions are evaluated using the dyadic KL first proposed by Osband et al. (2022). Results show that indeed random priors improve predictions, especially when considering joint predictions and hyperparameter tuning is limited, and that bootstrapping further helps when the input is highly non-isotropic. The bandit experiment runs a version of Thompson sampling based on ensembles on an artificially generated task. Here significant improvement in the cumulated regret is shown by using random priors, and further improvement by using also bootstrapping.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Addressing weakness 1 is a critical adjustment. This can be done by considering the case of bootstrapping without random priors in the experiments or by discussing why this is a bad idea from a theoretical standpoint.\n\nThe following adjustments, instead, are just suggestions on how to improve the manuscript:\n- To address weakness 2, you may test the different Thompson sampling variants on more realistic tasks like the ones considered in \"A Contextual Bandit Bake-off\" by Bietti et al. (2021), which are basically online-learning versions of classification problems and so should fit quite well within your framework\n- I would elaborate more, in the introduction, on why uncertainty estimation is important and why ensemble methods are one of the most used approaches, and what are the pros and cons of other approaches\n- I think evaluation metrics are a very important part of this work, and I suggest to give them more space in the main paper. For instance, I expect the concept of dyadic sampling to be new to most readers. Moving some of the content from appendix A to the main paper would be enough.\n- The final expression for $\\gamma_i$ on page 5 is confusing, since the argument of the conditional variance, $(X_t^\\top v_i)(\\theta_*^\\top v_i)$, seems to be deterministic conditional on $X_t$\n- Sentences that were unclear to me:  \"the agent's imaginary environment\" on page 3, \"this might be bit contrived\" on page 6\n\nFinally, a question: is \"uneven SNR\" something that could be addressed with some sort of pre-processing of the data? Would this make bootstrapping useless (at least in the supervised learning setting)?", "broader_impact_concerns": "I don't think a broader impact statement is necessary for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "avy2xQZsSM", "reviewer": "Reviewer_NvZp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Ensembles are a popular model and this paper contributes to understanding them better\n- The theoretical derivations seem to be correct\n- The paper is well-written and the main ideas are clearly presented\n\nWeaknesses:\n- Some related works seem to be missing from the discussion\n- The experiments could be improved towards more realistic problems\n\nMajor comments:\n- Theoretical properties of ensembles with prior functions have previously been discussed in [1], so this should be mentioned\n- There is a long line of research dealing with the topic of diversity in deep ensembles [e.g., 2-7], some of which also include certain theoretical convergence guarantees [e.g., 6]. Since the prior functions and bootstrapping studied in this paper also seem to mainly act by increasing diversity between the ensemble members, I think it should be discussed how this compares to these other proposed approaches to increase diversity. An experimental comparison would be great, but at least a theoretical discussion of the tradeoffs between these different methods seems warranted.\n- I appreciate the experiments on CIFAR10, but since this paper deals with uncertainty estimation, it would be good to see some standard evaluations of the predictive uncertainties, such as uncertainty calibration on CIFAR10-C and OOD detection on SVHN/CIFAR-100, similar to, e.g., [8]. The current experiments are great and I appreciate the joint probability evaluation, but they are hard to compare with the existing literature.\n- Given the theoretical results for linear regression, the bandit experiment is a nice confirmation of those, but the insights are not very realistic. It would be great to see a bandit experiment on a real problem, such as the ones in [9].\n\nMinor comments:\n- I feel like Props. 1 and 2 (or at least Prop. 1) could come a bit earlier in the text to improve the flow, since they are referenced one page before they appear.\n- Why are there no error bars in Fig. 3?\n- In Figs. 4 and 5, the y-axis is labeled \"value\", which could maybe rather be replaced by \"nll\" or similar and then the x-ticks could say \"marginal\" and \"joint\"\n- The experiment environment is variously referred to as \"neural testbed\", \"Neural Testbed\", and \"the Neural Testbed\", which should be harmonized.\n- Sec. 5: bit contrived -> a bit contrived\n- Sec. 5: as loss -> as the loss\n- Sec. 5.1: with same -> with the same\n- Sec. 5.1: as generative -> as the generative\n\n[1] https://openreview.net/forum?id=BJlahxHYDS\n\n[2] https://arxiv.org/abs/1802.07881\n\n[3] https://arxiv.org/abs/1912.02757\n\n[4] https://ojs.aaai.org/index.php/AAAI/article/view/5849\n\n[5] https://proceedings.neurips.cc/paper/2021/hash/466473650870501e3600d9a1b4ee5d44-Abstract.html\n\n[6] https://proceedings.neurips.cc/paper/2021/hash/1c63926ebcabda26b5cdb31b5cc91efb-Abstract.html\n\n[7] https://arxiv.org/abs/2101.05544\n\n[8] https://arxiv.org/abs/2106.04015\n\n[9] https://arxiv.org/abs/1802.09127", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors investigate the influence of the use of random prior functions and bootstrapping on the performance of ensemble models. They show theoretically in the case of linear regression that ensemble models with prior functions and bootstrapping asymptotically converge to the true posterior, while ensembles without these do not necessarily and in fact remain bounded away from the true posterior under mild assumptions. They then show empirically that these benefits of prior functions and bootstrapping translate to performance gains on several real datasets, although not on all of them.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My main requested changes would be an improved discussion of the relevant related work and additional uncertainty evaluations on CIFAR10. Nice-to-have, but not 100% required, would be more realistic bandit experiments and one or two alternative diversity-inducing ensemble methods as baselines.", "broader_impact_concerns": "I have no ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has received mixed reviews, with two reviewers leaning towards suggesting acceptance, and one reviewer arguing that the results of the paper are probably not interesting enough to warrant publication. 

Reviewers Vuf7 and NvZp were more positive, although they have pointed out that the empirical analysis is not entirely satisfying. Both of them took issue with the limited bandit experiments, but after discussion they seem to be OK with this part being included in the paper (as long as the authors clearly indicate that they are intended as illustration of how the implications of this work extend to sequential decision making). Several other references were pointed out by Reviewer NvZp, and the authors promised to cover these in an updated version of the paper.

Reviewer DqPm was less positive about the paper, focusing their criticism on the relatively small scale of the experiments, and the Bayesian linear regression setting studied in the theoretical half of the paper. To an extent I see where this criticism comes from, but in my view, the results in this paper clearly show the advantage of bootstrapping for uncertainty estimation (as a comparison of the lower bound of Thm 1 with the positive result of Prop 1 reveals). Clearly, this is a lesson that holds more generally beyond the setting studied in the paper, the implication being that some existing uncertainty quantification methods may not be sufficiently good for estimating posterior distributions.

Based on my reading of the paper and the reviews, I think that this is a good paper that makes a clear point and illustrates it nicely with some experiments. Sure enough, the empirical results are not very large-scale but they do drive the point home cleanly, which would have been perhaps difficult to do with larger experiments. 

Taking all of the above points into account, my conclusion is that the paper can be appropriate for publication at TMRL, given that the authors take the criticism raised in the reviews seriously, and actually implement the changes they promised in their responses.
Decision: Accept (poster)