Hypermodels for Exploration
Authors: Vikranth Dwaracherla
Xiuyuan Lu
Morteza Ibrahimi
Ian Osband
Zheng Wen
Benjamin Van Roy
Venue:  (2020)
Keywords: exploration
hypermodel
reinforcement learning

Abstract: We study the use of hypermodels to represent epistemic uncertainty and guide exploration.
This generalizes and extends the use of ensembles to approximate Thompson sampling. The computational cost of training an ensemble grows with its size, and as such, prior work has typically been limited to ensembles with tens of elements. We show that alternative hypermodels can enjoy dramatic efficiency gains, enabling behavior that would otherwise require hundreds or thousands of elements, and even succeed in situations where ensemble methods fail to learn regardless of size.
This allows more accurate approximation of Thompson sampling as well as use of more sophisticated exploration schemes.  In particular, we consider an approximate form of information-directed sampling and demonstrate performance gains relative to Thompson sampling.  As alternatives to ensembles, we consider linear and neural network hypermodels, also known as hypernetworks.
We prove that, with neural network base models, a linear hypermodel can represent essentially any distribution over functions, and as such, hypernetworks do not extend what can be represented.

Published as a conference paper at ICLR 2020
HYPERMODELS FOR EXPLORATION
Vikranth Dwaracherla, Xiuyuan Lu, Morteza Ibrahimi, lan Osband, Zheng Wen, Benjamin Van Roy
ABSTRACT
We study the use of hypermodels to represent epistemic uncertainty and guide exploration. This generalizes and extends the of ensembles approximate Thompson sampling. The computational cost of training an ensemble grows with its size, and such, prior work has typically been limited to ensembles with tens of elements. We show that alternative hypermodels can enjoy dramatic efficiency gains, enabling behavior that would otherwise require hundreds or thousands of elements, and even succeed in situations where ensemble methods fail to learn re- gardless of size. This allows more accurate approximation Thompson sampling as well as use of more sophisticated exploration schemes. In particular, we con- sider an approximate form of information-directte sampling and demonstrate per- formance gains relative to Thompson sampling. As alternatives to ensembles, we consider linear and neural network hypermodels, also known as hypernetworks. We prove that, with neural network base models, linear hypermodel can repre- sent essentially any distribution over functions, and as such, hypernetworks are no more expressive.
INTRODUCTION
Consider the sequential decision problem of an agent interacting with an uncertain environment, aiming to maximize cumulative rewards. Over each time period, the agent must balance between exploiting existing knowledge to accrue immediate reward and investing in exploratory behavior that may increase subsequent rewards. In order to select informative exploratory actions, the agent must have some understanding of what it is uncertain about. As such, an ability to represent and resolve epistemic uncertainty is core capability required of the intelligent agents.
The efficient representation of epistemic uncertainty when estimating complex models like neural networks presents important research challenge. Techniques include variational inference (Blun- dell et al. 2015), dropout"] (Gal & Ghahramani 2016) and MCMC (Andrieu et al. 2003). Another approach has been motivated by the nonparametrie bootstrap (Efron Tibshirani 1994) and trains an ensemble of neural networks with random perturbations applied to each dataset (Lu & Van Roy, (017]. The spirit is akin to particle filtering, where each element of the ensemble approximates sample from the posterior and variation between models reflects epistemic uncertainty. Ensem- bles have proved to relatively effective and to address some shortcomings of alternative posterior approximation schemes (Osband al. 2016 2018).
When training a single large neural network computationally intensive, training a large ensem- ble of separate models can be prohibitively expensive. As such, ensembles in deep learning have typically been limited to tens of models (Riquelme et al., 2018).. In this paper, we show that this parsimony severely limit the quality of the posterior approximation and ultimately the quality of the learning system. Further, we consider more general approach based on hypermodels that can realize the benefits of large ensembles without the prohibitive computational requirements.
hypermodel maps an index drawn from a reference distribution to a base model. An ensemble is type of hypermodel; it maps uniformly sampled base model index to that independently trained base model. We will consider additional hypermodel classes, including linear hypermodels, which will use map Gaitsien index base model parameters, and hypernetworks, for which the mapping is neural network (Ha et al. 2016 Our motivation is that intelligent
'DeepMind
Atthough later work suggests that this dropout approximation can be of poor quality Oshand 2016 Hron
2017).

---

Published as a conference paper at ICLR 2020
hypermodel design might be able amortize computation across the entire distribution of base models, and in doing so, offer large gains in computational efficiency.
We train hypermodels estimate a posterior distribution over base models conditioned ob- served data, in a spirit similar to that of the Bayesian hypermodel literature (Krueger et al. 2017). Unlike typical variational approximations to Bayesian deep learning, this approach allows com- putationally efficient training with complex multimodal distributions. In this paper, we consider hypermodels trained through stochastic gradient descent on perturbed data (see Section [2.] for full description). Training procedures for hypermodels are an important area of research, and it may be possible to improve on this approach. but that is not the focus of this paper. Instead, we aim to understand whether more sophisticated hypermodel architectures can substantially improve explo- ration. To do this we consider bandit problems of varying degrees of complexity, and investigate the computational requirements to achieve low regret over a long horizon.
To benchmark the quality of posterior approximations, we compare their efficacy when used for Thompson sampling (Thompson 1933; Russo al., 2018). In its ideal form, Thompson sampling (TS) selects each action by sampling a model from the posterior distribution and optimizing over actions. For some simple model classes, this approach is computationally tractable. Hypermodels enable approximate TS in complex systems where exact posterior inference intractable.
Our results address three questions:
Q: Can alternative hypermodels outperform ensembles?
A: Yes. We demonstrate through a simple example that linear hypermodels can offer dramatic im- provements over ensembles in the computational efficiency of approximate TS. Further, we demon- strate that linear hypermodels can be effective in contexts where ensembles fail regardless of ensem- ble size.
Q: Can alternative hypermodels enable more intelligent exploration?
A: Yes. We demonstrate that, with neural network hypermodels, a version of information-directe sampling (Russo & Van Roy 2014 2018) substantially outperforms TS. This exploration scheme would be computationally prohibitive with ensemble hypermodels but becomes viable with a hyper- network.
Q: Are hypernetworks warranted?
A: Not clear. We prove a theorem showing that, with neural network base models, linear hypermod- els can already represent essentially any distribution over functions. However, it remains to be seen whether hypernetworks can offer statistical or computational advantages.
Variational methods offer alternative approach approximating posterior distribution and sam- pling from it. O D'Donoghue et al.] (2018) consider such an approach for approximating Thompson sampling in reinforcement learning. Approaches approximating TS and information-direecte sampling (IDS) with neural networks base models have been studied in (Lu & Van Roy 2017]
Riquelme et al., 2018) and Nikolov al. (2019), respectively, using ensemble representations of uncertainty. Hypermodels have been a subject of growing interest over recent years. Ha et al. (2016)
proposed the notion of hypernetworks as a relaxed form of teight-sharing. Krueger al. (2017)
proposed Bayesian hypernetworks for estimation of posterior distributions and training algorithm based on variational Bayesian deep learning. A limitation this approach is in its requirement that the hypernetwork be invertible. Karaletsos et al. (2018) studied Bayesian neural networks with correlated priors, specifically considering prior distributions in which units in the neural network represented by latent variables and weights between units are drawn conditionally on the values of those latent variables. [Pawlowski et al.] (2017) introduced another variational inference based algorithm that interprets hypernetworks as implicit distributions, i.e. distributions that may have intractable probability density functions but allow for easy sampling. Hu et al. (2018) proposes the Stein neural sampler which samples from given (un-normalized) probability distribution with neural networks trained by minimizing variants of Stein discrepancies.
2 HYPERMODEL ARCHITECTURES AND TRAINING
We consider base models that are parameterized by an element of a parameter space θ. Given 0 @ and an input X, E NN,, a base model posits that the conditional expectation of the output
2

---

Published as a conference paper at ICLR 2020
hypermodel
gu(2)
X,
model
fo( s0 Y4+1
X,
model
(Xa) Yet1
(a) base model
(b) hypermodel
Figure 1: A base model generates output Y, given parameters and input X,, while hypermodel
generates base model parameters (z) given hypermodel parameters and an index 2.
Y E R is given by GIY--IXX, 0] (Xx), for some class of functions indexed by 0. Figure la depicts this class of parameterized base models.
A hypermodel is parameterized by parameters v, which identify function g, 2 O. We will refer to each E Z as an index, as it identifies a specific instance of the base model. In particular, given hypermodel parameters base model parameters 0 can be generated by selecting setting 9x (=). This notion of a hypermodel illustrated in Figure 1b Along with a hypermodel, in order represent distribution over base models, we must specify reference distribution p= that can be used to sample an element of Z. A hypermodel and reference distribution together represent distribution over base models through offering mechanism for sampling them by sampling an index and passing it through the mapping.
Z and
2.1 HYPERMODEL TRAINING
Given set data pairs {(X,, :...).., parameters that the implied distribution over base model parameters approximates its posterior. important that training algorithms be incremental. This enables scalability and also allows for ongoing modifications to the data set, those occurring in the bandit learning context, in which data samples accumulate as time progresses.
hypermodel training algorithm computes
One approach to incrementally training hypermodel involves perturbing data by adding noise to response variables, and then iteratively updating parameters via stochastic gradient descent. We will assume here that the reference distribution Ps is either an N--dimensional unit Gaussian uniform distribution over the Nz-dimensional unit hypersphere. Consider an augmented data set 0, ... T 1}, where each A, E RRV; is random vector that serves to randomize computations carried out by the algorithm. Each vector A, is independently sampled from N N(0, I) if p. is uniform over the unit hypersphere. Otherwise. A, independently sampled from the unit hypersphere.
D {(x,, Y(41,
We consider a stochastic gradient descent algorithm that aims to minimize the loss function
C(v, D) = - (d2)
Σ
(y + owa (x))
1 () Iug (2)
2ERRN
202
20
(2.3.a)DD
where Y/ is the initial vector hypermodel parameters. Each iteration of the algorithm entails calculating the gradient of terms summed over minibatch of (x, U. a.) tuples and random indices z. Note that aw z here represents a random Gaussian perturbation the response variable y- In particular, each iteration, a minibatch D is constructed by sampling a subset D uniformly with replacement, and a set & of indices is sampled i.i.d. from pz. An approximate loss function
D
ī(v, D.
Σ
(y+owa (z)()) 20 (l) ge. (2)113
20
D
2E
(u.V.)ED
defined based these Hypermodel parameters updated according + a Vv D. 2)/IDI where a, o2 and of algorithm hyperparameters. In experiments,
3

---

Published as a conference paper at ICLR 2020
will take the step size to be constant over iterations. It is natural to interpret og prior variance, though the prior distribution over base model parameters is o2 and o as the standard deviation of noise, as though the enror distribution Y, - fo (X;) ~ [0 o Note, though, that hypermodel can be trained on data generated by any process. One can think of the hypermodel
and base models as inferential tools in the mind of an agent rather than a perfect reflection of reality.
2.2 ENSEMBLE HYPERMODELS
An ensemble hypermodel is comprised of ensemble of N, base models, each identified by parameter vector in e = RN Letting indices Z be the set of Ny-dimensional one-hot vectors, represent an ensemble in terms of function Jv 2 O with parameters E ON,, In particular, given hypermodel parameters v E ON,, an index E 2 generates base model parameters (Z) vZ. For an ensemble hypermodel, the reference distribution P3 is taken to be uniform over the N2 elements of Z.
2.3 LINEAR HYPERMODELS
Suppose that θ NN and 2 RNN. Consider a linear hypermodel, defined by go)z) + Bz, where hypermodel parameters are given by == (a E RN,, B E RNNoNN=) and 2 is an index with reference distribution ps taken to be the unit Gaussian (O0 7) over N₂ Ny-dimensional vectors. Such hypermodel can be used in conjunction with any base model that is parameterized by a vector of real numbers.
The aforementioned linear hypermodel entails number of parameters that grows with the product of the number Ng of base model parameters and the index dimension N,, since B is a No x N₂ matrix. This can give rise to onerous computational requirements when dealing with neural network base models. For example, suppose that we wish to model neural network weights as a Gaussian random vector. This would require index of dimension equal to the number of weights, and the number of hypermodel parameters would become quadratic in the number of neural network weights. For large neural network, storing and updating that many parameters impractical. As such, it natural to consider linear hypermodels in which the parameters a and B are linearly constrained. Such linear constraints can, for example, represent independence or conditional independence structure among neural network weights.
2.4 NEURAL NETWORK HYPERMODELS
More complex hypermodels are offered by neural networks. In particular, consider the case in which du is a neural network with weights v, taking N, inputs and producing No outputs. Such representation is altermately refered to as hypernetwork. Let the reference distribution ps be the unit Gaussian N I) over ---dimensional vectors. As special case, neural network hypermodel becomes linear if there are no hidden layers.
2.5 ADDITIVE PRIOR MODELS
In order for stochastic gradient descent algorithm operate effectively, it is often important to structure the base model so that it is sum of prior model, with parameters fixed at initialization, and a differential model, with parameters that evolve while training. The idea here for the prior model to represent sample from prior distribution and for the differential to learn the difference between prior and posterior as training progresses. This additive decomposition was first introduced in (Osband et al. 2018), which demonstrated its importance in training ensemble hypermodels with neural network base models using stochastic gradient descent. Without this decomposition, to gen- erate neural networks that represent samples from a sufficiently diffuse prior, we would have to initialize with large weights. Stochastic gradient descent tends to train too slowly and thus becomes impractical if initialized in such a way.
We will consider a decomposition that uses neural network base models (including lincar base mod- els special case) though the concept is more general. Consider neural network model class (.g θ ) with θ R NNa where the parameter vector θ includes edge weights and node biases. Let the index set Z be N= Let D be diagonal matrix for which each element is the prior standard
4

---

Published as a conference paper at ICLR 2020
deviation of corresponding component of 6. Let B E &N be random matrix produced initialization. We will take 0 DB to be parameters of the prior (base) model. Note that, given an index 2, this generates prior model fØ fDB=- When wish to completely specify prior model distribution, we will need to define a distribution for generating the matrix B as well as reference distribution Pz-
Given a prior model of the kind we have described, we consider a base model of the form fo(x) = (a) + fa(x), where {fa 0 is the concatenation of and 0. With 0 DBz, the idea is compute parameters 0 such that fa fna= fo approximates a sample from posterior distribution, conditioned on data. As such, fa represents a difference between prior and posterior. This decomposition is motivated by the observation that neural network training algorithms are most effective if initialized with small weights and biases. we initialize 0 with small values, the initial values of fa will be small, and in this regime, fo = fDB2. which is appropriate since untrained base model should represent sample from prior distribution. In general, 0 is the output of neural network ÿa taking the same input as the prior hypermodel 0 DBz. As is discussed above, in the course of training, we will only update while keeping D and fixed.
} is another neural network model class satisfying fo = 0, and
3 EXPLORATION SCHEMES
Our motivation for studying hypermodels stems from their potential role in improving exploration methods. As context for studying exploration, we consider bandit problems. In partieular, we consider the problem faced by agent making sequential decisions, in each period selecting an action X₂ E X' and observing a response Y141 E R. Here, the action set X' a finite subset of RN. and Y2+1 is interpreted as a reward, which the agent wishes to maximize.
We view the environment as channel that maps X, to Y++1. and conditioned on X, and the en- vironment, Y1+1 is conditionally independent of Xo, Y1,...,, X,-1, Y,. In other words, actions do not induce delayed consequences. However, the agent learns about the environment from applying actions and observing outcomes, and such, its prediction of outcome V influenced by past observations Xn, V1,,,.,,,-1,, Yt-
X,-1,
A base model serves possible realization of the environment, while hypermodel encodes belief distribution over possible realizations. We consider an agent that represents beliefs about the environment through hypermodel, continually updating hypermodel paramerers P via stochastic gradient descent, described in Section 2.1 to minimize loss function based on past actions and observations. At each time t, the agent selects action X, based on the current hypermodel. Its selection should balance between exploring to reduce uncertainty indicated by the hypermodel and exploiting knowledge conveyed by the hypermodel to accrue rewards.
3.1 THOMPSON SAMPLING
TS is simple and often very effective exploration scheme that will serve as baseline in our experi- ments. With this scheme, each action X; is selected by sampling an index a ~ pa from the reference distribution and then optimizing the associated base model to obtain X, E arg max,5x Sonxxxoooeee (x). See (Russo et al. 2018 for an understanding when and why TS is effective.
3.2
CIOMMATION-DIRECTERR SAMPLING
IDS (Russo & Van Roy 2014 2018) offers alternative approach exploration that aims to more directly quantify and optimize the value of information. There are multiple versions of IDS, and consider here a sample-based version of variance-IDS (Russo & Van Roy] 2018). In each time period, this entails sampling new multiset 2 i.i.d. from p-. Then, for each action E X' compute the sample mean of immediate regret
1
Σ
max (nn())("")) (o))))())) x*Ex
ZE2
5

---

Published as a conference paper at ICLR 2020
and sample variance of reward across possible realizations of the optimal action
2
Σ (x) =EZ,.
Σ (a)(e)(a)) FE2
Ur
Z
[2
XEEX
Here, (2.- xx E 2) forms partition of 2 such that, x² is an optimal action for each E Ezr; that is, Zx. = {= E Ø|x* E arg max (s)())(z)) Then, probability vector x² E Ax is obtained by solving
E arg Bargmin min (Zrex T TEAx Exex TzU22
and action X2 sampled from π². Note that THe = 0f Zn is empty. As established by (Russo & Van Roy 2018), the minimum over E Ax is always attained by probability vector that has at most two nonzero components, and this fact can be used to simplify optimization algorithms.
Producing reasonable estimates of regret and variance calls for many distinct samples, and the num- ber required scales with the number of actions. An ensemble hypermodel with tens of elements does suffice, while alternative hypermodels we consider can generate very large numbers of distinct samples.
4 CAN HYPERMODELS OUTPERFORM ENSEMBLES?
Because training a large ensemble can be prohibitively expensive, neural network ensembles have typically been limited to tens of models (Riquelme et al.) (2018). In this section, we demonstrate that a linear hypermodel can realize the benefits of a much larger ensemble without the onerous computational requirements.
4.1 GAUSSIAN BANDIT WITH INDEPENDENT ARMS
We consider a Gaussian bandit with K independent arms where the mean reward vector 0* E R drawn from Gaussian prior N N(0, T) During each time period I, the agent selects action X, and observes noisy reward Y++1 8% W++1, where W4+1 is i..d. NO, o..) We let of 2.25 and 22 and we fix the time horizon 10,000 periods.
We compare an ensemble hypermodel and a diagonal linear hypermodel trained via SGD with per- turbed data. Our simulation results show that diagonal linear hypermodel requires about 50 100 times less computation than an ensemble hypermodel to achieve our target level of performance.
As discussed in Section 2.5 we consider base models of the form fo(x) fDD + fa(2) (æ:). where (DB=(x) is an additive prior model, and f (z) is trainable differential model that aims learn the difference between prior and posterior. For an independent Gaussian bandit, fa(z) fx( = 0_ for all 0 and x. Although the use of prior models is inessential in this toy example, we include it for consistency and illustration of the approach.
The index dimensional one-hot vectors. Each row of B E 32KNN, is sampled from N(0, 7), and D T. The ensemble (differential) hypermodel takes the form (2) = òz, where the parameters & E K N. are initialized to i.i.d. (0, 0.05²). Although initializing to small random numbers instead of zeros unnecessary for a Gaussian bandit, our intention here is to mimic neural network initialization and treat the ensemble hypermodel as a special case of neural network hypermodels.
NN of an ensemble hypermodel sampled uniformly from the set N
In linear hypermodel, to model arms independently. we let ,ı,..., E Rm each be drawn in- dependently from N (O,,)) I), and let the index 2 E V= be the concatenation of 21...., &Ks with N₂ Km. Let the prior parameters bi...., bx a block matrix with b bT....,,b by on the diago- nal and zero everywhere else. Let D T.I. I. The diagonal linear (differential) hypermodel takes the form (2) Cs -i where E œ and matrix C E x has a block diagonal struc- ture C diag(c), .....,c)., ck), with cg....cc CN E Rm. The hypermodel parameters = (C, μ) are initialized i.i.d. (O0, 0.052).
Rm be sampled uniformly from the
dimensional hypershpere, and let E R RKXN,

---

Published as a conference paper at ICLR 2020
We train both hypermodels using SGD with perturbed data. For ensemble hypermodel, the per- turbation of the data point collected at time is A 2, where A ~ (N,,, I). For a diagonal linear hypermodel, the perturbation is A xx,, where A, is sampled uniformly from the m-dimensional unit hypersphere.
We consider an agent to perform well if its average regret over 10,000 periods is below 0.01 K. We compare the computational requirements of ensemble and diagonal linear hypermodels across different numbers of actions. As a simple machine-independent definition, we approximate the number of arithmetic operations each time period:
computation = 7lgd X f2 X data X Apparams+
where Ragd is the number of SGD steps per time period, n. is the number of index samples per SGD step, Retata the data batch size, and Nprrams the number of hypermodel parameters involved in each index sample. We fix the data batch size to 1024 for both agents, and sweep over other hyper- parameters separately for each agent. All results are averaged over 100 runs. In Figure we plot the computation needed versus number of actions. Using diagonal linear hypermodel dramatically re- duces the amount of computation needed to perform well relative to using an ensemble hypermodel, with speed-up of around 50 to 100 times for large numbers of actions.
comgeehation
1.00
uaval resembor
compute
casertbl l'Itear
200
number ofuctions
Figure 2: Computation required for ensemble hypermodels versus diagonal linear hypermodels to perform well Gaussian bandits with independent arms.
4.2 NEURAL NETWORK BANDIT
In this section we show that linear hypermodels can also be more effective than ensembles in settings that require generalization between actions. We consider a bandit problem with rewards generated by neural network that takes vector-valued actions as inputs. We consider a finite action set A C Red with d 20, sampled uniformly from the unit hypersphere. We generate data using neural network with input dimension 20, 2 hidden layers of size 3, and scalar output. The output is perturbed by i.i.d. NO, 1) observation noise. The weights of each layer are sampled independently from (0, 2. 25, (0, 23) and NO,, 23)). respectively, with biases from N(0, 1).
We compare ensemble hypermodels with 10, 30, 100, and 300 particles, and a linear hypermodel with index dimension 30. Both agents use an additive prior fDB= (z), where f a neural network with the same architecture the used to generate rewards. For the ensemble hypermodel, each row of B is initialized by sampling independently from N(0, 7), and is diagonal with appropriate prior standard deviations. For the linear hypermodel, we enforce independence of weights across layers by choosing B to be block diagonal with 3 blocks, one for each layer. Each block has width 10. Within each block, each row initialized by sampling uniformly from the 10-dimensional unit hypersphere. For the trainable differential model, both agents use neural network architecture with 2 hidden layers of width 10. The parameters of the ensemble hypermodel are initialized to truncated (0, 0.05²). The weights of the linear hypermodel are initialized using the Glorot uniform initialization, while the biases are initialized to zero.
In our simulations, we found that training without data perturbation gives lower regret for both agents. In Figure 3 we plot the cumulative regret of agents trained without data perturbation. We see that linear hypermodels achieve the least regret in the long run. The performance of ensemble hy- permodels is comparable when the number of actions is 200. However, there is large performance gap when the number of actions is greater than 200, which, surprisingly, cannot be compensated by increasing the ensemble size. We suspect that this may have to do with the reliability of neural network regression, and linear hypermodels somehow able to circumvent this issue.

---

Published as a conference paper at ICLR 2020
15-
cumulative regret (k)
agest xirdogerdees IS
ceroelly egroely
cunemale 12
(nsorble 33
tmeemble 310
timestago
Figure 3: Compare (i) ensemble hypermodels, (ii) linear hypermodels, (iii) annealing e-greedy, and (iv) an agent assuming independent actions on neural network bandit.
We also compare with e-greedy agent with a tuned annealing rate, and an agent that assumes in- dependent actions and applies TS under the Gaussian prior and Gaussian noise assumption. The gap between the e-greedy agent and hypermodel agents grows as the number of actions becomes large, as e-greedy explores uniformly and does not write off bad actions. The performance of the agent that assumes independent actions degrades quickly as the number of actions increases, since it does not generalize across actions. In the appendix, we also discuss Bayes by Backprop (Blundell et al.) 2015) and dropout (Gal & Ghahramani 2016) as approximation methods for posterior sampling.
5 CAN HYPERMODELS ENABLE MORE INTELLIGENT EXPLORATION?
IDS, as we described earlier, requires a large number of independent samples from the (approxi- mate) posterior distribution to generate action. One way to obtain these samples is to maintain an ensemble of models, as is done by Nikolov et al. (2019). However, as the number of actions increases, maintaining performance requires large ensemble, which becomes computationally pro- hibitive. More general hypermodels offer an efficient mechanism for generating the required large number of base model samples. In this section, present experimental results involving problem and hypermodel stylized to demonstrate advantages of IDS transparent manner. This context inspired by the one-sparse linear bandit problem constructed by Russo & Van Roy (2018). However, the authors of that work do not offer a general computationally practical approach that implements IDS. Hypermodels may serve this need.
We generate data according to Y1+1 = X, 0" + W1+1 where 0" E RRNo is sampled uniformly from one-hot vectors and W++1 i.id. (0, noise. We consider linear base model fo(2) 0Ta, and hypermodel (z)) cxp( (22, + aa)/ Noo exp(Sv,,22, + a)), where a = 0.01, and = 10. As reference distribution we let p. be ((. Let the initial hypermodel parameters vo be the vector with each component equal to one. Note that our hypermodel is designed to allow representation of the prior distribution, vectors. For simplicity, let Ng be power of two. Let Z be the set of indicator vectors for all non- singleton sublists of indices in (1,....Ne) that can be obtained by bisecting the list one more times. Note that [ = No 2. Let the action space X' be comprised one hot-vectors and vectors (z/2: : e T).
-==1
well uniform distributions over subsets of one-hot
As with the one-sparse linear bandit of (Russo & Van Roy 2018), this problem is designed so that TS will identify the nonzero component of by applying one-hot actions rule out one component period, whereas IDS will essentially carry out bisection search. This difference in behavior stems from the fact that TS will only ever apply actions that have some chance of being optimal, which in this context includes only the one-hot vectors, whereas IDS can apply actions known to be suboptimal if they are sufficiently informative.
Figure x plots regret realized by TS and variance-IDS using the aforementioned hypermodel. trained with perturbed SGD. As expected, the difference performance is dramatic. Each plot is averaged over 500 simulations. We used SGD hyperparameters of 0.01 and of 1/ log No. The ex- periments are with N₂- = 200, 500 samples are used for computing the variance- based information ratio.
8

---

Published as a conference paper at ICLR 2020
250
cummulatie regret
200
155
al algorth hm
1bs
75
Uimestep
Figure 4: Cumulative regret of IDS and TS with with one-sparse models.
6 ARE HYPERNETWORKS WARRANTED?
Results of the previous sections were generated using ensemble and linear hypermodels. remains to be whether hypernetworks offer substantial benefits. One might believe that hypernetworks can benefit from the computational advantages enjoyed by linear hypermodels while offering the ability to represent a broader range probability distributions over base models. The following result refutes this possibility by establishing that, with neural network base models, linear hyper- models can represent essentially any probability distribution over functions with finite domain. We denote by Loo (X', B) the set of functions f X' + R such that |f < B, where X' is finite with |X| K.
Theorem Lel be the unit Gaussian distribution in R2K For all > 0, B, O, and probability measures μ over Loso (X, B), there exist transport map H from pz μ, a neural network Se : + R with linear output node and ReLU hidden nodes, and a linear hypermodel
gu + RNoo form go(z) [=f v²r such that
[[ - 10 Se
with probability at least 1 where f H()).
This result established Appendix [ To digest the result, first suppose that the inequality is satisfied with O. Interpret as the target probability measure we wish to approximate using the hypermodel. Note that (n)(s) and f are determined by ~ Ps. and f is distributed according to μ,, since is a transport function that maps pz to (u. ([ - r = then [g. is also distributed according to μ., and as such, the hypermodel perfectly represents the distribution. If e > 0, the representation becomes approximate with tolerance
Though result indicates that linear hypermodels suffice to represent essentially all distributions over functions, we do not rule out the possibility of statistical or computational advantages to us- ing hypernetworks. In particular, there could be situations where hypernetworks generalize more accurately given limited data, or where training algorithms operate more effectively with hypernet- works. In supervised learning, deep neural networks offer such advantages even though single hidden layer suffices to represent essentially any function. Analogous benefits might carry over to hypernetworks, though we leave this question open for future work.
CONCLUSION
Our results offer initial signs of promise for the role of hypermodels beyond ensembles in improving exploration methods. We have shown that linear hypermodels can offer large gains in computational efficiency, enabling results that would otherwise require ensembles of hundreds or thousands of ele- ments. Further, these efficiency gains enable more sophisticated exploration schemes. In particular, experiment with version of IDS sampling and demonstrate benefits over methods based on TS. Finally, we consider the benefits of hypernetworks and establish that, with neural network base models, linear hypermodels Hence, to the extent that hypernetworks offer advantages, this would not in terms of the class of distributions that can be represented.
already able to represent essentially any distribution over functions.

---

Published as a conference paper at ICLR 2020
REFERENCES
Christophe Andrieu, Nando De Freitas, Arnaud Doucet, and Michael I Jordan. An introduction to MCMC for machine learning. Machine learning, 50(1-2):5 43, 2003.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint arXiv:1505005424, 2015.
Bradley Efron and Robert J Tibshirani. An introduction to the bootstrap. CRC press, 1994.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pp. 1050-1059, 2016.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint arXiv: 09106, 2016.
Jiri Hron, Alexander de G Matthews, and Zoubin Ghahramani. Variational Gaussian dropout is not Bayesian. arXiv preprint arXiv:1711.02989 2017.
2017.
Tianyang Hu, Zixiang Chen, Hanxi Sun, Jincheng Bai, Mao Ye, and Guang Cheng. Stein neural sampler, 2018
Theofanis Karaletsos, Peter Dayan, and Zoubin Ghahramani. Probabilistic reta-repressntatioons of neural networks. arXiv preprint arXiv:1810.00555 2018.
David Krueger, Chin-Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypernetworks. arXiv preprint 171.1070, 2017.
Xiuyuan Lu and Benjamin Van Roy. Ensemble sampling. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems 30, pp. 3258-3266. Curran Associates, Inc., 2017. uttp::t::/pppee nips cttps.ccpaaper/55e0 6918 ensemble sampling..poe
Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: view from the width. In Advances in Neural Information Processing Systems 30, pp.
2017.
Nikolay Nikolov, Johannes Kirschner, Felix Berkenkamp, and Andreas Krause. Information- directed exploration for deep reinforcement learning. In 7th International Conference on Learn- ing Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL https: //openzeview www.e//owoomieeeeeeeeee
Brendan O' D'Donoguue, Ian Osband, Remi Munos, and Volodymyr Mnih. The uncertainty bellman equation and exploration. arXiv preprint (rx.vv:700.055880 2018.
Ian Osband. Risk versus uncertainty in deep learning: Bayes, bootstrap and the dangers dropout. In NIPS Workshop on Bayesian Deep Learning, 2016.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep explo- ration via bootstrapped DQN. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 4026-4034. Curran Associates, Inc., 2016. URL dqn.pdf 6501 deep- exp.oration www.ratiiooi.---tttttteeeeeeeeee www.sooottppee..o.oooo
nips. cc/paper/
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep re- inforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Newral Information Processing Systems 31, pp. 8617-8629, Curran Associates, Inc., 2018. URL http: httt:::/pppp....iitggt wttpppppc.ccccc////ccccpccpcpppt functions for
8080 - 1e:00--rnnnmizzedppprroooe pdf
- learning.
Nick Pawlowski, Andrew Brock, Matthew CH Lee, Martin Rajchl, and Ben Glocker. Implicit weight uncertainty in neural networks. arXiv preprint arXiv: 2017.
10

---

Published as a conference paper at ICLR 2020
Carlos Riquelme, George Tucker, and Jasper Roland Snoek. Deep Bayesian bandits show- down. In Sixth Interatioonal Conference on Learning Representations, 2018. URL https:
Daniel Russo and Benjamin Van Roy. Learning optimize via information-directedd sampling. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q- Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 1583 1591. Curran Associates. Inc.. 2014. URL http://papers: nips. cc/paper/ pdd
Daniel Russo and Benjamin Van Roy. Learning to optimize via information-direettd sampling. Operations Research, (6(1):230-252) 2018.
Daniel J. Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. A tu- torial on Thompson sampling. Foundations and Trends in Machine Learning, 11(1):1-96, 2018. ISSN 1935-8237. doi: 10.1561/22000000700. htthtt::://ae doi /0tg/10.10 1561/ 2200000070
William R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285-294, 1933.

---

Published as a conference paper at ICLR 2020
A UNIVERSAL APPROXIMATION VIA LINEAR HYPERMODELS
Assume that X' C R is a finite set with | = K, and j is a probability measure over the bounded functions f X' R such that f| B. First, we show that we can approximately sample function from l using ReLU model with an linear hypermodel, with the input to the hypermodel drawn from the K- K-dimensional uniform distribution. Our main result summarized below:
Theorem 2 Let ps be the uniform distribution over [0. For all 0, 8 E (0, 1), B> O. and probability measures y over (X, B), there exists a transport map H from ps to μs, a neural network So X + R with linear oulput node and ReLU hidden nodes, and linear hypermodel 9v : Z + Re with form gu(z) = [=T vT"]" such that
llfgu(=) - fl Sc,
with probability at least 1 5, where 1 = (2))
Proof of Theorem 2: Note that since | = K, the functions f : - R can be represented as probability measure over R Since p= is absolutely vectors in RRK and hence lr can be viewed
continuous with respect to the Lebesgue measure, from Brenier's theorem, there exists measurable transport map H 32 - RA from pa to μ. Notice that can always assume 1/ [=) & [0, 1] since all the probability mass under Pa is in [0, 1,11 and this assumption does not affect the measurability of H. To show that each component of H is Lebesgue integrable, let H (2)(a) denote the component of (2) corresponding to x. Note that
for
H=))(() = J0.1IK Hx))aa/dde & / 11.11K
Bdz = B,
A2
where the inequality follows from the fact that μ is Loo (2, B).
From Theorem 1 in Lu et al. (2017). for any e > 0 and E (0. 1). there exists ReLU model Ā: RRK + RKK s.t. for any x E X,
J RN
H())[z] (x) < c
where (x) and [x) are respectively the component of (2) and H2) corresponding to r. Note that the above inequality implies:
/ [1(=)) (1) RK
/
[1((2) (z)l Σ
- (=) < ed.
12
ZEX RK
Hence have
-1 10.11k
Piz-p
[H(2)) (2)llood S
[H(()) -= cd.
Note that we can always assume (=)]l I B for = E [0, 1|K If this assumption does not hold,
we can add some ReLU layers to cap H ensure [2)]) I B. Since (2())) < almost surely, this cap will not increase Ez-ps [[(=)) [=)]
We now discuss how to implement ReLU model h: x'x (0, 1] + I-B, BJ s.t. h(x, 2) = A (=) based on the ReLU implementation of the K-dimensional (z). Note that it is straightforward to use ReLU to implement the K-dimensional one-hot encoding for all x E 2. Since (=)]l x B, by defining
4 Σ max 4B 1(2 = =x) H())[=) 2B, 03 x
hxx,2) 2)
2B
we have h(2 z) = [z) Since both (z) and the one-hot encoding can be implemented by ReLU, h(,,, z) can also be implemented by ReL.U.
12

---

Published as a conference paper at ICLR 2020
Finally, we discuss how to construct the target ReLU model fo X - R and the linear hypermodel 9v based on h. Note that the ReLU corresponding to h has K + 1 imput nodes, corresponding to x (called InputxX) and K corresponding to (called InputZ). Thus, by treating & as part of the parameter vector 0, fo X + R is construct as follows: we make InputZ hidden nodes, and the only input node to each node in InputZ is InputX. Specifically, the input to the ith node in InputZ is 0 x + 2j with scalar weight 0 and bias 24, where 2; the ith component of Also note that given h, the components in 0 are either constant or components in 2, thus Ov is linear and Since the components in z are statistically independent, and components in f = (z) are either constant or components in 2, consequently, the components in 0 are statistically independent. Note that by definition, (=)(() = h(æx=) = [x)] By defining f (=),, we have Ez-p, [gl - [ < 66. From Markov's, with probability at least 1 - 6, we have (=) - f I E. q.e.d.
can be written as (=) |=_ y²
Finally, we prove Theorem 1 based on Theorem
Proof of Theorem 1 The proof is similar to that of Theorem 2 Recall that p can be viewed as probability measure over R* Since Ps (0, Ix) is absolutely continuous with respect the Lebesgue measure, from Brenier's 's theorem, there exists a measurable transport map H R² + R² from p= to ja, moreover, H=)) = V=Φ(2) () for a convex scalar function Φ R - R. Notice that for the given 8 E (0, 1), we can always choose large enough >> 0, such that P(= E BB) 2 1 - 8/2, where B, = {z ||=||2 < n/}.
We define an auxiliary function H' RRKK + RK as
(x()) -
H(2)
= B, otherwise
Since H is measurable, H' is also measurable. Moreover, since H2(x) = H(=) V=0(z) on the compact set B, thus H' (2) is bounded in B,- Thus, for any x X, we have
(2) de / (=) <o.
RR
B,
Similar to the proof for Theorem s.t.
2 for the given and 8, there exists ReLU model H R² - R
J (x) (=) eō/2.
Notice that
Ezmps [1) (=) = / [=)
(H( < 66/2
R
where with little bit abuse of notation (-)-) denotes the probability density function of the proba- bility measure ps. The first inequality follows from (=()) S (2m) k/2 < 1.
The subsequent analysis is similar to that in Theorem
2 Similarly, we can prove that with probability least 5/2, we have (l.f H' (x) S c, where fo is a ReLU model and 9u linear
hypermodel and can be written as (2)
"², Moreover, the components in 0 (2)
statistically independent. Recall that with probability at least 5/2, have H'(z) H()). Thus, from the union bound, we have with probability at least 1 Ō- we have ([fa=()) - H()) S By setting f = H(),, we have proved the theorem. q.e.d.
B ADDITIVE PRIORS
Osband et al. (0088) discuss the benefits of using an additive prior to represent prior uncertainty over models. In sequential decision making, it is particularly crucial to be able to represent prior uncertainties when there is little data available. In this section, we demonstrate the effectiveness of additive priors by comparing ensembles with and without additive priors the neural network bandit problem described in Section 4.2 Both ensembles are initialized randomly using standard neural network initializations. Since weights typically initialized to small values (otherwise
13

---

Published as a conference paper at ICLR 2020
cunquaatve popres to at 100k stops
100
1AD actisss
Din uchone
addittive prior
Fdaes
30 100
310
350
180 350
ensemble
Figure Compare ensembles with and without additive priors on a neural network bandit.
training could be difficult), the outputs of ensembles that do not use additive priors will be close zero and will not reflect prior uncertainties general. We see in Figure 3 that ensembles with additive priors achieve significantly lower regrets across different numbers of actions and ensemble sizes.
C ALGORITHM SENSITIVITY ANALYSIS
In order to analyze the sensitivity of the algorithm on different parameters, we present a series of experiments on the neural network bandit, similar to Section 4.2 The default values of the
parameters used these experiments are same as the values used to generate Figure 3 from the experiments are presented below.
The results
In Figure 6 we present some results from sensitivity analysis on observation noise and perturbation noise. The plot shows the regret of linear hypermodel 100k steps, with different perturbation scales Ow (standard deviation of the noise added to the response variable in the loss function) and standard deviations of the observation noise o Both ow and a take values from {0, 1, Observe that for ow = 0 and ow = 1, perturbation scale of O works the best; however, on increasing the observation noise dw 2, perturbation scale of 1 performs better than perturbation scale of O. The reason for this could be that SGD step is introducing sufficient amount of noise when om but it seems that need to inject additional noise for larger observation noise. From this, it is clear that we need to introduce perturbation into the loss function, as the observation noise grows. Even though there a discrepancy in the cumulative regret for different perturbation scales, the algorithm seems to be robust to the variation in perturbation scale across different levels of observation noise.
comolattee regret 6a at 100k stups
actione: 500
1686
7.5
perturbation seale
2.5-
standard deviation of observation noise d,
Figure 6: Performance of linear hypermodel with varying strengths in observation noise and pertur- bation
In Figure 7 we present results on how mis-specification in prior can affect the performance of the linear hypermodel. Plot shows the regret of linear hypermodel after 100k steps, when the prior is mis-specified. Prior is mis-speicified by drawing weights of the prior network from distribution such that the variance of these weights are a factor m times that of the variance the weights of the generator, call this value the prior weight multiplier. We that very small value
14

---

Published as a conference paper at ICLR 2020
of does not induce sufficient exploration and leads to huge regret. Similarly, large value of m can also induce more exploration than desired and leads to some additional regret.
cumulative regret at 100k steps
12.5
num actions
200
7.5
500
5-
2.5
4
prior weight multipher m
Figure 7: Performance of linear hypermodels for different values of multiplier m
In Figure 8 we show how performance of a linear hypermodel is affected by the index dimension. Recall from Section[4.2] that use disjoint segment of the index vector generate prior weights for each layer. We vary the index dimension per layer as 1, 2, 3, and 10 (corresponding to the entire index vector with dimension 3, 6, 9, 15, and 30) for 500 random seeds, and observe the average cumulative regret attained by the algorithm after 100k steps. Although there is some noise, observe that the index dimension increases the cumulative regret decreases. However, the improvement is marginal beyond an index dimension.
cunulative regret (3 at 100k steps
H
Index dimenston per layar
200
500 Durober of actions
10000
Figure 8: Performance of linear hypermodels for different index dimensions
D DIAGONAL LINEAR HYPERMODELS AND BAYES BY BACKPROP
An alternative approach for approximating posterior distributions for neural networks is variational methods such as Bayes by Backprop (Blundell et al. 2015). Bayes by Backprop assumes a diagonal Gaussian distribution as the variational posterior of neural network weights, which in effect uses diagonal linear hypermodel. Its training algorithm follows the variational inference framework and aims to minimize KL loss.
One can also train diagonal linear hypermodel using perturbed SGD stated in Section 2.1 Fixing the diagonal hypermodel architecture, one can ask whether perturbed SGD or whether Bayes by Backprop is better training algorithm when used for Thompson sampling. We test these algorithms on the neural network bandit problem in Section 4.2
We find that when training diagonal hypermodel using perturbed SGD, base models that use additive prior as in Section 2.5 are difficult to train. Instead, we consider base models that are a given by DBz 8, where N(0 I). The prior is
single neural network whose weights encoded in DBs, where matrix B has rows sampled from the unit hypersphere during initialization and D encodes appropriate standard deviations. The learnable parameter 0 (x / + Cz
15

---

Published as a conference paper at ICLR 2020
cumulative regret (k)
agent
diagonal hm hbb
3-
diagonal hm pogd
dropout
hm pagd
50
timesteps (k)
Figure 9: Compare (i) a diagonal hypermodel agent trained with perturbed SGD, (ii) diagonal hypermodel agent trained with Bayes by Backprop, (iii) linear hypermodel agent trained with perturbed SGD, and (iv) a dropout agent on a small neural network bandit.
where is diagonal matrix and both i and are initialized zero. Initially, the weights of the base model are dominated by DBz, which is desirable since want samples from the prior when there is little or no data.
Further, to make results comparable with the linear hypermodel results in Section 4.2 we increase the size of the base network for diagonal hypermodel agents so that the number of trainable pa- rameters is approximately
the same level that of linear hypermodel agent in Section 4.2 Specifically, we let the base network be an MLP with hidden layers of size 60.
We observed in experiments that Bayes by Backprop does not work well with its originally proposed KL regularization. We find that we had to decrease the strength of the KL regularization by an order get competitive performance. Further, Bayes by Backprop performs badly when the prior standard deviation of the weights is specified far from 0.1 to 0.3, which could suggest that Bayes by Backprop may only support very limited prior specifications. In Figure 9 cumulative regret of the best tuned Bayes by Backprop agent.
we show the
Compared to Bayes by Backprop, perturbed SGD is easier tune. We observed that perturbations do not make much difference in this toy example, and that regularization of base model parameters does not play big role here we are doing SGD. We plot the cumulative regret of the best tuned perturbed SGD agent in Figure | We see that given the diagonal hypermodel architecture, per- turbed SGD performs slightly better than Bayes by Backprop. Both agents are worse than linear hypermodel agent trained with perturbed SGD.
E DROPOUT AS POSTERIOR APPROXIMATION FOR NEURAL NETWORKS
Another popupar approach for approximating posterior distributions for neural networks is dropout (Gal Ghahramani 2016). The dropout approach applies independent Bernoulli masks to the activations during training, and Gal & Ghahramani (2016) argue that applying dropout masks once again a forward pass approximates sampling from the posterior distribution. To make the number of trainable parameters comparable of size 100. We then sweep over the probability of keeping each neuron, and find that keeping probability of 0.5 works well. In Figure g see that the performance of tuned dropout agent worse than the hypermodel agents.
other agents, choose the network to have hidden layers
16


Reviews:
[{"review_id": "r1gZ_23X5H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the possibility of using hypermodels in improving the exploration of bandit problems. By using SGD for training the hypermodel parameters, this paper introduces a computationally efficient alternative to ensemble methods. The idea of the paper is novel and interesting; however, I do have several concerns, mainly from numerical experiments that I would like the authors to address those in the rebuttal.\n\n1) My first and the most important concern is that the numerical experiments do not evaluate different aspects of the method. There are numerous ways to check the sensitivity of your method for the choice of hyperparameters that I think could be added to the appendix. In addition to testing various values for $\\sigma_p$, $\\sigma_w$, and $\\nu_0$, I think that multiple experiments are missing:\n    i) larger neural network\n    ii) I was expecting to see what would happen without additive prior. It could be one of the baselines in Figure 3. Even though (Osband et al., 2018) discuss the effect of this extension, but the use of this model is not numerically justified. \n    iii) How the experiments are sensitive to the noise of the output variable? What will happen if you do not add noise?\n\nThere are also other experiments possible such as testing on a real scenario that would significantly improve the presentation of the work. This is not a requirement though.\n\n2) I didn't get what is the purpose of the last term in the loss function defined in Section 2.1. Why you are looking preferring $\\nu$ to be close to $\\nu_0$? \n\n3) P4, \"it is natural to consider linear hypermodels in which parameters a and B are linearly constrained.\" This sentence needs to be clarified. I didn't comprehend how you are dealing with large neural network issues.\n\n4) In Section 6, I was expecting to see a simulation showing a comparison of linear hypermodel with hypernetworks. \n\n\nMinor:\n* On P2, \"informatino-directed\" -> \"informatino-directed\"\n* In the second paragraph of Section 2.1, it is mentioned that a hypermodel involves perturbing data. My understanding is that what is meant here by perturbing data is to add some noise to X. However, in the later formulae, there is no such thing as perturbing data. You could say that since our numerical experiments didn't show any improvement using data perturbation, we didn't include it in our notations. Please remove the confusion.\n* very minor, but I would suggest using a different notation for $a$ in Sections 2.1 and 2.3 to remove any possible confusion. \n* I think that the summation in computing the variance of IDS should be over $\\tilde{Z}_{x^*}$.\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HJgv4jb5YH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors demonstrate advantages of a linear hypermodel over an ensemble method in exploration guided by epistemic uncertainty. They perform an empirical study in the bandit setting and claim that their approach both outperforms the ensemble method and offers a significant increase in computational efficiency. The theoretical contribution is that they prove universality in the sense that an arbitrary distribution over functions can be represented by a linear hypermodel. The experiments support their claims. Some of the explanations, however, are confusing, and relations to prior work should be clarified. \n\nFigure 3 shows a surprisingly large performance gap between the hypermodel and the ensemble method as the number of actions increases. But how about comparing linear hypermodels with different index sizes? Do we also expect asymptotic improvement as we increase the index size?\n\nImprecise or confusing explanations in the paper:\n\n1) Page 2, first Q: In theory, the effectiveness of the ensemble method should converge to that of the hypermodel as the ensemble size increases. They only tried ensemble size [10, 30, 100, 300] and then concluded that linear hypermodel can be effective regardless of the size of ensembles. Why?\n\n3) Page 3, Section 2.1, second paragraph, first sentence: Please clarify a bit more what do you mean by perturbing data? Random shuffling of the dataset in each training epoch? What does ‘response variables’ mean?\n\n4) Page 3, Section 2.1, second paragraph, last 2 sentences about $A_t$: we guess it should be $A_t ~ N(0, I)$ if $p_z$ is unit Gaussian according to the description in this paragraph. The current text claims it is the other way around, perhaps a typo?\n\n5) Page 3, Section 2.1, first equation: Why take the inner product between $a$ and $z$ ? How does this reflect the randomized computation (the motivation for augmented random vector $A_t$)? The objective is to maximize the log-likelihood of the prediction under the Gaussian assumption. Please clarify the assumptions about random variables $Y_t$ at the beginning of this paragraph. \n\n6) Same place as in 5): Why regularize hypermodel parameters such that they are not too far from the initial vector? Is $\\nu_0$ actually the additive prior model described in Section 2.5?\n\n7) Page 3, Section 2.1, second equation: why multiply $|D|$ in the first term within the parentheses? Why not just $1/|D_tilde|$ to average the prediction error over the mini-batch?\n\n8) Page 3, Section 2.1, second equation: is the cardinality of the index set $|Z_tilde|$ independent of mini-batch size? I.e. for each training data point there could be multiple models realized by multiple indices $z$\n\n9) Page 4, Section 2.5: Why use this decomposition for training the hypermodel? If the intuition is to keep the initial weight small, what if we just simply initialize small values for $f_\\theta(x)$ without decomposition?\n\n10) Page 5, last second sentence: The notation of partition (the set notation after ‘Here,….’) is supposed to be $\\hat{\\mathcal{Z}}_{x^*} = \\{ z\\in \\hat{\\mathcal{Z}} | x^* in \\argmax_{x} f_{g_{\\nu}(z)}(x) \\}$\n\nMinor typos:\n\n- Page 2, third paragraph: ‘…we compare their [efficacy] when used...’ ->  [efficiency] ?\n- Page 2, the last paragraph before Section 2, first sentence: ‘Approaches to approximating TS and [informatino]-directed sampling...’ -> [information]\n\nRelations to prior work:\n\n1. Page 2: Hypernetworks (where one neural net learns to generate the weights of another net) are much older than this recent reference of 2016. One should relate this work to the original references since 1991 [FAST0-3a][FAST5][FASTMETA1-3][CO2] in section 8 of the overview http://people.idsia.ch/~juergen/deep-learning-miraculous-year-1990-1991.html \n\n2. Intro 2nd par: dropout was first published much earlier in 1990 as the stochastic delta rule: \nHanson, S. J.(1990). A Stochastic Version of the Delta Rule, PHYSICA D,42, 265-272. See also arXiv:1808.03578, 2018. \n\nWe might improve our rating provided the comments above were addressed in a satisfactory way in the rebuttal.\n\nEdit after rebuttal: The authors replied: \"Thanks for pointing out typos and citations that we will add.\" But apparently in the revised PDF this did not happen. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1eEz8qRKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper builds on a classical idea of sampling model parameters apart from learning them. Specifically, it combines hierarchical sampling with neural networks and proposes models that can help explore the parameter space efficiently. The proposal is evaluated appropriately.\n\nWhat exactly do we mean by intelligent exploration? Is this quantified via the #samples needed or variance of sampled parameters? Or is it via regret? \n\nThe paper is clearly written and the idea makes sense. However the experiments are essentially based on simulated data. It is not entirely clear as to how this would translate to real setups. \n\nIs it possible that the linear hypermodel is performing well because the data was generated according to a linear model in section 5?\n\nIf the baseline is a classical ensembling setup, then why not use classical performance measures to evaluate the benefit of hypermodeling? like accuracy etc. Why are we specifically talking about bandits? In other words, does the proposed hyper sampling allow for better weak learners in general as well? \n ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers ensemble of deep learning models in order to quantify their epistemic uncertainty and use this for exploration in RL. The authors first show that limiting the ensemble to a small number of models, which is typically done for computational reasons, can severely limit the approximation of the posterior, which can translate into poor learning behaviours (e.g. over-exploitation). Instead, they propose a general approach based on hypermodels which can achieve the benefits of a large ensemble of models without the computational issues. They perform experiments in the bandit setting supporting their claim. They also provide a theoretical contribution, proving that an arbitrary distribution over functions can be represented by a linear hypermodel.

The decision boundary for this paper is unclear given the confidence of reviewers and their scores. However, the tackled problem is important, and the proposed approach is sound and backed up by experiments. Most of reviewers concerns seemed to be addressed by the rebuttal, with the exception of few missing references which the authors should really consider adding. I would therefore recommend acceptance.
Decision: Accept (Poster)