Hypermodels for Exploration

1 INTRODUCTION . Consider the sequential decision problem of an agent interacting with an uncertain environment , aiming to maximize cumulative rewards . Over each time period , the agent must balance between exploiting existing knowledge to accrue immediate reward and investing in exploratory behavior that may increase subsequent rewards . In order to select informative exploratory actions , the agent must have some understanding of what it is uncertain about . As such , an ability to represent and resolve epistemic uncertainty is a core capability required of the intelligent agents . The efficient representation of epistemic uncertainty when estimating complex models like neural networks presents an important research challenge . Techniques include variational inference ( Blundell et al. , 2015 ) , dropout1 ( Gal & Ghahramani , 2016 ) and MCMC ( Andrieu et al. , 2003 ) . Another approach has been motivated by the nonparametric bootstrap ( Efron & Tibshirani , 1994 ) and trains an ensemble of neural networks with random perturbations applied to each dataset ( Lu & Van Roy , 2017 ) . The spirit is akin to particle filtering , where each element of the ensemble approximates a sample from the posterior and variation between models reflects epistemic uncertainty . Ensembles have proved to be relatively effective and to address some shortcomings of alternative posterior approximation schemes ( Osband et al. , 2016 ; 2018 ) . When training a single large neural network is computationally intensive , training a large ensemble of separate models can be prohibitively expensive . As such , ensembles in deep learning have typically been limited to tens of models ( Riquelme et al. , 2018 ) . In this paper , we show that this parsimony can severely limit the quality of the posterior approximation and ultimately the quality of the learning system . Further , we consider more general approach based on hypermodels that can realize the benefits of large ensembles without the prohibitive computational requirements . A hypermodel maps an index drawn from a reference distribution to a base model . An ensemble is one type of hypermodel ; it maps a uniformly sampled base model index to that independently trained base model . We will consider additional hypermodel classes , including linear hypermodels , which we will use to map a Gaussian-distributed index to base model parameters , and hypernetworks , for which the mapping is a neural network ( Ha et al. , 2016 ) . Our motivation is that intelligent ∗DeepMind 1Although later work suggests that this dropout approximation can be of poor quality ( Osband , 2016 ; Hron et al. , 2017 ) . hypermodel design might be able to amortize computation across the entire distribution of base models , and in doing so , offer large gains in computational efficiency . We train our hypermodels to estimate a posterior distribution over base models conditioned on observed data , in a spirit similar to that of the Bayesian hypermodel literature ( Krueger et al. , 2017 ) . Unlike typical variational approximations to Bayesian deep learning , this approach allows computationally efficient training with complex multimodal distributions . In this paper , we consider hypermodels trained through stochastic gradient descent on perturbed data ( see Section 2.1 for a full description ) . Training procedures for hypermodels are an important area of research , and it may be possible to improve on this approach , but that is not the focus of this paper . Instead , we aim to understand whether more sophisticated hypermodel architectures can substantially improve exploration . To do this we consider bandit problems of varying degrees of complexity , and investigate the computational requirements to achieve low regret over a long horizon . To benchmark the quality of posterior approximations , we compare their efficacy when used for Thompson sampling ( Thompson , 1933 ; Russo et al. , 2018 ) . In its ideal form , Thompson sampling ( TS ) selects each action by sampling a model from the posterior distribution and optimizing over actions . For some simple model classes , this approach is computationally tractable . Hypermodels enable approximate TS in complex systems where exact posterior inference is intractable . Our results address three questions : Q : Can alternative hypermodels outperform ensembles ? . A : Yes . We demonstrate through a simple example that linear hypermodels can offer dramatic improvements over ensembles in the computational efficiency of approximate TS . Further , we demonstrate that linear hypermodels can be effective in contexts where ensembles fail regardless of ensemble size . Q : Can alternative hypermodels enable more intelligent exploration ? . A : Yes . We demonstrate that , with neural network hypermodels , a version of information-directed sampling ( Russo & Van Roy , 2014 ; 2018 ) substantially outperforms TS . This exploration scheme would be computationally prohibitive with ensemble hypermodels but becomes viable with a hypernetwork . Q : Are hypernetworks warranted ? . A : Not clear . We prove a theorem showing that , with neural network base models , linear hypermodels can already represent essentially any distribution over functions . However , it remains to be seen whether hypernetworks can offer statistical or computational advantages . Variational methods offer an alternative approach to approximating a posterior distribution and sampling from it . O ’ Donoghue et al . ( 2018 ) consider such an approach for approximating Thompson sampling in reinforcement learning . Approaches to approximating TS and information-directed sampling ( IDS ) with neural networks base models have been studied in ( Lu & Van Roy , 2017 ; Riquelme et al. , 2018 ) and Nikolov et al . ( 2019 ) , respectively , using ensemble representations of uncertainty . Hypermodels have been a subject of growing interest over recent years . Ha et al . ( 2016 ) proposed the notion of hypernetworks as a relaxed form of weight-sharing . Krueger et al . ( 2017 ) proposed Bayesian hypernetworks for estimation of posterior distributions and a training algorithm based on variational Bayesian deep learning . A limitation of this approach is in its requirement that the hypernetwork be invertible . Karaletsos et al . ( 2018 ) studied Bayesian neural networks with correlated priors , specifically considering prior distributions in which units in the neural network are represented by latent variables and weights between units are drawn conditionally on the values of those latent variables . Pawlowski et al . ( 2017 ) introduced another variational inference based algorithm that interprets hypernetworks as implicit distributions , i.e . distributions that may have intractable probability density functions but allow for easy sampling . Hu et al . ( 2018 ) proposes the Stein neural sampler which samples from a given ( un-normalized ) probability distribution with neural networks trained by minimizing variants of Stein discrepancies . 2 HYPERMODEL ARCHITECTURES AND TRAINING . We consider base models that are parameterized by an element θ of a parameter space Θ . Given θ ∈ Θ and an input Xt ∈ < Nx , a base model posits that the conditional expectation of the output Yt+1 ∈ < is given by E [ Yt+1|Xt , θ ] = fθ ( Xt ) , for some class of functions f indexed by θ . Figure 1a depicts this class of parameterized base models . A hypermodel is parameterized by parameters ν , which identify a function gν : Z 7→ Θ . We will refer to each z ∈ Z as an index , as it identifies a specific instance of the base model . In particular , given hypermodel parameters ν , base model parameters θ can be generated by selecting z ∈ Z and setting θ = gν ( z ) . This notion of a hypermodel is illustrated in Figure 1b . Along with a hypermodel , in order to represent a distribution over base models , we must specify a reference distribution pz that can be used to sample an element of Z . A hypermodel and reference distribution together represent a distribution over base models through offering a mechanism for sampling them by sampling an index and passing it through the mapping . 2.1 HYPERMODEL TRAINING . Given a set of data pairs { ( Xt , Yt+1 ) : t = 0 , . . . , T−1 } , a hypermodel training algorithm computes parameters ν so that the implied distribution over base model parameters approximates its posterior . It is important that training algorithms be incremental . This enables scalability and also allows for ongoing modifications to the data set , as those occurring in the bandit learning context , in which data samples accumulate as time progresses . One approach to incrementally training a hypermodel involves perturbing data by adding noise to response variables , and then iteratively updating parameters via stochastic gradient descent . We will assume here that the reference distribution pz is either an Nz-dimensional unit Gaussian or a uniform distribution over the Nz-dimensional unit hypersphere . Consider an augmented data set D = { ( Xt , Yt+1 , At ) : t = 0 , . . . , T − 1 } , where each At ∈ < Nz is a random vector that serves to randomize computations carried out by the algorithm . Each vector At is independently sampled from N ( 0 , I ) if pz is uniform over the unit hypersphere . Otherwise , At is independently sampled from the unit hypersphere . We consider a stochastic gradient descent algorithm that aims to minimize the loss function L ( ν , D ) = ∫ z∈ < Nz pz ( dz ) 1 2σ2w ∑ ( x , y , a ) ∈D ( y + σwa > z − fgν ( z ) ( x ) ) 2 + 1 2σ2p ‖gν ( z ) − gν0 ( z ) ‖22 , where ν0 is the initial vector of hypermodel parameters . Each iteration of the algorithm entails calculating the gradient of terms summed over a minibatch of ( x , y , a ) tuples and random indices z . Note that σwa > z here represents a random Gaussian perturbation of the response variable y . In particular , in each iteration , a minibatch D̃ is constructed by sampling a subset of D uniformly with replacement , and a set Z̃ of indices is sampled i.i.d . from pz . An approximate loss function L̃ ( ν , D̃ , Z̃ ) = 1 |Z̃| ∑ z∈Z̃ 1 2σ2w |D| |D̃| ∑ ( x , y , a ) ∈D̃ ( y + σwa > z − fgν ( z ) ( x ) ) 2 + 1 2σ2p ‖gν ( z ) − gν0 ( z ) ‖22 is defined based on these sets . Hypermodel parameters are updated according to ν ← ν − α∇νL̃ ( ν , D̃ , Z̃ ) /|D| where α , σ2w , and σ2p are algorithm hyperparameters . In our experiments , we will take the step size α to be constant over iterations . It is natural to interpret σ2p as a prior variance , as though the prior distribution over base model parameters is N ( 0 , σ2pI ) , and σ 2 w as the standard deviation of noise , as though the error distribution is Yt− fθ ( Xt ) |θ ∼ N ( 0 , σ2w ) . Note , though , that a hypermodel can be trained on data generated by any process . One can think of the hypermodel and base models as inferential tools in the mind of an agent rather than a perfect reflection of reality . 2.2 ENSEMBLE HYPERMODELS . An ensemble hypermodel is comprised of an ensemble of Nν base models , each identified by a parameter vector in Θ = < Nθ . Letting indices Z be the set of Nν-dimensional one-hot vectors , we can represent an ensemble in terms of a function gν : Z 7→ Θ with parameters ν ∈ ΘNν . In particular , given hypermodel parameters ν ∈ ΘNν , an index z ∈ Z generates base model parameters gν ( Z ) = νZ . For an ensemble hypermodel , the reference distribution pz is taken to be uniform over the Nν elements of Z .

Summary: The authors demonstrate advantages of a linear hypermodel over an ensemble method in exploration guided by epistemic uncertainty. They perform an empirical study in the bandit setting and claim that their approach both outperforms the ensemble method and offers a significant increase in computational efficiency. The theoretical contribution is that they prove universality in the sense that an arbitrary distribution over functions can be represented by a linear hypermodel. The experiments support their claims. Some of the explanations, however, are confusing, and relations to prior work should be clarified.