Anti-Concentrated Confidence Bonuses For Scalable Exploration

√ T ) regret bounds for poly ( d ) fixed actions . We develop a practical variant for deep reinforcement learning that is competitive with contemporary intrinsic reward heuristics on Atari benchmarks . 1 INTRODUCTION . Optimism in the face of uncertainty ( OFU ) is a ubiquitous algorithmic principle for online decisionmaking in bandit and reinforcement learning problems . Broadly , optimistic decision-making algorithms augment their reward models with a bonus ( or intrinsic reward ) proportional to their uncertainty about an action ’ s outcome , ideally balancing exploration and exploitation . A vast literature is dedicated to developing and analyzing the theoretical guarantees of these algorithms ( Lattimore & Szepesvári , 2020 ) . In fundamental settings such as stochastic multi-armed and linear bandits , optimistic algorithms are known to enjoy minimax-optimal regret bounds . In modern deep reinforcement learning , many approaches to exploration have been developed with the same principle of optimism , with most empirical successes coming from uncertainty-based intrinsic reward modules ( Burda et al. , 2018b ; Pathak et al. , 2017 ; Osband et al. , 2016 ) . Such bonuses can be very useful , with prior work demonstrating impressive results on a wide array of challenging exploration problems . Several of these methods draw inspiration from theoretical work on multiarmed bandits , using ideas like count-based exploration bonuses . However , a related body of work on linear bandits provides tools for extending exploration bonuses to large but structured action spaces , a paradigm which may be appropriate for deep reinforcement learning . The Linear UCB ( LinUCB ) algorithm ( Auer , 2002 ; Dani et al. , 2008 ; Li et al. , 2010 ; Abbasi-Yadkori et al. , 2011 ) is attractive in this setting because it enjoys minimax-optimal statistical guarantees . To obtain these , LinUCB leverages a so-called elliptical bonus , the computation of which requires maintaining an inverse covariance matrix over action features . The principal challenge in generalizing the elliptical potential used in bandits to the deep setting lies in computing and storing this object . Due to the moving internal representation of the policy network , and the number of parameters used to compose it , a naive implementation of the LinUCB algorithm would require remembering all of the agent ’ s experience and constantly recomputing and inverting this matrix , which is likely extremely large . Clearly , such an approach is too computationally intensive to be useful . As we discuss in the next section , several works have used neural features for LinUCB-style bonuses , but all require an inversion of this sort , limiting their ability to scale to high dimensions . Towards bridging foundational algorithms with empirical frontiers , we develop a scalable strategy for computing LinUCB ’ s elliptical bonus , enabling us to investigate its effectiveness as an intrinsic reward in deep reinforcement learning . We use an ensemble of least-squares regressors to approximate these bonuses without explicitly maintaining the covariance matrix or its inverse . Our algorithm is both theoretically principled and computationally tractable , and we demonstrate that its empirical performance on Atari games is often competitive with popular baselines . 1.1 OUR CONTRIBUTIONS . We propose the use of anti-concentrated confidence bounds ( ACB ) to efficiently approximate the LinUCB bonus . ACB estimates per-action elliptical confidence intervals by regressing random targets on policy features . It anti-concentrates these bonuses by taking a maximum over the predictions from an ensemble of these regressors . First , we introduce ACB in the basic stochastic linear bandit setting . We show that these bonuses provably approximate LinUCB ’ s elliptical potential ; thus , optimistic exploration with ACB directly inherits standard analysis techniques for LinUCB . We derive near-optimal high-probability regret bounds for ACB , when the size of the action space is polynomial in the action feature dimension . We derive sufficient ensemble sizes for the special cases of multi-armed bandits and fixed actions , as well as the general case of changing actions . These follow from lower tail bounds for the maximum of independent Gaussians ; we conjecture that they are improvable using more sophisticated analytical tools . The main contribution of this work is empirical : we find that ACB provides a viable exploration bonus for deep reinforcement learning . After defining a suitable nonlinear analogue using action features from the policy network , we demonstrate that the intrinsic rewards produced by ACB are competitive with those from state-of-the-art algorithms in deep RL on a variety of Atari benchmarks ( Figure 1 ) . To the best of our knowledge , our work is the first to scalably study bonuses from the linear bandits literature in these deep reinforcement learning settings . 1.2 RELATED WORK . Linear bandits . Stochastic linear bandits were first introduced by Abe & Long ( 1999 ) . Optimistic algorithms are fundamental in this setting ( Auer , 2002 ; Dani et al. , 2008 ; Li et al. , 2010 ; Rusmevichientong & Tsitsiklis , 2010 ; Abbasi-Yadkori et al. , 2011 ) , and provide minimax-optimal regret bounds . Several works are concerned with designing more scalable optimism-based algorithms , with a focus on empirical bandit settings ( as opposed to deep RL ) . Jun et al . ( 2017 ) consider streaming confidence bound estimates ( obtaining per-iteration update costs independent of t ) in the generalized linear bandits model , but still incur the cost of maintaining the inverse covariance matrix via online Newton step updates . Ding et al . ( 2021 ) give strong regret guarantees for epoch-batched SGD with Thompson sampling , under more stringent assumptions ( i.i.d . action features , and a “ diversity ” condition to induce strong convexity ) . Korda et al . ( 2015 ) demonstrate that variance-reduced algorithms succeed in practice for estimating the LinUCB bonus . Most closely related to our work , Kveton et al . ( 2019 ) ; Ishfaq et al . ( 2021 ) study perturbed regressions for exploration , obtaining similar regret bounds to ours ( including the suboptimal √ logA factor in the regret bounds ) . The LinPHE algorithm in ( Kveton et al. , 2019 ) uses a single random regressor ’ s constant-probability anti-concentration to approximate the elliptical bonus . In work concurrent to ours , the main algorithm proposed by Ishfaq et al . ( 2021 ) also takes a maximum over random regressors , and plugs their analysis into downstream end-to-end results for linear MDPs . The algorithms presented in these works resemble the “ always-rerandomizing ” variant of ACB ( Algorithm 1 ) , which is still not suitable for large-scale deep RL . Our work introduces the “ lazily-rerandomizing ” and “ never-rerandomizing ” variants in an effort to close this gap , and presents corresponding open theoretical problems . Unlike the aforementioned works , we consider our primary contribution as empirical , with the deep learning variant of ACB ( Algorithm 2 ) offering performance comparable to commonly used deep RL bonuses on a wide array of Atari benchmarks without sacrificing theoretical transparency . The resulting algorithmic choices deviate somewhat from the bandit version ( as well as LinPHE/LSVIPHE ) , towards making ACB a viable drop-in replacement for typical bonuses used in deep RL . The use of ensembles in this work bears a resemblance to Monte Carlo techniques for posterior sampling ( Thompson , 1933 ; Agrawal & Goyal , 2013 ) . This is a starting point for many empiricallymotivated exploration algorithms , including for deep RL ( see below ) , but theory is limited . ( Lu & Van Roy , 2017 ) analyze an ensemble-based approximation of Thompson sampling ( without considering a max over the ensemble ) , and obtain a suboptimal regret bound scaling with A logA . Exploration in deep reinforcement learning . In deep reinforcement learning , most popular approaches use predictability as a surrogate for familiarity . The Intrinsic Curiosity Module ( ICM ) , for example , does this by training various neural machinery to predict proceeding states from current state-action pairs ( Pathak et al. , 2017 ) . The L2 error of this prediction is used as a bonus signal for the agent . The approach can be viewed as a modification to Intelligent Adaptive Curiosity , but relying on a representation that is trained to encourage retaining only causal information in transition dynamics ( Oudeyer et al. , 2007 ) . Stadie et al . ( 2015 ) instead use a representation learned by an autoencoder . A large-scale study of these “ curiosity-based ” approaches can be found in Burda et al . ( 2018a ) . Similar to these algorithms , Random Network Distillation ( RND ) relies on the inductive bias of a random and fixed network , where the exploration bonus is computed as the prediction error between this random network and a separate model trained to mimic its outputs ( Burda et al. , 2018b ) . In tabular settings , count-based bonuses are often used to encourage exploration ( Strehl & Littman , 2008 ) . This general approach has been extended to the deep setting as well , where states might be high-dimensional and infrequently visited ( Bellemare et al. , 2016 ) . Algorithms of this type usually rely on some notion of density estimation to group similar states ( Ostrovski et al. , 2017 ; Tang et al. , 2016 ; Martin et al. , 2017 ) . Other work proposes more traditional uncertainty metrics to guide the exploration process . For example , Variational Intrinsic Control ( Gregor et al. , 2016 ) uses a variational notion of uncertainty while Exploration via Disagreement ( Pathak et al. , 2019 ) and Bootstrapped DQN ( Osband et al. , 2016 ) model uncertainty over the predictive variance of an ensemble . More broadly , various approximations of Thompson sampling ( Thompson , 1933 ) have been used to encourage exploration in deep reinforcement learning ( Osband et al. , 2013 ; Guez et al. , 2012 ; Zhang et al. , 2020 ; Strens , 2000 ; Henaff et al. , 2019 ) . These methods have analogues in the related space of active learning , where ensemble and variational estimates of uncertainty are widely used ( Gal et al. , 2017 ; Beluch et al. , 2018 ) . Separate from these , there has been some work intended to more-directly deliver UCB-inspired bonuses to reinforcement learning ( Zhou et al. , 2020 ; Nabati et al. , 2021 ; Zahavy & Mannor , 2019 ; Bai et al. , 2021 ) . These efforts have various drawbacks , however , with many requiring matrix inversions , limiting their ability to scale , or simply not being competitive with more conventional deep RL approaches . ACB removes the covariance matrix inversion requirement , and as we demonstrate in Section A.2 , is competitive with RND and ICM on several Atari benchmarks . 0 1 2 3 4 5 Timesteps 1e3 0 1 2 3 4 5 6 Cu m ul at iv e Re gr et 1e2 OLS ensemble updates M=1 M=2 M=4 M=8 M=16 M=32 M=64 M=128 M=256 LinUCB 0 1 2 3 4 5 Timesteps 1e3 0 1 2 3 4 5 6 1e2 SGD ensemble updates Figure 2 : The cumulative regret for ACB ( incremental sampling ) with different ensemble sizes . We show a multi-armed bandit setting with 50 actions , comparing exact ordinary least squares ( left ) with SGD using Polyak averaging ( right ) . Up to a point , increasing ensemble size improves cumulative regret in both cases . 2 PRELIMINARIES . In this section , we provide a brief review and establish notation for the stochastic linear bandit setting and the LinUCB algorithm . For a more comprehensive treatment , see Agarwal et al . ( 2019 ) ; Lattimore & Szepesvári ( 2020 ) . At round t = 1 , . . . , T , the learner is given an action set At ⊆ A with features { xt , a ∈ Rd } a∈At , chooses an action at ∈ At , and receives reward rt ( at ) : = 〈xt , at , θ∗〉+ εt , where εt is a martingale difference sequence . We use A to denote maxt |At| . The performance of the learner is quantified by its regret , R ( T ) = T∑ t=1 〈 xt , a∗t , θ ∗〉− 〈xt , at , θ∗〉 , a∗t : = arg max a∈At 〈xt , a , θ∗〉 . This setting encompasses two commonly-considered special cases : • Fixed actions : the action sets At are the same for all t. • Multi-armed bandits : fixed actions , with At = [ d ] and xt , a = ea . Here , θ∗ is the vector of per-arm means . An OFU-based algorithm plays at round t at : = arg max a∈At { r̂t ( a ) + bonust ( a ) } , where r̂t ( a ) is an estimate for the reward rt ( a ) , and bonust ( a ) is selected to be a valid upper confidence bound for r̂t ( a ) , such that E [ rt ( a ) ] ≤ r̂t ( a ) + bonust ( a ) with high probability . The LinUCB algorithm uses r̂t ( a ) : = 〈 xt , a , θ̂t 〉 , where θ̂t = arg minθ∈Rd ∑t−1 τ=1 ( 〈xτ , aτ , θ〉 − ri ( ai ) ) 2 + λ‖θ‖2 is the 2-regularized least-squares regressor for the reward , and sets bonust ( a ) = β √ x > t , aΣ̃ −1 t xt , a , where Σ̃t : = λI + ∑t−1 τ=1 xt , atx > t , at . In an online learning context , this is known as the elliptical potential ( Cesa-Bianchi & Lugosi , 2006 ; Carpentier et al. , 2020 ) . We will quantify the theoretical performance of ACB by comparing it to LinUCB . Theorem 1 gives a high-probability regret bound for LinUCB , which is optimal up to logarithmic factors : Theorem 1 ( LinUCB regret bound ; Theorem 5.3 , ( Agarwal et al. , 2019 ) ) . If ||θ∗|| , ||xt , a|| ≤ O ( 1 ) , and each εt is O ( 1 ) -subgaussian , then there are choices of λ , β such that , with probability at least 1− δ , the regret of the LinUCB algorithm satisfies R ( T ) ≤ Õ ( d √ T ) . The Õ ( · ) suppresses factors polynomial in log ( 1/δ ) and log T . LinUCB can be implemented in O ( d2|At| ) time per iteration , by maintaining Σ̃−1t using rank-1 updates , and using this to compute the least-squares estimator θ̂t = Σ̃−1t ∑t−1 τ=1 xτ , aτ rτ ( aτ ) as well as bonust ( a ) . Algorithm 1 ACB for linear bandits Parameters : ensemble size M ; β , λ 1 : For all i = 1 , . . . , d , set warm-start features x−i+1 = √ λei and corresponding warm-start targets y ( j ) −i+1 ∼ N ( 0 , 1 ) for each j ∈ [ M ] 2 : for t = 1 , . . . , T : do 3 : Observe action features { xt , a } a∈At 4 : Update reward estimator : θ̂t : = arg min θ t−1∑ τ=1 ( 〈xτ , aτ , θ〉 − rτ ( aτ ) ) 2 + λ‖θ‖2 5 : Sample new random ensemble targets : ( re-randomized ) y ( j ) τ ∼ N ( 0 , 1 ) for each j ∈ [ M ] , τ ∈ { −d+ 1 , . . . , t− 1 } ( incremental ) y ( j ) t−1 ∼ N ( 0 , 1 ) for each j ∈ [ M ] 6 : Update bonus ensembles for each j ∈ [ M ] , w ( j ) t : = arg min w t−1∑ τ=−d+1 ( 〈xτ , aτ , w〉 − y ( j ) τ ) 2 7 : Compute per-arm bonuses bonust ( a ) : = β ·maxj∈ [ M ] ∣∣∣〈xt , a , w ( j ) t 〉∣∣∣ for each a ∈ At 8 : Choose action at : = arg maxa∈At〈xt , a , θ̂t〉+ bonust ( a ) 9 : Observe reward rt ( at ) = 〈xt , at , θ∗〉+ εt

Summary: This paper proposes a new intrinsic reward method for continuous-action-space deep RL algorithms. The proposed algorithm is inspired by the bonus term of LinUCB. The proposed algorithm improves the efficiency of its reward bound’s computation by using a randomized variant of the original reward bonus. The authors theoretically justified the optimality of this ACB bonus term in the linear case, showing that this is a reasonable replacement of the LinUCB reward bonus. Finally, the authors add the ACB bound to deep RL algorithms. Although this algorithm comes with no theoretical guarantee, experiments show that the ACB intrinsic reward is competitive compared to adopted baselines.