Anti-Concentrated Confidence Bonuses For Scalable Exploration

√ T ) regret bounds for poly ( d ) fixed actions . We develop a practical variant for deep reinforcement learning that is competitive with contemporary intrinsic reward heuristics on Atari benchmarks . 1 INTRODUCTION . Optimism in the face of uncertainty ( OFU ) is a ubiquitous algorithmic principle for online decisionmaking in bandit and reinforcement learning problems . Broadly , optimistic decision-making algorithms augment their reward models with a bonus ( or intrinsic reward ) proportional to their uncertainty about an action ’ s outcome , ideally balancing exploration and exploitation . A vast literature is dedicated to developing and analyzing the theoretical guarantees of these algorithms ( Lattimore & Szepesvári , 2020 ) . In fundamental settings such as stochastic multi-armed and linear bandits , optimistic algorithms are known to enjoy minimax-optimal regret bounds . In modern deep reinforcement learning , many approaches to exploration have been developed with the same principle of optimism , with most empirical successes coming from uncertainty-based intrinsic reward modules ( Burda et al. , 2018b ; Pathak et al. , 2017 ; Osband et al. , 2016 ) . Such bonuses can be very useful , with prior work demonstrating impressive results on a wide array of challenging exploration problems . Several of these methods draw inspiration from theoretical work on multiarmed bandits , using ideas like count-based exploration bonuses . However , a related body of work on linear bandits provides tools for extending exploration bonuses to large but structured action spaces , a paradigm which may be appropriate for deep reinforcement learning . The Linear UCB ( LinUCB ) algorithm ( Auer , 2002 ; Dani et al. , 2008 ; Li et al. , 2010 ; Abbasi-Yadkori et al. , 2011 ) is attractive in this setting because it enjoys minimax-optimal statistical guarantees . To obtain these , LinUCB leverages a so-called elliptical bonus , the computation of which requires maintaining an inverse covariance matrix over action features . The principal challenge in generalizing the elliptical potential used in bandits to the deep setting lies in computing and storing this object . Due to the moving internal representation of the policy network , and the number of parameters used to compose it , a naive implementation of the LinUCB algorithm would require remembering all of the agent ’ s experience and constantly recomputing and inverting this matrix , which is likely extremely large . Clearly , such an approach is too computationally intensive to be useful . As we discuss in the next section , several works have used neural features for LinUCB-style bonuses , but all require an inversion of this sort , limiting their ability to scale to high dimensions . Towards bridging foundational algorithms with empirical frontiers , we develop a scalable strategy for computing LinUCB ’ s elliptical bonus , enabling us to investigate its effectiveness as an intrinsic reward in deep reinforcement learning . We use an ensemble of least-squares regressors to approximate these bonuses without explicitly maintaining the covariance matrix or its inverse . Our algorithm is both theoretically principled and computationally tractable , and we demonstrate that its empirical performance on Atari games is often competitive with popular baselines . 1.1 OUR CONTRIBUTIONS . We propose the use of anti-concentrated confidence bounds ( ACB ) to efficiently approximate the LinUCB bonus . ACB estimates per-action elliptical confidence intervals by regressing random targets on policy features . It anti-concentrates these bonuses by taking a maximum over the predictions from an ensemble of these regressors . First , we introduce ACB in the basic stochastic linear bandit setting . We show that these bonuses provably approximate LinUCB ’ s elliptical potential ; thus , optimistic exploration with ACB directly inherits standard analysis techniques for LinUCB . We derive near-optimal high-probability regret bounds for ACB , when the size of the action space is polynomial in the action feature dimension . We derive sufficient ensemble sizes for the special cases of multi-armed bandits and fixed actions , as well as the general case of changing actions . These follow from lower tail bounds for the maximum of independent Gaussians ; we conjecture that they are improvable using more sophisticated analytical tools . The main contribution of this work is empirical : we find that ACB provides a viable exploration bonus for deep reinforcement learning . After defining a suitable nonlinear analogue using action features from the policy network , we demonstrate that the intrinsic rewards produced by ACB are competitive with those from state-of-the-art algorithms in deep RL on a variety of Atari benchmarks ( Figure 1 ) . To the best of our knowledge , our work is the first to scalably study bonuses from the linear bandits literature in these deep reinforcement learning settings . 1.2 RELATED WORK . Linear bandits . Stochastic linear bandits were first introduced by Abe & Long ( 1999 ) . Optimistic algorithms are fundamental in this setting ( Auer , 2002 ; Dani et al. , 2008 ; Li et al. , 2010 ; Rusmevichientong & Tsitsiklis , 2010 ; Abbasi-Yadkori et al. , 2011 ) , and provide minimax-optimal regret bounds . Several works are concerned with designing more scalable optimism-based algorithms , with a focus on empirical bandit settings ( as opposed to deep RL ) . Jun et al . ( 2017 ) consider streaming confidence bound estimates ( obtaining per-iteration update costs independent of t ) in the generalized linear bandits model , but still incur the cost of maintaining the inverse covariance matrix via online Newton step updates . Ding et al . ( 2021 ) give strong regret guarantees for epoch-batched SGD with Thompson sampling , under more stringent assumptions ( i.i.d . action features , and a “ diversity ” condition to induce strong convexity ) . Korda et al . ( 2015 ) demonstrate that variance-reduced algorithms succeed in practice for estimating the LinUCB bonus . Most closely related to our work , Kveton et al . ( 2019 ) ; Ishfaq et al . ( 2021 ) study perturbed regressions for exploration , obtaining similar regret bounds to ours ( including the suboptimal √ logA factor in the regret bounds ) . The LinPHE algorithm in ( Kveton et al. , 2019 ) uses a single random regressor ’ s constant-probability anti-concentration to approximate the elliptical bonus . In work concurrent to ours , the main algorithm proposed by Ishfaq et al . ( 2021 ) also takes a maximum over random regressors , and plugs their analysis into downstream end-to-end results for linear MDPs . The algorithms presented in these works resemble the “ always-rerandomizing ” variant of ACB ( Algorithm 1 ) , which is still not suitable for large-scale deep RL . Our work introduces the “ lazily-rerandomizing ” and “ never-rerandomizing ” variants in an effort to close this gap , and presents corresponding open theoretical problems . Unlike the aforementioned works , we consider our primary contribution as empirical , with the deep learning variant of ACB ( Algorithm 2 ) offering performance comparable to commonly used deep RL bonuses on a wide array of Atari benchmarks without sacrificing theoretical transparency . The resulting algorithmic choices deviate somewhat from the bandit version ( as well as LinPHE/LSVIPHE ) , towards making ACB a viable drop-in replacement for typical bonuses used in deep RL . The use of ensembles in this work bears a resemblance to Monte Carlo techniques for posterior sampling ( Thompson , 1933 ; Agrawal & Goyal , 2013 ) . This is a starting point for many empiricallymotivated exploration algorithms , including for deep RL ( see below ) , but theory is limited . ( Lu & Van Roy , 2017 ) analyze an ensemble-based approximation of Thompson sampling ( without considering a max over the ensemble ) , and obtain a suboptimal regret bound scaling with A logA . Exploration in deep reinforcement learning . In deep reinforcement learning , most popular approaches use predictability as a surrogate for familiarity . The Intrinsic Curiosity Module ( ICM ) , for example , does this by training various neural machinery to predict proceeding states from current state-action pairs ( Pathak et al. , 2017 ) . The L2 error of this prediction is used as a bonus signal for the agent . The approach can be viewed as a modification to Intelligent Adaptive Curiosity , but relying on a representation that is trained to encourage retaining only causal information in transition dynamics ( Oudeyer et al. , 2007 ) . Stadie et al . ( 2015 ) instead use a representation learned by an autoencoder . A large-scale study of these “ curiosity-based ” approaches can be found in Burda et al . ( 2018a ) . Similar to these algorithms , Random Network Distillation ( RND ) relies on the inductive bias of a random and fixed network , where the exploration bonus is computed as the prediction error between this random network and a separate model trained to mimic its outputs ( Burda et al. , 2018b ) . In tabular settings , count-based bonuses are often used to encourage exploration ( Strehl & Littman , 2008 ) . This general approach has been extended to the deep setting as well , where states might be high-dimensional and infrequently visited ( Bellemare et al. , 2016 ) . Algorithms of this type usually rely on some notion of density estimation to group similar states ( Ostrovski et al. , 2017 ; Tang et al. , 2016 ; Martin et al. , 2017 ) . Other work proposes more traditional uncertainty metrics to guide the exploration process . For example , Variational Intrinsic Control ( Gregor et al. , 2016 ) uses a variational notion of uncertainty while Exploration via Disagreement ( Pathak et al. , 2019 ) and Bootstrapped DQN ( Osband et al. , 2016 ) model uncertainty over the predictive variance of an ensemble . More broadly , various approximations of Thompson sampling ( Thompson , 1933 ) have been used to encourage exploration in deep reinforcement learning ( Osband et al. , 2013 ; Guez et al. , 2012 ; Zhang et al. , 2020 ; Strens , 2000 ; Henaff et al. , 2019 ) . These methods have analogues in the related space of active learning , where ensemble and variational estimates of uncertainty are widely used ( Gal et al. , 2017 ; Beluch et al. , 2018 ) . Separate from these , there has been some work intended to more-directly deliver UCB-inspired bonuses to reinforcement learning ( Zhou et al. , 2020 ; Nabati et al. , 2021 ; Zahavy & Mannor , 2019 ; Bai et al. , 2021 ) . These efforts have various drawbacks , however , with many requiring matrix inversions , limiting their ability to scale , or simply not being competitive with more conventional deep RL approaches . ACB removes the covariance matrix inversion requirement , and as we demonstrate in Section A.2 , is competitive with RND and ICM on several Atari benchmarks . 0 1 2 3 4 5 Timesteps 1e3 0 1 2 3 4 5 6 Cu m ul at iv e Re gr et 1e2 OLS ensemble updates M=1 M=2 M=4 M=8 M=16 M=32 M=64 M=128 M=256 LinUCB 0 1 2 3 4 5 Timesteps 1e3 0 1 2 3 4 5 6 1e2 SGD ensemble updates Figure 2 : The cumulative regret for ACB ( incremental sampling ) with different ensemble sizes . We show a multi-armed bandit setting with 50 actions , comparing exact ordinary least squares ( left ) with SGD using Polyak averaging ( right ) . Up to a point , increasing ensemble size improves cumulative regret in both cases . 2 PRELIMINARIES . In this section , we provide a brief review and establish notation for the stochastic linear bandit setting and the LinUCB algorithm . For a more comprehensive treatment , see Agarwal et al . ( 2019 ) ; Lattimore & Szepesvári ( 2020 ) . At round t = 1 , . . . , T , the learner is given an action set At ⊆ A with features { xt , a ∈ Rd } a∈At , chooses an action at ∈ At , and receives reward rt ( at ) : = 〈xt , at , θ∗〉+ εt , where εt is a martingale difference sequence . We use A to denote maxt |At| . The performance of the learner is quantified by its regret , R ( T ) = T∑ t=1 〈 xt , a∗t , θ ∗〉− 〈xt , at , θ∗〉 , a∗t : = arg max a∈At 〈xt , a , θ∗〉 . This setting encompasses two commonly-considered special cases : • Fixed actions : the action sets At are the same for all t. • Multi-armed bandits : fixed actions , with At = [ d ] and xt , a = ea . Here , θ∗ is the vector of per-arm means . An OFU-based algorithm plays at round t at : = arg max a∈At { r̂t ( a ) + bonust ( a ) } , where r̂t ( a ) is an estimate for the reward rt ( a ) , and bonust ( a ) is selected to be a valid upper confidence bound for r̂t ( a ) , such that E [ rt ( a ) ] ≤ r̂t ( a ) + bonust ( a ) with high probability . The LinUCB algorithm uses r̂t ( a ) : = 〈 xt , a , θ̂t 〉 , where θ̂t = arg minθ∈Rd ∑t−1 τ=1 ( 〈xτ , aτ , θ〉 − ri ( ai ) ) 2 + λ‖θ‖2 is the 2-regularized least-squares regressor for the reward , and sets bonust ( a ) = β √ x > t , aΣ̃ −1 t xt , a , where Σ̃t : = λI + ∑t−1 τ=1 xt , atx > t , at . In an online learning context , this is known as the elliptical potential ( Cesa-Bianchi & Lugosi , 2006 ; Carpentier et al. , 2020 ) . We will quantify the theoretical performance of ACB by comparing it to LinUCB . Theorem 1 gives a high-probability regret bound for LinUCB , which is optimal up to logarithmic factors : Theorem 1 ( LinUCB regret bound ; Theorem 5.3 , ( Agarwal et al. , 2019 ) ) . If ||θ∗|| , ||xt , a|| ≤ O ( 1 ) , and each εt is O ( 1 ) -subgaussian , then there are choices of λ , β such that , with probability at least 1− δ , the regret of the LinUCB algorithm satisfies R ( T ) ≤ Õ ( d √ T ) . The Õ ( · ) suppresses factors polynomial in log ( 1/δ ) and log T . LinUCB can be implemented in O ( d2|At| ) time per iteration , by maintaining Σ̃−1t using rank-1 updates , and using this to compute the least-squares estimator θ̂t = Σ̃−1t ∑t−1 τ=1 xτ , aτ rτ ( aτ ) as well as bonust ( a ) . Algorithm 1 ACB for linear bandits Parameters : ensemble size M ; β , λ 1 : For all i = 1 , . . . , d , set warm-start features x−i+1 = √ λei and corresponding warm-start targets y ( j ) −i+1 ∼ N ( 0 , 1 ) for each j ∈ [ M ] 2 : for t = 1 , . . . , T : do 3 : Observe action features { xt , a } a∈At 4 : Update reward estimator : θ̂t : = arg min θ t−1∑ τ=1 ( 〈xτ , aτ , θ〉 − rτ ( aτ ) ) 2 + λ‖θ‖2 5 : Sample new random ensemble targets : ( re-randomized ) y ( j ) τ ∼ N ( 0 , 1 ) for each j ∈ [ M ] , τ ∈ { −d+ 1 , . . . , t− 1 } ( incremental ) y ( j ) t−1 ∼ N ( 0 , 1 ) for each j ∈ [ M ] 6 : Update bonus ensembles for each j ∈ [ M ] , w ( j ) t : = arg min w t−1∑ τ=−d+1 ( 〈xτ , aτ , w〉 − y ( j ) τ ) 2 7 : Compute per-arm bonuses bonust ( a ) : = β ·maxj∈ [ M ] ∣∣∣〈xt , a , w ( j ) t 〉∣∣∣ for each a ∈ At 8 : Choose action at : = arg maxa∈At〈xt , a , θ̂t〉+ bonust ( a ) 9 : Observe reward rt ( at ) = 〈xt , at , θ∗〉+ εt

Summary: This paper studies using bonus for guiding exploration in reinforcement learning with large action spaces. The paper focuses on RL algorithms using LinUCB-style exploration bonuses. LinUCB, developed originally for stochastic linear bandits, has been a theoretically successful algorithm for problems admitting a linear reward structure. However, using such bonuses entails computing a matrix inversion, which may not be tractable in high dimensional problems.   The main contribution of the paper is to propose Anti-concentrated Confidence Bounds (ACB) with the aim of efficiently approximating the LinUCB bonus without matrix inversions. The main idea of ACB is that it maintains $M$ linear regressors trained to predict i.i.d. noise drawn from $\mathcal N(0,1)$. The algorithm then set the bonus proportional to the maximum deviation over the regressors from the mean. Two versions are considered: one for linear bandits and one for deep RL. For linear bandits, the authors prove a regret of $\tilde O(d\sqrt{T\log A})$ with high probability under mild assumptions (e.g., logarithmically in $T$ many regressors). The variant for deep RL, however, is examined through numerical experiments where the algorithm is shown to be competitive with state-of-the-art.