Regret Bounds and Reinforcement Learning Exploration of EXP-based Algorithms

1 INTRODUCTION . Multi-armed bandit ( MAB ) is to maximize cumulative reward of a player throughout a bandit game by choosing different arms at each time step . It is also equivalent to minimizing the regret defined as the difference between the best rewards that can be achieved and the actual reward gained by the player . Formally , given time horizon T , in time step t ≤ T the player choose one arm at among K arms , receives rtat among rewards r t = ( rt1 , r t 2 , . . . , r t K ) , and maximizes the total reward ∑T t=1 r t at or minimizes the regret . Computationally efficient and with abundant theoretical analyses are the EXP-type MAB algorithms . In EXP3.P , each arm has a trust coefficient ( weight ) . The player samples each arm with probability being the sum of its normalized weights and a bias term , receives reward of the sampled arm and exponentially updates the weights based on the corresponding reward estimates . It achieves the regret of the order O ( √ T ) in a high probability sense . In EXP4 , there are any number of experts . Each has a sample rule over actions and a weight . The player samples according to the weighted average of experts ’ sample rules and updates the weights respectively . Contextual bandit is a variant of MAB by adding context or state space S. At time step t , the player has context st ∈ S with s1 : T = ( s1 , s2 , . . . , sT ) being independent . Rewards rt follow F ( µ ( st ) ) where F is any distribution and µ ( st ) is the mean vector that depends on state st. Reinforcement Learning ( RL ) generalizes contextual bandit , where state and reward transitions follow a Markov Decision Process ( MDP ) represented by transition kernel P ( st+1 , rt|at , st ) . A key challenge in RL is the trade-off between exploration and exploitation . Exploration is to encourage the player to try new arms in MAB or new actions in RL to understand the game better . It helps to plan for the future , but with the sacrifice of potentially lowering the current reward . Exploitation aims to exploit currently known states and arms to maximize the current reward , but it potentially prevents the player to gain more information to increase local reward . To maximize the cumulative reward , the player needs to know the game by exploration , while guaranteeing current reward by exploitation . How to incentivize exploration in RL has been a main focus in RL . Since RL is built on MAB , it is natural to extend MAB techniques to RL and UCB is such a success . UCB ( Auer et al . ( 2002a ) ) motivates count-based exploration ( Strehl and Littman , 2008 ) in RL and the subsequent PseudoCount exploration ( Bellemare et al. , 2016 ) . New deep RL exploration algorithms have been recently proposed . Using deep neural networks to keep track of the Q-values by means of Q-networks in RL is called DQN ( Mnih et al . ( 2013 ) ) . This combination of deep learning and RL has shown great success . -greedy in Mnih et al . ( 2015 ) is a simple exploration technique using DQN . Besides -greedy , intrinsic model exploration computes intrinsic rewards by focusing on experiences . Intrinsic rewards directly measure and incentivize exploration if added to extrinsic ( actual ) rewards of RL , e.g . DORA ( Fox et al. , 2018 ) and ( Stadie et al. , 2015 ) . Random Network Distillation ( RND ) ( Burda et al. , 2018 ) is a more recent suggestion relying on a fixed target network . A drawback of RND is its local focus without global exploration . In order to address weak points of these various exploration algorithms in the RL context , the notion of experts is natural and thus EXP-type MAB algorithms are appropriate . The allowance of arbitrary experts provides exploration for harder contextual bandits and hence providing exploration possibilities for RL . We develop an EXP4 exploration algorithm for RL that relies on several general experts . This is the first RL algorithm using several exploration experts enabling global exploration . Focusing on DQN , in the computational study we focus on two agents consisting of RND and -greedy DQN . We implement the RL EXP4 algorithm on the hard-to-explore RL game Montezuma ’ s Revenge and compare it with the benchmark algorithm RND ( Burda et al . ( 2018 ) ) . The numerical results show that the algorithm gains more exploration than RND and it gains the ability of global exploration by not getting stuck in local maximums of RND . Its total reward also increases with training . Overall , our algorithm improves exploration and exploitation on the benchmark game and demonstrates a learning process in RL . Reward in RL in many cases is unbounded which relates to unbounded MAB rewards . There are three major versions of MAB : Adversarial , Stochastic , and herein introduced Gaussian . For adversarial MAB , rewards of theK arms rt can be chosen arbitrarily by adversaries at step t. For stochastic MAB , the rewards at different steps are assumed to be i.i.d . and the rewards across arms are independent . It is assumed that 0 ≤ rti ≤ 1 for any arm i and step t. For Gaussian MAB , rewards rt follow multi-variate normal N ( µ , Σ ) with µ being the mean vector and Σ the covariance matrix of the K arms . Here the rewards are neither bounded , nor independent among the arms . For this reason the introduced Gaussian MAB reflects the RL setting and is the subject of our MAB analyses of EXP3.P . EXP-type algorithms ( Auer et al . ( 2002b ) ) are optimal in the two classical MABs . Auer et al . ( 2002b ) show lower and upper bounds on regret of the order O ( √ T ) for adversarial MAB and of the order O ( log ( T ) ) for stochastic MAB . All of the proofs of these regret bounds by EXP-type algorithms are based on the bounded reward assumption , which does not hold for Gaussian MAB . Therefore , the regret bounds for Gaussian MAB with unbounded rewards studied herein are significantly different from prior works . We show both lower and upper bounds on regret of Gaussian MAB under certain assumptions . Some analyses even hold for more generally distributed MAB . Upper bounds borrow some ideas from the analysis of the EXP3.P algorithm in Auer et al . ( 2002b ) for bounded MAB to our unbounded MAB , while lower bounds are by our brand new construction of instances . Precisely , we derive lower bounds of order Ω ( T ) for certain fixed T and upper bounds of order O∗ ( √ T ) for T being large enough . The question of bounds for any value of T remains open . The main contributions of this work are as follows . On the analytical side we introduce Gaussian MAB with the unique aspect and challenge of unbounded rewards . We provide the very first regret lower bound in such a case by constructing a novel family of Gaussian bandits and we are able to analyze the EXP3.P algorithm for Gaussian MAB . Unbounded reward poses a non-trivial challenge in the analyses . We also provide the very first extension of EXP4 to RL exploration . We show its superior performance on two hard-to-explore RL games . A literature review is provided in Section 2 . Then in Section 3 we exhibit upper bounds for unbounded MAB of the EXP3.P algorithm and lower bounds , respectively . Section 4 discusses the EXP4 algorithm for RL exploration . Finally , in Section 5 , we present numerical results related to the proposed algorithm . 2 LITERATURE REVIEW . The importance of exploration in RL is well understood . Count-based exploration in RL relies on UCB . Strehl and Littman ( 2008 ) develop Bellman value iteration V ( s ) = maxa R̂ ( s , a ) + γE [ V ( s′ ) ] + βN ( s , a ) − 1 2 , where N ( s , a ) is the number of visits to ( s , a ) for state s and action a . Value N ( s , a ) − 1 2 is positively correlated with curiosity of ( s , a ) and encourages exploration . This method is limited to tableau model-based MDP for small state spaces , while Bellemare et al . ( 2016 ) introduce Pseudo-Count exploration for non-tableau MDP with density models . In conjunction with DQN , -greedy in Mnih et al . ( 2015 ) is a simple exploration technique using DQN . Besides -greedy , intrinsic model exploration computes intrinsic rewards by the accuracy of a model trained on experiences . Intrinsic rewards directly measure and incentivize exploration if added to extrinsic ( actual ) rewards of RL , e.g . DORA in Fox et al . ( 2018 ) and Stadie et al . ( 2015 ) . Intrinsic rewards in Stadie et al . ( 2015 ) are defined as e ( s , a ) = ||σ ( s′ ) −Mφ ( σ ( s ) , a ) ||22 where Mφ is a parametric model , s′ is the next state and σ is input extraction . Intrinsic reward e ( s , a ) relies on stochastic transition from s to s′ and brings noise to exploration . Random Network Distillation ( RND ) in Burda et al . ( 2018 ) addresses this by defining e ( s , a ) = ||f̂ ( s′ ) − f ( s′ ) ||22 where f̂ is a parametric model and f is a randomly initialized but fixed model . Here e ( s , a ) , independent of the transition , only depends on state s′ and drives RND to outperform other algorithms on Montezuma ’ s Revenge . None of these algorithms use several experts which is a significant departure from our work . In terms of MAB regret analyses focusing on EXP-type algorithms , Auer et al . ( 2002b ) first introduce EXP3.P for bounded adversarial MAB and EXP4 for contextual bandits . Under the EXP3.P algorithm , an upper bound on regret of the order O ( √ T ) is achieved , which has no gap with the lower bound and hence it establishes that EXP3.P is optimal . However these regret bounds are not applicable to Gaussian MAB since rewards can be infinite . Meanwhile for unbounded MAB , Srinivas et al . ( 2010 ) demonstrate a regret bound of order O ( √ T · γT ) for noisy Gaussian process bandits where a reward observation contains noise . The information gain γT is not well-defined in a noiseless Gaussian setting . For noiseless Gaussian bandits , Grünewälder et al . ( 2010 ) show both the optimal lower and upper bounds on regret , but the regret definition is not consistent with the one used in Auer et al . ( 2002b ) . We establish a lower bound of the order Ω ( T ) for certain T and an upper bound of the order O∗ ( √ T ) asymptotically on regret of unbounded noiseless Gaussian MAB following standard definitions of regret .

Summary: This paper contributes to the study of EXP-based algorithms in two aspects. One is on the theoretical aspect: It analyzes the lower and upper bounds of EXP-3 for Gaussian multi-bandit setting for which the reward can be unbounded. The other is on the empirical aspect: It applied EXP4, originally developed for MAB, to Rl applications and demonstrated its advanced empirical performance.