Regret Bounds and Reinforcement Learning Exploration of EXP-based Algorithms

1 INTRODUCTION . Multi-armed bandit ( MAB ) is to maximize cumulative reward of a player throughout a bandit game by choosing different arms at each time step . It is also equivalent to minimizing the regret defined as the difference between the best rewards that can be achieved and the actual reward gained by the player . Formally , given time horizon T , in time step t ≤ T the player choose one arm at among K arms , receives rtat among rewards r t = ( rt1 , r t 2 , . . . , r t K ) , and maximizes the total reward ∑T t=1 r t at or minimizes the regret . Computationally efficient and with abundant theoretical analyses are the EXP-type MAB algorithms . In EXP3.P , each arm has a trust coefficient ( weight ) . The player samples each arm with probability being the sum of its normalized weights and a bias term , receives reward of the sampled arm and exponentially updates the weights based on the corresponding reward estimates . It achieves the regret of the order O ( √ T ) in a high probability sense . In EXP4 , there are any number of experts . Each has a sample rule over actions and a weight . The player samples according to the weighted average of experts ’ sample rules and updates the weights respectively . Contextual bandit is a variant of MAB by adding context or state space S. At time step t , the player has context st ∈ S with s1 : T = ( s1 , s2 , . . . , sT ) being independent . Rewards rt follow F ( µ ( st ) ) where F is any distribution and µ ( st ) is the mean vector that depends on state st. Reinforcement Learning ( RL ) generalizes contextual bandit , where state and reward transitions follow a Markov Decision Process ( MDP ) represented by transition kernel P ( st+1 , rt|at , st ) . A key challenge in RL is the trade-off between exploration and exploitation . Exploration is to encourage the player to try new arms in MAB or new actions in RL to understand the game better . It helps to plan for the future , but with the sacrifice of potentially lowering the current reward . Exploitation aims to exploit currently known states and arms to maximize the current reward , but it potentially prevents the player to gain more information to increase local reward . To maximize the cumulative reward , the player needs to know the game by exploration , while guaranteeing current reward by exploitation . How to incentivize exploration in RL has been a main focus in RL . Since RL is built on MAB , it is natural to extend MAB techniques to RL and UCB is such a success . UCB ( Auer et al . ( 2002a ) ) motivates count-based exploration ( Strehl and Littman , 2008 ) in RL and the subsequent PseudoCount exploration ( Bellemare et al. , 2016 ) . New deep RL exploration algorithms have been recently proposed . Using deep neural networks to keep track of the Q-values by means of Q-networks in RL is called DQN ( Mnih et al . ( 2013 ) ) . This combination of deep learning and RL has shown great success . -greedy in Mnih et al . ( 2015 ) is a simple exploration technique using DQN . Besides -greedy , intrinsic model exploration computes intrinsic rewards by focusing on experiences . Intrinsic rewards directly measure and incentivize exploration if added to extrinsic ( actual ) rewards of RL , e.g . DORA ( Fox et al. , 2018 ) and ( Stadie et al. , 2015 ) . Random Network Distillation ( RND ) ( Burda et al. , 2018 ) is a more recent suggestion relying on a fixed target network . A drawback of RND is its local focus without global exploration . In order to address weak points of these various exploration algorithms in the RL context , the notion of experts is natural and thus EXP-type MAB algorithms are appropriate . The allowance of arbitrary experts provides exploration for harder contextual bandits and hence providing exploration possibilities for RL . We develop an EXP4 exploration algorithm for RL that relies on several general experts . This is the first RL algorithm using several exploration experts enabling global exploration . Focusing on DQN , in the computational study we focus on two agents consisting of RND and -greedy DQN . We implement the RL EXP4 algorithm on the hard-to-explore RL game Montezuma ’ s Revenge and compare it with the benchmark algorithm RND ( Burda et al . ( 2018 ) ) . The numerical results show that the algorithm gains more exploration than RND and it gains the ability of global exploration by not getting stuck in local maximums of RND . Its total reward also increases with training . Overall , our algorithm improves exploration and exploitation on the benchmark game and demonstrates a learning process in RL . Reward in RL in many cases is unbounded which relates to unbounded MAB rewards . There are three major versions of MAB : Adversarial , Stochastic , and herein introduced Gaussian . For adversarial MAB , rewards of theK arms rt can be chosen arbitrarily by adversaries at step t. For stochastic MAB , the rewards at different steps are assumed to be i.i.d . and the rewards across arms are independent . It is assumed that 0 ≤ rti ≤ 1 for any arm i and step t. For Gaussian MAB , rewards rt follow multi-variate normal N ( µ , Σ ) with µ being the mean vector and Σ the covariance matrix of the K arms . Here the rewards are neither bounded , nor independent among the arms . For this reason the introduced Gaussian MAB reflects the RL setting and is the subject of our MAB analyses of EXP3.P . EXP-type algorithms ( Auer et al . ( 2002b ) ) are optimal in the two classical MABs . Auer et al . ( 2002b ) show lower and upper bounds on regret of the order O ( √ T ) for adversarial MAB and of the order O ( log ( T ) ) for stochastic MAB . All of the proofs of these regret bounds by EXP-type algorithms are based on the bounded reward assumption , which does not hold for Gaussian MAB . Therefore , the regret bounds for Gaussian MAB with unbounded rewards studied herein are significantly different from prior works . We show both lower and upper bounds on regret of Gaussian MAB under certain assumptions . Some analyses even hold for more generally distributed MAB . Upper bounds borrow some ideas from the analysis of the EXP3.P algorithm in Auer et al . ( 2002b ) for bounded MAB to our unbounded MAB , while lower bounds are by our brand new construction of instances . Precisely , we derive lower bounds of order Ω ( T ) for certain fixed T and upper bounds of order O∗ ( √ T ) for T being large enough . The question of bounds for any value of T remains open . The main contributions of this work are as follows . On the analytical side we introduce Gaussian MAB with the unique aspect and challenge of unbounded rewards . We provide the very first regret lower bound in such a case by constructing a novel family of Gaussian bandits and we are able to analyze the EXP3.P algorithm for Gaussian MAB . Unbounded reward poses a non-trivial challenge in the analyses . We also provide the very first extension of EXP4 to RL exploration . We show its superior performance on two hard-to-explore RL games . A literature review is provided in Section 2 . Then in Section 3 we exhibit upper bounds for unbounded MAB of the EXP3.P algorithm and lower bounds , respectively . Section 4 discusses the EXP4 algorithm for RL exploration . Finally , in Section 5 , we present numerical results related to the proposed algorithm . 2 LITERATURE REVIEW . The importance of exploration in RL is well understood . Count-based exploration in RL relies on UCB . Strehl and Littman ( 2008 ) develop Bellman value iteration V ( s ) = maxa R̂ ( s , a ) + γE [ V ( s′ ) ] + βN ( s , a ) − 1 2 , where N ( s , a ) is the number of visits to ( s , a ) for state s and action a . Value N ( s , a ) − 1 2 is positively correlated with curiosity of ( s , a ) and encourages exploration . This method is limited to tableau model-based MDP for small state spaces , while Bellemare et al . ( 2016 ) introduce Pseudo-Count exploration for non-tableau MDP with density models . In conjunction with DQN , -greedy in Mnih et al . ( 2015 ) is a simple exploration technique using DQN . Besides -greedy , intrinsic model exploration computes intrinsic rewards by the accuracy of a model trained on experiences . Intrinsic rewards directly measure and incentivize exploration if added to extrinsic ( actual ) rewards of RL , e.g . DORA in Fox et al . ( 2018 ) and Stadie et al . ( 2015 ) . Intrinsic rewards in Stadie et al . ( 2015 ) are defined as e ( s , a ) = ||σ ( s′ ) −Mφ ( σ ( s ) , a ) ||22 where Mφ is a parametric model , s′ is the next state and σ is input extraction . Intrinsic reward e ( s , a ) relies on stochastic transition from s to s′ and brings noise to exploration . Random Network Distillation ( RND ) in Burda et al . ( 2018 ) addresses this by defining e ( s , a ) = ||f̂ ( s′ ) − f ( s′ ) ||22 where f̂ is a parametric model and f is a randomly initialized but fixed model . Here e ( s , a ) , independent of the transition , only depends on state s′ and drives RND to outperform other algorithms on Montezuma ’ s Revenge . None of these algorithms use several experts which is a significant departure from our work . In terms of MAB regret analyses focusing on EXP-type algorithms , Auer et al . ( 2002b ) first introduce EXP3.P for bounded adversarial MAB and EXP4 for contextual bandits . Under the EXP3.P algorithm , an upper bound on regret of the order O ( √ T ) is achieved , which has no gap with the lower bound and hence it establishes that EXP3.P is optimal . However these regret bounds are not applicable to Gaussian MAB since rewards can be infinite . Meanwhile for unbounded MAB , Srinivas et al . ( 2010 ) demonstrate a regret bound of order O ( √ T · γT ) for noisy Gaussian process bandits where a reward observation contains noise . The information gain γT is not well-defined in a noiseless Gaussian setting . For noiseless Gaussian bandits , Grünewälder et al . ( 2010 ) show both the optimal lower and upper bounds on regret , but the regret definition is not consistent with the one used in Auer et al . ( 2002b ) . We establish a lower bound of the order Ω ( T ) for certain T and an upper bound of the order O∗ ( √ T ) asymptotically on regret of unbounded noiseless Gaussian MAB following standard definitions of regret .

Summary: The authors consider analyzing the EXP3.P algorithm for the case of unbounded reward functions, in the sense that the rewards are governed by a Gaussian distribution. The authors first demonstrate a regret lower bound result on the Gaussian MABs when the time horizon is bounded from above. Then, the authors proceed to the analysis of the EXP3.P algorithm on the Gaussian MABs, and establish a regret bound similar to that of Auer et al. 2002. Finally, the authors apply the EXP3.P, where an expert corresponds to a Q-learning network, in the EXP4-RL algorithm, and evaluate it on multiple RL instances.