Q-learning with UCB Exploration is Sample Efficient for Infinite-Horizon MDP

A fundamental question in reinforcement learning is whether model-free algorithms are sample efficient . Recently , Jin et al . ( 2018 ) proposed a Q-learning algorithm with UCB exploration policy , and proved it has nearly optimal regret bound for finite-horizon episodic MDP . In this paper , we adapt Q-learning with UCB-exploration bonus to infinite-horizon MDP with discounted rewards without accessing a generative model . We show that the sample complexity of exploration of our algorithm is bounded by Õ ( SA 2 ( 1−γ ) 7 ) . This improves the previously best known result of Õ ( SA 4 ( 1−γ ) 8 ) in this setting achieved by delayed Q-learning ( Strehl et al. , 2006 ) , and matches the lower bound in terms of as well as S and A up to logarithmic factors . 1 INTRODUCTION . The goal of reinforcement learning ( RL ) is to construct efficient algorithms that learn and plan in sequential decision making tasks when the underlying system dynamics are unknown . A typical model in RL is Markov Decision Process ( MDP ) . At each time step , the environment is in a state s. The agent takes an action a , obtain a reward r , and then the environment transits to another state . In reinforcement learning , the transition probability distribution is unknown . The algorithm needs to learn the transition dynamics of MDP , while aiming to maximize the cumulative reward . This poses the exploration-exploitation dilemma : whether to act to gain new information ( explore ) or to act consistently with past experience to maximize reward ( exploit ) . Theoretical analyses of reinforcement learning fall into two broad categories : those assuming a simulator ( a.k.a . generative model ) , and those without a simulator . In the first category , the algorithm is allowed to query the outcome of any state action pair from an oracle . The emphasis is on the number of calls needed to estimate the Q value or to output a near-optimal policy . There has been extensive research in literature following this line of research , the majority of which focuses on discounted infinite horizon MDPs ( Azar et al. , 2011 ; Even-Dar & Mansour , 2003 ; Sidford et al. , 2018b ) . The current results have achieved near-optimal time and sample complexities ( Sidford et al. , 2018b ; a ) . ∗These two authors contributed equally . Without a simulator , there is a dichotomy between finite-horizon and infinite-horizon settings . In finite-horizon settings , there are straightforward definitions for both regret and sample complexity ; the latter is defined as the number of samples needed before the policy becomes near optimal . In this setting , extensive research in the past decade ( Jin et al. , 2018 ; Azar et al. , 2017 ; Jaksch et al. , 2010 ; Dann et al. , 2017 ) has achieved great progress , and established nearly-tight bounds for both regret and sample complexity . The infinite-horizon setting is a very different matter . First of all , the performance measure can not be a straightforward extension of the sample complexity defined above ( See Strehl & Littman ( 2008 ) for detailed discussion ) . Instead , the measure of sample efficiency we adopt is the so-called sample complexity of exploration ( Kakade et al. , 2003 ) , which is also a widely-accepted definition . This measure counts the number of times that the algorithm “ makes mistakes ” along the whole trajectory . See also ( Strehl & Littman , 2008 ) for further discussions regarding this issue . Several model based algorithms have been proposed for infinite horizon MDP , for example Rmax ( Brafman & Tennenholtz , 2003 ) , MoRmax ( Szita & Szepesvári , 2010 ) and UCRL-γ ( Lattimore & Hutter , 2012 ) . It is noteworthy that there still exists a considerable gap between the state-of-the-art algorithm and the theoretical lower bound ( Lattimore & Hutter , 2012 ) regarding 1/ ( 1− γ ) factor . Though model-based algorithms have been proved to be sample efficient in various MDP settings , most state-of-the-art RL algorithms are developed in the model-free paradigm ( Schulman et al. , 2015 ; Mnih et al. , 2013 ; 2016 ) . Model-free algorithms are more flexible and require less space , which have achieved remarkable performance on benchmarks such as Atari games and simulated robot control problems . For infinite horizon MDPs without access to simulator , the best model-free algorithm has a sample complexity of exploration Õ ( SA 4 ( 1−γ ) 8 ) , achieved by delayed Q-learning ( Strehl et al. , 2006 ) . The authors provide a novel strategy of argument when proving the upper bound for the sample complexity of exploration , namely identifying a sufficient condition for optimality , and then bound the number of times that this condition is violated . However , the results of Delayed Q-learning still leave a quadratic gap in 1/ from the best-known lower bound . This is partly because the updates in Q-value are made in an over-conservative way . In fact , the loose sample complexity bound is a result of delayed Q-learning algorithm itself , as well as the mathematical artifact in their analysis . To illustrate this , we construct a hard instance showing that Delayed Q-learning incurs Ω ( 1/ 3 ) sample complexity . This observation , as well as the success of the Q-learning with UCB algorithm ( Jin et al. , 2018 ) in proving a regret bound in finite-horizon settings , motivates us to incorporate a UCB-like exploration term into our algorithm . In this work , we propose a Q-learning algorithm with UCB exploration policy . We show the sample complexity of exploration bound of our algorithm is Õ ( SA 2 ( 1−γ ) 7 ) . This strictly improves the previous best known result due to Delayed Q-learning . It also matches the lower bound in the dependence on , S and A up to logarithmic factors . We point out here that the infinite-horizon setting can not be solved by reducing to finite-horizon setting . There are key technical differences between these two settings : the definition of sample complexity of exploration , time-invariant policies and the error propagation structure in Q-learning . In particular , the analysis techniques developed in ( Jin et al. , 2018 ) do not directly apply here . We refer the readers to Section 3.2 for detailed explanations and a concrete example . The rest of the paper is organized as follows . After introducing the notation used in the paper in Section 2 , we describe our infinite Q-learning with UCB algorithm in Section 3 . We then state our main theoretical results , which are in the form of PAC sample complexity bounds . In Section 4 we present some interesting properties beyond sample complexity bound . Finally , we conclude the paper in Section 5 . 2 PRELIMINARY . We consider a Markov Decision Process defined by a five tuple 〈S , A , p , r , γ〉 , where S is the state space , A is the action space , p ( s′|s , a ) is the transition function , r : S×A → [ 0 , 1 ] is the deterministic reward function , and 0 ≤ γ < 1 is the discount factor for rewards . Let S = |S| and A = |A| denote the number of states and the number of actions respectively . Starting from a state s1 , the agent interacts with the environment for infinite number of time steps . At each time step , the agent observes state st ∈ S , picks action at ∈ A , and receives reward rt ; the system then transits to next state st+1 . Using the notations in Strehl et al . ( 2006 ) , a policy πt refers to the non-stationary control policy of the algorithm since step t. We use V πt ( s ) to denote the value function under policy πt , which is defined as V πt ( s ) = E [ ∑∞ i=1 γ i−1r ( si , πt+i−1 ( si ) ) |s1 = s ] . We also use V ∗ ( s ) = supπ V π ( s ) to denote the value function of the optimal policy . Accordingly , we define Qπt ( s , a ) = r ( s , a ) + E [ ∑∞ i=2 γ i−1r ( si , πt+i−1 ( si ) ) |s1 = s , a1 = a ] as the Q function under policy πt ; Q∗ ( s , a ) is the Q function under optimal policy π∗ . We use the sample complexity of exploration defined in Kakade et al . ( 2003 ) to measure the learning efficiency of our algorithm . This sample complexity definition has been widely used in previous works Strehl et al . ( 2006 ) ; Lattimore & Hutter ( 2012 ) ; Strehl & Littman ( 2008 ) . Definition 1 . Sample complexity of Exploration of an algorithm ALG is defined as the number of time steps t such that the non-stationary policy πt at time t is not -optimal for current state st , i.e . V πt ( st ) < V ∗ ( st ) − . Roughly speaking , this measure counts the number of mistakes along the whole trajectory . We use the following definition of PAC-MDP Strehl et al . ( 2006 ) . Definition 2 . An algorithm ALG is said to be PAC-MDP ( Probably Approximately Correct in Markov Decision Processes ) if , for any and δ , the sample complexity of ALG is less than some polynomial in the relevant quantities ( S , A , 1/ , 1/δ , 1/ ( 1− γ ) ) , with probability at least 1− δ . Finally , recall that Bellman equation is defined as the following : { V πt ( s ) = Qπt ( s , πt ( s ) ) Qπt ( s , a ) : = ( rt + γPV πt+1 ) ( s , a ) , { V ∗ ( s ) = Q∗ ( s , π∗ ( s ) ) Q∗ ( s , a ) : = ( rt + γPV ∗ ) ( s , a ) , which is frequently used in our analysis . Here we denote [ PV πt ] ( s , a ) : = Es′∼p ( ·|s , a ) V πt+1 ( s′ ) . 3 MAIN RESULTS . In this section , we present the UCB Q-learning algorithm and the sample complexity bound . 3.1 ALGORITHM . Algorithm 1 Infinite Q-learning with UCB Parameters : , γ , δ Initialize Q ( s , a ) , Q̂ ( s , a ) ← 11−γ , N ( s , a ) ← 0 , 1 ← 24RM ln 11−γ , H ← ln 1/ ( ( 1−γ ) 1 ) ln 1/γ . Define ι ( k ) = ln ( SA ( k + 1 ) ( k + 2 ) /δ ) , αk = H+1H+k . for t = 1 , 2 , ... do 5 : Take action at ← arg maxa′ Q̂ ( st , a′ ) Receive reward rt and transit to st+1 N ( st , at ) ← N ( st , at ) + 1 k ← N ( st , at ) , bk ← c21−γ √ Hι ( k ) k . c2 is a constant and can be set to 4 √ 2 V̂ ( st+1 ) ← maxa∈A Q̂ ( st+1 , a ) 10 : Q ( st , at ) ← ( 1− αk ) Q ( st , at ) + αk [ r ( st , at ) + bk + γV̂ ( st+1 ) ] Q̂ ( st , at ) ← min ( Q̂ ( st , at ) , Q ( st , at ) ) end for Here c2 = 4 √ 2 is a constant . R = dln 3 ( 1−γ ) / ( 1 − γ ) e , while the choice of M can be found in Section . 3.3 . ( M = O ( ln 1/ ( ( 1− γ ) ) ) ) . The learning rate is defined as αk = ( H + 1 ) / ( H + k ) . H is chosen as ln 1/ ( ( 1−γ ) 1 ) ln 1/γ , which satisfies H ≤ ln 1/ ( ( 1−γ ) 1 ) 1−γ . Our UCB Q-learning algorithm ( Algorithm 1 ) maintains an optimistic estimation of action value function Q ( s , a ) and its historical minimum value Q̂ ( s , a ) . Nt ( s , a ) denotes the number of times that ( s , a ) is experienced before time step t ; τ ( s , a , k ) denotes the time step t at which ( st , at ) = ( s , a ) for the k-th time ; if this state-action pair is not visited that many times , τ ( s , a , k ) = ∞ . Qt ( s , a ) and Q̂t ( s , a ) denotes the Q and Q̂ value of ( s , a ) that the algorithm maintains when arriving at st respectively .

Summary: In this paper, the authors extend the UCB Q-learning algorithm by Jin et al. (2018) to infinite horizon discounted MDPs, and prove a PAC bound of \tilde{O}(SA/\epsilon^2 (1-\gamma)^7) for the resulting algorithm. This bound improves the one for delayed Q-learning by Strehl et al. (2006) and matches the lower-bound in terms of \epsilon, S, and A.