Nearly Minimax Optimal Reinforcement Learning for Discounted MDPs

) regret , where S is the number of states , A is the number of actions , γ is the discount factor and T is the number of steps . In addition , we construct a class of hard MDPs and show that for any algorithm , the expected regret is at least Ω̃ ( √ SAT/ ( 1− γ ) 1.5 ) . Our upper bound matches the minimax lower bound up to logarithmic factors , which suggests that UCBVI-γ is nearly minimax optimal for discounted MDPs . 1 Introduction . The goal of reinforcement learning ( RL ) is designing algorithms to learn the optimal policy through interactions with the unknown dynamic environment . Markov decision process ( MDPs ) plays a central role in reinforcement learning due to their ability to describe the time-independent state transition property . More specifically , the discounted MDP is one of the standard MDPs in reinforcement learning to describe sequential tasks without interruption or restart . For discounted MDPs , with a generative model [ 12 ] , several algorithms with near-optimal sample complexity have been proposed . More specifically , Azar et al . [ 3 ] proposed an Empirical QVI algorithm which achieves the optimal sample complexity to find the optimal value function . Sidford et al . [ 22 ] proposed a sublinear randomized value iteration algorithm that achieves a near-optimal sample complexity to find the optimal policy , and Sidford et al . [ 23 ] further improved it to reach the optimal sample complexity . Since generative model is a powerful oracle that allows the algorithm to query the reward function and the next state for any state-action pair ( s , a ) , it is natural to ask whether there exist online RL algorithms ( without generative model ) that achieve optimality . To measure an online RL algorithm , a widely used notion is regret , which is defined as the summation of sub-optimality gaps over time steps . The regret is firstly introduced for episodic and infinitehorizon average-reward MDPs and later extended to discounted MDPs by [ 15 , 30 , 35 , 35 ] . Liu and Su [ 15 ] proposed a double Q-learning algorithm with the UCB exploration ( Double Q-learning ) , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . which enjoys Õ ( √ SAT/ ( 1 − γ ) 2.5 ) regret , where S is the number of states , A is the number of actions , γ is the discount factor and T is the number of steps . While Double Q-learning enjoys a standard √ T -regret , it still does not match the lower bound proved in [ 15 ] in terms of the dependence on S , A and 1/ ( 1 − γ ) . Recently , Zhou et al . [ 34 ] proposed a UCLK+ algorithm for discounted MDPs under the linear mixture MDP assumption and achieved Õ ( d √ T/ ( 1− γ ) 1.5 ) regret , where d is the dimension of the feature mapping . However , directly applying their algorithm to our setting would yield an Õ ( S2A √ T/ ( 1− γ ) 1.5 ) regret1 , which is even worse that of double Q-learning [ 15 ] in terms of the dependence on S , A . In this paper , we aim to close this gap by designing a practical algorithm with a nearly optimal regret . In particular , we propose a model-based algorithm named UCBVI-γ for discounted MDPs without using the generative model . At the core of our algorithm is to use a “ refined ” Bernstein-type bonus and the law of total variance [ 3 , 4 ] , which together can provide tighter upper confidence bound ( UCB ) . Our contributions are summarized as follows : • We propose a model-based algorithm UCBVI-γ to learn the optimal value function under the discounted MDP setting . We show that the regret of UCBVI-γ in first T steps is upper bounded by Õ ( √ SAT/ ( 1− γ ) 1.5 ) . Our regret bound strictly improves the best existing regret Õ ( √ SAT/ ( 1− γ ) 2.5 ) 2 in [ 15 ] by a factor of ( 1− γ ) −1 . • We also prove a lower bound of the regret by constructing a class of hard-to-learn discounted MDPs , which can be regarded as a chain of the hard MDPs considered in [ 15 ] . We show that for any algorithm , its regret in the first T steps can not be lower than Ω̃ ( √ SAT/ ( 1− γ ) 1.5 ) on the constructed MDP . This lower bound also strictly improves the lower bound Ω ( √ SAT/ ( 1− γ ) +√ AT/ ( 1− γ ) 1.5 ) proved by [ 15 ] . • The nearly matching upper and the lower bounds together suggest that the proposed UCBVI-γ algorithm is minimax-optimal up to logarithmic factors . We compare the regret of UCBVI-γ with previous online algorithms for learning discounted MDPs in Table 1 . Notation For any positive integer n , we denote by [ n ] the set { 1 , . . . , n } . For any two numbers a and b , we denote by a ∨ b as the shorthand for max ( a , b ) . For two sequences { an } and { bn } , we write an = O ( bn ) if there exists an absolute constant C such that an ≤ Cbn , and we write an = Ω ( bn ) if there exists an absolute constant C such that an ≥ Cbn . We use Õ ( · ) and Ω̃ ( · ) to further hide the logarithmic factors . 2 Related Work . Model-free Algorithms for Discounted MDPs . A large amount of reinforcement learning algorithms like Q-learning can be regarded as model-free algorithms . These algorithms directly learn the action-value function by updating the values of each state-action pair . Kearns and Singh [ 13 ] firstly proposed a phased Q-Learning which learns an -optimal policy with Õ ( SA/ ( ( 1− γ ) 7 2 ) ) sample complexity for ≤ 1/ ( 1 − γ ) . Later on , Strehl et al . [ 25 ] proposed a delay-Q-learning algorithm , which achieves Õ ( SA/ ( ( 1− γ ) 8 4 ) ) sample complexity of exploration . Wang [ 29 ] proposed a randomized primal-dual method algorithm , which improves the sample complexity to Õ ( SA/ ( ( 1− γ ) 4 2 ) ) for ≤ 1/ ( 1− γ ) under the ergodicity assumption . Later , Sidford et al . [ 23 ] proposed a sublinear randomized value iteration algorithm and achieved Õ ( SA/ ( ( 1− γ ) 4 2 ) ) sample complexity for ≤ 1 . Sidford et al . [ 22 ] further improved the empirical QVI algorithm and proposed a variance-reduced QVI algorithm , which improves the sample complexity to Õ ( SA/ ( ( 1− γ ) 3 2 ) ) for ≤ 1 . Wainwright [ 28 ] proposed a variance-reduced Q-learning algorithm , which is an extension of the Q-learning algorithm and achieves Õ ( SA/ ( ( 1− γ ) 3 2 ) ) sample complexity . In addition , Dong 1Linear mixture MDP assumes that there exists a feature mapping φ ( s′|s , a ) ∈ Rd and a vector θ ∈ Rd such that P ( s′|s , a ) = 〈φ ( s′|s , a ) , θ〉 . It can be verified that any MDP is automatically a linear mixture MDP with a SA-dimensional feature mapping [ 2 , 35 ] . 2The regret definition in [ 15 ] differs from our definition by a factor of ( 1− γ ) −1 . Here we translate their regret from their definition to our definition for a fair comparison . A detailed comparison can be found in Appendix . et al . [ 9 ] proposed an infinite Q-learning with UCB and improved the sample complexity of exploration to Õ ( SA/ ( ( 1− γ ) 7 2 ) ) . Zhang et al . [ 33 ] proposed a UCB-multistage algorithm which attains the Õ ( SA/ ( ( 1− γ ) 5.5 2 ) ) sample complexity of exploration , and proposed a UCB-multistage-adv algorithm which attains a better sample complexity Õ ( SA/ ( ( 1− γ ) 3 2 ) ) in the high accuracy regime . Recently , Liu and Su [ 15 ] focused on regret minimization for the infinite-horizon discounted MDP and showed the connection between regret and sample complexity of exploration . Liu and Su [ 15 ] proposed a Double Q-Learning algorithm , which achieves Õ ( √ SAT/ ( 1− γ ) 2.5 ) regret within T steps . Furthermore , Liu and Su [ 15 ] constructed a series of hard MDPs and showed that the expected regret for any algorithm is lower bounder by Ω̃ ( √ SAT/ ( 1 − γ ) + √ AT/ ( 1− γ ) 1.5 ) . There still exists a 1/ ( 1− γ ) -gap between the upper and lower regret bounds . In contrast to the aforementioned model-free algorithms , our proposed algorithm is model-based . Model-based Algorithms for Discounted MDP . Our UCBVI-γ falls into the category of modelbased reinforcement learning algorithms . Model-based algorithms maintain a model of the environment and update it based on the observed data . They will form the policy based on the learnt model . More specifically , to learn the -optimal value function , Azar et al . [ 3 ] proposed an empirical QVI algorithm which achieves Õ ( SA/ ( ( 1− γ ) 3 2 ) ) sample complexity . Azar et al . [ 3 ] proposed an empirical QVI algorithm which improves the sample complexity to Õ ( SA/ ( ( 1− γ ) 3 2 ) ) for ≤ 1/ √ ( 1− γ ) S. Szita and Szepesvári [ 27 ] proposed an MoRmax algorithm , which achieves Õ ( SA/ ( ( 1− γ ) 6 2 ) ) sample complexity . Later , Lattimore and Hutter [ 14 ] proposed a UCRL algorithm , which achieves Õ ( S2A/ ( ( 1− γ ) 3 2 ) ) sample complexity in general and Õ ( SA/ ( ( 1− γ ) 3 2 ) ) sample complexity with a strong assumption on the state transition . Recently , Agarwal et al . [ 1 ] proposed a refined analysis for the empirical QVI algorithm which achieves Õ ( SA/ ( ( 1− γ ) 3 2 ) ) sample complexity when ≤ 1/ √ 1− γ . Upper and Lower Bounds for Episodic MDPs . There is a line of work which aims at proving sample complexity or regret for episodic MDPs ( MDPs which consist of restarting episodes ) [ 7 , 18 , 4 , 19 , 11 , 8 , 24 , 21 , 31 , 32 , 17 , 20 ] . Compared with the episodic MDP , discounted MDPs involve only one infinite-horizon sample trajectory , suggesting that any two states or actions on the trajectory are dependent . Such a dependence makes the learning of discounted MDPs more challenging . 3 Preliminaries . We consider infinite-horizon discounted Markov Decision Processes ( MDP ) which are defined by a tuple ( S , A , γ , r , P ) . Here S is the state space with |S| = S , A is the action space with |A| = A , γ ∈ ( 0 , 1 ) is the discount factor , r : S ×A → [ 0 , 1 ] is the reward function , P ( s′|s , a ) is the transition probability function , which denotes the probability that state s transfers to state s′ with action a . For simplicity , we assume the reward function is deterministic and known . A non-stationary policies π is a collection of function { πt } ∞t=1 , where each function πt : { S × A } t−1 × S → A maps history { s1 , a1 , ... , st−1 , at−1 , st = s } to an action . For any non-stationary policy π , we denote πt ( s ) = πt ( s ; s1 , a1 , ... , st−1 , at−1 ) for simplicity . We define the action-value function and value function at step t as follows : Qπt ( s , a ) = E [ ∞∑ i=0 γir ( st+i , at+i ) ∣∣∣∣s1 , ... , st = s , at = a ] , V πt ( s ) = E [ ∞∑ i=0 γir ( st+i , at+i ) ∣∣∣∣s1 , ... , st = s ] , where at+i = πt+i ( st+i ) , and st+i+1 ∼ P ( · |st+i , πt+i ( st+i ) ) . In addition , we denote the optimal action-value function and the optimal value function as Q∗ ( s , a ) = supπ Q π 1 ( s , a ) and V ∗ ( s ) = supπ V π 1 ( s ) respectively . Note that the optimal action-value function and the optimal value function are independent of the step t. For simplicity , for any function V : S → R , we denote [ PV ] ( s , a ) = Es′∼P ( ·|s , a ) V ( s′ ) . According to the definition of the value function , we have the following nonstationary Bellman equation and Bellman optimality equation for non-stationary policy π and optimal policy π∗ : Qπt ( s , a ) = r ( s , a ) + γ [ PV πt+1 ] ( s , a ) , Q∗ ( s , a ) = r ( s , a ) + γ [ PV ∗ ] ( s , a ) . ( 3.1 )

Summary: The authors study the problem of learning a discounted tabular-MDP. They adopt a notion of regret that is closely related to policy identification problem.  The authors make two main contributions: 1. They show how to adapt UCBVI to discounted MDP. This lead to an algorithm called UCBVI-\gamma algorithm. This model-based algorithm uses OFU principle by computing Bernstein-type bonus for empirical Q-function. The policy of the algorithm is updated every time step (online algorithm).  The author prove that its regret is upper bounded by \tilde{O}(\sqrt{SAT}/(1-\gamma)^1.5) where \gamma is the discount factor.  2. The authors also derive a minimax lower bound for the problem that essentially match the upper bound of the algorithm {\Omega}(\sqrt{SAT}/(1-\gamma)^1.5). This shows that their algorithm is minimax-optimal.