Navigating to the Best Policy in Markov Decision Processes

1 Introduction . Somewhat surprisingly , learning in a Markov Decision Process is most often considered under the performance criteria of consistency or regret minimization ( see e.g . [ SB18 , Sze10 , LS20 ] and references therein ) . Regret minimization ( see e.g . [ AJO09 , FCG10 ] ) is particularly relevant when the rewards accumulated during the learning phase are important . This is however not always the case : for example , when learning a game ( whether Go , chess , Atari , or whatever ) , winning or losing during the learning phase does not really matter . One may intuitively think that sometimes getting into difficulty on purpose so as to observe unheard situations can significantly accelerate the learning process . Another example is the training of robot prototypes in the factory : a reasonably good policy is first searched , regardless of the losses incurred , that can serve as an initialization for a second phase regret-minimization mode that starts when the robot is deployed . It is hence also of great practical importance to study the sample complexity of learning , and to work on strategies that might improve , in this perspective , on regret minimizing algorithms . In this work , we are interested in the best policy identification ( BPI ) problem for infinite-horizon discounted MDPs . This framework was introduced by [ Fie94 ] under the name of PAC-RL . In BPI the algorithm explores the MDP until it has gathered enough samples to return an ε-optimal policy with probability at least 1− δ. Crucially , the algorithm halts at a random time step , determined by a 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . stopping rule which guarantees that the probability of returning a wrong answer is less that δ . The optimality of BPI algorithms is measured through their sample complexity , defined as their expected stopping time . Best policy identification in MDPs has been mostly investigated under the lens of minimax-optimality . The minimax framework may be overly pessimistic by accounting for the worst possible MDP , whereas an algorithm with instance-specific guarantees can adapt its exploration procedure to the hardness of the MDP instance that it faces . Recently , a few works made the first attempts towards understanding the instance-specific sample complexity of reinforcement learning . However , they typically either make simplifying assumptions such as access to a generative model [ ZKB19 , MP21 ] , or restrict their attention to episodic MDPs [ WSJ21 ] . In practice however , the samples gathered rather correspond to a single , possibly infinite , trajectory of the system that we wish to control . This motivates our study of the full online setting where observations are only obtained by navigating in the MDP , that is by sequential choices of actions and following the transitions of the MDP . Our contributions . [ MP21 ] recently proposed an information-theoretical complexity analysis for MDPs in the case of access to a generative model . Here we extend their results to the online setting . Our main goal is to understand how the online learning scheme affects the sample complexity compared to the easier case where we have a generative model . A natural first step consists in understanding how the first order term T � log ( 1/δ ) changes . Thus we only focus on the asymptotic regime δ → 0 . Our key contributions can be summarized as follows : • First , we adapt the lower bound of [ MP21 ] to the online setting ( Proposition 2 ) . The new bound also writes as the value of a zero-sum two-player game between nature and the algorithm , where the loss function remains the same , but where the set of possible strategies for the algorithm is restricted to a subset Ω ( M ) of the simplex of dimension SA− 1 . We refer to the constraints defining Ω ( M ) as the navigation constraints . • We propose MDP-NaS , the first algorithm for the online setting1 with instance-dependent bounds on its sample complexity in the asymptotic regime δ → 0 ( Theorem 6 ) . A major challenge lies in the design of a sampling rule that guarantees that the sampling frequency of state-action pairs � Nsa ( t ) /t � s , a converges to some target oracle allocation ω� ∈ Ω ( M ) . Indeed , since we can no longer choose the next state of the agent , the tracking procedure which was developed by [ GK16 ] for multi-armed bandits and used in [ MP21 ] for MDPs with a generative model can no longer be applied in our setting . We propose a new sampling rule which performs exploration according to a mixture of the uniform policy and a plugin estimate of the oracle policy ( the policy whose stationary state-action distribution is ω� ) and prove that it satisfies the requirement above . The analysis of our sampling rule relies on an ergodic theorem for non-homogeneous Markov chains of independent interest ( Proposition 12 ) . • We investigate , depending on the communication properties of the ground-truth instance M , what is the minimal forced-exploration rate in our sampling rule that guarantees the consistency of the plug-in estimator of the oracle policy . Our findings imply that when M is ergodic , an exploration rate as low as 1/ √ t is sufficient . However , when M is only assumed to be communicating , one is obliged to resort to a much more conservative exploration rate of t− 1 m+1 , where m is a parameter defined in Lemma 4 that may scales as large as S − 1 in the worst case . • Finally , our stopping rule represents the first implementation of the Generalized Likelihood Ratio test for MDPs . Notably , we circumvent the need to solve the max-min program of the lower bound exactly , and show how an upper bound of the best-response problem , such as the one derived in [ MP21 ] , can be used to perform a GLR test . This improves upon the sample complexity bound that one obtains using the KL-Ball stopping rule of [ MP21 ] by at least a factor of 22 . 1Before publication of this work , but after a preprint was available online , [ WSJ21 ] proposed another algorithm with instance-dependent guarantees . 2Note that the stopping rule is independent of the sampling rule and thus can be used in both the generative and the online settings . Furthermore , one may even obtain an improved factor of 4 by using a deviation inequality for the full distribution of ( reward , next-state ) instead of a union bound of deviation inequalities for each marginal distribution . 1.1 Related work . Minimax Best-Policy Identification . BPI in the online setting has been investigated recently by a number of works with minimax sample complexity bounds . In the case of episodic MDPs [ KMDD+21 ] , [ MDJ+20 ] proposed algorithms that identify an ε-optimal policy at the initial state w.h.p . In contrast , in the case of infinite-horizon MDPs one is rather interested in finding a good policy at every state . Recent works provide convergence analysis for Q-learning [ LWC+20b ] or policy gradient algorithms [ AHKS20 ] , [ FYAY21 ] , [ ZCA21 ] . Their results typically state that if the algorithm is fed with the appropriate hyperparameters , it can return an ε-optimal policy w.h.p . after collecting a polynomial number of samples . In practice , a pure exploration agent needs a stopping rule to determine when to halt the learning process . In particular , the question of how to tune the number of iterations of these algorithms without prior knowledge of the ground truth instance remains open.3 Generative Model . A large body of literature focuses on the so-called generative model , where in each step , the algorithm may query a sample ( i.e. , observe a reward and a next-state drawn from the rewards and transitions distributions respectively ) from any given state-action pair [ KS98 ] , [ KMN99 ] , [ EDMM06 ] , [ AMK13 ] , [ DTC13 ] , [ Wan17 ] , [ SWW+18 ] , [ ZKB19 ] , [ AKY20 ] , [ LWC+20a ] , [ LCC+21 ] , [ MP21 ] . Instance-specific bounds . Instance-optimal algorithms for Best Arm Identification in multi armed bandits ( MDPs with one state ) have been obtained independently by [ GK16 ] , [ Rus16 ] . Here , we extend their information-theoretical approach to the problem of Best Policy Identification in MDPs . More recently , [ WSJ21 ] provided an algorithm for BPI in episodic MDPs with instance-specific sample complexity . A more detailed comparison with [ WSJ21 ] can be found in Section 6 . Outline . The rest of the paper is organized as follows : After introducing the setting and giving some notation and definitions in Section 2 , we derive in Section 3 a lower bound on the time required by any algorithm navigating the MDP until it is able to identify the best policy with probability at least 1− δ . The algorithm is presented in Section 4 . Section 5 contains our main results along with a sketch of the analysis . Finally , in Section 6 we compare our results with MOCA , the only other algorithm ( to the best of our knowledge ) with problem-dependent guarantees in the online setting . Most technical results and proofs are given in the appendix . 2 Setting and notation . Discounted MDPs . We consider infinite horizon MDPs with discount and finite state and action spaces S and A . Such an MDP M is defined through its transition and reward distributions pM ( ·|s , a ) and qM ( ·|s , a ) for all ( s , a ) . For simplicity , qM ( ·|s , a ) will denote the density of the reward distribution w.r.t . a positive measure λ with support included in [ 0 , 1 ] . Specifically , pM ( s�|s , a ) denotes the probability of transitioning to state s� after playing action a at state s while R ( s , a ) is the random instantaneous reward that is collected . Finally , γ ∈ [ 0 , 1 ) is a discounting factor . We look for an optimal control policy π� : S → A maximizing the long-term discounted reward : V πM ( s ) = EM , π [ �∞ t=0 γ tR ( st , π ( st ) ) ] , where EM , π denotes the expectation w.r.t the randomness in the rewards and the trajectory when the policy π is used . Classically , we denote by V �M the optimal value function of M and by Q�M the optimal Q-value function of M. Π� ( M ) = { π : S → A , V πM = V �M } denotes the set of optimal policies for M. Problem-dependent quantities . The sub-optimality gap of action a in state s is defined as Δsa = V � M ( s ) − Q�M ( s , a ) . Let Δmin = mins , a�=π� ( s ) Δsa be the minimum gap and sp ( V �M ) = maxs , s� |V �M ( s ) − V �M ( s� ) | be the span of V �M . Finally , we introduce the variance of the value function in ( s , a ) as Varp ( s , a ) [ V �M ] = Vars�∼pM ( .|s , a ) [ V � M ( s � ) ] . Active pure exploration with fixed confidence . When M is unknown , we wish to devise a learning algorithm A identifying , from a single trajectory , an optimal policy as quickly as possible with some given level of confidence . Formally , such an algorithm consists of ( i ) a sampling rule , selecting in each round t in an adaptive manner the action at to be played ; at depends on past observations , it 3Here we chose not to mention works that investigate the sample complexity in the PAC-MDP framework [ Kak03 ] . Indeed , in this framework the sample complexity is rather defined as the the number of episodes where the algorithm does not play an ε-optimal policy . As explained in [ DMKV21 ] , this objective is closer to regret minimization than to pure exploration . is FAt -measurable where FAt = σ ( s0 , a0 , R0 . . . , st−1 , at−1 , Rt−1 , st ) is the σ-algebra generated by observations up to time t ; ( ii ) a stopping rule τδ , a stopping time w.r.t . the filtration ( FAt ) t≥0 , deciding when to stop collecting observations ; ( iii ) a decision rule returning π̂�τδ an estimated optimal policy . An algorithm A is δ-Probably Correct ( δ-PC ) over some set M of MDPs , if for any M ∈ M , it returns ( in finite time ) an optimal policy with probability at least 1 − δ . In this paper , we aim to devise a δ-PC algorithm A with minimal sample complexity EM , A [ τδ ] . We make the following assumption . Assumption 1 . We consider the set M of communicating MDPs with a unique optimal policy . We justify the above assumption as follows . ( i ) We restrict our attention to the case where M is communicating , for otherwise , if it is Multichain there would be a non-zero probability that the algorithm enters a subclass of states from which there is no possible comeback . In this case it becomes impossible to identify the global optimal policy4 . ( ii ) About the uniqueness of the optimal policy , treating the case of MDPs with multiple optimal policies , or that of ε-optimal policy identification , requires the use of more involved Overlapping Hypothesis tests , which is already challenging in multiarmed bandits ( MDPs with a single state ) [ GK19 ] . We will analyze how to remove this assumption in future work . Notation . Z � S × A is the state-action space . Σ � { ω ∈ RS×A+ : � s , a ωsa = 1 } , the simplex of dimension SA − 1 . Nsa ( t ) denotes the number of times the state-action pair ( s , a ) has been visited up to the end of round t. We also introduce Ns ( t ) = � a∈A Nsa ( t ) . Similarly , for a vector ω ∈ Σ we will denote ωs � � a∈A ωsa . For a matrix A ∈ Rn×m , the infinity norm is defined as �A�∞ � max1≤i≤n �m j=1 |ai , j | . The KL divergence between two probability distributions P and Q on some discrete space S is defined as : KL ( P�Q ) � �s∈S P ( s ) log ( P ( s ) Q ( s ) ) . For Bernoulli distributions of respective means p and q , the KL divergence is denoted by kl ( p , q ) . For distributions over [ 0 , 1 ] defined through their densities p and q w.r.t . some positive measure λ , the KL divergence is : KL ( p�q ) � �∞ −∞ p ( x ) log � p ( x ) q ( x ) � λ ( dx ) . M denotes the set of communicating MDPs with a unique optimal policy . For two MDPs M , M� ∈ M , we say that M � M� if for all ( s , a ) , pM ( ·|s , a ) � pM� ( ·|s , a ) and qM ( ·|s , a ) � qM� ( ·|s , a ) . In that case , for any state ( action pair ) ( s , a ) , we define the KL divergence of the distributions of the one-step observations under M and M� when starting at ( s , a ) as KLM|M� ( s , a ) � KL ( pM ( ·|s , a ) �pM� ( ·|s , a ) ) +KL ( qM ( ·|s , a ) �qM� ( ·|s , a ) ) .

Summary: This work talks about best policy identification problem in discounted infinite MDP. Instead of assuming a generator, this work is on the online setting of MDP, where the agent can only collect state action pairs through online interaction with the environment. In theory, the work provides a lower bound of the BPI problem. Then the work provides a model-based algorithm MDP-NaS, which is shown to converge to the optimal policy.