Navigating to the Best Policy in Markov Decision Processes

1 Introduction . Somewhat surprisingly , learning in a Markov Decision Process is most often considered under the performance criteria of consistency or regret minimization ( see e.g . [ SB18 , Sze10 , LS20 ] and references therein ) . Regret minimization ( see e.g . [ AJO09 , FCG10 ] ) is particularly relevant when the rewards accumulated during the learning phase are important . This is however not always the case : for example , when learning a game ( whether Go , chess , Atari , or whatever ) , winning or losing during the learning phase does not really matter . One may intuitively think that sometimes getting into difficulty on purpose so as to observe unheard situations can significantly accelerate the learning process . Another example is the training of robot prototypes in the factory : a reasonably good policy is first searched , regardless of the losses incurred , that can serve as an initialization for a second phase regret-minimization mode that starts when the robot is deployed . It is hence also of great practical importance to study the sample complexity of learning , and to work on strategies that might improve , in this perspective , on regret minimizing algorithms . In this work , we are interested in the best policy identification ( BPI ) problem for infinite-horizon discounted MDPs . This framework was introduced by [ Fie94 ] under the name of PAC-RL . In BPI the algorithm explores the MDP until it has gathered enough samples to return an ε-optimal policy with probability at least 1− δ. Crucially , the algorithm halts at a random time step , determined by a 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . stopping rule which guarantees that the probability of returning a wrong answer is less that δ . The optimality of BPI algorithms is measured through their sample complexity , defined as their expected stopping time . Best policy identification in MDPs has been mostly investigated under the lens of minimax-optimality . The minimax framework may be overly pessimistic by accounting for the worst possible MDP , whereas an algorithm with instance-specific guarantees can adapt its exploration procedure to the hardness of the MDP instance that it faces . Recently , a few works made the first attempts towards understanding the instance-specific sample complexity of reinforcement learning . However , they typically either make simplifying assumptions such as access to a generative model [ ZKB19 , MP21 ] , or restrict their attention to episodic MDPs [ WSJ21 ] . In practice however , the samples gathered rather correspond to a single , possibly infinite , trajectory of the system that we wish to control . This motivates our study of the full online setting where observations are only obtained by navigating in the MDP , that is by sequential choices of actions and following the transitions of the MDP . Our contributions . [ MP21 ] recently proposed an information-theoretical complexity analysis for MDPs in the case of access to a generative model . Here we extend their results to the online setting . Our main goal is to understand how the online learning scheme affects the sample complexity compared to the easier case where we have a generative model . A natural first step consists in understanding how the first order term T � log ( 1/δ ) changes . Thus we only focus on the asymptotic regime δ → 0 . Our key contributions can be summarized as follows : • First , we adapt the lower bound of [ MP21 ] to the online setting ( Proposition 2 ) . The new bound also writes as the value of a zero-sum two-player game between nature and the algorithm , where the loss function remains the same , but where the set of possible strategies for the algorithm is restricted to a subset Ω ( M ) of the simplex of dimension SA− 1 . We refer to the constraints defining Ω ( M ) as the navigation constraints . • We propose MDP-NaS , the first algorithm for the online setting1 with instance-dependent bounds on its sample complexity in the asymptotic regime δ → 0 ( Theorem 6 ) . A major challenge lies in the design of a sampling rule that guarantees that the sampling frequency of state-action pairs � Nsa ( t ) /t � s , a converges to some target oracle allocation ω� ∈ Ω ( M ) . Indeed , since we can no longer choose the next state of the agent , the tracking procedure which was developed by [ GK16 ] for multi-armed bandits and used in [ MP21 ] for MDPs with a generative model can no longer be applied in our setting . We propose a new sampling rule which performs exploration according to a mixture of the uniform policy and a plugin estimate of the oracle policy ( the policy whose stationary state-action distribution is ω� ) and prove that it satisfies the requirement above . The analysis of our sampling rule relies on an ergodic theorem for non-homogeneous Markov chains of independent interest ( Proposition 12 ) . • We investigate , depending on the communication properties of the ground-truth instance M , what is the minimal forced-exploration rate in our sampling rule that guarantees the consistency of the plug-in estimator of the oracle policy . Our findings imply that when M is ergodic , an exploration rate as low as 1/ √ t is sufficient . However , when M is only assumed to be communicating , one is obliged to resort to a much more conservative exploration rate of t− 1 m+1 , where m is a parameter defined in Lemma 4 that may scales as large as S − 1 in the worst case . • Finally , our stopping rule represents the first implementation of the Generalized Likelihood Ratio test for MDPs . Notably , we circumvent the need to solve the max-min program of the lower bound exactly , and show how an upper bound of the best-response problem , such as the one derived in [ MP21 ] , can be used to perform a GLR test . This improves upon the sample complexity bound that one obtains using the KL-Ball stopping rule of [ MP21 ] by at least a factor of 22 . 1Before publication of this work , but after a preprint was available online , [ WSJ21 ] proposed another algorithm with instance-dependent guarantees . 2Note that the stopping rule is independent of the sampling rule and thus can be used in both the generative and the online settings . Furthermore , one may even obtain an improved factor of 4 by using a deviation inequality for the full distribution of ( reward , next-state ) instead of a union bound of deviation inequalities for each marginal distribution . 1.1 Related work . Minimax Best-Policy Identification . BPI in the online setting has been investigated recently by a number of works with minimax sample complexity bounds . In the case of episodic MDPs [ KMDD+21 ] , [ MDJ+20 ] proposed algorithms that identify an ε-optimal policy at the initial state w.h.p . In contrast , in the case of infinite-horizon MDPs one is rather interested in finding a good policy at every state . Recent works provide convergence analysis for Q-learning [ LWC+20b ] or policy gradient algorithms [ AHKS20 ] , [ FYAY21 ] , [ ZCA21 ] . Their results typically state that if the algorithm is fed with the appropriate hyperparameters , it can return an ε-optimal policy w.h.p . after collecting a polynomial number of samples . In practice , a pure exploration agent needs a stopping rule to determine when to halt the learning process . In particular , the question of how to tune the number of iterations of these algorithms without prior knowledge of the ground truth instance remains open.3 Generative Model . A large body of literature focuses on the so-called generative model , where in each step , the algorithm may query a sample ( i.e. , observe a reward and a next-state drawn from the rewards and transitions distributions respectively ) from any given state-action pair [ KS98 ] , [ KMN99 ] , [ EDMM06 ] , [ AMK13 ] , [ DTC13 ] , [ Wan17 ] , [ SWW+18 ] , [ ZKB19 ] , [ AKY20 ] , [ LWC+20a ] , [ LCC+21 ] , [ MP21 ] . Instance-specific bounds . Instance-optimal algorithms for Best Arm Identification in multi armed bandits ( MDPs with one state ) have been obtained independently by [ GK16 ] , [ Rus16 ] . Here , we extend their information-theoretical approach to the problem of Best Policy Identification in MDPs . More recently , [ WSJ21 ] provided an algorithm for BPI in episodic MDPs with instance-specific sample complexity . A more detailed comparison with [ WSJ21 ] can be found in Section 6 . Outline . The rest of the paper is organized as follows : After introducing the setting and giving some notation and definitions in Section 2 , we derive in Section 3 a lower bound on the time required by any algorithm navigating the MDP until it is able to identify the best policy with probability at least 1− δ . The algorithm is presented in Section 4 . Section 5 contains our main results along with a sketch of the analysis . Finally , in Section 6 we compare our results with MOCA , the only other algorithm ( to the best of our knowledge ) with problem-dependent guarantees in the online setting . Most technical results and proofs are given in the appendix . 2 Setting and notation . Discounted MDPs . We consider infinite horizon MDPs with discount and finite state and action spaces S and A . Such an MDP M is defined through its transition and reward distributions pM ( ·|s , a ) and qM ( ·|s , a ) for all ( s , a ) . For simplicity , qM ( ·|s , a ) will denote the density of the reward distribution w.r.t . a positive measure λ with support included in [ 0 , 1 ] . Specifically , pM ( s�|s , a ) denotes the probability of transitioning to state s� after playing action a at state s while R ( s , a ) is the random instantaneous reward that is collected . Finally , γ ∈ [ 0 , 1 ) is a discounting factor . We look for an optimal control policy π� : S → A maximizing the long-term discounted reward : V πM ( s ) = EM , π [ �∞ t=0 γ tR ( st , π ( st ) ) ] , where EM , π denotes the expectation w.r.t the randomness in the rewards and the trajectory when the policy π is used . Classically , we denote by V �M the optimal value function of M and by Q�M the optimal Q-value function of M. Π� ( M ) = { π : S → A , V πM = V �M } denotes the set of optimal policies for M. Problem-dependent quantities . The sub-optimality gap of action a in state s is defined as Δsa = V � M ( s ) − Q�M ( s , a ) . Let Δmin = mins , a�=π� ( s ) Δsa be the minimum gap and sp ( V �M ) = maxs , s� |V �M ( s ) − V �M ( s� ) | be the span of V �M . Finally , we introduce the variance of the value function in ( s , a ) as Varp ( s , a ) [ V �M ] = Vars�∼pM ( .|s , a ) [ V � M ( s � ) ] . Active pure exploration with fixed confidence . When M is unknown , we wish to devise a learning algorithm A identifying , from a single trajectory , an optimal policy as quickly as possible with some given level of confidence . Formally , such an algorithm consists of ( i ) a sampling rule , selecting in each round t in an adaptive manner the action at to be played ; at depends on past observations , it 3Here we chose not to mention works that investigate the sample complexity in the PAC-MDP framework [ Kak03 ] . Indeed , in this framework the sample complexity is rather defined as the the number of episodes where the algorithm does not play an ε-optimal policy . As explained in [ DMKV21 ] , this objective is closer to regret minimization than to pure exploration . is FAt -measurable where FAt = σ ( s0 , a0 , R0 . . . , st−1 , at−1 , Rt−1 , st ) is the σ-algebra generated by observations up to time t ; ( ii ) a stopping rule τδ , a stopping time w.r.t . the filtration ( FAt ) t≥0 , deciding when to stop collecting observations ; ( iii ) a decision rule returning π̂�τδ an estimated optimal policy . An algorithm A is δ-Probably Correct ( δ-PC ) over some set M of MDPs , if for any M ∈ M , it returns ( in finite time ) an optimal policy with probability at least 1 − δ . In this paper , we aim to devise a δ-PC algorithm A with minimal sample complexity EM , A [ τδ ] . We make the following assumption . Assumption 1 . We consider the set M of communicating MDPs with a unique optimal policy . We justify the above assumption as follows . ( i ) We restrict our attention to the case where M is communicating , for otherwise , if it is Multichain there would be a non-zero probability that the algorithm enters a subclass of states from which there is no possible comeback . In this case it becomes impossible to identify the global optimal policy4 . ( ii ) About the uniqueness of the optimal policy , treating the case of MDPs with multiple optimal policies , or that of ε-optimal policy identification , requires the use of more involved Overlapping Hypothesis tests , which is already challenging in multiarmed bandits ( MDPs with a single state ) [ GK19 ] . We will analyze how to remove this assumption in future work . Notation . Z � S × A is the state-action space . Σ � { ω ∈ RS×A+ : � s , a ωsa = 1 } , the simplex of dimension SA − 1 . Nsa ( t ) denotes the number of times the state-action pair ( s , a ) has been visited up to the end of round t. We also introduce Ns ( t ) = � a∈A Nsa ( t ) . Similarly , for a vector ω ∈ Σ we will denote ωs � � a∈A ωsa . For a matrix A ∈ Rn×m , the infinity norm is defined as �A�∞ � max1≤i≤n �m j=1 |ai , j | . The KL divergence between two probability distributions P and Q on some discrete space S is defined as : KL ( P�Q ) � �s∈S P ( s ) log ( P ( s ) Q ( s ) ) . For Bernoulli distributions of respective means p and q , the KL divergence is denoted by kl ( p , q ) . For distributions over [ 0 , 1 ] defined through their densities p and q w.r.t . some positive measure λ , the KL divergence is : KL ( p�q ) � �∞ −∞ p ( x ) log � p ( x ) q ( x ) � λ ( dx ) . M denotes the set of communicating MDPs with a unique optimal policy . For two MDPs M , M� ∈ M , we say that M � M� if for all ( s , a ) , pM ( ·|s , a ) � pM� ( ·|s , a ) and qM ( ·|s , a ) � qM� ( ·|s , a ) . In that case , for any state ( action pair ) ( s , a ) , we define the KL divergence of the distributions of the one-step observations under M and M� when starting at ( s , a ) as KLM|M� ( s , a ) � KL ( pM ( ·|s , a ) �pM� ( ·|s , a ) ) +KL ( qM ( ·|s , a ) �qM� ( ·|s , a ) ) .

Summary: This paper deals with online learning of policies for MDPs. Its main novelty is considering the "navigation constraints" related to this online setting where, in contrast with an episodic setting, the system must learn the policy in a single episode without being able to reset to an initial state. Bounds on sample complexity are derived and an algorithm for learning the best policy by exploiting the lower bounds is proposed.