On the Convergence of the Monte Carlo Exploring Starts Algorithm for Reinforcement Learning

1 INTRODUCTION . Perhaps the most famous algorithm in tabular reinforcement learning is the so-called Q-learning algorithm . Under very general conditions , it is well known that the Q-learning converges to the optimal Q-function with probability one ( Tsitsiklis , 1994 ; Jaakkola et al. , 1994 ) . Importantly , in order to guarantee convergence for Q-learning , it is only required that every state-action pair be visited infinitely often . Furthermore , as discussed in the related work , Q-learning converges for the infinite-horizon discounted problem as well as for the non-discounted terminal-state problem ( also known as the stochastic shortest-path problem ) . The Q-learning algorithm is inspired by dynamic programming and uses back-ups to update the estimates of the optimal Q-function . An alternative methodological approach , which does not use back-ups , is to use the Monte Carlo episodic returns to estimate the values of the Q-function . In order for such an algorithm to succeed at finding an optimal policy , the algorithm must include some form of exploration . A simple form of exploration is “ exploring starts , ” where at the beginning of each episode , a random state-action pair is chosen . In the classic book on reinforcement learning by Sutton & Barto ( 2018 ) , the authors describe such an algorithm , namely , Monte Carlo Exploring Starts ( MCES ) . In the MCES algorithm , after a ( random-length ) episode , the Q-function estimate is updated with the Monte Carlo return for each state-action pair along the episode , and the policy is improved in the usual fashion by setting it to the argmax of the current Q-function estimate . Exploration is performed by exploring starts , where the initial state-action pairs may be chosen with any distribution . We briefly remark here that AlphaZero is a Monte Carlo algorithm in that it runs episodes to completion and uses the returns from those episodes for the targets in the loss function ( Silver et al. , 2018 ) . AlphaZero additionally uses function approximators ( convolutional networks ) and planning ( Monte Carlo Tree Search ) , and is thus much more complicated than MCES . But AlphaZero is nonetheless fundamentally a Monte Carlo algorithm rather than a Q-learning-based algorithm . We mention AlphaZero here in order to emphasize that Monte Carlo algorithms are indeed used in practice , and it is therefore important to gain a deep understanding of their underlying theoretical properties . Since Q-learning converges under very general conditions , a natural question is : does MCES converge under equally general conditions ? In the 1996 book , Bertsekas & Tsitsiklis ( 1996 ) provide a counter-example showing that the MCES algorithm does not necessarily converge . See also Liu ( 2020 ) for numerical results in this direction . Thus , we see that the MCES convergence problem is fundamentally trickier than the Q-learning convergence problem . Instead of establishing a very general result as in Q-learning , we can at best establish convergence for a broad class of special-cases . Sutton and Barto write at the end of Section 5.3 : “ In our opinion , this is one of the most fundamental open theoretical questions in reinforcement learning ” . This paper is focused on this fundamental question . Although other questions , such as rates of convergence and regret bounds , are also important , in this paper our goal is to address the fundamental question of convergence . Tsitsiklis ( 2002 ) made significant progress with the MCES convergence problem , showing that almost sure convergence is guaranteed if the following three conditions hold : ( i ) the discount factor is strictly less than one ; ( ii ) the MCES algorithm is modified so that after an episode , the Q-function estimate is updated with the Monte Carlo return only for the initial state-action pair of the episode ; and ( iii ) the algorithm is further modified so that the initial state-action pair in an episode is chosen with a uniform distribution . As in the proof of Q-learning , Tsitsiklis ’ s proof is based on stochastic approximations . The conditions ( ii ) and ( iii ) combined ensure that the Q function estimates are updated at the same average rate for all state-action pairs , and both conditions appear to be crucial for establishing convergence in the proof in Tsitsiklis ( 2002 ) . However , these two conditions have the following drawbacks : • Perhaps most importantly , condition ( ii ) results in a substantially less efficient algorithm , since only one Q-function value is updated per episode . The original Sutton and Barto version is more efficient since after each episode , many Q-values are typically updated rather than just one . ( We also note as an aside that AlphaZero will also collect and use Monte Carlo return for all states along the episode , not just for the first state in the episode , as discussed on page 2 of Silver et al . ( 2017 ) ) • Similar to the idea of importance sampling , one may want to use a non-uniform distribution for the starting state-action pairs to accelerate convergence . • In some cases , we may not have access to a simulator to generate uniform exploring starts . Instead , we may run episodes by interacting directly with the real environment . Such natural interactions may lead to starting from every state , but not uniformly . An example would be playing blackjack at a casino rather than training with a simulator . In this paper we provide new convergence results and a new proof methodology for MCES . Unlike the result in Tsitsiklis ( 2002 ) , the results reported here do not modify the original MCES algorithm and do not require any of the conditions ( i ) − ( iii ) . Hence , our results do not have the three drawbacks listed above , and also allow for no discounting ( as in the stochastic shortest path problem ) . However , our proofs require restrictions on the dynamics of the underlying MDP . Specifically , we require that under the optimal policy , a state is never revisited . This class of MDPs includes stochastic feed-forward environments such as Blackjack ( Sutton & Barto , 2018 ) and also all deterministic MDPs , such as gridworlds ( Sutton & Barto , 2018 ) , Go and Chess ( when played against a fixed opponent policy ) , and the MuJoCo environments ( Todorov et al. , 2012 ) ( Episodic MuJoCo tasks fall into the category of OPFF MDPs because the MuJoCo simulation is deterministic ) . Moreover , if the trajectory horizon is instead fixed and deterministic , we show that the original MCES algorithm always converges ( to a time-dependent ) optimal policy , without any conditions on the dynamics , initial state-action distribution or the discount factor . Importantly , we also provide a new proof methodology . Our proof is very simple , making use of only the Strong Law of Large Numbers ( SLLN ) and a simple inductive argument . The proof does not use stochastic approximations , contraction mappings , or martingales , and can be done in an undergraduate course in machine learning . We believe that this new proof methodology provides new insights for episodic RL problems . In addition to the theoretical results , we present numerical experiments that show the original MCES can be much more efficient than the modified MCES , further highlighting the importance of improving our understanding on the convergence properties of the original MCES algorithm . 2 RELATED WORK . Some authors refer to an MDP with a finite horizon H as an episodic MDP . For finite horizon MDPs , the optimal Q-function and optimal policy are in general non-stationary and depend on time . Here , following Sutton & Barto ( 2018 ) , we instead reserve the term episodic MDPs for MDPs that terminate when the terminal state is reached , and thus the episode length is not fixed at H and may have a random length . Moreover , for such terminal-state episodic MDPs , under very general conditions , the optimal Q-function and policy are stationary and do not depend on time ( as in infinitehorizon discounted MDPs ) . When the dynamics are known and the discount factor equals 1 , the episodic optimization problem considered here is equivalent to the stochastic shortest path problem ( SSPP ) ( see Bertsekas & Tsitsiklis ( 1991 ) and references therein ; also see Chapter 2 of Bertsekas ( 2012 ) ) . Under very general conditions , value iteration converges to the optimal value function , from which an optimal stationary policy can be constructed . Convergence theory for RL algorithms has a long history . For the infinite-horizon discounted criterion , by showing that Q-learning is a form of stochastic approximations , Tsitsiklis ( 1994 ) and Jaakkola et al . ( 1994 ) showed that Q-learning converges almost surely to the optimal Q-function under very general conditions . There are also convergence results for Q-learning applied to episodic MDPs as defined in this paper with discount factor equal to 1 . Tsitsiklis [ 8 , Theorems 2 and 4 ( c ) ] proved that if the sequence of Q-learning iterates is bounded , then Q-learning converges to the optimal Q values almost surely . Yu & Bertsekas ( 2013 ) prove that the sequence of Q-learning iterates is bounded for episodic MDPs with or without non-negativity assumptions , fully establishing the convergence of Q-learning for terminal-state episodic RL problems . This paper is primarily concerned with the convergence of the MCES algorithm . In the Introduction we reviewed the important work of Sutton & Barto ( 1998 ) , Bertsekas & Tsitsiklis ( 1996 ) , and Tsitsiklis ( 2002 ) . Importantly , unlike Q-learning , the MCES algorithm is not guaranteed to converge for all types of MDPs . Indeed , in Section 5.4 of Bertsekas & Tsitsiklis ( 1996 ) , Example 5.12 shows that MCES is not guaranteed to converge for a continuing task MDP . However , if the algorithm is modified , as described in the Introduction , then convergence is guaranteed ( Tsitsiklis , 2002 ) . Recently , Chen ( 2018 ) extended the convergence result in Tsitsiklis ( 2002 ) to the undiscounted case , under the assumption that all policies are proper , that is , regardless of the initial state , all policies will lead to a terminal state in finite time with probability one . More recently , Liu ( 2020 ) relaxed the all policies being proper condition . As in Tsitsiklis ( 2002 ) , both Chen ( 2018 ) and Liu ( 2020 ) assume conditions ( ii ) − ( iii ) stated in the introduction , and their proofs employ the stochastic approximations methodology in Tsitsiklis ( 1994 ) . The results we develop here are complementary to the results in Tsitsiklis ( 2002 ) , Chen ( 2018 ) , and Liu ( 2020 ) , in that they do not require the strong algorithmic assumptions ( ii ) − ( iii ) described in the Introduction , and they use an entirely different proof methodology . In this work we focus on the question of convergence of the MCES problem . We briefly mention , there is also a large body of ( mostly orthogonal ) work on rates of convergence and regret analysis for Q-learning ( e.g . see Jin et al . ( 2018 ) ) and also for Monte Carlo approaches ( e.g. , see Kocsis & Szepesvári ( 2006 ) Azar et al . ( 2017 ) ) . To the best of our knowledge , these regret bounds assume finite-horizon MDPs ( for which the optimal policy is time-dependent ) rather than the terminal-state episodic MDPs considered here .

Summary: This paper studies Monte Carlo with exploration starts algorithm for solving the reinforcement learning problem. The writing is clear and I enjoyed reading this paper. As for the results, asymptotic convergence of the algorithm is established without needing strong assumptions in related literature. As pointed out by the authors, the result resolves an important open problem in RL. The proof is simple and intuitive. Numerical experiments corroborate theoretical findings.