On the Convergence of the Monte Carlo Exploring Starts Algorithm for Reinforcement Learning

1 INTRODUCTION . Perhaps the most famous algorithm in tabular reinforcement learning is the so-called Q-learning algorithm . Under very general conditions , it is well known that the Q-learning converges to the optimal Q-function with probability one ( Tsitsiklis , 1994 ; Jaakkola et al. , 1994 ) . Importantly , in order to guarantee convergence for Q-learning , it is only required that every state-action pair be visited infinitely often . Furthermore , as discussed in the related work , Q-learning converges for the infinite-horizon discounted problem as well as for the non-discounted terminal-state problem ( also known as the stochastic shortest-path problem ) . The Q-learning algorithm is inspired by dynamic programming and uses back-ups to update the estimates of the optimal Q-function . An alternative methodological approach , which does not use back-ups , is to use the Monte Carlo episodic returns to estimate the values of the Q-function . In order for such an algorithm to succeed at finding an optimal policy , the algorithm must include some form of exploration . A simple form of exploration is “ exploring starts , ” where at the beginning of each episode , a random state-action pair is chosen . In the classic book on reinforcement learning by Sutton & Barto ( 2018 ) , the authors describe such an algorithm , namely , Monte Carlo Exploring Starts ( MCES ) . In the MCES algorithm , after a ( random-length ) episode , the Q-function estimate is updated with the Monte Carlo return for each state-action pair along the episode , and the policy is improved in the usual fashion by setting it to the argmax of the current Q-function estimate . Exploration is performed by exploring starts , where the initial state-action pairs may be chosen with any distribution . We briefly remark here that AlphaZero is a Monte Carlo algorithm in that it runs episodes to completion and uses the returns from those episodes for the targets in the loss function ( Silver et al. , 2018 ) . AlphaZero additionally uses function approximators ( convolutional networks ) and planning ( Monte Carlo Tree Search ) , and is thus much more complicated than MCES . But AlphaZero is nonetheless fundamentally a Monte Carlo algorithm rather than a Q-learning-based algorithm . We mention AlphaZero here in order to emphasize that Monte Carlo algorithms are indeed used in practice , and it is therefore important to gain a deep understanding of their underlying theoretical properties . Since Q-learning converges under very general conditions , a natural question is : does MCES converge under equally general conditions ? In the 1996 book , Bertsekas & Tsitsiklis ( 1996 ) provide a counter-example showing that the MCES algorithm does not necessarily converge . See also Liu ( 2020 ) for numerical results in this direction . Thus , we see that the MCES convergence problem is fundamentally trickier than the Q-learning convergence problem . Instead of establishing a very general result as in Q-learning , we can at best establish convergence for a broad class of special-cases . Sutton and Barto write at the end of Section 5.3 : “ In our opinion , this is one of the most fundamental open theoretical questions in reinforcement learning ” . This paper is focused on this fundamental question . Although other questions , such as rates of convergence and regret bounds , are also important , in this paper our goal is to address the fundamental question of convergence . Tsitsiklis ( 2002 ) made significant progress with the MCES convergence problem , showing that almost sure convergence is guaranteed if the following three conditions hold : ( i ) the discount factor is strictly less than one ; ( ii ) the MCES algorithm is modified so that after an episode , the Q-function estimate is updated with the Monte Carlo return only for the initial state-action pair of the episode ; and ( iii ) the algorithm is further modified so that the initial state-action pair in an episode is chosen with a uniform distribution . As in the proof of Q-learning , Tsitsiklis ’ s proof is based on stochastic approximations . The conditions ( ii ) and ( iii ) combined ensure that the Q function estimates are updated at the same average rate for all state-action pairs , and both conditions appear to be crucial for establishing convergence in the proof in Tsitsiklis ( 2002 ) . However , these two conditions have the following drawbacks : • Perhaps most importantly , condition ( ii ) results in a substantially less efficient algorithm , since only one Q-function value is updated per episode . The original Sutton and Barto version is more efficient since after each episode , many Q-values are typically updated rather than just one . ( We also note as an aside that AlphaZero will also collect and use Monte Carlo return for all states along the episode , not just for the first state in the episode , as discussed on page 2 of Silver et al . ( 2017 ) ) • Similar to the idea of importance sampling , one may want to use a non-uniform distribution for the starting state-action pairs to accelerate convergence . • In some cases , we may not have access to a simulator to generate uniform exploring starts . Instead , we may run episodes by interacting directly with the real environment . Such natural interactions may lead to starting from every state , but not uniformly . An example would be playing blackjack at a casino rather than training with a simulator . In this paper we provide new convergence results and a new proof methodology for MCES . Unlike the result in Tsitsiklis ( 2002 ) , the results reported here do not modify the original MCES algorithm and do not require any of the conditions ( i ) − ( iii ) . Hence , our results do not have the three drawbacks listed above , and also allow for no discounting ( as in the stochastic shortest path problem ) . However , our proofs require restrictions on the dynamics of the underlying MDP . Specifically , we require that under the optimal policy , a state is never revisited . This class of MDPs includes stochastic feed-forward environments such as Blackjack ( Sutton & Barto , 2018 ) and also all deterministic MDPs , such as gridworlds ( Sutton & Barto , 2018 ) , Go and Chess ( when played against a fixed opponent policy ) , and the MuJoCo environments ( Todorov et al. , 2012 ) ( Episodic MuJoCo tasks fall into the category of OPFF MDPs because the MuJoCo simulation is deterministic ) . Moreover , if the trajectory horizon is instead fixed and deterministic , we show that the original MCES algorithm always converges ( to a time-dependent ) optimal policy , without any conditions on the dynamics , initial state-action distribution or the discount factor . Importantly , we also provide a new proof methodology . Our proof is very simple , making use of only the Strong Law of Large Numbers ( SLLN ) and a simple inductive argument . The proof does not use stochastic approximations , contraction mappings , or martingales , and can be done in an undergraduate course in machine learning . We believe that this new proof methodology provides new insights for episodic RL problems . In addition to the theoretical results , we present numerical experiments that show the original MCES can be much more efficient than the modified MCES , further highlighting the importance of improving our understanding on the convergence properties of the original MCES algorithm . 2 RELATED WORK . Some authors refer to an MDP with a finite horizon H as an episodic MDP . For finite horizon MDPs , the optimal Q-function and optimal policy are in general non-stationary and depend on time . Here , following Sutton & Barto ( 2018 ) , we instead reserve the term episodic MDPs for MDPs that terminate when the terminal state is reached , and thus the episode length is not fixed at H and may have a random length . Moreover , for such terminal-state episodic MDPs , under very general conditions , the optimal Q-function and policy are stationary and do not depend on time ( as in infinitehorizon discounted MDPs ) . When the dynamics are known and the discount factor equals 1 , the episodic optimization problem considered here is equivalent to the stochastic shortest path problem ( SSPP ) ( see Bertsekas & Tsitsiklis ( 1991 ) and references therein ; also see Chapter 2 of Bertsekas ( 2012 ) ) . Under very general conditions , value iteration converges to the optimal value function , from which an optimal stationary policy can be constructed . Convergence theory for RL algorithms has a long history . For the infinite-horizon discounted criterion , by showing that Q-learning is a form of stochastic approximations , Tsitsiklis ( 1994 ) and Jaakkola et al . ( 1994 ) showed that Q-learning converges almost surely to the optimal Q-function under very general conditions . There are also convergence results for Q-learning applied to episodic MDPs as defined in this paper with discount factor equal to 1 . Tsitsiklis [ 8 , Theorems 2 and 4 ( c ) ] proved that if the sequence of Q-learning iterates is bounded , then Q-learning converges to the optimal Q values almost surely . Yu & Bertsekas ( 2013 ) prove that the sequence of Q-learning iterates is bounded for episodic MDPs with or without non-negativity assumptions , fully establishing the convergence of Q-learning for terminal-state episodic RL problems . This paper is primarily concerned with the convergence of the MCES algorithm . In the Introduction we reviewed the important work of Sutton & Barto ( 1998 ) , Bertsekas & Tsitsiklis ( 1996 ) , and Tsitsiklis ( 2002 ) . Importantly , unlike Q-learning , the MCES algorithm is not guaranteed to converge for all types of MDPs . Indeed , in Section 5.4 of Bertsekas & Tsitsiklis ( 1996 ) , Example 5.12 shows that MCES is not guaranteed to converge for a continuing task MDP . However , if the algorithm is modified , as described in the Introduction , then convergence is guaranteed ( Tsitsiklis , 2002 ) . Recently , Chen ( 2018 ) extended the convergence result in Tsitsiklis ( 2002 ) to the undiscounted case , under the assumption that all policies are proper , that is , regardless of the initial state , all policies will lead to a terminal state in finite time with probability one . More recently , Liu ( 2020 ) relaxed the all policies being proper condition . As in Tsitsiklis ( 2002 ) , both Chen ( 2018 ) and Liu ( 2020 ) assume conditions ( ii ) − ( iii ) stated in the introduction , and their proofs employ the stochastic approximations methodology in Tsitsiklis ( 1994 ) . The results we develop here are complementary to the results in Tsitsiklis ( 2002 ) , Chen ( 2018 ) , and Liu ( 2020 ) , in that they do not require the strong algorithmic assumptions ( ii ) − ( iii ) described in the Introduction , and they use an entirely different proof methodology . In this work we focus on the question of convergence of the MCES problem . We briefly mention , there is also a large body of ( mostly orthogonal ) work on rates of convergence and regret analysis for Q-learning ( e.g . see Jin et al . ( 2018 ) ) and also for Monte Carlo approaches ( e.g. , see Kocsis & Szepesvári ( 2006 ) Azar et al . ( 2017 ) ) . To the best of our knowledge , these regret bounds assume finite-horizon MDPs ( for which the optimal policy is time-dependent ) rather than the terminal-state episodic MDPs considered here .

Summary: The paper studies the convergence of Monte Carlo Exploring Starts (MCES), in which the Q-function is estimated by averaging Monte Carlo returns and the policy is defined as the greedy policy w.r.t. this Q-function. The authors provide a technically simple proof of the convergence under different assumptions on the underlying MDP: (i) stochastic feed-forward MDP (in which states cannot be re-visited in an episode); (ii) optimal policy feed-forward (in which under the optimal policy states are not re-visited); (iii) finite-horizon MDPs. Given these additional assumptions, the analysis relaxes other assumptions employed in previous convergence proofs. Finally, experimental results to validate the considered set of assumptions is provided.