Multi-Step Reinforcement Learning: A Unifying Algorithm

Unifying seemingly disparate algorithmic ideas to produce better performing algorithms has been a longstanding goal in reinforcement learning. As a primary example, TD(λ) elegantly unifies one-step TD prediction with Monte Carlo methods through the use of eligibility traces and the tracedecay parameter λ. Currently, there are a multitude of algorithms that can be used to perform TD control, including Sarsa, Q-learning, and Expected Sarsa. These methods are often studied in the one-step case, but they can be extended across multiple time steps to achieve better performance. Each of these algorithms is seemingly distinct, and no one dominates the others for all problems. In this paper, we study a new multi-step action-value algorithm called Q(σ) that unifies and generalizes these existing algorithms, while subsuming them as special cases. A new parameter, σ, is introduced to allow the degree of sampling performed by the algorithm at each step during its backup to be continuously varied, with Sarsa existing at one extreme (full sampling), and Expected Sarsa existing at the other (pure expectation). Q(σ) is generally applicable to both onand off-policy learning, but in this work we focus on experiments in the on-policy case. Our results show that an intermediate value of σ, which results in a mixture of the existing algorithms, performs better than either extreme. The mixture can also be varied dynamically which can result in even greater performance. The Landscape of TD Algorithms Temporal-difference (TD) methods (Sutton and Barto 1998) are an important concept in reinforcement learning (RL) that combines ideas from Monte Carlo and dynamic programming methods. TD methods allow learning to occur directly from raw experience in the absence of a model of the environment’s dynamics, like with Monte Carlo methods, while also allowing estimates to be updated based on other learned estimates without waiting for a final result, like with dynamic programming. The core concepts of TD methods provide a flexible framework for creating a variety of powerful algorithms that can be used for both prediction and control. There are a number of TD control methods that have been proposed. Q-learning (Watkins 1989; Watkins and Dayan Copyright c  Authors contributed equally, and are listed alphabetically. 1992) is arguably the most popular, and is considered an offpolicy method because the policy generating the behaviour (the behaviour policy), and the policy that is being learned (the target policy) are different. Sarsa (Rummery and Niranjan 1994; Sutton 1996) is the classical on-policy control method, where the behaviour and target policies are the same. However, Sarsa can be extended to learn off-policy with the use of importance sampling (Precup, Sutton, and Singh 2000). Expected Sarsa is an extension of Sarsa that, instead of using the action-value of the next state to update the value of the current state, uses the expectation of all the subsequent action-values of the current state with respect to the target policy. Expected Sarsa has been studied as a strictly on-policy method (van Seijen et al. 2009), but in this paper we present a more general version that can be used for both onand off-policy learning and that also subsumes Q-learning. All of these methods are often described in the simple one-step case, but they can also be extended across multiple time steps. The TD(λ) algorithm unifies one-step TD learning with Monte Carlo methods (Sutton 1988). Through the use of eligibility traces, and the trace-decay parameter, λ ∈ [0, 1], a spectrum of algorithms is created. At one end, λ = 1, exists Monte Carlo methods, and at the other, λ = 0, exists onestep TD learning. In the middle of the spectrum are intermediate methods which can perform better than the methods at either extreme (Sutton and Barto 1998). The concept of eligibility traces can also be applied to TD control methods such as Sarsa and Q-learning, which can create more efficient learning and produce better performance (Rummery 1995). Multi-step TD methods are usually thought of in terms of an average of many multi-step returns of differing lengths and are often associated with eligibility traces, as is the case with TD(λ). However, it is also natural to think of them in terms of individual n-step returns with their associated n-step backup (Sutton and Barto 1998). We refer to each of these individual backups as atomic backups, whereas the combination of several atomic backups of different lengths creates a compound backup. In the existing literature, it is not clear how best to extend one-step Expected Sarsa to a multi-step algorithm. The Tree-backup algorithm was originally presented as a method to perform off-policy evaluation when the behaviour policy The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18)