Off-policy Multi-step Q-learning

1 INTRODUCTION . In recent years , Q-learning ( Watkins and Dayan , 1992 ) has achieved major successes in a broad range of areas by employing deep neural networks ( Mnih et al. , 2015 ; Silver et al. , 2018 ; Lillicrap et al. , 2016 ) , including environments of higher complexity ( Riedmiller et al. , 2018 ) and even in first real world applications ( Haarnoja et al. , 2019 ) . Due to its off-policy update , Q-learning can leverage transitions collected by any policy which makes it more data-efficient compared to onpolicy methods . Deep Q-learning , however , still has a very high demand for data samples which is limiting with regard to robot applications . One reason for the low data-efficiency is the long temporal horizon the reward signal has to propagate through . Data-efficiency of on-policy Temporal-Difference methods can be enhanced by the use of n-step returns , where a Monte Carlo rollout of length n is combined with a bootstrap of the value function . To employ n-step returns in an off-policy setting , subtrajectories of the exploratory policy have to be stored . These stored multi-step returns , however , will differ from the true value of the target-policy . In order to benefit from n-step data , the replay buffer has to be restricted in size or n has to be set to a small value to keep the samples close to the target-policy ( Barth-Maron et al. , 2018 ; Hessel et al. , 2018 ) . To avoid these problems , a dynamics model can be used for imaginary rollouts , the so-called Model-based Value Expansion ( MVE ) ( Feinberg et al. , 2018 ) . Alternatively , the full return can be composed of value functions with increasing discount , an approach called TD ( ∆ ) ( Romoff et al. , 2019 ) . In this work , we define a model-free Temporal-Difference formulation which follows the idea of multi-step learning while remaining off-policy . Our contributions are threefold . First , we introduce the Composite Q-learning algorithm . For its formulation , we define Truncated Q-functions , representing the return for the first n steps of a target-policy rollout w.r.t . the full action-value . In addition , we introduce Shifted Q-functions which represent the farsighted return after this truncated rollout . Both are then combined in a mutual recursive definition of the Q-function for the final algorithm . Second , we evaluate MVE within TD3 , leading to MVE-TD3 . And third , we introduce TD3 ( ∆ ) , an extension of TD ( ∆ ) to deep Q-learning . We discuss related work in Section 2 , describe the theoretical background in Section 3 and define Composite Q-learning , MVE-TD3 and TD3 ( ∆ ) in Section 4 . By breaking down the long-term return into a composition of several short-term predictions , our method increases data-efficency which we show in the tabular case and for three simulated robot tasks in Section 5 . We then conclude in Section 6 . 2 RELATED WORK . In order to correct for the deviation from the current target-policy , several methods suggest the use of Importance Sampling ( Precup et al. , 2000 ; 2001 ; Munos et al. , 2016 ) . Approaches based on Importance Sampling , however , can come with a vast increase in variance or can be of high cost and are not easily applicable to deterministic policies . One way to remain off-policy in multi-step Q-learning is to get the Monte Carlo rollout on the basis of the current target-policy applied to a learned dynamics model ( Feinberg et al. , 2018 ; Buckman et al. , 2018 ) . Due to accumulating errors of single-step models , this can lead to severe stability issues in the Q-update , which is also the conclusion of Feinberg et al . ( 2018 ) . The authors suggest to average all intermediate i-step returns to smoothen out the model error , the so-called TD-k trick . In contrast to our work , Feinberg et al . assume to have access to the true reward function . In order to balance the length of the model-assisted rollout , Buckman et al . ( 2018 ) couple it with an uncertainty estimate from value function and model ensembles . We alleviate the problem of accumulating error by estimating a multi-step dynamics model implicitly via consecutive bootstrapping . Most related to our approach is TD ( ∆ ) ( Romoff et al. , 2019 ) . Romoff et al . formalize a Bellmanoperator over the differences between value functions of increasing discount values . Their approach is on-policy and can therefore benefit directly from n-step returns . We extend TD ( ∆ ) to the offpolicy case below . Q-learning can also be extended to a SARSA-like tree-backup called Q ( σ ) ( Asis et al. , 2018 ; Hernandez-Garcia and Sutton , 2018 ) . However , it is an open question how to adjust this idea to continuous action-spaces and deterministic policies . In the Hybrid Reward Architecture ( HRA ) , van Seijen et al . ( 2017 ) suggest a decomposition of the reward and the estimation of value functions for each part of this decomposition which are then combined as an approximation of the full return . HRA addresses the problem of complex rewards and is thus complementary to our work focusing on long time scales . Concurrently to our work , Asis et al . ( 2019 ) introduced Fixed Horizon TD-learning , formalizing action-value functions for different horizons in time . In contrast to our work , however , Asis et al . define a consecutive bootstrapping formulation w.r.t . the target policies of the different truncated horizons and not the full return . A first general description of using intermediate predictions in Temporal Differences to restrict predictions to a fixed temporal horizon has been introduced in ( Sutton , 1988 ) . 3 BACKGROUND . We consider tasks modelled as Markov decision processes ( MDP ) , where an agent executes action at ∈ A in some state st ∈ S following its stochastic policy π . According to the dynamics model M of the environment , the agent transitions into some state st+1 ∈ S and receives scalar reward rt . The agent aims at maximizing the expected long-term return : Rπ ( st ) = Eaj≥t∼π , sj > t∼M T−1∑ j=t γj−trj ∣∣∣∣∣∣st , ( 1 ) where T is the ( possibly infinite ) temporal horizon of the MDP and γ ∈ [ 0 , 1 ] the discount factor . It therefore tries to find π∗ , s.t . Rπ∗ ≥ Rπ for all π . If the model of the environment is unknown , model-free methods based on the Bellman Optimality Equation over the so-called action-value can be used : Qπ ( st , at ) = Eaj > t∼π , sj > t∼M T−1∑ j=t γj−trj ∣∣∣∣∣∣st , at . ( 2 ) In the following , we abbreviate Eaj > t∼π , sj > t∼M [ ·|st , at ] by Et , π , M [ · ] . One popular representative of continuous model-free reinforcement learning is the Deep Deterministic Policy Gradient algorithm ( DDPG ) ( Lillicrap et al. , 2016 ) . In DDPG , actor µ is a deterministic mapping from states to actions , µ : S 7→ A , representing the actions that maximize the critic Qµ , i.e . µ ( st ) = arg maxaQ µ ( st , a ) . Q and µ are estimated by function approximators Q ( · , ·|θQ ) and µ ( ·|θµ ) , parameterized by θQ and θµ . The critic is optimized on the mean squared error between predictions Q ( sj , aj |θQ ) and targets yj = rj + γQ′ ( sj+1 , µ′ ( sj+1|θµ ′ ) |θQ′ ) , where Q′ and µ′ are target networks , parameterized by θQ ′ and θµ ′ . The parameters of µ are optimized following the deterministic policy gradient theorem ( Silver et al. , 2014 ) : ∇θµ ← [ 1 m ∑ j ∇aQ ( s , a|θQ ) |s=sj , a=µ ( sj |θµ ) ∇θµµ ( s|θµ ) , ( 3 ) and the parameters of the target networks are updated according to : θQ ′ ← [ ( 1− τ ) θQ′ + τθQ and θµ′ ← [ ( 1− τ ) θµ′ + τθµ , ( 4 ) with τ ∈ [ 0 , 1 ] . TD3 ( Fujimoto et al. , 2018 ) adds three adjustments to vanilla DDPG . First , the minimum prediction of two distinct critics is taken for target calculation to alleviate overestimation bias , an approach belonging to the family of Double Q-learning algorithms ( van Hasselt et al. , 2016 ) . Second , Gaussian smoothing is applied to the target-policy , addressing the variance in updates . Third , actor and target networks are updated every d-th gradient step of the critic , to account for the problem of moving targets . 4 OFF-POLICY MULTI-STEP Q-LEARNING . In this section , we introduce the Composite Q-learning algorithm , an off-policy multi-step reinforcement learning method to enhance data-efficiency , along with MVE-TD3 and TD3 ( ∆ ) as baselines . 4.1 COMPOSITE Q-LEARNING . The main motivation behind this work is the assumption that learning values on short time scales can be achieved faster than for the full temporal horizon of a task , which can be prohibitively long . As Murphy ( 2005 ) shows for the fixed-batch fitted Q-iteration case , the number of samples needed to achieve a certain generalization error is exponential in the horizon of the MDP . As discussed in Jin et al . ( 2018 ) , Q-learning with UCB-exploration has total regret polynomial in the horizon of the MDP and exponential with -greedy exploration ( Kearns and Singh , 2002 ) . We thus argue that truncated horizons for a fixed MDP translate to lower sample complexity of value-estimation . Building upon this idea , we estimate the return of n-step rollouts of the target-policy via Truncated Q-functions which we then combine to the full return with model-free Shifted Q-functions , an approach we call Composite Q-learning , while remaining purely off-policy . Since these quantities can not be estimated directly from single-step transitions , we introduce a consecutive bootstrapping scheme based on intermediate predictions . For an overview , see Figure 1a . The full algorithm is in the appendix . Code based on the implementation of TD31 can be found in the supplementary . 4.1.1 TRUNCATED Q-FUNCTIONS . In order to formalize the off-policy estimation of n-step returns , assume that n ( T − 1 ) and that ( T − 1− t ) mod n = 0 for task horizon T . We make use of the following observation : Qπ ( st , at ) = Et , π , M [ rt + γrt+1 + γ 2rt+2 + γ 3rt+3 + · · ·+ γT−1rT−1 ] = Et , π , M [ t+n−1∑ j=t γj−trj + γn t+2n−1∑ j=t+n γj− ( t+n ) rj + · · ·+ γT−n T−1∑ j=T−n γj− ( T−n ) rj ] . ( 5 ) That is , we can define the action-value as the combination of partial sums of length n. We can then define the Truncated Q-function as Qπn ( st , at ) = Et , π , M [ ∑t+n−1 j=t γ j−trj ] , which we plug into Equation ( 5 ) : Qπ ( st , at ) = Et , π , M [ Q π n ( st , at ) + γ nQπn ( st+n , at+n ) + · · ·+ γT−nQπn ( sT−n , aT−n ) ] . ( 6 ) Theorem 1 . Let Qπ1 ( st , at ) = rt be the one-step Truncated Q-function and Qπi > 1 ( st , at ) = rt + γEt , π , M [ Q π i−1 ( st+1 , at+1 ) ] the i-step Truncated Q-function . Then Q π i ( st , at ) represents the truncated return Qπi ( st , at ) = Et , π , M [ ∑t+i−1 j=t γ j−trj ] . Following Theorem 1 ( the proof can be found in the supplementary ) , we approximate Qπn ( st , at ) off-policy via consecutive bootstrapping . Let QTr ( · , ·|θQTr ) denote a function approximator with parameters θQ Tr and n outputs , subsequently called heads , estimating Qπi . Each output Q Tr i boot- straps from the prediction of the preceding head , with the first approximating the immediate reward function . The targets are therefore given by : yTrj,1 = rj and y Tr j , i > 1 = rj + γQ Tr′ i−1 ( sj+1 , µ ′ ( sj+1|θµ ′ ) |θQTr′i−1 ) , ( 7 ) where µ′ corresponds to the actor maximizing the full Q-value as defined in Section 3 . That is , QTr represents evaluations of µ at different stages of truncation and yTrj , i < n serve as intermediate predictions to get yTrj , n . We then only use Q Tr n , which implements the full n-step return , as the first part of the composition of the Q-target . Please note that in order to estimate Equation ( 6 ) , the dynamics model would be needed to get st+c·n of a rollout starting in st . In the following , we describe an approach to achieve an estimation of Equation ( 6 ) model-free .

Summary:   This paper proposes to split the value function into two separately learned components (a short-term truncated value function, and a long-term shifted value function) suggesting the short term truncated returns should learn faster as compared to the tail of the returns. They provide temporal difference formulations for a truncated value function and shifted value function, enabling efficient learning of the two components. They also provide derivations of other similar approaches to the off-policy case. Finally, they compare their algorithm to several approaches on a subset of the MuJoCo tasks, and a novel tabular domain.