Off-Policy Reinforcement Learning with Delayed Rewards

1 INTRODUCTION . Deep reinforcement learning ( RL ) aims at maximizing the cumulative reward of a MDP . To apply RL algorithms , the reward has to be given at every state-action pair in general ( i.e. , r ( s , a ) ) . With a good and high quality reward function , RL can achieve remarkable performance , e.g . AlphaGo Zero for Go ( Silver et al. , 2017 ) , DQN ( Mnih et al. , 2015 ) for Atari , SAC ( Haarnoja et al. , 2018b ) for robot control e.t.c . Recently , RL has been applied in many other real world settings beyond games and locomotion control . This includes industrial process control ( Hein et al. , 2017 ) , traffic optimization ( Gong et al. , 2019 ; Lin et al. , 2018 ) , molecular design ( Olivecrona et al. , 2017 ) and resource allocation ( Xu et al. , 2018 ) . However , in many of these real-world scenarios , Markovian and instant per-step rewards are hard or impossible to obtain or even clearly defined . In practice , it becomes more reasonable to use a delayed reward of several consecutive steps as feedback . For example , in traffic congestion reduction ( Gong et al. , 2019 ) , the amount of decreased congestion for a single traffic light switch is hard to define in practice while it is more adequate to use the average routing time consumed for the vehicles as the feedback . The latter is a delayed reward which can only be obtained after the vehicles have passed the congestion ( long after a single switch ) . In molecular design , only the molecule formed after the final operation can provide a meaningful evaluation for the whole process ( Olivecrona et al. , 2017 ) . In locomotion control where the feedback is generated by the interaction with the large environment , it is usually the case that the frequency of the rewards generated from the environment ’ s sensors is much lower than the frequency of the robot control , thus the rewards are given only every so often . Despite the importance and prevalence of the delayed reward setting , very few previous research in RL has been focusing on problems with non-Markovian properties and delayed rewards . Besides , current RL algorithms lack theoretical guarantee under non-Markovian rewards and perform unsatisfactorily in practice ( Gangwani et al. , 2020 ) . Thus , in this paper , our goal is to better understand the properties of RL with delayed rewards and introduce a new algorithm that can handle the delayed reward both theoretically and practically . A key to our approach lies in the definition of the past-invariant delayed reward MDPs . Based on this definition , theoretical properties and algorithmic implications are discussed to motivate the design of a new practical algorithm , which explicitly decompose the value function into components of both historical ( H ) and current ( C ) step information . We also propose a number of ways to approximate such HC decomposition , which can be readily incorporated into existing off-policy learning algorithms . Experiments demonstrate that such approximation can improve training efficiency and robustness when rewards are delayed . 2 PROBLEM FORMULATION . In order to characterize the delayed and non-Markovian reward signals , we introduce the Delayed Reward Markov Decision Process ( DRMDP ) . In this work , we focus on DRMDP that satisfies the Past-Invariant ( PI ) condition , which is satisfied in many real-world settings with delayed rewards . 2.1 PAST-INVARIANT DELAYED REWARD MDPS . In DRMDP , the transition of the environment is still Markovian and the agent can observe and interact with the environment instantly . However , rewards may be non-Markovian and delayed and are observed only once every few steps . More specifically , the time steps are divided into consecutive signal intervals of random lengths , and the reward signal generated during a signal interval may depend on the state-action sequence and is observed only at the end of the interval . We formally define DRMDP as follows . Definition 1 ( DRMDP ) . A Delayed Reward Markov Decision Process M = ( S , A , p , qn , r , γ ) is described by the following parameters . 1 . The state and action spaces are S and A respectively . 2 . The Markov transition function is p ( s′|s , a ) for each ( s , a ) ∈ S×A ; the initial state distribution is p ( s0 ) . 3 . The signal interval length is distributed according to qn ( · ) , i.e. , for the i-th signal interval , its length ni is independently drawn from qn ( · ) . 4 . The reward function r defines the expected reward generated for each signal interval ; suppose . τi = τt : t+ni = ( s , a ) t : t+ni = ( ( st , at ) , . . . ( st+n−1 , at+ni−1 ) ) is the state-action sequence during the i-th signal interval of length ni , then the expected reward for this interval is r ( τi ) . 5 . The reward discount factor is γ . In this work , we focus on the infinite-horizon DRMDP . We use ( τ , n ) = ( τi , ni ) i∈ { 1,2,3 , ... } to denote a trajectory , where τi is the state-action sequence during the i-th signal interval and ni is the corresponding length . We also let ti be first time step of the i-th signal interval , i.e. , ti =∑i−1 j=1 nj . Note that the reward r ( τi ) is revealed at time ti + ni − 1 = ti+1 − 1 . We finally define the discounted cumulative reward of the trajectory ( τ , n ) by R ( τ , n ) : = ∑∞ i=1 γ ti+1−1r ( τi ) . The objective for DRMDP is to learn a policy π that maximized the expected discounted cumulative reward J ( π ) : = E ( τ , n ) ∼π [ R ( τ , n ) ] . Augmented Policy Class . In DRMDP , the Markov policy class { π = π ( at|st ) } might not achieve satisfactory performance because of the non-Markovian nature of the reward function . Theoretically speaking , we need a more general policy class Πτ = { π = π ( at|τti : t ◦ st ) } ( i is the index of the signal interval that t belongs to ) . A formal statement is included in Appendix B . Unfortunately , policy optimization in such an exponentially large space Πτ is hard in practice , especially for highdimensional problems . Thus , as a trade-off , we consider the policy class Πs = { π = π ( at|st , t − ti ) } , which resembles the traditional Markov policy class , but is augmented with an extra parameter indicting the relative index of the current time step in the signal interval . Moreover , we focus on the PI-DRMDP problems , in which our algorithmic framework has theoretical guarantees . Definition 2 ( PI-DRMDP ) . A Past-Invariant Delayed Reward Markov Decision Process is a DRMDP M = ( S , A , p , qn , r , γ ) whose reward function r satisfies the following Past-Invariant ( PI ) condition : for any two trajectory segments τ1 and τ2 of the same length , and for any two equallength trajectory segments τ ′1 and τ ′ 2 such that the concatenated trajectories τa ◦ τ ′b are feasible under the transition dynamics p for all a , b ∈ { 1 , 2 } , it holds that r ( τ1 ◦ τ ′1 ) > r ( τ1 ◦ τ ′2 ) ⇐⇒ r ( τ2 ◦ τ ′1 ) > r ( τ2 ◦ τ ′2 ) . Roughly speaking , in PI-DRMDP , the relative credit for different actions at each st only depends on the experience in the future and is invariant of the past . This property may relieve the agent from considering the past experience for decision making . A simple example of r with PI condition is r ( τt : t+n ) = ∑t+n−1 i=t r̂ ( si , ai ) , where r̂ is a per-step reward function . This kind of tasks is studied in many previous work ( Zheng et al. , 2018 ; Oh et al. , 2018 ; Klissarov & Precup , 2020 ) . We refer this kind of reward functions as the sum-form . In the worst case , there exists some bizarre reward design in PI-DRMDPs that still requires the optimal policy to take history information into consideration ( see Appendix B ) . Despite the suboptimality issue , we will show that our algorithmic framework guarantees policy improvement in Πs , which is sufficient to support the design of actor-critic methods . Consequently , we are able to put forward a practical RL algorithm that can achieve the SOTA performance in delayed-reward tasks . The overall contribution of our work is summarized in Table 1 and we leave the optimality of the DRMDP problems as future works . General Reward Function . In Definition 1 , we define the reward as a function of the state-action sequence of its signal interval . In general , we may allow reward functions with longer inputs which overlap with the previous signal intervals , e.g. , maximal overlapping of c steps , r ( τti−c : ti+ni ) . The theoretical analysis in Section 3.1 and the empirical method in Section 3.2 can be directly extended to this general reward function . We provide detailed discussions in Appendix B for the general definition while we only consider c = 0 in the main text for the simplicity of the exposition . 2.2 OFF-POLICY RL IN PI-DRMDP . Deep Reinforcement Learning has achieved great success in solving high-dimensional MDPs problems , among which the off-policy actor-critic algorithms SAC ( Haarnoja et al. , 2018a ) and TD3 ( Fujimoto et al. , 2018 ) are the most widely used ones . However , since the rewards are delayed and non-Markovian in PI-DRMDP , directly applying these SOTA off-policy RL algorithms faces many challenges that degrade the learning performance . In this subsection , we briefly discuss the problems that arise in critic learning based on TD3 and similar problems also exist for SAC . First , in PI-DRMDP , value evaluation with off-policy samples brings in off-policy bias . TD3 minimizes over φ w.r.t . Lφ = ED [ ( Rt + γQ̂φ ( st+1 , a ′ t+1 ) −Qφ ( st , at ) ) 2 ] , ( 1 ) where ( st , at , Rt , st+1 ) is sampled from the replay bufferD , Rt = r ( τi ) if t = ti+1−1 and i = i ( t ) is the index of the reward interval that t belongs to , and Rt = 0 otherwise . Q̂φ represents the target value of the next state and a′t+1 is sampled from the smoothed version of the policy π . Furthermore , in practical implementation , samples in the replay buffer D are not sampled from a single behavior policy β as in traditional off-policy RL ( Sutton & Barto , 2018 ) . Instead , the behavior policy changes as the policy gets updated . Thus , we assume that samples are collected from a sequence of behavior policies β = { βk } Kk=1 . In the delayed reward setting , since the reward r ( τi ) depends on the trajectory of the whole signal interval ( i.e. , τi ) instead of a single step , the function Qφ ( st , at ) learned via Eq . ( 1 ) will also have to depend on the trajectory in the signal interval upto time t ( i.e. , τti : t ) rather than the single stateaction pair at time t. Since the samples are collected under a sequence of behavior polices β , different behavior policy employed at state st may lead to different distribution over τti : t in Eq . ( 1 ) . Consequently , Qφ ( st , at ) will be affected by this discrepancy and may fail to assign the accurate expected reward for the current policy . Please refer to Appendix B for detailed discussions . Second , in addition to the issue resulted from off-policy samples , Qφ ( st , at ) learned in Eq . ( 1 ) with on-policy samples in D may still be problematic . We formally state this problem via a simple sumform PI-DRMDP in Appendix B whose optimal policy is in Πs . In this example , the fix point of Eq . ( 1 ) fails to assign the actual credit and thus misleads policy iteration even when the pre-update policy is already the optimal . We refer this problem as the fixed point bias . Last but not least , the critic learning via Eq . ( 1 ) would suffer a relative large variance . Since Rt varies between 0 and r ( τi ) , minimization of TD-error ( Lφ ) with a mini-batch of data has large variance . As a result , the approximated critic will be noisy and relative unstable . This will further effect the policy gradients in Eq . ( 2 ) . ∇θJ ( πθ ) = ED [ ∇atQφ ( st , at ) ∣∣∣ πθ ( st ) ∇θπθ ( st ) ] ( 2 ) To sum up , directly applying SOTA off-policy algorithms in PI-DRMDP will suffer from multiple problems ( off-policy bias , fixed point bias , large traning noise , etc ) . Indeed , these problems result in a severe decline in performance even in the simplest task when n = 5 ( Gangwani et al. , 2020 ) .

Summary: The paper studies the problem of delayed reward, which imposes non-Markov behavior that hinders performance of "standard" RL algorithms. It then proposes a new Q-function definition based on segments of reward, together with a decomposition to historical-current parts. Lastly, the authors add experimental comparisons to several recently published methods.