Off-Policy Reinforcement Learning with Delayed Rewards

1 INTRODUCTION . Deep reinforcement learning ( RL ) aims at maximizing the cumulative reward of a MDP . To apply RL algorithms , the reward has to be given at every state-action pair in general ( i.e. , r ( s , a ) ) . With a good and high quality reward function , RL can achieve remarkable performance , e.g . AlphaGo Zero for Go ( Silver et al. , 2017 ) , DQN ( Mnih et al. , 2015 ) for Atari , SAC ( Haarnoja et al. , 2018b ) for robot control e.t.c . Recently , RL has been applied in many other real world settings beyond games and locomotion control . This includes industrial process control ( Hein et al. , 2017 ) , traffic optimization ( Gong et al. , 2019 ; Lin et al. , 2018 ) , molecular design ( Olivecrona et al. , 2017 ) and resource allocation ( Xu et al. , 2018 ) . However , in many of these real-world scenarios , Markovian and instant per-step rewards are hard or impossible to obtain or even clearly defined . In practice , it becomes more reasonable to use a delayed reward of several consecutive steps as feedback . For example , in traffic congestion reduction ( Gong et al. , 2019 ) , the amount of decreased congestion for a single traffic light switch is hard to define in practice while it is more adequate to use the average routing time consumed for the vehicles as the feedback . The latter is a delayed reward which can only be obtained after the vehicles have passed the congestion ( long after a single switch ) . In molecular design , only the molecule formed after the final operation can provide a meaningful evaluation for the whole process ( Olivecrona et al. , 2017 ) . In locomotion control where the feedback is generated by the interaction with the large environment , it is usually the case that the frequency of the rewards generated from the environment ’ s sensors is much lower than the frequency of the robot control , thus the rewards are given only every so often . Despite the importance and prevalence of the delayed reward setting , very few previous research in RL has been focusing on problems with non-Markovian properties and delayed rewards . Besides , current RL algorithms lack theoretical guarantee under non-Markovian rewards and perform unsatisfactorily in practice ( Gangwani et al. , 2020 ) . Thus , in this paper , our goal is to better understand the properties of RL with delayed rewards and introduce a new algorithm that can handle the delayed reward both theoretically and practically . A key to our approach lies in the definition of the past-invariant delayed reward MDPs . Based on this definition , theoretical properties and algorithmic implications are discussed to motivate the design of a new practical algorithm , which explicitly decompose the value function into components of both historical ( H ) and current ( C ) step information . We also propose a number of ways to approximate such HC decomposition , which can be readily incorporated into existing off-policy learning algorithms . Experiments demonstrate that such approximation can improve training efficiency and robustness when rewards are delayed . 2 PROBLEM FORMULATION . In order to characterize the delayed and non-Markovian reward signals , we introduce the Delayed Reward Markov Decision Process ( DRMDP ) . In this work , we focus on DRMDP that satisfies the Past-Invariant ( PI ) condition , which is satisfied in many real-world settings with delayed rewards . 2.1 PAST-INVARIANT DELAYED REWARD MDPS . In DRMDP , the transition of the environment is still Markovian and the agent can observe and interact with the environment instantly . However , rewards may be non-Markovian and delayed and are observed only once every few steps . More specifically , the time steps are divided into consecutive signal intervals of random lengths , and the reward signal generated during a signal interval may depend on the state-action sequence and is observed only at the end of the interval . We formally define DRMDP as follows . Definition 1 ( DRMDP ) . A Delayed Reward Markov Decision Process M = ( S , A , p , qn , r , γ ) is described by the following parameters . 1 . The state and action spaces are S and A respectively . 2 . The Markov transition function is p ( s′|s , a ) for each ( s , a ) ∈ S×A ; the initial state distribution is p ( s0 ) . 3 . The signal interval length is distributed according to qn ( · ) , i.e. , for the i-th signal interval , its length ni is independently drawn from qn ( · ) . 4 . The reward function r defines the expected reward generated for each signal interval ; suppose . τi = τt : t+ni = ( s , a ) t : t+ni = ( ( st , at ) , . . . ( st+n−1 , at+ni−1 ) ) is the state-action sequence during the i-th signal interval of length ni , then the expected reward for this interval is r ( τi ) . 5 . The reward discount factor is γ . In this work , we focus on the infinite-horizon DRMDP . We use ( τ , n ) = ( τi , ni ) i∈ { 1,2,3 , ... } to denote a trajectory , where τi is the state-action sequence during the i-th signal interval and ni is the corresponding length . We also let ti be first time step of the i-th signal interval , i.e. , ti =∑i−1 j=1 nj . Note that the reward r ( τi ) is revealed at time ti + ni − 1 = ti+1 − 1 . We finally define the discounted cumulative reward of the trajectory ( τ , n ) by R ( τ , n ) : = ∑∞ i=1 γ ti+1−1r ( τi ) . The objective for DRMDP is to learn a policy π that maximized the expected discounted cumulative reward J ( π ) : = E ( τ , n ) ∼π [ R ( τ , n ) ] . Augmented Policy Class . In DRMDP , the Markov policy class { π = π ( at|st ) } might not achieve satisfactory performance because of the non-Markovian nature of the reward function . Theoretically speaking , we need a more general policy class Πτ = { π = π ( at|τti : t ◦ st ) } ( i is the index of the signal interval that t belongs to ) . A formal statement is included in Appendix B . Unfortunately , policy optimization in such an exponentially large space Πτ is hard in practice , especially for highdimensional problems . Thus , as a trade-off , we consider the policy class Πs = { π = π ( at|st , t − ti ) } , which resembles the traditional Markov policy class , but is augmented with an extra parameter indicting the relative index of the current time step in the signal interval . Moreover , we focus on the PI-DRMDP problems , in which our algorithmic framework has theoretical guarantees . Definition 2 ( PI-DRMDP ) . A Past-Invariant Delayed Reward Markov Decision Process is a DRMDP M = ( S , A , p , qn , r , γ ) whose reward function r satisfies the following Past-Invariant ( PI ) condition : for any two trajectory segments τ1 and τ2 of the same length , and for any two equallength trajectory segments τ ′1 and τ ′ 2 such that the concatenated trajectories τa ◦ τ ′b are feasible under the transition dynamics p for all a , b ∈ { 1 , 2 } , it holds that r ( τ1 ◦ τ ′1 ) > r ( τ1 ◦ τ ′2 ) ⇐⇒ r ( τ2 ◦ τ ′1 ) > r ( τ2 ◦ τ ′2 ) . Roughly speaking , in PI-DRMDP , the relative credit for different actions at each st only depends on the experience in the future and is invariant of the past . This property may relieve the agent from considering the past experience for decision making . A simple example of r with PI condition is r ( τt : t+n ) = ∑t+n−1 i=t r̂ ( si , ai ) , where r̂ is a per-step reward function . This kind of tasks is studied in many previous work ( Zheng et al. , 2018 ; Oh et al. , 2018 ; Klissarov & Precup , 2020 ) . We refer this kind of reward functions as the sum-form . In the worst case , there exists some bizarre reward design in PI-DRMDPs that still requires the optimal policy to take history information into consideration ( see Appendix B ) . Despite the suboptimality issue , we will show that our algorithmic framework guarantees policy improvement in Πs , which is sufficient to support the design of actor-critic methods . Consequently , we are able to put forward a practical RL algorithm that can achieve the SOTA performance in delayed-reward tasks . The overall contribution of our work is summarized in Table 1 and we leave the optimality of the DRMDP problems as future works . General Reward Function . In Definition 1 , we define the reward as a function of the state-action sequence of its signal interval . In general , we may allow reward functions with longer inputs which overlap with the previous signal intervals , e.g. , maximal overlapping of c steps , r ( τti−c : ti+ni ) . The theoretical analysis in Section 3.1 and the empirical method in Section 3.2 can be directly extended to this general reward function . We provide detailed discussions in Appendix B for the general definition while we only consider c = 0 in the main text for the simplicity of the exposition . 2.2 OFF-POLICY RL IN PI-DRMDP . Deep Reinforcement Learning has achieved great success in solving high-dimensional MDPs problems , among which the off-policy actor-critic algorithms SAC ( Haarnoja et al. , 2018a ) and TD3 ( Fujimoto et al. , 2018 ) are the most widely used ones . However , since the rewards are delayed and non-Markovian in PI-DRMDP , directly applying these SOTA off-policy RL algorithms faces many challenges that degrade the learning performance . In this subsection , we briefly discuss the problems that arise in critic learning based on TD3 and similar problems also exist for SAC . First , in PI-DRMDP , value evaluation with off-policy samples brings in off-policy bias . TD3 minimizes over φ w.r.t . Lφ = ED [ ( Rt + γQ̂φ ( st+1 , a ′ t+1 ) −Qφ ( st , at ) ) 2 ] , ( 1 ) where ( st , at , Rt , st+1 ) is sampled from the replay bufferD , Rt = r ( τi ) if t = ti+1−1 and i = i ( t ) is the index of the reward interval that t belongs to , and Rt = 0 otherwise . Q̂φ represents the target value of the next state and a′t+1 is sampled from the smoothed version of the policy π . Furthermore , in practical implementation , samples in the replay buffer D are not sampled from a single behavior policy β as in traditional off-policy RL ( Sutton & Barto , 2018 ) . Instead , the behavior policy changes as the policy gets updated . Thus , we assume that samples are collected from a sequence of behavior policies β = { βk } Kk=1 . In the delayed reward setting , since the reward r ( τi ) depends on the trajectory of the whole signal interval ( i.e. , τi ) instead of a single step , the function Qφ ( st , at ) learned via Eq . ( 1 ) will also have to depend on the trajectory in the signal interval upto time t ( i.e. , τti : t ) rather than the single stateaction pair at time t. Since the samples are collected under a sequence of behavior polices β , different behavior policy employed at state st may lead to different distribution over τti : t in Eq . ( 1 ) . Consequently , Qφ ( st , at ) will be affected by this discrepancy and may fail to assign the accurate expected reward for the current policy . Please refer to Appendix B for detailed discussions . Second , in addition to the issue resulted from off-policy samples , Qφ ( st , at ) learned in Eq . ( 1 ) with on-policy samples in D may still be problematic . We formally state this problem via a simple sumform PI-DRMDP in Appendix B whose optimal policy is in Πs . In this example , the fix point of Eq . ( 1 ) fails to assign the actual credit and thus misleads policy iteration even when the pre-update policy is already the optimal . We refer this problem as the fixed point bias . Last but not least , the critic learning via Eq . ( 1 ) would suffer a relative large variance . Since Rt varies between 0 and r ( τi ) , minimization of TD-error ( Lφ ) with a mini-batch of data has large variance . As a result , the approximated critic will be noisy and relative unstable . This will further effect the policy gradients in Eq . ( 2 ) . ∇θJ ( πθ ) = ED [ ∇atQφ ( st , at ) ∣∣∣ πθ ( st ) ∇θπθ ( st ) ] ( 2 ) To sum up , directly applying SOTA off-policy algorithms in PI-DRMDP will suffer from multiple problems ( off-policy bias , fixed point bias , large traning noise , etc ) . Indeed , these problems result in a severe decline in performance even in the simplest task when n = 5 ( Gangwani et al. , 2020 ) .

Summary: This paper proposed a decision process that is Markov in state transitions but non-Markov in reward. The paper argued that this new decision process is a suitable model for applications where rewards are delayed. For this decision process, the paper proposed analogues of action-value function and policy improvement theorem. It further proposed to decompose the action-value into two parts, one of which only depends on the current state-action pair while the other one depends on the history. It then proposed an algorithm that extends SAC and estimates the two parts separately. Empirical results showed improved performance over SAC on delayed reward versions of several continuous control problems.