Reinforcement Learning with Random Delays

1 INTRODUCTION This article is concerned with the Reinforcement Learning ( RL ) scenario depicted in Figure 1 , which is commonly encountered in real-world applications ( Mahmood et al. , 2018 ; Fuchs et al. , 2020 ; Hwangbo et al. , 2017 ) . Oftentimes , actions generated by the agent are not immediately applied in the environment , and observations do not immediately reach the agent . Such environments have mainly been studied under the unrealistic assumption of constant delays ( Nilsson et al. , 1998 ; Ge et al. , 2013 ; Mahmood et al. , 2018 ) . Here , prior work has proposed different planning algorithms which naively try to undelay the environment by simulating future observations ( Walsh et al. , 2008 ; Schuitema et al. , 2010 ; Firoiu et al. , 2018 ) . We propose an off-policy , planning-free approach that enables lowbias and low-variance multi-step value estimation in environments with random delays . First , we study the anatomy of such environments in order to exploit their structure , defining Random-Delay Markov Decision Processes ( RDMDP ) . Then , we show how to trans- form trajectory fragments collected under one policy into trajectory fragments distributed according to another policy . We demonstrate this principle by deriving a novel off-policy algorithm ( DCAC ) based on Soft Actor-Critic ( SAC ) , and exhibiting greatly improved performance in delayed environments . Along with this work we release our code , including a wrapper that conveniently augments any OpenAI gym environment with custom delays . 2 DELAYED ENVIRONMENTS . We frame the general setting of real-world Reinforcement Learning in terms of an agent , random observation delays , random action delays , and an undelayed environment . At the beginning of each time-step , the agent starts computing a new action from the most recent available delayed observation . Meanwhile , a new observation is sent and the most recent delayed action is applied in the undelayed environment . Real-valued delays are rounded up to the next integer time-step . ∗equal contribution For a given delayed observation st , the observation delay ωt refers to the number of elapsed time-steps from when st finishes being captured to when it starts being used to compute a new action . The action delay αt refers to the number of elapsed time-steps from when the last action influencing st starts being computed to one time-step before st finishes being captured . We further refer to ωt + αt as the total delay of st. As a motivating illustration of real-world delayed setting , we have collected a dataset of communication delays between a decisionmaking computer and a flying robot over WiFi , summarized in Figure 2 . In the presence of such delays , the naive approach is to simply use the last received observation . In this case , any delay longer than one time-step violates the Markov assumption , since the last sent action becomes an unobserved part of the current state of the environment . To overcome this issue , we define a Markov Decision Process that takes into account the communication dynamics . 2.1 RANDOM DELAY MARKOV DECISION PROCESSES . To ensure the Markov property in delayed settings , it is necessary to augment the delayed observation with at least the last K sent actions . K is the combined maximum possible observation and action delay . This is required as the oldest actions along with the delayed observation describe the current state of the undelayed environment , whereas the most recent actions are yet to be applied ( see Appendix C ) . Using this augmentation suffices to ensure that the Markov property is met in certain delayed environments . On the other hand , it is possible to do much better when the delays themselves are also part of the state-space . First , this allows us to model self-correlated delays , e.g . discarding outdated actions and observations ( see Appendix A.1 ) . Second , this provides useful information to the model about how old an observation is and what actions have been applied next . Third , knowledge over the total delay allows for efficient credit assignment and off-policy partial trajectory resampling , as we show in this work . Definition 1 . A Random Delay Markov Decision Process RDMDP ( E , pω , pα ) = ( X , A , µ̃ , p̃ ) augments a Markov Decision Process E = ( S , A , µ , p ) with : ( 1 ) state-space X = S ×AK × N2 , ( 2 ) action-space A , ( 3 ) initial state distribution µ̃ ( x0 ) = µ̃ ( s , u , ω , α ) = µ ( s ) δ ( u− cu ) δ ( ω − cω ) δ ( α− cα ) , ( 4 ) transition distribution p̃ ( s′ , u′ , ω′ , α′ , r′|s , u , ω , α , a ) =fω−ω′ ( s′ , α′ , r′|s , u , ω , α , a ) pω ( ω′|ω ) pu ( u′|u , a ) , where s ∈ S is the delayed observation , u ∈ AK is a buffer of the last K sent actions , ω ∈ N is the observation delay , and α ∈ N is the action delay as defined above . To avoid conflicting with the subscript notation , we index the action buffers ’ elements using square brackets . Here , u [ 1 ] is the most recent and u [ K ] is the oldest action in the buffer . We denote slices by u [ i : j ] = ( u [ i ] , . . . , u [ j ] ) and u [ i : −j ] = ( u [ i ] , . . . , u [ K−j ] ) . We slightly override this notation and additionally define u [ 0 ] = a . The constants cu ∈ AK and cω , cα ∈ N initialize u , ω , α , since δ is the Dirac delta distribution . The transition distribution itself is composed of three parts : ( 1 ) The observation delay distribution pω modelling the evolution of observation delays . Note that this density function must represent a discrete distribution ( i.e . be a weighted sum of Dirac delta distributions ) . Furthermore , this process will repeat observations if there are no new ones available . This means that the observation delay can maximally grow by one from one time-step to the next . ( 2 ) The transition distribution for the action buffer pu ( u′|u , a ) = δ ( u′ − ( a , u [ 1 : −1 ] ) ) . ( 3 ) The distribution f∆ describing the evolution of observations , rewards and action delays ( Definition 2 ) . Definition 2 . For each change in observation delays ( ∆=ω−ω′ ) we define a variable step update distribution f∆ as f∆ ( s ′ , α′ , r′|s , u , ω , α , a ) =Es∗ , α∗ , r∗∼f∆−1 ( ·|s , u , ω , α , a ) [ p ( s ′ , r′−r∗|s∗ , u [ ω′︷ ︸︸ ︷ ω−∆+α′ ] ) pα ( α′|α∗ ) ] . ( 1 ) The base case of the recursion is f−1 ( s′ , α′ , r′ | s , u , ω , α , a ) = δ ( s′ − s ) δ ( α′ − α ) δ ( r′ ) . Here , pα is the action delay distribution which , similar to pω , must be discrete . The transition distribution of the underlying , undelayed MDP is p. The r′ − r∗ term accumulates intermediate rewards in case observations are skipped or repeated ( see Appendix A.4 ) . Since the observation delay can not increment by more than one , f−1 is used when ω is increasing , whereas f0 is used when there is no change in observation delay . A simple special case of the RDMDP is the constant observation and action delay case with pω ( ω′|ω ) = δ ( ω′ − cω ) and pα ( α′|α ) = δ ( α′ − cα ) . Here , the RDMDP reduces to a Constantly Delayed Markov Decision Process , described by Walsh et al . ( 2008 ) . In this case , the action and observation delays α , ω can be removed from the state-space as they don ’ t carry information . Examples of RDMDP dynamics are visualized in Figure 3 ( see also Appendix C ) . 3 REINFORCEMENT LEARNING IN DELAYED ENVIRONMENTS . Delayed environments as described in Section 2 are specific types of MDP , with an augmented statespace and delayed dynamics . Therefore , using this augmented state-space , traditional algorithms such as Soft Actor-Critic ( SAC ) ( Haarnoja et al. , 2018a ) ( Haarnoja et al. , 2018b ) will always work in randomly delayed settings . However , their performance will still deteriorate because of the more difficult credit assignment caused by delayed observations and rewards , on top of the exploration and generalization burdens of delayed environments . We now analyze how to compensate for the credit assignment difficulty by leveraging our knowledge about the delays ’ dynamics . One solution is to perform on-policy multi-step rollouts on sub-trajectories that are longer than the considered delays . On the other hand , on-policy algorithms are known to be sample-inefficient and therefore are not commonly used in real-world applications , where data collection is costly . This motivates the development of off-policy algorithms able to reuse old samples , such as SAC . Intuitively , in delayed environments , one should take advantage of the fact that actions only influence observations and rewards after a number of time-steps relative to the beginning of their computation ( the total delay ω + α ) . Since the delay information is part of the state-space , it can be leveraged to track the action influence through time . However , applying conventional off-policy algorithms in delayed settings leads to the following issue : the trajectories used to perform the aforementioned multi-step backups have been sampled under an outdated policy , and therefore contain outdated action buffers . In this section , we propose a method to tackle this issue by performing partial trajectory resampling . We make use of the fact that the delayed dynamics are known to simulate the effect they would have had under the current policy , effectively transforming off-policy sub-trajectories into on-policy sub-trajectories . This enables us to derive a family of efficient off-policy algorithms for randomly delayed settings . 3.1 PARTIAL TRAJECTORY RESAMPLING IN DELAYED ENVIRONMENTS . One important observation implied by Figure 3 is that , given the delayed dynamics of RDMDPs , some actions contained in the action buffer of an off-policy state did not influence the subsequent delayed observations and rewards for a number of time-steps . Therefore , if an off-policy sub-trajectory is short enough , it is possible to recursively resample its action buffers with no influence on the return . We propose the following transformation of off-policy sub-trajectories : Definition 3 . The partial trajectory resampling operator recursively updates action buffers as follows σπn ( s ∗ 1 , u ∗ 1 , ω ∗ 1 , α ∗ 1︸ ︷︷ ︸ x∗1 , r∗1 , τ ∗ n−1|x∗0 ; s1 , u1 , ω1 , α1︸ ︷︷ ︸ x1 , r1 , τn−1 ) =δ ( ( s∗1 , ω ∗ 1 , α ∗ 1 , r ∗ 1 ) − ( s1 , ω1 , α1 , r1 ) ) Ea0∼π ( ·|x∗0 ) [ δ ( u ∗ 1− ( a0 , u∗0 [ 1 : −1 ] ) ) ] σπn−1 ( τ∗n−1|x∗1 ; τn−1 ) ( 2 ) with trivial base case σ0 ( x∗0 ) = 1 This operator recursively resamples the most recent actions of each action buffer in an input subtrajectory τn , according to a new policy π . Everything else stays unchanged . A visual example is provided in Figure 4 with n = 2 and an action buffer of two actions . When resampled actions are delayed and would not affect the environment , they do not `` invalidate '' the sub-trajectory . The resampled trajectories can then be considered on-policy . Theorem 1 . The partial trajectory resampling operator σπn ( Def . 3 ) transforms off-policy trajectories into on-policy trajectories Eτn∼pµn ( ·|x0 ) [ σ π n ( τ ∗ n|x0 ; τn ) ] =pπn ( τ∗n|x0 ) ( 3 ) on the condition that none of the delayed observations depend on any of the resampled actions , i.e . ω∗t + α ∗ t ≥ t ( 4 ) where t indexes the trajectory τ∗n = ( s ∗ 1 , u ∗ 1 , ω ∗ 1 , α ∗ 1 , r ∗ 1 , . . . , s ∗ n , u ∗ n , ω ∗ n , α ∗ n , r ∗ n ) from 1 to n. The condition in Equation 4 can be understood visually with the help of Figure 3 . In the constant delay example it is fulfilled until the third time-step . After that , the observations would have been influenced by the resampled actions ( starting with a0 ) .

Summary: The paper introduces an algorithm for the case where actions have delayed effects in RL, and specifically in the case where the delay is random.  A resampling approach is applied to off policy buffered data in order to align it with the current policy and this approach is integrated into a SAC architecture, creating the new DCAC algorithm.  Empirical results in constant-delay and random-delay environments show the algorithm outperforming baselines.