Acting in Delayed Environments with Non-Stationary Markov Policies

1 INTRODUCTION . The body of work on reinforcement learning ( RL ) and planning problem setups has grown vast in recent decades . Examples for such distinctions are different objectives and constraints , assumptions on access to the model or logged trajectories , on-policy or off-policy paradigms , etc . ( Puterman , 2014 ) . However , the study of delay in RL remains scarce . It is almost always assumed the action is executed as soon as the agent chooses it . This assumption seldom holds in real-world applications ( DulacArnold et al. , 2019 ) . Latency in action execution can either stem from the increasing computational complexity of modern systems and related tasks , or the infrastructure itself . The wide range of such applications includes robotic manipulation , cloud computing , financial trading , sensor feedback in autonomous systems , and more . To elaborate , consider an autonomous vehicle required for immediate response to a sudden hazard on the highway . Driving at high speed , it suffers from perception module latency when inferring the surrounding scene , as well as delay in actuation once a decision has been made . While the latter phenomenon is an instance of execution delay , the former corresponds to observation delay . These two types of delay are in fact equivalent and can thus be treated with the same tools ( Katsikopoulos & Engelbrecht , 2003 ) . Related works . The notion of delay is prominent in control theory with linear time-invariant systems ( Bar-Ilan & Sulem , 1995 ; Dugard & Verriest , 1998 ; Richard , 2003 ; Fridman , 2014 ; Bruder & Pham , 2009 ) . While the delayed control literature is vast , our work intersects with it mostly in motivation . In the above control theory formulations , the system evolves according to some known diffusion or stochastic differential equation . Differently , the discrete-time MDP framework does not require any structural assumption on the transition function or reward . ˚Equal contribution A few works consider a delay in the reward signal rather than in observation or execution . Delayed reward has been studied on multi-armed bandits for deterministic and stochastic latencies ( Joulani et al. , 2013 ) and for the resulting arm credit assignment problem ( Pike-Burke et al. , 2017 ) . In the MDP setting , Campbell et al . ( 2016 ) proposed a Q-learning variant for reward-delay that follows a Poisson distribution . Katsikopoulos & Engelbrecht ( 2003 ) considered three types of delay : observation , execution , and reward . Chen et al . ( 2020b ) studied execution delay on multi-agent systems . The above works on MDPs employed state-augmentation with a primary focus on empirical evaluation of the degradation introduced by the delay . In this augmentation method , all missing information is concatenated with the original state to overcome the partial observability induced by the delay . The main drawback of this embedding method is the exponential growth of the state-space with the delay value ( Walsh et al. , 2009 ; Chen et al. , 2020a ) and , in the case of ( Chen et al. , 2020b ) , an additional growth that is polynomial with the number of agents . Walsh et al . ( 2009 ) avoided state-augmentation in MDPs with delayed feedback via a planning approach . By assuming the transition kernel to be close to deterministic , their model-based simulation ( MBS ) algorithm relies on a most-likely present state estimate . Since the Delayed-Q algorithm we devise here resembles to MBS in spirit , we highlight crucial differences between them : First , MBS is a conceptual algorithm that requires the state-space to be finite or discretized . This makes it highly sensitive to the state-space size , as we shall demonstrate in Sec . 7 [ Fig . 5 ( c ) ] , prohibiting it from running on domains like Atari . Differently , Delayed-Q works with the original , possibly continuous state-space . Second , MBS is an offline algorithm : it estimates a surrogate , non-delayed MDP from samples , and only then does it solve that MDP to obtain the optimal policy ( Walsh et al. , 2009 ) [ Alg . 2 , l. 16 ] . This is inapplicable to large continuous domains and is again in contrast to Delayed-Q . Recent studies considered a concurrent control setting where action sampling occurs simultaneously with state transition ( Ramstedt & Pal , 2019 ; Xiao et al. , 2020 ) . Both assumed a single action selection between two consecutive observations , thus reducing the problem to an MDP with execution delay of m “ 1 . Chen et al . ( 2020a ) have generalized it to an arbitrary number of actions between two observations . Hester & Stone ( 2013 ) addressed execution delay in the braking control of autonomous vehicles with a relatively low delay of m § 3 . All these works employ state-augmentation to preserve the Markov property of the process , whereas we are interested whether this restriction can be lifted . Additionally , they studied policy-gradient ( policy-based ) methods , while we introduce a Q-learning style ( value-based ) algorithm . Likewise , Firoiu et al . ( 2018 ) proposed a modified version of the policy-based IMPALA ( Espeholt et al. , 2018 ) which is evaluated on a single video game with delay values of m § 7 . To the best of our knowledge , our work is the first to tackle a delayed variant of the popular Atari suite ( Bellemare et al. , 2013 ) . Contributions . Revisiting RL with execution delay both in theory and practice , we introduce : 1 . Analysis of a delayed MDP quantifying the trade-off between stochasticity and delay . 2 . The first tight upper and lower complexity bounds on policy iteration for action-augmented MDPs . We stress that this is also a contribution to general RL theory of non-delayed MDPs . 3 . A new formalism of execution-delay MDPs that avoids action-embedding . Using it , we prove that out of the larger set of history-dependent policies , restricting to non-stationary deterministic Markov policies is sufficient for optimality in delayed MDPs . We also derive a Bellman-type recursion for a delayed value function . 4 . A model-based DQN-style algorithm that yields non-stationary Markov policies . Our algorithm outperforms the alternative standard and state-augmented DDQN in 39 of 42 experiments spanning over 3 environment categories and delay of up to m “ 25 . 2 PRELIMINARIES : NON-DELAYED STANDARD MDP . Here , we describe the standard non-delayed MDP setup . Later , in Sec . 5 , we introduce its generalization to the delayed case . We follow and extend notations from ( Puterman , 2014 ) [ Sec . 2.1. ] . An infinite horizon discounted MDP is a tuple pS , A , P , r , q where S and A are finite state and action spaces , P : S ˆA Ñ S is a transition kernel , the reward r : S ˆA Ñ R is a bounded function , and P r0 , 1q is a discount factor . At time t , the agent is in st and draws an action at according to a decision rule dt that maps past information to a probability distribution qdt over the action set . Once at is taken , the agent receives a reward rpst , atq . A decision rule can be history-dependent ( H ) or Markovian ( M ) , and randomized ( R ) or deterministic ( D ) . Denote by Ht the set of possible histories up to time t. Then , a history-dependent decision-rule is given by dt : Ht Ñ A with ht fiÑ qdtphtqp¨q . A Markovian decision-rule , on the other hand , maps states to actions , i.e. , dt : S Ñ A with s fiÑ qdtpsqp¨q . A policy ⇡ : “ pdtqt•0 is a sequence of decision rules whose type dictates that of the policy . It can be either Markovian deterministic ( ⇧MD ) or randomized ( ⇧MR ) , history-dependent deterministic ( ⇧HD ) or randomized ( ⇧HR ) . It is stationary if its decision rules do not depend on time , i. e. , dt “ d for all t • 0 . This defines the smaller class of stationary policies : deterministic ( ⇧SD ) and randomized ( ⇧SR ) . Note that stationary policies are inherently Markovian . Indeed , at time t “ 0 , d : H0 Ñ A is state-dependent because H0 “ S. Since the policy is stationary , i. e. , dt “ d @ t , subsequent decision rules are also state-dependent , thus Markovian . This makes ⇧HR the most general set and ⇧SD the most specific . We denote probability model by P⇡0 , where the subscript 0 stands for the delay value m “ 0 . The related random variables are denoted by s̃t P S , ãt P A and h̃t P pS ˆAqt ˆ S . The value function given policy ⇡ P ⇧HR is defined as v⇡psq “ E⇡0 „ ∞8 t “ 0 t rps̃t , ãtq ˇ̌ ˇ̌ s̃0 “ s ⇢ , where the expectation is taken with respect to ( w.r.t . ) P⇡0 p¨|s̃0 “ sq . Let the optimal value function v ˚psq : “ max ⇡P⇧HR v ⇡psq , @ s P S . ( 1 ) Our goal is to find a policy ⇡˚ that yields v˚ , and it is known that focusing on stationary deterministic policies ⇡ P ⇧SD is sufficient for reaching the optimum in ( 1 ) ( Puterman , 2014 ) [ Thm . 6.2.10. ] . 3 MDPS WITH DELAY : A DEGRADATION EXAMPLE In an MDP with execution delay1 m , any action chosen at time t is executed at t ` m. Therefore , at each step , the agent witnesses the current state and action being executed , but selects a new action that will be applied in a future state . We assume that m decided actions are already awaiting execution at t “ 0 , so at any given time , the queue of pending actions is of constant length m. As we illustrate in the next example , having a delay generally comes at a price . Example 3.1 ( Two-state MDP ) . Consider the MDP in Fig . 1 . It has two states and two actions : S “ ts0 , s1u , A “ ta0 , a1u . The transition kernel is independent of the action : for all s , s 1 P S s.t . s ‰ s1 , P ps1|s , aq “ P ps1|sq “ p where p P r0.5 , 1s . The reward is positive for one of the two actions only : rps0 , a0q “ rps1 , a1q “ 1 , rps0 , a1q “ rps1 , a0q “ 0 . We inspect the return obtained from the commonly used set of stationary deterministic policies ⇧SD . As expected , the highest possible return is attained when m “ 0 , but monotonically decreases with the delay , m , and increases with the level of certainty , p. We analytically quantify this effect in the following and give a proof in Appx . A.1 . Proposition 3.1 . For delay m P N and p P r0.5 , 1s , the optimal return of ⇡˚ P ⇧SD is 1 ` p2p´1qm2p1´ q . Remark 3.1 . This result demonstrates a clear tradeoff between stochasticity and delay . For p Ñ 0.5 or m Ñ 8 , the return goes to its minimal value of 0.5 { p1 ´ q. Contrarily , for p Ñ 1 or m Ñ 0 , it goes to its maximal value of 1 { p1 ´ q .

Summary: This paper investigated the problem of learning agents when there are execution delays. The authors (i) used a two-state MDP example to show some equivalence between execution delay and stochasticity of transitions; (ii) analyzed the action aggregation method, which cumulated all the history and then made decisions. They show a classic Policy Iteration (PI) method with the aggregation, unfortunately, has its iteration complexity exponentially depending on the delay time $m > 0$; (iii) formulated the Execution-Delay (ED)-MDP and showed that there exists a non-stationary Markov policy which attains optimal value, while any stationary policies will have suboptimal performance; (iv) proposed a model-based Q-learning method, delayed-Q, which used the predicted future state-action sequence to make decisions; (v) did experiments on Maze, CartPole, Acrobot and Atari tasks to verify the proposed delayed-Q method.