Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methods

1 Introduction . Deep reinforcement learning ( RL ) has demonstrated phenomenal achievements in a wide array of tasks , including superhuman game-playing [ 19 , 27 ] and controlling complex robots [ 8 , 13 ] . Most RL algorithms are based on an Markov Decision Process ( MDP ) , which is a discrete-time control process for the iteration of observing a state and performing an action . However , numerous real-world problems such as robotic manipulation and autonomous driving are defined in continuous time , which does not directly fit the MDP setting . To fill this gap , continuous time is often discretized by a discretization time scale δ , where the RL agent makes a decision at every δ . It has been shown that RL algorithms are greatly sensitive to this hyperparameter [ 1 , 36 ] . For instance , altering δ via frame skipping leads to drastic performance differences [ 1 , 4 ] . Indeed , an excessively high δ precludes the agent from making fine-grained decisions , which is likely to cause performance degradation . On average , the agent could perform equally well or better with a lower δ than with a higher δ , since the agent can make decisions more frequently . However , Baird [ 2 ] and Tallec et al . [ 36 ] theoretically proved that the standard Q-learning fails when δ → 0 as the action-value ( Q ) function collapses to the state-value ( V ) function , eliminating preferences between actions . As will be shown in Section 4.1 , policy gradient ( PG ) methods fail as well with an infinitesimal δ for the following three reasons : ( 1 ) The variance of the gradient estimator explodes . ( 2 ) Exploration ranges may become highly limited . ( 3 ) Infinitely many decision steps are required . The latter two also apply to Q-learning methods . Therefore , it is generally required to differently set an appropriate δ for each continuous environment . Indeed , continuous control environments in MuJoCo [ 37 ] have different discretization time scales from one another , ranging from 0.008s ( Hopper ) to 0.05s ( InvertedDoublePendulum ) . However , such 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . tuning of δ could be burdensome when applying RL algorithms to new environments , considering its significant influence on performance [ 1 , 36 ] . Furthermore , even if the optimal δ is found in simulation , trained policies may not be transferable to real-world settings , since physical sensors often have their unique sampling frequencies . There have been proposed some methods for robustness to discretization of time scales . δ-invariant methods could bring several advantages : ( 1 ) It obviates the need for tuning δ on each continuous control environment . ( 2 ) They can achieve better performance by utilizing more fine-grained control with a low δ . ( 3 ) Using a policy with adaptive decision frequencies ( as a variant of δ-invariant policies ) , the agent can efficiently take actions only when necessary , which could expedite training without losing agility . Tallec et al . [ 36 ] introduced an algorithm based on Advantage Updating [ 2 ] , which can make existing Q-learning methods ( e.g. , DQN [ 18 ] and DDPG [ 16 ] ) invariant to δ by preventing them from Q-function collapse . For PG methods , Munos [ 21 ] and Wawrzynski [ 38 ] proposed methods that can cope with fine time discretization . However , these methods either assume access to the gradient of the reward function or require an infinite number of decision steps ( or training steps ) when δ → 0 , both of which could hinder its application to real-world environments . We aim at proposing an efficient δ-invariant approach applicable to existing PG methods such as PPO [ 30 ] , TRPO [ 28 ] and A2C [ 20 ] . One straightforward approach may be to take durative actions by making policies produce both actions and their durations . Such an approach is practically equivalent to prior work on action repetition whose policies output both actions and the number of action repetitions [ 15 , 31 ] , since continuous control environments such as MuJoCo often already provide discretized time scales . However , prior approaches to action repetition possess some limitations . For example , there is no way to stop repeating a chosen action during a repetition period , which means that they are not capable of immediately handling unexpected events in stochastic environments . This could lead to catastrophic failure in some real-world settings such as autonomous driving . We thus propose an alternative approach named Safe Action Repetition ( SAR ) with the key idea of repeating an action until the agent exits its safe region . Our policy produces both an action and a safe region in the state space , only within which the chosen action is repeated . SAR enables any PG algorithm to not only be δ-invariant but also be robust to stochasticity such as unexpected events in the environment , because such situations lead the agent ’ s state to be outside of the safe region , immediately causing the cease of the current action . We apply the proposed method to several PG algorithms and empirically show that SAR indeed exhibits δ-invariance on various MuJoCo environments and outperforms baselines on both deterministic and stochastic settings . Our contributions can be summarized as follows : • We first provide a more general proof on the variance explosion of the PG estimator , which is the main reason why PG methods fail as δ → 0 . We then show that temporally extended actions can resolve the failure mode of PG algorithms with a low δ . • We introduce a novel δ-invariant method named SAR applicable to any PG method on continuous control domains . To the best of our knowledge , this is the first action repetition ( or durative action ) method that repeats an action based on the agent ’ s state , rather than a precomputed action duration . As a result , SAR can cope with unexpected situations in stochastic environments , which existing action repetition methods can not handle . • We apply SAR to three PG methods , PPO , TRPO and A2C , and empirically demonstrate that our SAR method is mostly invariant to δ on eight MuJoCo environments . We also verify its robustness to stochasticity via three different stochastic settings on each MuJoCo environment . Our method also outperforms previous δ-invariant approaches such as FiGARC [ 31 ] and DAU [ 36 ] on those settings . 2 Related Work . Continuous-time RL . Reinforcement learning in continuous-time domains has long been studied with various approaches [ 2 , 3 , 5 , 6 , 7 , 21 , 22 ] . Bradtke and Duff [ 3 ] extended existing Q-learning and temporal difference methods to semi-MDPs , which can be viewed as a continuous-time generalization of MDPs . Doya [ 5 ] developed a continuous actor-critic method based on the Hamilton-JacobiBellman ( HJB ) equation , a continuous-time counterpart of the Bellman equation , approximating policies and value functions with radial basis functions . Time discretization . Another line of research to cope with continuous-time environments is to use finely discretized MDPs . Baird [ 2 ] first informally presented that when δ → 0 , two different Q values on the same state will eventually collapse to the V value , such that Q ( s , a1 ) ≈ Q ( s , a2 ) ≈ V ( s ) , since the influence of each action is vanished by the infinitesimal time scale . They proposed Advantage Updating as a solution to avoid the collapse by appropriately scaling the advantage function A ( s , a ) = Q ( s , a ) −V ( s ) δ . More recently , Tallec et al . [ 36 ] theoretically proved the existence of collapse in low-δ settings and extended Advantage Updating for deep neural networks , showing its δ-invariance on classic control benchmarks . For PG methods , Munos [ 21 ] first demonstrated that the variance of the policy gradient estimate can be infinite when δ → 0 , and proposed an algorithm based on pathwise derivatives , in which the variance of the estimator decreases to 0 when δ → 0 . However , it assumes that the gradient of the reward function ∇r ( s , a ) is known to the agent . Korenkevych et al . [ 14 ] , Wawrzynski [ 38 ] proposed methods based on autocorrelated noise that could prevent the variance explosion . Notably , all of these approaches choose an action at every δ . However , this makes it infeasible to train when δ is nearly zero as the number of decision steps goes to infinity . In our work , we employ durative actions to achieve δ-invariance on PG methods , which could resolve the problem of variance explosion and infinite decision steps . Action repetition . Our proposed method is closely related to existing action repetition methods . Most works on action repetition let the policy also determine how many times ( or how long ) an action is repeated . DFDQN [ 15 ] doubled the action space by mapping a half of it to actions repeated r1 times and the other half to those repeated r2 times , where r1 and r2 are fixed hyperparameters . FiGAR [ 31 ] introduced a repetition policy π ( x|s ) in addition to the original action policy π ( a|s ) so that the agent repeats the action x times , where x is selected from a predefined set W ; e.g. , W = { 1 , 2 , . . . , 30 } . Metelli et al . [ 17 ] theoretically analyzed the performance of the optimal policy when a fixed action repetition count is given , and proposed a heuristic to approximately choose the optimal control frequency . On the other hand , we take a completely different approach where our policy produces not repetition counts ( or action durations ) but safe regions , which enables the agent to adaptively stop action repetition when facing unexpected events in stochastic environments . Finally , action repetition is related to the options framework [ 34 ] in that they both use temporally extended actions . In the options framework , the agent learns both a high-level inter-option policy and a low-level intra-option policy , where action repetition can be interpreted as a special case of an intraoption policy . However , one crucial difference between them is that within the options framework , the agent has to produce each δ-discretized low-level action ( even with open-loop options ) , which makes having δ-invariance non-trivial , unlike the action repetition approach . 3 Preliminaries . We consider a continuous-time MDPM = ( S , A , r , F , γ ) [ 3 , 5 ] , where S is a continuous state space , A is a bounded action space , F : S × A → S is a transition dynamics function , r : S × A → R is a reward function and γ ∈ ( 0 , 1 ] is a discount factor . For simplicity , we assume the environment has deterministic transition dynamics ; we refer to Munos and Bourgine [ 22 ] for the stochastic case involving Brownian motion . The transition dynamics and the return R ( τ ) are given by s ( t ) = s ( 0 ) + ∫ t 0 F ( s ( t′ ) , a ( t′ ) ) dt′ , R ( τ ) = ∫ ∞ 0 γt ′ r ( s ( t′ ) , a ( t′ ) ) dt′ , ( 1 ) where s ( t ) and a ( t ) respectively denote the state and action at time t , and τ denotes the whole trajectory consisting of states and actions . Following Tallec et al . [ 36 ] , we define a discretized version ofM asMδ = ( S , A , rδ , Fδ , γδ ) with a discretization time scale δ > 0 , where the agent observes a state and performs an action at every δ . We respectively denote the state and action at i-th step as si and ai , where si inMδ corresponds to s ( iδ ) inM and ai is maintained during the time interval [ iδ , ( i + 1 ) δ ) . In s ( t ) , t indicates the physical time , and i in si is the number of steps taken . The reward ri and return Rδ ( τ ) are defined as ri = rδ ( si , ai ) = r ( s ( iδ ) , a ( iδ ) ) δ , Rδ ( τ ) = ∞∑ i=0 γiδri , ( 2 ) where the discount factor is γδ = γδ . Given a deterministic policy π : S → A , the continuous value function V π ( s ) and the discretized one V πδ are defined as V π ( s ) = Eτ∼pπ ( τ ) [ R ( τ ) |s ( 0 ) = s ] and V πδ ( s ) = Eτ∼pπ ( τ ) [ Rδ ( τ ) |s0 = s ] , respectively . Tallec et al . [ 36 ] proved that V πδ converges to V π when δ → 0 under smoothness assumptions . In the rest of the paper , we will focus onMδ ( i.e. , M with time discretization ) and omit the subscript δ unless it is necessary . Also , as in ordinary discrete-time MDPs [ 35 ] , we consider stochastic transition dynamics p ( si+1|si , ai ) and stochastic policy π ( ai|si ) instead of deterministic ones .

Summary: This work is motivated by the performance dropoff of RL algorithms when the time scale $\delta$ decreases to 0. Although previous work has studied such an issue in Q-learning,  this paper rather focuses on policy-gradient (PG). Observing that the variance of PG explodes as $\delta\rightarrow 0$, the authors introduce Safe Action Repetition (SAR), which ensures bounded gradient regardless of the time scale $\delta$. Inspired by action repetition, the proposed technique is additionally able to handle unexpected events by interrupting repetition when the agent reaches an unsafe/uncertain region in the state-space.