Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methods

1 Introduction . Deep reinforcement learning ( RL ) has demonstrated phenomenal achievements in a wide array of tasks , including superhuman game-playing [ 19 , 27 ] and controlling complex robots [ 8 , 13 ] . Most RL algorithms are based on an Markov Decision Process ( MDP ) , which is a discrete-time control process for the iteration of observing a state and performing an action . However , numerous real-world problems such as robotic manipulation and autonomous driving are defined in continuous time , which does not directly fit the MDP setting . To fill this gap , continuous time is often discretized by a discretization time scale δ , where the RL agent makes a decision at every δ . It has been shown that RL algorithms are greatly sensitive to this hyperparameter [ 1 , 36 ] . For instance , altering δ via frame skipping leads to drastic performance differences [ 1 , 4 ] . Indeed , an excessively high δ precludes the agent from making fine-grained decisions , which is likely to cause performance degradation . On average , the agent could perform equally well or better with a lower δ than with a higher δ , since the agent can make decisions more frequently . However , Baird [ 2 ] and Tallec et al . [ 36 ] theoretically proved that the standard Q-learning fails when δ → 0 as the action-value ( Q ) function collapses to the state-value ( V ) function , eliminating preferences between actions . As will be shown in Section 4.1 , policy gradient ( PG ) methods fail as well with an infinitesimal δ for the following three reasons : ( 1 ) The variance of the gradient estimator explodes . ( 2 ) Exploration ranges may become highly limited . ( 3 ) Infinitely many decision steps are required . The latter two also apply to Q-learning methods . Therefore , it is generally required to differently set an appropriate δ for each continuous environment . Indeed , continuous control environments in MuJoCo [ 37 ] have different discretization time scales from one another , ranging from 0.008s ( Hopper ) to 0.05s ( InvertedDoublePendulum ) . However , such 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . tuning of δ could be burdensome when applying RL algorithms to new environments , considering its significant influence on performance [ 1 , 36 ] . Furthermore , even if the optimal δ is found in simulation , trained policies may not be transferable to real-world settings , since physical sensors often have their unique sampling frequencies . There have been proposed some methods for robustness to discretization of time scales . δ-invariant methods could bring several advantages : ( 1 ) It obviates the need for tuning δ on each continuous control environment . ( 2 ) They can achieve better performance by utilizing more fine-grained control with a low δ . ( 3 ) Using a policy with adaptive decision frequencies ( as a variant of δ-invariant policies ) , the agent can efficiently take actions only when necessary , which could expedite training without losing agility . Tallec et al . [ 36 ] introduced an algorithm based on Advantage Updating [ 2 ] , which can make existing Q-learning methods ( e.g. , DQN [ 18 ] and DDPG [ 16 ] ) invariant to δ by preventing them from Q-function collapse . For PG methods , Munos [ 21 ] and Wawrzynski [ 38 ] proposed methods that can cope with fine time discretization . However , these methods either assume access to the gradient of the reward function or require an infinite number of decision steps ( or training steps ) when δ → 0 , both of which could hinder its application to real-world environments . We aim at proposing an efficient δ-invariant approach applicable to existing PG methods such as PPO [ 30 ] , TRPO [ 28 ] and A2C [ 20 ] . One straightforward approach may be to take durative actions by making policies produce both actions and their durations . Such an approach is practically equivalent to prior work on action repetition whose policies output both actions and the number of action repetitions [ 15 , 31 ] , since continuous control environments such as MuJoCo often already provide discretized time scales . However , prior approaches to action repetition possess some limitations . For example , there is no way to stop repeating a chosen action during a repetition period , which means that they are not capable of immediately handling unexpected events in stochastic environments . This could lead to catastrophic failure in some real-world settings such as autonomous driving . We thus propose an alternative approach named Safe Action Repetition ( SAR ) with the key idea of repeating an action until the agent exits its safe region . Our policy produces both an action and a safe region in the state space , only within which the chosen action is repeated . SAR enables any PG algorithm to not only be δ-invariant but also be robust to stochasticity such as unexpected events in the environment , because such situations lead the agent ’ s state to be outside of the safe region , immediately causing the cease of the current action . We apply the proposed method to several PG algorithms and empirically show that SAR indeed exhibits δ-invariance on various MuJoCo environments and outperforms baselines on both deterministic and stochastic settings . Our contributions can be summarized as follows : • We first provide a more general proof on the variance explosion of the PG estimator , which is the main reason why PG methods fail as δ → 0 . We then show that temporally extended actions can resolve the failure mode of PG algorithms with a low δ . • We introduce a novel δ-invariant method named SAR applicable to any PG method on continuous control domains . To the best of our knowledge , this is the first action repetition ( or durative action ) method that repeats an action based on the agent ’ s state , rather than a precomputed action duration . As a result , SAR can cope with unexpected situations in stochastic environments , which existing action repetition methods can not handle . • We apply SAR to three PG methods , PPO , TRPO and A2C , and empirically demonstrate that our SAR method is mostly invariant to δ on eight MuJoCo environments . We also verify its robustness to stochasticity via three different stochastic settings on each MuJoCo environment . Our method also outperforms previous δ-invariant approaches such as FiGARC [ 31 ] and DAU [ 36 ] on those settings . 2 Related Work . Continuous-time RL . Reinforcement learning in continuous-time domains has long been studied with various approaches [ 2 , 3 , 5 , 6 , 7 , 21 , 22 ] . Bradtke and Duff [ 3 ] extended existing Q-learning and temporal difference methods to semi-MDPs , which can be viewed as a continuous-time generalization of MDPs . Doya [ 5 ] developed a continuous actor-critic method based on the Hamilton-JacobiBellman ( HJB ) equation , a continuous-time counterpart of the Bellman equation , approximating policies and value functions with radial basis functions . Time discretization . Another line of research to cope with continuous-time environments is to use finely discretized MDPs . Baird [ 2 ] first informally presented that when δ → 0 , two different Q values on the same state will eventually collapse to the V value , such that Q ( s , a1 ) ≈ Q ( s , a2 ) ≈ V ( s ) , since the influence of each action is vanished by the infinitesimal time scale . They proposed Advantage Updating as a solution to avoid the collapse by appropriately scaling the advantage function A ( s , a ) = Q ( s , a ) −V ( s ) δ . More recently , Tallec et al . [ 36 ] theoretically proved the existence of collapse in low-δ settings and extended Advantage Updating for deep neural networks , showing its δ-invariance on classic control benchmarks . For PG methods , Munos [ 21 ] first demonstrated that the variance of the policy gradient estimate can be infinite when δ → 0 , and proposed an algorithm based on pathwise derivatives , in which the variance of the estimator decreases to 0 when δ → 0 . However , it assumes that the gradient of the reward function ∇r ( s , a ) is known to the agent . Korenkevych et al . [ 14 ] , Wawrzynski [ 38 ] proposed methods based on autocorrelated noise that could prevent the variance explosion . Notably , all of these approaches choose an action at every δ . However , this makes it infeasible to train when δ is nearly zero as the number of decision steps goes to infinity . In our work , we employ durative actions to achieve δ-invariance on PG methods , which could resolve the problem of variance explosion and infinite decision steps . Action repetition . Our proposed method is closely related to existing action repetition methods . Most works on action repetition let the policy also determine how many times ( or how long ) an action is repeated . DFDQN [ 15 ] doubled the action space by mapping a half of it to actions repeated r1 times and the other half to those repeated r2 times , where r1 and r2 are fixed hyperparameters . FiGAR [ 31 ] introduced a repetition policy π ( x|s ) in addition to the original action policy π ( a|s ) so that the agent repeats the action x times , where x is selected from a predefined set W ; e.g. , W = { 1 , 2 , . . . , 30 } . Metelli et al . [ 17 ] theoretically analyzed the performance of the optimal policy when a fixed action repetition count is given , and proposed a heuristic to approximately choose the optimal control frequency . On the other hand , we take a completely different approach where our policy produces not repetition counts ( or action durations ) but safe regions , which enables the agent to adaptively stop action repetition when facing unexpected events in stochastic environments . Finally , action repetition is related to the options framework [ 34 ] in that they both use temporally extended actions . In the options framework , the agent learns both a high-level inter-option policy and a low-level intra-option policy , where action repetition can be interpreted as a special case of an intraoption policy . However , one crucial difference between them is that within the options framework , the agent has to produce each δ-discretized low-level action ( even with open-loop options ) , which makes having δ-invariance non-trivial , unlike the action repetition approach . 3 Preliminaries . We consider a continuous-time MDPM = ( S , A , r , F , γ ) [ 3 , 5 ] , where S is a continuous state space , A is a bounded action space , F : S × A → S is a transition dynamics function , r : S × A → R is a reward function and γ ∈ ( 0 , 1 ] is a discount factor . For simplicity , we assume the environment has deterministic transition dynamics ; we refer to Munos and Bourgine [ 22 ] for the stochastic case involving Brownian motion . The transition dynamics and the return R ( τ ) are given by s ( t ) = s ( 0 ) + ∫ t 0 F ( s ( t′ ) , a ( t′ ) ) dt′ , R ( τ ) = ∫ ∞ 0 γt ′ r ( s ( t′ ) , a ( t′ ) ) dt′ , ( 1 ) where s ( t ) and a ( t ) respectively denote the state and action at time t , and τ denotes the whole trajectory consisting of states and actions . Following Tallec et al . [ 36 ] , we define a discretized version ofM asMδ = ( S , A , rδ , Fδ , γδ ) with a discretization time scale δ > 0 , where the agent observes a state and performs an action at every δ . We respectively denote the state and action at i-th step as si and ai , where si inMδ corresponds to s ( iδ ) inM and ai is maintained during the time interval [ iδ , ( i + 1 ) δ ) . In s ( t ) , t indicates the physical time , and i in si is the number of steps taken . The reward ri and return Rδ ( τ ) are defined as ri = rδ ( si , ai ) = r ( s ( iδ ) , a ( iδ ) ) δ , Rδ ( τ ) = ∞∑ i=0 γiδri , ( 2 ) where the discount factor is γδ = γδ . Given a deterministic policy π : S → A , the continuous value function V π ( s ) and the discretized one V πδ are defined as V π ( s ) = Eτ∼pπ ( τ ) [ R ( τ ) |s ( 0 ) = s ] and V πδ ( s ) = Eτ∼pπ ( τ ) [ Rδ ( τ ) |s0 = s ] , respectively . Tallec et al . [ 36 ] proved that V πδ converges to V π when δ → 0 under smoothness assumptions . In the rest of the paper , we will focus onMδ ( i.e. , M with time discretization ) and omit the subscript δ unless it is necessary . Also , as in ordinary discrete-time MDPs [ 35 ] , we consider stochastic transition dynamics p ( si+1|si , ai ) and stochastic policy π ( ai|si ) instead of deterministic ones .

Summary: This paper studies the effect of discretization of the time on the policy space methods in RL. Since the underlying physics of the most of the RL systems is in continuous time, one needs to perform a discretization of the time horizon to employ policy gradient to find the optimal policy. However, as shown in the paper, policy space methods might suffer from high variance when the discretization parameter is small. In addition, a large discretization parameter is not also optimal, because it will result in a suboptimal policy, and it is not robust to stochastic behavior of the system. To deal with this problem, the authors propose SAR (safe action repetition). The idea is to let the algorithm repeat the same set of actions for a window of time, until the system exists some "safe region". To find this safe region, the policy outputs the radius of the safe region, and the algorithm needs to find the optimal policy which maximizes the reward while finding the optimal safe region. The authors compare their algorithm with the previous work experimentally, and they show that their algorithms outperforms FiGAR-C.