Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methods
Authors: Seohong Park
Jaekyeom Kim
Gunhee Kim
Venue: NeurIPS 2021 Poster (2021)
Keywords: Reinforcement learning

Abstract: In reinforcement learning, continuous time is often discretized by a time scale $\delta$, to which the resulting performance is known to be highly sensitive. In this work, we seek to find a $\delta$-invariant algorithm for policy gradient (PG) methods, which performs well regardless of the value of $\delta$. We first identify the underlying reasons that cause PG methods to fail as $\delta \to 0$, proving that the variance of the PG estimator can diverge to infinity in stochastic environments under a certain assumption of stochasticity. While durative actions or action repetition can be employed to have $\delta$-invariance, previous action repetition methods cannot immediately react to unexpected situations in stochastic environments. We thus propose a novel $\delta$-invariant method named Safe Action Repetition (SAR) applicable to any existing PG algorithm. SAR can handle the stochasticity of environments by adaptively reacting to changes in states during action repetition. We empirically show that our method is not only $\delta$-invariant but also robust to stochasticity, outperforming previous $\delta$-invariant approaches on eight MuJoCo environments with both deterministic and stochastic settings. Our code is available at https://vision.snu.ac.kr/projects/sar.

Time Discretizatin-IIvariint Safe Action Repetition for Policy Gradient Methods
Seohong Park Seoul National University artberryx@snu. ac
Jaekyeom Kim Seoul National University j jaekyeom@snu. ac.kr
Gunhee Kim Seoul National University gunhee@snu. ac
Abstract
In reinforcement learning, continuous time is often discretized by time scale 6, to which the resulting performance is known to be highly sensitive. In this work, we seek to find a d-invariant algorithm for policy gradient (PG) methods, which performs well regardless of the value of 8. We first identify the underly- ing reasons that cause PG methods to fail as + O, proving that the variance of the PG estimator can diverge to infinity in stochastic environments under a certain assumption of stochasticity. While durative actions or action repetition can be employed to have 5-invariance, previous action repetition methods can- not immediately react unexpected situations in stochastic environments. We thus propose a novel 5-invariant method named Safe Action Repetition (SAR) applicable to any existing PG algorithm. SAR can handle the stochasticity environments by adaptively reacting to changes in states during action repetition. We empirically show that our method is not only d-invariant but also robust to stochasticity, outperforming previous d-invariant approaches on eight MuJoCo environments with both deterministic and stochastic settings. Our code available at hittts://vvsioio snu wttprrooects//arr
Introduction
Deep reinforcement learning (RL) has demonstrated phenomenal achievements in wide array of tasks, including superhuman game-playing [99 (2) and controlling complex robots g 13. Most RL algorithms based on an Markov Decision Process (MDP), which is discrete-time control process for the iteration of observing a state and performing an action. However, numerous real-worle problems such robotic manipulation and autonomous driving are defined in continuous time, which does not directly fit the MDP setting. To fill this gap, continuous time is often discretized by discretization time scale 5, where the RL agent makes decision every 6. It has been shown that RL algorithms are greatly sensitive to this hyperparameter 36). For instance, altering via frame skipping leads to drastic performance differences 4] Indeed, an excessively high ō precludes the agent from making fine-grained decisions, which is likely to cause performance degradation.
On average, the agent could perform equally well better with lower 5 than with higher 8, since the agent can make decisions more frequently. However, Baird [2 and Tallec et al. [36] theoretically proved that the standard Q-learing fails when 5 - 0 as the action-value (Q) function collapses the state-valde (V) function, eliminating preferences between actions. As will be shown in Section 4.1 policy gradient (PG) methods fail well with an infinitesimal 5 for the following three reasons: (1) The variance of the gradient estimator explodes. (2) Exploration ranges may become highly limited. (3) Infinitely many decision steps required. The latter two also apply Q-learning methods.
Therefore, it is generally required to differently set an appropriate å for each continuous environment. Indeed, continuous control environments in MuJoCo [37] have different discretization time scales from one another, ranging from 0.008s (Hopper) to 0.05s (InvertedDoubleeeenduui) However, such
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

tuning of ó could be burdensome when applying RL algorithms to new environments, considering its significant influence on performance [ [6]. Furthermore, even if the optimal ò is found in simulation, trained policies may not be transferable to real-world settings, since physical sensors often have their unique sampling frequencies.
There have been proposed some methods for robustness to discretization of time scales, d-invariant methods could bring several advantages: (1) It obviates the need for tuning ô on each continuous control environment.
(2) They can achieve better performance by utilizing more fine-grained control with a low 5. (3) Using policy with adaptive decision frequencies (as variant of 6-invariant policies), the agent can efficiently take actions only when necessary, which could expedite training without losing agility. Tallec et al. [66] introduced an algorithm based on Advantage Updating [2]. which can make existing Q-learning methods (e.g., DQN (18] and DDPG ([66) invariant to by preventing them from Q-function collapse. For PG methods, Munos 21 and Wawrzynski [38 proposed methods that can cope with fine time discretization. However, these methods either assume access to the gradient of the reward function or require an infinite number of decision steps (or training steps) when 8 O, both of which could hinder its application to real-world environments.
We aim at proposing an efficient 6-invariant approach applicable existing PG methods such PPO [30], TRPO [28) and A2C [20]. One straightforward approach may be to take durative actions by making policies produce both actions and their durations. Such an approach is practically equivalent to prior work on action repetition whose policies output both actions and the number of action repetitions [TS BT, since continuous control environments such as MuJoCo often already provide discretized time scales. However, prior approaches to action repetition possess some limitations. For example, there is no way to stop repeating a chosen action during a repetition period, which means that they are not capable of immediately handling unexpected events in stochastic environments. This could lead to catastrophic failure in some real-world settings such as autonomous driving.
We thus propose an alternative approach named Safe Action Repetition (SAR) with the key idea of repeating an action until the agent exits its safe region. Our policy produces both an action and safe region in the state space, only within which the chosen action is repeated. SAR enables any PG algorithm not only be d-invariant but also be robust to stochasticity such as unexpected events in the environment, because such situations lead the agent's state to be outside of the safe region, immediately causing the cease of the current action. We apply the proposed method to several PG algorithms and empirically show that SAR indeed exhibits 5-invariance on various MuloCo environments and outperforms baselines on both deterministic and stochastic settings.
Our contributions can be summarized as follows:
We first provide a more general proof on the variance explosion of the PG estimator, which is the main reason why PG methods fail as 8 + O. We then show that temporally extended actions can resolve the failure mode of PG algorithms with a low
We introduce a novel 8-invariant method named SAR applicable to any PG method on continuous control domains. To the best of our knowledge, this is the first action repetition (or durative action) method that repeats an action based on the agent's state, rather than a precomputed action duration. As result, SAR can cope with unexpected situations in stochastic environments, which existing action repetition methods cannot handle.
We apply SAR to three PG methods, PPO, TRPO and A2C, and empirically demonstrate that our SAR method is mostly invariant to ó on eight MuloCo environments. We also verify its robustness to stochasticity via three different stochastic settings on each MuJoCo environment. Our method also outperforms previous 8-invariant approaches such FiGAR- C 31 and DAU (36 on those settings.
2 Related Work
Continuous-time RL. Reinforcement learning in continuous-tim time domains has long been studied with various approaches 2 3 6 22) Bradtke and Duff [3] extended existing Q-learning and temporal difference methods to semi-MDPs. which can be viewed as a continuous-time generalization of MDPs. Doya 5 developed a continuous actor-critic method based on the Hamilton-Jacobi- Bellman (HJB) equation, a continuous-time counterpart of the Bellman equation, approximating policies and value functions with radial basis functions.

---

Time discretization. Another line of research to cope with continuous time environments is to use finely discretized MDPs. Baird [2] first informally presented that when + 0, two different Q values on the same state will eventually collapse to the V value, such that Q(s, a1) a Q(s, a2) R V(s), since the influence of each action is vanished by the infinitesimal time scale. They proposed Advantage Updating as solution to avoid the collapse by appropriately scaling the advantage function A(s, a) ((s.))-V(o). More recently, Tallec et al. (36) theoretically proved the existence of collapse in low-ō settings and extended Advantage Updating for deep neural networks, showing its ò-invariance on classic control benchmarks. For PG methods, Munos [21] first demonstrated that the variance of the policy gradient estimate can be infinite when o + 0, and proposed an algorithm based on pathwise derivatives, in which the variance of the estimator decreases to when - However, it assumes that the gradient of the reward function Vr(s, a) is known to the agent. Korenkevych al. 14. Wawrzynski (38] proposed methods based on autocorrelated noise that could prevent the variance explosion. Notably, all of these approaches choose an action at every 5. However, this makes infeasible to train when ö is nearly zero as the number of decision steps goes to infinity. In our work, we employ durative actions to achieve --invariance on PG methods, which could resolve the problem of variance explosion and infinite decision steps.
Action repetition. Our proposed method is closely related to existing action repetition methods. Most works on action repetition let the policy also determine how many times (or how long) an action is repeated. DFDQN [[5] doubled the action space by mapping a half of it to actions repeated r1 times and the other half to those repeated r2 times, where rı and r2 are fixed hyperparameters. FiGAR (31 introduced a repetition policy <x)ss) in addition to the original action policy T(ass) so that the agent repeats the action times, where x is selected from a predefined set W: egg.. W = 21.2..... 30). Metelli et al. [17] theoretically analyzed the performance of the optimal policy when a fixed action repetition count is given, and proposed a heuristic to approximately choose the optimal control frequency. On the other hand, we take completely different approach where our policy produces not repetition counts (or action durations) but safe regions, which enables the agent to adaptively stop action repetition when facing unexpected events in stochastic environments.
Finally, action repetition is related to the options framework 34 in that they both use temporally extended actions. In the options framework, the agent learns both high-level inter-option policy and a low-level intra-option policy, where action repetition can be interpreted as a special case of an intra- option policy. However, one crucial difference between them is that within the options framework, the agent has to produce each 6-discretized low-level action (even with open-loop options), which makes having 8-invariance non-trivial, unlike the action repetition approach.
3 Preliminaries
We consider continuous-tim time MDP M (S, B 5). where S is continuous state space, A is bounded action space. F: S x A + S is transition dynamics function, r: S x A + IR is a reward function and 7 E (0, 1] is a discount factor. For simplicity, we assume the environment has deterministic transition dynamics; we refer to Munos and Bourgine [22) for the stochastic case involving Brownian motion. The transition dynamics and the return R(T) are given by
s(1) = =0)) + J." ((s(l))a(())d
fo "'r(s()),a(())))
R(F) R())=
where s(t) and a() respectively denote the state and action at time t, and t denotes the whole trajectory consisting of states and actions.
Following Tallec et al. [36], we define discretized version of M as MM (S,Ar rs, Fa, 78) with a discretization time scale 5 >> O. where the agent observes a state and performs an action at every 8. We respectively denote the state and action at i-th step as 8; and a,, where 8; in Ms corresponds to s(id) in M and a; is maintained during the time interval [i8, (i + 1)6). In s(t), L indicates the physical time, and i 8; is the number of steps taken. The reward r; and return (T) are defined as
20
r; = ra(s;; a:) = r(s(it), a(18))8
RB(T) = Ee Y
(2)
im
where the discount factor is 74 = S. Given a deterministic policy n: - A, the continuous value function (s))) and the discretized one V2 are defined as V (s) E, Er) (R(T)s(( = s] and
3

---

(s) = Er-p, (r) (r) (s s), respectively. Tallec et al. [36 proved that V converges to V when 6 + under smoothness assumptions.
In the rest of the paper, we will focus on Ms (i.e., M with time discretization) and omit the subseript 5 unless it is necessary. Also, as in ordinary discrete-time MDPs [5), consider stochastic transition dynamics p($(41|s; a.) and stochastic policy π (a:]s;) instead of deterministic ones.
Safe Action Repetition
We first show that 5 - leads to failure in policy gradient (PG) methods (Section 4.1 and propose that durative actions or action repetition can be solution to the failure mode. We then point out that existing action repetition methods have drawbacks the presence of unexpected events in stochastic environments (Section 4.2). As solution, we propose Safe Action Repetition (SAR) novel ō-invariant approach for PG algorithms,, which is robust to such stochasticity (Section 4.3 We
additionally suggest a variant of SAR to deal with non-Markovian environments (Section
4.1 Policy Gradients with Infinitesimal Discretization of Time Scale
smaller discretization time scale should not lead worse maximum returns on average, since allows the agent to perform more fine-ge tne-ggained actions. However, both Q-learning and PG methods are subjected to fail with too small We introduce three reasons why PG methods fail. We refer to Baird (2]. Tallec et al. 33 for further discussion on Q-learning methods.
Variance explosion of the policy gradient estimator. We show that the variance of the PG estimator can diverge to infinity with a decrease in While this was first shown in Munos (1] via a simple illustrative example, we provide more general proof without assuming particular environment. Specifically, we prove that the variance explosion problem can arise in any stochastic environment where the variance of its return conditioned on actions is greater than a small positive constant.
Let us consider a stochastic environment that has a finite physical time limit T and a discretization time scale 8. It follows that the number of decision steps (or actions) in single rollout is N T/8. Let the policy To (a)(ss) be parameterized by 6, the distribution over trajectories т (Su, ao...... SN) be given by pT(T) = p(sp) I To (a.|s; )(( 1s: a.) and (5 denote its state-marginal distribution. For simplicity, we assume that the policy is represented as multivariate normal distribution with learnable diagonal covariance matrix: Ta(a]ss)) ~ N (ue, (s:), E), where the mean
, (s:) [mo,.. (8z), ..... me,, K (s:)) T is modeled by neural network, Σ diag (o .......-. o x is the learnable variance that is independent of states (as in the original TRPO (28) and PPO (30))
Thus, 0 [0_ 01.... ..... Øk) T is the whole parameters of the policy no, and K dim(
The derivative of the RL objective function ((0) = Er>pp(r> [[][[]]]]]]]]]]]]))]]])][] can be written
[
(
V = BT-pp(t)
V log (agls))))))) R((T) A (
(3)
which is often referred as the policy gradient estimator.
We derive a lower bound for its total variation tr (Go())] where V[X] is the variance of a variable X (or the covariance matrix when X is multidimensional), and Lr is the trace operator.
Theorem If the environment is stochastic in the sense that for any reparameterized actions Co.N -1- if the variance of returns conditioned on the actions is lower bounded by a small positive constant e
(i.e., V N (1) [R(T)) that
where a, no (8:) Ee; and e;
iid (0 I)), it holds
Te
tr V (r))((+) [Go(T)]]
2
(4)
8. min(o],
We provide proof and further discussion Appendix A Equation (4) indicates that 8 - can lead the variance of the PG estimator to explode, especially in stochastic environments. We emphasize that the two primary causes of this explosion are the independence of actions and the infinitely growing number of decision steps, both of which correspond to Equation 22
(23 in Appendix A

---

Practically, existing PG algorithms are implemented with standard variance reduction techniques such as reward-to-go policy gradient and baseline functions. However, even if such techniques are applied, the variance of the PG estimator is still likely to explode as 6 - 0 considering the environment's stochasticity, if the learning rate and minibatch size remain the same.
Challenging exploration. Furthermore, existing PG methods are prone to perform worse with a low 6 due to the difficulty of exploration. The example in Figure I compares 2-D random walks of the same physical time limit with different 8's. is easier to get out of the blue box by chance with a high 8 than a low 8. Intuitively. when - 0, the range that the agent can move each step becomes smaller, which makes it challenging to reach distant states by pure exploration. This originates again from independently sampled actions. We provide a more formal explanation in Appendix B This exploration problem has also been addressed in Q-learning literature [66 36), which employs as a remedy autocorrelated noise such as an Ornstein- Uhlenbeck process.
(a) Low 6 (b) High 8
Figure I: An example of 2-D random walks hav- ing the same physical time limit with different 8's.
Infinite decision steps. For given time limit, the number of decision steps is inversely proportional to ó. This makes the size of the training data required for consuming the same number of episodes increase infinitely as - 0, and thereby impedes training in terms computational cost.
4.2 Durative Actions in Previous Works
For the aforementioned problems with PG methods given a low 8, durative actions can be a solution; that is, the policy decides actions only when it is necessary, rather than at every 8. With durative actions, (1) naturally makes (6-discretized) actions correlated with one another, and (2) it does not always require infinitely many decision steps even if 8 + O.
For durative actions, one may modify existing policies to produce both action and its duration In discretized continuous- time environments, this approach is practically equivalent to prior methods that produce repetition counts 15] 31 as continuous durations should be converted into the repetitions of time-discretieed actions. As one of such action repetition methods, FiGAR-A3C. [31 defines the policy as π (a, æ=|s) where a E A and
W. W is a predefined set of action repetition counts; egg.. W {1, 2, ................................ 300. At every decision step, FiGAR-A3C samples action a; and repetition count 2; from the policy, and then performs the action z; times, where denotes the i-th decision step. defines the n-step return (s;)
as
(x)
Σ Mite V ratk + Wh (8i-n); k=0
(5)
where the cumulative repetition counts (y:) is defined as yo and 9i+1 = y; + x; for i 2 0. The reward r; at the i-th decision step is given by the discounted sum of environment rewards over the holding time. While FiGAR-A3C) is based on the standard procedure of A3C (00). it is also applicable to other RL algorithms such as TRPO 28 and DDPG [6].
FiGAR can be naturally extended to its continuous variant, which we call FiGAR-C, by replacing <|a, with (a, t|s) where t E [0, fmax] stands for the duration of the action In d-discretized environments, it translates the action duration / into { repetition times. FiGAR-C is inherently ò-invariant since operates on the unit of physical time instead of the discretized time scale.
However, these approaches to action duration have two limitations. First, since it does not consider stopping an action during repetition, it cannot immediately react to unexpected events while repeating an action. This may lead to poor performance in stochastic environments. Second, in contrast to the fact that the optimal policy [[ass) of a fully observable MDP only depends on states s, not time (35], previous methods only consider the locality of the time variable f, without caring about underlying changes 8. This discrepancy can lead to performance degradation, since even during a small time period, can greatly vary in environments with stochastic or non-continuous dynamics. In the next section, we will demonstrate that such limitations can still bring back the variance explosion problem in these action repetition approaches.
5

---

4.3 Safe Action Repetition
We propose an alternative approach named Safe Action Repetition (SAR) that resolves the limitations of existing action repetition methods. SAR repeats actions based on state locality, taking the same action only when states are close.
Following 32 we define a perturbation set for a state E S as (s, = (>')A(s, S d). which corresponds to the closed ball of radius d centered at using the metric A in the state space. Employing the notion of perturbation sets, we propose the following action repetition scheme. At every decision step, SAR's policy T T(a dils:) produces both action a, and the radius d, of perturbation set, which we call the safe region. Then, SAR repeats the action ag only within the safe region B& (si, d:), or equivalently
A(s, st) S de.
(6)
where s denotes the current state during repetition. Once the agent goes outside of the safe region, SAR stops action repetition and selects new action.
Having action durations thresholded by Equation (6) grants two advantages. First, it naturally ensures 5-invariance since action duration is determined by the safe region radius, which is not related to how fine the discretization time scale is. Equation (6) is even completely agnostic to the physical time variable f. Second, the agent becomes robust stochasticity, e-g., encountering an unpredicted event, because such a situation would push the agent's 's state far away from the safe region, which immediately stops action repetition.
To intuitively differentiate SAR with previ- ous action repetition methods, we illustrate a simple example of a stochastic environment, where previous FiGAR-C fails to maintain an optimal policy due to the variance explo- sion of the PG estimator, while our method does not. Let us consider the following 5- discretized environment named AleriThenOfff whose physical time limit T is 1. The state
1 (alerted) off =
(alerted)
0
1 (alerted) off Figure 2: Illustration of AlerTThenOff environment.
E {0 (normal), 1 (alerted)} } with 50 o (normal). The 2-D action is vironment, 8 randomly changes once from 0 (normal) to (alerted) at time / E [0, 1]. When 1 (alerted), we should take an action of off 1 within [1, + x] that s can be back to 0 (normal), where æ > ō >> 0 is an environment parameter known to us. Otherwise, the environment produces a penalty reward of -v (a large negative number) and ends immediately. Conversely, if we set off = 1 when s = 0 (normal). it also causes a penalty of -v and ends immediately. We illustrate this in Figure 2 The reward (before discretization) given by r(t) (num), where f is an unknown reward function, and its discretized reward is given accordingly to Equation (2). When time reaches 1. noisy reward of ~ NN00 1) occurs. To sum up. we should perform off 1 as soon as possible only when we get to know that s becomes (alerted), while we should also perform appropriate num actions that maximize fnum)
[off, num]T where off 10,1) and num R. In this en-
To find an optimal policy in this environment, we consider a deterministic policy for off such that offf s, which we already know is optimal, and stochastic policy for num that num (num]s [s) WH,,, 1), where ja is the policy's 's parameter. Our goal is thus to find the optimal value of μ. If we assume that the penalty is infinitely large and f for simplicity, we obtain the following result.
Proposition 2 In AlertThenOff environment, for the optimal policy Ta, for FIGAR-C and the optimal policy na, for SAR, the following holds:
tr V po (r) (7)] + 00, tr [VTnno,,t) = 2,
when 8 + 0 x + 0, v + oa and f =
We provide a proof Appendix C From Proposition 2 we can conclude that in contrast to FiGAR-C, our SAR policy does not suffer from variance explosion in AleriThenOff environment. As the previous approach fails to maintain an optimal policy even in this very simple stochastic environment, it can also be at risk of failure in general stochastic environments. The intuition behind this failure is

---

that it has no choice but to infinitely shorten action durations in order to optimally handle stochasticity that requires immediate reactions from the agent, which leads both the number of decision steps and the variance of the policy gradient to explode. On the other hand, SAR can be free from variance explosion despite such stochasticity, if SAR sets appropriate safe regions so that such an exigent state locates outside of the safe regions, and thereby the number of decision steps can be bounded.
4.4 SAR on Non-Markovian Environments
We derived SAR based on the fact that the optimal policy in fully observable MDP only depends on states. However, if the Markovian property does not hold (e.g., environments with partially observable MDPs [T]] or time limits (23)), the optimal policy might not be fully determined by states alone. In this case, we can additionally incorporate temporal thresholds into SAR by extending the definition of the safe region as follows:
A s,) + A)jt t.l 1 d.,
where I denotes the time at the i-th step, è denotes the current time during repetition, and < X < is the coefficient that controls the trade-off between distance and time differences. Note that this variant of SAR, which we call A-SAR, is 6-invariant too since each term is independent from 6.
Experiments
We apply SAR to three policy gradient (PG) methods, PPO [30]. TRPO 28 and A2C [20]. and compare with baseline methods in multiple settings. We first demonstrate the d-invariance of our method on deterministic continuous control environments, compared to previous 5-invariant algorithms (Section 5.1p. We then evaluate SAR on stochastic environments to show its robustness to stochasticity (Section 5.2 We also provide illustrative examples for better understanding of our
method (Section 5.3) We describe the full experimental details in Appendix [
Experimental setup. We test SAR on eight continuous control environments from MuJoCo 137 InvertedPendullum-v2 InvertedDooubleeendiine
Hopper-v2, Walker2d-v2, HalfCheetah-v2, Ant-v2, Reacher-v2 and Swimmer-v2. We mainly compare our method to FiGAR-C described in Section 4.2 because it is the only prior method that is ö-invariant and does not always require infinite decision steps even if 5 - 0, but we also make additional comparisons with other baselines such as DAU 36 ARP 14, modified PPO as well in Section 5.1 and Appendices F.2 and
For SAR's distance function in Equation 6 we use (s,, = | - ss /dlmmd) where || 11 is the f, norm and is the state normalized by its moving average. This distance function corresponds to the average difference in each normalized state dimension, where the normalization permits sharing the hyperparameter dmax for all MuJoCo tasks. We also share mmx in FiGAR-C for all environments. Finally, we impose an upper limit of mmx on the maximum duration of actions in SAR for two reasons: (1) to further stabilize training and (2) ensure a fair comparison with FiGAR-C by setting the same limit on time duration. We provide an ablation study including an analysis of imposing an upper limit on t in Appendix !
5.1 Results on Deterministic Environments
We first train SAR and FiGAR-C with PPO, TRPO and A2C on the MuJoCo enviromments, which have deterministic transition dynamics (although they have randomized initial states), with various discretization time scales ranging from 5e 4 to 5e (details in Appendix [ We use suffixes such as 'PPPO' to denote the base PG algorithms. Figure shows the training curves of PPO and TRPO with SAR and with FiGAR-C on the eight MuJoCo environments, where the æ and y axes denote the number of decision steps and the total reward, respectively. We provide results on A2C and further comparison between SAR-PPO and PPO in Appendices Dand F Figure shooo that while vanilla PG algorithms fail to maintain their performance in lower-o settings, SAR is mostly robust to varying 5, often even achieving the best performance with the lowest 8. In a few environments such as Swimmer-v2, SAR's performance becomes slightly worse 5 decreases, which we hypothesize is because a higher aids the agent in getting out of local optima. Also, SAR exhibits similar or better performance compared FiGAR-C most deterministic environments.
For more comprehensive evaluation, we additionally compare with DAU [36], which another approach to d-invariance for Q-learning methods such as DQN [18] and DDPG [16). Note that
7

---

FRADONIEO
Chestah
Halt
Invoneed
Dooble
Peudahmn
Hagppe
Swinneee
Resehis
Walks 24
Inverted
Pendokam
118 Lowest
00
00 2ope
10 128
X2 MiBie
Millee Middle 6
95 11B Original 8
Low
Middl:
Original
Lowest 8
Low
Figure 3: Training curves deterministic MuloCo environments with various 8's (ranging from 5e 4 to 5e - 2). Shaded areas represent the 95% confidence intervals over eight runs. We compare SAR to FiGAR-C with base PG algorithms, PPO and TRPO. SAR mostly has d-invariance, showing similar or even better performance with lower 8's.
WAR PIP
DAU
Inveredllcche
Happpe
Walcorld
HallChath
Art
Switeee
Monter
Pestillers
Figure Bar plots comparing the final performance of SAR-PPO to DAU's on deterministic MuJoCo environments with various 8's. Error bars represent the 95% confidence intervals over eight runs.
SAR-PPO and DAU have different underlying algorithms and training schemes. While DAU is based on DDPG and chooses an action every environment step. SAR-PPO operates on PPO and makes action decisions only when needed. For the comparison, we use the official implementation of DAU 33.. Figure 4 compares the final performances of both methods with various 8's at the same physical time (equal to le6 environment steps in the original 8) on each environment. Overall SAR-PPO outperforms DAU and exhibits better 5-invariance in most of the environments. Also, our method requires about 17.6x fewer decision steps average in the lowest-ō settings via temporally extended actions.
5.2 Results on Stochastic Environments
To demonstrate SAR's robustness with stochastic dynamics, we modify existing MuloCo environ- ments by adding various types of stochasticity. (1) "External Force": we apply an external force with standard deviation of dext to the agent's body with probability of Pext at each decision step. (2) "Strong External Force (Perceptible)":: we make external forces perceptible by the agent, which allows it to react to stronger forces with dex2 Text- (3) "Action Noise": we apply noise with standard deviation of aue the action with probability of Pact at each decision step. Throughout this experiment,
the lowest-d settings. Figure compares SAR- APO's performance with

---

FIGAR PRY
Inverted
Choziih
Half
230
Inverted
Double
Penditmm
Aat
Hopper
Switmmer
Walkerdll
Realter
Miltie Ships
Sroe
Skipe
Strong Extemal Forr (Paceptill)
Stong Fatemal Force
Exsennal Force
Action Noise
(Perespcibte)
Extemaal Force
Acton Noise
Figure 5: Training curves of SAR-PPO and FIGAR-C-PPO MuJoCo environments with various types of stochasticity. Shaded areas represent the 95% confidence intervals over eight runs. SAR exhibits strong performance in the presence of stochasticity.
InvertedPendulum-v2 v2
(b) InvertedPemdulumm-y with external forces
Figure 6: Illustration of SAR on InvertedPendulum-yy with d 2e 3. Our policy produces safe regions (rhombus) within which the agent repeats actions. Circle markers represent the break points of action repetitions and red fists represent the external forces applied to the agent.
FIGAR---PPO's and shows that SAR outperforms FiGAR-C on most of the stochastic environments, often exhibiting drastic differences. We provide further details and the comparison TRPO in Appendices D and
5.3 Qualitative Analysis of SAR
order to provide further insights on SAR, we illustrate how SAR works on InvertedPendulum--22. where the goal is to maintain the balance of pendulum. The state space consists of four dimensions:
two for the position of the agent and the other two for its velocity; i.e., s = [x1, Ø2, U1, U2]. We demonstrate the behavior of SAR a 2-D plane. For better interpretability, we slightly modify method in this experiment: we use only the two (normalized) velocity dimensions for the distance function; that is, the agent stops action repetitions only by its velocity.
Figure 6a illlusstttess how a trained SAR model performs action repetition based on safe regions, where the x and axes correspond respectively to and U2, rhombuses represent safe regions, markers (either circle or cross) represent time-discretized states. can be observed that our policy is learned to produce large safe region when the agent's speed is low and, conversely, small safe region when the speed is high, which fits with the intuition because the risk of losing the balance of the pendulum rises the agent's speed increases.

---

Additionally, we demonstrate how SAR operates in the presence of stochastic external forces (de- scribed in Section 5 Figure [bb It shows how SAR can quickly and adaptively handle such stochasticity with safe regions by immediately stopping repetitions.
How SAR learns the adaptive sizes of safe regions? When the agent's velocity is low, bigger safe region becomes low-hanging fruit especially in the early stages of the training where the PG estimator mostly dominated by immediate rewards (i.e., the accumulated reward within a single action repetition), and thus SAR gravitates toward producing larger safe regions. On the other hand, when the velocity high and if safe regions are too large, the pendulum would easily lose the balance and thus lead to the end of the episode, which makes SAR in favor of smaller safe regions.
6 Conclusion
We proposed Safe Action Repetition (SAR), a novel 6-invariant action repetition method for policy gradient (PG) algorithms. SAR can handle infinitesimal-ò settings using temporally extended actions without suffering from the variance explosion problem of PG methods, which we proved for general stochastic environments under certain assumption. It can agilely cope with environment stochasticity via learned safe regions. We exhibited that our method achieves both d-invariance and robustness to stochasticity.
Limitations and future directions. SAR operates in environments where the distance functions can be properly defined in the state spaces. We experimented with the é norm as SAR's 's distance function, which might not applicable to discrete or very high dimensional environments such vision-based simulations. As such, expect that combining our method with representation learning techniques [9] I0, 27 would be an interesting future research direction. Also, since SAR in Section 4.3 assumes fully observable MDPs that it can exploit state locality, there is room for improvement in environments with noisy states or partially observable MDPs. Although we have suggested one possible solution in Section 4.4 this could also be the subject of future research.
7 Broader Impact
We expect that our method is especially useful in a variety of real- weal-world situations where the sampling frequencies of simulated environments and real-worke physical sensors are different or where the agent could encounter unexpected situations in the environment. However, in spite of the potential positive aspects, practitioners need to pay sufficient attention to various perspectives on their problems and our assumptions when trying to apply proposed method to real-world problems. For example, in some environments such as autonomous driving, minimizing risk may be more crucial than maximizing rewards, for which they may need to consider incorporating risk-averse methods [26 33 Also, they have to examine the degree to which the assumptions made in this work are satisfied in their problems. For instance, the Markovian property may not generally hold in practical settings, in which applying our method as-is might possess potential risks. They should analyze the ramifications of the assumption mismatch and handle them accordingly (see also Section GJ. With such considerations, we hope that our research provides new insights toward 5-invariant and robust reinforcement learning.
Acknowledgements
We thank the anonymous reviewers for their helpful comments. This work was supported by Samsung Advanced Institute of Technology. Brain Research Program by National Research Foundation of Korea (NRF) 7001773771147886
Institute of Information & communications Technology Planning & Evaluation (ITPP) grant funded by the Korea government (MSIT) and Institute of Information communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (No.2021 2022-00011333 Artificial Intelligence Graduate School Program (Scoul National University)). Gunhee Kim is the corresponding author.
SW StarLab)
References
[1] Marcin Andrychowicz, Anton Raichuk, Piotr Stanezyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain
10

---

Gelly, and Olivier Bachem. What matters for on-policy deep actor-critic methods? large-
scale study. In International Conference on Learning Representations, 2021. URL https 7/ppenreview
[2] L.. Baird. Reinforcement learning in continuous time: advantage updating. Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN"944 4:2448-2453 vol.4, 1994.
[3] Steven J. Bradtke and M. O. Duff. Reinforcement learning methods for continuous-time markov decision problems. In NIPS, 1994.
[4] Alexander Braylan, Mark Hollenbeck, Elliot Meyerson, and R. Miikkulainen. Frame skip is a powerful parameter for learning to play atari. AAAI Workshop: Leurning for General Competency in Video Games, 2015.
[5] K. Doya. Reinforcement learning in continuous time and space. Neural Computation, 12: 219-245 2000.
[6] Jianzhun Du, J. Futoma, and Finale Doshi-Velez. Model-based reinforcement learning for semi-markov decision processes with neural odes. ArXiv, abs/2006. 16210, 2020.
[7] Nicolas Frémaux, H. Sprekeler, and W. Gerstner. Reinforcement learning using continuous time actor-critic framework with spiking neurons. PLoS Computational Biology, 9, 2013.
[8] Shixiang Gu, Ethan Holly, T. Lillicrap, and S. Levine. Deep reinforcement learning for robotic manipulation with asynchronous off- ooflcc updates. 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 3389-33908 2017.
[9] Danijar Hafner, Timothy Lillierap, Jimmy Ba, and Mohammad Norouzi. Dream control: Learning behaviors by latent imagination. In International Conference on Learning Representa- tions, 2020. URL https: htttt::://opereviiii httpppparwrrrr.e..eeeeeorrrrrrra wtt..nnt.oorrmmmdss11TTT0t
[10] Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference Learning Representations, 2021. URL
https://openreviet net. met/ffoum?? http:/oormmmdddooaayoooooo Ooabbbyyybbuu
[11) M. Hausknecht and P. Stone. Deep recurrent q-learning for partially observable mdps. In AAAI Fall Symposia, 2015.
[12] Ashley Hill, Antonin Raffin, Maximilian Emestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https 2018
7/github. gretthhb.cm/hill....... a/ htt//saabebbaseeeeeee
[13] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, V. Vanhoucke, and S. Levine. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. ArXiv, abs/1806. 10293, 2018.
[14] Dmytro Korenkevych, A. Mahmood, Gautham Vasan, and J. Bergstra. Autoregressive policies for continuous control deep reinforcement learning. In IJCAI, 2019.
(15) Aravind Lakshminarayanan, Sahil Sharma, and Balaraman Ravindran. Dynamic action repe- tition for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2017.
[16] T. Lillicrap, Jonathan J. Hunt, A. Pritzel, N. Heess, Erez, Y. Tassa, D. Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, Abs/1109,,0991, 2016.
[17] Alberto Maria Metelli, Flavio Mazzolini, L. Bisi, Luca Sabbioni, and Marcello Restelli. Control frequency adaptation via action persistence in batch reinforcement learning. In ICMI, 2020.
[18] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, loannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602, 2013.

---

[19] V. Mnih, K. Kavukeuoglu, D. Silver, Andrei A. Rusu, J. Veness, Mare G. Bellemare, A. Graves, Martin A. Riedmiller, Andreas K. Fidjeland, Georg Ostrovski. Stig Petersen, C. Beattie, A. Sadik, loannis Antonoglou, Helen King, D. Kumaran, Daan Wierstra, S. Legg. and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 1158829333,, 2015.
[20] V. Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, A. Graves, T. Lillicrap, Tim Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning, ArXiv, abs/16022 102.0783..,, 2016.
[21] R. Munos. Policy gradient in continuous time. In J. Mach. Learn. Res., 2005.
[22] R. Munos and P. Bourgine. Reinforcement learning for continuous stochastic control problems. In NIPS, 1997.
[23] Fabio Pardo, Arash Tavakoli, Vitaly Levdik, and Petar Kormushev. Time limits in reinforcement learning. In International Conference Machine Learning, pages 4045-4054 PMLR, 2018.
[24] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-po pereoorrfore deep learning library. In Advances in Neural Information Processing Systems 32, 2019.
[25] Antonin Raffin, Ashley Hill, Maximilian Emestus, Adam Gleave, Anssi Kanervisto, and Noah Dormann. Stable baselines3.
2019.
[26] R Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at-risk for general loss distribu- tions. Journal of banking & finance, 26(())14443-) 2002.
[27] Julian Schrittwieser, Ioannis Antonoglou, T. Hubert, K. Simonyan, L. Sifre, Simon Schmitt, A. Guez, Edward Lockhart, Demis Hassabis. T. Graepel, T. Lillicrap, and D. Silver. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588 7839:604-609, 2020.
[28] John Schulman, Sergey Levine, P. Abbeel, Michael I. Jordan, and P. Moritz. Trust region policy optimization. ArXiv, abs/1502.05477, 2015.
[29] John Schulman, P. Moritz, Sergey Levine, Michael I. Jordan, and P. Abbeel. High- diggeeiinnsional continuous control using generalized advantage estimation. CoRR,
2016.
[30] John Schulman, F. Wolski, Prafulla Dhariwal, A. Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, abs//700..n 707.06647, 2017.
[31] Sahil Sharma, Aravind S. Lakshminarayanan, and Balaraman Ravindran. Learning to repeat: Fine grained action repetition for deep reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL httpps://ppenreveww net. met//foumm?d= B1GDWV5eg
[32] Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao. Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning, pages 8707-8718 18. PMLR, 2020.
[33] Rahul Singh, Qinsheng Zhang, and Yongxin Chen. Improving robustness via risk averse distributional reinforcement learning. In Learning for Dynamics and Control, pages 958-968. PMLR, 2020.
[34] R. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artif. Intell., 112:181-211 1999.
[35] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[36] C. Tallee, I.. Blier, and Y. Ollivier. Making deep q-learning methods robust to time discretization. In ICML, 2019.
12

---

[37] E. Tooooov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012.
[38] Pawel Wawrzynski. Control policy with autocorrelated noise in reinforcement learning for robotics, International Journal of Machine Learning and Computing, 5(2):91, 2015.
13


Reviews:
[{"review_id": "i9o2suwOxe", "reviewer": "Reviewer_hLjy", "summary": "The paper proposes an alternative to durative actions where instead of outputting an action and how long the action is to persist, it instead outputs an action and a radius. This radius defines a \"safe region\" where the action is to be repeated for as long as an agent is within it. Having the action duration depend on state makes it invariant to the environment's time discretization. They evaluate their proposal on the MuJoCo suite and demonstrate improved performance over an agent which bases its durative action on time alone.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper is overall well written and nicely organized. The motivating environment was really good for building intuition around key limitations of time-based action repetition, as well as the merit of a feature-based one.\n\nI'm recommending weak acceptance at this time, as I believe it meets the bar for publication, but feel it missed an important comparison for painting a clearer empirical understanding of the proposed method:\n\nCould the authors comment on how imposing t_max on SAR ensures a fair comparison between the two methods? For two orthogonal approaches for controlling action repetition, I think this instead blurs the comparison in that the paper lacks a clear/isolated evaluation of the proposed method. From the suggestion that imposing this limit stabilizes things, it would be informative to know the degree at which things are unstable without it, and some explanation for why it fails (e.g., are there notable degenerate cases that arise?). I think the results are promising in that the combination of the two appears to perform well, but think that such an omission detracts from a clear empirical understanding of the proposed method. Similarly, though more distant from this paper's scope, one could instead consider removing t_max from FiGAR-C to understand the role of this limit, e.g., is it a bias to make results more consistent or easier to tune, or is it *necessary* for getting the algorithm to work?\n\nI additionally have the following question, which largely didn't affect my rating: Have the authors tested alternative distance metrics? Can they comment on the sensitivity of the approach to such a choice, or give intuitions around how it might be chosen given a problem (e.g., correlated features in the observation, etc.)?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately addressed, but put in the supplementary material. I believe it is a requirement for it to be in the main text, as per the following post by the program chairs:\n\n\"Whereas the broader impacts section previously did not count towards the page limit, it must now fit within the page limit. However, we have extended the page limit from 8 pages to 9 pages...\"", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "fibUUFW-Pcp", "reviewer": "Reviewer_CWkP", "summary": "This work is motivated by the performance dropoff of RL algorithms when the time scale $\\delta$ decreases to 0. Although previous work has studied such an issue in Q-learning,  this paper rather focuses on policy-gradient (PG). Observing that the variance of PG explodes as $\\delta\\rightarrow 0$, the authors introduce Safe Action Repetition (SAR), which ensures bounded gradient regardless of the time scale $\\delta$. Inspired by action repetition, the proposed technique is additionally able to handle unexpected events by interrupting repetition when the agent reaches an unsafe/uncertain region in the state-space. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Originality*\n\nThis work brings $\\delta$-robustness forward by incorporating it into policy-gradient methods. To my knowledge, the idea behind SAR is new for tackling varying time scales. By accounting for spatial structure, SAR adds a new criterion for action repetition.\n\n*Quality and significance*\n\nI raise a few questions and concerns below:\n1. In Thm 1, why is the dependence on $\\sigma_1$ only? I would expect dependence on the whole covariance matrix, or on some norm of it. Is there an implicit assumption according to which the variance coefficients in $\\Sigma$ are ordered? If so, please mention it. \n\n2. l. 74 - \"this is the first action repetition method that repeats an action based on the agent's state\" and criterion in Eq. (6) therein: It is not clear to me how this distance threshold accounts for safety. Implicitly, it means that the furthest the agent is away from the reference state, the least safe it is. What is the motivation for such an approach? Why not repeat actions based on reward variation instead? For example, given some reference state and action (with corresponding reference reward), the agent could perform action repetition until the reward drops down. It would not be based on a norm because of the asymmetry of that criterion (since higher reward is desirable so you would repeat the action if doing so goes in your favor), but seems like an equally valid measure. \n\n3. In Proposition 2, several parameters vary at once, all of them tending to an extreme case ($\\delta,x\\rightarrow 0, \\nu\\rightarrow \\infty$ and $f=0$). As such, it is not clear how each of these parameters leads to variance explosion. What would happen if $\\delta \\rightarrow 0$ much slower than $\\nu\\rightarrow \\infty$? All of these interdependencies may affect the behavior of the variance. \n\n4. How does SAR evolve as the stochasticity level increases? In particular, I guess this would result in smaller safe regions and consequently reduce action duration. This brings me to considering Eq (8), which introduces a trade-off between distance and time: one would probably have to calibrate $\\lambda$ accordingly maybe by setting $lambda \\rightarrow 1$ as stochasticity increases. Such analysis is missing in the experiments. \n\n5. The first paragraph in Sec 6 reminds the setting of options, which should be addressed in related work:  Why not consider an option approach instead of action repetition? What would be missing there compared to SAR? \n\n*Significance*\n\nOverall, the problem is interesting and the proposed solution numerically promising, but the paper does not sufficiently challenge its method against other alternatives, which questions how much other researchers would later use SAR.  The limitations of SAR are hardly mentioned (only in Conclusion when suggesting an extension to POMDPs).\n\n\n*Clarity*\n\nAlthough the ideas are clearly described, there are several English mistakes that I enumerate below.\nAdditionally, below are some points that remain unclear to me:\n1. l. 283-284 \"Note that this variant of SAR.. is $\\delta$-invariant too\" - proof? Justification?\n2. l. 279 \"$\\lambda$ is the coefficient that controls the relative strength of the first term\" - it rather combines the first term with the second one by trading off distance with time difference. \n3. l. 296 \"While other previous ...\" - I did not understand the point of this sentence \n4. l. 351-353 \"... SAR can quickly and adaptively handle such stochasticity with safe regions by immediately stopping repetitions\" - How is this interpreted from the plot?\n\n \n\n*Typos and minor comments*\n\n- Redundancies in l. 1 - \"discretized\", \"discretization\"; l. 33 - \"differently\", \"different\"; \n- \"the\" should be removed in the following: l. 37 - \"the performance\"; l. 38 - \"the trained policies\"; l. 49 - \"the access\"; l. 56 - \"the prior\"; l. 93 - \"the collapse\"; l. 189 - \"the training\"; l. 190 - \"the computational cost\"; l. 215-216 - \"the state s, not the time t\"; l. 218 - \"the environments\"; l. 231 - \"the repetition\"; l. 232 - \"the action repetition\"; l. 238 - \"the action repetition\"; l. 257 - \"the time\"; l. 267 - \"the variance explosion\"; l. 268 - \"the optimal policy\"; l. 271 - \"the stochasticity\"; l. 273 - \"the variance explosion\"; l. 4 Fig 3 caption - \"the full results\"; l. 282 - \"the time at $s_i$\", \"the repetition\"; l. 304 - \"the training\"; l. 305 - \"the duration\"; l. 2 Fig 4 caption - \"the $95%$\"; l. 342 - \"the position\", \"the velocity\"; l. 364 - \"the locality\"\n- l. 18 \"assume an MDP\" --> \"are based on an MDP\" or \"rely on an MDP\" (MDP is not an assumption but a model)\n- l. 21 \"the MDP assumption\" --> \"the MDP setting\" or \"the MDP model\"\n- l. 20 \"are defined with continuous-time\" --> \"in continuous time\" (note hyphen removal as well)\n-  l. 21 \"the continuous-time\" --> \"the continuous timeline\"\n- l. 40 \"There have been proposed some methods\" --> \"(Some) methods have been proposed\" (\"Some\" is superfluous here also)\n- l. 45 \"its agility\" --> \"agility\"\n- l. 44 \"when it is necessary\" --> \"when necessary\"\n- l. 47 \"by preventing the collapse of the Q function\" --> \"by preventing from Q-function collapse\"\n- l. 59 \"failures\" --> \"failure\"\n- l. 63 \"the be not only $\\delta$-invariant\" --> \"to not only be $\\delta$-invariant\"\n- l. 72 \"any PG methods\" --> \"any PG method\" or \"all PG methods\"\n- l. 95 \"extended ... with\" --> \"extended ... to\"\n- l. 100 \"prevent the variance explosion\" --> \"prevent from variance explosion\"\n- l.121 + l. 205 \"given as\"--> \"given by\"\n- l. 138 \"a solution for\" --> \"a solution to\"\n- l. 193 \"but does only when it is necessary\" --> \"but does so when necessary only\"\n- l. 206 \"it is too applicable\" --> \"it is also applicable\"\n- l. 209 \"replacing ... with\" --> \"replace ... by\"\n- l. 210 \"times of repetition\" --> \"repetition times\"\n- l. 212 \"durations of actions\" --> \"action duration\"\n- l. 217 \"caring underlying changes\" --> \"caring about underlying changes\"; \"degradation in performance\" --> \"performance degradation\"\n- l. 215 \"for a fully observable MDP\" --> \"of a fully ....\"\n- l. 218 \"vary greatly\" --> \"greatly vary\"\n- l. 219 \"we will demonstrate\" --> \"we demonstrate\"\n- l. 231 \"outside the safe region\" --> \"outside of the safe region\"\n- l. 234 \"since its action duration, which is determined by the radius of a safe region, has no relation with how fine the discretization time scale is\" --> \"since action duration is determined by the safe region radius, which is not related to how fine the discretization timescale is\"\n- l. 236 \"and how it is scaled\" --> \"and the way it is scaled\"\n- l. 259 \"good num actions to maximize f(num)\" --> \"num actions that maximize f(num)\"\n- l. 260 \"the optimal policy\" --> \"an optimal policy\" (there may be more than one); \"for off that\" --> \"for off such that\"\n- l. 261 \"for num that\" --> \"for num such that\"\n- l. 266 \"the full proof in Appendix\" --> \"a full proof in the Appendix\"\n- l. 269 \"it can be also at risk of failure\" --> \"it can also be a risk of failure\"\n- l. 275 \"be finitely bounded\" --> \"be bounded\" (if it is were infinite then it would not be bounded)\n- l. 277 \"we derived SAR based on that the optimal policy ... depends only on states\" --> \"We derive SAR based on the fact that an optimal policy ... only depends on states\"\n- l. 4 Fig 3 caption \"We refer to Appendix\" --> \"to the Appendix\"\n- l. 287 \"We first demonstrate $\\delta$-invariance of\" --> \"the $\\delta$-invariance of\"\n- l. 282 \"repetition. $0 \\leq \\lambda \\leq 1$\" --> \"repetition,  $0 \\leq \\lambda \\leq 1$\"\n- l. 296 \"While other previous ... approaches\" --> \"Although previous ... approaches\"\n- l. 308 \"though they have randomized\" --> \"although they have randomized\"\n- l. 288 \"comparing with\" --> \"compared to\"\n- l. 313 \"in Appendix\" --> \"in the Appendix\"\n- l. 314 \"their performances\" --> \"their performance\"\n- l. 358 \"with the stochasticity of environments\" --> \"with stochasticity\" or \"with environment stochasticity\"\n- l. 364 \"the locality of states\" --> \"states locality\"\n\nPOST REBUTTAL: I am happy to give an additional point to the authors for their clarifications. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See details above", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "UUosIEL4nS", "reviewer": "Reviewer_HCbD", "summary": "Real world tasks are often in continuous time, whereas typical RL algorithms operate in discrete time. In such situations the algorithms are often sensitive to the chosen discretization time step.\n\nSmall discretization scales can lead to various problems such as increases in the gradient variance, high computational costs, and difficulties with exploration.\n\nThis paper proposes safe action repetition (SAR) to tackle these issues. SAR augments the action space with an additional variable that determines for how long the chosen action will be repeated. In particular, this additional variable will determine the size of an L1 ball around the current state, such that when the agent leaves this ball, a new action will be chosen, i.e. it determines the size of a \"safe region\" during which the action will be repeated.\n\nOne previous related work that tackled this problem was FiGAR that used a similar augmentated action, but which determined the number of action repeats, not the size of a safe region. A downside of this approach is that it can not react when something goes wrong during the action repeat period.\n\nThey perform experiments on MuJoCo continuous control tasks combining their algorithm with PPO, TRPO and A2C and show invariance to variance time discretization lengths and improved performance. They compared to FiGAR as well as DAU, another algorithm for discretization invariance (they compared to the official implementation). Their method usually considerably outperformed the other algorithms.\n\nThey also proved the explosion of the policy gradient variance when the discretization goes to 0 under the assumption that the variance of the return stays lower bounded.\n\n------------------------------------------------------------------------\nUpdate: See the comments for more discussion. The score remained unchanged.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general the paper was well written, the tackled problem relevant and the approach sensible. The experiments included ablation studies. They also included small toy examples explaining the benefits of their method. One thing that was still unclear to me was why the agent would simply not learn to set the safe region as small as possible to maximize the control frequency and the achievable performance? I hope the authors can explain this in their response. I also point out that the assumption that the variance of the return is lower bounded may not hold. \n\n**Variance may not be lower bounded**\n\nEven if the discretized environment is stochastic, as the discretization time step is reduced, the effect of the stochasticity can become smaller and the outcome can become deterministic in the continuous limit. For example, this will happen in the example by Munos that you cited, and actually, if one takes multiple samples and uses a mean baseline, the policy gradient variance would not explode in their example. One could also construct other examples where the return variance keeps decreasing as the discretization time scale is reduced. Based on this, I believe some of the claims are a bit too strong, and it should be emphasized that it relies on the assumption that the variance is lower bounded, which does not hold in general (though there are many examples where indeed the PG variance will explode, and the experiments in the appendix also demonstrated this empirically.)\n\n**Comment on proposition 2**\n\nThe example is OK, but I found it somewhat artificial to just place noise on the final reward to ensure that reward variance stays lower bounded. Another point that bothered me was that the choice of the size of the safe region has no effect on the task performance because there are only two states 0 and 1, and irrespective of the size of the safe region, the next action will be chosen right as the state changes. On the other hand, if the state were to change continuously, wouldn't the policy with $d_i \\to 0$ also be optimal, and lead to an infinite amount action steps and an explosion of the gradient variance also for SAR (this is basically related to my previous question of why the agent does not just learn to set the safe region as small as possible)?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No concern", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4h"}, "extra_text": {}}, {"review_id": "7WAuRi10cNv", "reviewer": "Reviewer_QGiy", "summary": "This paper studies the effect of discretization of the time on the policy space methods in RL. Since the underlying physics of the most of the RL systems is in continuous time, one needs to perform a discretization of the time horizon to employ policy gradient to find the optimal policy. However, as shown in the paper, policy space methods might suffer from high variance when the discretization parameter is small. In addition, a large discretization parameter is not also optimal, because it will result in a suboptimal policy, and it is not robust to stochastic behavior of the system. To deal with this problem, the authors propose SAR (safe action repetition). The idea is to let the algorithm repeat the same set of actions for a window of time, until the system exists some \"safe region\". To find this safe region, the policy outputs the radius of the safe region, and the algorithm needs to find the optimal policy which maximizes the reward while finding the optimal safe region. The authors compare their algorithm with the previous work experimentally, and they show that their algorithms outperforms FiGAR-C.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, it is a good work. However, I have some questions:\n- What is V(s) in equation (5)?\n- You used the notion of a distance in the state space to create the safe region. How can we extend this method to the systems where one cannot define a notion of distance on the state space?\n- In SAR, the radius d_i is one of the outputs of the policy and needs to be learned while we learn the optimal policy. What if the initialization of the policy assigns very small (or very large) radiuses in some states, and hence the algorithm will be divergent? In other word, I am concerned about the effect of the initialization of the policy (specially the safe region radius d_i) on the final policy achieved by the algorithm. How can we make the algorithm less sensitive to the initialization of the policy?\n- Although the authors provide figure 6 which shows that SAR produces reasonable safe regions, I was wondering how is the algorithm encouraged to find the best safe region? In other words, I want to know, in an intuitive sense, how the algorithm is rewarded to end up getting a policy which gets a larger safe region when the speed is low and bigger safe region when the speed is high?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Need for notion of distance in the state space", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel trick to address the time discretization issue with continuous-time policy gradient method. As a small time interval delta is used, the authors shows that variance of PG estimator can exploded when delta->0. To address the issue, they propose a Safe Action Repetition method, which lets the action repeat when the state changes within a small ball with diameter d. This approach guarantees safe and robust policy optimization that is delta-invariant and adaptive to exploration spped. All reviewers appreciate that this solution is interesting and potentially useful. 

The paper is still improvable. For example, the delta-invariance is achieved by reducing the time discretization problem to identifying a small neighborhood of the state space. This technique heavily depends on a priorly available state distance metric. Finding a good state distance metric is nontrivial, and would require further research beyond this paper.
Decision: Accept (Poster)