Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes an alternative to durative actions where instead of outputting an action and how long the action is to persist, it instead outputs an action and a radius. This radius defines a "safe region" where the action is to be repeated for as long as an agent is within it. Having the action duration depend on state makes it invariant to the environment's time discretization. They evaluate their proposal on the MuJoCo suite and demonstrate improved performance over an agent which bases its durative action on time alone.
main_review: I think the paper is overall well written and nicely organized. The motivating environment was really good for building intuition around key limitations of time-based action repetition, as well as the merit of a feature-based one.

I'm recommending weak acceptance at this time, as I believe it meets the bar for publication, but feel it missed an important comparison for painting a clearer empirical understanding of the proposed method:

Could the authors comment on how imposing t_max on SAR ensures a fair comparison between the two methods? For two orthogonal approaches for controlling action repetition, I think this instead blurs the comparison in that the paper lacks a clear/isolated evaluation of the proposed method. From the suggestion that imposing this limit stabilizes things, it would be informative to know the degree at which things are unstable without it, and some explanation for why it fails (e.g., are there notable degenerate cases that arise?). I think the results are promising in that the combination of the two appears to perform well, but think that such an omission detracts from a clear empirical understanding of the proposed method. Similarly, though more distant from this paper's scope, one could instead consider removing t_max from FiGAR-C to understand the role of this limit, e.g., is it a bias to make results more consistent or easier to tune, or is it *necessary* for getting the algorithm to work?

I additionally have the following question, which largely didn't affect my rating: Have the authors tested alternative distance metrics? Can they comment on the sensitivity of the approach to such a choice, or give intuitions around how it might be chosen given a problem (e.g., correlated features in the observation, etc.)?
limitations_and_societal_impact: Adequately addressed, but put in the supplementary material. I believe it is a requirement for it to be in the main text, as per the following post by the program chairs:

"Whereas the broader impacts section previously did not count towards the page limit, it must now fit within the page limit. However, we have extended the page limit from 8 pages to 9 pages..."
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the effect of discretization of the time on the policy space methods in RL. Since the underlying physics of the most of the RL systems is in continuous time, one needs to perform a discretization of the time horizon to employ policy gradient to find the optimal policy. However, as shown in the paper, policy space methods might suffer from high variance when the discretization parameter is small. In addition, a large discretization parameter is not also optimal, because it will result in a suboptimal policy, and it is not robust to stochastic behavior of the system. To deal with this problem, the authors propose SAR (safe action repetition). The idea is to let the algorithm repeat the same set of actions for a window of time, until the system exists some "safe region". To find this safe region, the policy outputs the radius of the safe region, and the algorithm needs to find the optimal policy which maximizes the reward while finding the optimal safe region. The authors compare their algorithm with the previous work experimentally, and they show that their algorithms outperforms FiGAR-C.
main_review: Overall, it is a good work. However, I have some questions:
- What is V(s) in equation (5)?
- You used the notion of a distance in the state space to create the safe region. How can we extend this method to the systems where one cannot define a notion of distance on the state space?
- In SAR, the radius d_i is one of the outputs of the policy and needs to be learned while we learn the optimal policy. What if the initialization of the policy assigns very small (or very large) radiuses in some states, and hence the algorithm will be divergent? In other word, I am concerned about the effect of the initialization of the policy (specially the safe region radius d_i) on the final policy achieved by the algorithm. How can we make the algorithm less sensitive to the initialization of the policy?
- Although the authors provide figure 6 which shows that SAR produces reasonable safe regions, I was wondering how is the algorithm encouraged to find the best safe region? In other words, I want to know, in an intuitive sense, how the algorithm is rewarded to end up getting a policy which gets a larger safe region when the speed is low and bigger safe region when the speed is high?

limitations_and_societal_impact: Need for notion of distance in the state space
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Real world tasks are often in continuous time, whereas typical RL algorithms operate in discrete time. In such situations the algorithms are often sensitive to the chosen discretization time step.

Small discretization scales can lead to various problems such as increases in the gradient variance, high computational costs, and difficulties with exploration.

This paper proposes safe action repetition (SAR) to tackle these issues. SAR augments the action space with an additional variable that determines for how long the chosen action will be repeated. In particular, this additional variable will determine the size of an L1 ball around the current state, such that when the agent leaves this ball, a new action will be chosen, i.e. it determines the size of a "safe region" during which the action will be repeated.

One previous related work that tackled this problem was FiGAR that used a similar augmentated action, but which determined the number of action repeats, not the size of a safe region. A downside of this approach is that it can not react when something goes wrong during the action repeat period.

They perform experiments on MuJoCo continuous control tasks combining their algorithm with PPO, TRPO and A2C and show invariance to variance time discretization lengths and improved performance. They compared to FiGAR as well as DAU, another algorithm for discretization invariance (they compared to the official implementation). Their method usually considerably outperformed the other algorithms.

They also proved the explosion of the policy gradient variance when the discretization goes to 0 under the assumption that the variance of the return stays lower bounded.

------------------------------------------------------------------------
Update: See the comments for more discussion. The score remained unchanged.
main_review: In general the paper was well written, the tackled problem relevant and the approach sensible. The experiments included ablation studies. They also included small toy examples explaining the benefits of their method. One thing that was still unclear to me was why the agent would simply not learn to set the safe region as small as possible to maximize the control frequency and the achievable performance? I hope the authors can explain this in their response. I also point out that the assumption that the variance of the return is lower bounded may not hold. 

**Variance may not be lower bounded**

Even if the discretized environment is stochastic, as the discretization time step is reduced, the effect of the stochasticity can become smaller and the outcome can become deterministic in the continuous limit. For example, this will happen in the example by Munos that you cited, and actually, if one takes multiple samples and uses a mean baseline, the policy gradient variance would not explode in their example. One could also construct other examples where the return variance keeps decreasing as the discretization time scale is reduced. Based on this, I believe some of the claims are a bit too strong, and it should be emphasized that it relies on the assumption that the variance is lower bounded, which does not hold in general (though there are many examples where indeed the PG variance will explode, and the experiments in the appendix also demonstrated this empirically.)

**Comment on proposition 2**

The example is OK, but I found it somewhat artificial to just place noise on the final reward to ensure that reward variance stays lower bounded. Another point that bothered me was that the choice of the size of the safe region has no effect on the task performance because there are only two states 0 and 1, and irrespective of the size of the safe region, the next action will be chosen right as the state changes. On the other hand, if the state were to change continuously, wouldn't the policy with $d_i \to 0$ also be optimal, and lead to an infinite amount action steps and an explosion of the gradient variance also for SAR (this is basically related to my previous question of why the agent does not just learn to set the safe region as small as possible)?
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work is motivated by the performance dropoff of RL algorithms when the time scale $\delta$ decreases to 0. Although previous work has studied such an issue in Q-learning,  this paper rather focuses on policy-gradient (PG). Observing that the variance of PG explodes as $\delta\rightarrow 0$, the authors introduce Safe Action Repetition (SAR), which ensures bounded gradient regardless of the time scale $\delta$. Inspired by action repetition, the proposed technique is additionally able to handle unexpected events by interrupting repetition when the agent reaches an unsafe/uncertain region in the state-space. 
main_review: *Originality*

This work brings $\delta$-robustness forward by incorporating it into policy-gradient methods. To my knowledge, the idea behind SAR is new for tackling varying time scales. By accounting for spatial structure, SAR adds a new criterion for action repetition.

*Quality and significance*

I raise a few questions and concerns below:
1. In Thm 1, why is the dependence on $\sigma_1$ only? I would expect dependence on the whole covariance matrix, or on some norm of it. Is there an implicit assumption according to which the variance coefficients in $\Sigma$ are ordered? If so, please mention it. 

2. l. 74 - "this is the first action repetition method that repeats an action based on the agent's state" and criterion in Eq. (6) therein: It is not clear to me how this distance threshold accounts for safety. Implicitly, it means that the furthest the agent is away from the reference state, the least safe it is. What is the motivation for such an approach? Why not repeat actions based on reward variation instead? For example, given some reference state and action (with corresponding reference reward), the agent could perform action repetition until the reward drops down. It would not be based on a norm because of the asymmetry of that criterion (since higher reward is desirable so you would repeat the action if doing so goes in your favor), but seems like an equally valid measure. 

3. In Proposition 2, several parameters vary at once, all of them tending to an extreme case ($\delta,x\rightarrow 0, \nu\rightarrow \infty$ and $f=0$). As such, it is not clear how each of these parameters leads to variance explosion. What would happen if $\delta \rightarrow 0$ much slower than $\nu\rightarrow \infty$? All of these interdependencies may affect the behavior of the variance. 

4. How does SAR evolve as the stochasticity level increases? In particular, I guess this would result in smaller safe regions and consequently reduce action duration. This brings me to considering Eq (8), which introduces a trade-off between distance and time: one would probably have to calibrate $\lambda$ accordingly maybe by setting $lambda \rightarrow 1$ as stochasticity increases. Such analysis is missing in the experiments. 

5. The first paragraph in Sec 6 reminds the setting of options, which should be addressed in related work:  Why not consider an option approach instead of action repetition? What would be missing there compared to SAR? 

*Significance*

Overall, the problem is interesting and the proposed solution numerically promising, but the paper does not sufficiently challenge its method against other alternatives, which questions how much other researchers would later use SAR.  The limitations of SAR are hardly mentioned (only in Conclusion when suggesting an extension to POMDPs).


*Clarity*

Although the ideas are clearly described, there are several English mistakes that I enumerate below.
Additionally, below are some points that remain unclear to me:
1. l. 283-284 "Note that this variant of SAR.. is $\delta$-invariant too" - proof? Justification?
2. l. 279 "$\lambda$ is the coefficient that controls the relative strength of the first term" - it rather combines the first term with the second one by trading off distance with time difference. 
3. l. 296 "While other previous ..." - I did not understand the point of this sentence 
4. l. 351-353 "... SAR can quickly and adaptively handle such stochasticity with safe regions by immediately stopping repetitions" - How is this interpreted from the plot?

 

*Typos and minor comments*

- Redundancies in l. 1 - "discretized", "discretization"; l. 33 - "differently", "different"; 
- "the" should be removed in the following: l. 37 - "the performance"; l. 38 - "the trained policies"; l. 49 - "the access"; l. 56 - "the prior"; l. 93 - "the collapse"; l. 189 - "the training"; l. 190 - "the computational cost"; l. 215-216 - "the state s, not the time t"; l. 218 - "the environments"; l. 231 - "the repetition"; l. 232 - "the action repetition"; l. 238 - "the action repetition"; l. 257 - "the time"; l. 267 - "the variance explosion"; l. 268 - "the optimal policy"; l. 271 - "the stochasticity"; l. 273 - "the variance explosion"; l. 4 Fig 3 caption - "the full results"; l. 282 - "the time at $s_i$", "the repetition"; l. 304 - "the training"; l. 305 - "the duration"; l. 2 Fig 4 caption - "the $95%$"; l. 342 - "the position", "the velocity"; l. 364 - "the locality"
- l. 18 "assume an MDP" --> "are based on an MDP" or "rely on an MDP" (MDP is not an assumption but a model)
- l. 21 "the MDP assumption" --> "the MDP setting" or "the MDP model"
- l. 20 "are defined with continuous-time" --> "in continuous time" (note hyphen removal as well)
-  l. 21 "the continuous-time" --> "the continuous timeline"
- l. 40 "There have been proposed some methods" --> "(Some) methods have been proposed" ("Some" is superfluous here also)
- l. 45 "its agility" --> "agility"
- l. 44 "when it is necessary" --> "when necessary"
- l. 47 "by preventing the collapse of the Q function" --> "by preventing from Q-function collapse"
- l. 59 "failures" --> "failure"
- l. 63 "the be not only $\delta$-invariant" --> "to not only be $\delta$-invariant"
- l. 72 "any PG methods" --> "any PG method" or "all PG methods"
- l. 95 "extended ... with" --> "extended ... to"
- l. 100 "prevent the variance explosion" --> "prevent from variance explosion"
- l.121 + l. 205 "given as"--> "given by"
- l. 138 "a solution for" --> "a solution to"
- l. 193 "but does only when it is necessary" --> "but does so when necessary only"
- l. 206 "it is too applicable" --> "it is also applicable"
- l. 209 "replacing ... with" --> "replace ... by"
- l. 210 "times of repetition" --> "repetition times"
- l. 212 "durations of actions" --> "action duration"
- l. 217 "caring underlying changes" --> "caring about underlying changes"; "degradation in performance" --> "performance degradation"
- l. 215 "for a fully observable MDP" --> "of a fully ...."
- l. 218 "vary greatly" --> "greatly vary"
- l. 219 "we will demonstrate" --> "we demonstrate"
- l. 231 "outside the safe region" --> "outside of the safe region"
- l. 234 "since its action duration, which is determined by the radius of a safe region, has no relation with how fine the discretization time scale is" --> "since action duration is determined by the safe region radius, which is not related to how fine the discretization timescale is"
- l. 236 "and how it is scaled" --> "and the way it is scaled"
- l. 259 "good num actions to maximize f(num)" --> "num actions that maximize f(num)"
- l. 260 "the optimal policy" --> "an optimal policy" (there may be more than one); "for off that" --> "for off such that"
- l. 261 "for num that" --> "for num such that"
- l. 266 "the full proof in Appendix" --> "a full proof in the Appendix"
- l. 269 "it can be also at risk of failure" --> "it can also be a risk of failure"
- l. 275 "be finitely bounded" --> "be bounded" (if it is were infinite then it would not be bounded)
- l. 277 "we derived SAR based on that the optimal policy ... depends only on states" --> "We derive SAR based on the fact that an optimal policy ... only depends on states"
- l. 4 Fig 3 caption "We refer to Appendix" --> "to the Appendix"
- l. 287 "We first demonstrate $\delta$-invariance of" --> "the $\delta$-invariance of"
- l. 282 "repetition. $0 \leq \lambda \leq 1$" --> "repetition,  $0 \leq \lambda \leq 1$"
- l. 296 "While other previous ... approaches" --> "Although previous ... approaches"
- l. 308 "though they have randomized" --> "although they have randomized"
- l. 288 "comparing with" --> "compared to"
- l. 313 "in Appendix" --> "in the Appendix"
- l. 314 "their performances" --> "their performance"
- l. 358 "with the stochasticity of environments" --> "with stochasticity" or "with environment stochasticity"
- l. 364 "the locality of states" --> "states locality"

POST REBUTTAL: I am happy to give an additional point to the authors for their clarifications. 


limitations_and_societal_impact: See details above
review_title: null

Meta-review: This paper proposes a novel trick to address the time discretization issue with continuous-time policy gradient method. As a small time interval delta is used, the authors shows that variance of PG estimator can exploded when delta->0. To address the issue, they propose a Safe Action Repetition method, which lets the action repeat when the state changes within a small ball with diameter d. This approach guarantees safe and robust policy optimization that is delta-invariant and adaptive to exploration spped. All reviewers appreciate that this solution is interesting and potentially useful. 

The paper is still improvable. For example, the delta-invariance is achieved by reducing the time discretization problem to identifying a small neighborhood of the state space. This technique heavily depends on a priorly available state distance metric. Finding a good state distance metric is nontrivial, and would require further research beyond this paper.
Decision: Accept (Poster)