TAAC: Temporally Abstract Actor-Critic for Continuous Control
Authors: Haonan Yu
Wei Xu
Haichao Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: temporal abstraction
action repetition
exploration
off-policy RL

Abstract: We present temporally abstract actor-critic (TAAC), a simple but effective off-policy RL algorithm that incorporates closed-loop temporal abstraction into the actor-critic framework. TAAC adds a second-stage binary policy to choose between the previous action and a new action output by an actor. Crucially, its "act-or-repeat" decision hinges on the actually sampled action instead of the expected behavior of the actor. This post-acting switching scheme let the overall policy make more informed decisions. TAAC has two important features: a) persistent exploration, and b) a new compare-through Q operator for multi-step TD backup, specially tailored to the action repetition scenario. We demonstrate TAAC's advantages over several strong baselines across 14 continuous control tasks. Our surprising finding reveals that while achieving top performance, TAAC is able to "mine" a significant number of repeated actions with the trained policy even on continuous tasks whose problem structures on the surface seem to repel action repetition. This suggests that aside from encouraging persistent exploration, action repetition can find its place in a good policy behavior. Code is available at https://github.com/hnyu/taac.

TAAC: Temporally Abstract Actor-Critic for Continuous Control
Haonan Yu, Wei Xu, Haichao Zhang
Horizon Robotics Cupertino, CA 95014
(haonan, yu, wei xu, xu, haichao htw..aacc....zaaglooorrooooo
Abstract
We present temporally abstract actor-critic (TAAC), simple but effective off- policy RL algorithm that incorporates closed-loop temporal abstraction into the actor-critic framework. TAAC adds a second-stage binary policy to choose between the previous action and new action output by an actor. Crucially, its "act-or-repeat" decision hinges on the actually sampled action instead of the expected behavior the actor. This post-acting switching scheme let the overall policy make more informed decisions. TAAC has two important features: a) persistent exploration, and b) a new compare-through Q operator for multi-step TD backup, specially tailored to the action repetition scenario. We demonstrate TAAC's advantages over several strong baselines across continuous control tasks. Our surprising finding reveals that while achieving top performance, TAAC is able "mine" significant number of repeated actions with the trained policy even on continuous tasks whose problem structures the surface seem to repel action repetition. This suggests that aside from encouraging persistent exploration, action repetition can find its place good policy behavior. Code is available at https: htt///gitttt www.coo///////oo www/hnyu/aa//ccccccccc taac.
Introduction
Deep reinforcement learning (RL) has achieved great success various continuous action domains such as locomotion and manipulation (Schulman et al., 2015; Lillicrap al., 2016; Duan et al., 2016; Schulman et al., 2017; Fujimoto et al., 2018; Haarnoja et al., 2018). Despite promising empirical results, these widely applicable continuous RL algorithms execute newly computed action every step of the finest time scale of problem. With no decision making at higher levels, they attempt to solve the challenging credit assignment problem over long horizon. As result, considerable sample efficiency improvements have yet to be made by them in complex task structures (Riedmiller al., 2018; 8; Li al., 2020; Lee et al., 2020b) and extremely sparse reward settings (Andrychowicz et al., 2017; Plappert et al., 2018; Zhang et al., 2021).
On the other hand, is argued that temporal abstraction (Parr and Russell, 1998; Dietterich, 1998; Sutton et al., 1999; Precup, 2000) is one of the crucial keys to solving control problems with complex structures. Larger steps are taken at higher levels of abstraction while lower-level actions only need to focus on solving isolated subtasks (Dayan and Hinton, 1993; Vezhnevets et al., 2017; Bacon et al., 2017). However, most hierarchical RL (HRL) methods are task specific and nontrivial to adapt. For example. the options framework (Sutton et al., 1999; Precup, 2000; Bacon et al., 2017) requires pre-defining an option space, while the feudal RL framework Vezhnevets al. (2017); Nachum et al. (2018); Zhang et al. (2021) requires tuning the hyperparameters of dimensionality and domain range of the goal space. In practice, their final performance usually hinges on these choices.
Perhaps the simplest form of an option or sub-policy would be just repeating an action for a certain number of steps, straightforward idea that has been widely explored (Lakshminarayanal al.,
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

2017; Sharma et al., 2017; Dabney et al., 2021; Metelli et al., 2020; Lee et al., 2020a; Biedenkapp et al., 2021). This line of works can be regarded as middle ground between "flat" RL and HRL. They assume a fixed candidate set of action durations, and repeat actions in an open-loop manner. Open-loop control forces an agent to commit to the same action over a predicted duration with no opportunity of early terminations. It weakens the agent's 's ability of handling emergency situations and correcting wrong durations predicted earlier. To address this inflexibility, a handful of prior works (Neunert et al., 2020; Chen et al., 2021) propose to output an "act-or-repeat" binary decision to decide if the action at the previous step should be repeated. Because this "act-or-repeat" decision will be examined at every step depending on the current environment state, this results in closed-loop action repetition.
All these action-repetition methods are well justified by the need of action persistence (Dabney et al., 2021; Amin et al., 2021; Zhang and Van Hoof, 2021; Grigsby et al., 2021) for designing a good exploration strategy, when action diversity should be traded for it properly. This trade-off is important because when reward is sparse or short-term reward is deceptive, action diversity alone only makes the agent wandering around its local neighborhood since any persistent trajectory has an exponentially small probability. In such a case, a sub-optimal solution is likely to be found. In contrast, persistence via action repetition makes the policy explore deeper (while sacrificing action diversity to some extent).
This paper further explores in the direction of closed-loop action repetition, striving to discover a novel algorithm that instantiates this idea better. The key question we ask is, how can we exploit the special structure of closed-looy repetition, so that our algorithm yields better sample efficiency and final performance compared existing methods? As an answer to this question, we propose temporally abstract actor-critic (TAAC), a simple but effective off-policy RL algorithm that incorporates closed- loop action repetition into an actor-critic framework. Generally, we add a second stage that chooses between candidate action output by an actor and the action from the previous step (Figure 1). Crucially, its "act-or-repeat" decision hinges on the actually sampled individual action instead of the expected behavior of the actor unlike recent works (Neunert et al., 2020; Chen et al., 2021). This post-acting switching scheme let the overall policy make more informed decisions. Moreover,
i) for policy evaluation, we propose a new compare-througe Q operator for multi-step TD backup tailored to the action repetition scenario, instead of replying on generic importance correction; ii) for policy improvement, we compute the actor gradient by multiplying a scaling factor to the aQ term from DDPG (Lillicrap et al., 2016) and SAC (Haarnoja et al., 2018), where the scaling factor is the optimal probability of choosing the actor's action in the second stage.
TAAC is much easier to train compared to sophisticated HRL methods, while it has two important features compared to "flat" RL algorithms, namely, persistent exploration and native multi-step TD backup support without the need of off- off-policyness correction.
We evaluate TAAC on 14 continuous control tasks, covering simple control, locomotion, terrain walking (Brockman et al., 2016), manipulation (Plappert et al., 2018), and self-driving (Dosovitskiy et al., 2017). Averaged over these tasks, TAAC largely outperforms 6 strong baselines. Importantly, our results show that it is our concrete instantiation of closed-loop action repetition that is vital to the final performance. The mere idea of repeating actions in closed-loop manner doesn't 't guarantee better results than the compared open-loop methods. Moreover, our surprising finding reveals that while achieving top performance, TAAC is able "mine" significant number of repeated actions with the trained policy even on continuous tasks whose problem structures on the surface seem to repel action repetition (Section 5.6.2). This suggests that aside from encouraging persistent exploration, action repetition can find its place in good policy behavior. This is perhaps due to that the action frequency of a task can be difficult be set exactly as the minimum value that doesn't 't comprise optimal control while leaving no room for temporal abstraction (Grigsby et al., 2021).
2 Related work
Under the category of temporal abstraction via action repetition, there have been various formulations. Dabney et al. (2021) proposes temporally extended é-greedy exploration where a duration for repeating actions is sampled from a pre-defined truncated zeta distribution. This strategy only affects the exploration behavior for generating off- off-policy data but does not change the training objective. Sharma et al. (2017) and Biedenkapp et al. (2021) learn a hybrid action space and treat the discrete
2

---

action as latent variable of action repeating steps. but the introduced temporal abstraction is open- loop and lacks flexibility. One recent work close to TAAC is PIC (Chen et al., 2021) which also learns to repeat the last action to address the action oscillation issue within consecutive steps. However, PIC was proposed for discrete control and its extension to continuous control is unclear yet. Also, PIC predicts whether to repeat the last action independent of newly sampled action, which requires its switching policy to make a decision regarding the core policy's expected behavior. In an application section, H-MPO (Neunert et al., 2020) explored how continuous control can benefit from meta binary action that modifies the overall system behavior. Again, like PIC their binary decision is made in parallel with newly sampled action. Different from PIC and H-MPO, TAAC only decides "act-or-repeat" after comparing the previous action with a newly sampled action. Moreover, TAAC employs new compare -trruuhh operator exploit repeated actions for multi-step TD backup, and is trained by much simpler actor gradient by absorbing the closed-form solution of the binary policy into the continuous action objective to avoid parameterizing discrete policy unlike H-MPÔ.
Our experiment design (Section 5.2) has covered most methods that consider action repetition. Table provides a checklist of the differences between TAAC and these methods.
3 Preliminaries
We consider the RL problem policy search in a Markov Decision Process (MDP). Let E RM denote a state, where a continuous action a E R. taken. Let π (als) be the action policy, and P(s 0z) the probability of the environment transitioning to St-1 after an action a, is taken
at 8t. Upon reaching 81+1, the agent receives a scalar reward r(s,, as Se+ 1). The RL objective
to find policy n² that maximizes the expected discounted return: Ex,P 1==00" 7'r(st, dr 81+1)],
where y E (0, 1) is discount factor. We also define Q" (s,, az) = Ex 12 ? "r(ss, de, 8e41)l as the discounted return starting from 9; given that a is taken and then π is followed, and (st) Eaew Q" (s,, ax) as the discounted return starting from s following æ.
In an off-policy actor-critic setting with π and Q parameterized by & and 8, a surrogate objective is usually used (Lillicrap et al., 2016; Haarnoja et al., 2018)
max E VØ V**(() BND
A max Φ
E end DaDa Q0(s,a).
This objective maximizes the expected state value over some state distribution, assuming that 1) sampled from a replay buffer D instead of the current policy, and 2) the dependency of the critic Qo(s, a) on the policy To is dropped when computing the gradient of Φ.. Meanwhile, 0 is learned separately via policy evaluation with typical TD backup.
4 Temporally abstract actor-critic
pol-Is,a )
((1s,
à O.
To enable temporal abstraction, we decompose the agent's s action decision into two stages (Figure 1): 1) sampling a new candidate action à TTo-Is, (a)ss,a)))) con- ditioned on the action a at the previous time step, and 2) choosing between put at the current step. The overall TAAC algorithm is summarized in Algorithm Appendix A.
Figure 1: TAAC's two-stage policy during inference. In the first stage, an action pol- icy To samples a candidate action â. In the second stage, a binary switching policy chooses between this candidate and the previous action a
and à as the actual out-
4.1 Two-stage policy
Formally, let [bb]s à, a) be the binary switching policy, where b = 0/1 means choosing a- /â. For simplicity, in the following we will denote B6 = [(b)s, à, a (always assuming its dependency on s, à, and a "). Then our two-stage policy xT"h for temporal abstraction is defined as
n (als, a A (alass a [30((a - a-) )+ + 315(a - a)] dā,
(2)
which can be shown to be proper probability distribution of a. This two-stage policy repeats previous actions through binary policy B, decision maker that compares a and â side by side
3

---

given the current state s. Repeatedly favoring b = 0 results in temporal abstraction of executing the same action for multiple steps. Moreover, this control is closed-loopp as does not commit to a pre-determined time window; instead it can stop repetition whenever necessary. As a special case, when $1 = 1, Th reduces to Tे; when So = 1, nThh (a|s, a = 6(a )-).
4.2 Policy evaluation with the compare-through operator
The typical one-step TD learning objective for a policy ι is
min E (Qo(s,a) a) 8 ( a))² with 5 Q8(s, = r(s,,, (s'),
(3)
8 (s,a,)))~D where B" is the Bellman operator, and slowly tracks @ to stabilize the learning (Mnih et al., 2015). For multi-step bootstrapping, importance correction usually needed, for example, Retrace (Munos et al., 2016) relies on a function of sfale)
x(ale) (μ is a behavior policy) to correct for the off-
of trajectory. Unfortunately, our T/ makes probability density computation challenging because of the marginalization over a Thus importance correction methods including Retrace cannot be applied to our case. To address this issue, below we propose new multi-step Q operator, called compare through... Then we explain how T² can exploit this operator for efficient policy evaluation.
For learning policy given trajectory (so; Go. 81, .1...... 8N, aN) from behavior policy μ,, let a. denote the actions sampled from æ at states 8n (n 2 1) and äg = ag (we do not sample from æ at 8o). We define (a point estimate of) the compare- -hrruugh operator T as
T=Qol50; an) 20 Σ rrr(se, ae 8441) + 7 (2( an),
where n = min (fn: ān * an) U (N)). Intuitively, given a sampled trajectory of length N from the replay buffer, the compare- compare-through operator takes an expectation, under the current policy the sampled states (from 81 to sx), over all the sub-trajectoriess (up to length N) of actions that match the sampled actions. Note that Eq. 4 is point estimate of this expectation. A formal definition of T is described by Eq. 17, and its relation to Retrace (Munos et al., 2016) is shown in Appendix L.
Theorem (Policy evaluation convergence) In tabular setting, the compare-through operator T whose point estimate defined by Eq. 4 (without the parameters 0) and expectation form defined by Eq. 17, has a unique fixed point Q", where ß is the current (target) policy.
For the detailed proof we refer the reader to Appendix L. Although the actual setting considered in this paper are continuous state and action domains with function approximators, Theorem I still provides some justification for our compare- compare-through operator.
Clearly, any discrete policy could exploit the compare-througee operator since there can be a non- zero chance of two discrete actions being compared equal. A typical stochastic continuous policy such as Gaussian used by SAC (Haarnoja et al., 2018) always has ān * On for n 2 1 w.r.t. any behavior policy μ.. In this case 7 no more than just a Bellman operator 8* However, if continuous policy is spe- cially structured to be faction-reproduee enjoy the privilege of using 8n for n > 1 as the target state. Our two-stage ~² is such a case, where each ac- tion o, (ā.,) is accompanied by repeating choice 6, (b,)). Starting from n = 1 with previous action ag. if by. bm. = 0 (both repeated) for all 1 S S then we know that a. = a, = _.. = ãj = 01 = ag- In other words, if two trajectories start with the same (sg, ag), we can compare their discrete (b,) sequences in place of the continuous )a,,) sequences. See Figure 2 for an illustration. Thus for multi-step TD learning we use T to replace Ee in Eq. 3.
0g
83
84
[A)
01
$0
81
$2
52
b2
1
Ø1
02
0g
$3
$4
An illustration of the compare- through operator by exploiting action repeti- tion of I² The upper branch is the trajectory sampled by a rollout policy; the lower one is sampled by the current policy during training. We have ă aj ag due bj by 0. The two trajectories diverge at $2 because ei- ther by = or by 1. For bootstrapping, we use $2 as the target state in this example.
Figure 2:
it will
The compare-through operator is not meant to replace multi-step TD learning with impor- tance correction general scenario, as is only effective for paction-rprroducible policies. Its bootstrapping has hard cutoff (by checking ä. * a,) instead of soft one as in Munos et al. (2016).
Remark
4

---

On the importance of reward normalization Because T computes bootstrapping targets based on dynamic step lengths and rewards are propagated faster with greater lengths, values bootstrapped by greater lengths might become overly optimistic/pessimisti are all positive/negative. This could affect the policy selecting actions according to their Q values. However, this side effect is only temporary and will vanish if all Q values are well learned eventually. In practice, we find it beneficial to normalize the immediate reward by its moving average statistics to roughly maintain zero mean and unit standard deviation (detailed in Appendix I.1).
(e.g., imagine a task with rewards that
4.3 Policy improvement with a closed-form ,3t
It can be shown (Appendix B) that the parameterized state value of '7"h is
VØ"h (s(a Gr
TF
[[L (ll-b))o(sssts a + bQo(s.ā)].
(5)
Intuitively, the actual Q value of each a n² is an interpolation by between the Q values of and â. Our policy improvement objective is then maxş,9 ((s,a-)<D Va"" (s|a Note that each time we sample the previous action at along with state s from the replay buffer. To encourage exploration, following prior works (Mnih et al., 2016; Riedmiller et al., 2018) we augment this objective with a joint entropy H..a- = [n [- log Bo - log n (āls, a "]] Thus the final policy improvement objective is
VØ (sla + aHH,a-
max E 9.9 (s,a (s;a-)~D
max
E
[1 (1- (o-b))QQss, + (Qo(s,)) (log 3o + (og ],
(6)
(s,a M a a
where a is lemperature parameter. Given any (s,a² a)), can derive closed-form solution of the noon-parametric) 3 policy for the innermost expectation 3 as
(s)
Qo(s,a) a) + exp
Q0(s,a (0(ls0))
B exp
exp
a
a
Then applying the re-parameterization trick à (o(e,8,a)), estimated actor gradient is
N(O, 1), one can show that the
8 (Q)
Olog To(ools,
AΦ
OΦ
dΦ 0 Olog (alas Oā
[Q0(s,)) Oā
Öfe
Ölog (alss
B
do
This gradient has a very similar form with SAC's (Haarnoja al., 2018), except that here scaling factor $7. We refer the reader to full derivation of the actor gradient in Appendix D.
8Q has
Z6
Remark 8 This closed-form solution is only possible after à is sampled. It's essentially comparing the Q values between a and à. This side-by-side comparison is absent in previous closed-loop repeating methods like PIC (Chen et al., 2021) and H-MPO (Neunert et al., 2020).
Remark on multi-step actor gradient According to Figure I, the newly generated action à; at the current step i, if repeated as future 11415.... ag (t' > 1), will also influence the maximization of future values: V1," (#++114441).... V0"" (s,)),)) As a result, in principle à, has a multi-step gradient. To exactly compute this multi-step gradient, a fresh rollout of the current ta via interacting with the environment is necessary. For reasons detailed in Appendix E, we end up truncating this full gradient to the first step, by letting Eq. 6 sample (s, am) pairs at non-consecutive time steps from the replay buffer. This one-step truncation is also (implicitly) adopted by H-MPO (Neunert et al., 2020) for their action repetition application case. Interestingly, the truncation results in a simple implementation of actor gradient of TAAC similar to SAC's.
Automatically tuned temperatures Given two entropy targets H' and H", we learn temperatures a and a from the objective
Iog(a')(- - H²)) + log(a")) log mo (ā]s, -7") |
min log(a'). ). log(a") (a.a j~D, dogg-
(8)
5

---

(lismmgg
SAC Neep
2017) (Riederkappal 2011)
SAC-Krp
SAC-EZ (Dahney 2001)
-lybili (Neument 2000
TAAC
Multi--tpp TD Chaeal-hup
Leumable durtion
Table 1: A summary of the 8 major comparison methods in our experiments.
by adjusting log(a) instead of a as in SAC (Haarnoja et al., 2018). We learn temperatures for ne and B separately, to enable finer control of the two policies and their entropy terms, similar to the separate policy constraints (Abdolmaleki et al., 2018; Neunert et al., 2020). Appendix shows how several equations slightly change if two temperatures are used (ox is replaced by a' or a").
Experiments
5.1 Tasks
To test if the proposed algorithm is robust and can be readily applied to many tasks, we perform experiments over 14 continuous control tasks under different scenarios:
a) SimpleControl: Three control tasks (Brockman et al., 2016) with small action and observation and InvertedDouble Penduuuuun b) Locomotion: Four locomotion tasks (Brockman et al., 2016) that feature complex physics and action spaces: Hopper, Ant, Walker2d, and HalsCheetah;
spaces: MountainCarContimenn
Terrain: Two locomotion tasks that require adapting to randomly generated terrains: Bipedal- Walker and FapedalWalkeHarddcore
d) Manipulation: Four Fetch (Plappert et al., 2018) tasks with sparse rewards and hard exploration (reward given only upon success): FetchReach, FetchPush, FetchSlide, and FetchPickAnddllce;; e) Driving: One CARLA autonomous-drivin high-dimensional multi-modal sensor inputs (camera, radar, IMU, collision, GPS, etc.): TownO1. The goal is to reach a destination starting from randomly spawned location in a small realistic town, while avoiding collisions and red light violations.
task (Dosovitskiy et al., 2017) that has complex
Among the 5 categories, (d) and (e) might benefit greatly from temporal abstraction because of hard exploration or the problem structure (e.g., driving naturally involves repeated actions). Categories (a)-(c) appear unrelated with temporal abstraction, but we test if seemingly unrelated tasks can also benefit from it. By comparing TAAC against different methods across vastly different tasks, we hope to demonstrate its generality, because adaptation to this kind of task variety requires few assumptions about the task structure and inputs/outputs. For more task details, we refer the reader Appendix I.
5.2 Comparison methods
While there exist many off-policy hierarchical RL methods that model temporal abstraction, for example Nachum et al. (2018); Riedmiller et al. (2018); Levy et al. (2019); Li et al. (2020); Zhang al. (2021), we did not find them readily scalable to our entire list of tasks (especially to high dimensional input space like CARLA), without considerable efforts of adaptation. Thus our primary focus is to compare TAAC with baselines of different formulations of action repetition: vanilla SAC (Haarnoja et al., 2018), SAC-Nrep. SAC-Krep (Sharma et al., 2017; Biedenkapp et al., 2021), SAC-EZ (Dabney et al., 2021), and SAC-Hybrid (Neunert al., 2020). Although originally some baselines have their own RL algorithm backbones, in this experiment we choose SAC as the common backbone because: 1) SAC is state-of-the-att among
actor-critic algorithms, 2) a common backbone facilitates reusing the same set of core hyperparameters for fairer comparison, and 3) by reducing experimental variables, it gives us a better focus on the design choices of action repetition instead of other orthogonal algorithmic components.
Let N be a parameter controlling the maximal number of action repeating steps. SAC-Nrep simply repeats every action N times. SAC-Krep, inspired by FiGAR (Sharma et al., 2017) and Tem- PoRL (Biedenkapp et al., 2021), upgrades an action a to pair (a, K), indicating that the agent will repeat action for the next K steps (1 < K < N) without being interrupted until finishing. To implement Sre-Krep, following Delalleau et al. (2020) we extended the original SAC algorithm to support mixture of continuous and discrete actions (Appendix G). Note that SAC-Krep's open-loop

---

control is in contrast to TAAC's closed-loop control. SAC-EZ incorporates the temporally extended e-greedy exploration (Dabney et al., 2021) into SAC. During rollout, the agent decides to explore, then the action is uniformly sampled and the duration for repeating that action is sampled from trun- cated zeta distribution zeta(n) X n PP S n S N. This fixed duration model encourages persistent exploration depending on the value of the hyperparameter μ.. The training step of SAC-EZ is the same with SAC. SAC-Hybrid shares a similar flavor of H-MPO (Neunert et al., 2020) for closed-loop action repetition. It defines hybrid policy to output the continuous action and binary switching action in parallel, assuming their conditional independence given the state. This independence between hybrid actions and how the Q values TAAC. We also apply Retrace (Munos et al.. 2016) to its policy evaluation with -sstep TD as done by H-MPO. We refer the reader to the algorithm details of EAC-Hybrid in Appendix H.
computed in SAC-Hybrid are the biggest differences with
In order to analyze the benefit of persistent exploration independent of that of multi-step TD learning, we also compare two additional methods. SAC-Ntd is a variant of SAC where a Q value is boot- strapped by an N-step value target with Retrace (Munos et al., 2016) to correct for off-policyness. For ablating TAAC, we evaluate TAAC-Itd ltd that employs a typical Bellman operator 8=" for one-step bootstrapping. Thus we have 8 methods in total for comparison in each task. See Table 1 for summary and Appendix for method details.
In our experiments, we set the repeating hyperparameter N to 3 on SimpleControl, Locomotion and Manipulation, and to 5 on Terrain and Driving. Here the consideration of N value mainly for open-loop methods like SAC-Nrep and SAC-Krep because they will yield poor performance with large values of N. TAAC is not sensitive to N for policy evaluation, as shown in Section 5.5.
5.3 Evaluation protocol
To measure the final model quality, we define score as the episodic return E r(st, at St+1) of evaluating (by tak- ing the approximate mode of a parameterized continuous policy: see Appendix C for details) a method for a task episode, where T is a pre-defined time limit or when the episode terminates early. Different tasks, even within the same group, can have vastly different reward scales (e.g.,
1.0
0.8
Nomalized Score
0.6
0.4
SAC
SAC-Mtd
SAC Nrap
0.2
SAC Krep
tens vs. thousands). So it is impractical to directly average their scores. It is not uncommon in prior works (Hessel al., 2018)
SAC EZ
0.0
Hybrid
TAAC-Itc (ours)
a performance range for each task sepa- rately, and normalize the score of that task to roughly [0, 1] before averaging scores of a method over multiple tasks. Similarly, to facilitate score aggregation across tasks, we adopt the metric of normalized score (и-sccre). For each task, we obtain the score (averaged over 100 episodes) of a random policy and denote by Zo. We also evaluate the best method on that task and obtain its average score Z1. Given a score Z, its normalized value calculated as 2,-2o With this definition, the n-score of each task category (a)-(e) can be computed as the averaged n-score across tasks within that category. Additionally, to measure training convergence speed, we approximate n-AUC (area under the n-score curve normalized by x value range) by averaging n-scores a n-score curve throughout the training. A higher n A-AUC value indicates faster convergence speed. n-AUC is a secondary metric to look at when two methods have similar final n-scores. Finally, by averaging n-score and A-AUC values over multiple tasks, we emphasize the robustness of an RL method.
TAAC tours)
-0.
0.0
0.5 Training Progress
1.0
Figure 3: Training curves (n-score vs. training progress) of the 8 compari- son methods in one plot. Each curve is mean of a method's 's n-score curves over all the 14 tasks, where the method is run with 3 random seeds on each task. See Figure 8 and Figure (Appendix K) for the complete set of individual training curves.
Z-20.
Given a task, we train each method for the same number of environment frames. Crucially, for fair comparisons we also make each method train 1) for the same number of gradient steps, 2) with the same mini-batch size and learning rate, 3) using roughly the same number of weights, and 4) with common set of hyperparameters (tuned with vanilla SAC) for the SAC backbone. More details of the experimental settings are described in Appendix J.
7

---

SAC
SAC-Nid
NSeC-Nrep
SrC-Krep
SAC-EZ
HSAC-Hybiid
TAAC-Id
TAAC
SimpleControl
0.64]0.03
0.54[0.07
0.54
0.05
0.65[0.03
0.88[0.14
0.83(0.33
0.
Locomotion
0.89
0.88(0.11
0.70
0.87
0.92
0.63
0.90 0.88
Final n-score
Terrain
0.35
0.48
0.54
0.48
0.15
0.79
0.00 0.06
(model quality)
Manipulation
0.60
0.75
0.07
0.05
0.68(0.07
0.49[0.07
0.09
0.09
Driving
0.85
0.84
0.03
0.95
0.71 0003
0.04
0.42
0.97 0.04
All
0.68
0.71
0.78
0.77
0.71
0.69 0.08
0.92
0. (0.03
SimpleControl
0.15
0.41[0.01
0.51
0.28
0.45(0.02
0.62[0.08
0.60
0. 0.120.01
Locomotion
0.69]
0.64
0.55
0.59
0.64
0.50(0.06
0.72(0.02
0. 0.74
R-AUC
Terrain
0.17
0.19
0.08
0.23
0.21 0.00
0.50
0.59 0.50
(convergence
Manipulation
0.044
0.50
0.066
0.01
0.49
0.38
0.71
0.07
speed)
Driving
0.38 0.002
0.41 0.04
0.02
0.03
0.32
0.02
0.60
0.00
All
0.07 0..00.
0.05
0.50
0.47[0.04
0,50 0.04
0.65
0. (.003
Table 2: n-score and n AUC results. Margins in brackets are computed by averaging the standard
deviations (across 3 random seeds) of individual tasks. The last two shaded columns are our methods.
5.4 Results and observations
The n-AUC and final n-score values are shown in Table 2, and the training curves of n-score are shown in Figure 3. First of all, we conclude that the tasks are diverse enough reduce the evaluation variance. The averaged standard deviations are small for most methods. Thus the comparison results are not coincidences and are likely to generalize to other scenarios. Overall, TAAC largely outperforms the 6 baselines regarding both final n-score (0.96[0.03) vs. second-best 0.78(0.05) and n-AUC (0.72[0.03] vs. second-best 0.55 0.044), with relatively small standard deviations. Note that the n-AUC gap naturally tends to be smaller than the final n-score gap because the former reflects a convergence trend throughout the training. Moreover, TAAC achieved top performance of n-AUC on each individual task category. Although some baselines achieved best final n-scores by slightly dominating TAAC, their performance not consistent over all task categories. More observations can be made below.
Persistent exploration and the compare-through operator are both crucial. Even with one-step TD, TAAC-Itd's performance (0.92[0.05) and (6.0). already outperforms the baselines. This shows that persistent exploration alone helps much. Furthermore, TAAC is generally better than vs. (0.0|). This shows that efficient
TAAC-Itd Itd (0.96[0.03) vs. 0.92[0.05] and policy evaluation by the compare- through operator is also a key component of TAAC.
A proper formulation of closed-loop action repetition is important. The idea of closed-loop action repetition alone is not a magic ingredient, as SAC-Hybrid only has moderate performance among the baselines. Notably, it performs worst on Locomotion and Manipulation. Our analysis revealed that its "act-or-repeat" policy tends to repeat with very high probabilities on Ant, PeetchiicknndPlee and FetchPush even towards the end of training. resulting in very poor n-scores. All these three tasks feature hard exploration. This result suggests that a good formulation of the idea is crucial. The two-stage decision IT Tāā]s, )x (b),, a a=, a) TAAC is clearly more informed than the parallel decisions π (ā]s, a ) (b]s, a a) of SAC-Hybrid. Furthermore, even with a latent "act-or-repeat" action, TAAC manages to maintain the complexity of the Q function ((s, a) as in the original control problem, while SAC-Hybrid has a more complex form Q((s, a ), (á, b)) (Appendix H). Naive action repetition works well. Interestingly, in this particular experiment, SAC-Nrep is the overall top-performing baseline due to its relatively balanced results on all task categories. However, when it fails, the performance could be very bad (Locomotion and Terrain). While its sample efficiency is good, it has a difficulty of approaching the optimal control (final mean n-scores S 0.92 in individual task categories) due to its lack of flexibility in the action repeating duration. This suggests that action repetition greatly helps, but a fixed duration is difficult to pick.
Limitation: action repetition hardly benefits tasks with densely shaped rewardy and frame skipping. We notice that TAAC is no better than SAC and SAC-EZ on Locomotion regarding the final n-score, although it has slight advantages of n-AUC. The locomotion tasks have densely shaped rewards to guide policy search. Thus action repetition hardly helps locomotion exploration, especially when the 4 tasks already have built-in frameskips (4 frames for Hopper and Walker2d; 5 frames for Ant and HalfCheetah). We believe that more sophisticated temporal abstraction (e.g., skills) is needed to improve the performance in this case.
5.5 Off-policyness experiments
To verify that our compare-through operator is not affected by off-policyness in policy evaluation, we compare TAAC to variant TAAC-Ntd which always bootstraps a Q value with an N N-step target,
8

---

Final newcore (model qualitt)
R-AUC (convergenos sprcod)
M
All
5
M
NA 10)
0.38 1o 0. 1.700111
1.
1
0.400.11
0.440
0.39
0.550.08
LA 101
1.000.02
0.94
0.94
03300.05
0.73
0.6700000
0.3910.02
0.73
0.600.000
0.080..3
Hyb-d
0.
0.630.8
100001
0.82
000
0.500..4
SAC-HHyyra-Coompli 0.890
0,6
0.775
0.08
0.90/00
0.74...111
0.18 0.
0.47
00. 0.00
0.00
0 0.50
Table 3: Off- ffffpolicyness experiments results. Error margins inside brackets are computed by averaging
the standard deviations (across 3 random seeds) of individual tasks in a category. S: SimpleControl;
L: Locomotion; T: Terrain; M: Manipulation; D: Driving.
regardless of (3's outputs and without importance correction'. We choose large trajectory length N = 10 to amplify the effect of off-policyness. Table 3 shows that -tstep TD without impor- tance correction significantly degrades the performance (0. (0.79[0.12) vs. (.93(..05) and 0.55[0.08] vs. 0.68[0.03). In contrast, the compare-through operator well addresses this issue for TAAC.
Furthermore, implement variant of SAC-Hybrid by replacing the Retrace operator with compare-througee operator, to see if the result difference between TAAC and SAC-Hybrid is mainly due to different ways of handling off- off-policyness. Table 3 shows that SAC-Hybrid H-bbridd---ooTTTh performs similarly to SAC-Hybrid (0.74(0.1) 11] vs. 0.69(0.08) and 0.50(0.06) vs. (.00() suggesting that it is indeed the formulation of SAC-Hybrid that creates its performance gap with TAAC.
5.6 Policy behavior visualization and analysis
In this section, we mainly answer two questions:
1) How is the exploration behavior of TAAC com- pared to that of SAC, a "flat" RL algorithm?
1.2.
1 00
2) What is the action repetition behavior of TAAC in a trained control task?
-02
-L375
pel
TMC
5.6.1 Exploration behavior
Figure 4: KDE plots of 100K state vectors visited by TAAC and SAC with their random- ized policies. The side plots on top and right represent ID marginal density functions. Left: MountainCarContinnan
TAAC introduces persistent exploration along pre- vious actions, having a better chance of escaping the local neighborhood around a state when acting randomly. Thus TAAC's exploration should yield a better state space coverage than SAC's does, as- suming other identical conditions. To verify this, visualize the state space coverage by SAC and TAAC during their initial exploration phases.
x is the car's posi- tion and is the car's 's velocity; Right: Bipedal- Walker: ry represent the top-2 principal com- ponents of the walker's hull.
Specifically, we select two tasks MountainCarContinun and Bipeda/Walker for this purpose. For either TAAC or SAC, we play a random version of its policy on either task for 50K environment frames, to simulate the initial exploration phase where the model parameters have not yet been altered by training. During this period, we record all the 50K state vectors for analysis. For CociutaainCrCComnnnuuu each state vector 2D, representing the car's "x-position" and "x-velocity" on the ID track. For Bipeda/Walker, each state vector is 24D, where the first 4D sub-vector indicates the statistics of the walker's ' hull: "angle", "angular velocity"". "x-velocity", and "y-velocity". For visualization, we first extract this 4D sub-vector and form combined dataset of 100K vectors from both TAAC and SAC. Then we apply PCA (Jolliffe, 1986) to this dataset and project each 4D vector down to 2D. After this, MountainCar CarContimuus and BipedalWalker Walker in Figure 4. We see that on both tasks, a random policy of TAAC is able to cover diverse states compared to SAC. This suggests that in general, TAAC is better at exploration compared to a "flat" RL method, thanks to its ability of persistent exploration.
able to draw KDE (kernel density estimate) plots for both
5.6.2 Action repetition behavior
In theory, to achieve optimal continuous control, the best policy should always sample a new action at every step and avoid action repetition at all. Thus one might assume that TAAC's second-stage policy shifts from frequently repeating actions in the beginning of training, to not repeating actions at all
We cannot apply Retrace TAAC-Ntd since the probability density of nt computationally intractable.

---

Tasks
MCC IJLC IDP HOP ANT WAL. HC BW
BWH
FR
FP
FS
FPP
TOW
Action repetition percentage 89% 74% 26% 37% 13% 26% 1% 25% 39%
9%
55%
54% 49% 55%
Table 4: Action repetition percentages by evaluating a trained TAAC model for 100 episodes on each of the 14 tasks. Refer to Section 5.1 for the full task names.
7+6
E+IT
(42
144
245
5
5 5
5
3
8(10)
(b)
(c)
Figure 5: Example frames (cropped) from up-performing evaluation episodes of TAAC. Each column contains consecutive frame(s) generated by the same action, where ": (n1) denotes n similar frames omitted due to space limit. (a): In CoanntainnarCootinnen
the car first backs up build gravitational potential and rushes down; (b): In Bipedal WalkerHardcore the bipedal walker jumps one step down;
(c): In PickAndiaee the robot arm approaches the object and lifts it to a goal location.
towards the end of training, in order to optimize the environment reward. However, some examples in Figure suggest that a significant repetition frequency still exists in TAAC's top-performing evaluation episodes. Note that because we take the mode of the switching policy during evaluation (Appendix C), this suggests that for those repeating steps, we have 3 > 34 by the trained model. In fact, if we evaluate 100 episodes for each of the three tasks in Figure 5 and compute the action repetition percentage (repeating steps divided by total steps), the percentages are surprisingly 89%, 39%, and 49%, even though the agents are doing extremely well! A complete list of action repetition percentages for all 14 tasks can be found in Table 4. Generally, TAAC is able to adjust the repetition frequency according to tasks, resulting in variety of frequencies across the tasks. More importantly, we inspect the repeated actions, they are often not even close to action boundaries {Amins amoz). ruling out the possibility of the policy being forced to repeat due action clipping/squashing.
We believe that there are two reasons for this surprising observation. First, with many factors such as function approximation, noise in the estimated gradient, and stochasticity of the environment dynamies, is hardly possible for a model to reach the theoretical upper bound of an RL. objective. Thus for new action and the previous action, if their estimated Q values are very similar, then TAAC might choose to repeat with a certain chance. For example in Figure (c), while the robot arm is lifting the object towards the goal in the air, it can just repeat the same lifting action for 3 steps (at t + 5), without losing the optimal return (up to some estimation error). Similar things happen to the bipedal walker (b) when it is jumping in the air (9 steps repeated at t 7), and to the mountain car (a) when it is rushing down the hill (11 steps repeated at t + 26). Second, as a function approximator, the policy network T have limited capacity, and it is not able represent the optimal policy every state in a continuous space. For non non-critical states that can be handled by repeating previous actions, TAAC might learn offload the decision making of ne onto 3, and save n representational power for critical states. For example, the robot arm in Figure 5 (c) invokes a fine-grainet control by n when it's grasping the object from the table, while later does not invoke nı for lifting it.
6 Conclusion
We have proposed TAAC, simple but effective off-policy RL algorithm that middle ground between "flat" and hierarchical RI. TAAC incorporates closed-loop temporal abstraction into actor- critic by adding a second-stage policy that chooses between the previous action and new action output by actor. TAAC yielded strong empirical results variety of continuous control tasks, outperforming prior works that also model action repetition. The evaluation and visualization revealed the success factors of TAAC: persistent exploration and a compare- step TD backup. We believe that our work has provided valuable insights into modeling temporal abstraction and action hierarchies for solving complex RL tasks in the future.
Q operator for multi-
Societal impact This paper proposes general algorithm to improve the efficiency of RL for continuous control. The algorithm can be applied to many robotics scenarios in the real world. How to address the potentially unsafe behaviors and risks of the deployed algorithm caused to the surroundings in real-world scenarios remains an open problem and requires much consideration.
10

---

Acknowledgements
The authors would like thank Jerry Bai and Le Zhao for helpful discussions on this project, and the Horizon Al platform team for infrastructure support.
References
Abdolmaleki, A., Springenberg, J. T., Degrave, J., Bohez, S., Tassa, Y., Belov, D., Heess, N., and Riedmiller, M. A. (2018). Relative entropy regularized policy iteration. arXiv.
Amin, S., Gomrokchi, M., Aboutalebi, H., Satija, H., and Precup. D. (2021). Locally persistent exploration in continuous control tasks with sparse rewards. In ICML..
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, (2017). Hindsight experience replay. In NeurIPS.
Bacon, P.-L.,, Harb, J., and Precup, D. (2017). The option-critic architecture. In AAAI.
Biedenkapp, A., Rajan, R., Hutter, F., and Lindauer, M. (2021). Temport: Learning when to act. In
ICML.
Brockman, G., Cheung, Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). Openai gym. CoRR, abs/1606.01544
Chen, C., Tang, H., Hao, J., Liu, W., and Meng, Z. (2021). Addressing action oscillations through learning policy inertia. AAAI.
Dabney, W., Ostrovski, G., and Barreto, A. (2021). Temporally-extendde epsilon-greeds exploration. In ICLR.
Dayan, and Hinton, G. E. (1993). Feudal reinforcement learning. In NeurlPS, pages 271-278.
Delalleau, O., Peter, M., Alonso, E., and Logut, A. (2020). Discrete and continuous action rep- resentation for practical RL in video games. In AAAI workshop on reinforcement learning in games.
Dietterich, T. G. (1998). The maxq method for hierarchical reinforcement learning. ICML, pages 118-126.
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., and Koltun, V. (201)), CARLA: An open urban driving simulator. In CoRL, pages 1-16.
Duan, Y., Chen, X., Houthooft, R., Schulman, J., and Abbeel, (2016). Benchmarking deep reinforcement learning for continuous control. In ICML.
Fujimoto, S., van Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods. In ICML, pages 1587-1596.
Grigsby, J., Yoo, J. Y., and Qi, Y. (2021). Towards automatic actor-critic solutions to continuous control. arXiv.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., and Levine, S. (2018). Soft actor-critic algorithms and applications. arXiv, abs/1812.05505.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In CVPR, pages 770-778.
Hessel, M., Modayil, van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan. D., Piot, B.. Azar, M. G., and Silver, D. (2018). Rainbow: Combining improvements in deep reinforcement learning. AAAL
Jolliffe, (1986). Principal component analysis and factor analysis. Principal Component Analysis. Springer New York.

---

Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. In ICLR.
Lakshminarayanan, A. S., Sharma, S., and Ravindran, B. (2017). Dynamic action repetition for deep reinforcement learning. In AAAI.
Lee, J., Lee, B.-J., and Kim, K.-E. (2020a). Reinforcement learning for control with multiple frequencies. In NeurIPS.
Lee, Y., Yang, J., and Lim, J. (2020b). Learning coordinate manipulation skills via skill behavior diversification. In ICLR.
Levy, A., Jr., R. P., and Saenko, K. (2019). Learning multi-level hierarchies with hindsight. In ICLR.
Li, C., Florensa, C., Clavera, I,, and Abbeel, (2020). Sub-policy adaptation for hierarchical reinforcement learning. In ICLR.
Lillicrap. T. Hunt, J., Pritzel, A., Heess, N., Erez, T., Tassa, Y. Silver, D., and Wierstra, D. (2016). Continuous control with deep reinforcement learning. In ICLR.
Metelli, A. M., Mazzolini, F., Bisi, L., Sabbioni, L., and Restelli, M. (2020). Control frequency adaptation via action persistence in batch reinforcement learning. In ICML.
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap. T. P., Harley, T., Silver, D., and Kavukcuoglu, K. (2016). Asynchronous methods for deep reinforcement learning. In ICML..
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540):529-53)
Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. (2016). Safe and efficient off off-pollcy reinforcement learning. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R., editors, NeurlPS.
Nachum, O., Gu, S., Lee, H., and Levine, S. (2018). Data-efficient hicrarchical reinforcement learning. In NeurIPS.
Neunert, M., Abdolmaleki, A., Wulfmeier, M., Lampe, T., Springenberg, J., Hafner, R., Romano, F., Buchli, J., Heess, N., and Riedmiller, M. (2020). Continuous-discrete reinforcement learning for hybrid control in robotics. CoRL.
Parr, R. and Russell, S. (1998). Reinforcement learning with hierarchies of machines. In NeurlPS, pages 1043-1049.
Plappert, M., Andrychowicz, M., Ray, A., McGrew, B., Baker, B., Powell, G., Schneider, J., Tobin, J., Chociej. M., Welinder, P., Kumar, V., and Zaremba, W. (2018). Multi-goal reinforcement lcarning: Challenging robotics environments and request for research. arXiv.
Precup, D. (2000). Temporal Abstraction in Reinforcement Learning. PhD thesis, University of Massachusetts Amherst.
Riedmiller, M., Hafner, R., Lampe, T., Neunert, M., Degrave, J., van de Wiele, T., Mnih, V., Heess, N., and Springenherg, J. T. (2. Learning by playing solving sparse reward tasks from scratch. In ICML.
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., and Abbeel, P. (2015). Trust region policy optimization. In ICML.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. In arXiv.
Sharma, S., Srinivas, A., and Ravindran, B. (2017). Learning to repeat: Fine grained action repetition for deep reinforcement learning. In ICLR.
12

---

Sutton, R. S., Precup, D., and Singh, S. (1999). Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence. 112(1):181 211.
Todorov, E., Erez, T., and Tassa, Y. (2012). Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026-5033.
Vezhnevets, A. S., Osindero, S., Schaul, T., Heess, N., Jaderberg. M., Silver, D., and Kavukcuoglu, K. (2017). Feudal networks for hierarchical reinforcement learning. JCML, page 3540-3549.
Zhang, J., Yu, H., and Xu, W. (2021). Hierarchical reinforcement learning by discovering intrinsic options. In ICLR.
Zhang, and Van Hoof, H. (2021). Deep coherent exploration for continuous control. ICML.
13


Reviews:
[{"review_id": "p_RmHHJegWH", "reviewer": "Reviewer_M15z", "summary": "The authors introduce TAAC, which stands for Temporally Abstract Actor Critic, an extension of the actor-critic framework, in the continuous control setting, to enable the policy to choose between selecting a new action or repeating the previous one. TAAC acts as a middle ground between hierarchical RL and “flat” RL, allowing the policy to perform temporal abstraction while not having to deal with the computational or complex formulation burden of hierarchical RL methods. TAAC requires only to change slightly the architecture of the actor-critic agent, i.e. replacing the policy had with a two-stage policy head and adding the previous action as input, and to change the policy evaluation and improvement update. In this setting, using off-policy correction for multi-step TD learning such as RETRACE is not applicable. Thus, the authors also introduce a compare-through operator to stabilize learning in this setting. The authors provide novel expressions of the policy evaluation and policy improvements updates, in this setting, as well as convergence proofs. Then the authors compare TAAC to 6 baselines and 1 ablation on a set of 14 benchmarks, grouped in 5 categories. They show that on average TAAC outperforms its competitors and especially shows a significant improvement on benchmarks that exhibit exploration difficulties and require planning over long horizons. Finally, the authors propose an in-depth study of the properties of their agents and notably show that it better cover the state space compared to SAC and also exploits often the action repetition leading to high performance even in settings where we would not expect it to be useful.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I warmly thank the authors for their work.\n\nClarity:\n- The paper is clear and well-written. I found it easy to read and to follow.\n\nSoundness of the claims, significance and novelty of the contribution, and relevance to the NeurIPS community:\n- The paper is well motivated and clearly positioned with respect to the literature\n- The method section is very clear and the derivations are well presented and motivated\n- The method seems fairly simple to implement and does not bring any additional computational cost, yet it seems to improve significantly the performance in many situations\n- The experimental study is sound. The authors considered a wide range of benchmarks and thoroughly compared the method to baselines and ablations.\n- The complementary analysis of the state space coverage and the frequency of re-used actions also make a lot of sense.\n\nLimitations of this work:\n- Line 88: the transition between the argument that computing $\\rho^{\\pi}(s)$ is usually intractable and the objective derived in the Equation is too fast and lacks maybe an argument. Also, $\\mathcal{D}$ (which I assume is a replay buffer) is used in the Equation but not introduced.\n- Line 186: The authors wrote: “to test if the algorithm generalizes”. I find this sentence misleading as when reading that I expect to see experiments about generalization in RL (generalization between environments, over initial states, etc …). I would recommend to change this sentence and maybe rather say something like: “to assess the performance of the algorithm” \n- Line 248: I don’t get precisely what taking the argmax of the policy means. In the discrete action setting, I would understand it as choosing the action with higher probability, however, we are in the continuous action setting here. It would be great if the authors can clarify this point.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper does not add limitations or potential negative societal impact to existing reinforcement learning methods. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Ne-W8fJeDJU", "reviewer": "Reviewer_Vncb", "summary": "The paper proposes to add a component to the standard RL actor-critic architecture, that learns when to repeat (or not) the last action executed by the agent. This component is a simple binary classifier, conditioned on $s_t$, $a^-$ (the previous action) and $\\hat{a}$ (what the actor wants to do now), that chooses between $a^-$ and $\\hat{a}$. The authors theoretically motivate their algorithm, compare it to a wide range of related work, and provide extended experimental results that show that the proposed algorithm outperforms the Soft Actor-Critic, and its recent variations.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written, and the idea it proposes is simple, in a good way. The experimental results are strong, and extremely well designed (many environments of many times, extra details in the appendix, well-motivated hyper-parameters). Overall, the contribution proposed in this paper has value, and, to me, clearly its place at NeurIPS.\n\nPossible improvements to the paper could be as follows:\n\n- In Section 4, two contributions are introduced at once: the introduction of the $\\beta$ binary classifier, and the addition of $a^-$ as input to the actor. I may have missed it, but I did not see an ablation study that shows how the algorithm performs with only the $\\beta$ classifier (leaving the actor as is), or only conditioning the actor on $a^-$ (without the binary classifier). I have the feeling that conditioning the actor on the previous action may help the agent with small amounts of partial observability (of full observability it has difficulties making sense of), by allowing the actor to choose an action at time-step $t$ that helps it make a decision at time $t+1$. I would therefore be very curious to empirically see the effect of that part of the contribution.\n- The paragraph after Theorem 1 is a bit difficult to understand. I understand that it tries to explain how finding which \"future action\" is different from the current action is possible, even when the action is continuous, but I did not understand everything. To me, it seems that storing the actual choice of $\\beta$ in the experience buffer, every time-step, allows a simple implementation of Equation 4 by just looking, in the buffer, at the first \"future action t+n\" for which $\\beta(t+n-1) =$ choose new action. Is this what the paragraph tries to convey?\n\nAuthor response: the authors give a motivation for why the actor always conditions on $a^-$, and say that they will do experiments without this conditioning, to see what impact it has. They will also make the paragraph after Theorem 1 clearer. Overall, this paper presents valuable work, and I recommend accepting it.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper makes the effort of having a small paragraph about societal impact, which is nice, especially since there is not much to say given that this is a theoretical paper, that positively impacts the sample-efficiency of RL, but has no real-world problem in mind. I don't think that a larger discussion would be necessary.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "K5gVNSm5Suk", "reviewer": "Reviewer_SDFG", "summary": "\nThis paper studies a single option: repetitive actions that can be selected via a two-layer architecture. First layer is a policy selected conditioned on previous action; second layer is a binary decision classifier to select the action by the 1st layer or the actual output at the current step. It is a very simple idea, with some seemingly flaws (see below), but perform well in a wide range of experiments. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nCompare-through operator: This definition’s explanation is bit confusing. It looks this equation loops from the beginning time step to find all the subsequence of actions that match the target policy. However, line 125 doesn’t tell much except the operation notationwise. This looks the n will be usually small. Otherwise, your behaviour policy almost very close to the target policy, which becomes on-policy learning (easy). \n\nHere you have a separate parameter for the target, apparently influenced by target network. Then I don’t get your Th. 1: what features does it deal with? If linear or tabular, why would you need the target “network parameter”? If it’s neural networks, then why Th. 1 is true? This is not a stable operator as far as 20 years of RL tells. \n\nExploration is an important aspect that this method may fail to address. Suppose one starts with a poor action. Then because of the binary decision on repeating it or not, there will be significant time repeating this single action in a long action sequence. Such method will only work with navigation in regular (rectangular) domains like Maze or Atari games. Regularizing with a joint entropy as in Section 4.3 may help, but I’m not convinced this is a good method at its heart for solving RL problems especially exploration is hard as we know. \n\nExperiments compared with 6 baselines on 5 kinds of control tasks (all continuous action). \nThe “surprising” fact found in the experiment is the large portion of repeating actions in tasks. It is interesting to find TAAC is able be top performers; although one can argue that these tasks may be just suitable for repeated actions. \n\nI think the empirical performance outweighs the seemingly inefficient exploration aspect, which is exactly the value  of this paper — a simple strategy with arguably inefficient exploration appearance but performing well in many tasks.  \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "0.8"}, "extra_text": {}}, {"review_id": "F4GIqFXPyjv", "reviewer": "Reviewer_VP6v", "summary": "This work proposes a temporally abstract actor-critic (TAAC). They do so by incorporating a second-stage binary policy which chooses between the previous action and a new action - this act or repeat strategy hinges on the actually sampled action as opposed to the expected behavior. The key contributions are 1) a new compare-through Q operator for multi-step TD backup for policy evaluation and 2) computing the actor gradient by multiplying a scaling factor to the change in Q function wrt to the actions for policy improvement. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**:\nThis work proposes a simple closed-loop off-policy RL algorithm TAAC, wherein a second stage chooses to act or repeat. The method shows how we can exploit the structure to maximize the PE-PI processes. The work proposed has the following strengths 1) The method is suitable for hybrid continuous-discrete action space including continuous control, 2) Enables persistent exploration and supports multi-step TD backups, and 3) The empirical results are strong in that the work is primarily empirical and shown to do well in a wide variety of tasks.  \n\nOne key feature of the method which can be very useful is that it serves as a middle ground between one-level RL and HRL. The combination of ideas pitched allow this work to scale better to most baselines compared. The multi-step Q operator called compare-through is shown to facilitate efficient PE which is also shown to converge.  \n\n\n**Weaknesses**:   \n\n* A primary concern is how the proposed method deviates a lot from existing HRL methods which also consider action-repeats? Perhaps the key added value is the extension to continuous control for hybrid action space. It might be useful to elaborate on this in the discussion/related work. I would encourage you to expand the related work section to cover more HRL literature and discuss where the paper is positioned in the context of broader HRL methods. \n* Arguably most HRL methods allow for the switching between the flat (one-step policies) and abstract actions (multi-step policies), especially the intra-option learning methods, which is the primary claim of this work. So what have we learned here which allows us to scale better to the list of tasks studied here? \n* What happens when we don’t specifically want action-reproducible policies? Currently a lot of the improvements seem to stem from the nature of tasks where action-repeat might be beneficial.\n* While the n-score and n-AUC is better as an average over tasks, TAAC is only marginally better or same as top baselines in most tasks as reported in the Figure 8: The unnormalized reward curves of the 14 tasks. It might be useful to expand on this in the main paper as to why you choose the evaluation metric to be n-score and n-AUC and consider averaging over all tasks as the primary metric.\n\n\n**Empirical Analysis**:   \n\nThe experiments evaluate TAAC in 5 categories of 14 continuous control tasks, covering simple control, locomotion, terrain walking (Brockman et al.,572016), manipulation (Plappert et al., 2018), and self-driving. \n\n* The baselines are chosen fairly and compare a wide range of action-repeats methods. How do these methods compare to non-action repeat baselines? Any insights on this would be great. In particular, I am curious if the action that is being repeated is not encouraging exploration or performance, how does the method overcome such a choice?\n* How many random seeds have been considered? Please mention it while reporting.\n\n\n**Writing and Presentation**:  \n* The paper can be presented better in terms of explaining how this approach differs from existing approaches which consider action repeats. Consider a summary table which can highlight the novelty of the approach as compared to many other HRL off policy approaches as mentioned by the authors in Sec 1. Sec 2, and  Sec. 5.2.\n* There are several insights in the more observations paragraph which is buried in 5.4. The paper might benefit from re-writing earlier parts to give some intuitions on key differences which this method brings to the table. For example 1)  Persistent exploration and the compare-through operator are crucial to the success of this approach. 2) The importance of the formulation of the closed-loop action repetition.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes the authors have discussed societal impact. The limitations of the work are also covered in different sections of the paper - it would be valuable to consolidate them and add them in the conclusion section perhaps. ", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The discussion helped clarifying points that were misunderstood by some reviewers initially. It also helped improving the clarity of the paper. There is a consensus on the scores and, now, the contribution is regarded as significant and simple. The method performs very well, it is simple enough to be implemented easily and it is computationally neutral. 
Decision: Accept (Poster)