TAAC: Temporally Abstract Actor-Critic for Continuous Control
Year: 2021

Abstract:
We present temporally abstract actor-critic (TAAC), a simple but effective offpolicy RL algorithm that incorporates closed-loop temporal abstraction into the actor-critic framework. TAAC adds a second-stage binary policy to choose between the previous action and a new action output by an actor. Crucially, its “act-or-repeat” decision hinges on the actually sampled action instead of the expected behavior of the actor. This post-acting switching scheme let the overall policy make more informed decisions. TAAC has two important features: a) persistent exploration, and b) a new compare-through Q operator for multi-step TD backup, specially tailored to the action repetition scenario. We demonstrate TAAC’s advantages over several strong baselines across 14 continuous control tasks. Our surprising finding reveals that while achieving top performance, TAAC is able to “mine” a significant number of repeated actions with the trained policy even on continuous tasks whose problem structures on the surface seem to repel action repetition. This suggests that aside from encouraging persistent exploration, action repetition can find its place in a good policy behavior. Code is available at https://github.com/hnyu/taac.

heading: 1 Introduction
text: Deep reinforcement learning (RL) has achieved great success in various continuous action domains such as locomotion and manipulation (Schulman et al., 2015; Lillicrap et al., 2016; Duan et al., 2016; Schulman et al., 2017; Fujimoto et al., 2018; Haarnoja et al., 2018). Despite promising empirical results, these widely applicable continuous RL algorithms execute a newly computed action at every step of the finest time scale of a problem. With no decision making at higher levels, they attempt to solve the challenging credit assignment problem over a long horizon. As a result, considerable sample efficiency improvements have yet to be made by them in complex task structures (Riedmiller et al., 2018; Li et al., 2020; Lee et al., 2020b) and extremely sparse reward settings (Andrychowicz et al., 2017; Plappert et al., 2018; Zhang et al., 2021).
On the other hand, it is argued that temporal abstraction (Parr and Russell, 1998; Dietterich, 1998; Sutton et al., 1999; Precup, 2000) is one of the crucial keys to solving control problems with complex structures. Larger steps are taken at higher levels of abstraction while lower-level actions only need to focus on solving isolated subtasks (Dayan and Hinton, 1993; Vezhnevets et al., 2017; Bacon et al., 2017). However, most hierarchical RL (HRL) methods are task specific and nontrivial to adapt. For example, the options framework (Sutton et al., 1999; Precup, 2000; Bacon et al., 2017) requires pre-defining an option space, while the feudal RL framework Vezhnevets et al. (2017); Nachum et al. (2018); Zhang et al. (2021) requires tuning the hyperparameters of dimensionality and domain range of the goal space. In practice, their final performance usually hinges on these choices.
Perhaps the simplest form of an option or sub-policy would be just repeating an action for a certain number of steps, a straightforward idea that has been widely explored (Lakshminarayanan et al.,
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
2017; Sharma et al., 2017; Dabney et al., 2021; Metelli et al., 2020; Lee et al., 2020a; Biedenkapp et al., 2021). This line of works can be regarded as a middle ground between “flat” RL and HRL. They assume a fixed candidate set of action durations, and repeat actions in an open-loop manner. Open-loop control forces an agent to commit to the same action over a predicted duration with no opportunity of early terminations. It weakens the agent’s ability of handling emergency situations and correcting wrong durations predicted earlier. To address this inflexibility, a handful of prior works (Neunert et al., 2020; Chen et al., 2021) propose to output an “act-or-repeat” binary decision to decide if the action at the previous step should be repeated. Because this “act-or-repeat” decision will be examined at every step depending on the current environment state, this results in closed-loop action repetition.
All these action-repetition methods are well justified by the need of action persistence (Dabney et al., 2021; Amin et al., 2021; Zhang and Van Hoof, 2021; Grigsby et al., 2021) for designing a good exploration strategy, when action diversity should be traded for it properly. This trade-off is important because when reward is sparse or short-term reward is deceptive, action diversity alone only makes the agent wandering around its local neighborhood since any persistent trajectory has an exponentially small probability. In such a case, a sub-optimal solution is likely to be found. In contrast, persistence via action repetition makes the policy explore deeper (while sacrificing action diversity to some extent).
This paper further explores in the direction of closed-loop action repetition, striving to discover a novel algorithm that instantiates this idea better. The key question we ask is, how can we exploit the special structure of closed-loop repetition, so that our algorithm yields better sample efficiency and final performance compared to existing methods? As an answer to this question, we propose temporally abstract actor-critic (TAAC), a simple but effective off-policy RL algorithm that incorporates closedloop action repetition into an actor-critic framework. Generally, we add a second stage that chooses between a candidate action output by an actor and the action from the previous step (Figure 1). Crucially, its “act-or-repeat” decision hinges on the actually sampled individual action instead of the expected behavior of the actor unlike recent works (Neunert et al., 2020; Chen et al., 2021). This post-acting switching scheme let the overall policy make more informed decisions. Moreover,
i) for policy evaluation, we propose a new compare-through Q operator for multi-step TD backup tailored to the action repetition scenario, instead of replying on generic importance correction;
ii) for policy improvement, we compute the actor gradient by multiplying a scaling factor to the ∂Q ∂a term from DDPG (Lillicrap et al., 2016) and SAC (Haarnoja et al., 2018), where the scaling factor is the optimal probability of choosing the actor’s action in the second stage.
TAAC is much easier to train compared to sophisticated HRL methods, while it has two important features compared to “flat” RL algorithms, namely, persistent exploration and native multi-step TD backup support without the need of off-policyness correction.
We evaluate TAAC on 14 continuous control tasks, covering simple control, locomotion, terrain walking (Brockman et al., 2016), manipulation (Plappert et al., 2018), and self-driving (Dosovitskiy et al., 2017). Averaged over these tasks, TAAC largely outperforms 6 strong baselines. Importantly, our results show that it is our concrete instantiation of closed-loop action repetition that is vital to the final performance. The mere idea of repeating actions in a closed-loop manner doesn’t guarantee better results than the compared open-loop methods. Moreover, our surprising finding reveals that while achieving top performance, TAAC is able to “mine” a significant number of repeated actions with the trained policy even on continuous tasks whose problem structures on the surface seem to repel action repetition (Section 5.6.2). This suggests that aside from encouraging persistent exploration, action repetition can find its place in a good policy behavior. This is perhaps due to that the action frequency of a task can be difficult to be set exactly as the minimum value that doesn’t comprise optimal control while leaving no room for temporal abstraction (Grigsby et al., 2021).

heading: 2 Related work
text: Under the category of temporal abstraction via action repetition, there have been various formulations. Dabney et al. (2021) proposes temporally extended -greedy exploration where a duration for repeating actions is sampled from a pre-defined truncated zeta distribution. This strategy only affects the exploration behavior for generating off-policy data but does not change the training objective. Sharma et al. (2017) and Biedenkapp et al. (2021) learn a hybrid action space and treat the discrete
action as a latent variable of action repeating steps, but the introduced temporal abstraction is openloop and lacks flexibility. One recent work close to TAAC is PIC (Chen et al., 2021) which also learns to repeat the last action to address the action oscillation issue within consecutive steps. However, PIC was proposed for discrete control and its extension to continuous control is unclear yet. Also, PIC predicts whether to repeat the last action independent of a newly sampled action, which requires its switching policy to make a decision regarding the core policy’s expected behavior. In an application section, H-MPO (Neunert et al., 2020) explored how continuous control can benefit from a meta binary action that modifies the overall system behavior. Again, like PIC their binary decision is made in parallel with a newly sampled action. Different from PIC and H-MPO, TAAC only decides “act-or-repeat” after comparing the previous action with a newly sampled action. Moreover, TAAC employs a new compare-through Q operator to exploit repeated actions for multi-step TD backup, and is trained by a much simpler actor gradient by absorbing the closed-form solution of the binary policy into the continuous action objective to avoid parameterizing a discrete policy unlike H-MPO.
Our experiment design (Section 5.2) has covered most methods that consider action repetition. Table 1 provides a checklist of the differences between TAAC and these methods.

heading: 3 Preliminaries
text: We consider the RL problem as policy search in a Markov Decision Process (MDP). Let s ∈ RM denote a state, where a continuous action a ∈ RN is taken. Let π(a|s) be the action policy, and P(st+1|st, at) the probability of the environment transitioning to st+1 after an action at is taken at st. Upon reaching st+1, the agent receives a scalar reward r(st, at, st+1). The RL objective is to find a policy π∗ that maximizes the expected discounted return: Eπ,P [ ∑∞ t=0 γ
tr(st, at, st+1)], where γ ∈ (0, 1) is a discount factor. We also define Qπ(st, at) = Eπ[ ∑∞ t′=t γ
t′−tr(st′ , at′ , st′+1)] as the discounted return starting from st given that at is taken and then π is followed, and V π(st) = Eat∼π Qπ(st, at) as the discounted return starting from st following π. In an off-policy actor-critic setting with π and Q parameterized by φ and θ, a surrogate objective is usually used (Lillicrap et al., 2016; Haarnoja et al., 2018)
max φ E s∼D
V πφ θ (s) , max
φ E s∼D,a∼πφ Qθ(s, a). (1)
This objective maximizes the expected state value over some state distribution, assuming that 1) s is sampled from a replay buffer D instead of the current policy, and 2) the dependency of the critic Qθ(s, a) on the policy πφ is dropped when computing the gradient of φ. Meanwhile, θ is learned separately via policy evaluation with typical TD backup.
4 Temporally abstract actor-critic
...
...
...
...
To enable temporal abstraction, we decompose the agent’s action decision into two stages (Figure 1): 1) sampling a new candidate action â ∼ πφ(·|s, a−) conditioned on the action a− at the previous time step, and 2) choosing between a− and â as the actual output at the current step. The overall TAAC algorithm is summarized in Algorithm 1 Appendix A.

heading: 4.1 Two-stage policy
text: Formally, let β(b|s, â, a−) be the binary switching policy, where b = 0/1 means choosing a−/â. For simplicity, in the following we will denote βb = β(b|s, â, a−) (always assuming its dependency on s, â, and a−). Then our two-stage policy πta for temporal abstraction is defined as
πta(a|s, a−) , ∫ â πφ(â|s, a−) [ β0δ(a− a−) + β1δ(a− â) ] dâ, (2)
which can be shown to be a proper probability distribution of a. This two-stage policy repeats previous actions through a binary policy β, a decision maker that compares a− and â side by side
given the current state s. Repeatedly favoring b = 0 results in temporal abstraction of executing the same action for multiple steps. Moreover, this control is closed-loop, as it does not commit to a pre-determined time window; instead it can stop repetition whenever necessary. As a special case, when β1 = 1, πta reduces to πφ; when β0 = 1, πta(a|s, a−) = δ(a− a−).

heading: 4.2 Policy evaluation with the compare-through operator
text: The typical one-step TD learning objective for a policy π is
min θ E (s,a,s′)∼D
[Qθ(s, a)− BπQθ̄(s, a)] 2 ,with BπQθ̄(s, a) = r(s, a, s′) + γV πθ̄ (s ′), (3) where Bπ is the Bellman operator, and θ̄ slowly tracks θ to stabilize the learning (Mnih et al., 2015). For multi-step bootstrapping, importance correction is usually needed, for example, Retrace (Munos et al., 2016) relies on a function of π(a|s)µ(a|s) (µ is a behavior policy) to correct for the off-policyness of a trajectory. Unfortunately, our πta makes probability density computation challenging because of the marginalization over â. Thus importance correction methods including Retrace cannot be applied to our case. To address this issue, below we propose a new multi-step Q operator, called compare-through. Then we explain how πta can exploit this operator for efficient policy evaluation.
For a learning policy π, given a trajectory (s0, a0, s1, a1, . . . , sN , aN ) from a behavior policy µ, let ãn denote the actions sampled from π at states sn(n ≥ 1) and ã0 = a0 (we do not sample from π at s0). We define (a point estimate of) the compare-through operator T π as
T πQθ̄(s0, a0) ≈ n−1∑ t=0 γtr(st, at, st+1) + γ nQθ̄(sn, ãn), (4)
where n = min ({n : ãn 6= an} ∪ {N}). Intuitively, given a sampled trajectory of length N from the replay buffer, the compare-through operator takes an expectation, under the current policy at the sampled states (from s1 to sN ), over all the sub-trajectoriess (up to length N ) of actions that match the sampled actions. Note that Eq. 4 is a point estimate of this expectation. A formal definition of T π is described by Eq. 17, and its relation to Retrace (Munos et al., 2016) is shown in Appendix L.
Theorem 1 (Policy evaluation convergence) In a tabular setting, the compare-through operator T π , whose point estimate defined by Eq. 4 (without the parameters θ̄) and expectation form defined by Eq. 17, has a unique fixed point Qπ , where π is the current (target) policy.
For the detailed proof we refer the reader to Appendix L. Although the actual setting considered in this paper are continuous state and action domains with function approximators, Theorem 1 still provides some justification for our compare-through operator.
...
...
Clearly, any discrete policy could exploit the compare-through operator since there can be a nonzero chance of two discrete actions being compared equal. A typical stochastic continuous policy such as Gaussian used by SAC (Haarnoja et al., 2018) always has ãn 6= an for n ≥ 1 w.r.t. any behavior policy µ. In this case T π is no more than just a Bellman operator Bπ. However, if a continuous policy is specially structured to be “action-reproducible”, it will enjoy the privilege of using sn for n > 1 as the target state. Our two-stage πta is such a case, where each action an (ãn) is accompanied by a repeating choice bn (b̃n). Starting from n = 1 with a previous action a0, if b̃m = bm = 0 (both repeated) for all 1 ≤ m ≤ n, then we know that ãn = an = . . . = ã1 = a1 = a0. In other words, if two trajectories start with the same
(s0, a0), we can compare their discrete {bn} sequences in place of the continuous {an} sequences. See Figure 2 for an illustration. Thus for multi-step TD learning we use T πta to replace Bπ in Eq. 3. Remark The compare-through operator is not meant to replace multi-step TD learning with importance correction in a general scenario, as it is only effective for “action-reproducible” policies. Its bootstrapping has a hard cutoff (by checking ãn 6= an) instead of a soft one as in Munos et al. (2016).
On the importance of reward normalization Because T πta computes bootstrapping targets based on dynamic step lengths and rewards are propagated faster with greater lengths, Q values bootstrapped by greater lengths might become overly optimistic/pessimistic (e.g., imagine a task with rewards that are all positive/negative). This could affect the policy selecting actions according to their Q values. However, this side effect is only temporary and will vanish if all Q values are well learned eventually. In practice, we find it beneficial to normalize the immediate reward by its moving average statistics to roughly maintain a zero mean and unit standard deviation (detailed in Appendix I.1).

heading: 4.3 Policy improvement with a closed-form β∗
text: It can be shown (Appendix B) that the parameterized state value of πta is
V π ta θ (s|a−) = E â∼πφ,b∼β
[ (1− b)Qθ(s, a−) + bQθ(s, â) ] , (5)
Intuitively, the actual Q value of each a ∼ πta is an interpolation by β between the Q values of a− and â. Our policy improvement objective is then maxφ,β E(s,a−)∼D V π ta
θ (s|a−). Note that each time we sample the previous action a− along with a state s from the replay buffer. To encourage exploration, following prior works (Mnih et al., 2016; Riedmiller et al., 2018) we augment this objective with a joint entropy Hs,a− = Eπφ(â|s,a−)βb [− log βb − log πφ(â|s, a−)]. Thus the final policy improvement objective is
max φ,β E (s,a−)∼D
[ V π ta θ (s|a−) + αHs,a− ]
= max φ,β E (s,a−)∼D â∼πφ,b∼β
[ (1− b)Qθ(s, a−) + bQθ(s, â)− α (log βb + log πφ(â|s, a−)) ] , (6)
where α is a temperature parameter. Given any (s, a−, â), we can derive a closed-form solution of the (non-parametric) β policy for the innermost expectation b ∼ β as
β∗1 = exp
( Qθ(s, â)
α
)/( exp ( Qθ(s, â)
α
) + exp ( Qθ(s, a −)
α
)) .
Then applying the re-parameterization trick â = fφ( , s, a−), ∼ N (0, I), one can show that the estimated actor gradient is
∆φ , ( β∗1 ∂Qθ(s, fφ)
∂φ − α∂ log πφ(fφ|s, a −) ∂φ ) = ( β∗1 ∂Qθ(s, â)
∂â − α∂ log πφ(â|s, a −) ∂â ) ∂fφ ∂φ − α∂ log πφ(â|s, a −) ∂φ .
(7)
This gradient has a very similar form with SAC’s (Haarnoja et al., 2018), except that here ∂Q∂â has a scaling factor β∗1 . We refer the reader to a full derivation of the actor gradient in Appendix D.
Remark on β∗ This closed-form solution is only possible after â is sampled. It’s essentially comparing the Q values between a− and â. This side-by-side comparison is absent in previous closed-loop repeating methods like PIC (Chen et al., 2021) and H-MPO (Neunert et al., 2020).
Remark on multi-step actor gradient According to Figure 1, the newly generated action ât at the current step t, if repeated as future a−t+1, . . . , a − t′ (t
′ > t), will also influence the maximization of future V values: V π ta
θ (st+1|a − t+1), . . . , V πta θ (st′ |a − t′ ). As a result, in principle ât has a multi-step
gradient. To exactly compute this multi-step gradient, a fresh rollout of the current πta via interacting with the environment is necessary. For reasons detailed in Appendix E, we end up truncating this full gradient to the first step, by letting Eq. 6 sample (s, a−) pairs at non-consecutive time steps from the replay buffer. This one-step truncation is also (implicitly) adopted by H-MPO (Neunert et al., 2020) for their action repetition application case. Interestingly, the truncation results in a simple implementation of actor gradient of TAAC similar to SAC’s.
Automatically tuned temperatures Given two entropy targets H′ and H′′, we learn temperatures α′ and α′′ from the objective
min log(α′), log(α′′)
{ E
(s,a−)∼D,â∼πφ,b∼β∗
[ log(α′)(− log β∗b −H′) + log(α′′)(− log πφ(â|s, a−)−H′′) ]} ,
(8)
by adjusting log(α) instead of α as in SAC (Haarnoja et al., 2018). We learn temperatures for πφ and β separately, to enable a finer control of the two policies and their entropy terms, similar to the separate policy constraints (Abdolmaleki et al., 2018; Neunert et al., 2020). Appendix F shows how several equations slightly change if two temperatures are used (α is replaced by α′ or α′′).

heading: 5 Experiments

heading: 5.1 Tasks
text: To test if the proposed algorithm is robust and can be readily applied to many tasks, we perform experiments over 14 continuous control tasks under different scenarios:
a) SimpleControl: Three control tasks (Brockman et al., 2016) with small action and observation spaces: MountainCarContinuous, LunarLanderContinuous, and InvertedDoublePendulum ; b) Locomotion: Four locomotion tasks (Brockman et al., 2016) that feature complex physics and action spaces: Hopper, Ant, Walker2d, and HalfCheetah; c) Terrain: Two locomotion tasks that require adapting to randomly generated terrains: BipedalWalker and BipedalWalkerHardcore; d) Manipulation: Four Fetch (Plappert et al., 2018) tasks with sparse rewards and hard exploration (reward given only upon success): FetchReach, FetchPush, FetchSlide, and FetchPickAndPlace; e) Driving: One CARLA autonomous-driving task (Dosovitskiy et al., 2017) that has complex high-dimensional multi-modal sensor inputs (camera, radar, IMU, collision, GPS, etc.): Town01. The goal is to reach a destination starting from a randomly spawned location in a small realistic town, while avoiding collisions and red light violations.
Among the 5 categories, (d) and (e) might benefit greatly from temporal abstraction because of hard exploration or the problem structure (e.g., driving naturally involves repeated actions). Categories (a)-(c) appear unrelated with temporal abstraction, but we test if seemingly unrelated tasks can also benefit from it. By comparing TAAC against different methods across vastly different tasks, we hope to demonstrate its generality, because adaptation to this kind of task variety requires few assumptions about the task structure and inputs/outputs. For more task details, we refer the reader to Appendix I.

heading: 5.2 Comparison methods
text: While there exist many off-policy hierarchical RL methods that model temporal abstraction, for example Nachum et al. (2018); Riedmiller et al. (2018); Levy et al. (2019); Li et al. (2020); Zhang et al. (2021), we did not find them readily scalable to our entire list of tasks (especially to high dimensional input space like CARLA), without considerable efforts of adaptation. Thus our primary focus is to compare TAAC with baselines of different formulations of action repetition: vanilla SAC (Haarnoja et al., 2018), SAC-Nrep, SAC-Krep (Sharma et al., 2017; Biedenkapp et al., 2021), SAC-EZ (Dabney et al., 2021), and SAC-Hybrid (Neunert et al., 2020). Although originally some baselines have their own RL algorithm backbones, in this experiment we choose SAC as the common backbone because: 1) SAC is state-of-the-art among open-sourced actor-critic algorithms, 2) a common backbone facilitates reusing the same set of core hyperparameters for a fairer comparison, and 3) by reducing experimental variables, it gives us a better focus on the design choices of action repetition instead of other orthogonal algorithmic components.
Let N be a parameter controlling the maximal number of action repeating steps. SAC-Nrep simply repeats every action N times. SAC-Krep, inspired by FiGAR (Sharma et al., 2017) and TempoRL (Biedenkapp et al., 2021), upgrades an action a to a pair of (a,K), indicating that the agent will repeat action a for the next K steps (1 ≤ K ≤ N ) without being interrupted until finishing. To implement SAC-Krep, following Delalleau et al. (2020) we extended the original SAC algorithm to support a mixture of continuous and discrete actions (Appendix G). Note that SAC-Krep’s open-loop
control is in contrast to TAAC’s closed-loop control. SAC-EZ incorporates the temporally extended -greedy exploration (Dabney et al., 2021) into SAC. During rollout, if the agent decides to explore, then the action is uniformly sampled and the duration for repeating that action is sampled from a truncated zeta distribution zeta(n) ∝ n−µ, 1 ≤ n ≤ N . This fixed duration model encourages persistent exploration depending on the value of the hyperparameter µ. The training step of SAC-EZ is the same with SAC. SAC-Hybrid shares a similar flavor of H-MPO (Neunert et al., 2020) for closed-loop action repetition. It defines a hybrid policy to output the continuous action and binary switching action in parallel, assuming their conditional independence given the state. This independence between hybrid actions and how the Q values are computed in SAC-Hybrid are the biggest differences with TAAC. We also apply Retrace (Munos et al., 2016) to its policy evaluation with N -step TD as done by H-MPO. We refer the reader to the algorithm details of SAC-Hybrid in Appendix H.
In order to analyze the benefit of persistent exploration independent of that of multi-step TD learning, we also compare two additional methods. SAC-Ntd is a variant of SAC where a Q value is bootstrapped by an N -step value target with Retrace (Munos et al., 2016) to correct for off-policyness. For ablating TAAC, we evaluate TAAC-1td that employs a typical Bellman operator Bπta for one-step bootstrapping. Thus we have 8 methods in total for comparison in each task. See Table 1 for a summary and Appendix J for method details.
In our experiments, we set the repeating hyperparameter N to 3 on SimpleControl, Locomotion and Manipulation, and to 5 on Terrain and Driving. Here the consideration of N value is mainly for open-loop methods like SAC-Nrep and SAC-Krep because they will yield poor performance with large values of N . TAAC is not sensitive to N for policy evaluation, as shown in Section 5.5.
5.3 Evaluation protocol
Figure 8 and Figure 9 (Appendix K) for the complete set of individual training curves.
To measure the final model quality, we define score as the episodic return ∑T t=0 r(st, at, st+1) of evaluating (by taking the approximate mode of a parameterized continuous policy; see Appendix C for details) a method for a task episode, where T is a pre-defined time limit or when the episode terminates early. Different tasks, even within the same group, can have vastly different reward scales (e.g., tens vs. thousands). So it is impractical to directly average their scores. It is not uncommon in prior works (Hessel et al., 2018) to set a performance range for each task separately, and normalize the score of that task to roughly [0, 1] before averaging scores of a method over multiple tasks. Similarly, to facilitate score aggregation across tasks, we adopt the metric of normalized score (n-score). For each task, we obtain the score (averaged over 100 episodes) of a random policy and denote it by Z0. We also evaluate the best method on that task and obtain its average score Z1. Given a score Z, its normalized value is calculated as Z−Z0Z1−Z0 . With this definition, the n-score of each task category (a)-(e) can be computed as the averaged n-score across tasks within that category. Additionally, to measure training convergence speed, we approximate n-AUC (area under the n-score curve normalized by x value range) by
averaging n-scores on a n-score curve throughout the training. A higher n-AUC value indicates a faster convergence speed. n-AUC is a secondary metric to look at when two methods have similar final n-scores. Finally, by averaging n-score and n-AUC values over multiple tasks, we emphasize the robustness of an RL method.
Given a task, we train each method for the same number of environment frames. Crucially, for fair comparisons we also make each method train 1) for the same number of gradient steps, 2) with the same mini-batch size and learning rate, 3) using roughly the same number of weights, and 4) with a common set of hyperparameters (tuned with vanilla SAC) for the SAC backbone . More details of the experimental settings are described in Appendix J.

heading: 5.4 Results and observations
text: The n-AUC and final n-score values are shown in Table 2, and the training curves of n-score are shown in Figure 3. First of all, we conclude that the tasks are diverse enough to reduce the evaluation variance. The averaged standard deviations are small for most methods. Thus the comparison results are not coincidences and are likely to generalize to other scenarios. Overall, TAAC largely outperforms the 6 baselines regarding both final n-score (0.96[0.03] vs. second-best 0.78[0.05]) and n-AUC (0.72[0.03] vs. second-best 0.55[0.04]), with relatively small standard deviations. Note that the n-AUC gap naturally tends to be smaller than the final n-score gap because the former reflects a convergence trend throughout the training. Moreover, TAAC achieved top performance of n-AUC on each individual task category. Although some baselines achieved best final n-scores by slightly dominating TAAC, their performance is not consistent over all task categories. More observations can be made below.
- Persistent exploration and the compare-through operator are both crucial. Even with one-step TD, TAAC-1td’s performance (0.92[0.05] and 0.65[0.04]) already outperforms the baselines. This shows that persistent exploration alone helps much. Furthermore, TAAC is generally better than TAAC-1td (0.96[0.03] vs. 0.92[0.05] and 0.72[0.03] vs. 0.65[0.04]). This shows that efficient policy evaluation by the compare-through operator is also a key component of TAAC. - A proper formulation of closed-loop action repetition is important. The idea of closed-loop action repetition alone is not a magic ingredient, as SAC-Hybrid only has moderate performance among the baselines. Notably, it performs worst on Locomotion and Manipulation. Our analysis revealed that its “act-or-repeat“ policy tends to repeat with very high probabilities on Ant, FetchPickAndPlace, and FetchPush even towards the end of training, resulting in very poor n-scores. All these three tasks feature hard exploration. This result suggests that a good formulation of the idea is crucial. The two-stage decision π(â|s, a−)π(b|s, a−, â) of TAAC is clearly more informed than the parallel decisions π(â|s, a−)π(b|s, a−) of SAC-Hybrid. Furthermore, even with a latent “act-or-repeat“ action, TAAC manages to maintain the complexity of the Q function Q(s, a) as in the original control problem, while SAC-Hybrid has a more complex form Q((s, a−), (â, b)) (Appendix H). - Naive action repetition works well. Interestingly, in this particular experiment, SAC-Nrep is the overall top-performing baseline due to its relatively balanced results on all task categories. However, when it fails, the performance could be very bad (Locomotion and Terrain). While its sample efficiency is good, it has a difficulty of approaching the optimal control (final mean n-scores ≤ 0.92 in individual task categories) due to its lack of flexibility in the action repeating duration. This suggests that action repetition greatly helps, but a fixed duration is difficult to pick. - Limitation: action repetition hardly benefits tasks with densely shaped rewards and frame skipping. We notice that TAAC is no better than SAC and SAC-EZ on Locomotion regarding the final n-score, although it has slight advantages of n-AUC. The locomotion tasks have densely shaped rewards to guide policy search. Thus action repetition hardly helps locomotion exploration, especially when the 4 tasks already have built-in frameskips (4 frames for Hopper and Walker2d; 5 frames for Ant and HalfCheetah). We believe that more sophisticated temporal abstraction (e.g., skills) is needed to improve the performance in this case.

heading: 5.5 Off-policyness experiments
text: To verify that our compare-through operator is not affected by off-policyness in policy evaluation, we compare TAAC to a variant TAAC-Ntd which always bootstraps a Q value with an N -step target,
regardless of β’s outputs and without importance correction1. We choose a large trajectory length N = 10 to amplify the effect of off-policyness. Table 3 shows that N -step TD without importance correction significantly degrades the performance (0.79[0.12] vs. 0.93[0.05] and 0.55[0.08] vs. 0.68[0.03]). In contrast, the compare-through operator well addresses this issue for TAAC.
Furthermore, we implement a variant of SAC-Hybrid by replacing the Retrace operator with our compare-through operator, to see if the result difference between TAAC and SAC-Hybrid is mainly due to different ways of handling off-policyness. Table 3 shows that SAC-Hybrid-CompThr performs similarly to SAC-Hybrid (0.74[0.11] vs. 0.69[0.08] and 0.50[0.06] vs. 0.50[0.04]), suggesting that it is indeed the formulation of SAC-Hybrid that creates its performance gap with TAAC.

heading: 5.6 Policy behavior visualization and analysis
text: In this section, we mainly answer two questions:
1) How is the exploration behavior of TAAC compared to that of SAC, a “flat” RL algorithm? 2) What is the action repetition behavior of TAAC in a trained control task?

heading: 5.6.1 Exploration behavior
text: TAAC introduces persistent exploration along previous actions, having a better chance of escaping the local neighborhood around a state when acting randomly. Thus TAAC’s exploration should yield a better state space coverage than SAC’s does, assuming other identical conditions. To verify this, we visualize the state space coverage by SAC and TAAC during their initial exploration phases.
Specifically, we select two tasks MountainCarContinuous and BipedalWalker for this purpose. For either TAAC or SAC, we play a random version of its policy on either task for 50K environment frames, to simulate the initial exploration phase where the model parameters have not yet been altered by training. During this period, we record all the 50K state vectors for analysis. For MountainCarContinuous, each state vector is 2D, representing the car’s “x-position” and “x-velocity” on the 1D track. For BipedalWalker, each state vector is 24D, where the first 4D sub-vector indicates the statistics of the walker’s hull: “angle”, “angular velocity”, “x-velocity”, and “y-velocity”. For visualization, we first extract this 4D sub-vector and form a combined dataset of 100K vectors from both TAAC and SAC. Then we apply PCA (Jolliffe, 1986) to this dataset and project each 4D vector down to 2D. After this, we are able to draw KDE (kernel density estimate) plots for both MountainCarContinuous and BipedalWalker in Figure 4. We see that on both tasks, a random policy of TAAC is able to cover more diverse states compared to SAC. This suggests that in general, TAAC is better at exploration compared to a “flat” RL method, thanks to its ability of persistent exploration.

heading: 5.6.2 Action repetition behavior
text: In theory, to achieve optimal continuous control, the best policy should always sample a new action at every step and avoid action repetition at all. Thus one might assume that TAAC’s second-stage policy shifts from frequently repeating actions in the beginning of training, to not repeating actions at all
1We cannot apply Retrace to TAAC-Ntd since the probability density of πta is computationally intractable.
towards the end of training, in order to optimize the environment reward. However, some examples in Figure 5 suggest that a significant repetition frequency still exists in TAAC’s top-performing evaluation episodes. Note that because we take the mode of the switching policy during evaluation (Appendix C), this suggests that for those repeating steps, we have β∗0 > β ∗ 1 by the trained model. In fact, if we evaluate 100 episodes for each of the three tasks in Figure 5 and compute the action repetition percentage (repeating steps divided by total steps), the percentages are surprisingly 89%, 39%, and 49%, even though the agents are doing extremely well! A complete list of action repetition percentages for all 14 tasks can be found in Table 4. Generally, TAAC is able to adjust the repetition frequency according to tasks, resulting in a variety of frequencies across the tasks. More importantly, if we inspect the repeated actions, they are often not even close to action boundaries {amin, amax}, ruling out the possibility of the policy being forced to repeat due to action clipping/squashing.
We believe that there are two reasons for this surprising observation. First, with many factors such as function approximation, noise in the estimated gradient, and stochasticity of the environment dynamics, it is hardly possible for a model to reach the theoretical upper bound of an RL objective. Thus for a new action and the previous action, if their estimated Q values are very similar, then TAAC might choose to repeat with a certain chance. For example in Figure 5 (c), while the robot arm is lifting the object towards the goal in the air, it can just repeat the same lifting action for 3 steps (at t+ 5), without losing the optimal return (up to some estimation error). Similar things happen to the bipedal walker (b) when it is jumping in the air (9 steps repeated at t+ 7), and to the mountain car (a) when it is rushing down the hill (11 steps repeated at t+ 26). Second, as a function approximator, the policy network πφ have a limited capacity, and it is not able to represent the optimal policy at every state in a continuous space. For non-critical states that can be handled by repeating previous actions, TAAC might learn to offload the decision making of πφ onto β, and save πφ’s representational power for critical states. For example, the robot arm in Figure 5 (c) invokes a fine-grained control by πφ when it’s grasping the object from the table, while later does not invoke πφ for lifting it.

heading: 6 Conclusion
text: We have proposed TAAC, a simple but effective off-policy RL algorithm that is a middle ground between “flat” and hierarchical RL. TAAC incorporates closed-loop temporal abstraction into actorcritic by adding a second-stage policy that chooses between the previous action and a new action output by an actor. TAAC yielded strong empirical results on a variety of continuous control tasks, outperforming prior works that also model action repetition. The evaluation and visualization revealed the success factors of TAAC: persistent exploration and a compare-through Q operator for multistep TD backup. We believe that our work has provided valuable insights into modeling temporal abstraction and action hierarchies for solving complex RL tasks in the future.
Societal impact This paper proposes a general algorithm to improve the efficiency of RL for continuous control. The algorithm can be applied to many robotics scenarios in the real world. How to address the potentially unsafe behaviors and risks of the deployed algorithm caused to the surroundings in real-world scenarios remains an open problem and requires much consideration.

heading: Acknowledgements
text: The authors would like to thank Jerry Bai and Le Zhao for helpful discussions on this project, and the Horizon AI platform team for infrastructure support.

Review summaries:
The paper proposes to add a component to the standard RL actor-critic architecture, that learns when to repeat (or not) the last action executed by the agent. This component is a simple binary classifier, conditioned on $s_t$, $a^-$ (the previous action) and $\hat{a}$ (what the actor wants to do now), that chooses between $a^-$ and $\hat{a}$. The authors theoretically motivate their algorithm, compare it to a wide range of related work, and provide extended experimental results that show that the proposed algorithm outperforms the Soft Actor-Critic, and its recent variations.

This work proposes a temporally abstract actor-critic (TAAC). They do so by incorporating a second-stage binary policy which chooses between the previous action and a new action - this act or repeat strategy hinges on the actually sampled action as opposed to the expected behavior. The key contributions are 1) a new compare-through Q operator for multi-step TD backup for policy evaluation and 2) computing the actor gradient by multiplying a scaling factor to the change in Q function wrt to the actions for policy improvement.  

The authors introduce TAAC, which stands for Temporally Abstract Actor Critic, an extension of the actor-critic framework, in the continuous control setting, to enable the policy to choose between selecting a new action or repeating the previous one. TAAC acts as a middle ground between hierarchical RL and “flat” RL, allowing the policy to perform temporal abstraction while not having to deal with the computational or complex formulation burden of hierarchical RL methods. TAAC requires only to change slightly the architecture of the actor-critic agent, i.e. replacing the policy had with a two-stage policy head and adding the previous action as input, and to change the policy evaluation and improvement update. In this setting, using off-policy correction for multi-step TD learning such as RETRACE is not applicable. Thus, the authors also introduce a compare-through operator to stabilize learning in this setting. The authors provide novel expressions of the policy evaluation and policy improvements updates, in this setting, as well as convergence proofs. Then the authors compare TAAC to 6 baselines and 1 ablation on a set of 14 benchmarks, grouped in 5 categories. They show that on average TAAC outperforms its competitors and especially shows a significant improvement on benchmarks that exhibit exploration difficulties and require planning over long horizons. Finally, the authors propose an in-depth study of the properties of their agents and notably show that it better cover the state space compared to SAC and also exploits often the action repetition leading to high performance even in settings where we would not expect it to be useful.

 This paper studies a single option: repetitive actions that can be selected via a two-layer architecture. First layer is a policy selected conditioned on previous action; second layer is a binary decision classifier to select the action by the 1st layer or the actual output at the current step. It is a very simple idea, with some seemingly flaws (see below), but perform well in a wide range of experiments.