TAAC: Temporally Abstract Actor-Critic for Continuous Control

1 Introduction . Deep reinforcement learning ( RL ) has achieved great success in various continuous action domains such as locomotion and manipulation ( Schulman et al. , 2015 ; Lillicrap et al. , 2016 ; Duan et al. , 2016 ; Schulman et al. , 2017 ; Fujimoto et al. , 2018 ; Haarnoja et al. , 2018 ) . Despite promising empirical results , these widely applicable continuous RL algorithms execute a newly computed action at every step of the finest time scale of a problem . With no decision making at higher levels , they attempt to solve the challenging credit assignment problem over a long horizon . As a result , considerable sample efficiency improvements have yet to be made by them in complex task structures ( Riedmiller et al. , 2018 ; Li et al. , 2020 ; Lee et al. , 2020b ) and extremely sparse reward settings ( Andrychowicz et al. , 2017 ; Plappert et al. , 2018 ; Zhang et al. , 2021 ) . On the other hand , it is argued that temporal abstraction ( Parr and Russell , 1998 ; Dietterich , 1998 ; Sutton et al. , 1999 ; Precup , 2000 ) is one of the crucial keys to solving control problems with complex structures . Larger steps are taken at higher levels of abstraction while lower-level actions only need to focus on solving isolated subtasks ( Dayan and Hinton , 1993 ; Vezhnevets et al. , 2017 ; Bacon et al. , 2017 ) . However , most hierarchical RL ( HRL ) methods are task specific and nontrivial to adapt . For example , the options framework ( Sutton et al. , 1999 ; Precup , 2000 ; Bacon et al. , 2017 ) requires pre-defining an option space , while the feudal RL framework Vezhnevets et al . ( 2017 ) ; Nachum et al . ( 2018 ) ; Zhang et al . ( 2021 ) requires tuning the hyperparameters of dimensionality and domain range of the goal space . In practice , their final performance usually hinges on these choices . Perhaps the simplest form of an option or sub-policy would be just repeating an action for a certain number of steps , a straightforward idea that has been widely explored ( Lakshminarayanan et al. , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 2017 ; Sharma et al. , 2017 ; Dabney et al. , 2021 ; Metelli et al. , 2020 ; Lee et al. , 2020a ; Biedenkapp et al. , 2021 ) . This line of works can be regarded as a middle ground between “ flat ” RL and HRL . They assume a fixed candidate set of action durations , and repeat actions in an open-loop manner . Open-loop control forces an agent to commit to the same action over a predicted duration with no opportunity of early terminations . It weakens the agent ’ s ability of handling emergency situations and correcting wrong durations predicted earlier . To address this inflexibility , a handful of prior works ( Neunert et al. , 2020 ; Chen et al. , 2021 ) propose to output an “ act-or-repeat ” binary decision to decide if the action at the previous step should be repeated . Because this “ act-or-repeat ” decision will be examined at every step depending on the current environment state , this results in closed-loop action repetition . All these action-repetition methods are well justified by the need of action persistence ( Dabney et al. , 2021 ; Amin et al. , 2021 ; Zhang and Van Hoof , 2021 ; Grigsby et al. , 2021 ) for designing a good exploration strategy , when action diversity should be traded for it properly . This trade-off is important because when reward is sparse or short-term reward is deceptive , action diversity alone only makes the agent wandering around its local neighborhood since any persistent trajectory has an exponentially small probability . In such a case , a sub-optimal solution is likely to be found . In contrast , persistence via action repetition makes the policy explore deeper ( while sacrificing action diversity to some extent ) . This paper further explores in the direction of closed-loop action repetition , striving to discover a novel algorithm that instantiates this idea better . The key question we ask is , how can we exploit the special structure of closed-loop repetition , so that our algorithm yields better sample efficiency and final performance compared to existing methods ? As an answer to this question , we propose temporally abstract actor-critic ( TAAC ) , a simple but effective off-policy RL algorithm that incorporates closedloop action repetition into an actor-critic framework . Generally , we add a second stage that chooses between a candidate action output by an actor and the action from the previous step ( Figure 1 ) . Crucially , its “ act-or-repeat ” decision hinges on the actually sampled individual action instead of the expected behavior of the actor unlike recent works ( Neunert et al. , 2020 ; Chen et al. , 2021 ) . This post-acting switching scheme let the overall policy make more informed decisions . Moreover , i ) for policy evaluation , we propose a new compare-through Q operator for multi-step TD backup tailored to the action repetition scenario , instead of replying on generic importance correction ; ii ) for policy improvement , we compute the actor gradient by multiplying a scaling factor to the ∂Q ∂a term from DDPG ( Lillicrap et al. , 2016 ) and SAC ( Haarnoja et al. , 2018 ) , where the scaling factor is the optimal probability of choosing the actor ’ s action in the second stage . TAAC is much easier to train compared to sophisticated HRL methods , while it has two important features compared to “ flat ” RL algorithms , namely , persistent exploration and native multi-step TD backup support without the need of off-policyness correction . We evaluate TAAC on 14 continuous control tasks , covering simple control , locomotion , terrain walking ( Brockman et al. , 2016 ) , manipulation ( Plappert et al. , 2018 ) , and self-driving ( Dosovitskiy et al. , 2017 ) . Averaged over these tasks , TAAC largely outperforms 6 strong baselines . Importantly , our results show that it is our concrete instantiation of closed-loop action repetition that is vital to the final performance . The mere idea of repeating actions in a closed-loop manner doesn ’ t guarantee better results than the compared open-loop methods . Moreover , our surprising finding reveals that while achieving top performance , TAAC is able to “ mine ” a significant number of repeated actions with the trained policy even on continuous tasks whose problem structures on the surface seem to repel action repetition ( Section 5.6.2 ) . This suggests that aside from encouraging persistent exploration , action repetition can find its place in a good policy behavior . This is perhaps due to that the action frequency of a task can be difficult to be set exactly as the minimum value that doesn ’ t comprise optimal control while leaving no room for temporal abstraction ( Grigsby et al. , 2021 ) . 2 Related work . Under the category of temporal abstraction via action repetition , there have been various formulations . Dabney et al . ( 2021 ) proposes temporally extended -greedy exploration where a duration for repeating actions is sampled from a pre-defined truncated zeta distribution . This strategy only affects the exploration behavior for generating off-policy data but does not change the training objective . Sharma et al . ( 2017 ) and Biedenkapp et al . ( 2021 ) learn a hybrid action space and treat the discrete action as a latent variable of action repeating steps , but the introduced temporal abstraction is openloop and lacks flexibility . One recent work close to TAAC is PIC ( Chen et al. , 2021 ) which also learns to repeat the last action to address the action oscillation issue within consecutive steps . However , PIC was proposed for discrete control and its extension to continuous control is unclear yet . Also , PIC predicts whether to repeat the last action independent of a newly sampled action , which requires its switching policy to make a decision regarding the core policy ’ s expected behavior . In an application section , H-MPO ( Neunert et al. , 2020 ) explored how continuous control can benefit from a meta binary action that modifies the overall system behavior . Again , like PIC their binary decision is made in parallel with a newly sampled action . Different from PIC and H-MPO , TAAC only decides “ act-or-repeat ” after comparing the previous action with a newly sampled action . Moreover , TAAC employs a new compare-through Q operator to exploit repeated actions for multi-step TD backup , and is trained by a much simpler actor gradient by absorbing the closed-form solution of the binary policy into the continuous action objective to avoid parameterizing a discrete policy unlike H-MPO . Our experiment design ( Section 5.2 ) has covered most methods that consider action repetition . Table 1 provides a checklist of the differences between TAAC and these methods . 3 Preliminaries . We consider the RL problem as policy search in a Markov Decision Process ( MDP ) . Let s ∈ RM denote a state , where a continuous action a ∈ RN is taken . Let π ( a|s ) be the action policy , and P ( st+1|st , at ) the probability of the environment transitioning to st+1 after an action at is taken at st . Upon reaching st+1 , the agent receives a scalar reward r ( st , at , st+1 ) . The RL objective is to find a policy π∗ that maximizes the expected discounted return : Eπ , P [ ∑∞ t=0 γ tr ( st , at , st+1 ) ] , where γ ∈ ( 0 , 1 ) is a discount factor . We also define Qπ ( st , at ) = Eπ [ ∑∞ t′=t γ t′−tr ( st′ , at′ , st′+1 ) ] as the discounted return starting from st given that at is taken and then π is followed , and V π ( st ) = Eat∼π Qπ ( st , at ) as the discounted return starting from st following π . In an off-policy actor-critic setting with π and Q parameterized by φ and θ , a surrogate objective is usually used ( Lillicrap et al. , 2016 ; Haarnoja et al. , 2018 ) max φ E s∼D V πφ θ ( s ) , max φ E s∼D , a∼πφ Qθ ( s , a ) . ( 1 ) This objective maximizes the expected state value over some state distribution , assuming that 1 ) s is sampled from a replay buffer D instead of the current policy , and 2 ) the dependency of the critic Qθ ( s , a ) on the policy πφ is dropped when computing the gradient of φ . Meanwhile , θ is learned separately via policy evaluation with typical TD backup . 4 Temporally abstract actor-critic ... ... ... ... To enable temporal abstraction , we decompose the agent ’ s action decision into two stages ( Figure 1 ) : 1 ) sampling a new candidate action â ∼ πφ ( ·|s , a− ) conditioned on the action a− at the previous time step , and 2 ) choosing between a− and â as the actual output at the current step . The overall TAAC algorithm is summarized in Algorithm 1 Appendix A .

Summary: The authors introduce TAAC, which stands for Temporally Abstract Actor Critic, an extension of the actor-critic framework, in the continuous control setting, to enable the policy to choose between selecting a new action or repeating the previous one. TAAC acts as a middle ground between hierarchical RL and “flat” RL, allowing the policy to perform temporal abstraction while not having to deal with the computational or complex formulation burden of hierarchical RL methods. TAAC requires only to change slightly the architecture of the actor-critic agent, i.e. replacing the policy had with a two-stage policy head and adding the previous action as input, and to change the policy evaluation and improvement update. In this setting, using off-policy correction for multi-step TD learning such as RETRACE is not applicable. Thus, the authors also introduce a compare-through operator to stabilize learning in this setting. The authors provide novel expressions of the policy evaluation and policy improvements updates, in this setting, as well as convergence proofs. Then the authors compare TAAC to 6 baselines and 1 ablation on a set of 14 benchmarks, grouped in 5 categories. They show that on average TAAC outperforms its competitors and especially shows a significant improvement on benchmarks that exhibit exploration difficulties and require planning over long horizons. Finally, the authors propose an in-depth study of the properties of their agents and notably show that it better cover the state space compared to SAC and also exploits often the action repetition leading to high performance even in settings where we would not expect it to be useful.