TAAC: Temporally Abstract Actor-Critic for Continuous Control

1 Introduction . Deep reinforcement learning ( RL ) has achieved great success in various continuous action domains such as locomotion and manipulation ( Schulman et al. , 2015 ; Lillicrap et al. , 2016 ; Duan et al. , 2016 ; Schulman et al. , 2017 ; Fujimoto et al. , 2018 ; Haarnoja et al. , 2018 ) . Despite promising empirical results , these widely applicable continuous RL algorithms execute a newly computed action at every step of the finest time scale of a problem . With no decision making at higher levels , they attempt to solve the challenging credit assignment problem over a long horizon . As a result , considerable sample efficiency improvements have yet to be made by them in complex task structures ( Riedmiller et al. , 2018 ; Li et al. , 2020 ; Lee et al. , 2020b ) and extremely sparse reward settings ( Andrychowicz et al. , 2017 ; Plappert et al. , 2018 ; Zhang et al. , 2021 ) . On the other hand , it is argued that temporal abstraction ( Parr and Russell , 1998 ; Dietterich , 1998 ; Sutton et al. , 1999 ; Precup , 2000 ) is one of the crucial keys to solving control problems with complex structures . Larger steps are taken at higher levels of abstraction while lower-level actions only need to focus on solving isolated subtasks ( Dayan and Hinton , 1993 ; Vezhnevets et al. , 2017 ; Bacon et al. , 2017 ) . However , most hierarchical RL ( HRL ) methods are task specific and nontrivial to adapt . For example , the options framework ( Sutton et al. , 1999 ; Precup , 2000 ; Bacon et al. , 2017 ) requires pre-defining an option space , while the feudal RL framework Vezhnevets et al . ( 2017 ) ; Nachum et al . ( 2018 ) ; Zhang et al . ( 2021 ) requires tuning the hyperparameters of dimensionality and domain range of the goal space . In practice , their final performance usually hinges on these choices . Perhaps the simplest form of an option or sub-policy would be just repeating an action for a certain number of steps , a straightforward idea that has been widely explored ( Lakshminarayanan et al. , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 2017 ; Sharma et al. , 2017 ; Dabney et al. , 2021 ; Metelli et al. , 2020 ; Lee et al. , 2020a ; Biedenkapp et al. , 2021 ) . This line of works can be regarded as a middle ground between “ flat ” RL and HRL . They assume a fixed candidate set of action durations , and repeat actions in an open-loop manner . Open-loop control forces an agent to commit to the same action over a predicted duration with no opportunity of early terminations . It weakens the agent ’ s ability of handling emergency situations and correcting wrong durations predicted earlier . To address this inflexibility , a handful of prior works ( Neunert et al. , 2020 ; Chen et al. , 2021 ) propose to output an “ act-or-repeat ” binary decision to decide if the action at the previous step should be repeated . Because this “ act-or-repeat ” decision will be examined at every step depending on the current environment state , this results in closed-loop action repetition . All these action-repetition methods are well justified by the need of action persistence ( Dabney et al. , 2021 ; Amin et al. , 2021 ; Zhang and Van Hoof , 2021 ; Grigsby et al. , 2021 ) for designing a good exploration strategy , when action diversity should be traded for it properly . This trade-off is important because when reward is sparse or short-term reward is deceptive , action diversity alone only makes the agent wandering around its local neighborhood since any persistent trajectory has an exponentially small probability . In such a case , a sub-optimal solution is likely to be found . In contrast , persistence via action repetition makes the policy explore deeper ( while sacrificing action diversity to some extent ) . This paper further explores in the direction of closed-loop action repetition , striving to discover a novel algorithm that instantiates this idea better . The key question we ask is , how can we exploit the special structure of closed-loop repetition , so that our algorithm yields better sample efficiency and final performance compared to existing methods ? As an answer to this question , we propose temporally abstract actor-critic ( TAAC ) , a simple but effective off-policy RL algorithm that incorporates closedloop action repetition into an actor-critic framework . Generally , we add a second stage that chooses between a candidate action output by an actor and the action from the previous step ( Figure 1 ) . Crucially , its “ act-or-repeat ” decision hinges on the actually sampled individual action instead of the expected behavior of the actor unlike recent works ( Neunert et al. , 2020 ; Chen et al. , 2021 ) . This post-acting switching scheme let the overall policy make more informed decisions . Moreover , i ) for policy evaluation , we propose a new compare-through Q operator for multi-step TD backup tailored to the action repetition scenario , instead of replying on generic importance correction ; ii ) for policy improvement , we compute the actor gradient by multiplying a scaling factor to the ∂Q ∂a term from DDPG ( Lillicrap et al. , 2016 ) and SAC ( Haarnoja et al. , 2018 ) , where the scaling factor is the optimal probability of choosing the actor ’ s action in the second stage . TAAC is much easier to train compared to sophisticated HRL methods , while it has two important features compared to “ flat ” RL algorithms , namely , persistent exploration and native multi-step TD backup support without the need of off-policyness correction . We evaluate TAAC on 14 continuous control tasks , covering simple control , locomotion , terrain walking ( Brockman et al. , 2016 ) , manipulation ( Plappert et al. , 2018 ) , and self-driving ( Dosovitskiy et al. , 2017 ) . Averaged over these tasks , TAAC largely outperforms 6 strong baselines . Importantly , our results show that it is our concrete instantiation of closed-loop action repetition that is vital to the final performance . The mere idea of repeating actions in a closed-loop manner doesn ’ t guarantee better results than the compared open-loop methods . Moreover , our surprising finding reveals that while achieving top performance , TAAC is able to “ mine ” a significant number of repeated actions with the trained policy even on continuous tasks whose problem structures on the surface seem to repel action repetition ( Section 5.6.2 ) . This suggests that aside from encouraging persistent exploration , action repetition can find its place in a good policy behavior . This is perhaps due to that the action frequency of a task can be difficult to be set exactly as the minimum value that doesn ’ t comprise optimal control while leaving no room for temporal abstraction ( Grigsby et al. , 2021 ) . 2 Related work . Under the category of temporal abstraction via action repetition , there have been various formulations . Dabney et al . ( 2021 ) proposes temporally extended -greedy exploration where a duration for repeating actions is sampled from a pre-defined truncated zeta distribution . This strategy only affects the exploration behavior for generating off-policy data but does not change the training objective . Sharma et al . ( 2017 ) and Biedenkapp et al . ( 2021 ) learn a hybrid action space and treat the discrete action as a latent variable of action repeating steps , but the introduced temporal abstraction is openloop and lacks flexibility . One recent work close to TAAC is PIC ( Chen et al. , 2021 ) which also learns to repeat the last action to address the action oscillation issue within consecutive steps . However , PIC was proposed for discrete control and its extension to continuous control is unclear yet . Also , PIC predicts whether to repeat the last action independent of a newly sampled action , which requires its switching policy to make a decision regarding the core policy ’ s expected behavior . In an application section , H-MPO ( Neunert et al. , 2020 ) explored how continuous control can benefit from a meta binary action that modifies the overall system behavior . Again , like PIC their binary decision is made in parallel with a newly sampled action . Different from PIC and H-MPO , TAAC only decides “ act-or-repeat ” after comparing the previous action with a newly sampled action . Moreover , TAAC employs a new compare-through Q operator to exploit repeated actions for multi-step TD backup , and is trained by a much simpler actor gradient by absorbing the closed-form solution of the binary policy into the continuous action objective to avoid parameterizing a discrete policy unlike H-MPO . Our experiment design ( Section 5.2 ) has covered most methods that consider action repetition . Table 1 provides a checklist of the differences between TAAC and these methods . 3 Preliminaries . We consider the RL problem as policy search in a Markov Decision Process ( MDP ) . Let s ∈ RM denote a state , where a continuous action a ∈ RN is taken . Let π ( a|s ) be the action policy , and P ( st+1|st , at ) the probability of the environment transitioning to st+1 after an action at is taken at st . Upon reaching st+1 , the agent receives a scalar reward r ( st , at , st+1 ) . The RL objective is to find a policy π∗ that maximizes the expected discounted return : Eπ , P [ ∑∞ t=0 γ tr ( st , at , st+1 ) ] , where γ ∈ ( 0 , 1 ) is a discount factor . We also define Qπ ( st , at ) = Eπ [ ∑∞ t′=t γ t′−tr ( st′ , at′ , st′+1 ) ] as the discounted return starting from st given that at is taken and then π is followed , and V π ( st ) = Eat∼π Qπ ( st , at ) as the discounted return starting from st following π . In an off-policy actor-critic setting with π and Q parameterized by φ and θ , a surrogate objective is usually used ( Lillicrap et al. , 2016 ; Haarnoja et al. , 2018 ) max φ E s∼D V πφ θ ( s ) , max φ E s∼D , a∼πφ Qθ ( s , a ) . ( 1 ) This objective maximizes the expected state value over some state distribution , assuming that 1 ) s is sampled from a replay buffer D instead of the current policy , and 2 ) the dependency of the critic Qθ ( s , a ) on the policy πφ is dropped when computing the gradient of φ . Meanwhile , θ is learned separately via policy evaluation with typical TD backup . 4 Temporally abstract actor-critic ... ... ... ... To enable temporal abstraction , we decompose the agent ’ s action decision into two stages ( Figure 1 ) : 1 ) sampling a new candidate action â ∼ πφ ( ·|s , a− ) conditioned on the action a− at the previous time step , and 2 ) choosing between a− and â as the actual output at the current step . The overall TAAC algorithm is summarized in Algorithm 1 Appendix A .

Summary: This work proposes a temporally abstract actor-critic (TAAC). They do so by incorporating a second-stage binary policy which chooses between the previous action and a new action - this act or repeat strategy hinges on the actually sampled action as opposed to the expected behavior. The key contributions are 1) a new compare-through Q operator for multi-step TD backup for policy evaluation and 2) computing the actor gradient by multiplying a scaling factor to the change in Q function wrt to the actions for policy improvement.