Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors introduce TAAC, which stands for Temporally Abstract Actor Critic, an extension of the actor-critic framework, in the continuous control setting, to enable the policy to choose between selecting a new action or repeating the previous one. TAAC acts as a middle ground between hierarchical RL and “flat” RL, allowing the policy to perform temporal abstraction while not having to deal with the computational or complex formulation burden of hierarchical RL methods. TAAC requires only to change slightly the architecture of the actor-critic agent, i.e. replacing the policy had with a two-stage policy head and adding the previous action as input, and to change the policy evaluation and improvement update. In this setting, using off-policy correction for multi-step TD learning such as RETRACE is not applicable. Thus, the authors also introduce a compare-through operator to stabilize learning in this setting. The authors provide novel expressions of the policy evaluation and policy improvements updates, in this setting, as well as convergence proofs. Then the authors compare TAAC to 6 baselines and 1 ablation on a set of 14 benchmarks, grouped in 5 categories. They show that on average TAAC outperforms its competitors and especially shows a significant improvement on benchmarks that exhibit exploration difficulties and require planning over long horizons. Finally, the authors propose an in-depth study of the properties of their agents and notably show that it better cover the state space compared to SAC and also exploits often the action repetition leading to high performance even in settings where we would not expect it to be useful.
main_review: I warmly thank the authors for their work.

Clarity:
- The paper is clear and well-written. I found it easy to read and to follow.

Soundness of the claims, significance and novelty of the contribution, and relevance to the NeurIPS community:
- The paper is well motivated and clearly positioned with respect to the literature
- The method section is very clear and the derivations are well presented and motivated
- The method seems fairly simple to implement and does not bring any additional computational cost, yet it seems to improve significantly the performance in many situations
- The experimental study is sound. The authors considered a wide range of benchmarks and thoroughly compared the method to baselines and ablations.
- The complementary analysis of the state space coverage and the frequency of re-used actions also make a lot of sense.

Limitations of this work:
- Line 88: the transition between the argument that computing $\rho^{\pi}(s)$ is usually intractable and the objective derived in the Equation is too fast and lacks maybe an argument. Also, $\mathcal{D}$ (which I assume is a replay buffer) is used in the Equation but not introduced.
- Line 186: The authors wrote: “to test if the algorithm generalizes”. I find this sentence misleading as when reading that I expect to see experiments about generalization in RL (generalization between environments, over initial states, etc …). I would recommend to change this sentence and maybe rather say something like: “to assess the performance of the algorithm” 
- Line 248: I don’t get precisely what taking the argmax of the policy means. In the discrete action setting, I would understand it as choosing the action with higher probability, however, we are in the continuous action setting here. It would be great if the authors can clarify this point.
limitations_and_societal_impact: This paper does not add limitations or potential negative societal impact to existing reinforcement learning methods.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: 
This paper studies a single option: repetitive actions that can be selected via a two-layer architecture. First layer is a policy selected conditioned on previous action; second layer is a binary decision classifier to select the action by the 1st layer or the actual output at the current step. It is a very simple idea, with some seemingly flaws (see below), but perform well in a wide range of experiments. 

main_review: 
Compare-through operator: This definition’s explanation is bit confusing. It looks this equation loops from the beginning time step to find all the subsequence of actions that match the target policy. However, line 125 doesn’t tell much except the operation notationwise. This looks the n will be usually small. Otherwise, your behaviour policy almost very close to the target policy, which becomes on-policy learning (easy). 

Here you have a separate parameter for the target, apparently influenced by target network. Then I don’t get your Th. 1: what features does it deal with? If linear or tabular, why would you need the target “network parameter”? If it’s neural networks, then why Th. 1 is true? This is not a stable operator as far as 20 years of RL tells. 

Exploration is an important aspect that this method may fail to address. Suppose one starts with a poor action. Then because of the binary decision on repeating it or not, there will be significant time repeating this single action in a long action sequence. Such method will only work with navigation in regular (rectangular) domains like Maze or Atari games. Regularizing with a joint entropy as in Section 4.3 may help, but I’m not convinced this is a good method at its heart for solving RL problems especially exploration is hard as we know. 

Experiments compared with 6 baselines on 5 kinds of control tasks (all continuous action). 
The “surprising” fact found in the experiment is the large portion of repeating actions in tasks. It is interesting to find TAAC is able be top performers; although one can argue that these tasks may be just suitable for repeated actions. 

I think the empirical performance outweighs the seemingly inefficient exploration aspect, which is exactly the value  of this paper — a simple strategy with arguably inefficient exploration appearance but performing well in many tasks.  


limitations_and_societal_impact: n/a
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes a temporally abstract actor-critic (TAAC). They do so by incorporating a second-stage binary policy which chooses between the previous action and a new action - this act or repeat strategy hinges on the actually sampled action as opposed to the expected behavior. The key contributions are 1) a new compare-through Q operator for multi-step TD backup for policy evaluation and 2) computing the actor gradient by multiplying a scaling factor to the change in Q function wrt to the actions for policy improvement. 

main_review: **Strengths**:
This work proposes a simple closed-loop off-policy RL algorithm TAAC, wherein a second stage chooses to act or repeat. The method shows how we can exploit the structure to maximize the PE-PI processes. The work proposed has the following strengths 1) The method is suitable for hybrid continuous-discrete action space including continuous control, 2) Enables persistent exploration and supports multi-step TD backups, and 3) The empirical results are strong in that the work is primarily empirical and shown to do well in a wide variety of tasks.  

One key feature of the method which can be very useful is that it serves as a middle ground between one-level RL and HRL. The combination of ideas pitched allow this work to scale better to most baselines compared. The multi-step Q operator called compare-through is shown to facilitate efficient PE which is also shown to converge.  


**Weaknesses**:   

* A primary concern is how the proposed method deviates a lot from existing HRL methods which also consider action-repeats? Perhaps the key added value is the extension to continuous control for hybrid action space. It might be useful to elaborate on this in the discussion/related work. I would encourage you to expand the related work section to cover more HRL literature and discuss where the paper is positioned in the context of broader HRL methods. 
* Arguably most HRL methods allow for the switching between the flat (one-step policies) and abstract actions (multi-step policies), especially the intra-option learning methods, which is the primary claim of this work. So what have we learned here which allows us to scale better to the list of tasks studied here? 
* What happens when we don’t specifically want action-reproducible policies? Currently a lot of the improvements seem to stem from the nature of tasks where action-repeat might be beneficial.
* While the n-score and n-AUC is better as an average over tasks, TAAC is only marginally better or same as top baselines in most tasks as reported in the Figure 8: The unnormalized reward curves of the 14 tasks. It might be useful to expand on this in the main paper as to why you choose the evaluation metric to be n-score and n-AUC and consider averaging over all tasks as the primary metric.


**Empirical Analysis**:   

The experiments evaluate TAAC in 5 categories of 14 continuous control tasks, covering simple control, locomotion, terrain walking (Brockman et al.,572016), manipulation (Plappert et al., 2018), and self-driving. 

* The baselines are chosen fairly and compare a wide range of action-repeats methods. How do these methods compare to non-action repeat baselines? Any insights on this would be great. In particular, I am curious if the action that is being repeated is not encouraging exploration or performance, how does the method overcome such a choice?
* How many random seeds have been considered? Please mention it while reporting.


**Writing and Presentation**:  
* The paper can be presented better in terms of explaining how this approach differs from existing approaches which consider action repeats. Consider a summary table which can highlight the novelty of the approach as compared to many other HRL off policy approaches as mentioned by the authors in Sec 1. Sec 2, and  Sec. 5.2.
* There are several insights in the more observations paragraph which is buried in 5.4. The paper might benefit from re-writing earlier parts to give some intuitions on key differences which this method brings to the table. For example 1)  Persistent exploration and the compare-through operator are crucial to the success of this approach. 2) The importance of the formulation of the closed-loop action repetition.

limitations_and_societal_impact: Yes the authors have discussed societal impact. The limitations of the work are also covered in different sections of the paper - it would be valuable to consolidate them and add them in the conclusion section perhaps.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes to add a component to the standard RL actor-critic architecture, that learns when to repeat (or not) the last action executed by the agent. This component is a simple binary classifier, conditioned on $s_t$, $a^-$ (the previous action) and $\hat{a}$ (what the actor wants to do now), that chooses between $a^-$ and $\hat{a}$. The authors theoretically motivate their algorithm, compare it to a wide range of related work, and provide extended experimental results that show that the proposed algorithm outperforms the Soft Actor-Critic, and its recent variations.
main_review: The paper is well-written, and the idea it proposes is simple, in a good way. The experimental results are strong, and extremely well designed (many environments of many times, extra details in the appendix, well-motivated hyper-parameters). Overall, the contribution proposed in this paper has value, and, to me, clearly its place at NeurIPS.

Possible improvements to the paper could be as follows:

- In Section 4, two contributions are introduced at once: the introduction of the $\beta$ binary classifier, and the addition of $a^-$ as input to the actor. I may have missed it, but I did not see an ablation study that shows how the algorithm performs with only the $\beta$ classifier (leaving the actor as is), or only conditioning the actor on $a^-$ (without the binary classifier). I have the feeling that conditioning the actor on the previous action may help the agent with small amounts of partial observability (of full observability it has difficulties making sense of), by allowing the actor to choose an action at time-step $t$ that helps it make a decision at time $t+1$. I would therefore be very curious to empirically see the effect of that part of the contribution.
- The paragraph after Theorem 1 is a bit difficult to understand. I understand that it tries to explain how finding which "future action" is different from the current action is possible, even when the action is continuous, but I did not understand everything. To me, it seems that storing the actual choice of $\beta$ in the experience buffer, every time-step, allows a simple implementation of Equation 4 by just looking, in the buffer, at the first "future action t+n" for which $\beta(t+n-1) =$ choose new action. Is this what the paragraph tries to convey?

Author response: the authors give a motivation for why the actor always conditions on $a^-$, and say that they will do experiments without this conditioning, to see what impact it has. They will also make the paragraph after Theorem 1 clearer. Overall, this paper presents valuable work, and I recommend accepting it.
limitations_and_societal_impact: This paper makes the effort of having a small paragraph about societal impact, which is nice, especially since there is not much to say given that this is a theoretical paper, that positively impacts the sample-efficiency of RL, but has no real-world problem in mind. I don't think that a larger discussion would be necessary.
review_title: null

Meta-review: The discussion helped clarifying points that were misunderstood by some reviewers initially. It also helped improving the clarity of the paper. There is a consensus on the scores and, now, the contribution is regarded as significant and simple. The method performs very well, it is simple enough to be implemented easily and it is computationally neutral. 
Decision: Accept (Poster)