Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper introduces a reinforcement learning algorithm with temporal abstraction using the options framework. It provides empirical results in a variety of domains, demonstrating that the algorithm can improve data efficiency.

The paper is well motivated. Data efficiency is an important concern in applications of reinforcement learning.  The approach is sufficiently novel. The empirical results are positive, showing performance improvements in a variety of domains. Results in simulated robotic manipulation tasks are particularly positive, as measured by average return. 

The paper can be improved by providing additional analysis to better understand the behaviour of the algorithm and the conditions under which it improves performance. For instance, it would be useful to see what type of options are being learned in the various domains and with different limits on the number of switches allowed. 

I had difficulty in evaluating the importance of the performance differences in Figure 5. In the main text of the paper, there is no information on the reward structure of the task. Additional information is present in the  appendix but I could not easily locate the relevant information (if it is indeed there). For instance, it would be useful to know what the behavioural difference is between average returns of 60 and 100.  

In figure 3, the number of switches are listed as 5. Some experimentation with various different values would be informative. In figure 5, seeing a longer training period would be informative. MPO and RHPO are still improving at the end of the learning curve. 

Please explain Equation 4 in some detail.  I could not follow. In particular,  I do not follow why the  $\pi^{L}$ term is there.

The paper is not easy to read. Many sentences do not communicate the intended meaning clearly. As an example, the first paragraph of the introduction would be difficult to understand by readers who are not already familiar with hierarchical reinforcement learning, the options framework, and the papers cited. And some of the writing is not clear regardless of the background of the reader. For instance, the first paragraph ends with "Overall, the interaction of algorithm and environment can become increasingly difficult, especially in an off-policy setting (Precup et al., 2006)." It is not clear what is meant by a "difficult" interaction here.

The writing could be more nuanced in the discussion of results presented in Figure 3. The authors write that “off-policy learning alone [..] improves data-efficiency and can suffice to outperform on-policy option algorithms such as DAC, IOPG and Option-Critic.” This is not true in every domain. Similarly, the authors write that they “achieve improvements when training mixture policies via RHPO”. Again, this is not true in all four domains. For instance, in Hopper-v2, RHPO lags behind MPO. 

In the pseudo code for Algorithm 1 on page 5, please specify inputs to the algorithm. And please do not use any undefined symbols (e.g., $\pi’$, $Q’$). 

There is no reference to Figure 6 in the text. 

In Figure 3, please include DAC, OC, and IOPG in the figure legend. 

In Figure 3, the way the performance of DAC, OC, and IOPG are presented on the plots is misleading. For each algorithm, a constant value is shown from step 0 until step $2 \times 10^6$ although these constants correspond only to average return obtained after $2 \times 10^6$ steps. Furthermore, my understanding is that these numbers have been taken from Zhang & Whiteson (2019). For best experimental  practice, these algorithms should be tested by the authors themselves along with the other algorithms shown in the plot (e.g., HO2). This would ensure that the performances are truly comparable and that there are differences in relevant experimental settings. In addition, it would allow the reader to compare the algorithms along the entirety of their learning curves.

Doina Precup's dissertation is listed twice in the references, with different publications years. 

Misuse of the comma is prevalent throughout the paper. 

The author response answered some of my questions. But I cannot say that I now better understand the behaviour of the algorithm and the conditions under which it improves performance. I agree with reviewer 2 that analysing behaviour and performance in a simpler setting would be informative. While the writing has improved, it stills lacks the clarity and nuance one would wish to see in a paper at this conference.  


review_title: Well-motivated paper that presents a new algorithm for hierarchical reinforcement learning using the options framework. Empirical results demonstrate gains in data efficiency in a number of problems but do not provide substantial insight into the behaviour of the algorithm. 

final_score: rating: 3: Clear rejection
review_content: ## Summary
This paper introduces a novel option-learning policy gradient method, HO2. The method learns a parameterized joint distribution over options and actions and uses a soft-continuation based approach to interrupt or "switch" between options before option termination. The method introduces a new meta-parameter which enforces a hard limit on the number of "switches" that can occur, significantly reducing the variance of the option-learning method and replacing softer loss penalization based approaches. The paper demonstrates the performance of the proposed algorithm on a handful of 3D virtualized environments as well as on robotic simulation tasks.

## Review
### Summary
I am currently leaning towards recommending reject for this paper. While the approach and algorithm are novel, they also appear to be highly complex, don't provide a noticeable or consistent improvement over much simpler benchmarks, and I fear that the improvements that _are_ seen are likely due to variance in the results or are hidden behind the additional machinery in the algorithm. I remain slightly skeptical of the utility of the proposed meta-parameter $n$ for setting a hard limit on the number of "switches" between active options, with my skepticism primarily due to concerns on the difficulty of tuning this parameter and the domain-specificity of the parameter.

### Details
I'm curious about some of the hidden complexities in the algorithm. In the problem setup, a policy is defined on an MDP as possibly being $\pi(a | h)$ where $h=\{s_t, a_{t-1}, s_{t-1}, a_{t-2}, \ldots, s_0\}$,  that is a full history of interactions. First, I suppose this means we are no longer dealing with the original MDP and are working in a modified MDP where the "Markov" state is a full history; already leading towards an exponential growth of the state-space. The algorithm itself depends on a recursive product of distributions for the entire length of a sampled trajectory as a result of this adapted Markov state. I'm initially worried at how difficult it is to keep this probability from decaying towards 0 rapidly. There appear to be multiple partial marginalizations (e.g. $\sum_{i = 0}^M p(X | Y = y_i) p(Y=y_i)$) which require rescaling the final product to stay within the standard simplex, perhaps this is used to prevent the product of distributions from decaying towards 0?

One of the primary motivations of the hard switching limit is that an auxiliary penalization on the objective is hard to tune. However, it isn't clear to me that the hard limit parameter $n$ would be any easier to tune. In fact, because the inclusion of $n$ seems to require more partial marginalizations, it almost seems as if this would cause additional complexity in the optimization problem. Did you find this meta-parameter easy to tune? What are the effects of choosing it to be $n=5$ for the experiments instead of (say) 10? It appears to play a very mild variance reduction role in the results (though with only 5 seeds, we _really_ can't say much about variance since this is severely underestimating variance). If the switching limit prefers to stay small, would this suggest that the best form of the algorithm is one without the options framework at all (e.g. the best version of the proposed algorithm is an unaltered actor-critic algorithm)?

The experiments in the paper start with deep neural networks with all of the necessary machinery to make Deep RL run at the moment, including experience replay, target networks, ADAM optimizer, layer norms, mini-batches, various types of activations on each layer, neural networks of different architectures for each of the three sets of weights, different stepsizes for each of the networks, etc. I fail to see why this approach couldn't have been studied in a much simpler linear function approximation setting where statistically significant results with fewer confounding variables could have been achieved. As it stands, it is entirely unclear to me if the proposed algorithm actually provides any benefit when, in the midst of all of the machinery, the modifications above the benchmark algorithms are modest. Given this, there certainly is something to be said for a novel algorithm that does perform favorably when included in the machinery of a Deep RL feat of engineering.

I , however, am unsure if the proposed algorithm does perform favorably. From the results, it appears that in most cases RHPO performs equivalently to the proposed; certainly not statistically significantly different. With only 5 random seeds, it would be very hard to make sound claims; especially considering the known variance issues in Deep RL (take Henderson et al. 2018 for a deeper discussion). The one place where the proposed algorithm _does_ outperform RHPO is in the robot simulator (though again with only 5 seeds, heavy skepticism is called for). I found this fascinating and am curious if there is some structure that the proposed algorithm is able to take advantage of in this domain that RHPO is unable to replicate.

# After discussion period:
I have read all other reviews, resulting conversation, and have read the edits to the paper. After extended conversation with the authors and a deeper investigation into the empirical components of the paper, I find I have further concerns than originally realized in my original review and that many of my original concerns remain. I am lowering my recommendation from a 5 -> 3 to reflect the new concerns; namely the validity of the ablation study as detailed in-depth below.

review_title: Official Blind Review #2

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies an important area in RL, hierarchical RL, which improves data efficiency by incorporating abstractions. In this paper, the authors proposes an efficient option learning algorithm, which utilizes a TD(0) type objective and constrains the learned policy being not too far away from the past policy. In terms of different abstractions, the paper studies action abstraction through a mixture policy, and temporal abstraction through explicitly limiting the maximum number of switches between options. 

The method is well-motived in general, however, I feel the notation in Alg 1 is a little bit unclear, what is \pi' and Q'? The ablation study is well-done, to separate the effects of different types, and gives practitioners some useful guidelines. There is one thing I am curious about, do you try the methods using some online data? Since the paper argues the improvement compared with online option learning, it would be great to also have some experiments using online data for a fair comparison. 

I am not that familiar with hierarchical RL, so I could not give a fair judgement of the novelty compared with previous option learning literature. In terms of quality, it is a well-motivated work, clearly written in most part of the paper and gives a method with reasonably good empirical performance. I feel the off-policy argument in this paper is less clear, is it just achieved by using a Q-learning based method? This can also be used online, and how is the online-version compared with the actor-critic option learning method?
review_title: This paper proposes an efficient option learning method based on TD(0) type objective. The overall objective relies on both action abstraction and temporal abstraction, an interesting ablation study is given to understand more the effect of each individual component.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper considers the Hierarchical Reinforcement Learning setting, Options in particular, and proposes an algorithm that allows to learn both the high-level and low-level (option) policies at once, from off-policy samples. An original aspect of the algorithm is that it is easy to constrain the learned policies on how often they terminate an option and start a new one. This prevents the agent from learning tiny options that immediately terminate. It is unclear whether it can also be used to prevent the agent from learning a single big option that does everything.

The paper is very interesting and has a high educational value. It combines many different approaches and is mathematically sound. The empirical evaluation shows encouraging results. However, the significance of the empirical results is difficult to measure, due to a few cons of this paper:

- The paper is quite difficult to follow, and requires several attentive reads to be understood (by someone having a deep knowledge about options, intra-option learning and the Option-Critic architecture). I believe that the lack of clarity comes from the brevity of the paper, that has to fit in the page limit. I would suggest the authors to remove Figures 1 and 2, that take place while still being very difficult to understand. The text helps understand the figures, the figures do not help understand the text, so I would remove the figures. "Problem setup" in the preliminaries could also be removed, and replaced with an introductory paragraph in "Method", that clearly states what the contribution will be, and what are its main components/properties.
- "multi-task learning", just before "Experiments", is then used in the experiments to increase the sample-efficiency of all the algorithms. Because it is used for all the algorithms and not ablated, I don't see how it contributes to the paper. I think that the paper already proposes many ideas, and that multi-task learning could be omitted and left for a future paper.
- In the experiments, comparing MPO with HO2 allows to see a benefit from the use of options, with the proposed learning algorithm. This is a good point. However, MPO is not a well-known baseline, and a quick glance at the plots does not allow the reader to see that MPO does not use options. I would suggest either to add a little "no options" next to MPO in the figures, or to replace it (or add to it) PPO, ACKTR, ACER or the Soft Actor-Critic (SAC). These algorithms are well-known baselines. Not all of them are off-policy, but I believe that comparing HO2 to state-of-the-art algorithms, without restriction to off-policy ones, would better allow to illustrate all the gains to be obtained by HO2.
- A good argument for the use of options is to use them for explainability: telling an observer what is the current option, and what it tries to achieve. Is it possible to have a small discussion of what do the options learned by HO2 do? Do they aim at goals, or do they perform small bits of trajectories?

In summary, I like the proposed algorithm, the core contribution of the paper, the contents of Section 3. However, the clarity of this paper is to me too low, and prevents fully grasping the impact of the proposed research. I therefore recommend against acceptance, and invite the authors to remove parts on multi-task, figures that do not help, and use well-known baselines in their evaluation. With the space gained with these changes, more text can be spent summarizing what the contribution will be, and motivating the use of options.

Author response: the authors answered my question about the absence of learning curves, and provided extra details. However, I still think that the paper could be clearer and more focused, a sentiment that I think I share with the other reviewers. Given my hesitation, I would therefore not vote for accepting this paper, but I acknowledge that the proposed method is original and interesting, so I would not mind if this paper were to be accepted.
review_title: Very interesting paper but difficult to follow

Meta-review: There was a fair amount of discussion about the paper.  Several reviewers felt that the paper would have been stronger if it tried to do less but better.  The reviews describe in detail what the reviewers would have found compelling, but the key suggestion is to remove the complexity that is not essential for the approach to provide consistent improvements.  Doing this requires a better understanding of the algorithm's behavior and a valid ablation study, a new concern raised during the discussion with the authors. 

The reviewers felt that the proposed approach is potentially interesting and would like to see this paper done well.
Decision: Reject