Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper proposes a principled way to update all options at once in an option-learning framework, and conducts extensive experiments to support the proposed algorithm.
main_review: I recommend accepting this paper for publication, conditional on the issues listed below being addressed in the final version.

**Quality:**

Overall the quality is very good. The proposed method is principled, and the experiments are numerous, extensive, and well-done.

**Clarity:**

The paper is written very clearly and is easy to understand. There are some grammatical errors throughout (the ones I noticed are listed below).

**Significance:**

The proposed method is very general and very effective, making it likely to be significant moving forward.

**Originality:**

The proposed algorithm is novel to the best of my knowledge.

**Misc. comments, questions, and suggestions for improvement:**

Line 89: should be "selected"

Line 99: "This options" should be "This option"

It's good to comment on the assumptions made in the paper, explaining why they're made, whether they're common, how restictive they are, etc.

Could the authors justify Assumption 2, and briefly comment on it in the paper? Assumption 2 seems to conflict with the idea of an initiation set by implying that the initiation set for all options must be all states. Presumably the initiation set was included in the options framework for a reason, so it would be good to comment on why this assumption is being made. Is it a simplifying assumption analogous to assuming that all actions are available in all states?

Line 130: "each" is repeated.

The discount factor can be interpreted as the probability of the agent terminating at each timestep. So why is the paper using discounting in addition to the termination function of each option?

Line 177: "hyperparamter" should be "hyperparameter". This also occurs in the Figure 2 caption.

Line 191: "that" should be "than"

Line 197: Missing a "to"

Line 205: "fo" should be "of"

Section 4.4: Using importance sampling in this way can lead to high-variance updates, right? Is this variance issue somewhat mitigated by the hyperparameter $\eta$ introduced in section 4.3? Additionally, using importance sampling rules out deterministic policies, which is a limitation of the algorithm that wasn't mentioned in the paper. In fact, the appendix describes how the original FourRooms options had to be modified to be stochastic so that they would work with importance sampling. It would be better to be clear about how exactly they were modified, and how incorporating importance sampling limits the policies allowed.

Line 234: "a good separation of the state-state has been learned."

Figures: What are the shaded regions? Confidence intervals? Standard error? It's best to plot confidence intervals because non-overlapping confidence intervals imply statistically significant results, whereas non-overlapping standard error does not. That way the reader can see at a glance whether results are statistically significant or not. If the confidence intervals end up overlapping, it can be ok to lower the confidence level; it still provides some evidence to support the proposed method.

Experiments: When a paper uses "ours" ("our approach", "our method", etc.) excessively it sounds like an annoying advertisement. Framing experiments this way makes it feel like the authors' only goal is to showcase the strengths of the proposed method, instead of conducting a fair and impartial comparison. What's weird is that the experiments seem to be done pretty fairly, so why the excessive salesmanship? It was starting to really annoy me by the end of the paper.

Relatedly, the discussion of some of the figures (Figure 2a, Figure 5a for example) feels biased and bordering on overclaiming. The results are really good; there is no need to exaggerate (discussion of figure 2a), and it's ok to point out situations where MOC took longer to learn initially (figure 5a)---it could be an avenue for further research.

Figure 2: What value of $\eta$ was used in 2a?

Line 259: should be "number of updates"

Appendix A.3.1: The caption for Table 2 says "FourRooms" when it should be sparse mountain car.

I was initially wary of the introduction of the hyperparameter $\eta$, because often papers will introduce a new hyperparameter, search for the best-performing value, and then claim a performance improvement with no guidance on how to set the hyperparameter on a new environment without testing. However, I was very happy to see that $\eta=1$ was (close to) the best performing value for the function approximation setting across different forms of function approximation and different base algorithms, which is really cool!

Line 327: "simultaenously"
limitations_and_societal_impact: The paper did not seem to address limitations at all. It would be good to state that using importance sampling rules out deterministic policies, clarify whether assumption 2 rules out the use of initiation sets, and state any further limitations that the authors are aware of.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: - This paper examines the possibility of improving sample efficiency in option learning by updating all options simultaneously, rather than the single option chosen for execution. The authors that posit that such an all-options update can improve sample efficiency both when a set of pre-computed options are provided to the agent a priori as well as when the agent must learn the options themselves from scratch along with how to best compose them
- The primary contributions of this paper center around the theoretical formulation that facilitates updating of multiple options derived from the experiences collected under a single option along with an empirical instantiation of the idea with tabular, linear, and non-linear function-approximation settings.
- Empirical results are presented confirming the efficacy of the proposed update when a set of pre-computed options are provided as well as when two options are being learned by the agent from scratch. Crucially, the authors are able to report modest performance gain across all tasks in the evaluation.
main_review: This paper scores highly on clarity, significance, and (to the best of my knowledge) originality. On that last point, these ideas seem analogous to those in the (flat) reinforcement learning setting [1] which may be an interesting point of exploration for more rigorous theoretical work in this space. However, I do see numerous issues on the axis of quality, with varying degrees of severity. If the authors can suitably resolve them, I would be happy to raise my score.

Based on how the authors have presented options, they have left themselves open to the more general setting where not all options are available at each state (as opposed to the fairly common, but less general, assumption that all options are available from all states). Under this setting, the initiation set does impact the value function definition and distributions outlined in Section 3. Fortunately, this can be fixed easily by simply defining the set of available options, as done for example by Khetarpal et al. 2020, or by introducing an appropriate initiation event and conditioning, as done in the original options paper by Sutton, Precup, & Singh 1999. That said, the rest of the paper seems to be assuming all options are available everywhere and, in fact, perhaps this is necessary to achieve ergodicity in Lemma 1 (otherwise some state-option pairs automatically become unreachable). In any case, the authors should either adjust the definitions accordingly or make this assumption explicit.

On the topic of Assumption 2, I wonder if there is some level of redundancy? In particular, I don't think one could achieve the ergodicity in Assumption 1 if either the termination function or the option policy yields a probability of zero for any state-option pair. Perhaps the authors should simply add the text of Assumption 2 to Assumption 1 so that the implications of ergodicity are made transparent to readers? That said, it seems like this assumption goes against the spirit of what options are trying to achieve. Naturally, we think of hierarchical multi-task reinforcement-learning agents endowed with a broad, diverse array of skills while the individual tasks presented to the agent may only demand a subset of them at a time. Even in the single-task setting, we expect some skills to have a natural sequentiality such that one would never execute skill B before having reached a particular state from the termination of skill A. Could the authors comment on why Assumption 2 is reasonable given the scale at which we expect hierarchical reinforcement-learning agents to be able to operate? Otherwise, it seems like the long-term possibilities for the proposed algorithm are quite limited.

In Lemma 1, I would have expected the stationary distributions appearing on the LHS and RHS of Equation 2 to match. Instead, the authors introduce a new distribution on the RHS, \bar{d}, which hasn't been defined anywhere in the paper. If this is not a typo, could the authors please clarify what this distribution represents? In the rest of this review, I'll assume this was a typo and the two distributions were meant to be the same, evaluated with different arguments on the LHS and RHS.

The authors correctly introduce new notation in L151 to distinguish the discounted stationary state-option distribution from the distribution defined by Equation 2. Given the analogous discussion for standard policy-gradient methods in (flat) reinforcement learning [2], could the authors comment on the mismatch between these two distributions, with an algorithm only being able to generate samples from the latter, and how that might affect the use of the intra-option policy gradient in their experiments (I suspect the effect would be shared among all the OptionCritic methods evaluated, not just MOC)? This question, while interesting, is perhaps beyond the scope of this work and, accordingly, the authors should not prioritize answering it in their rebuttal if space is constrained.

In Section 4.4, the introduction of importance-sampling ratios introduces two new assumptions of absolute continuity (one for each ratio used in the equation of that section). From a theoretical perspective, this again comes back to a question of reasonability for these assumptions and why it should be desirable for specific option policies to necessarily retain non-zero probability mass on all actions so as to entertain these off-policy updates? Intuitively, these assumptions seem overly restrictive and unreasonable. From a practical perspective, I wonder if there are any unmentioned reguarlization techniques or other heuristics (for example, entropy regularization) used to ensure that these absolute continuity conditions were met in the experiments?

The information radius criterion used in Figure 1c seems rather opaque as it is currently discussed. It would be of great benefit to readers if the authors could offer a bit more insight into the measure itself to clarify the importance of the result.

In Algorithm 1, the upon-arrival probability (roughly line 14) is currently an unbound variable; the authors should include psuedocode for its computation. The same is true of V_\Omega(s') two lines later although, presumably, this is just taking the expectation on Q_\Omega with respect to \mu_z.

One missing hyperparameter I can't find explicitly stated in each experiment is the total number of options learned. From scanning through, it seems as though no experiment learns more than two options; could the authors please confirm this? A natural follow-up question here is if the authors experimented with more options? Otherwise, it seems a bit difficult to claim that the proposed method does not suffer from issues of diversity when learning useful options (as diversity of two things is a rather low bar to clear). For a domain like FourRooms, I would have wanted one experiment to use precisely four options just to test that MOC does not fall into the degenerate solution of aliasing the four actions of the original MDP. I also wonder if this limitation to two options is done so as to allow the experimental setup to adhere to the aforementioned assumptions of this work; does MOC begin to break down when trying to learn 10 options? 20 options? 30?

===== Post-Rebuttal =====

I thank the authors for their response and clarifying many of the points I raised. I'm optimistic that they can suitably adjust the text in the main paper to reflect these clarifications. 

I still find the need to keep non-zero probability mass on all actions/options for all policies to be restrictive. Even in a transfer setting, the margin for uncertainty mentioned in the author response could simply pertain to a decision of whether or not the agent should attempt to expand its library of options/skills in this task as opposed to discerning optimal behavior solely with the options on hand. 

More importantly, I believe that further empirical investigation is needed to investigate MOC when learning far more than 2 options, and on domains far more complex than MountainCar. The now classic OptionCritic paper seems to go as far as 8 and, genuinely, if the premise of MOC to allow for updates across all options holds, then we should be able to push far past that point or, in an equally-important fashion, shine a spotlight on precisely where the breaking point resides. Even if the authors were to run such additional experiments (which I am not asking them to do), the critical piece would be the result and associated exposition in the main paper, which currently cannot be judged in the post-rebuttal phase. I'm incrementing my score to reflect the clarifications made, however, I believe the authors would be best served in providing a more rigorous empirical study with more learned options to sharpen their contribution. Hopefully, if the paper is accepted here, the authors will consider doing this anyways for the camera-ready version as I suspect this question/point of concern is a natural one apparent to anyone in the community hoping to leverage this approach.

[1] Ciosek, Kamil, and Shimon Whiteson. "Expected Policy Gradients for Reinforcement Learning." Journal of Machine Learning Research 21 (2020): 1-51.

[2] Nota, Chris, and Philip S. Thomas. "Is the Policy Gradient a Gradient?." In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pp. 939-947. 2020.
limitations_and_societal_impact: I believe this is clear from my main review.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a method to augment Option-Critic based methods with a modified objective that allows all options to be updated with every sample, as opposed to just the option that was executed. Performance improvements over Option-Critic and other baselines on multiple environments, the flexibility of their objective to augment many similar methods, and evaluations that are mostly comprehensive are why I recommend this paper to be accepted.

main_review: ## Paper Strengths
**Motivation**
The motivation is intuitive and sensible. We want to be able to simultaneously update many options, instead of just one, so that we can encourage better sample efficiency and possibly more useful options.

**Mathematical Formulation**
The assumptions made to formulate the objective are sensible and the modified objectives are formulated in a principled manner starting from the one-step TD update.

**Method**
The method is a series of objectives that can be applied across many option-learning architectures. It is also, to my knowledge, novel.

**Experiments**
The authors test across 3 domains, with multiple environments in the continuous control domain. They also perform interesting analysis on Ant, with option visualizations that show the usefulness of the second option. There are modest to strong performance improvements across their environments over baselines. 

## Paper Weaknesses
**Experiments**
One area where HRL/Option methods shine is sparse-reward, long-horizon tasks in continuous control (typically navigation environments). I would’ve liked to see evaluations on these types of tasks in the continuous control section. It’s also possible that visualizations on those environments would show interesting learned option behaviors.

## Questions
Shouldn’t Eq 1 be either $b_t + A_t \theta_t$ or there be $A$ be negated after line 117?

Why do you think the variance across runs is reduced despite any potential variance introduced by importance sampling?

limitations_and_societal_impact: No explicit discussion on limitations, I think the authors should add this to the discussion section.
review_title: null

Meta-review: This paper presents an intuitive approach to updating multiple options together while learning in a hierarchical deep reinforcement learning setting. The approach seems principled, and performs well empirically.
All reviewers have advocated for acceptance, with the expectation that the requested clarifications in the paper are made.
Decision: Accept (Spotlight)