Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper develops an approach for option discovery using meta-gradient descent to optimize a set of option networks. These option networks are then used to define the reward, and termination functions. The option policies are learned through normal RL in a separate option-network which is used to define options for a network which learns the behaviour policy using options and the base actions. They restrict their setting to that of multi-task reinforcement learning, where they sample from distributions of tasks with shared underlying regularities. They test their approach on several environments.
main_review: I think this is a very natural instantiation of the meta-learning for discovery idea proposed by [Veeriah et. al. 2019] to the option discovery problem. Many of the algorithms features are shared, and are directly applicable to the option discovery framework. I believe the ideas and paper have a clear place in the literature, but I'm concerned with the empirical evaluation as is. I'm leaning towards acceptance, but would like some of my points below to be seriously considered in the final revision thus I'm setting my score to a weak reject. I will raise my score if the following are addressed: (S2, S4, Q2, Q3, Q4, S6, S7)

## Introduction, Background, and Algorithm design

S1. I think temporal abstractions, temporally-extended actions, and options are all used interchangeably (which they all are basically interchangeable). This leads to some clarity issues. I think you should choose a term and use it.

S2. I like your hypothesis as it is stated. But I think it could be made more clear and more inline with the rest of the paper. I think the problem is in the focus on discovery in the hypothesis. Instead, maybe "If a temporal abstraction is /useful/ across /many training tasks/, then this abstraction captures regularities shared by the training tasks leading to a higher likelihood of being useful and reusable in new, /previously unseen tasks/." What I like about this rephrasing is it makes clear intuitions about why options are important, why you think the multi-task setting is the right place to test new algorithms, and gives a clear indication that what we are studying is the temporal abstraction (option). The discovery of such things is not important to the hypothesis itself.

Also within this hypothesis there are several terms which are ambiguous at stating. While they are later defined, having clear definitions at the stating of the hypothesis could be beneficial. This might mean moving the statement of the hypothesis to later sections after enough framework is built up.

S3. I'm not sure I would consider Coagent Networks as a technique for discovering options. I'm fine to disagree with that point, but the mention at line 166 should be cited appropriately (i.e. likely [Thomas, 2011] and [Kostas, 2019]).

S4. I think a problem statement would add clarity to the presented ideas. This will also remove ambiguity on why some baselines are included over others later in the paper. 

## Empirical Evaluation

Before I get into the statements and questions, I would like to focus on the baselines used for the empirical evaluation. Several notable algorithms are missing: [Machado et al. 2017 & 2018], [Harutyunyan et al. 2019], [Gregor et al, 2016], and [Eysenbach et al. 2018]. Now I don't think all these algorithms should be included (especially because the amount of compute is quite large), but I'm not sure I get the reasoning behind not including an unsupervised approach. The claim is because they are motivated by single-task RL they shouldn't transfer to the multi-task setting. I'm not sure I agree with this, and it is possible because they are learning options through an unsupervised mechanism (some without knowledge of the goal states) they could discover options which are useful for many tasks. [Gregor et al, 2016] even mentions as such in section 3. I believe this is the piece of the empirical evaluation that is difficult for me to accept, and is a large flaw in the paper (and a flaw in how option algorithms seem to be evaluated in general).

One point of concern I've noticed with the results presented in the four rooms environment: the amount of samples used for training all the competitors is significant. Given other option discovery approaches such as [Machado et al. 2018] and [Harutyunyan et al. 2019] use considerably less (on the order of 1000 episodes). If we assume worst case in [Harutyunyan et al. 2019] which is 5000 steps per episode (figure 4), that is at most 5mil samples, which is 100x less than what is used to train your networks. In reality [Harutyunyan et al. 2019] use far less than this. Maybe this is a property of the multi-task setting, but it is hard to know without a comparison.

Q1. What do you mean by usefulness for transfer? Do you mean how much they are used by the manager for testing tasks? Or something else? How are you measuring this?

Q2. What is the motivation for the input space for the four rooms environment? It seems like a lot of information about the whole environment is being passed through the network. Something like only the tabular representation (not the full map of the environment) or the (x,y) coordinates of the agent would be a more reasonable choice.

S5. The plots in figure 2 are quite difficult to read. Maybe it would be better if they were a 2x2 grid rather than 4 plots across the top.

Q3. What are the shown confidence intervals?

Q4. How many runs were used to tune the hyperparameters?

S6. I think including details on hyperparameters in the main paper is an important addition. I think we should always strive to be able to replicate the experimental setup entirely from the details presented in the main paper. Specific values of hyperparameters, and details about frameworks can definitely be pushed to the appendix.

S7. On the number of runs used for the empirical results. While your results look significant given the confidence intervals, I'm not very convinced given the insights from Deep Reinforcement Learning that Matters by Henderson. While computationally it may not be feasible to run more runs for this submission cycle, I think the paper would benefit from this. At the very least, providing confidence intervals using some of the methods discussed by Henderson et al. would be beneficial (say bootstrap). I have this issue for both the four rooms domain and the deep mind lab domain (where number of runs isn't reported).

#### Post response

From my conversation with the authors, I think I will increase my score to a 6 for now. There are still concerns over the bias introduced by the deliberation cost as it relates to the meta update, that I will have to think further on.

As of now, I view this conversation as I view most arguments between theory and empiricism. Because this paper relies much more heavily on being empirical rather than theoretical, these issues might not be problematic for this paper as it is. 
limitations_and_societal_impact: I think the limitations of the paper are not well discussed. Specifically the sample inefficiency. I discuss this lightly in my review, but feel this should be addressed with some more comparisons.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors introduce MODAC, a method that relies on meta-gradients for option discovery in hierarchical multi-task settings. The authors extend the options framework to the multi-task setting and use meta-gradients learning to discover task-independent options that can be successfully re-used across tasks. Their architecture is based on the standard hierarchical agent from Sutton et al. (1999) where a manager, that is conditioned by the current state and the task to achieve, can choose between a set of primitive actions and temporally extended options. Adding primitive actions to the action space of the manager adds flexibility and expressiveness and matches the original hierarchical framework. MODAC also implements option-policies, option-rewards and option-terminations. The training corresponds to two nested loops. An inner loop updates the manager to maximize the sum of discounted environment rewards and the option policies to maximize the discounted sum of option rewards. An outer loop evaluates the updated manager and option-policies and updates the option-rewards and option-terminations by back-propagating through the inner loop updates. This training setting enables MODAC option policies to optimize for different objectives that are parameterized by task-independent option-reward and termination that are discovered to be directly useful across many tasks. The authors evaluate MODAC in a grid world environment and in  DeepMind lab navigation tasks. They illustrate that MODAC does not only call primitive actions but also makes use of the options and that the discovered options are diverse and useful for the task at hand. They also compare MODAC to two baselines and show that MODAC outperforms them either in terms of asymptotic performance or sample efficiency or both.
main_review: I warmly thank the authors for their work.

Soundness of the claims, significance and novelty of the contribution, and relevance to the NeurIPS community:

- Discovering options, in an automated manner and without any priors, that can be reused to solve distributions of tasks, has been a long-standing goal in RL and I believe this method brings a step forward in that direction.  Using meta-gradients to learn option-rewards and terminations that are themselves used to train task-independent option-policies is novel and seems to work well.

- Allowing the manager to call primitive actions in addition to options is a strength of the method and the authors demonstrated the interest in doing so through examples and during the experimental study.

- The experimental study is thorough and well illustrates the strengths of the method as well as validates the claims made in the paper.



Limitations of this work:

- In part 1, I think the authors should briefly introduce the notations relative to MDPs. For instance, the state space S and the goal space G are never properly introduced. It would also be great if the authors may give more information about the nature of the different spaces. For instance, it is never mentioned that if the action space is discrete or continuous. Also it not clear to me what is the nature of the goal space G. Is it assumed to be a sub-part of the state space or a set of possible tasks?

- In part 1, the number of options K is introduced lately and almost implicitly when the authors describe the outer loop of the algorithm. I would recommend introducing it sooner in the part.

- It would be interesting if the authors could discuss to which extent MODAC could be extended to continuous action spaces and assess its performance on traditional HRL benchmarks such as the ones considered in HAC.
limitations_and_societal_impact: This work provides a method to discover options in multi-task HRL. Thus, it does not add limitations or potential negative societal impact to existing hierarchical reinforcement learning methods.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: Authors propose a meta-learning algorithm for learning temporarily extended actions within the options framework. In addition to the commonly used option modules defined in the original options framework (high-level policy, terminations, sub-policies) the algorithm also uses an aditional learned option-rewards module ($\{r^{o^i}\}$) to reward sub-policies when they perform actions. The algorithm is designed to extract useful temporarily extended actions from multiple environments by using meta-gradients (backpropagating through gradient). Meta-gradients are taken wrt. parameters of termination functions and option-rewards whereas sub-policies and high-level policy are updated in the inner update. The intuition behind using option-rewards and terminations as meta-parameters is that they should define task-agnostic sub-goals. 

Authors evaluate the performance of this new algorithm on one simple and one complex navigation tasks. The main performance metric considered is performance on a new unseen task (after pre-training on many tasks), although authors do show additional visualizations of learned options. Comparison is made with two additional algorithms for learning options  MLSH (designed for multi-task learning) and Option Critic (originally designed for single-task) and a single-task Actor-Critic baseline. In experiments, authors show that proposed algorithm outperforms all three baselines when options are transferred to previously unseen tasks. This suggests that proposed algorithm discovers options with better temporal structure.
main_review: The paper introduces an interesting novel method for meta-learning of options. It is clearly written and experiments show that the approach works well on both simple and challenging baseline. However, the method is mostly justified intuitively and I find the theoretical justification of the algorithm and the justification of some design choices that was provided quite weak. It is thus hard to see whether the method is theoretically sound and why it works so well when compared to prior methods. Authors also do not discuss the limitations of the approach. I would thus not recommend accepting the paper unless authors provide a clear derivation of the updates used in the algorithm from the main objective, better justify design choices and discuss limitations.  

Originality: The method builds on known techniques but includes a novel contribution (using gradient based meta-learning to learn hierarchical policies). Authors explain the differences and relation to related work although I am not completely sure if there isn't a connection missing between adding a negative reward for options switching and deliberation cost [2].

Quality: Claims are mostly supported by experiments, however, I did not find theoretical analysis of the approach sufficient. From the derivations provided in the paper and the appendix I cannot tell if derived updates are sound and as far as I can tell they seem to be at least a few missing dependencies (see detailed comments). Furthermore, in my opinion, the addition of learned option-reward module needs better justification than what is provided because it is usually not part of the options framework and I do not immediately see why "task reward may be insufficient to discover reusable, task-independent options", why such addition would be necessary and how it would combine with the options framework. I also did not find the limitations of the approach in the paper.

Clarity: The writing is mostly very clear and necessary concepts are well explained. The submission is also well organized. 

Significance: The submission tackles an interesting difficult task (learning hierarchical structure in multi-task learning) and according to experiments outperforms well-known baselines (also on non-trivial task). Additionally, it presents a novel idea: using gradient based meta-learning in combination with the options framework to learn such structure. I believe that this idea and its realization would be a contribution to the research community.

**Detailed Comments\Questions:**
- The option-reward module is only shortly introduced in line 95 with the intuition (line 105) that they define an option's subgoal and implicitly defines the option-policy (line 79). However, the connection of this module with the options framework is a bit unclear. I think that authors should provide an explanation why these options-rewards define sub-goals and show what kind of option-reward functions (subgoals) are learned in four rooms environment. I think this idea might be somehow connected to the Option Keyboard [3]. 

- I think that addition of "switching cost $c$" could be connected to the Deliberation cost [2]. Doesn't this interfere with the objective/update equations?

- From Eq. 3 and the learning algorithm, I understand that $r^o$ are not connected to real rewards in any way. I do not see the motivation for $G_t^o$ or why it is sensible to interpret terminations as a discount. Why does it make sense to interpret $\beta, r^o$ and $v^o$  as discounted returns? Why isn't normal discounting used? Since $\beta, r^o$ and $v^o$ are all meta-learned isn't $G_t^o-v^o(s_t)$ just an arbitrary 

- The objective and following derivation in the appendix (line 75) does not seem correct since it does not take into account the dependency of sampling distribution $p(G_t^M|\{\theta^o_i\},\theta^M, \{\eta^{\beta^o_i}\})$ on terminations $\beta$. Following line 81 in the appendix, why is only a single $\log\pi^o$ inside both expectations? As far as I can tell , it should be whole $p(G_t^M|\{\theta^o_i\},\theta^M, \{\eta^{\beta^o_i}\})$ decomposed into multiple components. Furthermore, I think that is not sufficient to differentiate through ${\theta^o}'$ since ${\theta^M}'$ (after inner updates) also depends on termination parameters (through sampling distribution as was explained in [1]) . Updates in Eqs. 1,2,5 are also written in terms of single samples but they are actually estimates of expectations which makes a difference in meta-learning setting.  

- In line 55 authors claim that all options directly optimizing the same (main task) reward may not be sufficient to discover reusable task-independent options. Why/In which settings is this a case?  

- In Fig 2 c) you show average return. However in such environments where reward for reaching a goal is one, one usually optimizes for discounted return (since return is always 1 if agents finds the goal). Is there any specific reason why you are showing return instead?  Can you provide plots for discounted return?

- Could you add the visualization of terminations and learned option-rewards module for four rooms task?   

Typos and minor details:
- line 32 cite
- line 63 use citations with MLSH and OC? (they are used later instead)
- use $\theta'$ on the left side in Eqs 1 and 2
- introduce L in algorithm box
- line 166 Coagent Networks citation
- Fig 2b text is very small
- line 269 sight instead of site
- line 299 as far as I remember Mankowitz et al. 2016 does not require hand-designed temporal abstractions

[1] DiCE: The Infinitely Differentiable Monte Carlo Estimator (Foerster et al. 2018)  
[2] When Waiting is not an Option : Learning Options with a Deliberation Cost (Harb et al. 2017)  
[3] The Option Keyboard: Combining Skills in Reinforcement Learning (Barreto et al. 2019) 

===== Post-Rebuttal ======
After the discussion with authors and other reviewers I decided to keep my score unchanged. Even though I do appreciate the effort that authors put into providing an additional baseline, I do not think that the issues related to theoretical justification of the method were properly addressed. While I realize that the work is mostly empirical, I do think that the empirical adjustments to the method should be better motivated and their effects on theoretical algorithm should be discussed in more detail especially since the performance of the algorithm is not that much better than the one of the single-task agent. 
limitations_and_societal_impact: Authors do discuss societal impact but do not discuss the limitations or drawbacks of the approach when compared to related methods.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The author’s propose an options-based hierarchical RL algorithm that uses meta-learned option-rewards and termination to train individual options.
The method is instantiated with an actor-critic RL method and experiments are conducted to test the transfer capabilities of the meta-learned options.
main_review: Strengths:
+ Very clear exposition of the method and good writing overall.
+ Detailed appendix with hyperparameter settings and method implementation details.
+ Strong experimental results for options with challenging DeepMind Lab environment and Atari.
+ Visualizations of learned options.

Weaknesses:
+ It would be good to also test against a subgoal based hierarchical RL method (f.e. HIRO)
+ shouldn’t it be $+\kappa^o v^o$ in equation 1 since value should be pushed up if it's underestimating n-step return?
+ Some more motivation/analysis for the benefit of using meta-learned rewards and terminations rather than sub-policies would be helpful. Right now the intuition seems to be that it is less task-dependent. In principle meta-learning could cause option-rewards to become task-dependent just like directly learned options?

Summary:
Novel approach to option discovery using meta-gradients in new way. Strong experimental results and provides a promising direction for options framework and multi-task transfer in RL.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors tackle the problem of learning options that would be useful across a range of similar tasks (for example similar mazes with different difficulties). The authors use a meta-gradient approach with appropriately defined parameters and meta-parameters. The paper contains quantitative and qualitative experiments on variants of the FourRooms task and on the more complicated DeepMind Lab (as well as experiments on Atari in the appendix)
main_review: Meta-gradients have been used previously for different settings in RL but this paper seems to be the first to apply it in a HRL setting. The combination of the two could lead to more flexible options that generalize better. The paper is also generally well written and cites many important related work.

That being said, I wish some of the theoretical and experimental details were better motivated.

In particular for the experimental aspect:
1. Although the choice of adding primitive actions to the options set is present in the original paper on options (Sutton et al.), it is not a current practice these days. Importantly, it is not clear whether OC and MLSH are using these primitive actions. It is good to know that options are used the majority of the time, but an ablation analysis looking at the role of adding primitive actions is necessary for me. The reason for this is that it has nothing to do with the meta-gradient approach and it would be enlightening for the community to understand the choice behind adding primitive actions.
2. The way the deliberation cost is not really principled and not motivated theoretically.  The authors don't explain why the switching cost is added to the manager but not to the options themselves. Notice that in Harb et al. 2018 the derivation indicates that it should be used for all updates. Moreover, shouldn't meta gradient to learn temporally extended options, if these options were generally useful? It is not clear why it is also necessary to add a deliberation cost.
3. The first claim of the paper is that "our approach can discover meaningful and diverse temporally-extended options". Personally, when I look at the qualitative results on DeepMind Lab, I am really not sure how the behaviour of each option is useful or interesting. Moreover, could we see similar options emerging from the baselines?
4. The MODAC algorithm, which uses learned rewards, is compared to HRL approaches that don't, that is Option-Critic (OC) and MLSH. I don't find this aspect particularly surprising. I think stronger baselines would be necessary here, at least for the smaller experiments.

In particular for the theoretical aspect:
1. Regarding Equation 3, the authors mention that when the option terminates, there is no bootstrapping. To me this seems simply wrong. Notice that in OC there is always bootstrapping and that the boostrapping is consistent with whether or not there was termination.
2. The derivation of the meta-gradient approach, which is the essential contribution of the paper, is very succinct. Even in the appendix, there is barely any details as to the approximations being considered. As a comparison, please consider the original Meta-Gradient paper (Xu et al.) where many more details are provided (basically the whole of section 1).
3. This leads to the question as to how the choice of the parameters vs hyper-parameters was made. Indeed, according to the original paper on options, the target for updating an option depends on the reward function, the termination function, the policy over options and even the other options' policies. What motivates which functions are considered parameters or meta-parameters? 


===== Post-Rebuttal ======

I appreciate the authors' efforts to address the points raised by the reviewers. In particular, I appreciate that the author providing a more competitive baseline where reward shaping is used for learning options. I would suggest to include all the details (such as hyperparameter search) for these additional experiments. Given the authors' efforts in addressing the issues, I will be raising my score to a weak accept even though it would be better to have an updated version of the paper to be convinced.

I think that the authors should be more clear in the approximations introduced in the work. This could be addressed by presenting complete and detailed derivations for their method. Many of the issues raised by the reviewers would then be apparent. As it is, the paper presents the update rules but the derivation of these updates are incomplete, and the answers provided during the rebuttal lead to more questions that remain up in the air.

In particular, both the way bootstrapping is done is not justified theoretically, even though intuitively there could be a motivation. The same goes for the deliberation cost. Why are the authors introducing actions, to then introduce a heuristic to penalize for choosing them? As we are learning options and their length can be changed, it is not necessary to include actions in the first place.

The meta-gradient method already introduces a set of approximations, and at some point it becomes hard to justify why more heuristics are necessary, especially as the empirical performance is not that much better than using a simple flat agent.
limitations_and_societal_impact: Learning options can lead to better interpretability. This could be highlighted.
review_title: null

Meta-review: 
The reviewers thought this was an interesting paper considering how to combine meta-gradient methods with HRL, and the empirical experiments were convincing. In the internal discussions two reviewers suggested the paper would be further strengthened by a more detailed discussion on the theoretical side. The authors are encouraged to consider the reviewers’ feedback in their revision for the camera ready.
Decision: Accept (Poster)