Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a meta-learning approach for learning hierarchically structured policies, where the agent is trained on some distribution of similar tasks to maximize the post-adaptation performance, as well as learning useful sub-policies in different time scales that enable fast adaptation. The overall model contains a high-level policy selecting over a set of options, and each option contains a sub-policy controlling action selection and a termination probability controlling the duration of low-level policy execution.  During training, the authors argue that the update for low-level policies and termination probabilities should be slower than the update for the high-level policy, to potentially encourage learning sub-policies that can be used in multiple tasks. This is the key idea of the proposed fast adaptation of modular policies (FAMP) algorithm. The experiments on Taxi and Ant Maze environments show FAMP can fast adapt and achieve better returns than other baseline meta-learning methods on unseen test tasks.

main_review: The paper tries to address an important problem in RL to learn hierarchical policies that enable better adaptation. The proposed algorithm follows the options framework and utilizes recent optimization methods, including MAML and DiCE. The paper has good coverage of related work and is distinguished from existing work by learning both sub-policies and termination conditions. Regarding experiments, the adaptation results look promising, and the selection of baselines is reasonable because MLSH does not utilize the environment id.

However, a few designing factors remain unjustified in the current paper. More ablation study is required to understand their effects, in addition to the comparison with MLSH. First, one key contribution, as claimed, is more flexible termination of sub-policies. However, it is unclear about its actual performance gain and how fixed termination would affect the learning sub-policies, and the adaptation performance remains to be checked. Second, the paper argues that fixing sub-policy in the inner-loop encourages better sub-policies. However, this intuition is not yet supported by any quantitative results. 

Also, it seems that FAMP does not require warm-up for sub-policies, while MLSH needs warm-up. Perhaps a related discussion on this is needed. How are sub-policies and termination conditions initialized? If the initial performance of sub-policies is undesired, fixing sub-policies in the inner-loop may lead to slower meta-learning.

Overall, the paper presents an interesting idea for hierarchical RL, and experiments are sufficient to prove its effectiveness in terms of better adaptation performance. However, the learned sub-policies on the Taxi task seem not easy to be interpreted. Moreover, IOPG may be computationally heavy. More analysis would strengthen the paper in understanding the effects of key designing components.

Typos:   
line 124, definition of $G_t$ should be $G_t(\tau) = \sum_{t’=t}^{T}\gamma^{(t’-t)}r_{t’}$.  
Eq. 1, $\sum_{\mathcal M_i \sim p(\mathcal M)}$ is not standard, perhaps to rewrite it as $\sum_{\mathcal M_i} p(\mathcal M_i) \mathcal L_{\mathcal M_i}(\cdots)$   

limitations_and_societal_impact: The authors discuss the limitation of the non-parallelizable responsibilities calculation of IOPG. I also don't foresee any negative social impact of this work.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes an option method where the high-level agent is a meta-learning agent that can fast learn to select proper options through sampled few episodes. DiCE is used to correct the gradients. The paper demonstrates the effectiveness of the approach on Taxi and Ant Maze environments and outperforms RL^2 and PPO baselines.
main_review: The paper's presentation is very clear and I appreciate the idea to combine meta-learning and the options framework, but I do not think that the experiments fully demonstrate the potential of the method. First of all, the experiments in the paper focus on navigation. Ant Maze environment is not a challenging task for other hierarchical RL methods. The difficulty comes from forbidding the agent from observing the maze layout. However, since the maze size is rather small, the number of possible layouts is also small. It is possible that that the agent simply remembers the solution to all map layouts and uses the meta-learning trajectories to identify the maze layout. The experiments do not demonstrate the generalizability of the novel map layout. In this case, why not simply learn a recognition model to reconstruct the maze layout [1] and then learn a goal-conditioned policy? The approach is not significant unless it can generalize to a novel layout in a larger maze. Otherwise, I would doubt that we should use search instead of meta-learning for solving complex planning tasks. Besides, since the number of options is small, and the results are not very interpretable, if we need options is questionable to me. The paper lacks a MAML-like meta-learning baseline to demonstrate that options are necessary.  

Besides, I am confused by the plots in the paper. Fig 4 shows that FAMP reaches 1200+ score after 40 episodes, but according to the pseudocode and Table 2, in each update step of the meta-agent, it needs to sample N\times L+1\times k episodes, which is larger than 40. Does this mean that there is no need to learn the meta-agent?

In short, the paper has good motivation, however, the experiments have flaws and are not strong enough to demonstrate the potential of their approach. I hope the authors can apply their approach to more complex domains and show its significance.

[1]Multi-task Batch Reinforcement Learning with Metric Learning
limitations_and_societal_impact: Yes, the author discussed the limitations of computation.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to use meta reinforcement learning in order to find a higher-level policy that can quickly update to achieve higher rewards and new tasks. The method uses the options framework in this case and uses meta-learning to find and quickly update the option selection policy given only a little bit of data from the environment. The method does show some improved performance using this particular structure over two different environments, one that was used in the previous work.

Pros
- The Meta-learning structure that's described in the paper does appear to be rather effective and somewhat novel. It shows some similarities to having a high-level policy that is more of a latent policy in this case in the middle learning performs a gradient step in order to adjust which options end up getting selected for a particular task. But the overall method is different enough from prior methods.
- The described method in the paper is also not significantly complicated and seems to fit in fairly well to previous hierarchical reinforcement learning designs in particular for using an options type setup that includes a termination function.


Cons
- While the results look promising they are applied to a limited number of environments. One environment being a discreet environment and the other being the environment from the MLSH paper. Well, potentially not necessary it could still help bolster the thoroughness of the findings in the paper if the evaluation is performed over additional environments. For example using some of the environments from meta world or other hierarchical reinforcement learning papers.
- Well the paper does outline its novel design structure for applying the meta-adaptation step on the higher-level policy there isn't any ablation analysis in the paper to really understand the most important components of the overall method. For example, additional analysis that includes all of the levels of the hierarchy inside of the inner meta adaptation step would improve the clarity of the experiments and the solidity of the findings. Overall the paper seems to be on solid ground but the additional analysis would add to the contributions of the paper.
main_review: Additional comments:

- It is interesting that the analysis in the paper indicates that the method's performance does not significantly depend on the number of options that are available to the agent. This is usually an issue for these types of methods. It would be helpful to provide some discussion on why the method does not suffer from this common issue.
- RL squared algorithm does not show any improvement in the evaluation environments. That meta-learning algorithm has been evaluated on similar tasks to this ant maze environment and has shown some success. It is possible this baseline is not applied correctly in the work.

----- Post Author Feedback -----
Given the author's feedback, my rating for the paper will stay the same.
limitations_and_societal_impact: N/A
review_title: null

Meta-review: This paper proposes a new hierarchical RL algorithm for fast adaptation. Following the options framework, the proposed method learns a set of sub-policies, a termination function, and a high-level policy via MAML with DiCE. The results on a simple grid world (Taxi) and a continuous control navigation task (Ant Maze) show that the proposed method adapts to new tasks more efficiently than a hierarchical (MLSH) and non-hierarchical baselines (PPO/RL^2). 

Although the idea is new and interesting, all of the reviewers agreed that the experiments are conducted only on simple domains. More crucially, while this paper emphasizes the importance of several ideas (e.g., learning termination function rather than using fixed-length options, fixing sub-policies during adaptation of high-level policy), there is no ablation study that justifies the claims. Thus, even though the results look encouraging, it is hard to understand the benefit of individual ideas. Therefore, this paper is not ready for publication at the moment, and I encourage the authors to conduct more comprehensive experiments for the future venue.
Decision: Reject