Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary
-------------
The authors propose a novel meta-rl problem where hierarchical tasks are characterized by a graph describing all sub-tasks and their dependencies. They propose a meta-rl approach to meta-train a policy that quickly infers the subtask graph from new task data. The approach is compared to relevant baselines from both the meta-rl and hierarchical rl literature on complex domains. In particular, the authors consider a large-scale Startcraft II experiment which proves the efficiency and scalability of the proposed methodology.

Major Comments
--------------

Meta-rl is a relevant direction for reducing the sample-complexity of rl agents and scaling them to large domains. This work presents interesting and novel ideas in these settings. In particular, the few-shot rl problem with subtask dependencies seems quite interesting for both encoding and solving large hierarchical rl problems. The proposed meta-rl algorithm is sound and simple to understand. The paper is well-organized, though sometimes it is difficult to follow the formalisms due to a large number of different symbols introduced. The experiments are quite interesting and convincing. In particular, the Starcraft domain should address all concerns about the scalability and efficiency of the proposed approach. Some comments/questions follow.

1. The state available to the agent includes the number of remaining time-steps and episodes. When/how are they used?

2. The paper requires the reader to be quite familiar with some previous works (e.g., Section 3.2 requires to know Song et al. 2018 to understand the test phase). It would be good to add more background/details about these works (at least in the supplementary), so that the paper is more self-contained.

3. In the Starcraft experiment, what is the difference between MSGI-meta and MSGI-GRProp? Furthermore, where is the "oracle" baseline (introduced in sec. 5) used in the experiments? I did not find any plot reporting it.

4. The main limitation is that this approach requires options for each subtask to be provided before-hand. Do the authors think that the method is easily generalizable to learn such options as well? Furthermore, I realized this limitation only after reading the very last lines of the paper. Since this is of major importance, I believe it should be clearly stated much earlier.

Minor Comments
--------------
1. First line of sec. 2.1: R_\tau should be R_G
2. I did not find a definition of o_t and d_t which appear, e.g., in Algorithm 1.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new meta-reinforcement learning algorithm, MSGI, which focuses on the problem of adapting to unseen hierarchical tasks through interaction with the environment where the external reward is sparse. The authors make use of subtask graph inference to infer the latent subtask representation of a task through interacting with the environment using an adaptation policy and then optimize the adaptation policy based on the inferred latent subtask structure. Each task in the paper is represented as a tuple of subtask precondition and subtask reward, which are inferred via logic induction and MLE of Gaussians respectively. At meta-test time, MSGI rollouts a subtask graph execution (SGE) policy based on the graph inferred from the interactions between the environment and the adaptation policy. The authors also propose a UCB-inspired intrinsic reward to encourage exploration when optimizing the adaptation policy. Experiments are conducted on two grid-world domains as well as StarCraft II.

Overall, this paper is mainly an extension of the prior work [1], which uses a subtask graph for tackling hierarchical RL problems. This work builds upon [1] by extending to meta-learning domains and studying generalization to new hierarchical tasks. While the contribution seems a bit incremental and the experimental setting is a bit unclear and limited to low-dimensional state space, the inference of task-specific subtask graphs based on past experiences and the proposal of a UCB-inspired reward shed some interesting insights on how to approach meta-hierarchical RL where long-horizon tasks and sparse rewards have been major challenges. Given some clarification on the experimental setup and additional results on more challenging domains in the author's response, I would be willing to improve my score.

Regarding the experimental setup, the set of subtasks is a Cartesian product of the set of primitive actions and a set of all types of interactive objects in the domain, while the state is represented as a binary 3-dimensional tensor indicating the position of each type of objects. Such a setup seems a bit contrived and is limited to low-dimensional state space and discrete action space, which makes me doubt its scalability to high-dimensional continuous control tasks. It would be interesting to see how/if MSGI can perform in widely used meta-RL benchmarks in Mujoco. I also wonder how MSGI can be compared to newly proposed context-based meta-RL methods such as PEARL.

As for the results, the authors don't provide an ablation study on the UCB exploration bonus though they claim they would show it in the paper. Moreover, the result of GRProp+Oracle is also missing in the comparison. I also don't understand why MSGI-Meta and RL2 would overfit in the SC2LE case and are unable to adapt to new tasks. Is that a limitation of the method? The authors also introduce MSGI-GRProp in this setting, which is never discussed before, and claim that MSGI-GRProp can successfully generalize to new tasks. It seems that the authors don't use a meta-RL agent in order to get this domain to work. I believe more discussion on this part is needed.

[1] Sungryull Sohn, Junhyuk Oh, and Honglak Lee. Hierarchical reinforcement learning for zero-shot
generalization with subtask dependencies. In NeurIPS, pp. 7156–7166, 2018.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The main problem that is tackled here are tasks that have a main goal that can only be reached by solving prerequisite tasks. They test their method on a simple game and a very complex one. 

Methodology and novelty
The authors combine various techniques (subtask graph inference, gradient based meta-learning and inductive logic programming). It is not clearly stated if the authors combined techniques and/or if they invented a new one. What is the big difference from the work by Sohn et al. (2018)?

Experiments
The authors evaluated one agent. It would have been better if they trained multiple agents and showed a performance distribution, so it is clear that the performance is not just achieved by luck (Fig 5.). 
The video material showed clearly how the complex game (StarCraft II) was solved much quicker than a baseline model. 

Presentation
Figure 3 does not give a description of the subtask graph (middle) and the StarCraft II. The video material clearly shows the performance of their method. Section 5.1.2 does not clearly explain the different datasets D1-D5 of Playground. 
review_title: Official Blind Review #564

Meta-review: This work formulates and tackles a few-shot RL problem called subtask graph inference, where hierarchical tasks are characterized by a graph describing all subtasks and their dependencies. In other words, each task consists of multiple subtasks and completing a subtask provides a reward. The authors propose a meta-RL approach to meta-train a policy that infers the subtask graph from any new task data in a few shots. Empirical experiments are performed on different domains, including Startcraft II, highlighting the efficiency and scalability of the proposed approach.

Most concerns of reviewers were addressed in the rebuttal. The main remaining concerns about this work are that it is mainly an extension of Sohn et al. (2018), making the contribution somewhat incremental, and that its applicability is limited to problems where subtasks are provided. However, all reviewers being positive about this paper, I would still recommend acceptance. 
Decision: Accept (Poster)