Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper proposes an approach to learning hierarchical policies in a lifelong learning context. This is achieved by stacking policies - an explicit "switch" policy is then used to decide whether to execute a primitive action or call the policy of the layer below it. Additionally, each task is encoded in a human-readable template, which provides interpretability.
Review:
Overall, I found the paper to be generally well-written and the core idea to be interesting. My main concern is about the performance against existing methods (no empirical results are provided), and while it does provide interpretability, I am not sure that other approaches (e.g. Tessler et al. 2017) could not be slightly modified to do the same. I think the paper could also benefit from at least one more experiment in a different, harder domain.

I have a few questions and comments about the paper:

The first paragraph claims "This precludes transfer of previously learned simple skills to a new policy defined over a space with differing states or actions". I do not see how this approach avoids suffering from the same problem? Additionally, approaches such as agent-space options [Konidaris and Barto. Building Portable Options: Skill Transfer in Reinforcement Learning, IJCAI 2007] get around at least the state part.

I do not quite follow what is meant by "a global policy is assumed to be executable by only using local policies over specific options". It sounds like this is saying that the inter-option policy can pick only options, and not primitive actions, which is obviously untrue. Can you clarify this sentence?

In section 3.1, it may be best to mention that the policy accepts both a state and task and outputs an action. This is stated shortly afterwards, but it was confusing because section 3.1 says that there is a single policy for a set of tasks, and so obviously a normal state-action policy would not work here.

At the bottom of page 6, are there any drawbacks to the instruction policy being defined as two independent distributions? What if not all skills are applicable to all items?

In section 5, what does the "without grammar" agent entail? How is the sampling from the switch and instruction policies done in this case?

While the results in Figures 4 and 5 show improvement over a flat policy, as well as the value of using the grammar, I am *very* surprised there is no comparison to existing methods. For example, Tessler's H-DRLN seems like one obvious comparison here, since it learns when to execute a primitive action and when to reuse a skill.

There were also some typos/small issues (I may have missed some):

pg 3: "In addition, previous work usually useS..."
pg 3. "we encode a human instruction to LEARN A..." (?)
pg 4. "...with A stochastic temporal grammar..."
pg 4. "... described above through A/THE modified..."
pg 6. "...TOTALLING six colors..."
There are some issues with the references (capital letters missing e.g. Minecraft)

It also would be preferable if the figures could appear after they are referenced in the text, since it is quite confusing otherwise. For example, Figure 2 contains V(s,g), but that is only defined much later on. Also, I struggled to make out the yellow box in Figure 2, and the positioning of Figure 3 on the side is not ideal either.
review_title: Hierarchical and Interpretable Skill Acquisition in Multi-Task Reinforcement Learning

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces an iterative method to build hierarchical policies. At every iteration, a new meta policy feeds in task id to the previous policy and mixes the results with an 'augmented' policy. The resulting policy is somewhat interpretable as the task id being sampled by the meta policy corresponds to one of the subgoals that are manually designed.

One of the limitation of the method is that appropriate subgoals and curriculum must be hand designed. Another one is that the model complexity grows linearly with the number of meta iterations. 

The comparison to non-hierarchical models is not totally fair in my opinion. According to the experiment, the flat policy performs much worse than the hierarchical, but it is unclear how much of this is due to the extra capacity of the model of the unfolded hierarchical policy and how much of that is due to the hierarchy. In other words, it is unclear if hierarchy is actually useful, or just the task curriculum and model capacity staging.

The paper does not appear to be  fully self contained in term of notations, in particular regarding the importance sampling I could not find the definitions of mu, and regarding the STG I could not find the definition of q and rho. 

The experimental results are a bit confusing. In the learning curves that are shown, it is not clear exactly when the set of task is expanded, nor when the hierarchical policy iteration occurs. Also, some curves are lacking the flat baseline.
review_title: Interesting model but not very clear explanation, a bit weak experimental section.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper aims to learn hierarchical policies by using a recursive policy structure regulated by a stochastic temporal grammar. The experiments show that the method is better than a flat policy for learning a simple set of block-related skills in minecraft (find, get, put, stack) and generalizes better to a modification of the environment (size of room). The sequence of subtasks generated by the policy are interpretable.

Strengths:
- The grammar and policies are trained using a sparse reward upon task completion. 
- The method is well ablated; Figures 4 and 5 answered most questions I had while reading.
- Theoretically, the method makes few assumptions about the environment and the relationships between tasks.
- The interpretability of the final behaviors is a good result. 

Weaknesses:
- The implementation gives the agent a -0.5 reward if it generates a currently unexecutable goal g’. Providing this reward requires knowing the full state of the world. If this hack is required, then this method would not be useful in a real world setting, defeating the purpose of the sparse reward mentioned above. I would really like to see how the method performs without this hack. 
- There are no comparisons to other multitask or hierarchical methods. Progressive Networks or Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning seem like natural comparisons.
- A video to show what the environments and tasks look like during execution would be helpful.
- The performances of the different ablations are rather close. Please a standard deviation over multiple training runs. Also, why does figure 4.b not include a flat policy?
- The stages are ordered in a semantically meaningful order (find is the first stage), but the authors claim that the order is arbitrary. If this claim is going to be included in the paper, it needs to be proven (results shown for random orderings) because right now I do not believe it. 

Quality:
The method does provide hierarchical and interpretable policies for executing instructions, this is a meaningful direction to work on.

Clarity:
Although the method is complicated, the paper was understandable.

Originality and significance:
Although the method is interesting, I am worried that the environment has been too tailored for the method, and that it would fail in realistic scenarios. The results would be more significant if the tasks had an additional degree of complexity, e.g. “put blue block next to the green block” “get the blue block in room 2”. Then the sequences of subtasks would be a bit less linear (e.g., first need to find blue, then get, then find green, then put). At the moment the tasks are barely more than the actions provided in the environment.

Another impedance to the paper’s significance is the number of hacks to make the method work (ordering of stages, alternating policy optimization, first training each stage on only tasks of previous stage). Because the method is only evaluated on one simple environment, it unclear which hacks are for the method generally, and which hacks are for the method to work on the environment.
review_title: A good idea, but with limited evaluation

Meta-review: This method has a lot of strong points, but the reviewers had concerns about baselines, comparisons, and hand-engineered aspects of the method. The authors gave a strong rebuttal and made substantial updates to the paper to address the concerns. I think that this has saved the submission and tipped the balance towards acceptance. 
Decision: Accept (Poster)