Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposed a method that enables hierarchical planning. Specifically, given human defined attributes, it learns a graph of attribute transitions. Given a test task and its target set of attributes and a current state, it infers the attribute of current state and search over paths through attribute spaces to get a high-level plan, and then use its low level policy to execute the plan.  Based on the relation (transition) of attributes, new and more complex tasks at test time can be solved compositionally. The proposed method is indeed technically sound and have some distinctions to other existing methods in literature, however, the novelty of this work does not seem to be significant as I will elaborate more.

1.	In this work, the attributes are provided by human, which certainty can incur a significant amount of effort hence limit is generalizable of the proposed method.  It might be more appealing if automatic attributes discovery can be incorporated into current framework to remove such restriction as well as better justify the assumption underlying the proposed method is that “the cost of the supervision required to identify the important features of an environment, or to describe the space of possible tasks within it, is not too expensive” 

2.	the definition of ignorabilty a little confusing. “transition between \rho_i and \rho_j should only depend on the attributes \rho, not exact state” should be written.

3.	When evaluating the model, the authors mentioned that “We recompute the path at intermediate steps in case we reach an attribute set we don’t expect”. What does “attribute set we don’t expect” mean? Do you mean the attribute never seen before?

4.	The author should give better account of the relation between the proposed method to other frameworks. The authors mentioned that the proposed method can be placed into the framework of option. However, the option frame is mainly dealing with temporal abstraction, whereas this work seems have much more to do state abstraction. 

5.	The current work is limited to dealing with problems with two levels of hierarchy 
 
6. Minor comments 
which properties of the environment we consider important -> which properties of the environment are important
a model that learns -> a method 
from the set of goals rho_j -> from the set of goals, 
GVF is undefined

review_title: Review for Composable Planning with Attributes

final_score: rating: 7: Good paper, accept
review_content: Summary: This paper proposes a method for planning which involves learning to detect high-level subgoals (called "attributes"), learning a transition model between subgoals, and then learning a policy for the low-level transitions between subgoals. The high-level task plan is not learned, but is computed using Dijkstra's algorithm. The benefit of this method (called the "Attribute Planner", or AP) is that it is able to generalize to tasks requiring multi-step plans after only training on tasks requiring single-step plans. The AP is compared against standard A3C baselines across a series of experiments in three different domains, showing impressive performance and demonstrating its generalization capability.

Pros:
- Impressive generalization results on multi-step planning problems.
- Nice combination of model-based planning for the high-level task plan with model-free RL for the low-level actions.

Cons:
- Attributes are handcrafted and pre-specified rather than being learned.
- Rather than learning an actual parameterized high-level transition model, a graph is built up out of experience, which requires a large sample complexity.
- No comparison to other hierarchical RL approaches.

Quality and Clarity:

This is a great paper. It is extremely well written and clear, includes a very thorough literature review (though it should probably also discuss [1]), takes a sensible approach to combining high- and low-level planning, and demonstrates significant improvements over A3C baselines when generalizing to more complex task plans. The experiments and domains seem reasonable (though the block-stacking domain would be more interesting if the action and state spaces weren't discrete) and the analysis is thorough.

While the paper in general is written very clearly, it would be helpful to the reader to include an algorithm for the AP.

Originality and Significance:

I am not an expert in hierarchical RL, but my understanding is that typically hierarchical RL approaches use high-level goals to make the task easier to learn in the first place, such as in tasks with long planning horizons (e.g. Montezuma's Revenge). The present work differs from this in that, as they state, "the goal of the model is to be able to generalize to testing on complex tasks from training on simpler tasks" (pg. 5). Most work I have seen does not explicitly test for this generalization capability, but this paper points out that it is important and worthwhile to test for.

It is difficult to say how much of an improvement this paper is on top of other related hierarchical RL works as there are no comparisons made. I think it would be worthwhile to include a comparison to other hierarchical RL architectures (such as [1] or [2]), as I expect they would perform better than the A3C baselines. I suspect that the AP would still have better generalization capabilities, but it is hard to know without seeing the results. That said, I still think that the contribution of the present paper stands on its own.

[1] Vezhnevets, A. S., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D., & Kavukcuoglu, K. (2017). FeUdal Networks for Hierarchical Reinforcement Learning. Retrieved from http://arxiv.org/abs/1703.01161
[2] Kulkarni, T. D., Narasimhan, K. R., Saeedi, A., & Tenenbaum, J. B. (2016). Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation. Advances in Neural Information Processing Systems.
review_title: Impressive generalization from single-task training to multi-step planning

final_score: rating: 4: Ok but not good enough - rejection
review_content: - This paper proposes a framework where the agent has access to a set of user defined attributes parametrizing features of interest. The agent learns a policy for transitioning between similar sets of attributes and given a test task, it can repurpose its attributes to reactively plan a policy to achieve the task. A grid world and tele-kinetically operated block stacking task is used to demonstrate the idea

- This framework is exactly the same as semi-MDPs (Precup, Sutton) and its several generalizations to function approximators as cited in the paper. The authors claim that the novelty is in using the framework for test generalization. 

- So the main burden lies on experiments. I do not believe that the experiments alone demonstrate anything substantially new about semi-MDPs even within the deep RL setup. There is a lot of new vocabulary (e.g. sets of attributes) that is introduced, but it dosen't really add a new dimension to the setup. But I do believe in the general setup and I think its an important research direction. However the demonstrations are not strong enough yet and need further development. For instance automatically discovering attributes is the next big open question and authors allude to it.

- I want to encourage the authors to scale up their stacking setup in the most realistic way possible to develop this idea further. I am sure this will greatly improve the paper and open new directions of researchers. 


review_title: review

Meta-review: Overall the reviewers appear to like the ideas in this paper, though this is some disagreement about novelty (I agree with the reviewer who believes that the top-level search can very easily be interpreted as an MDP, making this very similar to SMDPs). The reviewers generally felt that the experimental results need to more closely compare with some existing techniques, even if they're not exactly for the same setting.
Decision: Reject