Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Your paper needs a more thorough discussion of related work. In particular, please discuss in more detail work at your own home institution (Berkeley) by David Andre, Bhaskara Marti, Ron Parr and Stuart Russell on programmable HAMs and their variants. These provide exactly what your stated contributions aspire to achieve, namely high level abstract specifications of behavior using constructs that appear far more complex than the ones listed in your paper (subroutines, recursion, concurrency etc.). 


review_title: Relation to work on programmable HAMs

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper presents an approach to learning shared neural representations of temporal abstractions in hierarchical RL, based on actor-critic methods. The approach is illustrated in two tasks: gridworld with objects and a simplified Minecraft problem).  The idea of providing symbolic descriptions of tasks and learning corresponding "implementations" is potentially interesting and the empirical results are promising.  However, there are two main drawbacks of the current incarnation of this work.  First, the ideas presented in the paper have all been explored in other work (symbolic specifications, actor-critic, shared representations).  While related work is discussed, it is not really clear what is new here, and what is the main contribution of this work besides providing a new implementation of existing ideas in the context of deep learning. The main contribution if the work needs to be clearly spelled out.  Secondly, the approach presented relies crucially on curriculum learning (this is quite clear from the experiments).  While the authors argue that specifying tasks in simplified language is easy, designing a curriculum may in fact be pretty complicated, depending on the task at hand.  The examples provided are fairly small, and there is no hint of how curriculum can be designed for larger problems. Because the approach is sensitive to the curriculum, this limits the potential utility of the work. It is also unclear if there is a way to provide supervision automatically, instead of doing it based on prior domain knowledge.
More minor comments:
- The experiments are not described in enough detail in the paper. It's great to provide github code, but one needs to explain in the paper why certain choices were made in the task setup (were these optimized? What's this the first thing that worked?) Even with the code, the experiments as described are not reproducible
- The description of the approach is pretty tangled with the specific algorithmic choices. Can the authors step back and think more generally of how this approach can be formalized?  I think this would help relate it to the prior work more clearly as well.
review_title: Potentially interesting approach but novelty and utility are not clear at this a point

final_score: rating: null
review_content: No questions
review_title: No questions

final_score: rating: null
review_content: Dear authors,
 
 I am a little bit skeptical about the novelty of the approach proposed here which (as you briefly mention) is very close to what is done when learning to map instructions (natural language) to a policy. In that last case, the task is in fact harder than the one you intend to solve in this paper. Many articles are based on that (you cite some of them,but you could cite dozens of articles). Could you please provide a better comparison between what you are doing and these existing works ?  What are the advantages or drawbacks ?  So my only question is about the novelty since the paper is quite clear and well written (and the tasks are nice), and also about the lack of comparison with some baselines that could be extracted from this literature


review_title: pre-review question

final_score: rating: 3: Clear rejection
review_content: The paper proposes a new RL architecture that aims at learning policies from sketches i.e sequence of high-level operations to execute for solving a particular task. The model relies on a hierarchical structure where the sub-policy is chosen depending on the current operation to execute in the sketch . The learning algorithm is based on an extension of the actor-critic model for that particular case, and also involves curriculum learning techniques when the task to solve is hard. Experimental results are provided on different learning problems and compared to baseline methods. 

The paper is well-written and very easy to follow. I am not really convinced by the impact of such a paper since the problem solved here can be seen as an option-learning problem with a richer supervision (i.e the sequence of option is given). It thus corresponds to an easier problem with a limited impact. Moreover, I do not really understand to which concrete application this setting corresponds. For example, learning from natural langage instructions is clearly more relevant. So since the model proposed in this article is not a major contribution and shares many common ideas with existing hierarchical reinforcement learning methods,  the paper lacks a strong motivation and/or concrete application. So, the paper only has a marginal interest for the RL community

@pros: 
* Original problem with well design experiments
* Simple adaptation of the actor-critic method to the problem of learning sub policies


@cons:
* Very simple task that can be seen as a simplification of more complex problems like options discovery, hierarchical RL or learning from instructions
* No strong underlying applications that could help to 'reinforce' the interest of the approach

review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies the problem of abstract hierarchical multiagent RL with policy sketches, high level descriptions of abstract actions. The work is related to much previous work in hierarchical RL, and adds some new elements by using neural implementations of prior work on hierarchical learning and skill representations. 

Sketches are sequences of high level symbolic labels drawn from some fixed vocabulary, which initially are devoid of any meaning. Eventually the sketches get mapped into real policies and enable policy transfer and temporal abstraction. Learning occurs through a variant of the standard actor critic architecture. 

Experiments are provided through a standard game like domain (maze, minecraft etc.). 

The paper as written suffers from two problems. One, the idea of policy sketches is nice, but not sufficiently fleshed out to have any real impact. It would have been useful to see this spelled out in the context of abstract SMDP models to see what they bring to the table. What one gets here is some specialized invocation of this idea in the context of the specific approach proposed here. Second, the experiments are not thorough enough in terms of comparing with all the related work. For example, Ghavamzadeh et al. explored the use of MAXQ like abstractions in the context of mulitagent RL. It would be great to get a more detailed comparison to MAXQ based multiagent RL approaches, where the value function is explicitly decomposed. 
review_title: framework for multiagent hierarchical RL using policy sketches 

Meta-review: As per all the reviews, the work is clearly promising, but is seen to need additional discussion / formalization / experimental comparison with related work, and stronger demonstrations of the application of this technique.
 Further back-and-forth with the reviewers would have been useful, but there should be enough to go on in terms of directions. This work would benefit from being part of the workshop track.
Decision: Invite to Workshop Track