Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Description:

This paper presents a reinforcement learning architecture where, based on "natural-language" input, a meta-controller chooses subtasks and communicates them to a subtask controller that choose primitive actions, based on the communicated subtask. The goal is to scale up reinforcement learning agents to large-scale tasks.

The subtask controller embeds the subtask definition (arguments) into vectors by a multi-layer perceptron including an "analogy-making" regularization. The subtask vectors are combined with inputs at each layer of a CNN. CNN outputs (given the observation and the subtask) are then fed to one of two MLPs; one to compute action probabilities in the policy (exponential falloff of MLP outputs) and the other to compute termination probability (sigmoid from MLP outputs).

The meta controller takes a list of sentences as instructions embeds them into a sequence of subtask arguments (not necessarily a one-to-one mapping). A context vector is computed by a CNN from the observation, the previous sentence embedding, the previous subtask and its completion state. The subtask arguments are computed from the context vector through further mechanisms involving instruction retrieval from memory pointers, and hard/soft decisions whether to update the subtask or not.

Training involves policy distillation+actor-critic training for the subtask controller, and actor-critic training for the meta controller keeping the subtask controller frozen.

The system is tested in a grid world where the agent moves and interacts with (picks up/transforms) various item/enemy types.
It is compared to a) a flat controller not using a subtask controller, and b) subtask control by mere concatenation of the subtask embedding to the input with/without the analogy-making regularization.


Evaluation:

The proposed architecture seems reasonable, although it is not clear why the specific way of combining subtask embeddings in the subtask controller would be the "right" way to do it.

I do not feel the grid world here really represents a "large-scale task": in particular the 10x10 size of the grid is very small. This is disappointing since this was a main motivation of the work.

Moreover, the method is not compared to any state of the art alternatives. This is especially problematic because the test is not on established benchmarks. It is not really possible, based on the shown results, to put the performance in context of other works.

review_title: Potentially good architecture; insufficient evaluation for "large-scale" tasks, no comparison to other state-of-the-art methods

final_score: rating: null
review_content: I think there's a strong connection between the proposed paper and earlier work by Ron Parr, David Andre, and Stuart Russell on programmable hierarchies of abstract machines (HAMs). Recall that in HAMs, the policy is specified as a series of abstract machine invocations where the only point at which learning occurs are specific decision pints. Programmable HAMs extends this paradigm to the case where the policy is specified by a more or less full programmable language, with macros, loops, recursion etc. Many of the stated objectives in this paper can be fairly easily accomplished in PHAMs, I think. Of course, the underlying ML technology in HAMs did not exploit the current deep Q network approach, and that is certainly a valuable thing to pursue. So, I wonder whether the proposed work can be seen as an offshoot of PHAMs, adapted to the deep RL setting? 

Later work by Marthi and Russell extended this paradigm to concurrent actions as well, so it's certainly an important thread of work to compare against (both theoretically and experimentally). 

It's also worth pointing out that there is considerable work in genetic programming (Koza et al.) that explored the discovery of reusable procedures, function abstractions, recursion etc., based on evolutionary modification of LISP  like program abstractions. Again, the technology was different, but the underlying motivations are similar. 

This is nice work, in any case, and well worth pursuing as a way of scaling deep RL. 
review_title: Connections to programmable HAMs etc. 

final_score: rating: 7: Good paper, accept
review_content: This paper presents an architecture and corresponding algorithms for
learning to act across multiple tasks, described in natural language.
The proposed system is hierarchical and is closely related to the options
framework. However, rather than learning a discrete set of options, it learns
a mapping from natural instructions to an embedding which implicitly (dynamically)
defines an option. This is a novel and interesting new perspective on options
which had only slightly been explored in the linear setting (see comments below).
I find the use of policy distillation particularly relevant for this setting.
This, on its own, could be a takeaway for many RL readers who might not necessarily
be interested about NLP applications.

In general, the paper does not describe a single, simple, end-to-end,
recipe for learning with this architecture. It rather relies on many recent
advances skillfully combined: generalized advantage estimation, analogy-making
regularizers, L1 regularization, memory addressing, matrix factorization,
policy distillation. I would have liked to see some analysis but
understand that it would have certainly been no easy task.
For example, when you say "while the parameters of the subtask controller are
frozen", this sounds to me like you're having some kind of two-timescale stochastic gradient
descent. I'm also unsure how you deal with the SMDP structure in your gradient
updates when you move to the "temporal abstractions" setting.

I am inclined to believe that this approach has the potential to scale up to
very large domains, but paper currently does not demonstrate this
empirically. Like any typical reviewer, I would be tempted to say that
you should perform larger experiments. However, I'm also glad that you have
shown that your system also performs well in a "toy" domain. The characterization
in figure 3 is insightful and makes a good point for the analogy regularizer
and need for hierarchy.

Overall, I think that the proposed architecture would inspire other researchers
and would be worth being presented at ICLR. It also contains novel elements
(subtask embeddings) which could be useful outside the deep and NLP communities
into the more "traditional" RL communities.

# Parameterized Options

Sutton et. al (1999) did not explore the concept
of *parameterized* options originally. It only came later, perhaps first with
["Optimal policy switching algorithms for reinforcement
learning, Comanici & Precup, 2010"] or
["Unified Inter and Intra Options Learning Using Policy Gradient Methods", Levy & Shimkin, 2011].
Konidaris also has a line of work  on "parametrized skills":
["Learning Parameterized Skills". da Silva, Konidaris, Barto, 2012)]
or ["Reinforcement Learning with Parameterized Actions". Masson, Ranchod, Konidaris, 2015].

Also, I feel that there is a very important distinction to be made with
the expression "parametrized options". In your work, "parametrized" comes in
two flavors. In the spirit of policy gradient methods,
we can have options whose policies and termination functions are represented
by function approximators (in the same way that we have function approximation
for value functions). Those options have parameters and we might call them
"parameterized" because of that. This is the setting of Comanicy & Precup (2010),
Levy & Shimkin (2011) Bacon & Precup (2015), Mankowitz, Mann, and
Mannor (2016) for example.

Now, there a second case where options/policies/skills take parameters *as inputs*
and act accordingly. This is what Konidaris & al. means by "parameterized", whose
meaning differs from the "function approximation" case above.
In your work, the embedding of subtasks arguments is the "input" to your options
and therefore behave as "parameters" in the sense of Konidaris.

# Related Work

I CTRL-F through the PDF but couldn't find references to any of S.R.K. Branavan's
work. Branavan's PhD thesis had to do with using control techniques from RL
in order to interpret natural instructions so as to achieve a goal. For example,
in "Reinforcement Learning for Mapping Instructions to Actions", an RL agent
learns from "Windows troubleshooting articles" to interact with UI elements
(environment) through a Softmax policy (over linear features) learned by policy
gradient methods.

As you mention under "Instruction execution" the focus of your work in
on generalization, which is not treated explicitely (afaik) in Branavan's work.
Still, it shares some important algorithmic and architectural similarities which
should be discussed explicitly or perhaps even compared to in your experiments
(as a baseline).

## Zero-shot and UVFA

It might also want to consider
"Learning Shared Representations for Value Functions in Multi-task
Reinforcement Learning", Borsa, Graepel, Shawe-Taylor]
under the section "zero-shot tasks generalization". 


# Minor Issues

I first read the abstract without knowing what the paper would be about
and got confused in the second sentence. You talk about "longer sequences of
previously seen instructions", but I didn't know what clearly
meant by "instructions" until the second to last sentence where you specify
"instructions described by *natural language*." You could perhaps
re-order the sentences to make it clear in the second sentence that you are
interested in NLP problems.

Zero-generalization: I was familiar with the term "one-shot" but not "zero-shot".
The way that the second sentence "[...] to have *similar* zero-shot [...]" follows
from the first sentence might as well hold for the "one-shot" setting. You
could perhaps add a citation to "zero-shot", or define it more
explicitly from the beginning and compare it to the one-shot setting. It could
also be useful if you explain how zero-shot relates to just the notion of
learning with "priors".

Under section 3, you say "cooperate with each other" which sounds to me very much
like a multi-agent setting, which your work does not explore in this way.
You might want to choose a different terminology or explain more precisely if there
is any connection with the multi-agent setting.

The second sentence of section 6 is way to long and difficult to parse. You could
probably split it in two or three sentences.

review_title: Novel architectural ideas ; algorithmically complex

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper can be seen as instantiating a famous paper by the founder of AI John McCarthy on learning to take advice (which was studied in depth by other later researchers, such as Jack Mostow in the card game Hearts). The idea is that the agent is given high level instructions on how to solve a problem, and must distill from it a low level policy. This is quite related to how humans learn complex tasks in many domains (e.g., driving, where a driving instructor may provide advice such as "keep a certain distance from the car in front"). 

A fairly complex neural deep learning controller architecture is used, although the details of this system are somewhat confusing in terms of many details that are presented. A simpler approach might have been easier to follow, at least initially. The experiments unfortunately are on a rather simplistic 2D maze, and it would have been worthwhile to see how the approach scaled to more complex tasks of the sort usually seen in deep RL papers these days (e.g, Atari, physics simulators etc.). 

Nice overall idea, somewhat confusing description of the solution, and an inadequate set of experiments on a less than satisfactory domain of 2D grid worlds. 
review_title: RL by learning to take advice 

final_score: rating: null
review_content: Thank you for writing this nice paper. I have a few questions:

1. Please explain the notation in section 3.1. What are the parameters of the convolution layer? only b? Is the fully connected layer using W twice? the term fully connected seems to be an abuse of notation, otherwise please explain. Do you use this notation afterwards? if so, where? Is * denotes the convolution operation(3.1) or an inner product(3.2.1)? 
2. Section 3.2.1 Why is I in R^3? 
3. Subtask arguments, do you have an MLP for every possible sub-task?  is there a difference between a train and a test sub-task? do you train an MLP for the test sub-tasks as well? 
4. 3.2.2, please highlight the difference between c_t and the subtask termination signal (\beta). In particular, can you use the termination signal instead? what would be the impact of this? 
5. Table 1. The difference between Concat+analogy to parameter+analogy is more significant in the accuracy measure. How do you measure accuracy for the concat version? 
6. Can you please explain why the baseline(FLAT) models make a good comparison with your architecture? 
7. Can you provide details regarding the test tasks that were solved/unsolved by the different agents? For example, if the agent knows to solve the tasks "pick diamond" and "transfer cow", it is reasonable to assume that it will also be capable to "transfer diamond". Did you see such correlations? 
8. Figure 2, please increase it significantly in size and add more of the paper notation above it. 

I would like to suggest the authors to remove some of the technical details from the main text and include them in the appendix instead. While the paper presents many interesting contributions, it is slightly hard to follow. 
review_title: Questions

final_score: rating: 3: Clear rejection
review_content: The paper presents a hierarchical DRL algorithm that solves sequences of navigate-and-act tasks in a 2D maze domain. During training and evaluation, a list of sub-goals represented by text is given to the agent and its goal is to learn to use pre-learned skills in order to solve a list of sub-goals. The authors demonstrate that their method generalizes well to sequences of varying length as well as to new combinations of sub-goals (i.e., if the agent knows how to pick up a diamond and how to visit an apple, it can also visit the diamond). 

Overall, the paper is of high technical quality and presents an interesting and non-trivial combination of state-of-the-art advancements in Deep Learning (DL) and Deep Reinforcement Learning (DRL). In particular, the authors presents a DRL agent that is hierarchical in the sense that it can learn skills and plan using them. The skills are learned using a differential temporally extended memory networks with an attention mechanism. The authors also make a novel use of analogy making and parameter prediction. 

However, I find it difficult to understand from the paper why the presented problem is interesting and why hadn't it bee solved before. Since the domain being evaluated is a simple 2D maze, using deep networks is not well motivated. Similar problems have been solved using simpler models. In particular, there is a reach literature about planning with skills that had been ignored completely by the authors. Since all of the skills are trained prior to the evaluation of the hierarchical agent, the problem that is being solved is much more similar to supervised learning than reinforcement learning (since when using the pre-trained skills the reward is not particularly delayed). The generalization that is demonstrated seems to be limited to breaking a sentence (describing the subtask) into words (item, location, action). 

The paper is difficult to read, it is constantly switching between describing the algorithm and giving technical details. In particular, I find it to be overloaded with details that interfere with the general understanding of the paper. I suggest moving many of the implementation details into the appendix. The paper should be self-contained, please do not assume that the reader is familiar with all the methods that you use and introduce all the relevant notations. 

I believe that the paper will benefit from addressing the problems I described above and will make a better contribution to the community in a future conference. 

review_title: Review. 

Meta-review: The paper looks at how natural language instructions can be decomposed into sub-tasks for as-yet-unseen new tasks
 hence the zero-shot generalization, which is considered to be the primary challenge to be solved. 
 The precise problem being solved by the original paper is not clearly expressed in the writing. This left some reviewers asking for comparisons, while the authors note that for the specific nature of the problem being solved, they could not seen any particular methods that is known to be capable of tackling this kind of problem. The complexity of the system and simplicity of the final examples were also found to be contradictory by a subset of the reviewers, although this is again related to the understanding of the problem being solved.
 
 With scores of 3/4/5/7, the ideas in this paper were appreciated by a subset of reviewers. 
 At the time of the writing of this metareview, the authors have posted a fairly lengthy rebuttal (Jan 18) and significant revisions (Jan 18), with no further responses from reviewers as of yet. However, it is difficult for reviewers to do a full re-evaluation of the paper on such short notice. 
 
 While the latest revisions are commendable, it is unfortunately difficult to argue strongly in favor of this paper at present.
Decision: Reject