Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: In the context of multitask reinforcement learning, this paper considers the problem of learning behaviours when given specifications of subtasks and the relationship between them, in the form of a task graph. The paper presents a neural task graph solver (NTS), which encodes this as a recursive-reverse-recursive neural network. A method for learning this is presented, and fine tuned with an actor-critic method. The approach is evaluated in a multitask grid world domain.

This paper addresses an important issue in scaling up reinforcement learning to large domains with complex interdependencies in subtasks. The method is novel, and the paper is generally well written. I unfortunately have several issues with the paper in its current form, most importantly around the experimental comparisons.

The paper is severely weakened by not comparing experimentally to other learning (hierarchical) schemes, such as options or HAMs. None of the comparisons in the paper feature any learning. Ideally, one should see the effect of learning with options (and not primitive actions) to fairly compare against the proposed framework. At some level, I question whether the proposed framework is doing any more than just value function propagation at a task level, and these experiments would help resolve this.

Additionally, the example domain makes no sense. Rather use something more standard, with well-known baselines, such as the taxi domain.

I would have liked to see a discussion in the related work comparing the proposed approach to the long history of reasoning with subtasks from the classical planning literature, notably HTNs.

I found the description of the training of the method to be rather superficial, and I don't think it could be replicated from the paper in its current level of detail.

The approach raises the natural questions of where the tasks and the task graphs come from. Some acknowledgement and discussion of this would be useful.

The legend in the middle of Fig 4 obscures the plot (admittedly not substantially).

There are also a number of grammatical errors in the paper, including the following non-exhaustive list:
2: as well as how to do -> as well as how to do it
Fig 2 caption: through bottom-up -> through a bottom-up
3: Let S be a set of state -> Let S be a set of states
3: form of task graph -> form of a task graph
3: In addtion -> In addition
4: which is propagates -> which propagates
5: investigated following -> investigated the following
review_title: Neural Task Graph Execution

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
Summary: the paper proposes an idea for multi-task learning where tasks have shared dependencies between subtasks as task graph. The proposed framework, task graph solver (NTS), consists of many approximation steps and representations: CNN to capture environment states, task graph parameterization, logical operator approximation; the idea of reward-propagation policy helps pre-training. The framework is evaluated on a relevant multi-task problem.

In general, the paper proposes an idea to tackle an interesting problem. It is well written, the idea is well articulated and presented. The idea to represent task graphs are quite interesting. However it looks like the task graph itself is still simple and has limited representation power. Specifically, it poses just little constraints and presents no stochasticity (options result in stochastic outcomes).

The method is evaluated in one experiment with many different settings. The task itself is not too complex which involves 10 objects, and a small set of deterministic options. It might be only complex when the number of dependency layer is large. However, it's still more convinced if the paper method is demonstrated in more domains.


About the description of problem statement in Section 3:

- How the MDP M and options are defined, e.g. transition functions, are tochastic?

- What is the objective of the problem in section 3

Related work: many related work in robotics community on the topic of task and motion planning (checkout papers in RSS, ICRA, IJRR, etc.) should also be discussed.
review_title: Nice idea

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes to train recursive neural network on subtask graphs in order to execute a series of tasks in the right order, as is described by the subtask graph's dependencies. Each subtask execution is represented by a (non-learned) option. Reward shaping allows the proposed model to outperform simpler baselines, and experiments show the model generalizes to unseen graphs.

While this paper is as far as I can tell novel in how it does what it does, the authors have failed to convey to me why this direction of research is relevant.
- We know finding options is the hard part about options
- We already have good algorithms that take subtask graphs and execute them in the right order from the planning litterature

An interesting avenue would be if the subtask graphs were instead containing some level of uncertainty, or representing stochasticity, or anything that more traditional methods are unable to deal with efficiently, then I would see a justification for the use of neural networks. Alternatively, if the subtask graphs were learned instead of given, that would open the door to scaling an general learning. Yet, this is not discussed in the paper.

Another interesting avenue would be to learn the options associated with each task, possibly using the information from the recursive neural networks to help learn these options.


The proposed algorithm relies on fairly involved reward shaping, in that it is a very strong signal of supervision on what the next action should be. Additionaly, it's not clear why learning seems to completely "fail" without the pre-trained policy. The justification given is that it is "to address the difficulty of training due to the complex nature of the problem" but this is not really satisfying as the problems are not that hard. This also makes me question the generality of the approach since the pre-trained policy is rather simple while still providing an apparently strong score.


In your experiments, you do not compare with any state-of-the-art RL or hierarchical RL algorithm on your domain, and use a new domain which has no previous point of reference. It it thus hard to properly evaluate your method against other proposed methods.

What the authors propose is a simple idea, everything is very clearly explained, the experiments are somewhat lacking but at least show an improvement over more a naive approach, however, due to its simplicity, I do not think that this paper is relevant for the ICLR conference. 

Comments:
- It is weird to use both a discount factor \gamma *and* a per-step penalty. While not disallowed by theory, doing both is redundant because they enforce the same mechanism.
- It seems weird that the smoothed logical AND/OR functions do not depend on the number of inputs; that is unless there are always 3 inputs (but it is not explained why; logical functions are usually formalised as functions of 2 inputs) as suggested by Fig 3.
- It does not seem clear how the whole training is actually performed (beyond the pre-training policy). The part about the actor-critic learning seems to lack many elements (whole architecture training? why is the policy a sum of "p^{cost}" and "p^{reward}"? is there a replay memory? How are the samples gathered?). (On the positive side, the appendix provides some interesting details on the tasks generations to understand the experiments.)
- The experiments cover different settings with different task difficulties. However, only one type of tasks is used. It would be good to motivate (in addition to the paragraph in the intro) the cases where using the algorithm described in the paper may be (or not?) the only viable option and/or compare it to other algorithms. Even tough not mandatory, it would also be a clear good addition to also demonstrate more convincing experiments in a different setting.
- "The episode length (time budget) was randomly set for each episode in a range such that 60% − 80% of subtasks are executed on average for both training and testing." --> this does not seem very precise: under what policy is the 60-80% defined? Is the time budget different for each new generated environment?
- why wait until exactly 120 epochs for NTS-RProp before fine-tuning with actor-critic? It seems that much less would be sufficient from figure 4?
- In the table 1 caption, it is written "same graph structure with training set" --> do you mean "same graph structure than the training set"?
review_title: Review

Meta-review: Paper presents and interesting new direction, but the evaluation leaves many questions open, and situation with respect to state of the art is lacking
Decision: Reject