Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors present an interesting application of Graph Neural Networks to learning policies for controlling "centipede" robots of different lengths. They leverage the non-parametric nature of graph neural networks to show that their approach is capable of transferring policies to different robots more quickly than other approaches. The significance of this work is in its application of GNNs to a potentially practical problem in the robotics domain. The paper suffers from some clarity/presentation issues that will need to be improved. Ultimately, the contribution of this paper is rather specific, yet the authors show the clear advantage of their technique for improved performance and transfer learning on some agent types within this domain.

Some comments:
- Significant: A brief statement of the paper's "contributions" is also needed; it is unclear at first glance what portions of the work are the authors' own contributions versus prior work, particularly in the section describing the GNN theory.
- Abstract: I take issue with the phrase "are significantly better than policies learned by other models", since this is not universally true. While there is a clear benefit to their technique for the centipede and snake models, the performance on the other agents is mostly comparable, rather than "significantly better"; this should be reflected in the abstract.
- Figure 1 is instructive, but another figure is needed to better illustrate the algorithm (including how the state of the world is mapped to the graph state h, how these "message" are passed between nodes, and how the final graph states are used to develop a policy). This would greatly help clarity, particularly for those who have not seen GNNs before, and would make the paper more self-contained and easier to follow. The figure could also include some annotated examples of the input spaces of the different joints, etc. Relatedly, Sec. 2.2.2 is rather difficult to follow because of the lack of a figure or concrete example (an example might help the reader understand the procedure without having to develop an intuition for GNNs).
- There is almost certainly a typo in Eq. (4), since it does not contain the aggregated message \bar{m}_u^t.

Smaller issues / typos:
- Abstract: please spell out spell out multi-layer perceptrons (MLP).
- Sec 2.2: "servers" should be "serves"
- "performance By" on page 4 is missing a "."

Pros:
- The paper presents an interesting application of GNNs to the space of reinforcement learning and clearly show the benefits of their approach for the specific task of transfer learning.
- To the best of my knowledge, the paper presents an original result and presents a good-faith effort to compare to existing, alternative systems (showing that they outperform on the tasks of interest).

Cons:
- The contributions of the paper should be more clearly stated (see comment above).
- The section describing their approach is not "self contained" and is difficult for an unlearned reader to follow.
- The problem the authors have chosen to tackle is perhaps a bit "specific", since the performance of their approach is only really shown to exceed the performance on agents, like centipedes or snakes, which have this "modular" quality.

I certainly hope the authors improve the quality of the theory section; the poor presentation here brings down the rest of the paper, which is otherwise an easy read.
review_title: Though occasionally unclear, the authors present an interesting approach to solving a well-scoped problem.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The submission proposes incorporation of additional structure into reinforcement learning problems. In particular, the structure of the agent's morphology. The policy is represented as a graph neural network over the agent's morphology graph and message passing is used to update individual actions per joint.

The exposition is fairly clear and the method is well-motivated. I see no issues with the mathematical correctness of the claims made in the paper. However, the paper could benefit from being shorter by moving some details to the appendix (such as much of section 2.1 and PPO description).

Related work section could consider the following papers:

"Discrete Sequential Prediction of Continuous Actions for Deep RL"
Another approach that outputs actions per joint, although in a general manner that does not require morphology structure

"Generalized Biped Walking Control"
Considers the task of interactively changing limb lengths (your size transfer task) in a zero-shot manner, albeit with a non-neural network controller

The experimental results investigate the effects of various algorithm parameters, which is appreciated. However, a wider range of experiments would have been helpful to judge the usefulness of the proposed policy representation. In addition to robustness to limb length and disability perturbations, it would have been very nice to see multi-task learning that takes advantage of body structure (such as learning to reach for target with arms while walking with legs and being able to learn those independently, for example).

However, I do think using agent morphology is an under-explored idea and one that is general, since we tend to have access to this structure in continuous control tasks for the time being. As a result, I believe this submission would be of interest to ICLR community.
review_title: Investigates an under-explored idea, but evaluation could be more compelling

final_score: rating: 7: Good paper, accept
review_content: This paper proposes NerveNet to represent and learn structured policy for continuous control tasks. Instead of using the widely adopted fully connected MLP, this paper uses Graph Neural Networks to learn a structured controller for various MuJoco environments. It shows that this structured controller can be easily transferred to different tasks or dramatically speed up the fine-tuning of transfer.

The idea to build structured policy is novel for continuous control tasks. It is an exciting direction since there are inherent structures that should be exploited in many control tasks, especially for locomotion. This paper explores this less-studied area and demonstrates promising results.

The presentation is mostly clear. Here are some questions and a list of minor suggestions:
1) In the Output Model section, I am not sure how the controller is shared. It first says that "Nodes with the same node type should share the instance of MLP", which means all the "joint" nodes should share the same controller. But later it says "Two LeftHip should have a shared controller." What about RightHip? or Ankle? They all belongs to the same node type "joint". Am I missing something here? It seems that in this paper, weights sharing is an essential part of the structured policy, it would be great if it can be described in more details.

2) In States Update of Propagation Model Section, it is not clear how the aggregated message is used in eq. (4).

3) Typo in Caption of Table 1: CentipedeFour not CentipedeSix.

4) If we just use MLP but share weights among joints (e.g. the weights from observation to action of all the LeftHips are constrained to be same), how would it compare to the method proposed in this paper?

In summary, I think that it is worthwhile to develop structured representation of policies for control tasks. It is analogue to use CNN that share weights between kernels for computer vision tasks. I believe that this paper could inspire many follow-up work. For this reason, I would recommend accepting this paper.
review_title: A nice paper that learns structured policy for control

Meta-review: An interesting application of graph neural networks to robotics. The body of a robot is represented as a graph, and the agent’s policy is defined using a graph neural network (GNNs/GCNs) over the graph structure.

The GNN-based policy network perform on par with best methods on traditional benchmarks, but shown to be very effective for transfer scenarios: changing robot size or disabling its components.  I believe that the reviewers' concern that the original experiments focused solely on centepedes and snakes were (at least partially) addressed in the author response: they showed that their GNN-based model outperforms MLPs on a dataset of 2D walkers.

Overall:
-- an interesting application
-- modeling robot morphology is an under-explored direction
-- the paper is  well written
-- experiments are sufficiently convincing (esp. after addressing the concerns re diversity and robustness).


Decision: Accept (Poster)