Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper studies some of the optimization challenges with training large graph neural networks for reinforcement learning tasks, in particular locomotion control tasks. The paper proposes that one of the larger challenges in optimizing graph neural networks is the significant variance between updates across the network graph. The method proposes to randomly freeze nodes and components of nodes in the graph network. The freezing of these different blocks inside of the graph Network helps reduce the effects of large changes in the output of the network making the overall optimization less noisy and more stable resulting in better performance.

Pros
- The analysis in the paper does appear to be rather thorough. Especially going into the specifics about the difficulties in training neural networks for this particular type of problem. Also at the same time showing some of the limitations of training graph neural networks for example nerve net and its difficulty to perform well on larger agent morphologies.
- The paper then proposes a random perimeter freezing algorithm that increases the overall learning performance and final policy quality of the nerve net like design.
- Along with the performance gain the method is still able to show significant transfer capabilities across some different morphologies.

Cons
- One of the more subtle challenges in the contribution of this paper is that the method overall is somewhat simplistic. It could be thought of as performing a type of block dropout while training the network in order to reduce the amount of parameter change when making policy updates.
- One of the other more nuanced difficulties in the contribution of the paper is that it focuses on training graph neural networks for locomotion tasks on reinforcement learning. This could be seen as a somewhat specific area of research that may not have a more broad impact on the community.
- Related to the last point it would be helpful to compare this method to the amorphous work. That work calls out the challenges with training and the optimization of graph neural networks that may limit their overall application. In this new work proposed by the authors shows they've now been able to overcome some of the difficulties in training such that it may perform better than the amorphous algorithm as well.

main_review: Additional comments on the work:
- In figure 5 it shows that setting the learning rate to practically zero for the message part of the CNN leads to the highest final average award. This also seems to be corroborated in the recent paper Amorpeuos [1], which uses Transformers to train morphological type models. Does this indicate that the training of GNNs especially the message passing system makes the overall optimization complex and could indicate that GNNs are also generally limited by the difficulty of their optimization?
- [1] This reference should be fixed in the paper to reference the proper version of the paper that has been published at ICLR 2021.
- The paper mentions the Amorpeuos prior work that uses a transformer to represent locomotion policies yet it does not compared to in the paper. In order to get a broader picture of how this method compares to the most recent state-of-the-art, it would be important to compare to this structure. Especially since one of the main points in the Amorpeuos paper was that training GNNs, especially the message passing system, is too complex in that the benefits of GNNs will be hindered by the difficulties of this optimization. However, the method proposed in this paper should be addressing these optimization limitations and therefore should perform better than the Amorpheus work.
- The analysis in figure eight with respect to how the policy KL Divergence increases over training is somewhat counterintuitive. PPO on its own does learn to adjust the policy distribution over time and as the policy should converge to at least a local optimum one would think the KL Divergence would also shrink yet in these graphs we don't see a downward trend later in learning. Does this imply that the policy is not finished learning and there is still room to learn a more optimal policy?



Refernces:
[1] Kurin, V., Igl, M., Rocktaschel, T., Boehmer, W., & Whiteson, S. (2021, April). My body is a cage: the role of morphology in graph− based incompatible control. In Proceedings of the International Conference on Learning Representations. OpenReview.
limitations_and_societal_impact: To increase the contrubution of the paper it will be important to include comparisons to other prior works, like Amorpeuos.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper considers training RL agents by using graph neural networks (GNNs) as the function approximator, therefore inducing a useful inductive bias. Training GNNs is not without its hurdles and thus far previous work has focused on relatively small agents, putting into question scalability. This work attempts to address this issue by propposing parameter freezing in parts of the GNN. The hypothesis for this is that policy updates become unstable with GNNs, to which authors provide empirical evidence. The experiments also show better performance in more complex agents and zero-shot transfer than the NerveNet baseline.
main_review: The intersection of GNNs and RL is an interesting and promising field of study with many applications where a graph structure can be leveraged. One such way is to consider the graph formed by the agent's anatomy, which can then be used for better transfer as shown by the authors. As GNNs become hard to train with scaling agents, the paper proposes a perhaps surprising way to fix this issue: to freeze some parts of the GNN. 

The paper reads nicely and the motivation for freezing the weights is well defined, at least for the case when PPO is used. It is however not clear if this method would work or is relevant for the case then another algorithm is used, such as A2C or DDPG. What should we expect for such cases? 


Another potential downside is that the transfer experiments focused exclusively on the centipede-n domain. Although it is a challenging one, it is also a limited setting. However, throughout the paper the authors refer to the "transfer properties" of GNNs, which is quite a general claim (for example in L147, where there is also no citation backing up the claim) . Such a claim would need to be backed up by experimenting on different domains, or by being more specific as to the scope of the claim.

Also, do the authors have any hypothesis as to why the KL divergence (L156) is higher when using GNNs? The idea of overfitting is an interesting one, but then again why would GNNs overfit? Perhaps L183 should be expanded a bit to give this important insight. 

Figure 5 gives us a plot of the performance with respect to changes in individual parts of the GNNs (i.e. freeze one part at a time). However Snowflake freezes multiple parts of the GNNs at once. Is there any empirical evidence as to why this is done?

When freezing the message function, the authors acknowledge that this is very similar (at least in terms of performance) to using an identity function. The structure of the GNN then becomes more close to Graph Convolutional Networks [1]. Have the authors tried using GCN as a baseline?  In the same direction, it would be very informative to have an ablation or a baseline where instead of freezing the encoder and/or decoder, an identity function is used instead of a random network.

Concerning related work, the graph-based RL section could benefit from two relevant works [2,3] where GNNs are used in two quite different manners. In [2] the authors use GNNs for planning by doing more of a bread-first search approximation to the graph, where as in [3] the authors use GNNs for reward shaping by doing more of a depth-first search approximation to the graph. Adding these two references would highlight the flexibility with which GNNs can be combined with RL.

More of a minor comment, but in the background section of RL, I believe some standard references could be added as well as improving the notation (for example usually the expectation is usually explicit with respect to the policy $\pi$, as well as the value function and advantage function).

Finally, the number of runs is 6 form what I gathered in the appendix. I believe that for MuJoCo this could be easily pushed to 10 in order to provide more credibility, which is important in the current state of DRL.

[1] Kipf and Welling, Semi-Supervised Classification with Graph Convolutional Networks
[2] Deac et al., XLVIN: eXecuted Latent Value Iteration Nets
[3] Klissarov et al., Reward Propagation using Graph Convolutional Networks





=================Post-rebuttal===================

I appreciate the authors rebuttal. I think this paper is a solid contribution, and the authors rebuttal has added to this feeling. For this reason I will keep my score to accept.
limitations_and_societal_impact: The societal impact is discussed in the appendix, but in this context is not particularly significant.

For the limitations, the authors mention that in section 5 it is discussed how their approach is hard to scale beyond centipede-20. Is it possible to provide the wall clock performance across different configurations and compare it to an MLP agent?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a framework to address the challenge of scaling the application of graph neural network (GNN) based policies to locomotion challenges in reinforcement learning. The authors' hypothesize that the scaling challenges of GNN policies relate to overfitting of internal networks within the GNN architecture during RL training and unstable policy updates during on-policy RL training using PPO. The authors then introduce their framework, named Snowflake, to address this challenge which freezes distinct parts of the GNN during training to prevent overfitting and unstable policy updates.
The authors demonstrate their framework by applying NerveNet, a GNN based policy network, to the centipede environment in the Gym + Mujoco suite. In their experiments and subsequent analysis, the authors show supporting evidence for their claim that NerveNet doesn't scale to large locomotion challenges and that the policy updates are unstable as suggested by large KL divergence between the updates when compared to regular MLP policy updates. Subsequently, the authors show how common overfitting mitigations fall short of significantly improving the performance of NerveNet in the centipede setting and provide supporting evidence that the Snowflake increases the performance of NerveNet to match regular MLP policy performance in a variety of Mujoco tasks, including various sizes in the centipede environment. The authors also claim, and support that claim, that GNN based policies, such as NerveNet, provide better generalization abilities compared to MLP based policies.
main_review: Nits:
* the checklist should start on a separate page

Overall, I thought the paper outlined their problem description, hypothesis and supporting experimental evidence pretty well. The paper provided data for their initial analysis of why GNN policies have trouble scaling to larger locomotion tasks, which is improvement based on the ICML reviews, and then show that their method improves the performance of NerveNet on various Mujoco tasks.

**Originality:** The paper proposes a novel method to address an important challenge in applying GNN based architectures to RL settings. The method can be generally applied to all GNN based architectures that are used in RL settings and potentially in other supervised learning settings.

**Quality:** The paper outlines their hypothesis and the underlying well and provides supporting evidence for their analysis of the problem, such as KL policy divergence and experiments on common regularization techniques. The experiments and results when using the method generally the method generally support the authors' claims. To further improve the paper, it would be good to clarity on a couple of items:
* How does the Snowflake choose which parts of the GNN to freeze and which parts to train? Does the method enable different learning rates across the network for the unfrozen layers (I think the authors claim it does)? If so, how are those chosen? Does applying Snowflake basically increase the space of hyperparameters in the algorithm or is there a way to systemically determine the additional choices?
* Do you expect the method to perform similarly on off-policy RL methods, such as TD3 and SAC? Why or why not?
* A discussion on the limitations of the method

**Clarity:** The paper was generally well written and provides helpful presentation of the data. Further improvements in clarity can be made by having a diagram (or algorithm block) of how the Snowflake method chooses what to freeze as also mentioned above.

**Significance:** The results in the paper indicate that Snowflake can help the adoption of GNN policies to RL problems, which would enable solving more and more complex RL challenges with greater generalization properties. 
limitations_and_societal_impact: The discussion on potential societal impacts, data generation and computation infrastructure in the appendix is pretty thorough. More details on limitations and potential future work of the method would make the paper stronger.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: * The authors study a previously published method for learning locomotion in 3D agents, called NerveNet, which models the agent as a graph of joints linked by limbs and uses a Graph Neural Network (GNN) mechanism to compute joint forces. Previous work has shown that this method transfers better than MLPs across morphologies, but also scales poorly to larger agents.  

* The authors identify a problem that they suggest causes this poor scaling - namely, instability of policy updates, even at performance-optimal choices of hyperparameters. They choose to call this "overfitting" - a contentious choice no doubt, but why not. They show that a L2 regularization penalty on weights alleviates the problem a bit, but not enough. 

* Crucially, they also discover that reducing learning rates to zero for the encoding and decoding components of the model (i.e. using random frozen embeddings) considerably improves performance. 

* Somehow they decide to give a name to the fact of using frozen random embeddings, calling it "Snowflake". 

* They compare this method to standard NerveNet and MLPs on various locomotion tasks. They show that "Snowflake", unlike vanilla NerveNet, performs about as well as MLPs even on large morphologies (after enough training). Importantly, they also show that "Snowflake" trained on a given morphology transfers even better to a smaller morphology than NerveNet (and that MLPs simply can't do that at all).

* In short, the contribution of this paper (as I understand it) is an identification of a source for poor NerveNet performance, a solution to this problem (just use random frozen embeddings), and a measurement of the improved performance (matches MLPs in performance, considerably beats them in transfer).
main_review: The paper is interesting and convincing. My main question is about significance: does the contribution (improving a previously known technique) warrant a NeurIPS presentation?

I suppose it could, given the strength of the performance improvement in comparison to MLP (largely closing the gaps), the remarkable transfer results, and the possibility that this finding might apply to other GNN methods (which are evidently a hot topic right now). 

Conversely, I'm a bit puzzled by the decision to coin a new name for simply using frozen random embeddings. Is it really warranted? 

In the end, although I like the paper, I could see the decision go either way.

Separate from this overall uncertainty about significance, I have various clarity concerns that should be easily fixable.

1- In the (nice and useful) description of the GNN method, the notation makes it look like the inputs and outputs v_v and v'_v have the same dimensionality, and that the v' replace the v for the next step (since that's what happens between theta' and theta in the description of PPO that precedes it) ?? Surely that is not the case - it seems to me that the "input labels" v_v are fixed, unchanging descriptors of the vertices, while the "output labels" are basically just arbitrary outputs with little connection to the labels? If so this should be clarified (and perhaps the notation could be tweaked a bit).

2- Importantly, unless I missed it, the exact inputs are not clearly described.  In p.4, l. 129, we are told that they consist of "positional information" about joints and body parts. What does this information look like? The precise coding of inputs might have a great impact on performance and especially on generalization across body sizes. For example, is the "position" along the body axis fed as a simple natural integer from 0 to n/2, or is it a fractional number between 0 and 1, normalized to the total length n/2 of the body? Clearly the latter might help any architecture generalize to various lengths. Therefore, I believe a clear description of input format and information should be in the main text. 

3- Relatedly, the same line mentions body parts (the edges of the network), but in section 2.4 it seems that only joint (vertices) descriptors are used? Could this be clarified?

4- Supposedly, in the experimental comparisons, MLPs have a roughly similar number of trainable parameters to NervNet/Snowflake? Can this be confirmed? In addition, it would be nice to mention how much computational cost the method incurs in comparison to MLPs (see below).


limitations_and_societal_impact: Unless I missed it (very possible), it would be useful to have a brief mention of computational costs. Is NerveNet/Snowflake slower or faster than MLPs (with similar number of trainable parameters) *per timestep*, for the particular settings used in Section 5? Even a single sentence or two would be enough.

Societal impacts do not seem applicable here.
review_title: null

Meta-review: There was quite an extensive discussion between the reviewers on this paper after the author response. There is no doubt that the the paper proposes a simple (in a good way) fix to train GNNs with PPO. So it makes a valuable contribution to the community. There was also agreement that the paper could potentially be made a lot stronger with additional work, e.g., the paper is mainly based on empirical evidence for the proposed method so evaluating freezing part by part (reviewer PQSJ) would be very interesting, more comparisons to other work, etc. The main difference was in the weighting of these two aspects.
Decision: Accept (Poster)