Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a method to embed priors for motor functions to artificial neural networks. The authors demonstrate a rather simple example of using architecture of C.elegans to control swimmer in mujoco control suite problem. The proposed method has similarities with previous methods such as CPGs, but it has clear differences as the authors explain. Using the architecture proposed by the authors, the controller for the 5 link swimmer is reduced to 4 parameters. The training results show pretty much instant learning curve with converged policies comparable to MLPs trained with Reinforcement Learning or Evolutionary Strategies. 
strengths_and_weaknesses: The main strength of the paper is to add prior knowledge and structure to artificial neural networks for motor controlling tasks. This is a very important point for reinforcement learning and robotics. It reduces the number of samples required for learning and it introduces interpretability to the network. 

As one would expect, the results show that the same task can be learned with way less parameters in much smaller number of samples. Adding structure also provides some bounds to safety during learning, but the authors did not emphasize this point much.

The paper is very well written. The concepts and similar works are explained in details in a nice flow. It is very easy to follow the main idea while also understanding the details of the method.

In terms of weaknesses, as the authors acknowledge the authors demonstrate the method on a rather simple problem (swimmer), and they do not give a clear guideline to extend it to more complex motor control problems such as legged locomotion. 
questions: Would it be possible to give some guidelines on how to apply the given design principles to more complex problems such as legged locomotion (or even 2D walker problem)?
limitations: The main limitation is the lack of scalability of the method on more complex problems. It would be great to see authors give different architectures for different problems instead of a single one (swimmer).
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors introduce NCAP, a neural network architecture inspired by computational models of actual nematode locomotion. Experiments are done on the "swimmer" environment, which bears some relationship to nematode locomotion. They show that when optimized for forward velocity (via RL or ES), NCAP has much better initial performance than conventional architectures, despite having far fewer numerical parameters. Ablations reveal than the sign constraints in particular are crucial for performance. Finally, the authors show that training transfers to bodies with a different number of joints.
strengths_and_weaknesses: # Strengths

1) This is a unique perspective. It's rare for work in systems neuroscience to be fully translated into models that can be optimized with methods designed for deep learning.

2) The writing is clear throughout, though more insight into the motivation behind the unit types would be nice. E.g. way are oscillator units preferable to under ways of maintaining internal state (such as LSTMs)?

3) Ablations are thorough, though again more insight into these results would be appreciated.

# Weaknesses

1) It's unclear the extent to which learning matters here. Initial performance looks to be around 800 and final performance is perhaps 850. This is a weakness, because simply hardcoding a solution to a simple locomotion problem is not a significant AI result. I'm not convinced that this'll be useful for anything other than the exact task/body setup you use here.

2) The experimental results are very limited in scope. All of the AI papers mentioned in related works involve techniques being applied across a range of embodiments and tasks. And even within the constraints of the swimmer environment, none of these related works are compared against.

3) Misc. Figure 4 shouldn't use a log x axis (or at least have both versions visible somewhere). Transfer is stated as not possible for MLPs, but ignores work adapting deep learning methods to due just that: "One policy to control them all: Shared modular policies for agent-agnostic control". I'm not convinced that parameter-count is a meaningful metric in this regime, but even if it were "Weight Agnostic Neural Networks" could likely produce a single-parameter policy class with similar performance properties to NCAP.

#Takeaway

I would strongly encourage resubmitting after more general results are obtained. Alternatively, I think this work could be reframed as a contribution to neuroscience rather than AI, by emphasizing the testing the functional role of various nematode cells, etc and downplaying the potential utility in AI applications. Though this would still require considerable reworking and would likely be better served by a different publication venue.
questions: 1) Would NCAP work when optimizing a different reward function, like reaching a goal location?

2) Would NCAP work for non-swimmer environments? Particularly those with similarly structured action spaces (joint accelerations).

3) Why should we care about the parameter counts of small models independent of data efficiency? Even the largest model considered here isn't bottlenecked by memory or compute, and it's unclear these gains would hold for larger models. In particular, why is the reward/params metric in Figure 5 meaningful?

4) What do the transfer results (Figure 6) look like for a fresh initialized NCAP agent (i.e. no training)?
limitations: Yes.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper introduces a new neural network architecture, "NCAP" (Neural Circuit Architectural Priors), for performing a locomotion task (the Swimmer task from the DeepMind Control Suite).  The network is inspired by the locomotion circuits of C. elegans.  The authors train both NCAP, as well as various MLP architectures to perform the Swimmer task, and do so using 3 different RL training methods (PPO, DDPG and ES, an evolutionary black-box optimization method).  The NCAP architecture consistently achieves higher average reward earlier in training compared to the MLP architectures, while having significantly fewer trainable parameters. 
strengths_and_weaknesses: Strengths:
1. Originality: despite the connectome of C. elegans having been available for years, I don't think there are many papers trying to analyze the key components of the anatomy of C. elegans that enable these worms to learn efficiently, as well as the implications of the C. elegans connectome for building AI systems.  In this sense, the paper tackles interesting, under-explored questions
2.  Quality: the paper is relatively well written.  The authors are thorough in their review of Related Work, and the figures - especially the early figures that give the overview of their framework and the nematode - are clear and easy to understand.

Weaknesses: 
1. Clarity: Please clarify what the state at time $t$, $s_{t}$, corresponds to on line 240.  You give the inputs to NCAP in the equations of Section 3, but please clarify the exact inputs you are giving to the MLP comparison networks.  Is $s_{t}$ for the MLP networks just $\{o, q_{1}, q_{2} ,..., q_{N}\}$ or does this also include the speed $s$ and $r$ and $l$?  From Appendix B, it looks like you don't give the speed $s$ and $r$ and $l$ to the MLP.  
2. Clarity/quality: It seems strange to me that speed is given as an input to NCAP, given that reward is based on speed.  This is especially true if you only give speed as an input to NCAP and not to the MLP networks.  To what extent can the early success of NCAP relative to the MLP networks be attributed to the fact that speed is given as an input to NCAP?  It would have been nice to see an ablation study where the authors removed the speed input in order to get a sense of how crucial this input is.  Furthermore, it would nice to have been able to see an expanded x-axis for Figure 4A so as to know when NCAP performance first reached the avg reward of 800 (was NCAP achieving this reward from time=0 or time=$10^{4}$?)
3. Significance: while the results of NCAP on the Swimmer task are impressive, I am not sure of the broader significance of this paper.  This is especially true because of the fact that it appears that the asymptotic performance of NCAP is lower than that of some MLP architectures (Figure 4).  Are there other tasks where the NCAP architecture can be applied?  Or is the take-away message from this paper that sign constraints on weights in neural networks are important, and enable faster learning?  I am excited to hear what the authors have to say on this point: what do you think the broader implications of your paper are - beyond the Swimmer task?    

Minor points:
1.  Clarity: Are there not 6 parameters for NCAP?  $w_{prop}, w_{speed}, w_{turn}, w_{osc}, w_{ipsi}, w_{contra}$?  In Table 5, you say that there are 4 parameters
4. Clarity: there are many constrained parameters and outputs for the NCAP network.  While you clarify how weights are appropriately constrained to have the desired signs, I wasn't clear on how you constrain the output of the network to be in [-1, 1] (line 214).

Extremely minor point:
1. Quality (I only include this here because I noticed it when I zoomed into the figure and I would want to correct it if this was my paper): the x tick labels on Figure 6 are not vertically aligned
questions: My questions were outlined above, but I repeat these here for completeness:

1. Clarity: Please clarify what the state at time $t$, $s_{t}$, corresponds to on line 240.  You give the inputs to NCAP in the equations of Section 3, but please clarify the exact inputs you are giving to the MLP comparison networks.  Is $s_{t}$ for the MLP networks just $\{o, q_{1}, q_{2} ,..., q_{N}\}$ or does this also include the speed $s$ and $r$ and $l$?  From Appendix B, it looks like you don't give the speed $s$ and $r$ and $l$ to the MLP.  
2. Clarity/quality: It seems strange to me that speed is given as an input to NCAP, given that reward is based on speed.  This is especially true if you only give speed as an input to NCAP and not to the MLP networks.  To what extent can the early success of NCAP relative to the MLP networks be attributed to the fact that speed is given as an input to NCAP?  It would have been nice to see an ablation study where the authors removed the speed input in order to get a sense of how crucial this input is.  Furthermore, it would nice to have been able to see an expanded x-axis for Figure 4A so as to know when NCAP performance first reached the avg reward of 800 (was NCAP achieving this reward from time=0 or time=$10^{4}$?)
3. Significance: while the results of NCAP on the Swimmer task are impressive, I am not sure of the broader significance of this paper.  This is especially true because of the fact that it appears that the asymptotic performance of NCAP is lower than that of some MLP architectures (Figure 4).  Are there other tasks where the NCAP architecture can be applied?  Or is the take-away message from this paper that sign constraints on weights in neural networks are important, and enable faster learning?  I am excited to hear what the authors have to say on this point: what do you think the broader implications of your paper are - beyond the Swimmer task?    
limitations: The Discussion section for this paper was a little lackluster, and the authors could have been more explicit about the limitations of their work. For example, one limitation is that you've only demonstrated the utility of the NCAP architecture on the Swimmer task.  Another limitation is that the asymptotic performance of NCAP seems to be lower than that of the MLP architectures (Figure 4).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: 
Neural Circuit Architectural Priors for Embodied Control
In this paper, the authors are inspired by the Caenorhabditis elegans locomotion circuits, and design a structurally similar neural network structure. This prior structure provides valuable starting motion control knowledge and enables faster training in the task of the swimmer from the deepmind control suite.

strengths_and_weaknesses: 
Originality: This paper draws a very interesting connection between neuroscience and reinforcement learning. Since I have no previous knowledge in biology or neuroscience, I cannot comment on how novel this idea is. From the reinforcement learning point of view, I think it’s pretty novel and inspiring.

Quality and clarity: The paper is well written. The related work is pretty adequate, and the figures in the paper (for example Figure 1 and Figure 2) are pretty intuitive and easy to understand.
The experiments are well designed, with every experiment having clear purposes and explanations. 

Significance: How well does it scale to high-dimensional agents? How well does it scale to other tasks? I think these are some of the questions that would greatly affect how significant the proposed algorithm is in the community of reinforcement learning (I cannot comment on the neuroscience part unfortunately).
Without experiments on more creatures (for example the ant like creatures, the fish like creatures in the deepmind control suite), or more experiments for different tasks that aim to show the locomotion skills not only on swimming but also on reaching, avoiding, and even 3D swimming, the significance of the algorithm remains unclear.

questions: 1) From a neuroscience’s perspective, maybe the authors can comment on the connection between the proposed algorithm and some other neuro-inspired network structures? For example the feedback alignment network in [3], [4] and [5]
2) Back in the “Strengths And Weaknesses”, I mentioned the lack of experiments. Could the authors provide some comments on whether or not the algorithm can be applied to other creatures and other tasks?
3) The periodic phase generation module, while not exactly the same, was proposed and used in some character animation research such as [1], [2]. These are some literature I think is related to the oscillator units discussed in the paper.

limitations: 
I don’t see any potential negative societal impact in this work.

[1] Van de Panne, Michiel, Ryan Kim, and Eugene Fiume. "Virtual wind-up toys for animation." In Graphics Interface, pp. 208-208. CANADIAN INFORMATION PROCESSING SOCIETY, 1994.
[2] Holden, Daniel, Taku Komura, and Jun Saito. "Phase-functioned neural networks for character control." ACM Transactions on Graphics (TOG) 36, no. 4 (2017): 1-13.
[3] Lillicrap, Timothy P., Daniel Cownden, Douglas B. Tweed, and Colin J. Akerman. "Random synaptic feedback weights support error backpropagation for deep learning." Nature communications 7, no. 1 (2016): 1-10.
[4] Nøkland, Arild. "Direct feedback alignment provides learning in deep neural networks." Advances in neural information processing systems 29 (2016).
[5] Frenkel, Charlotte, Martin Lefebvre, and David Bol. "Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks." Frontiers in neuroscience 15 (2021): 629892.
review_title: null

Meta-review: metareview: This paper introduces the use of neural circuit architectural priors to build controllers for a physically simulated c-elegans-like swimmer implemented in MuJoCo as part of the DeepMind control suite. By leveraging the bio-inspired architectural priors, the controller starts with structured behavior (rather than highly erratic random movements as is commonly the starting point for embodied RL initial behavior). And the architectural prior supports continued learning from this starting point.

The work is seen as original, interesting, and quite clear. The work is also nicely self-contained.

That said, this paper has received mixed and borderline reviews (6, 4, 4, 6), and there were some concerns about scalability and utility to the AI community.  This paper was discussed with the SAC, and we decided that despite some of these legitimate concerns, this paper should be accepted. This paper has clear goals and can help us rethink some of our approaches to architectures.  Moreover the potential audience spans both neuroscience and AI. 

We (the AC and SAC) still highly encourage you to seriously consider comments from the reviewers. From both the positive-leaning and negative-leaning reviewers, there is respect for what was done as a work of modeling, but concerns about whether this constitutes only well-done computational modeling, or if it really amounts to anything that could be useful for AI more generally (and if it could scale to other bodies). You outlined some next steps, and how similar approaches could be used in other scenarios and with more complex bodies; we recommend that you include that discussion in this paper. We'd also strongly encourage you to avoid assertive claims about how neuroscience-inspired ideas can generally improve AI systems, and acknowledge limitations in this case study. While this case study is a provocative first step, the reviewers and AC tend to believe it will prove quite difficult to extend this strategy to more complex bodies.

Overall, focusing on what was achieved in this paper, nice work.
confidence: Less certain
Decision: Accept