Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper addresses a really important challenge: To understand innate contributions to neural circuits for motor control.
Its goal is to present a set of reusable architectural components and design principles for embodied control. 
They show that a resulting model can learn to swim more efficiently and requires fewer parameters while achieving similar accuracy as an MLP.

main_review: Unfortunately they only consider this one swimmer task, and the architecture was apparently designed with this task in mind. Furthermore this is not a very challenging control tas
It seems to me that their hand-crafted architecture (figure 3D) does most of the heavy lifting to solve this task, as can be seen by figure 5A, where the blue and the brown line do not require training at all to have a decent performance.
One interesting aspect of biological circuits is that synaptic weights cannot flip sign during training. But it is not clear whether their neurons are excitatory or inhibitory, like in biological circuits. This requires in addition that all weights from a neuron to other neurons have the same sign. Figure 5A shows that only the sign constraint really matters (a lot actually, but perhaps only because they don't allow negative weight initializations without this constraint) undermining their design principles from section 3.2.



summary_of_the_review: I do like the direction of research, as it tries to take inspiration from biology to improve ML 
However, the way I see it, what they really have is a nice model for a really specific task but not a lot beyond that.
The paper did not convince me that their design principles would be beneficial for other tasks, 
It is also not clear what exactly the principles are that make this single task work well.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a set of architectural components and design principles for constructing neural networks for embodied control. The proposed method is biologically inspired, and can be used to build neural networks specifically designed for the considered agent, e.g., the swimmer agent studied in the paper. The paper provides a literature review, describes the method, and evaluates it in an ablation study using the DDPG algorithm.
main_review: The paper investigates a method to introduce architectural priors for neural networks for control. These architectural and design choices need to be strictly adapted to the considered agent. For example, the authors accurately describe how their method has to be used for a Swimmer agent. I appreciate how the authors accurately explain the meaning of each component of their method. I also like the choice of the Swimmer agent as a case study, because it is clear how the method can be used for this use case, and the explanation provided by the authors is clear and informative.

Despite the good quality of writing and the clarity of the explanation, I have serious doubts about the contribution provided by this paper. As also stated by the authors, the proposed method has the limit of being strongly dependent on the knowledge of neuroscience and biology; however, most RL and AI researchers are not experts in these fields. I consider this a major drawback of this method, because it strongly limits the impact of this work. I do not think I would be able to use the proposed method in another scenario. Another drawback of the paper is that the claims about sample-efficiency and sparsity are not supported by any theoretical guarantee. It is fairly intuitive that these properties may hold empirically, but stronger motivations would be preferrable, especially because I think they strongly depend on the correctness of the architectural and design choices.

The experiments are fair results, far from impressive. I understand the importance of the evidence of sample-efficiency due to the big jump start, but I am concerned about the weaker performance compared to some baselines. How can the proposed method, that heavily designs a neural network to fit the considered agent, be outperformed by a “general-purpose” network? I understand that the outperforming network is significantly bigger, but it is not impractically big. In fact, it is a reasonable size that allow efficient forward-backward propagation on regular GPUs.

Overall, for the mentioned reasons, I find this paper to be significantly lower than the acceptance threshold.
summary_of_the_review: I think this paper has significant drawbacks that outweigh its strengths. The proposed method is very difficult to use, and although the authors do a good job explaining the single use case, the paper would greatly benefit from the study of more use cases. Despite the good quality of the writing and the clarity of the explanations, I think this paper should be rejected for the lack of usability of the proposed method, the absence of a theoretical study, and the need for a larger number of use cases.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a set of bio-inspired Neural Circuit Architectural Priors (NCAP) as ingredients for building trainable networks for continuous control. The authors gave one example of translating the C. elegans locomotion circuits to a network using NCAP to control a swimmer agent in a simulated environment. The proposed network trained by deep reinforcement learning methods (DRL) and evolution strategies (ES) showed comparable performance and more efficient computation with fewer parameters.
main_review: Pros

1. The paper is well written, especially the abstract, introduction, and related work sections. The work is correctly motivated since efficient learning is observed in the brain, and inspiration from biology is essential to improve existing learning-based methods.

2. The paper proposes a bio-inspired network trained by DRL methods while using much fewer parameters than an MLP. This shows that architectural priors induced by knowledge from biology can replace the pure-learning approach to some degree when the prior is correctly constructed.

3. The ablation studies performed by the authors demonstrated the effectiveness of the learning on the proposed network for the swimmer agent. It is interesting that the network can still work after learning with random init weights.

Cons

1. The set of NCAP and the network proposed by the paper share many similarities with existing CPG works. It is better to present the proposed approach as a fine-tuning method for CPG networks than as a general solution for robotics. Biological locomotor CPGs have inspired the design of many artificial networks for robot locomotion (for example, [Matsubara, Takamitsu, et al. 2006], [Lele, Ashwin Sanjay, et al. 2020], [Yao, Chenpeng, et al. 2021], [Lopez-Osorio, Pablo, et al. 2021]). These CPG works are based on similar concepts as the set of NCAPs proposed in this paper and explore different ways to fine-tune the CPG network. Though the fully differentiable network proposed in this work is interesting, its novelty and contribution are limited compared to previous works in the field.

2. Since the proposed approach is similar to CPG-based works, they share the same problems as well. First, these approaches are more suitable for producing stable rhythmic patterns in controlled environments, limiting their use in general real-world robot tasks. Second, as already listed in the discussion of the paper, the network's design heavily relies on domain knowledge and cannot be generalized across different tasks.

3. The authors performed experiments only on a 6-link swimmer in a simulated environment. This fails to show: 1) How can the NCAP approach be generalized to other robot tasks with different moving patterns? 2) How can the trained model adapt to more realistic environments compared with MLP-based approaches? The authors need to introduce additional case studies and experiments to prove that the proposed method is general enough to be applied to a broad spectrum of robot tasks.

4. The NCAP approach introduces strong heuristics to the network design, limiting the network's maximum performance. Thus, compared with MLP-based approaches, the NCAP-based network has comparable but poorer performance. The authors may need to think about relaxing the heuristics used in the network to improve the performance and generalizability. For example, the second ablation done by the authors in the ablation studies section might be a good direction by relaxing the excitation/inhibition restriction.

5. Other than fewer parameters, the paper failed to explore other advantages of the NCAP-based network compared with the MLP. One potential advantage can be the generalizability of the network with shared weights on different links of the swimmer. The authors can perform experiments on swimmers with different numbers of links using the same trained weight from the 6-link swimmer. The MLP cannot achieve this since the fully-connected network needs to be trained from scratch whenever input/output dimension changes.

Minor Problems

1. Equation f(x)=max(w, 0) in Fig. 2A should be f(x)=max(z, 0).

2. Fig. 3D is not clear. Different connections and neural units need more detailed legends and captions.

Matsubara, Takamitsu, et al. "Learning CPG-based biped locomotion with a policy gradient method." 2006.

Lele, Ashwin Sanjay, et al. "Learning to Walk: Bio-Mimetic Hexapod Locomotion via Reinforcement-Based Spiking Central Pattern Generation.", 2020.

Yao, Chenpeng, et al. "Humanoid adaptive locomotion control through a bioinspired CPG-based controller." 2021.

Lopez-Osorio, Pablo, et al. "Neuromorphic adaptive spiking CPG towards bio-inspired locomotion of legged robots." 2021.



summary_of_the_review: Overall, the reviewer recommends not accepting the paper. Data efficiency and generalizability are interesting and critical problems for deep learning approaches to robotics. Although the work is correctly motivated, the authors failed to prove that the proposed method is general enough to be applied to a broad spectrum of robot tasks. In addition, the proposed approach shares a lot of similarities with works on central pattern generator (CPG), due to which NCAP has very limited novelty. More detailed issues with the paper are listed in the cons in main review.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes both a set of artificial neural components and interesting principles for producing biologically-inspired neural networks for embodied control. This work aims to be at the intersection between neuroscience and machine learning for improving the design of artificial neural networks as well as improving our understanding of observed biological networks. Various components of biological networks are replicated in their framework, e.g., the balance between excitation and inhibition, intrinsic oscillators, or sparsity. The model is evaluated on C.elegans motion locomotion circuit using different types of training strategies. 

main_review: ### Clarity: 
This paper is well written and the authors clearly explain the motivations, and framework in which they plan to operate. Figure 1, seems a bit unnecessary as there is nothing particularly new and the font size is small and exactly repeats the text. Fig. 3, and 4 are hard to read because of the font size and the organization. We would suggest the authors reorganize the figures for more clarity, which would help with the explanation of the results as well. However, Figure 2 is very clear and the parallel with C.elegans is made clear.


### Strength:
This paper asks an important question that is how to combine knowledge of biological neural circuits and artificial neural networks in a principled way. It uses an interesting example that is C.elegans for which a lot is known about it's neural architecture and principles of locomotion. It uses unconventional neural components to the machine learning community and makes it differentiable, thus possible to train with standard learning methods. 

### Weakness: 
Hierarchical component models for building biologically inspired neural networks are not new. In particular, the entire connectome of C.elegans has been reconstructed, and little of that information is actually replicated (in the prior) in this work. 
Moreover, although the neural architecture is somewhat biologically plausible, the problem of weight sharing is not. It is also true for the learning rules used. Thus the use for understanding biological models is very limited, as it only vaguely replicated the neural structure but does not operate in a biological setting. 

From the experiments, it seems that training does almost nothing to the performance of the model, as show in Fig.3A and 3B. The model initial outperforms competing models but after training it underperforms it. The relevance of the model is thus questioned, it is only the neural structure that implies the performance and not the learning. As a result this type of model would only be useful for extremely simple organisms with few neurons like C.elegans with little learning/genetic encoding ratio. 









his allows us to design small, sparse networks that mirror neural circuits
yet are fully differentiable and therefore compatible with backpropagation-based learning algorithms.
In contrast to previous work on neuromechanical models of movement (Sarma et al., 2018; Boyle
et al., 2012; Izquierdo & Beer, 2015), our models are more abstract, include learned parameters, and
are used to control bodies significantly different from the originals in terms of mechanics, degrees of
freedom, and actuators.

We apply NCAP to translate C. elegans locomotion circuits into a simulated Swimmer agent, which we
train using both reinforcement learning (RL) and evolution strategies (ES). We show that our NCAP
architecture achieves comparable asymptotic performance with fully connected MLP architectures,
while dramatically improving data efficiency and requiring far fewer parameters. We further show
through an ablation analysis that principled excitation/inhibition and initialization play significant
roles in our NCAP architecture.
summary_of_the_review: This paper is interesting but does not proposes a significant improvement to the literature as the gap between the promises made in the motivation and actually delivered work is too wide. 

From a neuroscience point of view, this work does not provide substantial evidence of the importance of the model at either modeling or simulating biological neural systems. 

From a "theoretical" point of view, the model does not provide much advancement to the machine learning community either.
review_title: null

Meta-review: Meta Review for Neural Circuit Architectural Priors for Embodied Control

The motivation of this work is to address an important challenge: To understand innate contributions to neural circuits for motor control. This paper proposes both a set of reusable architectural components and design principles, and also interesting principles for producing biologically-inspired neural networks for embodied control. This work aims to be at the intersection between neuroscience and machine learning for improving the design of artificial neural networks and improving our understanding of observed biological networks. In their model, various components of biological networks are replicated (such as the balance between excitation and inhibition, sparsity, and oscillation). They show that a resulting model, inspired by C.elegans, can learn to swim more efficiently (when evaluated on the Swimmer RL environment) and requires fewer parameters while achieving similar accuracy as an MLP.

Most reviewers, including myself, recognize (and appreciate) the ambition of this work, and are excited at the goal of looking at problems from the perspectives of both system neuroscience and machine learning. The motivations of this paper are clearly explained, and the paper is well written (also the diagrams are great). I'm very excited about this work, and hope to see it succeed, but in the paper's current state (even with the revisions), I don't think it addresses the reviewers' main concerns.

After discussions and examining the paper and the reviews in detail, I feel reviewer GaKc best summarizes the main issues with the work at its current state:

- This paper is interesting but does not proposes a significant improvement to the literature as the gap between the promises made in the motivation and actually delivered work is too wide.

- From a neuroscience point of view, this work does not provide substantial evidence of the importance of the model at either modeling or simulating biological neural systems.

- From a "theoretical" point of view, the model does not provide much advancement to the machine learning community either.

So while the current work (especially in the revised state) I would consider to be an outstanding workshop paper, I cannot recommend it for acceptance at ICLR 2022. An advice I would give to the authors (as someone who publishes to ML conferences, and Sys-Neuro/Bio venues) is that for these ML conferences, it might be easier to make the narrative of the work narrower, and well-defined. If the method is supposed to demonstrate significant advantages of biologically inspired network architecture over current RL methods, the results should clearly demonstrate convincing experimental results that can persuade the (non-neuro) RL community to have interest in the method. If the method does not achieve SOTA results, then try to present the method capable of something really useful that existing RL methods simply fail at (and emphasize that as a core contribution). Conversely, if the narrative is to use a bio-inspired network to emulate biological behaviors, the method must have something important to offer for the community of people working on simulating biological neural systems. 

I look forward to seeing this work improved and eventually published at a journal or presented at a conference in the future, good luck!
Decision: Reject