Neural Circuit Architectural Priors for Embodied Control
Authors: Nikhil Xie Bhattasali
Anthony M. Zador
Tatiana A Engel
Venue: ICLR 2022 Submitted (2022)
Keywords: neuroscience-inspired AI
robotics
motor control

Abstract: Artificial neural networks coupled with learning-based methods have enabled robots to tackle increasingly complex tasks, but often at the expense of requiring large amounts of learning experience. In nature, animals are born with highly structured connectivity in their brains and nervous systems that enables them to efficiently learn robust motor skills. Capturing some of this structure in artificial models may bring robots closer to matching animal performance and efficiency. In this paper, we present Neural Circuit Architectural Priors (NCAP), a set of reusable architectural components and design principles for deriving network architectures for embodied control from biological neural circuits. We apply this method to control a simulated agent performing a locomotion task and show that the NCAP architecture achieves comparable asymptotic performance with fully connected MLP architectures while dramatically improving data efficiency and requiring far fewer parameters. We further show through an ablation analysis that principled excitation/inhibition and initialization play significant roles in our NCAP architecture. Overall, our work suggests a way of advancing artificial intelligence and robotics research inspired by systems neuroscience.

Under review as a conference paper at ICLR 2022
NEURAL CIRCUIT ARCHITECTURAL PRIORS FOR EMBODIED CONTROL
Anonymous authors Paper under double-blind review
ABSTRACT
Artificial neural networks for simulated motor control and robotics often adopt generic architectures like fully connected MLPs. While general, these tabula rasa architectures rely on large amounts of experience to learn, are not easily trans- ferable to new bodies, and have internal dynamics that are difficult to interpret. nature, animals are born with highly structured connectivity in their nervous systems shaped by evolution; this innate circuitry acts synergistically with learn- ing mechanisms to provide inductive biases that enable most animals to function well soon after birth and improve abilities efficiently. Convolutional networks inspired by visual circuitry have successfully encoded biases useful for vision tasks. However, it is unknown the extent which ANN architectures inspired by neural circuitry can yield useful inductive biases for other domains. In this work, we ask what advantages biologically inspired network architecture can provide in the context of motor control. Specifically, we translate C. elegans circuits for locomotion into an ANN model applied to a simulated Swimmer agent. On a locomotion task, our architecture achieves good initial performance and asymptotic performance comparable with MIPs, while dramatically improving data efficiency and requiring orders of magnitude fewer parameters. Our architecture is also more interpretable and transfers to new body designs. An ablation analysis shows that principled excitation/inhiiitio is crucial for learning, while weight initialization contributes to good initial performance. Our work demonstrates several advantages of ANN architectures inspired by systems neuroscience and suggests path towards modeling more complex animals.
INTRODUCTION
Artificial neural networks for simulated motor control and robotics often adopt generic architectures like fully connected multi-layered perceptrons (MLPs) (Pierson & Gashler 2017 Levine al.) 2016 Bin Peng et al.. 2020 Heess et al., 2016). While general, these tabula rasa architectures rely on large amounts of experience to learn. Data efficiency is especially important in motor control because, unlike computer vision and natural language processing which have greatly benefited from the availability of large datasets (Bommasani et al. 2021), gathering large-scale data for robotics is challenging. Experience must be gathered through interaction with the environment, which is difficult for reasons including time, human labor, safety, maintenance, and reproducibility (Kroemer et al.) 2020). Transfer is also a sigmificant problem, as experience is often specific to the robot body it was gathered on, and ANNs trained to control one body are not easily adapted a different one. In addition, rabula rasa architectures like MLPs are in general difficult to interpret, as they have internal dynamics that are distributed across units and non- non-trivial relate agent behavior (Merel et al. 2019a).
In nature, animals are born with highly structured connectivity in their brains and nervous systems that has been shaped over millennia by evolution (ador 2019). In some cases, this innate circuitry confers abilities with little or no learning: in others, it guides the learning process by providing strong inductive biases (Lake et al. 2017). These innate and learned mechanisms act synergistically, enabling most animals to function well soon after birth, while continuing to improve and acquire skills in a data-efficient manner, e.g. a horse learning to walk with only a couple hours of experience. Moreover, despite species-specins variations, there is significant amount of shared architecture (e.g. cerebellum, basal ganglia) and design principles (c.g. hicrarchical modularity, partial autonomy) even between distantly related species (Merel et al. 2019b). Biological circuit architecture is often highly

---

Under review as a conference paper at ICLR 2022
structured and sparse (Luo 2021). ). in sharp contrast to the all-to-all connectivity of MLPs. Moreover, evolution has progressively built more advanced abilities on lower-level circuits, leveraging modular structure transfer existing working designs to new animal bodies (Cisek] 2019 Merel et al. 2019b). Taken together, these findings from neuroscience suggest that structured neural circuits in animals instantiate efficient, transferable, and modular solutions for high-dimensional embodied control.
Capturing some of this structure in model architecture may enable ANNs to narrow the gap between artificial and natural systems. For example, convolutional networks inspired by visual circuitry have successfully encoded the inductive biases of spatial locality and weight sharing to improve performance, data efficiency, and parameter efficiency for vision tasks (Lindsay 2021). Since
their eecroscience-inspiree origins, convolutional networks have benefited from novel layer types. activation functions, and modules (Gu al. 2017). as well as novel architectures, e.g. LeNet (LeCun et al., 1989), AlexNet (Krizhevsky et al., 2012). VGG (Simonyan & Zisserman 2015). ResNet (He et 2015). The advantages of (non- biologiggyy inspired) architectural priors has also been established across other machine learning domains. In natural language processing, architectural priors have specialized to handle sequences with designs including recurrent networks, e.g. LSTM (Hochreiter & Schmidhuber 1997), and attention networks, e.g. Transformer (Vaswani et al. (017). However, it is unknown the extent to which ANN architectures inspired by neural circuitry can yield useful inductive biases for other domains.
In this work, we ask what advantages biologically inspired network architecture can provide in the context of motor control. Specifically, translate C. elegans circuits for locomotion into ANN model applied to simulated Swimmer agent. Our architecture is an instance of what we call "Neural Circuit Architectural Prior" (NCAP), to denote an ANN architecture that encodes prior structure / inductive biases inspired by biological neural circuits. In contrast to previous work on neuromechanical models of movement and central pattern generators (Sarma et al.) 2018 Izquierdo & Beer [0155] Jiao et al. 2021), our model is designed within the discrete- time ANN framework that is standard in machine learning and is fully differentiable, enabling us to train parameters with reinforcement learning (RI.) and evolution strategies (ES), and directly investigate the role of network architecture by comparing to MLPs. Further, our model controls an agent body from a standard benchmark (i.e. not custom-designed work with our architecture); this body is significantly different from C. elegans in terms of mechanics, degrees freedom, and actuators.
On a locomotion task, our architecture achieves good initial performance and asymptotic perfor- mance comparable with MLPs, while dramatically improving data efficiency and requiring orders of magnitude fewer parameters. Our architecture is more interpretable and transferable new body designs. An ablation analysis shows that principled excitation/inhibition significantly contributes to learning. Our work demonstrates several advantages of ANN architectures inspired by systems neuroscience and suggests a path towards modeling more complex animals.
The primary contributions of this work are:
1. A network architecture inspired by C. elegans circuits for locomotion, which combines the discrete-time ANN framework that is standard machine learning with features from computa- tional neuroscience like constraints on synapse sign (i.e. excitation vs. inhibition) and special cell types (i.e. intrinsic oscillators);
2. An evaluation of our model's initial performance, asymptotic performance, data efficiency, parameter efficiency, interpretability, and transfer compared to standard MLP architectures; and
3. An ablation analysis of the effects of sign constraints, weight initialization, weight sharing, and sparse connectivity on performance and learning.
Code and videos are available in the Supplementary Materials.
RELATED WORK
A robotic controller ultimately outputs generalized torques T to apply at each actuator. A neural network controller can directly output torques (Levine et al., 2016). or can output task-space analytic controller like
positions or accelerations è that converted into torques through operational space control (Khatib 1987.. However, some further level of abstraction and prior knowledge is often used to improve performance and simplify learning (Kroemer et al., 2020).
2

---

Under review as a conference paper at ICLR 2022
Trajectory Priors Desired movement is encoded through equations of motion. Dynamic Movement Primitives (DMP) (Schaal 2006, Pastor et al. 2009) use a set of differential equations to implement stable nonlinear attractor system capable of generating rhythmic and discrete trajectories, which are controlled via low-dimensional parameters specifying the motion's 's shape and goal. Policies Modulating Trajectory Generators (PMTG) (Iscen al. 2019) learn poliey to control predefined trajectory generator via low-dimensional parameters and also generate residual term to be added to the trajectory generator's output. For example, to produce legged locomotion, PMTG uses equations of motion composed from combination of sinusoidal functions and hand-engineered gait patterns, which are parameterized by stride length, walking height, and frequency. Generally, trajectory- centric methods can work well when the equations of motion capture good solutions for the desired movement; however, such equations can be difficult to design and make robust.
Behavioral Imitation Priors Desired movement is encoded through learnable functions, e.g. ANNs, trained to imitate reference motions, e.g. from motion capture or manual keyframing. Neural Probabilistic Motor Primitives (NPMP) (Merel et al., 2019c) train expert policies imitate human motion capture trajectories, then compress these experts into single generalist policy featuring a larinlee bottleneck, thereby creating common embedding space that a higher-Iever controller can use to interpolate and combine various motor primitives. Bin Peng et al. (2220) train expert policies to imitate animal motion capture trajectories using reinforcement learning and further deploy the learned policies on physical legged robot. Generally, imitation methods bypass the time and manual effort involved in designing equations of motion and have achieved diverse behaviors. This comes at the cost of compiling of reference motions from humans and animals. Further, learning individual expert policies does not make use of the shared structure between movements to learn more efficiently. In animal legged locomotion, for instance, neural circuit studies have suggested that different gait patterns (c.g. walk, trot, bound, gallop) could actually be different emergent dynamic modes of the same pattern generator network (Grillnerr & El Manira (020). In this case, a single policy with the right structure should be able to capture these diverse reference motions.
Architectural Priors Desired movement is encoded indirectly through the structure of ANNs.
establishing inductive biases to guide learning. Heess et al.] (201)) decompose policy into a low-level "spinal" network with access to only proprioceptive signals (e.g. joint positions) and a high-level "cortical" network with access to exteroceptive signals (e.g. distance to target, vision); this hierarchical architecture was loosely inspired by animal nervous systems. After pre-training the spinal network with shaped reward, the cortical network was able to learn complex locomotion tasks from sparse rewards by controlling the spinal network, while flat baseline architectures failed. This architecture was later shown to generalize to locomotion with challenging terrains and obstacles, providing greatly improved learning efficiency (Heess et al.) (0177). Generally, architectural priors can provide improved efficiency and abstraction while maintaining flexibility. However, the level of flexibility needs to be chosen appropriately. Too much flexibility can lead to learned behaviors that are not naturalistic and infeasibleddangerous to deploy in the real world without regularization (Bin Peng et al.) 2020), while too much constraint can impede the learning of diverse movements.
In this work, we design an architectural prior inspired by neural circuitry. In principle, this kind of prior should scale to diverse movements, since the architecture models the exact mechanisms that animals use achieve robust, flexible behavior. Moreover, since working "blueprints" already exist in nature, translating architectures from biology could prove more desirable and more scalable than hand-designing trajectory priors from scratch. Further, while we use reward-based, algorithms to learn parameters (i.e. RI. and ES), our architecture is largely agnostic to the learning algorithm, so it is possible that our architecture could also be trained within a behavioral imitation setting.
MODEL
We translate nematode (C. elegans) circuits for locomotion into an ANN model applied to a simulated Swimmer agent. In Section 3.1 we provide an overview of the nematode body structure and the modular neural circuits underlying locomotion. In Section 3.2 we describe our abstract integrator
and oscillator units that serve as building blocks for our architecture. In Section 3.3 we formalize the observation and action space of the Swimmer agent, and we propose our network architecture modeled on nematode circuits.
3

---

Under review as a conference paper at ICLR 2022
nematode body
nematode neural circuits
muscle
muscle
head
wave propagation
-> tail
dorsal
SMB
DB
DD
DB
DD
ventral
AVB
proprioception excitatory synapse
SMB
VB
VD
VD
inhibitory synapse
gap junction
muscle
muscle
Figure I: Nematode. (A) The nematode C. elegans achieves forward locomotion through alternating dorsal-ventral muscle contraction waves propagating down the body. (B) Muscle wave propagation, oscillation, steering, and speed control are coordinated by a highly stereotyped, modular, and repeated microcircuit. B neurons sense bending in the previous module via proprioception and excite ipsilateral muscles, while inhibiting contralateral muscles via neurons. Intrinsic oscillations in B neurons initiate waves. SMB neurons bias head and neck muscles for steering. AVB attenuates all B neurons via gap junctions for speed control.
3.1 NEMATODE
The nematode C. elegans has served as a useful model organism within neuroscience because is one of the simplest organisms with nervous system. Moreover, is unique in that its connectome, iee. wiring diagram, has been completely mapped (Hall & Altun 2008).
Nematode Body The nematode body is a I mm long, 50 um diameter tapered cylinder (Figure [A]. made up of 959 somatic cells, of which 302 are neurons comprising the nervous system, of which 75 are motor neurons that innervate the 95 body wall muscles distributed along the body. Forward and backward thrust is produced via alternating dorsal-ventral muscle contraction waves propagating down the body in the direction opposite to the direction of motion. Steering is produced by differential activation of the 20 anterior muscles in the head and neck (Gjorgjieva et al. 2014).
Nematode Neural Circuits The nematode forward locomotion circuit summarized below (Fig- ure BB) For more in-depth treatment, consider Gjorgjieva et al. (2014) and Wen et al. (2018).
Muscle wave propagation coordinated by 2 classes of neurons that innervate dorsal (D-) and ventral (V-) muscles. B neurons (DB and VB) act both sensory and motor neurons, expressing stretch receptors in their dendrites to sense bending 200 μm anterior to their somas, and sending excitatory output (via ACh) the muscles and D neurons. D neurons (DD and VD) send inhibitory output (via GABA) to the muscles. This microcircuit is highly stereotyped, modular, and repeated down the length of the body, and its logic is interpretable. For particular module, body bending in the previous module is sensed by B neurons, which then initiate bending on the same side (ipsilateral) while simultaneously inhibiting bending on the opposite side (contralateral) through neurons.
Muscle wave initiation is generated by intrinsic oscillators. While proprioception-onl circuits (with oscillators ablated) are capable of producing small waves on its own, oscillators are used initiate and entrain larger waves (Gjorgjieva et al. 2014). These oscillators were long believed to only reside in the head and neck, but recently work has shown them to in fact also be present in the body, as it is the B neurons themselves that produce intrinsic oscillations (Wen et al. 2018).
Steering is generated by the differential activation of SMB neurons biasing the head and neck muscles to bend dorsally ventrally (Izquierdo & Beer 2015).
Speed control is coordinated by the AVB command neuron, which connected through gap junctions with all B neurons. When AVB is in low state, the resting membrane potentials of B neurons hyperpolarized to prevent activation; when AVB is in a high state, B neurons are free to activate based on proprioceptive and oscillatory inputs.
4

---

Under review as a conference paper at ICLR 2022
integrator unit
oscillator unit
2 = b+ E. E.(
2
y
y
- f(()
-22
V6
y f(z)
width
excitatory weight
w* c* (w(w max (1r, 0)
square
[(I) J(1)
high
inhibitory weight
W (w) min(ue 0)
sine wave
amplitude
midline
activation function
(() max(2,
period
Architectural Components, (A) An integrator unit models simple neuron. The graded input signals are multiplied by weights that represent synaptic efficacy and which are constrained to be either excitatory (positive, green boxes) or inhibitory (negative, red boxes). The graded output signal is produced from an activation function. (B) An oscillator unit produces driving signals much like intrinsic pacemaker cells and network-based oscillators. The graded output signal is generated through periodic functions, e.g. square waves and sine waves.
Figure 2:
3.2 ARCHITECTURA, COMPONENTS
Our architecture is built from components that combine the discrete-time ANN framework that is standard in machine learning with features from computational neuroscience like constraints on synapse sign (i.e. excitation vs. inhibition) and special cell types (i.e. intrinsic oscillator). The components are fully differentiable and therefore compatible with backpropagation-ben -basdd learning algorithms, though not restricted them.
Integrator Units Signals in biological neural circuits processed and integrated by neurons. The integrator uni in Figure 2 similar to the standard ANN model. The graded inputs x; multiplied by synaptic weights w; produce the membrane potential 2, given the resting membrane potential b. A nonlinear activation function f(z) produces the graded output based on the membrane potential. Unlike the standard ANN model, however, we constrain synaptic weights by a sign constraint function c(w). This is done to reflect that in biological circuits a primary characteristic of a synapse whether is excitatory or inhibitory. In the standard model, synapses are initialized with random signs and are free to change during learning. We argue that principled excitation/inhibition is fundamental for interpreting and modeling the logic of neural circuits, and we show in the ablation analysis that they are critical for learning in our architecture (Section 4.5)
Neural circuits often feature components with specialized dynamics, with oscil- lators being a prominent example (Grillner & El Manira 2020). oscillator can be implemented through coupled activity between neurons within single neuron, similar to pacemaker cells in the heart (Bucher et al.) 2.. Oscillators serve as internal drivers of activity, good example of the fact that neural circuits do not exclusively react external inputs from the environment. The oscillator unit in Figure 3 uses a periodic function (t) to produce the graded output y. Example periodic functions include square wave and sine wave generators.
Oscillator Units
3.3
SWIMMER
The Swimmer is common body design in widely adopted continuous control benchmarks, e.g. DeepMind Control Suite (Tassa et al. (2200) and OpenAl Gym (Brockman et al. (016). We target this standard design rather than custom body like previous work (Sarma et al. 2018 [zquierdo & Beer 2015) in order to demonstrate the potential of using biologically inspired network architecture on tasks that have been tackled by the Al community.
Swimmer Body The Swimmer agent has an articulated body with N joints connecting N + links (Figure BA). Its movement is entirely within the ry-plane. Thrust is generated by the links pushing
'Simple neurons are often approximated as single integrator units (Torres & Varona 2012 However, sometimes neurons have multiple sites of integration, i.e. dendritic integration across multiple compartments. We prefer "integrator unit" te'neural unit" complex neuron may require multiple integrator units model.
5

---

Under review as a conference paper at ICLR 2022
A
swimmer body
head
tail
43
%
dorsal
ventral
42
94
B
swimmer network architecture oscillator
B neuron
muscle
B neuron
muscle
61
771
m2
42
nd
02
8
8
6
771
mc V
P
02
weights:
speed (6), prop (p),
(6)
Figure 3: Swimmer. (A) The Swimmer has an articulated body with N joints connecting N + links. Its observation space normalized joint positions g, and its action space is normalized joint accelerations q. (B) Our network architecture closely conforms to the modular microcircuit of the nematode. Each module senses bending in the previous module q--1 and drives neurons and muscles mi which are combined to create joint accelerations ij:
against the surrounding fluid, e.g. simulated via high-Reynokds fluid drag model Todorov et al. 2012). The observation space consists of normalized joint positions q E [-1, 1] between joint limits. The action space consists of normalized joint accelerations q E [-1,1] 1] between maximum acceleration counterclockwise and clockwise, respectively.
Swimmer Network Architecture Our network architecture is best explained visually (Figure 3B).
For muscle wave propagation, signals are integrated in B neurons and muscles; D neurons mainly serve to convert opposite-side B neuron signals from excitatory to inhibitory, and their role can be replicated directly the muscle integrator units. We model N modules control each of the N joints. For a particular module i S. N, the previous module joint position 9:11 is split into dorsal GL1 E [0, 1] and ventral -- E [0, 1] components, order to mirror signals from proprioceptive stretch receptors that sensitive to bending
side. B neurons modeled integrator units with outputs b/ and by, which receive same-side excitatory proprioceptive inputs. Muscles are modeled as integrator units with outputs m and m.. which receive same-side (ipsilateral) excitatory B neuron input as well as opposite-side (contralateral) inhibitory B neuron input. Finally, the joint acceleration q is calculated from dorsal and ventral muscle outputs, which act antagonistically.
For muscle initiation, the first module B neurons of and by receive inputs from oscillators dd and o", respectively, instead proprioception. We use square wave generators acting in anti anti-phase.
For steering, SMB outputs modeled as right turn signal r [0. 1] and a left turn signal / [O, 1], which serve as additional excitatory inputs to first module B neurons bf and bj, respectively.
For speed control, AVB outputs modeled speed signal E [0, 1). To approximate the effect of gap junctions, such that represents stopping and s represents maximum speed, 1 serves additional inhibitory input all neurons.
The complete architecture for module is therefore:
0-11 1 max (0ax(9/-11 (1:-1,
0/11 max( max(-99-1,0)
8 = ʃ U/ Wypeea( -8)
8))
6%. (upoopti11 + w Wapeed( (1 8))
m = ff (wita Weoncr
m f (1ita + wcontrab?)
05 m m

---

Under review as a conference paper at ICLR 2022
with special B neuron integrator units for i 1:
81 f ( o" wi "speed! (1 - -))
bi = (w 0" +witm!
"speed 8)
w
We weight sharing such that weights with the same within each module across sides. We initialize all weights with the correct signs and magnitudes 1.
shared across modules as well as
EXPERIMENTS
Learning Formalization We consider an agent formalized as policy function Te(ar(s;) that maps states to actions a, and which is represented by an ANN parameterized by weights 0 with our architecture described Section 3.3 We consider the standard agent-environment interaction model formalized as Markov Decision Process (MDP). At every timestep t, the agent in state st takes an action according to its policy a; ~ ro(ar(8)), receives a reward Tr, transitions to a new state 81+1. Policy parameters 0 are optimized to maximize the discounted return D TTT where I' the horizon of the episode, and 0 < 7 < 1 the discount factor.
Learning Algorithms We compare backpropagation-baste RL algorithms and a derivative-free ES algorithm for learning parameters in the architecture.
Proximal Policy Optimization (PPO): (Schulman et al. 2017 A model-free, on-policy, policy gradient RL method. uses a clipped surrogate objective limit the size of policy change each step, thereby improving stability. Since it assumes stochastic policies, perturb the deterministic actions with Gaussian noise E, ~ ~N,0
Deep Deterministic Policy Gradient (DDPG): (Lillierap al. 2019) A model-free, off-policy, policy gradient RL method. It uses off-policy data and the Bellman equation to learn the Q-function, which is iteratively used improve the policy.
Evolution Strategies (ES): (Salimans et al., 2017) An evolutionary black-box optimization method. creates population of policy parameter variants through perturbations with Gaussian noise, then combines them through averaging, weighted by the return collected across episodes.
We implement the Swimmer using the standard N = 5 body in the DeepMind Control Suite (Tassa et al. 2020) built upon the MuJoCo physies simulator (Todorov et al. 2012). We train the agent to swim using shaped rewards proportional to swimming speed (Appendix A.
Learning Setup
4.1 PERFORMANCE AND DATA EFFICIENCY
How well and data efficiently does learning occur in NCAP MLP architectures?
A
algorithms architecture
B
architectures per algorithms
MLP(256.256)
NCAP
PPO
DDPG
ES
1000
1000
256
avg Rewaru
800
raward DM
800
NCAP
128
600
600
400
400
DDPG
200
PPO
200
0
10s
10a
10¹
10a
10a
10a
107
timesteps
timesteps
timesteps
timesteps
timesteps
Figure 4: Performance and Data Efficiency. (A) Comparison of different algorithms for each architecture. Our architecture starts with high reward and improves with learning, achieving signifi- cantly better data efficiency and comparable performance. (B) Comparison of different architectures for each algorithm. Our architecture with parameters overperforms small MLPs (MLP(2,2) has 70 parameters) and is comparable to large MLPs (MLP(256, 256) has 77, 77,222 parameters). Plots show averages over random seeds (solid lines) and 95% bootstrap confidence intervals (shaded areas).

---

Under review as a conference paper at ICLR 2022
First, we compare different learning algorithms for either MLP or NCAP architectures (Figure 4A). We use an MLP with 2 hidden layers of dimensions (256, 256) and ReLU nonlinearities. We find that our NCAP architecture achieves substantially higher initial performance than MLPs as well as comparable asymptotie performance with MIPs, demonstrating the effectiveness of prior knowledge encoded in network architecture. Our NCAP architecture shows reduced variance during learning between trials with different random seeds, as well as reduced differences in asymptotic performance between algorithms. For both MLP and NCAP, ES requires roughly an order of magnitude more data to achieve comparable performance with the RL algorithms, consistent with previous work (Salimans et al. 2017). Qualitatively, both MLP and NCAP architectures yield reasonable swimming movement ( (Videos TA-B), though NCAP produces waves with large amplitudes resembling C. elegans movement, while MLP produces waves with small amplitudes more resembling tadpoles. This different movement shape explains the slightly lower asymptotic performance for NCAP because the body's 's direction of travel is less correlated with the head orientation, which is relevant for how rewards are calculated (Appendix A Videos IC). We note that we simplified our design from actual C. elegans circuits for pedagogical reasons (Section 4.5). and our goal is not to solve this swimming task per but rather to investigate the advantages of biologically inspired network architecture more generally. C. elegans circuits are not optimized for fast swimming with few segments (Section (4.4); future work may propose architectures better for this specific task, e.g. using larval zebrafish eircuits.
Second, we compare different architectures for each learning algorithm (Figure 4).. We use MIPs with hidden layers of varying dimension sizes and ReLU nonlinearities. We find that performance deteriorates across all algorithms for MLPs as hidden dimensions become smaller, with some algorithms like DDPG deteriorating dramatically, However, our NCAP architecture with 4 parameters overperforms small MLPs and is comparable to large MLPs. This is especially notable as the smallest MLP(2.2) has 70 parameters (1 order of magnitude more than NCAP) and the largest MLP (256,256) has 73,222 parameters (2 orders of magnitude more than NCAP). This suggests that the relatively simple structure of our NCAP architecture provides highly effective inductive biases.
parameter efficiency
4.2 PARAMETER EFFICIENCY
avg reward / params
10a
NCAP
DDPG PPO
How valuable are parameters in NCAP vs. ML. architectures?
10a
We compare the asymptotic performance divided by parameter count for each architecture (Figure 3). Across all algorithms, our NCAP architecture achieves more than an order of magnitude better parameter efficiency than MLPs of any size. A parameter in our NCAP architecture is "worth more" than one in MLPs.
10
10"
MLP
10-1
10a
10a
10
params
4.3 INTERPRETABILITY
Figure 5: Parameter Efficiency Asymptotic performance per unit parameter. Our architecture achieves better parameter effi- ciency than MLPs of any size.
How interpretable are unit dynamics in our NCAP architecture?
We visualize the dynamies of each network unit while performing a task (Videos 2). Since our NCAP architecture is a sparsely connected, modular network where units play constrained roles (e.g. excitatory or inhibitory), it is easier to relate unit dynamics to agent behavior, which helps in explanation and debugging.
different length bodies
1000
avg reward
800
4.4 TRANSFER
600
400
How well does our NCAP architecture adapt to new bodies?
DDPG PPO
We leverage the modular structure of our NCAP architecture to adapt trained network for N 5 joints to bodies with dif- ferent N' joints by duplicating/removing modules (Figure 6 We achieve robust swimming behavior for wide range of N' (Videos 3). Interestingly, zero-shot performance is better for longer bodies. likely reflecting that C. elegans circuits are evolved for its more segmented body. This kind of zero-shot transfer is not possible with monolithie, densely connected MLP architectures.
1
3
joints
Figure 6: Transfer. Zero-shot transfer to new bodies after train- ing on N 5 by leveraging ar- chitecture modularity.
8

---

Under review as a conference paper at ICLR 2022
A weight sharing, sign constraints, weight initialization
B sparse connectivity
PPO
ES
MLP(12.11010)
1000
init sign Init
1000
W5
bvg rewaay
800
WiS sign Init
aVg reward
800
600
600
400
400
DDPG
200
200
11
[ (m))
mim Im
10a
10
10
107
10a
10
10?
timesteps
timesteps
timesteps
Figure 7: Ablations. (A) Ablations of weight sharing, sign constraints, and weight initialization in different combinations. Sign constraints (i.e. principled excitation/inhibitio)) are crucial for learning. Weight initialization at large magnitudes is responsible for good initial performance. Weight sharing yields small gain in data efficiency. (B) Ablation of sparse connectivity yields an equivalently sized MLP (Appendix B). Learning is restored without sparse connectivity.
4.5 ABLATIONS
What are the effects of various features of our NCAP architecture performance and learning?
First, we investigate the role of weight sharing, sign constraints, and weight initialization (Figure A Without weight sharing, weights across modules and across sides are separate parameters, increasing the total number of parameters from 4 to 30. Without sign constraints, the identity function is applied instead of excitatory/nhiiiioryy constraint function. Without weight initialization, weight magnitudes are initialized through a uniform random distribution within [0, 1]. rather than at 1. If using sign constraints, weights are always initialized with the appropriate sign; otherwise, signs chosen randomly with equal probability. We find that sign constraints are crucial for learning. Without appropriate sign constraints, the NCAP architecture fails learn during the allotted timesteps, for both RL and ES algorithms. With appropriate sign constraints, even if the weights magnitudes are not initialized ideally, the NCAP architecture will learn. Weight initialization is responsible for good initial performance. Weight sharing has a smaller, but identifiable, contribution to data efficiency.
Second, we investigate the role of sparse connectivity that arises from the natural structure of neural circuits (Figure B). Our Swimmer architecture has the special property that it can be completely embedded within an MLP with 3 hidden layers of dimensions (12, 10, 10) and ReLU nonlinearities (Appendix B). Specifically, after ablating sign constraints and weight sharing, our architecture is identical to this MLP with highly pruned connectivity (mostly weights of O). We remove this sparsity and find that the MLP can learn the task with similar asymptotic performance NCAP.
Taken together, our results suggest that principled excitation/inhibition is an important design consid- eration in small, sparse architectures like our NCAP, but less important in MLPs. This may be related to the "Lottery Ticket Hypothesis" (Frankle & Carbin, 2019), which suggests that, upon initialization, the MLPs already contain subnetworks with initial weights and signs that do most of the work for learning, i.e. they are "winning tickets"; imposing sparsity eliminates these overlapping subnetworks.
DISCUSSION
We asked what advantages biologically inspired network architecture can provide in the context of motor control. Through our case study translating C. elegans locomotion circuits into an ANN model, we found that biologically inspired network architecture can achieve comparable asymptotic performance to MLPs with significantly improved initial performance, data efficiency, parameter efficiency, interpretability, and transfer. Therefore, while the norm of using tabula rasa architectures like MLPs may be general, architectural priors can provide useful inductive biases for motor control and should be investigated further. Future work can translate neural circuits that underlie a wider variety of animal bodies and movements, as well incorporate inductive biases from neural circuits that underlie visual, tactile, and auditory sensing. Overall, we believe that our work suggests a way of advancing artificial intelligence and robotics research inspired by systems neuroscience.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang- Wei Lee, Jie Tan, and Sergey Levine. Learning agile robotic locomotion skills by imitating animals. In Robotics: Science and Systems Science and Systems Foundation, July 2020.
XVI. Robotics:
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quiney Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus. Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Kohd, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zancle Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nic, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas. Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. 1rXiv:2108.07255 [cs], August 2021.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl Gym. arXiv: 540 [es], June 2016.
Dirk Bucher, Gal Haspel, Jorge Golowasch, and Farzan Nadim. Central pattern generators. November 2015.
Paul Cisek. Resynthesizing behavior through phylogenetic refinement. Attention, Perception, & Psychophysics, (1(7):2265-2255 October 2019.
Jonathan Frankle and Michael Carbin. The Lottery Ticket Hypothesis: Finding sparse, trainable neural networks. arXiv: 883800633333 [cs], March 2019.
Julijana Gjorgjieva, David Biron, and Gal Haspel. Neurobiology of Caenorhabditis elegans locomo- tion: Where do we stand? BioScience, 64(6):476-486, June 2014.
Sten Grillner and Abdeljabbar El Manira. Current principles of motor control, with special reference to vertebrate locomotion. Physiological Reviews, 100(1):271-320, January 2020.
Jiuxiang Gu, Zhenhua Wang. Jason Kuen, Lianyang Ma, Amir Shahroudy, Bing Shuai, Ting Liu, Xingxing Wang, Li Wang, Gang Wang, Jianfei Cai, and Tsuhan Chen. Recent advances in convolutional neural networks. 112.:115 [cx]. October 2017.
David H. Hall and Zeynep F. Altun. Elegans Atlas. Cold Spring Harbor Laboratory Press, Cold Spring Harbor, 2008.
2008.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv:1572.03385 [cs], December 2015.
Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers. arXiv: [cs], October 2016.
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S. M. Ali Eslami, Martin Riedmiller, and David Silver. Emergence of locomotion behaviours in rich environments. arXiv: 02286 [cs], July 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9()): 1735-1780, November 1997.
10

---

Under review as a conference paper at ICLR 2022
Atil Iscen, Ken Caluwaerts, Jie Tan, Tingnan Zhang, Erwin Coumans, Vikas Sindhwani, and Vincent Vanhoucke. Policies modulating trajectory generators. arXiv:1910.02812 [cs], October 2019.
Eduardo J. Izquierdo and Randall D. Beer. An integrated neuromechanical model of steering in C.
elegans. In ECAL 2015: The 13th European Conference Artificial Life, pp. 199-206, July 2015.
Yusheng Jiao, Feng Ling, Sina Heydari, Nicolas Heess, Josh Merel, and Eva Kanso. Learning to swim in potential flow. Physical Review Fluids, 6(5):050505, May 2021.
Goktug Karakasli. ESTorch: An evolution strategy library built around PyTorch, 2020.
O. Khatib. A unified approach for motion and force control robot manipulators: The operational space formulation. IEEE Journal on Robotics and Automation, 3(1):33-53, February 1987.
Alex Krizhevsky, Nya Sutskever, and Geoffrey E. Hinton. ImageNet classification with deep convo- lutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 25, Curran Associates, Inc., 2012.
Oliver Kroemer, Scott Niekum, and George Konidaris. A review of robot learning for manipulation: Challenges, representations, and algorithms. arXiv:1907.035 146 [cs], November 2020.
Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40:e253, 2017.
Y. LeCun, B. Boser, S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4):541-551, December 1989.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. 3xXiv:150400070 [cs], April 2016.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. DrXiv:/509.0299 [es, stat], July 2019.
Grace W. Lindsay. Convolutional neural networks as model of the visual system: Past, present, and future. Journal of Cognitive Neuroscience, 33(10))001720331 September 2021.
Liqun Luo. Architectures of neuronal circuits. Science, 373(6559), September 2021.
Josh Merel, Diego Aldarondo, Jesse Marshall, Yuval Tassa, Greg Wayne, and Bence Ölveczky. Deep neuroethology of virtual rodent.
[q-bio], November 2019a.
Josh Merel, Matthew Botvinick, and Greg Wayne. Hierarchical motor control in mammals and machines. Nature Communications, 10(1):5489, December 2019b.
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid con- trol. arXiv: 1PXiv:8111.11111 [cs], January 2019c.
Fabio Pardo. Tonic: A deep reinforcement learning library for fast prototyping and benchmarking. arXiv:2011.07537 [cs], May 2021.
Peter Pastor, Heiko Hoffmann, Tamim Asfour, and Stefan Schaal. Learning and generalization of motor skills by learning from demonstration. In 2009 IEEE International Conference on Robotics and Automation, pp. 763-768, Kobe, May 2009. IEEE.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach De Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, nigh-performance deep learning library. arXiv: /0tp:///////// [cs, stat], December 2019.
Harry A. Pierson and Michael S. Gashler. Deep learning in robotics: A review of recent research. Advanced Robotics, 31(16):821-83) August 2017.
11

---

Under review as a conference paper at ICLR 2022
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and llya Sutskever. Evolution strategies scalable alternative to reinforcement learning. arXiv:/703.0388 [cs, stat], September 2017.
Gopal P. Sarma, Chee Wai Lee, Tom Portegys, Vahid Ghayoomie, Travis Jacobs, Bradly Alicea, Matteo Cantarelli, Michael Currie, Richard C. Gerkin, Shane Gingell, Padraig Gleeson, Richard Gordon, Ramin M. Hasani, Giovanni Idili, Sergey Khayrulin, David Lung, Andrey Palyanov, Mark Watts, and Stephen D. Larson. OpenWorm: Overview and recent advances in integrative biological simulation of Caenorhabditis elegans. Philosophical Transactions of the Royal Society B: Biological Sciences, 373(1758):201700)) October 2018.
Stefan Schaal. Dynamic movement primitives a framework for motor control in humans and humanoid robotics. In Hiroshi Kimura, Kazuo Tsuchiya, Akio Ishiguro, and Hartmut Witte (eds.), Adaptive Motion of Animals and Machines, pp. 261-280. Springer- Verlag, Tokyo, 2006.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. urXiv:1707.063: [cs], August 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv: 1556 [cs], April 2015.
Yuval Tassa, Saran Tunyasuvunakool. Alistair Muldal, Yotam Doron, Piotr Trochim, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess. Dm control: Software and tasks for continuous control. Software Impacts, 6: 6:100022, November 2020.
Emanuel Todorov, Tom Frez, and Yuval Tassa. MuloCo: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, Vilamoura-Algare Algarve, Portugal, October 2012. IEEE.
Joaquin J. Torres and Pablo Varona. Modeling biological neural networks. In Grzegorz Rozenberg, Thomas Bäck, and Joost N. Kok (eds.). Handbook of Natural Computing, pp. 533-564. Springer Berlin Heidelberg, Berlin, Heidelberg. 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need.
[cs], December 2017.
Quan Wen, Shangbang Gao, and Mei Zhen. Caenorhabditis elegans excitatory ventral cord motor neurons derive rhythm for body undulation. Philosophical Transactions of the Royal Society B: Biological Sciences, 373(175)):0110377) October 2018.
Anthony M. Zador. A critique pure learning and what artificial neural networks can learn from animal brains. Nature Communications, 10(1):3770, December 2019,
12

---

Under review as a conference paper at ICLR 2022
A EXPERIMENTAL DETAILS
A.1 TASKS
The swim task aims to the agent's ability swim forwards at desired speed. It returns smooth reward that is O when stopped or moving backwards, and rises linearly to and saturates at when swimming at the desired speed.
A.2 IMPLEMENTATION
Libraries Neural networks implemented in PyTorch (Paszke al. 2019). The RL algorithms were implemented using Tonic (Pardo 2021). The ES algorithm was implemented using ES Torch
(Karakasli 2020).
Computational Resources Training was performed on a high performance computing cluster running the Linux Ubuntu operatin system. RL algorithm training runs were parallelized over 8 cores, while ES algorithm runs were parallelized over 32 cores.
A.3 HYPERPARAMETERS
RL Algorithms Standard hyperparameters for PPO and DDPG in Tonic (Pardo 2021 at commit 48 7b72; timesteps, 5e6.
ES Algorithm Population size, 256; noise standard deviation ø, 0.02; L2 weight decay, 0.005; optimized, Adam; learning rate, 0.01; timesteps. 5e7.
NCAP Swimmer Oscillator, square wave, period 60 timesteps, width 30 timesteps.
13

---

Under review as a conference paper at ICLR 2022
B SWIMMER ARCHITECTURE DETAILS
Our NCAP architecture has the special property that in can be completely embedded within a fully connected MLP of 3 hidden layers and ReLU nonlinearities. This enables to "interpolate" between our NCAP architecture and the MLP architecture, conducting fine-graines analysis of how various features of our architectural prior contribute to performance and learning.
By rearranging terms in the Swimmer network architecture diagram (Figure 3B), we arrive at the following network (N 5 shown):
weights: 00. speed
task module
motor module
weights:
(pl, losi contra
Ito all b))
11
41
92
b2
m2
43
03
44
01
1
by
m
45
mg
95
05
0/
mg
By removing weight sharing, sign constraints, and sparse connectivity, we arrive at a fully connected MLP of 3 hidden layers (N 2 shown):
od
0d
NCAP
o
MLP
no weight sharing sign constraints
rl
m
m
91
no sparse connectivity
9
m
6
m
42
Mig
22
d
771
92
42
02
42
by
m20
m
For N = 5, the resulting MLP has hidden layers of dimensions (12, 10, 10).
14


Reviews:
[{"review_id": "v5BC8JCh3GB", "reviewer": "Reviewer_GaKc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Clarity: \nThis paper is well written and the authors clearly explain the motivations, and framework in which they plan to operate. Figure 1, seems a bit unnecessary as there is nothing particularly new and the font size is small and exactly repeats the text. Fig. 3, and 4 are hard to read because of the font size and the organization. We would suggest the authors reorganize the figures for more clarity, which would help with the explanation of the results as well. However, Figure 2 is very clear and the parallel with C.elegans is made clear.\n\n\n### Strength:\nThis paper asks an important question that is how to combine knowledge of biological neural circuits and artificial neural networks in a principled way. It uses an interesting example that is C.elegans for which a lot is known about it's neural architecture and principles of locomotion. It uses unconventional neural components to the machine learning community and makes it differentiable, thus possible to train with standard learning methods. \n\n### Weakness: \nHierarchical component models for building biologically inspired neural networks are not new. In particular, the entire connectome of C.elegans has been reconstructed, and little of that information is actually replicated (in the prior) in this work. \nMoreover, although the neural architecture is somewhat biologically plausible, the problem of weight sharing is not. It is also true for the learning rules used. Thus the use for understanding biological models is very limited, as it only vaguely replicated the neural structure but does not operate in a biological setting. \n\nFrom the experiments, it seems that training does almost nothing to the performance of the model, as show in Fig.3A and 3B. The model initial outperforms competing models but after training it underperforms it. The relevance of the model is thus questioned, it is only the neural structure that implies the performance and not the learning. As a result this type of model would only be useful for extremely simple organisms with few neurons like C.elegans with little learning/genetic encoding ratio. \n\n\n\n\n\n\n\n\n\nhis allows us to design small, sparse networks that mirror neural circuits\nyet are fully differentiable and therefore compatible with backpropagation-based learning algorithms.\nIn contrast to previous work on neuromechanical models of movement (Sarma et al., 2018; Boyle\net al., 2012; Izquierdo & Beer, 2015), our models are more abstract, include learned parameters, and\nare used to control bodies significantly different from the originals in terms of mechanics, degrees of\nfreedom, and actuators.\n\nWe apply NCAP to translate C. elegans locomotion circuits into a simulated Swimmer agent, which we\ntrain using both reinforcement learning (RL) and evolution strategies (ES). We show that our NCAP\narchitecture achieves comparable asymptotic performance with fully connected MLP architectures,\nwhile dramatically improving data efficiency and requiring far fewer parameters. We further show\nthrough an ablation analysis that principled excitation/inhibition and initialization play significant\nroles in our NCAP architecture.", "summary_of_the_paper": "This paper proposes both a set of artificial neural components and interesting principles for producing biologically-inspired neural networks for embodied control. This work aims to be at the intersection between neuroscience and machine learning for improving the design of artificial neural networks as well as improving our understanding of observed biological networks. Various components of biological networks are replicated in their framework, e.g., the balance between excitation and inhibition, intrinsic oscillators, or sparsity. The model is evaluated on C.elegans motion locomotion circuit using different types of training strategies. \n", "summary_of_the_review": "This paper is interesting but does not proposes a significant improvement to the literature as the gap between the promises made in the motivation and actually delivered work is too wide. \n\nFrom a neuroscience point of view, this work does not provide substantial evidence of the importance of the model at either modeling or simulating biological neural systems. \n\nFrom a \"theoretical\" point of view, the model does not provide much advancement to the machine learning community either. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lQ8wZsR_Sjl", "reviewer": "Reviewer_qSdv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n\n1. The paper is well written, especially the abstract, introduction, and related work sections. The work is correctly motivated since efficient learning is observed in the brain, and inspiration from biology is essential to improve existing learning-based methods.\n\n2. The paper proposes a bio-inspired network trained by DRL methods while using much fewer parameters than an MLP. This shows that architectural priors induced by knowledge from biology can replace the pure-learning approach to some degree when the prior is correctly constructed.\n\n3. The ablation studies performed by the authors demonstrated the effectiveness of the learning on the proposed network for the swimmer agent. It is interesting that the network can still work after learning with random init weights.\n\nCons\n\n1. The set of NCAP and the network proposed by the paper share many similarities with existing CPG works. It is better to present the proposed approach as a fine-tuning method for CPG networks than as a general solution for robotics. Biological locomotor CPGs have inspired the design of many artificial networks for robot locomotion (for example, [Matsubara, Takamitsu, et al. 2006], [Lele, Ashwin Sanjay, et al. 2020], [Yao, Chenpeng, et al. 2021], [Lopez-Osorio, Pablo, et al. 2021]). These CPG works are based on similar concepts as the set of NCAPs proposed in this paper and explore different ways to fine-tune the CPG network. Though the fully differentiable network proposed in this work is interesting, its novelty and contribution are limited compared to previous works in the field.\n\n2. Since the proposed approach is similar to CPG-based works, they share the same problems as well. First, these approaches are more suitable for producing stable rhythmic patterns in controlled environments, limiting their use in general real-world robot tasks. Second, as already listed in the discussion of the paper, the network's design heavily relies on domain knowledge and cannot be generalized across different tasks.\n\n3. The authors performed experiments only on a 6-link swimmer in a simulated environment. This fails to show: 1) How can the NCAP approach be generalized to other robot tasks with different moving patterns? 2) How can the trained model adapt to more realistic environments compared with MLP-based approaches? The authors need to introduce additional case studies and experiments to prove that the proposed method is general enough to be applied to a broad spectrum of robot tasks.\n\n4. The NCAP approach introduces strong heuristics to the network design, limiting the network's maximum performance. Thus, compared with MLP-based approaches, the NCAP-based network has comparable but poorer performance. The authors may need to think about relaxing the heuristics used in the network to improve the performance and generalizability. For example, the second ablation done by the authors in the ablation studies section might be a good direction by relaxing the excitation/inhibition restriction.\n\n5. Other than fewer parameters, the paper failed to explore other advantages of the NCAP-based network compared with the MLP. One potential advantage can be the generalizability of the network with shared weights on different links of the swimmer. The authors can perform experiments on swimmers with different numbers of links using the same trained weight from the 6-link swimmer. The MLP cannot achieve this since the fully-connected network needs to be trained from scratch whenever input/output dimension changes.\n\nMinor Problems\n\n1. Equation f(x)=max(w, 0) in Fig. 2A should be f(x)=max(z, 0).\n\n2. Fig. 3D is not clear. Different connections and neural units need more detailed legends and captions.\n\nMatsubara, Takamitsu, et al. \"Learning CPG-based biped locomotion with a policy gradient method.\" 2006.\n\nLele, Ashwin Sanjay, et al. \"Learning to Walk: Bio-Mimetic Hexapod Locomotion via Reinforcement-Based Spiking Central Pattern Generation.\", 2020.\n\nYao, Chenpeng, et al. \"Humanoid adaptive locomotion control through a bioinspired CPG-based controller.\" 2021.\n\nLopez-Osorio, Pablo, et al. \"Neuromorphic adaptive spiking CPG towards bio-inspired locomotion of legged robots.\" 2021.\n\n\n", "summary_of_the_paper": "This paper proposes a set of bio-inspired Neural Circuit Architectural Priors (NCAP) as ingredients for building trainable networks for continuous control. The authors gave one example of translating the C. elegans locomotion circuits to a network using NCAP to control a swimmer agent in a simulated environment. The proposed network trained by deep reinforcement learning methods (DRL) and evolution strategies (ES) showed comparable performance and more efficient computation with fewer parameters.", "summary_of_the_review": "Overall, the reviewer recommends not accepting the paper. Data efficiency and generalizability are interesting and critical problems for deep learning approaches to robotics. Although the work is correctly motivated, the authors failed to prove that the proposed method is general enough to be applied to a broad spectrum of robot tasks. In addition, the proposed approach shares a lot of similarities with works on central pattern generator (CPG), due to which NCAP has very limited novelty. More detailed issues with the paper are listed in the cons in main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CWIdEMnc3JI", "reviewer": "Reviewer_6cvx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Unfortunately they only consider this one swimmer task, and the architecture was apparently designed with this task in mind. Furthermore this is not a very challenging control tas\nIt seems to me that their hand-crafted architecture (figure 3D) does most of the heavy lifting to solve this task, as can be seen by figure 5A, where the blue and the brown line do not require training at all to have a decent performance.\nOne interesting aspect of biological circuits is that synaptic weights cannot flip sign during training. But it is not clear whether their neurons are excitatory or inhibitory, like in biological circuits. This requires in addition that all weights from a neuron to other neurons have the same sign. Figure 5A shows that only the sign constraint really matters (a lot actually, but perhaps only because they don't allow negative weight initializations without this constraint) undermining their design principles from section 3.2.\n\n\n", "summary_of_the_paper": "The paper addresses a really important challenge: To understand innate contributions to neural circuits for motor control.\nIts goal is to present a set of reusable architectural components and design principles for embodied control. \nThey show that a resulting model can learn to swim more efficiently and requires fewer parameters while achieving similar accuracy as an MLP.\n", "summary_of_the_review": "I do like the direction of research, as it tries to take inspiration from biology to improve ML \nHowever, the way I see it, what they really have is a nice model for a really specific task but not a lot beyond that.\nThe paper did not convince me that their design principles would be beneficial for other tasks, \nIt is also not clear what exactly the principles are that make this single task work well.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2dwaVU6bGRW", "reviewer": "Reviewer_AX4S", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper investigates a method to introduce architectural priors for neural networks for control. These architectural and design choices need to be strictly adapted to the considered agent. For example, the authors accurately describe how their method has to be used for a Swimmer agent. I appreciate how the authors accurately explain the meaning of each component of their method. I also like the choice of the Swimmer agent as a case study, because it is clear how the method can be used for this use case, and the explanation provided by the authors is clear and informative.\n\nDespite the good quality of writing and the clarity of the explanation, I have serious doubts about the contribution provided by this paper. As also stated by the authors, the proposed method has the limit of being strongly dependent on the knowledge of neuroscience and biology; however, most RL and AI researchers are not experts in these fields. I consider this a major drawback of this method, because it strongly limits the impact of this work. I do not think I would be able to use the proposed method in another scenario. Another drawback of the paper is that the claims about sample-efficiency and sparsity are not supported by any theoretical guarantee. It is fairly intuitive that these properties may hold empirically, but stronger motivations would be preferrable, especially because I think they strongly depend on the correctness of the architectural and design choices.\n\nThe experiments are fair results, far from impressive. I understand the importance of the evidence of sample-efficiency due to the big jump start, but I am concerned about the weaker performance compared to some baselines. How can the proposed method, that heavily designs a neural network to fit the considered agent, be outperformed by a “general-purpose” network? I understand that the outperforming network is significantly bigger, but it is not impractically big. In fact, it is a reasonable size that allow efficient forward-backward propagation on regular GPUs.\n\nOverall, for the mentioned reasons, I find this paper to be significantly lower than the acceptance threshold.", "summary_of_the_paper": "This paper introduces a set of architectural components and design principles for constructing neural networks for embodied control. The proposed method is biologically inspired, and can be used to build neural networks specifically designed for the considered agent, e.g., the swimmer agent studied in the paper. The paper provides a literature review, describes the method, and evaluates it in an ablation study using the DDPG algorithm.", "summary_of_the_review": "I think this paper has significant drawbacks that outweigh its strengths. The proposed method is very difficult to use, and although the authors do a good job explaining the single use case, the paper would greatly benefit from the study of more use cases. Despite the good quality of the writing and the clarity of the explanations, I think this paper should be rejected for the lack of usability of the proposed method, the absence of a theoretical study, and the need for a larger number of use cases.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Meta Review for Neural Circuit Architectural Priors for Embodied Control

The motivation of this work is to address an important challenge: To understand innate contributions to neural circuits for motor control. This paper proposes both a set of reusable architectural components and design principles, and also interesting principles for producing biologically-inspired neural networks for embodied control. This work aims to be at the intersection between neuroscience and machine learning for improving the design of artificial neural networks and improving our understanding of observed biological networks. In their model, various components of biological networks are replicated (such as the balance between excitation and inhibition, sparsity, and oscillation). They show that a resulting model, inspired by C.elegans, can learn to swim more efficiently (when evaluated on the Swimmer RL environment) and requires fewer parameters while achieving similar accuracy as an MLP.

Most reviewers, including myself, recognize (and appreciate) the ambition of this work, and are excited at the goal of looking at problems from the perspectives of both system neuroscience and machine learning. The motivations of this paper are clearly explained, and the paper is well written (also the diagrams are great). I'm very excited about this work, and hope to see it succeed, but in the paper's current state (even with the revisions), I don't think it addresses the reviewers' main concerns.

After discussions and examining the paper and the reviews in detail, I feel reviewer GaKc best summarizes the main issues with the work at its current state:

- This paper is interesting but does not proposes a significant improvement to the literature as the gap between the promises made in the motivation and actually delivered work is too wide.

- From a neuroscience point of view, this work does not provide substantial evidence of the importance of the model at either modeling or simulating biological neural systems.

- From a "theoretical" point of view, the model does not provide much advancement to the machine learning community either.

So while the current work (especially in the revised state) I would consider to be an outstanding workshop paper, I cannot recommend it for acceptance at ICLR 2022. An advice I would give to the authors (as someone who publishes to ML conferences, and Sys-Neuro/Bio venues) is that for these ML conferences, it might be easier to make the narrative of the work narrower, and well-defined. If the method is supposed to demonstrate significant advantages of biologically inspired network architecture over current RL methods, the results should clearly demonstrate convincing experimental results that can persuade the (non-neuro) RL community to have interest in the method. If the method does not achieve SOTA results, then try to present the method capable of something really useful that existing RL methods simply fail at (and emphasize that as a core contribution). Conversely, if the narrative is to use a bio-inspired network to emulate biological behaviors, the method must have something important to offer for the community of people working on simulating biological neural systems. 

I look forward to seeing this work improved and eventually published at a journal or presented at a conference in the future, good luck!
Decision: Reject