Neural Circuit Architectural Priors for Embodied Control
Authors: Nikhil X. Bhattasali
Anthony M. Zador
Tatiana A Engel
Venue: NeurIPS 2022 Accept (2022)
Keywords: neuroscience
neural circuits
motor control

Abstract: Artificial neural networks for motor control usually adopt generic architectures like fully connected MLPs. While general, these tabula rasa architectures rely on large amounts of experience to learn, are not easily transferable to new bodies, and have internal dynamics that are difficult to interpret. In nature, animals are born with highly structured connectivity in their nervous systems shaped by evolution; this innate circuitry acts synergistically with learning mechanisms to provide inductive biases that enable most animals to function well soon after birth and learn efficiently. Convolutional networks inspired by visual circuitry have encoded useful biases for vision. However, it is unknown the extent to which ANN architectures inspired by neural circuitry can yield useful biases for other AI domains. In this work, we ask what advantages biologically inspired ANN architecture can provide in the domain of motor control. Specifically, we translate C. elegans locomotion circuits into an ANN model controlling a simulated Swimmer agent. On a locomotion task, our architecture achieves good initial performance and asymptotic performance comparable with MLPs, while dramatically improving data efficiency and requiring orders of magnitude fewer parameters. Our architecture is interpretable and transfers to new body designs. An ablation analysis shows that constrained excitation/inhibition is crucial for learning, while weight initialization contributes to good initial performance. Our work demonstrates several advantages of biologically inspired ANN architecture and encourages future work in more complex embodied control.

Neural Circuit Architectural Priors for Embodied Control
Nikhil X. Bhattasali, Anthony M. Zador, Tatiana A. Engel NeuroAl Program, Cold Spring Harbor Laboratory (bhattas, zador, engel)@cshl, edu
Abstract
Artificial neural networks for motor control usually adopt generic architectures like fully connected MLPs. While general, these tabula rasa architectures rely on large amounts of experience to learn, are not easily transferable to new bodies, and have internal dynamics that are difficult to interpret. In nature, animals are born with highly structured connectivity in their nervous systems shaped by evolution; this innate circuitry acts synergistically with learning mechanisms to provide inductive biases that enable most animals to function well soon after birth and learn efficiently. Convolutional networks inspired by visual circuitry have encoded useful biases for vision. However, it is unknown the extent which ANN architectures inspired by neural circuitry can yield useful biases for other AI domains. In this work, we ask what advantages biologically inspired ANN architecture can provide in the domain of motor control. Specifically, we translate C. elegans locomotion eircuits into ANN model controlling a simulated Swimmer agent, On a locomotion task, our architecture achieves good initial performance and asymptotic performance comparable with MLPs, while dramatically improving data efficiency and requiring orders of magnitude fewer parameters. Our architecture is interpretable and transfers to new body designs. An ablation analysis shows that constrained excitation/inhibition is crucial for learning, while weight initialization contributes to good initial performance. Our work demonstrates several advantages of biologically inspired ANN architecture and encourages future work in more complex embodied control.
Introduction
Artificial neural networks (ANNs) for motor control usually adopt generic architectures like fully connected multi-layered perceptrons (MLPs) (Pierson and (2sshler][01]]]ass
et al.] (016)] Bin Peng al., 2020 Heess al. 20166.. While general, these tabula rasa architectures rely on large amounts of experience to learn. Data efficiency is especially desirable because, unlike computer vision and natural language processing which have benefited greatly from available large datasets [Bommasani et al. 2021] motor control requires interacting with an environment to gather experience, which is come-consuming, laborious, and possibly unsafe [Kroemer et al., 2020). Transfer also a challenge, as experience is usually body-specific, and an ANN trained to control one body is not easily adapted new bodies. In addition, architectures like MLPs are difficult to interpret, as their internal dynamies are distributed across units and non-trivial to relate to agent behavior [Merel et al. 2019a
In nature, animals are born with highly structured connectivity in their brains and nervous systems that has been shaped over millennia by evolution [Zaorr] 2019]. In some cases, this innate circuitry confers abilities with little or no learning; in others, it guides the learning process by providing strong inductive biases [Lake et al.) 2017). These innate and learning mechanisms act synergistically, enabling most animals function well soon after birth, while continuing acquire and improve skills efficiently, e.g. horse learning to walk with only couple hours of experience. Moreover, despite
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

species speccif- variations, there is a significant amount of shared architecture (e.g. cerebellum, basal ganglia) and design principles (e.g. hierarchical modularity, partial autonomy) even between distantly related species |Merel et al., 2019h). The connectivity of neural circuits is often highly structured and sparse [Luo] 2021]. J. in sharp contrast to the all-to-all connectivity of ML.Ps Moreover, evolution has progressively built more advanced abilities on lower-level circuits, leveraging modular structure to transfer existing working designs to new animal bodies [Cisek] [0119] Merel et al. 2019b]. Taken together, these findings from neuroscience suggest that structured neural circuits in animals instantiate efficient, transferable, and modular solutions for high-dimensional embodied control.
Capturing some of this structure in model architecture may enable ANNs to narrow the gap between artificial and natural systems. For example, convolutional networks were inspired by biological visual circuitry, and the inductive biases of spatial locality and weight sharing that they encode have yielded substantial improvements in performance, data efficiency, and parameter efficiency for vision tasks [Lindsay] 2021]. However, it is unknown the extent to which ANN architectures inspired by neural circuitry can yield useful inductive biases for other AI domains.
In this work, we ask what advantages biologically inspired ANN architecture can provide in the domain of motor control. Specifically, we translate C. elegans locomotion circuits into an ANN model controlling a simulated Swimmer agent selected from a standard Al benchmark Tassa et al. 2020]. Our architecture is an instance of what we call a "Neural Circuit Architectural Prior" (NCAP), to denote an ANN architecture that encodes prior structure inspired by biological neural circuits.
On locomotion task, our architecture achieves good initial performance and asymptotic perfor- mance comparable with MLPs, while dramatically improving data efficiency and requiring orders of magnitude fewer parameters. Our architecture is interpretable and transfers to new body designs. An ablation analysis shows that constrained excitation/nhiibtion is crucial for learning, while weight initialization contributes to good initial performance. Our work demonstrates several advantages of biologically inspired ANN architecture and encourages future work in more complex embodied control.
In summary, the primary contributions of this work are:
1. An ANN architecture inspired by C. elegans locomotion circuits that combines the discrete- time ANN formalism that standard in machine learning with features from computational neuroscience like constraints on synapse sign (i.e. excitation vs. inhibition) and special cell types
(i.e. intrinsic oscillators);
2. An evaluation of our model's initial performance, asymptotic performance, data efficiency, parameter efficiency, interpretability, and transfer compared to standard MLP architectures, and
3. An ablation analysis of the effects of weight sharing, sign constraints, initialization, and sparse connectivity on performance and learning.
Code and videos are available at: https://siter wttssiitesgggge....eeee hot.le/.co//veew/caassiiiiii
2 Related Work
Movement Priors A motor controller ultimately generates joint-space torques 7 apply each actuator. While a learning- based controller can directly generate torques [Levine et al. 2016) or world- space positions accelerations ş that are transformed into torques (Khatib] 1987]. movement priors are often adopted to introduce abstraction, incorporate prior knowledge, and improve performance and learning speed Kroemer et al. 2020 Movement priors can be usefully grouped into 3 categories:
(I) Trajectory Priors encode desired movement through analytic equations of motion, c.g. Dynamic Movement Primitives use parameterized differential equations to implement stable attractor dynam- ical systems expressing different motion shapes [Schaal 2006 Pastor et al., 2009], and Policies Modulating Trajectory Generators augment hand ennineeeeeee trajectories with learned residual terms to enhance robustmess and flexibility [[seen et al., [019]. Generally, trajectory priors are interpretable and work well when desired movement is adequately captured by analytic primitives (e.g. sinusoids, splines); however, they can require much time and manual effort to design robustly.
(2) Behavioral Imitation Priors encode desired movement by training parameterized functions like ANNs to imitate reference motions generated from motion capture or manual keyframing, e.g. Bin Peng et al. 22000 train expert policies to imitate animal motion capture, and Neural

---

Probabilistic Motor Primitives compress expert policies trained to imitate human motion capture into a common, reusable embedding space [Merel al., 2019c). Generally, behavioral imitation priors have reproduced diverse movements and trade the time/effort of designing equations with that of compiling reference motions from humans/animals and retargeting the motions to agent bodies.
(3) Architectural Priors encode desired movement indirectly through specialized ANN architectures that establish inductive biases, e.g. HHees et al. [016] decompose an agent into a loosely neuro- inspired hierarchy consisting of a high-level "cortical network" with access to exteroceptive signals and low-level "spinal network" with access to only proprioceptive signals, and Heess et al.. (017] extend this architecture to more challenging locomotion tasks. Generally, architectural priors can significantly improve performance and data efficiency over generic/monolithic architectures; however, architectural priors still represent an mnder-explored space in the AI motor control literature, and much existing work has only specified high-level architectural structure while retaining generic low-level connectivity, e.g. the above-referenced "cortical" and "spinal" networks both use dense MLP connectivity, rather than connectivity that resembles biological cortical and spinal circuitry.
In this work, we investigate the potential of architectural priors that more closely resemble neural circuitry. In doing so, we demonstrate one path for how prior knowledge can be encoded in Al agents and how neuroscience can inspire more naturalistic Al Hasssbis et al. 2017].
Central Pattern Generators In bio-inspired motor control, an extensive literature has built upon models of Central Pattern Generators (CPGs), which are neural circuits found across animal species that produce rhythmic activity in the absence of rhythmic inputs and sensory feedback. These circuits underlie fundamental rhythmic movements, including breathing, chewing, digesting, swimming, walking, and running. Computational models of CPGs have been successfully applied to diverse robotic bodies. For review, refer to [[jppert [2008) and Yu et al. (2014).
While our concept of Neural Circuit Architectural Prior (NCAP) is related to CPGs, it differs in modeling constraints and formalization: (1) Neural-circuit-inspired connectivity can generate movement that includes, but not limited to, rhythmic movement. In animals, of course, discrete movements (c.g. reflexes, reaching, sitting, jumping) are generated by neural circuits too. This work highlights the case study of swimming because C. elegans locomotion circuits are simple explain. but related works have also explored circuits for reflexs [Liu et al. 2018 reaching [Schaal and Schweighofer [2005], and decision making [Lechner et al. [0019 Hasani et al. 2020]. (2) CPG models vary significantly, with some based on neural networks (i.e. architectural priors) and many based on other formalisms including systems of coupled oscillators, vector maps, and finite-state machines [[speert 20088. Neural-network-baseee CPGs further vary between the biophysically detailed, spiking, rate-coded, and population-level [Torres and Varona 2012 Bing et al., 2018 [[speert] 2008). This work adopts discrete-time ANN formalism that is familiar to the broader AI community. In addition to pedagogical benefits, our formalism produces models that are fully differentiable, enabling us to tune parameters with the same reinforcement learning (RL) and evolution strategies (ES) algorithms commonly used in AI motor control. Further, our Swimmer NCAP architecture has the special property of being embeddable within standard MLP of certain dimensions (Appendix B). enabling a direct, rigorous, and novel investigation of low-level connectivity in ANNs.
Neuromechanical Models In computational neuroscience, models of neural circuits have been increasingly combined with models of biomechanics to study setii-bodd--eiirommen interactions (Ausbor et al., 202T Rybak et al., 2015 Danner et al. 2017 Our work builds upon the rich insights gained from neuromechanical modeling, in particulat lar of C. elegans swimming and steering circuits [ooyle et al. 2012 Demin and Vityaev 2014 Izquierdo and Beer 2015 Sarma et al. 2018). However, while neuromechanical work primarily aims to elucidate biological principles and explain neural/behavioral data, our work has the distinct goal of translating insights to AI. To this end, we adopt the discrete-tima ANN formalism as described above, and we also target body from a standard Al benchmark: the DeepMind Control Suite [Tassa et al. 2020)) built on the MuJoCo physies simulator [odorov et al. 2012). In contrast, the above-mentionel C. elegans works use biophysically realistic bodies and muscles. Our Swimmer body is significantly different from C. elegans in mechanies, degrees of freedom, and actuators; it is not obvious that C. elegans circuits would be useful, and our findings may encourage future work for other MuloCo bodies as well, Ultimately, we hope our work can strengthen the bridge between the neuromechanical modeling and Al communities.
3

---

A nematode body
nematode neural circuits
muscle
muscle
head
wave propagation
tail
dorsal
SMB
DB
DD
DB
DD
ventral
AVB
propriccepiion
SMB
VD
excitatory synapse
inhibitory synapee
-W junction
muscle
muscle
Figure 1: Nematode. (A) The nematode achieves forward locomotion through dorsal- muscle contraction waves propagating down the body. (B) Muscle wave propagation, oscillation, steering, and speed control are coordinated by highly stereotyped, modular, and repeated microcireuit. B neurons sense bending in the previous module and excite ipsilateral muscles, while inhibiting contralateral muscles via D neurons. Intrinsic oscillations in B neurons initiate waves. SMB neurons bias head/neck muscles for steering. AVB attenuates all B neurons via gap junction for speed control.
3 Model
We translate nematode (C. elegans) locomotion circuits into an ANN model controlling simulated Swimmer agent. In In[Section 3.1 we provide an overview of the nematode body and the neural circuitry underlying locomotion. In Section 3.2 we describe the integrator and oscillator units that serve as building blocks for our NCAP architecture. In In/Section 3.3 we formalize the observation and action space of the Swimmer agent, and we propose our NCAP architecture inspired by nematode circuits.
3.1 Nematode
The nematode C. elegans has served as a useful model organism within neuroscience because it is one of the simplest organisms with a nervous system. Moreover, it is unique in that its connectome, i.c. wiring diagram, has been completely mapped [Hall and Altun 2008
Nematode Body The nematode body is I mm long, 50 um diameter tapered cylinder (Figure IA). It made up of 959 somatic cells, of which 302 neurons comprising the nervous system, of which 75 are motor neurons that innervate the 95 body wall muscles distributed along the body. Forward and backward thrust is produced via alternating dorsal-ventral muscle contraction waves propagating down the body in the direction opposite to the direction of motion. Steering is produced by differential activation of the 20 anterior muscles in the head and neck [Gjorgjieva et al. 2014).
Nematode Neural Circuits The nematode forward locomotion circuit is summarized below Fig ure BB). For an in-depth treatment, refer to Gjorgjieva et al. [2014] and Wen et al. (2018).
Muscle wave propagation is coordinated by 2 classes of neurons that innervate dorsal (D-) and ventral (V-) muscles. neurons (DB and VB) act as both sensory and motor neurons, expressing stretch receptors in their dendrites to sense bending 200 um anterior to their somas, and sending excitatory output (via ACh) to the muscles and to D neurons. D neurons (DD and VD) send inhibitory output (via GABA) to the muscles. This microcircuit is highly stereotyped, modular, and repeated down the length of the body, and its logic is interpretable. For particular module, body bending in the previous module is sensed by B neurons, which then initiate bending the same side (ipsilateral) while simultaneously inhibiting bending on the opposite side (contralateral) through D neurons.
Muscle wave initiation is generated by intrinsic oscillators. While proprioception-oni circuits (with oscillators ablated) are capable of producing small waves on its own, oscillators are used initiate and entrain larger waves [Gjorgjieva et al. 2014]. These oscillators were long believed to only reside in the head and neck, but recently work has shown them to fact also present in the body as there intrinsic oscillatory activity within the B neurons themselves [Wen et al. 2018

---

integrator unit
B
oscillator unit
2 = b b+ E.(wu)
2
2G
y
y
y = f(t)
-22
y f(2)
width
excitatory weight
w" c* (w) = max u, 0)
square wave
f(t)
high
low
inhibitory weight
(w) min(w, 0)
sine
((1)
amplitude
midline
activation function
f(2) max(z, 0)
period
Architectural Components. (A) An integrator unit models simple neuron. The graded input signals are multiplied by weights that represent synaptic efficacy and that are constrained to be either positive (excitatory, green boxes) or negative (inhibitory, red boxes). The graded output signal is produced by an activation function. (B) An oscillator unit produces driving signals much like intrinsic pacemaker cells and network-based oscillators. The graded output signal is generated by periodic functions, e.g. square waves and sine waves.
Figure 2:
Steering is generated by the differential activation of SMB neurons biasing the head and neck muscles to bend dorsally or ventrally [lzquierdo and Beer 2015].
Speed control is coordinated by the AVB command neuron, which is connected through gap junc- tions with all B neurons. When AVB is low, the resting membrane potentials of B neurons are hyperpolarized to prevent activation; when AVB is high, B neurons are free to activate.
3.2 Architectural Components
Our NCAP architecture is built from components that combine the discrete-time ANN formalism that is standard in machine learning with features from computational neuroscience like constraints on synapse sign (i.e. excitation vs. inhibition) and special cell types (i.e. intrinsic oscillator). The components are fully differentiable and therefore compatible with backpropagation-basee learning algorithms, though not restricted them.
Integrator Units Signals in biological neural circuits are processed and integrated by neurons. The integrator uni in Figure 2/4 is similar to the standard ANN model. The graded inputs i; are multiplied by synaptic weights wi to produce the membrane potential z, given the resting membrane potential b. nonlinear activation function f(z) produces the graded output y based on the membrane potential. Unlike the standard ANN model, however, we constrain synaptic weights by sign constraint function c(w). This is done to reflect that in biological circuits primary characteristic of a synapse is whether it is excitatory or inhibitory. In the standard model, synapses are initialized with random signs and are free to change during learning. We argue that constrained excitation/inhibition is fundamental for interpreting and modeling the logic of neural circuits, and we show in the ablation analysis that they are critical for learning in our architecture (Section 4.5).
Neural circuits often feature components with specialized dynamics, with oscilla- tors being a prominent example [Grillner and El Manira 2020)- An oscillator can be implemented through coupled activity between neurons or within a single neuron, similar to pacemaker cells in the heart [Bccher et al. 2015). Oscillators serve as internal drivers of activity, exemplifying the fact (less appreciated within the ML community) that neural circuits are not exclusively driven by external inputs from the environment. The oscillator unit in Figure 2B uses a periodic function f(() to produce the graded output y. Example periodic functions include square wave and sine wave generators.
Oscillator Units
3.3 Swimmer
The Swimmer is an agent body in widely adopted Al motor control benchmarks, e.g. DeepMind Control Suite [Tassa et al. [220]] and OpenAl Gym [Brockman et al.) 2016]. We target this standard body rather than biorealistic body like previous work [Sarma et al. 2018 Izquierdo and Beer 2015)
'Simple neurons are often approximated as a single integrator units Torres and Varona 2012]. However, sometimes neurons have multiple sites of integration, i.c. dendritic integration across multiple compartments. We prefer "integrator unit" to'neural unit" as complex neuron may require multiple integrator units to model.
5

---

A
swimmer body
head
tail
9
93
dorsal
ventral
94
swimmer network architecture
oscillator
0
B neuron
muscle
B neuron
muscle
771
m
02
8
771
m V
weights:
speed (6), prop (p),
(6)
Figure 3: Swimmer. (A) The Swimmer has an articulated body with N joints connecting + 1 links (N shown). Its observation space is normalized joint positions q. and its action space normalized joint accelerations q. (B) Our network architecture closely conforms to the modular microcircuit of the nematode. Each module senses bending in the previous module fi-1 and drives neurons b; and muscles m,, which are combined to create joint accelerations d;
in order to investigate the potential of biologically inspired network architecture in agents used by the broader AI community and inspire future work in other MuJoCo bodies as well.
Swimmer Body The Swimmer agent has an articulated body with joints connecting + links (Figure 3/). Its movement is entirely within the ry-plane. Thrust is generated by the links pushing against the surrounding fluid, c.g. simulated via high-Reynolds fluid drag model [Todorov et al. 2012). The observation space consists of normalized joint positions limits. The action space consists of normalized joint accelerations q E (-1, 1] between maximum acceleration counterclockwise and clockwise, respectively.
1] between joint
Swimmer Network Architecture Our NCAP architecture is best explained visually (Figure 3B).
For muscle wave propagation, signals are integrated in B neurons and muscles; D neurons mainly serve convert opposite-side neuron signals from excitatory to inhibitory, and their role can be replicated directly in the muscle integrator units. We model N modules control each of the N joints. For a particular module < i < N, the previous module joint position g:-1 1 is split into dorsal Gi11 [0, and ventral qp_ E [0, components, in order to mirror signals from proprioceptive stretch receptors that are sensitive to bending on one side. B neurons are modeled as integrator units with outputs by and by which receive same-side excitatory proprioceptive inputs. Muscles modeled integrator units with outputs m.! and m.. which receive same-side (ipsilateral) excitatory neuron input as well as opposite-side (contralateral) inhibitory neuron input. Finally, the joint acceleration q is caleulated from dorsal and ventral muscle outputs, which act antagonistically.
For muscle wave initiation, the first module B neurons of and bj receive inputs from oscillators od and o", respectively, instead of proprioception. We use square wave generators acting in anti-phase.
For steering, SMB outputs modeled as right turn signal r [0. 1] and a left turn signal / [O, 1], which serve as additional excitatory inputs to first module B neurons of and by, respectively.
For speed control, AVB outputs are modeled as speed signal E [0, 1]. To approximate the effect of gap junctions, such that represents stopping and represents maximum speed, 1 serves additional inhibitory input to all B neurons.

---

The complete architecture for module is therefore:
0i-1 max(qi-1,0)
4:-1 max( 1 max(-qi-o
6 f -
(ppoppii1 + (sppca(1-3))
6g = (ppopp + w Wapcod 1 ss)
m; = f (w { Weomma
m = f (w (iips + woommn i = min(m), 1) min(m,, 1)
with special neuron integrator units for 1:
0f = ( O + Whin" p + W "ispecd (1 - -s)
bj = (woteo" Wiun!
W Wapsos - 8)
We use weight sharing such that weights with the same name are shared across modules as well as within each module across sides. We initialize all weights with the correct signs and magnitudes of 1.
4 Experiments
Learning Formalization We consider agent formalized policy function no <a(a))s;) that maps states 8, to actions a, and which is represented by an ANN parameterized by weights 0 with our architecture described in Seection 3.3 We consider the standard agent-enviromment interaction model formalized as a Markov Decision Process (MDP). every timestep i, the agent state 8, takes action according to its policy a, To(a,)s)) receives reward rs, and transitions to new state 81+1. Policy parameters f are optimized maximize the discounted return E Et=o? T is the horizon of the episode, and 0 < S 1 is the discount factor.
7'r rt, where
Learning Setup
We implement the Swimmer using the standard N = 5 body in the DeepMind Control Suite [Tsssa et al. 2020 built upon the MuloCo physics simulator [Todorov al. 2012 We train the agent to swim using shaped rewards proportional to swimming speed (Appendix A.
Learning Algorithms We compare backpropagation baased RL algorithms (PPO, DDPG) and derivative-free ES algorithm (OpenAl-ES) for tuning parameters in the architecture (Appendix A.2).
4.1 Performance and Data Efficiency
How well and data efficiently does learning occur in NCAP vs. MLP architectures?
each architecture: different algorithms
B
each algorithm: different architectures
MLP(256,256)
NCAP
PPO
DDPG
ES
1000
1000
avg reward
avg reward
800
NCAP
PPO DDPG
200
200
ES
0
11T7
ETTI rm
10F
10F
107
10
10"
107
10F
10F
10¹
10"
107
timesteps
timesteps
timesteps
timesteps
timesteps
Figure 4: Performance and Data Efficiency. (A) Comparison across architectures, using different algorithms. Our architecture starts with high reward and improves with learning, achieving signifi- cantly better data efficiency and comparable performance. (B) Comparison across algorithms, using different architectures. Our architecture with 4 parameters overperforms small MLPs (c.g. MLP(2,2) has 74 parameters) and comparable large MLPs (e.g. MLP(256. 256) has 73,226 parameters). Therefore, structured connectivity (not simply parameter count) matters for performance. Plots show averages over 10 random seeds (solid lines) and 95% bootstrap confidence intervals (shaded areas).

---

First, we compare different learning algorithms for either MLP or NCAP architectures (Figure 4A). We use an MLP with 2 hidden layers of dimensions (256, 256) and ReLU nonlinearities. We find that our NCAP architecture achieves substantially higher initial performance than MIPs as well as comparable asymptotic performance with MIPss, demonstrating the effectiveness of prior knowledge encoded in network architecture. Our NCAP architecture shows reduced variance during learning between trials with different random seeds, as well as reduced differences in asymptotic performance between algorithms. For both MLP and NCAP, ES requires roughly an order of magnitude more data to achieve comparable performance with the RL algorithms, consistent with previous work (Slimaan et al. 2017). Qualitatively, both MLP and NCAP architectures yield reasonable swimming movement (Videos IA-B). though NCAP produces waves with large amplitudes resembling C. elegans movement, while MLP produces waves with small amplitudes more resembling tadpoles. This different movement shape explains the slightly lower asymptotic performance for NCAP hecause the body's is direction of travel is less correlated with the head orientation, which is relevant for how rewards are calculated (Appendix A. Videos 1C). We note that we simplified our design from actual C. elegans circuits for pedagogical reasons (Section 4.5). and our goal is not to solve this swimming task per se but rather to investigate the advantages of biologically inspired network architecture more generally. C. elegans circuits are not optimized for fast swimming with few segments (Section 4.4); future work may propose architectures better for this specific task, e.g. using larval zebrafish circuits.
Second, compare different architectures for each learning algorithm (Figure 4B). We use MLPs with 2 hidden layers of varying dimension sizes and ReLU nonlincarities. We find that performance deteriorates across all algorithms for MLPs as hidden dimensions become smaller, with some algorithms like DDPG deteriorating dramatically. However, our NCAP architecture with 4 parameters overperforms small MLPs and is comparable to large MLPs. This is especially notable the smallest MLP(2,2) has 74 parameters (1 order of magnitude more than NCAP) and the largest MLP(256,256) has 73,226 parameters (2 orders of magnitude more than NCAP). This suggests that the relatively simple structure of our NCAP architecture provides highly effective inductive biases.
4.2 Parameter Efficiency
parameter count & efficiency
logy ( reward /prams)
model
params
How valuable are parameters in NCAP vs. MLP architectures?
(256 256)
73,226
-6.3 0.2
(128, 128)
20.234
0.2
We compare the asymptotic performance divided by parameter count for each architecture (Figure 5). Across all algorithms, our NCAP architecture achieves more than an order of magnitude better parameter efficiency than MLPs of any size. A parameter in our NCAP architecture is "worth more" than one in MLPs.
(64,64)
6,026
-2.6 0.1
(32.32)
1,994
-1.1 0.2
(16,16)
746
(8.8)
1.4 0.4
(4.4)
146
1.3
(2.2)
74
2.6
NCAP
7.8 0.1
Figure 5:Parameter Efficiency.
4.3 Interpretability
Asymptotic reward per unit pa- rameter. Our model achieves better parameter efficiency than MLPs of all tested sizes.
How interpretable are unit dynamics in our NCAP architecture?
We visualize network unit dynamics during task (Videos 2). Since our NCAP architecture is a sparsely connected, modular network where units play constrained roles (e.g. excitatory or inhibitory), it is easier to relate unit dynamics to agent behavior, which facilitates debugging and engineering constraints for safety.
fewer/more joints
1000
800
600
4.4 Transfer
400
How well does our NCAP architecture adapt to new bodies?
200
PPO
DDPG
We leverage the modular structure of our NCAP architecture to adapt a trained network for N = 5 joints to bodies with different N' joints by adding/removing modules (Figure 6). Even without further training, we achieve robust swimming for a wide range of N' (Videos 3). Interestingly, performance is better for longer bodies, possibly reflecting that nematode circuits are evolved for a more segmented body. This kind of zero-shot transfer is not possible with monolithic, densely connected MLP architectures.
0
3
5
7
11
13
joints
Figure 6: Transfer. After train-
ing with N joints, our model can transfer to different N' by adding/removing modules.
8

---

A weight sharing, sign constraints, initialization
B sparse connectivity
PPO
ES
MLP(12,10,10)
1000
share sign ieik sgn
1000
avg reward
800
chars- and
son
avg reward
800
600
800
400
400
PPO
share hare ord inil
DDPG
ES
11
4
FTEI Triing Fm
10
10P
10P
107
10
10"
10'
timesteps
timesteps
timesteps
Figure 7: Ablations. (A) Ablations of weight sharing, sign constraints, and initialization in different combinations. Sign constraints are crucial for learning. Weight initialization contributes to good initial performance. Weight sharing yields small gain. (B) Ablation of sparse connectivity yields an equivalently sized MLP (Appendix B). Learning is restored without sparse connectivity.
4.5 Ablations
What are the effects of various features of our NCAP architecture on performance and learning?
First, we investigate the role of weight sharing, sign constraints, and weight initialization (Figure 7)). Without weight sharing, weights across modules and across sides are separate parameters, increasing the total number of parameters from 4 to 30. Without sign constraints, the identity function is applied to weights instead of the constraint function. Without typical weight initialization, weight magnitudes are initialized through uniform random distribution within [0, I]. rather than at 1. If using sign constraints, weights are always initialized with the appropriate sign; otherwise, signs are chosen randomly with equal probability. We find that sign constraints are crucial for learning. Without appropriate sign constraints, our NCAP architecture fails to learn during the allotted timesteps, for both RL and ES algorithms. With appropriate sign constraints, even if the weights magnitudes are not initialized large, our NCAP architecture will learn. Weight initialization is responsible for good initial performance. Weight sharing has a smaller, but identifiable, contribution to data efficiency.
Second, we investigate the role sparse connectivity that arises from the natural structure of neural circuits (Figure 7). Our Swimmer architecture has the special property that it can be completely embedded within an MI.P with 3 hidden layers of dimensions (12, 10, 10) and ReLU nonlinearities (Appendix B). Specifically, after ablating sign constraints and weight sharing, our architecture is identical to this MLP with highly pruned connectivity (mostly weights of O). We remove this sparsity and find that the MLP can learn the task with similar asymptotic performance our NCAP
Taken together, our results suggest that constrained excitation/inhiiition is an important consideration in small, sparse architectures like our NCAP, but less important in MLPs. This may be related to the "Lottery Ticket Hypothesis" [Frankle and Carbin 2019], which suggests that, upon initialization, the MLPs already contain subnetworks with initial weights and signs that do most of the work for learning, i.e. they are "winning tickets"; imposing sparsity eliminates these overlapping subnetworks.
Discussion
We asked what advantages biologically inspired ANN architecture can provide in the domain of motor control. Through our case study translating C. elegans locomotion circuits into an ANN model, we found that biologically inspired ANN architecture can achieve comparable asymptotic performance to MLPs with significantly improved initial performance, data efficiency, parameter efficiency, interpretability, and transfer. Therefore, while tabula rasa architectures are general, Neural Circuit Architectural Priors (NCAP) can provide useful inductive biases for motor control.
Nevertheless, case study here was limited: it focused relatively simple body and circuit, and it prioritized pedagogical simplicity over performance optimization and behavioral complexity. Future work should explore additional bodies, movement types, and circuit modalities:
(1) Bodies. Many animals generate movement through highly structured pattern generator circuits, including guadrupeds (c.g, mice, cats, dogs, horses) and bipeds (e.g. birds, humans) [[ispeert 2008

---

In particular, quadrupeds are commonly studied in both neuroscience and robotics [Rybak et al. 2015 Danner et al., 2017 Iscen et al., 2019]. Quadruped locomotion circuits are located primarily within the spinal cord and produce robust gaits (e.g. walk, bound, gallop) even when top-down connections from the brain are lesioned [Buschmann et al., 2013). Insights from these circuits could be translated to Al motor control bodies, and a Quadruped NCAP architecture may use similar integrators, oscillators, and constrained connections as our Swimmer NCAP
(2) Movement Types. Neural circuits coordinate diverse animal movements, both rhythmic (e.g. breathing, chewing, swimming, walking) and discrete (e.g. reflexes, reaching, sitting, jumping). Therefore, NCAP architectures that closely resemble neural circuit mechanisms could in principle be applied to a variety of additional movement types.
(3) Circuit Modalities. Biologically inspired ANN architecture may also provide useful inductive biases for upstream tasks involving perception and decision making. In animals, lower-level pattern generator circuits are modulated by higher-level control circuits. For instance, locomotion speed and direction are modulated by orientation and escape circuits in response to visual, auditory, and tactile stimuli. Such higher-level circuits could inspire NCAP architectures for additional modalities.
Ultimately, our work suggests way of advancing Al and robotics research inspired by systems neuroscience and encourages future work in more complex embodied control.
Acknowledgments and Disclosure of Funding
Thanks to Sergey Shuvaev, Pavel Tolmachev, Liam McCarty, Polina Kirichenko, Pavel Izmailov, Daniel Barabasi, Christopher Langdon, and Josh Merel for insightful discussions and manuscript feedback. This work was supported by Schmidt Futures (N.X.B.); the G. Harold and Leila Y. Mathers Charitable Foundation (A.M.Z.); the Eleanor Schwartz Charitable Foundation (A.M.Z.): NIH grants S $100D028632-01 and RF1DA055666 (T.A.E.); and the Alfred P. Sloan Foundation Research Fellowship (T.A.E.).
References
J. Ausbomn N. Shevtsova, and S. M. Danner. Computational modeling of spinal locomotor circuitry in the age of molecular genetics. International Journal of Molecular Sciences, 22(13))6835,, June 2021.
X. Bin Peng, E. Coumans, T. Zhang, T.WW Lee, J. Tan, and S. Levine. Learning agile robotic locomotion skills by imitating animals. In Robotics: Science and Systems XVI. Robotics: Science and Systems Foundation. July 2020.
Z. Bing, C. Meschede, F. Röhrbein, K. Huang, and A. C. Knoll. A survey of robotics control based on learning-inspired spiking neural networks. Frontiers in Neurorobotics, 12:35, July 2018.
Bommasani, D. A. Hudson, Adeli, Altman, Arora, von Arx, M. Bernstein, Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, Buch, D. Card, R. Castellon, N. Chatterji. A. Chen, K. Creel, J. Q. Davis, D. Demszky, C. Donahuc, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, L. Fei-Fei, C. Finn, T. Gale, L. Gillespie, K. Goel, N. Goodman, S. Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D. E. Ho, J. Hong, K. Hsu, J. Huang, T. Ieard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling, F. Khani,
Khattah, P. W. Kohd, M. Krass, R. Krishna, R. Kuditipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee. J. Leskovec. L. Levent, X. L. Li, X. Li, T. Ma. A. Malik, C. D. Manning. S. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair. A. Narayan, D. Narayanan, B. Newman, A. Nie. J. C. Niebles. H. Nilforoshan. Nyarko, G. Ogut, L. Orr. I. Papadimitriou. S. Park, C. Piech, E. Portelance. C. Potts, Raghunathan,
Reich, H. Ren, F. Rong, Y. Roohani, C. Ruiz, Ryan, C. Ré, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, F. Tramèr, R. Wang, W. Wang, B. Wu, J. Wu, Y. Wu, S. M. Xie, M. Yasunaga, J. You, M. Zaharia, M. Zhang, T. Zhang, X. Zhang, Y. Zhang, L. Zheng, K. Zhou, and P. Liang, On the opportunities and risks of foundation models.
S.ii:2008.0725:: [cz], Aug. 2021.
J. H. Boyle, S. Berri, and N. Cohen. Gait modulation in C. elegans: An integrated neuromechanical model. Frontiers in Computational Neuroscience, 6, 2012.
G. Brockman, V. Cheung, I.. Pettersson, J. Schneider, Schulman, Tang, and W. Zaremba. OpenAl Gym. arXiv:1606.01544 [es], June 2016.
D. Bucher. G. Haspel, J. Golowasch, and F. Nadim. Central pattern generators. In eLS. John Wiley & Sons, Ltd. Nov. 2015.
10

---

T. Buschmann, A. Ewald, A. von Twickel, and A. Büschges. Controlling legs for locomotion incssotion-insighht from robotics and neurobiology. Bioinspiration & Biomimeticx, 10(4):041001, June 2015.
P. Cisek. Resynthesizing behavior through phylogenetic refinement. Attention, Perception, & Psychophysics, 81 (7):2265 1122266-2287,, Oct. 2019.
M. Danner, N. Shevtsova, Frigon, and I. A. Rybak. Computational modeling of spinal circuits controlling limb coordination and gaits in quadrupeds. eLife, 6:e31050, Nov. 2017.
A. Demin and E. Vityaev. Learning in a virtual model of the C. elegans nematode for locomotion and chemotaxis. Biologically Inspired Cognitive Architectures, 7:9-14, Jan. 2014.
J. Frankle and M. Carbin. The Lottery Ticket Hypothesis: Finding sparse, trainable ncural networks. 1xX3v:8803.0363 [es], Mar. 2019.
J. Gjorgjieva, D. Biron, and G. Haspel. Neurobiology of Caenorhabditis elegans locomotion: Where do we stand? BioScience, 64(6):476 486, June 2014.
S. Grillner and A. El Manira. Current principles of motor control, with special reference vertebrate locomotion. Physiological Reviews, 100(1):271-322)) Jan. 2020.
H. Hall and Altun. C. Elegans Atlas. Cold Spring Harbor Laboratory Press, Cold Spring Harbor, N.Y. 2008.
R. Hasani, M. Lechner, A. Amini, D. Rus, and R. Grosu. natural lottery ticket winner: Reinforcement learning with ordinary neural circuits. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4082- 4093. PMLR, July 2020.
D. Hassabis, D. Kumaran, C. Summerfield, and M. Botvinick. Neuroscience-inspired artificial intelligence. Neuron, 95(2):245 258, July 2017.
N. Heess, G. Wayne, Y. Tassa, T. Lillicrap. M. Riedmiller. and D. Silver. Learning and transfer of modulated locomotor controllers. arXivv/610.05182 [cs], Oct. 2016.
N. Heess, D. TB, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, S. M. A. Eslami, M. Riedmiller, and D. Silver. Emergence of locomotion behaviours in rich environments. arXiv:1707.02286 [cs], July 2017.
A. J. Ijspeent. Central patterm generators for locomotion control in animals and robots: A review, Neural Networks, 2144)6642-653, May 2008
A. Iscen, K. Caluwaerts, Tan, T. Zhang, E. Coumans, V. Sindhwani, and V. Vanhoucke. Policies modulating trajectory generators. arXiv:1910.02812 [cs]. Oct. 2019.
E. Izquiendo and R. Beer. An integrated neuromechanical model of steering C. elegans. ECAL 2015: The 13th European Conference on Artificial Life, pages 199-206, July 2015.
G. Karakasli. ESTorch: An evolution strategy library built around Py 2020.
O. Khatib, A unified approach for motion and force control of robot manipulators: The operational space formulation. IEEE Journal on Robotics and Automation, 3(1):43-53, Feb. 1987.
O. Kroemer, S. Niekum, and G. Konidaris. A review of robot learning for manipulation: Challenges, representa- tions, and algorithms. arXiv: arXiv:1907.03: 196 [es), Nov. 2020.
B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40:e253, 2017.
M. Lechner, R. Hasani, M. Zimmer, T. A. Henzinger, and R. Grosu. Designing Worm-inspired Neural Networks Interpretable Robotic Control. In 2019 International Conference on Robotics and Automation (ICRA), pages 87-94, Montreal, QC, Canada, May 2019. JEEE
S. Levine, C. Finn, Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. arXiv:1504.00700 [cs), Apr. 2016.
T. P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, Continuous control with deep reinforcement learning. 10Xiv:150900297) [cs, stat], July 2019.
11

---

G. W. Lindsay. Convolutional neural networks as model of the visual system: Past, present, and future. Journal of Cognitive Neuroscience, 33(10):017722331 Sept. 2021.
X. Liu, Rosendo, S. Ikemoto, M. Shimizu, and K. Hosoda. Rohotic investigation effect of stretch reflex and crossed inhibitory response on bipedal hopping. Journal of The Royal Society Interface, 15(140)2201 80024, Mar. 2018.
L. Luo. Architectures of neuronal circuits. Science, 373(6559), Sept. 2021.
J. Merel, D. Aldarondo, J. Marshall, Y. Tassa, G. Wayne, and B. Ólveczky. Deep neurocthology of a virtual rodent. arXiv: 1911.9991 [q-bio), Nov. 2019a.
J. Merel, M. Botvinick, and G. Wayne. Hierarchical motor control in mammals and machines. Nature Communications, 10(1):5489. Dec. 2019b.
Merel, Hasenclever, Galashov, A. Ahuja, V. Pham, G. Wayne, Y. Teh, N. Heess. Neural probabilistic motor primitives for humanoid control. 1rX.1:1811.11111 [cs]. Jan. 2019c.
F. Pardo. Tonic: A deep reinforcement learning library for fast prototyping and benchmarking. arXiv:001 [cs], May 2021.
P. Pastor, H. Hoffmann, T. Asfour, and S. Schaal. Learning and generalization of motor skills by learning from demonstration. In 2009 IEEE International Conference on Robotics and Automation, pages 763-768, Kobe, May 2009, IEEE.
A. Paszke, Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang. Z. DeVito, M. Raison, A. Tejani, S, Chilamkurthy, B. Steiner, L. Fang. J. Bai, and S. Chintala. PyTorch: An imperative style, high-performance deep learning library. arXiv: 1912.01703 fes, stat], Dec. 2019.
H. Pierson and M. Gashler. Deep leamming in robotics: review of recent research. Advanced Robotics, 31 (16):821-335 Aug. 2017.
L. A. Rybak, K. Dougherty, and N. A. Shevtsova. Organization of the mammalian locomotor CPG: Review of computational model and circuit architectures based on genetically identified spinal interneurons. eneuro, 2 (5)EENEURO.009- 10069-155.000 15.2015, Sept. 2015.
T. Salimans, Нo, X. Chen, S. Sidor, and I. Sutskever. Evolution strategies scalable alternative reinforcement learning. arXiv: 0ee.1170300666 [cx, stat], Sept. 2017.
G. P. Sarma, C. W. Lee, T. Portegys, V. Ghayoomie, Jacobs, B. Alicea, M. Cantarelli, Currie, R. Gerkin, S. Gingell, P. Gleeson, R. Gordon, R. M. Hasami, G. Idili, S. Khayrulin, D. Lung, A. Palyanow, M. Watts, and S. D. Larson. OpenWorm: Overview and recent advances in integrative biological simulation of Caenorhabditis elegans. Philosophical Transactions of the Royal Society B: Biological Sciences. 373(1758): 20170382, Oct. 2018.
Schaal. Dynamic movement primitives a framework for motor control in humans and humanoid robotics. In H. Kimura, K. Tsuchiya, A. Ishiguro, and H. Witte, editors, Adaptive Motion of Animals and Machines, pages 261 261-280. Springer- Verlag, Tokyo, 2006.
S. Schaal and N. Schweighofer. Computational motor control in humans and robots. Current Opinion in Neurobiology, 15(6))675-682 682, Dec. 2005.
Schulman, Wolski, Dhariwal, Radford, and Klimov, Proximal policy optimization algorithms. arXiv:1707.06347 [ex], Aug. 2017.
Tassa, S. Tunyasuvunakoo,, A. Muldal, Y. Doron, P. Trochim, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lillicrrp. and N. Heess. Dm_control Software and tasks for continuous control. Software Impacts, 6:1 6:100022, Nov. 2020.
E. Todorov, T. Erez, and Y. Tassa. MuloCo: A physies engine for model-based control. In 2012 JEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026-5033, Vilamoura-Algarve, Portugal, Oct. 2012. IEBE.
J. J. Torres and P. Varona. Modeling biological neural networks. In G. Rozenberg, T. Bäck, and N. Kok, editors, Handbook of Natural Computing, pages 533-564 Springer Berlin Heidelberg, Berlin, Heidelberg. 2012.
12

---

Q. Wen, S. Gao, and M. Zhen. Caenorhabditis elegans excitatory ventral cord motor neurons derive rhythm for body undulation. Philosophical Transactions of the Royal Society B: Biological Sciences, 373(1758): 20170370, Oct. 2018.
J. Yu, M. Tan, J. Chen, and Zhang. A Survey CPG-Inspired Control Models and System Implementation. IEEE Transactions on Neural Networks and Learning Systems, 25(3))441-44) Mar. 2014.
A. M. Zador. A critique of pure learning and what artificial neural networks learn from animal brains. Nature Communications, 10(1):3770, Dec. 2019.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See SeefSection 2 Section S
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] See Supple- mental Material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See FFiuure 4fFigure 5|Figure
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See See/Appendix A
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Appendix A
(b) Did you mention the license of the assets? [Yes] See Appendix A
(c) Did you include any assets either in the supplemental material URL? [Yes] Code under MIT license.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14

---

A Experimental Details
A.1 Tasks
The swim task aims to test the agent's ability to swim forwards at a desired speed. It returns smooth reward that is 0 when stopped or moving backwards, and rises linearly and saturates when swimming at the desired speed.
A.2 Learning Algorithms
Proximal Policy Optimization (PPO) [Schulman al., 2017) A model-free, on-policy, policy gradient RI. method. uses clipped surrogate objective to limit the size of policy change at each step, thereby improving stability. Since it assumes stochastic policies, we perturb the deterministic actions with Gaussian noise E, ~ ~N,(0, o²).
Deep Deterministic Policy Gradient (DDPG) (Lillicrap al. 2019 A model-free, off-policy, policy gradient RL method. It uses off-policy data and the Bellman equation to learn the Q-function, which is iteratively used improve the policy.
Evolution Strategies (ES) Salimans et al., 2017] An evolutionary black-box optimization method. creates population of policy parameter variants through perturbations with Gaussian noise, then combines them through averaging, weighted by the return collected across episodes.
A.3
Implementation
Libraries
Neural networks were implemented in PyTorch (BSD) license)
Paszke al. (019]. The
RL algorithms were implemented using Tonic (MIT license) Pardo 2021 implemented using ES Torch (MIT license) [arakaslii 2020
The ES algorithm was
Computational Resources Training was performed on a high performance computing cluster running the Linux Ubuntu operating system. RI algorithm training runs were parallelized over cores, while ES algorithm runs were parallelized over 32 cores.
A.4 Hyperparameters
RL Algorithms Standard hyperparameters for PPO and DDPG in Tonic 48a7b72; timesteps, 5e6.
Pardo 2021
commit
ES Algorithm optimized, Adam; learning rate, 0.01; timesteps, 5e7.
Population size, 256; noise standard deviation 0.02; L2 weight decay, 0.005;
NCAP Swimmer Oscillator, square wave, period 60 timesteps, width 30 timesteps.
15

---

B Swimmer Architecture Details
Our NCAP architecture has the special property that can be completely embedded within fully connected MLP of hidden layers and ReLU nonlinearities. This enables us to "interpolate" between our NCAP architecture and the MLP architecture, conducting rigorous analysis of how various features of our architectural prior contribute to performance and learning.
By rearranging terms in the Swimmer network architecture diagram (Figure 3B), we arrive the following network (N = 5 shown):
weights: 00. speed
task module
motor module
weights: oee prop (o),
Ito b)
61
41
62
172
42
2
by
m2
25
43
43
45
44
44
01
4
m
4
0/
m0
45
05
0%
mg
By removing weight sharing, sign constraints, and sparse connectivity, we arrive at a fully connected MLP of 3 hidden layers (N 2 shown):
od
0d
NCAP
MLP
no weight sharing sign constraints
m
q
m
91
no sparse connectivity
9
m
9
b
m,
42
m
92
d
bg
m
42
42
42
by
m
42
m2
For N = 5, the resulting MLP has hidden layers of dimensions (12, 10, 10).
16


Reviews:
[{"review_id": "xx-clmzelKJ", "reviewer": "Reviewer_uQAH", "summary": "\nNeural Circuit Architectural Priors for Embodied Control\nIn this paper, the authors are inspired by the Caenorhabditis elegans locomotion circuits, and design a structurally similar neural network structure. This prior structure provides valuable starting motion control knowledge and enables faster training in the task of the swimmer from the deepmind control suite.\n", "questions": "1) From a neuroscience’s perspective, maybe the authors can comment on the connection between the proposed algorithm and some other neuro-inspired network structures? For example the feedback alignment network in [3], [4] and [5]\n2) Back in the “Strengths And Weaknesses”, I mentioned the lack of experiments. Could the authors provide some comments on whether or not the algorithm can be applied to other creatures and other tasks?\n3) The periodic phase generation module, while not exactly the same, was proposed and used in some character animation research such as [1], [2]. These are some literature I think is related to the oscillator units discussed in the paper.\n", "limitations": "\nI don’t see any potential negative societal impact in this work.\n\n[1] Van de Panne, Michiel, Ryan Kim, and Eugene Fiume. \"Virtual wind-up toys for animation.\" In Graphics Interface, pp. 208-208. CANADIAN INFORMATION PROCESSING SOCIETY, 1994.\n[2] Holden, Daniel, Taku Komura, and Jun Saito. \"Phase-functioned neural networks for character control.\" ACM Transactions on Graphics (TOG) 36, no. 4 (2017): 1-13.\n[3] Lillicrap, Timothy P., Daniel Cownden, Douglas B. Tweed, and Colin J. Akerman. \"Random synaptic feedback weights support error backpropagation for deep learning.\" Nature communications 7, no. 1 (2016): 1-10.\n[4] Nøkland, Arild. \"Direct feedback alignment provides learning in deep neural networks.\" Advances in neural information processing systems 29 (2016).\n[5] Frenkel, Charlotte, Martin Lefebvre, and David Bol. \"Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks.\" Frontiers in neuroscience 15 (2021): 629892.\n\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nOriginality: This paper draws a very interesting connection between neuroscience and reinforcement learning. Since I have no previous knowledge in biology or neuroscience, I cannot comment on how novel this idea is. From the reinforcement learning point of view, I think it’s pretty novel and inspiring.\n\nQuality and clarity: The paper is well written. The related work is pretty adequate, and the figures in the paper (for example Figure 1 and Figure 2) are pretty intuitive and easy to understand.\nThe experiments are well designed, with every experiment having clear purposes and explanations. \n\nSignificance: How well does it scale to high-dimensional agents? How well does it scale to other tasks? I think these are some of the questions that would greatly affect how significant the proposed algorithm is in the community of reinforcement learning (I cannot comment on the neuroscience part unfortunately).\nWithout experiments on more creatures (for example the ant like creatures, the fish like creatures in the deepmind control suite), or more experiments for different tasks that aim to show the locomotion skills not only on swimming but also on reaching, avoiding, and even 3D swimming, the significance of the algorithm remains unclear.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "uAI6X6hm1wo", "reviewer": "Reviewer_R6Ad", "summary": "The paper presents a method to embed priors for motor functions to artificial neural networks. The authors demonstrate a rather simple example of using architecture of C.elegans to control swimmer in mujoco control suite problem. The proposed method has similarities with previous methods such as CPGs, but it has clear differences as the authors explain. Using the architecture proposed by the authors, the controller for the 5 link swimmer is reduced to 4 parameters. The training results show pretty much instant learning curve with converged policies comparable to MLPs trained with Reinforcement Learning or Evolutionary Strategies. ", "questions": "Would it be possible to give some guidelines on how to apply the given design principles to more complex problems such as legged locomotion (or even 2D walker problem)?", "limitations": "The main limitation is the lack of scalability of the method on more complex problems. It would be great to see authors give different architectures for different problems instead of a single one (swimmer).", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of the paper is to add prior knowledge and structure to artificial neural networks for motor controlling tasks. This is a very important point for reinforcement learning and robotics. It reduces the number of samples required for learning and it introduces interpretability to the network. \n\nAs one would expect, the results show that the same task can be learned with way less parameters in much smaller number of samples. Adding structure also provides some bounds to safety during learning, but the authors did not emphasize this point much.\n\nThe paper is very well written. The concepts and similar works are explained in details in a nice flow. It is very easy to follow the main idea while also understanding the details of the method.\n\nIn terms of weaknesses, as the authors acknowledge the authors demonstrate the method on a rather simple problem (swimmer), and they do not give a clear guideline to extend it to more complex motor control problems such as legged locomotion. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "qmyFXQbBUB4", "reviewer": "Reviewer_8ee3", "summary": "This paper introduces a new neural network architecture, \"NCAP\" (Neural Circuit Architectural Priors), for performing a locomotion task (the Swimmer task from the DeepMind Control Suite).  The network is inspired by the locomotion circuits of C. elegans.  The authors train both NCAP, as well as various MLP architectures to perform the Swimmer task, and do so using 3 different RL training methods (PPO, DDPG and ES, an evolutionary black-box optimization method).  The NCAP architecture consistently achieves higher average reward earlier in training compared to the MLP architectures, while having significantly fewer trainable parameters. ", "questions": "My questions were outlined above, but I repeat these here for completeness:\n\n1. Clarity: Please clarify what the state at time $t$, $s_{t}$, corresponds to on line 240.  You give the inputs to NCAP in the equations of Section 3, but please clarify the exact inputs you are giving to the MLP comparison networks.  Is $s_{t}$ for the MLP networks just $\\{o, q_{1}, q_{2} ,..., q_{N}\\}$ or does this also include the speed $s$ and $r$ and $l$?  From Appendix B, it looks like you don't give the speed $s$ and $r$ and $l$ to the MLP.  \n2. Clarity/quality: It seems strange to me that speed is given as an input to NCAP, given that reward is based on speed.  This is especially true if you only give speed as an input to NCAP and not to the MLP networks.  To what extent can the early success of NCAP relative to the MLP networks be attributed to the fact that speed is given as an input to NCAP?  It would have been nice to see an ablation study where the authors removed the speed input in order to get a sense of how crucial this input is.  Furthermore, it would nice to have been able to see an expanded x-axis for Figure 4A so as to know when NCAP performance first reached the avg reward of 800 (was NCAP achieving this reward from time=0 or time=$10^{4}$?)\n3. Significance: while the results of NCAP on the Swimmer task are impressive, I am not sure of the broader significance of this paper.  This is especially true because of the fact that it appears that the asymptotic performance of NCAP is lower than that of some MLP architectures (Figure 4).  Are there other tasks where the NCAP architecture can be applied?  Or is the take-away message from this paper that sign constraints on weights in neural networks are important, and enable faster learning?  I am excited to hear what the authors have to say on this point: what do you think the broader implications of your paper are - beyond the Swimmer task?    ", "limitations": "The Discussion section for this paper was a little lackluster, and the authors could have been more explicit about the limitations of their work. For example, one limitation is that you've only demonstrated the utility of the NCAP architecture on the Swimmer task.  Another limitation is that the asymptotic performance of NCAP seems to be lower than that of the MLP architectures (Figure 4).  ", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Originality: despite the connectome of C. elegans having been available for years, I don't think there are many papers trying to analyze the key components of the anatomy of C. elegans that enable these worms to learn efficiently, as well as the implications of the C. elegans connectome for building AI systems.  In this sense, the paper tackles interesting, under-explored questions\n2.  Quality: the paper is relatively well written.  The authors are thorough in their review of Related Work, and the figures - especially the early figures that give the overview of their framework and the nematode - are clear and easy to understand.\n\nWeaknesses: \n1. Clarity: Please clarify what the state at time $t$, $s_{t}$, corresponds to on line 240.  You give the inputs to NCAP in the equations of Section 3, but please clarify the exact inputs you are giving to the MLP comparison networks.  Is $s_{t}$ for the MLP networks just $\\{o, q_{1}, q_{2} ,..., q_{N}\\}$ or does this also include the speed $s$ and $r$ and $l$?  From Appendix B, it looks like you don't give the speed $s$ and $r$ and $l$ to the MLP.  \n2. Clarity/quality: It seems strange to me that speed is given as an input to NCAP, given that reward is based on speed.  This is especially true if you only give speed as an input to NCAP and not to the MLP networks.  To what extent can the early success of NCAP relative to the MLP networks be attributed to the fact that speed is given as an input to NCAP?  It would have been nice to see an ablation study where the authors removed the speed input in order to get a sense of how crucial this input is.  Furthermore, it would nice to have been able to see an expanded x-axis for Figure 4A so as to know when NCAP performance first reached the avg reward of 800 (was NCAP achieving this reward from time=0 or time=$10^{4}$?)\n3. Significance: while the results of NCAP on the Swimmer task are impressive, I am not sure of the broader significance of this paper.  This is especially true because of the fact that it appears that the asymptotic performance of NCAP is lower than that of some MLP architectures (Figure 4).  Are there other tasks where the NCAP architecture can be applied?  Or is the take-away message from this paper that sign constraints on weights in neural networks are important, and enable faster learning?  I am excited to hear what the authors have to say on this point: what do you think the broader implications of your paper are - beyond the Swimmer task?    \n\nMinor points:\n1.  Clarity: Are there not 6 parameters for NCAP?  $w_{prop}, w_{speed}, w_{turn}, w_{osc}, w_{ipsi}, w_{contra}$?  In Table 5, you say that there are 4 parameters\n4. Clarity: there are many constrained parameters and outputs for the NCAP network.  While you clarify how weights are appropriately constrained to have the desired signs, I wasn't clear on how you constrain the output of the network to be in [-1, 1] (line 214).\n\nExtremely minor point:\n1. Quality (I only include this here because I noticed it when I zoomed into the figure and I would want to correct it if this was my paper): the x tick labels on Figure 6 are not vertically aligned", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "lNp1mAfUXhp", "reviewer": "Reviewer_QpQM", "summary": "The authors introduce NCAP, a neural network architecture inspired by computational models of actual nematode locomotion. Experiments are done on the \"swimmer\" environment, which bears some relationship to nematode locomotion. They show that when optimized for forward velocity (via RL or ES), NCAP has much better initial performance than conventional architectures, despite having far fewer numerical parameters. Ablations reveal than the sign constraints in particular are crucial for performance. Finally, the authors show that training transfers to bodies with a different number of joints.", "questions": "1) Would NCAP work when optimizing a different reward function, like reaching a goal location?\n\n2) Would NCAP work for non-swimmer environments? Particularly those with similarly structured action spaces (joint accelerations).\n\n3) Why should we care about the parameter counts of small models independent of data efficiency? Even the largest model considered here isn't bottlenecked by memory or compute, and it's unclear these gains would hold for larger models. In particular, why is the reward/params metric in Figure 5 meaningful?\n\n4) What do the transfer results (Figure 6) look like for a fresh initialized NCAP agent (i.e. no training)?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n1) This is a unique perspective. It's rare for work in systems neuroscience to be fully translated into models that can be optimized with methods designed for deep learning.\n\n2) The writing is clear throughout, though more insight into the motivation behind the unit types would be nice. E.g. way are oscillator units preferable to under ways of maintaining internal state (such as LSTMs)?\n\n3) Ablations are thorough, though again more insight into these results would be appreciated.\n\n# Weaknesses\n\n1) It's unclear the extent to which learning matters here. Initial performance looks to be around 800 and final performance is perhaps 850. This is a weakness, because simply hardcoding a solution to a simple locomotion problem is not a significant AI result. I'm not convinced that this'll be useful for anything other than the exact task/body setup you use here.\n\n2) The experimental results are very limited in scope. All of the AI papers mentioned in related works involve techniques being applied across a range of embodiments and tasks. And even within the constraints of the swimmer environment, none of these related works are compared against.\n\n3) Misc. Figure 4 shouldn't use a log x axis (or at least have both versions visible somewhere). Transfer is stated as not possible for MLPs, but ignores work adapting deep learning methods to due just that: \"One policy to control them all: Shared modular policies for agent-agnostic control\". I'm not convinced that parameter-count is a meaningful metric in this regime, but even if it were \"Weight Agnostic Neural Networks\" could likely produce a single-parameter policy class with similar performance properties to NCAP.\n\n#Takeaway\n\nI would strongly encourage resubmitting after more general results are obtained. Alternatively, I think this work could be reframed as a contribution to neuroscience rather than AI, by emphasizing the testing the functional role of various nematode cells, etc and downplaying the potential utility in AI applications. Though this would still require considerable reworking and would likely be better served by a different publication venue.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept