Neural Circuit Architectural Priors for Embodied Control

1 INTRODUCTION . Artificial neural networks for simulated motor control and robotics often adopt generic architectures like fully connected multi-layered perceptrons ( MLPs ) ( Pierson & Gashler , 2017 ; Levine et al. , 2016 ; Bin Peng et al. , 2020 ; Heess et al. , 2016 ) . While general , these tabula rasa architectures rely on large amounts of experience to learn . Data efficiency is especially important in motor control because , unlike computer vision and natural language processing which have greatly benefited from the availability of large datasets ( Bommasani et al. , 2021 ) , gathering large-scale data for robotics is challenging . Experience must be gathered through interaction with the environment , which is difficult for reasons including time , human labor , safety , maintenance , and reproducibility ( Kroemer et al. , 2020 ) . Transfer is also a significant problem , as experience is often specific to the robot body it was gathered on , and ANNs trained to control one body are not easily adapted a different one . In addition , tabula rasa architectures like MLPs are in general difficult to interpret , as they have internal dynamics that are distributed across units and non-trivial to relate to agent behavior ( Merel et al. , 2019a ) . In nature , animals are born with highly structured connectivity in their brains and nervous systems that has been shaped over millennia by evolution ( Zador , 2019 ) . In some cases , this innate circuitry confers abilities with little or no learning ; in others , it guides the learning process by providing strong inductive biases ( Lake et al. , 2017 ) . These innate and learned mechanisms act synergistically , enabling most animals to function well soon after birth , while continuing to improve and acquire skills in a data-efficient manner , e.g . a horse learning to walk with only a couple hours of experience . Moreover , despite species-specific variations , there is a significant amount of shared architecture ( e.g . cerebellum , basal ganglia ) and design principles ( e.g . hierarchical modularity , partial autonomy ) even between distantly related species ( Merel et al. , 2019b ) . Biological circuit architecture is often highly Under review as a conference paper at ICLR 2022 structured and sparse ( Luo , 2021 ) , in sharp contrast to the all-to-all connectivity of MLPs . Moreover , evolution has progressively built more advanced abilities on lower-level circuits , leveraging modular structure to transfer existing working designs to new animal bodies ( Cisek , 2019 ; Merel et al. , 2019b ) . Taken together , these findings from neuroscience suggest that structured neural circuits in animals instantiate efficient , transferable , and modular solutions for high-dimensional embodied control . Capturing some of this structure in model architecture may enable ANNs to narrow the gap between artificial and natural systems . For example , convolutional networks inspired by visual circuitry have successfully encoded the inductive biases of spatial locality and weight sharing to improve performance , data efficiency , and parameter efficiency for vision tasks ( Lindsay , 2021 ) . Since their neuroscience-inspired origins , convolutional networks have benefited from novel layer types , activation functions , and modules ( Gu et al. , 2017 ) , as well as novel architectures , e.g . LeNet ( LeCun et al. , 1989 ) , AlexNet ( Krizhevsky et al. , 2012 ) , VGG ( Simonyan & Zisserman , 2015 ) , ResNet ( He et al. , 2015 ) . The advantages of ( non-biologically inspired ) architectural priors has also been established across other machine learning domains . In natural language processing , architectural priors have specialized to handle sequences with designs including recurrent networks , e.g . LSTM ( Hochreiter & Schmidhuber , 1997 ) , and attention networks , e.g . Transformer ( Vaswani et al. , 2017 ) . However , it is unknown the extent to which ANN architectures inspired by neural circuitry can yield useful inductive biases for other domains . In this work , we ask what advantages biologically inspired network architecture can provide in the context of motor control . Specifically , we translate C. elegans circuits for locomotion into an ANN model applied to a simulated Swimmer agent . Our architecture is an instance of what we call a “ Neural Circuit Architectural Prior ” ( NCAP ) , to denote an ANN architecture that encodes prior structure / inductive biases inspired by biological neural circuits . In contrast to previous work on neuromechanical models of movement and central pattern generators ( Sarma et al. , 2018 ; Izquierdo & Beer , 2015 ; Jiao et al. , 2021 ) , our model is designed within the discrete-time ANN framework that is standard in machine learning and is fully differentiable , enabling us to train parameters with reinforcement learning ( RL ) and evolution strategies ( ES ) , and directly investigate the role of network architecture by comparing to MLPs . Further , our model controls an agent body from a standard benchmark ( i.e . not custom-designed to work with our architecture ) ; this body is significantly different from C. elegans in terms of mechanics , degrees of freedom , and actuators . On a locomotion task , our architecture achieves good initial performance and asymptotic performance comparable with MLPs , while dramatically improving data efficiency and requiring orders of magnitude fewer parameters . Our architecture is more interpretable and transferable to new body designs . An ablation analysis shows that principled excitation/inhibition significantly contributes to learning . Our work demonstrates several advantages of ANN architectures inspired by systems neuroscience and suggests a path towards modeling more complex animals . The primary contributions of this work are : 1 . A network architecture inspired by C. elegans circuits for locomotion , which combines the discrete-time ANN framework that is standard in machine learning with features from computational neuroscience like constraints on synapse sign ( i.e . excitation vs. inhibition ) and special cell types ( i.e . intrinsic oscillators ) ; 2 . An evaluation of our model ’ s initial performance , asymptotic performance , data efficiency , parameter efficiency , interpretability , and transfer compared to standard MLP architectures ; and 3 . An ablation analysis of the effects of sign constraints , weight initialization , weight sharing , and sparse connectivity on performance and learning . Code and videos are available in the Supplementary Materials . 2 RELATED WORK . A robotic controller ultimately outputs generalized torques τ to apply at each actuator . A neural network controller can directly output torques ( Levine et al. , 2016 ) , or it can output task-space positions x or accelerations ẍ that are converted into torques through an analytic controller like operational space control ( Khatib , 1987 ) . However , some further level of abstraction and prior knowledge is often used to improve performance and simplify learning ( Kroemer et al. , 2020 ) . Under review as a conference paper at ICLR 2022 Trajectory Priors Desired movement is encoded through equations of motion . Dynamic Movement Primitives ( DMP ) ( Schaal , 2006 ; Pastor et al. , 2009 ) use a set of differential equations to implement a stable nonlinear attractor system capable of generating rhythmic and discrete trajectories , which are controlled via low-dimensional parameters specifying the motion ’ s shape and goal . Policies Modulating Trajectory Generators ( PMTG ) ( Iscen et al. , 2019 ) learn a policy to control a predefined trajectory generator via low-dimensional parameters and also generate a residual term to be added to the trajectory generator ’ s output . For example , to produce legged locomotion , PMTG uses equations of motion composed from a combination of sinusoidal functions and hand-engineered gait patterns , which are parameterized by stride length , walking height , and frequency . Generally , trajectorycentric methods can work well when the equations of motion capture good solutions for the desired movement ; however , such equations can be difficult to design and make robust . Behavioral Imitation Priors Desired movement is encoded through learnable functions , e.g . ANNs , trained to imitate reference motions , e.g . from motion capture or manual keyframing . Neural Probabilistic Motor Primitives ( NPMP ) ( Merel et al. , 2019c ) train expert policies to imitate human motion capture trajectories , then compress these experts into a single generalist policy featuring a latent-variable bottleneck , thereby creating a common embedding space that a higher-level controller can use to interpolate and combine various motor primitives . Bin Peng et al . ( 2020 ) train expert policies to imitate animal motion capture trajectories using reinforcement learning and further deploy the learned policies on a physical legged robot . Generally , imitation methods bypass the time and manual effort involved in designing equations of motion and have achieved diverse behaviors . This comes at the cost of compiling of reference motions from humans and animals . Further , learning individual expert policies does not make use of the shared structure between movements to learn more efficiently . In animal legged locomotion , for instance , neural circuit studies have suggested that different gait patterns ( e.g . walk , trot , bound , gallop ) could actually be different emergent dynamic modes of the same pattern generator network ( Grillner & El Manira , 2020 ) . In this case , a single policy with the right structure should be able to capture these diverse reference motions . Architectural Priors Desired movement is encoded indirectly through the structure of ANNs , establishing inductive biases to guide learning . Heess et al . ( 2016 ) decompose a policy into a low-level “ spinal ” network with access to only proprioceptive signals ( e.g . joint positions ) and a high-level “ cortical ” network with access to exteroceptive signals ( e.g . distance to target , vision ) ; this hierarchical architecture was loosely inspired by animal nervous systems . After pre-training the spinal network with a shaped reward , the cortical network was able to learn complex locomotion tasks from sparse rewards by controlling the spinal network , while flat baseline architectures failed . This architecture was later shown to generalize to locomotion with challenging terrains and obstacles , providing greatly improved learning efficiency ( Heess et al. , 2017 ) . Generally , architectural priors can provide improved efficiency and abstraction while maintaining flexibility . However , the level of flexibility needs to be chosen appropriately . Too much flexibility can lead to learned behaviors that are not naturalistic and infeasible/dangerous to deploy in the real world without regularization ( Bin Peng et al. , 2020 ) , while too much constraint can impede the learning of diverse movements . In this work , we design an architectural prior inspired by neural circuitry . In principle , this kind of prior should scale to diverse movements , since the architecture models the exact mechanisms that animals use to achieve robust , flexible behavior . Moreover , since working “ blueprints ” already exist in nature , translating architectures from biology could prove more desirable and more scalable than hand-designing trajectory priors from scratch . Further , while we use reward-based algorithms to learn parameters ( i.e . RL and ES ) , our architecture is largely agnostic to the learning algorithm , so it is possible that our architecture could also be trained within a behavioral imitation setting . 3 MODEL . We translate nematode ( C. elegans ) circuits for locomotion into an ANN model applied to a simulated Swimmer agent . In Section 3.1 , we provide an overview of the nematode body structure and the modular neural circuits underlying locomotion . In Section 3.2 , we describe our abstract integrator and oscillator units that serve as building blocks for our architecture . In Section 3.3 , we formalize the observation and action space of the Swimmer agent , and we propose our network architecture modeled on nematode circuits . Under review as a conference paper at ICLR 2022 A head tailwave propagation dorsal ventral nematode body B muscle muscle muscle muscle … … … SMB AVB SMB DB DD VB VD DB DD VB VD nematode neural circuits A B head tail +1 –1 B neuron muscle … … … swimmer body r q2 q3 q4 q5 swimmer network architecture l 1− s od ov q1 d q1 v ! ! q2 d q2 d ! ! q2 v q2 v q1 b1 d oscillator b1 v m1 d m1 v B neuron muscle b2 d b2 v m2 d m2 v ! ! q1 d ! ! q1 v A Bintegrator unit oscillator unit square wave excitatory synapse inhibitory synapse gap junction proprioception dorsal ventral excitatory weight inhibitory weight activation function w+ = c+ ( w ) = max ( w , 0 ) w− = c− ( w ) = min ( w , 0 ) f ( z ) = max ( z , 0 ) z = b + c ( wi ) ⋅ xii∑ y = f ( z ) x1 x2 w1 + w2 − y y y = f ( t ) sine wave f ( t ) = f ( t ) = high low amplitude period width midline o s i weights : osc ( o ) , turn ( t ) , speed ( s ) , prop ( p ) , ipsi ( i ) , contra ( c ) p p o t t s s s i ii c c c c Figure 1 : Nematode . ( A ) The nematode C. elegans achieves forward locomotion through alternating dorsal-ventral muscle contraction waves propagating down the body . ( B ) Muscle wave propagation , oscillation , steering , and speed control are coordinated by a highly stereotyped , modular , and repeated microcircuit . B neurons sense bending in the previous module via proprioception and excite ipsilateral muscles , while inhibiting contralateral muscles via D neurons . Intrinsic oscillations in B neurons initiate waves . SMB neurons bias head and neck muscles for steering . AVB attenuates all B neurons via gap junctions for speed control .

Summary: This paper proposes a set of bio-inspired Neural Circuit Architectural Priors (NCAP) as ingredients for building trainable networks for continuous control. The authors gave one example of translating the C. elegans locomotion circuits to a network using NCAP to control a swimmer agent in a simulated environment. The proposed network trained by deep reinforcement learning methods (DRL) and evolution strategies (ES) showed comparable performance and more efficient computation with fewer parameters.