Neural Circuit Architectural Priors for Embodied Control

1 INTRODUCTION . Artificial neural networks for simulated motor control and robotics often adopt generic architectures like fully connected multi-layered perceptrons ( MLPs ) ( Pierson & Gashler , 2017 ; Levine et al. , 2016 ; Bin Peng et al. , 2020 ; Heess et al. , 2016 ) . While general , these tabula rasa architectures rely on large amounts of experience to learn . Data efficiency is especially important in motor control because , unlike computer vision and natural language processing which have greatly benefited from the availability of large datasets ( Bommasani et al. , 2021 ) , gathering large-scale data for robotics is challenging . Experience must be gathered through interaction with the environment , which is difficult for reasons including time , human labor , safety , maintenance , and reproducibility ( Kroemer et al. , 2020 ) . Transfer is also a significant problem , as experience is often specific to the robot body it was gathered on , and ANNs trained to control one body are not easily adapted a different one . In addition , tabula rasa architectures like MLPs are in general difficult to interpret , as they have internal dynamics that are distributed across units and non-trivial to relate to agent behavior ( Merel et al. , 2019a ) . In nature , animals are born with highly structured connectivity in their brains and nervous systems that has been shaped over millennia by evolution ( Zador , 2019 ) . In some cases , this innate circuitry confers abilities with little or no learning ; in others , it guides the learning process by providing strong inductive biases ( Lake et al. , 2017 ) . These innate and learned mechanisms act synergistically , enabling most animals to function well soon after birth , while continuing to improve and acquire skills in a data-efficient manner , e.g . a horse learning to walk with only a couple hours of experience . Moreover , despite species-specific variations , there is a significant amount of shared architecture ( e.g . cerebellum , basal ganglia ) and design principles ( e.g . hierarchical modularity , partial autonomy ) even between distantly related species ( Merel et al. , 2019b ) . Biological circuit architecture is often highly Under review as a conference paper at ICLR 2022 structured and sparse ( Luo , 2021 ) , in sharp contrast to the all-to-all connectivity of MLPs . Moreover , evolution has progressively built more advanced abilities on lower-level circuits , leveraging modular structure to transfer existing working designs to new animal bodies ( Cisek , 2019 ; Merel et al. , 2019b ) . Taken together , these findings from neuroscience suggest that structured neural circuits in animals instantiate efficient , transferable , and modular solutions for high-dimensional embodied control . Capturing some of this structure in model architecture may enable ANNs to narrow the gap between artificial and natural systems . For example , convolutional networks inspired by visual circuitry have successfully encoded the inductive biases of spatial locality and weight sharing to improve performance , data efficiency , and parameter efficiency for vision tasks ( Lindsay , 2021 ) . Since their neuroscience-inspired origins , convolutional networks have benefited from novel layer types , activation functions , and modules ( Gu et al. , 2017 ) , as well as novel architectures , e.g . LeNet ( LeCun et al. , 1989 ) , AlexNet ( Krizhevsky et al. , 2012 ) , VGG ( Simonyan & Zisserman , 2015 ) , ResNet ( He et al. , 2015 ) . The advantages of ( non-biologically inspired ) architectural priors has also been established across other machine learning domains . In natural language processing , architectural priors have specialized to handle sequences with designs including recurrent networks , e.g . LSTM ( Hochreiter & Schmidhuber , 1997 ) , and attention networks , e.g . Transformer ( Vaswani et al. , 2017 ) . However , it is unknown the extent to which ANN architectures inspired by neural circuitry can yield useful inductive biases for other domains . In this work , we ask what advantages biologically inspired network architecture can provide in the context of motor control . Specifically , we translate C. elegans circuits for locomotion into an ANN model applied to a simulated Swimmer agent . Our architecture is an instance of what we call a “ Neural Circuit Architectural Prior ” ( NCAP ) , to denote an ANN architecture that encodes prior structure / inductive biases inspired by biological neural circuits . In contrast to previous work on neuromechanical models of movement and central pattern generators ( Sarma et al. , 2018 ; Izquierdo & Beer , 2015 ; Jiao et al. , 2021 ) , our model is designed within the discrete-time ANN framework that is standard in machine learning and is fully differentiable , enabling us to train parameters with reinforcement learning ( RL ) and evolution strategies ( ES ) , and directly investigate the role of network architecture by comparing to MLPs . Further , our model controls an agent body from a standard benchmark ( i.e . not custom-designed to work with our architecture ) ; this body is significantly different from C. elegans in terms of mechanics , degrees of freedom , and actuators . On a locomotion task , our architecture achieves good initial performance and asymptotic performance comparable with MLPs , while dramatically improving data efficiency and requiring orders of magnitude fewer parameters . Our architecture is more interpretable and transferable to new body designs . An ablation analysis shows that principled excitation/inhibition significantly contributes to learning . Our work demonstrates several advantages of ANN architectures inspired by systems neuroscience and suggests a path towards modeling more complex animals . The primary contributions of this work are : 1 . A network architecture inspired by C. elegans circuits for locomotion , which combines the discrete-time ANN framework that is standard in machine learning with features from computational neuroscience like constraints on synapse sign ( i.e . excitation vs. inhibition ) and special cell types ( i.e . intrinsic oscillators ) ; 2 . An evaluation of our model ’ s initial performance , asymptotic performance , data efficiency , parameter efficiency , interpretability , and transfer compared to standard MLP architectures ; and 3 . An ablation analysis of the effects of sign constraints , weight initialization , weight sharing , and sparse connectivity on performance and learning . Code and videos are available in the Supplementary Materials . 2 RELATED WORK . A robotic controller ultimately outputs generalized torques τ to apply at each actuator . A neural network controller can directly output torques ( Levine et al. , 2016 ) , or it can output task-space positions x or accelerations ẍ that are converted into torques through an analytic controller like operational space control ( Khatib , 1987 ) . However , some further level of abstraction and prior knowledge is often used to improve performance and simplify learning ( Kroemer et al. , 2020 ) . Under review as a conference paper at ICLR 2022 Trajectory Priors Desired movement is encoded through equations of motion . Dynamic Movement Primitives ( DMP ) ( Schaal , 2006 ; Pastor et al. , 2009 ) use a set of differential equations to implement a stable nonlinear attractor system capable of generating rhythmic and discrete trajectories , which are controlled via low-dimensional parameters specifying the motion ’ s shape and goal . Policies Modulating Trajectory Generators ( PMTG ) ( Iscen et al. , 2019 ) learn a policy to control a predefined trajectory generator via low-dimensional parameters and also generate a residual term to be added to the trajectory generator ’ s output . For example , to produce legged locomotion , PMTG uses equations of motion composed from a combination of sinusoidal functions and hand-engineered gait patterns , which are parameterized by stride length , walking height , and frequency . Generally , trajectorycentric methods can work well when the equations of motion capture good solutions for the desired movement ; however , such equations can be difficult to design and make robust . Behavioral Imitation Priors Desired movement is encoded through learnable functions , e.g . ANNs , trained to imitate reference motions , e.g . from motion capture or manual keyframing . Neural Probabilistic Motor Primitives ( NPMP ) ( Merel et al. , 2019c ) train expert policies to imitate human motion capture trajectories , then compress these experts into a single generalist policy featuring a latent-variable bottleneck , thereby creating a common embedding space that a higher-level controller can use to interpolate and combine various motor primitives . Bin Peng et al . ( 2020 ) train expert policies to imitate animal motion capture trajectories using reinforcement learning and further deploy the learned policies on a physical legged robot . Generally , imitation methods bypass the time and manual effort involved in designing equations of motion and have achieved diverse behaviors . This comes at the cost of compiling of reference motions from humans and animals . Further , learning individual expert policies does not make use of the shared structure between movements to learn more efficiently . In animal legged locomotion , for instance , neural circuit studies have suggested that different gait patterns ( e.g . walk , trot , bound , gallop ) could actually be different emergent dynamic modes of the same pattern generator network ( Grillner & El Manira , 2020 ) . In this case , a single policy with the right structure should be able to capture these diverse reference motions . Architectural Priors Desired movement is encoded indirectly through the structure of ANNs , establishing inductive biases to guide learning . Heess et al . ( 2016 ) decompose a policy into a low-level “ spinal ” network with access to only proprioceptive signals ( e.g . joint positions ) and a high-level “ cortical ” network with access to exteroceptive signals ( e.g . distance to target , vision ) ; this hierarchical architecture was loosely inspired by animal nervous systems . After pre-training the spinal network with a shaped reward , the cortical network was able to learn complex locomotion tasks from sparse rewards by controlling the spinal network , while flat baseline architectures failed . This architecture was later shown to generalize to locomotion with challenging terrains and obstacles , providing greatly improved learning efficiency ( Heess et al. , 2017 ) . Generally , architectural priors can provide improved efficiency and abstraction while maintaining flexibility . However , the level of flexibility needs to be chosen appropriately . Too much flexibility can lead to learned behaviors that are not naturalistic and infeasible/dangerous to deploy in the real world without regularization ( Bin Peng et al. , 2020 ) , while too much constraint can impede the learning of diverse movements . In this work , we design an architectural prior inspired by neural circuitry . In principle , this kind of prior should scale to diverse movements , since the architecture models the exact mechanisms that animals use to achieve robust , flexible behavior . Moreover , since working “ blueprints ” already exist in nature , translating architectures from biology could prove more desirable and more scalable than hand-designing trajectory priors from scratch . Further , while we use reward-based algorithms to learn parameters ( i.e . RL and ES ) , our architecture is largely agnostic to the learning algorithm , so it is possible that our architecture could also be trained within a behavioral imitation setting . 3 MODEL . We translate nematode ( C. elegans ) circuits for locomotion into an ANN model applied to a simulated Swimmer agent . In Section 3.1 , we provide an overview of the nematode body structure and the modular neural circuits underlying locomotion . In Section 3.2 , we describe our abstract integrator and oscillator units that serve as building blocks for our architecture . In Section 3.3 , we formalize the observation and action space of the Swimmer agent , and we propose our network architecture modeled on nematode circuits . Under review as a conference paper at ICLR 2022 A head tailwave propagation dorsal ventral nematode body B muscle muscle muscle muscle … … … SMB AVB SMB DB DD VB VD DB DD VB VD nematode neural circuits A B head tail +1 –1 B neuron muscle … … … swimmer body r q2 q3 q4 q5 swimmer network architecture l 1− s od ov q1 d q1 v ! ! q2 d q2 d ! ! q2 v q2 v q1 b1 d oscillator b1 v m1 d m1 v B neuron muscle b2 d b2 v m2 d m2 v ! ! q1 d ! ! q1 v A Bintegrator unit oscillator unit square wave excitatory synapse inhibitory synapse gap junction proprioception dorsal ventral excitatory weight inhibitory weight activation function w+ = c+ ( w ) = max ( w , 0 ) w− = c− ( w ) = min ( w , 0 ) f ( z ) = max ( z , 0 ) z = b + c ( wi ) ⋅ xii∑ y = f ( z ) x1 x2 w1 + w2 − y y y = f ( t ) sine wave f ( t ) = f ( t ) = high low amplitude period width midline o s i weights : osc ( o ) , turn ( t ) , speed ( s ) , prop ( p ) , ipsi ( i ) , contra ( c ) p p o t t s s s i ii c c c c Figure 1 : Nematode . ( A ) The nematode C. elegans achieves forward locomotion through alternating dorsal-ventral muscle contraction waves propagating down the body . ( B ) Muscle wave propagation , oscillation , steering , and speed control are coordinated by a highly stereotyped , modular , and repeated microcircuit . B neurons sense bending in the previous module via proprioception and excite ipsilateral muscles , while inhibiting contralateral muscles via D neurons . Intrinsic oscillations in B neurons initiate waves . SMB neurons bias head and neck muscles for steering . AVB attenuates all B neurons via gap junctions for speed control .

Summary: This paper introduces a set of architectural components and design principles for constructing neural networks for embodied control. The proposed method is biologically inspired, and can be used to build neural networks specifically designed for the considered agent, e.g., the swimmer agent studied in the paper. The paper provides a literature review, describes the method, and evaluates it in an ablation study using the DDPG algorithm.