Deep neuroethology of a virtual rodent
Authors: Josh Merel
Diego Aldarondo
Jesse Marshall
Yuval Tassa
Greg Wayne
Bence Olveczky
Venue:  (2020)
Keywords: computational neuroscience
motor control
deep RL

Abstract: Parallel developments in neuroscience and deep learning have led to mutually productive exchanges, pushing our understanding of real and artificial neural networks in sensory and cognitive systems. However, this interaction between fields is less developed in the study of motor control. In this work, we develop a virtual rodent as a platform for the grounded study of motor activity in artificial models of embodied control. We then use this platform to study motor activity across contexts by training a model to solve four complex tasks. Using methods familiar to neuroscientists, we describe the behavioral representations and algorithms employed by different layers of the network using a neuroethological approach to characterize motor activity relative to the rodent's behavior and goals. We find that the model uses two classes of representations which respectively encode the task-specific behavioral strategies and task-invariant behavioral kinematics. These representations are reflected in the sequential activity and population dynamics of neural subpopulations. Overall, the virtual rodent facilitates grounded collaborations between deep reinforcement learning and motor neuroscience.

Published as a conference paper at ICLR 2020
DEEP NEUROETHOLOGY OF A VIRTUAL RODENT
Josh Merel" Diego Aldarondo" 22.3, .Jsse Marshall*3 Yuval Tassa', Greg Wayne', Bence Ölveczky 'DeepMind, London, UK.
Program in Neuroscience, 'Center for Brain Science, 4 'Department of Organismic and Evolutionary Biology, Harvard University, Cambridge, MA 02138, USA. jsmerel égoog] e. com, diegoa da rondod@ har rva wwwardd... edu, gesse_d.marsher lffass............. harvard. edu
ABSTRACT
Parallel developments in neuroscience and deep learning have led to mutually productive exchanges, pushing our understanding of real and artificial neural net- works in sensory and cognitive systems. However, this interaction between fields is less developed in the study of motor control. In this work, we develop a virtual rodent platform for the grounded study of motor activity in artificial models of embodied control. We then use this platform study motor activity across contexts by training model to solve four complex tasks. Using methods famil- iar to neuroscientists, we describe the behavioral representations and algorithms employed by different layers of the network using neuroethological approach characterize motor activity relative to the rodent's 's behavior and goals. We find that the model uses two classes of representations which respectively encode the task-specific behavioral strategies and fask-invariant behavioral kinematics. These representations reflected in the sequential activity and population dynamics of neural subpopulations. Overall, the virtual rodent facilitates grounded collabora- tions between deep reinforcement learning and motor neuroscience.
INTRODUCTION
Animals have nervous systems that allow them to coordinate their movement and perform diverse set of complex behaviors. Mammals, in particular, are generalists in that they use the same general neural network to solve wide variety of tasks. This flexibility in adapting behaviors towards many different goals far surpasses that of robots or artificial motor control systems. Hence, studies of the neural underpinnings of flexible behavior in mammals could yield important insights into the classes of algorithms capable of complex control across contexts and inspire algorithms for flexible control in artificial systems (Merel et al., 2019b).
Recent efforts at the interface of neuroscience and machine learning have sparked renewed interest in constructive approaches in which artificial models that solve tasks similar to those solved by animals serve as normative models of biological intelligence. Researchers have attempted to leverage these models to gain insights into the functional transformations implemented by neurobiological circuits, prominently in vision (Khaligh-Razav & Kriegeskorte] 2014, Yamins et al., 2014 Kar et al.) 2019).
but also increasingly in other areas, including audition (Kell et al. 2018 8) and navigation (Banino ct al. 20188] Cueva & Weil 2018). Efforts to construct models of biological locomotion systems have informed our understanding of the mechanisms and evolutionary history of bodies and behavior
(Grilnner et al. 2007]] [jspeert et al. 2007 Ramdya al. 2017 Nyakatura et al. 2019). Neural control approaches have also been applied to the study of reaching movements, though often in constrained behavioral paradigms (Lillicrap & Scott 2013). where supervised training is possible (Sussillo et al. 2015 Michaels et al. 2019).
While these approaches model parts of the interactions between animals and their environments (Chiel & Beer 1997). none attempt to capture the full complexity of embodied control, involving how an animal uses its senses, body and behaviors solve challenges in a physical environment.
* Equul contribution.

---

Published as a conference paper at ICLR 2020
The development of models of embodied control is valuable the field of motor neuroscience, which typically focuses on restricted behaviors in controlled experimental settings. is also valuable for Al research, where flexible models of embodied control could be applicable to robotics.
Here, we introduce a virtual model rodent to facilitate grounded investigation embodied motor systems. The virtual rodent affords control to biological data from real-world rodents, which are more experimentally accessible than humans. We draw inspiration from emerging deep reinforcement learning algorithms which now allow artificial agents to perform complex and adaptive movement in physical environments with sensory information that increasingly similar to that available to animals (Peng et al. 2016 2017 Heess et al. 2017 Merel et al. 9019a|e) Similarly, our virtual rodent exists in physical world, equipped with a set of actuators that must be coordinated for to behave effectively. also possesses sensory system that allows to use visual input from egocentric camera located on its head and proprioceptive input to sense the configuration of its body in space.
opportunity directly compare principles of artificial
There are several questions one could answer using the virtual rodent platform. Here we focus on the problem of embodied control across multiple tasks. While some efforts have been made to analyze neural activity in reduced systems trained to solve multiple tasks (Song et al. [017]] Yang iet al. 2019). those studies lacked the important element of motor control in physical environment. Our rodent platform presents the opportunity to study how representations of movements well as sequences of movements change as function of goals and task contexts.
To address these questions, trained virtual rodent solve four complex tasks within physi- cal environment, all requiring the coordinated control of its body. We then ask "Can a neuroscientist understand a virtual rodent?" more grounded take the originally satirical "Can biologist fix radio?" [Lazebnik] 2002) or the more recent "Could neuroscientist understand a microprocessor?" (Jonas & Kording 2017). We take a more sanguine view of the tremendous advances that have been made in computational neuroscience in the past decade, and posit that the supposed 'failure' of these approaches in synthetic systems partly misdirection. Analysis approaches neuroscience were developed with the explicit purpose of understanding sensation and action in real brains, and often implicitly rooted in the types of architectures and processing that are thought relevant in biological control systems. With this philosophy, we use analysis approaches common in neuroscience to plore the types of representations and dynamics that the virtual rodent's neural network employs to coordinate multiple complex movements in the service of solving motor and cognitive tasks.
APPROACH
2.1
VIRTUAL RODENT BODY
A
D
Figure 1: (A) Anatomical skeleton of a rodent (as reference; not part of physical simulation). (B) body designed around the skeleton match the anatomy and model collisions with the environment. (C) Purely cosmetic skin to cover the body. (D) Semi-transparent visualization of (A)-(C) overlain.
We implemented virtual rodent body (Figure 1 in MuJoCo (Todorov et al. 2012), based mea- surements of laboratory rats (see Appendix I..II I.. The rodent body has 38 controllable degrees of freedom. The tail, spine, and neck consist of multiple segments with joints, but controlled by tendons that co-activate multiple joints (spatial tendons MuJoCo). The rodent will be released as part of wwnt www.coontoo/locomttttooo
The virtual rodent has access to proprioceptive information as well as "raw" egocentric RGB-camerr 664 pixels) input from head-mounted camera. The proprioceptive inputs include internal
2

---

Published as a conference paper at ICLR 2020
joint angles and angular velocities, the positions and velocities of the tendons that provide actua- tion, egocentric vectors from the root (pelvis) of the body to the positions of the head and paws, a vestibular- like upright orientation vector, touch contact sensors in the paws, as well as egocentric acceleration, velocity, and 3D angular velocity of the root.
2.2
VIRTUAL RODENT TASKS
A
D
Figure 2: Visualizations four tasks the virtual rodent was trained to solve: (A) jumping over gaps ("gaps run"), (B) foraging in maze ("maze forage"). (C) escaping from hilly region ("bowl cape"), and (D) touching a ball twice with a forepaw with a precise timing interval between touches ("two-tap").
We implemented four tasks adapted from previous work in deep reinforcement learning and motor neuroscience (Merel et al. 2019a Tassa et al. 2018 Kawai et al. 2015) to encourage diverse motor behaviors in the rodent. The tasks are as follows: (I) Run along corridor, over "gaps", with reward for traveling along the corridor at a target velocity (Figure A)). (2) Collect all the blue orbs maze, with sparse reward for each orb collected (Figure 3). (3) Escape bowl-shaped region by traversing hilly terrain, with reward proportional to distance from the center of the bowl (Figure 2C). (4) Approach orbs in an open field, activate them by touching them with a forepaw, and touch them second time after precise interval of 800ms with tolerance of = 100ms; there is a time-out period the touch is not within the tolerated window and rewards are provided sparsely on the first context indicating its task. Rather, the agent had to infer the task from the visual input and behave appropriately.
and second touch (Figure Zp). We did not provide the agent with
2.3 TRAINING A MULTI-TASS POLICY
vision
core
I
(T,)
module
V(hg)
encoder
S
proprio
(S,)
policy module
encoder
-T(((hf)) ( at
The virtual rodent agent architecture. Egocentric visual image inputs encoded into features via a small residual network (He et al.) 2016) and proprioceptive state observations are encoded via small multi-layer perceptron. The features passed into recurrent LSTT module (Hochreiter & Schmidhuber 1997). The core module is trained by backpropogation during training of the value function. The outputs the core are also passed as features the policy module (with the dashed arrow indicating no backpropogation along this path during training) along with shortcut paths from the proprioceptive observations well as encoded features. The policy module consists of one or more stacked LSTMs (with or without skip connections) which then produce the actions a stochastic policy.
Figure 3:
Emboldened by recent results which end-to-end RL produces a single terrain-adaptive policy |Png et al.) 2016 2017 Heess al. 2017). trained single architecture on the multiple motor- control-relian tasks (see Figure 3). To train a single policy to perform all four tasks, we used IMPALA M-PAle-style setup for actor-critic DeepRL. (Espeholt al. 2018); parallel workers collected
3

---

Published as a conference paper at ICLR 2020
rollouts, logged them to replay, from which central learner sampled data to perform updates. The value-function critic was trained using off policy correction via V-trace. To update the actor, we used a variant of MPO (Abdolmaleki et al. 2018)) where the E-step is performed using advantages determined from the empirical returns and the value-function, instead of the Q-function (Song et al., 2019). Empirically, we found that the "escape" task was challenging learn during interleaved training relative to the other tasks. Consequently, we present results arising from training a single- task expert on the escape task and training the multi-task policies using kickstarting for that task (Schmitt et al. (018)), with a weak coefficient (.001 or .055). Kickstarting on this task made the seeds more reliably solve all four tasks, facilitating comparison of the multi-task policies with different architectures (i.e. the policy having 2, or layers, with or without skip connections across those layers). The procedure yields a single neural network that uses visual inputs to determine how to behave and coordinates its body to move in ways required to solve the tasks. See video examples of a single policy solving episodes cach task:
gaps forage, escape. and two-tap
3 ANALYSIS
A
Joint speed
Actuator forces
Skull
Skull
Fast kinematics
Slow kinematics
Elbow
Elbow
angw
Knee
Knee Angle
0.5
Leap
Run
Climb
Spin
Rear
Bowl escape
Maze forage
Jump
Stumble
Righting
Figure 4: Ethology of the virtual rodent. (A) Example jumping sequence in gaps run task with rep- resentative subset of recorded behavioral features. Dashed lines denote the time the corresponding frames (top). (B) tSNE embedding of 60 behavioral features describing the pose and kinematics of the virtual rodent allows identification of rodent behaviors. Points are colored by hand-labeling of behavioral clusters identified by watershed clustering. (C) The first two principal components of different behavioral features reveals that behaviors are more shared across tasks at short, 5-25 Hz timescales (fast kinematics), but no longer 0.3-5 Hz timescales (slow kinematics).
We analyzed the virtual rodent's neural network activity in conjunction with its behavior char- acterize how it solves multiple tasks (Figure [FA). We used analyses and perturbation techniques adapted from neuroscience, where a range of techniques have been developed highlight the prop- erties of real neural networks. Biological neural networks have been hypothesized to control, select, and modulate movement through a variety of debated mechanisms, ranging from explicit neural representations of muscle forces and behavioral primitives, to more abstract production of neural dynamics that could underly movement (Graziano) 2006) Kalaska 2009 Churchland et al. 2012). A challenge with nearly all of these models however is that they have largely been inspired by find- ings from individual behavioral tasks, making unclear how to generalize them to a broader range of naturalistic behaviors. To provide insight into mechanisms underlying movement in the virtual rodent, and to potentially give insight by proxy into the mechanisms underlying behavior in real rats, thus systematically tested how the different network layers encoded and generated different aspects of movement.
For all analyses we logged the virtual rodent's kinematics, joint angles, computed forces, sensory inputs, and the cell unit activity of the LSTMs in core and policy layers during 25 trials per task from each network architecture.
4

---

Published as a conference paper at ICLR 2020
3.1 VIRTUAL RODENTS EXHIBIT BEHAVIORAL FLEXIBILITY.
We began our analysis by quantitatively describing the behavioral repertoire of the virtual rodent. A challenge in understanding the neural mechanisms underlying behavior is that it can be described many timescales. On short timescales, one could describe rodent locomotion using a set of actuators that produce joint-specific patterns of forces and kinematies. However on longer timescales, these force patterns are organized into coordinated, re-used movements, such as running, jumping, and turning. These movements can further combined to form behavioral strategies or goal-directed behaviors. Relating neural representations to motor behaviors therefore requires analysis methods that span multiple timescales of behavioral description. behaviors these networks learn to generate and how they are differentially deployed across tasks, we developed sets of behavioral features that describe the kinematics of the animal on fast (5-25 Hz), intermediate (1-25 Hz) slow (0.3-5 Hz) timescales (Appendix (A.2 A.3 As validation that these features reflected meaningful differences across behaviors, embedding these features using ISNE (Maaten & Hinton, 2008) produced behavioral map in which virtual rodent behaviors, segregated to different regions of the map (Figure (see video). This behavioral repertoire of the virtual rodent consisted of many behaviors observed rodents, such as rearing, jumping, running, climbing and spinning. While the exact kinematics of the virtual rodent's behaviors did not exactly match those observed in real rats, they did reproduce unexpected features. For instance the stride frequency of the virtual rodent during galloping matches that observed in rats (Appendix A.3)
systematically examine the classes of
We next investigated how these behaviors were used by the virtual rodent across tasks. On short timescales, low-level motor features like joint speed and actuator forces occupied similar regions principal component space (Figure 4C). contrast, behavioral kinematics, especially on long, 0.3-5 Hz timescales, were more differentiated across tasks. Similar results held when examining overlap in other dimensions using multidimensional scaling. Overall this suggests that the network learned to adapt similar movements in selective manner for different tasks, suggesting that the agent exhibited a form of behavioral flexibility.
3.2 NETWORKS PRIMARILY REFLECT BEHAVIORS, NOT FORCES
Polcy
Sehavioral features
Dohanoor ser
Joint volcotty
rollyy
Achustor farois
Pelley
0.6
schovlor clustr
Posture
Dow
hilo milr
Actuator
iiterratics
Rebesfora frsture sroodig (CEA)
Skip
No skip
Sine linematics
ereodng
Behavional
Toloy
Pokay'
0.3
05
0.4
product (peraanthel
endurneenooding (OXA)
1ape
Hapee
Layer
aayer
Figure 5: Representational structure of the rodent's neural network. (A) Example similarity matrices of neural networks and behavioral descriptors. We grouped behavioral descriptors into 50 clusters that and we computed the average neural population vector during each cluster (AppendixA..) Similarity was assessed by computing the dot product of either the neural population vector or the behavioral feature vector within each cluster. (B) Centered Kernel Alignment (CKA) index of neural and behavioral feature similarity matrices for and policy layer architectures. (C) CKA index of feature similarity matrices across all pairs of network layers. (D) Average CKA index between core and policy layers and behavioral features, compared across architectures. Points show values from individual network seeds. Policy values averaged across layers.
We next examined the neural activity patterns underlying the virtual rodent's behavior to test net- works produced behaviors through explicit representations of forces, kinematics behaviors. As
5

---

Published as a conference paper at ICLR 2020
expected, core and policy units operate distinct timescales (See Appendix A.3 Figure 9). Units
in the core typically fluctuated over timescales of 1-10 seconds, likely representing variables asso- ciated with context and reward. In contrast, umits in policy layers were more active over subsecond timescales, potentially encoding motor and behavioral features.
To quantify which aspects of behavior were encoded in the core and policy layers. and how these pat- terns varied across layers, we used representational similarity analysis (RSA) (Kriegeskorte et al, 2008 Kriegeskorte & Diedrichsen 2019). RSA provides a global measure of how well different features are encoded in layers of a neural network by analyzing the geometries of network activity upon exposure several stimuli, such as objects. To apply RSA, first a representational similarity (or equivalently, dissimilarity) matrix is computed that quantifies the similarity of neural population responses to a set of stimuli. To test if different neural populations show similar stimulus encodings, these similarity matricies can then be directly compared across different network layers. Multiple metries, such as the matrix correlation or dot product can be used to compare these neural represen- tational similarity matricies. Here we used the linear centered kernel alignment (CKA) index, which shows invariance to orthonormal rotations of population activity (Korbblth et al. 2019h.
RSA can also be used to directly test how well a particular stimulus feature is encoded in a popula- tion. If each stimuli can be quantitively described by one or more feature vectors, a similarity matrix can also be computed across the set of stimuli themselves. The strength of encoding of particular set of features can by measured by comparing the correlation of the stimulus feature similarity ma- trix and the neuronal similarity matrix. The correlation strength directly reflects the ability of linear decoder trained on the neuronal population vector to distinguish different stimuli (Kriegeskorte & Diedrichsen| 2019). Unlike previous applications of RSA in the analysis of discrete stimuli such as objects, (Khaligh-Razavi & Kriegeskorte| 2014]] Yamins et al. 2014)) behavior evolves continuously. To adapt RSA behavioral analysis, we partitioned time by discretizing each behavioral feature into 50 clusters (Appendix (A.4).
As expected, RSA revealed that core and policy layers encoded somewhat distinct behavioral fea- tures. Policy layers contained greater information about fast timescale kinematics in manner that was largely conserved across layers, while core layers showed more moderate encoding of kinemat- ics that was stronger for slow behavioral features (Figure BC.. This difference in encoding was largely consistent across all architectures tested (Figure 5D).
The feature encoding of policy networks was somewhat consistent with the emergence of hier- archy of behavioral abstraction. In networks trained with three policy layers, representations were distributed in timescales across layers, with the last layer (policy 2) showing stronger encoding of fast behavioral features, and the first layer (policy 0) instead showing stronger encoding of slow behavioral features. However, policy layer activity, even close the motor periphery, did not show strong explicit encoding of behavioral kinematics or forces.
3.3 BEHAVIORAL REPRESENTATIONS ARE SHARED ACROSS TASKS
We then investigated the degree to which the rodent's 's neural networks used the same neural represen- tations to produce behaviors, such as running or spinning, that were shared across tasks, Embedding population activity into two-dimensions using multidimensional scaling revealed that core neuron representations were highly distinct across all tasks, while policy layers contained more overlap (Figure 6A) suggesting that some behavioral representations were re-used. Comparison of rep- resentational similarity matricies for behaviors that were shared across tasks revealed that policy layers tended to possess relatively similar encoding of behavioral features, especially fast behav- ioral features, over tasks (Figure 6C; Appendix AA.4b. This was validated by inspection of neural activity during individual behaviors shared across tasks (Appendix A.5) Figure I0). Core layer rep- resentations across almost all behavioral categories were more variable across tasks, consistent with encoding behavioral sequences task variables.
Interestingly, when comparing this cross-task encoding similarity across architectures, we found that one layer networks showed marked increase in the similarity of behavioral encoding across tasks (Figure (D). This suggests that in networks with lower computational capacity, animals must rely on smaller, shared behavioral representation across tasks.

---

Published as a conference paper at ICLR 2020
A
B
C
Policy
Policy 2
Bowl escape
Simiarity byehneitores encoding tasls (CKA)
Skip
No skip
Gaps run
Representation similnity (CKA)
Policy
forage
Twt-lap
0.7
Core
Core
0.5
0.5
0.5
Bowl escape
Gaps run
Maze forage
MbsS
Twe-tap
MDS
Mate forage
Two top
Layer
Layer
Layar
3ayer
Layer
1 Layer
Mare forage Bowi escape
tap
Geps run
Figure 6: Policy representations are shared across tasks. (A) Two-dimensional multidimensional
scaling embeddings of core and policy activity shows that while policy representations overlap across some tasks, core representations are largely distinct. (B) CKA index of the policy and network representations of behavioral features during behaviors shared across different tasks (Appendix (A.4). Policy 2, but not core networks show similar encoding patterns across the across the maze forage and two-tap tasks, well the gaps run and maze forage tasks, consistent with the shared behaviors used across these tasks. (C) The similarity of behavioral feature encoding (CKA index) across different architectures demonstrates that networks with fewer layers show greater sim- ilarity across tasks. Points show values from individual seeds.
3.4 NEURAL POPULATION DYNAMICS ARE SYNCHRONIZED WITH BEHAVIOR
B
Tap
Core
Policy 0
Policy 1
Policy
Sorted average activity
Time
Second
00.5 0.5 1o Time
-.5
1 0.5 0
0.5-1.5
1
0.5 0 0.5-1.5 1
00.5 0 0.5-1.5
Time (s)
Time (s)
Time (s)
Figure 7: Neurons in core and policy networks show sequential activity during stereotyped behavior. (A) Example video stills showing the virtual rodent engaged in the two-tap task (B) Average absolute z-scored activity traces of all 128 neurons each layer during performance the two-tap sequence. Traces sorted by the time of peak average firing rate. Dashed lines indicate the times of first and second taps. Sequential neural activity is present during the two-tap sequence.
While RSA described which behavioral features represented in and policy activity, were also interested in describing how neural activity changes over time to produce different behav- iors. We began by analyzing neural activity during the production of stereotyped behaviors. Activity

---

Published as a conference paper at ICLR 2020
patterns in the two-tap task showed peak activity in core and policy units that was sequentially orga- nized (Figure 7). uniformly tiling time between both taps of the two-tap sequence. This sequential activation observed across tasks and behaviors in the policy network, including during running (vdeo)) where, consistent with policy networks encoding short-timescale kinematic features in a task-invariant manner, neural activity sequences were largely conserved across tasks (See Appendix A.5 Figure(T0) These sequences were reliably repeated across instances of the respective behaviors, and in the case of the two-tap sequence. showed reduced neural variability relative surrounding
timepoints (See Appendix A.66 Figure 11
Core
Policy 2
C
JPC plane
12
bowl escape
gippo rin
3.5
riszcd (arage
Frequency (112)
Q
g
Fast
PC1
PC1 Running
Reward
Speed (a Ms
26
131
Policy Policy Policy
B
D
slow
85
Variance explained
30-
jPC1
jPC1
Task completion
Two Two-tap sequence
15
46
20
800
15
JPC4
Core
Policy O Policy 1 Policy 2
JPC3
JPC3
Figure 8: Latent network dynamics within tasks reflect rodent behavior on different timescales. (A) Vector field representation of the first two principal components of neural activity in the core and final policy layers during the two-tap task. PC spaces show signatures of rotational dynamics. (B) Vector field representation of first two jPC planes for the core and final policy layers during the two- tap task. Apparent rotations within the different planes are associated with behaviors and behavioral features of different timescales. labeled above. Columns denote layer (as in (A)), while rows denote jPC plane. (C) Characteristic frequency of rotations within each jPC plane. Groups of three points respectively indicate the first, second, and third jPC planes for given layer. Rotations in the core are slower than those in the policy. (D) Variance explained by each jPC plane.
The finding of sequential activity hints putative mechanism for the rodent's behavioral pro- duction. We next hoped to systematically quantify the types of sequential and dynamical activity present in core and policy networks without presupposing the behaviors of interest. To describe population dynamics in relation to behavior, we first applied principal components analysis (PCA) the activity during the performance of single tasks, and visualized the gradient of the population vector vector field. Figure 8A shows such vector field representation of the first two principal components of the core and final policy layer during the two-tap task. We generated vector fields by discretizing the PC space into two-dimensional grid and calculating the average neural activity gradient with respect to time for each bin.
The vector fields showed strong signatures of rotational dynamics across all layers, likely signature of previously described sequential activity. To extract rotational patterns, we used jPCA, a dimen- sionality reduction method that extracts latent rotational dynamics in neural activity (Churchland jet al. 2012). The resulting jPCs form an orthonormal basis that spans the same space as the first six traditional PCs, while maximally emphasizing rotational dynamics. Figure 83 shows the vector
8

---

Published as a conference paper at ICLR 2020
fields of the first two jPC planes for the core and final policy layers along with their characteristic frequency. Consistent with our previous findings, jPC planes in the core have lower characteristic frequencies than those in policy layers across tasks (Figure 8C). The jPC planes also individually explained large percentage of total neural variability (Figure (D).
These rotational dynamics in the policy and core jPC planes were respectively associated with the production of behaviors and the reward structure of the task. For example, in the two-tap task, rotations in the fastest jPC plane in the core were concurrent with the approach to reward, while rotations in the second fastest jPC were concurrent with long timescale transitions between running to the orb and performing the two-tap sequence. Similarly, the fastest jPC in policy layers was corre- lated with the phase of running, while the second fastest was correlated with the phase of the two-tap sequence (video). This trend of core and policy neural dynamics respectively reflecting behavioral and task-related features was also present in other tasks. For example, in the maze forage task, the first two jPC planes in the core respectively correlated with reaching the target orb and discovering the location of new orbs, while those in the policy were correlated with low-level locomotor features such as running phase (video). Along with RSA, these findings support a model in which the core layer transforms sensory information into contextual signal in a task-specific manner. This signal then modulates activity in the policy toward different trajectories that generate appropriate behav- iors in a more task-independent fashion. For more complete set of behaviors with neural dynamics visualizations overlaid, see Appendix A.7
3.5 NEURAL PERTURBATIONS CORROBORATE DISTINCT ROLES ACROSS LAYERS
To causally demonstrate the differing roles of core and policy units in respectively encoding task- relevant features and movement, we performed silencing and activation of different neuronal subsets in the two-tap task. We identified two stereotyped behaviors (rears and spinning jumps) that were reliably used in two different seeds of the agent to reach the orb in the task. We ranked neurons according to the degree of modulation of their z-scored activity during the performance of these behaviors. We then inactivated subsets of neurons by clamping activity to the mean values between the first and second taps and observed the effects of inactivation on trial success and behavior.
In both seeds analyzed, inactivation of policy units had stronger effect on motor behavior than the inactivation of core units. For instance, in the two-tap task, ablation of 64 neurons in the final policy layer disrupts the performance of the spinning jump (Appendix A.8 Figure 12 video). In contrast, ablation of Satavior-modulate affected the way in which the behavior directed toward objects the environment. For example, ablation of a subset of core units during the performance of spinning jump had a limited effect, but sometimes resulted in jumps that missed the target orbs (video) See Appendix A.8 Figure T2().
core units did not prevent the production of the behavior, but mildly
We also performed a complementary perturbation aimed to elicit behaviors by overwriting the cell state of neurons in each layer with the average time-varyse varying trajectory of neural activity measured during natural performance of target behavior. The efficacy of stimulation was found to depend on the gross body posture and behavioral state of an animal, but was nevertheless successful in some cases. For example, during the two-tap sequence, we were able to elicit spinning movements common to searching behaviors in the forage task (video) See Appendix [A.8] Figure [2D. E). The efficacy of this activation was more reliable in layers closer the motor output (Figure (2 In fact, activation of core units rarely elicited spins, but rather elicited sporadic dashes reminiscent of the searching strategy of many models during the forage task (video).
4 DISCUSSION
For many computational neuroscientists and artificial intelligence researchers, an aim is to reverse- engineer the nervous system at an appropriate level of abstraction. In the motor system, such an effort requires that we build embodied models of animals equipped with artificial nervous systems capable of controlling their synthetic bodies across range of behavior. Here we introduced virtual rodent capable of performing a variety of complex locomotor behaviors to solve multiple tasks using single policy. We then used this virtual nervous system to study principles of the neural control of movement across contexts and described several commonalities between the neural activity of artificial control and previous descriptions of biological control.

---

Published as a conference paper at ICLR 2020
A key advantage of this approach relative experimental approaches in neuroscience is that can fully observe sensory inputs, neural activity, and behavior, facilitating more comprehensive testing of theories related to how behavior of the connectivity, sources of variance, and training objectives of each component of the model, providing rare ground truth to test the validity of our neural analyses. With these advantages in mind, we evaluated our analyses based on their capacity to both describe the algorithms and representations employed by the virtual rodent and recapitulate the known functional objectives underlying its creation without prior knowledge.
be generated. Furthermore, we have complete knowledge
To this end, our description of core and policy as respectively representing value and motor pro- duction is consistent with the model's s actor-critic training objectives. But beyond validation, our analyses provide several insights into how these objectives are reached. RSA revealed that the cell activity of core and policy layers had greater similarity with behavioral and postural features than with short-timescale actuators. This suggests that the representation of behavior useful in the moment-to-mones production of motor actions in artificial control, model that has been pre- viously proposed in biological action selection and motor control (Mink] 1996; Graziano 2006). These behavioral representations were more consistent across tasks in the policy than in the core, suggesting that task context and value activity in the core engaged task-specific behavioral strategies through the reuse of shared motor activity in the policy.
Our analysis of neural dynamics suggests that reused motor activity patterns are often organized as sequences. Specifically, the activity of policy units uniformly tiles time in the production of several stereotyped behaviors like running, jumping, spinning, and the two-tap sequence. This finding is consistent with reports linking sequential neural activity the production of stereotyped motor and lask-oriented behavior in rodents (Berke et al., 2009 RReda--rozzo & Robbe [0033] Dhawale et al.) 2019). including during task delay periods (Akhlaghpour
al. 2016), as well in singing birds (Albert & Margoliash] 1996] [aahnloser et al.) 2002). Similarly. y. by relating rotational dynamics to the virtual rodent's s behavior, we found that different behaviors were seemingly associated with distinct rotations in neural activity space that evolved at different timescales. These findings are consistent with a hicrarchical control scheme which policy layer dynamics that generate reused behaviors are activated and modulated by sensorimotor signals from the core.
This work represents an early step toward the constructive modeling of embodied control for the purpose of understanding the neural mechanisms behind the generation of behavior. Incrementally and judiciously increasing the realism of the model's embodiment, behavioral repertoire, and neural architecture is natural path for future research. Our virtual rodent possesses far fewer actuators and touch sensors than a real rodent, uses a vastly different sense of vision, and lacks integration with olfactory, auditory, and whisker-based sensation (see /Zhuang al. (017). While the virtual rodent is capable of locomotor behaviors, an increased diversity of tasks involving decision making, memory-based navigation, and working memory could give insight into "cognitive" behaviors of which rodents are capable. Furthermore, chologically-inspree design of neural architectures and training procedures should facilitate comparisons real neural recordings and manipulations. We expect that this comparison will help isolate residual elements of animal behavior generation that are poorly captured by current models of motor control, and encourage the development of artificial neural architectures that can produce increasingly realistic behavior.
AUTHOR CONTRIBUTIONS
Josh and Yuval built the rodent MuloCo model, with measurements collected by Diego and Jesse. Josh trained the virtual rodent model. Jesse performed behavioral and neural representation analyses. Diego performed neural dynamics analyses. Josh, Jesse, and Diego drafted the manuscript. All authors contributed to the conception the project.
ACKNOWLEDGMENTS
The rodent skeleton reference model was purchased from Ieo3Dssdels on TurboSquid. Thanks to Max Cant for the rodent skin, and Marcus Wainwright for the skybox and ground textures. D.A. was supported by NSF GRFP DGE1745303. J.D.M was supported by a fellowship from the Helen Hay Whitney foundation sponsored by Vertex and K99/R00 award from the NINDS.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Abhas Abdolmaleki, Jost Tobias Springenherg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum a posteriori policy optimisation. In International Conference Learn- ing Representations, 2018.
Hessameddin Akhlaghpour, Joost Wiskerke, Jung Yoon Choi, Joshua P Taliaferro, Jennifer Au, and llana B Witten. Dissociated sequential activity and stimulus encoding in the dorsomedial striatum during spatial working memory. Elife, 5:e 2016.
Yu Albert and Daniel Margoliash. Temporal hierarchical control of singing in birds. Science, 273 (5283):1871-1875, 1996.
Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski, Alexander Pritzel, Martin Chadwick, Thomas Degris, Joseph Modayil, et al. Vector-based navigation using grid-like representations in artificial agents. Nature, 557(7705):429, 2018.
Joshua D Berke, Jason T Breck, and Howard Eichenbaum. Striatal versus hippocampal representa tions during win-stay maze performance. Journal of neurophysiology. 101()):555-15577 2009.
Gordon Berman, Daniel M Choi, William Bialek, and Joshua W Shaevitz. Mapping the stereotyped behaviour of freely moving fruit flies. Journal of The Royal Society Interface, (099):20140672) 2014.
Hillel Chiel and Randall D Beer. The brain has a body: adaptive behavior emerges from inter- actions of nervous system, body and environment. Trends in neurosciences, 20(12):553-57,, 1997.
Mark M Churchland, M Yu Byron, John P Cunningham, Leo P Sugrue, Marlene R Cohen, Greg Corrado, William T Newsome, Andrew M Clark, Paymon Hosseini, Benjamin B Scott, al. Stim- ulus onset quenches neural variability: a widespread cortical phenomenon. Nature neuroscience, 13(3):369, 2010.
Mark M Churchland, John P Cunningham, Matthew T Kaufman, Justin D) Foster, Paul Nuyujukian, Stephen Ryu, and Krishna V Shenoy. Neural population dynamics during reaching. Nature, 487 (7405):51, 2012.
Christopher J Cueva and Xue-Xin Wei. Emergence of grid-like representations by training recurrent neural networks to perform spatial localization. arXiv preprint 1803. 2018.
Ashesh K Dhawale, Steffen BE Wolff, Raymond Ko, and Bence P Ölveczky. The basal ganglia can control learned motor sequences independently of motor cortex. bioRxiv, pp. 827261, 2019.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, al. IMPALA: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference Machine Learning, pp. 1406-1415. 2018
Michael Graziano. The organization behavioral repertoire in motor cortex. Annu. Rev. Neurosci., 29:105-134, 2006.
Sten Grillner, Alexander Kozlov, Paolo Dario, Cesare Stefanini, Arianna Menciassi, Anders Lansner, and Jeanette Hellgren Kotaleski. Modeling vertebrate motor system: pattern generation, steering and control of body orientation. Progress in brain research, 165:221-234, 2007.
Richard HR Hahnloser, Alexay A Kozhevnikov, and Michale S Fee. An ultra-sparse code underli- esthe generation of neural sequences in songbird. Nature, 419(6902):55, 2002.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Nicolas Heess, TB Dhruva, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, Ali Eslami, et al. Emergence of locomotion behaviours rich environ- ments. arXiv preprint arXiv: 1oXiv:7707.000 1ol0::2020002 2017.

---

Published as a conference paper at ICLR 2020
Norman C Heglund and C Richard Taylor. Speed, stride frequency and energy cost per stride: how do they change with body size and gait? Journal of Experimental Biology, (38(1)301-313 1988.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Auke Jan Ijspeert, Alessandro Crespi, Dimitri Ryczko, and Jean-Marie Cabelguen. From swimming to walking with salamander robot driven by a spinal cord model. Science, 315(5817):1416- 1420, 2007.
Eric Jonas and Konrad Paul Kording. Could a neuroscientist understand microprocessor? PLoS computational biology, 13(1)=e 2017.
John F Kalaska. From intention to action: motor cortex and the control of reaching movements. In Progness in Motor Control, pp. 178. Springer, 2009.
Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias B Issa, and James J DiCarlo. Evidence that recurrent circuits are critical to the ventral streams execution of core object recognition behavior. Nature neuroscience, 22(6))974,, 2019.
Risa Kawai, Timothy Markman, Rajesh Poddar, Raymond Ko, Antoniu Fantana, Ashesh K Dhawale, Adam R Kampff, and Bence P Ólveczky. Motor cortex is required for learning but not for executing a motor skill. Neuron, 86(3):800-812, 2015.
Alexander JE Kell, Daniel LK Yamins, Erica N Shook, Sam V Norman-Haignere, and Josh H Mc- Dermott. task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals cortical processing hierarchy, Neuron, 98(3):630-644 2018.
Seyed-Mahdi Shaligh-Razav and Nikolaus Kriegeskorte. Deep supervised, but not unsupervised, models may explain it cortical representation. PLoS computational biology, 10(11):e103911, 2014.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. Similarity of neural network representations revisited. CoRR, 1001100550014, 2019.
Nikolaus Kriegeskorte and Jöm Diedrichsen. Peeling the onion of brain representations. Annual Review of Neuroscience, 42(1):407-433 2019.
Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis- connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:4, 2008
Yuri Lazebnik. Can biologist fix radio? Or, what I learned while studying apoptosis. Cancer cell, 273):179-182, 2002.
Timothy P Lillicrap and Stephen H Scott. Preference distributions of primary motor cortex neurons reflect control solutions optimized for limb biomechanics. Neuron, 77(1): -1916 2013.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9Novv) 2008
Josh Merel, Arun Ahuja, Vu Pham, Saran Tunyasuvunakool, Siqi Liu, Dhruva Tirumala, Nicolas Heess, and Greg Wayne. Hierarchical visuomotor control of humanoids. International Con- ference Learning Representations, 2019a.
Josh Merel, Matthew Botvinick, and Greg Wayne. Hierarchical motor control in mammals and machines. Nature Communications, 10(1):1-12, 2019b.
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid control. In International Conference on Learning Representations, 2019c.
Jonathan A Michaels, Stefan Schaffelhofer, Andres Agudelo-Toro, and Hansjörg Scherberger. A neural network model of flexible grasp movement generation. bioRxiv, pp. 742189, 2019.
12

---

Published as a conference paper at ICLR 2020
Jonathan W Mink.
The basal ganglia: focused selection and inhibition of competing motor pro- grams. Progress in neurobiology. 50(4):381-42) 1996.
John A Nyakatura. Kamilo Melo, Tomislav Horvat, Kostas Karakasiliotis, Vivian R Allen, Amir Andikfar, Emanuel Andrada, Patrick Arnold, Jonas Lauströer, John Hutchinson, et al. Reverse- engineering the locomotion of a stem amniote. Nature, 565(7739)3351, 2019.
Xue Bin Peng, Glen Berseth, and Michiel Van de Panne. Terrain-adaptive locomotion skills using deep reinforcement learning. ACM Transactions on Graphics (TOG), 35(4):81, 2016.
Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel Van De Panne. DeepLoco: Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Transactions on Graphics (TOG), 36(4):41, 2017.
Pavan Ramdya, Robin Thandiackal, Raphael Cherney, Thibault Asselborn, Richard Benton, Auke Jan Ijspeent, and Dario Floreano. Climbing favours the tripod gait over alternative faster insect gaits. Nature communications, 8: 8:144,4, 2017.
Alfonso Renart and Christian K Machens. Variability in neural activity and behavior. Current opinion neurobiology, 25:211-220, 2014.
Pavel E Rueda-Orozco and David Robbe. The striatum multiplexes contextual and kinematic infor-
mation to constrain motor habits execution. Nature neuroscience, 18(3):453, 2015.
Simon Schmitt, Jonathan J Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojeiech M Czarnecki, Joel Z Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, et al. Kickstart- ing deep reinforcement learning, arXiv preprint arXiv:1803.0388. 2018.
Francis Song, Guangyu R Yang, and Xiao-Jing Wang. Reward- training of recurrent neural networks for cognitive and value-based tasks. Elife, 6:e21492, 2017.
H Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, et al. V-MPO: On-Policy Maxi- mum Posteriori Policy Optimization for Discrete and Continuous Control. arXiv preprint 1:Xiv:1909.12238 2019.
Greg J Stephens, Bethany Jahnson-Kernee William Bialek, and William S Ryu. Dimensionality and dynamics in the behavior of elegans. PLOS computational biology, 4(4)ee100002 2008
David Sussillo, Mark Churchland, Matthew T Kaufman, and Krishna Shenoy. neural network that finds naturalistic solution for the production of muscle activity. Nature neuroscience. 18(7): 1025, 2015.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Ried- miller. DeepMind control suite, arXiv preprint arXiv:1801.00690 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optimin hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy Sciences, 111(2):8619-862) 2014.
Guangyu Robert Yang, Madhura R Joglekar, Francis Song, William T Newsome, and Xiao-Jing Wang. Task representations in neural networks trained to perform many cognitive tasks. Nature neuroscience, 22(2):297, 2019.
Chengxu Zhuang, Jonas Kubilius, Mitra JZ Hartmann, and Daniel L Yamins. Toward goal-driven neural network models for the rodent whisker- trigemmna system. In Advances in Neural Informa- tion Processing Systems, pp. 2555-2565, 2017.
13

---

Published as a conference paper at ICLR 2020
A APPENDIX
A. RAT MEASUREMENTS
To construct the virtual rodent model, we obtained the mass and lengths of the largest body segments that influence the physical properties of the virtual rodent. First, we dissected cadavers of two female Long-Evans rats, and measured the mass of relevant limb segments and organs. Next, we measured the lengths of body segments over the skin of animals anesthetized with 2% v/v isoflurane anesthesia oxygen. We confirmed that these skin based measurements approximated bone lengths by measuring bone lengths in third cadaver. The care and experimental manipulation of all animals were reviewed and approved by the appropriate Institutional Animal Care and Use Committee.
Animal (#)
64
Body part
Mass (g)
Average mass (g)
Hindlimb L
21
26
23.5
Hindlimb R
21
26
23.5
Tail
10
Forelimb
14
12.5
Forelimb L
12
13
12.5
Full torso
176
187
181.5
Head
26
26
26
Upper torso
71
74.5
Lower torso
98
114
106
Torso without organs
54
58
56
Intestines and stomach
22
32
27
Liver
26
17
21.5
Pelvis and kidneys
80
77
Jaw
2.43
4.70
3.57
Skull
23
21
22
Tail (base to mid)
5.92
7.20
6,56
Tail (mid tip)
1.78
2.30
2.04
Scapula I.
3.19
4.70
3.94
Humerus
6.25
4.70
5.48
Radius/ulna L
2.61
2.8
2.70
Forepaw
0.53
0.5
0.52
Scapula R
2.23
3,9
3.07
Humerus
6.08
6.7
6.39
Radius/ulna R
2.17
3.3
2.74
Forepaw R
0.53
0.5
0.52
Hindpaw
1.66
1.7
1,68
Tibia L
9
9
Femur L
13
16
14.5
Hindpaw R
1.81
1.6
1.71
Tibia R
5.5
Femur R
13
18
15.5
Total
281
301
291
Table 1: Before weighing, limb segments were divided at their respective joints. Mass of all seg- ments includes all bones, skin, muscle, fascia and adipose layers. L and R refer to the left and right sides of the animal. Precision of measurements listed without decimal places is +0.5g
14

---

Published as a conference paper at ICLR 2020
Animal (#)
48
62
55
56
64
63
62
Average + std
Age (days)
382
82
330
330
83
83
83
Mass (g)
325
273
389
348
283
269
273
309 = 47
Body part
Length (mm)
Ankle claw L
40.2
39.5
39.7
37.8
39.9
41.5
39.8
39.8 1.1
Ankle to toe L
38.4
38.12
37.7
35.6
36.6
39.3
38
37.7 17.7122
Ankle to pad
23.4
22.2
23
22.12
22.5
23.3
6.4
20.4
Ankle to claw R
38.2
40.4
38.3
39.3
39.6
38.3
39.0
Ankle
R
37
38.7
36.3
37.7
38.6
36.2
3.1
Ankle to pad R
22.4
23.3
21.9
21.8
23.1
24.1
2
Tibia I.
50
36.3
38.5
49.2
35.8
38.7
34.1
4.5
Femur L
44.5
31.6
32.1
37.9
33.4
35.35
32.4
3.6
Tibia R
36.7
39.1
37.9
35.1
38.4
36.18
1.5
Femur R
32.9
32.1
38.7
31.9
32.1
32.6
3.64
Pelvis
25.8
32
31.7
30.2
26.7
27.2
28.9
Wrist claw
15
18.8
17.6
18.6
16
19.02
19.2
1.6
Wrist to finger I.
16
15.8
17.4
15.5
17.07
17.6
16.6
Wrist pad I.
6.4
8.34
4.9
6.1
6.4
6.4
Wrist to olecranon L
29.1
34
32.5
31.7
33.9
32.1
29.9
3.99
Humerus L
31.9
29.52
31
28.2
27
31.2
25.4
29.2
Scapula L
22.7
24
26.4
29.3
25.9
29.1
26.2
26.2
Wrist claw
16.8
17.8
15.9
16.3
18.1
17.0
Wrist to finger R
14.1
13
15.6
15.6
15.3
16.9
15.1
Wrist to pad R
5.6
5.8
6.55
5.8
5.7
Wrist olecranon R
30.6
33.5
31.2
30.4
31.8
29.9
1.3
Humerus R
28.2
33.5
28.8
25
28.2
25.2
28.1 3.11
Scapula R
23.8
29.5
25.9
26.2
28.8
24.4
26.4 2.3
Headcap width
39
39
Headcap length
30
30
Skull width
38.8
23.35
23
21.8
22.8
23.9
22.2
25.1 6.1
Skull length
57
51.1
61
56.48
53.16
58.13
48
5.50
Skull height
21.59
21.5
21
0.3
Head thoracic
48.6
71.4
68.68
65
60.4
71.2
6.72
Thoracic to sacral
73.1
73.6
62.9
65.04
64.7
68.8
68.0
Head sacral
145
126
145.5
127.05
127.2
123.7
140.9
133.6
Head width
53.4
53.4
Ear Eye
18
17.55
19.3
17.9
19.2
18.8
18.5 - 0.7
7.2
8.25
8.6
8.8
8.2
8.3
8.2 0.6
Table 2: Length measurements of limb segments used construct the virtual rodent model from female Long-Evans rats. Measurements were performed using calipers either over the skin or over dissected bones (*). Thoracie and sacral refer vertebral segments. and R refer the left and right sides of the animal's body.
15

---

Published as a conference paper at ICLR 2020
A.2 BEHAVIORAL ANALYSIS
We generated features describing the whole-body pose and kinematics of the virtual rodent on fast, intermediate, and slow temporal scales. To describe the whole-body pose, we took the top 15 prin- cipal components of the virtual rodent's joint angles and joint positions to yield two 15 dimensional sets eigenpostures (Stephens al. 2008). We combined these into 30 dimensional set of postu- ral features. To describe the animal's whole-body kinematics, we computed the continuous wavelet transform of each eigenposture using Morlet wavelet spanning 25 scales. For each set of eigen- postures this yielded a 375 dimensional time- fmeefrequyncy representation of the underlying kinematics. We then computed the top 15 principal components of each 375 dimensional sentation and combined them yield 30 dimensional representational description of the animal's behavioral kinematics. To facilitate comparison kinematics to neural representations on differ- ent timescales, used three sets of wavelet frequencies on I to 25 Hz (intermediate), 0.3 to 5 Hz (slow) or 5-25 Hz (fast) timescales. kinematic information improves separation of animal behaviors in behavioral emheddings. There- fore, we combined postural and dynamical features, the later on intermediate timescales, to yield a 60 dimensional set of 'behavioral features' that we used to map the animal's behavior using (SNE (Figure 4C) (Berman et al. 2014). ISNEs were made using the Barnes-Hut approximation with perplexity of 30.
frequeryy repre-
separate work, we have found that combining postural and
A.3 POWER SPECTRAL DENSITY BEHAVIOR AND NETWORK ACTIVITY
A
Joint speed
Kinematics (fast)
B
Core
Policy 1
Forces
Kinematics (slow)
Policy 0
Policy 2
Power Spectral Density (a")/Hz
Locomotion
Power Spectral Density (o²)/Hz
Locomation
-2
4
-6
10
10
10"
10
Frequency (Hz)
Frequency (Hz)
Figure 9: (A) Power spectral density estimates of four different features describing animal behavior, computed by averaging the spectral density of the top ten principal components of each feature, weighted by the variance they explain. (B) Power spectral density estimates of four different net- work layers, computed by averaging the spectral density the top ten principal components of each matrix of activations, weighted by the variance they explain. Notice that policy layers have more power in high frequency bands than core layers. Arrows mark peaks in the power spectra corre- sponding locomotion. Notably, the 4-5 Hz frequency of galloping in the virtual rat matches that measured in laboratory rats (Heglund & Taylor 1988). Power spectral density was computed using Welch's 's method using 10 window size and 5 overlap.
16

---

Published as a conference paper at ICLR 2020
A.4 REPRESENTATIONAL SIMILARITY ANALYSIS
We used representational similarity analysis to compare population representations across different network layers and compute the encoding strength of different features describing animal behavior in the population. Representational similarity analysis has in the past been used to compare neural population responses tasks where behavioral stimuli are discrete, for instance corpuses of objects or faces (Kricgeskorte et al. 2008; Kriegeskorte & Diedrichsen 2019). A challenge in scaling such approaches to neural analysis the context of behavior is that behavior unfolds continuously in time. It is thus priori unclear how to discretize behavior into discrete chunks in which to compare representations.
Formally, we defined eight sets of features B.- describing the behavior of the animal on different timescales. These included features such as joint angles, the angular speed of the joint angles, eigenposture coefficients, and actuator forces that vary on short timescales, as well behavioral kinematics, which vary on longer timescales and 'behavioral features', which consisted of both kinematics and eigenpostures. Each feature set is matrix B, E gM where M is the number of timepoints in the experiment and ge is the number of features in the set. We discretized each set B; using k-means clustering with & = 50 yield partition of the timepoints in the experiment P.-
Using the discretization defined in P,. we can perform representational similarity analysis to com- pare the structure of population responses across neural network layers Lym and L,, or between given network layer and features of the behavior B,. Following notation in (Korbblith et al.) (019) let X E be matrix of population responses across p neurons and the k behavioral cat- egories in Pt. We let Y E Rigg be either the matrix of population responses from neurons in distinct network layer, or set of q features describing the behavior of the animal in the feature set B,
After computing the response matricies in given behavioral partition, we compared the representa- tional structure of the matricies X and YYT. To do so, we compute the similarity between these matricies using the linear Centered Kernel Alignment index, which is invariant under orthonormal rotations of the population activity. Following (Kornblihh et al. 2019), the CKA coeffient is:
|XY CKA(XX* ,YY") 1XXT||e|YYTT
(1)
Where ||= the Frobenius norm. For centered X and Y. the numerator is equivalent to the dot- dot-product between the vectorized responses ||YT|| = (vee(XX"))vve(x>"))
voc(YY
For given network layer I.m, and a behavioral partition P., we can denote Xx" = =D/,"" = DT Similarly, for a given feature set B;, let DP; DJ. Thus we are interested characterizing both
CKA(D D.)
and
CKA "KA" DD.
The former equation describes the similarity across two layers of the network, and the later describes the similarity of the network activity to a set of behavioral descriptors.
An additional challenge comes when restricting this analysis comparing the neural representations of behavioral across different tasks Ta, Th, where not all behaviors are necessarily used in each task. To make such a comparison, we denote B; (Ta) to be the set of behavioral clusters observed in task Te, and B. B,(T.) (T,).) to be the set of behaviors used each of the two tasks. We can then define a restricted partition of timepoints for each task Pi or pils.la that includes only these behaviors, and compute the representational similarity between the same layer across tasks:
CKA (DEr, D.r.)
We have presented means of performing representational similarity analysis across continuous time domains, where the natural units of discretization are unclear and likely manifold. While we focused on analyzing responses on the population level, it likely that different subspaces of the population may encode information about distinct behavioral features at different timescales, which still an emerging domain representational similarity analysis techniques.
17

---

Published as a conference paper at ICLR 2020
A.5
NEURAL POPULATION ACTIVITY ACROSS TASKS DURING RUNNING
Running activity
Bowl escape
Gaps run
Maze forage
Two-tap
Bowl escape
Gaps run
Sorted by:
O Absolute average z-scored activity
Maze forage
Two-tap
-0.1
0
0.1
-0.1
0
0.1
-0.1
0.1
-0.1
0
0.1
Time (s)
Figure 10: Average activity the final policy layer (policy 2) during running cycles across different tasks. In each heatmap, rows correspond to the absolute averaged z-scored activity for individ- neurons, while columns denote time relative the mid stance of the running phase. Across heatmaps, neurons sorted by the time of peak activity in the tasks denoted on the left, such that each column heatmaps contains the same average activity information with rearranged rows. Aligned running bouts were acquired by manually segmenting the the principal component space of policy 2 activity to find instances of mid-stance running and analyzing the surrounding 200 ms.
18

---

Published as a conference paper at ICLR 2020
A.6 STEREOTYPED BEHAVIOR INITIATION AND NEURAL VARIABILITY
During the exccution of stereotyped behaviors, neural variability reduced (Figure ITI. Recall that our setting, neurons have no intrinsic noise, but inherit motor noise through observations of the state (i.e. via sensory reafference). This effect loosely resembles, and perhaps informs line of interpretation the widely reported phenomenon of neural variability reducing with stimulus or task onset (Churchland et al., 2010). Our reproduction of this effect, which simply emerges from training, suggests that variance modulation may partly arise from moments in task that benefit from increased behavioral precision (Renart & Machens 2014).
1.4
Normalized standard deviation
1.2
poll
polioy2
Normalized Activity
1.0
.8
.6
.4
.2
2.5a
-1.5
-1.5
-1
Time (s)
Time (s)
Figure 11: Quantification of neural variability in inter-tap interval of two-tap task relative to the second tap. (A) Example normalized activity traces of ten randomly selected neurons the final policy layer. Lines indicate mean normalized activity whiles shaded regions range from the 20th percentile to the 80th percentile. Dashed lines indicate the times of first and second taps. (B) Standard deviation of normalized activity across all neurons in the final policy layer a function of time relative to the second tap. Lines indicate the mean standard deviation while shaded regions range from the 20th percentile to the 80th percentile. Observe that variability is reduced during the two-tap interval.
A.7 NEURAL DYNAMICS VISUALIZED DURING TASK BEHAVIOR
For completeness, we provide links to videos of few variants of neural dynamics for each task.
Network
Visualization
Task (link)
l-layer policy
PCA
gaps
PCA
forage
PCA
escape
PCA
two-tap
3-layer policy
PCA
gaps
PCA
forage
PCA
escape
PCA
two-tap
3-layer policy
jPCA
gaps
JPCA
forage
jPCA
escape
jPCA
two-tap
Table Links to representative visualizations of neural dynamics and behavior
19

---

Published as a conference paper at ICLR 2020
A.8
PERTURBATION RESULTS
First tap
Inter tap
Palicy 2
Bad spin jump
Pulyy
Proportion correct
08
Rear
06
04
02
First tap
Inter tap
Missed orb
inactivation
Proportonn correct
68
Spinning
Jump
68
16
64
120
Number of Neurons
First
Inter tap
Policy
Extra spin
Propostion eicted sprts
80
06
04
64
Number of Neurons
Figure 12: Causal manipulations reveal distinct roles for core and policy layers in the production of behavior. (A) Two-tap aceuracy during the inactivation of units modulated by idiosyncratic be- haviors within the two-tap sequence. Core inactivation has weaker negative effect trial success than policy inactivation for several levels of inactivation.
(B) Representative example of a failed trial during inactivation of the final policy layer in model that performs spinning jump during the two-tap sequence. The model is incapable producing the spinning jump behavior while inacti- vated. (C) Representative example of failed trial during
inactivation in model that performs a spinning jump during the two-tap sequence. The model still able to perform the spinning jump behavior, but misses the orb. (D) Proportion of attempts stimulation that successfully elicited spin behavior during the two-tap task. The efficacy of this activation was more reliable in layers closer the motor output. (E) Representative example of single trial in which extra spin occurs after policy 2 activation.
20


Reviews:
[{"review_id": "rkgMsbnQ9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is a fascinating paper that uses methods from computational neuroscience to characterize a neural network that controls a virtual rat (or, well, something ratlike). I really like the idea of trying to fuse neuroethology and animal behavior research in general with deep learning methods. To me, it seems like there was plenty of activity in this field around 20 years ago (with e.g. the robot models of animal behavior of Ijspeert and others, and the evolutionary robotics approach to study the evolution of behaviors) but that this research line has not merged with (alternatively informed, or learned from) modern deep learning. So I welcome this direction of research. This being said, it's not my field of research, so I'm unable to comment on several of the specifics here.\n\nThe learning of a network that can perform these four independent tasks is quite impressive in its own right. I would like the paper to comment on how hard or easy this was, if you attempted to learn other behaviors but failed, etc.\n\nThe actual analysis is a bit of a letdown - not because it seems to be wrong or incompetently done, rather the opposite - but in that there is so little to learn from it. Simply put, I do not understand anything more about networks that control simulated robots to perform multiple tasks (or about real rodents) after reading the paper. What does this mean? Is this an indictment of current neuroscience methods, that even when you have unambiguous non-noisy access to all of the state of the network, you cannot really find out much? If so, you should discuss this. If not, you should explain what's going on. At least from the perspective of an AI researcher, there just isn't enough understanding there.\n\nHowever, I don't think negative results (which this in a way is) should discourage from publication. I applaud the intent of the paper, the competence with which it was executed, and the learning of the network. So I want to see this published. But please remark on the points above.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rJl1PmhsYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors use a virtual rodent to study flexible behavior, a key concept at the intersection of AI and neuroscience. “Flexibility” is a common buzzword in this field used to describe what it is that mammals do better than deep nets, even as deep nets can outperform them on any one task in isolation with sufficient training. As the authors note, previous work addresses “cognitive flexibility” (Yang et al. 2019) in isolation without motor commands, or studies motor dynamics of real mammals but without any task flexibility—occluding the actual advantages that mammals have over machines. However, this work goes the full mile in requiring the agent to implement flexible cognition using a complex body in a physical environment with egocentric stimuli. From a strictly AI point of view, I could imagine arguments that this is not sufficiently “novel,” in that a lot of deep RL researchers have already trained agents with complex joint structures to succeed in such virtual worlds. But the real substance of the paper is the application of data analysis techniques imported directly from neuroscience. It is important to suss out what features of neural dynamics are peculiarities of biological computation and which are natural consequences of the tasks being performed. I find the paper to be overall well motivated.\n\nThe analyses were well ordered, starting with a high-level characterization of behavior that showed a hierarchy of features, where “low-level” or “fast” features are mixed across tasks while “high-level” or “slow” features are more task specific. They then moved on to find “neural correlates” at the behavior and task levels, driving home their primary point that core layers encode high-level variables in a way not shared across tasks while policy layers implement reusable core motor functions that are used in multiple contexts. This observation coincides with the measurable frequency content of core and policy activity, which is demonstrably faster in policy layers. Finally, the authors show a visualization of the dynamical systems that underlie these computations, with videos that very clearly illustrate how core and policy dynamics dissociate to specialize in cognition vs. motor generation. This part is where they really leverage the artificial setting, where the exact dynamics are known and do not need to be inferred from noisy neural data.\n\nWhile the end result of the analyses led to convincing visuals that made the intended point, the details of the analysis themselves were hard to follow and evaluate for rigor. An appendix explaining with notation how the behavioral “features” were computed, for example, would have been helpful. I acknowledge that it is a feature of the paper that so many different analysis techniques from neuroscience—dynamical systems to representation analysis to lesion studies—were applied, and the sheer breadth of the analysis was impressive, if difficult to evaluate.\n\nAlthough I recognize that to do so would be difficult, I think what would add a lot to this paper would be side-by-side comparisons of real neural data with “neural” data from the artificial rodent. Perhaps the behavior would have to be modified so that some tasks match specific neuroscience paradigms, or the sample size for analysis limited to what is realistic for experimental data. It would be neat to see if, say, preparatory activity signals could emerge from training such an artificial rodent to do many tasks, including a head-fixed timed lever pull or some other classic paradigm. Maybe future work can draw a more exact analogy between policy/core layers and specific brain regions.\n\nAs to topic fit, this is niche in the context of ICLR; however, showing how state-of-the-art deep RL techniques can be applied to neuroscientific questions may be of interest to machine learning researchers even if they are not also neuroscientists.\n\nOverall, I thoroughly enjoyed this paper and look forward to the new kinds of research that sprout out of it!", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_thoroughness_in_paper_reading": "N/A"}, "extra_text": {}}, {"review_id": "r1g-jDM6Fr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n=============================== Update after rebuttal ======================================================\n\nI thank the authors for their rebuttal and the revisions. I'm not entirely satisfied with the authors' response to my request for more architectural exploration, but I understand that there wasn't really enough time for this during the relatively short rebuttal period. I think the results in the paper are still valuable enough to merit publication, so I'm happy to increase my score and recommend accepting the paper. However, I still encourage the authors to take the issue of arhcitectural plausibility more seriously in the future, especially if the intended audience for this line of work is experimental neuroscientists. Currently, the architecture choice seems to be dictated primarily by trainability considerations (more specifically, trainability by current deep learning methods).\n\n========================================================================================================\n\nThis paper introduces a virtual rodent model with a complex set of actuators and visual and proprioceptive inputs. The model is simultaneously trained on four different tasks and the trained model is analyzed using various dimensionality reduction and visualization methods. The effort put into training and analyzing the rodent model is quite impressive. Moreover, the tools that the authors will make public can be useful for other researchers in this field. \n\nHowever, my main concern about the paper is that given the architecture choice made in the paper, most of the main results do not seem very surprising. On the other hand, the architecture choice itself is not motivated well enough. The differences between the dynamic and representational properties of the core and policy networks entirely depend on the fact that the core network is trained separately from the policy network. Why was this particular choice made? In a more realistic scenario (for example, in the actual brain of a rodent), everything will presumably be connected to most everything else to a certain degree, with no sharp separation between policy and core modules and the error signals flowing more broadly across the entire network. It seems to me that in such a scenario, there wouldn't be such a big difference between the dynamic and representational properties of different modules in the network. So, I am wondering if it is possible to train more models with alternative architectures (with presumably more realistic properties) and compare the results with the current results.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is somewhat unorthodox in what it sets out to do: use neuroscience methods to understand a trained deep network controlling an embodied agent. This is exciting, but the actual training of the virtual rodent and the performance it exhibits is also impressive in its own right. All reviewers liked the papers. The question that recurred among all reviewers was what was actually learned in this analysis. The authors responded to this convincingly by listing a number of interesting findings. 

I think this paper represents an interesting new direction that many will be interested in.
Decision: Accept (Spotlight)