IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2020.

1

Deep Learning of Neuromuscular and Visuomotor
Control of a Biomimetic Simulated Humanoid
Masaki Nakada, Tao Zhou, Honglin Chen, Arjun Lakshmipathy, and Demetri Terzopoulos, Fellow, IEEE

Abstract—We present a biomimetic framework for humanneuromuscular and visuomotor control that promises to be ofvalue to researchers developing humanoid robots. Our framework
features a biomechanically simulated human musculoskeletal
model, actuated by numerous skeletal muscles, with realistic
eyes driven by extraocular and intraocular muscles, whose optic
organs refract light, and whose retinas have many nonuniformly
distributed photoreceptors. The humanoid’s visuomotor control
system comprises 24 trained deep neural networks (DNNs)—
10 DNNs in its vision subsystem and 14 DNNs in its motor
subsystem—plus an additional 4 trained shallow neural networks(SNNs) that control the irises and lenses of the eyes. Of the
motor DNNs, a pair control the extraocular muscles, 6 per eye,responsible for eye movements, 2 control the 216 neck musclesof the cervicocephalic biomechanical complex, producing natural
head movements, 2 control the 443 core muscles of the torso,
and 2 control each limb; i.e., the 29 muscles of each arm
and 39 muscles of each leg. Directly from the foveated retinal
photoreceptor responses, a pair of foveation DNNs drive eye,head, and torso movements, while 8 limb vision DNNs extract
the visual information needed to direct arm and leg actions. By
synthesizing its own training data, our humanoid automaticallylearns efficient, online, active visuomotor control of its eyes, head,torso, and limbs in order to perform nontrivial tasks involving
the foveation and visual pursuit of moving target objects coupledwith visually-guided limb-reaching actions to intercept them. We
also demonstrate that it can balance itself in an upright stance,take steps, and perform certain simulated sports activities.
Index Terms—Biomimetics, Simulation and Animation, Deep
Learning in Robotics and Automation, Visual Learning, Humanoid Robots

I. I NTRODUCTION
NTHROPOMORPHIC physical robots, in some sense the“Holy Grail” of robotics, have been pursued by several
research groups [1], [2], [3], [4], [5], [6]. When musculoskeletal
structures are attempted, they usually are highly simplified
versions of their biological analogs, but researchers have shownthat such biomimetic robotic systems can work. For example,
Kurumaya et al. [7], [8] developed a humanoid robot with a
musculoskeletal structure employing multi-filament muscles
that have characteristics not unlike human muscles.
Visuomotor functionality in biological organisms refers
to the acquisition and processing of visual input and theA

Manuscript received September 10, 2019; revised December 12, 2019;
accepted January 11, 2020.
This paper was recommended for publication by Editor Nancy Amato
upon evaluation of the Associate Editor and Reviewers’ comments. The work
reported herein was presented to the RA community by D. Terzopoulos in an
ICRA 2019 keynote address, a privilege for which he is grateful to Gregory
Dudek and the other conference organizers.
The authors are with the Computer Science Department, University ofCalifornia, Los Angeles, CA 90095, USA.
Digital Object Identifier (DOI): see top of this page.

production of appropriate motor output responses to perform
desired tasks. Visuomotor systems have also been exploredfor several decades in robotics [9], [10], [11], [12], [13].
Again, biomimetic approaches have been severely challenged
by hardware constraints, but there have been some promisingattempts in recent years. Bjorkman and Kragic [14] and Kragic
et al. [15] introduced foveated visual processing using two
binocular cameras with narrow and wide fields of view toapproximate foveal and peripheral vision. Lesmana and Pai[16] and Lesmana et al. [17] employed a machine learning
approach to generate robotic eye saccade, pursuit and VOR
movements with oculomotor control systems.
In this paper, we present a strongly biomimetic simulation
framework for human visuomotor control that is free of
hardware constraints. Our framework is unique in that itfeatures a biomechanically simulated human musculoskeletal
model that currently includes no fewer than 823 anatomicallyaccurate skeletal muscles, which actuate head, torso, arm,and leg motions, complemented by a pair of biomechanicaleyes each with 2 intraocular muscles (iris sphincter and lensciliary muscle) for visual accommodation plus 6 extraocular
muscles responsible for realistic eye movements. Our simulated
humanoid is a greatly enhanced version of a preliminary model[18] that had a non-functional torso (immobilized spine) andsimplistic, kinematic eyes.
Our biomimetic visuomotor control system is unprecedented
both in its use of a sophisticated biomechanical human modeland in its use of modern machine learning methodologies. A
modular set of neural networks controls the realistic musculoskeletal system and performs online visual processing foractive, foveated perception and neuromuscular motor control.
The neural networks are automatically trained from datasynthesized by the human model itself.
Our muscle-actuated model represents a major advance inrealism over the previous generation of simulated humanoids
driven by rotational, “servomotor” joint actuators (e.g., [19]).
Hence, our work should be of value to researchers developing
sophisticated humanoid robot control systems, particularly
since it demonstrates, through the synergistic combination ofcompatibly biomimetic motor and sensory neural controllers,
what is possible with a simulated humanoid whose level ofrealism greatly exceeds current robotic hardware systems.
Furthermore, the benefit of a biomimetic simulated humanoid
operating within in its physics-based virtual environment isthat it can potentially provide a high-fidelity testbed for futureanthropomimetic robots, offering a quick and inexpensive
approach to developing and testing new biologically-inspired
sensorimotor control theories, thereby accelerating progress.

2IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2020.

(b)RVision Subsystem

(b)L

(a)

Right Retina

Left Retina
ONV

ONV

⋮⋮⋮ ⋮⋮⋮ ………
⋮⋮⋮ … ⋮⋮⋮

⋮
⋮

⋮

⋮

Δa1

Δa6

∆

(l)L DNNs 18,24

.

∆

∆

∆

.

(e)L Left Eye (section)

… ⋮⋮
⋮⋮ …

(m)

∆

∆

⋮⋮

⋮⋮

∆

.

⋮

∆

… ⋮⋮
⋮⋮ …
∆

⋮ … ⋮

∆

⋮⋮

∆

Δp

⋮⋮

(e)L

∆

⋮⋮ …… ⋮⋮

(e)R

⋮⋮

∆

∆

∆

⋮⋮

⋮⋮

∆

∆

(k)L DNNs 16,22⋮⋮ …… ⋮⋮∆(l)R DNNs 17,23(e)R Right Eye (section)

⋮⋮

∆

⋮⋮

∆

⋮⋮

∆

∆

⋮⋮ …… ⋮⋮

∆

… ⋮⋮
⋮⋮ …

⋮

Δa6

⋮⋮

∆

∆

∆

.

a6

.

⋮⋮

⋮ … ⋮

(k)R DNNs 15,21

Δθ(j) DNNs 14,20(i) DNNs 13,19.

Δa1(h)L DNN 12

Average

a6

Δl

∆

∆

∆

⋮

⋮

Δθ

∆

Average(h)R DNN 11

⋮

⋮

⋮

∆∆ ∆∆∆∆∆∆(c)L SNN 2

⋮

⋮

⋮

Δl

∆

(d)L SNN 4

⋮

⋮
Δp

∆∆ ∆∆∆

∆∆

∆

∆

⋮ … ⋮

(d)R SNN 3⋮ … ⋮(c)R SNN 1(f)L DNN 2⋮(g)L DNNs 4,6,8,10⋮(g)R DNNs 3,5,7,9⋮⋮⋮ ⋮⋮⋮ ………
⋮⋮ ⋮ … ⋮⋮⋮(f)R DNN 1

Motor Subsystem
Fig. 1: Architecture of the visuomotor system. The controllers in the system include a total of 24 DNNs and 4 SNNs (Fig. 3).
Vision Subsystem (top): Each retinal photoreceptor casts rays through the eye (Fig. 2a,b) and into the virtual world (a), which compute theirradiance at that photoreceptor. (b) The arrangement of the 14,400 photoreceptors (black dots) on the right R and left L foveated retinas.
Each eye outputs a 43,200-dimensional RGB Optic Nerve Vector (ONV). This feeds four trained visual accommodation SNNs (1–4); SNNs(c)R (1) and (d)R (3) control the muscles of the iris and lens of the right eye (e)R, and SNNs (c)L (2) and (d)L (4) do the same for the lefteye (e)L. The ONV also feeds ten trained vision DNNs (1–10). (f) A pair of foveation DNNs (1,2) produce outputs that drive the movementsof the eyes to foveate visual targets. (g) The eight limb vision DNNs (3–10) — (g)R (3,5,7,9) for the right eye and (g)R (4,6,8,10) for theleft eye — output observed limb-to-target discrepancy estimates.
Motor Subsystem (bottom): Fourteen trained neuromuscular motor DNNs (11–24) comprise the motor subsystem, including eight voluntary
motor DNNs (11–18) and six reflex motor DNNs (19–24). (h) The oculomotor DNNs (11,12), which are driven by the outputs of the foveation
DNNs, output muscle activation signals that control the six extraocular muscles of each eye to produce eye movements. Driven by theaveraged responses of the foveation DNNs, along with the current activations of the 216 neck muscles and 443 torso muscles, respectively, thecervicocephalic (i) voluntary motor DNN (13) and torso (j) voluntary motor DNN (14) each outputs muscle activation signals that contributeto actuating its associated neuromuscular complex. Driven by the bilaterally pairwise averaged responses of the limb vision DNNs, alongwith the current activations of the 29 muscles of each arm or 39 muscles of each leg, respectively, each of the four limb voluntary motor
DNNs (k) (l) (15–18) outputs muscle activation signals that contribute to actuating its associated neuromuscular complex. Each of the six
reflex motor DNNs (19–24) outputs muscle activation signals that contribute by stabilizing the muscle group of its associated musculoskeletal
complex (Fig. 4).

NAKADA et al.: DEEP LEARNING OF NEUROMUSCULAR AND VISUOMOTOR CONTROL OF A BIOMIMETIC SIMULATED HUMANOID

3

II. H UMANOID M ODELAs shown in Fig. 1, the visuomotor control system of ourhuman model consists of a set of 24 automatically-trained,
fully-connected Deep Neural Networks (DNNs) that operatecontinuously and synergistically, 14 of which are motor control
DNNs in the motor subsystem, while the other 10 are vision
DNNs in the vision subsystem. Additionally, 4 Shallow Neural
Networks (SNNs) associated with the biomimetic eyes, suffice
to control focal accommodation by activating the lens ciliarymuscles as well as the iris sphincter muscles to regulate theamount of light that reaches the foveated retinas and their
human-like, irregular photoreceptor distributions.
Directly from the retinal photoreceptor responses, a pairof foveation DNNs in the vision subsystem (upper Fig. 1)
drive eye, head, and torso movements, while eight limb vision
DNNs extract the perceptual information needed to drive
the actions of the limbs. In the motor subsystem (lower
Fig. 1), a pair of oculomotor DNNs control the 6 extraocular
muscles per eye to perform eye movements. Additionally,
two motor DNNs control the 216 neck muscles that actuate
the cervicocephalic musculoskeletal complex to balance thehead atop the flexible cervical column and produce controlled
head movements. Two more control the 443 core musclesof the thoracic/lumbar musculoskeletal complex that support
and actuate the torso. Finally, eight motor DNNs control the
musculoskeletal complexes of the four limbs; in particular, the29 muscles of each arm and the 39 muscles of each leg.
Thus, our simulated humanoid is capable of learning efficient,
online visuomotor control of its eyes, head, torso, and four
limbs to perform nontrivial motor tasks driven exclusively by
its egocentric, active visual perception.

(a)

(b)

(c) Left retina (d) Right retina

Fig. 2: Via raytracing (blue rays), the biomechanical humanoid’seyes (b) visually sample the environment (a) to compute theirradiance at the locations of RGB photoreceptors (black dots)
placed according to a foveated, noisy log-polar distribution on
their retinas (c) (d).
numerically simulated as a force-driven articulated multi-bodysystem.4
Each muscle actuator is activated by an independent, timevarying, efferent activation signal a(t). Given our human model,
the overall challenge in neuromuscular motor control is todetermine the activation signals for each of its 823 muscles
necessary to carry out various motor tasks.
B. Biomechanical Eye Model

We next summarize our biomimetic eye model, the detailsof which are presented in [23].
To model the eyes, we have taken into consideration
physiological data. As shown in Fig. 1e, we model the virtual
eye as a sphere of radius 12mm with the typical 7.5 g massof the human eyeball, which can be rotated with respect toits center around its vertical y axis by a horizontal angle ofθ, around its horizontal x axis by a vertical angle of φ, andaround the gaze z axis by a torsion angle ψ. The eyes lookforward in their neutral positions when θ = φ = ψ = 0◦ .
As with the human eye, each virtual eyeball is actuated by 6
A. Biomechanical Body Model
extraocular muscles, including the 4 rectus muscles that actuateFig. 1m displays the musculoskeletal system of our anatomimuch of the θ-φ movement, and the 2 oblique muscles thatcally accurate human model, which includes all of the relevant
induce torsion.
articular bones and muscles—193 bones connected by joints
Our biomimetic eye model includes a cornea that refracts
comprising 163 articular degrees of freedom, plus a total of 823
light rays, an iris with a finite-aperture pupil capable of dilation
muscle actuators embedded in a finite element model of theand constriction to regulate the incoming light, and a modelmusculotendinous soft tissues of the body.1 We designed ourof the ocular lens that further refracts light rays and, throughbiomechanical humanoid’s musculoskeletal system by referring
active lens deformation, can focus them onto the retinal surface.
to one of the most comprehensive, albeit purely geometric,See [23] for additional details.
commercially available human models, the Ultimate Human
1) Model of the Retina: The retina is the hemispherical inner
Model.2 Biomechanical parameters and muscle attachment
surface at the rear of the eyeball. Our virtual retina emulates
points were determined using the geometric model as a
how biological retinas sample scene radiance from the incidence
reference.3 Each skeletal muscle is modeled as a Hill-typeof light on photoreceptors. The irradiance at the locations ofuniaxial contractile actuator that applies forces to the bones
the photoreceptors, is computed using the raytracing technique
at its points of insertion and attachment. The human model isof computer graphics rendering [24]. Fig. 2a,b illustrates
the retinal “imaging” process. Each photoreceptor gathers
1 The finite element soft-tissue simulation, which produces realistic flesh
deformations, is unnecessary in the scope of the present paper and is suppressed; light from the environment through the finite-aperture pupil,
as follows: Sample rays emanate from the position of the
however, the skeletal mass distribution remains that of an adult male body.
2 https://www.turbosquid.com/3d-models/3d-human-anatomy-ultimate/1093983
photoreceptor, are refracted at the lens and corneal surfaces, and3 We opted against using the oversimplified biomechanical human models
cast into the 3D virtual world where they recursively intersect
available to the community, such as OpenSim [20], which is not a whole-bodymodel, and Anybody [21], a whole-body model with only 458 muscles. Per with the visible surfaces of virtual objects and sample the virtual
the Library of Congress (http://id.loc.gov/authorities/subjects/sh85088687.html), thehuman body has 650 named skeletal muscles and as many as 840 including
unnamed muscles.

4 For the details, see [22] and Appendix I. Appendices I, II, and III arefound in the supplemental document.

4IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2020.

∆

∆

∆

∆
∆

1000

500
1000

43,200‐D ONV

50

100

250

500

∆

25

1000

43,200‐D ONV

50

100

250

500

25

43,200‐D ONV(a) Iris and lens SNNs(b) Foveation DNNs(c) Limb vision DNNs
∆

∆
∆

∆

∆

∆

∆

∆

∆
∆
∆

∆

∆
300

300

300

300

300

.

∆

∆

300

300

(d) Ocular, cervicocephalic, and torso
voluntary motor DNNs

300

300

300

300

300

(e) Limb voluntary motor DNNs

.
300

300

300

300

300

300

(f) Reflex motor DNNs

Fig. 3: Network architectures of the (a)–(c) vision SNNs and DNNs, and (d)–(f) motor DNNs.

ઢaVoluntary
Motor DNN

Δav

Δe
.
Δe

ReflexMotor DNN

Δar

+ +

a

control DNNs in the motor subsystem, which drive head andtorso movements as well as the reaching actions of the limbs.
III. V ISUOMOTOR S YSTEM

Fig. 4: Architecture of a neuromuscular motor controller. The voluntary motor DNN inputs the discrepancy ∆ and, recurrently, the muscle
activations a, and outputs muscle activation updates ∆av , which
induce the desired actuation of the associated musculoskeletal complex.
The reflex motor DNN inputs the discrepancies in muscle strains eand strain rates ė, and outputs stabilizing activation updates ∆ar . The
output of the controller is a(t + ∆t) = a(t) + (∆av (t) + ∆ar (t)).

Fig. 1 overviews the visuomotor system, detailing its visionand motor subsystems. The figure caption describes theinformation flow and functions of its 24 DNN controllers(numbered 1–24 in the figure) and 4 SNNs. Fig. 3 illustrates thearchitectures of the various networks, which will be explainedin the ensuing sections.5
The vision subsystem includes 10 fully-connected, feedforward DNNs, numbered 1–10 in Fig. 1, which process the visual
information in the 43,200-dimensional ONV. These DNNs areof two types. The first, described in Section III-A1, controls theeye movements, as well as the head and torso movements via
the neck and torso neuromuscular motor controller. The second
type, described in Section III-A2, produces arm-to-target 3D
discrepancies [∆x, ∆y, ∆z] that drive the limbs via the limb
neuromuscular motor controllers.
The motor subsystem includes a pair of oculomotor controllers and six neuromuscular motor controllers associatedwith the cervicocephalic, torso, and four limb musculoskeletal

light sources in accordance with the standard Phong local
illumination model. The total irradiance at each photoreceptor
is the weighted sum of the associated sample ray irradiances.
Unlike the uniform, Cartesian grid arrangement of mostrobot imaging sensors, visual sampling in the primate retina
is known to be strongly space-variant with the density oftrichromatic (cone) photoreceptors decreasing radially from the
fovea toward the periphery [25]. The log-polar distribution isoften used in space-variant image sampling models. To simulate
5 As is detailed in Appendix III, we conducted experiments with various
biomimetic foveated perception, we position the photoreceptors
neural network architectures, activation functions, and other parameters toon the hemispherical retina according to a noisy log-polar determine their suitability for our purposes. The 6-hidden-layer architecture
distribution. On each retina (Fig. 2c,d), we include 14,400 common to all our DNNs has proven in our experiments to provide the best
overall performance. All the networks illustrated in Fig. 3 employ rectified
RGB photoreceptors.
linear units (ReLU). Their initial weights are sampled from the zero-mean2) Optic Nerve Vectors: The foveated retinal RGB “image” normal distribution with standard deviation p2/fan in, where fan in is thecaptured by each eye is output for further processing down the number of input units in the weight tensor [26]. Network unit/weight counts:
visual pathway, not as an array of pixels, but as a 14,400 × (a)–(c) ∼45K/∼44M; (d)–(e) ∼2K/∼400K–700K; (f) ∼2K–3K/∼400K–800K.
The DNN training data synthesis procedures are described in the ensuing
3 = 43,200-dimensional vector of photoreceptor responses, sections and in Appendix II. To train the networks, we use the mean-squaredwhich we call the Optic Nerve Vector (ONV). The raw visual error loss function and the Adaptive Moment Estimation (Adam) [27] stochastic−6−3
information encoded in the ONV supplies the vision DNNs. optimizer with learning rate η = 10 , step size α = 10 , forgetting
factors β1 = 0.9 for gradients and β2 = 0.999 for second moments ofThese directly control eye movements and extract perceptual gradients. Overfitting is avoided using an early stopping condition: negligible
information that is passed on to the neuromuscular motor improvement for 10 successive epochs.

NAKADA et al.: DEEP LEARNING OF NEUROMUSCULAR AND VISUOMOTOR CONTROL OF A BIOMIMETIC SIMULATED HUMANOID

I(a) t0I(b) t15I(c) t2(d) t3Fig. 5: Time sequence of photoreceptor responses in the left retina during a saccadic eye movement that foveates and tracks amoving white ball — (a) at time t0 the ball enters the visual periphery, (b) at t1 the eye movement is bringing the ball towards
the foveal region, (c) at time t2 the moving ball is centering within the foveal region, (d) at time t3 the ball is foveated/tracked.

I
I
I
Fig. 6: Simulation of saccadic eye movement dynamics driven by the oculomotor DNNs. Eye movement (red trace) from one
gaze direction to the next (blue rays) is triggered by sudden transitions of the visual target (white ball).

complexes. Fig. 4 illustrates the neuromuscular motor controller networks, as follows: A white ball is presented within the visualarchitecture, which is comprised of a voluntary motor DNN field. By raytracing the 3D scene from the perspective of theand a reflex motor DNN (c.f. [28]), both of which are fully- eye, the photoreceptors on its retina are stimulated and theirconnected, feedforward DNNs that produce muscle activation RGB components comprise the ONV input to the network. The
adjustment signals, ∆a. The muscle activation a feedback desired output of a foveation DNN is the angular differences
loop makes the neuromuscular controllers recurrent networks. ∆ between the actual gaze directions of the eyes and the
The voluntary motor DNNs, numbered 11–18 in Fig. 1, are known gaze directions that would foveate the ball. Repeatedly
discussed in Sections III-B1 to III-B4, and the reflex motor positioning the visual target at random locations in the visual
DNNs, numbered 19–24, are discussed in Section III-B5. field synthesizes a large training dataset of 1M input-output
The oculomotor controllers, DNNs 11–12, do not require pairs offline. The backpropagation-trained DNN serves as an
stabilization from reflex motor DNNs.
online foveation controller.
2) Limb Vision DNNs (3–10): The role of the left and rightlimb (arm and leg) vision DNNs is to estimate the separation
A. Vision Subsystem DNNs
1) Foveation DNNs (1,2): The first role of the left and in 3D space between the position of the end effector (handright foveation DNNs is to drive saccadic eye movements that or foot) and the position of a visual target, thus driving thealter the gaze directions in order to foveate visible objects of associated limb motor DNN to extend the limb in order tointerest, thereby observing them with maximal visual acuity, as touch the target.
The architecture of the limb vision DNNs (Fig. 3c) is
illustrated in Fig. 5 for a white ball in motion that enters thefield of view stimulating a number of peripheral photoreceptors identical to the foveation DNNs except for the size of theTon the retina. The eye performs a rapid saccadic rotation to output layer, which has 3 units ∆ = [∆x, ∆y, ∆z] , to encode
foveate the visual target, as shown in Fig. 6. Fine adjustments the estimated discrepancy between the 3D positions of the end
comparable to microsaccades are observed during fixation, effector and the visual target.
Our humanoid trains its four limb vision DNNs, as follows:which is followed by smooth pursuit eye movement as the
A ball is presented in the visual field and the trained foveation
foveated target moves.
The second role of the foveation DNNs is to control head DNNs foveate the visual target. Then, a limb (arm or leg) isand torso movements to facilitate fixation and visual tracking. extended towards the ball. The retinal photoreceptors in theThis is accomplished simply by driving the cervicocephalic eyes are stimulated and the visual stimuli are presented as
and torso neuromuscular motor controllers (DNNs 13,19,14,20 the RGB components of the ONVs. Given its ONV input, thein Fig. 1i,j) with the averaged outputs of the foveation DNNs. desired output of a limb vision DNN is the 3D discrepancy ∆The dynamic head and torso movements are thus coupled to between the known 3D positions of the end effector and visualtarget. Repeatedly placing the ball at random positions in theeye movements induced by visual attention.
As Fig. 3b reveals, the input layer of a foveation DNN visual field and articulating the limb to reach for it in space, acomprises 43,200 units, to accommodate the dimensionality of large training dataset of 1M input-output pairs is synthesized
the ONV, the output layer has 2 units, ∆ = [∆θ, ∆φ]T , and offline. The backpropagation-trained DNN serves as an online
there are six hidden layers. Our humanoid trains its foveation limb vision controller.

6IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2020.

B. Motor Subsystem DNNs
1) Oculomotor DNNs (11–12): For the oculomotor DNN
controllers (Fig. 3d), the input layer consists of 8 units, 2 units
for the angular discrepancies ∆ = [∆θ, ∆φ]T to the visualtarget and n = 6 units for the activations a = [a1 , . . . , an ]T ,of the six extraocular muscles. The output layer consists of
6 units providing the muscle activation adjustments ∆a =[∆a1 , . . . , ∆an ]T . Each of the six hidden layers contains 300
units.
To train the DNNs, our biomechanical humanoid synthesizes
training data as follows: Specifying a target orientation for
the eye yields angular discrepancies between it and thecurrent eye orientation. With the angular discrepancies ∆and current extraocular muscle activations a as input, thebiomechanical eye model computes inverse dynamics withminimal effort optimization of the muscles. This determinesmuscle activation adjustments ∆a that incrementally reduce
the angular discrepancies and serve as the desired output of
the oculomotor DNN. Repeatedly specifying random target eyeorientations, a large training dataset of 1M input-output pairs
was synthesized offline. The backpropagation-trained DNN
serves as online oculomotor controllers.
2) Cervicocephalic Voluntary Motor DNN (13): Similarly,
the input layer of the cervicocephalic voluntary motor DNN(Fig. 3d) consists of 218 units, 2 units for the head pose target
discrepancy angles ∆ = [∆θ, ∆φ]T and n = 216 units forthe activations a, of the 216 muscles of the cervicocephalicmusculoskeletal complex. The output layer consists of 216
units providing the muscle activation adjustments ∆av . Eachof the six hidden layers contains 300 units.
To train the DNN, our biomechanical humanoid synthesizes
training data as follows: Specifying a target orientation forthe head yields angular discrepancies between it and thecurrent head orientation. With the angular discrepancies ∆and current neck muscle activations a as input, our humanoidcomputes inverse kinematics (IK), inverse dynamics (ID),and muscle optimization (MO).6 This determines muscle
activation adjustments ∆av that incrementally reduce theangular discrepancies and serve as the desired output of thecervicocephalic DNN. Repeatedly specifying random angulardiscrepancies, a training set of 1M input-output training pairs
was synthesized offline. The backpropagation-trained DNN
serves in the online cervicocephalic neuromuscular motor
controller.
3) Core Voluntary Motor DNN (14): The core neuromuscular controller voluntary motor DNN (Fig. 3d) is architected
identically to the cervicocephalic DNN, except for the size ofthe input and output layers. The input layer consists of 446
units that comprise 3 units for the T1 vertebra target angulardiscrepancies ∆ = [∆α, ∆β, ∆γ]T and n = 443 units forthe activations a of the muscles of the torso musculoskeletal
complex. The output layer consists of 443 units providing
6 More specifically, given ∆, IK yields the desired change of each of thevertebra joint angles using a quadratic programming solver, and the required
joint accelerations are computed to obtain the desired angular modification
within a specified time. Then, ID determines the desired joint torques toachieve the desired joint accelerations. Finally, the MO computation provides
muscle activations that can produce the desired joint torques.

the muscle activation adjustments ∆av . The training proceeds
similarly to that for the cervicocephalic voluntary motor DNN,
but see Section III-B6.
4) Limb Voluntary Motor DNNs (15–18): The input layerof the limb voluntary motor DNNs (Fig. 3e) consists of 3 units
that specify ∆ = [∆x, ∆y, ∆z]T , the estimated discrepancy
between the 3D positions of the end effector and a specified
target position, as well as the current activations a of then = 29 arm muscles or those of the n = 39 leg muscles. The
output layer consists of equal numbers of units that encode themuscle activation adjustments ∆av .
To train the DNNs, our biomechanical humanoid again synthesizes training data: Specifying a target position, determines
the discrepancy between it and the end effector (hand or foot).
Given the discrepancy ∆ and current muscle activations aas the input, the desired output of the network is the muscle
activation adjustments ∆av , which are again computed throughIK followed by ID and MO of the limb muscles. Repeatedly
specifying random target positions, a large training datasetof 1M input-output training pairs was synthesized offline.
The backpropagation-trained DNNs serve in the limb online
neuromuscular motor controllers.
5) Reflex Motor DNNs (19–24): Fig. 3f illustrates thearchitecture of the 6 reflex motor DNNs. They are identical,
except for the sizes of their input and output layers, which are
determined by the number n of muscles in the associated musculoskeletal complex. The input layer consists of 2n units, which
represent the change in muscle strains ∆e = [∆e1 , . . . , ∆en ]Tand strain rates ∆ė = [∆ė1 , . . . , ∆ėn ]T . Like the voluntary
motor DNNs, the networks have six hidden layers with 300
units each. The output layer consists of n units providing
muscle activation adjustments ∆ar , which then additively
modify the muscle activations.
Training data for the reflex motor DNNs are computed
offline simultaneously with the associated voluntary motor
DNN training data synthesis. The input discrepancies in muscle
strains ∆e and strain rates ∆ė, are computed after the IK phase,and the desired ∆ar output is computed as a proportionalderivative (PD) controller. Because this computation is on a
per-muscle basis, the ID and MO phases are unnecessary. The
backpropagation-trained reflex motor DNNs serve in the online
neuromuscular motor controllers.
6) Core Training: Going beyond the independent training of
the five decoupled extremities in our preliminary biomechanicalmusculoskeletal model [18], which lacked a functional torso,
the six musculoskeletal complexes (torso and extremities) must
be regarded a unified whole in order to properly train the coreneuromuscular motor controller of the torso musculoskeletal
complex. This is because the articulated biomechanical skeletal
structure remains connected when in motion and each of themusculoskeletal complexes of the extremities include multiple
significant muscles that attach to major bones in the torso.
Hence, during torso training data synthesis, random muscleactuated motions of the extremities are induced, including headturning, arm reaching, and leg squatting, such that forces fromthe extremities propagate to the torso. Furthermore, if the centerof pressure approaches the boundary of the support polygon
determined by the feet, the biomechanical model is reset to an

NAKADA et al.: DEEP LEARNING OF NEUROMUSCULAR AND VISUOMOTOR CONTROL OF A BIOMIMETIC SIMULATED HUMANOIDI(a) Golf putting(b) Balanced, ready stance7I(c) Springing left(d) Leaping rightFig. 7: Our autonomous humanoid engaged in sports activities. (a) A frame from a simulated golf putting scenario. Holding
the club in a “preacher” grip, the golfer steps forward and putts the ball into the hole. (b)–(d) A sequence of frames from asimulated soccer goaltending scenario. The goaltender observes the ball’s trajectory, reaching out with its arms and leaping
toward the ball as necessary in order to deflect it from the goal.

upright posture and data synthesis is restarted, thus training
the core controller to maintain a balanced bipedal stance.7

V. C ONCLUSIONS

We have presented a sophisticated simulation framework
for exploring anthropomimetic visuomotor control. Our frameIV. S IMULATION R ESULTS
work is unique in that it features an anatomically accurate,
Fig. 7 shows our autonomous humanoid with its visuomotor biomechanically simulated humanoid whose realistic skeleton
control system engaged in two sports activities—golf and is actuated by a full complement of contractile skeletalsoccer.8 These were chosen to confirm that our humanoid can muscles. Furthermore, our model includes a pair of human-likebe dynamically controlled by its properly trained visuomotor eyes, each with a foveated retina accommodating numerous
system, while maintaining a balanced bipedal stance and photoreceptors, as well as a functional cornea, iris sphincter,demonstrating voluntary movement throughout its entire body, and deformable lens, which can adapt to illumination andwhich was impossible for our earlier, constrained model [18]. achieve focus. Note that our model is a more complete version
Afforded binocular vision by virtue of its functional eyes of the one presented in [18], which lacked a functional torso
as well as motor control to support the mass of its body and (immobilized lumbar and thoracic spine and pelvis) and utilizedbalance in gravity, our virtual golfer, shown in Fig. 7a, steps a simplistic eye model (kinematic, non-biomechanical eye
into position holding the putter with a “preacher” grip and putts movements and pinhole aperture).
Our elaborate anthropomimetic visuomotor system comprises
the ball into the hole. As shown in Fig. 7b–d, balancing itsasizablecollection of deep neuromuscular motor controllers
body in an upright ready stance, our virtual soccer goaltenderandvisioncontrollers. We have successfully demonstrated
successfully observes a moving visual target, the incomingitsrobustperformancein task scenarios that simultaneously(massless) ball. Under neuromuscular oculomotor control, its
involveeye
movement
control for saccadic foveation andeyes persistently track incoming soccer balls by making rapid,
smoothpursuitofvisualtargetsin conjunction with appropriatesaccadic foveation and smooth pursuit eye movements, which in
dynamic
head
and
torso
motioncontrol, plus visually-guided
turn drive more sluggish (due to the greater masses involved)
dynamic
limb
control
producingnaturalarm and leg extension
cervicocephalic and torso movements under neuromuscularactionsthatenablethehumanoidtointercept
moving target
musculoskeletal motor control. The goaltender’s eyes track theobjectsandperformothersportsactions.
Ourmodel is also
moving ball while its head and body are in motion, which
capableofperforming
balanced
stepping
maneuvers,
but
demonstrates Vestibulo-Ocular Reflex (VOR). Thus, by virtue
continuous
bipedallocomotion
isbest
actuated
andcontrolledof all its trained neuromuscular motor controllers operating inconcert, our virtual goaltender controls its arms, and legs to by incorporating low-level Central Pattern Generator (CPG)
reach out and even leap at approaching balls to deflect them neural circuits to produce rhythmic muscle activations [29].
In particular, our humanoid’s biomimetic eyes and visuomofrom the goal.
torsystem make it possible to autonomously synthesize natural
Appendix A provides additional details regarding the control
head-eyebehavior for robots. This is important for two reasons.
policies for the foregoing demonstrations.
First, because our muscle-actuated biomechanical eye models
7 The entire biomechanical body must be simulated during offline training
produce appropriate saccades and smooth pursuit movements,
data synthesis for the core neuromuscular motor controller; thus, on an Intel they automatically gaze at and track moving objects of interest,
Xeon E5-1650 v4 3.60 MHz CPU, 8.7 s on average are required to compute each which provides humanoid robots active perception of the
input-output training pair for its voluntary motor DNN and an additional 0.3 s
for its reflex motor DNN. By comparison, the neuromuscular motor controllers environment and goal-directed visual information suitable forof the extremities are trained independently; thus, for the cervicocephalictaking appropriate actions. Second, this meets the expectations
controller, the respective times are 17 ms and 2 ms, and for all four limbof people who interact with a robot, as eye and head behaviors
controllers, they are 4 ms and 1 ms. (Note that for the oculomotor controller,
offline training data synthesis requires approximately 0.1 s per input-output pair are perhaps those most critical to human sensitivities. Amongon an Intel Core i7-6700 3.4 GHz CPU.) Online, the trained neuromuscular its many potential uses, our model promises to be valuable
motor controllers compute muscle activations rapidly—the core controller inin human visual attention research, a topic that we wish to
2.8 ms, the cervicocephalic controller in 2.2 ms, and the limb controllers in
explore in future work. For this, as well as for other types of
0.7 ms on the Xeon CPU.
8 We refer the reader to our demonstration video.
visual processing, such as binocular stereopsis, we will want

8IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2020.

to increase the number of photoreceptors, experiment withother nonuniform photoreceptor distributions, and automaticallyconstruct 2D retinotopic maps from the 1D ONV inputs.
Our visuomotor control system should, in principle, betransplantable into an anthropomorphic physical robotic system,
were a compatible one available. However, building one would
involve continuing research across multiple fields including
materials science to create more highly biomimetic muscle
actuators [7] and sensing technologies to create eye-like,foveated imaging devices [14], [15]. Although these areambitious goals, our work reaffirms that they have great
potential.
A PPENDIX AC ONTROL P OLICIESa) Golfing: We set “ideal” lower-body postures for threestates (standing, stepping, and putting). Transitional movements
between the states are generated naturally by the trained
neuromuscular motor controllers. The control objective forthe torso is to keep the body upright (with some naturalforward-leaning angles) and balanced, as the center of pressure
is monitored during the training of the core controller. The
objective for the arms is to grip the club in the initial putting
posture, and then swing back to a desired position. The
objective for the eyes is to foveate the golf ball, and thecervicocephalic controller induces the head to follow the eye
movement.
b) Soccer: The policies for the eye, cervicocephalic, andtorso neuromuscular motor controllers are the same as forgolfing. The objective for both arms is to reach a incomingball observed by the eyes. Initially, the knees are slightly bent
in the natural preparatory stance of a goaltender ready for fast
reaction. When the eyes detect a ball approaching on one side of
the body, the opposite-side leg neuromuscular motor controller
triggers a rapid leg extension. This, in conjunction with themomentum of arm extensions generates leaping movements
toward the incoming ball.
Learning higher-level control policies for performing optimal
sports movements, possibly through reinforcement learning, isa good topic for future work.
R EFERENCES[1] I. Mizuuchi, Y. Nakanishi, Y. Sodeyama, Y. Namiki, T. Nishino, N. Muramatsu, J. Urata, K. Hongo, T. Yoshikai, and M. Inaba, “An advanced
musculoskeletal humanoid Kojiro,” in Proc. IEEE-RAS International
Conference on Humanoid Robots, 2007, pp. 294–299.
[2] H. G. Marques, M. Jäntsch, S. Wittmeier, O. Holland, C. Alessandro,
A. Diamond, M. Lungarella, and R. Knight, “ECCE1: The first of a seriesof anthropomimetic musculoskeletal upper torsos,” in Proc. IEEE-RAS
International Conference on Humanoid Robots, 2010, pp. 391–396.
[3] K. Ogawa, K. Narioka, and K. Hosoda, “Development of whole-bodyhumanoid “Pneumat-BS” with pneumatic musculoskeletal system,” inProc. IEEE/RSJ International Conference on Intelligent Robots andSystems, 2011, pp. 4838–4843.
[4] R. Niiyama, S. Nishikawa, and Y. Kuniyoshi, “Biomechanical approachto open-loop bipedal running with a musculoskeletal athlete robot,”
Advanced Robotics, vol. 26, no. 3-4, pp. 383–398, 2012.
[5] Y. Nakanishi, S. Ohta, T. Shirai, Y. Asano, T. Kozuki, Y. Kakehashi,
H. Mizoguchi, T. Kurotobi, Y. Motegi, K. Sasabuchi, et al., “Design
approach of biologically-inspired musculoskeletal humanoids,” International Journal of Advanced Robotic Systems, vol. 10, no. 4, pp. 216:1–13,
2013.

[6] Y. Asano, K. Okada, and M. Inaba, “Design principles of a humanmimetic humanoid: Humanoid platform to study human intelligence andinternal body system,” Science Robotics, vol. 2, no. 13, pp. eaaq0899:1–
11, 2017.
[7] S. Kurumaya, K. Suzumori, H. Nabae, and S. Wakimoto, “Musculoskeletal lower-limb robot driven by multifilament muscles,” Robomech Journal,
vol. 3, no. 18, pp. 1–15, 2016.
[8] S. Kurumaya, H. Nabae, G. Endo, and K. Suzumori, “Design of thin
McKibben muscle and multifilament structure,” Sensors and Actuators
A: Physical, vol. 261, pp. 66–74, 2017.
[9] B. Espiau, F. Chaumette, and P. Rives, “A new approach to visual servoing
in robotics,” IEEE Transactions on Robotics and Automation, vol. 8,
no. 3, pp. 313–326, 1992.
[10] W. J. Wilson, C. W. Hulls, and G. S. Bell, “Relative end-effector control
using Cartesian position based visual servoing,” IEEE Transactions on
Robotics and Automation, vol. 12, no. 5, pp. 684–696, 1996.
[11] M. Jagersand, O. Fuentes, and R. Nelson, “Experimental evaluation of
uncalibrated visual servoing for precision manipulation,” in Proc. IEEE
International Conference on Robotics and Automation, vol. 4, 1997, pp.
2874–2880.
[12] P. Hebert, N. Hudson, J. Ma, T. Howard, T. Fuchs, M. Bajracharya, and
J. Burdick, “Combined shape, appearance and silhouette for simultaneous
manipulator and object tracking,” in Proc. IEEE International Conferenceon Robotics and Automation, 2012, pp. 2405–2412.
[13] S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end trainingof deep visuomotor policies,” Journal of Machine Learning Research,
vol. 17, no. 1, pp. 1334–1373, 2016.
[14] M. Bjorkman and D. Kragic, “Combination of foveal and peripheral vision
for object recognition and pose estimation,” in Proc. IEEE International
Conference on Robotics and Automation, vol. 5, 2004, pp. 5135–5140.
[15] D. Kragic, M. Björkman, H. I. Christensen, and J.-O. Eklundh, “Vision
for robotic object manipulation in domestic settings,” Robotics andAutonomous Systems, vol. 52, no. 1, pp. 85–100, 2005.
[16] M. Lesmana and D. Pai, “A biologically inspired controller for fast eye
movements,” in Proc. IEEE International Conference on Robotics andAutomation, 2011, pp. 3670–3675.
[17] M. Lesmana, A. Landgren, P.-E. Forssén, and D. Pai, “Active gazestabilization,” in Proc. Indian Conference on Computer Vision, Graphics
and Image Processing, 2014, pp. 81:1–8.
[18] M. Nakada, T. Zhou, H. Chen, T. Weiss, and D. Terzopoulos, “Deeplearning of biomimetic sensorimotor control for biomechanical humananimation,” ACM Transactions on Graphics, vol. 37, no. 4, pp. 56:1–14,
2018, (Proc. ACM SIGGRAPH 2018, Vancouver, Canada, August 2018).
[19] P. Faloutsos, M. Van De Panne, and D. Terzopoulos, “Autonomous
reactive control for simulated humanoids,” in Proc. IEEE International
Conference on Robotics and Automation, vol. 1, 2003, pp. 917–924.
[20] S. L. Delp, F. C. Anderson, A. S. Arnold, P. Loan, A. Habib, C. T. John,
E. Guendelman, and D. G. Thelen, “OpenSim: Open-source software tocreate and analyze dynamic simulations of movement,” IEEE Transactions
on Biomedical Engineering, vol. 54, no. 11, pp. 1940–1950, 2007.
[21] J. B. Langholz, G. Westman, and M. Karlsteen, “Musculoskeletal
modelling in sports-evaluation of different software tools with focus
on swimming,” Procedia Engineering, vol. 147, pp. 281–287, 2016.
[22] S.-H. Lee, E. Sifakis, and D. Terzopoulos, “Comprehensive biomechanicalmodeling and simulation of the upper body,” ACM Transactions on
Graphics, vol. 28, no. 4, pp. 99:1–17, Aug. 2009.
[23] M. Nakada, A. Lakshmipathy, H. Chen, N. Ling, T. Zhou, and
D. Terzopoulos, “Biomimetic eye modeling and deep neuromuscularoculomotor control,” ACM Transactions on Graphics, vol. 38, no. 6, pp.
221:1–14, 2019, (Proc. ACM SIGGRAPH Asia 19 Conference, Brisbane,
Australia, November 2019).
[24] P. Shirley and R. K. Morley, Realistic Ray Tracing, 2nd ed. Natick,
MA, USA: A. K. Peters, Ltd., 2003.
[25] E. L. Schwartz, “Spatial mapping in the primate sensory projection:
Analytic structure and relevance to perception,” Biological Cybernetics,
vol. 25, no. 4, pp. 181–194, 1977.
[26] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers:
Surpassing human-level performance on ImageNet classification,” inProc. ICCV, 2015, pp. 1026–1034.
[27] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
arXiv preprint arXiv:1412.6980, 2014.
[28] S.-H. Lee and D. Terzopoulos, “Heads up! Biomechanical modeling andneuromuscular control of the neck,” ACM Trans. Graphics, vol. 23, no.
212, pp. 1188–1198, 2006.
[29] W. Si, S.-H. Lee, E. Sifakis, and D. Terzopoulos, “Realistic biomechanicalsimulation and control of human swimming,” ACM Transactions on
Graphics, vol. 34, no. 1, pp. 10:1–15, Nov. 2014.