Locally-Connected, Irregular Deep Neural Networks
for Biomimetic Active Vision in a Simulated Human
Masaki Nakada*, Honglin Chen*, Arjun Lakshmipathy, and Demetri Terzopoulos
Computer Science Department, University of California, Los Angeles, California, USA

Abstract—An advanced simulation framework has recently
been introduced for exploring human perception and visuomotor control. In this context, we investigate locally-connected,
irregular deep neural networks (liNets) for biomimetic active
vision. Like commonly used CNNs, liNets are locally-connected,
forming receptive fields, but unlike CNNs, they are suitable
for spatially irregular photoreceptor distributions inspired bythose found in foveated biological retinas. Compared to fullyconnected deep neural networks, liNets accommodate a muchgreater number of retinal photoreceptors to enhance visual acuity
without intractable memory consumption. LiNets serve well in thebiomimetic active vision system embodied in a simulated humanthat learns active visuomotor control and active appearancebased recognition.

I. I NTRODUCTION
Visuomotor functionality in biological organisms refers to
the process of continually acquiring and interpreting visualsensory information necessary to produce appropriate motor
responses that achieve desired goals. In humans, the vision
process begins with binocular optical sensing mediated by
nonuniform retinal photoreceptor topology [1] feeding neuralmechanisms that control eye movements and achieve higherlevel visuomotor control. Biological vision has inspired computational approaches that mimic the functionality of these neuralmechanisms. Recent breakthroughs in machine learning withartificial neural networks have proven effective in computer
vision; however, the application of Deep Neural Networks(DNNs) to embodied biomimetic visuomotor systems hasreceived little attention in the field.
We have recently introduced a unique human simulation
framework well suited to exploring bio-inspired vision andvisuomotor control [2] [3]. Our framework is unique in that itfeatures a biomechanically simulated, human musculoskeletal
model, which includes numerous skeletal muscle actuators. The
virtual human perceives its environment with eyes, capable ofeye movements, whose foveated retinas contain photoreceptors
arranged in a nonuniform distribution like that of biologicalretinas. Its prototype visuomotor system includes two dozen
automatically-trained, fully-connected DNNs, about half ofwhich comprise its active vision subsystem while the other half
comprise its neuromuscular motor control subsystem. In thiscontext, our contributions in the present paper are as follows:• In conjunction with a more realistic eye model, including
cornea, iris, pupil, lens, and foveated retina (Section III),
we propose locally-connected irregular DNNs (liNets)
for active vision (Section IV). Unlike the common*Co-primary author

Convolutional Neural Networks (CNNs) that are limited
to conventional images and form rectangular receptive
fields on regular pixel arrays, our networks are suitable tospatially irregular photoreceptor distributions like thosefound in foveated biological retinas.
• Relative to the fully-connected DNNs that we used in [2],which proved manageable for retinas with a few thousand
cone-like photoreceptors, our novel liNets accommodate
a dramatically greater number of photoreceptors. This
substantially enhances visual acuity. We investigate thememory efficiency and learning performance of liNets(Section V) applied to active, online visuomotor control
as well as in a novel 3D active face recognition application(Section VI).
The significance of our work includes how the retinal samplingis achieved on the hemispherical fundus of an optically-accurate
model of the eye, as well as our deep neural network solution toirregular visual processing, integrated within a comprehensive
virtual human model, which can hence emulate human active
vision with unprecedented realism, detail, and efficiency.
II. T HE V IRTUAL H UMAN M ODELOur human model [3] includes a large number of the relevant
articular bones and muscles—193 bones connected by joints
comprising 163 articular degrees of freedom, plus a total of
823 muscle actuators. Each skeletal muscle is modeled as a
Hill-type uniaxial contractile actuator that applies forces to thebones at its points of insertion and attachment. The humanmodel is numerically simulated as a force-driven articulated
multi-body system. Each muscle actuator is activated by anindependent, time-varying, efferent activation signal, which isprovided by its motor control subsystem.
Fig. 1 illustrates the architecture of our virtual human’s visuomotor control system, including its improved, biomimetic eyemodels [4]. The system incorporates a set of 24 automaticallytrained DNNs. In the motor subsystem, 12 of these DNNs
comprise 6 recurrent neuromuscular motor controllers that
actuate the muscle groups of the cervicocephalic complex (219muscles), torso (443 muscles), and four limbs (29 muscles perleg, 39 muscles per arm), and 2 oculomotor DNNs control theextraocular muscles (6 muscles per eye). The remaining 10
DNNs implement the sensory subsystem, which is devoted tovisual perception and continuously operates on the retinal
photoreceptor outputs. A pair of foveation DNNs drive asynergy of eye, head, and torso movements, while 4 pairsof vision DNNs extract the sensory information necessary tocontrol the arms and legs.

(b)R

Vision Subsystem

(b)L

(a)

Right Retina

Left Retina
ONV

ONV

⋮⋮⋮ ⋮⋮⋮ ………
⋮⋮⋮ … ⋮⋮⋮

⋮
⋮

⋮

⋮

Δa1

Δa6

∆

(l)L DNNs 18,24

.

∆

∆

∆

.

(e)L Left Eye (section)

… ⋮⋮
⋮⋮ …

(m)

∆

∆

⋮⋮

⋮⋮

∆

.

⋮

∆

… ⋮⋮
⋮⋮ …
∆

⋮ … ⋮

∆

∆

⋮⋮

⋮⋮

∆

∆

Δp

⋮⋮

(e)L

∆

⋮⋮ …… ⋮⋮

(e)R

⋮⋮

∆

∆

∆

⋮⋮

∆

(l)R DNNs 17,23(e)R Right Eye (section)(k)L DNNs 16,22⋮⋮ …… ⋮⋮

⋮⋮

∆

⋮⋮

∆

∆

⋮⋮

.

∆

⋮⋮ …… ⋮⋮

∆

… ⋮⋮
⋮⋮ …

⋮

Δa6

⋮⋮

∆

∆

∆

a6

.

⋮⋮

⋮ … ⋮

(k)R DNNs 15,21

Δθ(j) DNNs 14,20(i) DNNs 13,19.

Δa1(h)L DNN 12

Average

a6

Δl

∆

∆

∆

⋮

⋮

Δθ

∆

Average(h)R DNN 11

⋮

⋮

⋮

∆∆ ∆∆∆∆∆∆(c)L SNN 2

⋮

⋮

⋮

Δl

∆

(d)L SNN 4

⋮

⋮
Δp

∆∆ ∆∆∆

∆∆

∆

∆

⋮ … ⋮

(d)R SNN 3⋮ … ⋮(c)R SNN 1(f)L DNN 2⋮(g)L DNNs 4,6,8,10⋮(g)R DNNs 3,5,7,9⋮⋮⋮ ⋮⋮⋮ ………
⋮⋮ ⋮ … ⋮⋮⋮(f)R DNN 1

Motor Subsystem
Fig. 1: Architecture of the visuomotor system. The neural controllers in the system include a total of 24 DNNs and 4 SNNs.
Vision Subsystem (top): To compute irradiance via ray-tracing, each retinal photoreceptor casts rays through the eye (Fig. 2a,b,c) and into thevirtual world (a). (b) The arrangement of the photoreceptors (black dots) on the right R and left L foveated retinas. Each eye outputs an RGB
Optic Nerve Vector (ONV). This feeds four trained visual accommodation SNNs (1–4); SNNs (c)R (1) and (d)R (3) control the muscles of
the iris and lens of the right eye (e)R, and SNNs (c)L (2) and (d)L (4) do the same for the left eye (e)L. The ONV also feeds ten trained
vision DNNs (1–10). (f) A pair of foveation DNNs (1,2) produce outputs that drive the movements of the eyes to foveate visual targets. (g)The eight limb vision DNNs (3–10) — (g)R (3,5,7,9) for the right eye and (g)R (4,6,8,10) for the left eye — output observed limb-to-target
discrepancy estimates.
Motor Subsystem (bottom): Fourteen trained neuromuscular motor DNNs (11–24) comprise the motor subsystem, including eight voluntary
motor DNNs (11–18) and six reflex motor DNNs (19–24). (h) The oculomotor DNNs (11,12), which are driven by the outputs of the foveation
DNNs, output muscle activation signals that control the six extraocular muscles of each eye to produce eye movements. Driven by theaveraged responses of the foveation DNNs, along with the current activations of the 216 neck muscles and 443 torso muscles, respectively, thecervicocephalic (i) voluntary motor DNN (13) and torso (j) voluntary motor DNN (14) each outputs muscle activation signals that contributeto actuating its associated neuromuscular complex. Driven by the bilaterally pairwise averaged responses of the limb vision DNNs, along withthe current activations of the 29 muscles of each arm or 39 muscles of each leg, respectively, each of the four limb voluntary motor DNNs (k)(l) (15–18) outputs muscle activation signals that contribute to actuating its associated neuromuscular complex. Each of the six reflex motor
DNNs (19–24) outputs muscle activation signals that contribute by stabilizing the muscle group of its associated musculoskeletal complex.

C. Ray-Tracing-Based Irradiance Computation(a) Left retina(b) Right retina(c)Fig. 2: (a),(b) Foveated retinas with 14,400 photoreceptors(black dots) in noisy log-polar distributions. (c) Rays cast fromthe positions of photoreceptors on the retina through the lensand pupil out into the 3D scene by the ray-tracing procedurethat computes the the irradiance on photoreceptors.

III. T HE B IOMIMETIC E YE M ODELThe virtual human’s eyes are modeled in accordance withhuman physiological data [5].1 We model the virtual eye asa sphere of radius 12mm that can be rotated with respect toits center around its vertical y axis by an angle θ and aroundits horizontal x axis by an angle φ. The eyes are in theirneutral positions looking straight ahead when θ = φ = 0◦ . The
horizontal and vertical fields of view are 167.5◦ . To achieve
dynamic eye control, the six extraocular and the intraocular(ciliary and sphincter) muscles are modeled as compound Hilltype contractile actuators [4].
A. Cornea, Iris, and Lens Submodels
The eye model includes a cornea, iris/pupil, lens, andretina. The cornea refracts incoming light rays. The adjustable
iris/pupil accommodates to the brightness of the scene bycontrolling the quantity of light that enters the eye. Actuated bya ciliary muscle, the deformable lens further refracts incoming
light rays so as to focus them on the hemispherical retinal
surface at the back of the eyeball. Two shallow neural networks(SNNs) control the iris and lens accommodation [4].

The irradiance at any point on the hemispherical retinal
surface at the back of the eye is computed using the conventional ray-tracing technique of computer graphics rendering[7]. Sample rays from the positions of photoreceptors on thehemispherical retinal surface are cast through the eye and out
into the 3D virtual world where they recursively intersect withthe visible surfaces of virtual objects and query the virtual
light sources in accordance with the Phong local illumination
model. The irradiance values returned by these rays determine
the light impinging upon the retina at the position of thephotoreceptors. For our realistic eye model (Fig. 2c), eachphotoreceptor gathers light from the 3D environment througha finite aperture proportional to the area of the pupil. The
irradiance computation by each photoreceptor requires theweighted sum of multiple cast rays refracted at the surfaces ofthe lens and cornea.
D. Optic Nerve Vector (ONV)The foveated retinal RGB “image” captured by the P
photoreceptors of each eye is output for further processing
down the visual pathway, not as a 2D array of pixels, but
as a 1D Optic Nerve Vector (ONV) of length 3P . The raw
irradiance information encoded in this vector feeds the lowlevel perceptual neural networks that directly control pupil size,
focal accommodation and eye movements, intermediate-levelnetworks that control cervicocephalic, torso, and limb motions,and higher-level neural networks that enable face recognition.
IV. L OCALLY-C ONNECTED I RREGULAR DNN S

The human retina has approximately 5 million cones [8].
Enhancing the visual acuity of the virtual retinas requires theincorporation of many more photoreceptors than the 3,600
that we employed in [2]. For instance, the ONVs are 3 ×
14,400 = 43,200-dimensional for the retinas shown in Fig. 2.
Implementing and training a fully-connected DNN, as we didin [2], with this many inputs is infeasible due to the exorbitant
memory needed to accommodate the network’s weights. To
B. Retina
overcome this impediment, we introduce locally-connected
Unlike the uniform-resolution, Cartesian grid structure of irregular DNNs, or liNets, that effectively mitigate memory
most artificial imaging sensors, visual sampling in primate consumption, thereby enabling the use of larger numbers of
retinas is well known to be strongly nonuniform. The den- irregularly distributed photoreceptors.
sity of cones decreases radially with eccentricity, from the
foveal center toward the periphery. A log-polar photoreceptor A. LiNet Architecture
distribution is often used as a model [6].
In a fully-connected neural network, each neuron in a
To emulate foveated perception, we use a noisy log-polar
hiddenlayer is connected globally to all the neurons in the
distribution to model the nonuniform placement of P photoreprevious
layer, whereas like CNNs [9], each neuronal unit in a
ceptors on the hemispherical retina. Fig. 2a,b illustrates thehiddenlayer
of our liNet is connected only locally to a fixedplacement of P = 14,400 photoreceptors. Due to the additive
numberofneighboring
units in the previous layer. However, theGaussian noise, the photoreceptors are placed in different
implementationofaCNN
conforms to conventional, regularlypositions on each retina.
sampled images, the regular arrangement of units in tensor
data structures, and the sharing of weights and biases across all
1 The transverse size of an average eye is 24.2 mm and its sagittal sizeis 23.7 mm. The average mass is 7.5 g. The approximate field of view of
the units in each hidden layer (i.e., the convolutional property).
an individual eye is 30 degrees to superior, 45 degrees to nasal, 70 degrees The liNet shares none of these restrictions. Importantly, every
to inferior, and 100 degrees to temporal. When the two eyes are combined,
unit of a liNet has its own particular weights and bias, justthe field of view becomes about 135 degrees vertically and 200 degrees
horizontally.
like in a fully-connected network.

⋮

⋮

⋮

⋮

⋮

⋮

⋮
…

⋮

⋮
⋮
⋮

…

⋮

8000
7000
6000
5000
4000
3000
2000

hi
hi-1
h2
h1

hH

10 8
Locally-connected Network
Fully-connected Network

3

2.5

2

1.5

1

0.5

1000
0
10 3

3P-D ONV

3.5
Locally-connected Network
Fully-connected Network

9000

Number of Trainable Parameters⋮

Memory Usage of Training Process/MB

10000

0
10 4

10 5

10 6

10 7

ONV Dimension

(a)

10 8

10 3

10 4

10 5

10 6

10 7

10 8

ONV Dimension(b)Fig. 4: Semi-log plots of (a) memory consumption and (b)number of trainable parameters versus ONV dimension.

Fig. 3: The liNet architecture.
⋮⋮
Referring to Fig. 3, suppose that we have a liNet with H⋮⋮⋮⋮
hidden layers, each of which is comprised of Nh neuronal units.
⋮
Using a Euclidean-distance k-nearest-neighbor algorithm, each
⋮
θ
⋮
⋮
⋮
⋮
⋮
unit in hidden layer h = 1, . . . , H forms a receptive field withϕRh nearest neighbor units in the previous hidden layer h − 1,and it inherits a position in the 2D (ρ, θ polar coordinate
FC
43,200-D ONV
system) retinal domain that is the average of the positions
1,350
2,700
of the units in its receptive field. The receptive fields of the
5,400
units in the first hidden layer h = 1, are formed from R1
10,80021,600neighboring retinal photoreceptors. Given an irregular, foveatedphotoreceptor distribution, the overlapping receptive fields, Fig. 5: Foveation DNN architecture. A liNet backbone iswhich are illustrated by the white circles in the retinal domain followed by a fully-connected layer that outputs eye orientation
at the left of Fig. 3, naturally increase in size with eccentricity, control signals θ and φ.
from the denser foveal center outward to the sparser periphery.
Our flexible liNet architecture is defined by a set of H
connectivity matrices. Each Nh × Rh matrix Ch specifies the memory consumption and number of trainable parameters of theconnectivity between units in hidden layer h and Rh units in networks with increasing ONV dimensionality. Fig. 4a shows
hidden layer h − 1. Each row in a connectivity matrix stores that the memory consumption of fully-connected networks
the indexes of the units in its receptive field. The index of a increases rapidly as the input dimension increases—8,031 MB
unit in hidden layer h is 0 ≤ n < Nh , while the (ONV) index when 3P = 4.0 × 104 , whereas for the liNet it is only 543 MB.
of a retinal photoreceptor is 0 ≤ p < P .
For a 3P = 1.28 × 107 dimensional ONV, the liNet’s memory
Our liNets are implemented in PyTorch. We use Rectified consumption is 8,075 MB, comparable to that of the fullyLinear Units (ReLUs). Each ReLU computes a weighted sum connected network with a 4.0 × 104 dimensional ONV. Fig. 4bof the outputs of units (or photoreceptors in the case of units shows that the rate of increase in the number of trainablein hidden layer h = 1) within its receptive field. The weights, parameters (i.e., the weights and biases) of the fully-connectedwhich are stored in a set of weight matrices Wh of size network is much faster than that of the liNet.
Nh × Rh , and the biases, which are stored in bias vectors
The above observations indicate that, unlike fully-connectedof size Nh , are learned from training data using standard networks, liNets make it feasible to perform perceptual probackpropagation learning techniques.
cessing on retinal models whose photoreceptor counts are on
the order of the millions of cone photoreceptors in the human
V. L I N ET E FFICIENCY AND P ERFORMANCE
retina [1].
Next, to evaluate the learning effectiveness of the liNet, we
To assess the memory efficiency of the liNet, given P photoreceptors (3P -dimensional ONV), we compare its memory consider the task of executing eye movements to foveate objects
consumption to that of a fully-connected DNN with the same of interest. For this purpose, we employ retinas with P =14,400 photoreceptors (43,200-dimensional ONVs). Foveation
numbers of hidden layers and neurons in each hidden layer.
For example, consider setting the number of hidden layers functionality is accomplished by the foveation DNNs (denoted
to H = 4, and the number of units Nh in hidden layer h to the DNN 1 and DNN 2 in Fig. 1), each of which inputs the ONV
number Nh−1 in the previous layer divided by a constant factor from the eye and outputs activation signals to its 6 extraocularf = 5; i.e., Nh = bNh−1 /f c, such that the only difference muscles to produce eye movements.
is in the connectivity of the networks. In the liNet, every
We construct the foveation DNNs as illustrated in Fig. 5,
neuron in a hidden layer is locally-connected to only Rh = 5 consisting of a liNet backbone with H = 4 hidden layers, anearest neighboring units in the previous layer. Fig. 4 plots the receptive field size of Rh = 20, and f = 5, followed by a

I

I

(a)



(b)

(c)

Fig. 8: Frames from a simulation of the biomechanically-simulated virtual human sitting on a stool (with immobilized pelvis
and torso) and actively executing left-arm reaching motions to intercept a ball shot by the cannon. The ball is actively perceived
by the eyes (red lines indicate the gaze directions), processed by the liNet vision DNNs, foveated and tracked through eye
movements in conjunction with muscle-actuated head movements controlled by the neck-head motor DNN, and the reactive
reaching motion is muscle-actuated and controlled by the left arm motor DNN.
1
locally-connected network training loss
locally-connected network validation loss
fully-connected network training loss
fully-connected network validation loss

0.9

Mean Squared Error

0.8
0.025

0.7

0.02

0.6
0.5

0.015

0.4
0.01

0.3
0.005

0.2
0

0.1
0
0

10

20

30

40

50

60

70

80

90

100

Epoch

Fig. 6: Progress of the foveation DNN training process for thevision liNet (blue) and fully-connected foveation DNN (red)on the training (solid) and validation (dashed) datasets.

after only 9 epochs with a validation loss smaller than that of
the fully-connected network.
With their trained foveation DNNs, the eyes perform naturalvergence movements to converge onto visual targets. This is
illustrated in Fig. 7 for a white ball in motion that enters theeye’s field of view from the lower right, stimulating several
peripheral photoreceptors at the upper left of the retina. Theeye very rapidly foveates the visual target. Fine adjustments
comparable to microsaccades can be observed during fixation.
The above results confirm that our liNets are capable oflearning effective foveation with substantially smaller memory
and time requirements than the fully-connected networks we
employed in [2].
VI. A PPLICATIONS
A. Visuomotor Control

I(a) t0I(b) t1I(c) t2(d) t3Fig. 7: Time sequence (a)–(d) of photoreceptor responses in
the left retina during a saccadic eye movement that foveates
and tracks a moving white ball. At time t0 the ball becomesvisible in the periphery, at t1 the eye movement is bringing theball towards the fovea, and the moving ball is being fixated in
the fovea at times t2 and t3 .

fully-connected (FC) output layer that produces eye rotation
angles θ and φ.
Using a dataset of 20K training ONVs and a batch size of64, we trained a benchmark fully-connected foveation DNN
as we did in [2], as well as a liNet-based foveation DNN, tofoveate a white ball by stimulating the extraocular muscles toactuate saccadic eye movements. Fig. 6 plots the progress andvalidation losses of the backpropagation training processes. The
fully connected network converges to a small mean-squared
training loss below 0.01 after 47 epochs. The liNet does so

Fig. 8 presents a sequence of frames from a simulation
demonstrating the visuomotor system using the trained liNet
foveation DNNs and other vision DNNs (DNNs 3–10 in Fig. 1).
A cannon shoots a ball towards the virtual human, which
actively perceives the ball on its foveated retinas. The ONV
outputs of its eyes are continually processed by the liNet vision
DNNs to perform foveation and visual tracking of the incomingball. The motor DNNs control the extension of the arms andlegs to intercept and deflect the approaching ball. Thus, given
just the high level objective of deflecting the incoming ball,the virtual human successfully controls itself online to perform
this nontrivial dynamic visuomotor task.
While similar to our demonstration in [2], which employed
fully-connected vision DNNs to process retinas with a mere
3,600 photoreceptors, our novel liNets have enabled us to
implement retinas with 14,400 photoreceptors yet effectively
train the vision DNNs to perform foveation and tracking of thevisual target with eye movements and cervicocephalic control.
B. Active Face Recognition
As an example of higher-level visual processing with liNets,
we explore the task of 3D Active Face Recognition (AFR).
Fig. 9 shows the architecture of our AFR DNN. The network
includes a liNet backbone that has H = 5 hidden layers of

⋮
⋮

⋮
⋮

⋮
⋮

⋮

⋮

⋮

"

⋮
FC1

⋮
⋮

⋮

#
$%

FC2

⋮
&'

43,200-D ONV

1,350
2,700
5,400
10,80021,600Fig. 9: LiNet architecture for active face recognition. The
network outputs recognized person identity labels pi andhorizontal and vertical pose angle estimates α and β.

Fig. 12: Training and validation loss curves for 3D active facerecognition. Classification loss (blue) is the cross-entropy loss
Lclass for the classification subnet. Regression loss (red) is themean-squared loss Langle for the regression subnet.

dataset as follows: 70% training, 10% validation, 20% testing.

Fig. 10: The virtual human observes a 3D face model.

2) Training and Validation: To train our network, we use
the Adaptive Moment Estimation (Adam) stochastic optimizer
with a learning rate η = 0.01, step size α = 10−3 , andforgetting factors β1 = 0.9 for gradients and β2 = 0.999 forsecond moments of gradients. Overfitting is avoided by using
a validation split of 0.1 along with an early stopping condition.
We standardized the α and β outputs of the regression networkby computing the means and standard deviations over the entire
training dataset.
The hyper-parameter in the aforementioned loss function L
is set to λ = 6.0. The model parameters are recorded only
when the model simultaneously attains best regression loss andclassification accuracy on the validation set after each epoch.
Fig. 12 shows the training and validation loss curves. The
training process converged to a small error after 57 epochs,
which triggered the early stopping condition (no improvementfor 30 successive epochs).

ReLUs. Every hidden unit forms receptive fields with R = 20
nearest neighbors in the previous layer. The retina has P =14,400 photoreceptors (a 3P = 43,000 dimensional ONV),
and the number of units in each successive hidden layer isreduced by a factor of f = 2. The liNet feeds two single-layer,
fully-connected subnets—FC1 is a classification subnet, whose
output pi , for i = 1, . . . , N , has dimensionality equal to thenumber N of known person identities; FC2 is a regression
net that generates the estimated pose (horizontal and vertical3) Accuracy: The testing classification accuracy for therotation angles, α and β) of the observed face.
model was 97.97%, and the testing mean squared errors for
The AFR DNN is trained by minimizing the combined loss the α and β angles were 5.52 × 10−3 . This corresponds to
L = Lclass + λLangle , where Lclass is cross-entropy person an average error of 0.7◦ (after adjusting for the effect ofclassification loss, Langle is mean-squared pose estimation standardizing the output angles).2
regression loss, and parameter λ trades off between them.
Note that there exist no other datasets against which to1) Training Data Synthesis: To train our AFR model, we
benchmark the performance of the 3D active face recognition
use the Freiburg 3D face dataset, which comprises scansability of our virtual human. Futhermore, the biomimeticof 75 subjects, recorded using a CyberwareTM 3030PS laser
retina+liNet combination in its active vision system is notscanner as part of the University of Freiburg 3D morphable
meaningfully comparable against conventional CNNs trained on
faces database [10]. The virtual human (monocularly) observes
regular, passively-acquired, uniform-resolution, facial images.
the 3D faces of 50 different scanned subjects (Fig. 10).
Fig. 11 illustrates how we synthesize the training data. Raytracing the textured 3D geometry, the observer’s eye captures
2 Let a denote the target output and a the prediction of the network, etpmse
ONV “images” as the subject face is rotated by 2 degree denote themean squared error of the angle, edeg denote the corresponding◦◦
increments from −30 to 30 horizontally and vertically. Thus, error in degrees, and let µ and β denote the meanh and standard deviation
i2 of
we synthesize 50×30×30 = 45, 000 ONVs with corresponding the target angles in the dataset. We have emse = ( atσ−µ ) − ( apσ−µ ) and√face orientation and person ID labels. We partition the ONV we obtain edeg =| at − ap |= β emse . The standard deviation β is 8.96.

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 11: The virtual human foveating a 3D face in several different poses. Irradiance at the retinal photoreceptors is computedby ray-tracing; the white lines indicate rays cast from the positions of the photoreceptors, through the lens, pupil, and cornea,
and out into the 3D scene. Underneath, the photoreceptor responses comprising the ONV are visualized as retinal “images”.

VII. R ELATED W ORK
A number of researchers have proposed (locally-connected)
neural network architectures whose neuronal units have receptive fields; the best known are Fukushima’s “Neocognitron”[11] and the currently popular deep CNNs [9]. CNNs arecharacterized by the regular structure of their receptive fields,
enabling the sharing of weights across all units in a hidden
layer (i.e., the convolutional property). Closer to our liNets isthe non-convolutional LCNP [12]. However, unlike the liNet,
the aforecited networks are intended for use in conventional,
regularly-sampled pixel images common to computer vision,
which fundamentally differ from biological retinas.
Our biomimetic, foveated retina model employs a noisy logpolar photoreceptor distribution. Other placement patterns arereadily created, including more elaborate biomimetic procedural
models [8] or photoreceptor distributions empirically measured
from biological eyes [1], [13], all of which depart fundamentallyfrom the uniform-resolution, Cartesian images common tocomputer vision/graphics. However, foveated sensors have also
been of interest in computer vision and image processing [14].
Recently, Ozimek et al. [15] explored foveated sampling
and log-polar maps [6] to compute regular Cartesian corticalrepresentations of images to which deep CNNs may be appliedwith significant efficiency gains.
In the context of human active vision and visuomotor
control [16], Terzopoulos and Rabie deployed their “animat
vision” framework within a kinematic virtual human capableof bipedal locomotion, demonstrating active, vision-guided
tracking and pursuit [17], and a similar kinematic virtualhuman model, dubbed “Walter”, was employed by Sprague etal. [18] to study visuomotor control in the context of sidewalk
navigation tasks. The virtual humans demonstrated in [17] were
equipped with crudely-foveated eyes, implemented as coaxial
virtual cameras rendering composite polygon-shaded square
images through the GPU pipeline, quite unlike our biomimetic
ocular model that samples light in the 3D virtual environment
using ray-tracing so as to realistically emulate how irregularly
distributed photoreceptors respond to light irradiating the retina.
VIII. C ONCLUSIONS
Within a unique simulation framework for investigating
biomimetic human vision and visuomotor control, we have introduced and evaluated locally-connected, irregular deep neuralnetworks, or liNets. Unlike CNNs, our novel liNets are well
suited to a bio-inspired retinal model with irregularly distributed
photoreceptors, enabling greater numbers of photoreceptors
resulting in enhanced visual acuity.
We empirically demonstrated the utility, efficiency, andperformance of trained liNets in foveation, visuomotor control,and in a prototype 3D active facial recognition subsystem
incorporated into the brain of a simulated biomechanical humanmusculoskeletal model with biomimetic eyes. Our approachhas been to train the DNNs with large quantities of training
data that are synthesized by the virtual human itself.
Inevitably, our current retinal model is a gross simplificationof the biological human retina. Modeling the retinal ganglion
cells, as well as circular excitatory-inhibitory center-surroundand oriented receptive fields, the M-cell pathway, and theretinocortical map [6] are worthwhile avenues for future
research with our framework. We believe that our embodied,biomimetic modeling approach is suitable for such extensions.
Our foveation liNet generates saccadic eye movements to
foveate interesting objects in a variety of different scenarios;
hence, our model can be valuable in human visual attention
research [16], a topic that we wish to explore in future work.
R EFERENCES[1] C. A. Curcio, K. R. Sloan, R. E. Kalina, and A. E. Hendrickson, “Humanphotoreceptor topography,” Journal of Comparative Neurology, vol. 292,
no. 4, pp. 497–523, 1990. 1, 4, 8
[2] M. Nakada, H. Chen, and D. Terzopoulos, “Learning biomimetic
perception for human sensorimotor control,” in Proc. Second IEEE
CVPR Workshop on Mutual Benefits of Cognitive and Computer Vision(MBCC 2018), Salt Lake City, UT, June 2018, pp. 2030–2035. 1, 3, 5
[3] M. Nakada, T. Zhou, H. Chen, T. Weiss, and D. Terzopoulos, “Deeplearning of biomimetic sensorimotor control for biomechanical humananimation,” ACM Transactions on Graphics (TOG), vol. 37, no. 4, pp.
56:1–15, 2018, proc. ACM SIGGRAPH 2018, August 2018. 1[4] M. Nakada, A. Lakshmipathy, H. Chen, N. Ling, T. Zhou, and D. Terzopoulos, “Biomimetic eye modeling & deep neuromuscular oculomotor
control,” ACM Transactions on Graphics (TOG), vol. 38, no. 6, pp. 1–14,
2019, proc. ACM SIGGRAPH 2019, Brisbane, Australia, Nov 2019. 1, 3
[5] P. Riordan-Eva and E. Cunningham, Vaughan & Asbury’s General
Ophthalmology. McGraw Hill Professional, 2011. 3[6] E. L. Schwartz, “Spatial mapping in the primate sensory projection:
Analytic structure and relevance to perception,” Biological Cybernetics,
vol. 25, no. 4, pp. 181–194, 1977. 3, 8
[7] P. Shirley and R. K. Morley, Realistic Ray Tracing, 2nd ed. Natick,
MA, USA: A. K. Peters, Ltd., 2003. 3[8] M. F. Deering, “A photon accurate model of the human eye,” ACM Trans.
on Graphics (TOG), vol. 24, no. 3, pp. 649–658, 2005. 3, 8
[9] Y. LeCun, Y. Bengio et al., “Convolutional networks for images, speech,
and time series,” The Handbook of Brain Theory and Neural Networks,
vol. 3361, no. 10, p. 1995, 1995. 3, 8
[10] V. Blanz and T. A. Vetter, “Morphable model for the synthesis of 3D
faces,” in Proc. ACM SIGGRAPH 99 Conf., 1999, pp. 187–194. 6[11] K. Fukushima, “Neocognitron: A hierarchical neural network capable ofvisual pattern recognition,” Neural Networks, vol. 1, no. 2, pp. 119–130,
1988. 8[12] R. Uetz and S. Behnke, “Large-scale object recognition with cudaaccelerated hierarchical neural networks,” in IEEE International Conference on Intelligent Computing and Intelligent Systems, vol. 1. IEEE,
2009, pp. 536–541. 8[13] L. J. Grady, “Space-variant computer vision: A graph-theoretic approach,”
Ph.D. dissertation, Boston University, 2004. 8[14] M. Yeasin and R. Sharma, “Foveated vision sensor and image processing:A review,” in Machine Learning and Robot Perception. Springer, 2005,
pp. 57–98. 8[15] P. Ozimek, N. Hristozova, L. Balog, and J. P. Siebert, “A space-variant
visual pathway model for data efficient deep learning,” Frontiers in
Cellular Neuroscience, vol. 13, 2019. 8[16] J. M. Findlay, I. D. Gilchrist et al., Active vision: The psychology oflooking and seeing. Oxford University Press, 2003, no. 37. 8[17] T. F. Rabie and D. Terzopoulos, “Active perception in virtual humans,”in Proc. Vision Interface 2000, Montreal, Canada, 2000, pp. 16–22. 8[18] N. Sprague, D. Ballard, and A. Robinson, “Modeling embodied visual
behaviors,” ACM Transactions on Applied Perception (TAP), vol. 4, no. 2,
p. 11, 2007. 8