Biol. Cybern. 87, 116–140 (2002)
DOI 10.1007/s00422-002-0318-1
 Springer-Verlag 2002

Schema design and implementation of the grasp-related
mirror neuron system
Erhan Oztop, Michael A. Arbib
USC Brain Project, University of Southern California, Los Angeles, CA 90089-2520, USA
Received: 6 August 2001 / Accepted in revised form: 5 February 2002

Abstract. Mirror neurons within a monkey’s premotor
area F5 ﬁre not only when the monkey performs acertain class of actions but also when the monkey
observes another monkey (or the experimenter) performa similar action. It has thus been argued that theseneurons are crucial for understanding of actions byothers. We oﬀer the hand-state hypothesis as a new
explanation of the evolution of this capability: the basic
functionality of the F5 mirror system is to elaborate theappropriate feedback – what we call the hand state – foropposition-space based control of manual grasping ofan object. Given this functionality, the social role of theF5 mirror system in understanding the actions of others
may be seen as an exaptation gained by generalizing
from one’s own hand to an other’s hand. In other words,
mirror neurons ﬁrst evolved to augment the ‘‘canonical’’F5 neurons (active during self-movement based on
observation of an object) by providing visual feedbackon ‘‘hand state,’’ relating the shape of the hand to theshape of the object. We then introduce the MNS1
(mirror neuron system 1) model of F5 and related brain
regions. The existing Fagg–Arbib–Rizzolatti–Sakata
model represents circuitry for visually guided graspingof objects, linking the anterior intraparietal area (AIP)
with F5 canonical neurons. The MNS1 model extends
the AIP visual pathway by also modeling pathways,
directed toward F5 mirror neurons, which match arm–
hand trajectories to the aﬀordances and location of apotential target object. We present the basic schemas for
the MNS1 model, then aggregate them into three ‘‘grandschemas’’ – visual analysis of hand state, reach andgrasp, and the core mirror circuit – for each of which we
present a useful implementation (a non-neural visual
processing system, a multijoint 3-D kinematics simulator, and a learning neural network, respectively). With
this implementation we show how the mirror system
may learn to recognize actions already in the repertoireof the F5 canonical neurons. We show that the connectivity pattern of mirror neuron circuitry can be estab-

Correspondence to: M.A. Arbib (e-mail: arbib@pollux.usc.edu)

lished through training, and that the resultant network
can exhibit a range of novel, physiologically interesting
behaviors during the process of action recognition. We
train the system on the basis of ﬁnal grasp but thenobserve the whole time course of mirror neuron activity,
yielding predictions for neurophysiological experiments
under conditions of spatial perturbation, altered kinematics, and ambiguous grasp execution which highlightthe importance of the timing of mirror neuron activity.

1 Introduction
1.1 The mirror neuron system for grasping
The macaque inferior premotor cortex has been identiﬁed as being involved in reaching and grasping movements (Rizzolatti et al. 1988). This region has beenfurther partitioned into two subregions: F5, the rostral
region, located along the arcuate part; and F4, thecaudal part (see Fig. 1). The neurons in F4 appear to be
primarily involved in the control of proximal movements (Gentilucci et al. 1988), whereas the neurons of F5
are involved in distal control (Rizzolatti et al. 1988).
Rizzolatti et al. (1996a) and Gallese et al. (1996)
discovered a subset of F5 hand cells, which they called
mirror neurons. Like other F5 neurons, mirror neurons
are active when the monkey performs a particular classof actions, such as grasping, manipulating, and placing.
However, in addition the mirror neurons become active
when the monkey observes the experimenter or anothermonkey performing an action. The term F5 canonicalneurons is used to distinguish the F5 hand cells which do
not posses the mirror property but are instead responsive
to visual input concerning a suitably graspable object.
The canonical neurons are indistinguishable from themirror neurons with respect to their ﬁring during selfaction. However they are diﬀerent in their visual properties – they respond to object presentation and notaction observation per se (Murata et al. 1997).

117

Fig. 1. Lateral view of the monkey cerebral
cortex (intraparietal sulcus, IPS; superior
temporal sulcus, STS; and lunate sulcus
opened). The visuomotor stream for handaction is indicated by arrows (adapted from
Sakata et al. 1997)

Most mirror neurons exhibit a clear relation between
the observed and executed actions for which they are
active. The congruence between the observed and executed action varies. For some of the mirror neurons,
the congruence is quite loose; for others, not only must
the general action (e.g., grasping) match, but also theway the action is executed (e.g., power grasp) mustmatch as well. To be triggered, the mirror neurons
require an interaction between the hand motion andthe object. The vision of the hand motion or the object
alone does not trigger mirror activity (Gallese et al.
1996).
It has thus been argued that the importance of mirror
neurons is that they provide a neural representation thatis common to execution and observation of grasping
actions and thus that these neurons are crucial to thesocial interactions of monkeys, providing the basis forunderstanding of actions by others through their linkageof action and perception (Rizzolatti and Fadiga 1998).
Below we oﬀer the hand-state hypothesis, suggestingthat this important role is an exaptation of a moreprimitive role, namely that of providing feedback forvisually guided grasping movements. By exaptation wemean the exploitation of an adaptation of a system toserve a diﬀerent purpose (in this case for social understanding) than it initially developed for (in this case,visual control of grasping). We will then develop the
MNS1 (mirror neuron system 1) model and show thatthe system can exploit its ability to relate self-handmovements to objects to recognize the manual actions
being performed by others, thus yielding the mirror
property. We also conduct a number of simulation experiments with the model and show that these yieldnovel predictions, suggesting new neurophysiologicalexperiments to further probe the monkey mirror system.
However, before introducing the hand-state hypothesis
and the MNS1 model, we ﬁrst outline the FARS (Fagg–
Arbib–Rizzolatti–Sakata) model of the circuitry that
includes the F5 canonical neurons and provides theconceptual basis for the MNS1 model.

1.2 The FARS model of parietal–premotor
interactions in grasping
Studies of the anterior intraparietal sulcus (AIP; Fig. 1)
revealed cells that were activated by the sight of objects
for manipulation (Taira et al. 1990; Sakata et al. 1995).
In addition, this region has very signiﬁcant recurrent
corticocortical projections with area F5 (Matelli et al.
1994; Sakata et al. 1997). In their computational model
for primate control of grasping (the FARS model),Fagg and Arbib (1998) analyzed these ﬁndings of
Sakata and Rizzolatti to show how F5 and AIP may
act as part of a visuomotor transformation circuit,which carries the brain from sight of an object to the
execution of a particular grasp. In developing the FARS
model, Fagg and Arbib (1998) interpreted the ﬁndingsof Sakata (on AIP) and Rizzolatti (on F5) as showingthat AIP represents the grasps aﬀorded by the object
while F5 selects and drives the execution of the grasp.
The term aﬀordance (adapted from Gibson 1966) refers
to parameters for motor interaction that are signaled
by sensory cues without invocation of high-level object
recognition processes. The model also suggests how F5
may use task information and other constraints
encoded in prefrontal cortex (PFC) to resolve theaction opportunities provided by multiple aﬀordances.
Here we emphasize the essential components of themodel (Fig. 2) that will ground the version of the
MNS1 model presented below. We focus on the linkage
between viewing an aﬀordance of an object and thegeneration of a single grasp:
1. The dorsal visual stream (parietal cortex) extracts
parametric information about the object being attended. It does not ‘‘know’’ what the object is; it can
only see the object as a set of possible aﬀordances.
The ventral stream (from primary visual cortex toinferotemporal cortex, IT), by contrast, recognize
what the object is and passes this information to PFC
which can then, on the basis of the current goals of
the organism and the recognition of the nature of the118

this system evolved in the ﬁrst place to provide feedbackfor visually directed grasping, with the social role of themirror system being an exaptation as the hand-statemechanisms become applied to the hands of others aswell as to the hand of the animal itself.
2.1 Virtual ﬁngers

Fig. 2. The anterior intraparietal area, (AIP) extracts the aﬀordancesand F5 selects the appropriate grasp from the AIP ‘‘menu.’’ Various
biases are sent to F5 by the prefrontal cortex (PFC) which relies on
the recognition of the object by the inferotemporal cortex (IT). The
dorsal stream through AIP to F5 is replicated in the current version of
the mirror neuron system 1 (MNS1) model; the inﬂuence of IT and
PFC on F5 is not analyzed further in the present paper

object, bias F5 to choose the aﬀordance appropriate
to the task at hand.
2. AIP is hypothesized as playing a dual role in theseeing/reaching/grasping process, not only computing aﬀordances exhibited by the object but also, as
one of these aﬀordances is selected and execution of
the grasp begins, serving as an active memory of theone selected aﬀordance and updating this memory tocorrespond to the grasp that is actually executed.
3. F5 is hypothesized as ﬁrst being responsible for integrating task constraints with the set of grasps that are
aﬀorded by the attended object in order to select asingle grasp. After selection of a single grasp, F5
unfolds this represented grasp in time to govern therole of primary motor cortex (F1) in its execution.
4. In addition, the FARS model represents the way inwhich F5 may accept signals from areas F6 (pre-SMA
supplementary motor area), 46 (dorsolateral prefrontal cortex), and F2 (dorsal premotor cortex) to
respond to task constraints, working memory, andinstruction stimuli, respectively, and how these inturn may be inﬂuenced by object recognition processes in IT (see Fagg and Arbib 1988 for more details), but these aspects of the FARS model are not
involved in the current version of the MNS1 model.

2 The hand-state hypothesis
The key notion of the MNS1 model is that the brain
augments the mechanisms modeled by the FARS
model – for recognizing the grasping-aﬀordances of an
object (AIP) and transforming these into a program ofaction – by mechanisms that can recognize an action interms of the hand state which makes explicit the relation
between the unfolding trajectory of a hand and theaﬀordances of an object. Our radical departure from allprior studies of the mirror system is to hypothesize that

As background for the hand-state hypothesis, we ﬁrst
present a conceptual analysis of grasping. Iberall andArbib (1990) introduced the theory of virtual ﬁngers andopposition space. The term virtual ﬁnger is used todescribe the physical entity (e.g., one or more ﬁngers,
and the palm of the hand) that is used in applying force,and thus includes speciﬁcation of the region to bebrought in contact with the object (what we might callthe ‘‘virtual ﬁngertip’’). Figure 3 shows three types of
opposition: those for the precision grip, power grasp,
and side opposition. Each of the grasp types is deﬁned
by specifying two virtual ﬁngers, VF1 and VF2, and theregions on VF1 and VF2 which are to be brought into
contact with the object to grasp it. Note that the ‘‘virtual
ﬁngertip’’ for VF1 in palm opposition is the surface ofthe palm, while that for VF2 in side opposition is theside of the index ﬁnger. The grasp deﬁnes two ‘‘opposition axes’’: the opposition axis in the hand joining thevirtual ﬁnger regions to be opposed to each other, andthe opposition axis in the object joining the regions wherethe virtual ﬁngers contact the object. Visual perception
provides aﬀordances (diﬀerent ways to grasp the object);once an aﬀordance is selected, an appropriate opposition axis in the object can be determined. The task ofmotor control is to preshape the hand to form an
opposition axis appropriate to the chosen aﬀordance,
and to move the arm so as to transport the hand to bring
the hand and object axes into alignment. During the laststage of transport, the virtual ﬁngers move down the
opposition axis (the ‘‘enclose’’ phase) to grasp the object
just as the hand reaches the appropriate position.
2.2 The hand-state hypothesis
We assert as a general principle of motor control that if
a motor plant is used for a task, then a feedback system
will evolve to better control its performance in the faceof perturbations. We thus ask, as a sequel to the work of
Iberall and Arbib (1990), what information would beneeded by a feedback controller to control grasping in
the manner described in Sect. 2.1. Modeling of thisfeedback control is beyond the scope of this paper.
Rather, our aim is to show how the availability of such
feedback signals in the primate cortex for self-action for
manual grasping can provide the action recognition
capabilities which characterize the mirror system. Speciﬁcally, we oﬀer the following hypothesis.
The hand-state hypothesis. The basic functionality of theF5 mirror system is to elaborate the appropriate
feedback – what we call the hand state – for oppositionspace-based control of manual grasping of an object.

119Fig. 3a–c. Each of the three grasp types here is deﬁned by specifying
two ‘‘virtual ﬁngers’’, VF1 and VF2, which are groups of ﬁngers or apart of the hand such as the palm which are brought to bear on eitherside of an object to grasp it: a pad oppostion; b palm opposition; c side
opposition. The speciﬁcation of the virtual ﬁngers includes speciﬁcation of the region on each virtual ﬁnger to be brought in contact withthe object. A successful grasp involves the alignment of two‘‘opposition axes’’: the opposition axis in the hand joining the virtual
ﬁnger regions to be opposed to each other, and the opposition axis in
the object joining the regions where the virtual ﬁngers contact theobject (Iberall and Arbib 1990)

Given this functionality, the social role of the F5 mirror
system in understanding the actions of others may beseen as an exaptation gained by generalizing from one’s
own hand to another’s hand.
The key to the MNS1 model, then, is the notion ofhand state as encompassing data required to determine
whether the motion and preshape of a moving hand may
be extrapolated to culminate in a grasp appropriate toone of the aﬀordances of the observed object. Basically amirror neuron must ﬁre if the preshaping of the handconforms to the grasp type with which the neuron isassociated; and the extrapolation of hand state yields atime at which the hand is grasping the object along an
axis for which that aﬀordance is appropriate.
Our current representation of hand state deﬁnes aseven-dimensional trajectory

speciﬁc type of grasp, but they all must be available tomonitor a wide range of possible grasps. We have chosena set of variables of clear utility in monitoring the successful progress of grasping an object, but do not claimthat these and only these variables are represented in thebrain. Indeed, the brain’s actual representation will be a
distributed neural code, which we predict will correlatewith such variables, but will not be decomposable into acoordinate-by-coordinate encoding. However, we believe that the explicit deﬁnition of hand state oﬀered here
will provide a ﬁrm foundation for the design of new
experiments in kinesiology and neurophysiology.
The crucial point is that the availability of the handstate to provide feedback for visually directed grasping
makes action recognition possible. Notice that we havecarefully deﬁned the hand state in terms of relationships
between hand and object (though the form of the deﬁnition must be subject to future research). This has the
beneﬁt that it will work just as well for measuring how
the monkey’s own hand is moving to grasp an object as
for observing how well another monkey’s hand is moving to grasp the object. This, we claim, is what allows
self-observation by the monkey to train a system thatcan be used for observing the actions of others andrecognizing just what those actions are.

FðtÞ ¼ ½dðtÞ; vðtÞ; aðtÞ; o1 ðtÞ; o2 ðtÞ; o3 ðtÞ; o4 ðtÞwith the following components (see Fig. 4): threecomponents are hand conﬁguration parameters—a(t),index ﬁnger-tip and thumb-tip aperture; o3(t), o4(t), thetwo angles deﬁning how close the thumb is to the handas measured relative to the side of the hand and to theinner surface of the palm—and the remaining fourparameters relate the hand to the object. o1 and o2
components represent the orientation of diﬀerent components of the hand relative to the opposition axis forthe chosen aﬀordance in the object, whereas d and v
represents the kinematics properties of the hand withreference to the target location—o1(t), the cosine of theangle between the object axis and the (index ﬁnger tip–
thumb tip) vector; o2(t), the cosine of the angle between
the object axis and the (index ﬁnger knuckle–thumb tip)
vector; d(t), distance to target at time t; v(t), tangential
velocity of the wrist.
In considering the last four variables, note that only
one or two of them will be relevant in generating a

3 The MNS1 model
We now present a high-level view of the MNS1 model interms of the set of interacting schemas (functional units:
Arbib 1981; Arbib et al. 1998, Chap. 3) shown in Fig. 5,
which deﬁne the MNS1 model of area F5 and related
brain regions. As we now demonstrate, the connectivity
shown in Fig. 5 is constrained by the existing neurophysiology and neuroanatomy of the monkey brain. We
have already introduced areas AIP and area F5, dividing
the F5 grasp-related neurons into (i) F5 mirror neurons

120

Fig. 4. The components of handstate F(t) ¼ [d(t), v(t), a(t), o1(t),o2(t), o3(t), o4(t)]. Note that someof the components are purely
hand conﬁguration parameters(namely v, o3, o4, and a) whereas
others are parameters relating
the hand to the object

Fig. 5. The MNS1 model. Top diagonal: a portion of the FARS
model. Object features are processed by the caudal intraparietal
sulcus, (cIPS) and AIP to extract grasp aﬀordances, and these are sent
on to the canonical neurons of F5 that choose a particular grasp.
Bottom right: recognizing the location of the object provides
parameters to the motor programming area F4 which computes thereach. The information about the reach and the grasp is taken by the
motor cortex M1 to control the hand and the arm. New elements of
the MNS1 model: bottom left are two schemas, one to recognize theshape of the hand, and the other to recognize how that hand ismoving. Just to the right of these is the schema for hand–object spatial
relation analysis. It takes information about object features, themotion of the hand and the location of the object to infer the relation
between hand and object. The center two regions marked by the grayrectangle form the core mirror circuit. This complex associates thevisually derived input (hand state) with the motor program input fromF5 canonical neurons during the learning process for the mirror
neurons. The grand schemas introduced in section 3.2 are illustrated
as the following. The ‘‘core mirror circuit’’ schema is marked by thecenter grey box; the ‘‘visual analysis of hand state’’ schema is outlined
by solid lines just below it, and the ‘‘reach and grasp’’ schema isoutlined by dashed lines (Solid arrows: established connections; Dashedarrows: postulated connections. Details of the ascription of speciﬁcschemas to speciﬁc brain regions were deferred to a later paper.)which are, when fully developed, active during certain
self-movements of grasping by the monkey and during
the observation of a similar grasp executed by others,and (ii) F5 canonical neurons, namely those active during
self-movement and object vision but not for recognitionof the action of others. Other brain regions also play an

121important role in mirror-neuron-system functioning in
the macaque’s brain. Posterior parietal cortex and thecortex of caudal superior temporal sulcus (STS) havebeen subdivided into numerous areas mainly involved inspatial analysis of the visual environment and in thecontrol of spatially oriented behaviour (Maioli et al.
1998).
Based on cytoarchitectonic and connectional criteria
Brodmann’s area 7 (inferior parietal lobule) has beenfound to contain distinct regions including areas 7a and7b and area 7ip. Area 7 reaches its highest developmentin primates (Cavada and Goldman-Rakic 1989). Damage to this area can cause impairments in visually guided
reaching in addition to other spatial perceptual andmotor deﬁcits (Ratcliﬀ 1991; Stein 1991). The posterior
half of 7ip corresponds to the ventral intraparietal area
(Maunsell and Van Essen 1983) and the lateral intraparietal area (Andersen et al. 1985), and contains area
AIP. Areas 7a, 7ip, and 7b (to a lesser extent) arereciprocally connected with the cortex of the STS.
Although the density of 7b connections with the visualmotion cortex of STS is weak compared the extensive
connections of 7b with somatosensory areas, the interconnections of 7b with the visual regions are establishedthrough anterior 7ip and the transitional cortex 7ab
between 7a and 7b (Cavada and Goldman-Rakic 1989).
Findings from the same study also conﬁrm that AIP isconnected with area 7b.
Area F5 has no direct input from visual occipitalareas. Its main cortical input comes from inferior parietal lobe, and in particular area AIP and area 7b (Matelli et al. 1985; Gallese et al. 1996). The detailed
connections and organization of areas F4 and F5 arereviewed by Rizzolatti et al. (1998) and Geyer et al.
(2000). The caudal intraparietal sulcus, with projections
to AIP (Sakata et al. 1997) and area 7a (Cavada andGoldman-Rakic 1989, Fig. 7), has been shown to encode
object properties such as the orientation of object surfaces and object axes (Sakata et al. 1997).
We conclude our brief discussion of anatomical
connections by turning back to area 7ip. Area 7a receives input from the lateral intrparietal area (Andersen
et al. 1990; Lewis and Van Essen 2000), the medial intraparietal area (Boussaoud et al. 1990; Lewis and Van
Essen 2000; Bota 2001), and the ventral intraparietal
area (Lewis and Van Essen 2000; reviewed in Maunsell
1995). Interested readers can ﬁnd more details about
these connections at the NeuroHomology Database
Website (http://brancusi.usc.edu/scripts/webmerger.exe?
/database/homologies-main.html; Bota 2001). The premotor projections of these areas include the regions F2(not shown) and F4 (Luppino et al. 1999), as reviewed inGeyer et al. (2000).
The subsystem of the MNS1 model responsible forthe visuomotor transformation of objects into aﬀordances and grasp conﬁgurations, linking AIP and F5
canonical neurons, corresponds to a key subsystem of
the FARS model reviewed above. Our task is to complement the visual pathway via AIP by pathways directed toward F5 mirror neurons which allow themonkey to observe arm–hand trajectories and matchthem to the aﬀordances and location of a potential
target object. We will then show how the mirror system
may learn to recognize actions already in the repertoireof the F5 canonical neurons. In short, we will provide amechanism whereby the actions of others are ‘‘recognized’’ based on the circuitry involved in performing
such actions. (A topic of future research is to delineate
the circuitry – more apparent in chimpanzee thanmonkey but still rudimentary compared to human – forthe reverse process of imitation, going from the recognition of a novel action performed by others to the addition of that action to one’s own repertoire.) Sect. 4
provides the details of the implemented schemas, andSect. 5 confronts the overall model with virtual experiments and produces testable predictions.
3.1 Overall function
In general, the visual input to the monkey represents acomplex scene. However, we here sidestep much of thiscomplexity (including attentional mechanisms) by assuming that the brain extracts two salient subscenes; astationary object and in some cases a (possibly) moving
hand. The overall system operates in two modes:
1. Prehension. In this mode, the view of the stationary
object is analyzed to extract aﬀordances; then under
prefrontal inﬂuence F5 may choose one of these to act
upon, commanding the motor apparatus to performthe appropriate reach and grasp based on parameterssupplied by the parietal cortex. The FARS model
captures the linkage of F5 and AIP with PFC (Fig. 2).
In the MNS1 model, we incorporate the F5 and AIP
components from FARS (top diagonal of schemas inFig. 5), but omit IT and PFC from the present
analysis.
2. Action recognition. In this mode, the view of the stationary object is again analyzed to extract aﬀordances, but now the initial trajectory and preshape ofan observed moving hand must be extrapolated todetermine whether the current motion of the hand canbe expected to culminate in a grasp of the object
appropriate to one of its aﬀordances.
We do not prespecify all the details of the MNS1
schemas. Instead, we oﬀer a learning model which, given
a grasp that is already in the motor repertoire of the F5canonical neurons, can yield a set of F5 mirror neurons
trained to be active during such grasps as a result of selfobservation of the monkey’s own hand grasping thetarget object. (How such grasps may be acquired in theﬁrst place is a topic of current research.) Consistent withthe hand-state hypothesis, the result will be a system
whose mirror neurons can respond to similar actions
observed being performed by others. The current implementation of the MNS1 model exploits learning in
artiﬁcial neural nets.
The heart of the learning model is provided by theobject aﬀordance–hand state association schema and theaction recognition (mirror neurons) schema. These form
the core mirror (learning) circuit, marked by the gray

122

slanted rectangle in Fig. 5, which mediates the development of mirror neurons via learning. The simulation
results of this article will focus on this part of the model.
Section 4 presents in detail the neural network structureof the core circuit. As we note further in Sect. 5, thisleaves open many problems for further research, including the development of a basic action repertoire by
F5 canonical neurons through trial and error in infancy,
and the expansion and reﬁnement of this repertoire
throughout life.
3.2 Schemas explained
As shown in the caption of Fig. 5, we encapsulate theschemas shown there into the three ‘‘grand schemas’’ ofFig. 6a. These guide our implementation of MNS1.
Nonetheless, it seems worth providing speciﬁcations of
the more detailed schemas both to ground the deﬁnitionof the grand schemas and to set the stage for moredetailed neurobiological modeling in later papers. Ourearlier review of the neuroscience literature justiﬁes our
initial hypotheses, made explicit in Fig. 5, as to wherethese ﬁner-grain schemas are realized in the monkey
brain. However, after we explain these ﬁner-grain
schemas, we will then turn to our present simulationof the three grand schemas which is based on overall
functionality rather than neural regionalization, yetnonetheless yields interesting predictions for further
neurophysiological experimentation.
3.2.1 Grand schema 1: reach and grasp
Object features schema. The output of this schemaprovides a coarse coding of geometrical features of theobserved object. It thus provides suitable input to AIP
and other regions/schemas.
Object aﬀordance extraction schema. This schema transforms its input, the coarse coding of geometrical featuresof the observed object provided by the object features
schema, into a coarse coding for each aﬀordance of theobserved object.

Fig. 6a,b. a For purposes of simulation, we aggregate the schemas of
the MNS1 model of Fig. 5 into three ‘‘grand schemas’’ for visual
analysis of hand state, reach and grasp, and core mirror circuit. b For
detailed analysis of the core mirror circuit, we dispense withsimulation of the other two grand schemas and use other computational means to provide the three key inputs to this grand schema

Motor program (grasp) schema. We identify this schemawith the canonical F5 neurons, as in the FARS model.
Input is provided by AIP’s coarse coding of aﬀordances
for the observed object. We assume that the output of
the schema encodes a generic motor program for the
AIP-coded aﬀordances. This output serves as thelearning signal to the action recognition (Mirror neurons)
schema and drives the hand control functions of the
motor execution schema.
Object location schema. The output of this schemaprovides, in some body-centered coordinate frame,
the location of the center of the opposition axis for the
chosen aﬀordance of the observed object.
Motor program (reach) schema. The input is the
position coded by the object location schema, while
the output is the motor command required to transport
the arm to bring the hand to the indicated location. Thisdrives the arm control functions of the motor execution
schema.
The motor execution schema determines the course of
movements via activity in primary motor cortex M1 and‘‘lower’’ regions.
We next review the schemas which (in addition to thepreviously presented object features and object aﬀordance extraction schemas) implement the visual systemof the model.
3.2.2 Grand schema 2: visual analysis of hand state. The
hand-shape recognition schema takes as input a view of ahand, and its output is a speciﬁcation of the hand shape,
which thus forms some of the components of the handstate. In the current implementation these are a(t), o3(t),and o4(t). Note also that we implicitly assume that theschema includes a validity check to verify that the scene
does contain a hand.
The Hand-motion detection schema takes as input asequence of views of a hand and returns as output thewrist velocity, supplying the v(t) component of the handstate.
The Hand-object spatial relation analysis schema
receives object-related signals from the object features
schema, as well as input from the object location, handshape recognition, and Hand-motion detection schemas.
Its output is a set of vectors relating the current handpreshape to a selected aﬀordance of the object. The
schema computes such parameters as the distance of theobject to the hand, and the disparity between the opposition axes of the object and the hand. Thus the handstate components o1(t), o2(t), and d(t) are supplied bythis schema. The hand–object spatial relation analysis
schema is needed because, for almost all mirror neurons
in the monkey, a hand mimicking a matching grasp
would fail to elicit the mirror neuron’s activity unless thehand’s trajectory were taking it toward an object with agrasp that matches one of the aﬀordances of the object.
The output of this visual analysis is relayed to the object
aﬀordance–hand state association schema which drives
the F5 mirror neurons whose output is a signal
expressing conﬁdence that the observed trajectory will
extrapolate to match the observed target object using thegrasp encoded by that mirror neuron.

123

3.2.3 Grand schema 3: core mirror circuit. The action
recognition schema – which is meant to correspond to themirror neurons of area F5 – receives two inputs in ourmodel. One is the motor program selected by the motor
program schema; the other comes from the object
aﬀordance–hand state association schema. This schema
works in two modes: learning and recognition. When aself-executed grasp is taking place the schema is inlearning mode and the association between the observed
hand state (object aﬀordance–hand state association
schema) and the motor program (motor program
schema) is learned. While in recognition mode, themotor program input is not active and the schema acts
as a recognition circuit. If satisfactory learning (in termsof generalization and the range of actions learned) has
taken place via self-observation, then the schema will
respond correctly while observing another’s grasp actions.
The object aﬀordance–hand state association schemacombines all the hand-related information as well as theobject information available. Thus the inputs to theschema are from hand-shape recognition [componentsa(t), o3(t), o4(t)], hand motion detection [component v(t)],hand–object spatial relation analysis [o1(t), o2(t), d(t)] andfrom object aﬀordance extraction schemas. As will be
explained below, the schema needs a learning signal(mirror feedback). This signal is relayed by the action
recognition schema, and is basically a copy of the motor
program passed to the action recognition schema itself.
The output of this schema is a distributed representationof the object and hand-state match (in our implementation the representation is not prespeciﬁed but shapedby the learning process). The idea is to match the object
and the hand state as the action progresses during a
speciﬁc observed reach and grasp. In the current implementation, time is unfolded into a spatial representation of ‘‘the trajectory until now’’ at the input of theobject aﬀordance–hand state association schema, and theaction recognition schema decodes the distributed representation to form the mirror response (again, the decoding is not prespeciﬁed but is the result of the
backpropagation learning). In any case, the schema has
two operating modes. The ﬁrst is the learning mode
where the schema tries to adjust its eﬀerent and aﬀerent
weights to ensure the right activity in the action recognition schema. The second mode is the forward mode
where it maps the hand state and the object aﬀordance
into a distributed representation to be used by the action
recognition schema.
The key question for our present modeling will be toaccount for how learning mechanisms may shape theconnections to mirror neurons in such a way that anaction in the motor program repertoire of the F5 canonical neurons may become recognized by the mirror
neurons when performed by others.
To conclude this section, we note that our modeling issubject to two quite diﬀerent tests: (i) its overall eﬃcacy
in explaining behavior and its development, which canbe tested at the level of the schemas (functional units)
presented in this article; and (ii) its further eﬃcacy inexplaining and predicting neurophysiological data. As

we shall see below, certain neurophysiological predictions are possible given the current work, even though
the present implementation relies on relatively abstract
artiﬁcial neural networks.
4 Methods
4.1 Schema implementation
Having indicated the functionality and possible neuralbasis for each of the schemas that will make up each
grand schema, we now turn to the implementation ofthese three grand schemas. As noted earlier, the detailed
neurobiological modeling of the ﬁner-grain schemas is a
topic for further research. Here we implement the three
grand schemas so that each functions correctly in termsof its input–output relations, and so that the core mirror
circuit contains model neurons whose behavior can betested against neurophysiological data and yield predictions for novel neurophysiological experiments. The coremirror circuit is thus the principal object of study(Fig. 6b), but in order to study it there must be an
appropriate context, necessitating the construction of
the kinematically realistic reach and grasp simulator andthe visual analyzer for hand state. The latter will ﬁrst be
implemented as an analyzer of views of human hands,
and then will have its output replaced by simulated
hand-state trajectories to reduce computational expensein our detailed analysis of the core mirror circuit.
4.2 Grand schema 1: reach and grasp
We ﬁrst discuss the reach and grasp simulator that
corresponds to the whole reach and grasp command
system shown at the right of the MNS1 diagram (Fig. 5).
The simulator lets us move from the representation of
the shape and position of a (virtual) 3-D object and theinitial position of the (virtual) arm and hand to a
trajectory that successfully results in simulated graspingof the object. In other words the simulator plans a grasp
and reach trajectory and executes it in a simulated 3-Dworld. Trajectory planning (e.g., Kawato et al. 1987;
Jordan and Rumelhart 1992; Kawato and Gomi 1992;
Karniel and Inbar 1997; Breteler et al. 2001) and control
of prehension (Hoﬀ and Arbib 1993; Wolpert andGhahramani 2000, for a review), and their adaptation,
have been widely studied. However, our simulator is not
adaptive – its sole purpose is to create kinematicallyrealistic actions. A similar reach and grasp system was
proposed (Rosenbaum et al. 1999) where a movement isplanned based on the constraint hierarchy, relying on
obstacle avoidance and candidate posture evaluation
processes (Meulenbroek et al. 2001). However, the armand hand model was much simpler than ours as the arm
was modeled as a 2-D kinematics chain. Our reach/grasp
simulator is a nonneural extension of FARS model
functionality to include the reach component. It controls
a virtual arm/hand with 19 degrees of freedom (DOFs)(three at the shoulder, one for elbow ﬂexion/extension,

124

three for wrist rotation, two for each ﬁnger joints withadditional two DOFs for thumb one to allow the thumbto move sideways, and the other for the last joint in the
thumb) and provides routines to perform realistic
grasps. This kinematic realism is based on the literatureof primate reach and grasp experiments [for human see
Jeannerod et al. (1995), Hoﬀ and Arbib (1993) andcitations therein; for monkey see Roy et al. (2000)].
During a typical reach to grasp movement, the hand will
follow a bell-shaped velocity proﬁle (a single peaked
velocity curve). The kinematics of the aperture between
ﬁngers used for grasping also exhibits typical characteristics. The aperture will ﬁrst reach a maximum value thatis larger than the aperture required for grasping theobject, and then as the hand approaches to the target thehand encloses to match the actual required aperture forthe object. It is also important to note that in graspingtasks the temporal pattern of reaching and grasping issimilar in monkey and human (Roy et al. 2000). Of
course, there are intersubject and intertrial variability inboth velocity and aperture proﬁles (Marteniuk and
MacKenzie 1990). Therefore in our simulator we
captured the qualitative aspects of the typical reachand grasp actions, namely that the velocity proﬁles havesingle peaks and that the hand aperture has a maximum
value which is larger than the object size (see Fig. 7,
curves a(t) and v(t), for sample aperture and velocity
proﬁles generated by our simulator) . A grasp is plannedby ﬁrst setting the operational space constraints (e.g.,
points of contact of ﬁngers on the object) and thenﬁnding the arm–hand conﬁguration to fulﬁll the constraints. The latter is the inverse kinematics problem.
The simulator solves the inverse kinematics problem bysimulated gradient descent with noise added to thegradient (see Appendix A.2 for a grasp-planning example). Once the hand–arm conﬁguration is determined fora grasp action, then the trajectory is generated bywarping time using a cubic spline. The parameters of the
spline are ﬁxed and determined empirically to satisfy

aperture and velocity proﬁle requirements. Within thesimulator, it is possible to adjust the target identity,
position and size manually using a graphical userinterface or automatically by the simulator as, forexample, in training set generation.
Figure 7 (left) shows the end state of a power grasp,
while Fig. 7 (right) shows the time series for the handstate associated with this simulated power grasp trajectory. For example, the curve labeled d(t) shows the
distance from the hand to the object decreasing until thegrasp is completed, while the curve labeled a(t) shows
how the aperture of the hand ﬁrst increases to yield asafety margin larger than the size of the object and thendecreases until the hand contacts the object.
Figure 8a shows the virtual hand/arm holding a small
cube in a precision grip in which the index ﬁnger (or alarger ‘‘virtual ﬁnger’’) opposes the thumb. The powergrasp (Fig. 8b) is usually applied to big objects and ischaracterized by the hands covering the object, with theﬁngers as one virtual ﬁnger opposing the palm as
the other. In a side grasp (Fig. 8c), the thumb opposes
the side of another ﬁnger. To clarify the type of heuristics we use to generate the grasp, Appendix A.2 outlines the grasp planning and execution for a precisionpinch.
4.3 Grand schema 2: visual analysis of hand state
The visual analysis of hand state schema is a nonneurophysiological implementation of a visual analysis
system to validate the extraction of hand parametersfrom a view of a hand, by recovering the conﬁgurationof a model of the hand being seen. The hand model is a
3-D 14 DOF kinematic model, with a 3-DOF joint for
the wrist, two 1-DOF joints (metacarpophalangeal anddistalinterphalangeal) for each of the four ﬁngers, andﬁnally a 1-DOF joint for the metacarpophalangeal joint
and a 2-DOF joint for the carpometacarpal joint of theFig. 7. Left: The ﬁnal state ofarm and hand achieved by thereach/grasp simulator in executing a power grasp on the object
shown. Right: The hand-state
trajectory read oﬀ from thesimulated arm and hand during
the movement whose end state isshown at left. The hand statecomponents are: d(t), distance totarget at time t; v(t), tangential
velocity of the wrist; a(t), index
and thumb ﬁnger tip aperture;o1(t), cosine of the angle between
the object axis and the (index
ﬁnger tip–thumb tip) vector;
o2(t), cosine of the angle between
the object axis and the (index
ﬁnger knuckle—thumb tip) vector; o3(t), angle between the
thumb and the palm plane; o4(t),angle between the thumb and theindex ﬁnger

125

thumb. Note the distinction between ‘‘hand conﬁguration,’’ which gives the joint angles of the hand considered in isolation, and the ‘‘hand state’’ which comprises
seven parameters relevant to assessing the motion andpreshaping of the hand relative to an object. Thus thehand conﬁguration provides some–but not all–of thedata needed to compute the hand state.
To lighten the load of building a visual system torecognize hand features, we mark the wrist and the articulation points of the hand with distinctive colors. We
then use this color coding to help recognize key portionsof the hand and use this result to initiate a process ofmodel matching. Thus the ﬁrst step of the vision problem is color segmentation, after which the 3-D handshape is recovered.
4.3.1 Color segmentation and feature extraction. One
needs color segmentation to locate the colored regions on
the image. Gray-level segmentation techniques cannot
be used in a straightforward way because of the vectorial
nature of color images (Lambert and Carron 1999).
Split-and-merge is a well-known image segmentation
technique in image processing (Sonka et al. 1993), which
involves recursively splitting the image into smaller
pieces until some homogeneity criterion is satisﬁed as abasis for reaggregation into regions. In our case, thecriterion is having similar color throughout a region.
However, RGB (red–green–blue) space is not well suitedfor this purpose; HSV (hue–saturation–value) space isbetter suited since in the segmentation processes hue
usually corresponds to human perception and it ignores
shading eﬀects (Russ 1998, Chaps. 1, 6). However, thesegmentation system we implemented with HSV space,although better than the RGB version, was not satisfactory for our purposes. Therefore, we designed asystem that can learn the best color space.
Figure 9a shows the training phase of the color expert
system, which is a (one-hidden-layer) feed-forward network with a sigmoidal activation function. The learningalgorithm is backpropagation with momentum and an
adaptive learning rate. The given image is put through asmoothing ﬁlter to reduce noise in the image before
training. Then the network is given around 100 training
samples each of which is a vector of [(R, G, B), perceived
color code] values. The output color code is a vector
consisting of all zeros except for one component corresponding to the perceived color of the patch. Basically,
the training builds an internal nonlinear color space fromwhich it can unambiguously tell the perceived color. This
training is done only at the beginning of a session to learn
the colors used on the particular hand. Then the network
is ﬁxed as the hand is viewed in a variety of poses.
Figure 9b illustrates the actual segmentation processusing the color expert to ﬁnd each region of a single(perceived) color (see Appendix A.1 for details). The
output of the algorithm is then converted into a feature

Fig. 8a–c. Grasps generated by the simulator: a a precision grasp; b a power grasp; c a side grasp

Fig. 9a,b. a Training the color
expert, based on colored imagesof a hand whose joints arecovered with distinctively colored patches. The trained network will be used in thesubsequent phase for segmenting
image. b A hand image (notfrom the training sample) is fed
to the augmented segmentation
program. The color decision
during segmentation is done byconsulting to the color expert.
Note that a smoothing step (notshown) is performed before
segmentation

126

vector with a corresponding conﬁdence vector giving a
conﬁdence level for each component in the feature vector. Each ﬁnger is marked with two patches of the samecolor. Sometimes it may not be possible to determine
which patch corresponds to the ﬁngertip and which to
the knuckle. In those cases the conﬁdence value is set to0.5. If a color is not found (e.g., the patch may be obscured), a zero value is given for the conﬁdence. If aunique color is found without any ambiguity then the
conﬁdence value is set to 1. The segmented centers ofregions (color markers) are taken as the approximate
articulation point positions. To convert the absolutecolor centers into a feature vector we simply subtract thewrist position from all the centers found and put theresulting relative (x,y) coordinate into the feature vector(but the wrist is excluded from the feature vector as the
positions are speciﬁed with respect to the wrist position).
4.3.2 3-D hand model matching. Our model-matchingalgorithm uses the feature vector generated by thesegmentation system to attain a hand conﬁguration andpose that would result in a feature vector as close as
possible to the input feature vector (Fig. 10). The
scheme we use is a simpliﬁed version of Lowe’s (1991);
see Holden (1997) for a review of other hand recognition
studies.
The matching algorithm is based on minimization of
the distance between the input feature and model feature
vector, where the distance is a function of the two vectors and the conﬁdence vector generated by segmentation system. Distance minimization is realized by hill
climbing in feature space. The method can handle occlusions by starting with ‘‘don’t cares’’ for any joints
whose markers cannot be clearly distinguished in thecurrent view of the hand.
The distance between two feature vectors F and G iscomputed as follows:
qﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
DðF; GÞ ¼ ðF i  Gi Þ2 C fi C gi
where subscripting denotes components, and Cf, and Cg
denote the conﬁdence vectors associated with F and G.
Given this result of the visual processing – our handshape recognition schema – we can clearly read oﬀ thefollowing components of the hand state: a(t); aperture of
the virtual ﬁngers involved in grasping; and o3(t), o4(t),the two angles deﬁning how close the thumb is to thehand as measured relative to the side of the hand and to
the inner surface of the palm (see Fig. 4).

The remaining components can easily be computed
once the object aﬀordance and location is known. The
computation of the components d(t), v(t), o1(t), ando2 (t), deﬁned in Sect. 2.2, constitute the tasks of thehand–object spatial relation analysis schema and thehand-motion detection schema. These require visual inspection of the relation between hand and target, andvisual detection of wrist motion, respectively. It is clearthat they pose only minor challenges for visual processing compared with those we have solved in extracting the hand conﬁguration. We thus have completed our
exposition of the (nonbiological) implementation ofvisual analysis of hand state. Section 5.3 presents ajustiﬁcation of the visual analysis of hand state schemaby showing MNS1 performance when the hand state
was extracted by the described visual recognition system
based on a real video sequence. However, when we turnto modeling the core mirror circuit (grand schema 3) in
Sect. 4.4 we will not use this implementation of visual
analysis of hand state but instead, to simplify computation, we will use synthetic output generated by thereach/grasp simulator to emulate the values that could
be extracted with this visual system. Speciﬁcally, we use
the hand/grasp simulator to produce both (i) the visualappearance of such a movement for our inspection(Fig. 7 left), and (ii) the hand-state trajectory associatedwith the movement (Fig. 7, right). Especially for training
we need to generate and process too many grasp actions,which makes it impractical to use the visual processing
system without special hardware as the computationaltime requirement is too high. Nevertheless, we need toshow the similarity of the data from the visual systemand the simulator: we have already shown that the grasp
simulator generates aperture and velocity proﬁles thatare similar to those in real grasps. Of course, there is still
the question of how well the our visual system can extract these features and, more importantly, how similar
are the other components of the hand state that we did
not speciﬁcally craft to match the real data. Preliminary
positive evidence will be presented in Sect. 5.3.
4.4 Grand schema 3: core mirror circuit
As illustrated in Fig. 6b, our detailed analysis of the coremirror circuit does not require simulation of visual
analysis of hand state and of reach and grasp so long as
we ensure that it receives the appropriate inputs. Thus,
we supply the object aﬀordance and grasp command

Fig. 10. Illustration of the model-matching system. Left: markers located by feature extraction
schema. Middle and right: initial
and ﬁnal stages of model
matching. After matching is
performed a number of parameters for the hand conﬁguration
are extracted from the matched
3-D model

127

directly to the network at each trial. (Actually, we
conduct experiments to compare performance with andwithout an explicit input which codes object aﬀordance.)
The hand-state input is more interesting. Rather thanprovide visual input to the visual analysis of hand state
schema and have it compute the hand-state input to thecore mirror circuit, we use our reach and grasp simulator
to simulate the performance of the observed primate –and from this simulation we extract (as in Fig. 7) both agraphical display of the arm and hand movement thatwould be seen by the observing monkey, as well as thehand-state trajectory that would be generated in his
brain. We thus use the time-varying hand-state trajectory generated in this way to provide the input to themodel of the core mirror circuit of the observing monkey
without having to simultaneously model his visual
analysis of hand state. Thus, we have implemented thecore mirror circuit in terms of neural networks using as
input the synthetic data on hand state that we gather
from our reach and grasp simulator (however, see Sect.
5.3 for a simulation with real data extracted by ourvisual system). (Figure 13 shows an example of therecognition process together with the type of information supplied by the simulator.)
4.4.1 Neural network details. In our implementation, weused a feedforward neural network with one hidden layer.
In contrast to the previous sections, we can here identify
the parts of the neural network as Fig. 5 schemas in a
one-to-one fashion. The hidden layer of the model neuralnetwork corresponds to the object aﬀordance–hand state
association schema, while the output layer of the network
corresponds to the action recognition schema (i.e., we
identify the output neurons with the F5 mirror neurons).
In the following formulation MR (mirror response)
represents the output of the action recognition schemaand MP (motor program) denotes the target of the
network [copy of the output of motor program (grasp)
schema]. X denotes the input vector applied to the
network, which is the transformed hand state (and theobject aﬀordance). The transformation applied is described in Sect. 4.4.2. The learning algorithm used is backpropagation (Rumelhart et al. 1986) with a momentum
term. The formulation is adapted from Hertz et al. (1991).
Activity propagation (the forward pass) is given by
!!
X
X
Wij g
wjk XkMRi ¼ gjkLearning weights from the input layer to the hidden
layer are given by
Wij ¼ Wij þ gðtÞdWij þ lWold
ij ;
where
0

dWij ¼ g

X
j

Wold ¼ W

Wij g

X
k

!!
wjk XkðMPi  MRi Þ

Learning weights from the hidden layer to the output
layer are given by
wjk ¼ wjk þ gðtÞdwjk þ lwold
jk
where
0

dwjk ¼ g

X!
wjk Xk Xk

kwold
jk ¼ wjk
Thesquashing
function
g
we
used
was
gðxÞ ¼ 1=ð1 þ ex Þg and l are the learning rate and themomentum coeﬃcient, respectively. In our simulations,
we adapted g during training such that if the output
error was consistently decreasing then we increased g;otherwise we decreased g. We kept l as a constant set to
0.9. W is the 3 ð6 þ 1Þ matrix of real numbers
representing the hidden-to-output weights. w is the
6 ð210 þ 1Þ [6 ð220 þ 1Þ in the explicit aﬀordancecoding case] matrix of real numbers representing the
input to hidden weights, and X is the 210+1 (220+1 inthe explicit aﬀordance coding case)- component input
vector representing the hand state (trajectory) information. (The extra ‘‘1’’ comes from the fact that theformulation we used hides the bias term required forcomputing the output of a unit in the incoming signals
as a ﬁxed input clamped to 1.)
4.4.2 Temporal-to-spatial transformation. The input tothe network was formed in a way to allow encoding of
temporal information without the use of a dynamic
neural network, and solved the scaling problem. The
input at any time represented the entire input from thestart of the action until the present time t. To form the
input vector, each of the seven components of the handstate trajectory to time t is ﬁtted by a cubic spline (seeKincaid and Cheney 1991 for a formulation), and thesplines are then sampled at 30 uniformly spaced
intervals. The hand-state input is then a vector with
210 components: 30 samples from the time-scaled spline
ﬁtted to the seven components of the hand-state timeseries. Note then that no matter what fraction t is of thetotal time T of the entire trajectory, the input to the
network at time t comprises 30 samples of the hand state
uniformly distributed over the interval [0, t]. Thus the
sampling is less densely distributed across the trajectory
to date as t increases from 0 to T.
An alternative approach would be to use a simplerecurrent-neural-network style architecture to recognize
hand-state trajectories. However, this raises an extra
quantization or segmentation step to convert the continuous hand-state trajectories to discrete states. With
our approach, we avoid this extra step because thequantization is implicitly handled by the learning
process.
For MNS1, we chose to use the spline-based time-tospace transformation, deferring the investigation ofmodels based on recurrent networks (but not necessarily
simple recurrent networks) to our later development of

128

neurally realistic models of the ﬁner-grain schemas ofFig. 5.
Figure 11 demonstrates the preprocessing we use totransform time varying hand-state components into
spatial code. In the ﬁgure only a single component (theaperture) is shown as an example. The solid curve
indicates the available information when the 66% of thegrasp action is completed. In reality a digital computer(and thus the simulator) runs in discrete time steps, so
we construct the continuous curve by ﬁtting a cubic
spline to the collected samples for the value represented(the aperture value in this case). Then we resample 30
points from the (solid) curve to form a vector of size 30.
In eﬀect, this presents the network with the stretchedspline shown by the dotted curve in Fig. 11. Thismethod has the desirable property of avoiding the timescaling problem to establish the equivalence of actions
that last longer than shorter ones, as it is the case for agrasp for an object far from to the hand compared to a
grasp to a closer object. By comparing the dotted curve(what the network sees at t = 0.66) with the ‘‘solid plus
dashed’’ curve (the overall trajectory of the aperture), we
can see how much the network’s input is distorted. As
the action gets closer to its end the discrepancy between
the curves tends to zero. Thus, our preprocessing gives
rise to an approximation to the ﬁnal representation
when a certain portion or more of the input is seen.
Figure 12 samples the temporal evolution of the spatial
input the network receives.
4.4.3 Neural network training. The training set wasconstructed by making the simulator perform various
grasps in the following way.
1. The objects used were a cube of changing size (ageneric size cube scaled by a random scale factor
between 0.5 and 1.5), a disk (approximated as a thin

Fig. 11. The scaling of an incomplete input to form the full spatial
representation of the hand state As an example, only one componentof the hand state—the aperture—is shown. When the 66% of theaction is completed, the preprocessing we apply eﬀectively causes the
network to receive the stretched hand state (the dotted curve) as input
as a rerepresentation of the hand state information accessible to thattime (represented by the solid curve; the dashed curve shows theremaining, unobserved part of the hand state)prism), and a ball (approximated as a dodecahedron),again scaled randomly by a number between 0.75 and1.5. In this particular trial, we did not change the disk
size. In the training set formation, a certain object
always received a certain grasp (unlike the testingcase).
2. The target locations were chosen from the surface
patches of a sphere centered on the shoulder joint.
The patch is deﬁned by bounding meridian (longitude) and parallel (latitude) lines. The extent of themeridian and parallel lines was from 45 to 45. The
step chosen was 15. Thus the simulator made7 7 ¼ 49 grasps per object. The unsuccessful grasp
attempts were discarded from the training set.
3. For each successful grasp, two negative examples wereadded to the training set in the following way. The
inputs (a group of 30) for each parameter are randomly shuﬄed. In this way, the network was forced tolearn the order of activity within a group rather thanlearning the averages of the inputs (note that the
shuﬄing does not change mean and variance).
The second negative pattern was used to stress thatthe distance to target was important. The target
location was perturbed and the grasp was repeated (tothe original target position).
4. Finally, our last modiﬁcation in the backpropagation
training algorithm was to introduce a random input
pattern (totally random; no shuﬄing) on the ﬂy during training and ask the network to produce zero
output for those patterns. In this way we not only
biased the network to be as silent as possible during
ambiguous input presentation but also gave the network a greater chance of reaching global minima.
It should be emphasized that the network was trainedusing the complete trajectory of the hand state (analogous to adjusting synapses after the self-grasp is completed). During testing, in contrast, the preﬁxes of a
trajectory were used (analogous to the predictive
response of mirror neurons while observing a grasp
action). The network thus yielded a time-course ofactivation for the mirror neurons. As we shall see in
Sect. 5, initial preﬁxes yield little or no mirror neuron
activity, and ambiguous preﬁxes may yield transient
activity of the ‘‘wrong’’ mirror neurons.
We thus need to make two points to highlight thecontribution of this study:
1. It is, of course, trivial to train a network to paircomplete trajectories with the ﬁnal grasp type. Whatis interesting here is that we can train the system on
the basis of ﬁnal grasp but then observe the wholetime course of mirror neuron activity, yielding predictions for neurophysiological experiments by highlighting the importance of the timing of mirror neuron
activity.
2. Again, it is commonly understood that the training
method used here, namely backpropagation, is notintended to be a model of the cellular learning
mechanisms employed in cerebral cortex. This might
be a matter of concern were we intending to model the129

Fig. 12. The solid curve showsthe eﬀective input that the network receives as the action progresses. At each simulation cycle
the scaled curves are sampled (30samples each) to form the spatial
input for the network. Towards
the end of the action the networks input gets closer to theﬁnal hand statetime course of learning, or analyze the eﬀect of speciﬁc patterns of neural activity or neuromodulationon the learning process. However, our aim here isquite diﬀerent: we want to show that the connectivityof mirror neuron circuitry can be established throughtraining, and that the resultant network can exhibit arange of novel, physiologically interesting behaviors
during the process of action recognition. Thus, theactual choice of training procedure is purely a matterof computational convenience, and the fact that themethod chosen is nonphysiological does not weaken
the importance of our predictions concerning thetiming of mirror neuron activity.

5 Results
In this study we experimented with two types of
network. The ﬁrst has only the hand state as the
network input. We call this version the nonexplicit
aﬀordance coding network since the hand state will often
imply the object aﬀordance in our simple grasp world.
The second network we experimented with – the explicit
aﬀordance coding network – has aﬀordance coding as
one set of its inputs. The number of hidden layer units ineach case was chosen to be six, and there were three
output units, each one corresponding to a recognizedgrasp.

5.1 Nonexplicit aﬀordance coding experiments
We ﬁrst present results with the MNS1 model implemented without an explicit object aﬀordance input to thecore mirror circuit. We then study the eﬀects ofsupplying an explicit object aﬀordance input.
5.1.1 Grasp resolution. In Fig. 13, we let the (trained)
model observe a grasp action. Figure 13a demonstrates
the executed grasp by giving the views from three
diﬀerent angles to show the reader the 3-D trajectory

traversed. Figure 13b shows the extracted hand state(left) and the response of the (trained) core mirror
network (right). In this example, the network was able toinfer the correct grasp without any ambiguity, since asingle curve corresponding to the observed grasp reaches
a peak and the other two units’ outputs are close to zero
during the whole action. The horizontal axis for bothﬁgures is such that the onset of the action and thecompletion of the grasp are scaled to 0 and 1,
respectively. The vertical axis in the hand-state plot
represents a normalized (min = 0, max = 1) value forthe components of the hand state, whereas the output
plot represents the average ﬁring rate of the neurons (noﬁring = 0, maximum ﬁring = 1). The plotting schemethat is used in Fig. 13 will be used in later simulation
results as well.
It is often impossible (even for humans) to classify agrasp at a very early phase of the action. For example,
the initial phases of a power grasp and precision grasp
can be very similar. Figure 14 demonstrates this situation where the model changes its decision during theaction and ﬁnally reaches the correct result towards
the end of the action. To create this result we used the‘‘outer limit’’ of the precision grasp by having the model
perform a precision grasp for a wide object (using thewide opposition axis). Moreover, the network had notbeen trained using this object for precision grasp. In
Fig. 14b, the curves for power and precision grips cross
towards the end of the action, which shows the change
of decision of the network.
5.1.2 Spatial perturbation. We next analyze how themodel performs if the observed grasp action does notmeet the object. Since we constructed the training set tostress the importance of distance from hand to object,
we expected that network response would decrease withincreased perturbation of target location.
Figure 15 shows an example of such a case. However, the network’s performance was not homogeneousover the workspace: for some parts of the space the
network would yield a strong mirror response evenwith comparatively large perturbation. This could be130Fig. 13a,b. a A single grasp
trajectory viewed from three
diﬀerent angles to clearly show
its 3-D pattern. The wrist trajectory during the grasp ismarked by spatial markers, withthe distance between any twoconsecutive traveled in equal
time intervals. b Left: The input
to the network. Each componentof the hand state is labelled.
Right: How the network classiﬁes the action as a power grasp:
squares, power-grasp output;
triangles, precision grasp; circles,
side-grasp output. Note that the
response for precision and side
grasp is almost zeroFig. 14a,b. Power and precision
grasp resolution. The conventions used are as in Fig. 13. a
The curves for power and precision cross towards the end of theaction showing the change ofdecision of the network. b The
left panel shows the initial conﬁguration, and the right panel
shows the ﬁnal conﬁguration of
the handdue to the small size of the training set. However, interestingly the network’s response had some speciﬁcity
in terms of the direction of the perturbation. If theobject’s perturbation direction were similar to the direction of hand motion then the network would be
more likely to disregard the perturbation (since thetrajectory preﬁx would then approximate a preﬁx of avalid trajectory) and signal a good grasp. Note that the
network reduces its output rate as the perturbation
increases, however the decrease is not linear and after acritical point it sharply drops to zero. The critical
perturbation level also depends on the position in
space.
5.1.3 Altered kinematics. Normally, the simulator produces bell-shaped velocity proﬁles along the trajectoryof the wrist. In our next experiment, we tested action
recognition by the network for an aberrant trajectory
generated with constant arm-joint velocities. Thechange in the kinematics does not change the pathgenerated by the wrist. However the trajectory (i.e., timecourse along the path) is changed and the network is
capable of detecting this change (Fig. 16). The notable
point is that the network acquired this property without
our explicit intervention (i.e., the training set did notinclude any negative samples for altered velocity

proﬁles). This is because the input to the network at
any time comprises 30 evenly spaced samples of the
trajectory up to that time. Thus, changes in velocity canchange the pattern of change exhibited across those 30
samples. The extent of this property is again dependent
on spatial location.
It must be stressed that all the virtual experiments
presented in this section used a single trained network;
no new training samples were added to the training set
for any virtual experiment.
5.1.4 Grasp and object axes mismatch. The last virtual
experiment we present with nonexplicit aﬀordancecoding explores the model’s behavior when the object
opposition axis does not match the hand opposition
axis. This example emphasizes that the response of the
network is aﬀected by the opposition axis of the object
being grasped. Figure 17 shows the axis orientation
change for the object and the eﬀect of this perturbationon the output of the network. The arm simulator ﬁrst
performed a precision grasp to a thin cylinder. The
response of the mirror neuron model to this action
observation is shown in Fig. 17, leftmost panel. As canbe seen from the plot, the network conﬁdently activated the mirror neuron coding precision grip. The middle
panel shows the output of the network when the object

131

Fig. 15. Top: strong precisiongrip mirror response for areaching movement with a precision pinch. Bottom: Spatial
location perturbation experiment. The mirror response isgreatly reduced when the grasp
is not directed at a target object(Only the precision grasp related
activity is plotted; the other two
outputs are negligible.)Fig. 16. Altered kinematics experiment.
Left: the simulator executes the grasp withbell-shaped velocity proﬁle. Right: the simulator executes the same grasp with constantvelocity. Top row shows the graphicalrepresentation of the grasps and the bottom
row shows the corresponding output of the
network (Only the precision grasp related
activity is plotted; the other two outputs are
negligible.)

is changed to a ﬂat plate but the kinematics of thehand is kept the same. The response of the network
declined to almost zero in this case. This is an extreme
example – the objects in Fig. 17 (rightmost panel) have
opposition axes 90 apart, enabling the network to
detect the mismatch between the hand (action) and theobject. With less change in the new axis the network
would give a higher response and, if the opposition axisof the objects were coincident, the network would
respond to both actions (with diﬀerent levels of
conﬁdence depending on other parameters).

5.2 Explicit aﬀordance coding experiments
Now we switch our attention to the explicit aﬀordancecoding network. Here we want to see the eﬀect of object
aﬀordance on the model’s behavior. The new model issimilar to that given before except that it not only has
inputs encoding the current preﬁx of the hand state
trajectory (which includes hand–object relations), butalso has a constant input encoding the relevant aﬀordance of the object under current scrutiny. Thus, boththe training of the network and the performance of the

132

Fig. 17. Grasp and object axes mismatch experiment. Right: thechange of the object from cylinder to a plate (an object axis change of90). Left: the output of the network before the change (the network

turns on the precision grip mirror neuron). Middle: the output of the
network after the object change (Only the precision grasp related
activity is plotted; the other two outputs are negligible.)trained network will exhibit eﬀects of this additional,
aﬀordance input.
Due to the simple nature of the objects studied here,
the aﬀordance coding used in the present study only
encodes the object size. In general, one object will havemultiple aﬀordances. The ambiguity then would be
solved using extra cues such as the contextual state of
the network. We chose a coarse coding of object sizewith ten units. Each unit has a preferred value; the ﬁringof a unit is determined by the diﬀerence of the preferredvalue and the value being encoded. The diﬀerence ispassed through a non-linear decay function by which the
input is limited to the 0 to 1 range (the larger the difference, the smaller the ﬁring rate). Thus, the explicit
aﬀordance coding network has 220 inputs (210 handstate inputs, plus 10 units coarse coding the size). The
number of hidden layer units was again chosen as six
and there were again three output units, each one
corresponding to a recognized grasp.
We have seen that the MNS1 model without explicit
aﬀordance input displayed a biasing eﬀect of object sizein Sect. 5.1.1; the network was biased toward powergrasp while observing a wide precision pinch grasp (thenetwork initially responded with a power-grasp activity
even though the action was a precision grasp). The
model with full aﬀordance replicates the grasp resolution behavior seen in Fig. 12. However, we can now gofurther and ask how the temporal behavior of the modelwith explicit aﬀordance coding reﬂects the fact that
object information is available throughout the action.
Intuitively, one would expect that the object aﬀordance
would speed up the grasp resolution process (which isactually the case, as will be shown in Fig. 19).
In Sects. 5.2.1 and 5.2.2 we look at the eﬀect of
aﬀordance information in two cases: (i) where we study
the response to precision pinch trajectories appropriate
to a range of object sizes, and (ii) where on each trial weuse the same time-varying hand-state trajectory but
modify the object aﬀordance part of the input. In eachcase, we are studying the response of a network that has
been previously trained on a set of normal hand-state
trajectories coupled with the corresponding object
aﬀordance (size) encoding.

sizes, and then for each size drive the (previously trained)
network with both aﬀordance (object size) informationand the hand-state trajectory appropriate for a precisionpinch grasp appropriate to that size of object. For eachcase we looked at the model’s response. Figure 18 shows
the resultant level of mirror responses for four cases (tiny,
small, medium, and big objects). The ﬁlled squares
indicate the precision activity while the empty squares
indicate the power-grasp-related activity. When theobject to be grasped is small, the model turns on the
precision mirror response more quickly and with noambiguity (Fig. 18, top panels). The vertical bar drawn attime 0.6 shows the temporal eﬀect of object size
(aﬀordance). The curves representing the precision grasps
are shifted towards the end (time = 1), as the object size
gets bigger. Our interpretation is that the model gained
the property of predicting that a small object is more
likely to be grasped with a precision pinch rather than apower pinch. Thus the larger the object, the more of the
trajectory that had to be seen before a conﬁdent
estimation could be made that it was indeed leading toa precision pinch. In addition, as we indicated earlier, theexplicit aﬀordance coding network displays the grasp
resolution behavior during the observation of a precisiongrip being applied to large objects (Fig. 18, bottom
panels: right panel and, to a lesser degree, the left panel).
We also compared the general response time of thenonexplicit aﬀordance coding implementation with theexplicit coding implementation. The network with affordance input is faster to respond than the previous
one. Moreover, it appears that – when aﬀordance andgrasp type are well correlated – having access to theobject aﬀordance from the beginning of the action not
only lets the system make better predictions but alsosmoothes out the neuron responses. Figure 19 summarizes this: it shows the precision response of both theexplicit and nonexplicit aﬀordance case for a tiny object(dashed and solid curves, respectively).

5.2.1 Temporal eﬀects of explicit aﬀordance coding. To
observe the temporal eﬀects of having explicit coding of
aﬀordances to the model, we choose a range of object

5.2.2 Teasing apart the hand state and object aﬀordancecomponents. We now look at the case where the handstate trajectory is incompatible with the aﬀordance ofthe observed object. In Fig. 20, the upper-left panel
shows the system output for a precision grasp directed toa medium-sized object whose aﬀordance is supplied to
the network. We then repeatedly input the hand-state
trajectory generated for this particular action but in each

133

Fig. 18. The plots show the levelof mirror responses of the explicit aﬀordance coding object
for an observed precision pinch
for four cases (tiny, small, medium, and big objects). The ﬁlled
squares indicate the precision
activity while the empty squares
indicate the power-grasp-related
activity

Fig. 19. Solid curve: the precision grasp output, for the nonexplicit
aﬀordance case, directed to a tiny object. Dashed curve: the precision
grasp output of the model to the explicit aﬀordance case, for the same
objecttrial use an object aﬀordance discordant with theobserved trajectory aﬀordance (i.e., using a reduced orincreased size of the object). The plots in Fig. 20 show
the change of the output of the model due to the change
in the aﬀordance. The results shown in these plots tell us
two things. First, the recognition process becomes
fuzzier as the object gets bigger, because the larger
object sizes biases the network towards the power grasp.
In the extreme case the object aﬀordance can evenoverwhelm the hand state and switch the network
decision to power grasp (Fig. 20, lower right panel).
Moreover, for large objects, the large discrepancy

between the observed hand state trajectory and the sizeof the objects results in the network converging on a
conﬁdent assessment for neither grasp.
Secondly, the resolution point (the crossing point of
the precision and power curves) shows an interesting
temporal behavior. It may be intuitive to think that as
the object gets smaller the network’s precision decision
gets quicker and quicker (similar to what we have seen in
Sect. 5.2.1). However, although this is the case when theobject is changing size from big to small, it is not thecase when the object size is getting medium to tiny (i.e.,the crossing time has a local minimum between the two
extreme object sizes, as opposed to being at the tinyobject extreme). Our interpretation is that the network
learned an implicit parameter related to the absolutevalue of the diﬀerence of the hand aperture and theobject size such that the maximum ﬁring is achieved
when the diﬀerence is smallest; that is, when the hand
trajectory matches best with the object. This will explain
why the network has the quickest resolution for a size
between the biggest and the smallest sizes.
Figure 21 shows the time of resolution versus object
size in graphical form. We emphasize that the model
easily executes the grasp recognition task when thehand-state trajectory matches object aﬀordance. We do
not include all the results of these control trials, as they
are similar to the cases mentioned in Sect. 5.1.
5.3 Justifying the visual analysis of hand state schema
Before closing, we would like to present a simulation run
using a real video input to justify our claim that hand

134

Fig. 20. Filled squares indicate the precision-grasp-related cell activity,
while the empty squares represent the power-grasp-related cell activity.
The grasps show the eﬀect of changing the object aﬀordance while
keeping a constant hand-state trajectory. In each case, the hand-stateFig. 21. The graph is drawn to show the decision switch time versus
object size. The minimum is not at the boundary, that is, the network
will detect a precision pinch quickest with a medium object size. Notethat the graph does not include a point for ‘‘Biggest object’’ since there
is no resolution point in this case (see the lower right panel of Fig. 20)

state can be extracted from real video and used to drive
the core mirror circuit. The object aﬀordances aresupplied manually since we did not address object
recognition in our visual system. However, the rest of
the hand state is extracted by the hand recognition system
as described in Sect. 4.3. Figure 22 depicts the precisiongrasp action used as input video for the simulation.
The result of 3-D hand matching is illustrated inFig. 23. The color extraction is performed as describedin Sect. 4.3.1, but is not shown in the ﬁgure. It would bevery rewarding to perform all our MNS1 simulations
using this system. However, the quality of the video
equipment available and the computational power

trajectory provided to the network is appropriate to the medium-sized
object, but the aﬀordance input to the network encodes the size
shown. In the case of the biggest object aﬀordance, the eﬀect isenough to overwhelm the hand-state’s precision bias

Fig. 22. The precision-grasp action used to test our visual system is
depicted by superimposed frames (not all the frames are shown)requirements did not allow us to collect many grasp
examples to train the core mirror circuit. Nevertheless,
we did test the hand state extracted by our visual systemfrom this real video sequence on the MNS1 model thathas already been trained with the synthetic grasp
examples.
Figure 24 shows the recognition result when the actual visual recognition system provided the hand statebased on the real video sequence shown in Fig. 23.
Although the output of the network did not reach a highlevel of conﬁdence for any grasp type, we can clearly seethat the network favored the precision grasp over theside and power grasps. It is also interesting to note that asimilar competition (this time between side- andprecision-grasp outputs) took place as we saw (Fig. 14)
when the grasp action was ambiguous.

135

Fig. 23. The video sequence
used to test the visual system isshown together with the 3-D
hand-matching result (over eachframe). Again, not all the frames
are shown

Fig. 24. The output of the MNS1 model when driven by the visual
recognition system while observing the action depicted in Fig. 22. It
must be emphasized that the training was performed using the synthetic
data from the grasp simulator, whereas testing is performed using thehand state extracted by the visual system only. Dashed line, side grasp
related activity; solid line, precision grasp related activity. Power-grasp
activity is not visible as it coincides with the time axis

6 Discussion
6.1 The hand-state hypothesis
Because the mirror neurons within monkey premotor
area F5 ﬁre not only when the monkey performs a certainclass of actions but also when the monkey observes
similar actions, it has been argued that these neurons are
crucial for understanding of actions by others. We agreewith the importance of this role and indeed have built

upon it elsewhere, as we now brieﬂy discuss. Rizzolatti
et al. (1996b) used a positron-emission tomography study
to show that both grasping observation and object
prehension yield highly signiﬁcant activation in the
rostral part of Broca’s area (a signiﬁcant part of thehuman language system) as compared to the control
condition of object observation. Moreover, MassimoMatelli (in Rizzolatti and Arbib 1998) demonstrated ahomology between monkey area F5 and area 45 in thehuman brain (Broca’s area comprises areas 44 and 45).
Such observations led Rizzolatti and Arbib (1998),
building on Rizzolatti et al. (1996a), to formulate themirror-system hypothesis: human Broca’s area contains amirror system for grasping which is homologous to theF5 mirror system of monkey, and this provides theevolutionary basis for language parity; i.e., for anutterance to mean roughly the same for both speakerand hearer. This adds a neural ‘‘missing link’’ to the
tradition that roots speech in a prior system forcommunication based on manual gesture.
Arbib (2001) then reﬁnes this hypothesis by showing
how evolution might have bridged from an ancestralmirror system to a ‘‘language-ready’’ brain via increasingly sophisticated mechanisms for imitation of manual
gestures as the basis for similar skills in vocalization andthe emergence of protospeech. In some sense, then, thepresent paper can be seen as extending these evolutionary concerns back in time. Our central aim was to give acomputational account of the monkey mirror system byasking (i) what data must the rest of the brain supply tothe mirror system? and (ii) how could the mirror system
learn the right associations between classiﬁcation of its
own movements and the movement of others? In seeking

136

to ground the answer to (i) in earlier work on the controlof hand movements (Iberall and Arbib 1990), we were
led to extend our evolutionary understanding of themirror system by oﬀering the hand-state hypothesis: thebasic functionality of the F5 mirror system is to elaborate the appropriate feedback – what we call the handstate – for opposition-space-based control of manual
grasping of an object. Given this functionality, the socialrole of the F5 mirror system in understanding theactions of others may be seen as an exaptation gained bygeneralizing from one’s own hand to another’s hand.
The hand-state hypothesis provides a new explanation of the evolution of the ‘‘social capability’’ of mirror
neurons, hypothesizing that these neurons ﬁrst evolved
to augment the ‘‘canonical’’ F5 neurons (active during
self-movement but not speciﬁcally during the observation of grasping by others) by providing visual feedbackon ‘‘hand state,’’ relating the shape of the hand to theshape of the object.
6.2 Neurophysiological predictions
We introduced the MNS1 model of F5 and related brain
regions as an extension of the FARS model of circuitry
for visually guided grasping of objects that links parietal
area AIP with F5 canonical neurons. The MNS1 model
illustrated in Fig. 5 includes hypotheses as to how
diﬀerent brain regions may contribute to the functioningof the mirror system, and the region-by-region analysisof neural circuitry remains a target for current research
and future publications. However, the implementation
here took a diﬀerent approach, aggregating these regions
into three ‘‘grand schemas’’ – visual analysis of handstate, reach and grasp, and the core mirror circuit – foreach of which we present a detailed implementation. To
justify the claim that the model exhibits neurophysiologically interesting behaviors, we must look more carefullyat the structure of the implementation, stressing that it is
only the activity of mirror neurons in the core mirror
circuit for which we make this claim. We developed thevisual analysis of hand state schema simply to the pointof demonstrating algorithms powerful enough to takeactual video input of a hand (though we simpliﬁed the
problem somewhat by using colored patches) andproduce hand-state information. The reach and grasp
schema then represented all the functionality for taking
the location and aﬀordance of an object, and determining
the motion of a hand and arm to grasp it. However, theaim in the present paper was not to model the neuralmechanisms involved in these processes. Instead, we
showed that if we used the reach and grasp schema togenerate an observed arm–hand trajectory (i.e., torepresent the reach and grasp generator of the monkey
or human being observed), then that simulation could
directly supply the corresponding hand-state trajectory,
and we thus use these data so that we can analyze the coremirror circuit schema (Fig. 6b) in isolation from thevisual analysis of hand state. However, note that we havealso justiﬁed the visual analysis of hand state schema byshowing in a simulation that the core mirror circuit canbe driven with the visual system without any synthetic
data from the reach and grasp schema.
Moreover, this hand-state input (regardless of being
synthetic or real) was presented to the network in a way
which avoids the use of a dynamic neural network. To
form the input vector, each of the seven components of
the hand-state trajectory, up to the present time t, isﬁtted by a cubic spline. Then this spline is sampled at 30
uniformly spaced intervals; i.e., no matter what fraction
t is of the total time T of the entire trajectory, the input
to the network at time t comprises 30 samples of thehand state uniformly distributed over the interval [0, t].
The network is trained using the full trajectory of thehand state in a speciﬁc grasp; the training set pairs eachsuch hand-state history as input with the ﬁnal grasp typeas output. On the contrary, when testing the model withvarious grasp observations, the input to the network was
the hand-state trajectory that was available up to that
instant. This exactly parallels the way the biologicalsystem (the monkey) receives visual (object and hand)
information: when the monkey performs a grasp, thelearning can take place after the observation of thecomplete (self-) generated visual stimuli. On the other
hand, in the observation case the monkey mirror system
predicts the grasp action based on the partial visual
stimuli (i.e., before the grasp is completed). The network
thus yields a time course of activation for the mirror
neurons, yielding predictions for neurophysiologicalexperiments by highlighting the importance of the timingof mirror neuron activity. We saw that initial preﬁxes
will yield little or no mirror neuron activity, and ambiguous preﬁxes may yield transient activity of the‘‘wrong’’ mirror neurons.
Since our aim was to show that the connectivity of
mirror neuron circuitry can be established throughtraining, and that the resultant network can exhibit arange of novel, physiologically interesting behaviors
during the process of action recognition, the actual
choice of training procedure is purely a matter of computational convenience, and the fact that the method
chosen, namely back propagation, is nonphysiologicaldoes not weaken the importance of our predictions
concerning the timing of mirror neuron activity.
With this we turn to neurophysiological predictions
made in our treatment of the core mirror circuit, namelythe ‘‘grounding assumptions’’ concerning the nature of
the input patterns received by the circuit and the actual
predictions on the timing of mirror neuron activity
yielded by our simulations.
6.2.1 Grounding assumptions. The key to the MNS1model is the notion of hand state as encompassing datarequired to determine whether the motion and preshapeof a moving hand may be extrapolated to culminate in a
grasp appropriate to one of the aﬀordances of theobserved object. Basically a mirror neuron must ﬁre if
the preshaping of the hand conforms to the grasp typewith which the neuron is associated; and the extrapolation of hand state yields a time at which the hand isgrasping the object along an axis for which thataﬀordance is appropriate. What we emphasize here is137

not the speciﬁc decomposition of the hand state F(t)into the seven speciﬁc components [d(t), v(t), a(t), o1(t),o2(t), o3(t), o4(t)] used in our simulation, but rather thatthe input neural activity will be a distributed neural code
which carries information about the movement of thehand toward the object, the separation of the virtual
ﬁngertips, and the orientation of diﬀerent componentsof the hand relative to the opposition axis in the object.
The further claim is that this code will work just as wellfor measuring how well another monkey’s hand ismoving to grasp an object as for observing how themonkey’s own hand is moving to grasp the object,
allowing self-observation by the monkey to train asystem that can be used for observing the actions ofothers and recognizing just what those actions are.
We provided experiments to compare the performance of the core mirror circuit with and without theavailability of explicit aﬀordance information (in thiscase, the size of the object) to strengthen our claim that
it is indeed adaptive for the system to have this additional input available, as shown in Fig. 6b. Note that the‘‘grasp-command’’ input shown in that ﬁgure serves here
as a training input, and will, of course, plays no role in
the recognition of actions performed by others.
Also we have given a justiﬁcation of the visual analysis of hand state schema by showing in a simulationthat the core mirror circuit can be driven with the visualsystem we implemented without requiring the reach andgrasp simulator to provide synthetic data.
6.2.2 Novel predictions. Experimental work to date has
tended to emphasize the actions to be correlated with theactivity of each individual mirror neuron, while paying
little attention to the temporal dynamics of mirror
neuron response. By contrast, our simulations makeexplicit predictions on how a given (hand-state trajectory, aﬀordance) pair will drive the time course of mirror
neuron activity – with non-trivial response possibly
involving the activity of other mirror neurons in addition
to those associated with the actual grasp being observed.
For example, a grasp with an ambiguous preﬁx may
drive the mirror neurons in such a way that the systemwill, in certain circumstances, at ﬁrst give weight to thewrong classiﬁcation, with only the late stages of the
trajectory suﬃcing for the incorrect mirror neuron to bevanquished.
To obtain this prediction we created a scene where theobserved action consisted of grasping a wide object witha precision pinch (thumb and index ﬁnger opposing eachother). Usually this grasp is applied to small objects(imagine grasping a pen along its long axis versus
grasping it along its thin center axis). The mirror response
we got from our core mirror circuit was interesting. First,
the system recognized (while the action was taking place)the action as a power grasp (which is characterized byenclosing the hand over large objects; e.g. grasping anapple), but as the action progressed the model unit representing precision pinch started to become active andthe power-grasp activity started to decline. Eventually
the core mirror circuit settled on the precision pinch. Thisparticular prediction is testable and indeed suggests a

whole class of experiments. The monkey has to be presented with unusual or ambiguous grasp actions that
require a ‘‘grasp resolution.’’ For example, the experimenter can grasp a section of banana using precisionpinch from its long axis. Then we would expect to see
activity from power-grasp-related mirror cells followed
by a decrease of that activity accompanied by increasing
activity from precision-pinch-related mirror neurons.
The other simulations we made lead to diﬀerent
testable predictions such as the mirror response in caseof a spatial perturbation (showing the monkey a fakegrasp where the hand does not really meet the object)
and altered kinematics (perform the grasp with diﬀerent
kinematics than usual). The former is in particular ajustiﬁcation of the model, since in the mirror-neuron
literature it has been reported that the spatial contact of
the hand and the object is almost always required for the
mirror response (Gallese et al. 1996). On the other hand,the altered kinematics result predicts that an alterationof the kinematics will cause a decrease in the mirror
response. We have also noted how a discrepancy between hand-state trajectory and object aﬀordance may
block or delay the system from classifying the observed
movement.
In summary, we have conducted a range of simulation experiments – on grasp resolution, spatial perturbation, altered kinematics, temporal eﬀects of explicit
aﬀordance coding, and analysis of compatibility of thehand state to object aﬀordance – which demonstratethat the present model is not only of value in providing
an implemented high-level view of the logic of the mirror
system, but also serves to provide interesting predictions
ripe for neurophysiological testing, as well as suggesting
new questions to ask when designing experiments on the
mirror system. However, we must note that this study
has excluded some actions (such as tearing and twisting)for which mirror activity has been observed. As newneurophysiological studies on monkeys expand therange of actions for which the temporal response of themirror system is delimited, we will expand our model toexplain the new ﬁndings and suggest yet further classesof experiments to probe the structure and function of themirror system – as well as increasing the set of brain
regions in Fig. 5 for which increasingly realistic neuralmodels are made available.
Acknowledgments. This work was supported in part by a Human
Frontier Science Program grant to M.A.A which made possible
extensive discussions of data on the mirror system with Giacomo
Rizzolatti, Vittorio Gallese, and Massimo Matelli, to whom we
express our gratitude. Writing of the paper was also supported inpart by an ARC-IREX grant to Robyn Owens for the study of‘‘Hand movement recognition and language: assistive technologies
and neural mechanisms’’ while M.A.A was visiting as adjunctprofessor in the Computer Science Department at the University ofWestern Australia.

Appendix: Implementation details
The system was implemented in Java on a Linux
operating system. The grasp simulator can be accessed138

at http://java.usc.edu/~erhan/sim6.1. The material atthis URL also includes the action recognition circuitry.
The simulation environment enables the users to test thissimpliﬁed version of the action recognition ability of the
network.

since it is likely that the four split rectangles will have thesame color and will be connected (with our deﬁnition ofconnectedness).

A.2 Grasp planning and execution for a precision pinch
A.1 The segmentation system
The segmentation system as a whole works as follows:
1. Start with N rectangles (called nodes), set thresholds
for red, green, and blue variances as rV, gV, bV.
2. For each node calculate the red, green, blue varianceas rv, gv, bv.
3. If any of the variance is higher than the threshold(rv>rV or gv>gV or bv>bV), then split the node
into four equal pieces and apply steps 2 and 3 recursively.
4. Feed in the mean red, green, and blue values in that
region to the color expert to determine the color of
the node.
5. Make a list of nodes that are of the same color (addnode to the list reserved for that color).
6. Repeat steps 2–5 until no split occurs.
7. Cluster (in terms of Euclidean distance on the image)
the nodes and discard the outliers from the list (usethe center of the node as the position of the node).
The discarding is performed either when a region isvery far from the current mean (weighted center) or it
is not ‘‘connected’’ to the current center position. The
connectedness is deﬁned as follows. The regions A
and B are connected if the points lying on the line
segment joining the centers of A and B are the samecolor as A and B. Once again, the color expert is used
to determine the percentage of the correct colors (of A
and B) lying on the line segment. If this percentage isover a certain threshold (e.g., 70%) then the regions A
and B are taken as ‘‘connected.’’ (This strategy would
not work for a ‘‘sausage-shaped’’ region, but does
work for patches created by the coloring we used in
the glove.)
8. For each pruned list (corresponding to a color), ﬁnd
the weighted (by the area of the node) mean of theclusters (in terms of image coordinates).
9. Return the cluster mean coordinates as the segmented
region’s center.
So we do not exactly perform the merge part of the split–
merge algorithm. The return values from this procedure
are the (x,y) coordinates of the center of color patches
found. Another issue is how to choose the thresholds.
The variance values are not very critical; a too-smallvalue increases computation time but does not aﬀect the
number of colors extracted correctly (though thereturned coordinates may be shifted slightly). To see
why intuitively, one can notice that the center of arectangle and the centroid of the centers of the quarterrectangles (say after a split operation) would be thesame. This means that if a region is split unnecessarily(because the threshold variances were set to very smallvalues), it is likely to be averaged out by our algorithm

Precision pinch planning involves the folowing steps:
1. Determine the opposition axis to grasp the object.
2. Compute the two (outer) points A and B at which the
opposition axis intersects the object surface. These
serve as the contact points for the virtual ﬁngers thatwill be involved in the grasp.
3. Assign the real ﬁngers to virtual ﬁngers. The particular heuristic we used in the experiments was thefollowing. If the object is on the right (left) with respect to the arm, then the thumb is assigned to thepoint A if A is on the left of (at a lower level than), B;
otherwise the thumb is assigned to B. The index ﬁnger
is assigned to the remaining point.
4. Determine an approximate target position C, for thewrist. Mark the target for the wrist on the line segment connecting the current position of the wrist andthe target for the thumb a ﬁxed length (determined by
the thumb length) away from the thumb target.
5. Solve the inverse kinematics for only the wrist reach(ignore the hand).
6. Solve the inverse kinematics for grasping. Using thesum of distance squares of the ﬁnger tips to the target
contact points, do a random hill-climbing search tominimize the error. Note that the search starts withplacing the wrist at point C. However, the wrist position is not included in the error term.
7. The search stops when the simulator ﬁnds a conﬁguration with error close to zero (success) or after aﬁxed number of steps (failure to reach). In the successcase the ﬁnal conﬁguration is returned as the solution
for the inverse kinematics for the grasp. Otherwise‘‘failure to reach’’ is returned.
8. Execute the reach and grasp. At this point the simulator knows the desired target conﬁguration in termsof joint angles. So what remains to be done is toperform the grasp in a realistic way (in terms of kinematics). The simplest way to perform the reach is tolinearly change the joint angles from the initial conﬁguration to the target conﬁguration. But this does
not produce a bell-shaped velocity proﬁle (nor exactly
a constant-speed proﬁle either because of thenonlinearity in going from joint angles to end-eﬀector
position). The perfect way to plan an end-eﬀector
trajectory requires the computation of the Jacobian.
However we are not interested in perfect trajectories
as long as the target is reached with a bell-shaped
velocity proﬁle. To get the eﬀect it is usually suﬃcient
to modify the idea of linearly changing the joint angles a little bit. We simply modulate the change oftime by replacing time with a third-order polynomialthat will match our constraints for time (starts at 0
and climbs up to 1 monotonically). Note that we arestill working in the joint space and our method may

139

suﬀer from the nonlinearity in transforming the joint
angles to end-eﬀector coordinates. However, ourempirical studies showed that a satisfactory result, forour purposes, could be achieved in this way.

References
Andersen RA, Asanuma C, Cowan WM (1985) Callosal and prefrontal associational projecting cell populations in area 7A ofthe macaque monkey: a study using retrogradely transported
ﬂuorescent dyes. J Comp Neurol 232: 443–455
Andersen RA, Asanuma C, Essick G, Siegel RM (1990) Corticocortical connections of anatomically and physiologically deﬁned subdivisions within the inferior parietal lobule. J Comp
Neurol 296: 65–113
Arbib MA (1981) Perceptual structures and distributed motor
control. In: Brooks VB (ed) Handbook of physiology – The
nervous system II, Motor control, Part 1. American Physiological Society, Bethesda, Md., pp 1449–1480
Arbib MA (2002) The mirror system, imitation, and the evolution of language. In: Nehaniv C, Dautenhahn K (eds) Imitation in animals and artifacts. MIT Press, Cambridge, Mass.
Arbib MA, Érdi P, Szentágothai J (1998) Neural organization:
structure, function and dynamics. MIT Press, Cambridge,
Mass.
Bota M (2001) Neural homologies: principles, databases andmodeling. PhD thesis, Department of Neurobiology, University of Southern California, Los Angeles, Calif.
Boussaoud D, Ungerleider LG, Desimone R (1990) Pathways formotion analysis – cortical connections of the medial superior
temporal and fundus of the superior temporal visual areas in
the macaque. J Comp Neurol 296: 462–495
Cavada C, Goldman-Rakic PS (1989) Posterior parietal cortex inrhesus monkey: II. Evidence for segregated corticocortical
networks linking sensory and limbic areas with the frontal lobe.
J Comp Neurol 287: 422–445
Fagg AH, Arbib MA (1998) Modeling parietal–premotor interactions in primate control of grasping. Neural Netw 11: 1277–
1303
Gallese V, Fadiga L, Fogassi L, Rizzolatti G (1996) Action
recognition in the premotor cortex. Brain 119: 593–609
Gentilucci M, Fogassi L, Luppino G, Matelli M, Camarda R,
Rizzolatti G (1988). Functional organization of inferior area 6
in the macaque monkey. I. Somatotopy and the control ofproximal movements. Exp Brain Res 71: 475–490
Geyer S, Matelli M, Luppino G, Zilles K (2000) Functional neuroanatomy of the primate isocortical motor system. Anat
Embryol 202: 443–474
Gibson JJ (1966) The senses considered as perceptual systems.
Houghton Miﬄin, Boston
Hertz J, Krogh A, Palmer RG (1991) Introduction to the theory ofneural computation. Addison-Wesley, Redwood City, Calif.
Hoﬀ B, Arbib MA (1993) Models of trajectory formation andtemporal interaction of reach and grasp. J Mot Behav 25:
175–192
Holden EJ (1997) Visual recognition of hand motion. PhD thesis,
Department of Computer Science, University of Western
Australia, Perth, Australia
Iberall T, Arbib MA (1990) Schemas for the control of hand
movements: an essay on cortical localization. In: Goodale MA(ed) Vision and action: the control of grasping. Ablex,
Norwood, NJ, pp 163–180
Jeannerod M, Arbib MA, Rizzolatti G, Sakata H (1995) Grasping
objects: the cortical mechanisms of visuomotor transformations. Trends Neurosci 18: 314–320
Jordan MI, Rumelhart DE (1992) Forward models: supervised
learning with distal teacher. Cogn Sci 16: 307–354

Karniel A, Inbar GF (1997) A model for learning human reaching
movements. Biol Cybern 77: 173–183
Kawato M, Gomi H (1992) A computational model of four regionsof the cerebellum based on feedback-error learning. Biol
Cybern 68: 95–103
Kawato M, Furukawa K, Suzuki R (1987) A hierarchical neuralnetwork model for control and learning of voluntary movement. Biol Cybern 57: 169–185
Kincaid D, Cheney W (1991) Numerical analysis: Mathematics ofScientiﬁc Computing Brooks/Cole, Paciﬁc Grove, Calif.
Klein Breteler MDK, Gielen SC, Meulenbroek RG (2001) Endpoint constraints in aiming movements: eﬀects of approachangle and speed. Biol Cybern 85: 65–75
Lambert P, Carron T (1999) Symbolic fusion of luminance-huechroma features for region segmentation. Pattern Recogn 32:
1857–1872
Lewis JW, Van Essen (2000) Corticocortical connections of visual,sensorimotor, and multimodal processing areas in the parietal
loe of the macaque monkey. J Comp Neurol 428: 112–137
Lowe DG (1991) Fitting parameterized three-dimensional models
to images. IEEE Trans Pattern Anal Mach Intell 13: 441–450
Luppino G, Murata A, Govoni P, Matelli M (1999) Largely segregated parietofrontal connections linking rostral intraparietal
cortex (areas AIP and VIP) and the ventral premotor cortex(areas F5 and F4). Exp Brain Res 128: 181–187
Maioli MG, Squatrito S, Samolsky-Dekel BG, Sanseverino ER(1998) Corticocortical connections between frontal periarcuate
regions and visual areas of the superior temporal sulcus and the
adjoining inferior parietal lobule in the macaque monkey.
Brain Res 789: 118–125
Marteniuk RG, MacKenzie CL (1990) Invariance and variability inhuman prehension: implications for theory development. In:
Goodale MA (ed) Vision and action: the control of grasping.
Ablex, Norwood, NJ, pp 163–180
Matelli M, Camarda R, Glickstein M, Rizzolatti G (1986) Aﬀerent
and eﬀerent projections of the inferior area 6 in the macaquemonkey. J Comp Neurol 251: 281–298
Matelli M, Luppino G, Murata A, Sakata H (1994) Independent
anatomical circuits for reaching and grasping linking the inferior parietal sulcus and inferior area 6 in macaque monkey. Soc
Neurosci Abstr 20: 404.4
Maunsell JH (1995) The brains visual world: representation ofvisual targets in cerebral cortex. Science 270: 764–769
Maunsell JH, Van Essen DC (1983) The connections of the middle
temporal visual area (MT) and their relationship to a corticalhierarchy in the macaque monkey. J Neurosci 3: 2563–2586
Meulenbroek RG, Rosenbaum DA, Jansen C, Vaughan J, Vogt S(2001) Multijoint grasping movements. Simulated and observed
eﬀects of object location, object size, and initial aperture. Exp
Brain Res 138: 219–234
Murata A, Fadiga L, Fogassi L, Gallese V, Raos V, Rizzolatti G(1997) Object representation in the ventral premotor cortex(area F5) of the monkey. J Neurophysiol 78: 2226–2230
Ratcliﬀ G (1991) Brain and space: some deductions from clinicalevidence. In: Paillard J (ed) Brain and space, Oxford University Press, Oxford, pp 237–250
Rizzolatti G, Arbib MA (1998) Language within our grasp. Trends
Neurosci 21: 188–194
Rizzolatti G, Fadiga L (1998) Grasping objects and grasping action
meanings: the dual role of monkey rostroventral premotor
cortex (area F5). In: Sensory guidance of movement, Novartis
Foundation Symposium 218. Wiley, Chichester, pp 81–103
Rizzolatti G, Camarda R, Fogassi L, Gentilucci M, Luppino G,
Matelli M (1988) Functional organization of inferior area 6 in
the macaque monkey. II. Area F5 and the control of distal
movements. Exp Brain Res 71: 491–507
Rizzolatti G, Fadiga L, Gallese V, Fogassi L (1996a) Premotor
cortex and the recognition of motor actions. Cogn Brain Res 3:
131–141
Rizzolatti G, Fadiga L, Matelli M, Bettinardi V, Perani D, Fazio F(1996b) Localization of grasp representations in humans by

140
PET: 1. Observation versus execution. Exp Brain Res 111:
246–252
Rizzolatti G, Luppino G, Matelli M (1998) The organization of thecortical motor system: new concepts. Electroencephalogr Clin
Neurophysiol 106: 283–296
Rosenbaum DA, Meulenbroek RG, Vaughan J, Jansen C (1999)
Coordination of reaching and grasping by capitalizing on
obstacle avoidance and other constraints. Exp Brain Res 128:
92–100
Roy AC, Paulignan Y, Farne A, Jouﬀrais C, Boussaoud D (2000)
Hand kinematics during reaching and grasping in the macaquemonkey. Behav Brain Res 117: 75–82
Rumelhart DE, Hinton GE, Williams RJ (1986) Learning internal
representations by error propagation. In: Rumelhart DE,
McClelland JL (eds) Parallel distributed processing, vol 1:
Foundations. MIT Press, Cambridge, Mass., pp 151-193
Russ JC (1998) The image processing handbook. CRC, Boca
Raton, Flo.

Sakata H, Taira M, Murata A, Mine S (1995) Neural mechanismsof visual guidance of hand action in the parietal cortex of themonkey. Cereb Cortex 5: 429–438
Sakata H, Taira M, Kusunoki M, Murata A, Tanaka Y (1997)
The TINS lecture – The parietal association cortex in depth
perception and visual control of action. Trends Neurosci 20:
350–357
Sonka M, Hlavac V, Boyle R (1993) Image processing, analysis andmachine vision. Chapman and Hall, London, NY
Stein JF (1991) Space and the parietal association areas. In:
Paillard J (ed.) Brain and space. Oxford University Press,
Oxford, pp 185–222
Taira M, Mine S, Georgopoulos AP, Murata A, Sakata H (1990)
Parietal cortex neurons of the monkey related to the visual
guidance of hand movement. Exp Brain Res 83: 29–36
Wolpert DM, Ghahramani Z (2000) Computational principles of
movement neuroscience. Nature Neurosci 3: 1212–1217