Cognitive Brain Research 22 (2005) 129 – 151
www.elsevier.com/locate/cogbrainres

Research report

Mental state inference using visual control parameters
Erhan Oztopa,b,*, Daniel Wolpertc, Mitsuo Kawatoa,ba

JST-ICORP Computational Brain Project, 2-2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0288, Japan
ATR, Computational Neuroscience Laboratories, 2-2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0288, Japanc
Sobell Department of Motor Neuroscience, Institute of Neurology, University College London, Queen Square, London WC1N 3BG, UKb

Accepted 11 August 2004
Available online 11 September 2004

Abstract
Although we can often infer the mental states of others by observing their actions, there are currently no computational models of thisremarkable ability. Here we develop a computational model of mental state inference that builds upon a generic visuomanual feedback
controller, and implements mental simulation and mental state inference functions using circuitry that subserves sensorimotor control. Our
goal is (1) to show that control mechanisms developed for manual manipulation are readily endowed with visual and predictive processing
capabilities and thus allows a natural extension to the understanding of movements performed by others; and (2) to give an explanation on
how cortical regions, in particular the parietal and premotor cortices, may be involved in such dual mechanism. To analyze the model, we
simulate tasks in which an observer watches an actor performing either a reaching or a grasping movement. The observer’s goal is to estimate
the dmental stateT of the actor: the goal of the reaching movement or the intention of the agent performing the grasping movement. We showthat the motor modules of the observer can be used in a dsimulation modeT to infer the mental state of the actor. The simulations with different
grasping and non-straight line reaching strategies show that the mental state inference model is applicable to complex movements. Moreover,
we simulate deceptive reaching, where an actor imposes false beliefs about his own mental state on an observer. The simulations show thatcomputational elements developed for sensorimotor control are effective in inferring the mental states of others. The parallels between themodel and cortical organization of movement suggest that primates might have developed a similar resource utilization strategy for action
understanding, and thus lead to testable predictions about the brain mechanisms of mental state inference.
D 2004 Elsevier B.V. All rights reserved.
Theme: Neural basis of behavior
Topic: Cognition
Keywords: Reaching; Grasping; Cortex; Theory of mind; Mental simulation

1. Introduction
Although dmental state inferenceT, more generally
dtheory of mindT, has been a recent topic of interest incognitive neuroscience (see Refs. [4,9,26,28,83]), there areno computational models that go beyond conceptual arguments and none that offer testable predictions on the natureof mental state inference [27]. On the other hand, dmotor* Corresponding author. ATR, Computational Neuroscience Laboratories, Department of Cognitive Neuroscience, 2-2-2 Hikaridai, Seika-cho,
Soraku-gun, Kyoto 619-0288, Japan. Tel.: +81 774 95 1215; fax: +81 774
95 1236.
E-mail address: erhan@atr.jp (E. Oztop).
0926-6410/$ - see front matter D 2004 Elsevier B.V. All rights reserved.
doi:10.1016/j.cogbrainres.2004.08.004

imageryT or dmental simulationT has been widely studied bycognitive scientists, and recently by cognitive neuroscientists shedding light on the neural basis of motor imagery[12,47,48,61,66,67]. Our goal is to contribute to mental stateinference and motor imagery research by developing acomputational model that starts from a visuomanual control
mechanism and can be extended into a mental simulation
system allowing mental state inference, while conforming tohuman brain imaging and monkey neurophysiology data.
The first key observation for our proposal, that control
mechanisms for movement generation can be used to makeinferences about others’ behavior, is that a goal-directedmovement can be formulated as a feedback control problem.
For example, reaching a target in space requires, at least,

130

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151

minimization of the distance between the hand and the target.
Humans fixate certain landmarks critical for the control ofgrasping, such as where contact events take place, and thetime when gaze exits a given landmark coincides withkinematic events at that landmark [44]. This suggests thathumans monitor kinematic events to detect errors inexecution. It is not clear to what extent the visual feedbackis used during grasping, as there are contradictory results inthe literature [10]. In primates, the exact form of the features
extracted by the cerebral cortex for visual feedback control is
not known. However, the parietal cortex appears to be
involved in visuomotor aspects of manual manipulative
movements [1,7,21,20,30,46,53,82,90]. The feedforward
control, we assume, is a skill learned by self-observationof feedback-controlled movements, which involves inverse
model learning (e.g. feedback error learning), and is notaddressed in the current study (but see Refs. [52,92]). The
parietal areas with specific connections to the frontal lobe
mediate distinct sensorimotor transformations related to thecontrol of hand, arm, eye or head movements and spaceperception. The activity of the fronto-parietal circuits isthought to control actions requiring spatial processing [72]. It
is suggested that the premotor cortex may play a preferential
role in sensory or context-dependent processing related totask performance, whereas motor cortex may be more
involved in processing related to the purely motor aspectsof task performance [77]. In addition, some premotor cortex
neurons (mirror neurons) appear to be involved in both
movement generation and also action understanding [29,70].
Our model structure is based on the hierarchical view[36] of primate visuomotor control with visual features
being extracted by parietal cortex and used by the premotor
cortex to generate high-level motor signals that drive lower
motor centers thereby generating movement. The functionalframework is developed based on mirror neuron findings
from monkey and human brain imaging data, constrained bycomputational requirements. According to the model, action
recognition is mediated by unconscious or implicit mental
movement simulation that is implemented in the premotor
cortex, which subserves mental state inference.
There exists considerable evidence that humans areendowed with a mental simulation capability mediated by
the parietal and premotor cortices [31,48]. Chronometricstudies suggest that viewers decide if the presented stimulus
is a left or right hand, by engaging implicit motor imagery, asimulation that respects the rotational constraints of theobserver’s upper limbs [65]. In addition, humans use implicit
mental simulation when asked to decide how they will graspa handle appearing in a variety of orientations (over-hand vs.
under-hand), and subjects’ choices are in agreement with thegrips chosen during actual grasping [45]. Furthermore,experiments utilizing PET imaging have shown that similar
to actual and imagined movements, premotor, parietal andcerebellar regions (but not primary somatosensory and motor
cortices) are strongly activated during implicit mental
simulation [66] supporting our model. Although in thecurrent study, the utilization of mental imagery for mental
state inference was the emphasis, mental simulation mechanisms can also play an important role in motor planning as
suggested by the Prospective Action Model (PAM) in whichmovement alternatives are simulated and evaluated todetermine the most biomechanically efficient movement
for execution [45,48]. Note that the mental simulation we arereferring to is continuous (analog) in nature, and is a different
process than dvisual mental rotationT [11,40,78] that does not
involve motor imagery.
For different tasks, a multiplicity of specialized controllers
may be acquired. A structure similar to the MOSAIC model, a
modular adaptive controller, can be involved in such task
learning [39,95]. In this study, we investigate the mechanismsthat allow exploitation of motor circuits for mental stateinference rather than learning. As will be elaborated later, thekey to this mechanism is the existence of internal models.
Although theoretically perfect control does not necessarily
require any identifiable internal models [57], current evidence suggests that the central nervous system uses internal
models for transforming sensory stimuli into movement; i.e.
in planning, control and learning [51]. Internal models thatpredict the sensory consequences of a motor command are
known as forward models as they can be used to predict how
the motor system’s state changes in response to a given motor
command [94,95]. In a more general sense, a forward model
is a mechanism to predict a future event, given the set of
inputs such as the motor command and current state that canaffect the system’s future. Mehta and Schaal give an excellent
review of possible control schemes for the human brain withand without internal models and show with their experimental paradigm that visuomotor control involves at least one
forward model [57]. The role of forward models in control is
twofold: eliminating sensory processing delays [59,93] andmediating inverse model learning for establishing a feedforward control strategy [14]. Our model extends the function of
the forward model beyond motor control, highlighting the
key role played by the forward model in decoupling themental movement execution from the actual one.

2. Methods
We now introduce our model by describing a genericvisual control circuit and then show how it can be adapted
into a mental simulation and mental state inference circuit.
In Discussion, we elaborate on the connection between thecortical movement and action recognition areas and themodel structures, which together with simulation results
lead to predictions that can be tested experimentally.
2.1. Generic visual feedback circuit for goal-directedmovements
Based on experiments on goal-directed hand movements,
we propose the following neural processing steps. The

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151

parietal cortex extracts visual features relevant for thecontrol of a particular goal-directed action (X, called thecontrol variable) and relays this information to the premotor
cortex. For example, by taking X as the distance between theindex finger and a given target location, the circuit drives areaching movement. On the other hand, if we take X as thesum of the distance between index finger and its target
surface, and the thumb and its target surface, we have acircuit for grasping [80,81]. The premotor cortex computes
the motor signals to match the parietal cortex output (X) to
the desired neural code (X des) relayed by prefrontal cortex.
We assume that premotor cortex is an adaptive circuit that
outputs movement parameters (e.g. desired change in bodyconfiguration) to achieve the goals represented by theprefrontal and parietal cortices. The ddesired changeT
generated by the premotor cortex is then relayed to theprimary motor cortex and spinal cord for execution (Fig. 1).
We acknowledge the involvement of the basal ganglia andthe cerebellum in learning and dynamics control but do notinclude them in the model [33,43,50,76,92]. Although we
did not simulate the dynamics and the delay—hence did notneed a complex forward model, we assume that the sensory
forward model (FM) (Fig. 1) helps feedback control of
movement by eliminating delays in sensory feedback by
providing future sensory signals to the premotor cortex. In
general, the proposed FM needs to learn the composite
mapping of the dynamics controller, the controlled body andthe control variable computation and thus needs to accessmore resources than we have simulated (e.g. dynamics stateof the body). Kinematics and dynamics transformations ofhigh degrees-of-freedom mechanical system (humanoid
robots) can be successfully learned with recent learning131

architectures in an online fashion [86], thus we believe thebrain could acquire such a powerful FM as suggested
elsewhere [57,59]. Note that at this level, we define theforward model in functional terms. In a biological system,
multiple cortical structures may be involved in implementing such a composite forward model (see Discussion).
The model architecture proposed is orthogonal towhether a multiplicity of parallel circuits or a circuit withdifferent instantiation parameters allows deployment ofdifferent control strategies. Depending on the intentions ofan organism (e.g. grasping, tearing, etc.), the functionality ofthe parietal and premotor cortices is dswitchedT or
dmodulatedT by the prefrontal cortex providing the operational space (extrinsic, kinematic-based) signals and thevisual control servo required for the desired task execution.
Thus, the parietal and premotor cortices form a set of taskspecific operational space control mechanisms, which areengaged according to prefrontal influences. Our view is inaccordance with the experimental findings that control
strategies for goal-directed movements differ according to
the task in which the subject is involved [87].
2.2. Exploitation of control variables and mental stateinference
The second key observation that is crucial for the modelis that the time course of a control variable (e.g. distance)
extracted for visual feedback control of movement can bereadily used to recognize others’ movements if the variable
is invariant under translation and rotation. The invariancecan be embedded in the parietal processing (i.e. built in thecontrol variable) or an explicit coordinate transformation

Fig. 1. The visual feedback control mechanism proposed for goal directed actions is shown. The context and internal drives culminate in a mental state,
dwanting to do somethingT in prefrontal cortex which modulates parietal cortex for extraction of the required control variables to fulfil the goal. Premotor cortex(movement planning) receives signals from parietal cortex and prefrontal cortex, and computes the movement signals required to achieve X des=X. The
movement execution (lower motor areas: the primary motor cortex, spinal cord, the basal ganglia and the cerebellum) implements instructions relayed by the
premotor cortex.

132

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151

circuit can preprocess sensory inputs and relay the output tothe parietal cortex. For example, the angle between twoplanes in space is invariant under translation and rotation.
Along these lines, it has been shown that it is possible torecognize grasping movements and classify them into
different grip types [64].
Thus, we propose that neural circuits evolved for goaldirected manipulation skills (e.g. breaking a nut, holding acup) not only provide dexterity in task execution but also
mediate understanding of another’s intentions if used in amental simulation mode. The key element for this extended
function is the dual role of the FM that is mediated byinhibitory mechanisms. During task execution, the parietal
cortex, the premotor cortex and FM implement a visuomanual servo, where the FM compensates for visual feedback
delays. During mental state inference, the FM is utilized for
creating imaginary parietal signals based on the current
mental state estimate, which is decoupled from the action of
the observer, while the parietal cortex processes the visual
stimuli created by the observed movement. In brief,
behaviors are generated by the actor’s motor control system
which is internally parameterized by intention (not visible toan observer). The observer, by simulating the motor control
system with an initial estimate of this internal parameter, cancompare its simulated output with the observed act,
generating an error measure that allows updating of theobserver’s internal parameter until there is a good match
between the actor’s behavior and the observer’s prediction.
In the simulations we present in this paper, we analyze twokinds of mental state estimate correction. The first method isapplicable only when there is a discrete set of possible
mental states. This method simply finds the most likely
estimate by rejecting mental state hypotheses that lead tolarger errors. The second correction method is based onstochastic hill climbing, and it finds an approximate gradientin the (continuous) mental state space using stochastic
dmental state perturbationsT.
Fig. 2 illustrates the extension of the visuomanual control
circuit presented in Fig. 1 into a mental state inference
system. This extension represents a conceptual model ofprimate exaptation (the utilization of a structure for afunction other than that for which it was developed throughnatural selection) of the sensory forward model (FM)introduced in Fig. 1.
The basic processing steps carried out in the model canbe written as a cascade of transformations, with some of thetransformations depending on the current (and hypothesized) mental state of the system, which we index by j (andi). The transformations and variables are:
MP
DC
FD
CV
FM
X jn

movement planner
movement execution (dynamic controller)
body forward dynamics
control variable computation
forward model
actual control variable for mental state j at time nnpredicted value of the control variable at time n for
X j,predmental state jn
X j,observed
observed control variable for mental state j attime ndelay time delay to convert from movement plan to sensecontrolled variablehnj
state of the body
Dhnj desired change in the body state at time step n.

When the model is in the actor mode with mental state j:n1(1) X jn,pred =FMj (Dhn1, X j,pred)j(2) Dh jn =MPj (X jn,pred, X jndelay)(3) h jn+1=FD(DCj (Dhnj +hn1, hn1))jj(4) X jn+1=CVj (hn+1j )The forward model predicts thecontrol variable for the next time
step and cancels out the delay
involved in CV.
The motor planning generates
movement signals.
Dynamics controller fulfills hn+1
j i
hnj +Dhnj .
The result of the change in the
hand–object relation is captured
in control variable computation
which is approximated by FM(step 1), completing the operational space feedback loop.

When the model is in the observer mode with the
hypothesis of i as the actor’s mental state(1) X in,observed =CVi (Actor)(2) Mental simulation (m=1,. . .,n)mm 1(a) X i,pred=FMi (Dh im1, X i,pred)m(b) Dh im =MPi (X i,pred)

The control variable from theobservation of the actor is
extracted using CV of theobserver but directed to the
movement of the actor.
Predict the control variable ofthe next time step.
Generate movement signal to be
used by step (2a).

Then, the task of mental state inference in the observer is
to search over mental states (i) that will produce minimal
12
nmismatch between [X i,observed, X i,observed,. . ., X i,observed
] and
1
2
n
[X i,pred, X i,pred,. . ., X i,pred]. The mental simulation equations(2a and 2b) do not involve any movement and visual
processing, thus can be executed very rapidly.
Note that our simulations analyzed the observer; the actor
was implemented without delay and dynamics. In general,the dynamics require the h in the above equations to include
the time derivate (i.e. h D– (h,ḣ), and the dynamics control
loop (actor case step 3) can be run at a higher rate than theoperational space control loop, or alternatively, it can be
replaced with an inverse dynamics controller. Another issue
is that the forward prediction must be initialized to matchthe real control variable prior to operation. This initialization
can be accomplished by having no movement for a period of
0,1,. . .,delay
0,1,. . .,delay
delay or more (i.e. X j,pred=X j0 and Dh j,pred=0).
To give a concrete example, we use a reaching task andtrace the functioning of the model (see Fig. 2). Imagine an

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151

133

Fig. 2. The extension of goal-directed visuomanual controller into a dmental state inferenceT system can be seen by comparing the observer’s additional
circuitry (lower part) to the actor’s. The slashes (z) indicate inhibition or blockade of motor execution. The difference module computes the difference between
the control variables of the simulated movement (based on the current estimate of the mental state of the actor) and the actual observed movement.

actor being asked to freely select and reach to one of a set oftargets in space. The observer’s task is to guess to whichtarget the actor is reaching as soon as it can. Given this
scenario, we first describe what happens in our model fromthe actor’s viewpoint then we elaborate on observer’s
mental state estimation.
The type of movement (i.e. reaching) and its goal
constitute the mental state of the actor. During movement,
Parietal Cortex computes the distance between the target
and the hand while Premotor Cortex generates target joint
angles to reduce the distance information relayed by Parietal
Cortex. Low-level motor centers convert the output of
Premotor Cortex to neural signals that drive muscles
causing movement, which in turn modifies the parietal
distance signal. The output of Premotor Cortex is also
relayed to the FM to predict the sensory outcome that is
integrated with parietal output by premotor cortex tocompensate for delays and temporary interruptions in thesensory processing (e.g. due to occlusion). The acquisitionof the FM is mediated by the outputs of Premotor Cortex
and parietal cortex, which we assume has taken place

already prior to the reaching task. The visual feedback loop
continues until the desired zero-distance is achieved, driving
the hand towards the target specified by the mental state of
the actor.
The observer’s aim is to infer the goal of the actor’s
movement and starts with some hypothesis in the estimatedmental state. The premotor cortex does not receive input
from the parietal cortex and instead relies only on the FM
output to generate movement signals as observer’s parietal
cortex is engaged in computing the hand–target distancepertaining to actor’s movement. The FM receives the output
from the premotor cortex and computes the predicted
outcome (X pred) implementing the mental simulation loop.
Output of the observer’s premotor cortex is inhibited so thatthe mental simulation does not trigger overt movement. The
broken links (z) in Fig. 2 indicate the switch between theovert movement and simulated movement mode. The
difference module compares the stream of sensory signals
based on real movement (X) and mentally simulated
movement (X pred), and produces an error signal that is used
for modifying the current hypothesis about the actor’s goal.

134

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151

In the following sections, we present the simulation
environment and the experimental settings. In brief, thesimulation experiments (SE) conducted are: the straight-line
reaching simulations (SE1), where it is shown that themodel can perform mental state inference with stochastic
gradient descent in a continuous mental state; the spiralreaching simulations (SE2), where it is shown that themental state inference model is not limited to deasy to
predictT mental state inference tasks and can be generalized
to complex goal-directed movements as long as the control
is specified with respect to the goal; and the deceptive
reaching simulations (SE3 and SE4), aim at showing thesituation where the actor and observer have different
movement strategies. In this case, the mental state representations of the actor and the observer can be very
different. Finally, a tool-use simulation (SE5) shows thatthe mental state inference is applicable to grasping movements and a parallel implementation can be used to generatea probability distribution over the set of possible intentions
afforded by the context.
2.3. Simulation environment
We simulated two experimental setups. The first one
involves two agents sitting and facing a board on which
there are several target locations. As one agent reaches for atarget, the other’s task is to guess the first’s intended target
as rapidly as possible. The second setup involves a tool
(hammer) that can be grasped in several ways. The
observing agent watches another agent grab the tool andhas to infer the agent’s intention. In the former case, we
modeled the arm as a two-link kinematics chain, whereas in
the tool-use case, the hand was also included as the endeffector. The hand/arm is modeled to have 3DOF joints atthe shoulder to mimic a human shoulder ball joint and1DOF joint in the elbow for lower arm extension/flexion
movements (see Fig. 3). The wrist is modeled to have
3DOFs to account for extension/flexion, pronation/supina-tion, and ulnar and radial deviation movements of the hand.
Each finger has 2DOF whereas the thumb has 4DOF. The
finger and thumb joints are used to implement an open loop
enclosure of fingers at hand–object contact. In each case, theactor and observer models dknowT how to perform reaching
and grasping movements, implementing the control mechanism shown in Fig. 1.
2.4. Task implementations
Since the dynamics of the arms are not modeled, thedynamics control box shown in Fig. 1 is the identity
mapping for the simulations presented in this paper. The
task of the movement-planning module (the premotor
cortex) is, therefore, to plan a trajectory in three-dimensional space. The parietal cortex computes the control
variables required for task execution and observation. In
general, the forward model is a neural network that learns agoal-directed transformation (e.g. orientation with respect toan external reference) of the operational space forward
dynamics of the controlled body and the controller. In thissimulation for computational convenience, we use the exact
mapping instead of a neural network. The FM module
receives the joint angles as input and output (distance) for
reaching and (distance, orientation difference) for tool-use(grasping).
2.4.1. Reaching task implementations
In the first set of reaching simulations (continuous mental
state space), the mental state of each agent is modeled as thethree-dimensional coordinate of a target on the board plane(Fig. 4A). In the second set of reaching simulations (discretemental state space), the mental state is modeled as theidentity of the reach target (an integer number indicating atarget).
In the straight reaching case, agents produced a straightline hand path to reach a target. For spiral reaching, we
relaxed the straight path requirement, allowing spiral paths

Fig. 3. The arm/hand model has 19 DOFs (left). Zero posture of the arm/hand and the positive rotations of the arm are shown (right).

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151

135

Fig. 4. On the left (A), the reaching task simulation setup is shown (the spiral reaching paths for the eight targets are also shown). The demonstrator model
reaches for one of the targets (small squares) while the observer model tries to infer demonstrator’s goal by dmental simulationT. On the right (B), threeintentions of grabbing a hammer are shown. Left to right: hammering a nail (nailing task), pulling off a nail (prying task) and putting the hammer away
(holding task).

parameterized by the current position of the end-effector.
Finally, the deceptive reaching involved selection of a dfakeTtarget, which altered the reaching movement kinematics todeliberately deceive an observer (see Appendix A for
details). In reaching movements, to account for possible
inaccuracies during perception, zero-mean Gaussian noisewith a variance of 25 was added to the parietal distancecomputation. At the onset of movement, the end-effector–
target distance was 700–800 units depending on the target,and inter-target distance was 350 units vertically andhorizontally.
2.4.2. Tool-use (grasping) movement implementations
The tool-use experiment is set up with a hammer and two
agents that can grasp the hammer with three different
intentions. When a hammer is grasped by the handle so thatthe metal head points upwards at the end of movement is
considered as an intention of driving a nail (nailing task)
while a grasping movement that results the metal headpointing down is considered as an intention of pulling out anail (prying task). Finally, grasping the hammer merely
from its metal head is considered as an intention of putting it
away (holding task). The grasping movements mentioned
are illustrated in Fig. 4B. The tool-use experiment modeled
mental state as the intention of grasping the hammer(holding, nailing or prying a nail). The grasping movements
prior to driving and prying a nail require differentialalignment of the hand and the thumb. The knuckle vector(the vector pointing away from the little finger knuckle
towards the index finger knuckle) must point towards themetal head of the hammer prior to nailing while it has topoint just opposite prior to pulling out a nail. We haveassumed that a hammer is approximately grabbed from thecenter of the handle in these two grasps. The final grasp
indicative of holding rather than driving or prying a nail, ismodeled as the hand’s bottom approach towards the centerof the metal head while maintaining a vertical approach tothe hammer plane (plane spanned by the handle and themetal head). Appendix A provides details of these grasps.
2.5. Mental state inference implementation
As the control variables described for reaching andgrasping movements are defined in extrinsic space withrespect to target, the need for an explicit coordinatetransformation circuit for mental state inference is eliminated. Hence, the history of the control variables can beused to compare two movements (i.e. simulated and real
movements). For example, straight reaching requires thecontrol of distance between the end-effector and the target.
In general, the details of how the distance is reduced to zerodepend on an agent’s movement strategy as well as theagent’s intrinsic and extrinsic properties. With the assumption that both agents have similar mechanisms, we expect(in spite the variations in the agents) the different goals will
yield different distance trajectories, so that mental statecorrection can be performed using mentally simulated
movements. Similarly, for hammer grasping, we expect thatthe distance and orientation difference trajectories can beused to infer the intention of an agent grasping a hammer.
Note that although object centered formulation of thecontrol problem avoids explicit coordinate transformation,in general, some normalization might be required, as is thecase for distance (but not for orientation difference).
Distance normalization can easily be based on an initial
estimate of the distance of actor’s hand to a target. In thesimulations presented here, the actor and the observer wereplaced equidistant from the target board eliminating theneed for distance normalization.
If an observer model dknowsT the possible mental states(goals) of an actor as discrete entities, then it can perform
an exhaustive search in the mental state space. However, if
the number of possible mental states (and related targets) is
infinite, or the targets are not visible to the observer, then a136

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151

different search strategy must be applied. From a computational point of view, the mental state correction requires
the dcontrol variable differencesT (the difference box Fig.
2) to be converted into dmental state space adjustmentsT.
Although the current difference module implementation
does not perform scaling or any other advanced manipulations to compare two input time series, the differencecomputation involves an integral over sensory signals
produced during a feedback loop. Therefore, it is nottrivial to formulate a general deterministic gradient descentalgorithm. Nevertheless, we can perform stochastic dhill
climbingT in the error space. We have implemented boththe exhaustive and the stochastic gradient search methods,
for which we present algorithms, assuming that parietal
cortex (feature extraction/control variable) processing isserial (i.e. it can process the visual input related to a single
target at a time). The parallel version is straightforward
and not included here.

continuous metric space (e.g. in the reaching task, themental state is modeled as the two-dimensional open set
defined by the board borders.)
Algorithm. Stochastic gradient descent mental state search(for a continuous mental state space)
(1)

(2)

Algorithm. Exhaustive mental state search (for a mental
state space with M discrete elements)
(1)

(2)

(3)

Initialize: Set T k and S k to empty sequence(T k =S k =[]). Here T k and S k represent sequences ofobserved and mentally simulated vectors of visual
control variables extracted under the mental state k.
Repeat steps a–e from movement onset to movement
end.
(a) Pick next possible mental state ( j) (which can bethought of as an index to the possible target theactor is reaching to).
(b) Observe: extract the relevant control variables
based on the hypothesized mental state ( j), xij andadd to T j (T j =[T j ,xij ]). Here the superscript
notation i indicates that the collected data wasplaced in ith position in the visual control
variable sequence.
(c) Simulate: mentally simulate movement withmental state j while storing the simulated control
variables x j in S j (S j =[x0j ,x1j ,. . .,xNj ], where N is
the number of control variables collected during
movement observation).
(d) Compare: compute the discounted difference
between T j and S j , where N is the length of
T j and S j . P 

 i

ð1cÞ
N
i
i T
DN ¼ ð1c
W xsim xi cN i ,N þ1 Þ
i ¼ 0 xsimx
i aS and x i aT and W is a diagonal
where x simjjmatrix normalizing components of x i , and c is thediscount factor.
(e) If D N is smallest so far, set j min=j.
Return: j min (observer infers that j min is the actor’s
mental state).

In general, a mental space may not be indexed and finite
and thus the above algorithm cannot be applicable. The
following algorithm assumes that the mental space is a(3)

Initialize: Set A to an initial random mental state guess;set MaxIter to a positive integer (e.g. 20 for reachingtask); set D last=1e20 (arbitrarily large number).
Repeat a–c from movement onset to movement end.
(a) Observe: Add actor’s current kinematics (e.g.
end-effector position) in sequence K.
(b) Loop MaxIter times.
(i) Generate a random perturbation DA.
(ii) Apply the perturbation to A: A=A+DA.
(iii) Recompute: the vector of control variables
for mental state A using K and store in thesequence TA (TA=[x 0,x,. . .,x N ]). Here N is
the length of K, and the superscript notationi indicates that the data was placed in ith
position in TA .
(iv) Simulate: mentally simulate movement
for mental state A and store the vector ofcontrol variables generated during imagery
movement in S A .
ð1cÞ PN  ii T(v) Compare:D
N ¼ ð1c N þ 1 Þi¼0 xsimx i
W xsim  xi cNi, where xisimaS A andx i aTA and W is a diagonal matrix normalizing components of x i , and c is thediscount factor and N is the size of TA andSA.
(vi) If D lastbD N then undo with %20 penalty:
A=A1.2DA and go to step i.
(vii) If randb0.1 go to step i (even if theperturbation was good, with small probability make a random move).
(viii) Go to step ii (if the perturbation was good,use it once more).
Return: A (observer infers that A is the actor’s mental
state).

The discount factor for difference computation (D N ) waschosen as c=0.9 for all the simulations. The parameter callows forgetting of the early (and ambiguous due to noise)
portions of the movement observation. The normalization
matrix W was unit diagonal matrix in the reaching experiments and diag(0.7,0.3) in the tool-use grasping experiments. In general, W controls the relative weight of featuresappearing in the difference computation (e.g. distance andorientation difference).
The exhaustive search algorithm implements a serialparietal cortex that can dattendT a single target at a time
where the potential mental states are iterated seriallykeeping record of the best so far. For tool-use simulations,the parallel version of the exhaustive search algorithm isE. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151

used which does not cycle through the potential states but
computes all at the same time in a parallel fashion. The
output of the parallel difference computation (D N ) is thenconverted into beliefs using the softmax function:
XM
jipðAi jobservationÞ ¼ e20DNi
e20DNij¼1

where A i represents one of M mental states.
Certainly, a circuit capable of parallel processing
computes faster; however, the requirement of feature
extraction from multiple visual targets simultaneously(i.e. control variable computation for multiple goals) raises
the question of how many visual targets a biological agent
is able to monitor at the same time. This issue is related toattention and capacity of working memory, which we do
not elaborate here as it is beyond the scope of this paper.

3. Results
We first present simulation experiments of straight line(SE1) and spiral reaching (SE2) that use gradient search formental state inference (see Methods for details of thesimulations) where the actor and the observer have similar
movement strategies. In these cases, the mental state space
was modeled as the set of points on a bounded planesegment (the dboardT). The actor reached for fixed targets on
the board; however, the observer was not aware of the fixed
targets. Therefore, the observer used a continuous mental
state (target location on the board) search to infer the goal ofthe movement.
3.1. SE1: straight-line reaching actor vs. straight-line
reaching observer (continuous mental state search)
In this simulation, there are eight targets placed on theboard as shown in Fig. 4A (the center target is not used),
which are only visible to the actor.
Each target is equidistant from its vertical andhorizontal neighbors (350 units). The mental state searchrequires an initial guess of the actor’s mental state. In
general, the initial guess can be guided by the context orother cues. Here we initialized the estimated mental stateas the center of the target board. Fig. 5A shows the
trajectory of the mental state of the actor that represents
the goal location (gray line) which jumps from the initialcentral location to the target, and the predicted target of theobserver (black trace). Although the estimates of mental
states have variability (arising from noise on the visualfeedback and the stochastic nature of the search), thetrajectories arrive at the correct mental state. Fig. 5B
illustrates the temporal aspect of the mental state search,showing the distance between the observer’s estimate of
the actor’s goal and the actual goal of the actor as afunction of simulation time. The target of the movement

137

can be reliably estimated midway through the movement.
Note that here we present typical convergence patterns.
The speed of correct inference depends on the ambiguityof movement patterns, the noise level in the control
variable extraction and the number of cycles the stochastic
gradient descent search algorithm is run per observation
data point (see Methods for details). Although simple, thesesimulations show that the time course of the control variable
used in movement generation (distance) can be used toassess the similarity of a mentally simulated movement andobserved movement.
3.2. SE2: spiral reaching actor vs. spiral reaching observer(continuous mental state search)
Although reaching is not a very rich behavior in terms ofgoal-directedness, a general goal-directed reach can be
defined in the reference frame of a target. In this simulation
experiment, we show that the capacity of mental stateinference model is not limited to simple straight-line
reaching but can be generalized to general goal-directedmovements. We implement a spiral reaching movement
where the path followed by the actor is defined with respect
to the board rather than specified egocentrically. Intuitively,
this means that the turns of the spiral are not specified as,say, left to right but as from target 2 towards target 1 (seeMethods for more details). Fig. 6A shows the successful
mental state search patterns when the observer and actor use
the spiral reaching strategy (see Fig. 4A). As in all gradientmethods, the search suffers from local minima. Comparedto the straight line reaching, the spiral reaching presents
more local minima and more susceptibility to noise. The
central panel in Fig. 6B shows the best convergence andworst convergence patterns normalized and averaged overeight targets; in the worst case, the correct inference isdelayed. In spite of the local minima, our simulations showthat the time course of control variables used in movement
generation can be used to assess the similarity of amentally simulated movement and observed movement
even if the set of possible targets is unknown. Our
simulations also showed that a naive straight-line reachingobserver could not estimate the target of an actor’s goal
when the actor performed a spiral reaching (or vice versa)
since the mental simulation was very different from actor’s
movement. These simulation results are not shown for they
are similar to the results for deceptive actor vs. naive
observer case, which we present next.
3.3. SE3: deceptive reaching actor vs. straight-line reachingobserver (discrete mental state search)
Next, we present a series of simulations (SE3, SE4)
using exhaustive search method where the actor and theobserver can have different movement strategies (straightline and deceptive reaching). In these simulation experiments, the mental state space is modeled as finite (i.e. the138

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151

A

vertical position on the board

reach targets are fixed locations on the board) as illustrated
in Fig. 4A.
As a first step for information transfer between agents, we
introduced a reaching strategy by a ddeceptiveT actor who
initially chooses a fake target to aim for with the intention ofdeliberately forming a false belief (mental state estimate ofhis) in the observer’s mind. We expected that the first portionof the deceptive actor’s movement would fool a dnaiveT
observer who uses a straight-line reaching strategy until thelater portions of the movement. The simulation results
showed that the dnaiveT observer failed to estimate the goalof the actor’s reach since a straight-line reaching strategy
controls only the distance to a (real) target. In other words,
the mental state of a naive observer was lacking an importantcomponent, the possibility of the existence of a fake target.
By using a board with four targets, we can examine all
possible deception scenarios: when the actor chooses one offour real goals and therefore has three possible fake goals(leading to 12 possible scenarios). The movements arisingfrom all possible mental states of the deceptive actor are
depicted in the sub-panels in Fig. 7 (dashed line real target;
dotted line fake target). The vertical axis indicates the target,
whereas the horizontal axis represents simulation cycles
from movement onset to movement completion. The beliefof the observer about the actor’s reaching goal is superimposed as a solid line showing that the actor was deceived
during the deceptive movement observation (the overlap of
the solid and dotted lines).
3.4. SE4: deceptive reaching actor vs. deceptive reachingobserver (discrete mental state search)
To verify that SE3 is indeed a deception case, we
checked the goal estimation capability of an observer withthe same deceptive strategy as the actor. Thus, the observer
and actor had the same mental state space representation and

400

400

400

200

200

200

0

0

0

200

200

200

400
400

200

0

200

400

400
400

200

0

200

400

400
400

200

0

200

400

200

0

200

400

200

0

200

400

horizontal position on the board
400

400

200

200

0

0

200

200

400
400

200

0

200

400
400

400

400

400

400

200

200

200

0

0

0

200

200

200

400
400

200

0

200

400

400
400

200

0

200

400

400
400

Fig. 5. Panel A shows the simulation of the stochastic gradient descent is shown (straight-line reaching actor vs. straight-line reaching observer.) The straightline in each sub-panel indicates the actual goal of the actor. The black traces show the prediction of the observer. The initial estimate of the observer at the
movement onset of the actor is taken as the center of the board. Panel B shows the distance of the estimated target to the real target of the actor as a temporalplot. In addition, the center plot (in Panel B) shows the error as normalized and averaged over the eight trials.

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151

139

Fig. 5 (continued).

therefore the mental state search of the observer involved
finding out the fake target as well as the real target. The
result of this simulation set is shown in Fig. 8 with the same
conventions as in Fig. 7, except this time we used a sixtarget board to make the goal inference harder. The
movements get more ambiguous with more targets because
the trajectory for each movement becomes closer; hence, theeffect of the modeled noise becomes more severe (seeMethods). Nevertheless, the observer of this simulation cancorrectly estimate the mental state of the actor and thus infer
the goal of his movement much earlier than the naive
observer of previous simulation experiments (SE3).
3.5. SE5: tool-use simulations
The last simulation experiment (SE5) involves graspingwhere the mental state is modeled as the intention of graspinga hammer, either driving a nail, prying a nail or holding thehammer to put it away (Fig. 4B). The intention of an actor
determines what policy will be used to control the distanceand orientation of the hand with respect to the hammer toprovide a successful grasp. One of the important points that
we did not emphasize in the reaching simulations was what
the parietal cortices of the agent and the observer werecomputing during control and observation. In fact, when the
deceptive actor was generating a deceptive reaching, itsparietal cortex was computing the distance to both fake andreal targets whereas the naive observer was computing asingle distance for mentally simulating the movement. Sincevarious mental states may require different control mechanisms, the control variables that need to be computed dependon the current estimate or dhypothesisT of an observer. We
present the tool-use simulation results using a scheme toemphasize this fact as follows. We construct a time-varying(mental simulation)(observation) matrix to understandthe full dynamics of the observer (Fig. 9). The rows of thematrix represent the intention of the actor: (holding, nailing,prying). The columns represent the possible beliefs of theobserver: (holding, hammering, prying). Then each cell isused to show the similarity between mentally simulated
movement and the observed movement as a function ofsimulation time from movement onset to movement end.

140

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151

The similarity between observed and mentally simulated
control variables are not directly compared to find the most
likely mental state (as was case for reaching simulations) but
instead converted into a set of probabilities (beliefs).
Therefore, at any time step, each row sums to 1. The
convergence to unity of the beliefs along the diagonal plots
indicates that the observer could infer the mental state of theactor soon after the movement onset.

4. Discussion
4.1. General
The model presented an explicit account of how mental
simulation and mental state inference loops can be built overexisting visual feedback control mechanisms, mediating theunderstanding of the meanings of others’ actions. The key
ingredients for the reutilization of motor circuits forperception is the existence of forward models that predict
the sensory consequences of movement, and the goaldirectness of the motor control policy. Our simulations
showed that with these key ingredients, appropriate movement simulation and mental state search mechanisms enablemental state inference. Although similar views have been
put forward by several researchers [4,28], to our knowledge,
our study is the first computational approach to mental stateinference and theory of mind, respecting the existing
cognitive neuroscience findings.
The functional equivalence between action generation
and action perception has gained considerable attention by
the expanding human brain imaging studies [37]. Thesestudies showed that motor regions become active for both
action execution and observation, switching their function(e.g. Refs. [5,6,13,34,38,63,71]). One functional explanation of this dual activation is that during perception a copyof a motor schema is instantiated (see Refs. [2,3]) in motor
centers to be used for action understanding. Although thisA

Fig. 6. The actor and the observer use spiral reaching movement strategy for movement production and mental state inference. The conventions are as in Fig. 5(panel A: spatial solution path; panel B: temporal error plot) except that the center plot in panel B shows also the average normalized error for those trials where
mental state search converged very slowly (dotted curve).

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151

141

Fig. 6 (continued).

view is functionally attractive, there is no convincing
explanation of how this could be implemented in the brain.
The other alternative is to reuse the very same brain circuits
for both execution and observation, but switch the inputs
and outputs according to the task (i.e. motor control or
action understanding.) It is agreed that the descending motor
output to muscles must be inhibited during action observation. However, there are no studies on how and where theinput switching occurs in the brain. From a computational
perspective, switching is a complex task if it is to occur
downstream in motor control since most motor circuits
operate in intrinsic parameters such as joint angles, muscle
activities, etc. During perception, these values pertaining to
the actor have to be estimated accurately and channeled toobserver’s motor circuits so that a reuse of motor system ispossible.
Our model predicts that the switching must occur at
visual processing areas where the visual control parameters
are extracted in a goal-directed manner (e.g. in the parietal
cortex). Since the motor circuits are used only for mental
simulation, the problem of input switching at the motor

level does not occur in the model. The prediction isbiologically plausible because visual attention readily
allows mechanism that controls input switching forexecution and observation. When human subjects areengaged in grasping, the gaze is directed towards contact
targets on the object surface while the hand follows the
fixation point suggesting that eye movements are part of
motor planning [44]. The same pattern of eye fixations
emerges when subjects merely observe a demonstrator
performs the same manipulation [22]. In other words, theeye–hand coordination with respect to the object being
manipulated is identical regardless of whether the task is
executed or just observed during a demonstrator’s execution. Notice that the dhandT here is a pragmatic hand and it
does not matter whether it belongs to the actor or theobserver. Our model also assumes pragmatic feature
extraction within the parietal cortex. During execution,
visual processing is engaged in extracting control variable(s) based on the object and the controlled hand, whileduring observation, the control variables(s) extracted are
based on actor’s hand rather than the observer’s.

142

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151

Fig. 7. Four-target (0, 1, 2, 3) board is used to show the goal estimate of the observer in response to all possible mental states of the actor. The dashed horizontal
line in each panel represents the drealT target, whereas the dotted line represents the dfakeT target. The observer’s belief about the actor’s goal is shown by solidline. The horizontal axis represents simulation cycles starting from movement onset to movement completion. The vertical axis indicates the targets (the verticalordering has no spatial meaning on the target board.).

4.2. The mental state inference model and pretence as the
precursor of theory of mind
A normal child develops the ability to pretend andunderstand pretence by the age of two, starts to use mental
state terms by the age of three, and becomes capable ofsolving complex reasoning such as false belief tasks by theage of four [54,55]. However, a child has to learn todifferentiate between what is real and what is pretend inorder to build a consistent knowledge of the world [54,62].
In our model, the motor system subserves mental
simulation and mental state inference by implicitly assuming the existence of a mechanism to mark the mental state
representations as real or imagined. When in dobserver

Fig. 8. A deceptive observer can infer the mental state of a deceptive actor. The number of target points used was six but only six representative deception
scenarios are shown out of 65=30 possible. The conventions used are as in Fig. 7.

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151

modeT, the Estimated Mental State and Mental State areplace holders for fictitious states, namely the intention andgoals of the observed actor, which are decoupled fromreality (they are neither observer’s intention nor necessarily
the object/affordance the actor means to act upon) and canbe manipulated within the mental state inference mechanism. Our model provides an explicit mechanism as to
how to utilize an existing forward model such that it can
support a mental simulation system decoupled from thereal world, which is fundamental to mental state inference[54].
Our model is compatible with the suggestion that mind
reading is facilitated by imitative games and pretend play.
We see a child’s pretend play as an exercise to develop
the ability to manipulate the contents of mental State (seeFig. 2) with proper tagging of real or imagined. dIf thisbanana was a telephone what would I doT is answered bymodifying the representation of the (real) banana so thatthe affordances of a (virtual) phone is somehow (e.g.
based on the shape) anchored on the banana. This allows

143

the motor system to treat a banana as a phone using theanchored affordances to act on. Conversely, the child’s
understanding of the pretend play of others is an exercise
in mental state inference where the search is over the
possible modifications of the representation of the banana
so to make the representation compatible with theobserved action. In this sense, the mental state module(see Fig. 2) contains codes which are modifiable in part or
all by the actor either for self-action (pretence—true act if
not modified) or for understanding others’ action. As acorollary, the imitation ability offered by the current
model is based on reproducing the inferred intention of anobserver which may not respect the details of theobserved act. Imitation and its development is a vast
topic and a full discussion is beyond the scope of thispaper; the interested reader is referred to Ref. [89] andcitations therein.
It takes a child 2 more years to move from the pretend
play stage to adult like theory of mind that enables him/her
to appreciate false belief tasks [54,55]. Our model predicts

Fig. 9. The difference between visually extracted control variables and control variables obtained by mental simulation can be used to infer the intention of anactor. Each subplot shows the probability that the observed movement (rows) is the same as the mentally simulated one (columns). The horizontal axis
represents the simulation cycles starting from movement onset to movement completion. The control variables extracted for the comparison is based on thementally simulated movement. Thus, the first column inferences require the control parameters for holding (distance to metal head and the angle between thepalm normal and hammer plane) whereas the remaining columns require the control parameters for nailing and prying (distance to handle center and the angle
between the knuckle vector and hammer handle direction). The convergence to unity of the diagonal belief curves indicates correct mental state inference.

144

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151

that this delay is due to the time required for the child’s
dinventionT of a (neural) representation of abstract mental
states (believing vs. feeling hungry). Once it is acquired,
the proficiency acquired by pretend play enables a child toinstantiate representations of others’ mental states (believe
something) and perform the mental state inference loop
(searching something) to find out why, for example, whengoing out, his/her mother (erroneously) picked herumbrella on a sunny day (because she believed it was
raining).
4.3. Why mental state inference via mental simulation?
The ability to estimate the mental state of others could beacquired by learning the relation between one’s own mental
states and their visual consequences, and inverting therelation to infer mental states of others. However, theinversion is computationally hard due to the lack of a oneto-one correspondence between mental states and behavior.
In fact, considering the unobservable parameters, therelation between mental states and behavior (in either
direction) is not a well-defined function. Although one
may suggest that primates have developed specialized
dedicated neural circuits for dealing with such estimation
tasks, we emphasize two points that favor a mental
simulation mechanism over a dedicated circuit. First, mental
state inference using mental simulation allows interpretation
without prior experience since, as long as a movement or
behavior is in the repertoire of an organism, it will beinterpretable without any training. Second, a fixed amountof resource is required to implement the mental simulation
circuit as opposed to a dedicated neural circuit that requires
an ever-growing storage requirement with the increasing
number of behaviors to interpret. These two features makemental simulation a parsimonious evolutionary solution foraction understanding. On the other hand, for novel
behaviors or actions performed by agents with different
anatomical structures, the mental simulation strategy alone
may not be enough. Thus, we suggest that mental simulation
could be a strategy adopted by biological systems along theevolutionary scale, as it is cheap in terms of time andresource allocation. However, by the increasing complexityof environment and social life, other mechanisms might
have augmented this dcheap solutionT.
A functional parallel to our model has been the findings
on empathy and emotion understanding [8,88]. According
to these studies, a common face-emotion region for bothproduction and understanding appears to be the Insula[8,88]. Although it is tempting to accept this system as ademotional mirror neuron systemT, there is an importantdifference how these circuits could be learned (developed
in infancy), assuming that these circuits are not innate.
Hand actions provide self-vision directly; whereas facial
gestures can only be observed via a reflective substance orotherwise must be inferred, e.g. from somatosensory
perception (which itself has the problem of learning theinternal stateYvisual mapping). To match an observed
action onto an internal motor representation a comparison
must be performed, which requires the representations bein the same domain. Since an observer cannot infer either
the somatosensory perception or the motor commands ofthe actor, there is a major computational problem for thebrain, which requires a different solution than we havepresented with our model.
If we disregard the unlikely hypothesis that the
demotional mirror systemT is innately wired, then thesolution must be rooted in social interaction. bYou eat
AYyou have face expression X (visual)Q. bI eat AYI feel
disgust Y (internal state)Q. Therefore X (visual) must be Y
(feeling of disgust). The mechanism of how this example
could be elaborated in the brain is beyond the scope of thispaper but the important message that should be taken isthat the manual action understanding and facial emotion
understanding pose rather different problems to the primate
brain.
4.4. Biological considerations
Area F2 (caudal dorsal premotor cortex) in macaquemonkey receives somatosensory input from areas PEip andPEc, and visual input from areas MIP and V6A, andprojects to the primary motor cortex [32]. V6A reachrelated neurons can be useful in guiding the hand to reachits target as the majority of V6A neurons tested in areaching task were significantly modulated during theexecution of arm movements [19]. Area F2 can use the
MIP and V6A inputs in controlling arm position during thetransport of the hand to spatial targets [68]. It is possiblethat the dorsal premotor cortex (areas F2 and F7) functions
as a visuomotor controller that is involved in selection,
preparation and execution of movements [91]. However,there are differences in F2 and F7 connectivity andneurophysiological responses, thus it is not clear whether
they form separate circuits [32] or they are part of aunitary visuomotor circuit.
Similar to F2 and F7, area F4 (caudal ventral premotor
cortex) receives rich input from posterior parietal cortex, thedominant projection being from area VIP, and sends
descending projections to the brain stem and spinal cord[32]. In area F4, the space is coded in body-parts-centeredcoordinate frame (e.g. centered on the hand) [25], where
many neurons fire during reaching movements of theproximal arm indicating that VIP–F4 circuit can be involved
in reach control, transforming object locations into motorplans [68].
The neurons in area AIP (anterior part of the lateralbank of the intraparietal sulcus) discharge in response toviewing and/or grasping of three-dimensional objects
representing properties of objects relevant for grasping[60,75,73]. AIP neurons are active either in relation to
the grasping behavior alone or in relation to the vision of
objects, and often a mixture of the two [74,85]. Area

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151

AIP has very significant recurrent cortico-cortical projections with area F5 (rostral ventral premotor cortex) [56]that is involved in grasp planning and execution [69],and projects to motoneurons that control finger muscles[16]. However, F5 projections are not sufficient alone toperform a grasping action, as a lesion of the finger areain the primary motor cortex completely disrupts grasp
execution whereas inactivation of F5 neurons only
disrupts the coordination of fingers [24]. The latter
finding suggests that area F5 dinstructsT lower motor
centers for execution and is not directly involved incontrolling finger muscles supporting the structure of ourmodel. In addition, it is suggested that the premotor
cortex may play a preferential role in sensory or contextdependent processing related to task performance,
whereas motor cortex may be more involved in processing related to the purely motor aspects of task performance [77].
The mental state inference model is based on a systems
level abstraction of the briefly reviewed literature employing
an object centered representation (in the simulations
distance and orientation with respect to a target axis).
Although this representation is compatible with the reportsthat parietal cortex employs object centered representationof the world-for-action, the exact form of the code and thesubsequent transformations (e.g. normalization) are notknown [1,90]. To summarize, the model assumes thefollowing. For the control of goal-directed movement,parietal areas are involved in monitoring the relation ofhand with respect to an attended object or a point in space.
The premotor cortex, on the other hand, instructs theprimary motor cortex and related structures for movement
execution. The parietal and premotor cortices together withother lower centers form a visuomanual servo for a range ofgoal-directed movements, which are modulated by prefrontal cortex according to context and task requirements.
This hierarchical view might be questioned within thelight of recent findings that employed long (500 ms)
microstimulation of the monkey motor cortex [35]. In
this study, the prolonged stimulation generated complex
movements with repeatable robustness. The mapping of
the movements onto the motor cortex suggested that the
primary motor and premotor cortex might be complementary regions belonging to a larger map of manual
space [35,36]. However, it should be noted that the
prolonged microstimulation could be misleading and notconclusive due to the wide spread of applied current
over the cortex [84]. In addition, we note that the movements generated in prolonged microstimulation study
appear to be feedforward in nature and do not relate toan external goal. In contrast, our model attains the abilityof mental state inference by utilizing a feedback control
mechanism that is defined with respect to an externalgoal. Nevertheless, the direct and indirect projections of
the premotor cortex onto the spinal cord [16,17] and themodulation of primary cortex neuron outputs by the145

premotor cortex [79] suggest that the strict hierarchicalcontrol view he have adopted could be simplistic andother alternatives [17,35,36,49] should be considered for
future research.
It has been suggested that for imitation, a representationof a candidate movement is generated in the superior
temporal sulcus (STS) as a forward prediction mediated by
the mirror neurons and compared to the observed action
representation in STS [8,41,58]. If this hypothesis iscorrect, then we might associate STS with the difference
module, which generates an error signal between thementally simulated and observed sensory signals (see Fig.
2). If this view is correct, then our model predicts thatwithin STS a goal-directed movement is represented as atemporal profile of control parameters (e.g. grasping an
object might be represented as the set trajectories of finger
positions with respect to the object.) in a perspective- andactor-invariant way.
As the mental state inference model is a systems level
model, it leaves certain issues open for future research.
Firstly, the model’s biological instantiation needs to beaddressed, that is, how can the functional roles assigned tothe modules within the model be realized using realistic
neurons. An initial attempt would be to implement thepostulated functions of the modules using population ofneural units. This would then lead to neural level predictions
that can be compared against neurophysiological data
directly.
4.5. Mirror neurons and unconscious mental simulation
In one study, callosotomy patients could judge stimulus
handedness accurately when the handedness of thestimulus is contralateral to the hemisphere perceiving it
and was unable to judge it when the handedness of thestimulus was ipsilateral to the hemisphere perceiving it[67]. Response time analyses indicate patients, like
controls, mentally simulate reaching (dmoldingT) into
stimulus postures [67]. The finding shows that mental
simulation of a limb requires circuits specific to control ofthat limb in order to elicit correct response (i.e. recognition
or inference), supporting our model that movement understanding requires the proper functioning of the motor
circuit producing the movement.
The dmirror neuronsT in the ventral premotor cortex ofa macaque monkey become active when the monkey
observes goal-directed movements, suggesting that movement observation involves functioning of various cortical
motor centers [15,29,70]. In addition, unconscious motor
circuit involvement in movement observation is shown inhumans using non-invasive techniques [6,18,38,42]. However, the functional role of such involvement is still
unknown. Our model gives an answer by predicting thatthe activity in premotor areas is the result of unconscious
or implicit mental simulation of movement analogous to
the engagement of implicit motor imagery when subjects

146

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151

are asked to report whether the presented stimulus is a leftor right hand [65], or when they are asked to decide how
they will grasp a handle with varying orientations [45].
During a mental simulation, the brain regions implementing the body and a sensory forward model (FM) should
become active. Such a forward model could be implemented exclusively within the premotor cortex, inparticular by the mirror neurons. According to this view,
mirror neurons can be involved in the sensory forward
prediction of goal-directed hand movements, which areactivated for (1) mental simulation during action observation and (2) for feedback-delay compensation during
movement. The experimental findings so far are inaccordance with the FM account of mirror neurons: A
muscimol-induced lesion in area F5 where mirror neurons
were found does not impair grasping behavior but only
cause a slowing down in movement execution [24] which
supports the forward prediction for delay compensationrole of mirror neurons. On the other hand, the mirror
neuron discharge for action observation (see Refs.
[29,70]) is compatible with an action understanding
mechanism based on a mental simulation loop involving
mirror neurons.
However, the hypothesis that mirror neurons implementa FM (or part of) is not the only alternative: it is possible
to have a series of forward models implementing thefunctionality of the proposed FM. One possibility is tohave a visual forward model located in the parietal cortex,
mimicking the sensory processing from retina to parietal
centers, which is connected in serial with a cerebellar
dynamics forward model [51,92]. The latter alternative
predicts that movement observation must activate cerebellum (limb dynamics forward model) and some region
in the parietal cortex (sensory forward model) in addition
to premotor regions (movement planning), which conforms well with the neuroimaging findings employing
human subjects [6,43]. In the macaque, the primary
reports of mirror neuron findings are confined to a sectorof the inferior frontal cortex (area F5). Although somemirror-like neurons have been described in the anterior
inferior parietal cortex (area 7b) [23], human dmirror
regionsT appear to be more widely distributed thanmacaque mirror regions, including the parietal andpremotor cortices [6]. However, there might be mirror
areas in the macaque brain that have not been identified
yet.
Current experimental findings do not help us rule out
whether a circumscribed mirror region (i.e. in the premotor
cortex) implements the proposed FM, or a series of
forward models distributed over the cortex and the
cerebellum make up a composite FM. Careful experiments
must be designed with respect to the computational
account of mental simulation theory of action understanding. A class of behavioral and brain imaging experiments can be designed using the following basic
experimental setup. Subjects could be asked to infer thedemonstrator’s reach target [intention] as quick as possible, similar to our simulation experiments while simultaneously executing (or mentally imagining the executionof) reaching [grasping] movements. In such an experiment, our model would predict that the responses of thesubjects would be faster and more accurate when theobserver and the demonstrator’s movements are characterized by compatible intentions, whereas it would be
slower and less accurate with conflicting intentions. To
reveal the mirror neuron involvement in mental simulationand mental state estimation, neurophysiological experiments must involve synchronized kinematics recordings ofthe experimenter (or monkey) during action demonstration.
The correlation between the discharge profiles of mirror
neurons with various visual feedback parameters forgrasping, both when the monkey observes or executes agrasping movement, will reveal crucial information about
the computational role of mirror neurons during manualcontrol and action understanding, which can be used toverify/revise the model we presented in the paper.
The model we have developed presents a computational
account of how mental state estimation could be built
upon a visual feedback control mechanism. With theproposed localization of mental state estimation circuitry
and the alternatives suggested for forward model localization, we believe that the model will lead to experimental studies that will not only improve and detail ourmodeling, but place our understanding of mirror neurons
and mental state inference in a causally complete
computational framework.

Appendix A
A.1. Straight-line reaching
Straight line reaching is achieved using the pseudo-inverse method. The method is based on the geometric Jacobian (J),which is a time varying matrix relating end-effector Cartesian velocity to the angular velocities of the arm joints as follows
[96]:
2 3
2 3
h1
ḣ
ẋx
6 h2 74 ẏy 5 ¼ J6 ḣ
7 or in vector notation ṗ
p ¼ Jḣ
h
4 ḣ
h3 5
żz
h4
ḣ

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151

147

Representing the upper arm length and the lower arm length as l 1 and l 2, respectively, and abbreviating sin(h k ) and cos(h k )with s k and c k , where k is an index to the joint angle, the Jacobian matrix of our arm model can be written as:
20
J ¼4 l2 ð  s1 c2 s4  ðs1 s2 s3 c4 þ c1 c3 c4 ÞÞ þ l1 s1 c2
l2 ð  c1 c2 s4 þ c1 c4 s2 s3  c3 c4 s1 Þ  l1 c1 c2

l2 ð  c2 s4 þ s2 s3 c4 Þ  l1 c2
l2 ðc1 s2 s4 þ c2 s3 c1 c4 Þ þ l1 c1 c2
l2 ð  s1 s2 s4 þ c2 c4 s1 s3 Þ þ l1 s1 s2

 l2 ðc2 c3 c4 Þ
l2 ðc1 s2 c3 c4 þ s1 s3 c4 Þ
l2 ð  c3 c4 s1 s2  c1 c4 s3 Þ3l2 ð  s2 c4 þ c2 s3 s4 Þ5 l2 ð  c1 c2 c4  c1 s2 s3 s4 þ s1 c3 s4 Þ
l2 ð  s1 c2 c4  s1 s2 s3 s4  c1 c3 s4 Þ  l1 c1 c2

A desired end-effector position p desired can be achieved by iterating the below update rule until pip desired, where g
represents the update rate and J# is the pseudo-inverse of J. We adapted g as a function of end-effector–target distance (d) as
g=0.5+e0.02d to obtain faster convergence. Note that the visual feedback signal appearing in this computation is p desiredp.
htþ1 ¼ ht þ gJ#ht ðpdesired  pt Þ
A.2. Deceptive reaching
Deceptive reaching is characterized by a real ( p real) and a fake ( p fake) target. We denote the end-effector position by p. A
dfloating targetT for the inverse kinematics can be used to generate a deceptive trajectory. The desired target moves from thefake target to the real one. The function f is an increasing function between 0 and 1 in the domain of v, where v denotes the
distance between the board plane and end-effector (v 0 denotes the distance at the movement onset.)pdesired ðvÞ ¼ f ðvÞpreal þ ð1  f ðvÞÞpfake
pﬃﬃﬃﬃﬃﬃﬃﬃﬃ
In the simulations, we used f ðvÞ ¼ 1  v=v0 and ran the inverse kinematics algorithm (straight line reaching) with thetime varying target p desired. The visual feedback signal appearing in the inverse kinematics computation can be expanded asfollows:
pdesired ðvÞ  p ¼ f ðvÞðpreal  pÞ þ ð1  f ðvÞðpfake  pÞ
Thus, the control variables extracted by parietal cortex are taken as the distances from the end-effector to the fake and realtargets, hence this two distance values guided mental state search.
A.3. Spiral reachingSpiral reaching movements are generated by applying a visual feedback servo for a dfloating targetT based on a set of viapoints (constructed in an object centered manner), weighted according to the distance of the hand to the board. Desired endeffector position is given below using the following variables:
v: distance between the board plane and the end-effector ( p)v 0: value of v at movement onset
c: board center2
pﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ 2 cosðp þ kp=8Þ
8X
e 1v=v0 i=8 0:01 4
pdesired ðvÞ ¼ c þsinðp þ kp=8Þ
pﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
2 
8
X
k¼0
1v=v0 m=8
0:01
0
e

 sinðp þ kp=8Þcosðp þ kp=8Þ
0

3
0 

0 5 ptarget  c
1

m¼0

The movement kinematics was generated as in straight line reaching using the p desired as the target. The online feedback
control of this movement requires monitoring of eight virtual targets (this is evident via an expansion similar to the one indeceptive reaching). Thus, end-effector–virtual target distances (i.e. the terms of the summation in the above equation) are used
as control variables in the mental state inference.
A.4. Hammer-grasp for nailing and prying
Grasping of a hammer suitable for prying or driving a nail is modeled as a power grasp of the handle. The difference
between nailing and prying movements is characterized by the orientation of the hand in relation to the hammer. We modeled
the grasp as two parallel visual feedback control problems: (1) transporting the hand to the proximity of the center of the handle
and (2) orientating the knuckle vector to match the orientation of the handle (either fully aligned or p radians away). The
reaching component of the grasping movement is solved using the inverse kinematics algorithm introduced earlier. The

148

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151

Fig. 10. The control variables used in nailing and prying tasks are depicted. The desired Knuckle vector is (Handle vector) for prying and (Handle vector) for
nailing. The desired Distance for both cases is zero. The path of the hand is constrained with appropriate via-points to avoid collision.

orientation of the hand is adjusted by stochastic gradient descent based on feedback of the orientation error. Thus the control
variables that enable the generation of grasping movement were (1) the orientation difference (angle) between the knuckle
vector and hammer handle and (2) the distance of the hand to the handle center. The mental state search was therefore based on
the distance and orientation variables (see Fig. 10).
The algorithm used for simulating grasping kinematics performs various tasks starting from the onset of movement (time: 0)
until the completion of a grasp (time: 1):
0.00–1.00: Reach towards the handle center via a via-point (to avoid collision)
0.00–0.70: Extend fingers
0.30–0.95: Adapt hand orientation
0.95–1.00: Check for collision
If collision: Enclose fingers.
Note that the fingers play a rather decorative role in this simulation, as they are not part of feedback control in powergrasping.
A.5. Hammer-grasp for holding/putting away
Grasping of a hammer with no intention of tool-use is modeled as power grasping the metal head. We model the grasping as
two parallel visual feedback control problems: (1) transporting hand to the proximity of the center of the metal head and (2)adapting the hand orientation such that palm normal coincides with the hammer plane normal (see Fig. 11).
The reaching component of the grasping movement is solved using the inverse kinematics algorithm introduced earlier. The
orientation of the hand is adjusted by stochastic gradient descent based on the mismatch between palm and hammer plane
normal vectors. Thus, the control variables that enabled the generation of grasping movement were (1) the orientationdifference (angle) between the palm normal and hammer plane normal and (2) the distance of the hand to the metal head center.
The mental state search was, therefore, based on the distance and normal differences described. The algorithm used for
simulating grasping kinematics is the same as nailing and prying movement generation presented above.
Note that the orientation and distance not only comprise a minimal set of parameters that can be used to generate
movements for grasping a hammer, but also allows mental state inference without explicit coordinate transformation. Although
the grasp-planning algorithm we implemented is a general framework that can operate for simple scenarios, certain hammer
orientations/locations would require more advanced planning and control schemes (e.g. better collisions avoidance).
Nevertheless, the current simulations give us a simple framework to demonstrate how the goal-directed movement control
variables could be used to infer another actor’s intention.

Fig. 11. The control variables used in holding task is depicted. Desired palm normal is hammer plane normal. The desired distance is zero. However, the path of
the hand is constrained with a via-point avoiding collision.

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151

References[1] R.A. Andersen, L.H. Snyder, D.C. Bradley, J. Xing, Multimodal
representation of space in the posterior parietal cortex and its use inplanning movements, Annual Review of Neuroscience 20 (1997)
303 – 330.
[2] M.A. Arbib, Perceptual structures and distributed motor control, in:
V.B. Brooks (Ed.), Handbook of Physiology, Section 2: The Nervous
System, Vol. II, Motor Control, Part 1, American PhysiologicalSociety, 1981, pp. 1449 – 1480.
[3] M.A. Arbib, The Metaphorical Brain: 2, Neural Networks andBeyond, vol. xiv, Wiley, New York, NY, 1989, 458 pp.
[4] S.J. Blakemore, J. Decety, From the perception of action to theunderstanding of intention, Nature Reviews. Neuroscience 2 (2001)
561 – 567.
[5] E. Bonda, M. Petrides, D. Ostry, A. Evans, Specific involvement of
human parietal systems and the amygdala in the perception of
biological motion, Journal of Neuroscience 16 (1996) 3737 – 3744.
[6] G. Buccino, F. Binkofski, G.R. Fink, L. Fadiga, L. Fogassi, V.
Gallese, R.J. Seitz, K. Zilles, G. Rizzolatti, H.J. Freund, Action
observation activates premotor and parietal areas in a somatotopic
manner: an fMRI study, European Journal of Neuroscience 13 (2001)
400 – 404.
[7] Y. Burnod, P. Baraduc, A. Battaglia-Mayer, E. Guigon, E. Koechlin,
S. Ferraina, F. Lacquaniti, R. Caminiti, Parieto-frontal coding ofreaching: an integrated framework, Experimental Brain Research 129(1999) 325 – 346.
[8] L. Carr, M. Iacoboni, M.-C. Dubeau, J.C. Mazziotta, G.L. Lenzi,
Neural mechanisms of empathy in humans: a relay from neuralsystems for imitation to limbic areas, PNAS 100 (2003) 5497 – 5502.
[9] T. Chaminade, D. Meary, J.P. Orliaguet, J. Decety, Is perceptual
anticipation a motor simulation? A PET study, NeuroReport 12 (2001)
3669 – 3674.
[10] A. Churchill, Vision of the hand and environmental context in humanprehension, Experimental Brain Research 134 (2000) 81 – 89.
[11] L.A. Cooper, Mental transformations and visual comparison
processes: effects of complexity and similarity, Journal of Experimental Psychology. Human Perception and Performance 2 (1976)
503 – 514.
[12] D.J. Crammond, Motor imagery: never in your wildest dream, Trends
in Neurosciences 20 (1997) 54 – 57.
[13] J. Decety, J. Grezes, N. Costes, D. Perani, M. Jeannerod, E. Procyk, F.
Grassi, F. Fazio, Brain activity during observation of actions.
Influence of action content and subject’s strategy, Brain 120 (1997)
1763 – 1777.
[14] M. Desmurget, S. Grafton, Forward modeling allows feedback control
for fast reaching movements, Trends in Cognitive Sciences 4 (2000)
423 – 431.
[15] G. Dipellegrino, L. Fadiga, L. Fogassi, V. Gallese, G. Rizzolatti,
Understanding motor events—a neurophysiological study, Experimental Brain Research 91 (1992) 176 – 180.
[16] R.P. Dum, P.L. Strick, The origin of corticospinal projections from thepremotor areas in the frontal lobe, Journal of Neuroscience 11 (1991)
667 – 689.
[17] R.P. Dum, P.L. Strick, Motor areas in the frontal lobe of the primate,
Physiology & Behavior 77 (2002) 677 – 682.
[18] L. Fadiga, L. Fogassi, G. Pavesi, G. Rizzolatti, Motor facilitation
during action observation—a magnetic stimulation study, Journal of
Neurophysiology 73 (1995) 2608 – 2611.
[19] P. Fattori, M. Gamberini, D.F. Kutz, C. Galletti, dArm-reachingTneurons in the parietal area V6A of the macaque monkey, European
Journal of Neuroscience 13 (2001) 2309 – 2313.
[20] S. Ferraina, M.R. Garasto, A. Battaglia-Mayer, P. Ferraresi, P.B.
Johnson, F. Lacquaniti, R. Caminiti, Visual control of hand-reaching
movement: activity in parietal area 7m, European Journal of Neuroscience 9 (1997) 1090 – 1095.

149[21] S. Ferraina, A. Battaglia-Mayer, A. Genovesio, B. Marconi, P.
Onorati, R. Caminiti, Early coding of visuomanual coordination
during reaching in parietal area PEc, Journal of Neurophysiology 85(2001) 462 – 467.
[22] J.R. Flanagan, R.S. Johansson, Action plans used in action observation, Nature 424 (2003) 769 – 771.
[23] L. Fogassi, V. Gallese, L. Fadiga, G. Luppino, M. Matelli, G.
Rizzolatti, Coding of peripersonal space in inferior premotor cortex(area F4), Journal of Neurophysiology 76 (1996) 141 – 157.
[24] L. Fogassi, V. Gallese, L. Fadiga, G. Rizzolatti, Neurons responding to
the sight of goal-directed hand/arm actions in the parietal area PF (7b)of the macaque monkey, 28th Annual Meeting of Society for
Neuroscience, Los Angeles, 1998.
[25] L. Fogassi, V. Gallese, G. Buccino, L. Craighero, L. Fadiga, G.
Rizzolatti, Cortical mechanism for the visual guidance of handgrasping movements in the monkey—a reversible inactivation study,
Brain 124 (2001) 571 – 586.
[26] C.D. Frith, U. Frith, Interacting minds—a biological basis, Science
286 (1999) 1692 – 1705.
[27] H.L. Gallagher, C.D. Frith, Functional imaging of dtheory of mindT,
Trends in Cognitive Sciences 7 (2003) 77 – 83.
[28] V. Gallese, A. Goldman, Mirror neurons and the simulation theory ofmind-reading, Trends in Cognitive Sciences 2 (1998) 493 – 501.
[29] V. Gallese, L. Fadiga, L. Fogassi, G. Rizzolatti, Action recognition in
the premotor cortex, Brain 119 (1996) 593 – 609.
[30] C. Galletti, M. Gamberini, D.F. Kutz, P. Fattori, G. Luppino, M.
Matelli, The cortical connections of area V6: an occipito-parietal
network processing visual information, European Journal of Neuroscience 13 (2001) 1572 – 1588.
[31] E. Gerardin, A. Sirigu, S. Lehericy, J.-B. Poline, B. Gaymard, C.
Marsault, Y. Agid, D. Le Bihan, Partially overlapping neural networks
for real and imagined hand movements, Cerebral Cortex 10 (2000)
1093 – 1104.
[32] S. Geyer, M. Matelli, G. Luppino, K. Zilles, Functional neuroanatomyof the primate isocortical motor system, Anatomy and Embryology
202 (2000) 443 – 474.
[33] H. Gomi, M. Kawato, Adaptive feedback-control models of thevestibulocerebellum and spinocerebellum, Biological Cybernetics 68(1992) 105 – 114.
[34] S.T. Grafton, M.A. Arbib, L. Fadiga, G. Rizzolatti, Localization ofgrasp representations in humans by positron emission tomography: 2.
Observation compared with imagination, Experimental Brain Research 112 (1996) 103 – 111.
[35] M.S. Graziano, C.S. Taylor, T. Moore, Complex movements
evoked by microstimulation of precentral cortex, Neuron 34(2002) 841 – 851.
[36] M.S. Graziano, C.S. Taylor, T. Moore, D.F. Cooke, The corticalcontrol of movement revisited, Neuron 36 (2002) 349 – 362.
[37] J. Grezes, J. Decety, Functional anatomy of execution, mental
simulation, observation, and verb generation of actions: a metaanalysis, Human Brain Mapping 12 (2001) 1 – 19.
[38] R. Hari, N. Forss, S. Avikainen, E. Kirveskari, S. Salenius, G.
Rizzolatti, Activation of human primary motor cortex during action
observation: a neuromagnetic study, Proceedings of the National
Academy of Sciences of the United States of America 95 (1998)
15061 – 15065.
[39] M. Haruno, D.M. Wolpert, M. Kawato, MOSAIC model for
sensorimotor learning and control, Neural Computation 13 (2001)
2201 – 2220.
[40] M. Heil, F. Rosler, M. Link, J. Bajric, What is improved if a mental
rotation task is repeated—the efficiency of memory access, or the
speed of a transformation routine? Psychological Researchs 61 (1998)
99 – 106.
[41] M. Iacoboni, L.M. Koski, M. Brass, H. Bekkering, R.P. Woods, M.-C.
Dubeau, J.C. Mazziotta, G. Rizzolatti, Reafferent copies of imitatedactions in the right superior temporal cortex, PNAS 98 (2001)
13995 – 13999.

150

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151[42] M. Iacoboni, R.P. Woods, M. Brass, H. Bekkering, J.C. Mazziotta, G.
Rizzolatti, Cortical mechanisms of human imitation, Science 286(1999) 2526 – 2528.
[43] H. Imamizu, S. Miyauchi, T. Tamada, Y. Sasaki, R. Takino, B. Putz, T.
Yoshioka, M. Kawato, Human cerebellar activity reflecting an
acquired internal model of a new tool, Nature 403 (2000) 192 – 195.
[44] R.S. Johansson, G. Westling, A. Backstrom, J.R. Flanagan, Eye–handcoordination in object manipulation, Journal of Neuroscience 21(2001) 6917 – 6932.
[45] S.H. Johnson, Thinking ahead: the case for motor imagery inprospective judgements of prehension, Cognition 74 (2000) 33 – 70.
[46] P.B. Johnson, S. Ferraina, L. Bianchi, R. Caminiti, Cortical
networks for visual reaching: physiological and anatomical organization of frontal and parietal lobe arm regions, Cerebral Cortex 6(1996) 102 – 119.
[47] S.H. Johnson, P.M. Corballis, M.S. Gazzaniga, Within grasp but out ofreach: evidence for a double dissociation between imagined hand andarm movements in the left cerebral hemisphere, Neuropsychologia 39(2001) 36 – 50.
[48] S.H. Johnson, M. Rotte, S.T. Grafton, H. Hinrichs, M.S. Gazzaniga,
H.-J. Heinze, Selective activation of a parietofrontal circuit during
implicitly imagined prehension, NeuroImage 17 (2002) 1693 – 1704.
[49] S. Kakei, D.S. Hoffman, P.L. Strick, Sensorimotor transformations in
cortical motor areas, Neuroscience Research 46 (2003) 1 – 10.
[50] M. Kawato, H. Gomi, A computational model of 4 regions of thecerebellum based on feedback-error learning, Biological Cybernetics
68 (1992) 95 – 103.
[51] M. Kawato, D. Wolpert, Internal models for motor control, Sensory
Guidance of Movement 218 (1998) 291 – 307.
[52] M. Kawato, K. Furukawa, R. Suzuki, A hierarchical neural networkmodel for control and learning of voluntary movement, Biological
Cybernetics 57 (1987) 169 – 185.
[53] F. Lacquaniti, E. Guigon, L. Bianchi, S. Ferraina, R. Caminiti,
Representing spatial information for limb movement: role of area 5 in
the monkey, Cerebral Cortex 5 (1995) 391 – 409.
[54] A.M. Leslie, Pretense and representation: the origins of btheory of
mindQ, Psychological Review 94 (1987) 412 – 426.
[55] A.M. Leslie, Pretending and believing: issues in the theory of ToMM,
Cognition 50 (1994) 211 – 238.
[56] M. Matelli, G. Luppino, A. Murata, H. Sakata, Independent
anatomical circuits for reaching and grasping linking the inferiorparietal sulcus and inferior area 6 in macaque monkey, Society for
Neuroscience 20 (1994) 404.4.
[57] B. Mehta, S. Schaal, Forward models in visuomotor control, Journalof Neurophysiology 88 (2002) 942 – 953.
[58] R.C. Miall, Connecting mirror neurons and forward models, NeuroReport 14 (2003) 2135 – 2137.
[59] R.C. Miall, D.J. Weir, D.M. Wolpert, J.F. Stein, Is the cerebellum asmith predictor? Journal of Motor Behavior 25 (1993) 203 – 216.
[60] A. Murata, V. Gallese, M. Kaseda, H. Sakata, Parietal neurons related
to memory-guided hand manipulation, Journal of Neurophysiology 75(1996) 2180 – 2186.
[61] D.G. Nair, K.L. Purcott, A. Fuchs, F. Steinberg, J.A. Kelso, Corticaland cerebellar activity of the human brain during imagined andexecuted unimanual and bimanual action sequences: a functional MRI
study, Brain Research Cognitive Brain Research 15 (2003) 250 – 260.
[62] S. Nichols, S. Stich, A cognitive theory of pretense, Cognition 74(2000) 115 – 147.
[63] N. Nishitani, R. Hari, Temporal dynamics of cortical representation
for action, Proceedings of the National Academy of Sciences of theUnited States of America 97 (2000) 913 – 918.
[64] E. Oztop, M.A. Arbib, Schema design and implementation of thegrasp-related mirror neuron system, Biological Cybernetics 87 (2002)
116 – 140.
[65] L.M. Parsons, Temporal and kinematic properties of motor behavior
reflected in mentally simulated action, Journal of Experimental
Psychology. Human Perception and Performance 20 (1994) 709 – 730.

[66] L.M. Parsons, P.T. Fox, J.H. Downs, T. Glass, T.B. Hirsch, C.C.
Martin, P.A. Jerabek, J.L. Lancaster, Use of implicit motor imagery forvisual shape discrimination as revealed by PET, Nature 375 (1995)
54 – 58.
[67] L.M. Parsons, J.D. Gabrieli, E.A. Phelps, M.S. Gazzaniga, Cerebrally
lateralized mental representations of hand shape and movement,
Journal of Neuroscience 18 (1998) 6539 – 6548.
[68] G. Rizzolatti, L. Fadiga, Grasping objects and grasping action
meanings: the dual role of monkey rostraventral premotor cortex(area F5), Sensory Guidance of Movement, Novartis Foundation
Symposium 218, Wiley, Chichester, 1998, pp. 81 – 103.
[69] G. Rizzolatti, M. Gentilucci, R.M. Camarda, V. Gallese, G. Luppino,
M. Matelli, L. Fogassi, Neurons related to reaching–grasping arm
movements in the rostral part of area-6 (Area-6a-Beta), Experimental
Brain Research 82 (1990) 337 – 350.
[70] G. Rizzolatti, L. Fadiga, V. Gallese, L. Fogassi, Premotor cortex andthe recognition of motor actions, Cognitive Brain Research 3 (1996)
131 – 141.
[71] G. Rizzolatti, L. Fadiga, M. Matelli, V. Bettinardi, E. Paulesu, D.
Perani, F. Fazio, Localization of grasp representations in humans by
PET: 1. Observation versus execution, Experimental Brain Research
111 (1996) 246 – 252.
[72] G. Rizzolatti, L. Fogassi, V. Gallese, Parietal cortex: from sight toaction, Current Opinion in Neurobiology 7 (1997) 562 – 567.
[73] H. Sakata, M. Taira, A. Murata, S. Mine, Neural mechanisms of visual
guidance of hand action in the parietal cortex of the monkey, Cerebral
Cortex 5 (1995) 429 – 438.
[74] H. Sakata, M. Taira, M. Kusunoki, A. Murata, Y. Tanaka, The TINS
lecture—the parietal association cortex in depth perception andvisual control of hand action, Trends in Neurosciences 20 (1997)
350 – 357.
[75] H. Sakata, M. Taira, M. Kusunoki, A. Murata, Y. Tanaka, K. Tsutsui,
Neural coding of 3D features of objects for hand action in the parietal
cortex of the monkey, Philosophical Transactions of the Royal Societyof London Series B—Biological Sciences 353 (1998) 1363 – 1373.
[76] N. Schweighofer, J. Spoelstra, M.A. Arbib, M. Kawato, Role of thecerebellum in reaching movements in humans: II. A neural model ofthe intermediate cerebellum, European Journal of Neuroscience 10(1998) 95 – 105.
[77] L.M. Shen, G.E. Alexander, Preferential representation of instructed
target location versus limb trajectory in dorsal premotor area, Journalof Neurophysiology 77 (1997) 1195 – 1212.
[78] R.N. Shepard, L.A. Cooper, Mental Images and Their Transformations, MIT Press, Cambridge, MA, 1982.
[79] H. Shimazu, M.A. Maier, G. Cerri, P.A. Kirkwood, R.N. Lemon,
Macaque ventral premotor cortex exerts powerful facilitation of motor
cortex outputs to upper limb motoneurons, Journal of Neuroscience 24(2004) 1200 – 1211.
[80] J.B. Smeets, E. Brenner, A new view on grasping, Motor Control 3(1999) 237 – 271.
[81] J.B. Smeets, E. Brenner, Independent movements of the digits ingrasping, Experimental Brain Research 139 (2001) 92 – 100.
[82] J.F. Stein, Space and the parietal association areas, in: J. Paillard (Ed.),
Brain and Space, Oxford University Press, Oxford, England, 1991.
[83] V.E. Stone, S. Baron-Cohen, R.T. Knight, Frontal lobe contributions
to theory of mind, Journal of Cognitive Neuroscience 10 (1998)
640 – 656.
[84] P.L. Strick, Stimulating research on motor cortex, Nature Neuroscience 5 (2002) 714 – 715.
[85] M. Taira, S. Mine, A.P. Georgopoulos, A. Murata, H. Sakata, Parietal
cortex neurons of the monkey related to the visual guidance of hand
movement, Experimental Brain Research 83 (1990) 29 – 36.
[86] S. Vijayakumar, A. D’souza, T. Shibata, J. Conradt, S. Schaal,
Statistical learning for humanoid robots, Autonomous Robots 12(2002) 55 – 69.
[87] P. Weiss, M. Jeannerod, Getting a grasp on coordination, News inPhysiological Sciences 13 (1998) 70 – 75.

E. Oztop et al. / Cognitive Brain Research 22 (2005) 129–151[88] B. Wicker, C. Keysers, J. Plailly, J.P. Royet, V. Gallese, G. Rizzolatti,
Both of us disgusted in My insula: the common neural basis of seeing
and feeling disgust, Neuron 40 (2003) 655 – 664.
[89] J.H. Williams, A. Whiten, T. Suddendorf, D.I. Perrett, Imitation,
mirror neurons and autism, Neuroscience and Biobehavioral Reviews
25 (2001) 287 – 295.
[90] S.P. Wise, D. Boussaoud, P.B. Johnson, R. Caminiti, Premotor andparietal cortex: corticocortical connectivity and combinatorial computations, Annual Review of Neuroscience 20 (1997) 25 – 42.
[91] S.P. Wise, D. Boussaoud, P.B. Johnson, R. Caminiti, Premotor andparietal cortex: corticocortical connectivity and combinatorial computations, Annual Review of Neuroscience 20 (1997) 25 – 42.

151[92] D.M. Wolpert, M. Kawato, Multiple paired forward and inverse
models for motor control, Neural Networks 11 (1998) 1317 – 1329.
[93] D.M. Wolpert, Z. Ghahramani, M.I. Jordan, An internal model for
sensorimotor integration, Science 269 (1995) 1880 – 1882.
[94] D.M. Wolpert, R.C. Miall, M. Kawato, Internal models in the
cerebellum, Trends in Cognitive Sciences 2 (1998) 338 – 347.
[95] D.M. Wolpert, Z. Ghahramani, J.R. Flanagan, Perspectives andproblems in motor learning, Trends in Cognitive Sciences 5 (2001)
487 – 494.
[96] L. Sciavicco, B. Siciliano, Modelling and Control of Robot
Manipulators, Springer, London; New York, 2000, xxiii, 377, pp.
79–87.