Synergistic Intelligence Project as Humanoid Science
Minoru Asada1 , Koh Hosoda1 , Yasuo Kunioshi2 , Hiroshi Ishiguro1 , and Toshio Inui3
JST ERATO Asada Synergistic Intelligence Project (www.jeap.org)
1
Graduate School of Engineering, Osaka University
2
School of Information Sci. & Tech., University of Tokyo
3
Graduate School of Informatics, Kyoto University{asada,hosoda,kuniyoshi,ishiguro,inui}@jeap.org

Abstract— This paper presents a new JST ERATO Asada
Project on Synergistic Intelligence (hereafter, SI) that aims atunderstanding cognitive developmental natural agents (humans),building artificial agents (humanoids), and their mutual feedbacks. The project consists of four groups: (1) Physio-SI: dynamicmotions such as walking, running, jumping, and heading andtheir seamless connections based on pneumatic muscle actuators,(2) Perso-SI: cognitive developmental robotics including body
image, imitation, and language communication. (3) Socio-SI:
emergence of communication and society by androids, and (4)SI-mechanism: neuroscientific supports for Physio, Perso, andSocio-SIs. The purpose, plan, and goal are briefly described.
Index Terms— Cognitive Developmental Robotics, HumanoidScience, Synergistic Intelligence

I. I NTRODUCTION
The current technologies of hardware and its control havebeen advanced to the level that enables us to build humanoid
robots that have a large number of DoFs and various, many
sensors. Human-like motions are realized on these robots, andthe shape and appearance have come close to us. Figure 2
indicates these humanoids that can rise up dynamically ([1]:left figure) or show the human appearances ([2]: right figure).

Fig. 2.

Humanoids that show dynamic rise-up and human-like appearance.

What is missing in the current robotics is the faculties of language communication with ordinary people and of intelligent
behavior generation in various situations such as at home. The
fundamental relationship between humans and robots would
become more important since these robots would be introduced
into our lives in near future, and therefore the mechanisms ofadaptation and development of both humans and robots should
be taken into account in order to find the correct direction of
the technology in future.
“Synergistic Intelligence (hereafter, SI),” the title of ourproject, emerges intelligent behaviours through the interaction

with environment including humans. Synergistic effects withbrain science, neuroscience, cognitive science, and developmental psychology are expected. SI is one approach to a new
discipline called “Humanoid Science” that aims at providing a
new way of understanding ourselves and a new design theoryof humanoids through mutual feedback between the design ofhuman-like robots and human-related science.
”Humanoid Science” under which a variety of researchers
from robotics, AI, brain science, cognitive science, psychology
and so on are seeking for new understanding of ourselves byconstructivist approaches, that is expected to produce manyapplications.
SI adopts a methodology called “Cognitive DevelopmentalRobotics” (hereafter, CDR) [3] that consists of the designof self-developing structures inside the robot’s brain, and theenvironmental design: how to set up the environment so thatthe robots embedded therein can gradually adapt themselves tomore complex tasks in more dynamic situations. Unstructured
terrains are opponents for adaptive walkers to negotiate with inorder to generate dynamic motions. The caregiver’s behaviour
to a robot is one environmental design issue since parents,teachers, and other adults adapt themselves to the needs ofchildren according to each child’s level of maturity and theparticular relationship they have developed with that child.
One of the most formidable issues in SI is “Nature vs.
Nurture”: to what extent should we embed the structure, and towhat extent should we expect the development triggered by theenvironment? A symbolic issue is Language Acquisition.
How can robots emerge the symbol in the social context? Whatis the essential element in this process?
The goal of SI is to understand the fundamental developing
process of three-years old human child for robot realizationof this cognitive developmental process, and the topics we aregoing to attack are (1) emergence of dynamic, elastic motions
by artificial muscles, (2) cognitive developmental experiments
on baby robots, (3) realization and understanding of communication by android, and (4) understanding neuro-infrastructures
for the above processes based on brain functional imaging andanimal experiments.
II. P HYSICALLY S YNERGISTIC I NTELLIGENCE
Physio-SI in short, aims at understanding emergence ofhuman intelligence from dynamic connection between bodystructure and environment with artificial muscles.

Fig. 1.

An overview of Synergistic Intelligence Project

The agent behavior is determined by not only its dynamics
but also environmental one if its compliance is comparable tothat of the environment. This interaction may help the agent
to be adaptive and robust against changes of the environment,
which cannot be taken for the rigid agents with high-ratio
geared motors. In this project, we utilize such physical interactions to realize dynamic skills such as walking, jumping,and running, and try to figure out the methodology to emerge
synergistic intelligence:utlizing compliance and dymamic power provided by theantagonistic drive.
• Smooth transition between dynamic behaviors: We
will propose a framework to deal with different modes oflocomotion, walking and running, and develop a ”Motor
Skill Emerging Humanoid” that can realize these locomotion modes.
• Cognitive approach to locomotion: From the standpointof building a Motor SKill Emerging Humanoid, weapproach to the synergistic intelligence mechanisms andwill understand the foundation of emergence of dynamicbehaviors.
III. I NTERPERSONALLY-S YNERGISTIC I NTELLIGENCE

Fig. 3.

Dynamic motions

Walking on uneven terrains utlizing antagonistic
driven legs: By utilizing flexibility provided by antagonistic driven legs and ballistics of the body, we will derivea methodology to realize adaptive walking on uneventerrains.
• Realization of dynamic behaviors: We will realize
dynamic behaviors such as jumping and running by•Fusing together the capabilities of real world behavior andcommunication will be the critical step towards the truly
humanoid robot intelligence. Interpersonally-Synergistic Intelligence group (Perso-SI in short) aims at constructing synthetic
robotic models that account for the process of cognitive
development starting from bodily movements of fetuses andreaching rich meaningful communication of infants. This willbridge the research of the other two groups, namely, PhysioSynergistic Intelligence and Socio-Synergistic Intelligence.
Our developmental cognitive model would account for (butnot limited to) the following cognitive functionalities:
1) Acquisition of brain internal representation of self bodythrough whole body sensory-motor learning,2) Learning about objects and tools,
3) Emergence and development of behavior imitation,4) Concept/imagery formation by abstraction of behaviors,and
5) Fusing together the behavior concepts/imitation withinteractions through gestures, vocalizations, and facialexpressions.
Some of our models will be based on the neuroscientificmodels provided by the Synergistic Intelligence Mechanisms
group; The model of object manipulation for topic 2, imitation
learning for topic 3, and motor imagery for topic 4.

manipulation capabilities and their integration with audiovisual sensing, and will be used for experiments on imitation
learning, concept acquisition, and fusing with communication.
In addition, we will construct (c) a computer simulation of
sensory-motor learning of fetuses, in order to carry out learning
experiments and model validation for very early stages ofsensory-motor learning.
IV. S OCIALLY S YNGENETIC INTELLIGENCE
This group studies developmental mechanisms of communication and evaluations of intelligence based on human subjects
and the human society. We, humans, anthropomorphize theobject for conversation. This is an essential human abilityand a hint for investigating principles of communicative andintelligent machines by engineering and scientific approaches.
This idea guides us to a new cross-interdisciplinary framework
called Android Science [2]. In this framework, we tackle thefollowing issues

Fig. 4. A body representation: simulated baby body (left) and topographic
somatosensory cluster map (right)

Fig. 5.

Adaptation of body representation in tool use

Early stages of research on the above topics are alreadyin progress. Figure 4 shows the somato-sensory topographic
map (related to topic 1)self-organized based on temporalcorrelations of tactile signals from the skin of the simulated
baby body under random motion in a liquid medium [4]. Figure
5 shows the result of learning to use a stick as an extensionof self body [5]. The model is based on recent findings inneuroscience [6].
Two of the hardware platforms required for the above
research are; (a) whole-body sensory-motor learning robot,and (b) behavior imitation learning robot. We will developthese robots and use them for the above described research on
cognitive models. Robot (a) will have a whole body tactile
sensing skin and whole body motion capabilities, and will
be used for experiments such as body image acquisition bysensory integration learning. Robot (b) will emphasize object

Development of Man-Made Man (M3: M cube): In orderto realize robots that can be accepted by the humansociety, we need to develop a human-like robot. The robot
developed in this project is different from other robotlike humanoids and very humanlike androids [1]. Therobot M3 is going to have about 50 pneumatic actuators,
soft skin covering the whole body, and various humanlike
sensors with a child size.
• Mechanisms for establishing social relationships withhumans: We, humans, have various interactive behaviors
for establishing fundamental relationships with others as
humans, for example eye movement while looking at eachother, personal distance, shard attention, synchronized
movement while interacting, and reactions with tactile
sensation. These human behaviors are precisely measured
and the numerical models are developed [7]. Then, these
programmable models of the behaviors are going to beimplemented into M3.
• Constructive approach for understanding social mechanisms of humans and robots: We are going to implement
more complicated and social behaviors to the robot thathas the fundamental relationships with humans. This is
the constructive approach [8] that starts from the basic
elements and gradationally implements more complex
behaviors. Finally, we acquire knowledge on what isa human and what is the human society by using thedevelop robot.
•V. S YNERGISTIC I NTELLIGENCE M ECHANISM
SI-Mech in short, will conduct the following three issues:1) study of neural correlates of interactive control mechanism between hands and objects,2) investigation of the brain mechanism of imitation andimaging, and3) construction of an acquisition model for fundamental
functions of communication.

functions of left BA44 and BA45 in Broca’s area differ fromeach other; left BA45 is involved in motor sequence prediction
and left BA44 is implicated in motor sequence generation,interacting with the basal ganglia respectively. Second, left
BA45 makes contact with the temporal lobe through longdistance association fibers to integrate semantic and syntactic
information. Finally, in terms of ’the construction of an acquisition model for fundamental functions of communication’, weplan to construct an acquisition model through brain imaging
studies based on the ’motor sequence prediction learning
hypothesis’.
VI. W HAT ’ S EXPECTED ?
Fig. 6. Almost the same region was activated in the condition of pantomiming
and imaging of chopstick use [9]In study of neural correlates of interactive control mechanism between hands and objects, we will use tracing of line
drawing tasks under various conditions to identify the neuralcircuits involved in smooth manipulation of hands and objects.
We will also clarify the relationship between some motor andmental disorders and the neural substrates. Furthermore, Nagai
et al. [10] indicated that patients with Williams syndrome
can trace the drawings of objects, but can not copy them. To
investigate this dissociation between tracing and copying, wewill use functional imaging and neuropsychological methods
to clarify the difference in brain regions involved in thesebehaviors.
On the other hand, mechanisms of prediction and imitation
play important roles in acquiring functions of communication.
Imazu et al. [9] studied neural correlates of tool (chopstick)use with functional neuroimaging and revealed the following
finding: actual use of chopstick activates the right cerebellum,
whereas pantomiming and imagery of chopstick use activate
the left inferior parietal lobule (6). This result suggests thatdifferent neural networks are involved in actual use, andpantomime and imagery of tool use. This could explain thedissociation observed in patients with ideomotor apraxia: they
unable to pantomime with tools, despite their ability to manipulate tools actually.
For the purpose of ’the investigation of the brain mechanismof imitation and imaging’, we will first examine those of actionin the human brain with functional MRI. Second we will makea research on both the deficit and the learning process ofaction imitation of autistic children who have communication
deficits, modeling the relationship between action imitationand communication function. In addition, we are going tocompare the aforesaid study of William’s syndrome withthat of the autism and clarify the essence of communication
function.
Furthermore, the ’motor sequence prediction learning hypothesis’ on the basis of the brain imaging studies as well as
the anatomical ones has been proposed. There are two main
assumptions in the hypothesis. First, it is assumed that theIn this project, we will establish a new discipline that
explains the essential relationship between humans and robots
in future symbiotic society. Systematic and continuous modelling of embodiment,
autonomy, and sociality
from a viewpoint of emergence and development will beperformed, and constructivism will be shown by realizing thesemodels on real robots. Fundamental principles for adaptive
and developable robots will be revealed and may contribute tohuman society.
An expected goal is realization of three years old humanoidthat can show the capabilities of 1) seamless transition among
running, jumping, walking, and stop, and whole body motion
such as rope skipping, 2) naming game with toy playing
and novel behavior generation from imitation, and 3) selfexpression, attention, and conversation.
R EFERENCES[1] K. Terada, Y. Ohmura, and Y. Kuniyoshi. Analysis and control of wholebody dynamic humanoid motion – towards experiments on a roll-and-rise
motion. In Proc. of IEEE/RSJ International Conference on Intelligent
Robots and Systems 2003 (IROS ’03), pp. (CD–ROM), 2003.
[2] H. Ishiguro. Android science. In Proc. A CogSci 2005 Workshop on
Toward Social Mechanisms of Android Science, p. 25 and 26 July, 2005.
[3] M. Asada, K. F. MacDorman, H. Ishiguro, and Y. Kuniyoshi. Cognitive
developmental robotics as a new paradigm for the design of humanoidrobots. Robotics and Autonomous System, Vol. 37, pp. 185–193, 2001.
[4] Y. Kuniyoshi, Y. Yorozu, Y. Ohmura, K. Terada, T. Otani, A. Nagakubo,and T. Yamamoto. From humanoid embodiment to theory of mind.
In F. Iida, R. Pfeifer, L. Steels, and Y. Kuniyoshi, editors, Embodied
Artificial Intelligence, pp. 202–218. Springer, Lecture Note in Artificial
Intelligence 3139 (Berlin), 2004.
[5] C. Nabeshima, M. Lungarella, and Y. Kuniyoshi. Timing-based modelof body schema adaptation and its role in perception and tool use: Arobot case study. In The 4th International Conference on Developmentand Learning (ICDL’05) Osaka, Japan, July 2005, pp. 7–12, 2005.
[6] A. Iriki, M. Tanaka, and Y. Iwamura. Coding of modified bodyschema during tool use by macaque postcentral neurones. Cognitive
Neuroscience and Neuropsychology, Vol. 7, No. 14, pp. 2325–2330,
1996.
[7] T. Kanda, H. Ishiguro, M. Imai, and T. Ono. Development and evaluationof interactive humanoid robots. Proceedings of the IEEE, Vol. 92, No. 11,
pp. 1839–1850, 2004.
[8] H. Ishiguro. Toward interactive humanoid robots - a constructive
approach to developing intelligent robots. Proc. 1st Int. Joint Conf.
Autonomous Agents and Multi-Agent Systems
AAMAS2002), p.
Invited talk, 2002.
[9] S. Imazu, T. Sugio, S. Tanaka, and T. Inui. Differences between actual
and imagined usage of chopsticks. An fMRI study. Cortex, 2005.
[10] C. Nagai, M. Iwata, R. Matsuoka, and M. Kato. Visual recognition
disabilities in williams syndrome - why can they trace but cannot copy?
-. Japanese Journal of Neuropsychology, Vol. 17, pp. 36–44, 2001.