Robovie-IV: A robot enhances co-experience

Noriaki Mitsunaga

1

Takahiro Miyashita

mitunaga@atr.jp
Hiroshi Ishiguro

Yuichiro Yoshikawa

miyasita@atr.jp

1,2

Kiyoshi Kogure

ishiguro@ams.eng.osaka-u.ac.jp

1

1

1

y-yoshi@atr.jp

1

Norihiro Hagita

kogure@atr.jp

1

hagita@atr.jp

ATR Intelligent Robotics and Communication Laboratories
2-2-2 Hikaridai Keihanna-gakken-toshi, Kyoto

2

Graduate School of Engineering, Osaka University
2-1 Yamadaoka Suita City, Osaka, Japan

Abstract

ally be achieved upon recommendations by other people orby obtaining knowledge about a particular experience from

People establish and enhance their social relationships
through sharing experiences with others.

some media such as books, Web pages, and television. Are

Among media

these the only possible media? No. In fact, one of the mostused to promote such experiences, a human-size humanoid

promising candidate media is a humanoid robot whose sizeis the most promising medium. In addition to capabilities

is similar to that of humans. It's potential partly lies in itsof other media, such as the ability to collect and provide in-shape, which allows for a sense of human-like expressions

formation from the Internet and via ubiquitous sensors con-or emotions, wishes, and so forth. We can envision a futurenected through a network, a human-size humanoid has thescene where humanoid robots cohabit with us ubiquitously

potential to help people share experiences or information

to facilitate co-experience or as ubiquitous (co)experienceamong them owing to its shape. It enables people to ac-

media.

cept a sense of humanlike expressions or emotions, wishes,

We have been developing a series of communicationand so forth. We have recently developed a human-size hu-

robots called Robovie to investigate how a robot can servemanoid, called Robovie-IV, as a ubiquitous medium toto facilitate experience and co-experience. Based on knowl-enhance co-experience. This paper discusses the design re-edge obtained in the process of developing the previous

quirements of Robovie-IV and introduces an overview of its

Robovies, we have developed Robovie-IV as an ubiquitous

hardware and software architectures. There is also a brief

experience medium to enhance co-experience in human-

description and preliminary results of an experiment thathuman and human-robot-human settings. In the next sec-

we are currently conducting with Robovie-IV, in addition totion we discuss what is necessary for a robot as an ubiqui-conclusions.

tous experience medium and describe the design concept of
Robovie-IV. Then we present an overview of Robovie-IV's
hardware and the software architecture. Following that, we

1. Introduction

show an experiment we are currently doing with RobovieIV and nally give conclusions.

A person's experience often compels the person to improve the quality of her/his activities, and such an experi-

2. A robot enhances co-experience

ence may be utilized by other people. Therefore, sharing
an experience or knowledge on an experience is important.

What kinds of capabilities are necessary for a robot

Moreover, experience sharing or co-experience can serve to

as a ubiquitous experience medium?

establish or to enhance social relationships among people.

the following requirements to achieve natural and effective

However, a person may sometimes hesitate to participate

human-robot communication. First of all, it should be self-

in a novel experience or may not have a novel experience

contained. Although its computers, database systems, andmerely because of the person's ignorance of its existence.

even sensors can be outside of its body, it should have mech-

Facilitation of experience is important, and this can usu-

anisms to perform both verbal and nonverbal communica-

There are at least

Robovie-III was developed to enable involuntary motion fora robot [7]. It uses a wheeled inverted pendulum mechanism
for locomotion. Since the inverted pendulum is controlled
by feedback on its posture, involuntary motion of the wholebody occurs. In addition to effecting involuntary motion,
the wheeled inverted pendulum has a unique feature: Whensome external force is applied to the body from its front, it
moves backwards. Since its wheels are controlled to maintain its posture, it moves backwards to keep the posture as it
tilts backwards due to the applied force. We also adopt thewheeled inverted pendulum mechanism for Robovie-IV.
Fourth is human recognition. People expect a robot tobe able nd and identify them. On top of that, to be able
Robovie­IIS,

to share the memory of co-experience with humans, human

Robovie­IIF, and haptic interaction between

identication is mandatory. Many methods for human de-children and Robovie­IIS.

tection and identication using images have been proposed;

Figure

1.

From

left

to

right,

however, with current computational power and video camera resolution and viewing angles, conditions under whichtion by itself.

It should be able to move around with nowires for smooth communication because the distance between a person and a robot cannot be ignored in order toachieve effective communication. Also it should not havean outward appearance that frightens or discomforts people.
Second is the capability of haptic communication. Haptic communication is as important as vision and voice.
People who are familiar with each other often touch eachother's hair or hug each other; such haptic interaction reinforces their familiarity. If a communication robot equippedwith tactile sensors over its entire body could have thesame capability of haptic interaction as human do, the robot
would give us greater familiarity, thus shortening its communicative distance from people. To study haptic communication, we have previously developed two types of hu-

those methods work are still limited. Consequently, we decided to use a laser range sensor to nd human leg candidates and an optical/RF tag system for human identication.
Robovie-IV nds leg candidates using a laser range sensor,veries the results with its camera, and identies the detected human with a tag.
Based on the above discussion, Robovie-IV is designed
as a robot 1) whose height is the same as a child's; 2) whose
whole body is covered with soft, light-colored skin for a soft
look and touch; 3) with many tactile sensors are embeddedin the soft skin; 4) which can move in two modes, one isa normal wheeled robot mode with passive casters, and theother is the inverted pendulum mode; and 5) which has optical and RF tag readers and laser range sensors for humanidentication.

manoid robots, Robovie-IIS and Robovie-IIF, that have tactile sensors embedded in a soft skin that covers the robot's
entire body [8].

3. The hardware architecture

These robots were developed based on

Robovie-II [5]. Figure 1 shows overall views of Robovie-

Figure 2 shows front and side views of Robovie-IV. The

IIS, Robovie-IIF, and a scene of communication with a hu-

height of the robot is about 1 m, which is smaller thanman. Robovie-IV has tactile sensors based on the technique

Robovie-II/II-S/II-F (which are 1.2 m high). As in the g-

we used for these robots.

ure, it has two arms which have four dof each, one head withThird is the locomotion mechanism that can generate involuntary motions.

Human motion can be classied into

two types: voluntary and involuntary motion[1, 10, 2].

pan, tilt, and roll joints, four spare wheels (passive casters),
two powered wheels, and one jack to enable the inverted
pendulum state.

Robovie-IV is equipped with two video

Voluntary motions are a set of motions made to achieve

cameras with pan and tilt joints, one camera with an omni-

given tasks or intentions. Going to a certain place, mov-directional mirror, a microphone and a speaker, an opticaling an arm forward for a handshake, vocalization to say

tag reader (easily detachable, not shown in the gure), an

hello, and reacting to a pat in order to know who did it are

RF tag reader (the system is based on the active tag systemsuch examples. Involuntary motions, on the other hand, arecalled Spider by RF Code Inc. [11]), two laser range sen-a set of incidental motions such as feedback control aris-sors in the front and back, two gyros that sense the sameing in response to physical stimuli from the environment

axes (we use two to take the average of their outputs in or-

without prior planning or motivation.

These motions do

der to reduce random noise, and to detect failure so that the

not correspond to tasks directly but instead consist of mo-

robot does not fall over), and 56 tactile sensors. Figure 3

tions that make the robot appear to behave more naturally.

displays the arrangement of the sensor elements that are em-

2 gyro sensors(1 axis x2)

56 tactile
sensors

2 laser range
sensors

Front viewleft arm
controller (SH2)right arm (4dof)

right arm
controller (SH2)

2 wheels
Left side view

2 cameras with pan/tilt
omni camera

frame grabbersound in/out

200

4 spare wheels(passive casters)

1 jack(1dof)

left arm (4dof)

NTSC signals

56 tactile
sensors

connections
via RS-232
Sony EVI protocol

1 RF tagreader(diameter of wheels)

2 arms(4dof x2)

head controller
(SH2)

s

1 optical
tag reader

neck (3dof)bu

1 speaker and1 microphone22-4

2 cameras withpan and tiltneck
(3dof)

skin
sensorskin
sensorskin
sensorskin
processing(SH2)skinsensor
sensorprocessing(SH2)
processing(SH2)
processing
processing(SH2)(SH2)

RS-422

tactile sensors
tactile sensors
tactile sensors
tactile sensors
tactile sensors

RScamera withomni-directional
mirrorgyros
wheels (2dof)

wheel controller
(SH2)

jack

Figure 2. Front and left­side views of Robovie­

PC base computer(Pentium-M 2[GHz])

speaker2 laser range sensors

microphone

RF tag reader

IV. Fitted actuators and sensors are shown.

optical tag reader
Figure 4. Robovie­IV's hardware architecture.

bedded in the soft skin. The gure clearly shows that there
are tactile sensors under the nose, ears, and in the thumbs

There are four motor controllers, ve skin pro­
cessors, and one main computer.

for Robovie-IV to sense contact to those areas.

Corp.) and the main computer has a Pentium-M processor
that runs at 2 GHz. The PC can be connected to the sensors
via a wireless LAN.
Figure 5 shows the structure of a tactile sensor element
embedded in the soft skin. As the gure illustrates, the soft
skin consists of four layers. The outside layer is made ofthin silicone rubber, and the middle layer is made of thicksilicone rubber. We use these silicone rubber layers to realize humanlike softness. The inner layer is made of urethanefoam, which has a density lower than that of the silicone
rubber; the densities of the urethane foam and the silicone

33

Figure 3. The arrangement of the skin sensor

rubber are 0.03 g/cm

elements.

total density of the soft skin consisting of all layers is 0.6

There are 56 sensors in its soft

and 1.1 g/cm , respectively.

The

3g/cm . Robovie-IV's tactile sensor elements are lm-typeskin (#16 is not implemented).

piezoelectric sensors inserted between the thin and thick silicone rubber layers. These lm-type sensors, consisting of
polyvinylidene uoride (PVDF) and sputtered silver, output
Figure 4 shows Robovie-IV's hardware architecture.

a high voltage proportionate to changes in applied pressure.

There are four PID motor controllers connected to a PC

Since the middle and the inner layers deform easily upon

via RS-232 dedicated for wheels, left and right arms, andhuman contact with the skin, the sensor layer can easily de-neck joints. The jack and the inverted pendulum are con-tect the contact.

Two laser range sensors

Figure 6 illustrates how the powered wheels, the spareand optical and RF tag readers, and cameras with pan/tilt

wheels, and the jack are connected. The spare wheels arejoints are also connected to the PC via RS-232.

trolled by the motor controller.

Signalsconnected to the main body directly, while the poweredfrom the tactile sensors are fed to ve skin processors. The

wheels are connected via the jack. Robovie-IV can selectprocessed signals are then sent to the PC via the RS-422

two locomotion modes, with or without spare wheels, bybus. Images from three cameras are captured by a frame

controlling the jack.

grabber installed on the PC. A speaker and a microphonemoves as a normal robot with two powered wheels, but

are connected to the sound output and input of the PC. Each

without them, it moves in inverted pendulum mode.

controller has a SH2 micro-processor (Renesas Technology

utilize the wheeled inverted pendulum controller proposed

With the spare wheels, Robovie-IVWeences of the protocols from the robovie4. Process robomap4
is the one for self-localization. It receives information fromThin silicone rubber
Piezoelectric film (PVDF)
Thick silicone rubber
Urethane foam

the two laser range sensors and odometry from robobase4and compares it with the map of the environment.

Theestimated position is returned to robobase4 and sent to
robovie4. Then, robocam4 processes images from the cameras.

The processed results are directly sent to robovie4.

The PostgreSQL [13] is a popular database engine. We use
the database store and recall the co-experiences with huFigure 5. The skin's structure, which consistsof four layers.

The piezoelectric sensor isinserted between the outer and middle layers.

mans.
The julian process is a process of a grammar-based
recognition parser named Julian. Julian is a modied version of Julius [6], which is a high-performance, two-passlarge-vocabulary continuous speech recognition (LVCSR)
decoder software for speech-related researchers and developers. We have prepared manually-designed DFA grammar
as the language model and use it for the speech recognition
by Julian. The robovie4 process connects to the julian process and commands the selection of dictionary and so on.
The recognized words are sent back to robovie4.

4.1. Writing programs for Robovie­IV
The robobase4 process mainly consists of an event handler written in C++ and tasks written in Lua [12]. Lua is a
scripting language that can be embedded in a C or C++ program. Although we can write the whole program in C++,
we adopted a hybrid approach that enables us to write tasks
Figure 6. The powered wheels are connectedby a jack to the main body and spare wheels.
The

gure

shows

the

inverted

pendulummode.

interactively.

Tasks written in Lua run in parallel by thecollaborative multithreading supported by Lua.
The events from sensor values are detected in the event
handler, negating the need to copy an event detection foreach task. The event handler in robobase4 repeatedly 1) collects sensor values from julian, robocam4, and robobase4;2) detects events; 3) sends events to tasks; 4) receives com-by Ha and Yuta [4].

mands from tasks; and 5) sends commands to robobase4 as
in Fig. 7.
The tasks repeatedly 1) waits for events from the event

4. The software architecture

handler; 2) reads sensor values sent from robocam4, julian,or robobase4; reads from and writes to the database; andFigure 7 presents an overview of Robovie-IV's softwarearchitecture. The OS on the PC is Linux and a box withbold line in the gure indicates a process running on Linux.
There are six processes, robovie4, robobase4, robomap4,
robocam4, PostgreSQL, and julian. The processes are connected through FIFOs, sockets, and shared memories. The
process robovie4 makes decisions for the robot from theinternal state, information on the database, and the sen-3) decides the next action.
task written in Lua.

Figure8 shows an example

It repeatedly and innitely 1) waits

for touch events; and 2) prints out touch innitely. Fromthe example, it is clear that we can write the waiting foran event in a sequence, unlike in other objected oriented
approaches such as OPEN-R [3]. We decided to use thisapproach since many robot tasks can be easily written as
sequences of behaviors.

sor information gathered and processed by the other processes. Processors on the motor controllers control motors

4.2. Processing tactile sensor signals

as directed by the program called robobase4 running on the
PC. Process robobase4 handles the communication between
controllers and sensors, while robobase4 hide the differ-

Figure9 shows the raw data from a tactile sensor for

four kinds of touch behaviors by one of the authors. From

camerassensors and actuators

microphonerobovie4

julianrobocam4

1.

Capture images fromcameras, process them, andsend results to robovie4

1

robobase4

1

Collect sensor values fromsensors and send them torobovie4
Receive commands fromrobovie4 and transfer themto actuators

Event handler

1

Send results of speech
recognition to robovie4

2.
3.
4.
5.

5

Collect sensor
values
Detect events
Send events
Receivecommands
Sendcommands

Tasks (written in Lua)
Face tracking

3
Reflection to contact

Follow Human4robomap4
Map matching

Dialogue

PostgreSQL

Database engine.

Figure 7. Robovie­IV's software architecture. The PC's OS is Linux. There are six processes robovie4,
robobase4, robomap4, robocam4, PostgreSQL, and julian.

function task1(co)local self = TaskTable("task1",
co, task1)
Register(self)

processor reads the signal, it classies the signal by the algorithm shown in Fig. 10. It rst checks whether the signal
is classied as no touch/still contact or stroke/hit
by the signal's strength. If the signal is weak, it checks thecorrelation to the 60 Hz sine wave, and if the correlation iswhile (1) do
CatchEvent("R4Touch", self)
SleepTask()

high, the current signal is classied as still contact; otherwise, no touch. If the signal is strong, it checks the output
of BPF (ve-order BPF, low and high cut-off freq.: 70 Hz
and 100 Hz). The results are sent to the host PC via the RS-

print("Ouch");end
end

422 serial bus. We have implemented and tested these sensor processes, and found the results satisfactory. Although
this process is very simple to implement, we believe that thefour behaviors covered by it are the most important ones intactile communication.

Figure 8. An example task written in Lua.

It

repeatedly and innitely 1) waits for touch
events; 2) prints out touch.

the gure we can see that four behaviors create clearly different signals for the tactile sensor. We use the word notouch to indicate that he did not touch the skin, and stillcontact to indicate that he just placed his palm on the skinright on a sensor element. At this time, though the signal
is very weak, we can see the induced signal from the powerline (60 Hz in our case). We use hit to indicate that hehit the skin on the sensor element.

We see a strong im-

reading of ADC

20000
10000
0
no touch

still
contact-10000
0

hit

0.5

1
1.5
time [sec]stroke2pulse, which means the signal has a wide spectrum including higher frequencies. We use stroke to indicate that heFigure 9. The raw data from a tactile sensor

stroked a part of the skin close to the sensor element. We

for four kinds of touch behaviors by a human.

see that the signal is strong but it is not as peaky as thehit signal.

With the above knowledge gained from ourexperiment, we have implemented the tactile classication
program onto the skin sensor processor.

The tactile sen-sor output is read and processed at 200 Hz. Every time theRead sensor data
䎓

䎐䎓䎑䎕

䎓

䎕

䎗
䎙
䏗䏌䏐䏈䎾䏖䏀

䎛

䎔䎓

䎐䎓䎑䎕

䎐䎓䎑䎗

䎓

䎕

䎗
䎙
䏗䏌䏐䏈䎾䏖䏀

䎛

䎔䎓

䎓

䎕

䎗
䎙
䏗䏌䏐䏈䎾䏖䏀

䎛

䎔䎓

No

No
no touch

Does BPF output islarger than threshold?

䎐䎓䎑䎕

䎐䎓䎑䎗

䎐䎓䎑䎗

Does the correlation to 60[Hz]is larger than threshold?

䎓䏓䏒䏖䏌䏗䏌䏒䏑䎾䏐䏀No䏓䏒䏖䏌䏗䏌䏒䏑䎾䏐䏀䏓䏒䏖䏌䏗䏌䏒䏑䎾䏐䏀䎓Is it larger than threshold?

still contact

strokeFigure 11. The characteristics of the threehit

controls.
Figure 10. The algorithm to classify the tactile
sensor signal.

㪎㪅㪇㪇average㪍㪅㪇㪇standard deviation

points㪌㪅㪇㪇

5. What do people feel about involuntary mo-

㪋㪅㪇㪇
㪊㪅㪇㪇

tion?

㪉㪅㪇㪇
㪈㪅㪇㪇
control 1

As we have stated earlier, involuntary motions are im-control 2

control 3

portant for natural communication. However, different controls give different impressions since an involuntary motion

Figure 12. The points of controls for adjective

incidentally arises from the system's control mechanism.

pairs relevant to the extroversion factor.

We have prepared three different controls for the inverted
pendulum. Figure 11 shows the characteristics of the threecontrols when the robot is pushed by a force of the same
strength from the forward to reverse direction.

The rst

tives relevant to those factors.

In the questionnaire, we

one controls the wheels to keep the body in a certain po-placed three pairs of adjectives for each personality factor

sition and the velocity to zero.

The robot is pushed andfrom their adjective list (15 pairs in total) to conform to thestarts moving at 2 s, goes backwards to -0.4 m behind itsthemantic differential method. The pairs of adjectives were

initial position, then gradually returns to the rst position topresented at either end of a seven-point scale. For exam-

which it was moved. Note that in the graph, it seems that theple, `dumb' was placed at 1 on the scale, while `lively' wasrobot moved forward immediately after the push. However,placed at 7.

this just indicates the wheel's rotation forwards in order to

As a result, we found that three controls differ with re-compensate for the rebound, not the real movement. The

spect to the extroversion factor. Figure 12 shows the points

second one controls the wheels to keep the body in a certainof each control. The point is the average of the adjective

position and the velocity to zero, while the desired position

pairs relevant to the extroversion factor for each of ve sub-changes to -0.3 m after the robot feels the push. The gurejects. We see that the subjects felt greater extroversion fromshows that the robot gradually goes back to -0.3 m after thecontrol 1 than from controls 2 or 3. Although these are still

push. The third one controls the wheels to keep the body

preliminary results, they do indicate that involuntary mo-

velocity to zero. This one does not maintain the position.

tions are important factors in communication.

The gure shows that the robot goes back, gradually slows
down, and stops.
We gathered 15 subjects and asked ve of them to push

6. Conclusions

the robot with control 1, ve to push it with control 2, andve to push it with control 3. We asked them to ll in a

We have developed a human-size humanoid,

calledquestionnaire after they had pushed the robot until they un-Robovie-IV, as a ubiquitous medium to enhance co-derstood its movement (most of them pushed two or threeexperience. We discussed what is necessary for a robot astimes). We prepared the questionnaire to explore how eachan ubiquitous experience medium and described its design

subject feels about the robot's personality.

concept.

According to

Next, we introduced the hardware and softwarethe ve-factor model [9], the personality of a human isarchitectures of Robovie-IV. It has also been developed ascomposed of ve factors, that is, extroversion, agreeable-

a self-contained robot that fullls the requirements for aness, conscientiousness, neuroticism, openness, and intel-

ubiquitous experience medium. That is, Robovie-IV is aslect. Murakami and Murakami [9] found Japanese adjec-tall as human child, its whole body is covered by a soft-

skin containing 56 embedded tactile sensors, it locomotes[11] RF Code, Inc. http://www.rfcode.com/

by an inverted pendulum mechanism that causes involun-[12] The programing language Lua, http://www.lua.org/

tary motions, it has sensors for human recognition, and it[13] PostgreSQL Global Development Group.

has a database system to memorize co-experiences with humans. In addition, we introduced an experiment on involuntary motion, with the results revealing that the involuntary motions could be one of the most important factors inhuman-robot communication. We believe that the RobovieIV's architecture will be the basis of ubiquitous experience
media to enhance co-experience.

Acknowledgement
This research was supported by the National Institute ofInformation and Communications Technology of Japan.

References[1] H. Bruno.

Repp.: phase correction in sensorimotor syn-chronization: nonlinearities in voluntary and involuntary responses to perturbations. Hum. Mov. Science, 211(1):137,
2002.
[2] C.Wu. Analysis of voluntary movements for robotic control.
In Proceedings of the IEEE International Conference on
Robotics and Automation (ICRA91), pages 326331, 1991.
[3] M. Fujita. Development of an autonomous quadruped robot
for robot entertainment.

Autonomous Robots, 5(1):718,

1998.
[4] Y. Ha and S. Yuta. Trajectory tracking control for navigation of self-contained mobile inverse pendulum. In Proceedings of the IEEE/RSJ/GI International Conference on Intelligent Robots and Systems 1994 (IROS94), pages 18751882,
1994.
[5] H. Ishiguro, T. Ono, M. Imai, T. Maeda, T. Kanda, and
R. Nakatsu. Robovie: A robot generates episode chains inour daily life. In Proc. of Int. Symposium on Robotics, pages
13561361, 2001.
[6] A. Lee, T. Kawahara, and K. Shikano.

Julius  an open

source real-time large vocabulary recognition engine.

In

Proceedings of the European Conference on Speech Communication and Technology (EUROSPEECH), pages 1691
1694, 2001.
[7] T. Miyashita and H. Ishiguro.

Human-like natural behav-

ior generation based on involuntary motions for humanoid
robots. International Journal of Robotics and Autonomous
Systems, 48:203212, 2004.
[8] T. Miyashita,
N. Hagita.

T. Tajika,

H. Ishiguro,

K. Kogure,andHaptic communication between humans androbots. In Proceedings of the 12th International Symposiumof Robotics Research, 2005.
[9] Y. Murakami and C. Murakami.

BigFive Handbook (inJapanese). Gakugei-tosho, 2001.
[10] Y. Zheng. A neural gait synthesizer for autonomous bipedrobots. In Proceedings of the IEEE International Workshop
on Intelligent Robots and Systems (IROS90), pages 601
608, 1990.

http://www.postgresql.org/