Informing a User of Robot’s Mind by Motion
Kazuki KOBAYASHI1 and Seiji YAMADA2,1
The Graduate University for Advanced Studies
2-1-2 Hitotsubashi, Chiyoda, Tokyo 101-8430 Japankazuki@grad.nii.ac.jp
2
National Institute of Informatics
2-1-2 Hitotsubashi, Chiyoda, Tokyo 101-8430 Japanseiji@nii.ac.jp
1

Abstract
This paper describes a nonverbal approach for arobot to inform a human of its internal state. It is
implemented by executing particular motion. In practical cooperation of a human and a robot, the robot
often requires human’s help to achieve a task. In sucha situation, a robot needs to send useful information
on its internal state like intention, to guide a human tounderstand it and to make him/her to execute actions
for help. A simple solution for how to inform a human of the robot’s internal state is an explicit way inwhich a robot sends verbal information like speech synthesis. However, in terms of human-robot interaction,
implicit and effective approach is preferable because it
must be more natural and inexpensive like the interaction between humans. Thus we propose a motion-based
approach for a robot to inform a human of its internal
state. By such an approach, a robot is able to tell theinternal state to a human naturally, and he/she canunderstand the robot’s state effectively. Some experiments are made for evaluating the advantage of our
method over other nonverbal ways.

1

Introduction

In recent years, home robots including sweeping
robots and pet robots have been widely spread not
only in robotics laboratories but also ordinary homes.
By using such home robots, a user tries to achievetasks like sweeping cooperatively with them. Since a
robot often can not achieve its tasks by itself, it needs
to request a user to help it. For example, a sweeping
robot can not remove a heavy obstacle like a chair ora table, so it needs to request a user to remove it for
sweeping the region under it (Fig. 1).
Thus it is a significant problem how to inform a

Figure 1: Robot needs user’s help.

user of the robot’s internal state. We call such robot’sinternal state mind because it may correspond to human mind in psychology, especially theory of mind[1].
It is not a trivial problem to develop a method to inform robot’s mind to a user because the robot shouldtell him/her it as naturally as communication between
humans. The simple way to inform robot’s mind toa user is to use verbal communication with speechsynthesis like saying “Help me. Please remove the
obstacle.” However such verbal communication significantly depends on language and needs additional
expensive equipments for speech synthesis. We thenfocus on nonverbal communication because some researches show that nonverbal communication has richinformation. Watanabe et al.[2] have argued the importance of nonverbal information, head moving likenodding on the communication in their virtual space.
Komatsu[3] has reported that users can infer attitudes
of a machine by hearing its beep sounds. Matsumaru
et al.[4] have showed that their mobile robot indicates
its direction of movement by a laser pointer or an ani-Figure 2: Some nonverbal ways to inform a user ofrobot’s mind.

mated eye, and confirmed those availabilities. Nonverbal information is an essential factor for human-robot
social interaction[5] and instruction methods which a
robot observes human actions[6, 7]. Thus nonverbal
methods are preferable because of rich information, noor the least additional equipment and independence of
language.
In this paper, we propose a motion-based method
for a robot to inform a user of its mind in a nonverbal way. We can consider several nonverbal ways toinform a user of a robot’s mind. For example, a robot
can employ motion like struggle behavior, sound like
beep, lighting of LED and so on (Fig. 2). We consider
a motion-based informing to be the best way in termsof the feasibility and the effectiveness for designing theconcrete motion to inform a user of robot’s mind in an
obstacle-removing task. The motion is designed according to ethological policy. By applying ethologicalpolicy, we are able to execute motion so that humanor animal may perform so and narrow the candidatesof motion. In an obstacle-removing task, we designed
back-and-forth motion in front of an obstacle. We conducted experiments in comparison with other nonverbal methods, and obtained promising results.

between a human and a robot. However they did notinvestigate which modality is effective for mind reading of a robot, and we try to develop a motion-based
method to facilitate mind reading.
Psychology, in particular, TOM(theory of mind)[1]
is closely related to our work. In TOM framework, firsta person P-1 recognizes the other person P-2. Then
P-1 recognizes an object which P-2 gazes (joint attention), and the trinomial relation among P-1, P-2 andthe object occurs. P-1 eventually uses theory of mind
to infer the P-1’s mind. Our work deals with a situation in which a mobile robot faces an obstacle and can
not remove it for going ahead. Since P-1, P-2 and an
obstacle correspond to a human, a mobile robot and an
object respectively, our work is understood with TOM
framework. Though TOM is useful for describing ourtask, it does not provide how to design interaction forfacilitating human’s mind reading of a robot. We givea solution to such a problem.
In addition, our study has a close relation to the research about understanding of intentions. Dennett[9]has mentioned that human beings use three kinds ofstances when they try to understand a system’s actions. However, it is difficult to apply his ideas to designing a robot because the elements which the robot
should have for informing a user of its mind is still
unknown. Terada[10] and Sato[11] discuss artifacts
which behave in agreement with human intentions. In
general, intention understanding needs a high processing costs and much knowledge about robot’s tasks. In
contrast, we investigate the robot which indicates its
mind by simple methods in a simple task, and try toobtain general knowledge for designing robots.

3
2

Related work

Some previous work on human-robot interaction isconcerned with our study. Ono and Imai[8] studied
how human familiarity to a robot influenced humanrecognition of robot’s mind. At beginning of an experiment, a participant had experience in growing alife-like agent on a PC, and his/her familiarity to theagent significantly increased. Then the life-like agent
moved from the PC into a mobile robot and appeared
in a laptop PC on the mobile robot. Finally participants tried recognition of a robot’s noisy speech, andthe results showed that a robot with an agent is muchbetter than a robot without an agent. Their work isvery important in the sense of interesting attempt todevelop a concrete method to increase the familiarity

Informing a user of robot’s mind by
motionWe explain an obstacle-removing task and propose
a method to inform robot’s mind in the task.

3.1

Task: requesting a human to remove
an obstacle

We can easily imagine that a sweeping robot which
can not remove an obstacle like a chair requests a user
to remove it for sweeping the region under the obstacle(Fig. 1). We call such a task an obstacle-removing
task and employ it as a general test bed task for ourwork because it occurs frequently and easily in various
cooperative tasks of a human and a robot. In orderto achieve an obstacle-removing task, a robot needs

to inform a user of its mind which shows that it has
difficulty in removing the obstacle and wants him/her
to remove it.

3.2

Nonverbal approach

One of the main objectives in human-robot interaction is to construct natural interaction between ahuman and a robot. Thus as developing a method to
inform a user of the robot’s mind, the method should
be natural for a user and should not force him/her
cognitive load. Also TOM tells us that one of naturalinteraction between humans is nonverbal communication. Hence we consider nonverbal approach informing
robot’s mind to a user to be preferable to verbal one,and develop such a nonverbal method.

3.3

Advantage

We have some alternatives of modalities for such anonverbal method like sound, lighting, motion and soon. A robot can employ motion like struggle behavior,sound like beep, lighting of LED (Fig. 2). We consider
a motion-based informing to be the best way for thefollowing reasons.
• Feasibility: A robot must be designed to execute motion for achieving various tasks. Thus amotion-based informing method needs no additional and expensive implementation like a LED
or a speaker. In contrast, other nonverbal approaches need such implementation.
• Variation: By motion-based approach, we candesign motion as informing methods for various
tasks. The variation of motion-based informing
methods are far larger than that of other nonverbal methods.
• Less stress: Other nonverbal methods, particularly sound, may force a user to direct his/her attention to a robot and take more stress than motion. The motion-based method sends no bothering signal to a user, and the user may just see themotion naturally.
• Effectiveness: We intuitively consider motionbased informing to be more effective than othernonverbal ones. Because the interesting motion
seems to adequately attract user’s attention to arobot without stress.
Note that the feasibility, the variation and the lessstress properties of motion-based informing are valid,

Figure 3: Back-and-forth motion.

however the effectiveness is a assumption we believe
and it should be verified by the experiments. Thus
such experiments will be conducted in later sections.

3.4

Design policy of motion

We design concrete motion for informing robot’s
mind in an obstacle-removing task. We propose backand-forth motion as general motion which is neededin various tasks. A robot goes back and forward fourtimes in a short period of time in front of an obstacle along its trajectory. Fig. 3 shows behavior of the
back-and-forth motion.
We design back-and-forth motion according to
ethology. Most of animals have action patterns andrepeated them[12]. The back-and-forth motion expresses properties of such universal actions of the animals in terms of repetition and a sudden change ofmovement. A user can easily understand the robot’s
mind by looking its motion. We call this ethologicaldesign policy. Arkin et al.[13] have applied ethological models for robots and investigated its effectiveness.
In our study, some other types of motion for a robot
are available, however, we consider the back-and-froth
motion to be the most attractive for a human.
Back-and-forth motion is easily implemented because a robot may repeat just going back a little along
the trajectory and going ahead a little. Also this backand-forth is applicable to any situation in which an
obstacle is in front of a robot, thus it is considereda general method to achieve tasks including obstacleremoving.

4

Experiments

The purpose of the experiments is to verify theeffectiveness of our motion-based informing in an

Figure 5: Pattern of the behavior.
Figure 4: KheperaII
the ’off’. A buzzer which makes the sound of 6kHz
and 53dB (at 100mm from it) is equipped on its
top. We determined the sound pressure to be ahuman conversation level (50dB – 60dB) on theequal loudness curve (ISO226), and the experimental room has the sound pressure of 34dB.

obstacle-removing task. We compare the motionbased method of a robot with other two nonverbal
methods.

4.1

MethodFig.6 shows the experimental environment which
has a flat surface (400mm × 300mm), a wall surrounding it and two obstacles. It simulates an ordinary human working space like a desktop. The obstacles correspond to an object such as a pen-stab, a remote controller and so on and can be moved easily by a human.
We use a small mobile robot KheperaII (Fig.4). The
robot has eight infrared proximity and ambient lightsensors with up to 100mm range, a processor Motorola
68331 (25MHz), 512 Kbytes RAM, 512 Kbytes Flash
ROM, and two DC brushed servo motors with incremental encoders. The program written by C-language
runs on the RAM.
Participants observe the robot which sweeps out thefloor in the environment and indicates its mind in thefollowing three methods:(1) back-and-forth motion: the robot performs
back-and-froth motion which is composed of fourback-and-forth actions and four stop actions. The
robot behaves back-and-forth at the ’on’ in Fig.5,
and stops at the ’off’. It goes back for 0.075 sec.
and goes forward for 0.075 sec. in a back-andforth action.
(2) LED light: the robot performs LED lighting action. The robot turns the light on at the ’on’ in
Fig.5, and turns it off at the ’off’. The red colored LED with diameter of 3mm is equipped on
its top. We chose the red color of LED because itmeans warning like a traffic signal.
(3) beep sound: the robot performs beeping action
which is composed of beeping and muting. The
robot beeps at the ’on’ in Fig.5, and is silent at

The back-and-forth, the lighting and beeping are thesame in timing of the ’on’ and ’off.’ We consider thepattern of Fig.5 to be valid because the robot can performed easily and a user can observe the pattern certainly. The robot stops and performs their indication
when they meet an obstacle or a wall. After indicating, they turn left or right and then go ahead. If therobot senses an obstacle on its right (left), it turns left
(right) for a given length of time. The robot repeats
such actions while an experiment is performed. Notethat the robot cannot sweep the dust away.
Participants are instructed as follows: “This robot
is a sweeping robot. Actually, it cannot sweep the dust
away. So, please consider the floor to be cleaned by
the robot. You can move or touch everything in thisenvironment. Please help it if necessary.”
A participant performs a training and two trials and experiences three methods : back-and-forth, lighting andbeeping. The order of the methods for the participant
is random. A training or a trial is finished when the
robot meets obstacles three times, or the participant
replaces an obstacle when the robot indicates its mind.

4.2

Evaluation

We measure the number of times which the robot
meets obstacles until a participant moves the obstacle placed near it. It means ease of understanding
for robot’s mind. To measure the period from the
beginning of an experiment to the moment which aparticipant moves an obstacle is better for the evaluation, however, it is difficult because the time which
the robot reaches the first obstacle is somewhat different in each trial. Slips of robot’s wheels cause its
trajectory to be changed.

Figure 6: Experimental environment.

Figure 8: The experimental appearance.

5
5.1

Figure 7: The ratio which a participant moved theobject at robot’s first indication.

4.3

Results

Participants are 17 persons (male:11, female:6,
age:21–44). Fig.7 shows the results of the experiments.
In the figure, each bar represents the ratio which a participant moved the object at robot’s first indication.
A numerator represented in each bar is the numberof persons who moved the object, and a denominatoris the total number of persons who experienced themethod. The ratio of the motion is highest value. The
result of the statistical test, Pearson’s Chi-squared
test shows a significant difference among those threemethods (χ2 = 8.947, df = 2, P = 0.0114), and themultiple comparison, Ryan’s method also shows significant differences between the motion and the LED(diff.=0.562, RD=0.492, P=0.00568, α0 = 0.0167)
and between the motion and the beep (diff.=0.477,
RD=0.444, P=0.0221, α0 = 0.0333). Fig.8 shows theappearance of a experiment. Participants sat on thechar and helped the robot on the desk.

Discussion
Generality of our approach

In section 4, we obtained the promising results inthe experiments for comparison with other nonverbal
methods. However, how general are the results? How
widely are the results applied to? The answer mainly
depends on the generality of an obstacle-removing
task. As we mentioned before, a robot frequently andeasily needs the obstacle-removing in cooperative tasks
between a human and a robot. Thus we consider the
obstacle-removing task to be general in the cooperation of a user and a robot.
On the other hand, for other tasks without obstacleremoving, we may need to design another motion. We
consider that the ethological policy can be applied toother complicated tasks.

5.2

Manual free design

Users need to read manuals of machines when they
buy those newly or want to use those more conveniently. However, reading manuals forces us to impose
a high workload. It is better for a user to discoverrobot’s functions easily and naturally without reading
manuals.
The results of our experiments shows that the
motion-based indication enables users to understandrobots’ mind easily. We then consider the motionbased indication to be useful for making manual freemachines, and are currently constructing the procedure of discovering robot’s functions naturally without reading manuals. The procedure is composed ofthree steps: (1) indication of robot’s mind, (2) actionof its user, (3) reaction of the robot. The discovery ofrobot’s functions is achieves when the user finds thecausality between a user’s action and a robot’s one in
the steps. Our experiments satisfy the step (1) and(2), and the motion-based indication could contributefor human to discover such causality easily.

6

Conclusion

We proposed a motion-based method for informing
a user of a robot’s mind in a nonverbal way. There
are various nonverbal approaches like motion, sound,lighting and so on. We developed a motion-based informing as the best way in terms of the feasibility andthe effectiveness. Then an obstacle-removing task was
introduced as a general task for cooperation between
a human and a robot, and we designed the back-andforth motion to inform a user of robot’s mind to request removing an obstacle. The motion is designed
according to ethological policy. Eventually, we conducted experiments in comparison with other nonverbal methods, and obtained promising results.

References[1] S. Baron-Cohen, Mindblindness: An Essay on
Autism and Theory of Mind. MIT Press, 1995.
[2] T. Watanabe, M. Okubo, and M. Inadome, “Virtual communication system for human interaction
analysis,” in Proc. of the 7th IEEE International
Workshop on Robot and Human Communication,
1998, pp. 21–26.
[3] T. Komatsu, “Can we assign attitudes to a computer based on its beep sounds?” in Proceedingsof the Affective Interactions: The computer in theaffective loop Workshop at Intelligent User Interface 2005 (IUI2005), 2005, pp. 35–37.
[4] T. Matsumaru, K. Iwase, K. Akiyama, T. Kusada1, and T. Ito, “Mobile robot with eyeballexpression as the preliminary-announcement anddisplay of the robotrsquos following motion,” Autonomous Robots, vol. 18, no. 2, pp. 231–246,
2005.
[5] T. W. Fong, I. Nourbakhsh, and K. Dautenhahn,“A survey of socially interactive robots,” Robotics
and Autonomous Systems, vol. 42, no. 3–4, pp.
143–166, 2003.

[6] Y. Kuniyoshi, M. Inaba, and H. Inoue, “Learning
by watching: extracting reusable task knowledge
from visual observation of human performance,”
IEEE Transactions on Robotics and Automation,
vol. 10, no. 6, pp. 799–822, 1994.
[7] M. Nicolescu, M.N.and Mataric, “Learning andinteracting in human-robot domains,” IEEE
Transactions on Systems, Man and Cybernetics,
Part A, vol. 31, no. 5, pp. 419–430, 2001.
[8] T. Ono and M. Imai, “Reading a robot’s mind:
A model of utterance understanding based on thetheory of mind mechanism,” in Proc. of Seventeenth National Conference on Artificial Intelligence, 2000, pp. 142–148.
[9] D. C. Dennett, The Intentional Stance.
Press, 1987.

MIT[10] K. Terada and T. Nishida, “An active-affordancebased method for communication between humans and artifacts,” in Sixth International Conference on Knowledge-Based Intelligent Information and Engineering Systems (KES’02), 2002,
pp. 1351–1356.
[11] T. Sato, Y. Nishida, J. Ichikawa, Y. Hatamura,and H. Mizoguchi, “Active understanding of human intention by a robot through monitoring of
human behavior,” in Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS‘94), 1994, pp. 405–414.
[12] P. J. B. Slater, An Introduction to Ethology.
Cambridge University Press, 1985.
[13] R. C. Arkin, M. Fujita, T. Takagi, and
R. Hasegawa, “An ethological and emotional basis for human-robot interaction,” Robotics andAutonomous Systems, vol. 42, no. 3–4, pp. 191–
201, 2003.