Towards Ubiquitous Human-Robot Interaction

Hideaki Takeda, Nobuhide Kobayashi, Yoshiyuki Matsubara, and Toyoaki Nishida
Graduate School of Information Science,
Nara Institute of Science and Technology
8916-5, Takayama, Ikoma, Nara 630-01, Japan
takeda@is.aist-nara.ac.jp
http://ai-www.aist-nara.ac.jp/

Abstract
Multimodality for interaction tends to be considered
as use of di erent physical communication channels
for face-to-face interaction. But our usual communication is indeed more exible, e.g., communication
to others at a distance, communication to someone
who can reply, and communication with a group ofpeople together. We categorize intimate, loose, andcooperative interaction as extended multimodal interaction. In this paper we show how such di erent
types of interaction is realized as an integrated systemwith robots, people, and computers. Firstly, we introduce our multi-agent architecture to model the environment which includes people, robots, automatedinstruments, and computers. Each of them is modeled
as agent, and all communication is realized in an interagent communication language. Secondly, we describe
how intimate interaction is realized. We use gesturerecognition, gesture generation, and speech generation
for interaction. Thirdly, we introduce a software agent
called \watcher" to realize loose interaction. Watcher
always looks at the environment to detect whether
someone is requesting interaction. It also uses gesture recognition in a coarse level. Fourthly we provide
mediators for cooperative interaction. A mediator is
invoked each time interaction is requested in the environment. It can gather and enroll necessary agents forinteraction by planning and sometimes by consulting
other mediators.

Introduction

Recent advances in computer science and robotics
make robots more applicable to our daily life. Robots
and people will co-exist with sharing and cooperatingtasks in various ways (e.g., (Nauba, Powers, & Bircheld 1995)(Asoh et al. 1996)). The arising problem is
how to communicate and cooperate with people. We
need natural ways for people to communicate and cooperate with robots just as same as they do with otherpeople, i.e., people interact with other people anywhereat anytime. We call such kind of communication andcooperation \ubiquitous human-robot interaction".
The primitive way for human-robot interaction isinteraction through special instruments. People cancommunicate with robots by using instruments likecomputers. Recent technologies for multimodal communication can provide various communication channels like voice and gestures(e.g., (Darrell & Pentland 1993)). Interface agents (e.g., (Maes & Kozierok
1993))can be used for their communication. But people could not communicate with robots directly, andthey are bound to computer terminals.
Other way is direct interaction with people androbots. In addition to multimodal communication withcomputer, robots can use their bodies when they communicate to people. Although it is more restricted
than virtual interface agents because of their mechanical structures, physical motion are more natural andacceptable for people. We call such physical direct interaction between robot and people intimate interaction.
The intimate interaction enables people multimodal
direct interaction, but another problem arises. People
and robots should be close to each other to establish
such interaction. It is obstacle to realize ubiquitousinteraction among people and robots. We also needloose interaction such as interaction among people androbots who are apart from each other or interaction
among people and anonymous robots which are readyto response.
Although loose interaction absorbs the distanceproblem between people and robots, interaction is stillclosed within participants of interaction. We sometimes need more robots (or even people) involved toaccomplish interaction. For example, a robot is asked
to bring a book by a person, but it has no capacity tobring books. It should ask another robot which canbring books and the person should interact anotherrobot as a result. We call this type of interaction cooperative interaction. Cooperative interaction makes
interaction extensive, i.e., interaction can be extended
by introducing more robots and people as much as itneeds. It can solve the problem of limitation of functions of each robot so that interaction should not bebound to functions of robots which people are interacting.
In this paper, we show how we can realize such three
di erent types of interaction together. In Section 2,

Real-world Agent

query

KQML

reply
KQML comminication
sub-agent

DB sub-agent

hardwaresub-agent

Figure 1: Architecture of real-world agents
we introduce our multi-agent architecture to model
the environment which includes people, robots, automated instruments, and computers. Each of them ismodeled as agent, and all communication is realized
in an inter-agent communication language. In Section 3, we describe how intimate interaction is realized. We use gesture recognition, gesture generation,and speech generation for interaction which are dependent to situation of interaction. In Section 4, we introduce an agent called \watcher" to realize loose interaction. Watcher always looks at the environment to detect whether someone is requesting interaction. It alsouses gesture recognition in a coarse level. In Section
5, we provide mediators for cooperative interaction. A
mediator is invoked each time a task is thrown in theenvironment. It can gather and enroll necessary agentsfor the task by planning and sometimes by consulting
other mediators. We discuss our approach with related
work and conclude the paper in the last two sections.

Multi-agent architecture
for real-world agents

We have developed multi-agent architecture for realworld agents (Takeda et al. 1996). The basic idea
is that all participants in the environment are modeled as agents which can communicate to each otherin the knowledge level. We provide shared ontologies for object, place, and action, and all agents canuse them to some extent. The degree how muchontologies are understandable is dependent on physical and computational abilities of agents. Communication is realized as KQML message(Finin et al.
1994). KQML(Knowledge Query and Manipulation
Language) is a protocol for exchanging information andknowledge among agents. KQML is mainly designed
for knowledge sharing through agent communication.
We provide mainly two types of agents, software
agents and real-world agents. Software agents arethose which do not posses physical interaction meth-

Figure 2: Two mobile robots

Figure 3: Rack and door agentsods like cameras and motors, while real-world agent
is those which are agenti ed robots and instrumentsby adding software modules for message interpretationand knowledge- and data-bases. Figure 1 shows a typical architecture of a real-world agent which consistsof three sub-agents. KQML communication sub-agent
can interpret and translate KQML messages from otheragents into local messages, and vice versa. Databasesub-agent can memorize states of the real-world agent,and hardware sub-agent can operate actuators like motor and obtain sensed data from sensors.
We currently agenti ed a mobile robot with two manipulators called Kappa1a and a mobile robot without
manipulators called Kappa1b (see Figure 2). A manipulator has six degrees of freedom and a gripper. We
also have computer-controlled rack and door as realworld agents (see Figure 3).

Intimate human-robot interaction

The rst interaction we investigate is intimate interaction which is direct one-to-one interaction between
people and robots. We provide two communication
channels, i.e., gesture and vocal communication. People can tell their intention by using their gestures, andFigure 4: Extraction of motion area
(move (object righthand)(direct up))(move (object lefthand)(direct down))(move (object bothhand)(direct left))

Figure 5: Examples of primitive hand motions
the real-world agent can tell its intention by its gestures and voice 1 .
Gesture recognition is implemented in a relatively
simple way and can only extract gestures by hands.
Firstly the agent identi es motion areas of hands bysearching a black part in the scene and assuming it
person's head. Secondly, it de nes rectangle areas adjacent to both sides of the black part as motion areasof hands(see Figure 4 ). Thirdly, it detects motion ofhands by optical ow. The result is sequences of primitive hand motions which are speci ed by hand anddirection (see Figure 5 ). Then gestures are identied by comparing detected sequences of motions withknowledge on gestures. Figure 6 shows examples ofknowledge on gestures. We provide some gestures like\shake", \wave", and \move both hands".
There needs another step to know meaning of suchdetected gestures, because meaning of gestures is dependent on situation of interaction. In our system, thereal-world agent reacts to gestures according to predened state transition network. Each state has actions
that the real-world agent should take and some links toother states. Each link has conditions described withgestures of the person and its sensor modes. If one ofconditions of link of the current state is satis ed, thecurrent state is shifted to next state which is pointed
by the link. Since a single gesture can be included
in conditions of multiple links, multiple interpretationof gestures is possible. Figure 7 shows an example ofintimate interaction.
The real-world agent has variety of actions. One is

Speech recognition is not implemented yet, so people
cannot tell their messages by voice. The other researchgroup has implemented direction-sensitive speech recognition on the same robot, but it is not integrated to oursystem.
1(define wave"(struct(sequence((move (object righthand)(direct right))(move (object righthand)(direct left))(move (object righthand)(direct right))(move (object righthand)(direct left))))))(define shake(struct(sequence((move (object righthand)(direct up))(move (object righthand)(direct down))(move (object righthand)(direct up))(move (object righthand)(direct down))))))

Figure 6: Examples of de nition of gestures

Figure 7: An example of intimate interaction (\throw
it out")(define Come_on(content((behavior wave)(sourcecamera)(client?human)))(task((subject camera)(come (subject kappa1a)(destination ?human)))))Figure 9: Knowledge on task composition
Figure 8: Scene by camera for watcher
informative actions or gestures which cause no physical changes of the environment like \Yes", \No", and\Ununderstanding" using head motion, and \bye-bye",\raise both hands" using hand motion. Voice generation is also included in possible informative actions of
the real-world agent. Other is e ective actions whichcause physical changes of the environment like \grasp
something" and \release something" using hand motion, and \move to somewhere" using driving units.
We currently provide some interaction modes like\take a box", \Janken2 ", and \bye-bye". Some interaction is closed within the real-world agent and theperson, but others not. If the latter case, the realworld agent asks tasks to mediator in order to involveother real-world agents. We will discuss this process
as cooperative interaction in Section 5.

Loose human-robot interaction

Loose interaction is interaction between people androbots who are separated. Since robot may not see theperson, the same method for intimate interaction is notapplicable. We introduce an agent called \watcher"which watches a room to nd what is happening in
the room. It uses a camera to look over the room (seeFigure 8) and communication to other agents.
If watcher notices a request from someone to others,
it composes a task description and passes to mediator. Noti cation of requests comes by either recognition of camera scenes or communication from otheragents. Recognition of camera scenes is achieved by
the same way in gesture recognition in intimate interaction. Watcher currently observes two areas, i.e.,

It is a children's game in which two or more personshow one of three forms of hand to each other. The agent
uses hand motions instead of forming hands.
2around a door and a desk (two boxes in Figure 8). An
example of knowledge on task composition is shown in
Figure 9. This de nition tells \if it is found by camerathat someone is waving, compose a task that Kappa1ashould go to her/his position". As a result, the person
who waves can tell her/his intention to the real-worldagent even if it is not near her/him (see Figure 10). It is
important that watcher should not make direct orders
to real-world agents but tasks which can be scheduled
by mediator. If the appointed agents are busy to process other tasks, the composed task may be postponed
until the current task is nished, or be processed byother agents.

Cooperative human-robot interaction

As mentioned in Section 1, some interaction should be
extended to include agents needed to accomplish its
purpose, i.e., interaction should be performed cooperatively by more than two agents. Suppose that a person
is facing a robot which cannot take and carry objects
and asking the robot to bring an object to her/him.
The robot may try to do it by itself and nally nd
it cannot, or simply refuse her/his request because it
knows that it is impossible for it to do it. A better solution is that the robot should ask other robots which
can take and carry objects to perform the recognizedrequest. In this case, three agents, i.e., a person andtwo robots are necessary members to accomplish theinteraction.
We have developed cooperation among real-worldagents using mediator (Takeda et al. 1996). We here
extend it to process multiple asynchronous requests.
The basic idea is that every emerged interaction tries
to gather and control necessary agents independently.
Each mediator processes a single interaction by usingstate information of the environment and communication with other mediators if necessary.
In cooperative interaction, we abstract requests tointeraction as tasks. A task is described as an incomplete action which has some properties like subject

(fetch (object solaris)(destination human1))((move (subject Kappa1b)(from(at Kappa1b))(to(in_front_of Rack)))(handover (object
solaris)(from_place (on Rack))(to_place(on Kappa1b))(subjectRack))(move (subject Kappa1b)(from (atKappa1b))(to(in_front_of Human2)))(tell (subject Kappa1b)(atKappa1b)(content (talk Here_you_are))))Figure 10: An example of loose interaction (a camera
behind the robot detected human request and told therobot to go)

Figure 11: An example of completion and decomposition of tasks

and object. Incompleteness means that all properties
should not be speci ed. Unspeci ed properties will be
ful lled by mediators using the current state of theenvironment like what each agent is doing and where
objects are. Interaction is thus formed dynamicallyaccording to the current state of environment.
Requests to interaction are processed in the following way;
1. To compose tasks to realize the given requests. If requests are detected by cameras, this process is doneby watcher. Otherwise requesting agents themselvescompose tasks and send them to the watcher. Thenthe watcher invokes a mediator and delegates thereceived task to it.
2. To complete and decompose the given tasks by using
knowledge on object, place, and action, and current
state of the environment. This process is done bymediator, especially planner which is a componentof mediator. The result of this process is a sequenceof actions for real-world agents. Figure 11 shows anexample of completion and decomposition of tasks.
If some necessary agents are occupied by other mediator, the mediator asks to the occupying mediator
how long it would occupy those agents. According
to the answer, the mediator decides either it would
wait for release of those agents, or abandon the current plan. At the latter case, it tries to make analternative plan, otherwise it replies to the watcher
failure of planning.
3. To execute sequences of actions. It is done by executor which is the other component of mediator. It
repeats to ask agents to perform a single action andwait for its nish of all actions in the sequence. Eachagent of participants of the plan is occupied by the
executer from the beginning of the plan until all ac-

tions for it are nished. If any action is failed bysome reason, the executor noti es failure of the plan
to planner, and planner again generates a plan for
the task.
Figure 12 shows how the example of cooperative interaction mentioned above can be solved in our system.
The following numbers correspond those in Figure 12.
1. The person asks the mobile agent (Kappa1a) in frontof her/him to bring a manual for C++ by gestures.
2. The real-world agent understands the request, but
nds it is impossible for it to bring the manual, because the manual is now on the rack and it cannot
take objects on the rack. Then it decides to compose
a task for the request and sends it to the watcher.
3. The watcher invokes a mediator newly for the task,
and delegates the task to the mediator.
4. The mediator completes and decompose the task
into a sequence of actions each of which is executable
for an agent. The sequence means that another mobile agent (Kappa1b) will nd the position of thebook, receive it from the rack agent by working together, and bring it to the person. Then it executes
this sequence one by one.
5. In the above process, the mediator nds that
Kappa1a would be obstacle for Kappa1b to approach
the person. It composes a new task that Kappa1a
would go out, and sends it to the watcher.
6. Receiving the task, the watcher invokes another mediator and delegates the task to it.
7. The secondly invoked mediator also completes anddecompose the delegated task to a sequence of actions. Then it executes them one by one.

4. Task A is completed
and decomposed

7. Task B is completed
and decomposed

move (subject Kappa1b)
find (subject Kappa1b)
move (subject Kappa1b)
handover (subject Rack)
move (subject Kappa1b)
tell (subject Kappa1b)

tell (subject (kappa1a)
move (subject Kappa1a)

5. Task B (Move out
Kappa1a) is
composed
Mediator

Kappa1bRack

6. Task B isdelegated
WatcherMediator

2. Task A (Take amannual) is
composed
3. Task A isdelegated

Kappa1a

Human1

camera

1. A request is accepted
move
tell
tell

WorkStation

Take a mannual for C++
Coffeemakerkappa1a
find
move

move

move

kappa1bRack

handover

Door

Possible positions for persons

Figure 12: An example of cooperative interaction

Related work

There are many studies on robots co-existing with people. Most of them are to build autonomous robots intelligent enough to co-exist with people. For example,oce conversant robot (Asoh et al. 1996) is aiming
an autonomous robot highly integrated with learningfacilities. But real environments are still too complex
to learn. Our system compensates such complexity byagent communication among such robots, agenti ed instruments and computers.
Human-robot interaction is also an important issue
for this paper. Many researchers are interested in direct human-robot interaction, especially understanding of human behavior (e.g., (Kortenkamp, Huber, &Bonasso 1996)). The other way is to capture humanintention with various kinds of instruments. Sato etal.(Sato et al. 1994) proposed integrated use of various monitoring methods to understand human intention. Expression to people is equally important tounderstanding of human behavior. There are manystudies on software agents for man-machine interaction
from interface agent(e.g., (Maes & Kozierok 1993)) tohuman-like agents (e.g., (Maes 1995) and (Tosa 1995)),
but physical interaction can make interaction more reliable for people (e.g., (Nakata et al. 1995)). As indirect human-robot interaction, Suzuki et al.(Suzuki etal. 1995) evaluated various kind of communication between a human operator and multi-agent robot system,
but people are supervisors of robots not equal partners.
Coordination of agents is the other aspect of this paper(Cao et al. 1995). As centralized approach, Haighand Veloso(Haigh & Veloso 1996) showed robot control for asynchronous and incomplete requests. As
distributed approach, many methods like contractnet(Smith 1980) and negotiation (Brafman & Shoham1995) are used. Our method is medium-grain distribution, i.e., distribution by task. It is more exible
in hetrogeneousity for agents and tasks than fully distributed one because it is processed just as centralized
coordination after reserving agents.

Discussion and conclusion

We showed in this paper that multimodality for interaction should be re-de ned as that with more exibility for participants and relations among them. Multimodality for interaction tends to be considered asuse of di erent physical communication channels forface-to-face interaction. But our usual communication is indeed more exible, e.g., communication toothers at a distance, communication to someone who
can reply, and communication with some people together. They are also our communication channels.
De nition of modality for communication should include at least physical communication channels, physical relations among participants like distance, and nonphysical relations among them like speci ed or unspeci ed participants and direct or indirect interaction.

Our system is an attempt for interaction with the extended modality. We model all equipments in the environment as a kind of agent, so-called real-world agent
from robot to camera. Real-world agents are membersof the multi-agent systems in the computer network as
well as those of the physical environment. They can exchange information to each other directly, or via special
software agents like mediator and watcher. The agent
communication compensates for complexity of interaction in the physical environment. The physical interactions in voice and gestures is captured by the mostconvenient agents in the situation, and other interaction is performed by agent communication. As the result, a person in the environment should not consider
which she/he should communicate to, but just communicate to something in the environment. Then thesystem would coordinate necessary interaction. Since
intimate, loose, and cooperative interaction is thus integrated, we can say that people only have one communication channel, that is, communication to the environment.

References

Asoh, H.; Motomura, Y.; Hara, I.; Akaho, S.;
Hayamizu, S.; and Matsui, T. 1996. Comibing probabilistic map and dialog for robust life-long oce navigation. In Proceedings of the 1996 IEEE/RSJ International Conference on Intelligent Robots and Systems, volume 2, 807{812.
Brafman, R. I., and Shoham, Y. 1995. Knowledge
considerations in robotics and distribution of robotics
task. In Proceedings of IJCAI-95, 96{102.
Cao, Y. U.; Fukunaga, A. S.; Kahng, A. B.; andMeng, F. 1995. Cooperative mobile robotics: Antecedents and directions. In Proceedings of the 1995
IEEE/RSJ International Conference on Intelligent
Robots and Systems, volume 1, 226{234.
Darrell, T., and Pentland, A. 1993. Space-time gestures. In Proceedings of IEEE 1993 Computer Society
Conference on Computer Vision and Pattern Recognition, 335{340.
Finin, T.; McKay, D.; Fritzson, R.; and McEntire,
R. 1994. KQML: An information and knowledge exchange protocol. In Fuchi, K., and Yokoi, T., eds.,
Knowledge Building and Knowledge Sharing. Ohmsha
and IOS Press.
Haigh, K. Z., and Veloso, M. M. 1996. Interleaving
planning and robot execution for asynchronous user
requests. In Proceedings of the 1996 IEEE/RSJ International Conference on Intelligent Robots and Systems, volume 1, 148{155.
Kortenkamp, D.; Huber, E.; and Bonasso, R. P. 1996.
Recognizing and interpreting gestures on a mobile
robot. In Proceedings of AAAI-96, 915{921.
Maes, P., and Kozierok, R. 1993. Learning interface
agents. In Proceedings of AAAI-93, 459{465.

Maes, P. 1995. Arti cial life meets entertainment :lifelike autonomous agents. Communications of the
ACM 38(11).
Nakata, T.; Sato, T.; Mizoguchi, H.; and Mori, T.
1995. Synthesis of robot-to-human expressive behavior for human-robot symbiosis. In Proceedings of the
1995 IEEE/RSJ International Conference on Intelligent Robots and Systems, volume 3, 1608{1613.
Nauba, I.; Powers, R.; and Birch eld, S. 1995.
Dervish an oce-navigating robot. AI Magazine
16(2):53{60.
Sato, T.; Nishida, Y.; Ichikawa, J.; Hatamura, Y.; andMizoguchi, H. 1994. Active understanding of humanintention by a robot through monitoring of humanbehavior. In Proceedings of the 1994 IEEE/RSJ International Conference on Intelligent Robots and Systems, volume 1, 405{414.
Smith, R. G. 1980. The contract net protocol: Hightlevel communication and control in a distributed
problem solver. IEEE transaction on Computer C29(12).
Suzuki, T.; Yokota, K.; Asama, H.; Kaetsu, H.; andEndo, I. 1995. Cooperation between the human operator and the multi-agent robotic system: evaluationof agent monitoring methods for the human interface
system. In Proceedings of the 1995 IEEE/RSJ International Conference on Intelligent Robots and Systems, volume 1, 206{211.
Takeda, H.; Iwata, K.; Takaai, M.; Sawada, A.; andNishida, T. 1996. An ontology-based cooperative environment for real-world agents. In Proceedings ofSecond International Conference on Multiagent Systems, 353{360.
Tosa, N. 1995. Network neuro-baby with robotics
hand. In Anzai., ed., Symbiosis of Human and Artifact, 77{82. Elsevier Science B.V.