2nd International Conference on Autonomous Robots and Agents
December 13-15, 2004 Palmerston North, New ZealandInteraction Design with Commands Embedded in
Actions for Human-Robot Cooperative Task
Kazuki KOBAYASHIThe Graduate University for Advanced Studies
2-1-2 Hitotsubashi, Chiyoda
Tokyo 101-8430, Japankazuki@grad.nii.ac.jp

Seiji YAMADA
National Institute of Informatics
2-1-2 Hitotsubashi, Chiyoda
Tokyo 101-8430, Japanseiji@nii.ac.jp

Abstract
This paper proposes a novel interaction model for human-robot cooperative task. The point of the interaction
model is an introduction of CEA (Commands Embedded in Actions) by which a human work-load is reduced
because it need less inputs and outputs of a user of a robot than that of using direct commanding methods in aconventional interaction model. We build a procedure of the design for the proposed interaction model, and apply
the method of interaction design to a cooperative sweeping task by a human and a small mobile robot. We thenconduct experiments to confirm that reduction of a human work-load on our interaction design in the sweepingtask. Human cognitive loads are examined to evaluate human work-load while a human interacts with a robot, andloads between CEA and direct control of a robot are compared. The results of the experiments show that the CEA
minimizes a human cognitive load in comparison with other direct commanding methods.

Keywords: interaction design, cooperative task, sweeping, cognitive load
1 Introduction
There are robots spreading among people as a progression of its technologies. We can purchase pet robots
like AIBO [1] or cleaning robots like Roomba, andinteract with them in a home environment. We will see
tour-guide robots [2] in a museum in the near future.
Robots thus have transferred their scene from industrial environments to home environments. How a homerobot interacts with people is one of most important
issue to be accepted by people who want to share theirtime and spaces with robots.
Various researches have been studied in a field of
human-robot interaction. Most of the researches havedealt with methods of communication between ahuman and a robot such as gesture [3, 4, 5], speech[6, 7, 8], using control devices such as joysticks orcomputers [9, 10, 11], multiple methods [12] andothers [13, 14, 15, 16]. These types of interaction
focusing into a function of a robot are describedin Fig.1 in which the methods of communication
correspond to the arc of command, and tasks of ahuman correspond to the arc of action. This figure
assumes that a robot works based on the cycle ofsensing, interpretation, and execution. The arc of
human’s command means that a human gives a robot
information which a robot cannot sense or interpret,
and the arc of human’s action means human’s tasks
which a robot cannot execute it because of the difficultyof equipping necessary hardware for given tasks. For
instance, a commercial cleaning robot such as Roomba

301

Figure 1: Conventional Interaction

Figure 2: Suggested Interaction
can sweep out a room autonomously, however a humanactually needs to help the robot by removing obstacles
because of equipping no hardware for handling the
obstacles. Two tasks are assigned to the human: tocontrol the robot by remote controller and to remove
the obstacles.
The improvement of the arc of human’s command suchas gesture recognition, speech recognition and so on

2nd International Conference on Autonomous Robots and Agents
December 13-15, 2004 Palmerston North, New Zealandare insufficient for an actual task of a human in termsof his/her work-load: the human have to achieve twotasks. We therefore focus on a human’s given tasks represented by the arc of human’s action and deal with thiskind of work as a cooperative task between a humanand a robot to reduce the load of a human. We thenintroduce a new interaction model of Fig.2 in which
no direct interaction between a human and a robot are
employed. A human can control a robot by executing his/her actions to environment. We call this CEA(Commands Embedded in Actions) by which output
and input of a human can be reduced. A human does
not need to execute direct commanding to a robot andunderstand a way for communication with it. The existence of a robot is practically transparent for a human,and it leads reduction of a human work-load.
There are studies related our suggested interaction
model for a human-robot cooperative task. Their
cooperative task is to carry a long or big object by ahuman and a robot based on manipulator [17, 18, 19],
and outdoor cooperative tasks by a human and ahumanoid [20]. These studies are typical instances of
CEA. The robots can work well for helping a humanby sensing force of a shared object without directcommanding methods. However, interchanging force
is only allowed on such cooperation. In contrast,
interchanging various information including force isallowed by CEA between a human and a robot. Our
aim is to build up a general framework for human-robot
cooperation. Therefore, CEA is a novel approach in
the research area of human-robot interaction.
After this section, in Section 2, we describe the procedure of interaction design for a human-robot cooperative task. In Section 3, we apply our interaction
design to a cooperative sweeping task between a humanand a robot. A developing of a behavior-based robot
is described in Section 4. Section 5 describes experiments to compare CEA using suggested interaction design with direct control using conventional interaction
design such as voice or hand manipulation in terms ofhuman’s cognitive load. The experimental results areindicated in Section 6. Finally, Section 7 concludes
this paper.

2 Method of Interaction Design
We propose the procedure of interaction design for ahuman-robot cooperative task illustrated in Fig.2 as follow:
1. Divide a given task into a human’s task and arobot’s task.
(a) Determine a robot’s task as maximize robot’s
autonomy within given cost of hardware design.
(b) Determine a human’s task as assign the task
which a robot cannot execute autonomously.
(c) Determine cooperative behavior and information interchanging between a human and a robot.

302

2. Embed commands in actions.
Commands including the information of a robot
are communicated to a robot as a human executes
his/her actions. Actions are determined as minimize the change in space and time from a typicalhuman’s action for an assigned task.
3. Design functions of a robot.
Add the functions to a robot to sense or interpret
CEA.
CEA can be designed through above procedure. Although details depend on a given task, the procedure
can assist interaction design for reducing human workload. CEA has advantages described as follow:• No additional cognitive load in execution:
Since CEA has minimum additional actions totypical human’s actions to achieve an assigned
task, a human does not need additional cognitive
load by direct communication with a robot andsmoothly does cooperation with a robot byexecuting only typical actions.
• No understanding a way for communication:
Since direct communication is not necessary in
CEA, a human does not need to understand communication protocol like gesture commands, special speech commands. CEA thus releases a human from learning protocol and training to communicate with a robot.

3

Design on Cooperative Sweeping

We deal with a human-robot cooperative sweeping by
a human and a small mobile robot as a cooperative
work. A goal of the cooperative task is to sweep out
a desk including the region of under an object. In thissection, we first describe an experimental environment
and a specification of small robot. Then we apply theprocedure of our interaction design to the cooperative
sweeping.

3.1

Environment and robot

Fig.3 shows an experimental environment where a human and a robot work cooperatively. This environment
simulates a place used by a human routinely such as a
desktop. A desk swept out by a robot has a flat surface
and a wall which encloses the region for keeping arobot not to fall.
We use a small mobile robot KheperaII. The robot
has eight infrared proximity and ambient light sensors
with up to 100mm range, a processor Motorola 68331(25MHz), 512 Kbytes RAM, 512 Kbytes Flash ROM,and two DC brushed servo motors with incremental
encoders. The program written by C-language runs on
the RAM.

2nd International Conference on Autonomous Robots and Agents
December 13-15, 2004 Palmerston North, New ZealandLayer 3 : Interaction
44 cm(Cooperative Behavior)

Projected Image

Layer 2 : Sweeping(Autonomous Behavior)

WallSensor Value

Layer 1 : Obstacle Avoidance

Object33 cm

Action(Autonomous Behavior)

Figure 5: Subsumption architecture
KheperaII

1. A human picks up an object when a robot
approaches an object. Then a robot comes into
the region under the object picked up.

Hand

2. While a robot is in under an object, it keeps turning at the object’s edge.

Figure 3: Experimental environment

3. A robot goes out of the region when a human approaches an object to it.

4

Figure 4: Cooperative sweeping by CEA

3.2 Interaction Design
The procedure of our interaction design is applied forcooperative sweeping as follow:
1. Divide a sweeping task into a human’s task and arobot’s task.
(a) A robot’s task is to sweep out a desk
autonomously with strategy of a random turn.
The hardware resource of a robot is equal to thatof a commercial sweeping robot.
(b) A human’s task is to move an object because
a robot cannot move an object by itself.
(c) Cooperative behavior is that a robot sweeps
out the region under an object when a humanmoves the object.
2. Embed commands in actions
A command sent by a human makes a robot sweepout the region under an object. This command ismade by human actions to achieve his/her task.
Fig.4 shows CEA in which the human’s action has
no changes in the trajectory from the typical one,and it has little additional time to keep picking
up. A human does not need to move an object
to another place.
3. Design functions of a robot.
An extra infrared sensor which measures the distance in vertical direction is added on a robot’s
general I/O turret to sense CEA.
The detail of designed CEA in Fig.4 is described asfollow:

303

Behavior Design of Mobile Robot

Our robot performs obstacle avoidance, going forward(when no obstacles are on its front) and turning for
random direction. We use sweeping with selecting directions at random because it has lower cost than acalculated sweeping using a map. Many methods forregion covering have been developed [21], and mostof them need precise position of a robot. However,
our robot cannot use reliable methods like dead reckoning. We hence consider that the random sweeping
is adequate to our study because the most of consumer
sweeping robots adopt this kind of method.
A robot is implemented by behavior-based approach,and we adopt subsumption architecture [22] to manage
following behaviors: (1) obstacle avoidance, (2)autonomous sweeping while no object is sensed, and(3) interactive sweeping while an object is above a
robot. Fig.5 shows the robot’s behaviors into thethree layers in subsumption architecture. Each layer
asynchronously checks the applicability of behaviors
and executes applicable ones. Higher layers suppress
lower layer’s behaviors, and lower layers have morereactive behaviors. The behaviors of each layer consistof multiple actions. When the system obtains multiple
outputs, it generally selects the highest layer’s action.
Each layer has output frequency of action to control
the robot smoothly. We set the frequency as obstacle
avoidance: an action by 5msec, sweeping: 10msec,
interaction: 5msec, obstacle avoidance and interaction
occur most frequently.

5

Experiments

We conduct experiments to confirm that our interaction
design reduces work-load of human. We examine human cognitive load to evaluate human work-load, andcompare the load between CEA and direct control of a

2nd International Conference on Autonomous Robots and Agents
December 13-15, 2004 Palmerston North, New Zealand

Box

KheperaII

Sweeping Area

EC
(Embedded)
PC

Projector

SC(Sound)HC(Hand)

Figure 7: Three types of interactions
EC : A robot performs sweeping by CEA.
SC : A robot sweeps with receiving a sound commandby hand clapping.

USB Camera

HC : A robot sweeps with receiving a command byhand as blocking the robot’s line.
Figure 6: Experimental system
robot. The experiments are performed on our experimental system described in the next subsection.

5.1 Experimental system
Fig.6 shows the experimental system which can
indicate a robot’s trajectory. This system consists of asweeping area and a projection system. In experiments,
a human interacts with a robot on the sweeping areain Fig.3 indicating a swept location by the projection
system including a personal computer, a projector, anda USB camera.
The projection system detects a robot’s location using
a picture of two beams of infrared LEDs equipping on
the robot. The robot’s location is calculated by image
processing in the picture from the camera, and then an
image indicating the trajectory is created with the location. This image is ultimately projected on the sweeping area.
The projected image also includes small square cells.
A cell is lit when a robot enters the cell in real time.
These small cells therefore express the trajectory of arobot. The sweeping area having a width of 44 cm anda height of 33 cm is divided into 16 × 12 cells. Cells of
3 × 3 approximately correspond to the area of a robot.

5.2 Cognitive load measurement
We measure the cognitive load of a human interactingwith a robot by CEA, and compare the load to that of
direct control. Two direct control methods are chosen
as typical one without extra devices such as remotecontroller and a human can create a command by moving his/her body. These control methods are shown in
Fig.7, and the detail is described as follow:

304

The robots receiving such commands are prepared byadding extra sensors such as microphone or modifying
the program of a robot. When a robot controlled bydirect command receives a command from a human, it
turns from 90◦ to 180◦ randomly. A robot controlled
by CEA also turns from 90◦ to 180◦ randomly when it
runs out from under an object.
Measurement starts when a robot enters the region ofunder an object, and it continues until all cells of theregion are swept. A box whose size is 4 × 4 cells is
employed because sweeping time of the region is appropriate for subjects and the measurement. In the EC,
subjects keep to pick up an object until all the cellsof under the box are swept. In the SC and HC, first,
a human relocates a box to a corner of the sweeping
area, and then send a command for a robot to be turnby making sounds or approaching their hand to it when
it is likely to run out from the region of a box.
We use a dual task method to measure human cognitive load. Subjects have to do mental arithmetic as asecondary task while controlling the robot as a primarytask [9, 23]. They count backwards by three from arandomly selected three-digit number vocally. We obtain the average number of correct answers per second,and evaluate the human’s cognitive load for controlling each robot. Subjects are required to calculate as
quickly as possible, and to prioritize the controllinga robot over the counting. They practice controlling
robots and the counting well before experiments begin.
The order of EC, SC and HC for each subject is determined at random, and these experiments are recorded
three times respectively for a subject. Subjects are alsomeasured counting ability without operations of a robot
before a measuring of EC, SC, or HC respectively. The
counting ability is the number of correct answers of thecounting for 30 seconds. Fig.8 shows the experimental
appearance.

2nd International Conference on Autonomous Robots and Agents
December 13-15, 2004 Palmerston North, New ZealandFigure 8: Experimental Appearance

Therefore, CEA plays the significant role in a humanrobot cooperative task.

Figure 9: Results of scores and differences

5.3 Results
Subjects are eight men and four women between theage of 22 and 32. Each subject has three scores: EC,
SC, and HC. A score is the average of normalized
number of correct counting answers per second fora subject. The normalization is to divide the correct
answers per second by correct answers per second
without operations of a robot. Fig.9 shows averaged
scores, standard deviations, and differences tested
by Dunnett’s test. Each EC, SC and HC is theaverage of all subject’s scores. The number 1.0 meanscounting ability of each subject without operations ofa robot. EC has the highest average. The difference
between EC and HC has a significant difference(p << 0.01, α = 0.01,t = 3.938). The difference
between EC and SC also has a significant difference(p = 0.033, α = 0.05,t = 2.414).

6 Discussion
The results of the experiments show that the CEA reduces a human cognitive load in comparison with otherdirect commanding methods. CEA has a low cognitive
load because of minimizing cost of sending commands
and also shortening the trajectory of moving a box.

305

Physical loads concern the cognitive loads. Eachcommanding method accompanies motions of humanarms. Hence, the measured cognitive loads include
a load of the motions and a load of human attention.
However, we consider that the effects of humanmotions on the measured cognitive loads keep aminimum because the motions of human arms are
intuitive and natural. Actually, the subjects have no
choice except to clap his/her hands or to move his/herarms in the experiments. Therefore, the difference
between CEA and direct control methods is attention
for a robot. A subject has to repeat the cycle ofobservation of a robot and execution of moving his/herarms in the experiments of SC and HC. Contrastively,in the experiments of EC, a subject does not need toconcentrate his/her attention on a robot, and he/she caninteract with environment as Fig.2. In addition, theexperiments are set to be fair deal between CEA anddirect control methods in terms of controlling a robot
without specific devices.

7

Conclusion

We proposed a novel interaction model for a humanrobot cooperative task. The point of the interaction
model was an introduction of CEA (Commands Embedded in Actions) by which a human was able to control a robot through execution of his/her actions to environment. CEA led reduction of a human work-loadbecause it needed less inputs and outputs of a user of arobot than that of using direct commanding methods
in conventional interaction models. We developed aprocedure of the design for the proposed interaction
model. The procedure maximized robot’s autonomy
and minimized a human work-load. It also dealt withcooperative behavior of a human and a robot on a giventask. We applied the method of interaction design to acooperative sweeping task by a human and a small mobile robot. The experiments for conforming reductionof a human work-load on our interaction design wereconduced on the sweeping task. Human cognitive loads

2nd International Conference on Autonomous Robots and Agents
December 13-15, 2004 Palmerston North, New Zealandwere examined to evaluate human work-load while ahuman interacted with a robot, and loads between CEA
and direct control of a robot were compared. The results of the experiments showed that the CEA minimized a human cognitive load in comparison with otherdirect commanding methods. Therefore, we consider
CEA led by our interaction design method plays thesignificant role in a human-robot cooperative task.

References[1] G. S. Hornby, S. Takamura, J. Yokono, O. Hanagata,
T. Yamamoto, and M. Fujita. Evolving robust gaitswith aibo. In Proc. of IEEE International Conferenceon Robotics and Automation (ICRA‘00), pages 3040–
3045, 2000.
[2] Wolfram Burgard, Armin B. Cremers, Dieter Fox,
Dirk Hahnel, Gerhard Lakemeyer, Dirk Schulz, Walter
Steiner, and Sebastian Thrun. The interactive museum
tour-guide robot. In Proc. of the Fifteenth National Conference on Artificial Intelligence (AAAI/IAAI), pages
11–18, 1998.
[3] Jochen Triesch and Christoph von der Malsburg.
Robotic gesture recognition. In Proc. of the Bielefeld
Gesture Workshop, pages 233–244, 1997.
[4] Stefan Waldherr, Roseli Romero, and Sebastian Thrun.
A gesture based interface for human-robot interaction.
Autonomous Robots, 9(2):151–173, 2000.
[5] F. Marrone and M. Strobel. Cleaningassistant - a service
robot designed for cleaning tasks. In Proc. of Advanced
Mechatronic Systems (AIM‘01), 2001.
[6] Cynthia Breazeal and Lijin Aryananda. Recognitionof affective communicative intent in robot-directedspeech. Autonomous Robots, 12(1):83–104, 1 2002.
[7] Stanislao Lauria, Guido Bugmann, Theocharis Kyriacou, and Ewan Klein. Mobile robot programming using
natural language. Robotics and Autonomous Systems,
38(3–4):171–181, 3 2002.
[8] Shinya Kajikawa, Shigeyoshi Hiratsuka, Tadashi Ishihara, and Hikaru Inooka. Robot position control via
voice instruction including ambiguous expressions ofdegree. In Proc. of IEEE Int. Workshop on Robotand Human Interactive Communication (ROMAN‘03),
2003.
[9] J. W. Crandall and M. A. Goodrich. Characterizing
efficiency of human-robot interaction: A case studyof shared-control teleoperation. In Proc. of IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS‘02), 2002.
[10] Daisuke Katagami and Seiji Yamada. Active teachingfor an interactive learning robot. In Proc. IEEE Workshop Robot and Human Interactive Communication(ROMAN‘03), 2003.
[11] Alaa M. Khamis, Francisco J. Rodrı́guez, andMiguel A. Salichs. Remote interaction with mobile
robots. Autonomous Robot, 15(3), 2003.
[12] Kerstin Severinson-Eklundh, Anders Green, and HelgeHüttenrauch. Social and collaborative aspects of interaction with a service robot. Robotics and Autonomous
Systems, 42:223–234, 2003.

306[13] Tetsuo Ono and Michita Imai. Reading a robot’s mind:
A model of utterance understanding based on the theoryof mind mechanism. In Proc. of Seventeenth National
Conference on Artificial Intelligence, pages 142–148,
2000.
[14] John Travis Butler and Arvin Agah. Psychologicaleffects of behavior patterns of a mobile personal robot.
Autonomous Robots, 10(2):185–202, 2001.
[15] F. Kaplan, P. Oudeyer, E. Kubinyi, and A. Miklosi.
Robotic clicker training. Robotics and Autonomous
Systems, 38(3-4):197–206, 2002.
[16] Terrence W Fong, Illah Nourbakhsh, and Kerstin
Dautenhahn. A survey of socially interactive robots.
Robotics and Autonomous Systems, 42(3–4):143–166,
2003.
[17] Y. Hayashibara, Y. Sonoda, T. Takubo, H. Arai, andK. Tanie. Assist system for carrying a long object witha human – analysis of a human cooperative behavior
in the vertical direction –. In Proc. of 1999 IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS‘99), 1999.
[18] H. Arai, T. Takubo, Y. Hayashibara, and K. Tanie.
Human-robot cooperative manipulation using a virtual
nonholonomic constraint. In Proc. of 2000 IEEE
International Conference on Robotics and Automation(ICRA‘00), 2000.
[19] Kentaro Nakai, Kazuhiro Kosuge, and Yasuhisa Hirata.
Control of robot in singular configurations for humanrobot coordination. In Proc. of IEEE Int. Workshop
on Robot and Human Interactive Communication (ROMAN‘02), pages 356–361, 2002.
[20] K. Yokoyama, J. Maeda, T. Isozumi, and K. Kaneko.
Application of humanoid robots for cooperative tasks inthe outdoors,. In Proc. of IEEE/RSJ IROS Workshop on
Explorations towards Humanoid Robot Applications,2001.
[21] H. Choset. Coverage for robotics - a survey of recent results. Annals of Mathematics and Artificial Intelligence,
31:113–126, 2001.
[22] Rodney A. Brooks. A robust layered control system
for a mobile robot. IEEE Journal of Robotics andAutomation, 2(1):14–23, 1986.
[23] J. E. Condron and K. D. Hill. Reliability and validityof a dual-task force platform assessment of balance
performance : Effect of age, balance impairment, andcognitive task. Journal of American Geriatrics Society,50:157–162, 2002.