Toward a Robotic Attendant Adaptively Behaving
According to Human State- Attending position determination based on the target person’s state -Shuji Oishi1 , Yoshiki Kohari1 , and Jun Miura1

Abstract— This research aims to develop a robot which canadaptively attend a specific person according to the person’s
behavior. Transition of the person’s state is modeled with finite
state machine (FSM), and the robot recognizes events for thestate transitions and selects an appropriate attending action foreach state. We implemented attending actions for the person’s
walking and the sitting actions. When a person is walking, therobot takes a following action. When the person is sitting, therobot moves to a waiting position determined by considering the
comfort of the person and the others. We carried out attending
experiments using a real robot to show the effectiveness of the
proposed approach.

I. INTRODUCTION
In recent years, service robots have been attracting muchattention for applications such as self-reliance support of theelderly or guiding people in public places, and in a variety
of situations, the robots need to provide appropriate service
while attending a specific person. For such mobile robots,people detection and tracking are the essential functions andhave been well studied in robotics and computer vision fields
over the past decades.
Bellotto and Hu [1] develop a multi-sensor fusion technique for people tracking for mobile robots. The method
detects human legs in laser scan data considering possible
leg postures, and face detection in camera images is also performed to improve the accuracy of the discrimination. Fusing
the observations obtained with different sensors, the method
tracks people using a sequential unscented Kalman filter,and demonstrated the robustness of the proposed humantracking method in complex indoor environments. As anotherexample, a fast people tracking algorithm for service robots
using RGB-D data is proposed by Murano and Menegatti[2]. People in a scene are detected by performing 3Dclustering of the point cloud and calculating HOG featuresfrom RGB data of each cluster. The algorithm then tracks thedetected people considering the clusters’ motions, colors, anddetection confidences. They tested the proposed algorithm
using a variety of datasets and data obtained from a mobile
robot in crowded indoor environments, and demonstrated theoutstanding performance and robustness.
While a large number of person detection and tracking
algorithms have been developed and available for mobile*This work was supported by JSPS KAKENHI Grant Number 25280093.
1 S.Oishi, Y.Kohari, and J.Miura are with Department of Computer
Science and Engineering, Toyohashi University of Technology, Aichi, Japanoishi@cs.tut.ac.jp, kohari@aisl.cs.tut.ac.jp,
jun.miura@tut.jp

Fig. 1.

Adaptive attendance

robots, most of them focus on how well to follow thetarget person and do not pay much attention to the person’s
state. Leica et al. [3] develop a switched control strategy
of a guiding robot in order to allow the robot not only
to guide a specific person at a desired distance but also
switch the robot’s behavior based on the human-robot relative
position. The method basically defines several interaction
zones around the robot to enable the person to communicate
the intention, however, the person explicitly has to enter
the defined zone to switch the motion of the robot. When
working in social environments, the robot should not only
follow the person but also infer the person’s state andadaptively change its own behavior for providing better
service. For example, when the target person gets tired fromwalking and sits on a bench to rest, it is nice if the robot
could recognize the person’s state and move to an appropriate
position so as not to give unpleasant feelings by continuing
to follow persistently and stay in front of the target person.
In this paper, extending our past work [4], we propose
a new attendance methodology for mobile robots. As aprototype, we have developed an attendant robot which
recognizes a target person’s walking/sitting behavier withlaser range finders, and performs the corresponding attending
action according to the person’s state.
The rest of this paper is organized as follows. In section
II, the overview of the proposed method is shown to give thebrief idea of this work. In section III, we first introduce ourrobot system, and then describe the method of human state
estimation. The proposed attending algorithm based on theestimated human state is also explained. In section IV, wedemonstrate how the proposed algorithm works and verify
the effectiveness through experiments using a real robot.
Finally, in section V, we conclude this paper and discuss
the future work.
II. OVERVIEW OF PROPOSED METHODOLOGY
Although the main task of attendant robots is following
the target person for application such as guiding or nursing,appropriate attendance varies depending on the situations asshown in Fig.1. For example, while the robot only has to
track the target person avoiding obstacles when the person
is walking, the robot should stand by the sitting person at
a position where the robot does not obstruct the person.
In another example, when the person seems to be getting
lost, the attendant robot leads the person and guides tothe destination. Besides, when the person is talking withsomeone, the robot watches over the person at a distanceto refrain from interfering with them. As stated above,
depending on the state of the target person, the attendantrobot should select appropriate behavior to provide desirable
service.
This kind of advanced attendance is based on recognitionof the person’s state. In this study, we model the transitionof the person’s state with Finite State Machine (FSM). The
robot recognizes events for the state transitions from sequential range data obtained by laser range finders, and selects anappropriate attending action for each state. We implemented
attending actions for the person’s walking and the sitting
action as a prototype of the adaptive attending system. In
addition, with regards to the sitting action, we determine anappropriate waiting position based on proxemics in HumanRobot interaction. The details are explained in Sec.III-C
III. PROPOSED METHOD
A. The attendant robot system
In this research, we use an omni-directional attendantrobot (GRACE, KER) shown in Fig.2. It can controle
the driving and the steering torque efficiently based on
Differential-Drive Steering System (DDSS) [5] achieving
a high mobility even in narrow areas. The robot is also
equipped with a touch screen monitor as an interface where
the robot shows useful information to the user or allows theuser to input commands.
Two laser range finders (UTM-30LX, HOKUYO) are
mounted on the front and rear of waist-high (95cm) andshin-high (30cm) layers respectively in order to measureomni-directional range data at different heights. For person
tracking, we first extract leg-like segments in range data
focusing on that people in a scene cause local minima inthe distance histogram [6]. Next, leg clusters are detectedfrom these segments by calculating clusters’s features, suchas the length, the mean curvature, and the variance ratio
by PCA, and classifying them with Radial Basis Function
Support Vector Machine (RBFSVM) as a method of Zainudin

Fig. 2.

(a) Scene

An omni-directional attendant robot(b) Estimated position and orientationof the target person

Fig. 3.

Person detection with laser range finders

et al. [7]. These two steps are applied to each frame, and therobot tracks the target person’s legs position using Unscented
Kalman Filter (UKF). In order to elaborate the state of thetarget person, we extend the state variables in UKF using
upper range data so that it can estimate not only the position
but also the body orientation of the person, as shown in Fig.
3, by comparing the input torso shape data with the model
data, 360-degree torso shape data collected in advance.
When attending, the robot follows the person with planning a path using an algorithm proposed by Ardiyanto andMiura [4]. This algorithm calculates a shorter and safer pathto the destination in real time utilizing a randomized pathsearch, and enables the robot to move without collision evenin dynamic environments.
B. Modeling and estimating human states
Estimating the target person’s state is essential to performthe appropriate attendance. In this research, we adopt Finite
State Machine (FSM) to handle the states of the person. FSM
is a model of computation that consists of a finite number ofstates and is capable of managing transition from one state
to another. We model the transition of person’s state witha FSM where three states are defined: Initial, Walking, andSitting as shown in Fig.4. Before the recognition, the person’s
state is defined as ”Initial”. When the robot recognizes the
person walking, the state transits to ”Walking” state and therobot starts to follow the person. On the other hand, when
the robot detects a sitting behavior of the person, the FSM

(a) View from the robot
Fig. 5.
Fig. 4.

(b) Generated position

Robot destination when following the walking person

Finite state machine for adaptive attendancechanges its state to ”Sitting” and the robot nestle against theperson.
In order to recognize the transition of the target person’s state, we adopt Hidden Conditional Random Fields(HCRF)[8]. HCRF is an extension of conventional CRFs into
the temporal domain with hidden states and has demonstrated
outstanding performance in human gesture recognition. In
this research, we estimate the state of the person using
HCRF based on the target person’s behavior observed withlaser range finders. The behavior is described by sequential
features consisting of the walking speed, the distance andorientation to the nearest chair. Referring to a research of
Schindler et al. [9] which has reported that observations of
5 to 7 frames are enough to recognize most human gestures,
we have a HCRF learn the best discriminative structure from
5-frame consecutive features, and recognize the current stateof the person in real time based on same length of theconsecutive features.
C. Adaptive attendance1) Walking state: The attendant robot follows the target
person when the person is walking. Actually, the robot keeps
itself 1.0 m behind the person so that the robot can avoid
collision even when the person suddenly stops walking.
When the person steps into an area within 1.0 m from theattendant robot, it stops following until the person goes out of
the area. Fig.5 shows an example of generated destinations at
walking state where the robot (orange triangle) has detected
a target person walking in front of it (green dot) and moving
toward the target position (black cross).
2) Sitting state: On the other hand, when the target
person is sitting, the attendant robot moves to a appropriate
waiting position by considering not only the structure of theenvironment but also the comfort of the person and the othersin the environment. The position is determined based on thefollowing factors.
a) Collision safety: Risk of collision increases if the
waiting position is near from obstacles in the scene. In otherwords, the farther from the obstacles the robot is, the moresafely the robot can move. Therefore, we define an evaluation
function which gives higher scores to the positions that arefar from objects in the scene.

fob (xi, j ) = e−sob (di, j ) ,
ob 2

(1)

where xi, j denotes a 2D candidate position, di,obj a distancefrom the nearest obstacle, and sob a parameter which controles the distribution of the function fob .
b) Comfortable attendance: The appropriate distanceto the target person depends on what kind of service the robot
provides. In this research, the attendant robot is supposed toprovide service through a touch screen monitor mounted on
itself. We therefore give higher score to the area of around
0.5 m from the sitting person so that the person can reachthe monitor.
At the same time, it should be taken into account fromwhich direction the robot attends to the target person for
comfortable service. Wood et al. [10] investigated how aservice robot should approach and serve the target person
by carrying out experiments in real scenes, and revealed thatpeople generally prefer approaches from the front left and thefront right. Referring to the work, we define another functionthat gives a score to the candidate positions considering therelative distance and position to the target person as follows:
f pos (xi, j2−sd di,h j −dg
)=e




2
−sth thxi, j −thFL

×max e2 −sth thxi, j −thFR

,e

,

(2)

where di,h j is a distance between a candidate position and thetarget person, dg the target distance defined as 0.5 meter,thxi, j the relative angle from the front of the person toa candidate position, thFL and thFR angles of diagonallyforward left and right from the person, respectively. sd andsth controle the distribution of the function.
c) Social distance to the other parson: According toa proximics study of Hall [11], ”Social distance” has been
defined as the psychological zone (1.2m − 3.6m) for meetingand interacting with unfamiliar people while ”Personal distance” (0.45m − 1.2m) for well-known people. In order not
to give unpleasant feelings to the other people in the scene,
candidate positions within the social distance to the people
are rated lower as follows:TABLE I
R ECOGNITION RESULTS OF SITTING BEHAVIOR
Behavior Label
Pass by the chair
Sit on the chair
Total

Data
135
61
196

Detected
134
62
196

Correct
128
55
183

⎧


⎨ −ssd di,o j −dsd 2efsd (xi, j )⎩1

Precision
95.5
88.7
93.4

Recall
94.8
90.2
93.4

F1
95.2
89.4
93.4

(a)
i f di,o j < dsd ,otherwise(3)

where di,o j denotes the distance between a candidate position
and the other people, dsd the boundary between Social
distance and Personal distance (1.2m), and ssd a parameterfor the distribution.
d) Determination of the appropriate waiting position:
Integrating the above-mentioned factors, a total score foreach candidate position is calculated as follows:f (xi, j ) = fob (xi, j ) × f pos (xi, j ) × fsd (xi, j )

(b)

(c)

(d)

(e)

(4)

The appropriate waiting position is determined by finding
a candidate which has the highest score. When there areseveral candidates which have almost the same score as thebest one, the robot prefers to the nearest one for moving
efficiency.
Fig. 6 illustrates the procedure of the waiting position
calculation. In a scene shown in Fig. 6(a) where the target
person is sitting on a chair, the attendant robot calculates scores of candidate positions based on the distanceto obstacles (Fig.6(b)), relative position and angles to thetarget person (Fig.6(c)), and the social distance to the other(Fig.6(d)). Finally, by integrating these scores, the robot finds
the most appropriate waiting position as shown in Fig.6(e).

Fig. 6. The procedure of waiting position calculation: In a scene (a), therobot first measure the environment with laser range finders, and calculatesa score for each candidate position based on the distance to obstacles (b),the relative position and angle to the target person (c), and the social
distance to the other people (d). The most appropriate waiting position isthen determined by integrating these scores (e). Note that orange, green,
and blue dots denote the attendant robot, the target person, and the otherperson, respectively.

IV. EXPERIMENTS
A. Recognition of sitting behavior
We carried out experiments of sitting behavior recognition
to verify the performance of our recognition framework. We
first trained a one-class HCRF model using 235 sitting action
sequences as positive data and 548 passing by a chair action
sequences as negative data. Next, the trained HCRF model
was applied to a test dataset which consists of 135 positiveand 61 negative sequences. As shown in Table I, the attendantrobot successfully recognized the sitting behavior from thesimple features derived from range data obtained with laser
range finders.
B. Adaptive attending experiments
We also applied the proposed method to an actual attendant robot described in Sec.III-A. The software is implemented as a set of RT Components (RTCs) [12] andthe configuration is shown in Fig. 7. The entire system
works as follows: First, the attendant robot detects andstarts to track a target person estimating the position andbody orientation while it localizes itself using Monte Carlo
Localization [13] with the given environment map. Besides,

the system recognizes the person’s state and determines
the appropriate destination considering the state and thesurrounding environment. A collision-free path towards thedestination is then calculated by a randomized search [4],and the system finally drives the robot to the goal position.
To confirm whether the attendant robot could adaptively
select appropriate actions depending on the state of thetarget person, we carried out experiments according to thefollowing four different scenarios.
•

scenario 1: The target person sits on a chair, and there
is nobody around the chair.
• scenario 2: The target person sits on a chair, and there
is another person standing around the chair.
• scenario 3: The target person sits on a bench, and there
is nobody around the bench.
• scenario 4: The target person sits on a bench, and there
is another person standing around the bench.
In each scenario, the target person walks along a corridor, approaches to a chair or a bench, and sit on it.
The attendant robot follows the target person, and goes

Scenario 1

Fig. 7.

The entire system implemented using RT-middleware [12]Scenario 2

Scenario 3(a) Scene
Fig. 8.

(b) 2D map of the floor
Experimental setup

to the most appropriate waiting position considering thesurrounding environment when it recognizes sitting behaviorof the person. Note that the global 2D map of the floor
has been given, and the locations of chair and bench areregistered in the map in advance (Fig. 8).
Fig. 9 shows the experimental results for these four
scenarios. While the target person is walking, the attendantrobot follows the person trying to keep itself 1.0 m behind
and avoiding obstacles as well. On the other hand, when the
person is sitting, the attendant robot detects the transitionof the person’s state and determines the appropriate waiting
position considering the situation. These results demonstratethat the proposed method successfully enables the robot toattend the target person adaptively according to the state.
C. Evaluation based on questionnaire
In order to investigate how the robot’s behavior affects
to the target person’s feeling, we collected the subjects’
opinions using a questionnaire after each trial. Fifteen malesubjects were involved in this study, and we carried out
the same four scenarios for each subject with and without
applying the proposed adaptive attendance, totally 8 trials.
Note that in the ’without’ case the robot just follows the
subject in the same way as III-C.1 and does not recognize

Scenario 4
Fig. 9. Waiting position determination based on the proposed scoring. Notethat orange, green, blue, and red dots denote the attendant robot, the target
person, the other person, and the determined waiting position, respectively.

the state. The questionnaire allows a subject to rate thefollowing items with 5-point Likert scales where 1 denotes
highly negative, 2 fairly negative, 3 neutral, 4 fairly positive,
and 5 highly positive.
1) It was comfortable to be attended by the robot.
2) The robot was considerate to you.
3) During the attendance, the robot seemed to act considering the surrounding environment such as obstacles
and the other people.
We applied paired t-tests to investigate the differences
between the samples with and without the proposed method.
Table II shows the results of the questionnaires and the corresponding t-scores. Overall, the probabilities p were p < 0.05
in most of the cases. In other words, there were significant
improvements with regards to the subjects’s feelings when

TABLE II
Questionnaire results: T HE AVERAGES AND VARIANCES OF THE DIFFERENCES BETWEEN SCORES OF TRIALS WITH AND WITHOUT APPLYING THE
PROPOSED METHOD ARE CALCULATED . T HE TABLE ALSO HAS T- SCORES AND THE CORRESPONDING PROBABILITIES .

Question 1

Question 2

Question 3

Average scores w/ and w/o the proposed method
Average of the score difference
Variance of the score difference
t-value t(14)Probability p
Average scores w/ and w/o the proposed method
Average of the score difference
Variance of the score difference
t-value t(14)Probability p
Average scores w/ and w/o the proposed method
Average of the score difference
Variance of the score difference
t-value t(14)Probability p

the proposed adaptive attendance was applied enabling therobot to give pleasant impressions to the target person.
The probability p of the third question in the third scenario
was more than 0.05 indicating the proposed method did not
significantly improve the subjects’ impressions of the robot’s
awareness or consciousness. This was partly because in thethird scenario there was no one except the target person andthe robot could get good scores according to the algorithm[4] which generated collision-free paths to deal with thesituation.
In conclusion, the results indicate that the proposed
method allows the attendant robot to perform more considerate attendance, and also make the target person more
comfortable during the attendance.
V. CONCLUSION
In this paper, we developed an service robot which can
attend the target person considering the person’s state toprovide appropriate service. The transition of person’s stateis modeled using a Finite State Machine (FSM) to handle”Walking” and ”Sitting” states while the state is estimatedbased on sequential range data captured with laser range
finders. We implemented the proposed method in a mobile
robot and demonstrated that the robot could select an appropriate action according to the target person’s behavior
for comfortable attendance. We also evaluated the proposed
method based on questionnaires where the robot attended thetarget person in several situations. The study indicates thatthe proposed adaptive attendance makes the target person
more comfortable and also feel close to the attendant robot.
For future work, we extend the proposed method so that it
can recognize objects in the scene such as chairs, desk, doors,and so on for handling a variety of tasks. We also need to
discuss the use of FSM. FSM with a number of states may
enable the attendant robot to deal with more complicatedsituations, however, increasing merely the number of states
may not necessarily lead to natural interaction. For sophisticated interaction, we should extend the system carefully
considering psychological and socialcontextual factors.

Scenario 1
2.33 / 3.53
1.20
0.862
5.39
9.49 × 10−5
2.33 / 3.60
1.27
1.16
4.22
8.59 × 10−4
2.40 / 3.13
0.733
0.961
2.95
1.04 × 10−2

Scenario 2
2.13 / 2.93
0.800
0.862
3.59
2.93 × 10−3
2.20 / 3.33
1.13
0.915
4.79
2.85 × 10−4
1.73 / 3.33
1.60
1.35
4.58
4.26 × 10−4

Scenario 3
3.27 / 3.87
0.600
0.986
2.36
3.35 × 10−2
2.93 / 4.07
1.13
1.13
3.90
1.60 × 10−3
2.87 / 3.40
0.533
1.13
1.84
8.78 × 10−2

Scenario 4
2.67 / 4.00
1.33
0.976
5.29
1.14 × 10−4
2.60 / 4.13
1.53
0.743
7.99
1.39 × 10−6
1.67 / 4.33
2.67
0.900
11.5
1.65 × 10−8

R EFERENCES[1] N. Bellotto and Huosheng Hu, “Multisensor-based human detection
and tracking for mobile service robots,” IEEE Transactions on
Systems, Man, and Cybernetics, Part B: Cybernetics, vol. 39, no. 1,
pp. 167–181, 2009.
[2] Matteo Munaro and Emanuele Menegatti, “Fast RGB-D people
tracking for service robots,” Auton. Robots, vol. 37, no. 3, pp. 227–
242, oct 2014.
[3] Paulo Leica, Juan Marcos Toibero, Flavio Roberti, and Ricardo Carelli,“Switched control to robot-human bilateral interaction for guiding
people,” J. Intell. Robotics Syst., vol. 77, no. 1, pp. 73–93, jan 2015.
[4] Igi Ardiyanto and Jun Miura, “Real-time navigation using randomized kinodynamic planning with arrival time field,” Robotics andAutonomous Systems, vol. 60, no. 12, pp. 1579–1591, Dec 2012.
[5] Yuki Ueno, Takashi Ohno, Kazuhiko Terashima, Hideo Kitagawa,
Kazuhiro Funato, and Kiyoaki Kakihara, “Novel differential drive
steering system with energy saving and normal tire using spur gear foran omni-directional mobile robot,” in IEEE International Conferenceon Robotics and Automation (ICRA), 2010, pp. 3763 – 3768.
[6] Dirk Schulz, Wolfram Burgard, Dieter Fox, and Armin B. Cremers,“People tracking with a mobile robot using sample-based joint probabilistic data association filters,” International Journal of Robotics
Research (IJRR), vol. 22, no. 2, pp. 99–116, 2003.
[7] Zulkarnain Zainudin, Sarath Kodagoda, and Gamini Dissanayake,“Torso detection and tracking using a 2D laser range finder,” inAustralasian Conference on Robotics and Automation, 2010.
[8] Sy Bor Wang, A. Quattoni, L. P. Morency, D. Demirdjian, and
T. Darrell, “Hidden conditional random fields for gesture recognition,”in IEEE Computer Society Conference on Computer Vision and Pattern
Recognition (CVPR), 2006, vol. 2, pp. 1521–1527.
[9] K. Schindler and L. van Gool, “Action snippets: How many frames
does human action recognition require?,” in IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), June 2008, pp.
1–8.
[10] S. N. Woods, M. L. Walters, K. L. Koay, and K. Dautenhahn,“Methodological issues in hri: A comparison of live and videobased methods in robot to human approach direction trials,” in The
15th IEEE International Symposium on Robot and Human Interactive
Communication (ROMAN), Sept 2006, pp. 51–58.
[11] Edward T. (Edward Twitchell) Hall, The hidden dimension : man’s
use of space in public and private, London : Bodley Head, 1966.
[12] Noriaki Ando, Takashi Suehiro, and Tetsuo Kotoku, “A software
platform for component based rt-system development: Openrtm-aist,”in The 1st International Conference on Simulation, Modeling, and Programming for Autonomous Robots, Berlin, Heidelberg, 2008, SIMPAR’08, pp. 87–98, Springer-Verlag.
[13] “MRPT,” http://www.mrpt.org/.