Fourth-person sensing for a service robot
Kazuto Nakashima, Yumi Iwashita, Pyo Yoonseok, Asamichi Takamine, Ryo Kurazume
Kyushu University, Fukuoka, Japan
Email: k nakashima@irvs.ait.kyushu-u.ac.jp
Abstract—This paper proposes a new concept named ”fourthperson sensing” for a service robot. The proposed concept
combines wearable sensors (the ﬁrst-person viewpoint), sensors
mounted on robots (the second-person viewpoint), and sensors
embedded in the environment (the third-person viewpoint), andthe disadvantages in individual sensors are compensated by
combining sensory information from the ﬁrst, second, and thirdperson viewpoints. The fourth-person sensing is effective tounderstand a user’s intention and a context of the scene, thus itenables to provide a proper service by a service robot. As one
of applications of the fourth-person sensing, we develop a HCI
system combining the ﬁrst-person and the third-person sensing,and show the effectiveness of the proposed concept through robot
service experiments.

I.

I NTRODUCTION

This paper proposes a new concept named ”the fourthperson sensing” for a service robot, which combines conventional second and third person sensing with the ﬁrst-person
sensing obtained by wearable devices. In addition, to show
the effectiveness of the fourth-person sensing, we focus on an
ambiguous verbal communication between a user and a service
robot, and show a robot service experiment triggered by user’s
ambiguous voice commands.
II.

T HE FOURTH - PERSON SENSING

A. Concept of the fourth-person sensing

Due to the rapid aging of the population, a labor shortagein hospitals or care facilities is becoming a serious problem.
To mitigate the impact on this problem, the development ofa service robot which coexists with human in a daily-life
environment is an urgent challenge. On the other hand, sincea high level of safety is required for a service robot, the robot
needs to acquire a wide variety of surrounding information,and plans and executes a proper service task. However, thesefunctions are quite hard to be implemented in a single robot
due to a limitation of payload or processing and sensing
capabilities.
To tackle this problem, we have been developing an
informationally structured environment and its architecture
named Town Management System (TMS). In TMS, a variety
of sensors such as a laser range ﬁnder or a camera areembedded in an environment and a distributed sensor networkis organized [1], [2]. The captured information is integrated
and stored to a TMS database. A service robot accesses the
TMS database and obtains required information anytime andanywhere. In other words, the service robot is able to acquirea high sensing performance which cannot be realized by itself.
We also have started the development of a new TMS named
ROS-TMS, which adopts Robot Operating System (ROS) as a
middleware so that it makes more ﬂexible to add and replacesensors and robots [3].
Here, let us classify the environmental information managed by the ROS-TMS in terms of ”person”. The viewpointof the user can be regarded as the ﬁrst-person information,and the one of the service robot which provides a service task
is the second-person information. In addition, the information
obtained by the embedded sensors in the environment can be
regarded as the third-person information. From the second andthird-person viewpoints, a wide variety of information of theenvironment can be obtained, and the systems which combine
second and third-person information have been reported [4],[5]. However, in the area near a user which is important toprovide an appropriate service task, it is often difﬁcult to978-1-4799-8203-5/15/$31.00 ©2015 IEEEobtain sufﬁcient information in terms of the resolution and theaccuracy due to the occlusions or the distance from sensors.

1110

The fourth-person sensing is regarded as an imaginary
viewpoint from which we can understand the correct situationof the environment objectively by combining the ﬁrst, second,
and third-person information. Let us show an example forcorrect understanding about this concept with a ”novel”. Ina novel, the viewpoint of the main character provides theﬁrst-person information and his partner’s viewpoint gives thesecond-person information. Moreover, the viewpoints of people surrounding them provide the third-person information. In
real life, there is no way of obtaining the second and the thirdperson information directly. However, when we read a novel,
we can know these information explicitly from the sentences,imagine the story, and forecast the next scenario. This reader’sviewpoint, which we call the fourth-person information, will
be quite useful to understand the situation of the world in thenovel correctly. The extreme target of the fourth-person sensing
is to understand the user’s intention or the context correctly
by integrating there information from different viewpoints.
As explained in Section I, each viewpoint information has
pros and cons for a HCI system. The ﬁrst-person informationis quite useful to recognize user’s action and estimate his/her
intension. However, the measurement area is narrow and local
and fragmented information tends to be obtained. The secondperson information has high degrees of freedom in terms ofdata acquisition comparing with the third-person information
obtained by ﬁxed sensors, since the robot equipped withsecond-person sensors is able to move freely in an environment. However, the payload and the processing performanceof the robot is limited and it is almost impossible to acquiresufﬁcient information completely to perform a proper tasksafely. Though the third-person information is able to measure
the target, the user, and the environment comprehensively,
sensors tend to be located away from the user and the robot
in the environment and sometimes it is hard to obtain accurate
information due to occlusion or low resolution. Therefore, thecorrect recognition of the user’s intension from the third-person
information is sometimes very difﬁcult.

On the other hand, by fusing all-person information in acomplementary way, the following advantage will be expected
for the instruction to a service robot. Correct understanding
for an ambiguous instruction will be realized. The verbal
communication is effective for requesting a robot service since
it is induced by a user intentionally and can be an explicit
trigger for a service request. However in natural conversation,
it is often ambiguous and user’s intention or request are notclearly expressed. On the other hand, for instance, the ﬁrstperson images taken by a wearable camera contain a richinformation such as what the user is gazing or what the user
is doing now. These gaze information or action information
taken from the ﬁrst-person images may make an ambiguous
instruction more clear.

database. Position of objects, robots, and humans are tracked
by LRF, Vicon MX and the intelligent cabinet/refrigerator.
Especially the intelligent cabinet/refrigerator is able to detect
not only the object name by reading RFID tag attached on
the object but also the position of the object using the force
distribution measured by the load cells. These informations are
obtained from the embedded sensors in the environment, thus
we regard these sensors as the third-person sensors.
Vicon MX
RGB-D camera
Intelligent cabinet/refregerator

In the following sections, we introduce the ﬁrst, second,
and third-person sensors we utilized in the experiments.
Load cell,
RFID tag reader

B. The ﬁrst-person sensing
In recent years, several high performance wearable cameras
are provided in the market. Especially, smart glasses equippedwith on-board cameras are very popular as a HCI device andenable to capture ﬁrst-person images. In this research, we
adopted Moverio BT-200AV (Epson) smart glass as shownin Fig.1. First-person images have been used to sense theenvironment and the user’s activities from the user’s view
point, for various purposes of activity recognition in daily life,
sports scene, and so on [6].
3D LRF
RGB-D camera

2D LRF

Fig. 1. (Left) the ﬁrst-person sensor in a wearable device, (right) the secondperson sensors on a service robot.

C. The second-person sensing
A service robot needs to acquire user or environmental
information correctly by on-board sensors for executing aproper task safety. The obtained information is regarded as
the second-person information. The service robot used in theexperiments (SmartPAL V, Yaskawa, Fig.1) is equipped witha 3D laser range ﬁnder (HDL-32e, Velodyne) and a RGB-D
camera (Xtion, ASUS) on a head, and a 2D laser range ﬁnder(TopUrg, Hokuyo) on a body.
D. The third-person sensing
As introduced in Section I, we are developing the distributed sensor network and information management architecture named ROS-TMS. Figure 2 shows the experimental room
which is managed by ROS-TMS. In this room, laser range
ﬁnders (UTM-30LX-EW, Hokuyo), RGB-D cameras (Xtion,ASUS), the position tracking system (Vicon MX, Vicon),
RFID tag, intelligent cabinet/refrigerator system consistingof RFID-tag readers and load cells, etc. are embedded andenvironmental information is collected and stored in the TMS

1111

LRF
Informationally structured room
Intelligent cabinet system
Fig. 2.

III.

The third-person sensors embedded in an environment

A PPLICATIONS OF THE FOURTH - PERSON SENSING

The most promising application of the fourth-person sensing is the correct understanding of user’s instruction from an
ambiguous verbal communication. In this section, we focus
on a fetch-and-carry task by a service robot, and show anapplication of the fourth-person sensing which performs aservice task appropriately even if the user’s instruction is
ambiguous and the system cannot recognize it correctly by
the conventional second or third-person sensors.
The typical scenario is shown in Fig.3. There are threeobjects (”pet bottle”,”bucket”, and ”watering pot”) in theenvironment, which are all related to ”water”. We can imagine
that ”pet bottle”, ”bucket”, and ”watering pot” are related to”food”, ”cleaning”, and ”gardening”, respectively.
At ﬁrst, the user during a meal instructs the robot tobring ”water” by voice. However, since there are plural objects
related to ”water” in this environment as described above, therobot cannot determine the proper one which ﬁts the user’s
preference from such an ambiguous oral information. Theseambiguous instructions are often encountered in our daily life.
On the other hand, if the information about the user’s
action, for instance ”eating” in this case, can be obtained, therobot recognizes that the user might ask to take something
related to food, and is able to choose a pet bottle for drink fromplural candidates. As obviously shown in this example, humanaction is quite important to understand the user’s intension.
To recognize a user’s action, the ﬁrst-person vision is a
powerful information source comparing the second and thirdperson sensors. Though the second and third-person sensors
are also helpful for action recognition, the ﬁrst-person vision
can give a more reliable information since the human action
is tightly related to the image he/she has seen and the egomotion which might be closely related to the action can also
be captured directly from the ﬁrst-person vision.

The third-person
viewpoint

recognition. In the video datasets there are ﬁve categories: (i)reading a book, (ii) eating, (iii) watching a tree, (iv) watchinga robot, and (v) looking around.

Bring mesome water

Intension

1) Feature extraction: We utilize the Space-Time Interest
Points (STIP), which was proposed by Laptev [7], as alocal feature detector, and we describe feature vectors using
Histogram of Optical Flow (HOF) and Histogram of Oriented
Gradients (HOG). After extracting feature vectors, we apply
the principal component analysis (PCA) to original feature
vectors, while keeping 95 % contribution rate, for dimension
reduction.

Water

User
Service robot
The first-person viewpoint

Fig. 3.

An example of service scenarios.
100

100
IFV (HOF)
VLAD (HOF)
BoVW (HOF)

95

90

Accuracy rate [%]Accuracy rate [%]

95

85
80
75

For a more efﬁcient representation of motion informationin videos, we employ the concept of feature encoding. Morespeciﬁcally, we use three encoding methods: (i) visual words[10], (ii) vector of locally aggregated descriptors (VLAD) [12],and (iii) ﬁsher vector [11].

90
85
80
IFV (HOG)
75

VLAD (HOG)
BoVW (HOG)

70

70
65

65
0

50

100

150

Number of Visual Words(a) HOF

200

0

50

100

150

200

Number of Visual Words(b) HOG

Fig. 4. Accuracy rate for various numbers of Visual Word k: (a) Maximum
rate is 89.3% with ﬁsher vector, k=200 (b) Maximum rate is 96.2% with ﬁsher
vector, k=200.

In this section, we assume the scenario for taking a”water-related” object via oral instruction described above,and demonstrate the effectiveness of the fourth person sensing
combining the ﬁrst and third-person sensors.
A. System conﬁguration
We built a distributed processing system for a forth person
sensing consisting of a smart glass and a processing server.
1) Smart glass: We utilized a Moverio BT-200AV (Epson)
smart glass for capturing a ﬁrst-person images. Moverio isequipped with Android OS, an overlay display, a camera,
and a microphone, and is able to capture ﬁrst-person images
and voice at the same time. Captured images and voice arecompressed and transmitted to a processing server with a frame
rate of captured video periodically. The information such as
the server status and the recognized action is overlayed to thedisplay of the Moverio to conﬁrm the processing results.
2) Processing server: The processing server recognizes theuser’s action periodically, and it suggests candidate objects
after the server receives user’s voice. Both processes are
explained in the following section.
IV.

U NDERSTANDING A USER ’ S REQUEST

2) Activity recognition: We collected ﬁrst-person videos
using the camera of the Moverio. Each class has 50 sequences,thus totally there are 250 (=50×5) sequences. Each sequenceis 10 seconds with 30 fps frame rate, and image resolution is
320 × 240. After we obtain encoded feature vectors from allsequences, we utilize Linear Support Vector Machine (Linear
SVM) for activity recognition. Here, we used 2-folds forevaluations. The mean classiﬁcation accuracy was obtained byrepeating this random training-test splits for 100 times. The
number of clusters in visual words, which is used for the three
encoding methods, is changed as 10, 20, 50, 100, and 200.
Figure 4 shows the results of each feature encoding method
and each feature vector. From these results, we can see thatthe results of HOG descriptor showed better performance thanthose of HOF descriptor. Moreover, Fisher Vector shows thebest performance among all encoding methods. Thus in thefollowing experiments, we use HOG as a feature descriptor
and Fisher Vector as a feature encoding method.
B. Understanding a user’s request
In this system we take the advantage of ”tag” information,
which is assigned to each object stored in TMS database.
Tag information of an object consists of keywords which
explain the object. For example tags of ”drink”, ”tea”, and”water” are assigned to a bottle of tea. Each activity also
has tag information, so that we can count the number oftag information, which is the same tag information of theactivity, at each candidate object. We then give priority toeach candidate object, which is proportional to the numberof counted tag.
V.

In this section, ﬁrstly we explain several methods, which
are commonly used for activity recognition [8], [9]. Secondly,
we describe the way how to select the correct one frommultiple objects, which are candidates based on the user’s
voice commands.
A. Activity recognition using the ﬁrst person vision
In this section, ﬁrst we describe methods to extract local
features from videos, followed by three feature encoding methods for more efﬁcient representation of motion informationin videos. Then we show experimental results of the activity

1112

E XPERIMENT

A. Experimental settings
We explain experimental settings to evaluate the proposedsystem. A user wearing the Moverio asks to a robot to give himwater, while doing three different activities, ”reading a book”,”eating” and ”watching a tree”. A service robot gives him thecorrect one he wants, depending on his request. We assume
that a canned coffee is the right one for ”reading a book”, abottle of tea for ”eating”, and a watering pot for ”watching atree”. Table I shows a list of tag information of each activity,
and Table II shows object information related to water.

Bring me some water
Would you need a cancoffee ?

Would you need a green tea bottle ?
Bring me some water

Bring me some water

Would you need a watering pot ?

(a) Reading a book(b) Eating a meal(c) Gazing at a tree

Fig. 5. Experiment: Figures on upper row shows actual images and a user did some activities. Figures on lower row shows the screen of wearable camera.
Recognized results are shown as a ”User Activity” (red circles)
TABLE I.

TAGS ASSOCIATING TO ACTIVITIES
ActivityTagread a bookdrink, coffee
eat a meal
drink, teagaze at a tree
pot

TABLE II.
Category
CoffeeTea
Watering Pot

developing a system for much more accurate understanding ofa user’s intention and a context of the scene.
ACKNOWLEDGMENT
The present study was supported by a Grant-in-Aid for
Exploratory Research (26630099).

O BJECTS STORED IN THE DATABASE
Name
Tag
cancofee
drink, coffee, water
greentea bottle
drink, tea, water
watering pot
pot, water

R EFERENCES

B. Experimental results
We explain experimental results of the three activities.
Figure 5 (a) shows an actual image of ”reading a book” and thesystem estimated the user’s activity as ”read a book”, which isshown in ”User Activity” with red circle. After the user asked
the service robot to bring water for him, the robot successfully
chose a canned coffee. Figure 5 (b) shows the actual imageof ”eating”, and his activity was estimated as ”eat a meal”.
After he asked the service robot to bring water for him, therobot successfully chose a bottle of green tea. Figure 5 (c)shows the actual image of ”watching a tree”, and his activity
was estimated as ”gaze at a tree”. After he asked the service
robot to bring water for him, the robot successfully chose awatering pot. These experimental results show that the system
could successfully understand the user’s request based on his
activity, even though there are some ambiguities and multiple
candidates in the scene.
VI.

C ONCLUSIONSThis paper introduced a new concept of the fourth-person
sensing, which combines conventional second and third person
sensing for a service robot with the ﬁrst-person sensing. As an
example scenario of the fourth-person sensing, we focused on
an ambiguous verbal communication with a service robot, andwe developed a system which combines ﬁrst and third-person
sensing. Experimental results showed that the effectiveness ofthe proposed system and we conﬁrmed that the fourth-person
sensing contributes for more accurate understandings of user’s
requests.
The proposed system for the fourth-person sensing does notinclude the second-person sensing yet. Thus the future work
includes combining ﬁrst, second, and third-person sensing, and

1113[1] R. Kurazume, Y. Iwashita, K. Murakami, and T. Hasegawa, Introduction
to the robot town project and 3-d co-operative geometrical modelingusing multiple robots, in 15th International Symposium on Robotics
Research (ISRR 2011), 2011.
[2] O.M. Mozos, T. Tsuji, H. Chae, S. Kuwahata, T. Hasegawa, K.
Morooka, and R. Kurazum, The intelligent room for elderly care, in
1The ﬁfth International Workshop on the Interplay between Naturaland Artiﬁcial Computation (IWINAC2013), 2013.
[3] https://github.com/irvs/ros tms/wiki.
[4] F. Cavallo, M. Aquilano, M. Bonaccorsi, R. Limosani, A. Manzi, M.
Carrozza, and P. Dario, On the design, development and experimentationof the astro assistive robot integrated in smart environments, in IEEE
International Conference on Robotics and Automation (ICRA), 2013.
[5] F. Martinez, A. Carbone, and E. Pissaloux, Combining ﬁrstperson
and third-person gaze for attention recognition, in IEEE International
Conference and Workshops on Automatic Face and Gesture Recognition(FG), 2013.
[6] K.M. Kitani, T. Okabe, Y. Sato, and A. Sugimoto, Fast unsupervisedego-action learning for ﬁrst-person sports videos, in Conference on
Computer Vision and Pattern Recognition (CVPR), 2011.
[7] I. Laptev, On space-time interest points, International Journal of Computer Vision, vol. 64, no. 2-3, pp. 107-123, 2005.
[8] M.S. Ryoo and L. Matthies, First-person activity recognition: What
are they doing to me?, in Conference on Computer Vision and Pattern
Recognition (CVPR), 2013.
[9] Y. Iwashita, A. Takamine, R. Kurazume, and M.S. Ryoo, First-person
activity recognition from animal videos, in International Conference on
Pattern Recognition, 2014.
[10] G. Csurka, C. Dance, L. Fan, J. Willamowski, and C. Bray, Visual
categorization with bags of keypoints, in Workshop on statisticallearning in computer vision, European Conference on Computer Vision.
Prague, 2004, vol. 1, pp. 1-2.
[11] F. Perronnin, J. Sanchez, and T. Mensink, Improving the ﬁsher kernel forlarge-scale image classiﬁcation, in European Conference on Computer
Visions, pp. 143-156. Springer, 2010.
[12] H. Jegou, M. Douze, C. Schmid, and P. Perez, Aggregating local
descriptors into a compact image representation, in Computer Vision
and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE,
2010, pp. 3304-3311.