The final publication is available at IEEE via
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6705506

Robotic Service Coordination for Elderly People and Caregivers
with Ubiquitous Network Robot Platform
Masahiro Shiomi, Koji Kamei, Tadahisa Kondo, Takahiro Miyashita, Norihiro Hagita, Member, IEEE

Abstract— This paper proposed a method for deploying
robotic service coordination for elderly people and caregivers.
By showing two kinds of robotic services for them in realenvironments, we explain what important designs to deploy
robotic services are. For this purpose, we utilized an
infrastructure, called Ubiquitous Network Robot Platform(UNR-PF), allowing robot service coordination with robots,
smartphones, sensor networks. Three layered architecture of
UNR-PF supports developers of personalized robotic services
and their coordination to utilize different types of hardware
according to characteristics of users. In the case studies of
elderly care, robotic services should consider different types of
hardware (e.g., wheelchair type) due to characteristics of users;
it makes difficulties for design of robotics services.

I. INTRODUCTION
Robotic services consist of various robotic components,
such as sensor devices, actuation devices (i.e., robots) andsmartphones as both sensor and interaction devices, andservice applications that provide people with particularservices in real world environments by controlling suchdevices. Robotic services are expected to be an importantfactor to support daily life in the real world, particularly forsuper-aging society.
Japan had become super-aging society. In such society,one of important social problems to be solved is to encourage
elderly people in social participation, especially for those whoare living alone or who require nursing care. In this context,
robotic services are being expected for elderly care purposes
from the viewpoints about physical/mental support for suchelderly people (Figure 1).

Figure 1 Illustrations of robot services for elderly people* This research was supported by the Ministry of Internal Affairs andCommunications of Japan.
Masahiro Shiomi, Koji Kamei, Tadahisa Kondo, Takahiro Miyashita, andNorihiro Hagita are with the Advanced Telecommunications ResearchInstitute International, 2-2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto,
JAPAN (corresponding author to provide phone: 81-774-95-1432; fax:
81-774-95-1408; e-mail: m-shiomi@atr.jp).

In fact, many researchers have developed to support
elderly people from physical/mental support viewpoints For
example, from the viewpoint of physical assistance,
Dubowsky et al. have developed a robotic aid system which
supports elderly people’s mobility and monitoring [1]. Grafhave also developed an adaptive guidance system for robotic
walking aids [2]. Kulyukin et al have focused on carrying
baggage services, and then they developed a mobile cart robot
which works on a shopping mall to support shopping
physically. Mutlu et al. developed a conveyance robot to assist
hospital staffs, which is an indirect assistance for seniorcitizens [3].
From the viewpoint of mental care of elderly people,
several robotics research works have also been conducted. For
example, Paro is one of famous robotics products which used
for mental care for elderly people in nursing homes [4].
Several robots, which do conversations as reminders of
medicine schedules, are also used for mental cares of them,and used for preventing them from forgetting to take medicine[5][6]. Iwamura et al. have investigated the influences ofappearances and chats of interactive robots toward theimpressions that elderly people receive under shopping
support situations [7].
However, though several robotics services are achieved in
the level of “research,” they are still a little far from“commercialization.” To solve problems related tosuper-aging society, it is important to deploy various robotic
services into real world environments. For this purpose,developers should consider how to organize different kinds ofrobots for various services because appropriate robot
hardware varies depending on users’ characteristics.
Infrastructures for developing robotic components andmanaging their resources and assigning them for service
applications should be commonly used among developers andoperators to provide variations of robotic service applications
in particular service environments.
Though ROS [8] and RTM [9] have been developed as
middleware to accelerate developments of robot systems, they
however have mainly focused on relatively low-layer levels(robot component layer of UNR-PF) that enable control ofrobotic components, not strongly focused on robotic service
applications. Coordination between robotic services should
also be supported, because it is still difficult to realize astandalone robot that can provide multiple services only byitself.. For this purpose, to realize service coordination among
multiple robots, following four topics should be addressed,
such as, definitions of common representation for robots,classifications of robot abilities, managements of available
robot resources, and sharing knowledge between different
robots. Some of related works achieved robotic service
coordination by using networked robot systems [11] [12].

The final publication is available at IEEE via
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6705506

Moreover, RoboEarth [13] and CoTeSys [14] realized
information sharing among different types of robots. But,
these trials are relatively focused on multiple robot
coordination of information sharing, not strongly focused on
an infrastructure which enables developers to easily realize
robot services in real environments.
From these considerations, we have developed a robotic
service framework, ubiquitous network robot platform(UNR-PF) [14], to enable developers to realize various robotic
services applications easily. UNR-PF is consisted of threelayers: service application layer, UNR-PF layer, and robot
component layer. The service application layer is notdepended on devices (i.e., robots) so that application
developers can just focus on the design and development ofrobotic services by ignoring limitations of robot hardware.
The UNR-PF layer is designed to connect both application androbot component layers properly by consideringcharacteristics of person, robots and environments.
In this paper, we introduce the background and details of
UNR-PF and then describe two examples of real robotic
service coordination. First one is a shopping support in a
shopping mall (Figure 2). This service includes multiple
robots and smartphones. Elderly easily uses shopping support
applications through smartphone and interacts with the robots.
Second one is a robotics support in nursing home for bothcaregivers and elderly people. This project is in progress. The
system uses semi-autonomous wheelchair robots as a mobility
support and depth sensors for watching safety in the middle ofthe night. In both examples, UNR-PF realized robotic service
coordination with multiple robots considering personal
characteristics of users.

working robotic systems by adopting existing functional
modules in combination with their own software. Suchmodularized development process has accelerated thedevelopment of standalone robots as well as individualfunctional components; however, there exist gaps between
development of robots and service applications. Anotherabstraction layer for service development is required inaddition to the component oriented development.
In our previous project, six kind of robotic services havebeen investigated as case studies for verification oftechnologies and service coordination, such as shop guidance,
shopping support, touring support, active hearing, community
building support and health care services. Requirements forcommon functions arose from those case studies (Table 1),
and then the specifications of such functionalities have been
proposed as middleware architecture called UNR Platform(UNR-PF).
UNF-PF is a common infrastructure that supports
development of robotic services as applications of networked
robot technologies. The requirements for the platform are
mapped into the following four categories.
1. Multi-Robot Management: a mechanism to assign
suitable robots to the demands of services and users.
2. Multi-Area Management: a mechanism to share spatial
information of multiple areas such as map information
including dynamic location information of moving entities
and service oriented information in addition to static spatial
structures of the service environments.
3. User Attribute Management: to support the daily
activities of various users, especially the elderly and disabled,
it is important to understand users’ abilities and required
supports such as moving assistance, preference for visualcommunication than aural one, etc. Platform should allocate
robotic devices suitable for each user’s preference while a
service does not notice of it.
4. Service Coordination Management: to execute a numberof services across multiple areas, platform should provide
mechanisms for managing the state of service execution andthat for sharing information among services.

Figure 2 Robotic service selections via a smartphone
II. UNR-PF
As robotic services grow large, robotic systems also
become complicated and then development of such a system
also becomes harder and harder. Similar to traditional
software engineering, people in robotic technology started toseek modularity and reusability of basic functional
components and then invested development of common
libraries and middleware [8, 9, 15, 16].
Nowadays, robot developers can share functional modules
for robotic devices on common platforms and rapidly develop

Figure 3 shows a structure overview of the UNR-PF.
Following subsections describe the architecture, its internal
components, relationship with several international standards,
and then introduce current status of the implementation.

The final publication is available at IEEE via
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6705506

Figure 3 Overview of UNR-PF
A. System Architecture and Deployment
The middleware consists of two platform layers: a local
platform for the robotic system in a single area and a global
one for the robotic system that ranges over multiple areas
covered by a number of local ones (Figure 2). These
platforms serve as a middle-layer between the layer of service
applications and that of robotic components including
smartphones, agents sensor networks, etc.
B. Internal structure of the platform
First, service applications and robots in each environment
are connected to UNR-PF and register with it. The
applications and robots discover each other on the UNR-PF
and start interacting among themselves.
UNR-PF itself is composed of two platform layers: a local
platform (LPF) and a global platform (GPF). LPF is aplatform for the robotic system in a single area; GPF is aplatform for the robotic system that ranges over multiple
areas covered by a number of LPFs. These platforms serve asa middle-layer between the service application and therobotic component layers.
The platform is equipped with five database functions andthree management functions to provide common services tothe service applications and robots. The database functions
consist of robot, map, user, and operator registries, andservice cues. The management functions consist of state,
resource, and message managers (Table 2).
Table 1 Basic HRI component arose from case studies
HRI Component
Description
System information
Provides the information of thesystem such as status of the systemand position of the physical unit.
Person detection
Detects number of people.
Person localization
Detects position of people.
Person identification Identifies IDs (names) of people.
Face detection
Detects number of human faces.
Face localization
Detects position of human faces.
Sound detection
Detects number of sound sources.
Sound localization
Detects position of sound sources.
Speech recognition
Recognizes person’s speech.
Gesture recognition
Recognizes person’s gesture.
Speech synthesis
Generates robot speech.
Reaction
Performs specified reaction.
Navigation
Moves to specified target location.
Follow
Follows a specified target object.
Move
Moves to specified distance or
curve.

The final publication is available at IEEE via
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6705506

Table 2 Details of each registry and manager of UNR-PF
Description
RobotregistryMapregistry

User
registry

Operator
registry

ServiceregistryStatemanager

Resource
manager

Messagemanager

It contains information about the robots available inarea(s), such as robot IDs, shapes, mobility
capability, and transporting capacity. The platformrefers to this database to assign robots in a LPF
suitable for service applications.
It provides map information of the service execution
environment, including the floor properties andinformation about movable and no-go zones. The
map registry in GPF provides positional relations
among single areas covered by LPFs to improve
service linkage between areas.
It manages information about service users toprovide appropriate services and robots to users. This
database contains the attributes of each user, such as
user ID, call name, degree of walking ability, andsight and hearing abilities.
It manages information about the operators of robots
and/or services. The database contains the available
skills of all operators with their operator IDs.
Therefore, service applications can hire operator
assistance, if required, by the resource manager
described below.
It manages the invocation of services. This database
contains the ID of each registered service andinformation about conditions when invoked by theplatform. When a service application registers itself
to the platform, its ID and initiation conditions are
stored in the service queue.
It subscribe to the message manager for state
notifications registered in the service queue. Whenthe manager receives a state notification, it
determines if the state complies with the start
conditions in the service queue. If it does, themanager sends a message to the service application
to start the service.
It manages the assignment of such resources as
robots and operators. After receiving a commandexecution message from a service application, it
refers to the robot, user, and operator registries toreserve a suitable robot and an operator who canoperate the service and the robot depending on thesituation.
It exchanges between service applications androbotic components are handled by message
managers through a common interface. When the
GPF manager receives a message from a service
application, which is typically a request for acommand execution, it refers to the registeredprofiles of the robotic components and delivers it to
the suitable ones.

C. Standardization
To share information among the robots and the service
applications and achieve interoperability among different
robots, the specifications of the data structures and interfaces
should be standardized. In parallel to the prototype
development, the key elements of the platform technologies
have been proposed to several organizations as international
standards. Four key elements of the prototype system, such as
map information, location information, common interfaces,and platform architecture, have been standardized.
Map information is standardized in the Open Geospatial
Consortium (OGC), a consortium for standards associatedwith geographical information. Our request for extension of
the CityGML specifications for allowing maps to containrobot-specific information has been accepted and issued as an
example of application domain extension (ADE) of CityGML
version 2.0.0 in April 2012 [17].
To exchange location/pose information among various
networked robot elements and robotic services, this standard
proposes a new framework for robotic localization (RoLo) for
the representation and treatment of location information
specific to robotic usage.
A standard specification for describing and exchanging
location and pose information for robots has been issued as
the Robotic Localization Service (RLS) specification andupdated as RLS 1.1 in August 2012 [18].
The standardization of common interfaces between service
applications and robotic functional components is treated in
OMG as the Robotic Interaction Service (RoIS) Frameworkspecification. The initial version of the specification has been
issued as RoIS 1.0 in February 2013 [19].
The common platform architecture was discussed in theInternational Telecommunication Union Telecommunication
Standardization Sector (ITU-T), study group 16 (SG16). The
recommendation, initially called F.USN-NRP, was accepted
as a standardization work item in 2011 and has been released
as F.747.3 in March 2013 [20].

III. SERVICE APPLICATION I: SHOPPING SUPPORT IN A MALL
To evaluate the effectiveness, we have conducted a casestudy with elderly people. This service application provides a
touring service that was provided through three areas: theelderly person’s home, the shopping mall, and the operator
center.
Figure 4 shows an overview of the case study which uses ahumanoid robot. In this service application, firstly a user
makes a shopping list by using smartphone. When the user
visits the shopping mall, the service localizes the user’s
position by using environmental sensors then a humanoid
robot approaches to the user and greet to the user. Then, thehumanoid robot carries baggage and starts following the user
during shopping.
Figure 5 also shows another variation the case study inwhich a wheelchair robot is used for elderly people who havewalking difficulty. In this service application, firstly a user
reserves the shopping support service by using smartphone,too. When the user visits the shopping mall, the service
estimate arrival timing by using GPS information via
smartphone, then makes a wheelchair robot go to the entranceof the shopping mall with appropriate timing. As a result, theuser can use the appropriate wheelchair robot for his/her
shopping. We finally developed software which enables users
to select possible services depending on user’s characteristics(Figure 2).

The final publication is available at IEEE via
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6705506

support is quite important to their life, especially from theviewpoint of preventing homebound of elderly people.
However, in transfer support, a caregiver basically needs
to move together with an elderly person by pushingwheelchair. Although transfer support is important for
healthcare for elderly people, it is true for care givers that its
workload is high, particularly from the viewpoint of total timespent at work. For this problem, we proposed the use of
semi-autonomous wheelchair robot which transfers elderly
people safely with decreasing workloads of caregivers. On
behalf of care givers, it enables robot operators or families of
elderly people to support locomotion of wheelchair robots,
and talk with them via networks.
Figure 4 A shopping support with a humanoid robot

We designed semi-autonomous wheelchair robots use its
own sensors such as laser range finders and environmental
sensors such as 3D depth sensors installed in order tounderstand environments, such as avoiding people in acorridor while navigation. Moreover, UNR-PF provides
information about environment properties such as floor
characteristics and personal properties such as the nursing carelevel of users to adapt the behaviors of wheelchair robots.
Operators can watch the status of elderly people and thewheelchair via network, and they can talk with elderly people
or control wheelchair under transfer support. Even if aproblem happens to an elderly people riding on a wheelchair,
operators can instantly inspect the situation via network. We
are planning to extend our past tele-operation mechanism
which enables one operator to control five robots [21].

Figure 5 A Shopping support with a wheelchair robot

IV. SERVICE APPLICATION II: TRANSFER SUPPORT AND
WATCHING OF FALLING OUT OF A BED IN A NURSING HOME
Due to increase in the number of elderly people, elderly
care is becoming an important social problem in Japan. In
particular, hard works of caregivers and acute shortage of
caregivers are major problems in elderly care. Decreasing theworkload of caregivers by using robotic services would help
them to increase the quality of elderly care, and consequently
elderly people would be able to get more health. Moreover,use of robotic services would be easy and comfortable for
elderly people more than asking a help to caregivers in busy.
From these considerations, we are starting a new project
which uses robots and sensing technologies to decrease
workload of caregivers by supporting transfers of elderly
people and watching them of falling out of their beds in
midnight at a nursing home. For this purpose, UNR-PF is also
used for managing robotic components and service
applications by considering users characteristics such as
nursing care level (a degree of care-requirements in Japan).
A. Transfer Support
Figure 6 illustrates an overview of our transfer support
service application. Semi-autonomous wheelchair robots areused to support transfer of elderly people in a nursing home.
For elderly people who have walking difficulty, transfer

Figure 6 Overview of transfer support by wheelchair robots

B. Watching of Falling Out of a Bed
Figure 7 illustrates an overview of our watching service
application. Depth sensors in a room are used to detect
dangerous scenes such as falling out of a bed. For elderly
people, such accidents sometimes happen while they are
sleeping. Another problem related to falling out of a bed isdelay of finding of people. If such accident happens in
midnight and missing them long-time, it causes criticalinjuries for them.
This problem happens not only in nursing homes but also
in their homes, therefore it is one of main concerns of familiesof elderly people. Currently, floor sensors around a bed areused to detect such accident but such sensors often
miss-recognize the situations (e.g., when an elderly person

The final publication is available at IEEE via
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6705506

gets down of the bed to go to a restroom at night, the sensors
detects pressures and then sends a warning notification tocaregivers) and expensive running cost are often problems for
caregivers.
To solve such problems, we proposed use of depth sensors
to recognize both dangerous situations and safe ones
accurately. Current depth sensors are relatively inexpensive
and non-contact sensors, therefore use of depth sensors isappropriate from the viewpoint of cost merits.
For this purpose, we consider to use infrared-type depth
sensors such as Kinect to recognize situations where elderly
people falling out of beds. Moreover, such sensors can also be
used to observe daily activities of elderly people inenvironments; long-term data analysis would be helpful todetect anomaly situations such as wandering in theenvironment. We have already developed a system which
estimates future behaviors or positions of pedestrians by using
stored trajectories [22], we are planning to extend the system
to recognize and estimate behaviors and positions of elderly
people. Finally, the system can send warning notifications tocaregivers via network when the systems recognize or
estimate dangerous situations, so that the caregivers caneffectively support them.

Figure 7 Overview of watching support
V. CONCLUSION
This paper reports an overview of UNR-PF which is an
infrastructure to realize coordination of robotic services. We
also showed several case studies of robotic services for
elderly people and caregivers in real environments developedwith the three-layered architecture supported by UNR-PF.
ACKNOWLEDGMENT
We wish to thank the staffs at APITA Seika-dai Co., Ltd.
for their helpful participation. We also wish to thank Satoru
Satake, Satoshi Koizumi, Dylan F. Glas, Tetsushi Ikeda andMasaya Shimoyama for their help.

REFERENCES[1]

S. Dubowsky et al., “PAMM: A Robotic Aid to the Elderly for Mobility
Assistance and Monitoring: A "Helping-Hand" for the Elderly,” inProc. of the 2000 IEEE Intl. Conf. on Robotics and Automation(ICRA2000), pp. 570-576, 2000

[2]

B. Graf, “An Adaptive Guidance System for Robotic Walking Aids,” J.
of Computing and Information Technology, vol. 17, no. 1, pp. 109-120,
2009[3] B. Mutlu and J. Forlizzi, “Robots in organizations: the role of workflow,social, and environmental factors in human-robot interaction,” in Proc.
of the 3rd ACM/IEEE Int. Conf. on Human Robot Interaction(HRI2008), pp. 287-942, 2008
[4] T. Shibata, “An overview of human interactive robots for psychologicalenrichment,” Proceedings of the IEEE, vol. 92, no. 11, pp. 1749-1758,
2004[5] M. Montemerlo et al., “Experiences with a Mobile Robotic Guide for
the Elderly,” in Proc. of Int. Conf. on Artificial Intelligence, Edmonton,
pp. 587-592, 2002
[6] M. E. Pollack, “Intelligent technology for an aging population: The useof AI to assist elders with cognitive impairment,” AI magazine, vol. 26,
no. 2, pp. 9-24, 2005
[7] Y. Iwamura, M. Shiomi, T. Kanda, H. Ishiguro and N. Hagita, "Do
Elderly People Prefer a Conversational Humanoid as a Shopping
Assistant Partner in Supermarkets?," in Proc. of the 6th ACM/IEEE Int.
Conf. on Human Robot Interaction (HRI2011), pp. 449-456, 2011.
[8] M. Quigley et al., “ROS: an Open-Source Robot Operating System,” inProc. of the 2009 ICRA Workshop on Open-Source Software, 2009[9] N. Ando et al., “RT-Middleware: Distributed Component Middlewarefor RT (Robot Technology),” in Proc. of the 2005 IEEE/RSJ Int. Conf.
on Intelligent Robots and Systems (IROS2005), pp. 3933–3938, 2005[10] A. Sanfeliu, N. Hagita and A. Saffiotti, “Network Robot Systems,”
Robotics and Autonomous Systems, vol. 56, no. 10, pp. 793–797, 2008[11] P. Salvini, C. Laschi and P. Dario, “Do Service Robots Need a Driving
License?,” IEEE Robotics and Automation Mag., vol. 18, no. 2, pp.
12–13, 2011[12] M. Waibel et al., “RoboEarth,” IEEE Robotics and Automation Mag.,
vol. 18, no. 2, pp. 69–82, 2011[13] M. Tenorth et al., “Web-Enabled Robot,” IEEE Robotics andAutomation Mag., vol. 18, no. 2, pp. 58–68, 2011[14] K. Kamei, S. Nishio, N. Hagita, M. Sato,"Cloud networked robotics,"
IEEE Network, vol. 26, no. 3, pp. 28-34, 2012[15] H. Bruyninckx, “Open robot control software: the orocos project,” inProc. of the 2001 IEEE Int. Conf. on Robotics and Automation(ICRA2001), pp. 2523-2528, 2001[16] H. S. Jung et al., “Unified ubiquitous middleware for u-city,” in Proc.
of the 2007 Int. Conf. on Convergence Information Technology, pp.
2374-2379, 2007[17] Open Geospatial Consortium, Inc., “OpenGIS City Geography Markup
Language (CityGML) Encoding Standard version 2.0.0,” 2012[18] Object Management Group, “Robotic Localization Service (RLS)
version 1.1,” 2012[19] Object Management Group, “Robotic Interaction Service (RoIS)
Framework version 1.0,” 2013[20] ITU-T, “Recommendation F.747.3: Requirements and functionalmodel for ubiquitous network robot platform to support USN
applications and services,” 2013[21] K. Zheng, et al., “Supervisory Control of Multiple Social Robots for
Navigation,” in Proc. of the 8th ACM/IEEE Int. Conf. on Human Robot
Interaction (HRI2013), pp. 17-24, 2013[22] T. Kanda, D. F. Glas, M. Shiomi and N. Hagita, “Abstracting People's
Trajectories for Social Robots to Proactively Approach Customers,”
IEEE Transactions on Robotics, vol. 25, no. 6, pp. 1382-1396, 2009