Special Feature

Platform Technology for Ubiquitous
Services
Toshiki Iso†, Yoshinori Isoda, Kiyotaka Otsuji,
Hiroki Suzuki, Shoji Kurakake, and Toshiaki Sugimura
Abstract
This article introduces platform technology that i) makes it easy for users to access the services theywant among numerous services available everywhere in a ubiquitous environment and ii) uses a sensor
network to ascertain a user’s context when the user accesses a service and allows the service to be usedin the most convenient way.

1. Introduction
Over the last few years there has been a growing
need for computer networks that allow services to be
used comfortably at any time, anywhere, and by anyone. To achieve this, it is essential to develop services
that can be made available to users in a form suited totheir context by automatically discovering services
related to ubiquitous real-world computing resourcesand other equipment and objects.
In this article we discuss our research on technology associated with platforms for discovering and supporting services from a wide variety of real-world
objects and associated services in situations whereusers carry hand-held equipment with them. We also
discuss techniques for tailoring services to the user’s
context by gathering data from ubiquitous sensors situated in the environment.
2. Ubiquitous discovery service platform (UDSP)
To implement a ubiquitous discovery service(UDS) that associates objects in the real world withservices on a network, at least the following functions
are required:• A function that enables real-world objects to be
distinguished on a UDSP

• A function that lets one discover services on a
network that relate to real-world objects
This means that it is necessary to prepare techniques ranging from the management of IDs forobjects (including electronic content) to the management of services. However, although individual techniques have been proposed in various forms and byvarious groups, they do not form a comprehensive
system. For example, although Auto-ID Center provides a systematic platform for functions ranging
from ID management to service management, it does
this by introducing an independent ID format calledePC, which is incompatible with conventional IDs.
Moreover, it does not prescribe a general-purpose
service declaration for selecting services according tocontext or various types of service relationships.
Therefore in this article we propose a UDSP architecture that provides a systematic means of organizing and supporting functions for minting and managing IDs, which is compatible with conventional ID
codes and functions for reading IDs that can be usedeven in mobile environments, and which provides
functions for managing services associated with these
IDs.
Figure 1 shows a service platform that links realworld objects and services on the network. It therefore lies at the core of ubiquitous services. The UDS
mainly consists of the following key technologies.

† NTT DoCoMo Multimedia Laboratories
Yokosuka-shi, 239-8536 Japan
E-mail: iso@mml.yrp.nttdocomo.co.jp

82

NTT Technical Review

Special Feature• ID minting andmanagement• Management of services
related to IDs

UDSP (management of services bound to IDs)
• Universal ID issuing management system• BIND (binding ID with services) data management system• Universal ID capturing support system(3) Request service-related
information based on ID

ID⇔
serviceassociations(4) Provide servicerelated information

Service portal(2) Request services
based on ID• Service discovery system• Context information extraction
system(1) Read ID(5) Service discovery
Service provider
User

FOMA
(V type)Object ID(Visualtag)(6) Provide service

Fig. 1. UDS platform.

2.1 Identification of objects in mobile
environments
To allow objects to be identified wherever they are,they are labeled with Visualtag [1] markers representing their ID codes based on geometric invariants
that do not depend on the viewing direction. Compared with conventional ID-tags such as barcodes,
these markers have fewer constraints in terms ofviewing distance and angle. As a result they allow

objects to be identified easily by using a camera built
into a mobile terminal. By processing the images ofthese tags to take account of different lighting conditions, it is possible to extract their features stably,
even from images captured with the sort of camera inmobile terminals. As a result, there is no need for aseparate reading device (Fig. 2).

q1

q2

q5

q3

Real-world object
Visualtag

q4

ID can be scanned robustly from any perspective
p2

p1
p5

p3
p4

Captured image
IDs are acquired in
mobile environments
by using image processing
techniques that can tolerate
different lighting conditions.

Mobile terminal with a built-in camera

ID is derived from geometrical invariants.

Fig. 2. Object recognition with Visualtag.

Vol. 1 No. 8 Nov. 2003

83

Special Feature

2.2 Management and use of universal IDs
By introducing an ID management scheme that canperform integrated management encompassing theexisting ID code system of “universal IDs”, the platform can also handle existing barcodes and RF-IDs intheir original form.
2.3 Management of services associated with IDs
We have proposed a “BIND*1 data ticket” service
scheme that allows a wide variety of information to
be associated with all kinds of objects, thereby offering the freedom to do things such as linking together
multiple services, presenting services according to auser’s preferences or circumstances (e.g., location),and placing restrictions on the use of each service.
3. Example of a ubiquitous service using the UDS
platform
Here we describe the process that takes place in the
UDSP when it is used to support a ubiquitous service.
Figure 3 shows an example of a ubiquitous service ina mobile environment. When a user notices an interesting poster in the street, the following procedureshows how it is possible for information to beobtained from a real-world object (the poster), resulting in the provision of a related service.
First, the user uses a camera mounted on a handheld device to read in a Visualtag that has been added*1 BIND: binding ID with services

to the poster and transmits this image data to the
UDSP. A service portal system in the UDSP acquires
the user’s context data (user location, user preference,etc.), and sends the image data to a universal ID reading management system located within the UDS. The
universal ID reading management system analyzes
and extracts the geometrical invariants from theimage data and transmits them to a universal ID issuing management system in the UDS. The universal
ID issuing management system then extracts a universal ID as the code associated with these geometrical invariants, and transmits it to a BIND data management system in the UDS. The BIND data management system retrieves the various services associated with the universal ID and transmits them to theservice portal. In the service portal, the optimal services based on the user’s context are presented to theuser, allowing the user to select the required service(e.g., connecting to a website that presents a videocommercial, or to a sales information website).
As another example, a tourist information sign
could be made to provide users with video data even
if it does not have its own display device. Instead, theuser uses a hand-held device with a built-in camera toread in a Visualtag from the sign and transmit it to the
UDSP, which allows the user to see a tourist information video or a live video image of a particular destination.
Similar techniques can be used to implement personalized “info-attachment” services such as linking
the ID of a rail pass to an electronic timetable, or the
ID of a gift card to an online audio greeting.

PosterID (Visualtag)
Product information
Transmit ID(Visualtag) image

Door plate
W-C302Multimedia
Laboratories

Live video

UDS system

Video content

Service portal
Relatedinformation

Deliver service
ID (Visualtag)

FOMA(V type)

UDSP(manages IDs and associated services)

Tourist information
Tourist information board

Live video

ID ⇔service database

Kamakura Hachimangu
ID (Visualtag)

Fig. 3. Examples of ubiquitous services based on the UDS platform.

84

NTT Technical Review

Special Feature

4. Real-world information platforms andintelligent user support
Although the Internet has become an indispensable
information platform for human activity, it is still
considered a virtual world that exists separately fromthe real world. On the other hand, there is a growing
trend towards connecting cellphones and sensor
equipment such as cameras to the Internet, making it
possible to incorporate data that is closely related tohuman activities. Furthermore, there is growing
activity aimed at making computer resources and sensors ubiquitously available throughout the environment, as in ubiquitous computing for example. It isthus likely that in the future the Internet will develop
into a real-world platform incorporating informationon the diverse activities engaged in by humans in thereal world (Fig. 4). This real-world platform shouldmake it possible to provide intelligent user support,such as providing services tailored to the user’s context.
5. Experimental house and user support
systemTo create a system that supports users intelligently,
it will probably be necessary to promote interaction
between the environment, mobile terminals, andhumans rather than simply enhancing the ability ofmobile terminals or environment-side systems. Also,
to achieve smooth interaction it is essential to gather,store, and analyze the activity of individuals overlong periods and to utilize the results. A number ofother systems have been proposed for monitoring theactivity of humans, but the system we are aiming todevelop in this study differs in the following two
respects.
First, a ubiquitous sensor network is provided aspart of the environment so that information can begathered both from mobile terminals and from theenvironmental sensor network.
Second, the system is designed based on the maximthat interaction between users and the environment isfundamental to providing user support, rather thansimply enhancing the data storage or processing abilities of the mobile terminals. This makes it possible toutilize many different types of information in theubiquitous environment, and should make it easier toascertain a user’s context when providing support.
We have therefore built and started trials of a prototype system aimed at supporting users intelligently
based on cooperation between mobile terminals andenvironment-side systems. This system is installed in
an experimental house [2]. Users of the experimental
house move around indoors while carrying sensorequipped notebook PCs or cellphones (both of which
are referred to below as mobile terminals). These
mobile terminals provide the users with services tai-Symbiotic agent platform
Applications

Real-world information platform

Context
extraction

Personal activity
information

User-tailoredsupportUser

Environment information platform

Internet, remote equipment

Household appliances,office equipment

Mobile equipment

Sensor
networkFig. 4. Real-world information platform.

Vol. 1 No. 8 Nov. 2003

85

Special Feature

5.2 Virtual tags
To allow users to interact withthe environment in a variety ofways depending on their locaTransmit activity data
Experimental house
tions and the time, virtual tags
Hand-held terminals
can be set inside the house.
Used to display and select data,
These virtual tags are set byelicit responses, and gather terminal
sensor data
each user on a server in a
Omnidirectional visual sensor“5W1H” format (when, where,
Keyboard and mouseby who, for who, what, and
Acceleration sensorhow). Information that matches
Transmit virtual
these conditions is provided to
tag information
the mobile terminal of the same
Virtual tags
user or to other users. For examChannels for interaction with the environment
at designated points in time and space.
ple, it is possible to deliver
reminder data in the form ofTransmit user’s
memoranda (e.g., a reminder toposition
Visual sensor network
take one’s medicine or bring an
Ceiling cameras track user position
umbrella), or recommendation
data in the form of suggested´16
information (e.g., weather fore1F
casts or train timetables). InterWall cameras (omnidirectional visual sensors) store
2F
faces to electrical equipment
images and recognize user’s behaviors
inside the experimental house,such as the TV, can also be pro´25
vided as recommendation data.
Furthermore, it is possible toFig. 5. User support system.
provide users with information
by way of interaction between
lored to their context (Fig. 5).
users and the environment in the form of communication data. It is envisaged that this interactive informa5.1 Visual sensor networktion can be used to extract information about the
Two types of camera have been installed inside theuser’s characteristics, although this is not directly
house. The ceilings are fitted with 16 cameras incorlinked to supporting users.
porating conical mirrors that allow them to take pictures over a wider angle than ordinary cameras. The
5.3 Hand-held terminals
pictures taken by these ceiling cameras are subjected
When a user carries a notebook PC around in theto differential processing to extract the regions occuexperimental house, the PC acquires various types ofpied by humans and thereby detect the user positions.
information—omnidirectional images from a built-in
The positions of multiple users can then be tracked incamera, movement data from an acceleration sensor,
real time by combining the user position data
and data input via the mouse and keyboard—andobtained from multiple cameras. The walls are fitted
transmits it to an activity information server via awith 25 cameras called omnidirectional visual senwireless LAN (11 Mbit/s). If the user’s terminal is asors that can take pictures over a range of 360° in thecellphone, it uses an “i-application” to display rechorizontal direction. Of these cameras, the one closommendation information in the form of a list. The
est to the position of each user (as detected by theuser can access all kinds of information and control
ceiling cameras) is selected to provide a function fordomestic appliances by making selections from thissaving images as user activity monitoring informalist. For the PC user, reminder information is distion and a function for recognizing the behaviors ofplayed in the form of a pop-up window, while for theusers indoors, such as standing and sitting.
cellphone user it is delivered as a text message.
Communication data is only supported when theuser has a notebook PC. This allows the user to interData analysis
Movement patterns,
probabilistic models,motion primitives

86

Activity information server
Storage, retrieval, delivery

NTT Technical Review

Special Feature

act with the environment by choosing a preset
response phrase or inputting any text response whenpresented with an interactive phrase in a pop-up window. This interaction information is stored on theactivity information server as user activity information along with the sensor information obtained fromthe environment sensors and the hand-held terminals.
5.4 Activity information server
Data transmitted from each user is stored on theactivity information server in XML format. The
activity information server also supports a function
for delivering data in response to user requests foraccessing the activity information.

6. Applications
Sometimes a user may wish to access information
about his or her own activities or the activities ofother users. For this purpose, we have implemented
an awareness communication application whereby
the activity information for each user recorded in theexperimental house can be retrieved by the corresponding user or by other users. This application
allows the previous or current data for a user to beaccessed by inputting the name of the target user, thetime range, and a keyword (Fig. 6). Since it is important to consider the privacy of users in such cases, thedelivery of data in this system is controlled by definRetrieval conditions setting window

Acceleration sensor data

Omnidirectional image from environment

Omnidirectional image fromhand-held terminal

Fig. 6. Retrieval and access of user activity information.
(m)
14
12
10
8
6
4
2
0
0 2 4 6 8 10 12 14 (m)
(m)
14
12
10
8
6
4
2
0
0 2 4 6 8 10 12 14 (m)(a) Movement traces over a 10-minute period(b) Movement traces over a whole day (7 hours)

Fig. 7. User movement traces.

Vol. 1 No. 8 Nov. 2003

87

Special Feature

ing a protocol that specifies the users who may accessthe information, and parameters such as the types ofdata that can be disclosed and their playback rates.
We are also conducting experiments to gather andanalyze the activity information of users over long
periods to optimize applications that use virtual tags
to deliver information to them. Figure 7(a) shows the
movements of an individual user during a 10-minuteperiod, and Fig. 7(b) shows the movements of multiple users over a 7-hour period. Based on the results ofthese experiments, we are using techniques such asprobability models to express the behavior patterns
that are common to all users and the behavior patterns
characteristic to each individual user.

7. Conclusion
We have introduced two types of research we are
performing to enable ubiquitous services. In a ubiquitous environment where a wide variety of services
are present, it is important to be able to search for anddiscover services efficiently and to tailor these services to individual users. To further enhance these
services, topics for further study include statusdependent service discovery and management, andcontext-sensitive service customization.
References
[1]
[2]

88

T. Iso, S. Kurakake, and T. Sugimura, “Visualtag Reader: Image Capture by Cellphone Camera,” to be presented at IEEE ICIP’03, 2003.
Y. Isoda, K. Ohta, T. Sugimura and H. Ishiguro, “The Acquisition andUtilization of Human Activity Information for a Real-World Information Infrastructure”, IPSJ 64th assembly, 6G-01, 3.341-342, 2002.

Toshiki Iso
Senior Research Engineer, NTT DoCoMo
Multimedia Laboratories.
He received the M.S. degree in applied physics
from Waseda university, Tokyo in 1988. Sincejoining NTT DoCoMo Multimedia Laboratories
in 1988, he has been studying image processing,pattern recognition, and ubiquitous service systems. He is a member of the Institute of Electronics, Information and Communication Engineers (IEICE) and the Optical Society of Japan.

Hiroki Suzuki
NTT DoCoMo Multimedia Laboratories.
He received the M.E. degree in school of information science from Japan Advanced Institute of
Science and Technology, Tokyo in 2000. Sincejoining NTT DoCoMo Multimedia Laboratories
in 2000, he has been studying context-aware and
UDSP technology. He is a member of IEICE.

Yoshinori Isoda
Research Engineer, NTT DoCoMo Multimedia Laboratories.
He received the M.E. degree in systems engineering from Osaka University, Osaka in 1993.
Since joining NTT DoCoMo Multimedia Laboratories in 1993, he has been researching sensor
processing systems and ubiquitous computing
systems. He is a member of the Information Processing Society of Japan (IPSJ) and the Robotics
Society of Japan.

Shoji Kurakake
Executive Research Engineer, NTT DoCoMo
Multimedia Laboratories.
He received the B.S. and M.S. degrees in mathematical engineering and information physics
from the University of Tokyo, Tokyo in 1983 and1985, respectively. Since joining NTT in 1985,
he has been researching character recognition,
image recognition, media handling, mobile
agents, and ubiquitous services. He is a memberof IEICE, IEEE, and the Association for Computing Machinery (ACM).

Kiyota Otsuji
Senior Research Engineer, NTT DoCoMo
Multimedia Laboratories.
He received the M.S. degree in solid-statephysics from Osaka University, Osaka in 1989.
Since joining NTT DoCoMo Multimedia Laboratories in 1989, he has been engaged in R&D ofstatus recognition, speech recognition, and
UDSP technology. He is a member of IEICE.

Toshiaki Sugimura
Senior Executive Research Engineer, NTT
DoCoMo Multimedia Laboratories.
He received the B.S. and M.S. degrees in computer science from Tokyo Institute of Technology, Tokyo in 1978 and 1980, respectively. In
1980, he joined Nippon Telegraph and Telephone Public Corporation (now NTT). He moved
to NTT DoCoMo in 1999. He is engaged inresearch on natural language processing, intelligent processors, mobile multimedia, ubiquitous
information-communications environments, andother subjects. He is a member of IEICE, IPSJ,
and ACM.

NTT Technical Review