Personal Information Annotation
on Wearable Computer Users
with Hybrid Peer-to-Peer Communication
Koji Makita, Masayuki Kanbara, and Naokazu Yokoya
Nara Institute of Science and Technology,
8916–5 Takayama, Ikoma, Nara, 630–0192, JapanAbstract. This paper proposes a wearable annotation overlay system
which can correctly annotate dynamic users of wearable computers. To
provide users with the newest annotation information, a network shared
database system for wearable AR systems has been proposed. With thedatabase, a wearable annotation overlay system which can dynamicallyannotate users of wearable systems has been investigated. In conventional
systems, since dynamic users’ position is transmitted to wearable ARsystems via a shared database server, it is diﬃcult to overlay annotations
at a correct position because of low frequency of updating and delay ofclient-server communication. In this paper, we propose a new eﬀective
method for wearable AR systems to obtain dynamic users’ positions byusing hybrid peer-to-peer(P2P). In experiments, annotations on dynamic
users have been proven to be overlaid correctly enough to show whereusers are.

1Introduction

With an advancement in computing technologies, a wearable computer can be realized[1]. Therefore, many wearable augmented reality(AR) systems have been
investigated for displaying location-dependent annotation information[2-7]. At
the same time, the concept of network shared database system for wearable ARsystems has been proposed[8] in order to eﬃciently provide users with newest
information. Furthermore, a wearable annotation overlay system which can annotate dynamic users of wearable systems has been developed with the network
shared database[9].
Fig.1 shows an example of wearable annotation overlay system which canannotate dynamic users of wearable computers. The system has possibilities ofnew eﬀective applications. Some examples are mentioned in the following.
– When we want to ﬁnd our acquaintances in a crowd of people, wearable ARsystems can show us where our acquaintances are.
– In case of emergency, we can easily and fastly ﬁnd a policeman or a medical
doctor around us by using wearable AR systems.
Z. Pan et al. (Eds.): ICAT 2006, LNCS 4282, pp. 217–227, 2006.
c Springer-Verlag Berlin Heidelberg 2006
218

K. Makita, M. Kanbara, and N. Yokoya

Annotations on other users
HamaWada

Wearable PC user
User’ View
Fig. 1. An example of annotation overlay system which can annotate dynamic usersof wearable systems

To annotate dynamic users of wearable systems in a multi-user environment,a user’s computer needs to obtain annotation information of other users in realtime. At present, a network-shared annotation database via a wireless network
is usually prepared and annotation information of users is managed by the annotation database in a server. Thus wearable annotation overlay systems obtainannotation information and position of users from the database. Therefore, it isdiﬃcult to overlay annotations of dynamic users on the correct position in AR
images because of the low frequency of updating information and communication
delay. In the case that there are displacements of annotations, it is diﬃcult forusers to recognize correspondences between annotations and dynamic users ofwearable systems. Therefore, the conventional framework works well only when
there are a limited number of clients around.
High frequency of updating and low communication delay make it possible forannotation overlay systems to overlay annotations of dynamic users at correct
position. If the system can correctly overlay annotations of dynamic users, thesystem might be suitable for much broader range of applications because thesystem can work well in the crowds of people. The purpose of the present study isto construct a new eﬀective method for wearable AR systems to obtain dynamic
users’ positions. To realize high frequency of updating and low communication
delay, we use both client-server communication and hybrid P2P. In the method,a client system transmits its position to the server at some intervals. The clientsystem also transmits its position using P2P to other client systems by requestfrom the server.
This paper is structured as follows. Section 2 describes the proposed method
for wearable AR systems to obtain dynamic users’ positions. In Section 3,

Personal Information Annotation on Wearable Computer Users

219

experimental results with a prototype system are described. Finally, Section 4
describes summary and future work.

2Annotation Overlay to Dynamic Users with Hybrid
P2P

2.1

Overview of the Framework

To overlay annotation information of dynamic users, two types of data below
are usually needed.
1. Users’ personal information
2. Users’ positions
For example, in Fig.1, users’ names are needed as personal information andusers’ positions are needed to overlay annotations on correct positions. Since
these data are updated successively, client systems have to obtain the latest datain order to annotate to dynamic users. Users’ personal information is acquired
once in a while. On the other hand, users’ positions are updated in real time.
Thereby, client systems can overlay annotations by obtaining users’ personal
information at regular intervals. However, users’ positions must be consecutively
obtained with a small communication delay. If a server provides client users’positions, time lag which arises from a delay of client-server communication
is not small. Furthermore, the server must send and receive all of the users’
positions continuously in real-time. In this case, loads of the server and thenetwork are generally high. To solve these problems, we employ hybrid P2P
communication for obtaining users’ position from other clients. However, since
we have to consider a network limit and marginal performance of user’s system,
it is diﬃcult to send and receive multiple users’ positions by only using P2P.
Because of this situation, we install a database server which can be accessedwith a wireless network and the database manages annotation information. The
server is shared by multiple users of AR systems and has two functions below.
1. Providing multiple users’ personal information and positions
2. Management of timing of starting and stopping P2P communication
Since the database contains personal data of multiple users, we have to carefully
consider the security and privacy issue. In this study, we apply existing networkshared database design[8] to the database which manages multiple users’ personal information and positions. Moreover, the database manages user’s ID andIP address to control P2P communication. In Section 2.2, the outline of theproposed system is described. Section 2.3 describes how to update and obtainannotation information by client-server communication. Section 2.4 describes
how to obtain dynamic users’ positions.

220

K. Makita, M. Kanbara, and N. Yokoya
User ID
Position
IP address

Network-shared database
User information table
Annotation table
Position table
Annotation request table
User information
management server

Antenna

Position
AnnotationUserWirednetwork
Position
Annotation
Surveillancecamerasystem

䋺Server-client communication
䋨User’s position and annotation are sent䋩䋺P2P communication(User’s position is sent)

Fig. 2. Outline of proposed annotation overlay system

2.2

Outline of the Proposed System

Fig.2 shows an outline of the proposed annotation overlay system. The proposed system consists of a user information management server and two typesof client systems. Components of client systems and the database in the server
are described in detail in the following.
– Client systems
In this study, we assume an environment where there are multiple users ofwearable AR systems which can access network and obtain the position andorientation of the user’s viewpoint. We assume two types of client AR systems; wearable AR system and ﬁxed surveillance camera system. Wearable
AR system can be applied into a communication tool between multiple users.
On the other hand, ﬁxed camera system can be applied into a security camera system.
Client systems request annotation information by transmitting request
packet to the server. After obtaining annotation information, the systems
create AR images.
– User information management server
Personal information and position of users are managed by network-shared
database in the user information management server. The server has a user
information table, an annotation table, and a position table. Tables 1, 2,and 3 show sample lists of information. The server also has an annotation
request table. Table 4 shows a sample list of data. The data in Table 4 meansthat there are ﬁve users in an area, and client A has requested annotation
information of clients B and D.

Personal Information Annotation on Wearable Computer Users

221

Table 1. User information table

ID

PasswordGroupIP adress

User Aaaa

Group A

163.221.䊶䊶䊶User B

BBBBGroup D

163.221.䊶䊶䊶

User CcCcC---163.221.䊶䊶䊶

Table 2. Annotation table

2.3

ID

Annotation ID

Authorization

User A

A001

User B, C

User B

B001

User C

User C

C001

Group A

Updating and Obtaining Annotation Information by
Client-Server Communication

Each client system has speciﬁc ID and communicates with the server in a certain interval. Each wearable AR system automatically sends its speciﬁc ID, IP
address, and position to the server via a wireless network. On the contrary, theﬁxed camera system only sends its ID and IP address because its position is ﬁxedin the real world. Position table is updated by the positions sent from wearable
AR systems. In the case that a user of wearable AR system wants annotation
information, the system sends a request to the server. Annotation request table
is updated by request of client systems.
User information management server generates the P2P transmission list as
shown in Table 5. The P2P transmission list is generated for each system. Destination addresses of client systems are included in the P2P transmission list. TheP2P transmission list is updated by using data of table 1, 2, 3, and 4. Each clientsystem automatically obtains the P2P transmission list in a certain interval andTable 3. Position table

IDXYZ

Updated time

User A

0.2

1.0

1.1

19:00:00

User B

-3.2

15.3

1.2

19:39:11

User C

3.6

8.8

1.0

19:54:53

222

K. Makita, M. Kanbara, and N. Yokoya
Table 4. Annotation request table
Dst(*)ID

R. S(**)A

Client A

Table 5. P2P destination list

BCDE

Dst(*) ID

Dst(*) address

䂾㬍

䂾

㬍

User B

163.221.䊶䊶䊶

㬍㬍䂾User D

163.221.䊶䊶䊶

䂾㬍User E

163.221.䊶䊶䊶

Client B䂾

Client C䂾㬍Client D䂾䂾㬍

Client E䂾㬍㬍㬍
㬍

Dst(*) : Destination
R. S(**) : Request senderrefers to the list in order to transmit its position to other clients. For example,
if the list shown in Table 5 is for client A, the system of client A continuouslytransmits its position to systems of clients B, D, and E.
2.4

Obtaining Dynamic User’s Position

This section describes a procedure of obtaining users’ positions. The procedure
consists of nine processes as shown in Fig.3, which shows an example of the caseclient A requests annotation information of client B. Each process in Fig.3 isdescribed in detail in the following.
1. Request-transmission for users’ positionsFirst, client A requests annotation information of client B. Every client system is always able to transmit a request to the server. The request is transmitted with an user ID as well as a password of request sender(client A).
2. Client authentication
The server checks the user ID and the password to authenticate the validity of the client system. Only if the ID and the password are successfullyauthenticated by the server, we go to the next step.
3. Checking of obtaining annotation authorization
The server checks that client A is given permission for obtaining annotation
information of client B. Only if client A is given permission by client B, wego to the next step.
4. Update of annotation request table
The annotation request table is updated by the server with the request.
Table 4 shows that client A has requested annotation information of clients
B and D.
5. Generation of P2P transmission list
The server generates a P2P transmission list for each client system. Table
5 indicates that client A has to transmit its position to clients B, D, andE by P2P communication. The server generates the P2P transmission list
considering the speciﬁcation of client systems and network environment.

Personal Information Annotation on Wearable Computer Users

223

1䋺Request-transmission

2䋺Client authentication
3䋺Checking of obtaining
annotation authorization
4䋺Update of annotation
request table
5䋺Generation of P2P
transmission list
6䋺Determination of doing
P2P transmission

NY
7䋺Positiontransmission
instruction
8䋺Positiontransmission(P2P)
9䋺Positiontransmission

serverClient B(Position-sender)

Client A(Position-receiver)

Fig. 3. A procedure of obtaining user’s position via P2P communication

6. Determination of doing P2P transmission
The server checks the P2P transmission list for client B. If client A is in thelist, we go to Step 7. Otherwise, we go to Step 9.
7. Position-transmission instruction
In this step, the server orders client B to send its position to client A by P2Pcommunication.
8. Position-transmission by client system
Client B is ordered to transmit its position by receiving a P2P transmission list. In this case, since client A is included in the list, client B starts
transmitting its position to the system of client A by P2P communication.
9. Position-transmission from server
If client A is not included in a P2P transmission list, the server starts transmitting the position of client B to the system of client A.
When client A wants to stop obtaining annotation information, client A requests
the server to stop transmission. This stop-transmission is implemented in a similar way of obtaining user’s position.

224

K. Makita, M. Kanbara, and N. Yokoya

3

Experiments

3.1

Experimental Environment

We have carried out some experiments using the proposed method in our campus
where users of wearable AR systems can use a wireless LAN. Fig.4 illustrates
the experimental environment. We have developed a network-shared databaseof annotation information in a user information management server and set up
one ﬁxed camera system on the ceiling. The ﬁxed camera system consists ofa video camera and a standard PC. The user information management server
and the video camera system use a wired LAN. We have also set up invisible
visual markers on the ceiling for wearable computer users to estimate their position and orientation[7]. The area where user’s wearable system can estimate
its position and orientation is about 15m 3.5m. In experiments, three wearable system users exist in the environment. Users’ positions are estimated bytheir wearable systems and are transmitted to the location server every 1 second. Wearable systems and the video camera system obtained the annotation
information and users’ positions and generated annotation overlay images. Fig.5
illustrates a hardware conﬁguration of the wearable augmented reality system
which is used in these experiments. Table 6 shows hardware conﬁgurations of
the location server and the video camera system.
To generate annotation overlay images, position and orientation of the camera are needed. In experiments, the wearable system estimates the position andorientation of the user’s viewpoint by using the conventional marker-based registration method[7]. On the contrary, the position and orientation of the video
camera is estimated in advance.
LANNetworkshared
database
User information
management
serverAntenna
Invisible visual markers
䋨set up on the ceiling䋩User A

User B

Fixed camera(client)

User C

Users of wearable AR systems (clients)
PC for fixedcamera system
Fig. 4. Experimental environment

Personal Information Annotation on Wearable Computer Users

225

CCD camera
Infraredcamera

Qcam forNotebooks Pro(Logicool)

OriginalNotePC

Display
deviceLet’s Note CF-R4(Matsushita Electric Industrial)

SV-6(MicroOptical)

CPU 1.2GHz, weight 1.2kg,
memory 1GB

Fig. 5. Hardware conﬁguration of wearable AR system
Table 6. Hardware conﬁguration of server and ﬁxed camera system

3.2

Hardwarespecification

ServerPC

CPU䋺Pentium 3 1.2GHz
512MB memory 100Mbps Ethernet

Fixedcamerasystem

PC

CPU䋺Pentium D 3.0GHz
3.25GB memory 1Gbps Ethernet

camera

DCR-TRV900䋨Sony䋩 640㬍480 pixels

Annotation Overlay by Fixed Camera SystemFig.6 shows examples of annotation overlay images generated by the ﬁxed video
camera system. In the augmented images, translucent circles are overlaid on
wearable system users’ infrared cameras to indicate the positions of annotatedusers. The size of the circle is proportional to the distance of the ﬁxed video
camera system to a user. Users’ personal information are also shown on thecorners of the images. Figs.6 (a) through (c) show the annotation overlay images
when the users’ positions are obtained from the user information management
server. Figs.6 (d) through (i) show the annotation overlay images when the users’positions are directly transmitted from users by using P2P communication. FromFigs.6 (a) through (c), it is clear that the translucent circles are misaligned. Thisis because users’ position table in the server is updated at approximately every
second. On the other hand, the translucent circles in Figs.6 (d) through (i) arecorrectly overlaid on users’ infrared cameras. In this case, users’ positions aredirectly transmitted from users at about 10 times per second.

226K. Makita, M. Kanbara, and N. Yokoya(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Fig. 6. Overlay images generated by ﬁxed camera system

3.3Annotation Overlay by Wearable Computer Systems

Fig.7 shows the annotation overlay images generated by a wearable computer
system. In this experiment, three wearable computer users are walking in the area
and their position and orientation are transmitted to one another at about 10times per second. The method of showing annotations is same as the previous

(a)

(b)

(c)

Fig. 7. Overlay images generated by wearable AR system

Personal Information Annotation on Wearable Computer Users227experiment. As shown in Fig.7, the translucent circles are correctly overlaid
on users’ infrared cameras. Through this experiment, we have conﬁrmed thatwearable computer systems can generate appropriate annotation overlay imagesby using P2P communication and show where other users are.

4

SummaryThis paper has proposed a new method for annotating dynamic users of wearable
AR systems to intuitively indicate other users’ positions in user’s view. Users of
this annotation overlay system can see annotations on dynamic users and can correctly recognize where other users are. We have shown the feasibility of the method
through the demonstration with experiments in our campus. In the future, weshould further investigate the optimal management of P2P communication considering the speciﬁcation of wearable computers and wireless network.

Acknowledgments
This research was supported in part by Strategic Information and Communications R&D Promotion Programme (SCOPE) of Ministry of Internal Aﬀairs andCommunications of Japan.

References
1. S. Mann: “Wearable Computing: A First Step Toward Personal Imaging,” IEEE
Computer, Vol.30, No. 2, 2002.
2. S. Feiner, B. MacIntyre, T. Höllerer, and A. WebsterF “A Touring Machine: Prototyping 3D Mobile Augmented Reality Systems for Exploring the Urban Environment”, Proc. 1st Int. Symposium. on Wearable Computers, pp. 74-81, 1997.
3. R. Azuma: “A Survey of Augmented Reality,” Presence, Vol. 6, No. 4, pp. 355-385,
1997.
4. B. H. Thomas, V. Demczuk, W. Piekarski, D. Hepworth, and B. Gunther: “AWearable Computer System with Augmented Reality to Support Terrestrial Navigationh, Proc. Int. Symposium. on Wearable Computers, pp. 168-171, 1998.
5. R. Tenmoku, M. Kanbara, and N. Yokoya: ”Annotating user-viewed objects forwearable AR systems”, Proc. IEEE and ACM Int. Symposium. on Mixed Augmented Reality(ISMAR 05), pp. 192-193, 2005.
6. M. Kourogi, T. Kurata, and K. Sakaue: “A Panorama-based Method of Personal
Positioning and Orientation and Its Real-time Applications for Wearable Computers”, Proc. 5th Int. Symposium. on Wearable Computers, pp. 107-114, 2001.
7. Y. Nakazato, M. Kanbara, and N. Yokoya: ”Wearable augmented reality systemusing invisible visual markers and an IR camera”, Proc. IEEE Int. Symposium. on
Wearable Computers, pp. 198-199, 2005.
8. K. Makita, M. Kanbara, and N. Yokoya: ”Shared annotation database for networked wearable augmented reality system”, Proc. 5th Paciﬁc Rim Conf. on Multimedia, Vol. 3, pp. 499-507, 2004.
9. R. Tenmoku, A. Anabuki, M. Kanbara, and N. Yokoya: ”Annotation of Personal
Information on Wearable Users Using Shared Database”, Proc. Meeting on Image
Recognition and Understanding 2005, pp. 1598-1599, 2005. (in Japanese)