Annotation Overlay with a Wearable Computer Using Augmented Reality
Ryuhei TenmokuÝ, Masayuki Kanbara Ý, Naokazu Yokoya Ýand Haruo Takemura Þ½ Graduate School of Information Science, Nara Institute of Science and Technology

8916-5 Takayama-cho, Ikoma-shi, Nara, 630-0101, JAPANryuhie-t,masay-ka, yokoya@is.aist-nara.ac.jp
Cybermedia Center, Osaka University
takemura@cmc.osaka-u.ac.jp

Abstract
This paper describes an augmented reality system witha wearable computer to overlay an annotation relative tothe real world. To realize an augmented reality system, aposition of user’s viewpoint is needed for acquiring a relationship between the real and virtual coordinate systems.
However, it is difficult that the user’s viewpoint is accurately
measured in both in- and out-door scene. In this paper, awearable augmented reality system which can overlay anannotation on real scene is developed by using RFID tagand tag reader. The system can measure user’s position byRFID tag set up at appointed position where user thinks thatit need an information relative to real scene; for example,
a front of a guide plate and a blanch point. The feasibility of the system has been successfully demonstrated withexperimental.
Keywords: Wearable Computer, Augmented Reality, Annotation Overlay, RFID tag

1. Introduction
Since computers made great strides in these year, a wearable computer which can be loaded is realized[5, 6]. At thesame time, augmented reality which merges real and virtual worlds has received a great deal of attention as a new
method for displaying information or increasing the realityof virtual environments[2]. If an augmented reality can becombined with an wearable computer, many applications,such as navigation guide of museum or a sightseeing areas
can be realized.
To realized an augmented reality system, a position andorientation of user’s viewpoint are needed for acquiring arelationship between the real and virtual coordinate systems. This is called registration problem. However, it isdifficult that to accurately measure the position of user’s
viewpoint with a wearable sensor on which can be used

Figure 1. Appearance of the wearable augmented reality.

both in- and out-door scenes. Magnetic trackers which areusually used for indoor AR or VR system is difficult to beused under wide range of measurement such as outdoor environment. A global positioning system (GPS) is suitablefor outdoor use but cannot be used in indoor environment
where satellites for the system cannot be directly viewed.
In this paper, a wearable augmented reality system which
can overlay an annotation on real scene is developed byusing RFID tag and tag reader. The system detects user’s
approximate position by reading RFID tag placed at appointed position where user needs a information relative toreal scene; for example, a front of guide plate and a branchpoint. The orientation of user’s viewpoint is measured byusing inertial sensor attached to user’s head as shown in
Figure 1. In addition, it estimates drift error which causedorientation error of inertial sensor by using images captured
at user’s viewpoint as shown in Figure 1.
This paper is structured as follow. Section 2 describes
the wearable augmented reality using RFID tag and tag
reader. In Section 3, an experimental result with a prototype

trace him/her or so that statistic information such as how
many people have visited the place can be automaticallycollected. A user may add some additional information of
the place in RFID tag, such as rating of the spot such as rating of the restaurant at the spot. Such information may beshared by another user coming to the same place.
In our prototype system, we place an RFID tag reader on
a users wrist. Assuming that a user will explicitly reach for
RFID tag placed in the experimental environment.

outdoor

RFID tag reader

2.2. Hardware Configurationindoor
RFID tag

Figure 2. An example of position estimation
using RFID tags.

system is describes. In Section 4, an idea to replace RFID
tag and with IRDA marker is described. Finally, Section 5summarized the presented work.

2. Wearable Augmented Reality System
In this section, we describe a proto type system of awearable augmented reality system that can overlay information relative to the real scene. The system is developed
so that the system can be used for navigation guiding system. We assume that a user of the system navigate outdoor
and indoor environment. The system provides an appropriate direction to get to the user’s goal by showing useful
information in his/her view.

2.1. Localization using RFID tag
In order to make the system usable both indoor and outdoor environment, we choose RFID tags that are embeddedin the environment where the user of the system navigate.
An RFID tag is small ID tag of which ID can be read byusing Radio Communication. A reader that reads data outof a tag does not have to make any physical contact with thetag. Tag it self does not have any source of power and canbe embedded in various place. Typically, tag can hold dataof up to several kilobytes.
In our implementation, an RFID tag reader attached tothe user reads out ID information and detect where the useris located in the environment. One possible placement of
RFID tags is to embed them in the floor, road, and pavement and put a reader in a sole of a user’s shoe. This will
give relatively dense position information of the user to thesystem. (Fig 3)
Since RFID tag is a Read-Write device, a user of the system may leave their footprint in tags so that other user may

Figure 3 shows the hardware configuration of the proposed wearable augmented reality system. The system isstructured from three parts; these are sensors, computer anddisplay device.
Sensors The system has following three sensors. The sensors transmit data to the computer through USB connection.
RFID tag reader: The RFID tag reader (OMRON:V720) recognize RFID tag by reading a embedded dataof each RFID tag. In the system, the user’s position isdetermined by reading a RFID tag placed at appointedpoints.
Inertial sensor: The inertial sensor (Intersence: InterTrax2) measures the orientation of user’s view.
CCD camera: The camera (IO data: USB-CCD) attached near user’s viewpoint captures real scene which
is similar to user’s view.
Computer The computer generated an annotation overlay
images by determining the position and orientation ofuser’s viewpoint using the data from the sensors. In
this system, notebook PC (DELL: Inspiron8100 CPU
PentiumIII 1.2GHz, 512Mbytes memory) is used. This
PC is placed user has this in a shoulder bag. The computer has a database of annotation information.
Display device Small display device (MicroOptical: VGA
Clip on display) is attached to a glasses as shown inthe Figure 3. User can see annotation overlay image
by this device.

2.3. Algorithm for Generating Annotation Overlay
ImagesFigure 4 shows the flowchart of the proposed system. A
data of the sensors are fed into a computer. First, user’s position is determined using RFID tag reader. Next, since thedata of an inertial sensor includes a drift error, user’s orientation is estimated by using the data from inertial sensor.

CCD Camera
Real World Scene
ＲＦＩＤ Tag Reader Inertial Sensor

Position

Orientation

Display Device

Computer

Database ofAnnotation
Annotation Information Overlaid Image
Figure 3. Hardware configuration of the wearable augmented reality.

Determination of user’s position
Estimation ofdrift error
Detection of natural features
Feature tracking
Calculation of the drift error

Annotation overlay

Figure 4. Flow diagram of the prototype system.

Finally, annotation overlay image is generated using the position and orientation of user’s viewpoint and is output tothe display device that is worn by the user.
2.3.1. Determination of user’s position
To acquire the user’s position, the user reads RFID tag
which is placed at a guide plate or a blanch point using tagreader as shown in Figure 5. Therefore, location ID em-Figure 5. RFID tag and tag reader.
bedded the RFID tag can be recognized, and the annotation
information is selected from the database.
2.3.2. Estimation of drift error
A measurement data of inertial sensor usually includes
accumulation error that is called drift error. A drift error
is caused by temperature change and other reasons. In oursystem, the drift error is canceled by estimating amount ofdrift from the image captured by camera. The detail of thecalculation process is as follows.
step 1 In the first frame, natural features in a capture image are collected by using Harris’s interest operator [7]which detectes corners or cross-points of edges. Theposition of collected features are recorded.
step 2 In the second and later frame, every features foundin step 1 are tracked using Harris’s operator which isapplied inside of a searching window placed around(a) Main gate(b) Entrance(c) Hallway

Figure 6. Annotation overlay images.
each feature point in previous frames. This step is repeated until there are two consecutive frames wheremajority of positions of features is stable. When sucha condition is satisfied , we assume that the camera did
not move during these two frame time.
step 3 When stable two consecutive frames are found, weassume that a users head did not move in there twoframes. Therefore, we assume that the difference ofdata from inertial sensor between these frames is a constant drift error.
2.3.3. Annotation overlay image generation
This step is repeated when a user stays near the appointed
position where an RFID tag is placed. User’s position isdetermined by using the method in 2.3.1. User’s orientationis measured by the method described in 2.3.2.

3. Experiment
In this section, the experiment using the proposed wearable augmented reality system is described. We develop anavigation system which three tags placed to appointed positions at our school. The experiment is performed to guidea user coming from a nearest bus stop to our laboratory.
Figure 6 shows an example of overlaid annotation during
the experiment. Figure 6 (a) is an overlay at the main gateof our campus. Figure 6 (b) is an example of annotation
given at the entrance hall of our building. Figure 6 (c) is an

IRDA Markers

IRDA Detector
Figure 9. Allocation of multiple IRDA markers.

Figure 7. Prototype of IRDA Marker.
IrD A Port C C D C am era

IRM arker

HM DN otebook
PCIRM arker

IRM arker

Figure 8. An Example of a System using IRDA
Marker.

example of annotation given at the hallway of the third floor
guiding a user to Yokoya laboratory. As is observed fromFigure 6, the composition which is overlaid annotation onreal scene is realized. Through the experiment, what wehave found is that a rough position estimation using RFID
tag is still feasible for using navigation guidance purpose.
This is mainly because rough direction is sufficient to show
a route to a user. As we can also see from the experimental
results, the annotation can be composed at the correct position on the image according to movement of user’s viewpoint.

4. IRDA marker based position detection
In this section, we propose an IRDA marker which canbe used to expand utility of the system we have describedin previous sections. When RFID tags are used to locate
userfs position, userfs action to reach for a tag explicitly isrequired In other words, if a tag is not located in a prox-

imity of a user, the system can not determine where a user
is located in the environment. In order to compensate thisweakness of the system, we have developed an infrared beacon using IRDA protocol (IRDA marker). An IRDA marker
actively emits its ID information using IRDA test frame
packet. Therefore, it is possible to read an ID of IRDA tag
using a photo detector and using IRDA decoding software.
By enhancing infrared light emitting power, ID of a tag
can be received at the point where is several meters away
form the IRDA marker. When multiple tags are located ata userfs proximity, the system can be designed to receive
ID of a tag in specific direction by using directional photodetector. (Typically photo detector used for IRDA communication is unidirectional with 30 degrees to 60 degrees offield of view. FOV can be adjusted by using external opticalparts such as lens etc.)
We have built prototype of IRDA tag with SX micro controller (Ubicomp inc.). Figure 7 shows a photograph of theprototype. Figure 8 shows an example if IRDA Marker usage. IRDA detector is place on top of the HMD and userfs
approcimate location is estimated from ID of the maker.
Merits of using IRDA markers are;1. Even in a case that a user does not knows the potion ofthe maker, userfs position can be estimated as long as
IRDA beacon reaches the user,
2. By placing multiple markers as shown in Figure 9, thesystem may estimate approximate orientation or direction of a user; and3. An ordinary hand held PC or PDA can also be used todownload information from IRDA markers. It may bepossible to build navigation guide system compatible
with different kinds of mobile platform.
On the other hand, demerits of using IRDA markers are;
1. Under the strong sunlight, it is difficult to read ID from
IRDA markers;

2. Electric power source is required, continues transmission of RFID packet is required. This makes difficult
to reduce power consumption
Now, we are developing registration and position localization algorithm using IRDA markers. Possible expansionsof usage of IRDA markers are,
1. Stereo camera may be used to measure distance to theIRDA markers;
2. If more than three markers are seen from stereo camera, more accurate user position can be estimated.

5. Conclusion
This paper describes a augmented reality system that canoverlay annotation at appointed point of the real scene witha wearable computer to give a user information relative tothe real world. We also have developed a prototype wearable augmented reality system using RFID tag and reader
to get a user’s position in the real world. The prototype system can produce annotation overlay images at a campus ofour school. It is shown that the wearable augmented reality system with RFID tag is feasible and has the possibility
building up actual application.
In the future work, we should investigate methodology
for annotation overlay at arbitrary position of a real worldusing a global positioning system (GPS).

References[1] R. Azuma: “A Survey Of Augmented Reality, ” Presence, Vol. 6, No. 4, pp. 355-385, 1997.
[2] S. Julier, M. Lanzagorta, Y. Baillot, L. Rosenblum, S.
Feiner, T. Höller, and S. Sestito: “Information Filtering for Mobile Augmented Reality, ” Proc. Int. Symp.
on Augmented Reality, pp. 3-11, 2000.
[3] M. Billinghurst, S. Weghorst, and T. Furness III :“Wearable Computers for Three Dimensional CSCW,” Proc. Int. Symp. on Wearable Computers, 1997.
[4] S. Feiner, B. MacIntyre, T. Höller, A. Webster : “ATouring Machine: Prototyping 3D Mobile AugmentedReality Systems for Exploring the Urban Environment, ” Proc. Int. Symp. on Wearable Computers, pp74-81, 1997.
[5] K. Satoh, K. Hara, M. Anabuki, H.Yamamoto, andH.Tamura : “TOWNWEAR: An Outdoor wearable
MR system with high-precision registration, ” Proc.
Int. Symp. on Mixed Reality, pp 210-211, 2001.
[6] K. Satoh, M. Anabuki, H.Yamamoto, and H.Tamura: “A Hybrid Registration Method for Outdoor Augmented Reality, ” Proc. Int. Symp. on Augmented Reality, 2001.

[7] C. Harris and M. Steohens: “A Combined Corner andEdge Detector, ” Proc. Alvey Vision Conf., pp. 147151, 1988.