MVA2OO2 IAPR Workshop on Machine Vision Applications, Dec. 11 - 13,2002, Nara- ken New Public Hall, Nara, Japan

A Location Awareness System

13-1 5using Wide-angle Camera and Active IR-Tag
Muneyuki Sakata, Yoshihiro Yasumuro, Masataka Imura,
Yoshitsugu Manabe and Kunihiro Chihara
Graduate School of Information Science
Nara Institute of Science and Technology

Abstract
This paper proposes a new location-awareness system, ALTAIR(Automatic Location Tracking with Active IR-tag) that automatically detects and tracks thelocation of the mobile PC(Persona1 Computer) users.
IR(1nfraRed)-tag is stably detected and distinguishedby IR filtering camera. Combination of the IR-tag andwireless LAN (Local Area Network) enables ALTAIR
to control the IR-tags through the network to perform
stable tracking for multiple users. The position of thedetected IR-tag and its network information (e.g. I Paddress) are associated with each other by controllingthe emission pattern of the IR-tag in turn. ALTAIRstarts to aware the user's position when the user enters
the system area, using DHCP(Dynamic Host Configuration Protocol) service. Our experiments show the effective performance to tracking and recognize the users'positions in an office environment.

1Introduction

It becomes more and more common to use the personal computers (PCs) regardless of the place recently
as the mobile IT(1nformation Technology) devices suchas the cellular phones and PDA (Personal Digital Assistant) spread widely and the service of the wirelessnetwork starts in new locations;cafes, airports, sporting
arenas, public libraries in the town. Therefore, people
can visit anv- place with their mobile PCs with network
connection and may need to refer some information related to the location where the user is standing rightnow.
Our aim is to provide a location service for the mobile PC users in the facilities including the office buildings, the shopping moles and so on. GPS (Global Positioning System) is a powerful location method, but thepublic network services like "Hotspot", for instance,
are implemented within the buildings where out of thereach of GPS in manv cases. For indoors location. there
are also some meth;ds to acquire people's posiAon or
behavior in a room [I]. Most of the methods using color
images from a camera have a limit in the stability ofposition detection or user extraction by using matching processing of a color [2-41. Furthermore, each user
is required to do an explicit log-in operation, for example typing their names and passwords, so that theAnd thesvstem starts to aware and identifv" helshe.
,
method using the combination of the transmitter andreceiver of an electric wave or an ultrasonic wave, it isnecessary to install many sensors, while highly precise
position detection is possible [5,6].We propose a newframework of the location system to locate and identifv the mobile PC users without "log-in o~erationsto
be watched", based on a ubiquitous mobile computing
environment. We introduce a combination of the wideangle camera and the IR(1nfraRed)-tags which emit
IR light. IR-tags are connected to the system throughthe wireless network and the lighting patters of thetags can be controlled by the system. Our approach isto employ the wide-angle camera for both spatial andtemporal process. The spatial process traces the multiple users' positions and the temporal process identifies
each user and associates with his/her position.
In this paper, we describe the details about the process flow in the proposed system, ALTAIR in Section 2.
Our implementation and experimental setup are shownand discuss the performance of the system in Section
3, then conclude in Section 4.

-2
2.1

ALTAIR System
Proposed method

AE-rnail:{rnuneyu-s,yasumuro,irnura,manabe,chihara)
8is.aist-nara.ac.jp
Address: 8916-5 Takayarna-cho, Ikorna, Nara 630-0101
Japan.

Our system, named ALTAIR(Automatic Location
Tracking with Active IR-tag) , has a hierarchical configuration(Figure 1). The Location Server is on thetop level and manages the location information for theall users in the whole area. According to the demands
from the other application systems, information is sent
to an application server which performs location related services. Local Location Manager(LLM) locatesand integrates the users' positions in a section or aroom, where users are carrying the mobile PCs. A
LLM always communicates several Tag Detectors(TD).
A T D captures the image sequence from a wide-angle
camera to detect the lights from the IR-tags. The positions and lighting patterns of the IR-tags are transmitted to the LLM. Several camera systems of TDs areinstalled in each room. LLM manages and renews a listof the users' entry, according to the communications tothe DHCP server and the TDs. To identify the users,
LLM sends massages to a IR-tag to blink in order toassociates the blinking position with corresponding IPaddress.

Figure 2: IR tag(prototype)IR-tag is a small-sized marker with a IR LED whichis controlled by the mobile P C though a serial connection, RS232C. The lighting pattern of the I R LED on
the tag is changed according to the massages that theP C receives via network from the LLM. Our proto-typeIR-tag is about lOcm square size and easily put withthe mobile PCs (Figure 2).

2.3

IR wide-angle camera

Our camera system is constructed with color CCD
camera, optical IR-filter and wide-angle lens. Since IRfilter shuts off visible light and transmits IR(infrared)
light, our camera system can take images of nearinfrared range. And a wide-angle lens, we use a fisheye lens, helps a camera cover large area. Figure 3
shows comparing image of visible range and infraredrange. Both of images contains the user who is 5m
away from the camera. It is capable to extract thelight from the IR-tag without complicated image processing, while visible image needs it or cannot extract
user.

2.4

Tag Detector(TD)Tag Detector(TD) tracks lights of IR-tag from theimage sequence of the IR wide-angle camera and detects that one of the IR-tag is blinking. The T D sends
a message to the LLM to inform the light positions in
the world coordinate and which light blinks. The flow

Figure 3: Captured images with wide-angle camera:
visible range(left),infrared range(right)of detailed processing is as follows.
1. E x t r a c t i n g lights: The T D captures the image of
the IR wide-angle camera. Thresholding and labelingtechnique, the lights of IR-tags are extracted. Outputof this step is the light positions in camera coordinate.
2. T r a c k i n g l i g h t s a n d d e t e c t i n g blinks: The T D
updates a list of lights on each frame. The list contains
a light position in the camera coordinate, a lighting
state flag, and a blinking counter for each detected IRtag. The data of the list is updat.ed by searching thelight positions in the neighbor area between consecutive frames and checking the lighting state("off/on").
The number of the state transition from "off to "on"is counted up as the number of the blink.
3. C a l c u l a t i n g position: The light positions arecaptured and expressed by the camera coordinate system. In order to register the information from several
TDs that have different camera systems, transformation from camera coordinate t o world coordinate is necessary. We define the coordinates as Figure 4 shows.
Using data sheet of the wide-angle lens, 0 (the angle oflens axis and IR tag) can be approximated from 1 (the(A', Y.Z):Camera Position(II4Pop up a user from blinking queue+

Send "request message" to the userI&
Collect the l~ghtslnforrnat-4
For each light pointa certatn

brigin of World Coordinates
posit~onto bllnk~ngqueue

Figure 4: Camera coordinates and World coordinates

If the bllnk~ngcould not be found for a certain tlrnes,
Push agaln the user to bl~nklngqueue.

I

distance from the center in a picture to light points).
Then, vector
is expressed as

Figure 5: Flow of LLM

d = (X, Y, Z)+k(eG sin Ocos a+e; sinesin a+e: cosB)
with basis unit vector of camera coordinate e;, e;, e;and camera position (X, Y, 2 ) . In order to determine the 3-dimensional position of lights, the lights'
height(%)has to be assumed uniformly.
S e n d i n g message: Every several frames, for example
10 frames, T D calculates position of all light points andsends message to the LLM. The message contains positions of lights in the world coordinate and informationof blinking or not.

2.5LLM(Loca1 Location Manager)

A LLM merges and integrates the information sentfrom the TDs that cover an assigned section or a roomand update the user map. The information from multiple TDs are buffered and merged by a LLM. Comparing the buffered data with the latest map, tracking of
the each user's position is performed by nearest neighbor search. A LLM has a queue which contains all theusers entry in the section expressed by their IDS. LLM
sends a request message to a user's IR-tag to blink ina specified manner. The user to sent the message ischosen according to the queue. If the blinking LED is
detected by the TDs, LLM can identify the user andsuccessfully renew the location. Even in the case after
a user is com~letelvlost from the TDs because of visual
occlusion, for example, the location can be recoveredby this "roll call" strategy.
When a new user comes into a system area, theuser's Mobile PC acquires the information for connecting with LLM which manages the area, and with network environment, such as IP address using DHCP service etc. Based on the information acquired, a user's
Mobile PC connect to LLM. LLM receives the connection demand from a user, attaches ID to a newuser, and push the new user's ID to blinking queue.
A new user is distinguished preferentially and tracking
is started.

3

Implementation and Experiments

In order to verify whether a system can realize tracking and distinction of two or more users as we describedabove, we implemented one section of the proposed system and performed an experiment in an office environment.

3.1

Prototype Location System

As shown in Figure 6, we settled target area is 6msquare. Two IR wide-angle cameras(WATEC 202D,640x480 pixels) with fish-eye lenses were installed inthe area so that they face each other. The heightof the cameras are about 2.5m and depression angles
are about 45 deg. In the target area, an LLM andtwo TDs were implemented with the PCs listed in theTable 1. The IP addresses for the PCs are fixed inthis experiment. We performed manual calibration forcamera setup such as positioning, settling the centerand the range of the camera images. IR LEDs (5mm4, X=880nm, beam angle:40 deg.) are used for theTags. The blinking pattern of the IR-tag was settled
as the repetition of 50ms on and 50ms off, or 10Hz.
The height from the floor of IR tag was assumed tobe 1.2m for calculation of the positions. The mobile
PCs are carried so that LED might always turn to thedirection of one of the cameras.

3.2Results

Figure 7 shows the output of the LLM and unprocessed camera image for comparison. Each T D captured images with the speed of 18-22fps and stablecount of the blinking required the image sequence ofabout 13 frames. Updating rate of the users' position

ICPU

I RAM lPentium4 1.8GHz
PentimuII 400MHz

512MH
128M13, ,~i~~etector(2)Mobile PCs

Table 1: Specifications of the PCs

not fast, but "roll call" strategy ensures the identification of the individual IR-tags and we consider thatthe performance of the system is efficient for the office workers location. In the Figure 7, two users walk
across changing the directions. Visual occlusion may
occur and tracking error to exchange the users may
happen but, eventually our system performed correct
tracking and mapping.
On the other hand, some problerns were found in thesyst,em setup. We need to establish systematic method
to calibrate the camera systems otherwise the measurement error of the position was up to l m at maximum.
The system also suffered from frequent occlusion because of the directivity of IR LEDs. We are going toimprove the IR-tag emission, designing the arrangement of the multiple LEDs.
Figure 6: Experimental Space setup

4Conclusion

We have proposed ALTAIR system and implemented the core part to identify and grasp positions ofthe mobile PC users using IR-tag and wide-angle cameras. As the experimental results show the ALTAIR
system is capable of managing information of two or
more users in real-time process and recover from thetracking error because of visual oc:clusion and so on.
Our approach is practical and verifiable. Our future
work is to implement the system to connect more sections to cover larger area such as the whole building
and activate an organized application in an office environment.

Acknowledgments
This research is partly supported by CREST ofJST(Japan Science and Technology) and by Grant-inAid for Scientific Research (B) from Japanese Ministryof Education, Culture, Sports, Science and Technology(No.13558043).

References

Figure 7: Output of LLM(left),Video image for comparison(right)
map on the LLM was 2 fps and identification time forone person was 4 seconds. Even sparse setup using
only two cameras achieved real-time tracking for mult,iple users by the proposed approach. Tracing rate is[I] J . Hightower and G. Borriello, "Location system
for ubiquitous computing," C O M P U T E R , vol. 34,
no. 8, pp. 57-66, August 2001.
[2] J . Krumm, S. Harris, B. Meyers, B. Brumitt, M. Hale,and S. Shafer, "Multi-camera rnulti-person tracking for easy living," Proceedings 3rd IEEE Internationall Workshop Visual Surtleillance. Piscataway,N.J.: IEEE Press, 2000, pp. 3-10.
[3] J . Rekimoto and Y. Ayatsuka, "Cybercode: Designing augmented reality environments with visualtag," Proceedings of Designing Augmented Reality
Enuiromnments ( D A R E 4000), 2000.
[4] S. Thad, S. Mann, B. R.hodes, J . Levine, J . Healey,
D. Kirsch, R. Picard, and A. Pent.land, "Augmented
reality through wearable computing," Presence, vol. 6,
no. 4 , pp. 386-398, August 1997.
[5] A. Harter, A. Hopper, P. Steggles, A. Ward, andP. Webster, "The anat,omy of a context-aware application," Proceedings of the 5th International Conference on Mobile Computing anti Networkzng (MOBICOM'99), Seattle, Washington, USA, August 1999,
pp. 59-68.
[6] R. Want, A . Hopper, V. Falcao, and J . Gibbons,"The active badge locat.ion system," ACM Transactions on Informatzon Systems, 1992, pp. 91-102.