Deciphering Interactions from LED ID Tracking Data
Tetsuya Matsuguchi†, Yasuyuki Sumi†, Kenji Mase†‡†ATR Media Information Science Laboratories, ‡Nagoya University

1. Introduction
With the advent of ubiquitous and wearable
computing technology, we are now capable ofrecording large amounts of data in various forms
simultaneously [1]. However, there is a lack ofsystem that is capable of analyzing an enormous
amount of data quickly to output on the fly something
meaningful and useful.
Audio/Video data areespecially problematic due to the time-consuming
audio and image processing.
Here we report an implementation of a systemthat incorporates identity tags with an infrared lightemitting diode (LED tags) and infrared signal tracking
device (IR tracker) in order to record spatial andtemporal context along with audio/video data [2]. We
have developed a prototype for analyzing humaninteractions from large amount of data collected byubiquitous and wearable computers. This system isdesigned to analyze intricate schemes of humaninteractions [1]. Its fast processing without any image
processing makes it ideal for real-time applications aswell. We discuss our approaches to recognizing
human interactions from IR tracker data.
2. Setup and Demonstration
For the demonstration of our system, five booths
were set up in the sensor room. Each booth had two
sets of ubiquitous sensors that include video cameras
with IR trackers and microphones. LED tags wereattached to each of the posters and displays at thebooths. One presenter at each booth carried a set ofwearable sensors, which included a video camera withan IR tracker, a microphone, and an LED tag. Avisitor could choose to carry the same wearable system
as the presenters or just an LED tag, or nothing at all.
3. Data Analysis
During the two-day demonstration, with theparticipation of 80 users, we were able to collect ~300
hours of video data and over 380,000 tracker data.
Our task is to develop an analysis tool to extract
interactions from the tracker data and bring together
important video data to create a meaningful summaryof their interactions.

Figure 1. Setup of the room.

4. Tracker Data Analysis
Each tracking data consists of the time, the LED
ID, and the coordinate of the object within the view of
the tracker (x, y). Unfortunately, due to somehardware constraints, the detection rate was lower andthe error rate of the tracker was higher than what weexpected. Thus, any single tracking data by itself was
not dependable. It was necessary then to distinguishthe actual tracking data from the erroneously reported
data. To this end, we employed two parameters,
minInterval and maxInterval, to define a CAPTURED
event. A CAPTURED event is at least minInterval inlength, and times between tracking data that make up
an event is less than maxInterval. The idea is that itis less likely to have erroneous data of the same value
repeatedly. The minInterval also allows eliminationof events too short to be significant. The maxIntervalvalue compensates for the low detection rate of thetracker, however, if the maxInterval is too large, moreerroneous data will be utilized to make CAPTURED
events. The larger the minInterval and the smaller themaxInterval are, the fewer the significant events thatwill be recognized.
In order to adjust the parameters, we picked
video clips in which a user is in the view of the video
camera and annotated each frame with the values ofthe parameters that produce a CAPTURED event.
This process allowed us to easily visualize theappropriate values for the parameters. Interestingly,
we found that ubiquitous sensors (stationary) andwearable sensors (in motion) should have different

Figure 2. Basic Interaction Events

values for the parameters. As the result of theanalysis, we decided to use 5 sec for minInterval, 10sec for maxInterval of ubiquitous sensors, and 20 sec
for maxInterval of wearable sensors. In the futurewhen the detection and error rates are improved, theparameters can easily be changed using the sameanalysis.
5. Interaction Events
To simplify the analysis of human interactions,
we defined interaction events to be the building blocksof an interaction. Five basic interaction events wereused:
TALKEDTO,TOGETHERWITH,LOOKEDAT, VISITED, and STAREDAT (Figure 2).
S TALKEDTO/LOOKEDAT events occur
when UserA captures UserB/ObjectB at the same timeas UserB/ObjectB captures UserA. Another words,
two users (or a user and an object) are facing eachother.
S TOGETHERWITH or VISITED eventsoccur when two users, or a user and an object, werecaptured by the same IR tracker in the same timeinterval.
S STAREDAT events are “passive” interaction
events, in which a user is capturing another user or anobject. STAREDAT events are CAPTURED events
that are at least twice the minInterval.
6. Scenes: Clustering Events
A scene is made up of several interactive eventsand is defined based on time. It also has some
temporal dependence due to the clustering of sensors
at each booth. Precisely, all the events that overlap atleast minInterval/2 were clustered together to form ascene. In this prototype, we used simple rules toannotate the interactions in each scene. For example,
if TALKEDWITH event with UserB and VISITED
event at BoothC are clustered together, we inferredthat the user was talking UserB at BoothC. Using ourinteraction analysis, we were able to find ~1800 scenes
with an average length of 150 seconds per scene.
Although the annotation is very primitive at this stage,
this system has been useful and necessary in forming
more complex definitions for the analysis.
7. Video Production
Scene videos were created in a linear time fashionusing only one source of video at a time. In order todecide which video source to use to make up the scene

video, we established a priority list. The priority list
used was based on the following basic rules. Whensomeone is speaking (the volume of the audio isgreater than 0.1 / 1.0), a video source that shows theclose-up view of the speaker is used. If no one that isinvolved in the event is speaking, use ubiquitous video
camera source. In the time intervals where more than
one interaction event have occurred, the followingpriority
wasused:TALKEDWITH>
TOGETHERWITH > LOOKEDAT > VISITED >STAREDAT.
The audio for the scene videos were composed of
all audio sources of users and objects that are part ofeach scene in order to reconstitute conversations andthe atmosphere of the exhibition room.
8. Conclusions
At the two-day demonstration of our system, wewere able to provide users with their summary at theend of their experience on the fly. In the future, wewill develop a system that researchers can query forspecific interactions quickly with simple commands
and provides enough flexibility to suite various needs.
We plan to work together with such researchers toimprove our interaction pattern recognition.
Inaddition to the on-the-fly service, we can also use
audio and image processing to augment the data whenmore computing time is available and when detailed
reports are necessary. We foresee that this system isuseful not only for the study of human interactions, but
automatic cataloging of personal video collection.
Acknowledgement
We would like to thank Sidney Fels for helpful
discussions and assistance.
We also thank themembers of ATR MIS for the making thedemonstration possible. This research was supportedbytheTelecommunications
Advancement
Organization of Japan.
References[1] Sumi Y., Ito S., Matsuguchi T., Fels S., Utsumi A.,
Suzuki N., Nakahara A., Iwasawa S., Kogure K.,
Mase K., Hagita N. Collaborative capturing ofinteractions by multiple sensors, Interaction 2003,2003.
[2] Ito S., Sumi Y., Mase K. Interaction Capturing
Device with an Infrared ID Sensor, Interaction
2003, 2003.