Quick Realization of Function for Detecting Human Activity Eventsby Ultrasonic 3D Tag and Stereo Vision
Yoshifumi Nishida
Digital Human Research Center, AIST
2-41-6, Aomi, Koto, Tokyo 135-0064 Japany.nishida@aist.go.jp

Koji Kitamura
Tokyo University of Science
2641, Yamazaki, Noda-shi, Chiba 278-8510 Japank.kitamura@aist.go.jp

Toshio Hori
Digital Human Research Center, AIST
2-41-6, Aomi, Koto, Tokyo 135-0064 Japant.hori@aist.go.jp

Akifumi Nishitani
Tokyo University of Science
2641, Yamazaki, Noda-shi, Chiba 278-8510 Japan
a.nishitani@aist.go.jp

Takeo Kanade
Digital Human Research Center, AIST
2-41-6, Aomi, Koto, Tokyo 135-0064 Japan
Takeo.Kanade@cs.cmu.edu

Hiroshi Mizoguchi
Tokyo University of Science
2641, Yamazaki, Noda-shi, Chiba 278-8510 Japanhm@rs.noda.tus.ac.jp

Abstract
This paper proposes quick functions for setting up an ultrasonic tagging system for the detection of object handlingas part of daily human activities. The system provides robust measurement of the three-dimensional position of objects and robust detection of registered events in real time,and is supported by fast calibration and event registration
procedures. The system detects object manipulation in realtime using ultrasonic tags attached to objects and a robust position-estimation algorithm known as random sample consensus (RANSAC). Calibration of sensor position isachieved using a calibrating device fitted with only 3 or 4
sensors, and can be used to calibrate an area of arbitrary
size quickly and simply, making the system readily portable.
Registration of target activities is performed by the user using interactive software linked to a stereoscopic camera fitted with ultrasonic tags. The user creates simplified models of the objects, attaches virtual sensors, and associating
the virtual sensors with the target events.

1. Introduction
Information processing services centered around humanactivity in the real world have attracted increased attention
recently [1]. Human-centered applications require the facility to observe and recognize activities as a basis, and thepresent paper describes a method for quickly setting up asystem for robustly detecting daily human activity events in
the real world.
Generally, the problem of human activity recognition canbe formulated as a kind of pattern recognition problem asfollows.
P (Ŵ |Y ) = max
Wi

P (Y |Wi )P (Wi ),P (Y )(1)

where P (Wi |Y ) denotes the posterior probability that themeaning of an observed behavior pattern Y is Wi , P (Y )denotes the probability that a behavior pattern Y will beobserved, P (Wi ) denotes the probability that the behavior meaning Wi will occur, and P (Y |Wi ) denotes the conditional probability. Thus, the problem of human activity
recognition becomes a search for the maximum posterior
probability P (Ŵ |Y ).
There are three problems in realizing and utilizing afunction for recognizing human activity in the real world:
the robust observation of an activity pattern Y , the efficient
recognition of meaning W from the observed pattern, andquick implementation of a system for robustly observing
and efficiently recognizing human activity. Without solving the first problem, equation (1) cannot be formed. Without tackling the second problem, guaranteeing a solution to
the equation within the timeframe demanded by the application is impossible, and without dealing with the third problem, it is difficult to utilize any system in real applications
or research.

Proc. of 2nd IEEE International Conference on Pervasive Computing and Communications, pp. 43-54, 2004

1/12

As a method for efficient recognition of activities, theidea of object-based activity recognition has been proposed [2]. In theory, the behavior of handling objects inan environment such as an office or home can be recognized based on the motion of the objects. However,when applying the method to real environments, it is difficult to even achieve an adequate level of object recognition,
which is the basis of the method.
Separating the problems of object recognition and activity recognition is becoming increasingly realistic with theprogress in pervasive computing technology such as microcomputers, sensors, and wireless network technology.
It has now become possible to resolve object recognition
into the problems of sensorizing objects and tagging the objects with identification codes (IDs), and to address activity
recognition separately through the development of applied
technology.
The present authors have developed a three-dimensional(3D) ultrasonic location and tagging system as a fundamental system for robustly tracking objects. This system
introduces a new approach to tag-based activity recognition. In terms of cost and robustness against environmental noise, the ultrasonic system is superior to other location
techniques such as visual, tactile, and magnetic systems.
Although a number of ultrasonic location systems have already been proposed or commercialized [3, 4, 5], no method
has been developed for improving the robustness, accuracy,
and resolution of position when a person handles objects
tagged with ultrasonic location sensors. Although Shih [4]proposed a robust estimation method by “direct substitution”, the system had difficulty in maintaining the accuracyof position and calculation in real time. The system presented in the present paper has been developed specificallyto address the issue of robustness and accuracy in real time
when a person handle objects tagged with ultrasonic location sensors.
This paper focuses specifically on some supporting toolsthat can be used to set up a system for robustly detecting
daily human activity events quickly. Based on the fundamental ultrasonic tagging system for robustly measuring the
3D position of objects handled by a person, quick methods for the calibration of sensor position and the user registration of target activity events are introduced. The next
section describes the system for detecting human activity
events. Section 3 presents and evaluates the algorithms employed for robustly measuring the 3D position of objects
handled by a person. Section 4 describes the quick calibration method, and section 5 presents the quick registration
scheme for human activity based on stereoscopic video withultrasonic tags and interactive software.

2. System for Detecting Human Activity
EventsThis section describes the system for robustly observing
and efficiently recognizing daily human activities.

2.1. Overview
The configuration of the proposed system is shown in
Fig. 1. The system consists of an ultrasonic tag system, acalibration device, a stereoscopic camera with ultrasonictags, and a host computer. The system provides four key
functions: robust measurement of the 3D position of objects(Fig. 1(A)), quick calibration of sensor position (Fig. 1(B)),
quick registration of target activity events (Fig. 1(C)), androbust detection of registered events in real time (Fig. 1(D)).
The system realizes robust measurement of the 3D position of objects through the use of an ultrasonic tagging system and robust estimation algorithm known as random sample consensus (RANSAC). Quick calibration is achieved
through the use of a calibration device with three or more
ultrasonic transmitters, making the system portable. Quick
registration of target activity events is realized through theuse of a stereoscopic camera with ultrasonic tags and interactive software for creating a 3D shape model. In registration, simplified models of objects are defined and assigned
virtual sensors, and the virtual sensors are then associatedwith target events.

2.2. System setup
The steps required to establish the basic detection system are outlined below.
1. Install ultrasonic receivers in the target environment.
2. Calculate the 3D position of installed ultrasonic receivers using a calibration device (see section 4 for details).
3. Register target activity events using a stereoscopic
camera with ultrasonic tags and interactive software(see section 5 for details).
4. Detect the registered target events using the ultrasonictags and virtual sensors.

2.3. Advantages of the Proposed System
The advantages of the proposed system can be summarized as follows.
• Utilization of user knowledge Users know the target
activity to be detected, and the system can make fulluse of this knowledge by registering target events interactively.

Proc. of 2nd IEEE International Conference on Pervasive Computing and Communications, pp. 43-54, 20042/12

䎦 䎴䏘䏌䏆䏎䎃䏕䏈䏊䏌䏖䏗䏕䏄䏗䏌䏒䏑䎃䏒䏉䎃䏗䏄䏕䏊䏈䏗䎃䏄䏆䏗䏌䏙䏌䏗䏜䎃䏈䏙䏈䏑䏗䎋䏄䎌䎦䏕䏈䏄䏗䏈䎃䏖䏌䏐䏓䏏䏌䏉䏌䏈䏇䎃䎖䎧䎃䏖䏋䏄䏓䏈䎃䎃䎃䫹䎃䎃䏐䏒䏇䏈䏏䎃䏅䏜䎃䏘䏖䏌䏑䏊䎃䎃䎃䎃䎃䎃䎃䎃䎃䎃䎃䎃䏖䏗䏈䏕䏈䏒䏖䏆䏒䏓䏌䏆䎃䏆䏄䏐䏈䏕䏄䎳䏋䏜䏖䏌䏆䏄䏏䎃䏒䏅䏍䏈䏆䏗䎃䏚䏌䏗䏋䎃䏘䏏䏗䏕䏄䏖䏒䏑䏌䏆䎃䎖䎧䎃䏗䏄䏊

䎸䏏䏗䏕䏄䏖䏒䏑䏌䏆䎃䎖䎧䎃䏗䏄䏊䎋䏅䎌䎦䏕䏈䏄䏗䏈䎃䏉䏘䏑䏆䏗䏌䏒䏑䎃䏐䏒䏇䏈䏏䎃䏒䏉䎃䎃䫹䎃䏓䏋䏜䏖䏌䏆䏄䏏䎃䏒䏅䏍䏈䏆䏗䎃䏅䏄䏖䏈䏇䎃䏒䏑䫹䎃䏙䏌䏕䏗䏘䏄䏏䎃䏖䏈䏑䏖䏒䏕䎶䏗䏈䏕䏈䏒䏖䏆䏒䏓䏌䏆䎃䏆䏄䏐䏈䏕䏄䎃䏖䏜䏖䏗䏈䏐䎤䎸䏏䏗䏕䏄䏖䏒䏑䏌䏆䎃䎖䎧䎃䏗䏄䏊䎃䏖䏜䏖䏗䏈䏐䎵䏒䏅䏘䏖䏗䎃䏐䏈䏄䏖䏘䏕䏐䏈䏑䏗䎃䏒䏉䎃䏒䏅䏍䏈䏆䏗䏖䎊䎃䏓䏒䏖䏌䏗䏌䏒䏑䏖䎃䎸䏏䏗䏕䏄䏖䏒䏑䏌䏆䎃䏕䏈䏆䏈䏌䏙䏈䏕䏖䎦䏕䏈䏄䏗䏈䏇䎃䏙䏌䏕䏗䏘䏄䏏䎃䏖䏈䏑䏖䏒䏕

䎦䏕䏈䏄䏗䏈䏇䎃䏗䏄䏅䏏䏈䎃䏒䏉䎃䏕䏈䏊䏌䏖䏗䏈䏕䏈䏇䎃䏄䏆䏗䏌䏙䏌䏗䏜䎃䎋䏇䎌䎵䏈䏆䏒䏊䏑䏌䏝䏌䏑䏊䎃䏄䏆䏗䏌䏙䏌䏗䏜䎃䎃䎃䎃䎃䏌䏑䎃䏕䏈䏄䏏䎃䏗䏌䏐䏈䎸䏏䏗䏕䏄䏖䏒䏑䏌䏆䎃䎖䎧䎃䏗䏄䏊䎬䏑䏓䏘䏗䎃䎖䎧䎃䏇䏄䏗䏄䏄䏑䏇䎃䎬䎧

䎦䏄䏏䏌䏅䏕䏄䏗䏌䏒䏑䎃䏇䏈䏙䏌䏆䏈䎴䏘䏌䏆䏎䎃䏆䏄䏏䏌䏅䏕䏄䏗䏌䏒䏑䎃䏉䏒䏕䎃䏘䏏䏗䏕䏄䏖䏒䏑䏌䏆䎃䎖䎧䎃䏗䏄䏊䎃䏖䏜䏖䏗䏈䏐

䎋䏆䎌䎤䏖䏖䏒䏆䏌䏄䏗䏈䎃䏒䏘䏗䏓䏘䏗䎃䏒䏉䎃䎃䎃䎃䏙䏌䏕䏗䏘䏄䏏䎃䏖䏈䏑䏖䏒䏕䎃䏚䏌䏗䏋䎃䎃䎃䎃䎃䏗䏄䏕䏊䏈䏗䎃䏄䏆䏗䏌䏙䏌䏗䏜䎃䏈䏙䏈䏑䏗䎤䏑䏄䏏䏜䏝䏈䎃䏅䏜䎃䏙䏌䏕䏗䏘䏄䏏䎃䏖䏈䏑䏖䏒䏕䎧䏈䏗䏈䏆䏗
䏗䏄䏕䏊䏈䏗䎃䏄䏆䏗䏌䏙䏌䏗䏜䎶䏗䏈䏕䏈䏒䏖䏆䏒䏓䏌䏆䎃䏆䏄䏐䏈䏕䏄䏚䏌䏗䏋䎃䏘䏏䏗䏕䏄䏖䏒䏑䏌䏆䎃䎖䎧䎃䏗䏄䏊䎥䎧䏈䏗䏈䏆䏗䏌䏒䏑䎃䏒䏉䎃䏕䏈䏊䏌䏖䏗䏈䏕䏈䏇䎃䎵䏈䏄䏏䎃䎺䏒䏕䏏䏇䎧 䏄䏆䏗䏌䏙䏌䏗䏜䎃䏌䏑䎃䏕䏈䏄䏏䎃䏗䏌䏐䏈䎦䏒䏐䏓䏘䏗䏈䏕

Figure 1. Configuration of system for detecting human activity events• Efficient processing It is possible to minimize the system based on the number of ultrasonic receivers andtarget events considering also the installation environment and the activity events to be detected.
• Inexpensive It is possible to utilize inexpensive sensors such as ultrasonic tags (about $45 per sensor and$200 per tag). The stereoscopic camera (about $200)
is also inexpensive.
• Robustness The low cost of the sensors means that the
number of ultrasonic receivers can be readily increased
to ensure robust location ability (see section 3 for details).
• Ease of improvement The function for quick registration of target events allows the constructed system tobe easily improved by trial and error.

3. Robust Observation of Human Activity in
Handling Objects
3.1. Configuration of Ultrasonic Tagging System
Figure 2 shows the configuration of the ultrasonic tagging system. The system consists of an ultrasonic reception section, an ultrasonic transmission section, a time-offlight measurement section, a network and personal computer. The ultrasonic reception section receives ultrasonicpulses emitted from the ultrasonic transmitter and amplifies
the received signal. The time-of-flight measurement section
records the travel time of the signal from transmission to
reception. The network synchronizes the system and collects time-of-flight data from the ultrasonic reception section. The positions of objects are calculated based on threeor more time-of-flight results, which are obtained at a sampling frequency of 50 Hz.
The ultrasonic tagging system calculates the 3D position of an object by trilateration using three distance mea-

Proc. of 2nd IEEE International Conference on Pervasive Computing and Communications, pp. 43-54, 2004

3/12

PC
RS232(9.6k to 921.6kbps)
Collect data

SYNC signal
data

PIC

Measure time of flight

RS485(921.6kbps,1km)

࡮࡮࡮ < 256 devices
Radio transmitter& SYNC generator

PIC

PIC

PIC

VHF 314.9MHz
wireless RS232

Sen

࡮࡮࡮< 4096 devices

dS
C&
YN
ID

Ultrasonic receiver & amplifierRadio Ultrasonicreceiver transmitter
40kHz
PIC

Figure 3. Experimental living space

Battery

Ultrasonic 3D tag

28x20x17mm

Figure 2. Configuration of ultrasonic tagging
systemsurements. Two methods of multilateration are investigatedfor use with the proposed system: multilateration based ona least-squares method using redundant distance data, andmultilateration based on robust estimation.
The room used to conduct the experiments is shown in
Fig. 3. The room was 3.5 × 3.5 × 2.7 m in size, and was fitted with 307 ultrasonic receivers embedded in the wall andceiling. Tags were attached to various objects, including a
cup and a stapler as shown in Fig. 4. Some objects were fitted with two transmitters. The purpose of the experimentalroom was to clarify the effect of the use of redundant sensors and it should be noted that the algorithms presented in
the next section function efficiently with much fewer thanthe 300 sensors fitted in the experimental room in this case.

3.2. Multilateration method 1: linearization of aminimization problem
Trilateration or multilateration algorithms have been
proposed in the field of aerospace [6, 7]. This paper
presents some multilateration algorithms that are applicable to a more general case to deal with ultrasonic receivers
placed in arbitrary locations. Using distance data li , lj andthe receiver positions (xi , yi , zi ), (xj , yj , zj ), the following spherical equations can be obtained for the possible
position of the target.
(xi − x)2 + (yi − y)2 + (zi − z)2(xj − x)2 + (yj − y)2 + (zj − z)2

=
=

li2 ,
lj2 .

Small type(28x20x17mm)

Tiny type(12x12x20mm)

(2)
(3)

Long life
battery type(65x44x20mm)

Tag

Figure 4. Ultrasonic tags and example of attaching tags to objects

By subtracting Eq. (3) from Eq. (2), we obtain an equation for intersecting planes between the spheres, as shownin Fig. 5.

2(xj − xi )x + 2(yj − yi )y + 2(zj − zi )z =li2 − lj2 − x2i − yi2 − zi2 + x2j + yj2 + zj2(4)

Inputting pairs of (i, j) into the above equation gives thefollowing linear simultaneous equations.

Proc. of 2nd IEEE International Conference on Pervasive Computing and Communications, pp. 43-54, 2004

4/12

P=(x,y,z): intersection point

Simultaneous equations of planeon which an intersection line between the two spheres2( x j − xi ) x + 2( y j − yi ) y + 2( z j − zi ) z= li2 + l 2j − xi2 − yi2 − zi2 + x 2j + y 2j + z 2j

⎛ n1 ⎞⎜ ⎟⎜n ⎟A=⎜ 2⎟

⎜ ⎟
⎜n ⎟
⎝ M⎠

l2
l1
l3

n1

n2

⎛ x⎞
⎜ ⎟
x = ⎜ y⎟⎜z⎟
⎝ ⎠M-by-3 Matrix

Ax = bα: intersection planeniRank(A) ?

Figure 5. Planes of intersection between
spheres for position estimation

Rank(A)=1

Rank(A)=2

Rank(A)=3(Collinear)(Coplanar)(Non-coplanar)nCenter of sphere

AP
where

P

=
=

B, ⎞
⎛
x
⎝y ⎠ ,
z

(5)

B

(6)

The position (x̂, ŷ, ẑ) can then be calculated by a leastsquares method as follows.
= (AT A)−1 AT B.

Candidateof solutionx0nSolution is determinate.
A single solution exists.

A single position can be fixed.

x = ( At A) −1 At b

Pi = ( xi , yi , zi )⎞2(x0 − x1 ) 2(y0 − y1 ) 2(z0 − z1 )= ⎝2(x0 − x2 ) 2(y0 − y2 ) 2(z0 − z2 )⎠ , (7)2(x − x ) 2(y0 − y3 ) 2(z0 − z3 )⎞⎛2 0 2 3 2
l1 − l0 − x1 − y12 − z12 + x20 + y02 + z02⎜l22 − l02 − x22 − y22 − z22 + x20 + y02 + z02 ⎟⎟⎜= ⎜l2 − l2 − x2 − y 2 − z 2 + x2 + y 2 + z 2 ⎟ .(8)
0
3
3
3
0
0
0⎠
⎝3
..
.

P

Solution is indeterminate.
At most two solutions exist.

Candidateof solution⎛

A

Solution is indeterminate.
Infinite solutions exist.

(9)This method minimizes the square of the distance between the planes expressed by Eq. (4) and the estimated position. The algorithm is described in detail in Fig. 6. In actual usage, the rank of matrix A must be considered.

3.3. Multilateration method 2: Robust estimation
by RANSAC
Data sampled by the ultrasonic tagging system is easily
contaminated by outliers due to reflections. Method 1 above
is unable to estimate the 3D position with high accuracy ifsampled data includes outliers deviating from a normal distribution. In the field of computer vision, robust estimation

Candidateof solution
Candidateof solution

If there are conditions to selectone solution from the two,a single position can be fixed.
1) Solve the minimum norm solution

x 0 = A+ b

A+ is Moore-Penrose inverse matrix.

A position cannot be fixed.

x 0 is the minimum norm solution.
2) Solve two positions using
the equations below.

x = tn + x 0( x − x i ) 2 + ( y − y i ) 2 + ( z − z i ) 2 = li2

n is a base vector of nullspace of A.
3) Select a single solution using
conditions such as(x − p co n d) ⋅ n co n d > 0

Figure 6. Algorithm for estimating 3D position by a least-squares method considering
the rank of A

methods that are effective for sampled data including outliers have already been developed. In this work, the random
sample consensus (RANSAC) [8, 9] estimator is adopted toeliminate the undesirable effects of outliers. The procedure
is as follows.
1. Randomly select three distances measured by three receivers (jth trial).
2. Calculate the position (xcj , ycj , zcj ) by trilateration.
3. Calculate the error εcji for all receivers (i = 0, 1, ..., n)by Eq. (10), and find the median εmj of εcji .

Proc. of 2nd IEEE International Conference on Pervasive Computing and Communications, pp. 43-54, 2004

5/12

εcji

=

¬
¬
Ô
¬
¬
(xi − xmj )2 + (yi − ymj )2 + (zi − zmj )2 ¬
¬li −

εmj
(x̂, ŷ, ẑ)==

medj |εcji |min εmj

Density

4. Repeat steps 1 to 3 as necessary to find the combination of measurements giving the minimum error, andadopt the corresponding 3D position.
0.04
0.02
0
30

(10)
(11)
(12)

m]
y[m
-30

150

m]
m
[
x

-30

0.6
0.5

100
50
0

0.4
0.3
0.2

0

10

20

30

40

50

60
0.1

The number of sensors

0250

Deviation [mm]

30

200

Density

Deviation [mm]

250

30

-30

-20

-10

0

10

20

30

Z [mm]

200

Figure 8. Resolution in the x and y directions(upper) and z direction (lower) (grid size: 2 ×
2 mm)

150
100
50
0
0

10

20

30

40

50

60

The number of sensors
Figure 7. Relationship between resolution
and the number of sensors for the leastsquares method (upper) and RANSAC (lower)

added to the distance data for the selected receiver to represent an outlier. Each point was derived from 30 estimations of the position. The 5 lines in the figures represent theestimations for 5 different locations of the transmitter. The
resolution increases with the number of receivers, and the
RANSAC method provides a more stable estimation withhigher resolution compared to the least-squares method.
The resolution in the x, y, and z directions is illustrated
in Fig. 8, which shows the probability density distribution
for 1000 estimations using RANSAC. The resolution in thex and y directions is about 15 mm, while that in the z direction is about 5 mm.

3.4. Resolution
3.5. Positioning accuracy
Figure 7 shows the relationship between the number ofreceivers and the deviation of the estimated position for 4,
6, 9, 24, and 48 receivers in the ceiling. To compare the effect of the RANSAC method and that of the least-squares
method, one receiver is selected randomly and 500 mm is

Figure 9 shows the relationship between the number ofreceivers and the error of the estimated position for 4, 6, 9,
24, and 48 receivers. The error is taken as the distance fromthe position measured by a visual motion capture system.

Proc. of 2nd IEEE International Conference on Pervasive Computing and Communications, pp. 43-54, 2004

6/12

300

䎖䎘䎓

䎓
䎳䏒䏖䏌䏗䏌䏒䏑䎃䏒䏉䎃䏖䏈䏑䏖䏒䏕250

䏝䎠䎕䎓䎓䎓
䏝䎠䎔䎘䎓䎓
䏝䎠䎔䎓䎓䎓
䏝䎠䎘䎓䎓䎕䎚䎓䎓

Error [mm]

350

200
150

䎗䎓䎓䎓

100
50
10

20

30

40

50

60
䏝䎃䎾䏐䏐䏀

The number of sensors
350䎖䎖䎖䎾䏐䏐䏀

0

䎕䎓䎓䎓
䎔䎘䎓䎓
䎔䎓䎓䎓
䎖䎓䎓䎓

䎘䎓䎓
䎓

䎔䎓䎓䎓
䎕䎓䎓䎓

䏛䎃䎾䏐
䏐

䎖䎘䎓䎓

150
100

䎓

䏀

䎗䎓䎓䎓

䎖䎘䎓䎓

䎖䎓䎓䎓

䎖䎓䎓䎓

䎕䎘䎓䎓

䎕䎘䎓䎓

䎕䎓䎓䎓

䎕䎓䎓䎓

䎼䎃䎾䏐䏐䏀

200

䎖䎓䎓䎓

䏐䏐
䏀

䎔䎓䎓䎓

250

䎖䎖䎖䎾䏐䏐䏀

䎕䎓䎓䎓
䎓

䏜 䎃䎾

300

䎼䎃䎾䏐䏐䏀

Error [mm]䎋䏅䎌䎃䎳䏒䏖䏌䏗䏌䏒䏑䏖䎃䏒䏉䎃䎗䎛䎃䏖䏈䏑䏖䏒䏕䏖䎋䏄䎌䎃䎰䏈䏄䏖䏘䏕䏈䏇䎃䎔䎗䎓䎓䎃䏓䏒䏌䏑䏗䏖

0

䎔䎘䎓䎓
䎔䎓䎓䎓

50

䎔䎓䎓䎓

䎘䎓䎓

0

䎓

10

20

30

40

50

䎘䎓䎓

䎓

䎘䎓䎓

䎔䎓䎓䎓

䎔䎘䎓䎓

60

3.6. Robustness to occlusion
As in other measuring techniques such as vision-based
methods, it is necessary to increase the number of sensors
to solve the problem of sensor occlusion, where the line of䎖䎓䎓䎓䎼䎃䎾䏐䏐䏀

䎖䎘䎓䎓

䎗䎓䎓䎓

䎗䎘䎓䎓

䎓

䎘䎓䎓

䎔䎓䎓䎓

䎔䎘䎓䎓

䎖䎓䎓䎓

䎖䎓䎓䎓

䎕䎘䎓䎓

䎕䎘䎓䎓

䎕䎓䎓䎓

䎕䎓䎓䎓

䎔䎘䎓䎓

䎕䎓䎓䎓

䎕䎘䎓䎓

䎖䎓䎓䎓

䎻䎃䎾䏐䏐䏀
䎖䎘䎓䎓

䎖䎘䎓䎓

䎗䎓䎓䎓

䎗䎘䎓䎓

䎋䎽䎠䎔䎓䎓䎓䎾䏐䏐䏀䎌

䎔䎘䎓䎓
䎔䎓䎓䎓

䎔䎓䎓䎓

䎘䎓䎓

䎘䎓䎓
䎓

䎓

䎋䎽䎠䎘䎓䎓䎾䏐䏐䏀䎌

䎖䎘䎓䎓

䎓

䎘䎓䎓

䎔䎓䎓䎓

䎔䎘䎓䎓

䎕䎓䎓䎓

䎕䎘䎓䎓

䎻䎃䎾䏐䏐䏀

䎕䎓

One receiver is selected randomly and 500 mm is added tothe distance data for the selected receiver as an outlier. Eachpoint was derived from 30 estimations of the position. The
5 lines in the figures represent the estimation for 5 different locations of the transmitter. The error decreases as thenumber of receivers is increased, and the RANSAC method
is appreciably more accurate with fewer receivers. It is considered that the least-squares method is easily affected byoutliers, whereas the RANSAC method is not.
Figure 10 shows the 3D distribution of error for 1400
measured positions in the room. The figures show that theerror is lowest (20–80 mm) immediately below the 48 receivers in the ceiling, increasing toward the edges of theroom.
These results demonstrate that it is possible to improve
the accuracy and resolution by increasing the number of receivers, and that the undesirable effect of outliers can be
mitigated through the use of RANSAC estimation.

䎕䎘䎓䎓䎻䎃䎾䏐䏐䏀

The number of sensors
Figure 9. Relationship between positioning
accuracy and the number of receivers for the
least-squares method (upper) and RANSAC(lower)䎕䎓䎓䎓䎼䎃䎾䏐䏐䏀

0

䎔䎘䎓䎓

䎗䎓

䎖䎓䎓䎓

䎖䎘䎓䎓

䎗䎓䎓䎓

䎗䎘䎓䎓

䎓

䎓

䎘䎓䎓

䎔䎓䎓䎓

䎙䎓

䎛䎓

䎔䎓䎓

䎔䎘䎓䎓

䎕䎓䎓䎓

䎕䎘䎓䎓

䎖䎓䎓䎓

䎻䎃䎾䏐䏐䏀

䎋䎽䎠䎔䎘䎓䎓䎾䏐䏐䏀䎌

䎔䎕䎓

䎔䎗䎓

䎔䎙䎓

䎖䎘䎓䎓

䎗䎓䎓䎓

䎗䎘䎓䎓

䎋䎽䎠䎕䎓䎓䎓䎾䏐䏐䏀䎌

䎔䎛䎓

䎕䎓䎓

䎨䏕䏕䏒䏕䎃䎾䏐䏐䏀

Figure 10. Three-dimensional distribution oferror in the experimental roomsight to the target object is obstructed by other objects suchas walls or room occupants. In the present tagging system,
the problem of occlusion occurs often when a person moves
or operates an object. These situations give rise to two separate problems; a decrease in the number of usable sensors
for the target, and an increase in reflections due to obstruction and movement. As one of the most typical situations
where occlusion occurs, this section focuses on occlusion
due to a hand.
Figure 11 shows how the error increases and the numberof usable sensor decreases as a hand approaches an object
fitted with an ultrasonic transmitter. The results are shownfor both the least-squares and RANSAC methods. Although
the error increases significantly by both methods when thehand approaches the object, the RANSAC method is muchless affected than the least-squares method. This demonstrates that the proportion of outliers increases when occlusion occurs, and that RANSAC is more robust in this situa-

Proc. of 2nd IEEE International Conference on Pervasive Computing and Communications, pp. 43-54, 2004

7/12

tion because it can mitigate the effect of such outliers.

cell phone

200
180

Least Square
RANSAC160

The number of valid sensors

40

140
30

120
100

20

80
60

10

40
20

0

0
10

15

20

25

30

35

40

45

The number of valid sensors

Error [mm]

cup

chair

trash

documents
stapler

50

Distance from object [cm]Figure 12. Measured trajectory for movementof several objects one after anotherFigure 11. Accuracy of the ultrasonic tagging
system when occlusion due to a hand occurs

3.7. Real-time position measurement
Figure 12 shows the measured trajectory for a person
moving a cup to a chair, the floor, and a desk. The figure
demonstrates that the system can robustly measure the positions of the objects in most places of the room regardlessof occlusion by a hand or body.
In the current system, the sampling frequency is about
50 Hz. This frequency decreases to 50/n Hz when n objects are being monitored. However, it is possible to maintain a high sampling frequency by selecting which transmitters to track dynamically. For example, a transmitter can be
attached to a person’s wrist, and the system can select transmitters in the vicinity of the wrist to be tracked, thereby reducing the number of transmitters that need to be tracked
at one time and maintaining the highest sampling frequency
possible. Figure 13 shows the measured trajectory in a dynamic selection mode. The red sphere in the figure shows
the position of the hand.

Figure 13. Dynamic selection of transmitterswhich the receivers are fitted, making it difficult to calibratelarge volumes. Here, a calibration method that requires a
relatively small number of transmitters and which is independent of room size is presented.

Receivers

Pr

Measurement

4. Quick Calibration Method for Ultrasonic
3D Tag System
4.1. Measurement and calibration
The ultrasonic tagging system involves calibration of receiver position and measurement of transmitter position, as
shown in Fig. 14. Both problems are essentially the same.
As described in the previous section, the robustness of theultrasonic tagging system can be improved by increasing
the number of ultrasonic receivers. However, simple calibration requires a calibration device with a large number oftransmitters and which is equivalent in size to the space inL

Calibration| Pri − Pt j |= Li,j

Pt
Transmitters

Figure 14. Calibration and measurement

4.2. Quick Calibration Method
The procedure for quick calibration is as follows.

Proc. of 2nd IEEE International Conference on Pervasive Computing and Communications, pp. 43-54, 2004

8/12

1. Move the calibration device arbitrarily to multiple positions (A, B, and C in Fig. 15).
2. Calculate positions of receivers in a local coordinate
system, with local origin set at the position of the calibration system.

Move calibration device arbitrarily%CNEWNCVGF&RQUKVKQPUQHTGEGKXGTU
KPCNQECNEQQTFKPCVGU[UVGO

Calculate 3D positions of receivers
in local coordinate system

2000

2000

1500

1500

1000

1000

500

500

0

0
1500
1500

1000
1000

500
500

3. Select receivers for which the positions can be calculated in more than two positions of the calibration system. The pink points in Fig. 15 denote the points thatcan be calculated in more than two positions.

00Select receivers that could be calculated
in more than two places

-500

-500

500

0

2000
1500

%CNEWNCVGFRQUKVKQPU
QHVTCPUOKVVGTU

1000
500
0
5000

Calculate the positions of calibration
device in world coordinate system

4000
3000
2000
1000
0
0

4. Select a world coordinate system from the local coordinate systems and calculate the positions of the calibration device in the global coordinate system using
the receivers selected in Step 3. Then calculate transformation matrices (M1 andM2 in Fig. 15).

500
-1000

0

800
600
400
200

2000
1500

Calculate all the positions of receivers

1000
5000
4000
3000
2000
1000
0

800
600
400
200
0

Figure 16. Flow of quick calibration method

5. Calculate receiver positions using the receiver position
calculated in Step 2 and the transformation matrices
calculated in Step 4.
Figure 16 shows the flow of the calibration method.
Calculated positions
Calculated positionsof calibration device(Transmitter)

80 receivers

[mm]
2000

Receivers

1500
1000

z

500
0

x
4000[mm]yPlace B
Place CCalibration device(Transmitters)

Place A

3500
3000[mm] 0

2500
2000

1

1500

1000

First position ofcalibration device(Transmitter)

M1

1000
500
2000

0
-500

M2

Figure 15. Quick calibration method

4.3. Experimental Results
Figure 17 shows the experimental results. A total of
80 receivers were calibrated using 4 transmitters. The experimental results demonstrate that the proposed calibration method is very effective (maximum positioning error:
103 mm) and can be used in spaces much larger than the calibration system.
The proposed method makes it possible to make the ultrasonic tagging system portable. Figure reffig:calibrationdevice-pic.eps shows a portable-type ultrasonic tagging system consisting of a case, tags, receivers, and a calibration
device. The portable system allows human activities to bemeasured in the area where the activities occur with quick
installation and calibration.

Figure 17. Experimental results for proposed
calibration method

5. Quick Registration of Human Activity Events
Quick registration of target human activity events is performed using a stereoscopic camera fitted with ultrasonictags as shown in Fig. 19 in combination with interactive
software. The operation involves simplification of the 3D
object and the physical phenomena relating to target events.
The software abstracts the shapes of objects in the realworld as simple two-dimensional shapes such as lines, circles, or polygons. In order to describe the real-world events
when a person handles the objects, the software abstracts
the function of objects as simple phenomena such as touch,detouch, or rotation. The software adopts the concept of virtual sensors and effectors to allow the user to define theProc. of 2nd IEEE International Conference on Pervasive Computing and Communications, pp. 43-54, 2004

9/12

Ultrasonic sensors

Ultrasonic 3D tag

Portable case
Calibration device
built in sections

Stereoscopic camera
Figure 18. Portable ultrasonic taggin system
function of the objects easily through simple mouse operations.
For example, to define the activity ”put a cup on thedesk”, the user simplifies the cup and the desk as simple
two-dimensional models of a circle and rectangle using thephoto-modeling function of the software. Using a function
for editing virtual sensors, the user then adds a ”touch” virtual sensor to the model of the desk, and adds a ”bar” effector to the model of the cup.

Figure 19. Ultravision (stereoscopic camera
with ultrasonic tags) for creating simplified
3D shape models

5.1. Software for Quick Registration of Human
Activity Events(a) Creating simplified models of objects Figure 21 shows
examples of simplified models of objects such as a tissue, acup, a desk and a stapler. The cup is expressed as a circle
and the desk as a rectangle. The simplification is performed
using a stereoscopic camera fitted with ultrasonic tags incombination with photo-modeling software. The camera isfitted with multiple ultrasonic tags, allowing the system totrack its position and posture. Therefore, it is possible tomove the camera freely while the user creates simplified
models, and the system can integrate the created models
into the global coordinate system.
(b) Creating models of object functions using virtual sensors/effectors The software creates a model of an object’sfunction by attaching virtual sensors and effectors to themodel created in step (a). Virtual sensors and effectors areprepared in advance by the software and function as sensors and effects affecting the sensors. The current system
has an ”angle sensor” for detecting rotation, a ”bar effector” to represent touch, and a ”touch sensor” for detecting
touch. In the right part of Fig. 22, red indicate a virtual bar+OCIGUHTQOUVGTGQUEQRKEECOGTC

5RGEKHKPIEJCTCEVGTKUVKERQKPVU

Figure 20. Photo-modeling using the stereoscopic camera system

Proc. of 2nd IEEE International Conference on Pervasive Computing and Communications, pp. 43-54, 2004

10/12

#EVKXKV['XGPVGZ ;QWRWVVJGEWRQPVJGYJKVGVCDNG

Figure 21. Simplified shape models
$CT'HHGEVQTVQWEJGU6QWEJ5GPUQT

effector, and green indicates a virtual touch sensor. Using
simple mouse operations, it is possible to add virtual sensors/effectors to the 3D shape model.

$CT'HHGEVQT

6QWEJ5GPUQT

$CT'HHGEVQTVWTPUTGF
YJGPKVKUVQWEJKPIYKVJ6QWEJ5GPUQT

Figure 22. Model of object function using virtual sensors/effectors(c) Associating output of object function model with activity event Human activity can be described using the outputof the virtual sensors created in Step (b). In Fig. 23, red indicates that the cup touches the desk, and blue indicates thatthe cup does not. By creating a table describing the relationship between the output of the virtual sensors and the target
events, the system can output symbolic information such as”put a cup on the desk” when the states of the virtual sensors change.
(d) Detecting human activity events in real time When thesoftware inputs the position data of the ultrasonic tag, thesoftware can detect the target events using the virtual sensors and the table defined in Steps (a) to (c), as shown in
Fig. 24.

6. Conclusion
This paper described a system for robustly detecting
daily human activity events in the handling of objects in

Figure 23. Association between output of virtual sensors and target activity eventthe real world, with supporting calibration and registration
tools to allow the system to be set up quickly. The systemrobustly measures the 3D position of objects and robustly
detects the registered events in real time. Estimation of the
3D position with high accuracy, high resolution, and robustness to occlusion is performed using a RANSAC-based estimation method. The system was tested in an experimentalroom (3.5 × 3.5 × 2.7 m) fitted with 307 ultrasonic receivers
embedded in the walls and ceiling, and it was demonstrated
that it is possible to improve the accuracy, resolution, androbustness to occlusion by increasing the number of ultrasonic receivers and by adopting a robust estimator such as
RANSAC to estimate the 3D position based on redundantdistance data. The resolution of the system is 15 mm horizontally and 5 mm vertically using sensors in the ceiling,
and the total spatially varying position error is 20–80 mm.
It was also confirmed that the system can track moving objects in real time, regardless of obstructions.
The quick calibration method involves the use of a calibration device with 3 or 4 ultrasonic transmitters. By arbitrarily placing the device at multiple positions and measuring distance data in local coordinates, the positions of the receivers can be calculated. The experimental results showedthat the positions of 80 receivers could be calculated using
4 transmitters with a positioning error of only 103 mm.
The quick method for registering target human activity events in the handling of objects involves the use of astereoscopic camera fitted with ultrasonic tags in combination with interactive software. The effectiveness of the function was confirmed by application of the registration procedure to examples such as ”put a cup on the desk”. Using
simplified 3D shape models and event descriptions, users
can easily input new events using simple mouse operations.
Further development of the system will include refine-Proc. of 2nd IEEE International Conference on Pervasive Computing and Communications, pp. 43-54, 2004

11/12

Position datafrom ultrasonic tag system

Simplified shape model
Physical function model
The number of tags and their IDs

Input
Functional model

ment of the method for measuring the 3D position withhigher accuracy and resolution, miniaturization of the ultrasonic transmitters, development of a systematic methodfor defining and recognizing human activities based on thetagging data and data from other sensor systems, and development of new applications based on human activity data.

References

Hold blue cup

Rotate stapler[1] T. Hori. Overview of Digital Human Modeling. Proceedingsof 2000 IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS2000), Workshop Tutorial Note,
pp. 1–14, 2000
[2] H. Mizoguchi, T. Sato, and T. Ishikawa. Robotic Office
Room to Support Office Work by Human Behavior Understanding Function with Networked Machines. IEEE/ASME
Transactions on Mechatronics, 1, (3):237-244, September
1996[3] A. Hopper, P. Steggles, A. Ward, and P. Webster. The
Anatomy of a Context-Aware Application. Proceedings of
5th Annual International Conference Mobile Computing and
Networking (Mobicom99), 59-68, 1999
[4] S. Shih, M. Minami, H. Morikawa, and T. Aoyama. An Implementation and Evaluation of Indoor Ultrasonic Tracking
System. Proceedings of the 2001 IEICE Domestic General
Conference, B-7-20, 2001
[5] http://www.isense.com[6] K. C. Ho. Solution and Performance Analysis of Geolocation by TDOA. IEEE Transaction on Aerospace and Electronic Systems, 29(4):1311-1322, October 1993
[7] D.E. Manolakis, Efficient Solution and Performance Analysis of 3-D Position Estimation by Trilateration. IEEE Trans.
on Aerospace and Electronic Systems, 32(4):1239-1248, October 1996
[8] P. J. Rousseeuw, and A. M. Leroy. Robust Regression andOutlier Detection. Wiley, New York, 1987
[9] M.A. Fishler, and R. C. Bolles. Random Sample Consensus: A Paradigm for Model Fitting with Application to Image Analysis and Automated Cartography. Communicationof the ACM, 24(6):381-395, June 1981

Move three physical objects

Figure 24. Recognition of human activity inreal time using the function model

Proc. of 2nd IEEE International Conference on Pervasive Computing and Communications, pp. 43-54, 2004

12/12