Cros
ssMotio
on: Fusing Dev
vice andd Imagee Motionn for Us
ser
Identific
Ication, TrackinTng and D
Device Associiation
Anddrew D. Wils
son and Hrv
rvoje Benkoo
Microosoft Researc hRedmondd, WA 98052 USA
awilsoon@microsoft..com, benko@@microsoft.coom

Figure 1. CrosssMotion matchees phone device accelerationawitth accelerations observed in thee infrared and ddepth images of the Kinect
v2 camera. (a)): the phone’s accceleration is calculated in the Earth’sEcoordinnate frame, show
wn by the axis; ((b) CrossMotion
n finds thephone held by thet user (red ma
arker); (c) imag
ge and device accceleration matcch image (darkerr is better); (d) ttoys and other oobjects canbe traccked by embedd
ding or attachingg the device (thee device itself dooes not need to b
be directly visib
ble to the cameraa).
Recognnizing and trackinng mobile devicees from video is difficult fora numbber of reasons. M
Many smartphonees are small, shinny, and dark,
makingg them difficult tto image clearlyy. It might be im
mpossible todifferenntiate two devicees of the same mmodel. Handheld ddevices may
be partiially occluded bby the hand, while those kept inn a purse or
clothes pocket can’t be seen at all. Actiive markers suchh as infrared
LEDs ccan assist in traccking and identiification [16]. F
For example,
Xbox OOne and Sony PS
S4 controllers usse infrared and vvisible LEDs
to assisst in tracking and associatingg controllers w
with players.
Howeveer, such active mmarkers are rare, aand require a linee of sight.

A
ABSTRACTT
Identifying and tracking people and mobile deevices indoors hashmmany applicationns, but is still a chhallenging probleem. We introducee a
ccross-modal sensor fusion approaach to track mobile devices and thetuusers carrying them.
t
The CrossMotion technnique matches thetaacceleration of a mobile device, asa measured by ana onboard internnalmmeasurement uniit, to similar accceleration observ
ved in the infrarredaand depth imagess of a Microsoftt Kinect v2 cameera. This matchiing
pprocess is concep
ptually simple annd avoids manyy of the difficultiies
tyypical of morre common ap
ppearance-basedd approaches. In
pparticular, CrossM
Motion does not require a modell of the appearannceoof either the userr or the device, nor in many casses a direct line ofssight to the devicce. We demonstrate a real time im
mplementation th
hat
ccan be applied tot many ubiquittous computing scenarios. In ouroeexperiments, CrossMotion found the person’s bod
dy 99% of the timme,oon average within
n 7cm of a refereence device positiion.

We preesent a sensor ffusion approach to locating andd tracking amobile device and its uuser in video. C
CrossMotion mattches device
acceleraation with accelleration observedd in the infraredd and depth
images of the Kinect caamera (Figure 1)). It uses the ineertial sensors
on to many mobille devices to findd device acceleraation. Device
commonand imaage accelerationss are compared inn the 3D coordinnate frame ofthe env
nvironment, thannks to the abbsolute orientatiion sensing
capabiliities common inn today’s smartpphones, as well aas the range
sensingg capability of com
mmodity depth ccameras.

C
Categories anda Subject Descriptorss
I.4.8 Image Proceessing and Compputer Vision: Sceene Analysis –SSensor Fusion, Tracking
Tr

Previouus works ([4], [113], [15], [11]) eexplore the fusioon of device
sensors and visual feattures to find thhe user carrying the device.
These reely on external mmeans of determiining candidate oobjects in thevideo. F
For example, ShakeID considers which of up to four tracked
hands hholds the device.. Rather than com
mpare the motioon of a small
numberr of candidate objects in the video, CrossMootion sensor
fusion iis performed at eevery pixel in the video image andd requires noseparatee process to sugggest candidate obbjects to track. Thhe techniquerequiress no knowledge of the appearance of the devicee or the user,
therebyy avoiding many of the difficultiees of traditional appearancebased ap
approaches, and aallows for a widee range of camerra placement
options and applicationns. An interestingg and powerful cconsequenceof the teechnique is that tthe device user, aand in many casees the device
itself, m
may be reliably tracked even iff the device is in the user’s
pocket ((Figure 4), or em
mbedded in anothher object (Figuree 1d).

K
Keywords
S
Sensor fusion; deepth cameras; ineertial measuremen
nt units

11. INTRODDUCTION
T
Tracking a mobile device and itss owner is usefu
ul for a number of
uubiquitous compu
uting scenarios th
hat rely on identiifying and trackiingthhe device’s ow
wner to provide location-based services, such as
cconnecting the sm
martphone with nearbyninfrastruccture such as a wallw
ddisplay. In this paper we consider the problem off tracking a mob
bileddevice user with a video camera.
Permission to make digital or hard copiess of all or part of thiis work for personal oroclassroom use is grannted without fee prov
vided that copies are notn made or distributeed
for profit or commerccial advantage and thaat copies bear this nottice and the full citatioonon the first page. Cop
pyrights for componen
nts of this work owneed by others than ACM
M
must be honored. AbstractingAwith creedit is permitted. To copy otherwise, ororepublish, to post on
o servers or to reedistribute to lists, requiresrprior speciffic
permission and/or a fee.
f Request permissio
ons from Permissions@acm.org.
ICMI '14, November 12 - 16 2014, Istanbu
ul, Turkey
Copyright 2014 ACM
M 978-1-4503-2885-2
2/14/11…$15.00
http://dx.doi.org/10.1145/2663204.266327
70

In this paper, we revview related woork, detail the C
CrossMotion
algorithhm, discuss our implementation, present the ressults of two
evaluatiions of the algorithm’s pperformance, aand discussconsideerations and limittations in applyinng the technique..

216

window of time (1s). A Kalman filter is used to estimate theacceleration of each hand. The hand with the most similar pattern ofacceleration is determined to be holding the device. This previous
work further studies the correlation of contacts on a touch screen by
the opposite hand. Ultimately touch contacts are associated with theheld device by way of the Kinect tracked skeleton that is seen to beholding the device.

2. RELATED WORK
2.1 Device Association Using Sensor Fusion“Sensor fusion” refers to the combination of multiple disparate
sensors to obtain a more useful signal. Of particular relevance to thepresent work are fusion techniques that seek to associate two
devices by finding correlation among sensor values taken from both.
For example, when two mobile devices are held together andshaken, accelerometer readings from both devices will be highly
correlated ([3], [7]). Detecting such correlation can cause
application software to pair or connect the devices in some usefulway. Similarly, when a unique event is observed to happen at thesame time at both devices, various pairings may be established.
Perhaps the simplest example is connecting two devices by pressing
buttons on both devices simultaneously [10], but the same idea canbe applied across a variety of sensors. For example, two devices thatare physically bumped together will measure acceleration peaks at
the same moment in time. Hinckley et al. [2] refers to theseinteractions as “synchronous gestures.”

All of the above previous works require that a small number ofcandidate objects are first tracked. The subsequent correlation
process involves determining which of these object’s motion mostclosely matches that of the device. The step of generating candidate
objects can be prone to failure. For example, ShakeID compares the
motion of the tracked hands of the one or two users detected by the
Kinect sensor skeletal tracking process. If the device is not held in
the hand, or if the Kinect skeletal tracking fails, the device cannot be
tracked. Furthermore, holding a mobile device can impact the hand
tracking process to an extent that estimating hand acceleration
robustly is difficult. Kinect skeletal tracking requires a frontoparallel view of the users. Thus relying on Kinect skeletal tracking
constrains where the camera may be placed. For example, skeletal
tracking fails when the camera is mounted in the ceiling for anunobstructed top-down view of the room.

It can be particularly useful to establish correlations across very
different modalities, since often such modalities complement eachother. We mention just a few of these “cross-modal” approaches: amobile phone may be located and paired with an interactive surface
by correlating an acceleration peak in the device with theappearance of a touch contact [12], or when the surface detects thevisible flashing of a phone at the precise moment it is triggered [18].
An object tagged with an RFID chip can be detected and located as
it is placed on an interactive surface by correlating the appearance ofa new surface contact with the appearance of an RFID tag [8].
A small number of previous works investigate the idea of correlating
mobile device inertial sensor readings with movement observed in avideo camera.

In comparison to previous work, CrossMotion avoids the difficultyof choosing candidate objects by matching low level motion features
throughout the image. It may be used in many situations where
skeletal tracking is noisy or fails outright and thus can be used in awide variety of application scenarios. Whereas most of the related
work performs matching over a significant window in time,
CrossMotion uses a fully recursive formulation that relies on storing
only the previous frame’s results, not a buffer of motion history. In
fact, the recursive nature of the computation allows it to be applied
everywhere in the image in real time, avoiding the need to track
discrete objects.

Kawai et al. [4] propose correlating accelerometers worn at the
waist with visual features to track young children in school. They
consider tracking head-worn red LEDs, as well as tracking theposition of motion blobs. For the accelerometer measurements, they
consider integrating to obtain position for direct comparison with thevisual tracking data, as well as deriving pedometer-like features.
While the paper lacks specifics, the authors favor pedometerfeatures in combination with markerless motion blob visual features.

We argue that to correlate image and device motion for the purposesof locating the device or the user carrying it, the best approach is tomatch image motion directly, since as with “synchronous gestures”
the pattern of image motion will provide the discriminative power torobustly detect the device or its user. Making fewer assumptions
about the appearance of the device or user extends the range ofapplicability of the approach, and makes the technique less complex,
more robust, and ultimately more useful.

Shigeta et al. [13] propose computing normalized cross-correlation
between the motion trajectory of an object and device accelerometerreadings to determine which of several tracked objects contains thedevice. Their approach requires a window of many samples toperform correlation and relies on an external process to find andtrack objects from monocular video. Plötz et al. [9] use a similar
approach to synchronize inertial sensors and video cameras.

We take some inspiration from models of visual search whichfeature a pre-attentive, massively parallel processing stage in which
low level features are computed across the visual field, to be
selected by further higher-level top-down processes [19]. For
example, a pedestrian can detect interesting motion in the periphery,such as a car approaching a crosswalk, even on a windy day wherethe visual scene is full of motion.

Teixeria et al. [15] propose identifying and tracking people acrossmultiple existing security cameras by correlating mobile device
accelerometer and magnetometer readings. They describe a hidden
Markov model-based approach to find the best assignment of sensed
devices to tracked people. As with Shigeta et al., they rely on an
external process to generate tracked objects and use a large
matching window, though they demonstrate how their approach canrecover from some common tracking failures.

Maki et al. [5] and Stein et al. [14] similarly propose matching
trajectories of a set of features tracked using optical flow. Trajectory
acceleration magnitude is compared to device acceleration
magnitude to find the closest matching trajectory. These previous
works require that the tracked points be periodically resampled tomaintain even distribution throughout the image. Maki et al.
demonstrates tracking 512 points, while our recursive approach uses
optical flow rather differently, considering all pixel locations in theimage without relying on a set of proposed features, effectively
tracking a few hundred thousand points. Furthermore, theseprevious works compare acceleration magnitude which is invariant
to orientation. CrossMotion instead uses the full 3D acceleration in a
known coordinate frame, and thus avoids many simple scenariosthat lead to false matches when using only acceleration magnitude

2.2 Correlating Image and Device Motion

Most closely related to the present work is Rofouei et al.’s ShakeID
system, which matches smartphone accelerometer values with theacceleration of up to four hands tracked by the Microsoft Kinect
sensor [11]. The hand holding the phone is inferred by matching thedevice acceleration with acceleration of hand positions over a short217

(e.g., a motion too the left will hav
ve the same acceleration magnitu
ude
aas a similar motio
on to the right).

paper, tthe mobile devicce reports orienntation in a standdard “ENU”(east, nnorth, up) coorrdinate system. While magnettic north isdisturbeed by the presencce of metal and oother magnetic fi
fields present
in indooor environments,, in practice it tennds to be constannt in a given
room. FFor our purposess it is only imporrtant that magnettic north notchange dramatically as the device movees about the areaa imaged by
the Kineect sensor.

33. CROSSM
MOTION ALGORITHM
A
M
TThe CrossMotioon algorithm matchesmdevice (e.g.,(smartphonne)aacceleration withh acceleration obbserved in the infrarediand deppth
immages of the Kiinect camera (Fiigure 1). This matchingmprocess is
pperformed at eachh pixel in the inffrared image (Figgure 1c). By virttue
oof the absolute orrientation sensin
ng available on th
he smartphone andathhe ability to deteermine the 3D position of an obsserved point in thetK
Kinect color im
mage, the match is performed ini a common 3D3ccoordinate frame (world referencee frame).

Mobile device acceleroometers report deevice acceleratioon in the 3D
coordinnate frame of the device. H
Having computeed absoluteorientattion using the maagnetometers, gyyros and acceleroometers, it iseasy to transform the acccelerometer outtputs to the ENU
U coordinate
frame aand subtract accceleration due to gravity. Of course, theaccuraccy of this estim
mate of device acceleration inn the ENU
coordinnate frame is onlyy as good as thaat of the orientatiion estimate.
While mmany smartphonne SDKs include functions to repport absoluteorientattion, for reasonns explained lateer we implemennt our own
algorithhm.

Inn this paper, wee present results using the Kinecct for Windows v2
ssensor, which inccludes a time off flight depth caamera. Other dep
pthssensors may alsoo be used. For example,e
we haave a CrossMotiionimmplementation th
hat uses the firstt Kinect for Wind
dows sensor whiich
iss based on a structured light depthh sensor.
T
The algorithm maay be summarizeed as:

Our proototype implemenntation of CrossM
Motion transmits this device
acceleraation (ENU coorrdinates, gravityy removed) over WiFi to thehost PC
C that performs seensor fusion.

11. Find device acceleration: During runtimee, mobile deviice
acceleration iss continually tran
nsmitted to a hosst PC (Figure 1a).
22. Find image motion: Simulttaneously, densee optical flow iscomputed on the Kinect infraared image (Figuure 2a). Each flo
owvector is conv
verted to a 3D motionmusing thee depth image, andatransformed too the coordinate frame of the mobbile device.
33. Estimate image acceleration: 3D accelerationn is estimated byya
Kalman filter at each point off the image, with the 3D flow at thetpoint provided
d as input (Figurre 2b).
44. Match devicee and image acceelerations: The differencedbetweeen
image and deevice accelerationn is computed ata each pixel in thetKinect infrareed image (Figurre 2cd). Small valuesv
indicate thetpossible preseence of the devicce at those pixel locations.
l

3.2 IImage Motioon
The CrrossMotion algorrithm compares the 3D accelerration of themobile device with 33D accelerationn observed in video. Our
approacch to find accelerration in video is to first computee the velocity
of movvement at every pixel in the Kinnect infrared immage using astandardd optical flow ttechnique. This 2D image-spacee velocity isaugmennted with depth iinformation and converted to vellocity in realworld 33D coordinates (mmeters per seconnd). Acceleration is estimatedman filter. We nnext describe
at each point in the imaage using a Kalmeach off these steps in deetail.

3.2.1 Finding 2D V
Velocity with Optical Floww

W
We next describee each of the abovve steps in detail.

Rather tthan track the poosition of a disccrete set of know
wn objects inthe scenne, image motionn is found by coomputing dense optical flowon the eentire Kinect inffrared image. Deense optical flow
w algorithms
model tthe motion obseerved in a pair oof images as a ddisplacement, at eeach pixel. Therre are a variety of optical flow
w algorithms.
Our proototype implemenntation uses the algorithm propoosed by Broxet al. [11], which is knoown for its accuuracy. While opttical flow istypicallyy used to compuute the motion forrward from time − 1 to theframe aat time , for reaasons explained later CrossMotioon computes
flow froom the current frrame at time to the frame at timme − 1. The
velocityy , at each point , we denoote as , and , . We notethat , are integer-valuued, while , arare real-valued.

3.2.2 Converting too 3D Motion
Depth ccameras such as the Microsoft K
Kinect sensor repport distanceto the nearest surfacee at every poinnt in its depth image. The
Windows v2 SDK provides an APII to compute
Microsooft Kinect for W
the 3D position of a pooint in the depthh camera in real world units(meters)). We denote thhe 3D position ccorresponding too a 2D point, in tthe infrared imagge at time as , , .

F
Figure 2. Stages of the CrossMo
otion algorithm: (a) dense opticcal
flflow is computed from the infrared imagge; (b) per-pix
xel
aacceleration estimates using KalmanK
filter; (c) instantaneoousmmatch image (daarker is better) anda (d) filtered match image.

Rather tthan convert 2D
D velocities (as ccomputed by optiical flow) to
3D quaantities directly,, CrossMotion uuses a Kalman filter-based
techniquue that estimatess velocity and accceleration at eachh pixel.

33.1 Device Motion
M
M
Many mobile devicedAPIs offfer real time devicedorientatiion
innformation. To
oday orientationn is computedd by combiniing
innformation fromm the onboard accelerometers, gyroscopes andammagnetometers. BecauseBthis orieentation is with respectr
to magneetic
nnorth (as measureed by the magneetometer) and graavity (as measurred
bby the accelerom
meter, when the device is not moving),m
it is oftten
cconsidered an “aabsolute” orientaation. In the worrk reported in thhis

3.2.3 Estimating A
Acceleration
CrossM
Motion uses a K
Kalman filter tto estimate accceleration ofmovingg objects in the image. The Kalman filter incorporates ourknowleddge of sensor nooise and is recurrsive (that is, it incorporates
all prevvious observationns). The techniqque thus allows much better
estimatees of acceleratioon compared to the approach off using finite

218

differences. While it is beyond the scope of the paper to fully
explain the Kalman filter (see [17] for a good introduction), we
attempt to describe the basics of the technique by way of explaining
the particular formulation used in CrossMotion.

motion estimates spatially, along patches of the image whose
motion is being estimated. In this way the Kalman filter can use
many observations to accurately estimate the acceleration of a given
patch of an object as it moves about the image.

The Kalman filter is closely related to the simpler “exponential”
of a scalarusingfilter which computes a smoothed estimate
the recursive relation:

Flow quantities , and , (which we abbreviate as and ) are
∗
∗
∗
and
from
incorporated by predicting
, , ,
, , ,
, ,
,
,
and
using
the
equations
of
,
,
,
,
,
,
motion as above. In practice, , , , , , and , , are stored as a
2D array the same dimension as the Kinect infrared image, but,
because + and + are real valued, quantities
,
,
, and
are best computed by bilinear,,,,
interpolation on the 2D array. Finally, observation , , is simply
the 3D world coordinate position at image coordinates , . In thisprocess, the Kalman filter at , updates motion estimates found at+ , + in the previous time step, and motion estimates follow
along or “track” the objects whose motion is being estimated.

+ (

=

)

−

where the gain ∈ (0,1) controls the degree to which the filter−. The smaller the gain, theincorporates the “innovation”
less the filter follows the observation , and the more the signal issmoothed. An improved version of the exponential filter is
∗

=

+ (

−

∗)

where ∗ is a prediction of given(for example, by assuming
constant velocity). The Kalman filter is essentially this “improved”
exponential filter, and moreover includes a principled means to set
the value of the gain given our uncertainty in both the prediction∗
and observation .

This interpolation finally motivates computing optical flow inreverse fashion, from time to time − 1: , and , are definedfor all integer values , . Computing flow in the usual fashion fromtime − 1 to time might leave some pixels without “predecessors”
from the previous frame, even if previous motion estimates are
distributed across multiple pixels using bilinear interpolation.
Computing flow from time to time − 1 avoids this problem.

For our problem of estimating acceleration from image motion, we
first consider the motion of a single object in 3D. The familiar
equations of motion predict the object’s position ∗ , velocity ∗ andacceleration ∗ from previous values,
,
, and
:
1
∗
=
+
∆ +
∆
2
∗
=
+
∆
∗

3.3 Sensor Fusion
3.3.1 Common Coordinate System
In the following, we describe a one-time calibration procedure
which obtains the camera’s orientation with respect to the ENU
coordinate frame of the mobile device. Motion observed in thecamera may then be transformed to ENU coordinates and compared
to device accelerations directly.

=

Given observationof the 3D position of a tracked object, we
correct the predictions of position, velocity and acceleration with
=

∗

+

∗(

−

∗)

=

∗

+

∗(

−

∗)

=

∗

+

∗(

−

∗)

While there are many ways to compute the relative orientation of theKinect camera to the coordinate system used by our mobile device,
we adopt a straightforward semi-automatic procedure that is easy to
implement and gives good results. First the mobile device is placed
display-side down on a plane that is easily observed by the Kinect
camera, such as a wall or desk. Viewing the color video stream of
the camera, the user clicks on three or more points on the plane.

where ∗ denotes element-wise multiplication. Kalman gains , ,relate the innovation, or error in the prediction of position, tochanges in each of our estimates of position, velocity andacceleration. Kalman gain is computed as described in [17], and isrelated to our uncertainty in our predictive model ∗ andobservations. In particular, it is crucial to assign a highuncertainty to our estimate of acceleration
to reflect our belief
that acceleration of the object varies over time (indeed, this is thequantity we wish to estimate). Similarly, our uncertainty inisrelated to the noise of our sensor.

The 3D unit normalof the plane in Kinect coordinates iscomputed by first calculating the 3D position of each clicked pointand fitting a plane by a least-squares procedure. The same normal
in ENU coordinates is computed by rotating the unit vector(out of the display of the device) by the device orientation.
Similarly, gravity unit vector
in camera coordinates is taken fromthe 3-axis accelerometer built in to the Kinect sensor. Gravity
inthe ENU coordinate frame is by definition – .

Finally, we note that the usual formulation of Kalman gain is timevarying. However, if the uncertainty of our predictive model andobservations is constant, Kalman gain converges to a constant value[17], as presented above. This leads to a simplified implementationof the update equations, and further underscores the relationshipbetween the Kalman filter and the simpler exponential filter.

The 3×3 rotation matrix M
that brings a 3D camera→
point to the ENU coordinate frame is calculated by matching thenormalsand, as well as gravity vectors
and
, and
forming orthonormal bases K and W by successive cross products:

3.2.4 Incorporating Flow
CrossMotion maintains a Kalman filter of the form described aboveto estimate 3D acceleration at each pixel location in the image. We
denote our estimated position, velocity and acceleration at eachpixel location , as , , , , , and , , respectively.

==

Optical flow information is used in two ways: first, the flow at apoint in the image is a measurement of the velocity of the object
under that point. It thus acts as input to our estimate of acceleration
using the Kalman filter. Second, we can use flow to propagate

,

,

=‖

=‖
M

219

×
×

×

,

=

×

,K=

,

=

×

,W=

= K

W

‖

‖

×

→

is transmitted to a host PC over WiFi at a rate of approximately
50Hz. The Kinect camera is configured using the Microsoft Kinect
for Windows v2 Developer Preview SDK to acquire infrared anddepth images at resolution 512×424 at 30Hz. CrossMotion uses theBrox optical flow API in OpenCV 2.4.7. This optical flow
implementation uses the GPU to achieve real time performance, andis written in Nvidia’s CUDA GPU programming language. Our hostPC runs Windows and includes an Nvidia GeForce GTX 660Ti
graphics card hosting CUDA 5.5.

While this procedure uses a mobile device to place the Kinect
camera in ENU coordinates, we note that this calibration need only
be performed once when the camera is mounted. An unfamiliar
device will work with the system without further calibration as longit also reports orientation in ENU coordinates.

3.3.2 Matching
3D image accelerations are estimated at each pixel and transformed
to the ENU coordinate system as described above. The acceleration
observed at each pixel may be compared directly to the device
acceleration :
, ,

=

, ,

All image processing runs at 30Hz. Optical flow calculations are
performed on the full resolution infrared image. Per-pixel Kalmanfilter updates and sensor fusion matching is implemented in CUDA.
Optical flow parameters (e.g., smoothness) and Kalman filter
parameters (sensor noise and process noise) are derived empirically.

−

Regions of the image that move with the device will give smallvalues of , , . In particular, the hope is that pixels that lie on thedevice will give the smallest values (Figure 2c). If we assume thatthe device is present in the scene, it may suffice to locate its positionin the image by finding ∗ , ∗ that minimizes , , . However, otherobjects that momentarily move with the device, such as those rigidly
attached (e.g., the hand holding the device and the arm) may also
match well.

device and image acceleration, X

acceleration, g5device acceleration, X
image acceleration, X
0-5

In practice, locating the device by computing the instantaneous
minimum over , , will fail to find the device when it ismomentarily still or moving with constant velocity. In these cases
device acceleration may be near zero and so matches many parts ofthe scene that are not moving, such as the background. We address
this by smoothing , , with an exponential filter to obtain , , .
This smoothed value is “tracked” using optical flow and bilinear
interpolation, in the same manner as the Kalman motion estimates(Figure 2d). Small values over the smoothed value , , will pickout objects that match device acceleration over the recent past(depending on smoothing parameter ) and “remember” themoments when some non-zero device acceleration uniquely
identified it in the image. In the case where the device stops moving,
the small values , , will stay with the device for some time,
hopefully until the device moves again.

device and image acceleration, Y

acceleration, g5device acceleration, Y
image acceleration, Y
0-5
device and image acceleration, Z

acceleration, g

5

Our current implementation takes a further optional step to avoid the
problem of matching static backgrounds by adding a small penalty
to pixel locations that exhibit little motion.

device acceleration, Z
image acceleration, Z0-50

An important consideration in performing the above matchingprocess is that the latency of the Kinect sensor is much greater thanthat of the mobile device, including WiFi communications. Without
accounting for this difference, the measure of similarity , , will beinaccurate. CrossMotion accounts for the relative latency of theKinect sensor by artificially lagging the mobile device readings bysome small number of frames. In our prototype implementation thislag is tuned empirically to five frames, approximately 80ms.

0.5

1

1.5

2
2.5
time, seconds

3

3.5

4

4.5

Figure 3. Device acceleration compared to image acceleration at
the point ∗ , ∗ in each of the coordinate axes. The device washeld in by the user and moved back and forth. Note how device
and image acceleration track each other closely.

4.1.1 Computing Orientation
Today’s mobile devices combine gyros, accelerometers andmagnetometers to compute absolute orientation. Gyros measure
angular velocity and are used to provide fast, accurate updates toorientation, while the noisier magnetometers and accelerometers areused to take observations of magnetic north and gravity. The
absolute orientation information in the observations of magnetic
north and gravity is applied to gradually remove the inevitable drift
encountered when using gyros alone. Accelerometers measure thedirection of gravity only when the device is motionless. Thus, if thedevice is undergoing significant motion, the orientation estimate
may be distorted for some time. We find that when using the built-in
Windows Phone 8 API for orientation, orientation estimates are
disrupted by vigorous motion, and several seconds are required torecover. Because CrossMotion relies on accurate orientation
information to transform accelerometer outputs to the ENU

Figure 3 shows a typical trace of device acceleration and image
acceleration at ∗ , ∗ . Considering that these values are computed invery different ways, they track each other surprisingly well.
In some applications it may not be appropriate to assume that thedevice is in the scene. For example, the user holding the device may
leave the camera’s field of view. In this case the minimum value
over , , can be checked against a threshold to reject matches ofpoor quality. We denote the minimum value at ∗ , ∗ as ∗ .

4. IMPLEMENTATION
Our prototype implementation of CrossMotion uses a Microsoft
Kinect for Windows v2 sensor and a Nokia Lumia 920 running
Windows Phone 8. Device acceleration and orientation information220

coordinate frame, when using the built-in API vigorous motion
causes CrossMotion fusion to fail for a few seconds.

Participants were told that the order of strokes in performing theletter gestures was unimportant.

Our own implementation of an algorithm to compute absoluteorientation allows us to make trade-offs that differ from the built-in
implementation. Our implementation opts for near instantaneous
drift correction when the device is deemed motionless, andotherwise performs updates to the orientation estimate using the
gyro running at 250Hz.

This set of trials was repeated but with the introduction of a
distractor user (one of the experimenters) who did their best to
mimic the precise motion of the participant. This distractor stoodnext to the participant and performed the same letter in the samestroke order, and at the same pace. This second set of trials was
included to test the performance of CrossMotion in the presence
other people, as well as give an initial indication of how easy it is to“spoof” a user’s motion.

5. EVALUATION
CrossMotion may be useful in applications where the user interacts
with a large display using speech and gesture, or with a phone
application that is connected with the display. In these settings it
may be valuable to know the identity of the user, their location in
front of the display, and possibly the location of the device.

During every CrossMotion frame, including pauses, software logged
the 3D position corresponding with the algorithm’s best match aswell as the right hand joint position returned by the Kinect SDK
body tracker. Software also logged whether the position of
CrossMotion’s solution ∗ , ∗ lies on the participant’s body as
determined by the Kinect SDK’s “body index” image.

To evaluate CrossMotion’s performance we considered twoparticular modes of use that may be useful in interacting with large
displays by mobile device: gesturing towards the display while
holding the device in the hand, and wearing the device in a pocket
while moving about in front of the display.

5.1.2 Results
Across all trials, of the 76,296 video frames (about 41 min), 1,629
frames (2.1%) occurred when the WiFi connection from the mobile
device dropped temporarily. Of the rest, 98.9% correctly placed∗ ∗,on the participant’s body. Taking the right hand position as
ground truth, average error in 3D was 6.9cm (s.d. = 11.6cm).
Position error in the “near” condition was 6.5cm (s.d. = 8.7cm),
while error in the “far” condition was 7.4cm (s.d. = 14cm). The
increase in error in the “far” condition may be attributed to one
participant in that condition. We suspect that in that trial themagnetometer had fallen out of calibration. In any case, there is no
reason to suspect that the difference in near and far is statisticallysignificant. CrossMotion never placed its solution on the distractor
user, thus the distractor condition had no effect on the results, andwe do not report them separately.

5.1 Experiment 1: Gesturing with the Device
In our first experiment, we tested CrossMotion’s ability to track themobile device as the user gestures in a front display, holding thedevice in the moving hand. This configuration approximates how auser might interact with a remote display application by gesturingwith the device. As we are interested to see if CrossMotion
performance is impacted by increasing distance from the camera, wevaried the participants’ distance to the display. In a second set oftrials we introduced a distractor user (one of the experimenters)
which did their best to mimic the motion of the participant. This was
repeated for each distance condition, giving a 2×2 design with
distance and the presence of a distractor as experimental conditions.
For the purposes of this study we take as ground truth the output of
the Kinect SDK body tracking, which includes the 3D position of
the hands, hip and middle of the spine.

5.2 Experiment 2: Wearing the Device
In the second experiment, we tested CrossMotion’s ability toreliably pick out the person carrying the phone; i.e., does ∗ , ∗ lie
on the user’s body. We are also interested to see if CrossMotion canfurther locate the device on the person. In many circumstances wemight not expect to be able to locate the device, since, as noted
earlier, CrossMotion does not track the device, but instead finds
regions of the image which are consistent the motion of the device.
Thus parts of the user’s body which are rigidly attached to thedevice may match as well as locations on the device itself.

5.1.1 Procedure
We solicited five adult participants (one female, four male) from theauthors’ institution. All were right-handed and were familiar withusing smartphones.
Participants were instructed to stand on a marked spot on the floor
1.5m from the display (near condition) or 2.5m from the display (farcondition), and were directed to attend to the display, a 24” LCD.
The Kinect camera was mounted above the display, about 180 cm
above the floor, to observe the area in front of the display. An
experimenter instructed the participant to hold the device in theirright hand, orienting it so that it is approximately vertical, with thesmartphone’s screen facing them.

We consider two locations of the device on the body: hanging froma lanyard hung around the user’s neck, and placed in the right front
pocket of a jacket. These positions were selected because they bothhave good analogues in Kinect’s body tracking outputs: the lanyard
position matches well with the “middle spine” position, while thejacket pocket corresponds well with the “right hip” position. We
used the same participants as in the first experiment.

The experimenter explained the following set of trials, repeated foreach distance condition: a letter was shown on the display for four
seconds. During that time, the participant was instructed to “draw”
the letter with the device by moving the device in the space in frontof their body, and to finish approximately in four seconds. Thisduration was conveyed by a progress bar animation which finishedat the end of four seconds. At the end of four seconds the screen was
blanked. After a three second pause, during which time theparticipant was instructed to return to a comfortable center positionin front of their body, another letter was presented. After a few
practice trials, the participant performed the same task for each of
the unistroke letters in the alphabet (17 letters, e.g., B, C, D, etc.).
We note the letter stimulus was employed to merely cause the user
to exhibit a variety of gestures, not to test letter recognition.

5.2.1 Procedure
The placement of camera and display was the same as in the firstexperiment. In the “middle spine” condition, the phone was placedin a wearable lanyard-style badge holder, of the kind sometimes
used at conferences. In the “right hip” condition, the phone wasplaced in the right front pocket of a light fleece jacket. In bothconditions, the participant was instructed to move to any one of four
markers on the floor when the remote display indicated, but to do so
in way that they continually faced the remote display. Two of themarkers were the same used in the first experiment, while two more
were located at distance of 2m from the display, but at a distance of
0.5m on either side of the line connecting the first two points. The

221

ffour markers thuss formed a diamo
ond shape. The participant
p
moved
da
tootal of 34 times ini a session (4 seeconds between eachemove).

trackingg, but if the deviice is lying on a desk, CrossMottion will not
be able to detect it againnst a static backgground.

S
Software logged the same inform
mation as in thee first experiment,
eexcept rather thaan logging the riight hand positio
on returned by thetK
Kinect SDK, the right hip and mid
ddle spine positio
ons were logged..

Our inittial experiments show that a simpple threshold on the value of∗is noot enough to relliably determine if the smartphoone is in theview oof the camera or off camera. By inspectingg some test
sequencces, it seems thaat this is primariily because durinng moments
when thhe user is not mooving, static partss of the physical eenvironment
imaged by the camera can sometimes give good matcches. In ourown expperiments we haave been reluctannt to add applicaation-specificheuristiccs to the basic loocalization algoriithm to address thhis problem.
Partly, tthis is to investiggate the power oof the simplest veersion of thealgorithhm. However, w
we note that thhere are a num
mber of preprocessiing steps that would be reasoonable to incorrporate in a
producttion version off CrossMotion. For example,, the staticbackgroound could be mmodeled and reemoved from coonsideration.
Similarlly, in many appllications it may bbe possible to usse the “bodyindex” ssegmentation maade available by the Kinect SDK.. This coarse
segmenntation assigns eaach pixel of the ddepth map to onee of up to six
people or the backgrouund, and could bbe used as a maask or given
some hyysteresis at the leevel of the user.

55.2.2 Results
A
Across all trialss, of the 43,06
60 video framess (about 24 miin)
aanalyzed, 1,352 framesf(3.1%) occcurred when th
he WiFi connectiion
frfrom the mobile device dropped temporarily. Th
hese frames can be
cconsidered invalid. Of the rest, 99
9.0% correctly pllaced ∗ , ∗ on thetpparticipant’s bod
dy. Considering “middle spine”” and “right hiip”
ppositions togetherr, average error in
i 3D was 20.5ccm (s.d. = 14.3cm
m).
P
Position error in the “middle spiine” condition was
w 17.1cm (s.d.. =77.8cm), while errror in the “right hip” condition was
w 23.9cm (s.d.. =220.7cm). The resu
ults suggest a treend of larger erroor for the right hip,hw
which is consisttent with inform
mal observation of the system in
oobservation. In partp this may be explained by th
he inherent lack ofaagreement between our choice off ground truth (KKinect SDK trackked
bbody points) andd the actual position of the dev
vice. In fact, if wew
eexamine the averrage error as a 3D
D vector quantity
y we see that muuchoof the error is inn the z coordinatte, consistent witth the fact that thetddevice is worn ou
utside the body, whilew
the body sk
keleton reported byK
Kinect typically lies
l inside the boddy.

An inteeresting questionn is whether it is possible for one mobile
device user to “spoof” another mobilee device user byy mimicking
motion simultaneously. While ouur first experimeent suggests
their mthat it iss very difficult too do, in theory itt is possible. In ssuch cases itshould bbe possible to at least detect that two devices apppear to matchthe samme image motionn, or themselves, and block furrther action.
Anotherr approach is tto use CrossMootion’s results tto bootstrap
further vision-based anaalysis to achievee a two-factor auuthorization.
For exaample, device orr user identity annd location withiin the image
could seeed a targeted facce recognition prrocess.

C
Comparing averaage error in position across both
h experiments, wew
nnote that error inn the second exp
periment is greatter than that of thetffirst. This is unsu
urprising, since (as( discussed in the next section)) assmaller moving objectomore preciisely indicates po
osition than a larrge
rigid moving objeect. In fact, we area pleasantly surrprised at how wellw
ked when it is woorn.
thhe device is track

55.3 Discussiion

Finally,, we note that thee CrossMotion fu
fusion described in this paper
assumess that the devicce motion and ccorresponding immage motion
consistss only of translattion, and not rotaation. Consider, ffor example,
spinningg the mobile devvice in place. Inn this case CrossM
Motion may
not findd the device, sinnce the device acccelerometers wiill give very
low valuues, while there may be extensivve motion observved about thedevice. In practice this rrarely is an issue, as the rotation oof the device
and user is often acccompanied by significant trannslation. We
envisionn extending thee model to morre completely mmodel device
motion by including anngular velocity aas reported by ddevice gyros
and obsserved by rotatioon in the optical flow field aboutt the device.
Includinng angular veelocity may im
mprove the preecision androbustnness of the matchiing process.

W
While CrossMotiion does not req
quire an appearaance model of thetddevice, in order to detect and trackt
the user itt requires that thetccamera have a lin
ne of sight to som
me part of the usser that moves withthhe device. Conttrast this with otherovision-baseed approaches thhatrequire observinng the device directly.
d
Our exxample sequencces
ddemonstrate that the technique will
w often track thet person holdiingthhe device if the device is in the user’s shirt pockket or pants pockket(F
Figure 4). In thhese cases CrosssMotion finds objectsothat moove
pprecisely with th
he device. This behavior may beb appropriate forfaapplications that require tracking
g at the level off the user. In thooseccases placing thee point ∗ , ∗ annywhere on thee user would bee a
ppositive match.
F
Furthermore, ourr examples show
w that CrossMotiion often finds thet
pposition of the device itself, evenn if it is not direectly in view. Th
hisaability requires that the device motion is unnique against thetbbackground of th
he rest of the mootion in scene, att least periodically.
W
While our initial experiments are promising exam
mples of where thhisiss often true, it will require moore work to show that this worrks
ggenerally since it depends on the surrounding conttext of motion. For
Feexample, the abillity to localize th
he device on the user’s leg whenn it
iss in their pants pocket dependss in part on the leg’s occasionallymmoving differenttly than the torsso. Conversely, when holding thetddevice it is not unncommon for it too find some part of the arm holdiingthhe device (however, we are surprrised by the systeem’s ability to fiind
thhe phone in the user’s shirt poccket, since it wo
ould seem that anyappoint on the torso
o should match eq
qually as well).

Figure 4. Image of , , for phone located in (a) shirtt pocket and(bb) pants pocket. Darker pixels iindicate better mmatch.

6. AP
PPLICATIOONS
Becausee it makes few
w assumptions abbout the appearrance of themobile device or the usser carrying it, C
CrossMotion can be used in awide vaariety of applicatiion settings.

B
By the same tok
ken, because it matches devicee motion to imaage
mmotion, CrossMo
otion cannot initially detect the mobilemdevice iff it
ddoes not move. Iff the device is beeing held by its user,u
it is likely th
hateeven a slight motion of the useer is enough too establish reliabble

Robust indoor person trracking is usefull in providing loccation-based
servicess to individualls. Way-findingg and deliverinng targeted
advertissing are two ofteen-cited example applications. C
CrossMotion
can be used in many inndoor person traacking applicatioons where a222

[2] Hinckley, K., Ramos, G., Guimbretiere, F., Baudisch, P., andSmith, M. Synchronous gestures for multiple persons andcomputers. In Proc. UIST 2003. 149-158.
[3] Holmquist, L.E., Mattern, F., Schiele, B., Alahuhta, P., Beigl,M., and Gellersen, H. Smart-Its friends: A technique for users
to easily establish connections between smart artefacts. In
Proc. Ubicomp 2001. 116-122.
[4] Kawai, J., Shintani, K., Haga, H., and Kaneda, S. Identificationand positioning based on motion sensors and a video camera.
In Proc. 4th IASTED Int. Conf. on Web-Based Education, 2005.
[5] Maki, Y., Kagami, S., and Hashimoto, K. Accelerometer
detection in camera view based on feature point tracking. In
Proc. IEEE/SICE Int’l Symp. On System Integration. 2010.
448-453.
[6] Marquardt, N., Hinckley, K., and Greenberg, S. Cross-device
interaction via micro-mobility and F-formations. In Proc. UIST
2012. 13-22.
[7] Mayrhofer, R., and Gellersen, H. Shake well before use:
intuitive and secure pairing of mobile devices. IEEE
Transactions on Mobile Computing, 8(6). 2009. 792-806.
[8] Olwal, A., and Wilson, A.D. SurfaceFusion: Unobtrusive
tracking of everyday objects in tangible interfaces. In Proc.
Graphics Interface 2008. 235-242.
[9] Plötz, T., Chen, C., Hammerla, N. Y., and Abowd, G.A.
Automatic synchronization of wearable sensors and videocameras for ground truth annotation- a practical approach. In
Proc. 16th Int. Symp. on Wearable Computers, 2012.
[10] Rekimoto, J., Ayatsuka, Y., and Kohno, M. SyncTap: Aninteraction technique for mobile networking. In Proc. Mobile
CHI 2003. 104-115.
[11] Rofouei, M., Wilson, A.D., and Brush, A.J. Your phone ormine?: Fusing body, touch, and device sensing for multi-user
device-display interaction. In Proc. ACM SIGCHI. 2012. 19151918.
[12] Schmidt, D., Chehimi, F., Rukzio, E., and Gellersen, H.
PhoneTouch: A technique for direct phone interaction on
surfaces. In Proc. UIST 2010. 13-16.
[13] Shigeta, O., Kagami, S., and Hashimoto, K. Identifying amoving object with an accelerometer in a camera view. In
Proc. Int. Conf. Intelligent Robots and Systems, 2008.
[14] Stein, S., and McKenna, S.J. Accelerometer localization in theview of a stationary camera. In Proc. Ninth Conference on
Computer and Robot Vision, 2012. 109-116.
[15] Teixeira, T., Jung, D., and Savvides, A. Tasking networked
CCTV cameras and mobile phones to identify and localize
multiple people. In Proc. Ubicomp, 2010. 213-222.
[16] Want, R., Hopper, A., Falcão, V., and Gibbons, J. The active
badge system. ACM Transactions on Information Systems,
10(1), 1992. 91-102.
[17] Welch, G., and Bishop, G. An introduction to the Kalmanfilter. Technical Report TR 95-041. University of North
Carolina at Chapel Hill Dept. of Computer Science. 1995.
[18] Wilson, A.D., and Sarin, R. BlueTable: connecting wirelessmobile devices on interactive surfaces using vision-basedhandshaking. In Proc. Graphics Interface 2007. 119-1.
[19] Wolfe, J.M. Guided search 2.0: a revised model of visual
search. Psychonomic Bulletin & Review 1, 2 (1994), 202-238.

computer vision-based approach would be acceptable, and whereusers are likely carrying a mobile device such as a smartphone ortablet. Computer vision-based approaches to person tracking are
attractive because they can yield fine-grained position information
but they typically either rely on visible markers (as in [16]) orcomplex models of the user’s appearance. Other approaches require
the user to carry new devices and therefore are unlikely to be
adopted outside of critical applications. As an example way-finding
application, consider providing directions to a visitor as they
approach key hallway intersections, or when they leave an elevator.
A CrossMotion-equipped depth camera could be installed at theselocations to direct the visitor.
CrossMotion’s ability to track and identify users by way of theirmobile device makes it particularly appropriate for various device
association problems. For example, a connection between a mobile
device user and a wall display could be automatically established as
the user approaches the display. Knowing the precise position withrespect to the display can be used in number of ways. For example,
when there are multiple simultaneous users, the wall display canrender each user’s own set of virtual objects nearby. Users can“flick” objects from their device onto the display. Fine-grained
person tracking and device association can also be used to detect
when users bring their devices near to each other, which can used
for a variety of applications [6].
CrossMotion may be useful in various settings where it is valuable
to track objects of interest as they move about the environment,
particularly when it is undesirable to attach visual markers to thoseobjects. For example, young children wearing small wrist worn
sensor packages could be tracked in school [4], a child’s stuffedanimal could be tracked throughout the house, while real animals
could be tracked to study their patterns of movement. The XboxOne Skype app performs an automatic digital pan and zoom tocapture the active participants in the room, based on simple image
motion processing techniques. CrossMotion might be used to limit
this selection to particular users. Similarly, given a future long-range
depth camera, a calibrated pan/tilt/zoom camera might follow aparticular soccer player on the field.

7. CONCLUSION
We introduce CrossMotion, a cross-modal sensor fusion technique
to detect and track mobile devices and the people carrying them.
Because it matches inertial device motion with motion observed invideo, it makes very few assumptions about the appearance of either
the device or the user. This paper details a real time implementationof the technique based on estimating image acceleration fromoptical flow. Our initial experiments with the technique demonstrate
its ability to find the user reliably (99% of the time). In many cases
it can find the device itself even when it is not in direct view of thecamera. While there are a number of considerations in applying thetechnique, we believe it is a unique and potentially useful option in
many ubiquitous computing scenarios.

8. REFERENCES[1] Brox, T., Bruhn, A., Papenberg, N., and Weikert, J. Highaccuracy optical flow estimation based on a theory for warping.
In Proc. 8th European Conference on Computer Vision, vol. 4.
2004. 25-36.

223