SenseShapes: Using Statistical Geometry for Object Selection in aMultimodal Augmented Reality System
Alex Olwal
Hrvoje Benko
Steven Feiner
Department of Computer Science, Columbia University
500 W. 120th St. 450 CS Building, New York, NY{aolwal,benko,feiner}@cs.columbia.edu
Abstract
We introduce a set of statistical geometric tools
designed to identify the objects being manipulated
through speech and gesture in a multimodal augmented
reality system. SenseShapes are volumetric regions ofinterest that can be attached to parts of the user’s body toprovide valuable information about the user’s interaction
with objects. To assist in object selection, we generate arich set of statistical data and dynamically choose whichdata to consider based on the current situation.

1. Introduction
A major problem in developing systems that support
multimodal interaction through speech and gesture is todetermine the object(s) to which a user is referring, sinceirrelevant objects will likely fall into the user’s gaze andpointing direction. We introduce SenseShapes, volumetricregions of interest that can be attached to the user, to provide valuable statistical information about the user’s interaction with objects in the environment. We have created a multimodal augmented reality (AR) system inwhich the statistical data generated by SenseShapes assists the fusion of multimodal input into executable commands.
Our multimodal input devices consist of a modified
Essential Reality P5 glove to sense hand gestures, a headset microphone, and two InterSense IS900 six-degree-offreedom trackers to monitor head and hand position andorientation. IBM ViaVoice 10 is used for speech recognition, and a Sony LDI-D100B optical see-through headworn display presents the AR overlay. Our system receives the gesture events shown in Figure 1 (currently“point,” “grab,” and “thumbs up”) and speech commands
that, with the aid of the SenseShapes, are integrated intovalid actions. SenseShapes are also being used in a different multimodal AR system [2] that, unlike the one described here, employs mutual disambiguation of unimodal
recognizers, does not track the user’s fingers, and uses astatic weighted combination of statistics instead of thedynamic integration process described in Section 3.

2. SenseShapes
SenseShapes are volumetric primitives (spheres, cubes,cylinders, and cones) that we attach to the user (e.g., apointing cone attached to the hand). Previous work in ARand VR has included selection of objects using rays orFigure 1. An AR user interacting through gesture,
speech and SenseShapes. The tracked glove andthree possible gestures are shown at the right.

cones attached to the user’s hand [3] and head [4] andcomputing object intersections with these volumes, or has
used the projection of a tracked point on a glove to perform selection on the image plane [5]. In contrast, a SenseShape keeps a history of all objects in the scene that intersect it, and stores statistical data about these objects.
Our statistical data currently provides us with five different rankings for an object, which are relative to a specificobject’s behavior in a certain SenseShape during a timeperiod.
The time ranking (Trank) is derived from the fraction oftime (Tobject) the object spent in a volume over a specifictime period (Tperiod). The more time the object spends inthe volume, the higher the ranking.
TrankTobject,Tperiod

1 t Trank ! 0.

The distance ranking (Drank) is based on an object’sdistance Dobject from the volume’s origin (which can bearbitrarily chosen) compared to other objects, where Dmax
is the distance of the most distant object in the volume
from the volume’s origin. The closer the object is to thevolume, the higher the ranking.
Drank 1 Dobject,Dmax

1 t Drank ! 0.

The stability ranking (Srank) measures an object’s presence in the volume relative to other objects based on Eobject, the number of times an object enters and exits the volume, and Emax, the most times any object enters and exits
the volume. Fewer entries/exits yield a higher ranking (amore stable object).
S rankEmax  1  Eobject
Emax  1

Proceedings of the Second IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR ’03)
0-7695-2006-5/03 $17.00 © 2003 IEEE,

1 t S rank ! 0.

The most stable objects don’t leave the volume, and have
Eobject = 0 and Srank = 1.
The visibility and center-proximity rankings reflect anobject’s visibility relative to selected SenseShapes. We
compute the visibility ranking of a cone by rendering alow-resolution version of the scene, from a center of projection at the cone’s apex, into an off-screen object buffer[1] containing the cone’s base. Each object is renderedwith a unique color, allowing it to be identified in theframe through the pixel color, as shown in Figure 2.
We currently generate two object buffers, one for aneye cone and one for a hand cone. The visibility ranking(Vrank) is defined as¦ visiblePixelsobject ,Vrank
1 t Vrank t 0,
¦ pixelsInFrame
where visiblePixelsobject are the visible pixels an object has
in a frame, and pixelsInFrame are all the pixels in theframe.
desk #pixels = 106

TV #pixels = 30

minCenterDistance = 24
avgCenterDistance = 28
Vrank = 0.026
Cmin = 0.247
Cavg = 0.241

minCenterDistance = 28
avgCenterDistance = 30
Vrank = 0.007
Cmin = 0.118
Cavg = 0.069

table #pixels = 14

floor #pixels = 1215
minCenterDistance = 0
avgCenterDistance = 16
Vrank = 0.297
Cmin = 1
Cavg = 0.509

minCenterDistance = 26
avgCenterDistance = 30
Vrank = 0.003
Cmin = 0.197
Cavg = 0.193

couch #pixels = 929
minCenterDistance = 6
avgCenterDistance = 18
Vrank = 0.227
Cmin = 0.510
Cavg = 0.424

chair #pixels = 934
minCenterDistance = 1
avgCenterDistance = 15
Vrank = 0.228
Cmin = 0.969
Cavg = 0.529

Figure 2. Off-screen object buffer with a 64-pixel
diameter. Each object is listed with visibility andcenter-proximity information and rankings.

The center-proximity rankings (Cmin and Cavg) indicate
how close the visible portion of an object is to the centerof the shape. The pixel distance is calculated as theEuclidean distance from the center of the object buffer.
C min 1 Cavg

1minDistanceToCenterobject
maxDistanceToCenterframe
avgDistanc eToCenterobject
maxDistanc eToCenterframe,

1 t C min t 0,,

1 t Cavg t 0,

where maxDistanceToCenterframe is the object buffer radius, minDistanceToCenterobject is the smallest distance ofany pixel of the specific object, and avgDistanceToCenterobject is the average distance for all pixels.
Only the statistics that are applicable to a particular
SenseShape are considered during integration of statistics(e.g., visibility ranking might not be relevant for a handcentered “grabbing sphere”).

3. Dynamic Integration
Our preliminary experience with SenseShapes showsthat it is useful to have the system perform dynamic integration, in which an integration strategy for determining
which statistics to use is chosen based on the current gesture, speech and SenseShape. For example, we have usedspatial cues in speech (such as “this/these/that/those”) to
select among a set of alternative rankings:
Ranking(“make this desk red”) = Trank(hand)uVrank(head)uVrank(hand) uCmin(head)uCmin(hand)uDrank(hand)
Ranking(“make that desk red”) = Trank(hand)uVrank(head)uVrank(hand)uCavg(head)uCavg(hand)u(1–Drank(hand))In this example, closer objects will be weighted higher
when “this” is used and “lower when “that” is used. Furthermore, due to the inherent imprecision of pointing at adistance, the average center proximity rank is used for“that,” while the minimum center proximity rank is used
for “this.” When no spatial cues are detected, the system
uses both the average and the minimum center proximity
rankings, but discards the distance rankings altogether.

4. Conclusions and Future Work
SenseShapes are a set of statistical tools that use instrumented volumes to determine the user’s intentions ina multimodal AR system. Our preliminary experiments
show that the dynamic integration of SenseShape statistics increases the predictability of selection. We plan tofurther improve SenseShapes to dynamically adapt theirposition, orientation, size, and geometry to accommodate
different situations. Following that, we plan to conduct auser study to measure the effectiveness and relevance ofeach of the ranking rules, to allow for more effective dynamic integration.

Acknowledgments
We wish to thank Sajid Sadi and Avanindra Utukuri
for their support with the P5 glove. This research was
funded in part by Office of Naval Research Contracts
N00014-99-1-0394, N00014-99-1-0683, and N00014-991-0249, NSF Grant IIS-00-82961, and a gift from Microsoft.

References[1] Atherton, P. R. 1981. A Method of Interactive Visualization of CAD Surface Models on a Color Video Display. In
Computer Graphics (Proceedings of ACM SIGGRAPH 81),15(3). 279–287.
[2] Kaiser, E., Olwal, A., McGee, D., Benko, H., Corradini,A., Li, X., Feiner, S., and Cohen, P. An Architecture for 3D
Multimodal Interaction in Augmented and Virtual Reality. To
appear in Proc. ICMI 2003 (Int. Conference on MultimodalInterfaces), Vancouver, BC. November 5–7, 2003.
[3] Liang, J., Green, M. JDCAD: A Highly Interactive 3D
Modeling System. Computers and Graphics, 18(4). 499–506.
1994.
[4] Mine, M. Exploiting Proprioception in VirtualEnvironment Interaction. Ph.D. Thesis, Department of Computer Science, University of North Carolina at Chapel Hill.
1997.
[5] Piekarski, W. and Thomas, B. H. Using ARToolKit for 3D
hand position tracking in mobile outdoor environments. FirstIEEE Int. Workshop on Augmented Reality Toolkit. 2002.

Proceedings of the Second IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR ’03)
0-7695-2006-5/03 $17.00 © 2003 IEEE