ITS 2010: Devices & Algorithms

November 7-10, 2010, Saarbrücken, Germany

Z-touch: An Infrastructure for 3D gesture interactionin the proximity of tabletop surfaces
Yoshiki Takeoka
The University of Tokyo
7-3-1 Hongo
Bunkyo-ku, Tokyo, JAPANyoshikinoko@gmail.com

Takashi Miyaki
The University of Tokyo
miyaki@acm.org

between finger contact and finger hovering, interaction techniques similar to the mouse-over mode used in the graphic
user interfaces (GUIs) would become available. This would
enable the user to inspect the object’s information by approaching it with his/her finger before actually touching it.
Similarly, if the system can detect the inclination and direction of a finger, they can be used to control objects. For
example, a user would be able to rotate an onscreen dialknob using a finger by controlling the direction of the finger
and manipulate the shapes of onscreen objects by controlling
their contour curves by using finger position and angle. We
may also use various 3D hand gestures on a tabletop.

ABSTRACT

Sensing the depth (distance from the surface) of fingers/hands
near a tabletop is very important. It allows us to use threedimensional (3D) gesture interaction in multi-touch applications as we do in the real world. We introduce Z-touch, amulti-touch table that can sense the approximate postures offingers or hands in the proximity of the tabletop’s surface. Ztouch uses a vision-based posture sensing system. Multilayered infrared (IR) laser planes are synchronized with shutter
signals from a high-speed camera, which captures each layerof the laser images. A depth map is obtained by using thecaptured image. Our prototype works at ∼30 fps. Z-touch
not only uses with the finger/hand contact points but also theangle of the hovering fingers. The interaction with the finger
angles is unique and allows users to control multiple parameters by using a single finger. In this study, we introduce
the principle of the method of finger detection and its applications (e.g., drawing, map zooming viewer, Bezier curvecontrol).

To enable these features, several attempts have been madeto use depth information obtained from surface computing
systems. A pen-based interaction system that uses the angleof the pen has been proposed [7]. Capacitive sensor-based
depth detection systems, such as SmartSkin [6], detect finger
proximity and can be used with LCD displays. Some visionbased systems TouchLight [9], Perceptive Workbench [5],and BiDi Screen [4] use correlation-based stereo matchingof finger images obtained from two cameras or multiple image sensors and detect the hand’s distance from the surface. However, capacitive-based systems measure the distance inaccurately because of noise and effects from theuser’s body. Stereo-based systems often require more computational power than systems using a single camera. They
also require very accurate camera calibration and are less robust against planar texture regions. Although systems using
a depth camera [1, 10] have been proposed, these are still
very expensive, and the obtained depth accuracy is inadequate especially for clearly distinguishing between touching
and hovering. A method for estimating angles of fingers byusing an image of the contact fingers [2] has been proposed.
However, it cannot detect hovering fingers.

H5.2 [Information interfaces and presentation]: User Interfaces. - Input devices and strategies.
ACM Classification:
General terms:

Jun Rekimoto
The University of Tokyo,
Sony Computer Science
Laboratories, Inc.
rekimoto@acm.org

Multi-touch, Interactive Surface, GestureInteraction.
Multi-touch Table, Interactive Surface, 3D Hand
Gesture Interaction.
Keywords:

INTRODUCTION

Most surface computing systems are capable of tracking multiple finger positions [3]. Users of these systems can use bothhands and multiple fingers for interacting with various objects on an interactive surface.
While these systems are useful, we think these interactions
can be greatly improved if the system can detect a finger’s
three-dimensional (3D) shape and its distance from the surface. For example, if the system can correctly distinguish

To overcome these limitations, we introduce Z-touch, a multitouch tabletop system that detects finger positions and distances from an interactive surface [8]. Z-Touch uses multiple
laser layers to detect the distance of each finger from the surface (Figure 2). These multiple layers are illuminated in turn,and a high-speed camera installed under the surface captures
an image of each layer independently by synchronizing withthe timing of laser layers. This system mainly provides twofeatures. The first one is hand posture detection. This featureis useful for detection of the vertical/horizontal orientationof the fingers touching near the surface. The other feature isPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.
ITS’10, November 7-10, 2010, Saarbrücken, Germany.
Copyright 2010 ACM 978-1-4503-0399-6/10/11...$10.00.

91

ITS 2010: Devices & Algorithms

November 7-10, 2010, Saarbrücken, Germany

Figure 3: Finger angle detection. The system candetect the finger angles by matching the appearanceof finger postures in laser plane images obtained atdifferent heights.

Figure 1: Z-Touch sensing principle. Multiple infraredlaser planes are projected for recognizing finger distance from the interaction surface. Infrared line laser
modules are synchronized with a high-speed camera’s
shutter signal.

Figure 2: Captured depth map of a hand near the top
panel of Z-touch. RGB channels correspond to thelaser plane: R, highest; G, middle; B, lowest (veryclose to the tabletop’s surface). The image at thebottom is that of the thumb not touching the surface;hence, the silhouette is pale red in color.

Figure 4: Z-touch hardware setup. The dimensions ofZ-touch are 1.0 m × 0.6 m × 0.6 m. The framework
is a 40 mm wide steel angle frame.

hover detection. The system can detect finger positions while
the finger is within proximity if not touching to the surface.
In this paper, we describe the sensing architecture of Z-touchand propose several techniques of interaction for the systemusing 3D finger depth information.

parallel to the top panel. The maximum detectable height isalmost the same as the typical distance between the tip ofthe finger and the second knuckle. The lowest laser planeis the closest to the surface of the top panel. The top panel(control area) is a 420 mm wide and 6 mm thick square glass
covered with a transparent projection sheet, so that only visible light emitted from the projector diffuses in the surfaceof the top panel but reflected IR light passes through (Figure5). Figure 4 shows the Z-touch hardware setup. Its dimensions are 1 m × 0.6 m × 0.6 m. For safety, a small black
felt curtain is attached to the edge of the top glass panel toprevent the IR laser from reflecting. We used a 40 mm steel
angle frame for the framework. The projector was placed in
the middle frame, and the mirror used for the projection isin the bottom frame. The high-speed camera was installedin the bottom frame. We used a Point Grey Grasshopper, ahigh-speed camera which can capture 8-bit gray-scale VGA
images at 200 fps. This camera generates external trigger
signal to laser control circuit for every capturing.

PROPOSED APPROACH
Principle of Z-touch

Z-touch is a multi-touch table infrastructure for 3D gestureinteraction in the proximity of tabletop surfaces. Figure 1
shows its system architecture. The key components are multilayered infrared (IR) line laser modules and a high-speedcamera. Our proposed approach expands the multi-touch implementation method of the laser light plane (LLP). However, only a single laser plane has been used in the LLP,
whereas Z-touch uses a multilayered laser plane. These laser
planes are synchronized with a high-speed camera’s shutter
signal.
Hardware setup

To prevent the flickering effects of environmental illumination and projection light, we used IR line laser modules andan IR band-pass filter. The power of IR line laser is 5 mW,and its wavelength is 780 ± 10 nm. Peak transmission of theIR band-pass filter is 780 nm. We choose this wavelength parameter, based on the spectacle response of the image sensorof high speed camera. Eight sets of triple-line laser modules
are placed at the center of the edge and at the vertex of thetop panel. The IR laser planes are calibrated such that thedistance between two laser planes is 11.5 mm, and they are

Finger tracking

Figure 2 shows a depth map obtained by Z-touch. A blob
will appear at different positions in the captured frames, until
the surface of the fingers/hand is completely perpendicular tothe laser plane. Therefore, the system can detect the finger
angles by matching the appearance of finger postures in thelaser plane images at different heights (Figure 3). We used
three-step image processing to track and estimate the postureof the fingers/hands as follows.
92

ITS 2010: Devices & Algorithms

November 7-10, 2010, Saarbrücken, Germany

Figure 5: Top panel of Z-touch. Eight sets of multilayered line laser modules are placed and directed asindicated by the orange triangles. Transparent screen
sheet is attached to the glass panel.

Figure 6: Touch detection. The images in the bottom
row are depth maps. Colors representing the depth
are the same as in Figure 2. Rectangles show thepositions of detected blobs in each laser layer image.
While the hand on the right is hovering over the surface,no blob is detected in the lowest laser plane (shown inblue on the left).

1. Image binarization.
2. Blob detection in the binarized image.
3. Silhouette matching of the detected blob regions with thatin layers at different heights.
The silhouette matching process finds a match for the blob byusing the following equations. a and b are blobs detected in
at different heights, a ∈ Blobheight , and b ∈ Blobheight+1 .
s(a, b) is the size of the set-theoretic difference between aand b, d(a, b) is the Euclidean distance between the blobs,and T is the distance threshold.
match(a) = arg min c(a, b)

Figure 7: Drawing. The
pixels’ drawing colors
corresponding to thedepth map values.

Figure 8: Map zooming. The zoom levelof the map is changedaccording to the lowest
height of fingers/handsdetected.

(1)baccuracy of depth sensing, color information of the drawing
pixels is corresponded to the values of the depth map.

{s(a, b) × k + d(a, b) d(a, b) < T, k > 0c(a, b) =∞else.

Map zooming
Figure 8 illustrates the map zooming viewer. Users can simultaneously control zooming and scrolling by pointing afinger and varying its height. The zooming level of the map
is controlled by the proximity of the finger to the surface.

This matching method detects the finger heights at the threelevels. Because adjusting the angle and the power of thelaser is very hard, we do not estimate the granularity of finger height. We used the process described above for ease ofimplementation. More efficient processes (such as particle
filter tracking) could also be used. Figure 6 is an applicationthat demonstrates the touch/hover of fingers/hands near thesurface. The position of a finger near the tabletop is shownby the red and green circles. The circles change color when
the finger contacts the surface, and a message “Touch!” isdisplayed near the point where contact was established. The
position of the rectangle indicates where a blob has been detected, and the color shows the laser layer images of the blob.

Bezier curve control
Figure 9 illustrates an application for Bezier curve control.
Users can control multiple curve parameters by using justone finger. The direction of a Bezier curve’s handle corresponds to the horizontal angle of the finger, θ, and the length
corresponds to its vertical angle φ. The endpoint of the curveis the contact point of the user’s finger F (xf , yf ). The position of the curve handle H(xh , yh ) is calculated as follows.
H(xh , yh ) = F (xf , yf ) +−l(sin θ, cos θ), l > 0. (2)tan φ

APPLICATIONS ON Z-touch

To demonstrate the effectiveness of the proposed depth sensing architectures, we implemented three types of interaction
techniques on the basis of Z-touch.

DISCUSSION
Limitations on finger tracking

Some limitations exist on finger tracking. Because it is difficult to align the laser, we cannot estimate the granularityof finger’s height. The algorithm for angle estimation is af-Drawing
Figure 7 illustrates the drawing application. To evaluate the93

ITS 2010: Devices & Algorithms

November 7-10, 2010, Saarbrücken, Germanyarchitecture and these interaction techniques will enhance thepossibilities of existing user interfaces for tabletop computing. Users can intuitively manipulate detailed information ona tabletop. Further work on hardware configuration adjustment or vision recognition can be done to develop practicalapplications.
REFERENCES

1. H. Benko and A. D. Wilson. DepthTouch: Using
Depth-Sensing Camera to Enable Freehand Interactions
On and Above the Interactive Surface. Technical Report
MSR-TR-2009-23, Microsoft Research, 2009.
2. C. T. Dang, M. Straub, and E. André. Hand distinction
for multi-touch tabletop interaction. In ITS ’09: Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces, pages 101–108. ACM,
2009.
3. J. Y. Han. Low-cost multi-touch sensing through frustrated total internal reflection. In UIST ’05: Proceedings of the 18th annual ACM symposium on User interface software and technology, pages 115–118. ACM,
2005.
4. M. Hirsch, D. Lanman, H. Holtzman, and R. Raskar.
BiDi screen: a thin, depth-sensing LCD for 3D interaction using light fields. In SIGGRAPH Asia ’09: ACM
SIGGRAPH Asia 2009 papers, pages 1–9. ACM, 2009.

Figure 9: Bezier curve control. Endpoint of the curve isthe contact position of the finger F (xf , yf ). The direction of the curve handle corresponds to the horizontal
angle θ of the finger. The length of the curve handle
corresponds to the vertical angle φ. Equation( 2) relates the position of the endpoint F (xf , yf ) and theposition of the curve handle H(xh , yh ).

5. B. Leibe, T. Starner, W. Ribarsky, Z. Wartell, D. Krum,B. Singletary, and L. Hodges. The perceptive workbench: Towards spontaneous and natural interaction insemi-immersive virtual environments. In VR ’2000:
IEEE Virtual Reality 2000 Conference, pages 13–20.
IEEE Computer Society, 2000.
6. J. Rekimoto. SmartSkin: an infrastructure for freehand
manipulation on interactive surfaces. In CHI ’02: Proceedings of the SIGCHI conference on Human factors
in computing systems, pages 113–120. ACM, 2002.

fected by changes in illumination such as occlusion by fingers/hands. When large objects like the user’s arm are placedon the tabletop, the silhouette of the fingers changes dramatically, and finger detection may fail temporarily. We deployeda laser module at 20 cm intervals on the bezel to prevent theocclusion effects (Figure 5). Our suggested algorithm usedmatching for different laser layer also has a limitation fortracking. This is due to the coarse matching nature foundin the silhouette matching algorithm. So any mismatchingof blob region will happen temporarily, and will produce incorrect finger’s angle estimation result. However we could
improve the accuracy by using gradation of image for matching.

7. S. Subramanian, D. Aliakseyeu, and A. Lucero. Multilayer interaction for digital tables. In UIST ’06: Proceedings of the 19th annual ACM symposium on Userinterface software and technology, pages 269–272.
ACM, 2006.
8. Y. Takeoka, T. Miyaki, and J. Rekimoto. Z-touch: amulti-touch system for detecting spatial gestures near
the tabletop. In SIGGRAPH ’10: ACM SIGGRAPH2010 Talks. ACM, 2010.

CONCLUSION AND DIRECTION FOR FUTURE WORK

9. A. D. Wilson. TouchLight: an imaging touch screen
and display for gesture-based interaction. In ICMI
’04: Proceedings of the 6th international conference
on Multimodal interfaces, pages 69–76. ACM, 2004.

In this paper, we presented Z-touch, a multi-touch table thatcan sense the distance and posture of fingers near the tabletop’s surface even if the fingers/hands are not touching thesurface. Z-touch uses a multilayered IR laser plane synchronized with a high-speed camera’s shutter signal. By combining multiple levels of laser plane images, the proximity andtilt angle of an approaching finger can be extracted using avision-based technique in real time. We also developed applications that use Z-touch such as map zooming and multiple parameter control by using a single finger. Our proposed

10. A. D. Wilson. Depth-Sensing Video Cameras for 3D
Tangible Tabletop Interaction. In TABLETOP ’07:
IEEE International Workshop on Horizontal Interactive Human-Computer Systems, pages 201–204. IEEE
Computer Society, 2007.
94