Dart-It: Interacting with a Remote Display by Throwing Your Finger Touch
Chi-Chiang Huang∗Rong-Hao Liang∗†
Liwei Chan∗∗†National Taiwan UniversityAcademia Sinica

Bing-Yu Chen∗‡
UEI Research‡

estimatedeyes
positionskinecttargetabc

depth Kinect
imagedFigure 1: Dart-It is a lightweight system that allows users to throw their finger touch to a remote display. In the matching game, a user can(a) lift his arm and (b) rapidly select a target without a cursor, and then (c) swipe to make a match. (d) Occlusion-free eye-tracking based onoptical-flow.

1

Introduction

Hand tracking technologies allow users to control a remote display
freely. The most prominent freehand remote controlling method isthrough a body-centric cursor, e.g. Kinect. Using that method, auser can first place the cursor to a rough position on the remote display, move the cursor to the exact position, and then commit theselection by a gesture. Although controlling the body-centric cursor is intuitive, it is not efficient for novel users who are not familiar with their proprioception. Inaccurate cursor placement resultsin long dragging movement, and therefore causes consequent arm
fatigue problems.
Perspective-based pointing [Pierce et al. 1997] is another freehand
remote pointing method that allows users to select a target on aremote display by directly pointing at where they see. When a user
points at an on-screen target, the remote pointing position is definedby the ray casting from the user’s dominant eye to his/her fingertip.
Perspective-based pointing is efficient because users are allowed
to aim the target accurately with their eyes. Nonetheless, since it
requires reliable face and fingertip tracking, it usually needs extra
cameras and motion trackers in real-world deployment [Banerjeeet al. 2012], which is too heavyweight for general usage. Hence, wepresent Dart-It (Figure 1), a lightweight system that uses only one
RGBD camera to enable perspective-based remote direct-pointing
on any deployed remote display.

2Implementation

A Kinect sensor (Figure 1(a)) is used to extract the users’ 3D eye
positions, fingertips, as well as the skeleton information. If the user
steadily points at a remote screen for a 100-ms moment, a selection
can be confirmed. The short dwell time form the user a sense of”throwing” a finger touch to the position where he/she is looking at.
Our main technical challenge is hand occlusion. When performing
perspective-based pointing, the user’s hand pointing at the remote
display may occlude his/her eyes making the tracking invalid. To
mitigate this issue, we designed an optical-flow-based occlusionfree eye-tracking method (Figure 1(d)). Once the occlusion occurred, our system calculates the optical flow of the user’s head.
Based on the skeleton information, our algorithm first excludes

the background and user’s occluding hand from the depth image,and then calculates the remaining pixels’ optical flows, which aremainly caused by the movement of the user’s head. Based on theresults, we can correct the previous valid eyes’ positions until theeyes are not occluded anymore.
Using GPU, the computational performance of our system is consistently at 30 fps. A 12-participant user study also indicates that,by using our method, users are able to select 13.1cm-width onscreen targets with 95% accuracy, when they were standing or sitting at arbitrary positions in 1.6m ∼ 2m away from the display.

3Application: A Matching GameWe use a matching game, Candy Crush Saga1 , as an application
to demonstrate the usefulness and possible generalization of thistechnique. The object of this game is to make lines of three candies
by selecting one candy and swapping it toward an adjacent one. An
7 × 7 grid of 15cm-width targets are shown on an 100-inch 4:3
projection screen with a Kinect sensor placed on its center bottom.
In this game, a user first selects a desired target by simply lifting hisarm (Figure 1(a)) and then pointing at it (Figure 1(b)), as if throwing his finger touch to the display. Once a target is selected, theuser can swipe the selected target toward its adjacent one to makethe match (Figure 1(c)). No cursors and additional committing gestures are required during the selection. During playing this game,the user can freely move his position or sit down for more comfortable control. The user also can further alternatively use his anotherhand or even bi-manually use both hands for better performance.
Multiuser interaction is also supported in this game.
Acknowledgments: This research was supported in part by theNational Science Council of Taiwan under grants NSC101-2219E-002-026.

References
BANERJEE , A., B URSTYN , J., G IROUARD , A., AND V ERTE GAAL , R. 2012. MultiPoint: Comparing laser and manual pointing as remote input in large display interactions. IJHCS 70, 10,690–702.
P IERCE , J. S., F ORSBERG , A. S., C ONWAY, M. J., H ONG , S.,Z ELEZNIK , R. C., AND M INE , M. R. 1997. Image plane interaction techniques in 3D immersive environments. In Proc. I3D’97, 39–43.
1 http://about.king.com/games/candy-crush-saga/