2013 Second IAPR Asian Conference on Pattern Recognition

Where are you looking at? – Feature-Based Eye Tracking on Unmodiﬁed Tablets

Shoya Ishimaru, Kai Kunze, Yuzuko Utsumi, Masakazu Iwamura, Koichi Kise
Dept. of Computer Science and Intelligent Systems
Osaka Prefecture University{ishimaru,kunze}@m.cs.osakafu-u.ac.jp {utsumi,iwamura,kise}@m.cs.osakafu-u.ac.jp
Front Camera Image

Abstract—This paper introduces our work towards
implementing eye tracking on commodity devices. We
describe our feature-based approach and the eye tracking system working on a commodity tablet. We recorded
data of 5 subjects following an animation on screen asreference. On the assumption that the position of device
and user’s head is stable, the average distance error
between estimated gaze point to actual gaze point isaround 12.23 [mm] using user-dependent training.

Eye detection
eye coordinate

Trim eye image
trimmed eye image

Iris detection

Re-Trimeye image
trimmed eye image

Eye Cornerdetection
eye corner coordinates

Eye gazeestimation

iris coordinate

Eye Gaze Coordinate

Figure 1. Overview of our proposed method and implementation.

I. I NTRODUCTION
the CIDetector library (an iOS built-in face detectionlibrary) as a rough detection to reduce the processing
cost and then improve the iris detection by a more ﬁnegrained method. After trimming, we detect coordinatesof inner corner and outer corner using the Harris cornerdetection from OpenCV [5].
Eye gaze estimation: In gaze estimation phase, thecoordinates of the user’s gaze are calculated by regression. We use the x and y image coordinates of righteye’s iris as (RIx ,RIy ) , left eye’s iris as (LIx ,LIy ),right eye’s inner corner as (RCx ,RCy ) and left eye’s
inner corner as (LCx ,LCy ). The x and y values of eye
gaze coordinate are calculated in Equations (1,2).

Eye gaze can give us unique insides about a person,
especially as it is directly connected to cognition andother brain functions. Therefore there is more and more
research focusing on eye tracking [1], [4], [2], [3].
However, the usage of existing dedicated eye trackers
is still quite limited, as they are very expensive. We
wonder if it is possible to use commodity tablets andphones to detect the eye gaze of the user while he’s
looking on the screen of the device. The contributionsof the paper are as follows: (1) we present an featurebased eye tracking system working on a tablet. (2) Weshow that our system works with an average distanceerror of 12.23 [mm] for 5 users with a user dependent
training phase, on the assumption that the position ofdevice and user’s head is stable.
II. M ETHOD AND I MPLEMENTATION(1)

y = a2 (RIy + LIy )/2 + b2(2)a1 , a2 are the coefﬁcients and b1 , b2 are scalars. Bothare estimated using regression.

We introduce our feature-based method and discuss
the implementation to estimate user’s eye gaze on
mobile tablets. Our eye tracking system is divided into
two-phases. First we detect the eye features. Second
we estimate the eye gaze coordinate using linear regression.
Feature detection: In the feature detection phase, wedetect the eye features (coordinates of iris and eyecorners) from the image taken by front camera. The
detailed ﬂow chart is depicted in Figure1. We use
978-1-4799-2190-4/13 $26.00 © 2013 IEEE
DOI 10.1109/ACPR.2013.190

x = a1 (RIx − RCx + LIx − LCx ) + b1

III. E XPERIMENTAL S ETUPThe experiments are conducted on the Apple iPad4,
with a 1 GHz dual-core A6X processor, 1.0 GB memory, and a screen which resolution is 2048 * 1536. Thescreen’s width is 147.8[mm], the height is 19.71[mm].
The built-in front camera has a resolution of QuadVGA (1280 * 960) as maximum. In this experiment,
we used VGA (640 * 480) images with 132 dpi.
738

Figure 2. Ground truth and the actual gaze points as given by the animated dot for some experiments. The square is the edge of iPadscreen. The ’x’ marks are the coordinates of actual gaze point and ’o’ marks are the coordinates of estimated gaze point by the system.
as given by the animated dot for some experiments. The square is the edge of iPad screen. The ’x’ marks are the coordinates of actual
gaze point and ’o’ marks are the coordinates of estimated gaze point by the system. the left ﬁgure shows a correct estimation, the middle
depicts problems when the inner eye corner cannot be detected correctly, the right shows problems due to vertical head movement.

V. C ONCLUSION AND F UTURE W ORK

There are 5 test subjects with two experimental runs
per participant: One for training and the other for
testing. The distance between user’s face to iPad isﬁxed to 30[cm]. To establish a reliable performancebaseline, we used a portable easel to ﬁx the iPad. For
each experimental run, the subjects keep watching a
dot moving on the screen. The dot moves from leftto right. It jumped to next line and moved from left
to right again. We record the coordinates of the dot
and images from front camera. Since subjects kept
watching the dot, its coordinates on screen are very
close to the actual gaze point. We validated it with theSMI mobile eye tracker for a couple of trials.

In this paper, we present a method to estimate eyegaze on an unmodiﬁed tablet using only the front
facing camera. We propose a feature-based systemusing regression. From the initial experimental result,the accuracy of our system is promising assuming ﬁxed
head and tablet positions. Most importantly, the next
experiments will focus on more natural reading andtablet usage scenarios, using a stationary eye tracker
for ground truth.
R EFERENCES[1] R. Biedert, J. Hees, A. Dengel, and G. Buscher. A robust
realtime reading-skimming classiﬁer. In Proceedings of
the Symposium on Eye Tracking Research and Applications, pages 123–130. ACM, 2012.

IV. R ESULTS AND D ISCUSSION
We evaluate our system using the distance error(Euclidean distance) between actual gaze point and theestimated gaze point. When we set the coordinates ofactual gaze point as (AGx , AGy ) and the coordinatesof estimated gaze point as (EGx , EGy ), error E [mm]is calculated as follows from Equation (3). dpi is theresolution of the taken image. 1 [inch] is 25.4 [mm].

[2] K. Kunze, A. Bulling, Y. Utsumi, S. Yuki, and K. Kise.
I know what you are reading – recognition of document
types using mobile eye tracking. In in Proc. of ISWC
2013, 2013.
[3] K. Kunze, S. Ishimaru, Y. Utsumi, and K. Kise. Myreading life – towards utilizing eyetracking on unmodiﬁed tablets and phones. In Adjunct Proceedings atUbiComp, pages 455–464, 2013.

(EGx − AGx )2 + (EGy − AGy )2 ∗ 25.4/dpi(3)
Figure2 shows the ground truth and the actual gazepoints as given by the animated dot. The mean error
for each subject are 10.74[mm], 24.04[mm], 8.36[mm],13.47[mm] and 4.54[mm]. The average is 12.23 [mm].
There are two problems related to the experimental
recordings: one related to eye corner detection, theother related to head movement (seen in Figure 2).
E=[4] W. Sewell and O. Komogortsev. Real-time eye gaze
tracking with an unmodiﬁed commodity webcam employing a neural network. In CHI’10 Extended Abstracts, pages 3739–3744. ACM, 2010.
[5] V. Vezhnevets and A. Degtiareva. Robust and accurate
eye contour extraction. In Proc. Graphicon, pages 81–84. Citeseer, 2003.

739