IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS

1

Appearance-based Gaze Estimation with Online
Calibration from Mouse Operations
Yusuke Sugano, Yasuyuki Matsushita, Yoichi Sato, and Hideki Koike

Abstract—This paper presents an unconstrained gaze estimation method using an online learning algorithm. We focus ona desktop scenario where a user operates a personal computer,and use the mouse-clicked positions to infer where on the screen
the user is looking at. Our method continuously captures theuser’s head pose and eye images with a monocular camera,and each mouse click triggers learning sample acquisition. In
order to handle head pose variations, the samples are adaptively
clustered according to the estimated head pose. Then local
reconstruction-based gaze estimation models are incrementally
updated in each cluster. We conducted a prototype evaluation in
real-world environments, and our method achieved an estimation
accuracy of 2.9 degrees.
Index Terms—Eye movement, tracking, Computer Vision,
Human-computer interface

I. I NTRODUCTION
Gaze estimation is the process of detecting at what the eyes
are looking. Many applications have been proposed in thefield of human-computer interaction, including attentive userinterfaces [1], [2] that use user attention for the goal of naturalinteraction. Contrary to traditional gaze-pointing applications,gaze and attention are intended to play a supplemental role toassist user interaction [3].
Current techniques of gaze estimation suffer from technicallimitations such as a lengthy calibration process, extensive
hardware requirements, and difficulty in handing head posevariations. Creating a calibration-free gaze estimator that uses
simple and low-cost equipment while allowing users to freely
move their heads is an open challenge.
One limitation of existing gaze estimation techniques is thatusers actively participate in the calibration task. Users are
typically asked to look at several reference points to acquire
the ground-truth data. Since such an active task interrupts theuser interaction, even a short calibration step can be a criticallimitation in application scenarios that assume a natural stateof attention.
Our goal is to make a completely passive, non-contact,
single-camera system for estimating gaze direction that does
not require an explicit calibration stage yet still allows headpose movement. To achieve this goal, we develop a new
appearance-based system of estimating gaze direction basedon an online-learning approach.
Y. Sugano is with Perceptual User Interfaces Group, Max Planck Institute for Informatics, Campus E1 4, Saarbrücken, 66123, Germany. e-mail:sugano@mpi-inf.mpg.de
Y. Sato is with the Institute of Industrial Science, The University of Tokyo.
Y. Matsushita is with Microsoft Research Asia.
H. Koike is with the Graduate School of Information Science and Engineering, Tokyo Institute of Technology.

Our system incorporates advances of the single-camera
three-dimensional (3-D) estimation of head poses to continuously capture users’ head poses and eye images. We limit
our scenario to a desktop environment with a camera mounted
on the monitor. During the operation of a personal computer(PC), the user looks at the mouse cursor when he or she clicks
the mouse button. Using the clicked coordinates as gaze labels,
the system automatically collects learning samples while users
are operating the PC. Our method continuously and adaptively
learns the mapping between eye appearance and gaze direction
without the need for lengthy calibrations.
Although the idea of using mouse clicks to explicitly recalibrate an eye tracker has been presented in [4], in this work
we show that mouse clicks can serve as training data without
the user’s intention for eye tracker calibration. Using our
method, the user’s natural behavior can serve as a calibration
process and the gaze estimation process can be integrated into
desktop user interfaces. This leads to a scenario where users
can install a software-based gaze estimation system which
gradually learns to predict gaze positions on her/his own
PC. Although it does not directly enable the quick use of
the system, the mapping function is often device- and userdependent and the system can be used by the target user once
the function is learned.
In prior work we utilized visual saliency of a displayed
video to infer focus of attention [5], [6]. Chen et al. [7] applieda similar idea to the case of model-based gaze estimation, andAlnajar et al. [8] proposed to directly use actual human gaze
patterns collected from other viewers for the calibration-freegaze estimation task. These approaches have complementary
scopes of application. Although the saliency-based technique[5]–[7] can be applied to completely passive systems without
active user interaction, it is quite difficult to compute accurate
visual saliency maps in desktop environments. Further, reusing
gaze patterns that are obtained from other users as in [8] isdifficult in our interactive setting, where the gaze behavior isheavily person-dependent. Here we demonstrate the practicaladvantage of the proposed method that achieves comparable
accuracy while running in real time as a component of ainteractive system.
This paper extends our prior work [9] by considering: 1) Online refinement of gaze labels, 2) an improved approach fordiscarding inappropriate training samples, and 3) subpixel eyeimage alignment and blink detection. We also present results
using a web browsing scenario.
The rest of the paper is organized as follows. Section II
presents related work, and Section III describes the architecture of our system. Section IV explains our gaze esti-

IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS

mation method based on an incremental-learning algorithm.
The details of implementing the head tracking and eye-image
cropping are in Section V. Proof of concept evaluations appear
in Section VI. Section VII concludes the paper.
II. R ELATED WORK
A. Model-based methods
Model-based approaches use an explicit geometric model ofan eye and estimate the eye’s gaze direction using geometriceye features [10], [11]. While model-based approaches tend to
produce more accurate results than appearance-based methods,
they typically require a high-resolution camera for accurately
locating the geometric features in the eyes.
In addition, model-based methods often require additional
hardware such as multiple cameras or calibrated light sources
to handle head movements [12]–[20]. Such requirements result
in large systems with special equipment that are not readily
available to end-users. Also, the algorithms are specialized totheir own hardware configuration; therefore, it is difficult toimplement a similar approach with only a single web camera.
There have been methods proposed to remove such restrictions in model-based approaches. Ishikawa et al.’s method [21]uses an active appearance model [22] to extract eye featuresand head poses only with a monocular camera. Yamazoe etal. ’s method [23] estimates gaze direction by fitting a 3-D
eye model to 2-D eye images. These approaches work in thereal-world, and ordinary low-resolution cameras are used inboth methods. Unfortunately, their methods are limited to only
computing coarse features, such as the edge of the iris and thecorners of the eyes, due to the low resolution of the camera,
and result in lower accuracy in comparison with other modelbased methods.
B. Appearance-based methods
Appearance-based approaches directly compute featuresfrom the appearance of eye images and estimate the gazepoints by learning the mapping between eye image featuresand gaze points [24]–[28]. Compared to model-based methods,
appearance-based methods have an advantage of simpler andless restrictive systems and have robustness against outliers
even when implemented with relatively low-resolution cameras. The downsides are 1) typically more data are neededin comparison with the model-based methods, and 2) the
estimation accuracy is in general not as high as with modelbased methods.
With appearance-based approaches, it is difficult to deal
with changes in head pose and head pose variation introduces
the requirement for additional training samples. Baluja etal.’s method [24] allows for some head movements among
the appearance-based methods. Their method collects training
samples for each different head pose while the range of headpose change is limited. They describe two major difficulties: 1)
the appearance of an eye looking at the same point drasticallyvaries with the head pose. Therefore, additional information
about the head pose is necessary and 2) the training samples
have to be collected across the pose space to account for head2

movements. This results in a large number of training samples
and an unrealistically lengthy calibration stage.
To address head pose changes. Lu et al.’s method [29]utilizes additional training data, i.e., a video of the target
person rotating her/his head while fixating on a calibration
target, to learn an error compensation function caused by head
movements. They also proposed an approach to synthesize eye
images for unknown head poses using additional reference
images to estimate pixel flows [30]. Although it relies onan additional RGB-D input, Funes et al.’s method [31] uses
front-facing eye images that are warped using estimated 3D
facial shapes. Valenti et al. [32] proposed a pose-retargetedgaze estimation method that adaptively maps the calibration
plane and gaze displacement vectors to target planes according
to 3D head poses. However, they require specially-designed
calibration processes at reference head poses, and cannot beapplied to our problem setting which results in an uncontrolled
stream of training samples.
III. A RCHITECHTURE
The process flow of our approach is illustrated in Figure 1.
The input to the system is a continuous video stream fromthe camera as well as the display coordinates of the clicked
points. The 3-D model-based head tracker [33] keeps runningduring the entire process to capture the head pose p and tocrop the eye image x.
Our approach assumes that the user’s gaze is directed at themouse cursor on the monitor when the user clicks a mousebutton. With this assumption, we collect learning samples by
capturing mouse cursor positions when clicking as well as eye
images and head poses. We create a training sample at eachmouse click using the screen coordinates of the mouse position
as the gaze label, g, associated with the appearance features(head pose p and eye image x). More training samples are
obtained the more the user clicks. Our system incrementally
updates the mapping function between appearance features andgaze positions using the labeled samples.
In the learning stage, incremental learning is performed
in a reduced principal components analysis (PCA) [34], [35]subspace to decrease the computational cost of dealing withmulti-dimensional image features. The samples are adaptively
clustered according to their head poses, and the local appearance manifold is updated in each sample cluster.
When the new training samples are not given to the system,
the system runs in a prediction loop. In the prediction loop, theinputs to the system are only the head pose p and eye image x,
and the system produces gaze estimates ĝ. The gaze estimateĝ is produced by local linear interpolation of the accumulated
training samples. As more samples are accumulated in thelearning loop, the sample clusters and local manifolds are
updated; therefore, it produces more reliable gaze estimates.
IV. A LGORITHM
The heart of our gaze estimator is to learn the mapping
between appearance features {x, p} and the gaze label g. Oncethe mapping is established, our method predicts the unknown
label ĝ from the unlabeled features {x, p}. Our method uses a

IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS

3

Gaze estimator

Input
User activity (mouse click)

PCA subspace

Gaze point
3D head tracker

Learning

Eye image
Head pose

OutputPrediction

Gaze point

Fig. 1. Learning and prediction flow for the proposed framework. Our method continuously takes gaze points g, eye images x, and head poses p from themouse clicks and synchronously captured images of the user for learning. For the prediction stage, the method takes only eye images x and head poses p as
input to produce gaze estimates ĝ.

local linear interpolation method that is similar to [26], [36],i.e., we predict the unknown label ĝ by choosing k nearest
neighbors from the labeled samples and interpolating theirlabels using distance-based weights.
For the accurate interpolation, it is critical to choose thecorrect neighbors from the appearance manifold, which models
appearance changes of different gaze directions. Tan et al. [26]use 2-D topological information about the coordinates of thegaze labels as a constraint. Two eye images are assumed tobe neighbors on the manifold in their method when they have
similar gaze directions instead of simply evaluating by the similarity of their appearances. This assumption, however, does
not always hold if head pose changes are considered. With
the head pose variations, two different gaze directions yield
similar appearances, or conversely, similar gaze directions leadto very different appearances.
To overcome this problem, we compute the sample clusters
with similar head poses and create a local manifold foreach sample cluster. This model is inspired by the locallyweighted projection regression (LWPR) algorithm [37]. The
local linear regressors are adaptively created and learned in
LWPR according to the distance of input features. We employ
similar adaptive architecture to create pose-dependent clustersof eye images.
In our method, the similarity measure of the cluster, i.e.,the distance between the head pose and the sample cluster, isdefined as a product of the Gaussian functions of head translation and rotation. Given a pose p, specified by translation t
and rotation r in 3-D, the distance sk between the head posep to a certain cluster (say, the k-th cluster) is computed as
1||t − t̄||2sk (p) = p
exp −2κt σt22πκt σt21||r − r̄||2
p
,
(1)
exp −
2κr σr22πκr σr2
where t̄ and σt2 are the average and variance of head translation
calculated from the samples in the cluster. Likewise, r̄ and σr2
are the average and variance of head rotation. The constantweights κt and κr are empirically set.

In Equation (1), the Euclidean distance measure is used forboth translation and rotation vectors. In our method, the rotation vector is represented by quaternions. With the quaternion
representation, the distance can be measured by an angulardistance ωd , i.e., the angle of rotation from one quaternion
to the other. However, we approximate it using the Euclidean
distance. Because we incrementally and continuously update
the clusters, calculating the average r̄ is computationally expensive in the angular-distance measure. However, the averageorientation in the Euclidean-distance measure can easily beobtained as an arithmetic average of the quaternions [38]. The
Euclidean distance ||r − r̄||2 = ||I − r̄r −1 ||2 = 4 sin2 (ωd /4)
can also be a good approximation of the angular distance when
the two rotations are close.
Given a labeled sample {x, p, g}, the eye image x is firstused to update the PCA subspace of the eye images. The
subspace that we use is described with N eigenvectors asx ≈ x̄ + U a,(2)

where x̄ is the mean eye image, U is the matrix whose
columns are composed of the first N eigenvectors, and a is an
N -dimensional vector of PCA coefficients. After updating thesubspace, the sample is added to all clusters whose similaritysk (pt )) is greater than the predefined threshold τx . If nosuitable clusters are found, a new cluster is created to only
contain the new sample. In the prediction stage, given an
unlabeled feature {x, p}, the output gaze ĝ is computed
as a weighted average of the candidate predictions obtained
from multiple sample clusters. The following sections IV-A
and IV-B describe further details of prediction and learning
methods. The prediction and learning algorithms are outlinedin Algorithm 1.
A. Prediction
When unlabeled data {x, p} are given, the system predicts
the gaze estimate ĝ from the learnt data. First, the eye image
x is projected onto the current PCA subspace computed fromall the training samples as
a = tU (x − x̄),(3)

IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS

4

Algorithm 1 Adaptive clustering framework

eye image a to the samples (vertices) of the triangle. The
samples on the triangle as well as the samples adjacent to
the triangle are regarded as neighboring samples. By selecting
such neighboring samples that are located near the triangle,
the sample set for interpolation is restricted to have a limited
amount of gaze variations. To ensure computational efficiency,
the above process of finding neighboring triangles is performed
using the Ns closest samples in the cluster. If none of the Ns
samples forms a triangle, Ns is increased by ns until a triangle
set is found.
Using the selected sample set Np , we compute the interpolation weights w = (w1 , w2 , . . . , w|Np | ). The interpolation
weights w are computed by minimizing the reconstruction
error as
2XX
w = argmin a −
wi ai
s.t.
wi = 1, (4)Prediction: Given input features {x, p}
Project image x into the current subspace: a = tU (x − x̄)
for k = 1 to K (the number of clusters) do
Calculate the interpolated gaze ĝk and prediction confidence ck (Section IV-A).
end forCompute finalP
P prediction as a weighted average: ĝ =
c
ĝ
/
k
k
k
k ck .
Learning: Given the i-th learning sample {x, p} associatedwith the gaze label gUpdate the image subspace using incremental PCA: meanx̄, eigenvectors U , eigenvalues λ, coefficients {a1 . . . ai }.
The input x is approximated as x ≈ x̄ + U ai .
for k = 1 to K (with respect to each of all K clusters) do
if sk (pi ) > τx thenAdd sample to the cluster and update its local manifold(Section IV-B).
end ifend for
if none of sk (pi ) is greater the threshold τx thenCreate new (K + 1)-th cluster and add the sample.
K ← K + 1.
end if

w

i∈Np

i∈Np

where wi denotes the weight of the i-th neighbor’s appearance
ai . Finally, assuming the local linearity, the intermediate gazeestimate gk from the k-th cluster is computed asXĝk =w i gi .
(5)i∈Np

To reduce negative effects from the clusters that do notcontain a sufficient number of samples, we define a reliability
measure for the interpolation that represents how well the input
appearance a can be described by the selected neighbors as!
P(a − i∈Np wi ai )2.
(6)
rk (a) = exp −2ςr2
In other words, we discard samples from the clusters where thereconstruction error of the input appearance a is significant.
In Equation (6), the factor ςr is empirically set. We define theprediction confidence ck as a product of the reliability r(a)and the pose similarity s(p) as
ck = sk (p)rk (a).

Fig. 2. Example of gaze triangulation shown in screen coordinates. Eacheye image (flipped horizontally for a better visualization) is located at thecorresponding gaze point, and the lines indicate Delaunay edges between thegaze points.

using the mean eye image x̄, the basis matrix U whose
columns comprises the first N eigenvectors. a is the projected
N -dimensional vector. An intermediate gaze estimate ĝk isthen computed in each cluster using the projected eye image(PCA coefficients) a and the local interpolation of its neighbors. The neighboring samples of the projected eye image aare selected from the manifold, and the gaze labels of theneighbors are interpolated to determine the intermediate gazeestimate ĝk from the k-th cluster.
As in Tan et al. [26], we use the Delaunay triangulationof the gaze label for creating the appearance manifold. Figure 2 shows the visualization of the appearance manifoldwith Delaunay triangulation. Given the projected eye imagea, our method finds neighboring triangles in the appearance
subspace. The distance from the projected eye image a to atriangle is measured by the average distance from the projected(7)

In this manner, the prediction confidence embeds the reliabilityof the k-th cluster as well as the similarity with the neighboring
samples. The final gaze prediction ĝ is computed as a weighted
average of the intermediate predictions ĝk using the prediction
confidence ck asPk ck ĝkĝ = P.
(8)k ck
To assess the overall reliability of the gaze estimate ĝ,
we further compute the weighted average of rk (a) using thesimilarity measure sk as weights:Psk (p)rk (a)r̄(p, a) = kP.
(9)k sk (p)Figure 3 shows the angular error plot of the gaze estimate ĝacross the prediction reliability r̄. The plot shows that theaccuracy of estimation increases as the reliability measure
increases. From this observation, we stabilize the gaze estimateĝ by taking a weighted temporal average obtained fromconsecutive frames based on the reliability r̄. The effect of
the temporal averaging is discussed in Section VI.

IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS

5

Angular error [degree]

12
10
8
6
4
2
0
0.0

0.2

0.4
0.6
Prediction reliability

0.8

1.0

Fig. 3. Angular error of the gaze estimate ĝ against prediction reliability r̄
(Eq. (9)). The bold rectangles represent local averages with a window widthof 0.1 along the reliability axis.

Interpolated gazefrom the triangle 1

Target gaze

Fig. 4. The gaze label of the sample g is refined using the weighted averageof interpolated labels gi on the neighboring triangles (within the distancethreshold τg ).

B. Learning
When the user clicks a mouse button, our system takes the
clicked position (as the labeled gaze g) as well as the user’s
head pose p and eye image x as a training sample. Given
the i-th training sample {xi , pi , gi }, our method updates theappearance subspace using Skocaj et al. [35]’s incremental
PCA method. The mean eye image x̄ and the basis matrix
U in Equation (2) as well as all the previous coefficients{a1 . . . ai−1 } are updated at the same time.
After updating the appearance subspace, the reduced learning sample {ai , pi , gi } is added to a pose cluster only when its
head pose pi is sufficiently close to the cluster’s center. With
this approach, all clusters are guaranteed to contain samples
with similar head poses. Tan et al. [26] use a topologicalmanifold model with a similar setting to ours. In their method,
however, the gaze label g is treated as a static quantity without
any error. In reality, humans cannot gaze at a point with apixel-level accuracy. Even when a user is looking at a target
carefully, a certain level of fixational eye movement occurs.
About 1-degree of microsaccades occur during fixation [39].
In our case, since the user is not forced to look carefully at themouse cursor when clicking, larger errors can be included in
the gaze label g. Moreover, there could be meaningless samples due to random mouse clicks without sufficient attention.
For these reasons, we avoid directly using the clicked
coordinates gc as a gaze label g. Instead, we estimate theprobable gaze label g constrained by gc using a refining

approach starting with gc as an initial value for g. To refine thegaze label g of the sample, we first select all existing samples
whose distance to the incoming sample’s click point gc areunder a threshold τg in the gaze space. Using these existing
samples and incoming-sample g, the set of all combinations of
three samples that are nearby and enclose incoming-sample gcan be computed (see Figure 4). Using the method describedin the previous section, interpolated gaze label gi can becomputed from each triangle. All the interpolated labels areaggregated as a Gaussian-weighted average around gc asP
gc + i ri qi gi
P
,
(10)
g=
1 + i ri qi
where the Gaussian weighting factor qi is defined as||gi − gc ||2.
qi = exp −
2ςq2

(11)

In the above equations, i is the triangle index, and ri is thereliability measure calculated as in Equation (6). The factor
ςq is empirically set. The clicked point gc is added with thefull-weight 1.
As mentioned above, there are incoming samples that are
inadequate as learning samples, e.g., clicks without due attention. These samples do not convey the correlation between theappearance and gaze label (clicked point). To avoid such outliers, we assess the data through cross validation. In addition
to computing the interpolated gaze label g, we can computea standard interpolation ġ without the constraint of gc asP
ri giġ = Pi.
(12)i ri
If the distance dg = ||ġ−gc ||2 is too large, i.e., the interpolated
gaze ġ is too far from the clicked point gc , the sample can beconsidered as an outlier. In that case, we eliminate the sample
from the cluster instead of refining its gaze label.
To avoid biased distribution of the training samples in thegaze space, we further prune the learning samples to improve
the quality of the training data when the density of the training
samples becomes high. If there is more than one sample withinradius τr around the position of incoming gaze label g, we
keep the nearest sample (with the lowest dg ) and eliminate
the other samples. The threshold value τr should be set withrespect to both the size of the display area and memory
capacity. For example, Tan et al. [26] used one sample per
2.76 cm2 . Whenever a new incoming sample is provided, thedata resampling process described above is executed for every
sample in the clusters.
Once a sample is added to the cluster, we incrementally2
update the cluster mean t̄k and variance σt,kof Equation (1)asnt̄ + t,(13)n+1
nσt2 + (t − t̄new )(t − t̄old )
σt2 ←
,
n+1
where n denotes the number of samples in the cluster before
updating, t represents the translation vector of the incoming
sample, and t̄old and t̄new are the cluster means before and aftert̄ ←IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS

Fig. 5. Head pose tracking and eye image capturing. (a) Estimation of thehead pose. The lines represent the normal directions of the bounding box ofthe user’s head. (b) Cropped eye image. The eye image is cropped based on
the predefined eye region on the face mesh (rectangle in (a)). (c) Cropped
eye image after post-processing. The eye location is aligned and intensity isenhanced. This image is used as the eye appearance feature.

the update, respectively. The updating procedure is applied
to the rotation component r̄ and its variance σr2 . After,
the Delaunay triangulation in the gaze-label coordinates isrecomputed.
V. I MPLEMENTATIONIn this section we describe the methods of obtaining input
features, i.e., the head pose p and eye image x from a sequenceof gray-scale input images. We also explain the method of
detecting blinks for improving the accuracy of gaze estimates.
A. Head pose tracking
Our method uses the head-tracking method [33] based ona multi-linear model, which represents face shape variations
by two separate factors: variations across people and facial
expressions. The head data are represented as the appearanceof the face and 3-D positions of 10 feature points defined in
the local-coordinate system of the user’s head (Figure 5 (a)).
In this work, we ignore the shape variations caused byfacial expressions and use a simplified linear model to adapt
to the variations across people. Our method precomputes 8
eigenshapes from the database of facial shapes, and the faceshape is represented by the linear combination of the basis
shapes. Using the model, our system simultaneously tracks
the 3-D head pose using a particle filter [40] and estimates
the face shape based on bundle adjustment [41]. As a result,
the tracker outputs the user’s 3-D head pose p = {t, r}, wheret = t(x, y, z) is a 3-D translation and r = t(q1 , q2 , q3 , q4 ) is a
4-D rotation vector defined by four quaternions. Figure 5 (a)shows an example of head pose tracking. The crosses indicate
the positions of the detected feature points, and the lines
represent the head pose.
B. Eye image cropping
Once the head pose p is estimated, the system crops the eye
image x. Using the estimated head position p, it first extracts arough eye region from the input image using the predefined eye
location in the generic 3-D face model. Based on the distance
between the two eye corners in the image coordinates, the6rectangular region with a fixed aspect ratio (the rectangle in
Figure 5 (a)) is cropped. The rectangle is then re-scaled to anormalized W1 ×H1 image I1 (Figure 5 (b)). We further apply
histogram equalization to normalize its brightness to obtain the
final eye image I2 .
While head pose tracking is robust, there still remains asmall error when cropping eye images. This error appears asa small amount of jittering in the eye image sequence. For an
appearance-based method, accurate alignment of eye images
is crucial. To improve the alignment, we apply a subspacealignment method as described below.
The eye image I2 of size W2 ×H2 (Figure 5 (c)) is cropped
from the larger image I1 of size W1 ×H1 with a top left margind = t(x, y). As described in Section IV, the PCA subspaceused in the learning algorithm is updated incrementally using
the labeled samples. Our method tries to find the eye image I2that maximizes the correlation with the reconstruction image
I´2 . created from the appearance subspace. The vector form of
the reconstruction image x́2 is computed asx́2 = x̄ + U tU (x2 − x̄),

(14)

where x2 is a vector form of the eye image I2 , x̄ is the average
eye image, and U is a matrix that consists of eigenvectors of
the PCA subspace.
To find the optimal cropping region, a correlation map C
is computed in a brute force manner in the area of (W1 −W2 + 1) × (H1 − H2 + 1). The value in the correlation map
C(x, y) corresponds to the correlation between I2 and I´2 with
an offset d = t(x, y). The offset in the pixel level accuracy isdetermined by taking the point (x, y) that gives the maximal
value of C(x, y). Using this solution as the initial guess, we
further compute the sub-pixel alignment using a simple 2-D
parabola fitting described as#"∂C(x+δx ,y+δy )∂x∂C(x+δx ,y+δy )
∂y

= 0,

(15)

where δx and δy are the subpixel displacement along the xand y-axes, respectively. Equation (15) can be approximatedby a Taylor expansion around (x, y) as 0   00
00
Cxx Cxy
Cx
δx
+
= 0,
(16)
00
00
Cy0
Cxy
Cyy
δy
and the subpixel displacement (δx , δy ) is obtained by

0
00
0
00
yy
 δx = C00y Cxy00 −Cx C00 )2
Cxx Cyy −(Cxy
.
0
00
0
00
 δy = C00x Cxy00 −Cy C00 xx2
C C −(C )
xx

yy

(17)

xy

Here, C 0 and C 00 are the 1st and 2nd order derivatives of C
at (x, y).
Finally, I2 is cropped with the offset (x+δx , y+δy ) to create
the vectorized form of the eye image x. In our configuration,
the size of the final image is set to (W2 , H2 ) = (70, 30). In
most cases eye images were around 80 ∼ 100 pixels wide,and larger than the final resolution.

IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS

7

Correlation

1.000
0.995
0.990
0.985
Blink

0.980
0.975
0.970

Number of frames

Fig. 6. Blink detection. The graph shows the correlation between the cropped
eye image I2 and reconstruction image I´2 . The eye images correspond to I2 .
The correlation drops significantly when the user blinks.

C. Blink detection
If a user blinks when clicking, the data are inappropriate fortraining. To eliminate such samples, our method automaticallydetects blinks based on the correlation of the incoming eye
image and the accumulated eye images. As described in
the previous section, cropping of eye images is performedby finding the optimal offset where the correlation with thereconstructed image is maximized. However, if the input eye
image is dissimilar to any samples that span the subspace(e.g., the blinking case), the maximum correlation becomes
relatively small. From this observation, our method finds theblinking eye images by evaluating the correlation as illustrated
in Figure 6. If the correlation is lower than a pre-defined
threshold τb , we treat the sample as an inappropriate sample.
A blink of an eye usually lasts for about 150 ms, which islong enough to appear in multiple video frames as illustrated
in Figure 6. Therefore, we discard the neighboring frames ofthe detected blinks within a certain time range.
VI. P ROOF OF CONCEPT EVALUATIONS
A. Apparatus
Our system consists of a VGA resolution camera (PointGrey
Flea) and a Windows PC with a 2.67 GHz dual core CPU and 3
GB of RAM. The processing times are about 2 ms for the headtracking, 20 ms for eye cropping and alignment, 20 ms forgaze estimation, and 25 ms for learning. The entire estimation
process including display rendering runs at about 20 fps inour research implementation. Throughout the experiments, weused the following parameters: κt = κr = 2.0, τx = 0.001,
Ns = 30, ns = 10, ςr2 = 25000, ςq2 = 2500, τg = 100 px,
τr = 30 px, and τb = 0.99. We used a 17-inch display with aresolution of 1280 × 1024 pixels (96 dpi).
B. Evaluation with random targets
1) Participants: Ten (nine male and one female) users who
did not wear glasses participated. Their ages ranged from 27
to 32, and the average age of the participants was 28.9 witha standard deviation of 1.8.
2) Procedure: We first conduct the experiment using random click targets. A target for clicking is randomly shown tothe user in a full-screen window. To simulate a typical target,
like a button or an icon on the desktop, we use a circle with

Fig. 7. System setting for the experiments. Left figure shows the screenshot of the full-screen window shown to the user. Right figure shows theexperiment setup.

a 64-pixel diameter (Figure 7). The users were asked to click
the displayed targets as usual. During the operation, the users
are allowed to freely move their head poses. Experiments wereconducted for about 20 minutes (until about 1200 clicks) toevaluate the performance variation across the running time anddiverse head pose variations.
3) Dependent measures: Whenever a new labeled sample
is given, the prediction is performed prior to the learning.
The estimation error is evaluated as the distance between the
clicked position gt and the estimated gaze position ĝt . The
angular error θ is computed as
Dm (gt , ĝt )θt = tan−1,(18)
zt − dcam
where Dm indicates the distance between two points in themetric unit, zt is the depth of the estimated head pose at time
t in the camera coordinate system, and dcam is the pre-defined
distance between the camera and the display. The dcam is 10
cm in our configuration.
4) Results: Table I shows the angular and pixel errors(denoted as average ± standard deviation), click count, thenumbers of clusters. “Normal” error corresponds to the errorof the raw output ĝ, and “weighted” error indicates the errorof the results with the weighted temporal averaging (taking
the past 5 frames in this experiment) based on the weight r̄
in Equation (9). “Used” clicks denote the number of clicks
that are not discarded by the rejection process describedin Section V-C. The last six columns show the ranges ofhead movement of each user. Translation ranges x, y, andz correspond to horizontal, vertical, and depth directions, androtation ranges φ, θ, ψ correspond to angles around the z-, x-,
and y-axes, respectively. The average range in the experimentwas 23 × 7 × 35 cm and 15 × 32 × 24 degrees.
The angular error is consistently low across different users(around 3 degrees), and it demonstrates the better performance
when the weighted temporal averaging is used. Figure 8 showsthe evolution of the average of weighted angular error against
the number of clicks. There are some variations across users,in the early stage of the learning, e.g., less than 400 clicks,
and the errors generally tend to be higher due to an insufficientnumber of learning samples. However, the errors consistently
converge to a certain range after 600 clicks and do not diverge
throughout the sessions.
Table II compares our method to a commercial gaze tracker

IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS

8TABLE IR ESULT USING RANDOM TARGETS . “N ORMAL” ERROR CORRESPONDS TO THE RAW OUTPUT ĝ, OF THE SYSTEM , AND “ WEIGHTED ” ERRORCORRESPONDS TO THE TEMPORAL WEIGHTED AVERAGE BASED ON THE WEIGHT r̄ IN E Q . (9). φ, θ, AND ψ CORRESPOND TO THE ROTATION ANGLESAROUND THE z-, x-, AND y- AXES , RESPECTIVELY.

Person
A
B
C
D
E
F
G
H
I
J

Pixel error px
Weighted
NormalNum. clicks
Used/AllNum.
clusters

2.5 ± 1.5
3.0 ± 2.2
2.4 ± 1.5
3.1 ± 1.5
3.0 ± 2.3
3.2 ± 2.0
2.8 ± 2.0
3.1 ± 2.2
2.8 ± 2.2
3.3 ± 2.6
2.9 ± 2.1

89 ± 58
135 ± 101
102 ± 65
107 ± 73126 ± 92
150 ± 95
120 ± 85
150 ± 105110 ± 89
145 ± 117123 ± 881313/1313
1293/1302
1305/1308
1301/1302
1226/1248
1318/1319
1308/1312
1305/1308
1278/1309
1267/1315

13
11
7
7
21
17
11
7
13
17

2.9 ± 1.8
3.7 ± 2.8
3.0 ± 1.9
3.7 ± 2.7
3.3 ± 2.4
3.6 ± 2.3
3.5 ± 3.0
3.6 ± 2.6
3.1 ± 2.4
3.7 ± 2.9
3.4 ± 2.5

105 ± 67
165 ± 130126 ± 82
129 ± 92140 ± 101
170 ± 112148 ± 130174 ± 125
122 ± 99
164 ± 132144 ± 107

Person

6.0
5.5
5.0
4.5
4.0
3.5
3.0
2.5
2.0
1.5
1.0
0.5
0.0

B
C
D
E
F

400

600
800
1000
Number of clicks

Movement
range cm
23 × 7 × 35
29 × 12 × 31
-

7
10
3
8
6
7
6
8
10
8
7

5
9
6
7
16
18
17
8
41
22
15

18
32
32
29
33
36
23
29
43
42
32

31
36
37
31
52
37
34
31
34
23
35

17
19
12
21
29
25
28
16
29
43
24

3.5
3.0
2.5
2.0

I

0.5

Temporal weighted average

J

0.0

1200

Missing
estimation rate
35%-

(Tobii TX3001 ) and camera-based gaze estimation methods
that allows free head movements [29], [30], [32]. Average estimation error of TX300 is evaluated with the same experimental
setting, i.e., users were instructed to click random targets
under free head movement. The second column in Table IIshows an average angular error of 10 users, where each of theusers clicked about 300 times. The other columns show head
movement range during the experiments and the percentage offrames with missing estimation results. While the estimation
1 http://www.tobii.com/

16
27
23
22
27
34
19
23
21
17
23

Raw output

TABLE IIC OMPARISONS WITH A COMMERCIAL GAZE TRACKER AND
CAMERA - BASED GAZE ESTIMATION METHODS THAT ALLOWS FREE HEAD
MOVEMENTS [29], [30], [32].
Angularerror deg
2.9 ± 2.1
1.7 ± 2.4
2.38 [29]2.24 [30]
3 ∼ 5 [32]Rot. degθψ

Output without clustering

Fig. 8. Evolution of the average angular error. The graph shows the averageof the weighted angular error in the random-target experiments against the
number of clicks. Each line corresponds to each user in Table I.

MethodProposed
TX300
Lu et al. [29]Lu et al. [30]
Valenti et al. [32]

φ

1.0

H

200

Trans. cm
y
z

1.5

G

0

x

4.0

A
Angular error [degree]

Angular error [degree]

Average

Angular error deg
Weighted
Normal

0

200

400

600
800
Number of clicks

1000

1200

Fig. 9. Comparison of errors with and without the clustering. The plotshows the average errors of User A using random targets. The red andblue lines correspond to the results with clustering (normal and weighted
output, respectively), and the green line corresponds to normal output withoutclustering.

accuracy of TX300 is better than our camera-based approach,when the users are freely moving their heads, the error tends tobecome higher than 1 degree and the system frequently returns
missing estimation results. The overall missing estimation rateof TX300 was about 35%. For the other three methods, theirreported estimation errors are shown. These methods dependon explicit calibration stages, and [32] is the only method
which is reported to work in real-time. Despite a lack ofreliable calibration data obtained through an active calibration
scheme, our method can achieve similar accuracy to thesemethods.
To assess the effectiveness of our clustering approach, we
compared the performance with and without the clustering
method. Figure 9 shows one of the results. The plots represent
the evolution of the average angular errors of the gaze estimates for User A. The middle and the bottom lines correspondto the clustering results (normal and weighted output), andthe top line corresponds to the output without clustering, i.e.,
all samples are added to a single cluster. From the plot, theclustering approach consistently improves the performance.
The error gradually increased and did not converge to a certain

IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS

9

4.0
3.5

Angular error [degree]

Angular error [degree]

4.5

3.0
2.5
2.0
1.5

Head pose error

1.0

Eye cropping error

0.5
0.0

0

1

2

3

4

5

Gaussian noise deviation[cm] for head pose / [px] for eye cropping

Fig. 10. Estimation error with respect to input noise. Two lines correspond to
average angular errors plotted against standard deviation of the noise added
to head translation and eye cropping position.

6.0
5.5
5.0
4.5
4.0
3.5
3.0
2.5
2.0
1.5
1.0
0.5
0.0

C. Evaluation with desktop environment
Five users from the prior evaluation browsed web pages forabout 30 minutes. The average click count for a user is around600. To create a natural desktop environment for testing, we
implemented a global system hook that ran as a backgroundprocess to capture the click events and positions. The system
continuously captures the user’s head poses and eye images
as a background process.
Table III shows the angular and pixel errors, click count, thenumbers of clusters, and the range of head pose movement.
As in the previous experiment, clicked coordinates are usedas the ground-truth gaze positions. Our method achieves anangular error of 2.6 degrees on average. Although the click
counts and head pose variations are limited, the accuracy ofgaze estimation is comparable with the result using randomtargets. Figure 11 shows the average of the weighted angularerror with respect to the click count. We observed that theerror converged in a similar manner with the previous result.

A
B
C
D
E

0

100

200

300
400
500
Number of clicks

600

Fig. 11. Evolution of the average angular error in the real-world setting.

0

error without the clustering. In addition, by comparing the
middle and bottom lines, we can see that the estimation error
is greatly reduced by taking the weighted temporal average.
The percentage of the reduced error is about 85% on average
for all users when the weighted temporal averaging is used.
We further conducted a performance comparison with artificial error to quantitatively evaluate the robustness against
input noise. In Figure 10, Gaussian noise was added to two
factors of the input information and average angular errors of
10 users were plotted against the standard deviation of thenoise. For head pose, the noise was added to estimated headtranslations with the standard deviation ranging from 0 to 5
cm. The estimation error does not diverge greatly and our
method can robustly handle noisy head pose. This is mainly
because the head pose is indirectly used as a cue to evaluate
cluster similarity in our method and it does not heavily rely on
geometric information. For eye cropping, alignment error wasadded to the re-scaled eye image I1 with the standard deviation
ranging from 0 to 5 pixels. Although the error is greater when
there is larger cropping noise than, e.g., 3 pixels, our method
can compensate small cropping noise through the sub-pixelalignment step.

Person

x

0

1280

y
1024
Fig. 12. Distribution of clicked points by User A in the desktop-environment
scenario. Clicked points are distributed in the 1280 × 1024 desktop space.

One of the most important factors in this experiment setting
was the biased distribution of the click positions. Figure 12
shows the distribution of the clicked positions by User A.
We can see more clicks on menu buttons and at the top of
the desktop. The distribution of click points in the real-world
scenario is expected to be biased, like this example. For theareas with sparse gaze labels, it is hard to achieve a good
estimation. However, it is expected that the possibility of theuser to steadily look at such areas is low. Another interestingobservation is that the distribution of the gaze labels varies
with the tasks and application scenarios. Our method can get
updated with the changes in the distribution because of theincremental learning approach.
VII. D ISCUSSION
We proposed an appearance-based gaze estimation method
using an incremental learning approach. The proposed method
is developed for a desktop scenario, where a user clicks amouse in PC operations. The clicked position is used as agaze label, and the head pose and appearance of the eye arerecorded with a standard desktop camera. To allow free head
movement, we used a 3-D head pose tracker and proposed aclustering-based method for learning pose-dependent mapping

IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS

10TABLE IIIR ESULT USING DESKTOP ENVIRONMENT. F ROM LEFT TO RIGHT, ANGULAR AND PIXEL ERRORS , CLICK COUNT, THE NUMBER OF CLUSTERS , AND RANGEOF HEAD POSE MOVEMENT ARE SHOWN .
PersonA
B
C
D
E
Average

Angular error deg
Weighted
NormalPixel error px
Weighted
Normal

3.0 ± 2.8
2.7 ± 2.1
2.3 ± 1.7
2.7 ± 1.7
2.3 ± 1.9
2.6 ± 2.1

111 ± 108119 ± 95
121 ± 90112 ± 7392 ± 78111 ± 89

3.4 ± 3.2
3.0 ± 2.4
2.8 ± 2.0
3.3 ± 2.1
2.7 ± 2.2
3.0 ± 2.4

127 ± 124
136 ± 110148 ± 104134 ± 90105 ± 88130 ± 103

functions between eye appearances and gaze points. We further
introduced methods of subspace eye alignment and gaze label
refinement to enhance the estimation accuracy.
While a limitation of our approach is the time is takes forthe gaze estimation process, the effectiveness of the proposed
method is validated through proof of concept evaluations, andour method achieved an estimation accuracy of 2.9 degrees
without temporal smoothing. Although less accurate than stateof-the-art commercial products, our method works with asingle camera without any special hardware and does not
require active participation in the calibration task. This enables
a scenario where users can use a gaze estimation system that
adaptively learns to predict their gaze positions through theusers’ daily activities on a standard PC. The performance ofthe proposed method is accurate enough to infer the user’s areaof interest and their corresponding desktop UI components.
ACKNOWLEDGMENT
This research was supported by the Microsoft Research
IJARC Core Project and JST CREST.
R EFERENCES[1] E. Horvitz, C. Kadie, T. Paek, and D. Hovel, “Models of attention
in computing and communication: from principles to applications,”
Communications of the ACM, vol. 46, no. 3, pp. 52–59, 2003.
[2] R. Vertegaal, J. Shell, D. Chen, and A. Mamuji, “Designing for augmented attention: Towards a framework for attentive user interfaces,”
Computers in Human Behavior, vol. 22, no. 4, pp. 771–789, 2006.
[3] D. Rozado, “Mouse and keyboard cursor warping to accelerate andreduce the effort of routine hci input tasks,” Human-Machine Systems,
IEEE Transactions on, vol. 43, no. 5, pp. 487–493, 2013.
[4] R. J. K. Jacob, “Eye movement-based human-computer interaction
techniques: Toward non-command interfaces,” Advances in HumanComputer Interaction, vol. 4, pp. 151–190, 1993.
[5] Y. Sugano, Y. Matsushita, and Y. Sato, “Calibration-free gaze sensing using saliency maps,” in IEEE Computer Society Conference on Computer
Vision and Pattern Recognition (CVPR2010), 2010, pp. 2667–2674.
[6] ——, “Appearance-based gaze estimation using visual saliency,” IEEE
Transactions on Pattern Analysis and Machine Intelligence, vol. 35,
no. 2, pp. 329–341, 2013.
[7] J. Chen and Q. Ji, “Probabilistic gaze estimation without active personal
calibration,” in Proceedings of the IEEE International Conference on
Computer Vision and Pattern Recognition (CVPR 2011), 2011, pp. 609–
616.
[8] F. Alnajar, T. Gevers, R. Valenti, and S. Ghebreab, “Calibration-freegaze estimation using human gaze patterns,” in Proceedings of the IEEE
International Conference on Computer Vision (ICCV 2013), 2013, pp.
137–144.
[9] Y. Sugano, Y. Matsushita, Y. Sato, and H. Koike, “An incremental
learning method for unconstrained gaze estimation,” in Proceedings ofthe 10th European Conference on Computer Vision (ECCV 2008), 2008,
pp. 656–667.

Num. clicks
Used/AllNum.
clusters

717/728706/718700/700
618/619679/692

4
6
3
1
3

x

Trans. cm
y
z

14
8
18
3
4
10

6
5
2
4
4
4

28
31
28
13
10
22

φ

Rot. deg
θ
ψ

9
26
4
7
8
11

23
47
22
9
21
25

17
20
10
11
16
14

[10] T. E. Hutchinson, K. P. White Jr., W. N. Martin, K. C. Reichert, and
L. A. Frey, “Human-computer interaction using eye-gaze input,” IEEE
Transactions on Systems, Man and Cybernetics, vol. 19, no. 6, pp. 1527–
1534, 1989.
[11] R. J. K. Jacob, “What you look at is what you get: eye movementbased interaction techniques,” in Proceedings of the SIGCHI conferenceon human factors in computing systems, 1990, pp. 11–18.
[12] Y. Matsumoto, T. Ogasawara, and A. Zelinsky, “Behavior recognition
based on head pose and gaze direction measurement,” in Proceedingsof the IEEE/RSJ International Conference on Intelligent Robots andSystems (IROS 2000), vol. 3, 2000, pp. 2127–2132.
[13] C. Morimoto, A. Amir, and M. Flickner, “Detecting eye position andgaze from a single camera and 2 light sources,” in Proceedings ofthe 16th International Conference on Pattern Recognition (ICPR 2002),
2002, pp. 314–317.
[14] J. G. Wang and E. Sung, “Study on eye gaze estimation,” IEEE
Transactions on Systems, Man and Cybernetics, Part B, vol. 32, no. 3,
pp. 332–350, 2002.
[15] D. Beymer and M. Flickner, “Eye gaze tracking using an active
stereo head,” in Proceedings of the IEEE International Conference on
Computer Vision and Pattern Recognition (CVPR 2003), vol. 2, 2003,
pp. 451–458.
[16] S. W. Shih and J. Liu, “A novel approach to 3-d gaze tracking using
stereo cameras,” IEEE Transactions on Systems, Man and Cybernetics,
Part B, vol. 34, no. 1, pp. 234–245, 2004.
[17] Z. Zhu, Q. Ji, and K. P. Bennett, “Nonlinear eye gaze mapping function
estimation via support vector regression,” in Proceedings of the 18th
International Conference on Pattern Recognition (ICPR 2006), vol. 1,
2006, pp. 1132–1135.
[18] C. Hennessey, B. Noureddin, and P. Lawrence, “A single camera eyegaze tracking system with free head motion,” in Proc. 2006 symposiumon eye tracking research & applications, 2006, pp. 87–94.
[19] D. H. Yoo and M. J. Chung, “A novel non-intrusive eye gaze estimation
using cross-ratio under large head motion,” Computer Vision and Image
Understanding, vol. 98, no. 1, pp. 25–51, 2005.
[20] F. L. Coutinho and C. H. Morimoto, “Free head motion eye gaze tracking
using a single camera and multiple light sources,” in Proceedings ofthe Brazilian Symposium on Computer Graphics and Image Processing,
2006, pp. 171–178.
[21] T. Ishikawa, S. Baker, I. Matthews, and T. Kanade, “Passive driver gaze
tracking with active appearance models,” in Proceedings of the 11th
World Congress on Intelligent Transportation Systems, 2004.
[22] T. F. Cootes, G. J. Edwards, and C. J. Taylor, “Active appearance models,” IEEE Transactions on Pattern Analysis and Machine Intelligence,
vol. 23, no. 6, pp. 681–685, 2001.
[23] H. Yamazoe, A. Utsumi, T. Yonezawa, and S. Abe, “Remote gazeestimation with a single camera based on facial-feature tracking without
special calibration actions,” in Proceedings of the 2008 symposium oneye tracking research & applications, 2008, pp. 245–250.
[24] S. Baluja and D. Pomerleau, “Non-intrusive gaze tracking using artificial
neural networks,” Advances in Neural Information Processing Systems(NIPS), vol. 6, pp. 753–760, 1994.
[25] L. Q. Xu, D. Machin, and P. Sheppard, “A novel approach to real-time
non-intrusive gaze finding,” in Proceedings of the British Machine Vision
Conference, 1998, pp. 428–437.
[26] K. H. Tan, D. J. Kriegman, and N. Ahuja, “Appearance-based eye gazeestimation,” in Proceedings of the Sixth IEEE Workshop on Applicationsof Computer Vision (WACV 2002), 2002, pp. 191–195.

IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS[27] O. Williams, A. Blake, and R. Cipolla, “Sparse and semi-supervised visual mapping with the S3 GP,” Proceedings of the 2006 IEEE Conference
on Computer Vision and Pattern Recognition, pp. 230–237, 2006.
[28] F. Lu, Y. Sugano, T. Okabe, and Y. Sato, “Inferring human gaze fromappearance via adaptive linear regression,” in Proc. the 13th IEEE
International Conference on Computer Vision (ICCV 2011), 2011, pp.
153–160.
[29] F. Lu, T. Okabe, Y. Sugano, and Y. Sato, “A head pose-free approach
for appearance-based gaze estimation.” in Proc. 22nd British Machine
Vision Conference (BMVC2011), 2011, pp. 1–11.
[30] F. Lu, Y. Sugano, T. Okabe, and Y. Sato, “Head pose-free appearancebased gaze sensing via eye image synthesis,” in Proc. 21st International
Conference on Pattern Recognition (ICPR2012), 2012, pp. 1008–1011.
[31] K. A. Funes Mora and J.-M. Odobez, “Gaze estimation from multimodal
kinect data,” in Proc. CVPR 2012 Workshop on Gesture Recognition,
2012, pp. 25–30.
[32] R. Valenti, N. Sebe, and T. Gevers, “Combining head pose and eye
location information for gaze estimation,” IEEE Transactions on Image
Processing, vol. 21, no. 2, pp. 802–815, 2012.
[33] Y. Sugano and Y. Sato, “Person-independent monocular tracking of face
and facial actions with multilinear models,” in Proceedings of the IEEE
International Workshop on Analysis and Modeling of Faces and Gestures(AMFG2007), 2007, pp. 58–70.
[34] I. T. Jollife, Principal Component Analysis, ser. Springer Series in
Statistics. New York: Springer-Verlag, 1986.
[35] D. Skocaj and A. Leonardis, “Weighted and robust incremental method
for subspace learning,” in Proceedings of the Ninth IEEE International
Conference on Computer Vision (ICCV 2003), 2003, pp. 1494–1501.
[36] S. T. Roweis and L. K. Saul, “Nonlinear dimensionality reduction by
locally linear embedding,” Science, vol. 290, no. 5500, pp. 2323–2326,
2000.
[37] S. Vijayakumar, A. D’Souza, and S. Schaal, “Incremental online learningin high dimensions,” Neural Computation, vol. 17, no. 12, pp. 2602–
2634, 2005.
[38] M. Humbert, N. Gey, J. Muller, and C. Esling, “Determination of a meanorientation from a cloud of orientations. application to electron backscattering pattern measurements,” Journal of Applied Crystallography,
vol. 29, pp. 662–666, 1996.
[39] S. Martinez-Conde, S. L. Macknik, and D. H. Hubel, “The role offixational eye movements in visual perception,” Nature Reviews Neuroscience, vol. 5, pp. 229–240, 2004.
[40] M. Isard and A. Blake, “CONDENSATION – conditional density
propagation for visual tracking,” International Journal of Computer
Vision, vol. 29, no. 1, pp. 5–28, 1998.
[41] B. Triggs, P. F. McLauchlan, R. I. Hartley, and A. W. Fitzgibbon,“Bundle adjustment – a modern synthesis,” Lecture Notes in Computer
Science, vol. 1883, pp. 298–372, 1999.

11

Yusuke Sugano is a postdoctoral researcher at thePerceptual User Interfaces Group at the Max Planck
Institute for Informatics. He received his M.S. and
Ph.D. degrees in information science and technology from the University of Tokyo, Japan, in 2007
and 2010 respectively. His research interests include
eye tracking, computer vision and human-computerinteraction.

Yasuyuki Matsushita received his B.S., M.S. and
Ph.D. degrees in EECS from the University of Tokyoin 1998, 2000, and 2003, respectively. He joined
Microsoft Research Asia in April 2003. He is a
Senior Researcher in Visual Computing Group. His
areas of research are computer vision (photometrictechniques, such as radiometric calibration, photometric stereo, shape-from-shading), computer graphics (image relighting, video analysis and synthesis).
He is a senior member of IEEE.

Yoichi Sato is a professor at Institute of Industrial
Science, the University of Tokyo. He received his
B.S. degree from the University of Tokyo in 1990,
and his MS and PhD degrees in robotics from Schoolof Computer Science, Carnegie Mellon University in
1993 and 1997 respectively. His research interests
include physics-based vision, reflectance analysis,
image-based modeling and rendering, and gaze andgesture analysis.

Hideki Koike is a professor of Department ofComputer Science, Tokyo Institute of Technology.
He received his B.E. in mechanical engineering, andM.E. and Dr. Eng. in information engineering fromthe University of Tokyo in 1986, 1988, and 1991,
respectively. His research interests includes visionbased HCI, information visualization, and network
security.