IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

1

Appearance-based Gaze Estimation using
Visual Saliency
Yusuke Sugano, Yasuyuki Matsushita, and Yoichi Sato
Abstract—We propose a gaze sensing method using visual saliency maps that does not need explicit personal calibration. Our
goal is to create a gaze estimator using only the eye images captured from a person watching a video clip. Our method treats
the saliency maps of the video frames as the probability distributions of the gaze points. We aggregate the saliency maps basedon the similarity in eye images to efficiently identify the gaze points from the saliency maps. We establish a mapping between
the eye images to the gaze points by using Gaussian process regression. In addition, we use a feedback loop from the gazeestimator to refine the gaze probability maps to improve the accuracy of the gaze estimation. The experimental results show thatthe proposed method works well with different people and video clips and achieves a 3.5-degree accuracy, which is sufficient for
estimating a user’s attention on a display.
Index Terms—Gaze estimation, Visual attention, Face and gesture recognition.

F

1 I NTRODUCTION
Gaze estimation is important for predicting human attention, and therefore can be used to better understandhuman activities as well as interactive systems. There isa wide range of applications for gaze estimation including
market analysis of online content and digital signage, gazedriven interactive displays, and many other human-machineinterfaces.
In general, gaze estimation is achieved by analyzing theappearance of a person’s eyes. There are two categories ofcamera-based remote sensing methods: Model-based andappearance-based. Model-based methods use a geometriceye model and its associated features. Using specialized
hardware such as multiple synchronized cameras and infrared light sources, they extract the geometric features ofan eye to determine the gaze direction. Appearance-based
methods, on the other hand, use the natural appearances ofeyes observed from a commodity camera without requiring any dedicated hardware. Various implementations ofcamera-based gaze estimators have been proposed includingcommercial products (see [1] for a recent survey).
One of the key challenges in previous gaze estimators isthe need for explicit personal calibration to adapt to individual users. The users in these existing systems are always required to actively participate in calibration tasks by fixating
their eyes on explicit reference points. Another problem thatmost estimation methods suffer from is calibration drift, andtheir calibration accuracy highly depends on the users andinstallation settings. An interactive local calibration scheme
with, e.g., user feedback [2], is sometimes required in• Y. Sugano and Y. Sato are with the Institute of Industrial Science,
the University of Tokyo, 4-6-1 Komaba, Meguro-ku, Tokyo, 153-8505,
Japan. E-mail: {sugano,ysato}@iis.u-tokyo.ac.jp• Y. Matsushita is with Microsoft Research Asia, 13F, Building 2, No.
5 Dan Ling Street, Haidian district, Beijing, 100080, China. E-mail:yasumat@microsoft.com

practical application systems to correct personal calibration
errors. In many scenarios, such active personal calibration
is too restrictive as it interrupts natural interactions andmakes unnoticeable gaze estimation impossible. Although
the number of reference points for personal calibration canbe reduced using specialized hardware such as multiple
light sources [3], [4], [5] and stereo cameras [6], it stillrequires a user to actively participate in the calibration task.
It is also well known in the class of model-based approaches that the gaze direction can be approximately estimated as the direction of the optical axis without requiring
personal calibration [7]. However, its offset with the visual
axis, which corresponds to the actual gaze direction, canbe as large as 5 degrees [1], [4], and the accuracy varies
significantly based on the individual. More importantly,
such hardware-based attempts add a strong constraint tothe application setting, and this naturally limits the user
scenarios.
There are previous studies that aim at completely removing the need for explicit personal calibration processes. Yamazoe et al. use a simple eyeball model for gaze estimationand perform automatic calibration by fitting the model tothe appearance of a user’s eye while the user is moving
his/her eyes [8]. In Sugano et al.’s method, in a similar
spirit to [2], a user’s natural mouse inputs are used for theincremental personal calibration of the appearance-based
gaze estimation without any calibration instructions [9].
Both methods use only a monocular camera, however, theseapproaches still have some limitations. Yamazoe et al.’s
approach suffers from inaccuracy due to the simplified
eyeball model, and Sugano et al.’s approach can only beapplied to interactive environments with user inputs.
Apart from these gaze estimation studies, computationalmodels of visual saliency have been studied to estimate
the visual attention on an image, which is computed ina bottom-up manner. In contrast to gaze estimation approaches that aim to determine where peoples’ eyes actually

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

Auto-calibration
for gaze estimation

Saliency map

Fig. 1. Illustration of our method. Our method uses
saliency maps computed from video frames in bottomup manner to automatically construct a gaze estimator.

look, visual saliency computes the image region that is
likely to attract human attention. Biologically, a humantends to gaze at an image region with high saliency, i.e.,a region containing unique and distinctive visual featurescompared to the surrounding regions. After Koch andUllman’s original concept [10] of visual saliency, various
bottom-up computational models of visual saliency maps
have been proposed in [11], [12], [13], [14], [15]. Experiments show that there is a correlation between bottom-up
visual saliency and fixation locations [16]. However, thevisual attention mechanism is not yet fully understood. It
is already known that fixation prediction becomes much
more difficult under natural dynamic scenes, in which ahigh-level task and knowledge have a stronger influence on
the gaze control [17].
Gaze estimation (top-down) and visual saliency (bottomup) models are closely related. Nonetheless, not manystudies exist that bridge these two subjects. Kienzle etal. [18], [19] propose a method for learning the computational models of bottom-up visual saliency by using thegaze estimation data. A visual saliency map is modeled intheir work as a linear combination of the Gaussian radialbasis functions, and their coefficients are learned using a
support vector machine (SVM). Judd et al. [20] and Zhao
and Koch [21] also use this approach with different features
and a larger database. The linear weights of low-level image
features (e.g., color and intensity) and high-level features(e.g., face detector) are learned via the SVM in [20]. In [21],the optimal feature weights are learned by solving a nonnegative least squares problem using an active set method.
These approaches learn accurate saliency models using gazepoints. In contrast to these methods, our goal is to createa gaze estimator from the collection of visual saliency
maps. To our knowledge, this is the first work using visualsaliency as prior information for gaze estimation.
In this paper, we propose a novel gaze sensing methodthat uses computational visual saliency, as illustrated inFig. 1. Our approach is based on the assumption thatbottom-up visual saliency is correlated with actual gazepoints. By computing the visual saliency maps from a video
and relating them with the associated eye images of a user,
our method automatically learns the mapping from the eye
images to the gaze points. We aggregate the saliency maps
based on the similarity of the eye images to produce reliablemaps, which we call gaze probability maps in this paper, tohandle low prediction accuracy of raw saliency maps. Oncethe gaze probability maps are obtained, our method learns

2

the relationship between the gaze probability maps andthe eye images. In addition, a feedback scheme optimizes
the feature weights used to compute the visual saliency
maps. The feedback loop enables us to further strengthen
the correlation between the gaze probability maps and theeye images. From one point of view, our method closes
the bottom-up visual saliency and top-down gaze estimates
loop; the visual saliency determines the likely locationof the gaze points, and the gaze points in return refine
the computation of the visual saliency. We demonstrate
our approach through extensive user testing and verify
the effectiveness of the use of visual saliency for gazeestimation.
Our method takes a set of eye images recorded insynchronization with any video clip as the input. Fromsuch an input, our method automatically determines therelationship between the eye images and gaze directions.
In addition, our method does not distinguish between thetest data and training data, i.e., one dataset can be used
for both the calibration and estimation at the same time.
Therefore, when only the gaze estimates for a particular
video clip are needed, a user only needs to watch thevideo clip once. Once the relationship is learned, our gazeestimator can be used in other application scenarios as longas the configuration between the camera and user remains
unchanged. In this manner, the proposed framework leads
to a gaze estimation technique that exempts the users fromthe active personal calibration.
In general, a fundamental trade-off between the accuracy
and a system’s portability exists. Our system aims at
minimizing the hardware and calibration constraints for developing a fully ambient gaze estimation technique, which
is a key factor for opening up a new way of attentive userinterface [22], [23]. For example, to collect the gaze dataover a film clip on a public display, the film creator may
only have to place a camera to capture the eye images of
the audience. Similarly, movie players on PCs can naturally
obtain gaze data for media understanding without the users’
notice. In addition, the calibrated gaze estimators can beused for gaze-based interactions. Our method can further
enhance the calibration accuracy during the gaze estimation
process by using the eye images as input. By closing theloop of calibration and estimation in this manner, thiswork aims at enhancing the approach of calibrating gazeestimators through daily activities [9].
The preliminary version of this work appeared in [24].
A closely related work is recently introduced by Chen andJi [25]. They use the idea of using visual saliency maps
for model-based gaze estimation of a person looking atstill pictures. While Chen and Ji’s approach achieves ahigher level of accuracy and allows for free head movement,
their results rely on a model-based setup with a longer
recording time on a single image. In contrast, our system
uses an appearance-based estimation and is built using onlya monocular camera. While it is often discussed that a gazeprediction from saliency maps is more reliable when using
static photographs than when using video clips, our method
avoids this problem via the aggregation of the saliency

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

3

Eye image

Eye image

Average eye image

Video frame

Saliency map

Gaze probability map

Gazeestimator

Eye image

Estimator
construction
Gaze point
Saliency
aggregation

Saliency
extraction

Optimization ofsaliency model parameters

Fig. 2. Illustration of proposed approach. Our method consists of four steps. The saliency extraction step
computes saliency maps from the input video. The saliency aggregation step combines saliency maps to produce
gaze probability maps. Using the gaze probability maps and associated average eye images, the estimator
construction step learns the mapping from an eye image to a gaze point. A feedback loop is used to optimize thefeature weights to improve the accuracy by using cross validation.

maps, which results in statistically accurate and stable gazeprobability maps.
This paper is organized as follows. In Section 2, we
describe the proposed gaze estimation method that autocalibrates from the bottom-up saliency maps. Section 3
describes the feedback loop from the estimated gaze point
to the saliency weight computation. This feedback loop
is intended to bridge the gap between the top-down gazepoint and the bottom-up visual saliency, and improvesgaze estimation accuracy. Finally, we validate the proposed
method by conducting user tests in Section 4. Our results
show that our method can achieve a 3.5-degree of accuracy without needing any specialized hardware or explicit
personal calibration processes.

2 G AZE
M APS

E STIMATIONFROMS ALIENCY

Our goal is to construct a gaze estimator without ancalibration stage. Our method assumes a fixed head poseand fixed relative positions among the user’s head, camera, and display. The term calibration indicates obtaining
the mapping function from an eye image to a point on
the display coordinate. The relationship between the eye
images (input) and the gaze points (output) is expressed asa single regression function in an appearance-based gazeestimation, and our goal is to estimate the parameters of
the gaze estimation function without using explicit trainingdata.
The inputs for our system are N video frames{I1 , . . . , IN } and associated feature vectors {e1 , . . . , eN }extracted from the eye images of a person who is watchinga video clip with a fixed head position. The implementation
details of the feature vector e are described in Section 4.1;
but our framework does not depend on specific image
features. For presentation clarity, we denote e simply as
eye image in this paper. In our setting, the eye images andvideo frames are synchronized. The i-th eye image ei iscaptured at the same time as frame Ii is shown to theperson. Using this dataset {(I1 , e1 ), . . . , (IN , eN )}, a gazeestimation function from an eye image e∗ to an unknown
gaze point g ∗ is built.
Our method consists of four steps: Saliency extraction,
saliency aggregation, estimator construction, and feature
weights optimization, as illustrated in Fig. 2. Once thesaliency maps are computed in the saliency extraction step,
the saliency aggregation step produces gaze probability
maps that have a higher concentration of gaze point estimates than the saliency maps. The average eye images
are computed by clustering the eye images, and all thesaliency maps are aggregated according to the eye image
similarities to compute the gaze probability maps. Using thegaze probability maps and associated average eye images,
the estimator construction step learns the mapping froman eye image to a gaze point by using a variant of theGaussian process regression. Our method further optimizes
the feature weights that are used for the saliency computation by using the feedback loop. By optimizing the weights
in a cross-validation manner, this fourth step improves theaccuracy of the gaze estimator. The resulting gaze estimator
outputs the gaze points for any eye image of the user. In
the following subsections, we describe the details of thesaliency extraction, aggregation, and estimator construction
steps, and in Section 3 the feature weight optimization.
2.1 Saliency Extraction
This step extracts the visual saliency maps from the input
video frames {I1 , . . . , IN }. As shown in Fig. 3, our method
adopts six features to compute the saliency maps: five lowlevel features and one high-level feature.
Each frame I is first decomposed into multiple featuremaps F . We use commonly-used feature channels, i.e.,color, intensity, and orientations as the static features, andflicker and motion are used as dynamic features in our
method. The intensity channel indicates the grayscale luminance, two color channels are red/green and blue/yellow
differences, and four orientation channels are the responses
from the 2D Gabor filters with orientation at 0◦ , 45◦ , 90◦ ,IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

4

Feature Extraction

Color

Intensity

FlickerFace Detection

MotionOrientation

Graph-Based Visual Saliency Computation

Fig. 3. Computation of visual saliency maps. Our method uses six features to compute the saliency maps:
Face detection-based high level saliency, and five low-level features obtained by using a graph-based saliency
computation; color, intensity, orientation, flicker, and motion. s(1) ∼ s(6) show examples of computed saliency
maps.
and 135◦ , respectively. The flicker channel indicates anabsolute intensity difference from the previous frame, andfour motion channels use the spatially-shifted differences
between the Gabor responses. The feature maps are computed at three levels of the image pyramid which are 1/2,
1/4, and 1/8 of the original image resolution. As a result,
36 (3 levels × (1 intensity + 2 color + 4 orientation + 1
flicker + 4 motion)) feature maps F are computed.
The saliency maps are then computed from the featuremaps F using a Graph-based Visual Saliency (GBVS) [14].
Computation in the GBVS algorithm is conducted in twostages: Activation and normalization. Activation maps A
are first computed from the feature maps F to locate theregions with prominent image features. Greater values are
assigned to the pixels in activation maps A where they havedistinct values compared with their surrounding regions in
the feature maps. In the GBVS algorithm, this computationis performed in a form of a steady-state analysis of a
Markov chain GA . Each node of GA corresponds to a pixel
position in feature maps F , and a transition probabilityΩa between nodes (i, j) and (p, q) is defined based on adissimilarity between the two corresponding pixels in F asΩa ((i, j), (p, q)) , Ωd |F (i, j) − F (p, q)|,

By computing the equilibrium distribution of GN as described above, the resulting maps are concentrated so thatthey have fewer important peaks. These normalized activation maps are averaged within each channel, and as a result,
five low-level saliency maps s(1) , . . . , s(5) are computed.
It is well known that humans tend to fixate on faces,
especially on the eyes, which are highly salient for humans.
With this observation, Cerf et al. [26] proposed a facechannel-based saliency model using a face detector. We follow this approach to produce reliable saliency maps using
the facial features. We use a facial feature detector OKAO
Vision library developed by OMRON Corporation [27]to obtain facial features. This sixth saliency map s(6) ismodeled as 2-D Gaussian circles with a fixed variance at
the center of two detected eye positions. When the detector
only detects a face but not the eyes, e.g., due to a limited
resolution, the facial saliency is defined at the center of the
facial region.
Finally, our method computes the temporal average of(1)(6)
each saliency map s̄i , . . . , s̄i within a temporal window
ns as
(f )

s̄i

(1)

=

i
X
1
(f )
s ,
ns + 1 j=i−n j(4)swhere Ωd indicates the Gaussian weight evaluating the
Euclidean distance between (i, j) and (p, q). In this way,
the nodes (= pixels) with a higher dissimilarity to theirsurroundings have higher transition probabilities. Therefore,
by iteratively computing the equilibrium distribution da (araster-scanned vector form of A) of GA that satisfiesΩa da = da ,(2)

where Ωa is the transition probability matrix consisting ofΩa , the salient pixels in F have larger values in A.
Since the resulting activation maps often have many
insignificant peaks, the GBVS algorithm further normalizes
them to suppress the local maxima. Using the computed
activation maps A, a Markov chain GN is defined in asimilar way with a transition probability Ωn as:Ωn ((i, j), (p, q)) , Ωd |A(p, q)|.

(3)

(f )

where sj
is the raw saliency map of the f -th
feature computed from the j-th frame, and ns is
the number of frames used for temporal averaging.
This is because humans cannot instantly follow rapid
scene changes, and only the past frames are used forthe smoothing to account for the latency. As a result, synchronized pairs of saliency maps and eye im(1)
(6)
(1)
(6)
ages Ds = {(s̄1 , . . . , s̄1 , e1 ), . . . , (s̄N , . . . , s̄N , eN )}are produced.
2.2

Saliency Aggregation

Although it is assumed that the saliency maps can predictgaze points, their accuracy is insufficient for determining
the exact gaze point locations as discussed in previous
studies [17]. In this section, we describe our method toIEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

compute the probability distribution of the gaze point by
aggregating the computed saliency maps.
The computed saliency maps {s̄(f ) } encode the distinctive visual features of video frames. While the saliency
map does not provide the exact gaze point, highly salient
regions in the saliency map are likely to coincide withthe actual gaze point. Suppose we have a set of saliency
maps that statistically have high saliency scores aroundthe actual gaze point and random saliency scores at otherregions. Since we assume a fixed head position, there isa one-to-one correspondence between the gaze points andthe eye images; actual gaze points would be almost thesame between visually similar eye images. Therefore, byaggregating the saliency maps based on the similarity of
the associated eye images, we can assume that the image
region around the actual gaze point has a vivid peak ofsaliency compared with other regions. The aggregated map
can be used as the gaze probability map, i.e., the probability
distribution of the gaze point.
The similarity score ws of a pair of eye images ei andej is defined as
ws (ei , ej ) = exp(−κ2s ||ei − ej ||2 ),(5)

where the factor κs controls the similarity score. The
similarity score ws is higher when the appearances of two
eye images are similar, i.e., the gaze points of the eye
images are close. Since the appearance variation of the eye
images is quite large for different people, the optimal valueof κs in Eq. (5) is highly person-dependent. Therefore, inthis work, the factor κs is indirectly defined via the range ofthe values taken by ws . More specifically, κs is optimized
by minimizing an error defined asκs = argmin ||Ts − det(Ws )||2 ,

(6)

κs

where Ws ∈ RNs ×Ns is a similarity weight matrix computed using randomly selected Ns eye images in Ds . Ts is
the target value of the determinant that is empirically defined, e.g., by quantitatively checking a sample dataset. The
factor κs is determined to adapt to the person-dependency
by minimizing Eq. (6) via the gradient descent.
We eliminate the eye images that are not useful for gazeestimation from the dataset, e.g., eye images of blinking,prior to the computation of the gaze probability maps. On
the other hand, the eye images recorded during fixation
are useful as training data. To automatically identify such
eye images, we use a fixation measure of an eye image edefined as
we (ei ) = exp(−αe κ2s Var(ei )),(7)

where αe is a weighting factor, and Var(ei ) denotes thevariance in the eye images {ei−nf , . . . , ei+nf } over a5

temporal window 2nf + 1 centered at i,i+nfX

Var(ei ) =||ej − µei ||2 ,(8)

j=i−nfµei =12nf + 1

i+nf

X

ek .

(9)

k=i−nfEq. (7) evaluates the stability of the eye regions, and it
is assumed that there are no significant changes in thelighting conditions during the temporal window. Since theappearances of the eye images rapidly change during thefast movement of the eyes, we (ei ) becomes small whenei is captured during eye movement or blinking. A sub(1)
(6)
(1)
(6)
set Ds0 = {(s̄1 , . . . , s̄1 , e1 ), . . . , (s̄N 0 , . . . , s̄N 0 , eN 0 )}is created from Ds by removing the eye images where thewe scores are lower than a predefined threshold τf .
Since variation in the gaze points is limited in Ds0 andthere can be many samples that share almost the same gazepoint, the eye images are clustered according to similarity
ws to reduce redundancy and computational cost. Using thesimilarity score (Eq. (5)), each eye image ei is sequentiallyadded to the cluster whose average eye image ē is the mostsimilar to ei . A new cluster is adaptively created if thehighest similarity among all existing clusters is lower thana threshold τe . M clusters and their average eye images{ē1 , . . . , ēM } are computed from these computations.
(f )After these steps, a gaze probability map p̄i of eachfeature f is computed as
PN 0
(f )
(f )
j=1 ws (ēi , ej )(s̄j − s̄all )
(f )
,
(10)
p̄i =
PN 0
j=1 ws (ēi , ej )
(f )

(f )

(f )

where s̄all is the average of all the maps s̄1 , . . . , s̄N 0 in
Ds0 . It is known that a center bias of visual saliency [20],[21] exists because man-made pictures usually have ahigher saliency at the center of the image. The average(f )saliency map s̄all is used to eliminate this center bias ina gaze probability map. Without this, the gaze probability
map tends to have a higher value at the center regardless(f )of the eye image ēi . The gaze probability map p̄i canalso have negative values. In our case, only the relative
differences matter, and therefore, we used the computed
results by normalizing the values to a fixed range. We again
use the graph-based normalization scheme (Eq. (3)) to the(f )gaze probability maps p̄i in order to enhance the peaksin the gaze probability maps.
The final gaze probability map p̄i is computed as aweighted sum of all the feature-dependent maps p̄(f ) as
p̄i =

6
X

(f )

ωf p̄i ,

(11)

f =1

where ωf is a weight for f -th feature. p̄i is thennormalized to a fixed range, and we obtain a dataset
Dp = {(p̄1 , ē1 ), . . . , (p̄M , ēM )}. We followed many existing visual saliency map models and used equal weights at
this step to aggregate feature maps. However, it is often

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

Fig. 4. Examples of gaze probability maps p̄ andcorresponding average eye images ē. The overlaid
dots depict the actual gaze points of ē to illustrate thecorrespondence between the gaze points and the gazeprobability. The true gaze points are obtained using acalibration-based gaze estimator.
True positive rate

6

We compare the gaze probability maps with the original
raw saliency maps to assess the correlation improvementwith the actual gaze points. Fig. 5 shows the correlation improvement using a receiver operating characteristic(ROC) curve. We sweep the threshold value for the gazeprobability maps and raw saliency maps to obtain the plots,
and assess all the ground truth gaze points that we obtainthrough the experiment. The horizontal axis represents thefalse positive rate, i.e., the rate of the pixels in a map abovea threshold value. The vertical axis is the true positive rate,
which indicates the rate of frames whose saliency value atthe gaze point is greater than the threshold. The area under
the curve (AUC) of the gaze probability maps is 0.93, andthat of the raw saliency maps is 0.82. This result shows thatthe correlation is significantly enhanced by the aggregation
process.

1.0

2.3

0.8
0.6
Gaze probability maps

0.4

Raw saliency maps

0.2
False positive rate
0.0
0.0

0.2

0.4

0.6

0.8

1.0

Fig. 5. ROC curves of raw saliency maps and gazeprobability maps. The horizontal axis indicates thefalse positive rate, i.e., the rate of the pixels above athreshold. The vertical axis indicates the true positive
rate, i.e., the rate of the frames that have a higher
saliency value than the threshold at the true gaze point.
The thin line (AUC = 0.82) indicates the performanceof the raw saliency maps obtained by the processdescribed in Section 2.1. The bold line (AUC = 0.93)
indicates the performance of the gaze probability maps
described in Section 2.2.

pointed out that the contribution of each feature is not
uniform, and there is a certain degree of data dependency.
We use a feedback scheme to optimally adjust the weightparameters to address these issues. The feedback scheme isdiscussed in Section 3.
Fig. 4 shows examples of the obtained gaze probability
maps p̄ for six people. The eye images shown at the topleft of each sub-figure indicate the corresponding average
eye images ē, and the overlaid dots indicate the actual
gaze points of ē. Note that ē is a prototype of the eye
images synthesized through the above process, and theactual gaze points are unknown. Therefore, we used theestimates from the appearance-based gaze estimator using
explicit calibration, which is described in Section 4, toobtain the true gaze points as a reference. Although thegaze probability maps p̄i are generated without knowing
the actual gaze points, they have a significant correlation
with the actual gaze points.

Estimator Construction

In the previous step, M average eye images {ē1 , . . . , ēM }and corresponding gaze probability maps {p̄1 , . . . , p̄M } are
obtained. This section describes our method for creatinga gaze estimator using them as the training dataset. Our
goal is to establish a mapping from the eye image togaze points. We develop a method based on Gaussian
process regression that has been successfully applied toboth appearance-based [28] and model-based [29] gazeestimation to efficiently achieve this task.
With the standard Gaussian process regression framework, an estimator is built to output the probability distribution P (g ∗ |e∗ , Dg ) of an unknown gaze point g ∗from an eye image e∗ , given the labeled training data
Dg = {(g1 , ē1 ), . . . , (gM , ēM )}, which consists of the eye
images and corresponding gaze points. In our case, however, we only know Dp = {(p̄1 , ē1 ), . . . , (p̄M , ēM )} where
the gaze probability map p̄i only provides the probability
distribution of the gaze point gi . Therefore, instead of
directly applying the standard Gaussian process regression,
we work on a marginalized probability where explicit
training labels are not required.
After normalizing the gaze probability maps, we define
the gaze probability distribution P (g|p̄) asp̄(g)P (g|p̄) = P P
x

y p̄

,

(12)

where p̄(g)indicates the value of p̄ at the gaze point
P P
g, andxy p̄ is the overall summation of p̄. In theabove equation, we describe the estimation of a onedimensional scalar g to simplify the notation, but two
regressors are independently built for each X- and Y direction in the actual implementation. Using Eq. (12),the target distribution P (g ∗ |e∗ , Dp ) can be obtained bymarginalizing over all the possible combinations of M gazepoints D̂g = {ĝ1 , . . . , ĝM } asX
P (g ∗ |e∗ , Dp ) =P (g ∗ |e∗ , D̂g )P (D̂g |Dp ),
(13)
D̂g

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

where
P (D̂g |Dp ) =MYP (ĝi |p̄i ).

(14)

i

In Eq. (13), g ∗ indicates an unknown gaze point associatedwith the eye image e∗ , and ĝi is a candidate of the gazepoint that corresponds to ēi .
However, the summations of the Eq. (13) are computationally expensive. Various approximation techniques
for the Gaussian process regression [30] can reduce thecomputational cost of computing P (g ∗ |e∗ , D̂g ), however
they cannot directly approximate Eq. (13) itself. For
these reasons, we solve Eq. (13) using a Monte Carloapproximation. We randomly produce ng sets of samng
(l)
(l)
(l)
ples Dg = {(g1 , ē1 ), . . . , (gM , ēM )}l=1
according to theprobability distribution defined by Eq. (12). In particular,(l)gi in the l-th set is generated according to the distribution
P (gi |p̄i ) defined by the i-th probability map. It has been
experimentally shown in Fig. 5 that gaze probability maps
have a high correlation with the actual gaze points. From
this observation, we discard the low saliency values fromthe gaze probability maps to reduce the number of samples.
We use a threshold τs to set the probability to zero if p̄(x, y)
is lower than the threshold. Using these sets, Eq. (13) canbe approximated as∗

is Ki∗ = k(ēi , e∗ ), and Gl ∈ R1×M is a vector of the(l)(l)
gaze points, where its i-th element is Gi = gi . As a
∗ ∗result, the distribution P (g |e , Dp ) can be estimated as a
Gaussian distribution N (µ, σ 2 ) with
µ=

(15)

Gaussian Process Regression.
We assume a noisy observation model for a gaze pointgi = f (ei ) + i , i.e., a gaze point gi is given as a functionof the eye image ei with a noise term i = N (0, γ 2 ςi2 ).
The data-dependent noise variance γ 2 ςi2 is defined as being
proportional to ςi2 , which is an actual variance of the gener(n )(1)ated samples {gi , . . . , gi g }. It explicitly assigns a higher
noise variance for samples from ambiguous saliency mapsthat have several peaks. The function f (ei ) is assumed to
be a zero-mean Gaussian with a covariance function k:
2k(ei , ej ) = α exp(−β ||ei − ej || ),

l=1

l=1

(19)

Tuning Hyperparameters.
There are three hyperparameters, α, β, and γ, in theabove formulation, which need to be optimized for eachdataset. We use a cross-validation approach [31] for the
optimization in our method. The optimal parameters canbe estimated by maximizing a leave-one-out log predictive
probability L defined as
L({Dg(l) }, θ) =

ng M
X
X

(l)

log p(gi |{Dg(l) }−i , θ),(20)

l=1 i=1

where θ is a set of hyperparameters θ = {α, β, γ}, and(l){Dg }−i is a set of generated samples that excludes the
samples with the i-th eye image. The predictive probability(l)(l)
p(gi |{Dg }−i , θ) is defined as a Gaussian function aslog p(gi |{Dg(l) }−i , θ) =(l)

1 (gi − µ−i )21
1 2−− log 2π, (21)
− σ−i
2
2
2
2σ−i
2

(l)

Finally, P (g ∗ |e∗ , Dg ) for each l is computed based on theGaussian process regression as follows [30].

2ng

The variance σ 2 is simply the same as σ12 , because σl2 of
Eq. (18) is actually independent of the index l.

l=1

2

ng

1 X
1 X 2
µl , σ 2 =σl = σ12 .
ng
ng

(l)

ng

1 X
P (g ∗ |e∗ , Dg(l) ).
P (g |e , Dp ) =
ng
∗

7

2
where µ−i and σ−iare the estimated mean and variance(l)using the sample set {Dg }−i .
In our case, however, the actual gaze points of the eye(l)images in the samples {Dg } also have a center bias forthe same reason discussed in Section 2.2. This results in
fewer samples in the peripheral region, and the errors inthis region become overly small. When Eq. (20) is directly
minimized, the optimization result also tends to be biased
to the center. Therefore, we modify Eq. (20) to remove thecenter bias by normalizing the predictive probability as

L=

(16)

with hyperparameters α and β. With this assumption,(l)
P (g ∗ |e∗ , Dg ) is derived as a Gaussian distribution
N (µl , σl2 ) that has a mean µl and variance σl2 :µl = K ∗ K −1 G(l) ,(17)

σl2 = k(e∗ , e∗ ) − K ∗ K −1tK ∗ ,

(18)

and

where the matrix K ∈ RM ×M is a covariancematrix where its (i, j)-th element is defined as
Kij = k(ēi , ēj ) + γ 2 ςi2 δij . The vector K ∗ ∈ R1×M
represents a covariance vector of the input eye
image and average eye images, whose i-th element

ng M
X
X
l=1

1

n
i=1 (i,l)(l)

log p(gi |{Dg(l) }−i , θ),(22)

where n(i,l) is the total number of samples that have the(l)same gaze points as gi . By evaluating the average errors ofeach gaze point, Eq. (22) can evaluate the estimation errors
in an unbiased manner on the display coordinates. Using
partial derivatives with respect to the hyperparameters,
Eq. (22) is maximized via a conjugate gradient method.
The readers are referred to [30] for a detailed derivation of
the partial derivatives.
2.4

Gaze Estimation

Once we have matrices K, S, and {G(1) , . . . , G(ng ) } in
Eqs. (17) and (18), a gaze point can be estimated by taking
a new eye image e as an input. The estimated distributions

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

Gaze
estimator

Error minimization

Fig. 6. Illustration of feature weight optimization. Target
attention maps {a} are generated based on leave-oneout estimates, and feature weights are optimized byminimizing a sum of squared residuals between target
maps and sum maps {p̄}.

between the combined saliency maps and gaze points isrefined using this feedback loop.
Once the gaze estimator is built (Section 2.3), it canbe used to estimate the gaze points from the associatedaverage eye images ē in Dp . Using this dataset, ourmethod optimizes the feature weights so that the correlation
between the peak of the gaze probability map p̄ and thegaze point estimate is higher.
Our approach is illustrated in Fig. 6. As described in
Section 2.3, leave-one-out estimates {µ−1 , . . . , µ−M } are
obtained for each average eye image in both the X- and Y directions. Using these estimated gaze coordinates, target
attention maps {a1 , . . . , aM }, which represent the topdown gaze point distribution, are generated by drawing
circles at the gaze points with a fixed radius. Since theseleave-one-out estimates can include a large error, the radius
is set to a relatively larger value (∼ 4 degrees in our current
implementation) than the central area of human vision (∼ 1
degree). Using the target attention maps a, the feature
weights ω = t(ω1 , . . . , ω6 ) are optimized by minimizing
the sum of the squared residuals as

for each X- and Y-direction, N (µx , σx2 ) and N (µy , σy2 ),are converted to the display coordinates N (µ̂x , σ̂x2 ) andN (µ̂y , σ̂y2 ) byµ̂x = xo +WIHIµx , µ̂y = yo +
µy ,
Ws
Hs

(23)

HI 2
WI 2
σ , σ̂y2 =
σ ,
Ws x
Hs y

(24)

8

ω = arg minωMX||ai −

i=1

6
X

(f )

ωf p̄i ||2 ,

(25)

f =1

with a non-negativity constraint
ω ≥ 0.

(26)andσ̂x2 =

where Ws , Hs indicates the width and height of thesaliency maps, WI , HI indicates the actual width andheight of the displayed images {I1 , . . . , IN }, and (xo , yo )indicates the display origin of the images. The average(µ̂x , µ̂y ) corresponds to the estimated gaze point g. The
estimated variances are not directly used in our current
system, however the estimation accuracy will be improvedby incorporating techniques such as the Kalman filter asin [28].

3

F EATURE W EIGHT O PTIMIZATION

The previous section describes the complete pipeline of
the training the gaze estimator and testing method. In thesaliency aggregation step (Section 2.2), all six saliency
features {p̄(f ) } are independently aggregated, and the aggregated maps are linearly combined by Eq. (11) to produce
the summed map p̄.
Although most existing methods use equal weights for
simplicity [11], [14], [26], how to optimally tune theweights to achieve an even higher correlation between thevisual saliency and gaze points remains unclear. Somestudies use a data-driven learning approach to optimallytune the weight parameters using known gaze points [18],[19], [20], [21]. In our scenario, we do not have accessto ground truth gaze points. Thus, the feature weights arerefined in our method by using the estimated gaze points
using the data-driven learning approach. The correlation

To reduce the number of equations in Eq. (25), na points
are randomly sampled from both the positive and zeroregions in all the attention maps a. The matrix form ofEq. (25) can be written asω = arg minωMX||A − P̄ ω||2 ,

(27)

i=1

where A ∈ R2M na ×1 is a vector that consists of values
at the selected points and P̄ ∈ R2M na ×6 contains thecorresponding feature values in each row. Eq. (27) issolved by using the non-negative least-squares algorithmof Lawson and Hanson [32] to obtain the optimal set offeature weights.

4

E XPERIMENTAL R ESULTS

In this section, we present our experimental results toevaluate our method. We use a set of 80 video sources inthe experiments that are downloaded from the Vimeo website [33], which include various types of video clips, e.g.,music videos and short films. Some example frames areshown in Fig. 7. 30-second video sequences are randomly
extracted from each video source without an audio signal,
and resized to a fixed resolution of 960 × 540. These 80
short clips are divided into four datasets A, B, C, D, andseven novice test subjects t1 , . . . , t7 are asked to watch allof them. The video clips are shown at 25 fps; therefore, thenumber of frames is N = 15000 in each set, and the display
resolution is set to WI = 1920 and HI = 1080. Although
it highly depends on the algorithms and can be done prior

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

9(a)Fig. 7. Examples of video clips used in experiments.
We use a set of 80 video clips downloaded from theVimeo website [33]. All the pictures are licensed undera Creative Commons License.2
to the recording session of the eye images, the most time
consuming part of the proposed framework is the saliency
extraction step. For an efficient computation, the saliency
maps are calculated at a smaller resolution, Ws = 32 andHs = 18, in our experiments. One pixel corresponds toabout 1.35 × 1.35 degrees in our current setting, which is
close to the limit of the central area of human vision.
Throughout the experiment, the parameters are set
at ns = 5, τe = 0.45, Ts = −300, αe = 0.008, nf = 3,
τf = 0.2, ng = 1000, na = 40, and τs is adaptively set
to retain the top 15% of pixels and the remaining 85%
are set to zero in each map. These parameter settings are
empirically obtained from our experiment. In our current
implementation, when M ' 100, it took about 1 minutefor parameter optimization, and 1 millisecond per frame for
estimation using a 3.33-GHz Core i7 CPU with a simple
code parallelization using OpenMP [34].
4.1

Experiment Details

A chin rest is used during the experiments to fix the peoples’ head positions, and a 23-inch Full HD (508.8 × 286.2mm) display is placed about 630 mm away from the subject
to show the video clips. A VGA-resolution camera is placed
under the display to capture eye images.
We use the OMRON OKAO Vision library to detect thecorners of each eye. We use template matching around thedetected corners of each eye to ensure alignment accuracy.
Small template images of the corners of each eye areregistered during the initialization, and the locations withthe highest normalized cross-correlation are used as thealigned corner positions. Based on the aligned positions
as illustrated in Fig. 8 (a), the eye images are cropped to a
fixed size of 70 × 35 pixels.
The eye images are histogram-equalized and pixels withintensity lower than the given threshold are truncated tozero so that images contain only eye regions (Fig. 8 (b)) tominimize effects caused by lighting changes. The thresholdvalue is automatically decided using Otsu’s method [35].
2. From top to bottom, left to right: “The Eyewriter” by Evan Roth(http://vimeo.com/6376466), “Balloons” by Javi Devitt (http://vimeo.com/
10256420), “Tenniscoats - Baibaba Bimba — A Take Away Show” by La
Blogotheque (http://vimeo.com/11046286), “Persona” by superhumanoids(http://vimeo.com/13848244), “MADRID LONGBOARD” by Juan Rayos(http://vimeo.com/12132621), and “MATATORO” by Matatoro Team(http://vimeo.com/13487624).

(b)Fig. 8. Examples of eye images. (a) The eye images
are cropped to a fixed size based on the detected
positions. (b) The images are histogram-equalized andthresholded so that they contain only the iris and eye
contours.
Average Error [mm][deg.]

60

Tobii TX300
Calibrated estimator

50

5.4
4.5

40

3.6

30

2.7

20

1.8

10

0.9

0

0.0

t1

t2

t3

t4

t5

t6

t7 Average
Test subject

Fig. 9. Error comparison of commercial gaze estimator(Tobii TX300) and calibrated estimation method. A
target point (the ground truth) is explicitly displayed
to the test subjects, and the estimation accuracy isevaluated by assessing the deviation from the groundtruth.

Finally, we apply a discrete Fourier transform to obtainthe 4900-dimensional feature vectors e, which consists ofFourier magnitudes of both the left and right eye images.
Ground truth.
We use a Tobii TX300 gaze tracker [36] to obtain the
ground truth to quantitatively assess the effectiveness of
our proposed method. The Tobii gaze tracker is placed
within our setup and run in parallel with our method to
obtain the ground truth gaze points. In addition, we also
run a standard appearance-based gaze estimation methodthat uses an explicit calibration (in short, what we calla calibrated method hereafter) as a baseline method forfurther assessing our method. We show 16 × 9 reference
points for each test subject at a regular interval on thedisplay to train the estimator of the calibrated method. The
eye images are recorded during the calibration to establish
the mapping between the eye image and the gaze points.
Once the pairs of reference gaze points and eye images are
obtained, a learning process, which is the same manneras described in Section 2.3, is performed to obtain the
mapping.
It is important to discuss the accuracy of the reference
method that we use as the ground truth. The catalog
specification of the accuracy of the Tobii TX300 is less than
1 degree. However, the error can be larger depending on
the test subjects and installation conditions. We conduct a
preliminary experiment using our setting to verify the actual
accuracy of both the Tobii gaze tracker and the calibrated
gaze estimator. A total of 120 target points are randomlyIEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

shown on the display to each of the seven test subjects.
The subjects are asked to look at these target points, andwe assess the gaze estimation accuracy using the target
points as the ground truth gaze points. Fig. 9 shows theaverage distance errors between the ground truth target
points and the estimated gaze points by the commercial
and calibrated gaze estimators. As shown in the plot, theestimation accuracy depends on the subjects in our setting.
The average errors are similar in these two approaches;
it is about 30 mm (' 2.7 degrees) with the commercial
estimator, and 25 mm (' 2.3 degrees) with the calibrated
estimator. We observe that there is a fundamental limit in
the accuracy evaluation of a gaze estimator from theseexperiments. As described above, we use the output of
the commercial gaze estimator as the ground truth becauseof its availability to the readers and reproducibility in theexperiments.
4.2

Gaze Estimation Result

We examine the performance of the proposed method byusing the following procedure. First, we use the entire
dataset for both training and testing to assess the upperbound accuracy of the proposed method, i.e., the training
data performance. Second, we divide the dataset into two,
one for training and the other for testing, to evaluate thegeneralizability of the proposed gaze estimator, i.e., the test
data performance.
Performance evaluation.
We first assess the performance, where the same dataset
is used for both the training and testing. We perform thisevaluation using four dataset A, B, C, D independently. The
estimation results are summarized in Table 1. Each row corresponds to the result using dataset A, B, C, D, where all
20 video clips are used for both the training and testing. The
first two columns indicate the AUCs of the average ROC
curves of the raw saliency maps s and the gaze probability
maps p̄. The remaining columns list the estimation errorsof the proposed method and the calibrated appearancebased estimator for both the distance and angular errors.
The errors are described by their (average ± standard
deviation) form. The distance error is evaluated by using the
Euclidean distance between the estimated and the groundtruth gaze points, and the angular errors are computed using
the distance between the eyes and the display.
Similarly, Table 2 lists the estimation error of eachsubject t1 , . . . , t7 . Each row corresponds to the averageof the results of the corresponding test subject using thefour datasets A, B, C, D. The columns list the AUCs andestimation errors in the same manner as in Table 1. The
overall average error is 39 mm (' 3.5 degrees). It can beseen from Table 1 and Table 2 that the performance does
not have a strong dependency on the dataset and subjects.
Although our method has a larger error than the calibrated
estimator, our method can still achieve an accuracy of 3.5
degrees, which is sufficient for obtaining the regions ofattention in images.

10Error [mm]
75

A
B
C
D

65

[deg.]
6.8
5.9

55

5.0

45

4.1

35

3.2
Number of Training Video Clips

25

5

10

15

2.3

20

Fig. 10. Estimation errors w.r.t. different amounts oftraining video clips. Each point is plotted by choosing
a limited number of learning video clips from the corresponding dataset, and the average error is computedfrom the results from the seven test subjects.
Performance on training data

Error [mm]Performance on test data

60

[deg.]
5.4

50

4.5

40

3.6

30

2.7

20

1.8

10

0.9
0.0

0
A

B

C

D
DatasetFig. 11. Comparison of two estimation methods. Allfour datasets are divided into two subsets. One subset
is used to train the gaze estimators, and the estimation accuracy is independently examined using eachsubset. The dark bars represent the training data performance, and the light bars represent the test data
performance.

Performance variation w.r.t. the amount of trainingdata.
We conduct a test using varying amounts of training
video data to analyze the performance variations withrespect to the amount of training data. Fig. 10 shows acomparison of the estimation errors for different amountsof training video clips. The X-axis represents the numberof video clips used from each dataset. The Y-axis shows theaverage error that is computed from all seven test subjects.
For this evaluation, five video clips are randomly selectedfrom the training dataset and used as the test data. The
result shows that the larger amount of training data results
in a more accurate estimation result, though improvement
slows after 10 training clips.
Performance on test data.
We use the test data sampled from the training dataset
to verify the upper-bound on the accuracy in the above
experiments. Our gaze estimator can also be applied tounseen video clips after training. We perform a test by
separating the training and test datasets to evaluate thegeneralizability of the proposed method. Each of the four
dataset A, . . . , D in this test are divided into two subsets

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

11

TABLE 1
Average error for each dataset. Two AUC columns show the AUCs of the average ROC curves of the raw
saliency maps s and the gaze probability maps p̄. The remaining columns are the distance and angularestimation errors (average ± standard deviation) when using the two estimation methods.
DatasetA
B
C
D
Average

s
AUC
0.80
0.83
0.81
0.83
0.82

p̄
AUC
0.92
0.95
0.94
0.91
0.93

Proposed method
error [mm]error [deg.]
41 ± 26
3.7 ± 2.3
36 ± 23
3.3 ± 2.1
41 ± 25
3.7 ± 2.3
36 ± 25
3.3 ± 2.3
39 ± 25
3.5 ± 2.3

Calibrated method
error [mm]error [deg.]
33 ± 153.0 ± 1.4
24 ± 132.2 ± 1.2
27 ± 152.5 ± 1.4
34 ± 163.1 ± 1.530 ± 152.7 ± 1.4

TABLE 2Average error of each subject. The columns indicate the AUCs of the average ROC curves and estimation
errors in the same manner as in Table 1.
Subject
t1
t2
t3
t4
t5
t6
t7

s
AUC
0.80
0.79
0.83
0.81
0.84
0.83
0.82

p̄
AUC
0.93
0.92
0.93
0.94
0.92
0.94
0.90

Proposed method
error [mm]error [deg.]
41 ± 28
3.7 ± 2.6
41 ± 27
4.0 ± 2.4
33 ± 223.0 ± 2.042 ± 24
3.8 ± 2.2
35 ± 23
3.2 ± 2.1
36 ± 223.2 ± 2.0
39 ± 27
3.6 ± 2.5

Calibrated method
error [mm]error [deg.]
29 ± 162.6 ± 1.4
30 ± 142.7 ± 1.3
34 ± 153.1 ± 1.4
30 ± 142.7 ± 1.3
27 ± 152.5 ± 1.4
24 ± 122.2 ± 1.1
33 ± 18
3.0 ± 1.7

Error [mm]Proposed method

60

Fig. 12. Gaze estimation results. The estimation results of our method are rendered as 2-D Gaussian
circles. The corresponding input eye images are shownat the top-left corner. The overlaid cross shapes represent the ground truth gaze points obtained by the commercial gaze tracker (Tobii TX300), and the solid circles
indicate the gaze points obtained from the calibrated
estimator.

that consist of 10 video clips each. One subset is used
to train our gaze estimator, and the other subset is used
for testing. At the same time, we evaluate the performanceusing the original training data as the test data for a comparison. Fig. 11 shows the comparative results of these two
estimation scenarios. The dark bars in the figure indicate
the training data performance, and the light bars correspondto the test data performance. In most of the datasets,
the training data performance showed the higher expected
accuracy. However, the test data performance is comparable
to it without any significant performance degradation.
Fig. 12 shows some examples of the gaze estimation
results. The output of our method is rendered as a 2D Gaussian circle centered at (µ̂x , µ̂y ) with a variance(σ̂x2 , σ̂y2 ) given by Eq. (19), and the mean coordinate(µ̂x , µ̂y ) is used as the estimated gaze point. The input eye

Sample-wise optimization[deg.]
5.4

50

4.5

40

3.6

30

2.7

20

1.8

10

0.9

0

t1

t2

t3

t4

t5

t6

t7

Average

0.0

Test subject

Fig. 13. Comparison of hyperparameters tuning
method. The dark bars indicate the estimation errors
after optimization using Eq. (22), and the light bars
indicate the optimization results using Eq. (20). The
proposed method without using Eq. (22) results inlower errors.
images are shown at the top-left corner. The overlaid crossshape represents the ground truth gaze point obtained by
the commercial gaze tracker, and the solid circle shows thegaze point estimated by the calibration-based estimator.
4.3

Effects of Parameter Optimization

In this section, we quantitatively evaluate the effectivenessof our parameter optimizations. We first assess the effectiveness of the hyperparameter tuning described in Section 2.3,
and second, we evaluate the feature weight optimization
described in Section 3.
Hyperparameter tuning results.
The effectiveness of the hyperparameter tuning of theGaussian process regression is summarized in Fig. 13.
The dark bars indicate the estimation errors after ourIEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

A

12Error [mm]B

Optimized weights

60

Equal weights

50

Face
Color
Flicker
Intensity
MotionOrientation

C[deg.]
5.4
4.5

40

3.6

30

2.7

20

1.8

10

0.9

D

0.0

0
A

Fig. 14. Optimized weights of saliency features. The
pie graph shows the ratio of the optimized feature
weights averaged over all test subjects.
NSS

B

C

D
DatasetFig. 16. Error comparison between before and afterfeature weight optimization. Optimized weights led toslightly reduced error for 3 of 4 datasets.

Optimized weights

3.5

Equal weights

3.0
2.5
2.0
1.5
1.0
0.5
0.0
A

B

C

D
DatasetFig. 15. Normalized Scanpath Saliency (NSS) scoresof gaze probability maps. Dark bars indicate NSS
values using optimized weights, and light bars indicate NSS values using initial equal weights. Optimized
weights always outperform equal weights.

optimization using Eq. (22), and the light bars show the
optimization results using Eq. (20), which corresponds tothe standard formulation used to evaluate the sample-wiseerror. Our optimization method reduces the error by about
0.5 degrees.
Feature weight optimization results.
This experiment assesses the effectiveness of the feature
weight optimization by our feedback loop, where the feature weight is updated to further enhance the accuracy of
the gaze estimation. The original feature weights are all
set to one. After feature weight optimization, the ratio of
the feature weights is optimized, as shown in Fig. 14. The
pie graph shows the ratio of the average weight computedfrom the data from all test subjects. After optimization, faceand orientation features have greater weight compared withthe others. This result is consistent with the report given by
Zhao et al. [21], which optimized the feature weights using
known gaze points.
We compute the Normalized Scanpath Saliency(NSS) [37] to evaluate the correlation between the gazeprobability map and the estimated gaze point. The original
definition of NSS is the normalized average of the saliency
values at the fixation locations, i.e., the saliency maps
are linearly normalized to have a zero mean and a unit
standard deviation, and the NSS is computed as the averageof the saliency values at the fixated positions. Therefore,a higher NSS score indicates a greater correlation. In
our case, instead of using the fixation locations, we use
the true gaze points that are associated with the average
eye images to compute the NSS. Given all the sets ofaverage eye images and the gaze probability maps, wefirst compute the ground-truth gaze points. Since theaverage eye images are synthesized features as discussed
in Section 2.2, we use the estimates from the calibrated
method instead of the commercial gaze tracker. Using
the ground-truth gaze points, we compute the NSS using
the normalized gaze probability maps. Fig. 14 shows thesummary of the results; dark bars indicate NSS values
after optimization, and light bars indicate NSS valuesof initial equal weights. NSS scores are improved after
optimization for all datasets.
Fig. 16 shows an error comparison between the before
and after feature weight optimization. Dark bars indicate
errors after feature weight optimization, and light bars
represent errors when initial equal weights are used. Whilethe improvement is rather small, the optimized weights
yield a higher accuracy in 3 out of 4 datasets. One possible
explanation for this small improvement would be thatthe aggregated feature maps p̄(f ) are already sufficiently
accurate for predicting gaze points, and the estimation
accuracy is almost saturated in our setting. However, thisimprovement suggests that it is useful to incorporate thefeedback loop for saliency optimization, especially whenmore complex saliency map models with many features areincorporated.
4.4

Spatial Bias in Error

The accuracy of our method depends on the distribution of
the training samples. Fig. 17 shows the spatial distributionof the average estimation errors in the display coordinate. In
the figure, each grid corresponds to the ground truth gazelocation, and the magnitude and direction of the average
estimation errors are rendered. In Fig. 17 (a), the higher
intensity indicates a greater magnitude of estimation errors.
In (b), the error directions are color coded, where eachcolor corresponds to a certain direction that is illustrated
in the reference circle, and a higher saturation indicates agreater error magnitude, like in Fig. 17 (a). There exists a

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE(a) Magnitude

[mm]
160

8
(b) Direction

Fig. 17. Spatial distribution of estimation errors in
display coordinate: (a) higher intensity corresponds togreater numbers of estimation errors, and (b) color
wheel indicates error directions while saturation indicates error magnitudes.
Average saliency map

Distribution of gaze points

Fig. 18. Average saliency map and spatial histogram ofgaze points. The image on the left shows the averageof all the raw saliency maps extracted from the four
video clips used in the experiment. The image on theright shows the spatial histogram of the ground truth
gaze points from the experimental dataset. A higher
intensity corresponds to larger counts of gaze points.

our method automatically establishes the mapping fromthe eye image to the gaze point using video clips. Taking
a synchronized set of eye images and video frames, ourmethod trains the gaze estimator by regarding the saliency
maps as the probabilistic distributions of the gaze points.
In our experimental setting with fixed head positions, ourmethod achieves an accuracy with about a 3.5-degree error.
We took an appearance-based gaze estimation approach.
Appearance-based methods have a significant benefit in thatthey can be constructed using only a monocular camera
without requiring a specialized hardware device. However,
one of the biggest technical challenges common among
existing appearance-based methods is the difficulty in handling the head pose movements. This is mainly due tothe fact that allowing a head pose movement significantly
expands the space of the training samples, and thus, thetraining becomes more difficult. There is some effort put
forth to handle the head pose variations in an appearancebased setting [9], [38], and our future work includes
adopting these techniques to allow head pose movement.
In addition, as shown in [25], it is an alternative future
direction to incorporate our approach in a model-based gazeestimation framework to further improve the estimation
accuracy.
Naturally, the estimation accuracy of our method depends
on the quality of the raw saliency maps extracted from the
input video clips. The human gaze control mechanism is
not yet completely understood, and there is a wide range ofpossibilities for more advanced saliency models to be usedin our method to improve the gaze estimation accuracy.

ACKNOWLEDGMENTS
This research was supported by CREST, JST.

R EFERENCES[1]

systematic bias, i.e., large errors are observed around the
peripheral areas, and the error directions are biased toward
the center of the display.
Fig. 18 shows the average saliency map and spatial
histogram of the gaze points. The image on the left shows
the average of all raw saliency maps extracted from alldatasets used in our experiment. The image on the rightshows the spatial histogram of the ground truth gaze points
obtained from the same datasets, in which the brighter
intensity represents a higher frequency. Since salient objects
are typically located near the centers of video frames, theaverage for the saliency maps is lower around the display
boundary. The actual gaze points also tend to concentratearound the center of the display. As a result, the numberof learning samples at the display edges is limited, and abias in the estimation accuracy exists because of this.

5

C ONCLUSION

We propose a novel gaze estimation framework in thispaper that auto-calibrates by using saliency maps. Unlike
the previous approaches that require an explicit calibration,

13

[2]
[3]
[4]

[5]

[6]

[7]

[8]

D. W. Hansen and Q. Ji, “In the eye of the beholder: A survey ofmodels for eyes and gaze,” IEEE Transactions on pattern analysis
and machine intelligence, vol. 32, no. 3, pp. 478–500, 2010.
R. J. K. Jacob, “Eye tracking in advanced interface design,” in Virtualenvironments and advanced interface design, W. Barfield and T. A.
Furness, Eds. Oxford University Press, 1995, pp. 258–288.
T. Ohno, “One-point calibration gaze tracking method,” in Proceedings of the 2006 symposium on Eye tracking research & applications(ETRA ’06), 2006, pp. 34–34.
E. D. Guestrin and M. Eizenman, “Remote point-of-gaze estimation
requiring a single-point calibration for applications with infants,” inProceedings of the 2008 symposium on Eye tracking research &applications (ETRA ’08), 2008, pp. 267–274.
A. Villanueva and R. Cabeza, “A novel gaze estimation system with
one calibration point,” IEEE Transactions on Systems, Man, andCybernetics, Part B: Cybernetics, vol. 38, no. 4, pp. 1123–1138,
2008.
T. Nagamatsu, J. Kamahara, T. Iko, and N. Tanaka, “One-pointcalibration gaze tracking based on eyeball kinematics using stereo
cameras,” in Proceedings of the 2008 symposium on Eye tracking
research & applications (ETRA ’08), 2008, pp. 95–98.
E. D. Guestrin and M. Eizenman, “General theory of remote gazeestimation using the pupil center and corneal reflections,” Biomedical
Engineering, IEEE Transactions on, vol. 53, no. 6, pp. 1124 –1133,
2006.
H. Yamazoe, A. Utsumi, T. Yonezawa, and S. Abe, “Remote gazeestimation with a single camera based on facial-feature tracking
without special calibration actions,” in Proceedings of the 2008
symposium on Eye tracking research & applications (ETRA ’08),
2008, pp. 245–250.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE[9]

Y. Sugano, Y. Matsushita, Y. Sato, and H. Koike, “An incremental
learning method for unconstrained gaze estimation,” in Proceedingsof the 10th European Conference on Computer Vision (ECCV 2008),
2008, pp. 656–667.
[10] C. Koch and S. Ullman, “Shifts in selective visual attention: towards
the underlying neural circuitry,” Human neurobiology, vol. 4, no. 4,
pp. 219–227, 1985.
[11] L. Itti, C. Koch, and E. Niebur, “A model of saliency-based visual
attention for rapid scene analysis,” IEEE Transactions on patternanalysis and machine intelligence, vol. 20, no. 11, pp. 1254–1259,
1998.
[12] C. Privitera and L. Stark, “Algorithms for defining visual regionsof-interest: Comparison with eye fixations,” IEEE Transactions on
pattern analysis and machine intelligence, vol. 22, no. 9, pp. 970–
982, 2000.
[13] L. Itti and P. F. Baldi, “Bayesian surprise attracts human attention,” inAdvances in Neural Information Processing Systems, Vol. 19 (NIPS
2005), 2006, pp. 547–554.
[14] J. Harel, C. Koch, and P. Perona, “Graph-based visual saliency,” inProceedings of Advances in neural information processing systems(NIPS 2007), vol. 19, 2007, pp. 545–552.
[15] N. D. B. Bruce and J. K. Tsotsos, “Saliency, attention, and visual
search: An information theoretic approach,” Journal of Vision, vol. 9,
no. 3, pp. 1–24, 2009.
[16] D. Parkhurst, K. Law, and E. Niebur, “Modeling the role of saliencein the allocation of overt visual attention,” Vision research, vol. 42,
no. 1, pp. 107–123, 2002.
[17] A. C. Schütz, D. I. Braun, and K. R. Gegenfurtner, “Eye movements
and perception: A selective review,” Journal of Vision, vol. 11, no. 5,
2011.
[18] W. Kienzle, F. A. Wichmann, B. Scholkopf, and M. O. Franz, “A
nonparametric approach to bottom-up visual saliency,” in Proceedings of Advances in neural information processing systems (NIPS
2006), vol. 19, 2006, pp. 689–696.
[19] W. Kienzle, B. Scholkopf, F. A. Wichmann, and M. O. Franz, “Howto find interesting locations in video: a spatiotemporal interest point
detector learned from human eye movements,” in Proceedings ofthe 29th Annual Symposium of the German Association for Pattern
Recognition (DAGM 2007), 2007, pp. 405–414.
[20] T. Judd, K. Ehinger, F. Durand, and A. Torralba, “Learning topredict where humans look,” in Proceedings of the Twelfth IEEE
International Conference on Computer Vision (ICCV 2009), 2009.
[21] Q. Zhao and C. Koch, “Learning a saliency map using fixatedlocations in natural scenes,” Journal of Vision, vol. 11, no. 3, 2011.
[22] E. Horvitz, C. Kadie, T. Paek, and D. Hovel, “Models of attention
in computing and communication: from principles to applications,”
Communications of the ACM, vol. 46, no. 3, pp. 52–59, 2003.
[23] R. Vertegaal, J. Shell, D. Chen, and A. Mamuji, “Designing for augmented attention: Towards a framework for attentive user interfaces,”
Computers in Human Behavior, vol. 22, no. 4, pp. 771–789, 2006.
[24] Y. Sugano, Y. Matsushita, and Y. Sato, “Calibration-free gaze sensing
using saliency maps,” in IEEE Computer Society Conference on
Computer Vision and Pattern Recognition (CVPR2010), 2010, pp.
2667–2674.
[25] J. Chen and Q. Ji, “Probabilistic gaze estimation without active
personal calibration,” in IEEE Computer Society Conference on
Computer Vision and Pattern Recognition (CVPR2011), 2011.
[26] M. Cerf, J. Harel, W. Einhauser, and C. Koch, “Predicting humangaze using low-level saliency combined with face detection,” inProceedings of Advances in neural information processing systems(NIPS 2008), vol. 20, 2008, pp. 241–248.
[27] OMRON OKAO Vsion library, https://www.omron.com/r d/coretech/vision/okao.html.
[28] O. Williams, A. Blake, and R. Cipolla, “Sparse and semi-supervised
visual mapping with the S3 GP,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR2006),
vol. 1, 2006, pp. 230–237.
[29] D. W. Hansen, J. S. Agustin, and A. Villanueva, “Homography
normalization for robust gaze estimation in uncalibrated setups,”
in Proceedings of the 2010 Symposium on Eye-Tracking Research&#38; Applications (ETRA ’10), 2010, pp. 13–20.
[30] C. E. Rasmussen and C. K. I. Williams, Gaussian processes formachine learning. The MIT Press, 2006.
[31] S. Sundararajan and S. S. Keerthi, “Predictive approaches for choosing hyperparameters in gaussian processes,” Neural Computation,
vol. 13, no. 5, pp. 1103–1118, 2001.

14[32] C. L. Lawson and R. J. Hanson, Solving least squares problems.
Society for Industrial Mathematics, 1987.
[33] Vimeo, http://vimeo.com/.
[34] OpenMP, http://openmp.org/.
[35] N. Otsu, “A threshold selection method from gray-level histograms,”
Systems, Man and Cybernetics, IEEE Transactions on, vol. 9, no. 1,
pp. 62 –66, 1979.
[36] Tobii Technology, http://www.tobii.com/.
[37] R. Peters, A. Iyer, L. Itti, and C. Koch, “Components of bottom-up
gaze allocation in natural images,” Vision Research, vol. 45, no. 18,
pp. 2397–2416, 2005.
[38] F. Lu, Y. Sugano, O. Takahiro, and Y. Sato, “A head pose-free
approach for appearance-based gaze estimation,” in Proceedings ofthe 22nd British Machine Vision Conference (BMVC 2011), 2011.

Yusuke Sugano received his B.S., M.S. and
Ph.D. degrees in information science andtechnology from the University of Tokyo in
2005, 2007 and 2010 respectively. He iscurrently a Project Research Associate at
Institute of Industrial Science, the University of Tokyo. His research interests include
computer vision and human-computer interaction.

Yasuyuki Matsushita received his B.S.,
M.S. and Ph.D. degrees in EECS from the
University of Tokyo in 1998, 2000, and 2003,
respectively. He joined Microsoft Research
Asia in April 2003. He is a Lead Researcher
in Visual Computing Group of MSRA. His
major areas of research are computer vision(photometric techniques, such as radiometriccalibration, photometric stereo, shape-fromshading), computer graphics (image relighting, video analysis and synthesis). Dr. Matsushita is on the editorial board member of IEEE Transactions on
Pattern Analysis and Machine Intelligence (TPAMI), International
Journal of Computer Vision (IJCV), IPSJ Journal of Computer Vision
and Applications (CVA), The Visual Computer Journal, and Encyclopedia of Computer Vision. He served/is serving as a Program
Co-Chair of PSIVT 2010, 3DIMPVT 2011 and ACCV 2012. He is
appointed as a Guest Associate Professor at Osaka University (April
2010-) and National Institute of Informatics, Japan (April 2011-). He
is a senior member of IEEE.

Yoichi Sato is a professor at Institute of Industrial Science, the University of Tokyo. He
received his B.S. degree from the Universityof Tokyo in 1990, and his M.S. and Ph.D.
degrees in robotics from School of Computer Science, Carnegie Mellon University
in 1993 and 1997 respectively. His researchinterests include physics-based vision, reflectance analysis, image-based modelingand rendering, and tracking and gesture
analysis. He is currently on the Editorial
Board of International Journal of Computer Vision, IPSJ Journal ofComputer Vision and Applications, and IET Computer Vision. He
also served as Associate Editor of IEEE Transactions on PatternAnalysis and Machine Intelligence. He is a Program Co-Chair of
ECCV 2012.