Improving Eye Tracking Accuracy by Head Motion History
Masahiro TOYOURA† (Member), Takumi TANAKA† , Atsushi SUGIURA† , Xiaoyang MAO† (Member)† University of Yamanashi<Summary> We focus on eye tracking by head motion. This type of eye tracking does not provide themost accurate results, but it does not require a user to wear cumbersome sensors like cameras on glasses. The
approach works for many applications, such as the extraction of human attention by surveillance camera
or an intuitive interface for tablet devices. Through a preliminary experiment, we conﬁrmed that headdirection is often largely diﬀerent from eye direction. We propose to estimate accurate eye direction by using
head motion history. A sequence of head directions and the diﬀerentials provide richer information thanhead direction at one moment. Using multiple regression analysis (MRA) and dynamic coupled component
analysis (DCCA), we examined the relationship between eye direction and head motion history, and reduced
the error rate by 7.2% and 0.8% on average.
Keywords: eye tracking, ego-motion, gyro sensing, visual saliency

1.

Introduction

Eye direction is a promising cue for understanding a

Wearable eye tracker(Answer direction)
NAC Technology
EMR-9

user’s attention. For example, eye tracking data obtainedby surveillance camera images detect human attention for
public signs. The most common technique for estimating
eye direction is capturing images of human eyes1). Cameras installed on a person’s glasses or at the frontal sideof a monitor display are user for the purpose. However,

9D gyro sensor
ATR TSND121
1.4 m

Stimulus images
on 70-inch LCD display

Fig. 1 Environment of experiment

this method forces a user to wear cumbersome devices, indata logging. Figure 1 shows experimental environment.

case of cameras on glasses, or to sit on a chair, in case ofcameras on a monitor display. To understand human at-

The gyro sensor will be replaced by a surveillance camera
or other lighter devices in future works; however, this istention in a real context, the method is not applicable. In
contrast, eye tracking by head motion can be applied for

not the focus of this paper. Also note that head motion
history cannot represent eye direction. It is clear that

crowds on a street. It can be used on infant subjects who

humans can change eye direction without moving theirhesitate to wear glasses or other special devices. However,heads. The contribution of our method is to improve theeye movement estimated with head motion is not as accurate as tracking eye movement with cameras. Sankara-

accuracy of eye tracking by head motion history.
When the directions of the head and the chest are thenarayanan et. al. proposed an eye tracking method forsame, eye direction estimated only from head direction

eye tracking of pedestrians captured in multiple surveillance camera video images2). They assumed that a person’s face direction is the same as his or her eye direction.

at the moment is the same as the direction of the chest.

The assumption is often used for such applications, but

at the next moment. Humans see an object by collabo-

they have not focused on its accuracy.
In this paper, we focus on improving the accuracy ofratively moving their eyes and heads. It is called “eyehead coordination”3). This topic is well discussed in theeye tracking data by head motion. This is a pilot studyof using head motion history, not only head direction at acontext of cognitive psychology. Many experiments havebeen conducted within limited components such as dis-certain moment. We employ a gyro sensor on the head forplacement from the center position. The diﬀerence ofIf the head is directed to the right at the previous moment, we can expect that the eye will direct to the left

individuals should be taken into account for engineering
applications, but it tends to be ignored. In the context
of computer vision or motion estimation, Okinaka et al.4)
also focused on eye-head coordination, and tried to improve the accuracy of eye tracking. They employed the
speed and acceleration of head direction. Our employedhead motion history includes the trajectory of the headdirection in addition to speed and acceleration. A moresophisticated regression model improves the accuracy.
We have proposed a method for improving the accuracyof eye tracking by head motion in5). We further discuss

Fig. 2 Error of eye tracking by head motion
Front
↓
Right

on which stimulus image contributes to more improve theaccuracy of eye tracking in this paper. The distribution

Right
↓
Front

of salient regions in the stimulus image tends to aﬀect thedegree of improvement.

Fig. 3 Diﬀerence of eye direction from diﬀerent headmotion history

We ﬁrst examine the accuracy of eye tracking by headmotion in Section 2. The method for improving the accuracy of eye tracking by head motion history is represented
in Section 3. Experimental results are shown in Section
4, and we conclude the paper and discuss future work in
Section 5.

Convenonal
Head direcon

Eye direcon

ProposedCompareHead direcon2.

Accuracy of Eye Tracking

State model(MRA & DCCA)

We ﬁrst examined the accuracy of eye tracking. As
shown in Figure 1, still stimulus images were displayedon a 70-inch LCD display monitor. A subject stood 1.4m
away from the monitor. Our employed eye tracker NAC
Technology EMR-9 estimates eye direction with an accuracy of 0.1 degrees. The eye tracker has three cameras capture images of two eyeballs and outward. WhenEye direcon

Head moonhistory

Fig. 4 Improving the accuracy of eye tracking by headmotion history

as compensating movement and synergistic movement6). Related to the moment of eye movement, the compen-

assuming the directions of eyes and head are the same,the eye points always come to the center of images captured by the camera that shoots outward. The distance
between the eye position and the center of an image issating movement of head is observed between -240ms to+80ms, and the synergistic movement is observed -240ms

equal to the error of eye tracking by head direction at a

Figure 4 shows the overview of this paper. The conventional model treated the eye direction as the headdirection. Our proposed method introduces head motion history and two state models for integrating eye andhead direction. The head direction and head motion history are logged by wearable devices in the experiment.
Although we understand the wearable devices are not realistic for a practical use, since our focus is to clarify thepossibility of head motion history for accurate eye track-

to 0ms. The head motion history provides a cue for correctly estimating the eye direction.

given moment.
An example of root mean square error (RSME) of theangular error is plotted in Figure 2. The average isabout 30 degrees, and the maximum is more than 70 degrees. The accuracy is much worse than with eye trackers
on glasses.

3.

Eye Tracking by Head Motion History

We tried to improve the accuracy of eye tracking. The
core idea was using head motion history, not only headdirection at a moment. As shown in Figure 3, the direction of head is not same as the direction of eyes especiallying, we keep the problem as future work. We compare
the accuracy of eye directions estimated with the conventional method and proposed method.
We used multiple regression analysis (MRA) and dynamic coupled component analysis (DCCA)7)to learn thein motion. The human eye-head coordination is known
2

relationship between head motion history and eye direction. MRA has a simple model, so it is easy to use, but it
is diﬃcult to learn non-linear angular relationships with
MRA. DCCA elucidates the relationships between different dimensional datasets. It does not directly project
the input vector into the output vector. DCCA assumes
a lower dimensional space substantively controls the in-(a) Conventional model

put and output, and shows the projections of input andoutput vectors for the lower dimensional space. DCCA
works well even though the input and output vectors are
represented in a high dimensional spaces. DCCA is robust for non-linear data, compared with simple MRA.
Ma and Deng8)applied DCCA to synthesize avatar models with eye motion. They employed DCCA to control
eye motion of the avatar by human head tracking data.
The introduction of smoothness function makes it robust(b) Multiple regression analysis (MRA)

even for noisy data. A disadvantage is the lower speed ofconvergence because of the complexity of the model.
Figure 5 shows the diﬀerence among three types ofh, pitch φht−k andlearning. The input vector is roll θt−khyaw ψt−k of head direction at time t, its previous timet − k (k = 0, · · · , n) and constant 1. Total dimension is3(n+1)+1. The output vector is roll θte , pitch φet , and yaw
ψte of eye direction at time t. The dimension is 3. In theconventional model shown in Figure 5(a), eye direction
is estimated as the same direction of the head. MRA, in
Figure 5(b), shows the relationship between the input andoutput vectors as a matrix that projects the input vector
into the corresponding output vector. DCCA, in Figure5(c), shows the relationship by two matrices that project(c) Dynamic coupled component analysis (DCCA)
Fig. 5 Diﬀerences among three learning models

DCCA treats the projection matrices B and C as follows. S is a temporal smoothing matrix. The matrices
B and C contribute to project the signal to lower dimensional space, and the matrix S contributes smoothing of

the input into the corresponding lower dimensional point,
and the point is projected into the output with small
error. In the learning phase, we gathered the input andoutput vectors with the glasses-type eye tracker and thethe signal. B and C are essential, and S is complemental
in DCCA. The hat notations mean that the dimensions

gyro sensor mounted on the head. By MRA, a projection

are reduced by principle component analysis (PCA) orother techniques. λ1 and λ2 are arbitrary constants forweighting the terms.

matrix A is optimized to minimize energy E deﬁned by
the input vectors and output vectors as the following.
⎞
e
θt−n
⎟
⎜ e
⎜ φt−n ⎟⎜ e⎟⎜ ψt−n ⎟
⎛ e ⎞
⎜
⎟
⎜ . ⎟
θt
.
⎜
⎟
⎜
. ⎟
⎟,
et = ⎝ φet ⎠ , ht = ⎜⎜⎟⎜ θte ⎟eψt⎜⎟
⎜ φe ⎟⎜⎟t⎜⎟⎝ ψte ⎠⎛

Estatic =E=||et − Aht ||.

||ĥt − Cet ||22 + λ1 ||et − Bht ||22 ,Edynamic = Estatic + λ22||et − Set−1 ||22 .

(2)
(3)

Estatic represents the energy that the input vectors andoutput vectors are projected, to the same corresponding
points of the common lower space, and Edynamic controls

1
t


t


(1)

the temporal smoothness of the output vectors.
3subject should see the images.
An eye tracker often misses eye direction. According
to the speciﬁcations sheet, the eye tracker we used cancorrectly estimate ±20o horizontally and ±40o vertically.
The range could not cover the whole of the monitor, thenthe eye tracker lost the eye direction when the subject saw
Fig. 6 Wearable eye tracker and 9D gyro sensor for theexperiment. Subjects stare at the stimulus images with wearing the devices.

the corners of the monitor. We ignored the eye tracking
data including the missing frames both in learning andexecution phases. Only the eye tracking data within the<Learning><Execution>

25 images

stimulus images was used for the regression model and thecalculation of error. In our experimental environment,
the limits were ±29o in horizontal and ±17o in vertical.
We set no limit for the direction of head.
We adopted our proposed methods for eye tracking and

1 image

6 subjects

head motion in time sequences. MRA and DCCA show
the relationship of eye direction and head motion history

6 subjectsof six subjects from the data for 25 stimulus images in theinter-user study as shown in Figure 7(a). The estima-(a) Inter-user study<Learning>tion error was calculated by adopting the learning model<Execution>

into the data for one unused stimulus image. We validated how the learned model is generalized to images inthis study. The situation of ﬁve subjects with 26 stimulus images in the intra-user study is shown in Figure 7(b).
The error was calculated by adopting the learning model

26 images

26 images

into the data of one unused subject. We validated how
5 subjects

the learned model is generalized between subjects in this1 subject

study. The design is called as leave-one-out cross validation. We set λ1 = 0.40 and λ2 = 0.80 for DCCA, n = 5(b) Intra-user study
Fig. 7 Design of subject studies

4.

for the history frames, which were the best parameters
through our trials and errors.
Figure 8 shows the reduction percentage of estima-

Experiments

We conducted a subject study to investigate whether

tion error for each stimulus image in the inter-user andthe proposed method improves accuracy. Six subjects

intra-user study. Figures 8(a) and 8(c) indicate the error(ﬁve males and one female; university students in their

reduction achieved by head motion history with MRA,
and Figures 8(b) and 8(d) indicate the error reduction20s) stood 1.4m away from a 70-inch monitor, as shownin Figure 1. A 9D gyro sensor was mounted on theirachieved by head motion history with DCCA. Figures
8(a) and 8(b) are the results of the inter-user study. Fig-

heads, and they wore an eye tracker as shown in Figure
6. The equipment recorded head motion history and eye

ures 8(a) and 8(b) show the median of estimation error

directions in time sequence.
Twenty-six still stimulus images were displayed for 15
seconds each. Between two stimulus images, a white dot

for each image given by 6 subjects, and Figures 8(c) and

was displayed at the center of the monitor, and we asked

accuracy of eye tracking was improved.
The average error for all stimulus images was -7.2%
with MRA, and -0.8% with DCCA in the inter-user study.

8(d) show the median of estimation error for each subject given by all images. Negative values mean that thethe subjects to stare at it. The dot controlled the initial
position for each image. Because the monitor reﬂected
objects in the room, we kept the room dark so as to

For most stimulus images and subjects, head motion history improved the accuracy of eye tracking. The absolutevalues of original estimation error were 29.1o (S.D. 2.9o )on average. By introducing the regression models, thereduce reﬂections. We had the subjects freely see still
stimulus images. Each stimulus image was shown for 15
seconds. We did not give any instructions on how the4

(a) Inter study, MRA(b) Inter study, DCCA(c) Intra study, MRA(d) Intra study, DCCA

Fig. 8 Estimation errors of subject studies

error became 26.5o (S.D. 2.4o ) on average in MRA ando

the result that we tried to estimate B and C with ﬁxed S
as the unit matrix, the error was almost same. Therefore,
we concluded that the introduction of projection to low
dimensional space could not contribute to estimate moreaccurate regression model.
There were large diﬀerences in results among the stim-o29.2 (S.D. 2.6 ) on average in DCCA. Compared with
MRA of the simple model, DCCA did not improve theaccuracy. According to the results of a one-sided t-test,
only the result with MRA in the inter-study (Figure 8(a))
was signiﬁcant (p < 0.05). There was no signiﬁcant dif-ulus images. Figure 9 shows the images that scoredthe best and the worst for improving the accuracy of eye
tracking. We generated saliency map with a graph-based
visual saliency (GBVS) algorithm9); and overlaid it on thestimulus images. The best images have the tendency toinclude batch of areas with high saliency values, whereas
the worst images include cluttered small areas. From theeye tracking data, the worse images tend to attract theference for the results with DCCA.
In the intra-user study, the average error was increased
by 4.3% with MRA, and 9.6% with DCCA. The results
for subject 1 were much worse than those for the othersubjects, which degrades the total accuracy. The relationship of eye direction and head motion history in subject 1
could not be modeled well by the data of other subjects.
Some private habits of viewing or poor eyesight of subject

gaze of subjects in not salient regions, and individual subjects saw diﬀerent regions of the images. The best images
attracted the gaze of many subjects in similar way andin similar regions, which would contribute more accurate
estimation of eye direction. However, there are exceptions within the images representing eye tracking data.
The tendency would diﬀer from much more subjects. We
would like to examine how much the accuracy can be
improved with much more data as future work. In thiswork, we at least conﬁrmed that it is possible to improve
the accuracy with head motion history, and MRA is more

1 could have aﬀected the results.
We expected that DCCA would show the relationshipof eye direction and head motion well, but the resultsof MRA were much better than those of DCCA. The eye
movement containing high frequency components and theexistence of missing or incorrect frames by eye blinking
degrades the accuracy of estimation. It is possible thata simple model derived better results. In addition, it isclear that humans can change eye direction without moving their heads; therefore, some head motion is not related
to eye direction, as we described in the introduction. As
5

(a) Best stimulus images(b) Worst stimulus images
Fig. 9 The best and worst stimulus images for improving the accuracy. (Top)
Saliency maps for stimulus images, (middle) eye tracking data of subject
1, (bottom) eye tracking data of subject 4. The cross marks (x) in theﬁgure represent that we failed to track the eye direction in its previous orfollowing frames because the subject saw outside of the image, or blinked
his/her eyes. We conﬁrmed that eye tracking data includes high frequency
components and we failed to track the eye direction in many frames.

promising than DCCA for the purpose.

5.

Discussions and Future Work

Saccades do not involve the movement of head, therefore images with distributed saliency points are diﬃcult

We proposed a method for improving eye direction es-

gives worse results in the experiment. This is the limitation of our method. Our method would work well in thecase that saccades are not observed.

timation by head motion history. In the experiments, theaccuracy of eye tracking was improved with MRA andDCCA models. As an important future work, we will analyze the results from the view of subjects and stimulus
images for specifying the valid situations. In many applications, discrete-time eye tracking data may be more6

useful than constant eye tracking data.

Masahiro TOYOURA (Member)He received the B.Sc. degree in Engineering,
M.Sc. and Ph.D. degrees in Informatics from
Kyoto University in 2003, 2005 and 2008 respectively. He is currently an Associate Professor at Interdisciplinary Graduate School,
University of Yamanashi, Japan. His researchinterests are augmented reality, computer andhuman vision. He is a member of ACM andIEEE Computer Society.

Acknowledgement
This work was supported by JSPS KAKENHI Grant
Numbers JP17H00737, JP16K12784, JP17H00738.

References
1) EMR-9, NAC Image Technology.
2) K. Sankaranarayanan, M. C. Chang, N. Krahnstoever: “Tracking gaze direction from far-ﬁeld surveillance cameras,” Proc.
Winter Conference on Applications of Computer Vision,
pp.519–526 (2011).

Takumi TANAKAHe received the B.Sc. degree in Engineeringfrom University of Yamanashi, Japan. His research interests include computer vision andhuman sensing.

3) C. Lee: “Eye and head coordination in reading: roles of head
movement and cognitive control,” Vision Research, Vol. 39,
pp.3761–3768 (1999).
4) Y. Okinaka, I. Mitsugami, Y. Yagi: “Gaze Estimation Based on
Eyeball-Head Dynamics,” Proc. IPSJ CVIM, Vol. 2016-CVIM202, Article 18 (2016).
5) M. Toyoura, T. Tanaka, A. Sugiura, X. Mao: “Eye Tracking
by Head Motion History,” Proc. International Workshop on
Image Electronics and Visual Computing, pp.1–5, Article 4B-3
(2017).

Atsushi SUGIURAHe received the B.Sc. degree in Engineering,
M.Sc. and Ph.D. degrees in Engineering from
University of Yamanashi in 2002, 2012 and2015 respectively. He is currently an Assistant Professor at Interdisciplinary Graduate
School, University of Yamanashi, Japan. His
research interests are augmented reality, computer and human vision.

6) W. Einhüser, F. Schumann, S. Bardins, K. Bartl, G. Böning,
E. Schneider, P. König: “Human eye-head coordination in natural exploration,” Network: Computation in Neural Systems,
pp.267–297 (2007).
7) F. De la Torre, M. J. Black: “Dynamic coupled component
analysis,” Proc. CVPR, Vol. 2, pp.II-643-II-650 (2001).
8) X. Ma, Z. Deng: “Natural eye motion synthesis by modelinggaze-head coupling,” Proc. IEEE VR, pp.143–150 (2009).

Xiaoyang MAO (Member)
She received her B.Sc. in Computer Sciencefrom Fudan University, China, M.Sc. and
Ph.D. in Computer Science from Universityof Tokyo. She is currently a Professor at
Interdisciplinary Graduate School, University
of Yamanashi, Japan. Her research interests
include texture synthesis, non-photo-realistic
rendering and their application to scientiﬁcvisualization. She is a member of ACM SIGRRAPH and IEEE Computer Society.

9) J. Harel, C. Koch, P. Perona: “Graph-based visual saliency,”
Advances in Neural Information Processing Systems, pp.545–552 (2006).

(Received July 17, 2017)(Revised November 10, 2017)7