From: AAAI Technical Report FS-93-04. Compilation copyright © 1993, AAAI (www.aaai.org). All rights reserved.

Non-Intrusive

Gaze Tracking Using Artificial
Networks

NeuralDean Pomerleau & Shumeet Baluja

1. Introduction

School of ComputerScience
Carnegie Mellon University
2.2. Neural Network Based Gaze Tracking

The goal of gaze tracking is to determine where aperson is looking from the appearance of their eye.
Numerousattempts have been made to create better gaze trackers. The interest in gaze tracking
exists because of the vast number of potential
applications.
Twoof the most commonuses of agaze tracker are as an alternative to the mouseas
an input modality [Ware & Mikaelian, 1987] and as
an analysis tool for human-computer interaction
studies [Nodineet. al, 1992].

One of the primary benefits of the NNbased gaze
tracker is that it is non-intrusive; the user isallowed to movehis head freely. In order to accountfor the shifts in the relative positions of the camera
and the eye, the eye must be located in each image
frame. In the current system, the eye is located bysearching, in the image of the user’s face, for thespecular reflection of a stationary light. This canusually be distinguished by a small bright region
surrounded by a very dark region. The reflection’s
location is used to limit the search for the eye in thenext frame. A 15x30 window surrounding thereflection is extracted; the image of the eye islocated within this window.[Figure 2].

Viewedin the context of machinevision, successful
gaze tracking requires techniques to handle imprecise data, noisy images, and a possibly infinitely
large image set. The most accurate gaze tracking
has come from intrusive systems which either
require the subject to keep their head stable,
through chin rests etc., or systems which require
the user to wear cumbersome equipment, ranging
from special contact lenses to a camera placed on
the user’s head to monitor the eye. The systemdescribed here attempts non-intrusive gaze tracking, in which the user is neither required to wear
any special equipment, nor required to keep hisheadstill.
2.

To determine the coordinates of the gaze, the 15x30
windowis used as the input to the neural network.
The forward pass is simulated in the NN, and thecoordinates of the gaze are determined by reading
the output units. The output units are organizedwith 50 output units for specifying the X coordinate, and 50 units for the Y coordinate [Figure 3].
The gaussian output representation used is similar
to that used in ALVINN[Pomerleau, 1992].
The network is trained by the user movingthe cursor with the mousearound the screen, while visually tracking the cursor. The image of the eye isdigitized, and paired with the (X,Y) coordinates
the cursor. A total of 1500 image/position pairs aregathered. The network is trained for approximately
200 epochs, using standard back propagation.

Gaze Tracking

2.1. Standard Gaze Tracking
In standard gaze trackers, the image of the eye isprocessed in three basic steps. First, the specularreflection of a stationary light source is found in theeye’s image. Second, the pupil’s center is found.
Finally, the relative position of the light’s reflection
and the pupil’s center is calculated. Frominformation about their relative positions, the gaze direction is determined [Figure 1]. In many of thecurrent gaze tracker systems, the user is required
to remain motionless, or special headgear must beworn by the user to maintain a constant offset
between the position of the cameraand the eye.

3. Results
The current system works at 10 hz.. The best accuracy we have achieved is 1.5 degrees with the freedom of head movement up to 30cm.. Although wehave not yet matched the best gaze tracking systems, which have achieved approximately 0.75
degree accuracy, our system is non-intrusive, anddoes not require the expensive hardware which153many other systems require. Figure 4 shows how
our system compares with a commercial gazetracking system, the ISCAN.

as a tool for data collection
interaction experiments.

in human-computer

References
4. Future

Directions

Oneof the largest problemsin existing eye trackers
is the inability to handle user motion. To address
the problem of user motion, a mobile camera could
be used. To control the camera, another neural network can be trained to keep the user’s face in thecenter of the image. An extension of this idea is touse a neural network to find the eye in the image ofthe face. Currently, the heuristic used to find theeye in the imageof the face is to locate a bright spotsurrounded by dark regions. However, neural networks have been applied to facial feature tracking[Hutchinson, 1990] [Debenham, 1991] and thistechnology mayhelp here. Another extension of theidea of facial tracking is using extra inputs units torepresent the head position. Finally, for increased
accuracy, a higher resolution image of the eye may
yield improved results. The current resolution is
only 15x30 gray-scale pixels. Increasing the resolution may, however, decrease the speed of the gaze
tracker.
In order to rapidly train the neural network to newusers, one method may be to use a multiple network architecture,
as was used in the MANIAC
autonomousroad following system [Jochem, 1993].
In this system, several smaller "expert" networks
were trained on different road types, ie. one lane,
two lane etc.. An arbitrating
network, which
resided "on top" of the smaller networks, is used to
select which of the expert networks is yielding thebest response to the current road. In a similar manner, expert networks can be trained on several different people, with retraining only having tochange the weights in the arbitrating network.
Arbitration between networks could also be constructed through the use of metrics which estimate
the output reliability [Pomerleau, 1993]. A benefitof the expert network approach is that each of theexpert networks can be trained independently of
the 6thers.
This flexibility of head position makesthe connectionist gaze tracker muchless intrusive than existing systems. Wewould like to test the viability ofreplacing the mouse with the connectionist gazetracker. Other potential uses for the system, which
wewill be tested in the future, include helping disabled people interact with their environment, and154

Hutchinson, R.A. (1990), "Development of an MLP
feature location technique using preprocessedimages". In proceedings of International Neural Networks Conference, 1990. Kluwer.
Jochem, T.M., D.A. Pomerleau, C.E. Thorpe (1993),"MANIAC:A Next Generation Neurally Based
AutonomousRoad Follower". In Proc. of International Conference on Intelligent AutonomousSystems (IAS-3).
Debenham, R.M. & S.C.J. Garth (1991), "The
Detection of Eyes Using Radial Basis Functions".
In Proceedingsof the 1991International Conference ofArtificial Neural Networks ICANN-P]. Amsterdam,Netherlands, North-Holland.
Fahlman, S.E. & Lebiere, C. (1991) "The Cascade
Correlation Learning Architecture". Advances in
Neural Information Processing Systems 2. D.S.
Touretzky (ed.) MorganKaufmann,pp. 524-532.
Nodine, C.F., H.L. Kundel, L.C. Tote & E.A.
Krupinksi (1992) "Recording and analyzing eyeposition data using a microcomputer workstation",
Behavior Research Methods, Instruments & Computers 24 (3) 475-584.
Pomerleau,D.A. (1991) "Efficient Training of Artificial Neural Networks for AutonomousNavigation,"
Neural Computation3:1, Terrence Sejnowski(Ed).
Pomerleau, D.A. (1992)Neural NetworkPerception forMobile Robot Guidance. Ph.D. Thesis, Carnegie Mellon University. CMU-CS-92-115.
Pomerleau, D.A. (1993) "Input Reconstruction Reliability Estimation", Neural Information Processing
Systems 5. Hanson, Cowan, Giles (eds.) MorganKaufmann, pp. 270-286.
Starker, I. & R. Bolt "A Gaze-ResponsiveSelf Disclosing Display", In CHI-90. Addison Weseley, Seattle, Washington.
Ware, C. & Mikaelian, H. (1987) "An Evaluation
an Eye Tracker as a Device for ComputerInput", InJ. Carrol and P. Tanner (ed.) HumanFactors inComputingSystems - IV. Elsevier.

Figures

SpecularReflectionLooking
LightatLooking
Above
Light

Figure 1. Relative

position

Looking

BelowLightof specular

Figure 2. The 15x30 window extracted
user’s face.

155

reflection

Looking

Left
Lightand pupil.

from the image of theof

O OO

50 X output
Units

000

50 Y output
Units
4 HiddenUnitsiiiiiiiiiii!iiiiiiiiiiii!iiiiiiiiiii~iiii~:i15x30
Input
RetinaFigure 3. Network Architecture

ProjectedISC;~f50

ISCAN~

NNGaze~ker100

1.50200

250

CameraField of View(ram)Figure 4. NNgaze tracker

156

vs. ISCAN.

300