Measuring the Utility of Gaze Detection
for Task Modeling: A Preliminary Study
Penelope Brooksc, Khoo Yit Phangc, Rachael Bradleya, Douglas Oard,a,b Ryen Whiteb, François GuimbretièrecaCollege of Information Studies/ bUMIACS /cDepartment of Computer Science, University of Maryland, College Park, MD 20742{pnoe,khooyp,rlb,oard,ryen,fguimbre}@umd.edu
important, we will increasingly need to look beyond querydocument term matching to identify observable behaviors thatindicate an analyst’s interests. In this paper, we focus on
associating visual activity and report writing with a known(stimulated) interest.

ABSTRACT
Search engines present readers with a list of documents ranked by
predicted relevance to a keyword query. Salient sections of
documents that are selected for examination can be highlighted
using similar techniques. An ability to restructure information
presentation based on an analyst’s initial interactions with aninformation space might improve search outcomes. This study
takes the first steps towards designing an analysis environment inwhich structural overlays evolve in response to an analyst’s
actions. Two sources of implicit feedback are explored: topicalsimilarity to material included in a very brief written report, andeye behavior patterns. Results indicate that using eye-tracking canbe as effective as lexical overlap, but more work is needed.

Based on the increased interest in eye-tracking and theinformation that can be gathered by monitoring eye movements,
we used an eye-tracker to monitor participants reading a
document and creating a brief focused report. We hypothesizedthat eye-tracking would add useful evidence unavailable fromlexical overlap with the content of the report. This approach was
inspired by the Inferring Relevance from Eye Movements
Challenge 2005, a multi-phase competition that seeks to advancethe study of eye-movement data [3]. The Challenge recommends
monitoring fixation duration, regression patterns and pupil size,
while taking into account re-fixations and word skipping.

Categories and Subject Descriptors
H.5.2 [Information Systems]: Information Interfaces andPresentation – User Interfaces.

Our study builds upon research conducted in several fields,
primarily eye-tracking, IR, psychology and linguistics.
Vertegaal’s work focuses on the challenges of designing anattentive interface by defining criteria that measure interest andaddressing the discrepancy between visual interest and cognitive
interest [9]. Eye-tracking researchers have monitored fixation,saccades (ballistic eye-movement from one target to another), andpupil size to indicate visual interest [1][2][5][6]. Granka et al.
used eye-tracking to study visual fixation while using a searchengine [1]. Our study differs from theirs in our focus on fulldocuments rather than search-engine result lists. Puolmaki et al.
conducted research with similar goals to ours, applying machinelearning to infer interest from reading behavior. They
demonstrated a system that ranked documents in a mannerconsistent with the user’s expectations [2]. We extend their workby measuring regression and pupil dilation, and by incorporating
evidence from lexical overlap.

General Terms
Measurement, Design, Experimentation, Human Factors

Keywords
Eye-tracking, implicit feedback, search, document summarization

1. INTRODUCTION
In the classical Information Retrieval (IR) paradigm, searchsystems rank documents in decreasing order of predicted topicalrelevance to a query. Using iterative query reformulation,
searchers can often produce useful ranked lists, while brief queryfocused summaries provide a basis for recognizing documents in
those lists that are actually interesting. Selecting a link in the list
can then reveal internal navigation cues to help users rapidly focus
on the salient parts of a selected document. Modern IR systems
accomplish these tasks based on lexical overlap with the query.
While this can work well for carefully edited documents in whichsalient content exhibits strong locality, informal genre such as
blogs or bulletin board postings frequently lack the explicit
structure typically found in sources for which many present
systems were designed. As informal sources become increasingly

Our intent for this initial study was to focus on implicit feedback
gathered by observing users’ interaction with the system,
associating those observations with content that is presented andreports that are created. Although our ultimate goal is toseparately model interest and topical relevance, the design of acontrolled user study requires a dependent variable that iscomparable across study participants. For the study reported inthis paper, we therefore stimulated a common interest by asking
each participant to focus on a common set of topical questions
while performing an IR task. We show that a combination oflexical overlap and eye tracking measures is always at least as
useful, and sometimes more useful, than any single measure
would be.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,requires prior specific permission and/or a fee.

2. EXPERIMENTIUI ’06 Workshop on Intelligent User Interfaces for Intelligence Analysis,
Jan 29, 2006, Sydney, Australia.
Copyright 2006 ACM 1-58113-000-0/00/0004…$5.00.

Our experiment aims to determine if interest, an indirect variable
representing the internal state of the reader, can be inferred from1

Figure 1. Screenshot of Trial.
eye behavior. Based on previous work, we make the assumptionthat the amount of information processing, reflected by eyemovements during reading, correlates with visual interest [4].
Additionally, we assume that visual interest is equivalent tocognitive interest, as prior studies have suggested this to be true
for relatively complex tasks such as in our study [3][9].

Figure 2. Fixations overlaid onto sections
motivated by previous work associating them to cognitive
processes [4][7]. While those studies did not focus on reading, we
speculate that the measures would also be useful for reading tasks.
We hypothesized that we can generate useful list of relevant
sections ranked by: higher number of fixations [H1]; longer
fixation duration [H2]; larger pupil size [H3]; and greater numberof regressions [H4]. We compare these interest-based ranking to aterm-based ranking system and hypothesized that combining eyetracking and lexical overlap would rank relevant sections moreaccurately than term-based ranking alone [H5].

For this study, we designed an analytical reading task that
required participants to answer a question with relevant
information from a document. We define (topical) relevance as
the information within a document needed to answer a question.
By providing a question for the reader to answer, we controlled
what they found relevant. We therefore rely on relevance as asurrogate for interest in this study. One question was designed foreach document. To establish ground truth for relevance, eachdocument was manually divided by the experimenters into
sections – defined as a coherent group of words such as aparagraph, list item, or header. Five independent assessors thencoded each section in the document as relevant or irrelevant withrespect to the question.

We post-processed the eye-tracker data to determine the numberof fixations, fixation duration, pupil size and number ofregressions for each section. We measured fixations, when the eye“stops” for at least 60 milliseconds, in a window of 10x15 pixels
with a time of 66 ms (determined by a pilot study). A fixation
location may not be unique: we considered each fixation to thesame section separately. For each fixation location, the fixation
duration was measured. We defined a regression as a repeatedfixation that involved eye movement to a section from anothersection or from the area on the screen where the answer was to beentered. We recorded the number of regressions that a participant
made to each section. Pupil size (the average area of the pupil
during a fixation) is provided by the eye-tracker in units of camera
pixels. We recorded the average pupil size for each fixation
location.

Three Congressional Research Service (CRS) reports, two 7screen documents (“Kuwait” and “Internet”) and one 18-screen
document (“Cyberattack”), were read by each participant. CRS
reports were selected because they are written in a standardized
style that facilitated manual division into sections for ground truth
relevance assessment, it is unlikely that participants would haveread them, and they are long enough to ensure that both relevant
and irrelevant information (to the question we posed) would be
provided. These document lengths were selected based on a pilot
study which indicated that reading multiple longer documents
would result in fatigue.

2.1 Protocol
We used a within-subjects design, allowing us to compare results
across participants. Each participant experienced two levels of
relevance (relevant and irrelevant) in each trial. To address ordereffects from document length and topic, we fully counterbalanced
the order of the texts presentation. Eleven volunteers participatedin our experiment, responding to an email sent out to university
graduate and undergraduate students. Participants were not paid,
but were provided with refreshments. Data from two participants
were eliminated, one due to difficulty wearing the eye-tracker andthe other due to excessive squinting. To maintain a fully counterbalanced dataset, we therefore arbitrarily deleted data from threeother participants for the preliminary analysis and reported on the
final six participants in this paper.

For each document, one question was devised by one of theauthors. The questions were open-ended, designed to be ofmoderate difficulty, and to draw on information from several
portions of the document. For example, for the CRS report
entitled Terrorist Capabilities for Cyberattack: Overview and
Policy Issues, the question was: “What are some of thecomplications in linking cybercrime with terrorism?” In thisexample, “cybercrime” and “terrorism” appear frequently in thereport, but “complication” does not appear at all and determining
the answer requires synthesis across the whole document. Answer
lengths between two and four sentences were expected.
We used an eye-tracker to measure the number of fixations,fixation duration, number of regressions, and pupil size of theparticipants’ eyes. The results were then used as estimators ofinterest to rank sections. These measurement choices were

Before the experiment, written instructions were given to eachparticipant and read from a script by the experimenter. Eachparticipant performed the experiment individually and began with2

a visual acuity test and reading speed assessment. The participant
then dons the ISCAN ETL-500 head-mounted eye-tracker. Next,
participants made seating adjustments while the experimenter
ensured proper alignment of the eye-tracker optics.

sizes, so we restrict our analysis to comparison of alternative
measurements with assessor and document held constant.
For the eye-tracking measurements, we computed a set of scoresfor each section by normalizing each measurement to fall within a[0,1] range such that a higher score represented a greater expected
degree of relevance per our hypotheses. We then used theresulting scores to compute a set of ranked lists in which theelements in the list were sections and scored these ranked lists
using trec_eval for each document-assessor pair.

After calibration, each participant was given a short practice trial,
which served to clarify the instructions and to ensure comfort withthe system. Next, the participant was presented with the firstquestion and given an opportunity to clarify the question before
proceeding to the timed trial. After indicating they were ready toproceed, the participant was presented with a document on the left
side of the screen containing information pertinent to the question
and the question and answer region on the right side of the screen(Figure 1).

In addition to fixations, fixation duration, pupil size, andregressions, three additional compound measures were alsosynthesized and used as the basis for constructing ranked lists.
We observed during the experiment that participants were reading
entire sections at the beginning of a document, but that they readless as they progressed through the document. To account for thisobservation, we computed a fixation difference measure, definedas the number of fixations minus the 7-section moving-average.
The 7-section window was chosen to reduce the skew effect fromshort sections (e.g., titles and headers). The other two compound
measures combine results from one eye-tracking measurementwith the results from term-based search. We combined the scoresby taking the maximum of number of fixations and Lucene termmatch score for each section (MaxLucFix and MeanLucFix) in
one case, and by replacing the maximum operator with thearithmetic mean of the two scores in the other case. Because these
combinations were chosen post hoc, they should be treated only assuggestive of what might be achieved from evidence combination
strategies.

Each participant was given five minutes to read the shorter
documents and ten minutes to read the longer document. If an
answer had not been completed when time expired, the document
was removed from the screen and the participant was allowed tocomplete their answer. The time limit was intended to simulate
the constraints in a real-world information acquisition task,
forcing participants to read only as much as necessary, and limited
their ability to pre-read and then answer the question frommemory. At the end of the experiment, each participant wasdebriefed regarding the purpose of this research and comments
were solicited. Additionally, they were asked to participate in a
post-experiment questionnaire where their interest and knowledgeof the topics presented was gauged.
The experiment was run on a 19” Dell LCD monitor, which had anative resolution of 1280x1024. The eye tracker was calibrated foreach participant, both with the program provided with the eyetracker and with an experiment-specific program. Minimizing
calibration errors with our current head-mounted eye-tracker
configuration proved to be challenging, in part due to discomfort
and fatigue. We plan to use a Tobii desk-mounted system in the
future, which may partially mitigate those effects.

Controlling for confounding variables posed some challenges.
Differences in writing style or formatting could affect reading
speed, so we chose documents from a single source. Anothersource of concern was prior knowledge by our participants. We
had selected topics that we felt were likely to be new for theparticipants. Testing for a priori knowledge and a priori interest
before the experiment would have been impractical, so we reliedon post-experiment self-report data to screen for those factors. We
used a post-experiment questionnaire asking each participant
about their knowledge of and interest in the topics on a scale from
1 to 5. Responses to these questions were fairly consistent, so nodata was excluded based on those factors. Another possible
confounding variable was reading speed, since faster readers havebeen shown to make shorter fixations and fewer regressions [3].

3. DATA ANALYSIS
We used Lucene (available from http://lucene.apache.org) tocompute a score for each section by indexing the sections as“documents” and then each participant’s answer as a query. This
produced a set of ranked lists, one for each participant. We thenused trec_eval (from http://trec.nist.gov) to compute the Mean
Uninterpolated Average Precision (MAP) for each measurement
for each ground truth assessor. MAP is designed to model user
satisfaction when traversing a ranked list. Given the output of theranked list, Uninterpolated Average Precision is defined as:

AP =1 R "i%($ 'R i=1 # ri &4. RESULTS
Our preliminary analysis indicated that we may have given
participants too little time to read the short “Kuwait” and“Internet” documents, so for this paper we have chosen to focus
on the long “Cyberattack” document. For purposes ofpresentation, we have sorted the assessors in increasing order ofthe number of relevant sections that they identified in the“Cyberattack” document (2, 5, 7, 16, and 21 out of 83 sections,
respectively). Based on this analysis we removed Assessor 1’sscores, as that assessor only selected two relevant sections out of
83 sections; MAP values are overly sensitive to quantization noise
when few relevant items are known. For each remaining assessor,
MAP values averaged across the six-participant counterbalanced
group are reported for each measure.

(Eq. 1)

where R is the total number of relevant sections in the ranked list;
ri is the rank of the relevant section i. The mean is then taken
across participants,
rather than across topics (as would be standard!
in a search engine evaluation).
We therefore obtained one MAP value for each assessordocument combination (a 4 by 3 matrix). Variations in document
characteristics and assessor opinions make comparison of MAP
across assessors and/or documents problematic with small sample

3Figure 3. Eye-tracking measurements for Cyberattack

Figure 4. Cyberattack Fixations, Lucene, Lucene+fixations

Of the eye-tracking measurements, number of fixations (Fixationsin Figure 3) results in the highest MAP score on average, followedby number of regressions (Regressions). Fixation duration(Duration) and pupil size (Pupil Size) yield much lower MAP
scores. These observations tend to support hypotheses H1 and H4
more strongly than hypotheses H2 or H3. Surprisingly, fixationdifference did not score better than number of fixations. We thinkthat for relevant paragraphs, fixations increase more than it isbeing reduced by the position of the section in the document.

choice of time limits for shorter documents and calibrationdifficulties with the eye-tracker we used will need to be addressedin future studies. Using well structured documents for theseexperiments facilitated creation of ground truth judgments, but if
we are to achieve our ultimate goals we also need to begin towork with the informal genre that motivate this line of research.

6. ACKNOWLEDGMENTS
This work was partially supported by DARPA’s GALE program.

We speculate that Regressions may have turned out to be a better
indicator of interest because of the way participants answered thequestion. Since they could answer as they were reading, they
could frequently refer back to the relevant sections. Fixations
would tend to increase as well due to re-reading. These factors
suggest that we might want to treat Regressions from the answer
area and Regressions from other sections separately in our
analysis. This type of behavior would reasonably be expected tooccur any time analysts are taking notes, so these two
measurements may be particularly useful as predictors of interest.

7. REFERENCES[1] Granka, L. A., Joachims, T., and Gay, G. 2004. Eye-tracking
analysis of user behavior in WWW search. SIGIR ‘04.
[2] Puolamäki, K., Salojärvi, J., Savia, E., Simola, J., and Kaski,S. 2005. Combining eye movements and collaborative
filtering for proactive information retrieval. SIGIR ‘05.
[3] Rayner, K. 1998. Eye Movements in Reading andInformation Processing: 20 Years of Research. Psychological
Bulletin 124(3), 1998, 372-422.

Comparing Fixations with Lucene reveals little difference forthree assessors and an apparent preference for Lucene forAssessor 4 (Figure 4). The composite scores MaxLucFix and
MeanLucFix are at least 0.05 higher than Lucene alone for two of
the four assessors, a criterion suggested by Sparck Jones as being
noticeable to the user [7], and to be comparable to the better ofLucene or Fixations for the other two assessors. These results
therefore tend to support hypothesis H5.

[4] Rudmann, D. S., McConkie, G. W., and Zheng, X. S. 2003.
Speech and Gaze: Eye-tracking in cognitive state detectionfor HCI. ICMI ‘03.
[5] Salojärvi, J., Puolamäki, K., Simola, J., Kovanen, L., Kojo,
I., and Kaski, S. (2005) Inferring Relevance from Eye
Movements: Feature Extraction. Publications in Computerand Information Science, Report A82 (3 March 2005)[6] Santella, A. and DeCarlo, D. 2004. Robust clustering of eye
movement recordings for quantification of visual interest.
ETRA’2004.

5. CONCLUSION
As with any initial foray into a new area, our results raise as manyquestions as they answer. Perhaps our most surprising result wasthat eye-tracking did as well as it did, yielding rankings that weretypically nearly good as those obtained using term-matching.
Combination of evidence seems to offer some promise, and we
gained one insight (regarding Regressions from the answer area)that might help to design more nuanced evidence combination
strategies. Several aspects of our study design, including the useof topical relevance as a surrogate for interest and the use ofmultiple independent assessors worked well, but our infelicitous[7] Spärck Jones, K. Automatic indexing.
Documentation, 30:393-432, 1974.

Journalof[8] Velichkovsky, B. M., Dornhoefer, S. M., Pannasch, S., andUnema, P. J. 2000. Visual fixations and level of attentional
processing. ETRA ‘00.
[9] Vertegaal, R. 2002. Designing attentive interfaces. ETRA
‘02.

4