Junji Watanabe*
PRESTO Japan Scienceand Technology Agency
3-1 Morinosato Wakamiya
Atsugi, Kanagawa 243-0198, JapanandNTT Communication Science
Laboratories
NTT Corporation
3-1 Morinosato Wakamiya
Atsugi, Kanagawa 243-0198, Japan
Hideyuki Ando
Taro Maeda
NTT Communication Science
Laboratories
NTT Corporation
3-1 Morinosato Wakamiya
Atsugi, Kanagawa 243-0198, Japan
Susumu Tachi
Graduate School of Information
Science and Technology
University of Tokyo
7-3-1 Hongo
Bunkyo, Tokyo, 113-8656, Japan

Gaze-Contingent Visual
Presentation Based on Remote
Saccade Detection

Abstract
Pursuing new display techniques based on insights into human visual perception canreveal new possibilities for visual information devices. Here, we propose a novel information presentation technique that exploits the perceptional features during rapid eye
movements called saccades by using a fast remote eye-measuring method. When lightsources are fixed on a vertical line, and the flashing pattern is changed quickly during ahorizontal saccade, 2D images can be perceived due to spatio-temporal integration in
the human vision system. We use this phenomenon to present 2D images with only
one-dimensional light sources, and to show these images even in midair. The flashingcycle and flash timing of light sources are important elements in developing the design
theory for this display technique. The flashing cycle determines the maximum resolutionof a perceived 2D image. The flash timing is a crucial issue for our purpose because 2D
images are perceived only when the timing of the saccade coincides with the flash timing. Therefore, in this paper, we describe the relationship between a flashing cycle andthe maximum resolution of a perceived 2D image, and then propose a concise saccade
detection method. By using this method, saccades can be detected and the light sources
can be flashed within the saccade interval as it occurs in real time, and 2D images canbe successfully presented.

1

Introduction

The display of 2D images is fundamental to nearly all forms of visual presentation, and pursuing display techniques based on insights into human visual
perception may reveal many new possibilities. In this paper, we propose a visual information display that can effectively present 2D images with only a 1D
light source by remotely detecting a viewer’s saccadic eye movements.
When a vertical line of light sources flashes quickly in a temporal patternduring a horizontal saccadic eye movement, 2D images can be perceived because of the spatio-temporal integration of the human visual system. The flashing pattern is expanded into a spatial pattern by the eye movement as represented in Figure 1(a). During eye movement, the vertical light array travels on
the retina while changing the flashing pattern. Then, the different vertical images at different retinal locations are integrated into a 2D image. This phenomenon has been used for artistic expressions and entertainment tools. For

Presence, Vol. 16, No. 2, April 2007, 224 –234
© 2007 by the Massachusetts Institute of Technology*Correspondence to watanabe@avg.brl.ntt.co.jp.

224 PRESENCE: VOLUME 16, NUMBER 2

Watanabe et al. 225

Figure 1. Representation of the moving light array in space-timecoordinates. The horizontal axis is space, and the vertical axis is time.
During a saccadic eye movement, the vertical light array travels on
the retina while the flashing pattern changes. The different verticalimages at different retinal positions are integrated into a 2D image(projection to the space axis in space-time coordinates).
(a) Retinally painted image, (b) Perceived image.

example, Exploratorium displays an artwork “Light
Stick” based on this phenomenon by the artist Bill Bell
(Exploratorium, 1993). Rave parties such as “Burning
Man” (Burning Man Project, 1989) and the Media performance group “cell/66b” (Watanabe et al., 2004)
employ this phenomenon as a visual effect.
In addition, when we consider this phenomenon as avisual information display principle, it thus enables presenting 2D images with only 1D light sources, and showing
images even in midair in a dim environment (see Figure2). These features can be applied to an augmented reality
visual display that can superimpose various types of information onto real environments, to the field of entertainment, and to light devices for commercial advertising.
However, a feature of this phenomenon is that if the timing of the saccade does not coincide with the flash timingof the light source, only 1D light will be perceived, insteadof a 2D image. This point is crucial when we use this phenomenon as a visual information display principle. Here,
we propose a novel information presentation techniqueachieved by combining the saccade-contingent phenomenon with a remote and fast eye-measuring method. Saccades can be detected and the light sources can be flickered within the saccade interval as it occurs in real time,

Figure 2. Example of displaying 2D images using 1D light sourcesand viewers’ eye movements (conceptual image).

and then 2D images can be successfully presented. Moreover, if we can detect where the saccade is made, we canselectively present different 2D images to different viewers
using only one light array.
We discuss the design theory of saccade-contingent
visual display in Section 2 and our remote saccade detection method in Section 3. The novel aspect brought
up in our study is the concept of timing the display tosaccades as our eye tracking method remotely detects
them. Although the saccade-contingent phenomenon
and remote eye measuring methods may be previously
known and partly established, pursuing the possibility ofthe visual display technique studied in this paper shouldbe fruitful for investigating new possibilities for designing visual information display devices.

2

Design Theory of Saccade-Contingent
Display
2.1 Features of the Perisaccadically
Presented Image

When the described phenomenon is employed as abasis for a visual display principle, investigating the per-

226 PRESENCE: VOLUME 16, NUMBER 2

ceptual features of perisaccadically presented images isimportant. Previous studies have reported that while theeye is moving, the visual system becomes blind. Various
aspects of sensitivity loss occur around the time of thesaccade (Latour, 1962; Volkmann, Schick, & Riggs,
1968; Zuber & Stark, 1966). Thus, the images presented during saccades can be suppressed. Some studies,
however, showed that sensitivity to images with highcontrast and high spatial frequency is not suppressed(Burr, Morrone, & Ross, 1994; Uchikawa & Sato,
1995). Actually, 2D images generated with a high luminance LED (light emitting diode) can be observed, aslong as environmental light is dim (that is to say in highcontrast to the light source). For example, when flashing patterns are presented with vertically arranged LEDs
in dim light (around 50 lx) and the usual saccades occur(10 –20° in amplitude), images 50 pixels in width can beperceived. People, even naı̈ve viewers, can recognize
3–5 characters in one saccade eye movement.
One interesting feature of the perceived image thathas been reported is that when a light stimulus is presented through a saccade, the width of the perceived
image is only half of the retinal image, the width ofwhich is saccade amplitude, as in Figure 1(b) (Hershberger, 1987; Hershberger & Jordan, 1992; Jordan &Hershberger, 1994; Hershberger, Jordan, & Lucas,
1998; Sogo & Osaka, 2001; Noritake, Kanzai, Terao, &Yagi, 2005; Watanabe, Maeda, & Tachi, 2005a; Watanabe, Noritake, Maeda, Tachi, & Nishida, 2005b).
When 20° saccades occur, images 10° in width can beperceived. This indicates that the perceived 2D images
are not just the result from a retinally bleached afterimage, but are instead the result of spatio-temporal information processing in the visual stream. (See the detailsof the perceptional features of spatio-temporal integration in the paper by Watanabe et al., 2005b).

2.2 Maximum Resolution of the Image
When a vertical light array flashes and a horizontal saccade occurs, the vertical resolution of theperceived image coincides with the number of vertically arranged light sources. On the other hand, thehorizontal resolution of the perceived image can be

Figure 3. Definition of the maximum resolution by (a) spatial
factors and (b) temporal factors. (c, d) D is the width of light source,
and V is the velocity of the eye movement. Light source travels V ⫻ T
on the retina during one flashing cycle. If V ⫻ T is larger than D (c),we can present 2D images properly. If V ⫻ T is smaller than D (d),the image presented at time t overlaps the images at time t ⫹ T.

determined by the flashing cycle of the light source.
Here, we discuss the relationship between the flashing cycle of the light array and the maximum resolution of the perceived 2D.
The maximum resolution of the perceived image can be
described by spatial or temporal factors as in Figure 3(a) or
3(b). The maximum resolution described as X in the figure,can be defined as X ⫽ W/D (W is the expanded width ofimage on the retina; D is the width of the light source) orX ⫽ ␶/T (␶ is the time of information presentation during asaccade; T is the flashing cycle). When we define X based on
the spatial factors as in Figure 3(a), the flashing cycle of a lightsource has to be changed during the saccade according to theposition of the light source on the retina, because the velocityof the eye movements change over time during a saccade.
Therefore, to achieve the maximum resolution as X ⫽ W/D
is difficult. Consequently, we consider the maximum resolution based on temporal factors (X ⫽ ␶/T ) as in Figure 3(b).
We can present 2D images properly only when the velocity ofeye movement exceeds a threshold V␶, and V␶ can be calculated by the width of light source D and the flashing cycle T.
We assume that, if the flashing cycle is T, the light source isturned on for very a short moment and is turned off for T.
The light source moves V ⫻ T on the retina during one flashing cycle (V is the velocity of eye movement). If V ⫻ T islarger than D as in Figure 3(c), we can present 2D images
properly. If V ⫻ T is, however, smaller than D as in Figure3(d), the image presented at time t overlaps the image at timet ⫹ T. Therefore, to present each pixel of an image without

Watanabe et al. 227

Figure 4. (a) Assumed velocity of eye movement (normal
distribution). (b) Simulation of maximum resolution according to
flashing cycle.

overlapping, V ⫻ T must be larger than D, and the thresholdV␶ can be calculated by D ⫽ V␶ ⫻ T. In summary, the maximum resolution X is calculated by X ⫽ ␶/T, ␶ is determinedby V␶, and V␶ ⫽ D/T. If the width of light source D andtime course of the velocity of eye movement are known, X
can be a function of T.
For example, assuming that the width of light source
D is 0.1° in visual angle (corresponding to the width
when one sees a light source 5 mm in width from the
distance of 3 m), 10 and 20° saccades, and the timecourses of velocity are as shown in Figure 4(a), we cansimulate the maximum resolution X corresponding to
the flashing cycle T. The results of such a simulation are
shown in Figure 4(b). The horizontal axis represents
flashing cycle T (ms) and the vertical axis represents X(pixels). When 10° saccades occur, 24.7 pixels can bepresented with at most a 0.4 ms flashing cycle. When
20° saccades occur, 50.4 pixels can be presented with a
0.3 ms flashing cycle. Therefore, one pixel for thesemaximum resolutions can be perceived at about 0.2° invisual angle in both conditions, and this pixel width issufficient for viewers with normal visual acuity to perceive. Actually, we confirmed in our observation thatimages with these resolutions could be perceived in thedescribed conditions.

3

Design Theory of Real-Time Remote
Saccade Detection

The flash timing of the light source is essential
for successfully presenting visual information basedon the saccade-contingent phenomenon, because 2D
images can be perceived only when the timing of thesaccade coincides with the flash timing of the lightsource. As described above, we detect a viewer’s saccade as it occurs in real time, and then, a single lightarray is flickered immediately after detecting the viewer’s saccades for effective information presentation. In
vision science experiments, presenting stimuli triggered by saccadic eye movements has been achieved,
and a saccade-contingent presentation was accomplished in a virtual reality study (Triesch, Sullivan,
Hayhoe, & Ballard, 2002). However, they used a limbus eye tracking technique. Although eye tracking
performed remotely and without viewer constraints ispreferable for our presentation principle, the viewers
were forced to attach sensors to their heads and theirheads were fixed on a chinrest.
Most previous techniques for remotely measuring
eye movement used a video camera and infrared
LEDs. The camera captured high-resolution images
around the iris to measure eye movements with highaccuracy. However, the frame rate in previous studies(such as Ohno, Mukawa, & Yoshikawa, 2002) withnormal high resolution cameras was about 60 Hz, orroughly 16 ms per frame. This is not fast enough toaccurately detect a saccade because its maximum velocity is more than 700°/s and its duration is less than
50 ms.
Recently, a few fast remote eye tracking systems,
such as R-HS-S6 by Applied Science Laboratory, havebeen developed (Applied Science Laboratories,2006). However, they are very expensive and requirelarge scale equipment. Considering our purpose, thesimpler method of fast and single horizontal line
scanning, instead of 2D scanning, is sufficient because
detection of saccades, not measuring eye position, is
required. Moreover, only horizontal saccades withlarge amplitudes and high frequencies are useful forour visual presentation technique. Therefore, toachieve visual presentation using our method basedon a saccade-contingent phenomenon, we propose aconcise and economical saccade detection method
based on a fast pupil detection technique.

228 PRESENCE: VOLUME 16, NUMBER 2

3.1 Principle of Proposed Method
Recently, a number of fast and robust pupil detection techniques have been proposed (Ebisawa, 1995;
Ebisawa, 1998; Morimoto, Koons, Amir, & Flickner,
2000; Ji & Yang, 2001; Nguyen, Wagner, Koons, &Flickner, 2002). The basic principle is as follows. As in
Figure 5(a), two light sources (one is on the optical axisof a camera, the other is off the axis) generate an image
with a bright pupil and an image with a dark pupil(Young & Sheena, 1975; Hutchinson, White, Reichert,& Frey, 1989). By subtracting the level of brightness ineach position of the two images, we can localize thepositions of the pupils due to the brightness differencein the pupils. In addition to that, even when the eyegaze is not directed toward the camera, the bright pupil
can be observed, as is described in the following experiment. Therefore, the eye movements can be detected bymeasuring the displacement of pupil position as in Figure 5(b). However, as mentioned above, this subtraction technique was performed by using an ordinary
camera that provides good accuracy but provides poor
temporal resolution. Therefore, detecting saccades withhigh temporal resolution is not possible.
Consequently, we propose a concise way of detecting
saccades remotely based on the subtraction technique.
We capture a horizontal line image around a viewer’s
pupil with high temporal resolution, and then measure
the pupil position and its velocity. When the velocity
exceeds a certain threshold, the onset of a saccade canbe detected. Immediately after detecting the saccade
onset, a light source starts to flicker to present 2D images.
Figure 5(c) shows the arrangement of the proposed
method. The arrangement is composed of infrared
LEDs, a half mirror, and a camera. LEDs for obtaining
an image of a bright pupil are placed on the optical axis.
LEDs for the dark pupil are placed relatively far fromthe optical axis. The flash is synchronized with the camera’s frame rate. Pupil positions are acquired by subtracting the level of brightness in each position of thecaptured images. We used a partial scan camera that canchange the spatial resolution and frame rate, but with aspatio-temporal tradeoff. When we detect a viewer’s

Figure 5. Diagrams of (a) acquisition of pupil’s position,(b) observation of displacement by eye movement, (c) diagramsof system arrangement.

saccades, first of all, the pupil position must be localized. Therefore, we first use the camera to capture 2D
images with normal temporal resolution to specify the
pupil positions before scanning a line image around theeye. This first procedure follows the same strategy as in
the subtraction technique presented in previous work by
Morimoto et al. (2000). Then, the camera performs 1D
fast scanning around the eye and the image is processedfor detecting saccades.
In summary, remote pupil detection techniques have

Watanabe et al. 229

Figure 6. (a) Diagram of retinal reflection according to light source
position. (b) Diagram of retinal reflection according to gaze angle.
(c) Spatial arrangement of experiment.

been developed. However, the frame rates of the cameras
they were using were not fast enough to detect saccades inreal time. Therefore, we used horizontal line images with ahigh temporal frequency. The fast pupil detection technique and 1D measurement specific to our purpose accomplishes this real-time saccade detection.

3.2 Retinal Retroreflective Feature
To develop our technique, we had to investigate
the retroreflective feature of the retina, specifically the
relationship between the gap between the light source
and the optical axis, the amount of retinal reflection[Figure 6(a)], the relationship of gaze angle to the axis,
and the amount of retinal reflection [Figure 6(b)]. Although previous researchers have partly identified therequired parameters of the retina (Miller, Hall, Greivenkamp, & Guyton, 1995; Nguyen et al., 2002), we con-ducted an experiment to clarify this in a comprehensive
manner. Figure 6(c) represents the experimental arrangement.
The subject gazed at one of 10 fixation points arranged from ⫺40 to ⫹50° in 10° intervals. We placed
an infrared LED at one of the 11 positions, arranged
from ⫺5 to ⫹5° at 1° intervals. We used 110 combinations for LED positions and fixation points. The
amount of retinal reflection was measured with a camera
placed 25 cm from the subject’s eye. The camera’s resolution was 320 ⫻ 240 pixels, and subject’s pupil occupied about 50 pixels horizontally.
While the subject kept his/her gaze fixed on one
point, the position of the LED was changed over 11
steps, and the eye image was captured at each step. Additionally, in the case of fixation at 0°, a measurement
was done in two lighting conditions, in a dark room
(0.1 lx) and with indoor lighting (100 lx). For all otherangles, measurements were performed only inside thedark room. Two subjects participated in this experiment. Typical data is shown in Figure 7.
Figure 7(a) shows the amount of retinal reflection
when the subject fixed his/her eye at 0°. The horizontal
axis represents the gap angle (°) between the LEDs position and the optical axis of the camera. The verticalaxis represents the sum of brightness in the pupil area.
The square and triangle symbols represent the respective
results from the dark room and the indoor lighting. In
both conditions, when the gap was 0°, the sum ofbrightness reached the maximum, and decreased as the
gap angle increased until around 3°. This means thatthe retroreflected light from the retina was unobservable
when the LED position was separated by more than 3°from the camera axis. This value is consistent with themeasurements of Miller et al. (1995).
Figure 7(b) shows the results with regard to the fixation positions from ⫺20 to ⫹30°. The horizontal andvertical axes are the same as in Figure 7(a). In all data,
retinal reflection could no longer be observed when the
gap reached 3°. This means that our technique can detect eye positions even when large eye movements occur. As the fixation point drifted from 0°, the maximum
value decreased. In Figure 7(c), the square and triangle
symbols represent the area of the pupil, and the bright-

230 PRESENCE: VOLUME 16, NUMBER 2㻥㼕㼌㼊㼋㼗㼑㼈㼖㼖㻠㼄㼕㼈㼄㻃㼒㼉㻃㼓㼘㼓㼌㼏㻃㼛㻃㼅㼕㼌㼊㼋㼗㼑㼈㼖㼖㻃㼓㼈㼕㻃㼓㼌㼛㼈㼏㻧㼄㼕㼎㻃㼕㼒㼒㼐㻯㼌㼊㼋㼗㻃㼈㼑㼙㼌㼕㼒㼑㼐㼈㼑㼗㻃

㻕㻓㻓㻓㻓
㻔㻛㻓㻓㻓
㻔㻙㻓㻓㻓
㻔㻗㻓㻓㻓
㻔㻕㻓㻓㻓
㻔㻓㻓㻓㻓
㻛㻓㻓㻓
㻙㻓㻓㻓
㻗㻓㻓㻓
㻕㻓㻓㻓
㻓

㻐㻘

㻐㻗

㻐㻖

㻐㻕

㻐㻔

㻓

㻔

㻕

㻖

㻗

㻘

㻧㼌㼖㼓㼏㼄㼆㼈㼐㼈㼑㼗㻃㼒㼉㻃㼌㼑㼉㼕㼄㼕㼈㼇㻃㻯㻨㻧㻃㼉㼕㼒㼐㻃㼒㼓㼗㼌㼆㻃㼄㼛㼌㼖㻃㼒㼉㻃㼆㼄㼐㼈㼕㼄䚭㻋䁯㻌㻋㼄㻌㻃㻤㼐㼒㼘㼑㼗㻃㼒㼉㻃㼕㼈㼗㼌㼑㼄㼏㻃㼕㼈㼉㼏㼈㼆㼗㼌㼒㼑㻃㼄㼗㻃㼉㼌㼛㼄㼗㼌㼒㼑㻏㻃㻓䁯

㻕㻓㻓㻓㻓

㻐㻕㻓䁯

㻐㻔㻓䁯

3.3 Required Spatial Resolution for
Saccade Detection

㻓䁯

㻔㻙㻓㻓㻓
㻔㻕㻓㻓㻓
㻛㻓㻓㻓
㻗㻓㻓㻓
㻓
㻕㻓㻓㻓㻓

㻐㻘㻐㻗 㻐㻖 㻐㻕 㻐㻔 㻓 㻔 㻕 㻖 㻗 㻘 㻐㻘㻐㻗 㻐㻖 㻐㻕 㻐㻔 㻓 㻔 㻕 㻖 㻗 㻘 㻐㻘㻐㻗 㻐㻖 㻐㻕 㻐㻔 㻓 㻔 㻕 㻖 㻗 㻘
㻋㼇㼈㼊㻌

㻔㻙㻓㻓㻓
㻔㻕㻓㻓㻓
㻛㻓㻓㻓
㻗㻓㻓㻓
㻓

㻖㻓䁯

㻔㻓䁯

㻕㻓䁯

㻋㼅㻌㻃㻤㼐㼒㼘㼑㼗㻃㼒㼉㻃㼕㼈㼗㼌㼑㼄㼏㻃㼕㼈㼉㼏㼈㼆㼗㼌㼒㼑㻃㼄㼆㼆㼒㼕㼇㼌㼑㼊㻃㼗㼒㻃㼊㼄㼝㼈㻃㼇㼌㼕㼈㼆㼗㼌㼒㼑㻜㻓

$UHDRISXSLO%ULJKWQHVVSHUSL[HO

㻕㻘㻓

㻛㻓

㻕㻗㻓

㻚㻓

㻕㻔㻓

㻙㻓

㻔㻛㻓

㻘㻓

㻔㻘㻓

㻗㻓

㻔㻕㻓

㻖㻓

㻜㻓

㻕㻓

㻙㻓

㻔㻓

㻖㻓

㻓
㻋㼓㼌㼛㼈㼏㻌㻐㻗㻓 㻐㻖㻓 㻐㻕㻓 㻐㻔㻓 㻓 㻔㻓 㻕㻓 㻖㻓'HJUHHRIIL[DWLRQ㻋䁯㻌㻗㻓㻘㻓

ness per pixel, respectively. As the eyeball rotated, thebrightness per pixel was not changed regardless of any
decrease in the pupil area. Therefore, it is likely that theamount of retinal reflection decreased due to a decreasein the captured pupil area.
In summary, the LEDs for the bright and dark pupil
should be separated by more than 3° to extract the pupil
position efficiently, and the pupil position is detectableeven after large eye movements like saccades. The variationof the retinal reflection is caused by the change in the captured pupil area since the retroreflective feature is a stablephenomenon regardless of the fixation position.

Our next investigation explored how much spatial
resolution is required to accurately detect saccades. The
experimental setup was generally the same as for theprevious experiment, and we captured 10 and 15° rightward saccades by using a high frequency camera at 200
Hz (Point Grey Research Inc. SCORPION). Ten trials
were conducted for each amplitude. The infrared LED
at 0° was always lighted during the measurements.
In this experiment, the subtraction of the two images
with bright and dark pupil was not performed because
pupil position was given. We performed image processing to detect the saccade onset from the captured images as in Figure 8(a). First we extracted one horizontal
line image that passed through the pupil center. Fromthe brightness distribution of the line image, the pupiledges were found by using a threshold. We defined thecenter of two edges as the pupil position. This procedure was conducted on all captured frames. Then, thevelocity of pupil displacement was calculated as the positional difference between two sequential frames. Whenthe velocity exceeded a certain threshold, this techniquedetected the onset of a saccade.

㻓㻋㼆㻌㻃㻤㼕㼈㼄㻃㼒㼉㻃㼓㼘㼓㼌㼏㻃㼄㼑㼇㻃㼄㼐㼒㼘㼑㼗㻃㼒㼉㻃㼕㼈㼉㼏㼈㼆㼗㼌㼒㼑㻃㼓㼈㼕㻃㼓㼌㼛㼈㼏

Figure 7. (a) Amount of reflection at fixation 0° (dark room andindoor condition) according to LED’s position. (b) Amount of reflection
according to gaze direction. (c) Area of pupil and amountof reflection per pixel.

Watanabe et al. 231

Figure 8. (a) Performed image processing for detecting saccades. (b) Elements for determining spatial
resolution.

To evaluate the required spatial resolution, we considered the value P as described in Figure 8(b), which isthe number of pixels per centimeter according to the
distance from the measured person. The value of P iscalculated with the formula on the right in Figure 8(b),assuming that the camera’s angle of view is ␪, the camera’s horizontal resolution is R, and the distance of themeasured subject from the camera is L. In this experimental condition, P is 60.6 pixel/cm, R is 1280 pixels,
␪ is 55.8°, and L is 25 cm. We made low spatial resolution images by down-sampling the original images by
1/2, 1/5, and 1/10. The down-sampled images areshown in Figure 9. The corresponding values of P are
30.3, 12.1, and 6.0 pixel/cm. The same image processing was conducted in each scaling condition.
Figure 10 shows the averages (thin line) and standard
deviations (error bars) of the obtained velocities of all scaling conditions of 10 and 15° saccades. The horizontal axis
represents time from saccade onset (ms) (physical onset ofthe saccade was defined as the time when the velocity exceeds 10 cm/s) and the vertical axis represents obtained
velocity (cm/s). The bold line is the threshold for detecting saccades. The bold line is extended to the time whensaccades can certainly be detected (i.e., until the thresholdfalls below the error bar). As the value of P decreased, thestandard deviation increased.
When 40 mm/s is used as the threshold for detecting

saccades, in the conditions of P ⫽ 60.6, 30.3, and 12.1,
we could detect a saccade near its onset. However, whenP ⫽ 6.0, the standard deviation was too large to detect thesaccade at its early stage. This means that a value of P
greater than 12.1 is required to accurately detect saccades.
This value corresponds to about 1 pixel per millimeter. In
a previous study, it was reported that eye movements were
measurable with 0.5° accuracy when the pupil image iscaptured in 160 pixels horizontally, and this corresponds
to about 6 pixels per millimeter (Matsuda & Nagami,1997). Therefore, when we detect a saccade using the proposed technique, the required spatial resolution is 1/6 thatof previous techniques.
Additionally, the proposed technique can allow head
movements, although not without restrictions. For example, assuming that R is 1200 pixels, ␪ is 60 or 90°,head movements are allowed within 60 cm (R/2 divided by P) at the distance L ⫽ 104, or 60 cm (P ⫽ 10
pixel/cm), respectively.

3.4 Discussion
The proposed method can detect the initiationand direction of a viewer’s saccade by measuring thevelocity change of the pupil movement. Basically thismethod functions properly in dim light conditions because the pupil size increases. Considering that perisac-

232 PRESENCE: VOLUME 16, NUMBER 2

Figure 9. Down-sampled images.

cadically presented images can also be perceived in dim
conditions, this display technique can be used effectively
in dimly lighted environments.
In order to use the described method, the distance
between the camera and the observer must be known.
In the experiments discussed above, 40 mm/s is used as
the threshold for detecting saccades. This threshold forthe captured image differs according to the distance.
Therefore, the distance must be measured in some way
before detecting saccades. In this method, the positionsof the two pupils are measurable, so we can calculate theapproximate distance by comparing the general distance
between the two pupils with the measured distance.
One of the characteristics of our method is that itallows head movements. However, head movements
cause displacement of pupil position. When translational
movements to front and back occur, the pupils of botheyes move in the opposite direction, although saccades
made eye movements in the same direction. Therefore,
we can segregate the saccade and the movement tofront and back by observing the directions of the pupils’
movements.
As to the segregation of the movements right andleft, we think that by focusing on their velocity wecan segregate these movements. The maximum velocity of a saccade reaches somewhat more than 700°/s,
and this is equivalent to 16 cm/s horizontal move-

Figure 10. Difference of accuracy according to spatial resolution.
Top four graphs, 10° saccades; bottom four graphs, 15° saccades.
The bold bar is the threshold for detecting saccades and extends tothe time when the saccade is surely detected, that is, when the error
bar exceeds 40 mm/s. The resolution of each set is 60.0, 30.3, 12.1,
or 6.0 pixel/cm.

ment of the pupil. Assuming that the hip of a subject
is in a fixed position on a chair while the measurement is being performed, the subject would find it
difficult to move his or her head faster than 10 cm/s.
As to the segregation of the head, since the featuresof the vestibulo-ocular reflex accompanying head rotation is similar to the feature of the saccade, the segregation with one camera is difficult. However, obvi-

Watanabe et al. 233

ously, if two cameras or a marker on the head areused for measuring head position, we can easily discriminate velocities by eye and head movement.

3.5 Summary of Measurement Method
We proposed a concise and remote saccade detection method, and clarified the feasibility of this methodby investigating the retinal retroreflective features andthe required spatial resolution for accurately detecting
saccades. Our results indicates that this method can detect saccades in their initial stages, and this can be usefulfor presenting 2D images by exploiting the saccadecontingent phenomenon.

4

Conclusion

In our current paper, we proposed a novel 2D
information presentation technique that uses perceptional features during saccades and a concise remotesaccade detection method. We discussed the relationship between a flashing cycle and the maximum resolution of perceived 2D images, and confirmed the feasibility of this real-time saccade detection technique.
Through these discussions, we have described how to
specify a suitable flashing cycle that maximizes the resolution of perceived images. We also explained how asaccade-contingent display can be achieved by detecting
a viewer’s saccade and displaying images within the saccade’s duration.

References
Applied Science Laboratories. (2006). Eye tracking expertise.
Available at www.a-s-l.com. Retrieved November 1, 2006.
Burning Man Project. (1989). Available at www.burningman.
com. Retrieved November 1, 2006.
Burr, D. C., Morrone, M. C., & Ross, J. (1994). Selective
suppression of the magnocellular visual pathway during saccadic eye movements. Nature, 371, 511–513.
Ebisawa, Y. (1995). Unconstrained pupil detection techniqueusing two light sources and the images difference method.

Visualization and Intelligent Design in Engineering andArchitecture (VIDEA 95), 79 – 89.
Ebisawa, Y. (1998). Improved video-based eye-gaze detection
method. IEEE Transactions on Instrumentation and Measurement, 47, 948 –955.
Exploratorium: The Museum of Science, Art and Human Perception. (1993). Available at www.exploratorium.edu. Retrieved November 1, 2006.
Hershberger, W. (1987). Saccadic eye movements and theperception of visual direction. Perception & Psychophysics, 41,
35– 44.
Hershberger, W., & Jordan, J. S. (1992). Visual direction
constancy: Perceiving the visual direction of perisaccadic
flashes. In E. Chekaluk (Ed.), The role of eye movements inperceptual processes (pp. 1– 43). North-Holland: Elsevier.
Hershberger, W., Jordan J. S., & Lucas, D. R. (1998). Visualizing the perisaccadic shift of spatiotopic coordinates. Perception & Psychophysics, 60, 82– 88.
Hutchinson, T. E., White, K. P., Jr., Reichert, K. C., & Frey,
L. A. (1989). Human-computer interaction using eye-gaze
input. IEEE Transactions on Systems, Man and Cybernetics,
19, 1527–1533.
Ji, Q., & Yang, X. (2001). Real time visual cues extraction formonitoring driver vigilance. Proceedings of International
Workshop on Computer Vision Systems, 107–124.
Jordan, J. S., & Hershberger, W. (1994). Timing the shift inretinal local signs that accompanies saccadic eye movements.
Perception & Psychophysics, 55, 657– 666.
Latour, P. L. (1962). Visual threshold during eye movements.
Vision Research, 2, 261–262.
Matsuda, K., & Nagami, T. (1997). All-purpose measurement
system of the eye positions. Proceedings of the 12th Symposium on Biological and Physiological Engineering, 173–176(in Japanese).
Miller, J. M., Hall, H. L., Greivenkamp, J. E., & Guyton,
D. L. (1995). Quantification of Bruckner test for strabismus. Investigative Ophthalmology & Visual Science, 36,
897–905.
Morimoto, C. H., Koons, D., Amir, A., & Flickner, M.
(2000). Pupil detection and tracking using multiple lightsources. Image and Vision Computing, 18, 331–335.
Nguyen, K., Wagner, C., Koons, D., & Flickner, M. (2002).
Differences in the infrared bright pupil response of humaneyes. Proceedings of the Symposium on Eye Tracking Research& Applications 2002, 133–138.
Noritake, A., Kanzai, K., Terao, M., & Yagi, A. (2005). A con-234 PRESENCE: VOLUME 16, NUMBER 2

tinuous lit stimulus is perceived to be shorter than a flickering
stimulus during a saccade. Spatial Vision, 18, 297–153.
Ohno, T., Mukawa, N., & Yoshikawa, A. (2002). FreeGaze: A
gaze tracking system for everyday gaze interaction. Proceedings of the Symposium on Eye Tracking Research & Applications 2002, 125–132.
Sogo, H., & Osaka, N. (2001). Perception of relation of stimuli locations successively flashed before saccade. Vision Research, 41, 935–942.
Triesch, J., Sullivan, B. T., Hayhoe, M. M., & Ballard, D. H.
(2002). Saccade contingent updating in virtual reality. Proceedings of the Symposium on Eye Tracking Research & Applications 2002, 95–102.
Uchikawa, K., & Sato, M. (1995). Saccadic suppression of
achromatic and chromatic responses measured by increment-threshold spectral sensitivity. Journal of the OpticalSociety of America A, 12, 661– 666.
Wantanabe, J., Maeda, T., & Tachi, S. (2005). Time course of
localization for a repeatedly flashing stimulus presented atperisaccadic timing. Systems and Computers in Japan, 36(9),77– 86.
Watanabe, J., Noritake, A., Maeda, T., Tachi, S., & Nishida,S. (2005). Perisaccadic perception of continuous flickers.
Vision Research, 45, 413– 430.
Watanabe, J., Tavata, T., Verdaasdonk, M. A., Ando, H.,
Maeda, T., & Tachi, S. (2004). Illusory interactive performance by self eye movement. SIGGRAPH 2004 Conference
DVDROM Sketches.
Volkmann, F. C., Schick, A. M., & Riggs, L. A. (1968).
Time course of visual inhibition during voluntary saccades. Journal of the Optical Society of America, 58, 562–
569.
Young, L., & Sheena, D. (1975). Methods and designs: Survey of eye movement recording methods. Behavioral Research Methods and Instrumentation, 7, 397– 429.
Zuber, B. L., & Stark, L. (1966). Saccadic suppression: Elevation of visual threshold associated with saccadic eye movements. Experimental Neurology, 16, 65–79.