ASIAGRAPH 2008 PROCEEDINGS

視線方向に応じた焦点可変画像による立体提示手法
Active Focus Control with Eye Gaze Detection for Depth Cognition
大谷慶一郎/東京大学,吉田匠/東京大学大学院,新居英明/東京大学大学院,川上直樹/東京大学大学院,舘暲/東京大学大学院
Keiichiro Otani, Takumi Yoshida, Hideaki Nii, Naoki Kawakami, and Susumu Tachi The University of Tokyo
t60518@mail.ecc.u-tokyo.ac.jp, takumi_yoshida@ipc.i.u-tokyo.ac.jp, hideaki_nii@ipc.i.u-tokyo.ac.jp,
kawakami@star.t.u-tokyo.ac.jp, tachi@star.t.u-tokyo.ac.jp

Abstract:

This paper describes the display system which has an internal active focus control system with the camera for an eye gazedetection to enhance depth information.. In order to enhance the depth information of two-dimensional images, blur effect isoften used in creating the CG image. However, in this situation, even if the observer gazes at unfocused objects, he cannot
see the objects clearly. Hence, such the effect is static effect and different from the focal accommodation of eyes. We propose
a prototype system for actively controlling the focus of an image on the basis of the point of gaze of the observer in order tosimulate the effect of focal accommodation; our system can be mounted to the existing displays. By using this system, we
conducted comparative experiments using an all-in-focus image and a controlled-focus image with regard to the gaze of theobserver. The results indicated that we can improve the three-dimensional appearance of two-dimensional images.

Keywords:

three-dimensional appearance, active focus control, eye gaze detection, depth-of-field blur

1. Introduction
It is possible to improve the sense of reality byenhancing the depth information of two-dimensional images.
By using techniques to present two-dimensional images withhigh sense of reality, it is possible to design equipments for
attractions in theme parks or transmit the visual informationof a camera of a remote vehicle to its operator with highsense of reality. Thus, a technique for enhancing the depth
information of two-dimensional images is required invarious fields.
In the real world, various depth cues influence our depth
cognition. The cues are classified into three categories[John01]—binocular depth cues such as binocular disparityand motion disparity, monocular depth cues such as
convergence, and pictorial cues such as shade and linear
perspective. There are many methods that use these depth
cues to enhance the depth information of two-dimensional
images. In this study, we focus on one of the pictorial
cues—lens blur.

1.1. Related study
In order to enhance the depth information of twodimensional images, blur effects is often used. For example,
the method of rendering with regard to the depth-of-field isan established technique in CG image. Using this method,
the creator attracts the attention of the audience to an object
by emphasizing it by focusing on it and unfocusing the otherareas. Potmesil and Chakravarty [Potmesil81] first proposed
the simulation of an optical lens to simulate depth-of-field
blur. Barsky [Barsky04] proposed another method for vision
realistic rendering, which accurately simulates the image weperceive in the real world. The common point among thesemethods is that the focus point is constant and the position
is dependent on the creator of the CG movie. On the other
hand, when we see the real world, the objects near the pointof gaze appear clear, and the objects in the other areas
appear blurred, as shown in figure 1. This phenomenon is
attributed to focal accommodation. Therefore, theconventional methods do not accurately simulate the effects
of focal accommodation. Hillaire et al. [Hillaire08]
improved the three-dimensional appearance of images byusing the observer’s focus point as input. The researchshows that focusing actively on the point of gaze of theobserver improves sensations and perception of the virtualenvironment. But the near objects from the observer which
require the exact focal accommodation are unconsidered.

(a)

(b)
Figure 1: In the real world, the object we gaze at appears clear
and the other objects appear blurry. When we gaze at the object
in the front, the vision resembles that in (a). When we gaze at
the object in the back, the vision resembles that in (b).

1.2. Purpose of this study

3. Implementation

The purpose of this study is to develop the display which
has internal active focus control system with the camera foreye gaze detection to enhance depth information.
By conducting comparative experiments using a
prototype system, we studied whether the depth cognitionof the two-dimensional images are improved by simulating
the effect of focal accommodation. From the experiments,
we conclude that our method may have great possibilities.
In Section 1, we present the background of this study,
related studies, and purpose of our study. In Section 2, we
present our proposed active focus control system. In Section
3, we describe in detail the techniques used in the system. In
Section 4, we describe the comparative experiments
conducted using the system. In Section 5, we provide theresults; in Section 6, we discuss the results. In Section 7, wesummarize this study and provide a conclusion.

In this section, we explain the eye gaze detection
algorithm and blur generation algorithm.

2. The eye gaze detection and blur
generation system
We propose the display which has internal active focus
control system with the camera for eye gaze detection toenhance depth information. The system actively focuses the
CG object on the point of gaze of the observer. By focusing
actively on the point of gaze, the system accurately
simulates the phenomenon wherein objects situated at thesame distance as the depth of focal plane appear clear andthe other objects appear blurry. Figure 2 shows the structureof the system. We describe the process flow.
Infra-red
cameraInfra-red LED

3.1. Eye gaze detection
We describe an algorithm for detecting a user’s eye gaze.
There are various techniques to detect eye gaze, such as the
EOG method, optical lever method, search coil method, andpupil-center-corneal reflection method [Yusuke04]. Eachmethod has both merits and demerits. We adopt the pupilcenter-corneal reflection method because it does not task theobserver and it is easy to implement. The pupil-centercorneal reflection method involves the computation of thepoint of gaze of the observer by using the pupil position andthe first Purkinje image. The first Purkinje image is formed
from light reflected on the corneal; it corresponds to lightborrowed from infrared LED. The algorithm for thistechnique is described in detail as follows.
First, we obtain the observer’s pupil position. The
infrared camera captures the image of the eye through theinfrared filter. We convert the raw image into a binary
image. Subsequently, the contour of the pupil is detected bysearching the most likely contour length. Then, the contour
is approximated to an ellipse by the least square method.
The center of the ellipse is the observer’s pupil position.
Second, we obtain the observer’s first Purkinje image
position. Similarly, we convert the raw image, which is
captured when obtaining the pupil position, into a binary
image. Now, we detect not only the first Purkinje image but
also the second Purkinje image. However, we use only theformer to compute the point of gaze. Hence, we set the ROI(region of interest) around the pupil which do not include
the second Purkinje image, and calculate the barycenter of
ROI. The barycenter is the observer’s first Purkinje image
position. Figure 3 shows the pupil and the first Purkinje
image obtained by the above process.

Infra-redfilterpupil
observer

DisplayFirst purkinje image

Figure 2: Structure of the proposed system.

Figure 3: Pupil and first Purkinje image.

Firstly, the eye gaze detection component actively
measures the eye motion of the observer. The eye isilluminated by an infrared LED. The image of the eye iscaptured by an infrared camera through an infrared passfilter. Second, the image is transmitted to a computer, whichcalculates the point of gaze by using the pupil-centercorneal reflection method. After the point of gaze iscomputed, the depth of focal plane is calculated. Then, theblur is calculated and the blur per pixel is drawn on the CG
image.

The point of gaze on the display is calculated by using
the pupil position and the first Purkinje image position. By
using a conversion matrix F , we represent the relationshipbetween the three points, as shown in equation 1.

[u

v]

[x1
[x2

t
y1 ] : The pupil positiont: The point of gaze position on the displayy 2 ] : The first Purkinje image positiont⎡ x1 ⎤
⎢y ⎥
⎡u ⎤
⎢ 1⎥
=
F
⎢v ⎥
⎢ x2 ⎥⎣ ⎦⎢ ⎥⎣ y2 ⎦

3.3. Drawing the blur on the display

L (1)Here, F represents the 2 × 4 conversion matrix. F haseight parameters; hence, we must calculate them after
calibration of the system. The observer gazes in turns at thenine fixed grid points, and at the time the system detects thepupil position and the first Purkinje image position. Thenthe grid point positions, the pupil positions, and the first
Purkinje image positions are substituted into equation 1. The
least squares solutions of the simultaneous equation are theeight parameters of F . The calibration is required in eachtime the system gets started.

Based on equation (2), we draw the blur per pixel on
the display. In this study, we apply blur to an image by
averaging the color of the pixels. The pixels used for the
averaging are the pixels which have their centers inside theblur circle. In case the blur appears around the pixel in(i j ) , as shown in figure 5, the color of the pixel is theaverage of the color of the shaded blue pixels.

i
Blur circle

j3.2. Scale of blur circle
We describe the lens blur effect in our eye. When an
object is situated at a distance different from the distance ofthe focal plane, blurring appears in the image on the retina.
First, we derive the formula for the scale of the blur. We
consider the following assumptions.
1: As shown in figure 4, the crystal lens is a single
converging lens and the retina is a single plane. They areparallel to one another [Rokita96].

: pixels used for averaging
Figure 5: Drawing the blurred pixels in the display. The pixels
of which center is inside the blur circle are used for averaging.

3.4. Prototype system
2: The blur scale depends on the distances of the object fromthe crystal lens.
3: The radius of the crystal lens is equivalent to that of thepupil.

eye side

outsidef

objectbdO

Focal plane

Aperture(Pupil)rBluraCrystal lensFigure 4: Eye model

RetinaAccording to the first assumption, the shape of the blur
appearing on the retina is a circle. According to the second
and third assumptions, geometrically, the radius of the blurcircle rBlur is expressed by the following formula. Here,f denotes the depth of focal plane; d , the distance of theobject from the eye; a , the distance from the crystal lens to
the retina; and b , the radius of the crystal lens.

rBlur = a ⋅ b

1 1
−
f d

L ( 2)In this section, we describe the prototype system that wehave implemented. Figure 6 shows the structure of the
prototype system we have implemented.
The system consists of the following three components:
an eye gaze detection component, a computer to create twodimensional images with blurring at distances, except thatequivalent to the depth of focal plane, and a display to show
the image. In the eye gaze detection component, an infrared
camera captures images of the observer’s eye through theinfrared filter over a half mirror. The camera should belocated near the eye to capture the clear image of the eye.
However if the camera is in front of the eye, it blocks theobserver’s view. Hence we locate the camera at a point
which is optically near to the eye by using the half mirror.
Then the system detects the point of gaze by image dataprocessing by using the pupil-center-corneal reflection
method. A two-dimensional image blurred with the focus
point on the basis of the point of gaze is created on the GPU.
The display presents the two-dimensional image. The
process mentioned above is performed in real time.
Figure 7 shows the actual system, and figure 8 shows theimage of the eye gaze detection component.
We used a personal computer with an Intel PentiumR 4
CPU (2.4 GHz, 512 MB RAM) and a SAPPHIRE X1950
PRO graphic board. The infrared camera is the Dragonfly
Express from Point Grey Research. The size of the display is
17 inches and the distance from the observer to the display
is 18.5 inches. The resolution of the display is 1280 × 1024
pixels and the resolution of the image is also 1280 × 1024

pixels. The eye gaze detection rate is approximately 25
frames per second.

Half mirror

Right eyeInfra-red LED

4. Experiment
We conducted comparative experiments to quantitatively
evaluate the effects of presenting the image by using theprototype system. The images we compared are an all-infocus image and focus-controlled image. Figure 7 shows the
pictorial content of the images we presented.

Infra-redfilterInfra-red
cameraChin support

DisplayComputerVGA

to 1394 port

Figure 6: Structure of the implemented system.

Display
Figure 9: Image we presented in the experiment. The fiveteapots are positioned within the same scale. The wall andfloor are drawn and represented by a checked pattern forobservers to perceive the blur. The closeups image of thearea inside the red square are shown in figure 10 and figure
11.

Eye gaze
detection
component
Figure 7: The experimental setup

Figure 10: Closeup image of the all-in-focus image. Every pixel
is drawn clearly.

(a)Figure 8: Image of the eye gaze detection component.

(b)Figure 11: The closeup image of focus-controlled image. In theimage (a), the point of gaze of the observer is the cover handleof the yellow teapot. In the image (b), the point of gaze of theobserver is the wall.

4.1. Experimental procedure
At the beginning of the experiment, the eye gazedetection system was calibrated. Then, the two types ofimages were presented in turns to the subjects. First, thesystem showed the all-in-focus images to the subject for
approximately 20 s. Then, the system showed the focuscontrolled images for approximately 20 s. Subsequently, thesubject was given three questionnaires and asked to scorepoints ranging from 1 to 5 for each image by comparing theimages. The questionnaires were based on “depth
cognition,” “stress,” and “reality.” The questionnaire on
depth cognition required the evaluation of the amount of
depth information. The questionnaire on stress required theevaluation of the amount of stress experienced whenobserving the image. The questionnaire on reality required
the evaluation of the extent of reality conveyed by the image.
Further, the subjects were asked to comment or remark
freely. Seven men and one woman participated in theexperiment.

5. Results
The results of depth cognition, observation stress, andreality are shown in figure 12, figure 13, and figure 14,
respectively.

As shown in figure 13, one subject answered that stress wasincreased by focus-controlled image; however, the other fivesubjects observed the all-in-focus images with less stressthan that that required for focus-controlled images. Many ofthem said that they felt stress when observing focuscontrolled images because the images flickered during
blurring.
5
4

score
3

All-in-focus
全焦点画像
image
焦点可変画像Focus controlled
image

2
1
0
1

2

3

4

5

6

7

8

Participant No.

Figure 14: The score of the reality per subjects.

As shown in figure 14, two subjects answered that thereality is higher in observing focus-controlled images ascompared to that in observing all-in-focus images. However,
the other five subjects answered to the contrary. The reality
tended to be decreased by using this system as a whole.

5

6. Discussion

4
score
3

All-in-focus
image
Focus controlled
image

2
1
0
1

2

3

4

5

6

7

8

Participant No.

Figure 12: Score of the depth cognition per subject.

As shown in figure 12, all the subjects answered that thefocus-controlled images could enhance more depth
information as compared to the all-in-focus images. The
subjects commented that it was easy to perceive the depth
information of focus-controlled images.
5
4

score
3

All-in-focus
全焦点画像
image
焦点可変画像Focus controlled
image

2
1
0
1

2

3

4

5

6

7

8

Participant No.

Figure 13: Score of the stress per subject.

The results clearly indicate that our proposed systemenhances the depth information of a two-dimensional image.
However, the stress and reality experienced with this system
shoed the tendency to decrease. We discuss the reasons forthis by considering the opinions and comments of thesubjects.
A cause is that the system does not simulate the time for
focal accommodation. When we shift our gaze from one
object to another in the real world, it takes time toaccommodate the focus of the eyes. However, this system
could not simulate this phenomenon; hence, when theobserver shifted his/her gaze from one object to anotherobject in the images presented by the system , the focal
plane was changed in no delay. We consider this as thecause of the flicker of the image during blurring, which
resulted in the stress. In order to decrease the stress due toflicker, we improved the system to allow for more time forchanging the focal plane smoothly. The improved systemdecreased the stress due to flicker.
Another cause is the low accuracy of eye gaze detection.
The accuracy is significantly different for each subject
because it depends on the contrast of the pupil or the lengthof the contour of the pupil. When the accuracy is low, thepoint of gaze calculated by the system and the point at
which the subject gazes may be different, and the point at
which object the observer gazes cannot be drawn clearly. In
order to improve the eye gaze detection, we must use acamera with higher resolution or change the least square
method for approximating the contour of the pupil.

As discussed above, improving the stress and reality ofthe system by improving the accuracy of the eye gazedetection is the topic of our further study.

7. Conclusions
We have studied the display which has internal active
focus control system with the camera for eye gaze detection
to enhance depth information. By using the methods we
described in sections 3, which detect the eye gaze andgenerate the blur per pixel, we implemented the prototype
system and mounted the system on an existing display. We
conducted comparative experiments by using this system inorder to qualitatively evaluate the effect of presenting the
focus-controlled image. As a result, the system succeeds inenhancing the depth information. Our system presents
images with less stress and high sense of reality by
improving the accuracy of the eye gaze detection. In the
future, it will become easy to perceive the depth informationby combination our method and the other methods using
other depth cues such as binocular disparity or convergence.

References
Barsky,B.A. 2004. Vision-realistic rendering: Simulation of thescanned foveal image from wavefront data of human subjects.
Proceedings of Symposium on Applied Perception in Graphics
and Visualization, 73–81.
Hillarie,S., Lecuyer,A., Cozot,R., and Casiez,G. 2008. Using an
Eye-Tracking System to Improve Camera Motions and Depthof-Field Blur Effects in Virtual Environments.
John Krantz. 2001. Experiencing Sensation and Perception,
Prentice Hall. New Jersey.
Potmesil,M. and Chakravarty,I. 1981. A lens and aperture camera
model for synthetic image generation. In Proceedings of ACM
SIGGRAPH, vol. 19, 298–306.
Rokita,P. 1996. Generating depth-of-field effects in virtual reality
applications. In Prodeedings of IEEE Computer Graphics andApplications, vol.16, 18–21.
Yusuke,S., Hironobu,F., and Yutaka,H. 2004. Measurement of 3D
Eye Movements Using Image Processing.