Gaze detection apparatus, gaze detection method, and gaze detection program

A gaze detection apparatus performs a calibration operation including displaying a target image in multiple positions on a display screen sequentially, setting, in the display screen, corresponding areas corresponding to the target images displayed on the display screen sequentially, and calculating positional data of points of regard in display periods in which the respective target images are displayed; and determines, with respect to each of the target images, whether the positional data of the point of regard that is calculated in the calibration operation is present within the corresponding area.

BACKGROUND

1. Technical Field

The disclosure relates to a gaze detection apparatus, a gaze detection method, and a gaze detection program.

2. Description of the Related Art

A gaze detection apparatus that detects a position on an observation surface, such as a monitor screen, that an operator or a subject regards has been proposed. As a method of detecting a direction of a line of sight of a subject without wearing the device on the face, a technique of applying detection light to an eyeball of the subject, calculating a center of the pupil and a center of the corneal curvature from an image of the eye to which the detection light is applied, and detecting a vector toward the center of the pupil from the center of the corneal curvature as the direction of the line of sight of the subject is known. For such gaze detection apparatuses, a method of having an operator or a subject regard a target position on a monitor screen and performing calibration for each operator or subject in order to detect a line of sight of the subject or the operator more accurately has been proposed (For example, see Japanese Laid-open Patent Publication No. H8-321973).

According to H8-321973, when an error occurs in a calibration operation, a notification of the error is made and a return to the default state for performing the calibration operation again is made. In order to perform the calibration operation again thereafter, however, the operator, or the like, has to make manual inputs. For this reason, depending on the subject, the concentration may scatter and the accuracy of detection may lower.

SUMMARY

The disclosure was made in view of the above-described circumstances and an object of the disclosure is to provide a gaze detection apparatus, a gaze detection method, and a gaze detection program that make it possible to inhibit accuracy of detection from lowering.

A gaze detection apparatus according to the present disclosure comprising: a display screen on which an image is displayed; a light source configured to apply detection light to at least one of eyeballs of a subject; an image data acquisition unit configured to acquire image data on the eyeball to which the detection light is applied; a position detector configured to detect, from the acquired image data, positional data of a pupil center representing a center of a pupil of the eyeball to which the detection light is applied and positional data of a corneal reflection center representing a center of corneal reflection; a point-of-regard detector configured to calculate, based on a position of the pupil center and a position of the corneal reflection center, positional data of a point of regard of the subject on a plane containing the display screen; and a controller configured to perform a calibration operation including displaying a target image in multiple positions on the display screen sequentially, setting, in the display screen, corresponding areas corresponding to the target images displayed on the display screen sequentially, and calculating positional data of the points of regard in display periods in which the respective target images are displayed; determine, with respect to each of the target images, whether the positional data of the point of regard that is calculated in the calibration operation is present within the corresponding area; when the number of valid target images that are the target images on each of which it is determined that the positional data of the point of regard is present within the corresponding area is at or above a threshold, output the positional data of the points of regard of the valid target images as calibration data; and, when the number of the valid target images is under the threshold, cause the calibration operation to be repeated.

A gaze detection method according to the present disclosure comprising: applying detection light to at least one of eyeballs of a subject; acquiring image data on the eyeball to which the detection light is applied; detecting, from the acquired image data, positional data of a pupil center representing a center of a pupil of the eyeball to which the detection light is applied and positional data of a corneal reflection center representing a center of corneal reflection; calculating, based on a position of the pupil center and a position of the corneal reflection center, positional data of a point of regard of the subject on a plane containing a display screen; and performing a calibration operation including displaying a target image in multiple positions on the display screen sequentially, setting, in the display screen, corresponding areas corresponding to the target images displayed on the display screen sequentially, and calculating positional data of the points of regard in display periods in which the respective target images are displayed; determining, with respect to each of the target images, whether the positional data of the point of regard that is calculated in the calibration operation is present within the corresponding area; when the number of valid target images that are the target images on each of which it is determined that the positional data of the point of regard is present within the corresponding area is at or above a threshold, outputting the positional data of the points of regard of the valid target images as calibration data; and, when the number of the valid target images is under the threshold, causing the calibration operation to be repeated.

A non-transitory computer readable recording medium storing therein a gaze detection program according to the present disclosure that causes a computer to execute a process including: applying detection light to at least one of eyeballs of a subject; acquiring image data on the eyeball to which the detection light is applied; detecting, from the acquired image data, positional data of a pupil center representing a center of a pupil of the eyeball to which the detection light is applied and positional data of a corneal reflection center representing a center of corneal reflection; calculating, based on a position of the pupil center and a position of the corneal reflection center, positional data of a point of regard of the subject on a plane containing a display screen; and performing a calibration operation including displaying a target image in multiple positions on the display screen sequentially, setting, in the display screen, corresponding areas corresponding to the target images displayed on the display screen sequentially, and calculating positional data of the points of regard in display periods in which the respective target images are displayed; determining, with respect to each of the target images, whether the positional data of the point of regard that is calculated in the calibration operation is present within the corresponding area; when the number of valid target images that are the target images on each of which it is determined that the positional data of the point of regard is present within the corresponding area is at or above a threshold, outputting the positional data of the points of regard of the valid target images as calibration data; and, when the number of the valid target images is under the threshold, causing the calibration operation to be repeated.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

An embodiment of a gaze detection apparatus, a gaze detection method, and a gaze detection program of the disclosure will be described based on the drawings. The embodiment does not limit the invention. The components in the embodiment described below include ones that are replaceable by those skilled in the art and that are easily replaceable or ones that are substantially the same.

In the following description, a three-dimensional global coordinate system is set and a positional relationship of each unit will be described. A direction parallel to a first axis of a given surface serves as an X-axis direction, a direction parallel to a second axis of a given surface orthogonal to the first axis serves as a Y-axis direction, and a direction parallel to a third axis orthogonal to each of the first axis and the second axis serves as a Z-axis direction. The given surface contains an X-Y plane.

Gaze Detection Apparatus

FIG.1is a perspective view schematically illustrating an example of a gaze detection apparatus100according to a first embodiment. As illustrated inFIG.1, the gaze detection apparatus100includes a display device101, a stereo camera device102, and an illumination device103.

The display device101contains a flat panel display, such as a liquid crystal display (LCD) or an organic electroluminescence display (OLED). In the embodiment, the display device101includes a display screen101S. The display screen101S displays an image. In the embodiment, the display device101S displays, for example, an index for evaluating the visual function of a subject. The display screen101S is substantially parallel to the X-Y plane. The X-axis direction is the horizontal direction of the display screen101S, the Y-axis direction is the vertical direction of the display screen101S, and the Z-axis direction is a depth direction orthogonal to the display screen101S.

The stereo camera device102includes a first camera102A and a second camera102B. The stereo camera device102is arranged below the display screen101S of the display device101. The first camera102A and the second camera102B are arranged in the X-axis direction. The first camera102A is arranged in a negative X-direction with respect to the second camera102B. Each of the first camera102A and the second camera102B contains an infrared camera and includes an optical system capable of transmitting infrared light having a wavelength of, for example, 850 [nm] and an imaging device capable of receiving the infrared light.

The illumination device (light source)103includes a first light source103A and a second light source103B. The illumination device103is arranged below the display screen101S of the display device101. The first light source103A and the second light source103B are arranged in the X-axis direction. The first light source103A is arranged in the negative X-direction with respect to the first camera102A. The second light source103B is arranged in the positive X-axis direction with respect to the second camera102B. Each of the first light source103A and the second light source103B contains a light emitting diode (LED) light source and is capable of emitting infrared light of a wavelength of, for example, 850 [nm]. The first light source103A and the second light source103B may be arranged between the first camera102A and the second camera102B. Alternatively, the stereo camera102may be arranged above the display screen101S of the display device101.

The illumination device103emits infrared light that is detection light, thereby illuminating an eyeball111of the subject. The stereo camera device102captures an image of part of the eyeball111(collectively referred to as an “eyeball”) with the second camera102B when the detection light that is emitted from the first light source103A is applied to the eyeball111and captures an image of the eyeball111with the first camera102A when detection light that is emitted from the second light source103B is applied to the eyeball111.

A frame synchronization signal is output from at least one of the first camera102A and the second camera102B. The first light source103A and the second light source103B emit detection light based on the frame synchronization signal. The first camera102A captures image data on the eyeball111when detection light that is emitted from the second light source103B is applied to the eyeball111. The second camera102B captures image data of the eyeball111when detection light that is emitted from the first light source103A is applied to the eyeball111.

When the detection light is applied to the eyeball111, part of the detection light reflects on a pupil112and the light from the pupil112is incident on the stereo camera device102. When the detection light is applied to the eyeball111, a corneal reflection image113that is a virtual image of the cornea is formed on the eyeball111and the light from the corneal reflection image113is incident on the stereo camera device102.

Appropriately setting relative positions among the first camera102A, the second camera102B, the first light source103A, and the second light source103B lowers the intensity of light that is incident on the stereo camera device102from the pupil112to the stereo camera device102and increases the intensity of light that is incident on the stereo camera device102from the corneal reflection image113. In other words, an image of the pupil112that is captured by the stereo camera device102has a low brightness and an image of the corneal reflection image113has a high brightness. The stereo camera device102is able to detect a position of the pupil112and a position of the corneal reflection image113based on the brightness of the captured images.

FIG.2is a diagram illustrating an example of a hardware configuration of the gaze detection apparatus100according to the embodiment. As illustrated inFIG.2, the gaze detection apparatus100includes the display device101, the stereo camera device102, the illumination device103, a computer system (controller)20, an input-output interface device30, a drive circuit40, an output device50, and an input device60.

The computer system20, the drive circuit40, the output device50, and the input device60communicate data via the input-output interface device30. The computer system20contains a processing unit20A and a storage device20B. The processing unit20A contains a microprocessor, such as a central processing unit (CPU). The storage device20B contains a memory, such as a read only memory (ROM) and a random access memory (RAM), or a storage. The processing unit20A performs operations according to a computer program20C that is stored in the storage device20B.

The drive circuit40generates drive signals and outputs the drive signals to the display device101, the stereo camera device102, and the illumination device103. The drive circuit40supplies image data on the eyeball111that is captured with the stereo camera device102to the computer system20via the input-output interface device30.

The output device50contains a display device, such as a flat panel display. Note that the output device50may contain a printing device. The input device60is operated and thus generates input data. The input device60contains a keyboard or a mouse for the computer system. Note that the input device60may contain a touch sensor that is provided on a display screen of the output device50that is a display device.

In the embodiment, the display device101and the computer system20are independent devices. Note that the display device101and the computer system20may be integrated. For example, when the gaze detection apparatus100contains a tablet personal computer, the computer system20, the input-output interface device30, the drive circuit40, and the display device101may be mounted on the tablet personal computer.

FIG.3is a functional block diagram illustrating an example of the gaze detection apparatus100according to the embodiment. As illustrated inFIG.3, the input-output interface device30includes an input-output unit302. The drive circuit40includes a display device driver402that generates a drive signal for driving the display device101and outputs the drive signal to the display device101; a first camera input-output unit404A that generates a drive signal for driving the first camera102A and outputs the drive signal to the first camera102A; a second camera input-output unit404B that generates a drive signal for driving the second camera102B and outputs the drive signal to the second camera102B; and a light source driver406that generates dive signals for driving the first light source103A and the second light source103B and outputs the drive signals to the first light source103A and the second light source103B. The first camera input-output unit404A supplies image data on the eyeball111that is captured by the first camera102A to the computer system20via the input-output unit302. The second camera input-output unit404B supplies the image data on the eyeball111that is captured by the second camera102B to the computer system20via the input-output unit302.

The computer system20controls the gaze detection apparatus100. The computer system20includes a display controller202, a light source controller204, an image data acquisition unit206, an input data acquisition unit208, a position detector210, a curvature center calculator212, a point-of-regard detector214, an area setting unit216, a determination unit218, a processor220, a calibration controller222, a storage224, and an output controller226. The functions of the computer system20are implemented by the processing unit20A and the storage device20B.

The display controller202displays an image to be shown to the subject on the display screen101S of the display device101. The display controller202is capable of displaying, for example, a target image in a calibration operation in multiple positions (target positions) on the display screen101S. The display controller202may display the target image in each of the target positions such that the target image switches sequentially or display the target image such that the target image shifts sequentially to the target positions within the display screen101S. The number of target positions in which the target image is displayed can be, for example, set by the operator by making an input via the input device60, or the like.

The light source controller204controls the light source driver406, thereby controlling the operational state of the first light source103A and the second light source103B. The light source controller204controls the first light source103A and the second light source103B such that the first light source103A and the second light source103B emit detection light at different sets of timing.

The image data acquisition unit206acquires image data on the eyeball111of the subject of which image is captured by the stereo camera device102including the first camera102A and the second camera102B from the stereo camera device102via the input-output unit302.

The input data acquisition unit208acquires the input data that is generated by operating the input device60from the input device60via the input-output unit302.

The position detector210detects positional data of a pupil center based on the image data on the eyeball111that is acquired by the image data acquisition unit206. Based on the image data of the eyeball111that is acquired by the image data acquisition unit206, the position detector210detects positional data of a corneal reflection center. The pupil center is the center of the pupil112. The corneal reflection center is the center of the corneal reflection image113. The position detector210detects the positional data of the pupil center and the positional data of the corneal reflection center with respect to the eyeball111to which the detection light is applied.

Based on the image data on the eyeball111that is acquired by the image data acquisition unit206, the curvature center calculator212calculates positional data of the center of corneal curvature of the eyeball111.

Based on the image data on the eyeball111that is acquired by the image data acquisition unit206, the point-of-regard detector214detects positional data of the point of regard of the subject. In the embodiment, the positional data of the point of regard refers to positional data of the intersection between a line-of-sight vector of the subject that is defined by the three-dimensional global coordinate system and a plane containing the display screen101S of the display device101. Based on the positional data of the pupil center and the positional data of the center of corneal curvature that are acquired from the image data on the eyeball111, the point-of-regard detector214detects a light-of-sight vector of each of the left and right eyeballs111. After the line-of-sight vector is detected, the point-of-regard detector214detects the positional data of the point of regard representing the intersection between the line-of-sight vector and the display screen101S.

In a display period in which the target image is displayed on the display screen101S, the area setting unit216sets a corresponding area corresponding to the target image that is displayed on the display screen101S.

In the display period in which the target image is made on the display screen101S, the determination unit218determines whether the point of regard is present within the corresponding area. The determination unit218determines whether the point of regard is present within the corresponding area, for example, regularly at given intervals. The given interval may be, for example, the period of the frame synchronization signal (for example, every 50 [msec]) that is output from the first camera102A and the second camera102B.

Based on data on the determination by the determination unit218, the processor220counts the number of times it is determined that the point of regard is present within the corresponding area. The processor220includes a counter that counts, for the corresponding area, the number of times the determination is made.

The calibration controller222causes the calibration operation to be performed. The calibration operation includes displaying a target image in multiple positions on the display screen101S, setting a corresponding area corresponding to the target image that is displayed on the display screen101S on the display screen101S, and calculating positional data of a point of regard in a display period in which each target image is displayed.

The calibration controller222determines, with respect to each target image, whether the positional data of the point of regard that is calculated in the calibration operation is present within the corresponding area. When performing the determination, the calibration controller222determines, with respect to each target image, whether the point of regard is present within the corresponding area based on the number of times the determination is made that is counted by the processor220. For example, with respect to each target image, when the number of times it is determined that the point of regard is present within the corresponding area is at or above a given ratio to the number of sets of detection of positional data of the point of regard, the calibration controller222determines that the point of regard is present within the corresponding area with respect to the target image. A target image on which it is determined that the positional data of the point of regard is present within the corresponding area is referred to as a valid target image below. With respect to each target image, when the number of times it is determined that the point of regard is present within the corresponding area is under the given ratio to the number of sets of detection of positional data of the point of regard, the calibration controller222determines that the point of regard is not present within the corresponding area with respect to the target area.

The calibration controller222determines whether the number of valid target image is at or above a threshold. The threshold can be set, for example, by the operator by making an input via the input device60, or the like. When the number of valid target images is at or above the threshold, the calibration controller222outputs the positional data of the points of regard with respect to the valid target images as calibration data. When the number of valid target images is under the threshold, the calibration controller222causes the calibration operation to be repeated.

When causing the calibration operation to be repeated, the calibration controller222displays a target image in each of the same multiple target positions as those in the latest calibration operation and, when the target image that is displayed in the target position that is the same between the repeated calibration operation and the latest calibration operation or a calibration operation before the latest calibration operation is successively determined as a valid target image, determines other all valid target images in the latest calibration operation or a calibration operation before the latest calibration operation among the successive calibration operations as valid target images in the repeated calibration operation.

When causing the calibration operation to be repeated, the calibration controller222is able to display a target image whose appearance differs from that of the target image that is displayed in the latest calibration operation. In this case, for example, based on an instruction of the calibration controller222, the display controller selects one of multiple types of target images and displays the selected one.

When the calibration operation is performed for a given number of times and the number of valid target images is under the threshold in every calibration operation, the calibration controller222makes an output indicating that the calibration operation is erroneous. The given number of times the calibration operation is caused to be performed can be set, for example, by the operator by making an input via the input device60, or the like.

The storage224stores various types of data and programs on the above-described gaze detection. The storage224stores display data (images, videos, etc.) corresponding to the number of positions in which the target image is displayed. The storage224stores, for example, the display data on multiple types of target images with different appearances. The storage224stores positional data of the point of regard that is calculated in each calibration operation.

The storage224stores a gaze detection program that causes a computer to execute a process of applying detection light to at least one of eyeballs of a subject; a process of acquiring image data on the eyeball to which the detection light is applied; a process of detecting, from the acquired image data, positional data of a pupil center representing a center of a pupil of the eyeball to which the detection light is applied and positional data of a corneal reflection center representing a center of corneal reflection; a process of calculating positional data of a point of regard of the subject on a plane containing a display screen based on a position of the pupil center and a position of the corneal reflection center; and a process of performing a calibration operation including sequentially displaying a target image in multiple positions on the display screen, sequentially setting, in the display screen, corresponding areas corresponding to the target images displayed on the display screen, and calculating positional data of the point of regard in a display period during which each of the target images is displayed; determining, with respect to each of the target images, whether the positional data of the point of regard that is calculated in the calibration operation is present within the corresponding area; when the number of valid target images that are target images on each of which it is determined that the positional data of the point of regard is present within the corresponding area is at or above a threshold, outputting the positional data of the points of regard with respect to the valid images as calibration data; and, when the number of valid images is under the threshold, causing the calibration operation to be repeated.

The output controller226outputs data to at least one of the display device101and the output device50.

An overview of a process performed by the curvature center calculator212will be described. The curvature center calculator212calculates positional data of the center of corneal curvature of the eyeball111based on the image data on the eyeball111.FIGS.4and5are schematic diagrams for explaining a method of calculating positional data of a corneal curvature center110according to the embodiment.FIG.4illustrates an example in which the eyeball111is illuminated with a single light source103C.FIG.5illustrates an example in which the eyeball111is illuminated with the first light source103A and the second light source103B.

First of all, the example illustrated inFIG.4will be described. The light source103C is arranged between the first camera102A and the second camera102B. A pupil center112C is the center of the pupil112. A corneal reflection center113C is the center of the corneal reflection image113. InFIG.4, the pupil center112C represents the center of the pupil at the time when the eyeball111is illuminated with the single light source103C. The corneal reflection center113C represents the center of corneal reflection at the time when the eyeball111is illuminated with the single light source103C. The corneal reflection center113C is present on a straight line connecting the light source103C and the corneal curvature center110. The corneal reflection center113C is positioned at an intermediate point between the surface of the cornea and the corneal curvature center110. A corneal curvature radius109is a distance between the surface of the cornea and the corneal curvature center110. Positional data of the corneal reflection center113C is detected by the stereo camera device102. The corneal curvature center110is present on a straight line connecting the light source103C and the corneal reflection center113C. The corneal curvature center calculator212calculates, as positional data of the corneal curvature center110, positional data that makes the distance from the corneal reflection center113C on the straight line be at a given value. The given value is a value that is determined previously from a value of a general corneal curvature radius and the given value is stored in the storage224.

The example illustrated inFIG.5will be described. In the embodiment, the first camera102A and the second light source103B and the second camera102B and the first light source103A are arranged at horizontally symmetrical positions with respect to a straight line passing through the intermediate position between the first camera102A and the second camera102B. It can be thought that a virtual light source103V is in the intermediate position between the first camera102A and the second camera102B. A corneal reflection center121represents a center of corneal reflection in an image of the eyeball111that is captured with the second camera102B. A corneal reflection center122represents a center of corneal reflection in an image of the eyeball111that is captured with the first camera102A. A corneal reflection center124represents a center of corneal reflection corresponding to the virtual light source103V. Positional data of the corneal reflection center124is calculated based on positional data of the corneal reflection center121and positional data of the corneal reflection center122. The stereo camera device102detects the positional data of the corneal reflection center121and the positional data of the corneal reflection center122in the three-dimensional local coordinate system that is defined in the stereo camera device102. For the stereo camera device102, camera calibration by a stereo calibration method is performed previously and a conversion parameter to convert the three-dimensional local coordinate system into a three-dimensional global coordinate system is calculated. The conversion parameter is stored in the storage224. Using the conversion parameter, the curvature center calculator212converts the positional data of the corneal reflection center121and the positional data of the corneal reflection center122into positional data in the three-dimensional global coordinate system. Based on the positional data of the corneal reflection center121and the positional data of the corneal reflection center122that are defined in the three-dimensional global coordinate system, the curvature center calculator212calculates the positional data of the corneal reflection center124in the three-dimensional global coordinate system. The corneal curvature center110is on a straight line123connecting the virtual light source103V and the corneal reflection center124. The curvature center calculator212calculates the positional data that makes the distance from the corneal reflection center124on the straight line123be a given value as the positional data of the corneal curvature center110. The given value is a value that is determined previously from a value of a general corneal curvature radius and the given value is stored in the storage224.

As described above, even when there are two light sources, the corneal curvature center110is calculated in the same manner as that in the case of a single light source.

The corneal curvature radius109is the distance between the surface of the cornea and the corneal curvature center110. By calculating the positional data of the surface of the cornea and the positional data of the corneal curvature center110, the corneal curvature radius109is calculated.

Gaze Detection Method

An example of the gaze detection method according to the embodiment will be described. In the gaze detection method according to the embodiment, after performing the calibration process, a process of detecting a point of regard is performed.FIG.6is a schematic diagram for explaining an example of the calibration process according to the embodiment. In the calibration process, a target position130is set for a subject to regard. The target position130is defined in the three-dimensional global coordinate system. The display controller202displays a target image in the target position130. A straight line141is a straight line connecting the virtual light source103V and the corneal reflection center113C. A straight line142is a straight line connecting the target position130and the pupil center112C. The corneal curvature center110is the intersection between the straight line141and the straight line142. The corneal curvature center calculator212is able to calculate positional data of the corneal curvature center110based on positional data of the virtual light source103V, positional data of the target position130, positional data of the pupil center112C, and positional data of the corneal reflection center113C. The point-of-regard detector214is able to calculate positional data of the point of gaze from the calculated center of corneal curvature. Details of the calibration process will be described below.

A point-of-regard detection process will be described. The point-of-regard detection process is performed after the calibration process. Based on the image data on the eyeball111, the point-of-regard detector214calculates a line-of-sight vector of the subject and positional data of the point of regard.FIG.7is a schematic view for explaining an example of the point-of-regard detection process according to the embodiment. InFIG.7, a point of regard165represents a point of regard that is calculated from the corneal curvature center that is calculated using the value of a general curvature radius. A point of regard166represents a point of regard that is calculated from a center of corneal curvature that is calculated from the corneal curvature center that is calculated using a distance126that is calculated in the calibration process. The pupil center112C represents the center of the pupil that is calculated in the calibration process and the corneal reflection center113C represents the center of corneal reflection that is calculated in the calibration process. A straight line173is a straight line connecting the virtual light source103V and the corneal reflection center113C. The corneal curvature center110is the position of the corneal curvature center that is calculated from the value of a general curvature radius. The distance126is the distance between the pupil center112C and the corneal curvature center110that are calculated by the calibration process. A corneal curvature center110H represents the position of the post-correction corneal curvature center obtained by correcting the corneal curvature center110using the distance126. The corneal curvature center110H is calculated from the fact that the corneal curvature center110is on the straight line173and that the distance between the pupil center112C and the corneal curvature center110is the distance126. Accordingly, a line of sight117that is calculated in the case where a general curvature radius is used is corrected into a line of sight178. The point of regard on the display screen101S of the display device101is corrected from the point of regard165to a point of regard166.

Calibration Process

The calibration process described above will be described. In the calibration process, the display controller202displays a target image in multiple target positions.FIG.8is a diagram illustrating a display example in which a target image M is displayed on the display screen101S. As illustrated inFIG.8, the display controller202displays the target image M in the target positions130sequentially. The number of the target positions130in which the target image M is displayed can be set by the operator previously via the input device60, or the like. The display controller202displays the target images M sequentially in the set number of the target positions130one by one. For example, when the number of the target positions130is set at “5”, as illustrated inFIG.8, the display controller202displays the target image M in each of five target positions130that are a first position131, a second position132, a third position133, a fourth position134, and a fifth position135on the display screen101S such that the target image switches sequentially.FIG.8represents the five target images M on the five target positions130and, practically, the single target image M is sequentially displayed in each target position. The display controller202displays the target image M such that the period of display is equal among the target positions130.

The area setting unit216sets a corresponding area A corresponding to the target image M that is displayed on the display screen101S. As illustrated inFIG.8, when the target image M is displayed on the five target positions130from the first position131to the fifth position135, the area setting unit216sets corresponding areas A corresponding to the respective target images M in the display periods in which the respective target positions130are displayed. For example, the area setting unit216sets a corresponding area A1corresponding to the target image M in the first position131in the display period in which the target image M is displayed in the first position131. In the display period in which the target image M is displayed in the second position132, the area setting unit216sets a corresponding area A2corresponding to the target image M in the second position132. In the display period in which the target image M is displayed in the third position133, the area setting unit216sets a corresponding area A3corresponding to the target image M in the third position133. In the display period in which the target image M is displayed in the fourth position134, the area setting unit216sets a corresponding area A4corresponding to the target image M in the fourth position134. In the display period in which the target image M is displayed in the fifth position135, the area setting unit216sets a corresponding area A5corresponding to the target image M in the fifth position135. The area setting unit216sequentially switches the setting of the corresponding area A from the corresponding area A1to the corresponding area A5in synchronization with the sets of timing at which the position of display of the target image M switches sequentially from the first position131to the fifth positions135.

As described above, in each of the display periods in which the target image M is displayed, the image data acquisition unit206acquires the image data on the left and right eyeballs, the position detector210detects the positional data of the pupil center and the positional data of the center of corneal reflection, and the point-of-regard detector214calculates the positional data of the point of regard. The process is performed according to the period of the frame synchronization signal that is, for example, output from the first camera102A and the second camera102B (for example, every 50 [msec]). The first camera102A and the second camera102B capture images synchronously.

FIG.9is a diagram illustrating an example of detection areas where points of regard are detected by the calibration operation. Detection areas P1to P5illustrated inFIG.9schematically illustrate areas of detected points of regard in the respective display periods in which the target image M is displayed in the first to fifth positions131to135. The points of regard that are calculated by the calibration operation are not displayed on the display screen101S practically.

The calibration controller222determines, on each of the target images M, whether the positional data of the point of regard that is calculated in the above-described calibration operation is present within the corresponding area A. As illustrated inFIG.9, in the display period in which the target image M is displayed in the first position131, the area of the calculated point of regard is present within the corresponding area A1. Thus, the calibration controller222determines that the positional data of the point of regard is present within the corresponding area A1. In this case, the target image M that is displayed in the first position131serves as a valid target image.

In the display period in which the target image M is displayed in the second position132, the area of the calculated point of regard is present within the corresponding area A2. Thus, the calibration controller222determines that the positional data of the point of regard is present within the corresponding area A2. In this case, the target image M that is displayed in the second position132serves as a valid target image.

In the display period in which the target image M is displayed in the third position133, the area of the calculated point of regard is not present within the corresponding area A3. Thus, the calibration controller222determines that the positional data of the point of regard is not present within the corresponding area A3. In this case, the target image M that is displayed in the third position133does not serve as a valid target image.

In the display period in which the target image M is displayed in the fourth position134, the area of the calculated point of regard is present within the corresponding area A4. Thus, the calibration controller222determines that the positional data of the point of regard is present within the corresponding area A4. In this case, the target image M that is displayed in the fourth position134serves as a valid target image.

In the display period in which the target image M is displayed in the fifth position135, the area of the calculated point of regard is not present within the corresponding area A5. Thus, the calibration controller222determines that the positional data of the point of regard is not present within the corresponding area A5. In this case, the target image M that is displayed in the fifth position135does not serve as a valid target image.

As a result, in the above-described calibration operation, three valid target images are detected. The calibration controller222determines whether the number of the valid target images that are detected as described above is at or above the threshold. The threshold can be set, for example, by the operator by making an input via the input device60, or the like.

The number of the valid target images is three here and thus, for example, when the threshold is set at or under “3”, the calibration controller222determines that the number of the valid target images is at or above the threshold. When the number of the valid target images is at or above the threshold, the calibration controller222outputs positional data on the points of regard with respect to the valid images as calibration data. In other words, the calibration controller222outputs, as calibration data, positional data of the point of regard in the detection area P1corresponding to the target image M in the first position131. The calibration controller222outputs, as calibration data, positional data of the point of regard in the detection area P2corresponding to the target image M in the second position132. The calibration controller222outputs, as calibration data, positional data of the point of regard in the detection area P4corresponding to the target image M in the fourth position134.

On the other hand, when the threshold is set at or above “4”, the calibration controller222determines that the number of the valid target images is under the threshold. When the number of the valid target images is under the threshold, the calibration controller222causes the calibration operation to be repeated. In this case, the calibration controller222stores the positional data of the points of regard that are detected in the above-described calibration operation in the storage224.

When the calibration operation is caused to be repeated, the display controller202displays the target image M in the same target positions130as those in the latest calibration operation. In other words, the display controller202displays the target image M in the five target positions130from the first position131to the fifth position135. The area setting unit216sets a corresponding area A in a period in which the target image M is displayed in each target position130by the same process as that of the latest calibration operation. The image data acquisition unit206acquires image data on the left and right eyeballs, the position detector210detects positional data of the pupil center and positional data of the corneal reflection center, and the point-of-regard detector214calculates positional data of the point of regard. In this manner, the computer system20automatically repeats the calibration operation and thus it is possible to efficiently acquire more natural and accurate measurement results while maintaining concentration of the subject without hindering the flow of the gaze detection process. When repeating the calibration operation, the computer system20shifts to the calibration operation without making a notification of the fact, which thus inhibits the concentration of the subject from lapsing.

FIG.10is a diagram illustrating an example of detection areas where points of regard are detected by the repeated calibration operation. In the example illustrated inFIG.10, in the display periods in which the target image M is displayed in the first position131, the third position133and the fourth position134, the areas of the calculated points of regard are present respectively within the corresponding areas A1, A3and A4. Thus, in the repeated calibration operation, the three target images that are displayed in the first position131, the third position133and the fourth position134are detected as valid target images.

The above-described calibration operation is of a system in which the target image M that is displayed sequentially in the target positions130on the display screen101S is followed and regarded. The inventors found that, in the system, a subject who performs following and regarding correctly is characterized in performing following and regarding in the same way of viewing in each calibration operation. In other words, when distribution of positional data of the points of regard coincides or similar among the multiple numbers of calibration operation, it can be estimated that the subject views in the same manner. For this reason, when the target image M that is displayed in the same position as that of the target image M that is determined as a valid target image in the repeated calibration operation is successively determined as a valid target image in the latest calibration operation or a calibration operation before the latest calibration, it can be considered that the subject views in the same manner.

Thus, when causing the calibration operation to be repeated, the calibration controller222stores positional data of a center point that is calculated in the latest calibration operation in the storage224with respect to each target position130. The calibration controller222then displays the target image M in the same target positions130as those in the latest calibration operation, that is, each of the first to fifth positions131to135. Thereafter, when the target image M that is displayed in the target position130that is the same between the repeated calibration operation and the latest calibration operation or a calibration operation before the latest calibration operation is successively determined as a valid target image, the calibration controller222determines other all valid target images in the latest calibration operation or a calibration operation before the latest calibration operation among the successive calibration operations as valid target images in the repeated calibration operation.

In the calibration operation illustrated inFIG.9and the repeated calibration operation illustrated inFIG.10, two target images M that are displayed in the first position131and the fourth position134are detected as valid target images together. Accordingly, the calibration controller222determines that other all valid target images in the latest calibration operation are valid target images in the repeated calibration operation. In other words, the target image M that is displayed in the second position132and that is determined as a valid target image in the latest calibration operation is also detected as a valid target image in the repeated calibration operation. As a result, in the calibration operation illustrated inFIG.10, four valid target images are detected.

When the repeated calibration operation illustrated inFIG.10is performed and then the further repeated calibration operation is performed (referred to as the further repeated calibration operation below), the calibration controller222stores positional data of a center point that is calculated in the latest calibration operation in the storage224with respect to each target position130. The calibration controller222then displays the target image M in each of the first to fifth positions131to135. Thereafter, when, at least one of the target images M that are displayed in the first position131and the fourth position134that are the same target positions130as those in the latest two calibration operations is determined as a valid target image, the calibration controller222determines other all valid target images in the two calibration operations as valid target images in the further repeated calibration operation.

As described above, the calibration controller222stores the positional data of the point of regard that is calculated in each calibration operation and, when a valid target image in the repeated calibration operation is determined, uses the positional data of the point of regard that is stored in the storage as feedback data.

The calibration controller222determines whether the number of the valid target images detected as described above is at or above the threshold and, when the number of the valid target images is at or above the threshold, outputs positional data on the points of regard with respect to the valid target images as calibration data. When the number of the valid target images is under the threshold, the calibration controller222causes the calibration operation to be repeated.

FIG.11is a diagram illustrating another example of the detection areas where points of regard are detected by the repeated calibration operation. In the example illustrated inFIG.11, in the display period in which the target image M is displayed in the third position133and the fifth position135, the areas of the calculated points of regard are present within the corresponding areas A3and A5, respectively. Thus, in the repeated calibration operation, the two target images M that are displayed in the third position133and the fifth position135are detected as valid target images.

On the other hand, in the repeated calibration operation illustrated inFIG.11, there is no valid target image that is detected at the same position as that in the calibration operation illustrated inFIG.9that is the latest calibration operation. Thus, in this case, the calibration controller222does not determine the valid target images in the latest calibration operation as valid target images in the repeated calibration operation. As a result, in the calibration operation illustrated inFIG.11, two valid target images are detected.

In the example illustrated inFIGS.10and11, the mode in which, when performing the repeated calibration operation, the display controller202displays the target image whose appearance is the same as the target image M displayed in the latest calibration operation has been described; however, the calibration operation is not limited to this.FIG.12is a diagram illustrating another display example in which the target image is displayed on the display screen101S. As illustrated inFIG.12, when causing the calibration operation to be repeated, the display controller202is able to display a target image MA whose appearance differs from that of the target image M that is displayed in the latest calibration operation. In this case, the display controller202displays the target image M in the same target positions130as those in the latest calibration operation. In this manner, displaying the target image MA whose appearance differs from that of the target image M in the latest calibration operation makes it possible to inhibit the concentration and regarding rate of the subject from lowering because the subject gets bored.

When the above-described calibration is performed for a given number of times and the number of target images is under the threshold in every calibration operation, the calibration controller222makes an output indicating that the calibration operation is erroneous. The given number of times can be set, for example, by the operator by making an input via the input device60, or the like.

The example of the gaze detection method according to the embodiment will be described with reference toFIG.13.FIG.13is a flowchart illustrating an example of the calibration process in the gaze detection method according to the embodiment. As illustrated inFIG.13, first of all, the calibration controller222sets a number (X) of the target positions130that are displayed on the display screen101S (step S101). The calibration controller222sets the number X of the target positions130based on a result of an input in the input device60. Note that X is an integer that is 2 or larger.

The calibration controller222sets a threshold (Y) of valid target images (step S102). The calibration controller222sets the threshold Y of valid target images based on a result of an input in the input device60. Note that Y is an integer that is 1 or larger or X or smaller.

The calibration controller222sets a number of retries (R) in which the repeated calibration operation is performed (step S103). The calibration controller222sets the number of retries R based on, for example, the result of an input in the input device60. Note that R is an integer that is 1 or larger. Note that R may be 0. In this case, a setting is made such that the repeated calibration is not performed.

After the number X of the target positions130, the threshold Y of valid target images, and the number of times R the repeated calibration operation is performed are set, the calibration operation is caused to be performed (step S104). In the calibration operation, a target image M is displayed in the multiple target positions130based on the set number X of the target positions130and corresponding areas A corresponding to the target images M are set in the periods in which the target images M are displayed, respectively. In the display period in which the target image M is displayed, the image data acquisition unit206acquires image data on the left and right eyeballs, the position detector210detects positional data of the pupil center and positional data on the corneal reflection center, and the point-of-regard detector214calculates positional data of the point of regard.

After the calibration operation, the calibration controller222calculates a number (Z) of the target images M that are determined as valid target images (step S105). At step S105, when the target image M that is displayed in the same target position130between the repeated calibration operation and the latest calibration operation or a calibration operation before the latest calibration operation is successively determined as a valid target image, the calibration controller222determines other all valid target images in the latest calibration operation or a calibration operation before the latest calibration operation among the successive calibration operations as valid target images in the repeated calibration operation. After calculating the number Z of the valid target images, the calibration controller222determines whether the number Z of the valid target images is at or above the threshold Y (step S106). When it is determined that the number Z of the valid target images is at or above the threshold Y (YES at step S106), the calibration controller222outputs the positional data of the points of regard on which valid target images are determined as calibration data and ends the calibration operation.

On the other hand, when it is determined that the number Z of the valid target images is under the threshold Y (NO at step S106), the calibration controller222determines whether the number of retries in which the repeated calibration operation is performed until the determination reaches the set number of retries R (step S107). When the number of retries of the calibration operation reaches the set number R as a result of the determination (YES at step S107), the calibration controller222makes an output indicating that the calibration operation is erroneous (step S110) and ends the process.

When the number of retries of the calibration operation does not reach the set number R (NO at step S107), the calibration controller222causes the repeated calibration operation to be performed. In the repeated calibration operation, when the type of target image M is changed to a target image with different appearance, the calibration controller222transmits an instruction to change the type of target image M to the display controller202(step S108). In this case, in the repeated calibration operation, a target image whose appearance differs from that in the latest calibration operation is displayed on the display screen101S. When causing the repeated calibration operation to be performed, the calibration controller222stores positional data on points of regard that is a detection result of the latest calibration operation in the storage224(step S109). Thereafter, the operation at and after step S104is caused to be performed repeatedly.

As described above, the gaze detection apparatus according to the embodiment includes the display screen101S on which an image is displayed; the illumination device103that applies detection light to at least one of eyeballs of a subject; the image data acquisition unit206that acquires image data on the eyeball to which the detection light is applied; the position detector210that detects, from the acquired image data, positional data of a pupil center representing the center of a pupil of the eyeball to which the detection light is applied and positional data of a corneal reflection center representing the center of corneal reflection; the point-of-regard detector214that calculates, based on the position of the pupil center and the position of the corneal reflection center, positional data of a point of regard of the subject on a plane containing the display screen101S; and the computer system20that performs a calibration operation including displaying a target image M in multiple target positions130on the display screen101S sequentially, sequentially setting, in the display screen101S, corresponding areas A corresponding to the target images M displayed on the display screen101S, and calculating positional data of the points of regard in display periods in which the respective target images M are displayed, determines, with respect to each of the target images M, whether the positional data of the point of regard that is calculated in the calibration operation is present within the corresponding area A, when a number Z of valid target images that are target images M on each of which it is determined that the positional data of the point of regard is present within the corresponding area A is at or above a threshold Y, outputs the positional data of the points of regard with respect to the valid target images as calibration data, and, when the number Z of the valid target images is under the threshold Y, causes the calibration operation to be repeated.

The gaze detection method according to the embodiment includes: applying detection light to at least one of eyeballs of a subject; acquiring image data on the eyeball to which the detection light is applied; detecting, from the acquired image data, positional data of a pupil center representing the center of a pupil of the eyeball to which the detection light is applied and positional data of a corneal reflection center representing the center of corneal reflection; calculating, based on the position of the pupil center and the position of the corneal reflection center, positional data of a point of regard of the subject on a plane containing the display screen101S; and the computer system20that performs a calibration operation including displaying a target image M in multiple target positions130on the display screen101S sequentially, sequentially setting, in the display screen101S, corresponding areas A corresponding to the target images M displayed on the display screen101S, and calculating positional data of the points of regard in display periods in which the respective target images M are displayed, determining, with respect to each of the target images M, whether the positional data of the point of regard that is calculated in the calibration operation is present within the corresponding area A, when a number Z of valid target images that are target images M on each of which it is determined that the positional data of the point of regard is present within the corresponding area A is at or above a threshold Y, outputting the positional data of the points of regard with respect to the valid target images as calibration data, and, when the number Z of the valid target images is under the threshold Y, causing the calibration operation to be repeated.

The gaze detection program according to the embodiment causes a computer to perform a process including: applying detection light to at least one of eyeballs of a subject; acquiring image data on the eyeball to which the detection light is applied; detecting, from the acquired image data, positional data of a pupil center representing the center of a pupil of the eyeball to which the detection light is applied and positional data of a corneal reflection center representing the center of corneal reflection; calculating, based on the position of the pupil center and the position of the corneal reflection center, positional data of a point of regard of the subject on a plane containing the display screen101S; and the computer system20that performs a calibration operation including displaying a target image M in multiple target positions130on the display screen101S sequentially, sequentially setting, in the display screen101S, corresponding areas A corresponding to the target images M displayed on the display screen101S, and calculating positional data of the points of regard in display periods in which the respective target images M are displayed, determining, with respect to each of the target images M, whether the positional data of the point of regard that is calculated in the calibration operation is present within the corresponding area A, when a number Z of valid target images that are target images M on each of which it is determined that the positional data of the point of regard is present within the corresponding area A is at or above a threshold Y, outputting the positional data of the points of regard with respect to the valid target images as calibration data, and, when the number Z of the valid target images is under the threshold Y, causing the calibration operation to be repeated.

Because of such a configuration, the computer system20automatically performs the repeated calibration operation and thus it is possible to efficiently acquire more natural and accurate measurement results while maintaining concentration of the subject without hindering the flow of the gaze detection process. Thus, it is possible to inhibit the detection accuracy from lowering.

When causing the calibration operation to be repeated, the calibration controller222displays the target image M in each of the first to fifth positions131to135that are the same target positions130as those of the latest calibration operation and, when the target image M that is displayed in the target position130that is the same between the repeated calibration operation and the latest calibration operation or a calibration operation before the latest calibration operation is determined as a valid target image successively, determines other all valid target images in the latest calibration operation or the calibration operation before the latest calibration operation among the successive calibration operations as valid target images in the repeated calibration operation. Accordingly, using the characteristics of the subject makes it possible to perform an efficient calibration operation.

When causing the calibration operation to be repeated, the calibration controller222displays the target image MA whose appearance differs from that of the target image M that is displayed in the latest calibration operation. This makes it possible to inhibit the concentration and regarding rate of the subject from lowering because the subject gets bored.

When the calibration operation is performed for a given number of time R and the number Z of valid target images is under the threshold Y in every calibration operation, the calibration controller222makes an output indicating that the calibration operation is erroneous. Thus, when the number Z of valid target images is under the threshold Y even when the repeated calibration operation is performed for the given number of times R, it is possible to close the calibration operation.

The embodiments of the disclosure have been described, and the content of the embodiments does not limit embodiments. The components described above contain ones easily assumed by those skilled in the art and ones substantially the same, that is, ones in the range of equivalence. Furthermore, the above-described components can be combined as appropriate. Furthermore, various omissions, replacements and changes can be made on the components within the scope of the embodiments described above.

According to the disclosure, it is possible to provide a gaze detection apparatus, a gaze detection method, and a gaze detection program that make it possible to inhibit accuracy of detection from lowering.