ELECTRONIC APPARATUS

An electronic apparatus according to the present invention acquires right line-of-sight information on a line-of-sight of a right eye of a user, and a left line-of-sight information on a line-of-sight of a left eye of the user, determines a first distance, which is a distance from the user to an intersection of the line-of-sight of the right eye and the line-of-sight of the left eye, and a second distance, which is a distance from the user to an object existing between the user and the intersection, on a basis of the right line-of-sight information and the left line-of-sight information; and controls setting of predetermined processing based on the right line-of-sight information and the left line-of-sight information, on a basis of a difference between the first distance and the second distance.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an electronic apparatus that can acquire line-of-sight information of a user.

Description of the Related Art

In recent years, the automation and intelligence of head mounted type display devices having a line-of-sight detection function, such as a spectacle type device utilizing mixed reality (MR) or augmented reality (AR), are progressing.

Japanese Patent Application Publication No. 2012-196279 discloses a technique to determine whether a user (viewer) is intentionally viewing a certain portion (whether the viewer is gazing at a certain portion) based on a time-dependent change of a pupil diameter of the user. Further, Japanese Patent Application Publication No. 2012-113450 discloses a technique to determine an alertness of a user (driver) based on a convergence angle of the user.

However, even if the techniques disclosed in Japanese Patent Application Publication No. 2012-196279 and Japanese Patent Application Publication No. 2012-113450 are used, in some cases it is difficult to determine accurately whether or not the user is gazing at the object, and to appropriately perform predetermined processing based on the line-of-sight information. For example, in the case of the technique disclosed in Japanese Patent Application Publication No. 2012-196279, whether or not the pupil is constricted is determined, but the pupil diameter changes depending on the quantity of light that enters the eye of the user, hence it is difficult to determine accurately whether or not the user is gazing. In the case of Japanese Patent Application Publication No. 2012-113450, the convergence angle of the user is used, but it is difficult to determine accurately whether or not the user is gazing by the convergence angle alone.

SUMMARY OF THE INVENTION

The present invention provides a technique to perform predetermined processing based on the line-of-sight information appropriately.

The present invention in its first aspect provides an electronic apparatus including at least one memory and at least one processor which function as: an acquisition unit configured to acquire right line-of-sight information on a line-of-sight of a right eye of a user, and left line-of-sight information on a line-of-sight of a left eye of the user; a determination unit configured to determine a first distance, which is a distance from the user to an intersection of the line-of-sight of the right eye and the line-of-sight of the left eye, and a second distance, which is a distance from the user to an object existing between the user and the intersection, on a basis of the right line-of-sight information and the left line-of-sight information; and a control unit configured to control setting of predetermined processing based on the right line-of-sight information and the left line-of-sight information, on a basis of a difference between the first distance and the second distance.

The present invention in its second aspect provides an electronic apparatus including at least one memory and at least one processor which function as: a first acquisition unit configured to acquire first information on a position in a depth direction of a field-of-view of a user, the first information being based on first line-of-sight information on a line-of-sight of a right eye of the user and second line-of-sight information on a line-of-sight of a left eye of the user; a second acquisition unit configured to acquire second information on a position in the depth direction of a target which is set based on at least one of the first line-of-sight information and the second line-of-sight information; and a control unit configured to control predetermined processing based on at least one of the first line-of-sight information and the second line-of-sight information, on a basis of the first information and the second information.

The present invention in its third aspect provides a control method of an electronic apparatus, including: acquiring right line-of-sight information on a line-of-sight of a right eye of a user, and left line-of-sight information on a line-of-sight of a left eye of the user; determining a first distance, which is a distance from the user to an intersection of the line-of-sight of the right eye and the line-of-sight of the left eye, and a second distance, which is a distance from the user to an object existing between the user and the intersection, on a basis of the right line-of-sight information and the left line-of-sight information; and controlling setting of predetermined processing based on the right line-of-sight information and the left line-of-sight information, on a basis of a difference between the first distance and the second distance.

DESCRIPTION OF THE EMBODIMENTS

Embodiment 1 of the present invention will be described. In Embodiment 1, an example of applying the present invention to an optical see-through type display device will be described, but the present invention is also applicable to a video see-through type display device, for example. The optical see-through type display device displays a graphic (e.g. virtual object) on a display surface (display surface which the light from a real space (outer world) transmits through), for example. In this case, the user can view the graphic displayed on the display surface. Further, the user can also view the real space (outer world) directly on the display surface. The video see-through type display device combines the image capturing the real space (virtual space) with the graphic when necessary, and displays the composite image on the display surface (display surface which light from the real space does not transmit through). In this case, the user can not view the real space directly, but can indirectly view the real space in the displayed image.

In Embodiment 1, an example of applying the present invention to a head mounted type display device will be described, but the present invention is also applicable to a handheld type display device and a stationary type display device, for example. The head mounted type display device is smart glasses (augmented reality (AR) glasses) or a head mounted display (HMD), for example. The handheld display device is a smartphone or a tablet terminal, for example. The display device which the user holds by hand and attaches (applied) to the head is a type of handheld display device, and also a type of head mounted type display device. The smartphone mounted to a head mounted adaptor (e.g. virtual reality (VR) goggles) is a type of head mounted type display device.

The present invention is also applicable to electronic apparatuses other than the display devices. The present invention is applicable to various electronic apparatuses that can acquire line-of-sight information of the user. For example, the present invention is applicable to a control device (e.g. controller or personal computer (PC)) connected to a display device. Example1is an example of applying the present invention to AR, but the present invention is also applicable to other cross realities (XRs), such as mixed reality (MR) and VR.

Description of Configuration

FIGS.1A and1Bare external views of a display device100according to Embodiment 1.FIG.1Ais a front perspective view, andFIG.1Bis a rear perspective view. The display device100is optical see-through type AR glasses. The display device100can independently detect a line-of-sight of a right eye and a line-of-sight of a left eye of a user wearing the display device100on his/her head. Hereafter the user wearing the display device100on his/her head is simply referred to as a “user”.

The lenses10are optical members that face the eyes of the user. The user can view the outer world through the lenses10. Displays11display a virtual object (virtual image of a virtual object) for both eyes of the user (both left and right eyes) based on control (display control) from a CPU2, which will be described later. The user can view the displayed virtual object as if the virtual object exists in the outer world. Light source drive circuits12drive the light sources13aand13b. Each of the light sources13aand13bis a light source to illuminate an eye of the user, and is an infrared light-emitting diode, for example, which emits an infrared light invisible to the user. A part of the lights emitted from the light sources13aand13band reflected by the eyes of the user converges on eye image pickup elements17respectively by light-receiving lenses16. These members are disposed for the right eye and the left eye respectively. For example, the eye image pickup elements17are the right image pickup element that images the right eye, and the left image pickup element that images the left eye.

FIG.2is a block diagram depicting an electric configuration of the display device100. The CPU2is a central processing unit of a microcomputer embedded in the display device100, and controls the display device100in general. The display11, the light source drive circuit12, a line-of-sight detection circuit15and a memory unit3are connected to the CPU2.

The memory unit3has a storage function to store video signals from the eye image pickup element17, and a storage function to store later mentioned line-of-sight correction parameters (parameters to correct an individual difference of the line-of-sight).

The line-of-sight detection circuit15A/D-converts the outputs from the eye image pickup elements17(eye images capturing the eye) in a state where an optical image of the eye is formed on the eye image pickup element17, and sends the result to the CPU2. The CPU2extracts the feature points required for the line-of-sight detection from the eye image in accordance with the later mentioned predetermined algorithm, and detects the line-of-sight of the user based on the positions of the feature points. For example, the CPU2acquires right line-of-sight information related to the line-of-sight of the right eye based on the right eye image acquired by the right image pickup element, and acquires left line-of-sight information related to the line-of-sight of the left eye based on the left eye image acquired by the left image pickup element.

Description on Line-of-Sight Detection Method

The line-of-sight detection method will be described with reference toFIGS.3,4A,4B and5. The line-of-sight of the right eye and the line-of-sight of the left eye are both detected by the following line-of-sight detection method.FIG.3is a diagram for describing the principle of the line-of-sight detection method, and is a schematic diagram of an optical system to detect a line-of-sight. As illustrated inFIG.3, the light sources13aand13bare disposed at approximately symmetric positions with respect to the optical axis of a light-receiving lens16, and illuminate an eyeball14of the user. A part of the lights emitted from the light sources13aand13band reflected by the eyeball14is collected to the eye image pickup element17by the light-receiving lens16.FIG.4Ais a schematic diagram of an eye image captured by the eye image pickup element17(optical image of the eye projected to the eye image pickup element17), andFIG.4Bis a diagram indicating the output intensity of the eye image pickup element17.FIG.5is a flow chart of the line-of-sight detection processing.

When the line-of-sight detection processing inFIG.5starts, in step S101, the CPU2controls the light sources13aand13busing the light source drive circuit12, so that the infrared light is emitted toward the eyeball14of the user. The optical image of the eye of the user illuminated by the infrared light forms an image on the eye image pickup element17via the light-receiving lens16, and is photoelectrically-converted by the eye image pickup element17. Thereby processable electric signals of the eye image can be acquired.

In step S102, the CPU2acquires an eye image (image data, image signals) from the eye image pickup element17via the line-of-sight detection circuit15.

In step S103, the CPU2detects the coordinates of points corresponding to the corneal reflex images Pd and Pe of the light sources13aand13band the pupil center c, from the eye image acquired in step S102.

The infrared lights emitted from the light sources13aand13billuminate a cornea142of the eyeball14of the user. Here the corneal reflex images Pd and Pe formed by a part of the infrared lights reflected on the surface of the cornea142are collected by the light-receiving lens16, form images on the eye image pickup element17, and become cornea reflex images Pd′ and Pe′ in the eye image. In the same manner, the lights from the edges a and b of the pupil141also form images on the eye image pickup element17, and become pupil edge images a′ and b′ in the eye image.

FIG.4Bindicates the brightness information (brightness distribution) of a region α in the eye image inFIG.4A. InFIG.4B, the brightness distribution in the X axis direction is indicated, where the X axis direction is the horizontal direction of the eye image, and the Y axis direction is the vertical direction thereof. In Embodiment 1, it is assumed that the coordinates of the corneal reflex images Pd′ and Pe′ in the X axis direction (horizontal direction) are Xd and Xe, and the coordinates of the pupil edge images a′ and b′ in the X axis direction are Xa and Xb. As indicated inFIG.4B, at the coordinates Xd and Xe of the corneal reflex images Pd′ and Pe′, an extremely high level of brightness is acquired. In the region from the coordinate Xa to the coordinate Xb, which corresponds to a region of the pupil141(region of the pupil image acquired when the light from the pupil141forms an image on the eye image pickup element17), an extremely low level of brightness is acquired, excluding the coordinates Xd and Xe. Then in a region of an iris143outside of the pupil141(region of an iris image outside the pupil image acquired when the light from the iris143forms an image), an intermediate level of brightness, between the above mentioned two types of brightness, is acquired. For example, in a region of which X coordinate (coordinate in the X axis direction) is larger than the coordinate Xa, and a region of which X coordinate is smaller than the coordinate Xb, the intermediate level of brightness, between the above mentioned two types of brightness, is acquired.

From the brightness distribution indicated inFIG.4B, the X coordinates Xd and Xe of the corneal reflex images Pd′ and Pe′ and the X coordinates Xa and Xb of the pupil edge images a′ and b′ can be acquired. For example, coordinates at which the brightness is extremely high can be acquired as the coordinates of the corneal reflex images Pd′ and Pe′, and coordinates at which the brightness is extremely low can be acquired as the coordinates of the pupil edge images a′ and b′. If a rotation angle θx of an optical axis of the eyeball14, with respect to an optical axis of the light-receiving lens16, is small, the coordinate Xc of the pupil center image c′ (center of the pupil image), acquired when the light from the pupil center c forms an image on the eye image pickup element17, can be given by Xc≈(Xa+Xb)/2. In other words, the coordinate Xc of the pupil center image c′ can be calculated from the X coordinates Xa and Xb of the pupil edge images a′ and b′. In this way, the coordinates of the corneal reflex images Pd′ and Pe′ and the coordinate of the pupil center image c′ can be estimated.

In step S104, the CPU2calculates an image forming magnification of the eye image. The image forming magnification β is a magnification determined by the position of the eyeball14with respect to the light-receiving lens16, and can be calculated using the function of the distance (Xd−Xe) of the corneal reflex images Pd′ and Pe′.

In step S105, the CPU2calculates a rotation angle of the optical axis of the eyeball14with respect to the optical axis of the light-receiving lens16. The X coordinate of the mid-point between the corneal reflex image Pd and the corneal reflex image Pe approximately matches with the X coordinate of a center of curvature O of the cornea142. Hence if Oc is a standard distance from the center of curvature O of the cornea142to the center c of the pupil141, the rotation angle θx of the eyeball14on the Z-X plane (plane vertical to the Y axis) can be calculated using the following Formula 1. The rotation angle θy of the eyeball14on the Z-Y plane (plane vertical to the X axis) also can be calculated using a method similar to the method for calculating the rotation angle θx.

In step S106, the CPU2estimates a viewpoint of the user on the lens10using the rotation angles θx and θy calculated in step S105. If the coordinates of the viewpoint (Hx, Hy) are coordinates corresponding to the pupil center c, the coordinates of the viewpoint (Hx, Hy) can be calculated using the following Formulas 2 and 3. The viewpoint may be regarded as a position at which the line-of-sight is directed, or as a position where the user is looking, or as a line-of-sight position.

A parameter m in Formulas 2 and 3 is a constant that is determined by the configuration of an optical system to perform the line-of-sight detection processing, and is a conversion coefficient to convert the rotation angles θx and θy into the coordinates corresponding to the pupil center c on the lens10. It is assumed that the parameter m has been determined and stored in the memory unit3in advance. The parameters Ax, Bx, Ay and By are line-of-sight correction parameters to correct the individual difference of the line-of-sight, and are acquired by calibrating the line-of-sight detection. The line-of-sight correction parameters Ax, Bx, Ay and By are stored in the memory unit3before starting the line-of-sight detection processing.

In step S107, the CPU2stores the coordinates of the viewpoint (Hx, Hy) in the memory unit3, and ends the line-of-sight detection processing.

The line-of-sight detection method is not limited to the above mentioned method, but may be any method that can acquire the line-of-sight information (information on the line-of-sight) from the eye image. As the final line-of-sight information, information indicating the line-of-sight direction (direction of line-of-sight), instead of information indicating the viewpoint, may be acquired. For example, instead of acquiring the coordinates of the viewpoint (Hx, Hy), the processing up to acquiring the rotation angle (Ax×θx+Bx or Ay×θy+By) may be performed.

FIG.6is a diagram indicating a field-of-view of the user (range where the user can view via the lens10), and indicating a state where the display11is operating. As indicated inFIG.6, the display11displays a frame or the like at a current viewpoint A (estimated position). The CPU2may control the display11, so that the information on a real object which the user is consciously viewing is displayed as a user interface (UI) in accordance with the result of the line-of-sight detection (line-of-sight information). The real object is an object that actually exists in the outer world (real space). Viewing consciously may be regarded as “gazing”.

Method for Determining Gaze

A viewpoint can be estimated by performing the line-of-sight detection processing. However, it cannot be estimated whether the user is consciously (intentionally) viewing the object. The user may or may not be viewing an object consciously. For example, a user may be overlooking the entire visual field, including the object. Even in such a way of viewing, a viewpoint is estimated by the line-of-sight detection processing. Since the viewpoint in the case of overlooking is not a viewpoint intended by the user, processing unexpected by the user may be performed as predetermined processing based on the viewpoint (predetermined processing based on the right line-of-sight information and the left line-of-sight information).

The predetermined processing based on the viewpoint is processing to display an item at the viewpoint, for example. The item is a frame (pointer) that is displayed at the viewpoint A inFIG.6, for example. Using this pointer, the user may select a button displayed on the display11, or select a real object, for example. In the case where a real object is selected, information on the selected real object is displayed on the display11, for example.

In the case of overlooking, however, display of the pointer is not intended by the user, and interferes with visibility of the user. In some cases, a button or a real object may be selected in error, the opposite of the intension of the user.

The predetermined processing based on the viewpoint may be processing to create a heat map of the viewpoint (processing to collect information (e.g. viewpoint, time, age) to create a heat map). By creating the heat map, an advertisement can be displayed at an appropriate position (position where advertisement effect is high, such as position the user frequently looks at). The heat map may be created associating the viewpoint with time. Then an advertisement can be displayed at an appropriate position at each timing. The heat map may be created by collecting information on a plurality of users. In this case, the heat map may be created associating the viewpoint with age. Then an advertisement appropriate for each age group can be displayed at an appropriate position.

However, if the viewpoints in the case of overlooking are collected and used for creating a heat map, a desired heat map (heat map indicating distribution of the positions the user intentionally looked at) cannot be created, and an advertisement cannot be displayed at an appropriate position, for example.

Therefore in Embodiment 1, the CPU2determines a visual distance and an object distance based on right line-of-sight information (viewpoint of the right eye) and left line-of-sight information (viewpoint of the left eye). The visual distance is a distance from the user to an intersection between the line-of-sight of the right eye and the line-of-sight of the left eye (viewpoint with both eyes, a position viewed with both eyes). The object distance is a distance from the user to an object (a real object or a virtual object) which exists between the user and a viewpoint with both eyes. When the difference of the visual distance and the object distance is smaller than a threshold TH, the CPU2determines that the user is consciously viewing the object (the user is gazing at the object) and the degree of gaze is high. When the difference of the visual distance and the object distance is larger than the threshold TH, on the other hand, the CPU2determines that the user is overlooking the entire visual field including the object, and the degree of gaze is low. When the difference between the visual distance and the object distance is the same as the threshold TH, the CPU2may determine that either the degree of gaze is high or the degree of gaze is low.

Then depending on the determination result on whether the degree of gaze is high or low, the CPU2controls the setting of the predetermined processing based on the right line-of-sight information and the left line-of-sight information. For example, the CPU2sets the predetermined processing to enable if it is determined that the degree of gaze is high, and to disable if it is determined that the degree of gaze is low. Thereby the viewpoint in the case of overlooking is not used for the predetermined processing, and the above mentioned problem can be solved. For example, display of unnecessary items can be prevented, and a drop in reliability of the heat map can be prevented.

The CPU2may control the setting of the predetermined processing based on the right line-of-sight information and the left line-of-sight information, depending on the difference between the visual difference and the object distance, without determining whether the degree of gaze is high or low.

Method for Determining Visual Distance

The convergence angle formed by the line-of-sight of the right eye and the line-of-sight of the left eye is smaller as the visual distance is longer; and the convergence angle is larger as the visual distance is shorter. For example, as indicated inFIG.7, the convergence angle θ is smaller as the viewpoint P with both eyes is more distant from the user, and the convergence angle θ is larger as the viewpoint P is closer to the user. Therefore in Embodiment 1, the CPU2determines the convergence angle based on the right line-of-sight information and the left line-of-sight information, and determines the visual distance in accordance with the convergence angle.

Method for Changing Setting of Line-of-Sight Processing

A method for changing the setting of the line-of-sight processing (predetermined processing based on the right line-of-sight information and the left line-of-sight information) will be described with reference toFIGS.8to10.FIG.8is a flow chart of the setting change processing (processing to change the setting of the line-of-sight processing).FIG.9is a diagram depicting a positional relationship of a user, a viewpoint with both eyes, and a position of an object.FIG.10is a graph indicating a relationship between a degree of divergence (difference of the visual distance and the object distance) and a degree of gaze. For example, when the display device100starts up, the setting change processing inFIG.8starts. The CPU2repeatedly performs the setting change processing inFIG.8.

When the setting change processing inFIG.8starts, the CPU2performs the line-of-sight detection processing inFIG.5in step S201, and acquires the right line-of-sight information (viewpoint of right eye) and the left line-of-sight information (viewpoint of left eye).

In step S202, the CPU2determines (calculates) the convergence angle θ based on the right line-of-sight information and the left line-of-sight information acquired in step S201. The convergence angle θ is an angle formed by the line-of-sight of the right eye and the line-of-sight of the left eye, and can be calculated from the rotation angle of the right eye and the rotation angle of the left eye, for example. Then the CPU2determines (calculates) the visual distance L1from the convergence angle θ and the distance between both eyes, using a trigonometric function. As indicated inFIG.9, the visual distance L1is a distance from the user to the viewpoint P with both eyes.

In step S203, the CPU2determines an object (real object or virtual object) that exists between the user and the viewpoint P. For example, as indicated inFIG.9, the CPU2determines an object that exists in a direction from the center of the display device100to the viewpoint P (an object that exists on the center line C). Then the CPU2determines the distance from the user to the determined object as an object distance L2. The method for determining the object distance L2is not especially limited. Here a case of determining the distance to the real object as the object distance L2will be considered. In this case, for example, a camera (imaging apparatus) is disposed in the display device100in advance, in order to image the outer world. Then the object distance L2is determined using the depth information acquired by the camera. The object distance L2may be determined using the light detection and ranging (LiDAR) method. Now a case of determining the distance to a virtual object as the object distance L2will be considered. In this case, the object distance L2is determined using a three-dimensional position that is set as a position of the virtual object.

In step S204, the CPU2calculates the difference of the visual distance L1determined in step S202and the object distance L2determined in step S203(that is, the degree of divergence L3=L1−L2), and determines whether the degree of divergence L3is not more than a threshold TH. The CPU2advances the processing to step S205if it is determined that the degree of divergence L3is not more than the threshold TH, or to step S207if it is determined that the degree of divergence L3is more than the threshold TH.

In step S205, the CPU2determines that the degree of gaze is high. In step S206, the CPU2sets the line-of-sight processing to enable (line-of-sight processing is executed).

In step S207, the CPU2determines that the degree of gaze is low. In step S208, the CPU2sets the line-of-sight processing to disable (line-of-sight processing is stopped).

The threshold TH to be compared with the degree of divergence L3may or may not be a predetermined fixed value. As the visual distance L1increases, the convergence angle θ decreases, and the calculation error of the visual distance L1increases. As the object distance L2increases, the calculation error of the object distance L2increases. In the case where the user is gazing at a nearby object, a small degree of divergence that is not more than 1 m is acquired, but in the case where the user is gazing at a distant object, a large degree of divergence that is about several meters is acquired. Therefore the CPU2may use the threshold TH, which is larger as the object distance L2is longer. For example, as indicated inFIG.10, the CPU2may use a threshold TH1if the object distance L2is long, and use a threshold TH2, which is smaller than the threshold TH1, if the object distance L2is short.

The above is the example of comparing the degree of divergence L3with the threshold TH, but the CPU2may determine a degree of gaze based on the degree of divergence L3, and compare the degree of gaze with a threshold. In this case, the CPU2determines that the degree of gaze is high if the degree of gaze is higher than the threshold, and determines that the degree of gaze is low if the degree of gaze is lower than the threshold. The method for determining the degree of gaze is not especially limited, but the correspondence between the degree of divergence L3and the degree of gaze may be determined in advance, so that the degree of gaze is determined from the degree of divergence L3in accordance with this correspondence. If the degree of divergence L3is constant, the CPU2may determine a value that is higher as the object distance is longer, as the degree of gaze. For example, as indicated inFIG.10, the correspondence in the case where the object distance L2is long and the correspondence in the case where the object distance L2is short may be determined in advance, as the correspondence of the degree of divergence L3and the degree of gaze. In this case, processing similar to the case of the above mentioned thresholds TH1and TH2can be implemented using only one threshold TH3that is compared with the degree of gaze.

According to the setting change processing inFIG.8, when the state where the user is gazing at the object changes to a state where the user is directing his/her line-of-sight to the area beyond the object, setting of the line-of-sight processing is changed. It is also possible that the display device100may be mounted on a pseudo-person (e.g. a head model having pseudo-eyeballs made of glass or the like), and operation similar to the above mentioned operation may be reproduced. In this case, the direction of the line-of-sight of the pseudo-eyeballs is adjusted by adjusting the angle of the pseudo-eyeballs.

In the example described above, one object exists between the user and the viewpoint P, but a plurality of objects may exist.FIG.11indicates this state. InFIG.11, three objects O1, O2and O3exist in the direction from the center of the display device100to the viewpoint P. In this case, the CPU2may determine a plurality of object distances which correspond to the plurality of objects respectively. Then the CPU2may control the setting of the line-of-sight processing based on a plurality of differences between the visual distance and the plurality of object distances.

For example, in step S203, the CPU2determines an object distance L21up to the object O1, an object distance L22up to the object O2, and an object distance L23up to the object O3. In step S204, the CPU2calculates the degrees of divergence L31, L32and L33. The degree of divergence L31is a difference between the visual distance L1determined in step S202and the object distance L21determined in step S203, that is, the difference (L1−L21). The degree of divergence L32is a difference between the visual distance L1and the object distance L22, that is, the difference (L1−L22). The degree of divergence L33is a difference between the visual distance L1and the object distance L23, that is, the difference (L1−L23). Then the CPU2determines whether at least one of the degrees of divergence L31, L32and L33is not more than a threshold TH. The CPU2advances the processing to step S205if it is determined that at least one of the degrees of divergence L31, L32and L33is not more than the threshold TH, or advances the processing to step S207if it is determined that all of the degrees of divergence L31, L32and L33are more than the threshold TH.

Conclusion

As described above, according to Embodiment1, the setting of the line-of-sight processing (predetermined processing based on the line-of-sight information) is controlled based on the difference between the visual distance and the object distance. Thereby the line-of-sight processing can be performed appropriately. For example, it can be accurately determined whether or not a user of an optical see-through type display device is gazing at an object, regardless the environment of the surroundings (ambient luminance, quantity of light that enters the eye of the user), and the setting of the line-of-sight processing can be appropriately controlled.

Embodiment 2 of the present invention will be described. In the following, description on aspects the same as Embodiment 1 (e.g. configuration and processing the same as Embodiment 1) will be omitted, and aspects different from Embodiment 1 will be described. In Embodiment 1, the CPU2sets the line-of-sight processing to enable if it is determined that the degree of gaze is high, and sets the line-of-sight processing to disable if it is determined that the degree of gaze is low. In Embodiment 2, on the other hand, the CPU2sets the line-of-sight processing to enable if it is determined that the degree of gaze is low, and sets the line-of-sight processing to disable if it is determined that the degree of gaze is high. The line-of-sight processing is processing to perform display of encouraging gazing, for example.

The user can go out in a state of wearing a head mounted type display device (e.g. optical see-through type AR glasses). User friendliness improves by the display of encouraging gazing at a specific object (object to be viewed). For example, safety of the user is improved by the display of encouraging gazing at an obstacle (e.g. bicycle left on a sidewalk) or gazing at a traffic light.

However, if the user is consciously viewing (gazing at) a specific object, the display of encouraging gazing is a display unintended by the user, and interrupts with the view of the user.

Therefore in Embodiment 2, the CPU2performs the display of encouraging gazing if it is determined that the degree of gaze is low, and does not perform the display of encouraging gazing if it is determined that the degree of gaze at a specific object is high. Thereby unnecessary display can be prevented. The conditions to perform the display of encouraging gazing may include a condition in which the viewpoint with both eyes is deviated from a specific object.

The method for detecting a specific object is not especially limited. For example, a camera (imaging apparatus) is mounted on the display device100in advance, so as to capture images of the outer world. The CPU2detects the specific object in the image captured by the camera using at least one of the various known techniques. To detect the specific object, an arithmetic unit (learned model) can be used, where a captured image is inputted and the detection result is outputted. The CPU2may function as the arithmetic unit, or the display device100may include a graphics processing unit (GPU) that functions as the arithmetic unit.

As described above, according to Embodiment 2, the line-of-sight processing is set to enable if it is determined that the degree of gaze is low, and the line-of-sight processing is set to disable if it is determined that the degree of gaze is high. The line-of-sight processing can also be performed appropriately by this method. For example, the user can be encouraged to gaze at a specific object, so as not to miss viewing the object.

The above embodiments (including modifications) are merely examples, and the configurations acquired by appropriately modifying or changing the configurations of the above embodiments within the scope of the spirit of the present invention are also included in the present invention. The configurations acquired by appropriately combining the configurations of the above embodiments are also included in the present invention.

According to the present invention, predetermined processing based on the line-of-sight information can be performed appropriately.

Other Embodiments

This application claims the benefit of Japanese Patent Application No. 2022-126021, filed on Aug. 8, 2022, which is hereby incorporated by reference herein in its entirety.