Patent Description:
Camera-based eye tracking technologies may be utilized in many fields, for example, a viewpoint tracking-based super multi-view autostereoscopic three-dimensional (3D) display. A camera-based eye tracker may operate normally in a high illumination environment (for example, <NUM> Lux), but may not operate normally in a low illumination environment due to a low quality of a camera image. In a dark place as well as a bright place, a user may frequently watch a television (TV) or use a mobile device. Also, driving at night-time needs to be considered in a technology of a next-generation 3D head-up display (HUD) for vehicles.

For example, when user's eyes are tracked using a color camera in a low illumination environment, for example, a dark place or nighttime, a success rate and an accuracy of eye tracking may decrease due to a reduction in an image quality, and accordingly an infrared camera may be used. However, when the infrared camera is used, a success rate and an accuracy of eye tracking of a person wearing glasses may be reduced due to a reflection on lenses of the glasses. <NPL> discloses a methodology for tracking and detecting eyes robustly in indoor environments in real-time. In this methodology of Haro et al, infrared lighting is used to capture the physiological properties of eyes, Kalman trackers are used to model eye/head dynamics and a probabilistic based appearance model is used to represent eye appearance. By combining these three separate modalities, with specific enhancements within each modality, the approach of Haro et al allows eyes to be treated as robust features that can be used for other higher-level processing. <CIT> describes a system and method for actively illuminating and capturing images of an object, such as a driver of a vehicle to reduce glare. The system includes a video imaging camera orientated to generate images of the subject object (e.g., eye(s)) in successive video frames. The system also includes first and second light sources operable to illuminate the object one at a time. The system further includes a processor for processing the image by performing a functional operation to reduce glare in the image. The system may remove glare caused by the illuminators and by external energy sources. <CIT> relates to a device and method for automatic detection of a camera lens belonging to a camera registering on the sly. In a first step of this method according to <CIT>, a lighting source is switched on and a region to be checked is irradiated. Then, images in both the visible and the infrared wavelength range are obtained, whereby an optical filter switcher is used in order to switch between the two wavelengths ranges. The images in the visible and in the infrared wavelength range are subsequently output to memory storage. Further, an image processor reads the images in both the visible and the infrared wavelength range and a brightness number matrix of the infrared light image and a brightness number matrix of the visible light image are subtracted from each other. The difference data obtained by this subtraction is subsequently compared with a threshold value. If the difference data results in pixels being higher than said threshold value, a region is identified as most likely corresponding to a camera lens registering on the sly.

It is the object of the present invention to provide a method and an apparatus that allows tracking eyes also in low illumination environments with great accuracy.

According to an aspect of an example embodiment, there is provided a method of detecting a reflection according to claim <NUM>.

The acquiring of the input image includes activating the infrared light source during an on interval, and the acquiring of the reference image includes deactivating the infrared light source during an off interval.

The acquiring of the input image further includes generating the input image, based on a first plurality of rays that is received from the object, during the on interval, and the acquiring of the reference image further includes generating the reference image, based on a second plurality of rays that is received from the object, during the off interval.

The generating of the input image may include collecting first intensities of the first plurality of rays received from the object, during a first portion of first frames in the on interval, and determining the first intensities as first pixel values of first pixels of the input image, to generate the input image. The generating of the reference image may include collecting second intensities of the second plurality of rays received from the object, during a second portion of second frames in the off interval, and determining second intensities as second pixel values of second pixels of the reference image, to generate the reference image.

The extracting of the reflection region includes generating a difference map by subtracting first pixel values of the reference image from second pixel values of the input image respectively corresponding to the first pixel values, the difference map indicating a difference between the input image and the reference image, and extracting the reflection region from the input image, based on the difference map.

The extracting of the reflection region further includes determining elements of the difference map, the elements having difference values exceeding a threshold, and determining, as the reflection region, pixels of the input image that correspond to the elements.

The method may further include tracking a position of the object, and designating at least one of a plurality of infrared light sources included in an infrared ray array, the at least one of the plurality of infrared light sources corresponding to the position that is tracked. The acquiring of the input image may include activating the at least one of the plurality of infrared light sources that is designated, during an on interval, and the acquiring of the reference image may include deactivating the at least one of the plurality of infrared light sources that is designated, during an off interval.

The acquiring of the input image may include acquiring the input image, based on an infrared region of a first ray that is received from the object during an on interval in which the infrared light source is activated, and the acquiring of the reference image may include acquiring the reference image, based on a visible ray region of a second ray that is received from the object during an off interval in which the infrared light source is deactivated.

The method may further include dynamically adjusting a first length of an on interval in which the infrared light source is activated and a second length of an off interval in which the infrared light source is deactivated.

The method may further include removing the reflection region from the input image, and tracking a gaze of a user, based on the input image from which the reflection region is removed.

The infrared light source and an image acquirer are arranged so that a predetermined angle is formed by a first direction in which the infrared light source emits a first ray to the object and by a second direction in which the image acquirer receives a second ray from the object.

The method may further include gradually increasing a ray intensity of the infrared light source from a first start timing to a first intermediate timing in an on interval in which the infrared light source is activated, gradually decreasing the ray intensity from the first intermediate timing to a first end timing in the on interval, gradually decreasing the ray intensity from a second start timing to a second intermediate timing in an off interval in which the infrared light source is deactivated, and gradually increasing the ray intensity from the second intermediate timing to a second end timing in the off interval.

The method may further include periodically deactivating the infrared light source, in response to a detection of a transparent object that causes a light reflection between a user and the apparatus.

The acquiring of the input image may include increasing an intensity of the infrared light source from an off level to an on level, during an on interval; and the acquiring of the reference image may include decreasing the intensity of the infrared light source from the on level to the off level, during an off interval.

A non-transitory computer-readable storage medium according to claim <NUM> stores instructions that, when executed by a processor, cause the processor to perform the method.

According to another aspect of the invention, there is provided an apparatus according to claim <NUM>.

The apparatus may further include an infrared ray array including a plurality of infrared light sources, the infrared ray array being spaced apart from the image acquirer. The processor may be further configured to activate at least one of the plurality of infrared light sources during an on interval, and deactivate the at least one of the plurality of infrared light sources during an off interval.

The processor is further configured to generate a difference map by subtracting first pixel values of the reference image from second pixel values of the input image respectively corresponding to the first pixel values, the difference map indicating a difference between the input image and the reference image, and extract the reflection region from the input image, based on the difference map.

The image acquirer may be further configured to receive, from the object, a first ray during an on interval in which the infrared light source is activated and a second ray during an off interval in which the infrared light source is deactivated. The processor may be further configured to acquire the input image, based on an infrared region of the first ray, and acquire the reference image, based on a visible ray region of the second ray.

The apparatus may further include an infrared ray array spaced apart from the image acquirer so that a predetermined angle is formed by a first direction in which the infrared ray array emits a first ray to the object and a second direction in which the image acquirer receives a second ray from the object. The processor may be further configured to activate at least one of a plurality of infrared light sources included in the infrared ray array, so that the predetermined angle is maintained at a position of the object, in response to a movement of the object being detected.

According to an aspect of an example embodiment, there is provided a method of detecting a reflection, the method being performed by an apparatus for detecting the reflection, and the method including acquiring an input image of an object, based on an activation of an infrared light source, acquiring a reference image of the object, based on a deactivation of the infrared light source, and generating a difference map by subtracting first pixel values of the reference image from second pixel values of the input image respectively corresponding to the first pixel values. The method further includes determining elements of the difference map, the elements having difference values exceeding a threshold, and removing a reflection region from the input image, the reflection region corresponding to the elements.

The method may further include adjusting a first length of an on interval in which the infrared light source is activated to be shorter than a second length of an off interval in which the infrared light source is deactivated.

The above and/or other aspects will be more apparent by describing example embodiments with reference to the accompanying drawings, in which:.

Reference will now be made in detail to example embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout.

The terminology used herein is for the purpose of describing the example embodiments only and is not to be limiting of the examples. As used herein, the singular forms are intended to include the plural forms as well, unless the context clearly indicates otherwise. It may be further understood that the terms "include/comprise" and/or "have" when used in this specification, specify the presence of stated features, integers, steps, operations, elements, components and/or combinations thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

Unless otherwise defined, all terms including technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments belong. It will be further understood that terms, such as those defined in commonly-used dictionaries, may be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

When describing the example embodiments with reference to the accompanying drawings, like reference numerals refer to like constituent elements and a repeated description related thereto will be omitted. When it is determined detailed description related to a related known function or configuration they may make the purpose of the example embodiments unnecessarily ambiguous in describing the example embodiments, the detailed description will be omitted here.

<FIG> is a diagram illustrating an example in which a gaze tracker tracks a user's gaze according to an example embodiment.

Referring to <FIG>, a gaze tracker <NUM> may track a gaze of a user <NUM>. The gaze tracker <NUM> may detect a position of a pupil <NUM> of the user <NUM> and may track a movement of the pupil <NUM>. For example, the gaze tracker <NUM> may extract a face region including a face of the user <NUM>, and may extract an eye region from the face region. The gaze tracker <NUM> may detect the pupil <NUM> in the eye region. However, the example embodiment is not limited thereto.

The gaze tracker <NUM> may track the gaze of the user <NUM> using infrared rays. Using infrared rays, the gaze tracker <NUM> may accurately track the gaze of the user <NUM> even in a relatively low illumination environment. However, when a transparent object <NUM> that may cause a light reflection is located between the gaze tracker <NUM> and the user <NUM>, a reflection phenomenon <NUM> may appear on an image captured by the gaze tracker <NUM>. For example, the user <NUM> may wear the transparent object <NUM>, and a reflection phenomenon due to the transparent object <NUM> may hinder a detection of a gaze. The transparent object <NUM> may include, for example, glasses or sunglasses.

In example embodiments, a reflection may refer to a phenomenon in which a ray emitted from a light source is reflected from an arbitrary object (for example, a transparent object, such as glasses) and is incident on an image acquirer (for example, a camera) at an intensity that is close to or greater than a maximum intensity that may be sensed. The image acquirer may determine, as a saturated value, a value of a pixel corresponding to a reflection region in which the reflection occurs. The saturated value may be, for example, a value corresponding to a maximum intensity that may be sensed by the image acquirer.

<FIG> is an image captured by a gaze tracker before a reflection region is removed.

An object image <NUM> of <FIG> may be an image captured by a gaze tracker. The object image <NUM> may be an image including an object, and an object may include, for example, at least a part of a human body. The object image <NUM> may be an image acquired by capturing a face of a person.

As shown in <FIG>, a reflection region <NUM> may appear on a transparent object (for example, glasses) that may cause a light reflection. The reflection region <NUM> may have a higher intensity than those of neighboring regions, and may have, for example, a maximum intensity that may be sensed by a sensor. A pupil is brightly detected in comparison to an iris from a user's face, and accordingly an accuracy of a pupil detection may decrease when the reflection region <NUM> appears closer to an eye.

<FIG> and <FIG> are flowcharts illustrating a method of detecting a reflection according to an example embodiment.

Referring to <FIG>, in operation <NUM>, an apparatus for detecting a reflection (hereinafter, referred to as a reflection detection apparatus) acquires an input image based on an activation of an infrared light source. The infrared light source is a light source configured to emit infrared rays, and may include, for example, an infrared light-emitting diode (LED). The infrared light source may emit a bunch of infrared rays. The reflection detection apparatus may include an infrared ray array including a plurality of infrared light sources. The input image may be an image acquired by capturing an object, and may be used to track a gaze of a user that is an object. For example, the input image may be generated based on an infrared region and a visible ray region.

An activation of an infrared ray may refer to an operation of adjusting an intensity of a ray emitted from the infrared light source to an on level. An intensity corresponding to the on level may vary depending on a design. For example, the reflection detection apparatus may control the infrared light source to emit a ray with an intensity greater than or equal to a threshold intensity, to activate the infrared light source.

A ray intensity corresponding to the on level may be designed as an intensity that is sufficient to allow a processor to extract a feature point of an object form the input image. For example, an infrared ray emitted with an intensity corresponding to an on level may be protected to an object. In this example, an image acquirer may receive an infrared ray that is reflected from the object in response to a projection of the infrared ray to the object. Also, the image acquirer may generate an input image based on the received infrared ray, and the processor may extract a feature point of the object from the input image. In response to an increase in the ray intensity corresponding to the on level, the image acquirer may acquire a sharper input image, and the processor may more accurately extract a feature point of the object.

A feature point of the object may be a point indicating a feature of the object in an image. For example, when the object is a face of a person, a feature point of the face may be a point corresponding to an eye, a point corresponding to a nose, a point corresponding to a mouth, and a point corresponding to an ear. However, the feature point of the object is not limited thereto.

In operation <NUM>, the reflection detection apparatus acquires a reference image based on a deactivation of the infrared light source. The reference image may be an image acquired by capturing an object, and is used to detect a reflection. The reference image is generated mainly based on a visible ray region.

The deactivation of the infrared light source may refer to an operation of adjusting an intensity of a ray emitted from the infrared light source to an off level. An intensity corresponding to the off level may vary depending on a design. For example, the reflection detection apparatus may control the infrared light source to emit a ray with an intensity less than a threshold intensity, to deactivate the infrared light source. Also, the reflection detection apparatus may cut off a power supply to the infrared light source, to deactivate the infrared light source. When the power supply is cut off, a ray intensity of the infrared light source may be zero or may converge to zero.

A ray intensity corresponding to the off level may be designed as an intensity that is sufficient to allow the processor to extract a reflection region form the input image. For example, the deactivated infrared light source may interrupt emitting of an infrared ray. Also, an infrared ray emitted with an intensity corresponding to an off level may be projected to an object, and may be mostly absorbed to the object or attenuated. In response to a decrease in the ray intensity corresponding to the off level, the image acquirer may acquire a reference image that may not include a reflection region or that may include a relatively small reflection region.

In operation <NUM>, the reflection detection apparatus extracts a reflection region from the input image based on the input image and the reference image. The reflection detection apparatus calculates a difference between the input image and the reference image, and generates a difference map. A region with a great difference between the input image and the reference image may indicate a reflection region generated by an infrared ray with an intensity corresponding to an on level, because the input image is generated mainly based on an infrared region and the reference image is generated mainly based on a visible ray region.

The difference map indicates the difference between the input image and the reference image. For example, the difference map may include a plurality of elements. Each of the plurality of elements may have a value corresponding to a difference between a pixel of an input image corresponding to a corresponding element and a pixel of the reference image. A number of the plurality of elements may be equal to, for example, a number of pixels in the input image or a number of pixels in the reference image.

The reflection detection apparatus may periodically deactivate the infrared light source during a detection of a transparent object that may cause a light reflection between a user and the reflection detection apparatus. For example, while a user wears glasses, the reflection detection apparatus may repeat the activation and deactivation of the infrared light source, to enhance an accuracy of a pupil detection.

<FIG> illustrates an example of the method of <FIG>.

Referring to <FIG>, in operation <NUM>, the reflection detection apparatus may activate the infrared light source during an on interval. In example embodiments, the on interval may refer to an interval designated to activate the infrared light source, within a predetermined period. The on interval will be further described below with reference to <FIG>, <FIG> and <FIG>.

In operation <NUM>, the reflection detection apparatus may generate the input image based on a ray received from the object, during the on interval. For example, the reflection detection apparatus may collect intensities of a plurality of rays from the object during at least a portion of frames in the on interval. For example, when the on interval includes a plurality of frames, the reflection detection apparatus may collect intensities of a plurality of rays during a last frame in the on interval. The reflection detection apparatus may determine an intensity corresponding to each of the plurality of rays as a pixel value of each of pixels included in the input image, and may generate the input image. Also, the reflection detection apparatus may accumulate intensities collected during a portion of frames in the on interval, and may generate the input image.

For example, the reflection detection apparatus may acquire the input image based on an infrared region of a ray received from the object during the on interval. The ray received during the on interval may include rays corresponding to an infrared region and a visible ray region. Thus, each of the pixels in the input image may have a value obtained by adding a ray intensity of the infrared region and a ray intensity of the visible ray region.

In example embodiments, the infrared region may indicate a frequency band that is classified as an infrared ray among frequencies of electromagnetic waves. Also, the visible ray region may indicate a frequency band visible to human eyes in electromagnetic waves.

In operation <NUM>, the reflection detection apparatus may deactivate the infrared light source during the off interval. In example embodiments, the off interval may refer to an interval designated to deactivate the infrared light source, within a predetermined period. For example, the off interval may be an interval other than the on interval in the predetermined period. The off interval will be further described below with reference to <FIG>.

In operation <NUM>, the reflection detection apparatus may generate a reference image based on the ray received from the object, during the off interval. For example, the reflection detection apparatus may collect intensities of a plurality of rays from the object during at least a portion of frames in the off interval. For example, when the off interval includes a plurality of frames, the reflection detection apparatus may collect intensities of a plurality of rays during a last frame in the off interval. The reflection detection apparatus may determine an intensity corresponding to each of the plurality of rays as a pixel value of each of pixels included in the reference image, and may generate the reference image. Also, the reflection detection apparatus may accumulate intensities collected during a portion of frames in the off interval, and may generate the reference image.

The reflection detection apparatus may acquire the reference image based on a visible ray region of a ray received from the object during the off interval. For example, the ray received during the off interval may dominantly include rays corresponding to a visible ray region. Thus, each of the pixels in the reference image may have a value corresponding to a ray intensity of the visible ray region.

In operation <NUM>, the reflection detection apparatus generates a difference map indicating a difference between the input image and the reference image, by subtracting a pixel value of the reference image corresponding to a pixel value of the input image from the pixel value of the input image. The difference map may include the same number of elements as a number of pixels in the input image and a number of pixels in the reference image. Each of the elements of the difference map may have a value corresponding to a difference between a pixel value of the input image and a pixel value of the reference image.

In operation <NUM>, the reflection detection apparatus extracts a reflection region based on the difference map. The reflection detection apparatus determines elements that have values exceeding a threshold in the difference map. The reflection detection apparatus may determine pixels of the input image that correspond to elements to be the reflection region.

However, the example embodiment is not limited thereto, and the reflection detection apparatus may select an element having a value higher than a value of a neighboring element from the difference map. The reflection detection apparatus may determine a pixel of the input image that corresponds to the selected element to be the reflection region.

The reflection detection apparatus may remove the extracted reflection region from the input image. The reflection detection apparatus may track a gaze of a user based on the input image from which the reflection region is removed. Thus, the reflection detection apparatus may more accurately determine a position of a user's pupil, based on an input image from which a reflection is removed.

<FIG> is a diagram illustrating a structure in which an image acquirer and an infrared light source of a reflection detection apparatus are spaced apart according to an example embodiment.

Referring to <FIG>, a reflection detection apparatus <NUM> includes an image acquirer <NUM> and an infrared light source <NUM>. The image acquirer <NUM> and the infrared light source <NUM> are spaced apart from each other. The image acquirer <NUM> and the infrared light source <NUM> are arranged so that a predetermined angle <NUM> is formed by a direction of the image acquirer <NUM> and a direction of the infrared light source <NUM>.

The direction of the infrared light source <NUM> may indicate a direction in which an infrared ray <NUM> is emitted from the infrared light source <NUM> and propagates. For example, when a bunch of infrared rays is emitted from the infrared light source <NUM>, a propagation direction of the infrared ray <NUM> corresponding to a central ray among the infrared rays may be the direction of the infrared light source <NUM>. The direction of the image acquirer <NUM> may indicate a direction corresponding to a central ray among rays that may be received by the image acquirer <NUM>. The rays that may be received by the image acquirer <NUM> may be, for example, rays included in a field of view (FOV) of the image acquirer <NUM>.

The image acquirer <NUM> receives a ray <NUM> that corresponds to an infrared region and a visible ray region from an object <NUM>. For example, the object <NUM> may reflect an infrared ray projected by the infrared light source <NUM> or reflect a visible ray projected by an external device.

<FIG>, <FIG>, <FIG>, <FIG>, <FIG> and <FIG> are images acquired based on a degree to which an image acquirer and an infrared light source are spaced apart according to an example embodiment.

<FIG> illustrate input images acquired by the image acquirer based on an arrangement of the image acquirer and the infrared light source.

<FIG> illustrates an example in which a direction difference between the image acquirer and the infrared light source is zero. For example, when a direction of the image acquirer and a direction of the infrared light source are identical to each other, a reflection region <NUM> may appear as shown in <FIG>. In <FIG>, the reflection region <NUM> may appear around an eye.

<FIG> illustrates an input image acquired in a structure in which an angle of <NUM> degrees is horizontally formed by the direction of the image acquirer and the direction of the infrared light source. In the input image of <FIG>, a reflection region <NUM> may appear to cover at least a portion of an eye.

<FIG> illustrates an input image acquired in a structure in which an angle of <NUM> degrees is horizontally formed by the direction of the image acquirer and the direction of the infrared light source. In the input image of <FIG>, a reflection region <NUM> may be reduced in size in comparison to <FIG> and <FIG>.

<FIG> illustrates an input image acquired in a structure in which an angle of <NUM> degrees is vertically formed by the direction of the image acquirer and the direction of the infrared light source. In <FIG>, a reflection region may not appear.

<FIG> illustrates an input image acquired in a structure in which an angle of <NUM> degrees is vertically formed by the direction of the image acquirer and the direction of the infrared light source. The input image of <FIG> may not include a reflection region, however, may include a shadow region that appears on an eye and a cheek. The shadow region may have a darker pixel value than a neighboring region.

<FIG> illustrates an input image acquired in a structure in which an angle of <NUM> degrees is vertically formed by the direction of the image acquirer and the direction of the infrared light source. The input image of <FIG> may not include a reflection region, however, may include a shadow region that is wider than the shadow region of <FIG>.

Thus, the image acquirer and the infrared light source are arranged so that an optimum angle is formed by the direction of the image acquirer and the direction of the infrared light source. A reflection detection apparatus that includes the image acquirer and the infrared light source is arranged to optimize the direction difference acquires an input image in which a reflection region and a shadow region are minimized. An angle of <NUM> degrees is formed by the direction of the image acquirer and the direction of the infrared light source.

<FIG> is a diagram illustrating an example in which a reflection detection apparatus is implemented in a vehicle according to an example embodiment.

For example, the reflection detection apparatus may be included in a vehicle <NUM>. The reflection detection apparatus may include an image acquirer <NUM> and an infrared light source <NUM>. The image acquirer <NUM> may be located in a dashboard that is in front of a driver. The infrared light source <NUM> may be located on a rearview mirror of the vehicle <NUM>.

During driving of the vehicle <NUM>, the reflection detection apparatus may acquire an input image from which a reflection is removed, based on an activation and a deactivation of the infrared light source <NUM>. For example, even in a low illumination environment (for example, nighttime), the reflection detection apparatus may accurately detect a gaze of a user based on an input image from which a reflection by an infrared ray is removed.

However, an arrangement of the image acquirer <NUM> and the infrared light source <NUM> is not limited to the above description. For example, the image acquirer <NUM> and the infrared light source <NUM> is arranged so that a predetermined angle is formed by a direction of the image acquirer <NUM> and a direction of the infrared light source <NUM>.

<FIG> is a diagram illustrating a spatial adjustment of an infrared light source based on a movement of an object according to an example embodiment.

Referring to <FIG>, a reflection detection apparatus may include an image acquirer <NUM> and an infrared ray array <NUM>. The infrared ray array <NUM> may include a plurality of infrared light sources. The plurality of infrared light sources may be horizontally arranged. However, the example embodiment is not limited thereto, and the plurality of infrared light sources may be arranged vertically or in a two-dimensional (2D) structure of "n" rows and "m" columns in which n and m may be integers greater than or equal to "<NUM>. " However, a structure of the infrared ray array <NUM> is not limited thereto, and the plurality of infrared light sources may be variously arranged.

At least a portion of the infrared light sources in the infrared ray array <NUM> may emit infrared rays. For convenience of description, in <FIG>, a first infrared light source <NUM> among the plurality of infrared light sources may emit an infrared ray, however, there is no limitation thereto. At least two infrared light sources may emit infrared rays. The image acquirer <NUM> may receive an infrared ray that is emitted from the first infrared light source <NUM> and that is reflected from an object <NUM>. The image acquirer <NUM> may generate an input image <NUM> including an object region <NUM> representing the object <NUM> based on the received infrared ray together with a visible ray. In this example, a predetermined angle may be formed by a direction of the first infrared light source <NUM> and a direction of the image acquirer <NUM>.

Also, the reflection detection apparatus may track a position of the object <NUM>. The reflection detection apparatus may detect a change in the position of the object <NUM>, based on the image acquirer <NUM>. For example, a movement to an object region <NUM> representing the object <NUM> may be sensed in the input image <NUM> acquired by the image acquirer <NUM>.

The reflection detection apparatus may designate an infrared light source corresponding to the tracked position from the infrared ray array <NUM> that includes the plurality of infrared light sources. For example, the reflection detection apparatus may designate a second infrared light source <NUM> that is mapped to the object region <NUM> in the input image <NUM>, from the infrared ray array <NUM>. In this example, the above-described angle may be formed by a direction of the second infrared light source <NUM> and the direction of the image acquirer <NUM>.

For example, the reflection detection apparatus may assign the infrared light sources of the infrared ray array <NUM> for each region of the input image <NUM>. Because the infrared light sources are horizontally arranged as shown in <FIG>, the reflection detection apparatus may assign the infrared light sources for each of regions into which the input image <NUM> is horizontally divided. Because the object <NUM> horizontally moves to the right, the reflection detection apparatus may designate the second infrared light source <NUM> located further leftward in comparison to the first infrared light source <NUM>. In response to a movement of the object <NUM> being detected, the reflection detection apparatus may select an infrared light source that is activated during an on interval from the infrared ray array <NUM> so that a predetermined angle formed by the direction of the image acquirer <NUM> and a direction of an infrared ray may be maintained at the position of the object <NUM>.

The reflection detection apparatus may activate the designated infrared light source during an on interval, and may deactivate the designated infrared light source during an off interval. The reflection detection apparatus may periodically repeat an activation and deactivation of the second infrared light source <NUM> until the position of the object <NUM> is changed again.

For example, the reflection detection apparatus may quickly designate an infrared light source corresponding to the position of the object <NUM> immediately when the movement of the object <NUM> is detected. Thus, the reflection detection apparatus may select an appropriate infrared light source instead of individually turn on or off the infrared light sources of the infrared ray array <NUM>. For example, when a face position is changed due to a change in a user's posture during driving of a vehicle including the reflection detection apparatus, the reflection detection apparatus may quickly remove a reflection by selecting an appropriate infrared light source.

<FIG>, <FIG> and <FIG> are diagrams illustrating a temporal adjustment of an infrared light source according to example embodiments.

<FIG> illustrates a periodic repetition of an activation and a deactivation of an infrared light source.

A reflection detection apparatus may periodically repeat an activation and a deactivation of the infrared light source, to acquire an input image <NUM> and a reference image <NUM>. For example, the reflection detection apparatus may activate the infrared light source during an on interval <NUM>, and may deactivate the infrared light source during an off interval <NUM>.

For example, the reflection detection apparatus may increase a ray intensity <NUM> of the infrared light source from an off level to an on level during the on interval <NUM>. Also, the reflection detection apparatus may decrease the ray intensity <NUM> from the on level to the off level during the off interval <NUM>.

The reflection detection apparatus may generate the input image <NUM> based on a visible ray and an infrared ray received during the on interval <NUM>. Also, the reflection detection apparatus may generate the reference image <NUM> based on a visible ray received during the off interval <NUM>. The reflection detection apparatus calculates a difference map <NUM> by subtracting the reference image <NUM> from the input image <NUM>. The reflection detection apparatus determines elements of the difference map <NUM> that have values exceeding a threshold to be a reflection region <NUM>.

<FIG> illustrates an example of an activation and a deactivation of an infrared light source.

Referring to <FIG>, a length of an on interval <NUM> may be less than a length of an off interval <NUM>. For example, when a frame rate of an image acquirer is "<NUM>" frames per second (fps), a predetermine period <NUM> may correspond to ten frames. The off interval <NUM> may correspond to nine frames, and the on interval <NUM> may correspond to one frame. A reflection detection apparatus may reduce the length of the on interval <NUM>, to reduce a cumulative amount of an infrared ray intensity <NUM>. The reflection detection apparatus may adjust the on interval <NUM>, to minimize an influence of infrared rays on a human body.

Also, the reflection detection apparatus may dynamically adjust the length of the on interval <NUM> in which the infrared light source is activated, and the length of the off interval <NUM> in which the infrared light source is deactivated. In an example, in an environment (for example, a dark environment) in which an accuracy of a gaze detection decreases, the reflection detection apparatus may increase the length of the on interval <NUM>. In another example, in an environment (for example, a bright environment) with a relatively high accuracy of the gaze detection, the reflection detection apparatus may reduce the length of the on interval <NUM>. Thus, the reflection detection apparatus may enhance an accuracy of a pupil detection while minimizing an influence of infrared rays on a human body.

<FIG> illustrates a gradual activation and deactivation of an infrared light source.

A reflection detection apparatus may gradually increase a ray intensity <NUM> of the infrared light source from a start timing <NUM> to an intermediate timing <NUM> in an on interval <NUM>. The reflection detection apparatus may gradually decrease the ray intensity <NUM> from the intermediate timing <NUM> to an end timing <NUM> in the on interval <NUM>. The reflection detection apparatus may gradually decrease the ray intensity <NUM> from a start timing <NUM> to an intermediate timing <NUM> in an off interval <NUM>. The reflection detection apparatus may gradually increase the ray intensity <NUM> from the intermediate timing <NUM> to an end timing <NUM> in the off interval <NUM>. The end timing <NUM> in the off interval <NUM> may be identical to the start timing <NUM> of the on interval <NUM>. For example, the reflection detection apparatus may adjust the ray intensity <NUM> in a form of a sinusoidal wave, for example, a sine wave or a cosine wave. However, the example embodiment is not limited thereto, and the reflection detection apparatus may consecutively change the ray intensity <NUM>.

The reflection detection apparatus may generate an input image by accumulating intensities of rays that are reflected from an object and received during the on interval <NUM>. Also, the reflection detection apparatus may generate a reference image by accumulating intensities of rays that are reflected from an object and received during the off interval <NUM>.

<FIG> is an image from which a reflection region is removed according to an example embodiment.

A reflection detection apparatus may remove a reflection region acquired using the method of <FIG> from an input image <NUM>. For example, the reflection detection apparatus may perform a hole-filling algorithm (for example, inpainting) on a reflection region of the input image <NUM>. The reflection detection apparatus may remove the reflection region from the input image <NUM>, and may fill a hole region generated by removing the reflection region to be similar to a neighboring region. In an example, the reflection detection apparatus may fill values of pixels in the hole region with values of pixels in the neighboring region, using the hole-filling algorithm. In another example, the reflection detection apparatus may fill values of pixels in the hole region with values similar to values of pixels in the neighboring region, using the hole-filling algorithm. Thus, the reflection detection apparatus may process the input image <NUM> so that the hole region and the neighboring region may become blurred.

The reflection detection apparatus may extract a feature point form the input image <NUM> from which the reflection region is removed, and may determine an upper face region <NUM> based on the feature point. The reflection detection apparatus may determine an eye region from the upper face region <NUM> and may track a pupil <NUM> of a user.

<FIG> and <FIG> are block diagrams illustrating examples of a configuration of a reflection detection apparatus according to example embodiments.

Referring to <FIG>, a reflection detection apparatus <NUM> includes an image acquirer <NUM> and a processor <NUM>.

The image acquirer <NUM> acquires an input image based on an activation of an infrared light source, and acquires a reference image based on a deactivation of the infrared light source. The image acquirer <NUM> may receive rays that correspond to a visible ray region and an infrared region. The image acquirer <NUM> may include a camera configured to capture the visible ray region and the infrared region. For example, the image acquirer <NUM> may include a high-speed camera with a frame rate that is greater than or equal to "<NUM>" fps. The input image that is mainly based on the infrared region and the reference image that is mainly based on the visible ray region may represent similar object shapes. A region with a great different in pixel values between the input image and the reference image may be a reflection region.

The processor <NUM> extracts a reflection region from the input image based on the input image and the reference image. The input image may be acquired based on an infrared region corresponding to a ray received from an object during an on interval, and the reference image may be acquired based on a visible ray region corresponding to a ray received from an object during an off interval.

Referring to <FIG>, a reflection detection apparatus <NUM> may further include an infrared ray array <NUM> and a memory <NUM>. The processor <NUM> may activate at least a portion of infrared light sources included in the infrared ray array <NUM> during an on interval, and may deactivate at least a portion of the infrared light sources in the infrared ray array <NUM> during an off interval. Also, the processor <NUM> generates a difference map indicating a difference between the input image and the reference image by subtracting a pixel value of the reference image corresponding to a pixel value of the input image from the pixel value of the input image, and extracts the reflection region based on the difference map.

The infrared ray array <NUM> may include a plurality of infrared light sources and may be spaced from the image acquirer <NUM>. For example, the infrared ray array <NUM> is spaced from the image acquirer <NUM> so that a predetermined angle is formed by a direction of the image acquirer <NUM> and a direction of an infrared ray emitted from the infrared ray array <NUM>.

The memory <NUM> may store information used to perform a method of detecting a reflection. For example, the memory <NUM> may temporarily or permanently store the input image, the reference image and the difference map.

The reflection detection apparatus <NUM> may generate an input image from which a pupil may be detected in a low illumination environment (for example, an environment of an illumination less than or equal to <NUM> Lux). For example, the reflection detection apparatus <NUM> may be applicable to a head-up display (HUD) configured to provide an autostereoscopic three-dimensional (3D) image by tracking a gaze. Also, the reflection detection apparatus <NUM> may be applicable to a personal computer (PC) connected to a monitor. In addition, the reflection detection apparatus <NUM> may be applicable to a tablet device or a smartphone configured to provide an autostereoscopic 3D image.

For example, during infrared imaging in a low illumination environment (for example, driving at night-time), the reflection detection apparatus <NUM> may detect and remove an infrared reflection on lenses of glasses. Thus, the reflection detection apparatus <NUM> may enhance an accuracy of a gaze tracking in a relatively dark environment (for example, an environment of an illumination less than or equal to <NUM> Lux).

The example embodiments described herein may be implemented using hardware components, software components, or a combination thereof. A processing device may be implemented using one or more general-purpose or special purpose computers, such as, for example, a processor, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a field programmable array, a programmable logic unit, a microprocessor or any other device capable of responding to and executing instructions in a defined manner. The processing device may run an operating system (OS) and one or more software applications that run on the OS. The processing device also may access, store, manipulate, process, and create data in response to execution of the software. For purpose of simplicity, the description of a processing device is used as singular; however, one skilled in the art will appreciated that a processing device may include multiple processing elements and multiple types of processing elements. For example, a processing device may include multiple processors or a processor and a controller. In addition, different processing configurations are possible, such a parallel processors.

The software may include a computer program, a piece of code, an instruction, or some combination thereof, to independently or collectively instruct or configure the processing device to operate as desired.

The method according to the above-described example embodiments may be recorded in non-transitory computer-readable media including program instructions to implement various operations that may be performed by a computer. The program instructions recorded on the media may be those designed and constructed for the purposes of the example embodiments, or they may be of the well-known kind and available to those having skill in the computer software arts. Examples of non-transitory computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD ROM discs and DVDs; magneto-optical media such as optical discs; and hardware devices that are configured to store and perform program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, and the like. Examples of program instructions include both machine code, such as code produced by a compiler, and files containing higher level code that may be executed by the computer using an interpreter. The described hardware devices may be configured to act as one or more software modules to perform the operations of the above-described example embodiments, or vice versa.

Claim 1:
A method of detecting a reflection, the method being performed by an apparatus (<NUM>) for detecting the reflection, and the method comprising:
acquiring (<NUM>) an input image of an object (<NUM>), by activating an infrared light source (<NUM>) configured to emit infrared rays, wherein the infrared light source (<NUM>) and an image acquirer (<NUM>) are arranged so that a predetermined angle (<NUM>) of <NUM> degrees is formed by a first direction in which the infrared light source (<NUM>) emits a first ray to the object (<NUM>) and by a second direction in which the image acquirer (<NUM>) receives a second ray from the object (<NUM>);
acquiring (<NUM>) a reference image of the object (<NUM>), by deactivating the infrared light source (<NUM>) and generated mainly using a visible ray region;
generating (<NUM>) a difference map (<NUM>) by calculating a difference between the input image and the reference image;
determining elements of the difference map (<NUM>), the elements having difference values exceeding a threshold; and
extracting (<NUM>) a reflection region (<NUM>) from the input image, the reflection region (<NUM>) corresponding to the elements.