Apparatus and method for predicting eye position

An apparatus and method for predicting an eye position includes a storer configured to store detected position information of an eye of user during a sample time interval, a calculator configured to calculate a weighted average value of a variation of the detected position information, and a predictor configured to generate prediction position information of the eye of user at a target time based on the weighted average value, and the calculator is configured to apply a weight to the variation of the detected position information such that the weight increases as the target time is approached during the sample time interval.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the priority benefit of Korean Patent Application No. 10-2014-0177448, filed on Dec. 10, 2014, in the Korean Intellectual Property Office, the entire contents of which are incorporated herein by reference in its entirety.

BACKGROUND

At least one example embodiment relates to an eye tracking based three-dimensional (3D) display.

2. Description of the Related Art

Recently, three-dimensional (3D) displays have gained more attention due to their ability to represent an image in a more realistic manner than a two-dimensional (2D) display. Various 3D display methods may include an eye-tracking based glassless 3D display method. An eye-tracking based glassless 3D display may utilize a display method of detecting an eye position of user and outputting a left eye image and a right eye image included in a 3D image to the respective eyes of the user. In the eye tracking based glassless 3D display, accurately outputting an image to the eye position of user may be significant. Therefore, there is a desire for accurately detecting the eye position of user when the image is output.

SUMMARY

Some example embodiments relate to a three-dimensional (3D) display apparatus.

In some example embodiments, the 3D display apparatus may include an eye position tracker configured to determine position information on an eye of a user, an eye position predictor configured to generate prediction position information on the eye of user at a target time based on a variation of the position information during a sample time interval, and an image processor configured to render a 3D image based on the prediction position information.

The eye position predictor may be configured to generate the prediction position information at the target time based on a sum of the position information at the target time and a weighted average of the variation of the position information during the sample time interval.

The eye position predictor may be configured to calculate a weighted average value of the variation of the position information, and generate the prediction position information based on the weighted average value.

The eye position predictor may be configured to apply a weight to the variation of the position information, the weight increasing as the target time approaches during the sample time interval.

The weight may be determined based on a normal distribution having a maximum value at the target time.

The prediction position information may include first prediction position information generated based on the variation of the position information and second prediction position information generated based on a variation of the first prediction position information.

The image processor may be configured to render the 3D image based on the second prediction position information.

The eye position predictor may include a storer configured to store the position information on the eye of user during the sample time interval, and the storer is configured to shift position information stored in the storer in advance and store the new position information in the storer, in response to receiving new position information.

The image processor may be configured to render the 3D image corresponding to the position information at the target time, in response to an amount of the position information stored in the storer being less than or equal to a desired (or alternatively, predetermined) reference.

The eye position tracker may be configured to determine the position information based on a user image.

Other example embodiments relate to an apparatus for predicting an eye position.

In some example embodiments, the apparatus for predicting an eye position may include a storer configured to store position information on an eye of a user during a sample time interval, a calculator configured to calculate a weighted average value of a variation of the position information, and a predictor configured to generate prediction position information on the eye of user at a target time based on the weighted average value.

The calculator may be configured to calculate a second weighted average value of a variation of the prediction position information during the sample time interval, and the predictor is configured to generate second prediction position information on the eye of user at the target time based on the second weighted average value.

The calculator may be configured to apply a weight to the variation of the position information, the weight increasing as the target time approaches during the sample time interval.

The predictor may be configured to generate the prediction position information at the target time based on a sum of the position information at the target time and a weighted average of the variation of the position information during the sample time interval.

The storer is configured to shift position information stored in the storer in advance and store the new position information in the storer, in response to receiving new position information.

Other example embodiments relate to a method of predicting an eye position.

In some example embodiments, the method of predicting an eye position receiving position information on an eye of a user during a sample time interval, calculating a weighted average value of a variation of the position information, and generating prediction position information on the eye of user at a target time based on the weighted average value.

The method of predicting an eye position may further include calculating a second weighted average value of a variation of the prediction position information during the sample time interval, and generating second prediction position information on the eye of user at the target time based on the second weighted average value.

The calculating may include applying a weight to the variation of the position information, the weight increasing as the target time approaches during the sample time interval.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

It should be understood, however, that there is no intent to limit this disclosure to the particular example embodiments disclosed. On the contrary, example embodiments are to cover all modifications, equivalents, and alternatives falling within the scope of the example embodiments. Like numbers refer to like elements throughout the description of the figures.

Various example embodiments will now be described more fully with reference to the accompanying drawings in which some example embodiments are shown. In the drawings, the thicknesses of layers and regions are exaggerated for clarity.

FIGS. 1A and 1Billustrate examples of rendering a detected eye position and a predicted eye position according to at least one example embodiment.

FIG. 1Aillustrates an example of rendering a detected eye position, andFIG. 1Billustrates an example of rendering a predicted eye position. An eye position of a user may be detected for an eye-tracking based glassless three-dimensional (3D) display. A 3D display apparatus may display a 3D image corresponding to the detected eye position of user. The 3D image may include an image output to a right eye of the user and an image output to a left eye of the user. To provide a user with a clear 3D image, the 3D display apparatus may accurately detect an eye position of the user and accurately output the 3D image to the eye position of user.

When the user moves, a process of detecting the eye position of user and accurately outputting the 3D image to the eye position of user may be difficult. For example, referring toFIGS. 1A and 1B, the user may move to a rendering position from an initial position, through a detecting position. Here, the detecting position may be an eye position of user at a point in time the 3D display apparatus detects the eye position of user, and the rendering position may be an eye position of user at a point in time the 3D display apparatus renders the 3D image.

A desired (or alternatively, predetermined) time may pass while rendering an input image to a 3D image and detecting the eye position of user from a user image by the 3D display apparatus. Accordingly, when the user moves, crosstalk may be generated due to a delay caused by the desired (or alternatively, predetermined) time. For example, referring toFIG. 1A, when the eye position of user moves to the rendering position from the detecting position, crosstalk may be generated when the 3D display apparatus outputs a 3D image to the detecting position at which an eye of user is detected. Referring toFIG. 1B, when the 3D display apparatus predicts the rendering position of the eye of user and outputs the 3D image to the rendering position, crosstalk may be reduced (or alternatively, prevented). Accordingly, predicting an eye position based on a movement of user and outputting a 3D image to the predicted eye position is desired.

FIG. 2is a block diagram illustrating an example of a 3D display apparatus according to at least one example embodiment.

Referring toFIG. 2, a 3D display apparatus100includes an eye position tracker110, an eye position predictor120, and an image processor130.

The eye position tracker110may determine position information on an eye of a user based on a user image. The eye position tracker110may include an image sensor. The eye position tracker110may sense the user image using the image sensor, and determine the position information on the eye of user. The position information may be information on an actual position of the eye of user detected from the user image. The position information may include a coordinate value of the actual position of the eye of user. The eye position tracker110may detect a face of the user from the user image, and determine position information on eye of user from the detected face of user. The eye position tracker110may determine the position information based on data trained with respect to various types of faces and eyes, or relative positions of faces and eyes.

The eye position tracker110may track the eye position of user in a 3D space expressed by an x-y-z coordinate system. The position information may include 3D coordinate values to indicate a position of a right eye of the user and a position of a left eye of the user.

The eye position predictor120may generate prediction position information on the eye of user at a target time based on a variation of the position information during a sample time interval. The prediction position information may be information on prediction position of the eye of user calculated from the position information. For example, the position information may correspond to the detecting position illustrated inFIG. 1, and the prediction position information may correspond to the rendering position illustrated inFIG. 1. The prediction position information may include a coordinate value of a prediction position of eye of the user.

The target time may be a point in time at which rendering is complete. The target time may be referred to as a current time. The sample time interval may be a time interval providing a sample for generating the prediction position information at the target time. The sample time interval may be a desired (or alternatively, predetermined) time interval prior to the target time. For example, when a time interval “t” is “n−10” through “n”, the sample time interval is “n−10” through “n”, and the target time is “n”.

The eye position predictor120may generate the prediction position information on the eye of user at the target time based on the variation of the position information during the sample time interval. The prediction position information may include first prediction position information generated based on the variation of the position information and second prediction position information generated based on a variation of the first prediction position information. Concisely, the eye position predictor120may generate the first prediction position information based on the variation of the position information, and generate the second prediction position information based on the variation of the first prediction position information. Hereinafter, a process for generating the first prediction position information and the second prediction position information will be described in detail.

The process for generating the first prediction position information will be described with reference to the following descriptions.

The first prediction position information may be position information generated based on the variation of the position information. The first prediction position information will be described in detail with reference toFIG. 5.

FIG. 5illustrates an example of a process of generating first prediction position information according to at least one example embodiment.

Referring toFIG. 5, a graph of position information during a sample time interval n−10 through n and first prediction position information at target time n is illustrated.

The position information and the first prediction information may include a 3D x-y-z coordinate, and the position information and the first prediction information inFIG. 5may be coordinate values of any one axis among the 3D coordinates. In addition, the position information and the first prediction information may be coordinate values of any one of a left eye or a right eye of the user. Accordingly, when the following descriptions are applied to the remaining axes and eyes, 3D first prediction position information on both eyes may be obtained.

The eye position predictor120may generate first prediction position information using Equation 1.

In Equation 1, “p1(n)” denotes first prediction position information at “n”, and “m(n)” denotes position information at “n”. “m(t)−m(t−1)” denotes a variation of position information at “t” and “t−1”. “w(t)” denotes a weight of “m(t)−m(t−1)”, and “W” denotes a total sum of the weight “w(t)”. “α” denotes a multiple factor. “α” may be controlled by a user and/or based on empirical evidence. For example, when a delay in generating of the position information or rendering of a 3D image increases, the user may set “α” as a relatively high value. “T” denotes a number of the position information used for generating the first predict position information. Hereinafter, identical parameters have identical meanings.

The weight may be a relatively high value close to a target time during a sample time interval. For example, the weight may be determined based on a normal distribution having a maximum value at the target time. The weight may have a relatively low value at a time further away from the target time. As such, it should be understood that the weight applied to the variation of the position information may increase as the target time is approached during the sample time interval. Equation 2 denotes “w(t)” according to at least one example embodiment.

Equation 2 denotes a normal distribution having a maximum value at a target time. In Equation 2, “σ2” denotes a variance of the normal distribution. “σ” may be controlled by a user. Equation 3 denotes “W”.
W=Σt=n-Tt=nw(t)  [Equation 3]

Based on the aforementioned descriptions of parameters, the right hand side of Equation 1 denotes a sum of the position information at the target time and a weighted average of the variation of the position information during the sample time interval. Accordingly, the eye position predictor120may generate the first prediction position information at the target time based on the sum of the position information at the target time and the weighted average of the variation of position information during the sample time interval. In addition, the eye position predictor120may calculate a weighted average value of a variation of the position information, generate the first prediction position information based on the weighted average value. The eye position predictor120may generate second prediction position information based on the first prediction position information. Hereinafter, the second prediction position information will be described in detail.

The process for generating the second prediction position information will be described with reference to the following descriptions.

The second prediction position information may be position information generated based on the variation of the first prediction position information. The second prediction position information will be described in detail with reference toFIG. 6.

FIG. 6illustrates an example of a process of generating second prediction position information according to at least one example embodiment.

Referring toFIG. 6, a graph of position information during a sample time interval “n−10” through “n”, first prediction position information during the sample time interval “n−10” through “n”, and second prediction position information at a target time “n” is illustrated.

The position information and the prediction information may include a 3D x-y-z coordinate, and the position information and the prediction information inFIG. 6may be a coordinate value of any one axis among the 3D coordinates. In addition, the position information and the prediction information inFIG. 6may be a coordinate value of any one of a left eye or a right eye of the user. Accordingly, when the following descriptions are applied to the remaining axes and eyes, 3D second prediction position information on both eyes may be obtained.

The eye position predictor120may generate second prediction position information using Equation 4.

In Equation 4, “p2(n)” denotes second prediction position information at “n”, and “m(n)” denotes position information at “n”. When “p1′(t)” is “t=n”, “p1′(t)” denotes “m(t)”. When “p1′(t)” is “t≠n”, “p1′(t)” denotes “p1(t)”. “p1′(t)−p1(t−1)” denotes a variation of first prediction position information at “t” and “t−1”. “w(t)” denotes a weight of “p1′(t)−p1(t−1)”, and “W” denotes a total sum of the weight “w(t)”. “α” denotes a multiple factor. With respect to “α”, the descriptions of Equation 1 may be applied to “α”. “T” denotes a number of the position information used for generating the prediction position information.

The weight may be a relatively high value close to a target time during a sample time interval. For example, the weight may be determined based on a normal distribution having a maximum value at the target time. The weight may have a relatively low value at a time further away from the target time. As such, it should be understood that the weight applied to the variation of the position information may increase as the target time is approached during the sample time interval. With respect to “w(t)” and “W”, the descriptions of Equation 2 and Equation 3 may be applied to “w(t)” and “W”.

Based on the aforementioned descriptions of parameters, the right hand side of Equation 4 denotes a sum of the position information at the target time and a weighted average of a variation of the first prediction position information during the sample time interval. Accordingly, the eye position predictor120may generate the second prediction position information at the target time based on the sum of the position information at the target time and the weighted average of the variation of first prediction position information during the sample time interval. In the present disclosure, a weighted average value of first prediction position information during a sample time interval may be referred to as a second weighted average value.

The eye position predictor120may calculate the second weighted average value of the variation of first prediction position information, and generate the second prediction position information based on the second weighted average value.

Referring back toFIG. 2, a 3D display apparatus100will be described. Referring toFIG. 2, an image processor130generates an output image of an input image based on prediction position information. Here, the prediction position information may be the first prediction position information or the second prediction position information described above. In addition, the input image may be a stereo image, and the output image may be a 3D image rendered based on the prediction position information. The output image may be a 3D image corresponding to a coordinate value based on the prediction position information.

To predict an eye position, a sample more than or equal to a desired (or alternatively, predetermined) reference, for example, a sufficient amount of position information, may be desired. For example, referring toFIGS. 5 and 6which will be described hereinafter, position information corresponding to 10 frames may be required. When a sample sufficient to predict the eye position, for example, the sufficient amount of position information, is not stored, the image processor130may render a 3D image corresponding to the position information until the sufficient sample is stored. The image processor130may render the 3D image corresponding to the position information at the target time, in response to an amount of the position information stored in advance being less than or equal to the desired (or alternatively, predetermined) reference.

FIG. 3is a block diagram illustrating an example of an eye position predictor according to at least one example embodiment.

Referring toFIG. 3, the eye position predictor120includes a storer121, a calculator122, and a predictor123.

The storer121stores position information on an eye of a user during a sample time interval. The storer121may receive the position information from the eye position tracker110, and store the received position information. The storer121may shift the position information stored in the storer121in advance, in response to receiving new position information. The operation of the storer121will be described in detail with reference toFIG. 4.

FIGS. 4A through 4Cillustrate an example of a storing process of a storer according to at least one example embodiment.

Referring toFIG. 4, the storer121ofFIG. 3and a coordinate value of position information “LE[k]” are illustrated. In an example, the storer121may be a buffer. The coordinate value “LE[k]” may be referred to as a coordinate value of a left eye. The storer121may store the coordinate value “RE[k]” referred to as a coordinate value of a right eye. Each of “LE[k]” and “RE[k]” may include a 3D x-y-z coordinate value. Referring toFIG. 4, “n” denotes as a number of frames used in the storer121.

FIG. 4Aillustrates position information stored in advance “LE[n−1]” through “LE[n−7]”.FIG. 4Billustrates a shift of the position information stored in advance “LE[n−1]” through “LE[n−7]”, prior to storing new position information “LE[n]”.FIG. 4Cillustrates a storing of the new position information “LE[n]”, subsequent to the shift. The storer121may store the new position information “LE[n]” through the processes ofFIGS. 4A through 4C. Since the position information is stored in the storer121according to a time flow, the storer121may manage the position information without a need for further indexing. The calculator122may calculate a weighted sum by applying a highest weight to a most recently stored value.

Referring back toFIG. 3, the eye position predictor120will be described. The calculator122calculates a weighted average value of a variation of the position information. The calculator122may calculate the weighted average value based on the weight. The calculator122may apply the weight to the variation of position information, the weight being a relatively high value according to a closed to a target time during a sample time interval. In addition, the calculator122may calculate second weighted average value based on a variation of first prediction position information during the sample time interval.

The predictor123generates prediction position information on eye of user at the target time based on the weighted average value. The predictor123may transmit the generated prediction position information to the image processor130. The predictor123may generate the prediction position information at the target time based on a sum of the position information at the target time and the weighted average of the variation of the position information during the sample time interval. The predictor123may generate second prediction position information on eye of user at the target time based on a second weighted average value calculated by the calculator122.

FIG. 7is a flowchart illustrating an example of a method of generating a 3D image according to at least one example embodiment.

Referring toFIG. 7, in operation310, a 3D display apparatus100determines position information on an eye of user based on a user image. The 3D display apparatus100may include an image sensor to determine the position information. The 3D display apparatus100may sense the user image using the image sensor, and determine the position information on the eye of user. The position information may be information on an actual position of the eye of user detected from the user image. The position information may include a coordinate value of the actual position of the eye of user.

In operation320, the 3D display apparatus100generates prediction position information on the eye of user at the target time based on a variation of the position information during a sample time interval. The prediction position information may be information on a prediction position of the eye of user. The prediction position information may include first prediction position information and second prediction position information. With respect to the position information and the prediction position information, the aforementioned descriptions may be applied to the position information and the prediction position information.

In operation330, the 3D display apparatus100renders a 3D image based on the prediction position information. The 3D display apparatus100may render the 3D image corresponding to a coordinate value based on the prediction position information. Even when a user moves, the user may view a clear 3D video corresponding to the prediction position information.

FIG. 8is a flowchart illustrating an example of a method of predicting an eye position according to at least one example embodiment.

Referring toFIG. 8, in operation321, an apparatus for predicting an eye position receives position information on an eye of user during a sample time interval. The position information may include a coordinate value of the eye position of the user.

In operation322, the apparatus for predicting an eye position may store the position information. The apparatus for predicting an eye position120may shift position information stored in advance and store the new position information, in response to receiving new position information. When a sample sufficient to predict the eye position, for example, the sufficient amount of position information, is not stored, the apparatus for predicting an eye position120may render a 3D image corresponding to the position information until the sufficient sample is stored. The apparatus for predicting an eye position120may render the 3D image corresponding to the position information at the target time, in response to an amount of the position information stored in advance being less than or equal to the desired (or alternatively, predetermined) reference.

In operation323, the apparatus for predicting an eye position120calculates a weighted average value of a variation of the position information. The apparatus for predicting an eye position120may calculate the weighted average value based on a weight. The apparatus for predicting an eye position120may apply the weight to the variation of the position information, the weight being a relatively high value according to a closed to the target time during the sample time interval. In addition, the eye position prediction information may calculate second weighted average value based on a variation of first prediction position information on the sample time interval.

In operation324, the apparatus for predicting an eye position120generates prediction position information on the eye of user at the target time based on the weighted average value. The apparatus for predicting an eye position may generate the prediction position information at the target time and a weighted average of the variation of the position information during the sample time interval. The apparatus for predicting an eye position120may generate second prediction position information on the eye of user at the target time based on the second weighted average value.

The units and/or modules described herein may be implemented using hardware components and software components. For example, the hardware components may include microphones, amplifiers, band-pass filters, audio to digital converters, and processing devices. A processing device may be implemented using one or more hardware device configured to carry out and/or execute program code by performing arithmetical, logical, and input/output operations. The processing device(s) may include a processor (i.e., a special purpose processor), a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a field programmable array, a programmable logic unit, a microprocessor or any other device capable of responding to and executing instructions in a defined manner. The processing device may run an operating system (OS) and one or more software applications that run on the OS. The processing device also may access, store, manipulate, process, and create data in response to execution of the software. For purpose of simplicity, the description of a processing device is used as singular; however, one skilled in the art will appreciated that a processing device may include multiple processing elements and multiple types of processing elements. For example, a processing device may include multiple processors or a processor and a controller. In addition, different processing configurations are possible, such a parallel processors.