Method and apparatus for generating three-dimensional image

A method for generating a three-dimensional (3D) image may detect a current eye position of a user and render a 3D image based on at least one of a previously detected eye position of the user and previously generated stereo images. A cycle at which the current eye position of user is detected and a cycle at which a 3D image is rendered may be asynchronous.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the priority benefit of Korean Patent Application No. 10-2014-0169678, filed on Dec. 1, 2014, in the Korean Intellectual Property Office, the entire contents of which are incorporated herein by reference in its entirety.

BACKGROUND

At least one example embodiment relates to a method and/or apparatus for generating a three-dimensional (3D) image, and more particularly, to a method and/or apparatus for generating the 3D image in which a plurality of processes are asynchronously operated.

2. Description of the Related Art

Three-dimensional (3D) video displays may be categorized as a glasses type display and a glasses-free type display. A method of providing a 3D video in a glasses-free type display may provide left and right images to both eyes of a user. To provide the images of the left and right images to the respective eyes, positions of the right eye and the left eye may be tracked. For example, positions of the right eye and the left eye may be detected, and a 3D video may be provided based on the detected positions. When the positions of the right eye and the left eye change (i.e., a user changes position), a user may view a 3D video having a deteriorated image quality.

SUMMARY

At least one example embodiment relates to a method of generating a three-dimensional (3D) image.

In at least one example embodiment, the method may include detecting an eye position of a user, and rendering a 3D image based on at least one of an eye position (e.g., a current eye position) of the user detected in advance and stereo images generated in advance. The detecting and the rendering are performed in parallel.

A cycle of the detecting and a cycle of the rendering may be asynchronized.

The rendering may include estimating a current eye position based on a stored eye position of the user, and rendering the 3D image based on the estimated current eye position and the stereo images generated in advance.

The method of generating a 3D image may further include generating respective stereo images of both eyes of the user based on the eye position of the user detected in advance.

The generating may include estimating a current eye position based on a stored eye position of the user, and generating the respective stereo images based on the estimated current eye position.

The estimating may include calculating a speed vector of numerous detected eye positions and estimating the current eye position based on the speed vector.

The generating may include different operation cycles based on types of contents to be played.

The method of generating a 3D image may further include receiving an image, and the generating may include generating the stereo images using the image.

The generating may include detecting a foreground of the image, calculating a motion vector of the detected foreground, estimating a current position of the foreground in the image based on the calculated motion vector, and generating the stereo images based on the estimated position.

The generating respective stereo images may include generating the stereo images by receiving stereo images generated in advance.

At least one example embodiment relates to an apparatus for generating a three-dimensional (3D) image.

In at least one example embodiment, the apparatus may include an eye tracker configured to detect an eye position (e.g., a current eye position) of a user, and a renderer configured to render a 3D image based on at least one of an eye position of the user detected in advance and stereo images generated in advance. The eye tracker and the renderer are performed in parallel.

A cycle at which the eye tracker may be performed and a cycle at which the renderer is performed are asynchronized.

The apparatus for generating a 3D image may further include a stereo image generator configured to generate respective stereo images of both eyes of the user based on the eye position of the user detected in advance.

The stereo image generator may be configured to estimate a current eye position based on a stored eye position of the user, and generate the respective stereo images based on the estimated current eye position.

The stereo image generator may be configured to calculate a speed vector of numerous detected eye positions, and estimate the current eye position based on the speed vector.

The stereo image generator may be configured to include different operation cycles based on types of contents to be played

The apparatus for generating a 3D image may further include an image receiver configured to receive an image. The stereo image generator may be configured to generate the stereo images using the image.

The stereo image generator may be configured to detect a foreground of the image, calculate a motion vector of the detected foreground, estimate a current position of the foreground in the image based on the calculated motion vector, and generate the stereo images based on the estimated position.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

It should be understood, however, that there is no intent to limit this disclosure to the particular example embodiments disclosed. On the contrary, example embodiments are to cover all modifications, equivalents, and alternatives falling within the scope of the example embodiments. Like numbers refer to like elements throughout the description of the figures.

Various example embodiments will now be described more fully with reference to the accompanying drawings in which some example embodiments are shown. In the drawings, the thicknesses of layers and regions are exaggerated for clarity.

Hereinafter, reference will now be made in detail to examples with reference to the accompanying drawings, wherein like reference numerals refer to like elements throughout.

Various alterations and modifications may be made to the examples. Here, the examples are not construed as limited to the disclosure and should be understood to include all changes, equivalents, and replacements within the idea and the technical scope of the disclosure.

The terminology used herein is for the purpose of describing particular examples only and is not to be limiting of the examples. As used herein, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “include/comprise” and/or “have” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, components, and/or combinations thereof, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, elements, components, and/or groups thereof.

When describing the examples with reference to the accompanying drawings, like reference numerals refer to like constituent elements and a repeated description related thereto will be omitted. When it is determined detailed description related to a related known function or configuration they may make the purpose of the examples unnecessarily ambiguous in describing the examples, the detailed description will be omitted here.

FIG. 1is a diagram illustrating a viewing of a three-dimensional (3D) video using both eyes of a user according to at least one example embodiment.

An apparatus100for displaying a 3D video may detect positions of a left eye110and a right eye120of a user viewing a video, and enable the user to sense a 3D effect by displaying respective videos at the detected positions.

In an example, the apparatus100may generate respective stereo images for the left eye110and the right eye120of the user. The apparatus100may render the 3D image using the generated stereo images. The apparatus100may display the 3D video using a plurality of rendered 3D images.

In another example, when images input or received are stereo images, the apparatus100may render a 3D image using the stereo images. The apparatus100may display a 3D video using a plurality of 3D images.

According to at least one example embodiment, when frames or images of a video to be displayed are obtained, the apparatus100may detect positions of both eyes of a user. When the positions of both eyes of user are detected, the apparatus100may generate stereo images based on the detected positions of both eyes and the obtained images. When the stereo images are generated, the apparatus100may render a 3D image using the detected positions of both eyes and the generated stereo images.

Accordingly, the apparatus100may perform a subsequent operation (e.g., a rendering operation) when a previous operation (e.g., an eye tracking operation) is performed. The 3D image generating method may be a synchronous method.

In a synchronous 3D image generating method, a time required for each operation may be accumulated. When the positions of both eyes of user are changed, the synchronous 3D image generating method is unable to reflect the change, thereby deteriorating an image quality of the 3D video to be displayed. For example, the deterioration in image quality may be due to crosstalk.

As still another example, the apparatus100may asynchronously detect positions of both eyes, generate stereo images, and render a 3D image.

When the detecting, the generating, and the rendering are asynchronously performed, accumulated time delay may be reduced (or alternatively, prevented). Since there is no time delay, even when the positions of both eyes of the user are changed, deterioration of the image quality of the 3D video may be mitigated (or alternatively, prevented).

An asynchronous 3D image generating method will be described in detail with reference toFIGS. 2 through 7.

FIG. 2is a block diagram illustrating a configuration of a 3D image generating apparatus according to at least one example embodiment.

Hereinafter, a 3D image generating apparatus100may be simply referred to as an apparatus100.

The apparatus100may include an image receiver210, an eye tracker220, a stereo image generator230, a renderer240, and a displayer250. One or more of these elements may be implemented by, for example, one or more special purpose processors.

The image receiver210may receive a basic image of a 3D video to be displayed.

In an example, the image receiver210may receive respective frames of a two-dimensional (2D) video as the basic image. In another example, the image receiver210may receive a 3D graphics file or computer graphics metafile.

The eye tracker220may detect positions of both eyes of a user.

The stereo image generator230may generate stereo images based on the basic image and the positions of both eyes of the user.

The renderer240may generate a 3D image based on the stereo images and the positions of both eyes of the user.

The displayer250may display a 3D video based on the 3D image.

The image receiver210, the eye tracker220, the stereo image generator230, the renderer240, and the displayer250will be described in detail with reference toFIGS. 3 through 7. A 3D image generating method with reference toFIGS. 3 through 7are descriptions of asynchronous method.

FIG. 3is a flowchart illustrating a 3D image generating method according to at least one example embodiment.

In operation310, the eye tracker220may detect an eye position of a user viewing the apparatus100. As used herein, an eye may refer to both eyes of a user.

For example, the eye tracker220may photograph the user using a camera. The eye tracker220may detect the eye position of the user using the photographed image of the user.

The eye tracker220may photograph the user at a photographing cycle (e.g., a maximum photographing cycle) of the camera.

The eye tracker220may detect the eye position at a first cycle. The first cycle may be the maximum photographing cycle of the camera.

In another example, the eye tracker220may release a ray to the eyes of user. The eye tracker220may detect the eye position of user using the ray reflected from the eyes.

The eye tracker220may store the detected eye position of user.

In operation320, the stereo image generator230may generate respective stereo images of both eyes based on an eye position of the user detected in advance.

The eye position of the user detected in advance may be a recently stored eye position (e.g., a most recently stored eye position).

The stereo image generator230may use the recently stored eye position of user detected in advance.

The stereo image generator230may generate the respective stereo images based on the eye position of the user detected in advance and a received image.

The received image may be an image which is the most recently received.

The stereo image generator230may update the stereo images by generating the stereo images in lieu of operation330of rendering a 3D image.

The stereo image generator230may store the generated stereo images.

The stereo image generator230may generate respective stereo images at a second cycle.

The first cycle may be different from the second cycle. The first cycle and the second cycle may be asynchronized. The stereo image generator230may include different operation cycles based on types of contents to be displayed. In an example, when an image is displayed, the stereo image generator230may be operated once. In another example, when a video is displayed, the stereo image generator230may set an operation cycle based on a frame rate of the video. As still another example, when a graphics image is displayed, the stereo image generator230may set a cycle based on a rendering speed of the graphics image. The graphics image may be an image including 3D information.

According to at least one example embodiment, operation320may be an operation of receiving stereo images. The stereo image generator230may receive stereo images generated in advance. The stereo image generator230may generate the stereo images by receiving the stereo images generated in advance.

For example, the stereo image generator230may receive the stereo images from a camera generating stereo images. A plurality of cameras may be provided.

When the stereo images generated in advance are received, the second cycle may be a cycle at which the stereo image generator230receives stereo images.

In operation330, the renderer240may render a 3D image based on an eye position of the user detected in advance and at least one of stereo images generated in advance.

The eye position detected in advance may be a recently stored eye position.

The stereo images generated in advance may be recently stored stereo images.

The renderer240may render a 3D image at a third cycle. The third cycle may be different from the first and the second cycles. The third cycle may be asynchronized with the first and the second cycles.

Operations310,320, and330may be performed in parallel.

Since operations320and330are performed in parallel, operation330may be performed when operation310is performed and an eye position of a user is updated, even when stereo images are not updated because operation320is not performed. When the detected eye position of user is updated and differs from a desired (or alternatively, predetermined) eye position, the renderer240may render a 3D image based on the updated eye position of user.

For example, when the second cycle at which stereo images are received or generated is longer than the first and the third cycles, operations310and330may be performed even when operation320is not performed. In this example, in operation330, the renderer240may render a new 3D image based on the updated eye position of user and desired (or alternatively, predetermined) stereo images.

FIG. 4illustrates a time flow of performing an asynchronous 3D image generating method according to at least one example embodiment.

A cycle of eye tracking operation410may be referred to as a cycle at which the aforementioned operation310is performed. It should be appreciated thatFIG. 4shows how the operations ofFIG. 3are performed in parallel.

The eye tracker220may photograph a user using a camera and detect an eye position of the user using the photographed image. For example, the eye tracker220may detect the eye position of user at processing sections of E1and E2of eye tracking operation410, using the photographed image.

When the processing sections of E1and E2conclude, the eye tracker220may update the eye position of user. The eye tracker220may update the eye position of user by storing the detected eye position of user. The eye tracker220may update the eye position of user by storing the detected eye position of user.

For example, an eye position may be updated at conclusion411of the processing section of E1and conclusion412of the processing section of E2.

A cycle of stereo image generating operation420may be referred to as a cycle at which the aforementioned operation320is performed.

For example, the stereo image generator230may consecutively generate stereo images. Concisely, the stereo image generator230may consecutively generate the stereo images for processing sections of S1, S2, S3, S4, and S5.

The stereo image generator230may update the stereo images by storing the generated stereo images.

In an example, the stereo image generator230may generate stereo images at the processing section of S1, based on an eye position of a user updated at a processing section of E0(not shown). At the processing section of S2, since the processing section of E1is ongoing, the stereo image generator230may generate stereo images based on a desired (or alternatively, predetermined) eye position of the user. At the processing section of S3, the stereo image generator230may generate stereo images based on an eye position of the user updated at the processing section of E1.

A cycle of 3D image rendering operation430may be a cycle at which the aforementioned operation330is performed.

The renderer240may render a 3D image based on the recently updated eye position and at least one of the most recently generated stereo images.

For example, the renderer240may consecutively render a 3D image. Concisely, the renderer240may consecutively render a 3D image at processing sections of R1through R10.

For example, at the processing section of R2, since a processing of E1and S1is ongoing, the renderer240may render a 3D image based on data of E0and S0. At the processing section of R3, the renderer240may render a 3D image based on data associated with E1and S1. At the processing section of R5, the renderer240may render a 3D image based on data associated with E1and S2.

Since an eye position and stereo images received by an input at the processing section of R3are updated, a 3D image generated based on a processing result of R3may be different from a 3D image generated based on a processing result of R2. When the generated 3D image is different from the previous 3D image, the 3D image may be updated.

When the 3D image is updated, an image of video to be output to the displayer250may be changed.

Reference is made to an input at a processing section of the renderer240when at least one of the detected eye position and the generated stereo images are updated. When the detected eye position and at least one of the generated stereo images are updated, the generated 3D image may be different from the 3D image generated at the previous processing section. In this case, the renderer240may update a 3D image.

For example, at processing sections of R3, R5, and R8, the renderer240may render a 3D image based on an updated eye position.

FIG. 5is a flowchart illustrating a method of generating stereo images according to at least one example embodiment.

The aforementioned operation320may include operations510and520.

In operation510, the stereo image generator230may estimate a current eye position based on a stored eye position of a user.

For example, the stereo image generator230may estimate the current eye position based on a plurality of detected eye positions (e.g., previously detected eye positions). The stereo image generator230may generate a speed vector of an eye position. The stereo image generator230may estimate the current eye position based on the speed vector. For example, the stereo image generator230may calculate a speed vector of numerous detected eye positions, and estimate the current eye position based on the speed vector.

In another example, the stereo image generator230may estimate a current eye position using a Kalman filter.

In operation520, the stereo image generator230may generate stereo images based on the estimated current eye position.

FIG. 6is a flowchart illustrating a 3D image generating method according to at least one example embodiment.

The aforementioned operation330may include operations610and620.

In operation610, the renderer240may estimate a current eye position based on a stored eye position (e.g., a previously detected eye position) of a user.

For example, the renderer240may estimate a current eye position based on a plurality of detected eye positions (e.g., a plurality of previously detected eye positions). The renderer240may generate a motion vector of an eye position. The renderer240may estimate the current eye position based on the motion vector.

In operation620, the renderer240may render a 3D image based on the estimated current eye position and stereo images generated in advance.

FIG. 7is a flowchart illustrating a 3D image generating method according to at least one example embodiment.

In operation710, the image generator210may receive an image.

Operation710may be performed in parallel with operations310,320, and330ofFIG. 3.

Operation320may include operations722through728detailed in the following.

In operation722, the stereo image generator230may detect a received frame or a foreground of an image.

In operation724, the stereo image generator230may calculate a motion vector of the detected foreground. The stereo image generator230may calculate the motion vector of the foreground based on a plurality of frames or images.

In operation726, the stereo image generator230may estimate a current position of the foreground based on the motion vector.

In operation728, the stereo image generator230may generate stereo images based on the estimated current position of the foreground.

The units and/or modules described herein may be implemented using hardware components and software components. For example, the hardware components may include microphones, amplifiers, band-pass filters, audio to digital convertors, and processing devices. A processing device may be implemented using one or more hardware device configured to carry out and/or execute program code by performing arithmetical, logical, and input/output operations. The processing device(s) may include a processor (i.e., a special purpose processor), a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a field programmable array, a programmable logic unit, a microprocessor or any other device capable of responding to and executing instructions in a defined manner. The processing device may run an operating system (OS) and one or more software applications that run on the OS. The processing device also may access, store, manipulate, process, and create data in response to execution of the software. For purpose of simplicity, the description of a processing device is used as singular; however, one skilled in the art will appreciated that a processing device may include multiple processing elements and multiple types of processing elements. For example, a processing device may include multiple processors or a processor and a controller. In addition, different processing configurations are possible, such a parallel processors.