Method and apparatus for processing omni-directional image

A method and apparatus are provided for processing information about an omni-directional image. The method includes generating a first two-dimensional (2D) image projected from a first omni-directional image, by setting points on the first omni-directional image, which intersect a straight line passing through a first position that is a center of the first omni-directional image and a second position that is a center of a second omni-directional image, to a first pole and a second pole, generating a second 2D image projected from the second omni-directional image, by setting points on the second omni-directional image, which intersect the straight line passing through the first position and the second position, to a third pole and a fourth pole, and generating a third 2D image corresponding to a 2D image projected from a third omni-directional image centered in a third position between the first position and the second position, based on the first 2D image and the second 2D image.

BACKGROUND

The present disclosure relates generally to a method and apparatus for processing an omni-directional image, and more particularly, to a method and apparatus for obtaining an omni-directional image having a new center from omni-directional images having different centers.

2. Description of Related Art

Along with technologies associated with virtual reality (VR) and/or augmented reality (AR), improvements have been made in processing and transmitting omni-directional images (e.g., three-dimensional (3D) images) for display in devices capable of providing VR or AR.

An omni-directional image may be generated from an image acquired by an omni-directional camera or a plurality of images acquired by a plurality of two-dimensional (2D) cameras. To provide a realistic VR experience to a user in response to movement of a VR device, a plurality of omni-directional images having different centers are provided and an omni-directional image selected from among a plurality of omni-directional images is provided in response to the user's movement.

Because it impossible to acquire omni-directional images by using a camera in all the possible positions, an omni-directional image centered in a region may not be acquired directly by a camera if the region is different from that of where the camera is arranged. Accordingly, an omni-directional image having a different center than the centers of previously acquired omni-directional images should be generated to provide a realistic VR experience to the user.

SUMMARY

Embodiments of the present disclosure are to address at least the above-mentioned problems and/or disadvantages and to provide at least the advantages described below.

According, an aspect of the disclosure is to provide a method and apparatus for generating an omni-directional image having a different center than those of already acquired omni-directional images.

In accordance with an aspect of the disclosure, a method is provided for processing information about an omni-directional image. The method includes generating a first two-dimensional (2D) image projected from a first omni-directional image, by setting points on the first omni-directional image, which intersect a straight line passing through a first position that is a center of the first omni-directional image and a second position that is a center of a second omni-directional image, to a first pole and a second pole, generating a second 2D image projected from the second omni-directional image, by setting points on the second omni-directional image, which intersect the straight line passing through the first position and the second position, to a third pole and a fourth pole, and generating a third 2D image corresponding to a 2D image projected from a third omni-directional image centered in a third position between the first position and the second position, based on the first 2D image and the second 2D image.

In accordance with another aspect of the disclosure, an apparatus is provided for processing information about an omni-directional image. The apparatus includes a communication interface and a processor electrically connected with the communication interface, in which the processor may generate a first 2D image projected from a first omni-directional image, by setting points on the first omni-directional image, which intersect a straight line passing through a first position that is a center of the first omni-directional image and a second position that is a center of a second omni-directional image, to a first pole and a second pole, to generate a second 2D image projected from the second omni-directional image, by setting points on the second omni-directional image, which intersect the straight line passing through the first position and the second position, to a third pole and a fourth pole, and to generate a third 2D image corresponding to a 2D image projected from a third omni-directional image centered in a third position between the first position and the second position, based on the first 2D image and the second 2D image.

DETAILED DESCRIPTION

Embodiments of the present disclosure are described with reference to the accompanying drawings.

Similar reference numerals may be used to indicate similar components that perform substantially the same functions throughout the specification.

As used herein, terms such as “first” and “second” or “1st” “2nd” may be used to simply distinguish a corresponding component from another, and do not limit the components in other aspect (e.g., importance or order). Therefore, a first element mentioned below may be a second element, or vice versa, within the technical spirit of the disclosure.

FIG. 1illustrates cameras and omni-directional images corresponding thereto, according to an embodiment.

Referring toFIG. 1, each of a first camera C1and a second camera C2may be an omni-directional camera or a set of a plurality of cameras. A first omni-directional image O1may be captured and generated by the first camera C1. A second omni-directional image O2may be captured and generated by the second camera C2. Each of the first omni-directional image O1and the second omni-directional image O2may be rendered in the shape of a sphere. A center L1of the first omni-directional image O1corresponds to a location of the first camera C1. A center L2of the second omni-directional image O2corresponds to a location of the second camera C2.

Points at which a straight line passing through the center L1of the first omni-directional image O1and the center L2of the second omni-directional image O2intersects the first omni-directional image O1and the second omni-directional image O2are referred to as epipoles (e.g., poles) e1, e2, e3, and e4.

Data regarding a third omni-directional image O3is generated based on data regarding the first omni-directional image O1and data regarding the second omni-directional image O2. The third omni-directional image O3corresponds to an image acquired by a third camera C3. The third camera C3may be a virtual camera. A center L3of the third omni-directional image O3corresponds to a location of the third camera C3. The center L3of the third omni-directional image O3is located between the center L1of the first omni-directional image O1and the center L2of the second omni-directional image O2on the straight line passing through the center L1of the first omni-directional image O1and the center L2of the second omni-directional image O2.

An axis passing through a north pole N1and a south pole S1of the first omni-directional image O1and an axis passing through a north pole N2and a south pole S2of the second omni-directional image O2may not be parallel with each other. In this case, a two-dimensional 2D image projected from the first omni-directional image O1in an ERP manner with respect to the north pole N1and the south pole S1of the first omni-directional image O1and a 2D image projected from the second omni-directional image O2in the ERP manner with respect to the north pole N2and the south pole S2of the second omni-directional image O2may not be aligned with each other. Generating data regarding a third omni-directional image O3from the non-aligned 2D images increases the complexity of processing, which increases a system load and, in turn, increases power consumption of the system.

FIG. 2illustrates positions of a particular point indicated on a first omni-directional image and a second omni-directional image, according to an embodiment.

Referring toFIG. 2, the north poles N1and N2and the south poles S1and S2of the omni-directional images O1and O2may be adjusted (e.g., reset) to positions of the first through fourth epipoles e1, e2, and e3, and e4. More specifically, the north pole N1and the south pole S1of the first omni-directional image O1may be set to the first epipole e1and the second epipole e2, respectively, and the north pole N2and the south pole S2of the second omni-directional image O2may be set to the third epipole e3and the fourth epipole e4, respectively.

A first point P1on a space may be indicated at a first position p11on the first omni-directional image O1and a second position p12on the second omni-directional image O2. Based on the adjusted north poles N1and N2and south poles S1and S2of the first and second omni-directional images O1and O2, the first position p11and the second position p12have the same longitude as each other. A position on the first omni-directional image O1where an arbitrary point is the first point P1on a space is indicated, and a position on the second omni-directional image O2where an arbitrary point is indicated (e.g., P12) have the same longitude as each other.

Based on the adjusted north poles N1and N2and south poles S1and S2of the first and second omni-directional images O1and O2, ERP projection may be performed with respect to the first and second omni-directional images O1and O2.

FIG. 3illustrates first and second 2D images that are ERP-projected from a first omni-directional image and a second omni-directional image, according to an embodiment.

Referring toFIG. 3, a first 2D image E1and a second 2D image E2are generated by being ERP-projected from the first omni-directional image O1and the second omni-directional image O2, respectively. A left side and a right side of the first 2D image E1correspond to the first epipole e1and the second epipole e2, respectively. In an ERP-projection as illustrated inFIG. 2, a y coordinate of a particular position on a 2D image corresponds to a longitude of a corresponding position on an omni-directional image. An x coordinate of the particular position on the 2D image corresponds to an angle between the corresponding position on the omni-directional image and the north pole (e.g., an epipole) with respect to a center of the omni-directional image. Thus, the y coordinate of the first position p11on the first 2D image E1and the y coordinate of the second position p12on the second 2D image E2may be the same as each other. Likewise, the y coordinate of an arbitrary position on the first 2D image E1and the y coordinate of a corresponding position on the second 2D image E2may be the same as each other.

That is, according toFIGS. 2 and 3, corresponding positions on the 2D images E1and E2are aligned on the same y coordinates, simplifying a process for generating the data regarding the third omni-directional image O3based on the 2D images E1and E2.

FIG. 4illustrates a method for generating a third 2D image based on first and second 2D images, according to an embodiment.

Referring toFIG. 4, a second point P2on a space may be indicated at a third position p21on the first 2D image E1and a fourth position p22on the second 2D image E2. The third 2D image E3corresponds to an image that is ERP-projected from the third omni-directional image O3. The second point P2may be indicated at a fifth position p23on the third 2D image E3.

Y coordinates of the third position p21, the fourth position p22, and the fifth position p23may be the same as one another as described above with reference toFIGS. 2 and 3.

A disparity value between an x coordinate of the third position p21and an x coordinate of the fourth position p22may be expressed with λ. That is, when the x coordinate of the third position p21is x1and the x coordinate of the fourth position p22is x2, λ=x1−x2. When an x coordinate of the fifth position p23is x3, x3=x1−(a)×λ. Accordingly, (a) indicates a ratio of a distance between the center L1of the first omni-directional image O1and the center L3of the third omni-directional image O3with respect to a distance between the center L1of the first omni-directional image O1and the center L2of the second omni-directional image O2. By identifying positions on the third 2D image E3, which correspond to points on a space, in this way, the third 2D image E3may be generated. The third 2D image E3may correspond to a 2D image generated by setting points e5and e6(e.g., epipoles) on the third omni-directional image O3, which intersect the straight line passing through the center L1of the first omni-directional image O1and the center L2of the second omni-directional image O2, to the north poles and the south poles and projecting the third omni-directional image O3as an ERP.

FIG. 5illustrates a method for generating a third 2D image based on first and second 2D images, according to an embodiment.

Referring toFIG. 5, a third point P3on a space may be indicated at a sixth position p31on the first omni-directional image O1, a seventh position p32on the second omni-directional image O2, and an eighth position p33on the third omni-directional image O3. An x coordinate of a position on an ERP-projected 2D image corresponding to a particular position on an omni-directional image corresponds to an angle between the north pole (e.g., an epipole) of the omni-directional image and the particular position on the omni-directional image with respect to the center of the omni-directional image. Thus, an x coordinate of the sixth position p31on the first 2D image E1corresponds to θL. An x coordinate of the seventh position p32on the second 2D image E2corresponds to θR. A disparity value between the x coordinate of the sixth position p31on the first 2D image E1and the x coordinate of the seventh position p32on the second 2D image E2may be expressed as θL−θRwhich equals α+β. α is an angle from the third point P3between the center L1of the first omnidirectional image O1and the center L3of the third omnidirectional image O3or an angle between the sixth position p31and the eighth position p33. β is an angle from the third point P3between the center L2of the second omnidirectional image O2and the center L3of the third omnidirectional image O3or an angle between the seventh position p32and the eighth position p33.

An x coordinate of the eighth position p33on the 2D image that is ERP-projected from the third omni-directional image O3corresponds to θC, where θC=θL−α, Herein, α is given as shown in Equation (1) below.

Y coordinates of the sixth position p31, the seventh position p32, and the eighth position p33on the 2D image may be the same as one another.

By identifying positions on the 2D image ERP-projected from the third omni-directional image O3, which correspond to points on a space, in this way, the 2D image may be generated by setting the epipoles e5and e6to the north poles and the south poles and projecting the third omni-directional image O3in the ERP manner.

FIG. 6illustrates a method for setting a center of a third omni-directional image, according to an embodiment.

Referring toFIG. 6, the center L3of the third omni-directional image O3may be selected from among quantized positions A1, A2, and A3between the center L1of the first omni-directional image O1and the center L2of the second omni-directional image O2on the straight line passing through the center L1of the first omni-directional image O1and the center L2of the second omni-directional image O2. The number of quantized positions may be set differently according to embodiments. Intervals between the center L1of the first omni-directional image O1, the center L2of the second omni-directional image O2, and the plurality of quantized positions A1, A2, and A3may be set uniformly, but the disclosure is not limited thereto.

In various embodiments, among the plurality of quantized positions A1, A2, and A3, a position closest to a device D for displaying an omni-directional image or a user of the device D may be set as the center L3of the third omni-directional image O3.

The second quantized position A2may be set as the center L3of the third omni-directional image O3. When each distance between each of the plurality of quantized positions A1, A2, and A3and the device D for displaying an omni-directional image or the user of the device D is larger than the distance between the center L1of the first omni-directional image O1and the device D for displaying an omni-directional image, or larger than the distance between the center L1of the first omni-directional image O1and the user of the device D, an image corresponding to the first omni-directional image O1may be displayed in the device D for displaying an omni-directional image. The third omni-directional image O3may not be generated or may not be delivered to the device D for displaying an omni-directional image.

When the device D or the user thereof, is moving, a ratio of a distance between the closest quantized position among the plurality of quantized positions A1, A2, and A3and the device D with respect to a distance between a center of an omni-directional image currently displayed on the device D and the device D is less than or equal to a threshold value, the device D displays an omni-directional image centered in the closest quantized position.

Accordingly, an apparatus (e.g., a server) for transmitting data regarding an omni-directional image generates data regarding an omni-directional image centered in the closest quantized position and transmits the generated data to the device D.

Referring toFIG. 6, when an omni-directional image centered in the first quantized position A1is currently displayed on the device D, a ratio of a distance d2between the device D and the second quantized position A2with respect to a distance d1between the device D and the first quantized position A1is less than or equal to a threshold value, the device D displays an omni-directional image centered in the second quantized position A2.

An apparatus for transmitting data regarding an omni-directional image previously generates and stores data regarding an omni-directional image centered in each of the plurality of quantized positions A1, A2, and A3, and transmits, to the device D, data regarding the omni-directional image, which is selected due to a need from among previously generated and stored data regarding a plurality of omni-directional images.

The device D stores the data regarding the omni-directional image centered in each of the plurality of quantized positions A1, A2, and A3, retrieves the data regarding the omni-directional image, which is selected due to a need from among the previously stored data regarding the plurality of omni-directional images, and displays the selected omni-directional image.

FIG. 7is a flowchart illustrating a method for processing an omni-directional image, according to an embodiment.

Referring toFIG. 7, in step710, a first omni-directional image is projected as a first 2D image by setting epipoles of the first omni-directional image to poles, and a second omni-directional image is projected as a 2D image by setting epipoles of the second omni-directional image to poles. Detailed operations of step710are similar as described with referenceFIGS. 2 and 3.

In step720, a third 2D image corresponding to a third omni-directional image is generated from the first 2D image and the second 2D image. Detailed operations of step720may be the operations described with reference toFIG. 4or the operations described with reference toFIG. 5.

Steps710and720may be performed by an apparatus (e.g., a server) for transmitting data regarding an omni-directional image. In various embodiments, steps710and720may be performed by an apparatus (e.g., a VR device) for displaying data regarding an omni-directional image.

FIG. 8is a block diagram of an apparatus for processing an omni-directional image, according to an embodiment.

Referring toFIG. 8, an apparatus800for processing an omni-directional image may be an apparatus for transmitting data regarding an omni-directional image (e.g., a server). The apparatus800generates data regarding the third omni-directional image O3and transmits the generated data using the above-described methods. The data regarding third omni-directional image O3is based on a 2D image corresponding to a 2D image that is ERP-projected from the third omni-directional image O3. The apparatus800may transmit data regarding the 2D image corresponding to the 2D image that is ERP-projected from the third omni-directional image O3by using a protocol such as the Moving Picture Experts Group (MPEG).

For convenience of a display in a receiving side, the apparatus800converts the generated 2D image into a 2D image corresponding to a 2D image that is ERP-projected based on the different poles of the third omni-directional image O3. Transmission of the data regarding the 2D image may be based on the converted 2D image. In various embodiments, packing according to region (e.g., region-wise) may be performed on the 2D image. Region-wise packing means dividing the 2D image into a plurality of regions and performing transformation, rotation, re-sampling, or re-arrangement with respect to each of the plurality of regions.

The apparatus800includes a processor810, a memory820, and a communication interface830. The processor810substantially performs and controls operations performed in the apparatus800. The processor810is electrically connected with the memory820and the communication interface830for communication, and controls operations of the memory820and the communication interface830. Thus, operations performed directly by the memory820or the communication interface830may be interpreted as being substantially performed by the processor810. The memory820transitorily or non-transitorily stores data needed for operations of the apparatus800or the processor810. The memory820stores instructions or codes that are executable by the processor810. The communication interface830may be configured to transmit and/or receive data.

FIG. 9is a block diagram of an apparatus for processing an omni-directional image, according to an embodiment.

Referring toFIG. 9, an apparatus900may be an apparatus for displaying an omni-directional image (e.g., a VR device). The apparatus900receives data regarding an omni-directional image and displays the received data. For example, the apparatus may receive data regarding the third omni-directional image O3based on an image corresponding to an image that is ERP-projected from the third omni-directional image O3generated by the above-described methods, and display at least a part of the third omni-directional image O3based on the received data.

The apparatus900receives data regarding the first omni-directional image O1and data regarding the second omni-directional image O2. The data regarding the first omni-directional image O1may be data regarding the first 2D image E1, and the data regarding the second omni-directional image O2may be data regarding the second 2D image E2. The apparatus900generates data regarding a 2D image corresponding to an image that is ERP-projected from the third omni-directional image O3by using the above-described methods based on the data regarding the first 2D image E1and the data regarding the second 2D image E2. The apparatus900displays at least a part of the third omni-directional image O3based on the generated data regarding the 2D image.

The apparatus900includes a processor910, a memory920, a communication interface930, and a display940. A description of the processor910, the memory920, and the communication interface930is substantially similar to that of the processor810, the memory820, and the communication interface830of the apparatus800ofFIG. 8. The display940displays an image under control of the processor910.

Accordingly, embodiments of the disclosure provide, at least, a method and apparatus for generating an omni-directional image having a different center than those of existing omni-directional images. Additionally, the method and apparatus provided may reduce a computational load necessary for processing omni-directional images.