Patent Description:
In order to meet the demand for wireless data traffic soaring since the <NUM> communication system came to the market, there are ongoing efforts to develop enhanced <NUM> communication systems or pre-<NUM> communication systems. For the reasons, the <NUM> communication system or pre-<NUM> communication system is called the beyond <NUM> network communication system or post LTE system.

For higher data transmit rates, <NUM> communication systems are considered to be implemented on ultra high frequency bands (mmWave), such as, e.g., <NUM>. To mitigate pathloss on the ultra high frequency band and increase the reach of radio waves, the following techniques are taken into account for the <NUM> communication system: beamforming, massive multi-input multi-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beamforming, and large scale antenna.

Also being developed are various technologies for the <NUM> communication system to have an enhanced network, such as evolved or advanced small cell, cloud radio access network (cloud RAN), ultra-dense network, device-to-device (D2D) communication, wireless backhaul, moving network, cooperative communication, coordinated multi-point (CoMP), and reception interference cancellation.

There are also other various schemes under development for the <NUM> system including, e.g., hybrid FSK and QAM modulation (FQAM) and sliding window superposition coding (SWSC), which are advanced coding modulation (ACM) schemes, and filter bank multi-carrier (FBMC), non-orthogonal multiple access (NOMA) and sparse code multiple access (SCMA), which are advanced access schemes.

Meanwhile, the Internet is evolving from the human-centered connection network by which humans create and consume information to the Internet of Things (IoT) network by which information is communicated and processed between things or other distributed components. The Internet of Everything (IoE) technology may be an example of a combination of the Big data processing technology and the IoT technology through, e.g., a connection with a cloud server.

To implement the IoT, technology elements, such as a sensing technology, wired/wireless communication and network infra, service interface technology, and a security technology, are required. There is a recent ongoing research for inter-object connection technologies, such as the sensor network, Machine-to-Machine (M2M), or the Machine-Type Communication (MTC).

In the IoT environment may be offered intelligent Internet Technology (IT) services that collect and analyze the data generated by the things connected with one another to create human life a new value. The IoT may have various applications, such as the smart home, smart building, smart city, smart car or connected car, smart grid, health-care, or smart appliance industry, or state-of-art medical services, through conversion or integration of existing IT technologies and various industries.

Thus, there are various ongoing efforts to apply the <NUM> communication system to the IoT network. For example, the sensor network, machine-to-machine (M2M), machine type communication (MTC), or other <NUM> techniques are implemented by schemes, such as beamforming, multi-input multi-output (MIMO), and array antenna schemes. The above-mentioned application of the cloud radio access network as a Big data processing technique may be said to be an example of the convergence of the <NUM> and IoT technologies.

Meanwhile, a <NUM>-degree image processing device for virtual reality (VR) may generate a 3D image by mapping <NUM>-degree multi-view images captured by multiple cameras to a surface, such as a three-dimensional (3D) model, e.g., a sphere, a cube, or a cylinder, and a display device (HMD) may render a scene corresponding to a particular view of the generated 3D image and displays the rendered scene to the user. At this time, required to transmit <NUM>-degree image content to a user who is located far away are a device capable of compressing, storing, and transmitting the <NUM>-degree image content and a format for transmission, but no such device and transmission format exist currently.

Thus, a need exists for research on devices capable of compressing, storing, and transmitting <NUM>-degree image content and transmission formats.

<CIT>, is about a method for representing an omni directional-like image by capturing an omni-directional-like image and determining a representation format that is commonly applicable to any image acquisition system that is able to capture the omnidirectional-like image. The captured omni-directional-like image is mapped onto the determined representation format.

<CIT>, is about a method for mapping a panoramic image to a 3D virtual object of which a projection is made for display on a screen.

<CIT>, is about system and method for digitally rendering omnidirectional images includes capturing images of a region <NUM> degrees about an origin point in a single plane, and assembling the images in a digital format to create a complete spherical image surrounding the origin point. The spherical image is projected onto faces of a cube surrounding the spherical image.

<NPL>, is a review on map projection distortion.

<NPL>, is a paper describing algorithms for creating, storing and viewing high-resolution immersive surround videos.

<NPL>, proposes a technique of region adaptive video smoothing that exploits characteristics of equirectangular mapping to provide bitrate savings.

<NPL>, is a paper describing an approach to creating full view panoramic mosaics from image sequences.

According to an embodiment of the present disclosure, there are proposed a method and apparatus for processing <NUM>-degree images.

According to an embodiment of the present disclosure, there are proposed a method and apparatus for converting <NUM>-degree images into 2D images.

According to an embodiment of the present disclosure, there are proposed a method and apparatus for creating metadata for parameters considered upon converting <NUM>-degree images into 2D images.

According to an embodiment of the present disclosure, there are proposed a method and apparatus for restoring 2D images into <NUM>-degree images based on parameters considered upon converting the <NUM>-degree images into the 2D images.

Other aspects, advantages, and core features of the present disclosure will be apparent to one of ordinary skill in the art from the following detailed description of exemplary embodiments of the present disclosure, taken into conjunction with the drawings.

An embodiment of the present disclosure has the effect of processing <NUM>-degree images.

According to an embodiment of the presents the effect of converting <NUM>-degree images into 2D images.

According to an embodiment of the present disclosure provides the effect of creating metadata for parameters considered upon converting <NUM>-degree images into 2D images.

According to an embodiment of the present disclosure gives the effect of restoring 2D images into <NUM>-degree images based on parameters considered upon converting the <NUM>-degree images into the 2D images.

Particular preferred embodiments of the present disclosure and the foregoing and other aspects, features, and advantages will be apparent from the following detailed description taken in conjunction with the accompanying drawings, wherein:.

It should be noted that the same or similar reference denotations may be used to refer to the same or similar elements, features, or structures throughout the drawings.

Hereinafter, embodiments of the present disclosure are described in detail with reference to the accompanying drawings.

According to various embodiments of the present disclosure, the electronic devices include virtual reality (VR) devices (e.g., <NUM>-degree image cameras, head-mounted devices (HMDs), or smart glasses), smartphones, tablet personal computers (PCs), mobile phones, video phones, electronic book readers, desktop PCs, laptop PCs, netbook PCs, personal digital assistants (PDAs), portable multimedia players (PMPs), MP3 players, mobile medical devices, cameras, wearable devices (e.g., HMDs), electronic clothing, electronic bracelets, electronic necklaces, electronic appcessories, electronic tattoos, or smartwatches.

Further, according to an embodiment of the present disclosure, the electronic device may include, e.g., a smart home device, e.g., a television, a digital video disk (DVD) player, an audio player, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave oven, a washer, a drier, an air cleaner, a set-top box, a TV box, a gaming console, an electronic dictionary, a camcorder, or an electronic picture frame.

Further, according to an embodiment of the present disclosure, the electronic device includes, e.g., a medical device, a navigation device, a global positioning system (GPS) receiver, an event data recorder (EDR), a flight data recorder (FDR), an automotive infotainment device, a sailing electronic device, an aviation electronic device, a security device, or an industrial or home robot.

According to various embodiments of the present disclosure, an electronic device may be a combination of the above-listed devices. It should be appreciated by one of ordinary skill in the art that the electronic device is not limited to the above-described devices.

<FIG> is a block diagram illustrating components of an image processing device processing a <NUM>-degree image.

Referring to <FIG>, the image processing device illustrated includes a mapper <NUM>, a converter <NUM>, an encoder <NUM>, a storage unit <NUM>, a decoder <NUM>, an inverse-converter <NUM>, and a playback unit <NUM>.

A <NUM>-degree multi-view image <NUM> captured by multiple cameras is input to the mapper <NUM>, and the mapper <NUM> maps the <NUM>-degree multi-view image <NUM> to a 3D model, e.g., the surface of a sphere, generating and outputting a 3D image <NUM>.

The 3D image <NUM> is input to the converter <NUM>, and the converter <NUM> converts the input 3D image <NUM> into a 2D image <NUM> and outputs the 2D image. The 2D image <NUM> is input to the encoder <NUM>, and the encoder <NUM> encodes the input 2D image <NUM> using a predetermined encoding scheme and outputs the encoded image. The encoded 2D image is stored in the storage unit <NUM>.

The 2D image output from the storage unit is input to the decoder <NUM>, and the decoder <NUM> decodes the input image using a predetermined decoding scheme and outputs the decoded image. The decoded 2D image is input to the inverse-converter <NUM>, and the inverse-converter <NUM> inverse-converts the input 2D image into a 3D image and outputs the 3D image.

The inverse-converted 3D image <NUM> is input to the playback unit <NUM>, and the playback unit <NUM> displays the input image.

The above-described <NUM>-degree multi-view image may be at least one input image containing an image for each direction. Further, the above-described 3D image may be any one of a <NUM>-degree video, an omnidirectional video, or omnidirectional media. The omnidirectional media is a head-mounted device (HMD). The 2D image may be a projected frame or a packed frame.

Specifically, projection means one set of input images being projected as the projected frame. The projected frame means a frame specified by a <NUM> video projection format indicator. The packed frame means a frame that originates from a region-wise packing of the projected frame. In a stereoscopic <NUM>-degree video, input images at one time instance are stitched, generating projected frames representing two views, one for each eye. The two views are mapped to the packed frames that are the same.

As such, the image processing device shown in <FIG> generates the 3D image <NUM> based on the input <NUM>-degree multi-view image <NUM>, then converting into the 2D image <NUM> and storing the 2D image <NUM>, then inverse-converting into the 3D image <NUM> and transmitting the 3D image <NUM>. The operations of the image processing device may include a generating step in which the mapper <NUM> generates the 3D image <NUM> based on the <NUM>-degree multi-view image <NUM>, a storing/transmitting step in which the converter <NUM>, the encoder <NUM>, and the storage unit <NUM> convert the 3D image <NUM> into the 2D image <NUM> and store and transmit the 2D image, and a presenting/playing step in which the playback unit <NUM> inverse-converts the 2D image <NUM> into the 3D image <NUM> and displays the 3D image <NUM>.

<FIG> are views illustrating examples of conversion/inverse-conversion between a 2D image and a 3D image.

<FIG> illustrates an example of mapping a 2D image to the surface of a sphere which is a 3D model. Assuming that the coordinates of the 2D image are (s,t), the 3D image coordinates on the surface of the sphere may be expressed as x=r sin(πt) sin(2πs), y=r sin(nt) cos(2πs), z=r cos(πt), and conversion/inverse-conversion between the 2D image and the 3D image may be performed via the 2D image coordinates and the 3D image coordinates.

<FIG> illustrates an example of mapping a 2D image to the surface of a cylinder which is a 3D model. Assuming that the coordinates of the 2D image are (s,t), the 3D image coordinates on the surface of the cylinder may be expressed as x=r sin(2πs), y=r cos(2πs), z=t, and conversion/inverse-conversion between the 2D image and the 3D image may be performed via the 2D image coordinates and the 3D image coordinates.

<FIG> illustrates an example of mapping a 2D image to a cube which is a 3D model. The 2D image may be represented with six textures corresponding to the six faces of the cube, and each of the textures may be mapped to a respective one of the faces of the cube, thereby converting into the 3D image. The 3D image may be inverse-converted into the 2D image by demapping each of the faces of the cube to a respective one of the six textures.

Mapping the 2D image to the surface of which one of the sphere, cylinder, and cube which are 3D models may be determined by a generator that generates VR content (i.e., the 3D image), e.g., the mapper, and information related to the mapping is transmitted to the user terminal in the form of metadata.

Although the mapping schemes shown in <FIG> have their own pros and cons, they all may commonly cause image distortion during the conversion. In particular, in the case of equirectangular projection that converts <NUM>-degree spherical images into 2D images, the top, bottom, and corners of the 2D image may be distorted relatively more than the center, considerably deteriorating immersion when the top, bottom, and corners of the 2D image are viewed on the HMD device.

<FIG> is a flowchart illustrating operations by an image processing device to convert a <NUM>-degree image into a 2D image according to an embodiment of the present disclosure.

Referring to <FIG>, in step <NUM>, the image processing device determines a 3D model to which to map a <NUM>-degree image. Here, the 3D model may be, e.g., a sphere, a cylinder, a cube, a pyramid, or a polyhedron. In step <NUM>, the image processing device determines a partition size for the 3D image mapped to the 3D model selected in step <NUM>. The partition size may be denoted with, e.g., the number of horizontal pixels and the number of vertical pixels, and may be, e.g., <NUM>, <NUM>, <NUM>, <NUM>, or <NUM> pixels. As another example, the partition size may be represented as an angular interval along the horizontal and vertical directions, and may be, e.g., <NUM>°, <NUM>°, <NUM>°, <NUM>°, or <NUM>°. In step <NUM>, the partition size is determined as a value to minimize the entire ratedistortion (RD) cost considering the degree of distortion of the entire image and the amount of metadata generated.

In step <NUM>, the image processing device determines a rotating angle related to each of three directions, i.e., x, y, and z axes, for the 3D image. The rotating angle may be, e.g., <NUM>, <NUM>, or <NUM> degrees. In step <NUM>, the image processing device determines an interpolation method to apply upon converting (or mapping) the 3D image to the 2D image. The interpolation method may be, e.g., nearest neighbor interpolation, bilinear interpolation, or b-spline interpolation. In step <NUM>, the image processing device determines whether to apply control grid interpolation upon converting the 3D image into the 2D image. Here, control grid interpolation means warping into a different shape by shifting corners of regions partitioned in polygonal shape, e.g., triangle or rectangle, by a predetermined distance (dx, dy) when partitioning the 3D image according to the partition size determined in step <NUM>.

In step <NUM>, the image processing device checks whether converting the 3D image into the 2D image based on the result determined in steps <NUM> to <NUM> is the optimal way to minimize distortion at a predetermined bitrate. Where, as a result of checking in step <NUM>, converting the 3D image into the 2D image based on the result determined in steps <NUM> to <NUM> is the optimal method, the image processing device proceeds with step <NUM>, converting the <NUM>-degree image into the 2D image based on the final result determined and creating metadata for information related to the conversion. Here, the conversion-related information basically includes information related to the 3D model determined in step <NUM>, information related to the partition size determined in step <NUM>, information about the rotating angle determined in step <NUM>, and information about the interpolation method determined in step <NUM>, and may additionally include the information about whether to apply control grid interpolation as determined in step <NUM>.

In step <NUM>, the image processing device stores the converted 2D image data and the metadata related to the conversion.

<FIG> and <FIG> are views illustrating an example of adaptively determining a partition size considered upon converting a 3D image into a 2D image according to an embodiment of the present disclosure.

Where a 3D image for VR exists as a <NUM>-degree spherical image, the spherical surface may be partitioned into various sizes as shown in <FIG>. Further, the <NUM>-degree spherical image may be converted into the 2D image shown in <FIG> using cartography mapping, such as equirectangular projection. At this time, the pixel value corresponding to each partitioned region of the 2D image is determined to be the same as the pixel value corresponding to each partitioned region of the spherical surface, and mapping a particular region among the partitioned regions of the 2D image to a particular region among the partitioned regions of the spherical image is determined by an algorithm used to convert the 3D image into the 2D image.

Where one pixel of the <NUM>-degree spherical image is precisely mapped to one pixel of the 2D image upon converting the 3D image into the 2D image, the pixel value of the 2D image is determined as the corresponding pixel value of the <NUM>-degree spherical image. However, where one pixel of the <NUM>-degree spherical image is mapped to a point midway between the pixels of the 2D image, the pixel value of the 2D image is determined by interpolation that is based on the value of the neighbor pixels.

For memory bandwidth and pipeline processing, mapping in units of blocks are efficient, and the partitioned regions as shown in <FIG> are mapped to the partitioned regions as shown in <FIG>. At this time, the partitioned regions of the 2D image as shown in <FIG> differ in shape depending on relevant positions, and thus, the converting method (or equation) and the degree of distortion may be varied for each of the partitioned regions of the 2D. To minimize the distortion of the whole image, thus, material contents in the image may be arranged in low-distortion ones among the partitioned regions while relatively immaterial contents, such as the background, in the image are placed in high-distortion regions. Resultantly, it is very critical to determine the partitioned region of the 2D image, to which each of the partitioned region of the 3D image is to be mapped per content, and such mapping relation may be designated by providing a guide to the mapper which generates the contents, by analyzing the image content to allow the material object to be less distorted, or by converting the 3D image into the 2D image as per each mapping position and then comparing the degrees of distortion for the entire image so that the distortion is minimized.

The mapping relation indicating which partitioned region of the 2D image each partitioned image of the 3D model surface is mapped to may be expressed with indexes. Assuming that the 3D model is a sphere, the partitioned regions of the 3D model surface which are designated with θx and θy may be mapped with indexes that are designated, starting from the upper and left position of the 2D image, in the order according to raster scanning scheme. Here, raster scanning scheme means a scanning scheme in which horizontal scan lines, constituted of pixes, are scanned one-by-one, from the top line to the bottom on the screen, while sweeping from one side of each line to the other. The mapping relation between the 3D image and the 2D image is stored in the bitstream, and at this time, the mapping relation is created as metadata that is then stored in the form of a look-up table.

The metadata indicates information about the position (θx, θy) and the region (dθx, dθy), in the surface of the sphere, a partition of the planar image, i.e., the 2D image, which has a width of w pixels along the x axis, which indicates the horizontal position, and a height of h pixels along the y axis, which indicates the vertical position, is mapped to.

Meanwhile, the size and shape of each partitioned region may adaptively be determined. <FIG> illustrates an example in which the partition size is adjusted. Typically, when the partition size is relatively large, the load of computation increases and the freedom of mapping between the respective partitions of the 3D image and the 2D image reduces, but more accurate interpolation is rendered possible. When the partition size is relatively small, the load of computation and the interpolation accuracy decrease, but the freedom of mapping between the partitions of the 3D and 2D images increases. As the partition size decreases, the freedom of mapping increases, therefore the amount of metadata increases as much. In turn, the partition size and shape are determined to minimize the entire RD cost in consideration of the amount of metadata generated and the degree of distortion in the entire image.

The partition size for the partitioned regions of the 3D model surface is created as metadata, and the metadata necessary to represent the partition size includes unsigned int partition Width, unsigned int partitionHeight, and int interpolType. Here, unsigned int partitionWidth denotes the width of the partitions according to the relevant partition size, unsigned int partitionHeight denotes the height of the partitions according to the relevant partition size, and int interpolType denotes the interpolation method. The int interpolType is defined in the form of a look-up table as shown in Table <NUM> below.

In Table <NUM>, nearest neighbor interpolation represented as 0x00 means a method in which mesh vertex of a predetermined interval are taken as interpolation points, and the value of the point closest thereto is determined as the value of the interpolation points. Bi-linear interpolation represented as 0x01 means a two-dimensional expansion to the method of linearly determining a value between two points according to the straight-line distances to the two points. B-spline interpolation represented as 0x02 means a method of obtaining a smooth function with a loworder polynomial by dividing the entire section into subsections. Besides, 0x03-0xFF means values reserved to indicate other interpolation schemes than nearest neighbor interpolation, bilinear interpolation, and B-spline interpolation.

<FIG> is a view illustrating an example of adaptively determining a rotating angle considered upon converting a 3D image into a 2D image according to another embodiment of the present disclosure.

Referring to <FIG>, where a 3D image for VR is a <NUM>-degree spherical image, the <NUM>-degree spherical image may be rotated in three directions, e.g., x, y, and z axis. <FIG> illustrates an example in which the <NUM>-degree spherical image is rotated at <NUM> degrees, <NUM> degrees, and <NUM> degrees.

The mapping relation indicating which partitioned region of the 2D image each partitioned image of the 3D model surface is mapped to is represented with indexes in the form of a look-up table in which case the amount of metadata stored in the bitstream is significantly large. To reduce the entire RD cost, the amount of metadata needs to be reduced. To reduce the amount of metadata, the mapping relation may be expressed with the index indicating the degree of rotation.

Although the <NUM>-degree spherical image is rotated in the three directions and is then partitioned into a plurality of regions that are then mapped to the 2D image, the position of the object in the raw image is substantially changed. That is, for the <NUM>-degree spherical image rotated in the three directions, the rotating angle for each rotating direction may be expressed as an index, and although only the index for the rotating angle is stored in the bitstream and is transmitted, the freedom of mapping between the partitioned regions of the 3D and 2D images increases.

The mapping relation indicating which partitioned region of the 2D image each partitioned region of the 3D model surface is mapped to may be created as metadata, and the metadata necessary to represent the mapping relation by adaptive rotation contains int angleX, int angleY, and int angleZ. Here, int angleX denotes the rotating angle for the x-axis direction, int angle Y denotes the rotating angle for the y-axis direction, and int angleZ denotes the rotating angle for the z-axis direction.

<FIG> is a view illustrating an example of adaptively determining whether to apply a control grid interpolation method considered upon converting a 3D image into a 2D image according to still another embodiment of the present disclosure.

Referring to <FIG>, where a 3D image for VR is a <NUM>-degree spherical image, and the surface of the <NUM>-degree spherical image is assumed to be partitioned into a plurality of regions, the data corresponding to the triangular or rectangular partitioned regions may be converted/inverse-converted from the 3D image to the 2D image or from the 2D image to the 3D image, while maintaining the shape. However, for more accurate mapping per partitioned region, conversion may be performed with the corners of the partitioned region shifted to the left and right by a predetermined distance (dx, dy). At this time, if the shape of the prior partitioned regions was square, the converted regions may be warped to be rectangular or trapezoidal in shape.

That is, assuming that the <NUM>-degree spherical image has a plurality of control points <NUM> and a control polygon <NUM> constituted of the plurality of control points <NUM>, if control grid interpolation is applied to the control points arrayed as denoted with reference number <NUM>, the shape of the partitioned regions may be warped as denoted with reference number <NUM>.

<FIG> is a flowchart illustrating operations by an image processing device to restore a 2D image into a <NUM>-degree image according to an embodiment of the present disclosure.

Referring to <FIG>, in step <NUM>, the image processing device parses stored metadata. Here, the metadata means data related to converting the <NUM>-degree image into the 2D image and includes at least one of, e.g., data related to the rotating angle for each of the x, y, and z axes and the partition size for the <NUM>-degree image, data related to the 3D model to which the <NUM>-degree image is mapped, data related to the interpolation scheme applied upon mapping the <NUM>-degree image to the 2D image, and data related to whether control grid interpolation applies.

In step <NUM>, the image processing device generates an image memory based on the 3D model-related data among the parsed metadata. In step <NUM>, the image processing device identifies the image data in units of the partition size indicated by the partition size-related data in the stored 2D image, based on the partition size-related data among the parsed metadata. It is assumed here that the image data in the corresponding positions are identified, starting from the upper and left region among the partitioned regions of the 2D image, in the order according to raster scanning scheme.

In step <NUM>, the image processing device maps the image data identified in step <NUM> to the 3D image surface based on the interpolation-related data among the parsed metadata. Here, the 3D model considered upon mapping to the 3D image surface follows the 3D model-related data of step <NUM>. In step <NUM>, the image processing device determines whether it is needed to apply control grid interpolation based on the data related to whether to apply control grid interpolation among the parsed metadata, and as necessary, applies control grid interpolation.

In step <NUM>, the image processing device checkes whether the region in the 3D image restored via steps <NUM> to <NUM> is the last region among the partitioned regions. If the region in the restored 3D image is the last region among the partitioned regions as a result of the check of step <NUM>, the image processing device proceeds with step <NUM>, rotating the whole 3D image data based on the data related to the rotating angle for each of the x, y, and z axes for the <NUM>-degree image and hence restoring the <NUM>-degree image.

Claim 1:
A method for encoding <NUM>-degree multi-view images, the method comprising:
- generating a three-dimensional, 3D, image (<NUM>) by mapping (<NUM>) a <NUM>-degree multi-view image to a 3D model,
- generating a 2D image (<NUM>) by projecting the 3D image (<NUM>) to a planar space with a rotated x axis, a rotated y axis and a rotated z axis according to a rotation angle for each of an x axis, a y axis, and a z axis, wherein the rotation angle is determined so that a distortion related to the 3D image (<NUM>) is minimized;
- encoding (<NUM>) the 2D image (<NUM>); and
- transmitting the encoded 2D image and metadata, the metadata including information related to the rotation angle and a projection-format indicator specifying a projection format of the 2D image for the projecting of the 3D image to the planar space.