Patent Description:
A <NUM>° omnidirectional image refers to all the scenes around an observation point in space which are composed of all the light that can be received by this observation point, and a sphere can describe all the scenes around the observation point. Since spherical images are difficult to store, and an existing image codec is designed for common non-omnidirectional images and is not ideal for the coding effect of the spherical images, it is necessary to convert the spherical images into omnidirectional image formats through projection, and the common projection formats include Equirectangular Projection (ERP), Cubemap Projection (CMP), etc. For omnidirectional images in different formats, an existing coding mode for common non-omnidirectional images such as HEVC, AVS2, AV1, etc. may be adopted for coding, and a locally adjusted coding method may also be adopted for different formats.

ERP sampling is shown in <FIG>, and points on a sphere may be expressed with longitude ϕand latitudeθ. The sphere is mapped to a rectangle with an aspect ratio of <NUM>:<NUM> in a ϕ - θ coordinate system, and then the rectangle is uniformly sampled. The ERP format has oversampling and image texture overstretching issues in the high-latitude areas.

As shown in <FIG>, a sphere is projected to six faces of a cube through a perspective projection, and then the cube is uniformly sampled. Compared with the ERP, the CMP samples the sphere more uniformly, and has higher coding efficiency. However, due to the characteristics of the perspective projection, the corners of each face of the cube have a higher sampling density than the central region. Correspondingly, for the sphere, the sampling uniformity of the sphere is insufficient.

After determining a position mapping relationship between the sphere and the omnidirectional image, the omnidirectional image may be sampled, so as to generate a pixel value for each pixel position, and then the omnidirectional image is coded and decoded. The position mapping relationship between the sphere and the omnidirectional image may also be utilized to perform format conversion of the omnidirectional image, and the ERP format may be converted to the CMP format or other formats.

In view of the defects in the prior art, the present invention provides a novel omnidirectional image processing method and device.

For the same area, in a Cubemap Projection (CMP) format, the central region of each face corresponds to a larger region on a sphere than marginal regions, which leads to non-uniform sampling on the sphere and redundant information in the marginal regions of the cube, and reduces the representation efficiency.

A main idea of the present invention is to enable the arc length of the unit interval at each latitude in an omnidirectional image to be as equal as possible on the sphere by adjusting a stretching relationship between the omnidirectional image and the sphere in different positions, thereby reducing information redundancy in the marginal regions in Cubemap projection, and improving the representation efficiency.

For this purpose, the present invention adopts the following technical solution:
A first objective of the present invention is to provide an omnidirectional image processing method, including:
decoding a code stream to obtain an omnidirectional image coding format information; and determining a mapping relationship between a position (x, y) in at least one region, which represents one face of a cube projected from a sphere, in the omnidirectional image and a position (θ, ϕ) on the sphere according to the format information, wherein the mapping relationship is:.

Preferably, the omnidirectional image format information is one of the following:.

A second objective of the present invention is to provide an omnidirectional image processing method, including the following steps:
expressing at least one image region, which represents one face of a cube projected from a sphere, in an omnidirectional image by the following mapping relationship, wherein the mapping relationship between a position (x, y) in the image region and a position (θ, ϕ) on the sphere is:.

Preferably, the omnidirectional image processing method further comprises one region <NUM> in the decoded omnidirectional image is the region described in claim <NUM>,a position (x1, y1) in a region <NUM> and their respective corresponding positions (θ<NUM>, ϕ<NUM>) on a sphere according to the format information is the position (x, y) in the region and the position (θ, ϕ) on a sphere described in claim <NUM>, in addition to region <NUM>, there is at least another adjacent region <NUM>, determining mapping relationships between a position (x2, y2) in a region <NUM> and their respective corresponding positions (θ2, ϕ2) on a sphere according to the format information, wherein the mapping relationships are:.

A third objective of the present invention is to provide an omnidirectional image processing device, including the following modules:.

A fourth objective of the present invention is to provide an omnidirectional image processing device, including the following modules:
a position mapping module, wherein output of the position mapping module is omnidirectional image format information, and a mapping relationship between a position (x, y) in at least one region, which represents one face of a cube projected from a sphere, in an omnidirectional image and a position (Θ, cp) on the sphere is determined in the module as:.

Compared with the prior art, by designing the special position mapping relationship between the sphere and the omnidirectional image, the present invention enables the arc length of the unit interval at each latitude in the omnidirectional image to be as equal as possible on the sphere, and the omnidirectional image to be distributed more uniformly on the sphere, so that the spherical uniformity of expression is improved, the sampling loss is reduced under the same sample number, and the coding efficiency is improved.

In combination with the accompanying drawings, the principle of the present invention can be explained from embodiments given below.

The accompanying drawings illustrated here are to provide further understanding of the present invention and constitute a part of this application. Various embodiments of the present invention are merely to explain the present invention and are only some particular cases, and the application range of the present invention is not limited to these embodiments. In the drawings:.

The embodiment of the present invention provides an omnidirectional image processing method.

The omnidirectional image processing method provided by the present embodiment includes the following steps:
A code stream is decoded to obtain omnidirectional image coding format information. A mapping relationship between a position (x, y) in at least one region in an omnidirectional image obtained by decoding the code stream and a position (θ, ϕ) on a sphere is determined according to the format information. The mapping relationship is: x and ϕ are in linear relationship, and y and <MAT> are in linear relationship.

x represents a first dimensional coordinate position of the region in the decoded omnidirectional image, y represents a second dimensional coordinate position of the region in the decoded omnidirectional image, ϕ represents a longitude position of the sphere, θ represents a latitude position of the sphere, a line mapped to the sphere by a first dimensional coordinate axis is an equator line, and a line mapped to the sphere by a second dimensional coordinate axis is a prime meridian.

In the present embodiment, the decoded omnidirectional image is in a Cubemap Projection (CMP) format, the region is a front face of the cube in <FIG>, and the mapping relationship between (x, y) and (θ, ϕ) is: <MAT> and <MAT>.

In this embodiment, the format information is a format number, and this format number specifies by default that a direction of a first dimensional coordinate of the region in the omnidirectional image is a horizontal direction, as shown in <FIG>.

The omnidirectional image processing method provided by the present embodiment includes the following steps:
A code stream is decoded to obtain omnidirectional image coding format information. A mapping relationship between a position (x, y) in at least one region in an omnidirectional image obtained by decoding the code stream and a position (θ, ϕ) on a sphere is determined according to the format information. The mapping relationship is:.

In the present embodiment, the decoded omnidirectional image is in a CMP format, the region is a front face of the cube in <FIG>, and the mapping relationship between (x, y) and (θ, ϕ) is: <MAT> and <MAT>.

In this embodiment, the format information includes a format number and direction information of a first dimensional coordinate of the region, and this direction information specifies that a direction of the first dimensional coordinate of the region in the omnidirectional image is a vertical direction, as shown in <FIG>.

The omnidirectional image processing method provided by the present embodiment includes the following steps:
At least one image region in an omnidirectional image is expressed by the following mapping relationship. The mapping relationship between a position (x, y) in the image region and a position (θ, ϕ) on a sphere is:.

In the present embodiment, the omnidirectional image is in a CMP format, the region is a front face of the cube in <FIG>, and the mapping relationship between (x, y) and (θ, ϕ) is: <MAT> and <MAT>.

In this embodiment, format information is a format number, and this format number specifies by default that a direction of a first dimensional coordinate of the region in the omnidirectional image is a horizontal direction, as shown in <FIG>.

The omnidirectional image expressed by the mapping relationship is coded and written to a code stream.

The format information of the mapping relationship is also written to the code stream.

The sequence of writing the omnidirectional image and the format information to the code stream may be arbitrary. That is, the omnidirectional image may be coded and written to the code stream first, and then the format information of the mapping relationship is also written to the code stream. Or, the format information of the mapping relationship may be written to the code stream first, and then the omnidirectional image expressed by the mapping relationship is coded and written to the code stream.

In the present embodiment, the omnidirectional image is in a CMP format, the region in the omnidirectional image is a front face of the cube in <FIG>, and the mapping relationship between (x, y) and (θ, ϕ) is: <MAT> and <MAT>.

In this embodiment, format information includes a format number and direction information of a first dimensional coordinate of the region, and this direction information specifies that a direction of the first dimensional coordinate of the region in the omnidirectional image is a vertical direction, as shown in <FIG>.

The embodiment of the present invention provides a method of sampling by using a mapping relationship.

The sampling method provided by the present embodiment includes the following steps:
At least one image region is included in a to-be-generated omnidirectional image. The mapping relationship between a position (x, y) in the image region and a position (θ, ϕ) on a sphere is:.

In the present embodiment, the to-be-generated omnidirectional image is in a CMP format, the region is a front face of the cube in <FIG>, and the mapping relationship between (x, y) and (θ, ϕ) is: <MAT> and <MAT>.

Pixel sampling positions in a vertical direction are shown in <FIG>, each meridian is sampled with equal arc length, but the arc lengths between adjacent sampling points on different meridians are different. Pixel sampling positions in a horizontal direction are shown in <FIG>, the equator is sampled with equal arc length, and other positions are sampled with approximately equal arc length.

During sampling, first, the equator is sampled with equal arc length in the horizontal direction, and then meridians corresponding to all the horizontal sampling positions are sampled with equal arc length in the vertical direction.

According to the pixel sampling positions in the to-be-generated omnidirectional image, each pixel point in the to-be-generated omnidirectional image is interpolated on the sphere to obtain its corresponding pixel value.

<FIG> shows an image generated according to the position mapping relationship.

The omnidirectional image processing method provided by the present embodiment includes the following steps:
A code stream is decoded to obtain omnidirectional image coding format information.

Mapping relationships between a position (x1, y1) in a region <NUM> and a position (x2, y2) in a region <NUM> and their respective corresponding positions (θ<NUM>, ϕ<NUM>) and (θ<NUM>, ϕ<NUM>) on a sphere are determined according to the format information since at least one pair of adjacent regions, i.e. the region <NUM> and the region <NUM>, is present in a decoded omnidirectional image. The mapping relationships are:.

In this embodiment, a front face and a right face in <FIG> are taken as the adjacent regions in the omnidirectional image, the front face is the region <NUM>, and the right face is the region <NUM>. The relationship between (x1, y1) and (ϕ<NUM>, θ<NUM>) is: <MAT> , and <MAT>.

The relationship between(x2, y2) and (ϕ<NUM>, θ<NUM>) is: <MAT> , and <MAT>.

In this embodiment, first dimensional directions of the region <NUM> and the region <NUM> are both horizontal directions, as shown in <FIG>.

In this embodiment, a front face and a right face in <FIG> are taken as the adjacent regions in the omnidirectional image, the front face is the region <NUM>, and the right face is the region <NUM>. The relationship between (x1, y1) and (ϕ<NUM>, θ<NUM>) is: <MAT> and <MAT>.

The relationship between(x2, y2) and (ϕ<NUM>, θ<NUM>) is: <MAT> and <MAT>.

In this embodiment, first dimensional directions of the region <NUM> and the region <NUM> are both vertical directions, as shown in <FIG>.

The embodiment of the present invention provides an omnidirectional image format conversion method.

The omnidirectional image format conversion method provided by the present embodiment includes the following steps:
At least one image region is included in a to-be-generated omnidirectional image. A mapping relationship between a position (x, y) in the image region and a position (θ, ϕ) on a sphere is:.

In this embodiment, a first dimensional direction is a horizontal direction, as shown in <FIG>.

A position on the sphere is mapped to a source format. An Equirectangular Projection (ERP) is taken as the source format, and a mapping relationship between the position (u, v) in the ERP and the position (θ, ϕ) on the sphere is: <MAT> and <MAT>.

The values of u and v are both in [<NUM>,<NUM>],.

Interpolation is performed on the ERP to generate pixel values in the to-be-generated omnidirectional image.

<FIG> shows a source format image of the ERP, and <FIG> show an omnidirectional image in an unfolded CMP format.

In this embodiment, the region in the omnidirectional image is a rear face of the cube in <FIG>. Coordinate systems shown in <FIG> are established on this face. In the X-Y coordinate system, when y=-<NUM>, <MAT>, and when y=<NUM>, <MAT>, then
the mapping relationship between (x, y) and (θ, ϕ) is: <MAT> and <MAT>.

In the X'-Y' coordinate system, when y'=-<NUM>, <MAT>, and when y'=<NUM>, <MAT>, then the mapping relationship between (x', y') and (θ, ϕ) is: <MAT> and <MAT>.

The region where y=±<NUM> is the effective content corresponding to the sphere in the omnidirectional image. Conversion between (x', y') and (x, y) is as follows: <MAT> and <MAT>.

Through the above conversion, the mapping relationship between the position (x', y') in the omnidirectional image and the position (θ, ϕ) on the sphere can be derived as: <MAT> , and <MAT>.

The embodiment of the present invention provides an omnidirectional image processing device.

The omnidirectional image processing device provided by the present embodiment includes the following modules:.

<FIG> shows a schematic diagram of this processing device.

In the present embodiment, the omnidirectional image is in a CMP format, the region is a front face of the cube in <FIG>, and the mapping relationship between (x, y) and (θ, ϕ) is: <MAT> and <MAT> In this embodiment, the format information includes a format number and direction information of a first dimensional coordinate of the region in the omnidirectional image, and this direction information specifies that a direction of the first dimensional coordinate of the region in the omnidirectional image is a vertical direction, as shown in <FIG>.

The omnidirectional image processing device provided by the present embodiment includes the following modules:
a position mapping module, wherein output of the position mapping module is omnidirectional image format information, and a mapping relationship between a position (x, y) in at least one region in an omnidirectional image and a position (θ, ϕ) on a sphere is determined in the module as:.

The embodiment of the present invention provides an omnidirectional image generation method.

The omnidirectional image generation method provided by the present embodiment includes the following steps:
At least one image region is included in a to-be-generated omnidirectional image. A mapping relationship between a position (x, y) in the image region and a position (θ, ϕ) on a sphere is specified as:.

In the present embodiment, the to-be-generated omnidirectional image is in a CMP format. As shown in <FIG>, coordinate systems are respectively established on six faces of the cube. For the faces on the first line of the CMP format, namely, the left, front and right faces, a first dimensional direction is a horizontal direction. When y=-<NUM> and x=<NUM>, <MAT>, and when y=<NUM> and x=<NUM>, <MAT>, then the mapping relationship between (x, y) and (θ, ϕ) is: <MAT> and <MAT>.

In <FIG>, the region enclosed by thick lines represents an effective region in the face of the cube, the height of this region is L', and the side length of the face of the cube is L. In the front face, the region outside the thick lines is a non-effective region, the pixels in this region are filled with the pixels in the line where y=-<NUM>, and the pixels in non-effective regions of the left and right faces are also filled with the pixels in the lines where y=-<NUM> in the corresponding regions.

For the faces on the second line of the CMP format, namely, the upper, lower and rear faces, a first dimensional direction is the horizontal direction. When y=<NUM> and x=<NUM>, <MAT>, and when y=-<NUM> and x=<NUM>, <MAT>, then the mapping relationship between (x, y) and (θ, ϕ) is: <MAT> and <MAT>.

In the rear face, the region outside the thick lines is a non-effective region, the pixels in this region are filled with the pixels in the line where y=<NUM>, and the pixels in non-effective regions of the upper and lower faces are also filled with the pixels in the lines where y=<NUM> in the corresponding regions.

The omnidirectional image generation method provided by the present embodiment includes the following steps:
At least one image region is included in a to-be-generated omnidirectional image. A mapping relationship between a position (x, y) in the image region and a position (θ, ϕ) on a sphere is:.

In <FIG>, the region enclosed by thick lines represents an effective region in the face of the cube, the height of this region is L', and the side length of the face of the cube is L. In the front face, the region outside the thick lines is a non-effective region, the pixels in this region are generated by interpolation on the sphere according to the same mapping relationship as in the effective region, and at this time, the value of y is less than-<NUM>. The pixels in non-effective regions of the left and right faces are generated by interpolation on the sphere according to the same mapping relationship as in the corresponding effective regions.

For the faces on the second line of the CMP format, namely, the upper, lower and rear faces, a first dimensional direction is a vertical direction. When x=-<NUM>, <MAT>, and when x=<NUM>, <MAT>, then the mapping relationship between (x, y) and (θ, ϕ) is: <MAT> , and <MAT>.

Claim 1:
An omnidirectional image processing method, comprising:
decoding a code stream to obtain an omnidirectional image coding format information; and determining a mapping relationship between a position (x, y) in at least one region, which represents one face of a cube projected from a sphere, in the omnidirectional image and a position (θ, ϕ) on the sphere according to the format information, wherein the mapping relationship is:
x and ϕ are in linear relationship, and y and <MAT> are in linear relationship,
where x represents a first dimensional coordinate position on a first dimensional coordinate axis of the region in the decoded omnidirectional image, y represents a second dimensional coordinate position on a second dimensional coordinate axis of the region in the decoded omnidirectional image, ϕ represents a longitude position of the sphere, θ represents a latitude position of the sphere, a line mapped to the sphere by the first dimensional coordinate axis is an equator line, and a line mapped to the sphere by the second dimensional coordinate axis is a prime meridian.