Image processing apparatus and image processing method to increase encoding efficiency of two-dimensional image

A padding processing section replaces, by using an input image which is a two-dimensional image in which plural fisheye images configuring a predetermined azimuth range are provided side by side, an image to be used in a prediction process outside a valid region of a fisheye image with an image at a corresponding position in a valid region of another fisheye image based on continuity of the plural fisheye images to reduce prediction errors. The padding processing section calculates the position in the valid region of the other fisheye image corresponding to the position outside the valid region based on fisheye image information indicative, for each fisheye image, of a center position, a fisheye radius, or a state of reversal or rotation of the fisheye image in the two-dimensional image. A reversible encoding section includes the fisheye image information into an encoded stream of the two-dimensional image.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Phase of International Patent Application No. PCT/JP2018/045458 filed on Dec. 11, 2018, which claims priority benefit of Japanese Patent Application No. JP 2018-050903 filed in the Japan Patent Office on Mar. 19, 2018. Each of the above-referenced applications is hereby incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present technology relates to an image processing apparatus and an image processing method.

BACKGROUND ART

Conventionally, a technology is disclosed in PTL 1 in which, in an imaging system for imaging all directions of 360°, an omnidirectional image is encoded efficiently without performing such a process as interpolation for an image and without allowing a seam to stand out. Further, in PTL 2, motion prediction and compensation of a panorama video are performed utilizing a spatial correlation between a right side boundary portion and a left side boundary portion of a panorama video including all directions of 360°. Further, in NPL 1, it is described that, in the case where a fisheye image of a view angle of 180° or more in a gaze direction and another fisheye image of a view angle of 180° or more in the opposite direction to the gaze direction are acquired to obtain an omnidirectional still image, an image process such as splicing of the fisheye images is performed and a resulting image is converted, for example, into an equirectangular cylinder image and then encoded and recorded. Further, it is described that, in the case where an omnidirectional moving image is to be obtained, splicing, a conversion process into an equirectangular image and so forth are not performed in a system but plural acquired fisheye images are arranged side by side on a two-dimensional plane and then an image in which the fisheye images are arranged side by side is encoded and recorded by a conventional encoding technology.

CITATION LIST

Patent Literature

Non Patent Literature

SUMMARY

Technical Problems

Incidentally, a plurality of fisheye images is not used in PTL 1 and PTL 2. Further, if circular fisheye images are arranged side by side on a two-dimensional plane as in NPL 1, then an invalid region exists outside the circumference. Therefore, in the case of intra prediction in the proximity of a circumference, since a prediction image is produced from the invalid region, there is the possibility that the encoding efficiency in the proximity of the circumference may degrade. Also in inter prediction, since a region capable of being used for search for a motion vector is limited in the proximity of a circumference of the fisheye image, the encoding efficiency decreases.

Therefore, the present technology provides an image processing apparatus and an image processing method in which the encoding efficiency of a two-dimensional image in which plural fisheye images configuring a predetermined azimuth range are provided and arranged side by side can be increased.

Solution to Problems

The first aspect of the present technology resides in

an image processing apparatus including

a padding processing section configured to perform, by using a two-dimensional image in which plural fisheye images configuring a predetermined azimuth range are provided side by side, a padding process of replacing an image at a position outside a valid region of a fisheye image with an image at a corresponding position in a valid region of another fisheye image on the basis of continuity of the plural fisheye images.

In the present technology, the padding processing section performs, by using a two-dimensional image in which plural fisheye images configuring a predetermined azimuth range, for example, 360 degrees in all directions are provided side by side, a padding process of replacing, for example, an image at a position to be used in a prediction process outside a valid region of a fisheye image with an image at a corresponding position in a valid region of another fisheye image on the basis of the continuity of the plural fisheye images. The valid region of the fisheye image is a region within a fisheye radius, and the padding processing section calculates a position within a valid region of another fisheye image corresponding to a position outside the valid region according to a fisheye radius, an arrangement state of the plurality of fisheye images, or reversal or rotation for each fisheye image. Further, in the case where each of a plurality of fisheye images includes the corresponding position in the valid region thereof, the padding processing section performs replacement using an image of a fisheye image in which a distance from a fisheye center of the fisheye image that includes the corresponding position in the valid region thereof to a position outside the valid region for which the padding process is to be performed is shortest.

The image processing apparatus further includes an encoding section configured to perform a prediction encoding process of the two-dimensional image to generate an encoded stream, and the encoding section performs the prediction encoding process using the two-dimensional image for which the padding process has been performed. For example, the encoding section performs a quantization process and a prediction process using the two-dimensional image for which the padding process has been performed. Alternatively, the padding process before the prediction encoding process may be performed for an image that is an adjacent block of an intra prediction process outside the valid region of the fisheye image or an image that is a search range for a motion vector in an inter prediction process. Further, the encoding section may perform a prediction process using the two-dimensional image before the padding process is performed and the padding processing section may perform the padding process when an image of an adjacent block to a block for which the intra prediction process is to be performed or an image within a search range for a motion vector in a block for which the inter prediction process is to be performed is outside the valid region. The encoding section includes fisheye image information indicative, for each fisheye image, of a center position, a fisheye radius, and a state of reversal or rotation of the fisheye image in the two-dimensional image into an encoded stream.

The image processing apparatus further includes a decoding section configured to decode an encoded stream generated by performing a prediction encoding process of the two-dimensional image, and the decoding section replaces, in the case where an image to be used in the prediction process, that is, an image of an adjacent block to a block for which the intra prediction process is to be performed or an image of a reference block based on motion vector information of a block for which motion compensation is to be performed and reference frame information is at a position outside the valid region, the image to be used for the prediction process into a decoded image at a corresponding position in the valid region of the another fisheye image. The decoding section calculates the corresponding position in the valid region of the another fisheye image on the basis of fisheye image information obtained by decoding the encoded stream. Further, the decoding section replaces an image outside of the valid region of the fisheye image in the two-dimensional image after decoding with an invalid image.

The second aspect of the present technology resides in

an image processing method including

replacing in a padding processing section, by using a two-dimensional image in which plural fisheye images configuring a predetermined azimuth range are provided side by side, an image at a position outside a valid region of a fisheye image with an image at a corresponding position in a valid region of another fisheye image on the basis of continuity of the plural fisheye images.

Advantageous Effects of Invention

With the present technology, a padding process of replacing, by using a two-dimensional image in which plural fisheye images configuring a predetermined azimuth range are provided side by side, an image at a position outside a valid region of a fisheye image with an image at a corresponding position in a valid region of another fisheye image on the basis of continuity of the plural fisheye images. Accordingly, even if an image to be used in prediction is at a position outside the valid region, it is replaced with an image at a corresponding position in the valid region. Thus, the encoding efficiency of a two-dimensional image in which plural fisheye images are provided side by side can be increased. It is to be noted that the advantageous effects described in the present specification are exemplary to the end and are not restrictive, and some other additional effects may be applied.

DESCRIPTION OF EMBODIMENTS

In the following, modes for carrying out the present technology are described. If fisheye images are arranged on a two-dimensional plane, then in a rectangular region that circumscribes each fisheye image (or includes the entire fisheye image), an invalid region having no image exists outside a valid region indicative of the fisheye image. Therefore, in the present technology, a padding process of replacing an image outside a valid region with an image at a corresponding position in a valid region of another fisheye image is performed such that, in intra prediction and inter prediction, even if the position of an image to be used in the prediction is outside a valid region, an image in the valid region can be used to make it possible to perform a prediction encoding process having a high encoding efficiency even if a fisheye image is applied. It is to be noted that the description is given in the following order.1. Plural Fisheye Images and Padding Process2. Image Processing apparatus2-1. First Embodiment of Image Encoding Apparatus2-1-1. Configuration of First Embodiment26-1-2. Operation of First Embodiment2-1-3. Another Operation of First Embodiment2-1-4. Example of Operation of Intra Prediction and Inter Prediction2-2. Second Embodiment of Image Encoding Apparatus2-2-1. Configuration of Second Embodiment2-2-2. Operation of Second Embodiment2-3. Image Decoding Apparatus2-3-1. Configuration of Embodiment2-3-2. Operation of Embodiment3. Modification of Padding Process4. Application Example

1. Plural Fisheye Images and Padding Process

FIG. 1is a view illustrating a plurality of fisheye images configuring 360 degrees in all directions. For example, a virtual sphere is provided with reference to a position CP. Here, if an image sensor that uses a fisheye lens captures a fisheye image indicative of an upper side hemisphere SF1with respect to an x-y plane and another fisheye image indicative of a lower side hemisphere SF2with respect to the x-y plane, then the two fisheye images configure 360 degrees in all directions.

FIG. 2is a view illustrating continuity of two fisheye images configuring 360 degrees in all directions. A fisheye image FE1on the left side inFIG. 2is an image obtained by developing a fisheye image indicative of the upper side hemisphere SF1ofFIG. 1on a two-dimensional plane. A fisheye image FE2on the right side inFIG. 2is an image obtained by developing a fisheye image indicative of the lower side hemisphere SF2ofFIG. 1on the two-dimensional plane. It is to be noted that the fisheye image FE1and the fisheye image FE2are generated such that they are equal in fisheye radius R to each other. Further, in the present technology, the region in the fisheye radius is a valid region of the fisheye image.

If it is assumed that a position P outside the valid region of the fisheye image FE1is included in the valid region of the fisheye image FE2and the position P is a position (x1, y1) indicated by coordinates, then the position (x1, y1) corresponds to a position (x2, y2). Accordingly, an image outside the valid region of the fisheye image FE1is padded using an image at a corresponding position in the valid region of the fisheye image FE2. It is to be noted that a position outside a valid region that is a padding target is referred to also as a padding point.

Now, a procedure of a padding process is described with reference toFIG. 3. It is to be noted that, inFIG. 3and images hereinafter described in which a plurality of fisheye images is continuous, the left upper is made a reference (0, 0).

In a procedure 1, polar coordinates (r1, θ1) of the position (x1, y1) outside the valid region of the fisheye image FE1is calculated. An expression (1) indicates a calculation formula of the radius r1 of the polar coordinates, and an expression (2) indicates a calculation formula of a declination θ1.
[Math. 1]
r1=√{square root over ((x1−R)2+(y1−R)2)}  (1)
θ1=cos−1((x1−R)/r1)  (2)

In a procedure 2, on the basis of the radius r1 calculated by the procedure 1, a radius r2 of the position (x2, y2) corresponding to the position (x1, y1) is calculated. An expression (3) indicates a calculation formula of the radius r2 of the coordinates (x2, y2).
r2=2R−r1  (3)

In a procedure 3, on the basis of the declination θ1 calculated by the procedure 1, a declination θ2 of the position (x2, y2) corresponding to the position (x1, y1) is calculated. An expression (4) indicates a calculation formula of the declination θ2 of the coordinates (x2, y2).
θ2=π−θ1  (4)

In a procedure 4, the polar coordinates of the position (x2, y2) are converted into orthogonal coordinates. The diameter and the declination of the polar coordinates indicative of the position (x2, y2) have been calculated by the procedure 2 and the procedure 3. Accordingly, in the procedure 4, an expression (5) and an expression (6) are used to calculate orthogonal coordinates of the position (x2, y2).
x2=r2 cos(θ2)+3R(5)
y2=r2 sin(θ2)+R(6)

In a procedure 5, the image at the position (x1, y1) is replaced with the image at the position (x2, y2).

If such processes as described above are performed, then it is possible to perform a padding process by replacing an image outside the valid region of the fisheye image FE1with an image in the valid region of the fisheye image FE2.

2. Image Processing Apparatus

In an image processing apparatus, a prediction encoding process of a plurality of fisheye images configuring a predetermined azimuth range or a decoding process of an encoded stream generated by this prediction encoding process is performed. The image encoding apparatus that performs the prediction encoding process improves the encoding efficiency by using a padding process of replacing an image outside a valid region of a fisheye image with an image within a valid region of another fisheye image. Meanwhile, the image decoding apparatus that performs the prediction encoding process performs a decoding process of an encoded stream generated by a prediction encoding process using a padding process to generate a plurality of fisheye images configuring a predetermined azimuth range.

2-1. First Embodiment of Image Encoding Apparatus

2-1-1. Configuration of First Embodiment

FIG. 4exemplifies a configuration of a first embodiment of the image encoding apparatus. An image encoding apparatus10performs encoding of image data using a prediction process to generate an encoded stream that complies with, for example, the H265./HEVC standard.

The image encoding apparatus10includes a padding processing section21, a screen rearrangement buffer22, a calculation section23, an orthogonal transform section24, a quantization section25, a reversible encoding section26, an accumulation buffer27, and a rate controlling section28. The image encoding apparatus10further includes a dequantization section31, an inverse orthogonal transform section32, a calculation section33, a deblocking filter34, an SAO (Sample Adaptive Offset) filter35, a frame memory36, and a selection section37. Furthermore, the image encoding apparatus10includes an intra prediction section41, an inter prediction section42, and a prediction selection section43. It is to be noted that the encoding section corresponds to the screen rearrangement buffer22to the prediction selection section43.

A two-dimensional image in which plural fisheye images acquired from plural image sensors and configuring a predetermined azimuth range, for example, 360 degrees in all directions, are provided side by side is inputted as an input image to the padding processing section21. In the two-dimensional image, regions other than valid regions of the fisheye images form an invalid image (for example, a monochromatic image). Fisheye image information relating to the fisheye images provided in the two-dimensional image is inputted to the padding processing section21and the reversible encoding section26. The fisheye image information indicates, for each fisheye image, the position of the center, fisheye radius R, rotation, reversal information and so forth of the fisheye image in the two-dimensional image. The fisheye image information may further include, syntax elements (for example, information of a scale, an angle of view, a direction of a camera and so forth) indicated by “JCTVC-AB0032 Omnidirectional fisheye video SEI message” in regard to which standardization is being progressed by JCT-VC (Joint Collaboration Team—Video Coding) that is a standardization organization, position information indicative of a position at which the fisheye image is acquired and like information.

The padding processing section21uses the input image to perform, on the basis of the fisheye image information, a padding process of replacing, on the basis of continuity of a plurality of fisheye images, an image at a position outside of a valid region of a fisheye image with an image at a corresponding position in a valid region of another fisheye image. It is to be noted that a valid region is a region in a fisheye radius. The padding processing section21outputs the image after the padding process to the screen rearrangement buffer22.

The screen rearrangement buffer22stores images supplied from the padding processing section21and rearranges the stored frame images from a displaying order into an order for encoding (encoding order) according to a GOP (Group of Picture) structure. The screen rearrangement buffer22outputs image data (original image data) of the frame images having the encoding order to the calculation section23. Further, the screen rearrangement buffer22outputs the original image data to the SAO filter35, the intra prediction section41, and the inter prediction section42.

The calculation section23subtracts, for each pixel, prediction image data supplied from the intra prediction section41or the inter prediction section42through the prediction selection section43from the original image data supplied from the screen rearrangement buffer22to obtain residual data indicative of prediction residuals and outputs the residual data to the orthogonal transform section24.

For example, in the case of an image for which intra encoding is to be performed, the calculation section23subtracts the prediction image data generated by the intra prediction section41from the original image data. On the other hand, in the case of an image for which inter encoding is to be performed, the calculation section23subtracts the prediction image data generated by the inter prediction section42from the original image data.

The orthogonal transform section24performs an orthogonal transform process for residual data supplied from the calculation section23. For example, the orthogonal transform section24performs, for each of one or more TUs set in each CTU (Coding Tree Unit), orthogonal transform such as discrete cosine transform, discrete sine transform, or Karhunen-Loeve transform. The orthogonal transform section24outputs transform coefficients of a frequency region obtained by performing the orthogonal transform process to the quantization section25.

The quantization section25quantizes the transform coefficients outputted from the orthogonal transform section24. The quantization section25outputs the quantization data of the transform coefficients to the reversible encoding section26. Further, the quantization section25outputs the generated quantization data also to the dequantization section31.

The reversible encoding section26performs a reversible encoding process for the quantization data inputted from the quantization section25for each CTU, for example, a reversible encoding process of CABAC (Context-Adaptive Binary Arithmetic Coding). Further, the reversible encoding section26acquires parameters of a prediction mode selected by the prediction selection section43, for example, parameters of information indicative of an intra prediction mode or the like or parameters of information indicative of an inter prediction mode and motion vector information. Furthermore, the reversible encoding section26acquires parameters relating to a filter process from the SAO filter35hereinafter described. Further, the reversible encoding section26acquires block information indicative of in what manner a CTU, a CU, a TU, and a PU are to be set to an image. The reversible encoding section26encodes the quantization data and accumulates the acquired parameters relating to an encoding process as syntax elements of the H.265/HEVC standard and as part of header information for an encoded stream into the accumulation buffer27. Further, the reversible encoding section26includes fisheye image information inputted to the image encoding apparatus10as a syntax element of an encoded stream or as SEI (Supplemental Enhancement Information) that is additional information into the encoded stream.

The accumulation buffer27temporarily retains data supplied from the reversible encoding section26and outputs the data as an encoded image that is encoded or outputted as an encoded stream, for example, to a recording apparatus or a transmission line not depicted in a succeeding stage at a predetermined timing.

The rate controlling section28controls the rate of quantization operation of the quantization section25on the basis of compressed images accumulated in the accumulation buffer27such that no overflow or no underflow may occur.

The dequantization section31dequantizes quantization data of transform coefficients supplied from the quantization section25by a method corresponding to that of the quantization performed by the quantization section25. The dequantization section31outputs the resulting dequantization data to the inverse orthogonal transform section32.

The inverse orthogonal transform section32performs inverse orthogonal transform of dequantized data supplied thereto by a method corresponding to that of the orthogonal transform process performed by the orthogonal transform section24. The inverse orthogonal transform section32outputs a result of the inverse orthogonal transform, namely, restored residual data, to the calculation section33.

The calculation section33adds prediction image data supplied from the intra prediction section41or the inter prediction section42through the prediction selection section43to residual data supplied from the inverse orthogonal transform section32to obtain a locally decoded image (decoded image). For example, in the case where the residual data corresponds to an image for which intra encoding is to be performed, the calculation section33adds the prediction image data supplied from the intra prediction section41to the residual data. On the other hand, for example, in the case where the residual data corresponds to an image for which inter encoding is to be performed, the calculation section33adds the prediction image data supplied from the inter prediction section42to the residual data. The decoded image data that is a result of the addition is outputted to the deblocking filter34. Further, the decode image data is outputted as reference image data to the frame memory36.

The deblocking filter34suitably performs a deblocking process to remove block distortion of decoded image data. The deblocking filter34outputs a result of the filter process to the SAO filter35.

The SAO filter35performs an adaptive offset filter process (also called SAO (Sample Adaptive Offset) process) for the decoded image data after the filter by the deblocking filter34. The SAO filter35outputs an image after the SAO process to the frame memory36.

The reference image data accumulated in the frame memory36is outputted to the intra prediction section41or the inter prediction section42through the selection section37at a predetermined timing. For example, in the case of an image for which intra encoding is to be performed, reference image data for which a filter process by the deblocking filter34or the like is not performed is read out from the frame memory36and outputted to the intra prediction section41through the selection section37. On the other hand, for example, in the case where inter encoding is to be performed, reference image data for which a filter process by the deblocking filter34or the like is performed is read out from the frame memory36and outputted to the inter prediction section42through the selection section37.

The intra prediction section41executes an intra prediction process for each of one or more PUs set in each CTU on the basis of original image data and decoded image data. A PU can be divided into one or more TUs. For example, the intra prediction section41evaluates a cost function value based on a prediction error and a code amount to be generated for each of prediction mode candidates specified by H.265/HEVC. Then, the intra prediction section41selects a prediction mode by which the cost function value becomes in the minimum, namely, a prediction mode by which the compression ratio becomes highest, as an optimum prediction mode. Further, the intra prediction section41generates prediction image data in accordance with the selected optimum prediction mode. Then, the intra prediction section41outputs information relating to intra prediction including prediction mode information indicative of the optimum prediction mode, the corresponding cost function value, and prediction image data to the prediction selection section43.

The inter prediction section42executes an inter prediction process (motion detection and motion compensation) for each of one or more PUs set in each CTU on the basis of original image data and decoded image data. For example, the inter prediction section42evaluates a cost function value based on a prediction error and the code amount to be generated for each of prediction mode candidates included in the search range for a motion vector specified by H.265/HEVC. Then, the inter prediction section42selects a prediction mode that indicates the lowest cost function value, namely, a prediction mode by which the compression ratio becomes highest, as an optimum prediction mode. Further, the inter prediction section42generates prediction image data in accordance with the selected optimum prediction mode. Then, the inter prediction section42outputs the information relating inter prediction, the corresponding cost function value, and prediction image data to the prediction selection section43.

The prediction selection section43sets a prediction mode for each of CTUs or CUs on the basis comparison of cost function values inputted from the intra prediction section41and the inter prediction section42. In regard to a block for which the intra prediction mode is set, the prediction selection section43outputs prediction image data generated by the intra prediction section41to the calculation sections23and33and outputs information relating to intra prediction to the reversible encoding section26. Meanwhile, in regard to a block for which the inter prediction mode is set, the prediction selection section43outputs prediction image data generated by the inter prediction section42to the calculation sections23and33and outputs information relating to inter prediction to the reversible encoding section26.

2-1-2. Operation of First Embodiment

Now, operation of the first embodiment of the image encoding apparatus is described.FIG. 5is a flow chart exemplifying operation of the first embodiment of the image encoding apparatus.

In step ST1, the image encoding apparatus performs a padding process. The padding processing section21of the image encoding apparatus10replaces an image at a position to be used in a prediction process outside a valid region of a fisheye image with an image at a corresponding position in a valid region of another fisheye image. For example, while the padding processing section21is scanning the input image in a raster order in a unit of an encoding block (CTU), it decides whether the concerned encoding block unit includes a boundary of a fisheye image or the encoding block unit is outside the valid region of the fisheye image. In a case that is one of a case in which decision is one of that the concerned encoding block unit includes a boundary of the fisheye mage and another case in which the concerned block unit is outside the valid region of the fisheye image, the padding processing section21performs a padding process. On the other hand, in the case where the concerned encoding block unit is within the valid region and does not include a boundary of the fisheye image, the padding processing section21does not perform a padding process.

In step ST2, the image encoding apparatus performs a screen rearrangement process. The screen rearrangement buffer22of the image encoding apparatus10rearranges the input images after the padding process that are ordered in the displaying order into an encoding order and outputs a result of the rearrangement to the intra prediction section41, the inter prediction section42, and the SAO filter35.

In step ST3, the image encoding apparatus performs an intra prediction process. The intra prediction section41of the image encoding apparatus10uses reference image data read out from the frame memory36to perform intra prediction of pixels of a block of a processing target in all intra prediction modes that are candidates to generate prediction image data. Further, the intra prediction section41uses the generated prediction image data and the original image data to calculate a cost function value. It is to be noted that decoded image data that has not been subjected to a filter process by the deblocking filter34or the like is used as the reference image data. The intra prediction section41selects an optimum intra prediction mode on the basis of the calculated cost function value and outputs prediction image data generated by intra prediction of the optimum intra prediction mode, parameters and the cost function value to the prediction selection section43.

In step ST4, the image encoding apparatus performs an inter prediction process. The inter prediction section42of the image encoding apparatus10performs inter prediction of pixels of a block of a processing target in all inter prediction modes that become candidates to generate a prediction image. Further, the inter prediction section42calculates a cost function value by using the generated prediction image data and the original image data. It is to be noted that decoded image data that has been subjected to a filter process by the deblocking filter34or the like is used as the reference image data. The inter prediction section42determines an optimum inter prediction mode on the basis of the calculated cost function values and outputs a prediction image generated by the optimum inter prediction mode, parameters, and the cost function value to the prediction selection section43.

In step ST5, the image encoding apparatus performs a prediction image selection process. The prediction selection section43of the image encoding apparatus10determines one of the optimum intra prediction mode and the optimum inter prediction mode as an optimum prediction mode on the basis of the cost function values calculated in step ST2and step ST3. Then, the prediction selection section43selects and outputs the prediction image data of the determined optimum prediction mode to the calculation sections23and33. It is to be noted that the prediction image data is used in calculation in steps ST5and ST11hereinafter described. Further, the prediction selection section43outputs parameters relating to the optimum prediction mode to the reversible encoding section26.

In step ST6, the image encoding apparatus performs a difference calculation process. The calculation section23of the image encoding apparatus10calculates the difference between the original image data rearranged in step ST2and the prediction image data selected in step ST5and outputs residual data that is a result of the difference to the orthogonal transform section24.

In step ST7, the image encoding apparatus performs an orthogonal transform process. The orthogonal transform section24of the image encoding apparatus10orthogonally transforms the residual data supplied from the calculation section23. In particular, the orthogonal transform section24performs orthogonal transform such as discrete cosine transform and outputs transform coefficients obtained by the orthogonal transform to the quantization section25.

In step ST8, the image encoding apparatus performs a quantization process. The quantization section25of the image encoding apparatus10quantizes the transform coefficients supplied from the orthogonal transform section24. Upon this quantization, the rate is controlled as described in connection with a process in step ST17hereinafter described.

The quantization information generated in such a manner as described above is locally decoded in the following manner. In particular, in step ST9, the image encoding apparatus performs a dequantization process. The dequantization section31of the image encoding apparatus10dequantizes the quantization data outputted from the quantization section25with a characteristic corresponding to that of the quantization section25.

In step ST10, the image encoding apparatus performs an inverse orthogonal transform process. The inverse orthogonal transform section32of the image encoding apparatus10inversely orthogonally transforms the dequantized data generated by the dequantization section31with a characteristic corresponding to that of the orthogonal transform section24to generate residual data and outputs the residual data to the calculation section33.

In step ST11, the image encoding apparatus performs an image addition process. The calculation section33of the image encoding apparatus10adds the prediction image data outputted from the prediction selection section43to the locally decoded residual data to generate a locally decoded (namely, locally decoded) image.

In step ST12, the image encoding apparatus performs a deblocking filter process. The deblocking filter34of the image encoding apparatus10performs a deblocking filter process for the image data outputted from the calculation section33to remove block distortion and outputs resulting image data to the SAO filter35and the frame memory36.

In step ST13, the image encoding apparatus performs an SAO process. The SAO filter35of the image encoding apparatus10performs an SAO process for the image data outputted from the deblocking filter34. By the SAO process, a type and a coefficient for the SAO process are determined for each LCU that is a maximum encoding unit and a filter process is performed using them. The SAO filter35stores image data after the SAO

process into the frame memory36. Further, the SAO filter35outputs parameters relating to the SAO process to the reversible encoding section26such that they are reversibly encoded in step ST15as hereinafter described.

In step ST14, the image encoding apparatus performs a storage process. The frame memory36of the image encoding apparatus10stores the image before the filter process therefor is performed by the deblocking filter34or the like and the image after the filter process is performed by the deblocking filter34or the like.

On the other hand, the transform coefficients quantized in step ST8described hereinabove are outputted also to the reversible encoding section26. In step ST15, the image encoding apparatus performs reversible encoding. The reversible encoding section26of the image encoding apparatus10encodes the transform coefficients after the quantization or the residual data after the dequantization outputted from the quantization section25and the parameters supplied thereto.

In step ST16, the image encoding apparatus performs an accumulation process. The accumulation buffer27of the image encoding apparatus10accumulates the encoded data. The encoded data accumulated in the accumulation buffer27is suitably read out and transmitted to the decoding side through a transmission line or the like.

In step ST17, the image encoding apparatus performs rate control. The rate controlling section28of the image encoding apparatus10controls the rate of the quantization operation of the quantization section25such that the encoded data accumulated in the accumulation buffer27may not suffer from overflow or underflow.

The image encoding apparatus performs such processes as described above up to the end of images for each CTU.

According to such a first embodiment as described above, by performing a padding process, an image at a position to be used in a prediction process outside a valid region of a fisheye image is replaced with an image at a corresponding position in a valid region in another fisheye image. Accordingly, for example, in generation of a prediction image using an invalid image in an intra prediction process or in an inter prediction process, generation of a prediction image using a motion vector of low accuracy arising from that an invalid image is included in a search range for a motion vector can be prevented from being performed. Therefore, even with a fisheye image, the prediction error in an inter prediction process and an inter prediction process can be reduced similarly as in the related art. Consequently, the encoding efficiency of a fisheye image can be enhanced in comparison with that in an alternative case in which the padding process is not performed.

2-1-3. Another Operation of First Embodiment

Now, another operation of the first embodiment is described. In this operation, a padding process is performed in the case where, using an image at a position to be used in a prediction process as an image at a position to be used in a prediction process of an encoded block in which a fisheye image is included, the image at the position to be used in the prediction process is outside a valid region.

FIG. 6is a flow chart exemplifying another operation of the first embodiment and corresponds to a modification of the process in step ST1in the flow chart depicted inFIG. 5.

In step ST21, the padding processing section21decides whether the concerned block includes a boundary of a fisheye image. While the padding processing section21scans the input image in a raster order in an encoded block unit (CTC), it decides whether the concerned block includes a boundary of the fisheye image. In the case where the padding processing section21decides that the concerned block includes a boundary of the fisheye image, the processing advances to step ST22, but in the case where the padding processing section21detects that the concerned block does not include a boundary of the fisheye image, the processing advances to step ST23.

In step ST22, the padding processing section21performs a padding process for the outer side of the boundary. The padding processing section21replaces an image of the region on the outer side of the boundary with an image at a corresponding position in a valid region of another fisheye image. Then, the processing advances to step ST26.

In step ST23, the padding processing section21decides whether the concerned block is outside a valid region. In the case where the padding processing section21decides that the concerted block is outside the valid region of the fisheye image, the processing advances to step ST24, but in the case where the padding processing section21decides that the concerned block is within the valid region, the processing advances to step ST26.

In step ST24, the padding processing section21decides whether the concerted block is included in the search range. In the case where the padding processing section21decides that the concerned block is included in the search range for a motion vector, the processing advances to step ST25, but in the case where the padding processing section21decides that the concerned block is not included in the search range, the processing advances to step ST26.

In step ST25, the padding processing section21performs a padding process for the concerted block. Since the concerned block is included in the search range but outside the valid region of the fisheye image, the padding processing section21replaces the image of the concerned block with an image at a corresponding position in the valid region of the other fisheye image. Then, the processing advances to step ST26.

In step ST26, the padding processing section21decides whether the process for the blocks is completed. In the case where the padding processing section21does not complete the process of the blocks in the picture, the processing returns to step ST21, but in the case where the padding processing section21completes the process of the block of one picture, it performs the process for a next picture.

Where the padding processing section21performs such a padding process as described above, an image only in a region that is necessitated in a prediction process of an encoded block including a fisheye image is replaced with an image at a corresponding position in a valid region of another fisheye image. Accordingly, the padding process can be performed efficiently.

2-1-4. Example of Operation of Intra Prediction and Inter Prediction

FIG. 7is a view illustrating action of a padding process upon intra prediction. A case in which a block100is intra predicted is described.

If a padding process is performed, then images, for example, outside of valid regions of the block100and adjacent regions101to105, which are adjacent the block100, in the fisheye image FE1are replaced with corresponding images in valid regions of regions201to205adjacent a block200of the fisheye image FE2corresponding to the block100. Accordingly, the intra prediction section41can perform an intra prediction process using the images in the valid regions by using an image indicated by a slanting line region of the peripheral blocks of the block100.

FIG. 8is a view illustrating action of a padding process upon inter prediction. If the padding process is performed, then an image in a search range outside the valid region of the fisheye image FE1is replaced with an image at a corresponding position in the valid region of the fisheye image FE2. Accordingly, in the case where search for a motion vector is performed and the position at which the error from a block300is in the minimum is a block301, an inter prediction process can be performed using a motion vector MV1, namely, a motion vector MV2, as an optimum motion vector.

In such a manner, by using an image after a padding process, even with a fisheye image, an intra prediction process and an inter prediction process can be performed similarly as in the related art. Therefore, even if an encoding target image is a fisheye image, encoding can be performed efficiently.

2-2. Second Embodiment of Image Encoding Apparatus>

The first embodiment of the image encoding apparatus described above is directed to a case in which quantization and prediction processes are performed after a padding process is performed. However, in a second embodiment, quantization and prediction processes are performed using a two-dimensional image for which a padding process is not performed as yet, and in the case where an image to be used in the prediction process is at a position outside a valid region, the image is replaced with an image at a corresponding position in a valid region of another fisheye image.

2-2-1. Configuration of Second Embodiment

FIG. 9exemplifies a configuration of a second embodiment of the image encoding apparatus. An image encoding apparatus10aperforms encoding of image data using a prediction process to generate an encoded stream compatible, for example, with HEVC.

The image encoding apparatus10aincludes a screen rearrangement buffer22, a calculation section23, an orthogonal transform section24, a quantization section25, a reversible encoding section26, an accumulation buffer27, and a rate controlling section28. The image encoding apparatus10further includes a dequantization section31, an inverse orthogonal transform section32, a calculation section33, a deblocking filter34, an SAO (Sample Adaptive Offset) filter35, a frame memory36, and a selection section37. Furthermore, the image encoding apparatus10aincludes an intra prediction section41a, an inter prediction section42a, and a prediction selection section43.

An input image, in particular, a two-dimensional image in which a plurality of fisheye images acquired from a plurality of image sensors and, for example, configuring 360 degrees in all directions is provided, is inputted to the screen rearrangement buffer22. Further, fisheye image information inputted together with the fisheye images is inputted to the intra prediction section41a, the inter prediction section42a, and the reversible encoding section26.

The screen rearrangement buffer22stores an input image and rearranges such input images of a displaying order stored therein into an order for encoding (encoding order) according to a GOP (Group of Picture) structure. The screen rearrangement buffer22outputs the image data of the frame images in the encoding order (original image data) to the calculation section23. Further, the screen rearrangement buffer22outputs the original image data to the SAO filter35, the intra prediction section41, and the inter prediction section42.

The calculation section23subtracts, for each pixel, prediction image data supplied from the intra prediction section41or the inter prediction section42through the prediction selection section43from the original image data supplied from the screen rearrangement buffer22and outputs residual data indicative of a prediction residual to the orthogonal transform section24.

The orthogonal transform section24performs an orthogonal transform process for the residual data supplied from the calculation section23and outputs obtained transform coefficients in a frequency region to the quantization section25.

The quantization section25quantizes the transform coefficients outputted from the orthogonal transform section24. The quantization section25outputs quantization data of the transform coefficients to the reversible encoding section26. Further, the quantization section25outputs the generated quantization data also to the dequantization section31.

The reversible encoding section26performs a reversible encoding process of the quantization data inputted from the quantization section25for each CTU. Further, the reversible encoding section26acquires bock information indicative of how parameters of a prediction mode, parameters relating to a filter process, CTUs and so forth are to be set. The reversible encoding section26encodes the quantization data, encodes parameters and fisheye image information, and accumulates them as part of header information into the accumulation buffer27.

The accumulation buffer27temporarily retains the data supplied from the reversible encoding section26and outputs, at a predetermined timing, the data as an encoded image that is in an encoded state and as an encoded stream, for example, to a recording apparatus, a transmission line or the like not depicted in a succeeding stage.

The rate controlling section28controls the rate of the quantization operation of the quantization section25on the basis of the compression images accumulated in the accumulation buffer27such that an overflow or an underflow does not occur.

The dequantization section31dequantizes the quantization data of the transform coefficients supplied from the quantization section25by a method corresponding to that of the quantization performed by the quantization section25. The dequantization section31outputs obtained dequantization data to the inverse orthogonal transform section32.

The inverse orthogonal transform section32inverse orthogonal transforms the dequantization data supplied thereto by a method corresponding to that of the orthogonal transform process performed by the orthogonal transform section24. The inverse orthogonal transform section32outputs a result of the inverse orthogonal transform, namely, restored residual data, to the calculation section33.

The calculation section33adds the prediction image data supplied from the intra prediction section41or the inter prediction section42through the prediction selection section43to the residual data supplied from the inverse orthogonal transform section32to obtain a locally decoded image (decoded image) and outputs the locally decoded image to the deblocking filter34. Further, the calculation section33outputs decoded image data as reference image data to the frame memory36.

The deblocking filter34suitably performs a deblocking filter process to remove block distortion of decoded image data. The deblocking filter34outputs a result of the filter process to the SAO filter35.

The SAO filter35performs an adaptive offset filter process for the decoded image data after the filter by the deblocking filter34. The SAO filter35outputs an image after the SAO process to the frame memory36.

Reference image data accumulated in the frame memory36is outputted to the intra prediction section41aor the inter prediction section42aat a predetermined timing through the selection section37.

The intra prediction section41aexecutes an intra prediction process for each of one or more PUs set in each CTU on the basis of original image data and decoded image data. The intra prediction section41aevaluates a cost function value based on a prediction error and a code amount to be generated for each of prediction mode candidates specified by H.265/HEVC. Further, in the case where the prediction mode candidate is outside a valid region of a fisheye image, the intra prediction section41aperforms a padding process of replacing an image outside the valid region with an image at a corresponding position in a valid region of another fisheye image. The intra prediction section41aselects a prediction mode in which the cost function value is in the minimum, namely, a prediction mode in which the compression ratio is in the maximum, as an optimum prediction mode. Further, the intra prediction section41agenerates prediction image data in accordance with the selected optimum prediction mode. Then, the intra prediction section41aoutputs information relating to intra prediction including the prediction mode information indicative of the optimum prediction mode, the corresponding cost function value, and the prediction image data to the prediction selection section43.

The inter prediction section42aexecutes an inter prediction process (motion detection and motion compensation) for each of one or more PUs set in each CTU on the basis of the original image data and the decoded image data. The inter prediction section42evaluates a cost function value based on a prediction error and a code amount to be generated for each of prediction mode candidates included in a search range for a motion vector specified by H.265/HEVC. Further, in the case where the search range includes a region outside a valid region of a fisheye image, the inter prediction section42aperforms a padding process of replacing an image outside the valid region with an image at a corresponding position in a valid region of another fisheye image. The inter prediction section42aselects a prediction mode in which the cost function value is in the minimum, namely, a prediction mode in which the compression ratio is highest, as an optimum prediction mode. Further, the inter prediction section42agenerates prediction image data in accordance with the selected optimum prediction mode. Then, the inter prediction section42aoutputs information relating to inter prediction, the corresponding cost function value, and the prediction image data to the prediction selection section43.

The prediction selection section43sets a prediction mode for each CTU or for each CU on the basis of comparison of cost function values inputted from the intra prediction section41aand the inter prediction section42a. In regard to a block to which the intra prediction mode is set, the prediction selection section43outputs prediction image data generated by the intra prediction section41ato the calculation sections23and33and outputs information relating to intra prediction to the reversible encoding section26. Further, in regard to a block to which the inter prediction mode is set, the prediction selection section43outputs prediction image data generated by the inter prediction section42to the calculation sections23and33and outputs information relating to inter prediction to the reversible encoding section26.

2-2-2. Operation of Second Embodiment

Now, operation of the second embodiment of the image encoding apparatus is described.FIG. 10is a flow chart exemplifying operation of the second embodiment of the image encoding apparatus.

In step ST51, the image encoding apparatus performs a screen rearrangement process. The screen rearrangement buffer22of the image encoding apparatus10arearranges input images of a displaying order into an encoding order and outputs the rearranged input images to the intra prediction section41a, the inter prediction section42a, and the SAO filter35.

In step ST52, the image encoding apparatus performs an intra prediction process. The intra prediction section41aof the image encoding apparatus10performs generation of prediction image data using reference image data read out from the frame memory36to intra predict pixels of blocks of a processing target in all intra prediction modes that are candidates and calculates a cost function value using the generated prediction image data and the original image data. It is to be noted that, as the reference image data, decoded image data (image data encoded already) that are not subject to a filter process by the deblocking filter34and so forth is used. The intra prediction section41selects an optimum intra prediction mode on the basis of the calculated cost function values and outputs the prediction image data generated by the intra prediction of the optimum intra prediction mode, parameters and the cost function values to the prediction selection section43. The intra prediction section41selects an optimum intra prediction mode on the basis of the calculated cost function values and outputs the prediction image data generated by the intra prediction of the optimum intra prediction mode, parameters and the cost function values to the prediction selection section43.

Further, in the case where the block of a processing target includes an image of an invalid region, the intra prediction section41aperforms a padding process of using, as an image of the invalid region, an image of a corresponding region from the valid region encoded already of another fisheye image.

FIG. 11is a flow chart exemplifying padding operation upon intra prediction. In step ST71, the intra prediction section41adecides whether the fisheye image corresponding to the invalid region is encoded already. In the case where the intra prediction section41adecides that there is an encoded image corresponding to the invalid region, the processing advances to step ST72, but in the case where the intra prediction section41adecides that there is no encoded image corresponding to the invalid region, the processing advances to step ST73.

In step ST72, the intra prediction section41aperforms a padding process. The intra prediction section41aperforms a padding process to replace the image of the invalid region with an image of the region corresponding region from the valid region encoded already of the other fisheye image. Then, the processing advances to step ST73.

In step ST73, the intra prediction section41aperforms a prediction process. The intra prediction section41aperforms a process similar to that of the related art. In particular, the intra prediction section41aperforms intra prediction in all intra prediction modes to generate prediction image data and performs selection of an optimum intra prediction mode and so forth, on the basis of cost function values calculated using the generated prediction image mode and the original image mode.

FIGS. 12A and 12Bdepict an example of a padding operation upon intra prediction. It is to be noted that a slanting line region indicates an encoded region.

As depicted inFIG. 12A, in the case where the concerned block is the block200of the fisheye image FE2, since the block100of the fisheye image FE1corresponding to the block200is in an already encoded state, an image outside a boundary of the fisheye image of the block200is replaced with an image of a valid region corresponding to the block100.

On the other hand, as depicted inFIG. 12B, in the case where the concerned block is a block101of the fisheye image FE1, a block201of the fisheye image FE2corresponding to the block101is not in an encoded state as yet, and therefore, a padding process cannot be used. Therefore, an intra prediction process is performed without performing a padding process.

In step ST53ofFIG. 10, the image encoding apparatus performs an inter prediction process. The inter prediction section42of the image encoding apparatus10performs inter prediction of pixels of the block of the processing target in all inter prediction modes to generate prediction image data and calculates a cost function value using the generated prediction image data and the original image data. It is to be noted that, as the reference image data, decoded image data for which a filter process has been performed by the deblocking filter34or the like is used. The inter prediction section42determines an optimum inter prediction mode on the basis of the calculated cost function values and outputs a prediction image generated in the optimum inter prediction mode, parameters, and the cost function value to the prediction selection section43.

Further, in the case where a block indicated by a motion vector is outside the picture frame, the inter prediction section42performs a padding process of using an image of a region corresponding to a valid region of another fisheye image.

FIG. 13is a flow chart exemplifying a padding operation upon inter prediction. In step ST81, the inter prediction section42aacquires a motion vector. The inter prediction section42aacquires a motion vector of an encoding target block, and the processing advances to step ST82.

In step ST82, the inter prediction section42adecides whether a padding process using a fisheye image is to be performed. In the case where the inter prediction section42ais to perform a padding process using a fisheye image in the case where the block indicated by the motion vector acquired in step ST81is outside the picture frame, the processing advances to step ST83, but in the case where the fisheye image is not to be used, the processing advances to step ST84.

In step ST83, the inter prediction section42aperforms a padding process using the fisheye image. The inter prediction section42aperforms a padding process using the fisheye image and uses, as the region outside the picture frame, an image of a corresponding region from the valid region encoded already of the other fisheye image. Thereafter, the processing advances to step ST85.

In step ST84, the inter prediction section42aperforms a conventional out-of-frame padding process. The intra prediction section41areplaces the image outside the frame with the invalid image, and the processing advances to step ST85.

In step ST85, the inter prediction section42aperforms a prediction process. The inter prediction section42auses the image obtained by the padding process in step ST83or step ST84as prediction image data and performs selection of an optimum inter prediction mode and so forth, on the basis of the generated prediction image data and cost function values calculated using the original image data.

FIG. 14depicts an example of padding operation upon inter prediction. When a motion vector MVa acquired in step ST81indicates outside the picture frame in regard to a concerned block300, decision of whether fisheye image information is to be used in padding is performed. Here, in the case where fisheye image information is to be used, in the padding process, an image of a reference block310indicated by the motion vector MV1from a radius and a center position of the fisheye image is replaced with an image of a corresponding block410in the valid region of the fisheye image FE2. Further, in the case where the fisheye image information is to be used, in the padding process, an image of the reference block310indicated by the motion vector MV1from the radius and the center position of the fisheye image is replaced with an image of black, gray or the like.

In step ST54ofFIG. 10, the image encoding apparatus performs a prediction image selection process. The prediction selection section43of the image encoding apparatus10determines one of the optimum intra prediction mode and the optimum inter prediction mode as an optimum prediction mode, on the basis of the cost function values calculated in step ST52and step ST53. Then, the prediction selection section43selects and outputs the prediction image data of the determined optimum prediction mode to the calculation sections23and33. It is to be noted that the pieces of prediction image data are utilized for calculation in step ST60hereinafter described. Further, the prediction selection section43outputs parameters relating to the optimum prediction mode to the reversible encoding section26.

In step ST55to step ST66, processes similar to those in step ST6to step ST17ofFIG. 5are performed.

In such a manner, according to the second embodiment of the image encoding apparatus, even if a padding process is performed in intra prediction or inter prediction, a prediction encoding process of a high encoding efficiency can be performed similarly as in the first embodiment.

Now, a configuration and operation of an image processing apparatus that performs a decoding process of an encoded stream generated by the image encoding apparatus10(10a) are described.

2-3-1. Configuration of Embodiment

FIG. 15exemplifies a configuration of an embodiment of the image decoding apparatus, and the image decoding apparatus50is an image decoding apparatus corresponding to the image encoding apparatus10(10a) depicted inFIG. 4(FIG. 9). An encoded stream generated by the image encoding apparatus10(10a) is supplied to and decoded by the image decoding apparatus50.

The image decoding apparatus50includes an accumulation buffer61, a reversible decoding section62, a dequantization section63, an inverse orthogonal transform section64, a calculation section65, a deblocking filter66, an SAO filter67, and a screen rearrangement buffer68. The image decoding apparatus50further includes a frame memory71, a selection section72, an intra prediction section73, and a motion compensation section74.

The accumulation buffer61receives and accumulates an encoded stream transmitted thereto. This encoded stream is read out at a predetermined timing and outputted to the reversible decoding section62.

The reversible decoding section62reversibly decodes the encoded stream and outputs parameters of information indicative of an intra prediction mode obtained by the reversible decoding and fisheye image information to the intra prediction section73. Further, the reversible decoding section62outputs parameters of the information indicative of the inter prediction mode and motion vector information and the fisheye image information to the motion compensation section74.

The dequantization section63dequantizes quantization data obtained by the decoding by the reversible decoding section62by a method corresponding to the quantization method of the quantization section25ofFIG. 4(FIG. 9). The dequantization section63outputs the dequantized data to the inverse orthogonal transform section64.

The inverse orthogonal transform section64performs inverse orthogonal transform by the method corresponding to the orthogonal transform method of the orthogonal transform section24ofFIG. 4(FIG. 9) and obtains and outputs decoded residual data corresponding to residual data before the orthogonal transform by the image encoding apparatus10(10a) to the calculation section65.

To the calculation section65, prediction image data is supplied from the intra prediction section73or the motion compensation section74. The calculation section65adds the decoded residual data and the prediction image data to obtain decoded image data corresponding to the original image data before the prediction image data is subtracted by the calculation section23of the image encoding apparatus10. The calculation section65outputs the decoded image data to the deblocking filter66.

The deblocking filter66performs a deblocking filter process to remove block distortion of the decoded image. The deblocking filter66outputs the image data after the filter process to the SAO filter67.

The SAO filter67performs an SAO process for the image data after the filter by the deblocking filter66. The SAO filter67performs a filter process for the image data after the filter by the deblocking filter66for each LCU by using the parameters supplied from the reversible decoding section62and outputs the resulting image data to the screen rearrangement buffer68.

The screen rearrangement buffer68performs rearrangement of images. In particular, the order of frames rearranged in an encoding order by the screen rearrangement buffer22ofFIG. 4(FIG. 9) is changed into the original order for display.

The output of the SAO filter67is further supplied to the frame memory71. The selection section72reads out image data to be used in intra prediction from the frame memory71and outputs the image data to the intra prediction section73. Further, the selection section72reads out image data to be inter processed and image data to be referred to from the frame memory71and outputs the image data to the motion compensation section74.

To the intra prediction section73, pieces of information obtained by decoding header information (prediction mode information, motion vector information, reference frame information, flags, fisheye image information, various parameters and so forth) information and so forth are supplied from the reversible decoding section62. In the case where it is decided from the information from the reversible decoding section62that the prediction mode is intra prediction, the intra prediction section73generates prediction image data from the image data acquired from the frame memory71, on the basis of the prediction mode information and so forth, and outputs the prediction image data to the calculation section65. Further, in the case where, in generation of prediction image data, an image of an adjacent block to a block for which an intra prediction process is to be performed is at a position outside the valid region, the intra prediction section73performs a padding process on the basis of the fisheye image information to replace an image of the adjacent block with an image decoded at a corresponding position in the valid region of the other reference image.

To the motion compensation section74, information obtained by decoding header information is supplied from the reversible decoding section62. In the case where it is indicated by the information from the reversible decoding section62that the prediction mode is inter prediction, the motion compensation section74generates prediction image data from the image data acquired from the frame memory71, on the basis of the prediction mode information, motion vector information, reference frame information and so forth, and outputs the prediction image data to the calculation section65. On the other hand, in the case where the image of the reference block based on the motion vector block information of a block for which motion compensation is to be performed in generation of prediction image data and the reference frame information is a position outside the valid region, the intra prediction section73performs a padding process on the basis of the fisheye image information to replace the image of the reference block with an image decoded already at a corresponding position in the valid region of the other fisheye image.

2-3-2. Operation of Embodiment

Now, operation of the embodiment of the image decoding apparatus is described.FIG. 16is a flow chart exemplifying operation of the image decoding apparatus.

After the decoding process is started, the image decoding apparatus performs an accumulation process in step ST101. The accumulation buffer61of the image decoding apparatus50receives and accumulates an encoded stream transmitted thereto.

In step ST102, the image decoding apparatus performs a reversible decoding process. The reversible decoding section62of the image decoding apparatus50decodes an encoded stream supplied from the accumulation buffer61. Prior to decoding of a picture, the reversible decoding section62decodes parameter information such as motion vector information, reference frame information, prediction mode information (intra prediction mode or inter prediction mode), and fisheye image information. In the case where the prediction mode information is intra prediction mode information, the prediction mode information and the fisheye image information are supplied to the intra prediction section73. In the case where the prediction mode information is inter prediction mode information, motion vector information and so forth corresponding to the prediction mode information and the fisheye image information are outputted to the motion compensation section74.

In step ST103, the image decoding apparatus performs a prediction image generation process. The intra prediction section73or the motion compensation section74of the image decoding apparatus50individually performs a prediction image generation process corresponding to the prediction mode information supplied from the reversible decoding section62.

In particular, in the case where intra prediction mode information is supplied from the reversible decoding section62, the intra prediction section73generates an intra prediction image of the intra prediction mode. In the case where inter prediction mode information is supplied from the reversible decoding section62, the motion compensation section74performs a motion compensation process of the inter prediction mode to generate an inter prediction image. Further, the intra prediction section73and the motion compensation section74perform a padding process on the basis of fisheye image information to generate prediction images. The prediction image generated by the intra prediction section73(intra prediction image) or the prediction image generated by the motion compensation section74(inter prediction image) is outputted to the calculation section65.

In step ST104, the image decoding apparatus performs a dequantization process. The dequantization section63of the image decoding apparatus50dequantizes the quantization data obtained by the reversible decoding section62by a method corresponding to the quantization method of the quantization section25ofFIG. 4(FIG. 9) and outputs the dequantization data to the inverse orthogonal transform section64.

In step ST105, the image decoding apparatus performs an inverse orthogonal transform process. The inverse orthogonal transform section64of the image decoding apparatus50performs inverse orthogonal transform by a method corresponding to the orthogonal transform method of the orthogonal transform section24ofFIG. 4(FIG. 9) to obtain decoded residual data corresponding to residual data before the orthogonal transform in the image encoding apparatus10(10a), and outputs the decoded residual data to the calculation section65.

In step ST106, the image decoding apparatus performs an image addition process. The calculation section65of the image decoding apparatus50adds the prediction image data supplied from the intra prediction section73or the motion compensation section74and the decoded residual data supplied from the inverse orthogonal transform section64to generate decoded image data. The calculation section65outputs the generated decoded image data to the deblocking filter66and the frame memory71.

In step ST107, the image decoding apparatus performs a deblocking filter process. The deblocking filter66of the image decoding apparatus50performs a deblocking filter process for the image outputted from the calculation section65. This removes block distortion. The decoded image from the deblocking filter66is outputted to the SAO filter67.

In step ST108, the image decoding apparatus performs an SAO process. The SAO filter67of the image decoding apparatus50performs an SAO process for the image after the filter by the deblocking filter66by using parameters relating to an SAO process supplied from the reversible decoding section62. The SAO filter67outputs the decoded image data after the SAO process to the screen rearrangement buffer68and the frame memory71.

In step ST109, the image decoding apparatus performs a storage process. The frame memory71of the image decoding apparatus50stores the decoded image data before the filter process supplied from the calculation section65and the decoded image data for which the filter process by the deblocking filter66and the SAO filter67has been performed.

In step ST110, the image decoding apparatus performs a screen rearrangement process. The screen rearrangement buffer68of the image decoding apparatus50accumulates the decoded image data supplied from the SAO filter67and outputs the accumulated decoded image data in a displaying order that is an order before the pieces of decoded image data are rearranged by the screen rearrangement buffer22of the image encoding apparatus10(10a).

It is to be noted that, in regard to a two-dimensional image indicated by decoded image data, because a padding process has been performed by the image encoding apparatus, an image is sometimes displayed at a position outside the valid region of a fisheye image. Accordingly, the image decoding apparatus may be configured such that it decides a valid region of a plurality of fisheye images provided in a two-dimensional image on the basis of fisheye image information to perform a padding process of replacing an image outside the valid region of the fisheye image with an invalid image.

In such a manner, in the image decoding apparatus, since decoding of an encoded stream generated by the image encoding apparatus10(10a) described hereinabove is performed in such a manner as described hereinabove, an input image inputted to the image encoding apparatus10(10a) can be outputted from the image decoding apparatus50. It is to be noted that the present technology can be applied not only to a case in which generation or decoding of an encoded stream compatible with the H265./HEVC standard but also to a case in which generation or decoding of an encoded stream compatible with another standard such as, for example, the standard of H.264/AVC.

3. Modification of Padding Process

In the foregoing description of the embodiment described above, the padding process is described assuming that two fisheye images have no rotation and no reversal and are fixed in terms of the continuity and the positional relation of the fisheye images. However, depending upon the configuration of the inputting section, a plurality of fisheye images is sometimes outputted as images for which a reversal process or a rotation process is performed. In this case, coordinates of a valid image to be padded are corrected from rotation information or reversal information to obtain a correct image.

FIGS. 17A, 17B, 17C, and 17Dexemplify a case in which a reversal process is performed. As depicted inFIG. 17A, for example, a fisheye image FEf by imaging of a front hemisphere SFf and a fisheye image FEb by imaging of a rear hemisphere SFb are acquired by an imaging section provided on a moving body BD. Further, an object OBa and an object OBb are provided on the left side and the right side, respectively, with respect to a route of movement of the moving body BD.

At time t1, for example, fisheye images FEf1and FEb1depicted inFIG. 17Bare acquired by the imaging section provided on the moving body BD. Thereafter, at time t2, for example, fisheye images FEf1and FEb1depicted inFIG. 17Care acquired by the imaging section provided on the moving body BD. In the fisheye image FEf1ofFIG. 17B, the object OBa and the object OBb are displayed on the left side and the right side, respectively. However, in a fisheye image FEb2ofFIG. 17C, the object OBa and the object OBb are displayed on the right side and the left side, respectively. Therefore, if a reversal process is performed such that, even if the moving body BD passes the positions of the objects OBa and OBb, the display positions of the objects OBa and OBb are not reversed, then they are displayed without the reversal of the display positions of the objects OBa and OBb as depicted inFIG. 17D. In other words, by the reversal process, an object positioned on the right side can be displayed on the right side while an object positioned on the left side can be displayed on the left side.

FIG. 18depicts a case in which one of two fisheye images is reversed horizontally. In this case, an expression (7) and an expression (8) are used in place of the expression (5) and the expression (6) indicated in regard to a case in which horizontal reversal is not performed, respectively. Accordingly, for the position (x1, y1) outside the valid region of the fisheye image FE1, a padding process is performed using an image at the position (x2, y2) included in the valid region of the fisheye image FE2.
x2=−r2 cos(θ2)+3R(7)
y2=r2 sin(θ2)+R(8)

FIG. 19exemplifies a case in which a reversal process is present in inter prediction. It is to be noted that a case in which there is no reversal process corresponds toFIG. 8.

When a padding process is performed, an image within a search range outside the valid region of the fisheye image FE1is replaced with an image at a corresponding position in the valid region of the fisheye image FE2. It is to be noted that the image at the corresponding position in the valid region of the fisheye image FE2is an image of a block401at a position when a block400for which horizontal reversal is not performed is horizontally reversed with reference to the middle position of the fisheye image FE2. Accordingly, in the case where search for a motion vector is performed and the position at which the error from the block300is in the minimum is the block301, an inter prediction process can be performed using a motion vector MV3as an optimum motion vector.

FIG. 20is a flow chart exemplifying operation of a padding process that takes a reversal process into consideration. The padding process is performed by the padding processing section21. Otherwise, the padding process may be performed by the intra prediction section41and the inter prediction section42as described hereinabove.

In step ST201, the padding processing section calculates coordinates of the fisheye image FE2. The padding processing section21calculates the coordinates the fisheye image FE2depicted inFIG. 18, on the basis of a fisheye radius R, and the processing advances to step ST202.

In step ST202, the padding processing section decides whether the fisheye image FE2is not in a reversed state. In the case where the padding processing section21decides on the basis of the fisheye image information relating to the input image that the fisheye image FE2is not in a reversed state, the processing advances to step ST208, but in the case where the padding processing section21decides that the fisheye image FE2is in a reversed state, the processing advances to step ST203.

In step ST203, the padding processing section decides whether the reversal is vertical and horizontal reversal. In the case where the padding processing section21decides on the basis of the fisheye image information that the reversal is vertical and horizontal reversal, the processing advances to step ST204, but in the case where the padding processing section21decides that the reversal is not vertical and horizontal reversal, the processing advances to step ST205.

In step ST204, the padding processing section corrects the x coordinate and the y coordinate. The padding processing section21performs coordinate correction according to the vertical and horizontal reversal with reference to the center of the fisheye image FE2for the x coordinate and the y coordinate. Then, the processing advances to step ST208.

In step ST205, the padding processing section decides the reversal is horizontal reversal. In the case where the padding processing section decides on the basis of the fisheye image information that horizontal reversal is performed, the processing advances to step ST206, but in the case where the padding processing section decides that the reversal is not horizontal reversal, namely, that the reversal is vertical reversal, the processing advances to step ST207.

In step ST206, the padding processing section corrects the x coordinate. The padding processing section21performs correction of the coordinate according to the horizontal reversal with reference to the center of the fisheye image FE2for the x coordinate, and then the processing advances to step ST208.

In step ST207, the padding processing section corrects the y coordinate. The padding processing section21performs correction of the coordinate according to the vertical reversal with reference to the center of the fisheye image FE2for the y coordinate, and the processing advances to step ST208.

In step ST208, the padding processing section performs a padding process using the image in the valid region. The padding processing section21replaces, on the basis, in the case where reversal is not performed, of a coordinate or coordinates that have not been corrected, but on the basis, in the case where reversal is performed, of a coordinate or coordinates after correction, an image outside the valid region in the fisheye image FE1with an image at a corresponding position in the valid region of the fisheye image FE2. It is to be noted that, althoughFIG. 18toFIG. 20illustrate a case in which reversal of a fisheye image has been performed, also in the case where a fisheye image is in a rotated state, it is sufficient if the calculation expressions of the coordinates are changed according to rotation of a fisheye image similarly.

Now, a case in which fisheye images are different in arrangement is described as a modification of the padding process.

FIGS. 21A and 21Bdepict a case in which fisheye images are arranged side by side continuously in the vertical direction. Here,FIG. 21Adepicts a relation between adjacent regions, and adjacent blocks in upper side four blocks and lower side four blocks are denoted by like reference characters A to D.FIG. 21Bdepicts the position of a valid region of a second fisheye image corresponding to an invalid region of a first fisheye image. In the case where the fisheye images are continuous in the vertical direction, a padding process can be performed by changing the expression (5) to expression (7) given hereinabove for the case where they are arranged side by side horizontally to an expression (9) to an expression (11), respectively.
θ2=−θ1  (9)
x2=r2 cos(θ2)+R(10)
y2=r2 sin(θ2)+3R(11)

FIG. 22depicts a case in which three fisheye images are arranged side by side continuously in the horizontal direction. In this case, if an offset in the horizontal direction is added to the padding process of the fisheye image FE1and the fisheye image FE2, then expansion to a padding process of the fisheye image FE2and a fisheye image FE3can be achieved. In other words, as an image outside the valid region of the fisheye image FE2, an image at a corresponding position in the valid region of the fisheye image FE3can be used.

FIG. 23depicts a case in which four fisheye images are arranged side by side. In the case where four fisheye images are concerned, image choices for valid regions that can be utilized in a padding process increase. A position outside a valid region of a fisheye image sometimes corresponds to a position of a valid region of a plurality of other fisheye images. For example, a position P1(coordinates (x1, y1)) outside the valid region of the fisheye image FE1corresponds to a position P2(coordinates (x2, y2)) of the valid region of the fisheye image FE2and a position P3(coordinates (x3, y3)) of the valid region of the fisheye image FE3. In the case where a position outside the valid region of the fisheye image FE1corresponds to positions that are present in a plurality of other fisheye images in such a manner, the padding processing section21adds a process for determining the image in which one of the fisheye images is to be used.

For a determination method of a fisheye image to be used in the padding process, for example, the distance from the center of the fisheye image is used. According to the method of using the distance from the center of a fisheye image, the distance from a padding target position outside the valid region of a fisheye image to the center of a fisheye image in which a position corresponding to the padding target position exists in the valid region thereof is calculated for each of the other fisheye images, and the image of the position corresponding to the valid region in the fisheye image having the shortest distance is used.

FIG. 24is a flow chart exemplifying operation of a padding process that uses a distance from the center of a fisheye image and illustrates operation in the case where fisheye images in which a position corresponding to a padding target position exists in a valid region thereof are fisheye images FE2, FE3, and FE4.

In step ST301, the padding processing section calculates the distance from a padding target position to the center of the fisheye image. The padding processing section21calculates the distance from the padding target position outside the valid region in the fisheye image FE1to the center of each of the other fisheye images FE2, FE3, and FE4, and then, the processing advances to step ST302.

In step ST302, the padding processing section decides whether the center of the fisheye image FE2is nearest. In the case where the distance to the center of the fisheye image FE2is shortest on the basis of the distances calculated in step ST301, the padding processing section21decides that the center of the fisheye image FE2is nearest, and the processing advances to step ST303. On the other hand, in the case where the shortest distance is not the distance to the center of the fisheye image FE2, the processing advances to step ST304.

In step ST304, the padding processing section decides whether the center of the fisheye image FE3is nearest. In the case where the distance to the center of the fisheye image FE3is shortest on the basis of the distances calculated in step ST301, the padding processing section21decides that the center of the fisheye image FE3is nearest, and the processing advances to step ST305. On the other hand, in the case where the shortest distance is not the distances to the centers of the fisheye images FE2and FE3, the processing advances to step ST306.

In step ST307, the padding processing section decides whether the padding process is to be ended. In the case where the padding processing section21has replaced the images at all padding target positions with images in the valid region of the other fisheye image, it ends the padding process. On the other hand, in the case where there remains a padding target position for which the padding process has not been performed, the padding processing section21sets a new padding target position, and the processing returns to step ST301.

If such processes as described above are performed, even if a plurality of fisheye images includes a position corresponding to the padding target position in the valid region thereof, a fisheye image to be used for the padding process can be selected.

It is to be noted that, although the padding process described above exemplifies a case in which an image outside the valid region of the fisheye image FE1is replaced with an image at a corresponding position of a valid region of another fisheye image, also in the case where an image different from the fisheye image FE1, for example, an image outside the valid region of the fisheye image FE2, is to be replaced with an image at a corresponding position in the valid region of another fisheye image, it is sufficient if a padding process is performed similarly to the image outside the valid region of the fisheye image FE1.

Further, while the padding process described above exemplifies a case in which a plurality of fisheye images configuring 360 degrees in all directions is used, it is sufficient if a plurality of fisheye images configures a predetermined azimuth range, and this is not limited to 360 degrees in all directions.

4. Application Examples

Now, application examples of the image processing apparatus of the present technology are described. The image processing apparatus of the present technology can be applied, for example, to an imaging apparatus that captures a moving image using a fisheye lens. In this case, if the image encoding apparatus10(10a) is provided in the imaging apparatus, then the encoding efficiency of a fisheye image is enhanced to allow the imaging apparatus to record an encoded stream on a recording medium or output an encoded stream to external equipment. On the other hand, if the image decoding apparatus50is provided in an imaging apparatus, then the imaging apparatus can decode an encoded stream to perform recording and reproduction of a fisheye image. Further, if an imaging apparatus that includes the image encoding apparatus10(10a) is incorporated in a moving body of any of types such as, for example, an automobile, an electric automobile, a hybrid electric automobile, a motorcycle, a bicycle, a personal mobility, an airplane, a drone, a ship, a robot, a construction machine, and an agricultural machine (a tractor and so forth), then it becomes possible to record or transmit an image of the entire surroundings of the moving body efficiently to external equipment. Furthermore, if the image processing apparatus of the present technology is provided in a portable electronic apparatus having a function of capturing a moving image using a fisheye lens, then when a fisheye image is to be recorded on a recording medium, the data amount can be reduced in comparison with that in an alternative case in which a padding process is not performed.

Further, if the image processing apparatus that performs a decoding process is provided in an image reproduction apparatus, then it becomes possible to display an omnidirectional image on a head-mounted display or the like.

The series of processes described in the specification can be executed by hardware, software, or a composite configuration of them. In the case where the processes by software are to be executed, a program in which the processing sequence is described is installed into a memory in and executed by a computer incorporated in hardware for exclusive use. Alternatively, the program can be installed into and executed by a computer for universal use that can execute various processes.

For example, the program can be recorded into a hard disk, an SSD (Solid State Drive), or a ROM (Read Only Memory) as a recording medium in advance. Alternatively, the program can be stored (recorded) temporarily or non-transitorily in a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disk, a DVD (Digital Versatile Disc), a BD (Blu-Ray Disc (registered trademark)), a magnetic disk, or a semiconductor memory card. Such a removable recording medium as just described can be provided as what is generally called package software.

Further, the program may be installed into a computer from a removable recording medium or may be transferred to a computer by wireless or wired transfer from a download site through a network such as a LAN (Local Area Network) or the Internet. The computer can receive the program transferred in such a manner and install the program into a recording medium such as a hard disk built therein.

It is to be noted that the effects described in the present specification are exemplary to the last and are not restrictive and additional effects that are not described may be applicable. Further, the present technology shall not be interpreted restrictively to the embodiments of the technology described above. The embodiments of the present technology disclose the present technology by way of exemplification and it is obvious that those skilled in the art can perform modification of and substitution in the embodiments without departing from the subject matter of the present technology. In other words, in order to decide the subject matter of the present technology, the claims should be taken into consideration.

It is to be noted that the image processing apparatus of the present technology can also take such a configuration as described below.

An image processing apparatus, including:

a padding processing section configured to perform, by using a two-dimensional image in which plural fisheye images configuring a predetermined azimuth range are provided side by side, a padding process of replacing an image at a position outside a valid region of a fisheye image with an image at a corresponding position in a valid region of another fisheye image on the basis of continuity of the plural fisheye images.

The image processing apparatus according to (1), in which the padding processing section replaces an image at a position to be used in a prediction process outside the valid region of the fisheye image with an image at the corresponding position.

The image processing apparatus according to (1) or (2), in which

the valid region of the fisheye image is a region within a fisheye radius.

The image processing apparatus according to (3), in which

the padding processing section calculates the position in the valid region of the another fisheye image corresponding to the position outside the valid region according to the fisheye radius and an arrangement state of the plural fisheye images. (5)

The image processing apparatus according to (3) or (4), in which

the padding processing section calculates the position in the valid region of the another fisheye image corresponding to the position outside the valid region according to reversal or rotation for each fisheye image.

The image processing apparatus according to any one of (3) to (5), in which,

in a case where each of a plurality of fisheye images includes the corresponding position in the valid region of the fisheye images, the padding processing section performs replacement using an image of a fisheye image in which a distance from a fisheye center of the fisheye image that includes the corresponding position in the valid region of the fisheye image to the position outside the valid region for which the padding process is to be performed is shortest.

an encoding section configured to perform a prediction encoding process of the two-dimensional image to generate an encoded stream, in which

the encoding section performs the prediction encoding process using an image for which the padding process has been performed.

The image processing apparatus according to (7), in which

the encoding section performs a quantization process and a prediction process using the two-dimensional image for which the padding process has been performed.

The image processing apparatus according to (8), in which

the encoding section performs the padding process for an image that is an adjacent block of an intra prediction process outside the valid region of the fisheye image or an image that is a search range for a motion vector in an inter prediction process.

The image processing apparatus according to (7), in which the encoding section performs a prediction process using the two-dimensional image before the padding process is performed, and

the padding processing section performs the padding process when an image of an adjacent block to a block for which the intra prediction process is to be performed or an image of a search range for a motion vector in a block for which the inter prediction process is to be performed is outside the valid region.

The image processing apparatus according to any one of (7) to (10), in which

the encoding section includes fisheye image information relating to the fisheye images provided in the two-dimensional image into the encoded stream.

The image processing apparatus according to (11), in which

the fisheye image information includes information indicative, for each of the fisheye images, of a center position of each of the fisheye images in the two-dimensional image, a fisheye radius, and a state of reversal or rotation for each fisheye image.

a decoding section configured to decode an encoded stream generated by performing a prediction encoding process of the two-dimensional image, in which

the decoding section replaces, where an image to be used in the prediction process is at a position outside the valid region, the image to be used in the prediction process with an image after decoding at a corresponding position in the valid region of the another fisheye image.

The image processing apparatus according to (13), in which

the image to be used in the prediction process includes an image of an adjacent block to a block for which the intra prediction process is to be performed or an image of a reference block based on motion vector information of a block for which motion compensation is to be performed and reference frame information.

The image processing apparatus according to (13) or (14), in which

the encoded stream includes fisheye image information relating to the fisheye images provided in the two-dimensional image, and

the decoding section calculates the corresponding position in the valid region of the another fisheye image on the basis of the fisheye image information obtained by decoding the encoded stream.

The image processing apparatus according to any one of (13) to (15), in which

the decoding section replaces an image outside the valid region of the fisheye images in the two-dimensional image after decoding with an invalid image.

INDUSTRIAL APPLICABILITY

According to the image processing apparatus and the image processing method of the present technology, a two-dimensional image in which plural fisheye images configuring a predetermined azimuth range are provided side by side is used to perform a padding process of replacing, on the basis of the continuity of the plural fisheye images, an image at a position outside a valid region of a fisheye image with an image at a corresponding position in a valid region of another fisheye image. Therefore, even if an image to be used in prediction is a position outside the valid region, it is replaced with an image at a corresponding position in the valid region. Thus, the encoding efficiency of a two-dimensional image in which plural fisheye images are provided side by side can be increased. Accordingly, the image processing apparatus and the image processing method are suitable for electronic equipment, a mobile apparatus and so forth that perform recording, reproduction, transmission or the like of fisheye images.

REFERENCE SIGNS LIST