CODING APPARATUS AND METHOD, AND STORAGE MEDIUM

A coding apparatus that performs coding processing on each block of an input image comprises a coding unit that executes coding processing using an inter-prediction mode, an intra-prediction mode or a multi-reference prediction mode, an output unit that outputs a result of the coding processing as coded data; and a calculation unit that calculates a first coding cost if the block is coded in the inter-prediction mode and a second coding cost if the block is coded in the intra-prediction mode using a predetermined function. The coding unit executes the coding processing using the multi-reference prediction mode if a difference between the first coding cost and the second coding cost is equal to or less than a threshold, and using the prediction mode with a smaller one of the first coding cost and the second coding cost if the difference is larger than the threshold.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an coding apparatus and method, and storage medium, and more particularly to predictive coding technology.

Background Art

In order to efficiently transmit, record, and reproduce moving images, a moving image coding apparatus that compresses and encodes moving image data and a moving image decoding apparatus that decodes and decompresses moving image data are used. As moving image encoding methods, H.264/AVC (Advanced Video Coding) and H.265/HEVC (High Efficiency Video Coding), which utilizes hybrid coding that reduces spatial and temporal redundancy of moving image data, are mainly used currently. Further, as a moving image encoding method of next generation, International standardization of H.266/VVC (Versatile Video Coding) is in progress (see “Versatile Video Coding (Draft 10)”, JVET-S2001-vH, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 2020).

In this moving image encoding method, one screen is divided into small blocks called CU (Coding Unit), and coding/decoding processing is performed for each block. Further, in such a video coding method, a predicted image is normally generated based on a local decoded image obtained by encoding/decoding an input image, and a prediction error (sometimes called a “difference image” or “residual image”) obtained by subtracting the predicted image from the input image is encoded. Inter-prediction and intra-prediction can be selected as a method for generating a predicted image.

In inter-slicing, which of intra-prediction and inter-prediction is used in coding is generally determined by comparing the cost value (coding cost) obtained using a cost function based on the prediction error value and the code amount of the prediction error. Here, since the predicted images obtained by intra-prediction and inter-prediction are different, if blocks encoded by intra-prediction and inter-prediction are adjacent to each other in the screen of an image to be encoded, the difference between the predicted images of the blocks may be visually recognized as an image quality difference. In general, people have a visual characteristic that they tend to notice changes in flat areas, so in flat areas of an image, the difference in image quality between blocks encoded by intra-prediction and inter-prediction due to the difference in the predicted images described above is more conspicuous, and may be perceived as deterioration of image quality.

In order to solve this problem, Japanese Patent Laid-Open No. 2006-094081 proposes the following technique. That is, the cost value is corrected such that the lower the activity indicating the flatness of each block, the intra-prediction mode is less likely to be selected, thereby preventing blocks encoded by intra-prediction and inter-prediction from being adjacent to each other in a flat portion of an image.

However, the technique described in Japanese Patent Laid-Open No. 2006-094081 does not completely prevent blocks of intra-prediction and inter-prediction from being adjacent to each other. In other words, even if the technique described in Japanese Patent Laid-Open No. 2006-094081 is used, blocks of intra-prediction and inter-prediction may be adjacent to each other depending on the degree of flatness of the image. In this case, the difference in image quality between blocks of intra-prediction and inter-prediction may be perceived as deterioration in image quality.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the above problems, and aims to make image quality deterioration due to differences in predicted images inconspicuous in a case where one image is coded using different predicted images.

According to the present invention, provided is an coding apparatus that divides an input image into blocks and performs coding processing on each block of the input image, comprising: a coding unit that executes coding processing on each block using one of prediction modes of an inter-prediction mode that generates an-inter predicted image by performing inter-prediction on a block and encodes a difference between the block and the inter-predicted image, an intra-prediction mode that generates an intra-predicted image by performing intra-prediction on a block and encodes a difference between the block and the intra-predicted image, and a multi-reference prediction mode that generates a multi-predicted image by combining an inter-predicted image and an intra-predicted image generated from a block, and encodes a difference between the block and the multi-predicted image; an output unit that outputs a result of the coding processing as coded data; and a calculation unit that calculates a first coding cost in a case where the block is coded in the inter-prediction mode and a second coding cost in a case where the block is coded in the intra-prediction mode using a predetermined function, wherein the coding unit executes the coding processing on the block using the multi-reference prediction mode in a case where a difference between the first coding cost and the second coding cost is equal to or less than a threshold, and executes the coding processing on the block using the prediction mode with a smaller one of the first coding cost and the second coding cost in a case where the difference between the first coding cost and the second coding cost is larger than the threshold.

Further, according to the present invention, provided is An coding method that divides an input image into blocks and performs coding processing on each block of the input image, comprising: a step of executing coding processing on each block using one of prediction modes of an inter-prediction mode that generates an inter-predicted image by performing inter-prediction on a block and encodes a difference between the block and the inter-predicted image, an intra-prediction mode that generates an intra-predicted image by performing intra-prediction on a block and encodes a difference between the block and the intra-predicted image, and a multi-reference prediction mode that generates a multi-predicted image by synthesizing an inter-predicted image and an intra-predicted image generated from a block, and encodes a difference between the block and the multi-predicted image; a step of outputting a result of the coding processing as coded data; and a step of calculating a first coding cost in a case where the block is coded in the inter-prediction mode and a second coding cost in a case where the block is coded in the intra-prediction mode using a predetermined function, wherein the coding processing is executed on the block using the multi-reference prediction mode in a case where a difference between the first coding cost and the second coding cost is equal to or less than a threshold, and the coding processing is executed on the block using the prediction mode with a smaller one of the first coding cost and the second coding cost in a case where the difference between the first coding cost and the second coding cost is larger than the threshold.

Furthermore, according to the present invention, provided is a non-transitory computer-readable storage medium, the storage medium storing a program that is executable by the computer, wherein the program includes program code for causing the computer to function as an coding apparatus that divides an input image into blocks and performs coding processing on each block of the input image, comprising: a coding unit that executes coding processing on each block using one of prediction modes of an inter-prediction mode that generates an-inter predicted image by performing inter-prediction on a block and encodes a difference between the block and the inter-predicted image, an intra-prediction mode that generates an intra-predicted image by performing intra-prediction on a block and encodes a difference between the block and the intra-predicted image, and a multi-reference prediction mode that generates a multi-predicted image by combining an inter-predicted image and an intra-predicted image generated from a block, and encodes a difference between the block and the multi-predicted image; an output unit that outputs a result of the coding processing as coded data; and a calculation unit that calculates a first coding cost in a case where the block is coded in the inter-prediction mode and a second coding cost in a case where the block is coded in the intra-prediction mode using a predetermined function, wherein the coding unit executes the coding processing on the block using the multi-reference prediction mode in a case where a difference between the first coding cost and the second coding cost is equal to or less than a threshold, and executes the coding processing on the block using the prediction mode with a smaller one of the first coding cost and the second coding cost in a case where the difference between the first coding cost and the second coding cost is larger than the threshold.

DESCRIPTION OF THE EMBODIMENTS

First Embodiment

FIG.1is a block diagram showing the configuration of an image coding apparatus100that performs predictive coding according to the embodiment. The image coding apparatus100may be arranged, for example, in an image processing engine incorporated in an image capturing apparatus.

A frame buffer101stores an input image to be encoded, and outputs image data of pixels of a required block size to an adder102, intra-prediction unit112, motion compensation unit114, and feature detection unit116.

The adder102generates difference image data between the image data output from the frame buffer101and predicted image data output from a prediction mode selection unit117described later, and outputs the difference image data to an orthogonal transformation unit103.

The orthogonal transformation unit103performs orthogonal transform such as discrete cosine transform (DCT) and discrete sine transform (DST) on the difference image data, generates transform coefficients, and outputs the transform coefficients to a quantization unit104.

The quantization unit104quantizes the transform coefficients output from the orthogonal transformation unit103according to the quantization step size (or quantization parameter) output by a quantization control unit106, and outputs the results to an entropy coding unit105, and an inverse quantization unit107.

The entropy coding unit105performs variable-length coding on the quantized transform coefficients, motion vectors, parameters necessary for decoding processing, and the like, formats them into a predetermined format, and outputs them as a coded stream. Also, the entropy coding unit105calculates a generated code amount for each block and outputs it to the quantization control unit106.

The quantization control unit106determines a quantization step size for each block and a quantization parameter Qp from the generated code amount output from the entropy coding unit105and a preset target code amount, and outputs them to the quantization unit104and the prediction mode selection unit117.

The inverse quantization unit107performs inverse quantization on the quantized transform coefficients output from the quantization unit104, generates transform coefficients, and outputs the transform coefficients to an inverse orthogonal transformation unit108.

The inverse orthogonal transformation unit108applies inverse transformation of the orthogonal transformation applied by the orthogonal transformation unit103to the transform coefficients output from the inverse quantization unit107, generates difference image data, and outputs the difference image data to an adder109.

The adder109adds the difference image data generated by the inverse orthogonal transformation unit108and predicted image data output from the prediction mode selection unit117, which will be described later, to generate locally decoded reference image data, and outputs it to a reference frame buffer110and a loop filter111.

The loop filter111includes a deblocking filter, an adaptive loop filter, etc., removes block noise from the reference image data output from the adder109, and outputs the result to the reference frame buffer110. Note that the reference image data output from the adder109may be stored in the reference frame buffer110without going through the loop filter111as described above. In other words, the filtering process by the loop filter111can be omitted.

The reference frame buffer110stores the supplied reference image data, and outputs the stored reference image data to the intra-prediction unit112and the motion compensation unit114at predetermined timings.

The intra-prediction unit112performs intra-prediction using the image data of a coding target block output from the frame buffer101and reference image data of blocks around the coding target block read out from the reference frame buffer110.

Here, intra-prediction modes selectable by the intra-prediction unit112will be described with reference toFIG.2.

FIG.2is a diagram showing types of intra-prediction modes (mode numbers). The intra-prediction unit112can select an intra-prediction mode from planar prediction (mode number0), DC prediction (mode number1), and 65 types of directional prediction (mode numbers2to66).

Further, the intra-prediction unit112selects the intra-prediction mode with the smallest cost value according to the cost function indicating the coding efficiency among the intra-prediction modes. Various functions are conceivable as the cost function, and the following equation (1) is used as a representative equation in this embodiment.

In the equation (1), IntraCost represents a cost value that indicates the coding efficiency in intra-prediction mode, and SATD represents a value obtained by taking a difference between input image data to be encoded and the reference image data, applying Hadamard transform to obtained difference data, and taking a sum of absolute differences of the transformed difference data. Qp represents a quantization parameter used in the quantization unit104, and Mdcost represents a cost value corresponding to a code amount obtained in an intra-prediction mode.

The intra-prediction unit112generates a predicted image for intra-prediction using the intra-prediction mode selected using the equation (1). In addition, the intra-prediction unit112outputs the selected intra-prediction mode, the cost value when the selected intra-prediction mode is applied, and the predicted image data (intra-predicted image) generated by the intra-prediction, to a multi-reference prediction unit115and the prediction mode selection unit117.

A motion information memory113is a memory for temporarily holding motion vectors output from a motion compensation unit114, which will be described later. The motion information memory113may also hold the motion vectors of the blocks located around the coding target block, and the motion vector of each of the blocks of already coded frame or frames. Information of the motion vectors held in the motion information memory113is referred to by the motion compensation unit114, which will be described later.

The motion compensation unit114selects the optimal motion vector from among two modes: an inter mode for newly searching for a motion vector and a merge mode for selecting the optimal motion vector from motion vector candidates of already coded blocks.

Here, the operation of the motion compensation unit114will be explained usingFIG.3.

In step S301, the motion compensation unit114uses the image data of the coding target block output from the frame buffer101and the reference image data read out from the reference frame buffer110to search for the optimum motion vector in the inter mode. More specifically, the motion compensation unit114searches for the motion vector with the lowest cost value calculated using the cost function that indicates coding efficiency. Although various functions are conceivable as the cost function, the equation (2) is used as a representative one in this embodiment.

In Equation (2), InterCost represents a cost value indicating coding efficiency in the inter mode. SATD represents a value obtained by taking a difference between input image data to be encoded and the reference image data after motion vector is applied, applying Hadamard transform to obtained difference data, and taking a sum of absolute differences of the transformed difference data. Qp is a quantization parameter used in the quantization unit104, and Mvcost is a cost value corresponding to a code amount of motion vector.

The motion compensation unit114temporarily holds the information of the motion vector determined using the equation (2) and the cost value obtained when the motion vector is applied

In step S302, the motion compensation unit114selects the optimum motion vector in the merge mode from a plurality of motion vector candidates already stored in the motion information memory113. More specifically, the motion compensation unit114applies the equation (2) to each of the plurality of motion vector candidates and selects the motion vector with the lowest cost value. The motion compensation unit114also calculates an index value (merge index) indicating the motion vector candidate.

The motion compensation unit114temporarily holds the merge index and the cost value obtained when the motion vector indicated by the merge index is applied.

In step S303, the motion compensation unit114compares the cost value in the inter mode calculated in step S301and the cost value in the merge mode calculated in step S302, and selects the mode with the lowest cost value.

In step S304, the motion compensation unit114uses the motion vector in the selected mode and the reference image data read out from the reference frame buffer110to generate predicted image data for inter-prediction. In the merge mode, the motion vector can be determined by referring to the merge index calculated in step S302.

The motion compensation unit114outputs a flag (merge flag) indicating whether or not the merge mode has been selected, information on motion vector or merge index, the cost value in the selected mode, and the generated predicted image data for inter-prediction (inter-predicted image) to the multi-reference prediction unit115and the prediction mode selection unit117.

The multi-reference prediction unit115combines the predicted image data for intra-prediction output from the intra-prediction unit112and the predicted image data for inter-prediction output from the motion compensation unit114to generates new predicted image data.

As a technique for combining predicted image data for intra-prediction and predicted image data for inter-prediction, in “Algorithm description for Versatile Video Coding and Test Mode 10 (VTM10)”, JVET-S2002-v1, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 2020, a technology called Combined Inter and Intra Prediction (CIIP) is disclosed. The method of combining the predicted image data for intra-prediction and predicted image data for inter-prediction is also based on CIIP. The method of combining the predicted image data by intra-prediction and predicted image data for inter-prediction is also based on CIIP. On the other hand, in “Algorithm description for Versatile Video Coding and Test Mode 10 (VTM10)”, it is described, as application conditions of CIIP, that intra-predicted images are applied with PLANAR prediction, inter-predicted images are applied with merge mode, and so forth. However, the method of combining the predicted image data for intra-prediction and the predicted image data for inter-prediction in this embodiment is not necessarily limited to these conditions, and it is also possible to apply directional prediction, and like, to the predicted image data for intra-prediction.

In CIIP, predicted images for intra-prediction and inter-prediction are blended pixel by pixel, so that differences in predicted images between blocks can be further reduced. Based on this, in the present embodiment, CIIP is applied to blocks that exist in areas where intra-prediction and inter-prediction blocks are likely to be adjacent to each other, and image quality differences between blocks are likely to be noticeable, such as flat areas of an image. This reduces image quality degradation caused by using different prediction modes to generate predicted images of adjacent blocks.

Next, details of a method of combining predicted image data for intra-prediction and predicted image data for inter-prediction performed by the multi-reference prediction unit115will be described below.

The multi-reference prediction unit115combines predicted image data by intra-prediction and predicted image data by inter-prediction for each pixel using the following equation (3).

In the equation (3), Pintra represents the pixel value of predicted image data for intra-prediction, Pinter represents the pixel value of predicted image data for inter-prediction, and Pciip is the pixel value of predicted image data after combining them. wt represents a weighting coefficient indicating the ratio of pixel values for intra-prediction and inter-prediction, and can take an integer value from 1 to 3 here.

Here, a method of determining the weighting coefficient wt used in the multi-reference prediction unit115will be described with reference toFIG.4.

In step S401, the multi-reference prediction unit115initializes isIntraTop and isIntraLeft to 0, and the weighting coefficient wt to 1. isIntraTop is a flag indicating whether or not the prediction mode of the encoded block that is adjacent to the coding target block in the upward direction is intra-prediction. Also, isIntraLeft is a flag indicating whether or not the prediction mode of the encoded block adjacent to the coding target block in the left direction is intra-prediction. Note that the prediction modes of these encoded blocks are held by the prediction mode selection unit117as described later, and the multi-reference prediction unit115can appropriately refer to this information.

In step S402, the multi-reference prediction unit115determines whether or not the prediction mode of the encoded block adjacent to the coding target block in the upward direction is intra-prediction. If the prediction mode of the encoded block adjacent in the upward direction is intra-prediction (Yes in step S402), the multi-reference prediction unit115sets isIntraTop to 1 in step S403. If the prediction mode of the encoded block adjacent in the upward direction is not intra-prediction (No in step S402), the multi-reference prediction unit115advances the process to step S404.

In step S404, the multi-reference prediction unit115determines whether or not the prediction mode of the encoded block adjacent to the coding target block in the left direction is intra-prediction. If the prediction mode of the encoded block adjacent in the left direction is intra-prediction (Yes in step S404), the multi-reference prediction unit115sets isIntraLeft to 1 in step S405. If the prediction mode of the encoded block adjacent in the left direction is not intra-prediction (No in step S404), the multi-reference prediction unit115advances the process to step S406.

In steps S406to S410, the multi-reference prediction unit115determines the value of weighting coefficient wt according to the sum of isIntraTop and isIntraLeft. If the sum of isIntraTop and isIntraLeft is 2 (Yes in step S406), the process proceeds to step S407where the multi-reference prediction unit115sets the weighting coefficient wt to 3. If the sum of isIntraTop and isIntraLeft is 1 (No in step S406and Yes in step S408), the process proceeds to step S409where the multi-reference prediction unit115sets the weighting coefficient wt to 2. If the sum of isIntraTop and isIntraLeft is 0 (No in step S406and No in step S408), the process proceeds to step S410where the multi-reference prediction unit115sets the weighting coefficient wt to one.

The processes of steps S406to S410adaptively change the combining percentages between the pixel values of the predicted image data for intra-prediction and the pixel values of the predicted image data for inter-prediction at the time of combining the predicted images, according to the prediction mode of the blocks surrounding the coding target block. As a result, in a case where multi-reference prediction is selected as the prediction mode in the prediction mode selection unit117, which will be described later, image quality deterioration due to predicted images of the coding target block and its surrounding blocks being different from each other can be reduced.

Note that the method of synthesizing the predicted image data for intra-prediction and the predicted image data for inter-prediction is not limited to the example shown in the equation (3) andFIG.4. For example, the multi-reference prediction unit115may perform the synthesis using the equation (4) below.

As another example, the weighting coefficient wt can be adaptively changed for each pixel according to the pixel position of each pixel in the coding target block. Furthermore, the weighting coefficient wt can be adaptively changed according to the intra-prediction mode (DC prediction, PLANAR prediction, etc.) output from the intra-prediction unit112, the block size of the coding target block, and the like.

The multi-reference prediction unit115uses predicted image data (multi-predicted image) synthesized using the equation (3) to calculate a cost value based on a cost function that indicates coding efficiency. Various functions are conceivable as the cost function, but in this embodiment, the equation (5) is used as a representative one.

In the equation (5), CiipCost represents a cost value indicating coding efficiency in multi-reference prediction. SATD represents a value obtained by taking a difference between input image data to be encoded and the synthesized predicted image data, applying Hadamard transform to obtained difference data, and taking a sum of absolute differences of the transformed difference data. Qp is a quantization parameter used in the quantization unit104, and Mpcost is a cost value corresponding to a code amount of parameters required in multi-reference prediction. Parameters required in multi-reference prediction include an intra-prediction mode output from the intra-prediction unit112, merge flag, merge index, or motion vector output from the motion compensation unit114.

The multi-reference prediction unit115outputs the cost value calculated using the equation (5) and the synthesized predicted image data to the prediction mode selection unit117.

The feature detection unit116generates a flat area flag indicating whether the coding target block belongs to a flat area of the image and outputs it to the prediction mode selection unit117. Here, a method of generating the flat area flag in the feature detection unit116will be described with reference toFIG.5

FIG.5is a block diagram showing an example of the internal configuration of the feature detection unit116.

A flatness determination unit501acquires an input image of the coding target block and generates a flag (flatness flag) indicating whether or not the coding target block is flat. Various methods are conceivable for determining whether or not the block is flat. In this embodiment, the flatness is defined as the variance of the pixel values of the coding target block, and if the flatness is lower than a predetermined threshold, the block is determined to be flat. In the present embodiment, the flag value is “1” if the block is determined to be flat, and the flag value is “0” if the block is determined to be not flat.

A flatness flag buffer502temporarily holds the flatness flag output from the flatness determination unit501. The flatness flag buffer502also holds flatness flags of a plurality of encoded blocks, and a flat area determination unit503, which will be described later, can appropriately refer to this information. The flatness flag buffer502in this embodiment holds at least the flatness flags of the encoded blocks adjacent to the coding target block in the upper direction, the upper left direction, and the left direction.

The flat area determination unit503determines whether the coding target block belongs to a flat area of the image based on the flatness flag of the coding target block and the flatness flags of a plurality of encoded blocks held in the flatness flag buffer502. Various methods are conceivable as a method of determining whether or not the coding target block belongs to a flat area. In this embodiment, if the flag values of the flat flags of the encoded blocks adjacent in the upper direction, the upper left direction and the left direction are all “1”, it is determined that the coding target block belongs to a flat area. In the present embodiment, the flag value is set to “1” if the block is determined that it belongs to a flat area, and the flag value is set to “0” if the block is determined that it does not belong to a flat area.

Note that the method of determining whether or not the coding target block belongs to a flat area is not limited to the example described usingFIG.5. For example, the feature detection unit116may simply assume that a coding target block belongs to a flat area if its flatness is lower than a predetermined threshold. Alternatively, the feature detection unit116may determine a flatness flag for each of the blocks of a frame and store the flatness flags in the flatness flag buffer502before starting the coding processing of the frame. This allows the flat area determination unit503to improve determination accuracy of whether or not the coding target block belongs to a flat area by referring not only to the flat flags of blocks adjacent to the coding target block, but also to the flat flags of blocks further away from the coding target block.

The prediction mode selection unit117selects one of the intra-prediction, inter-prediction, and multi-reference prediction. Here, the determination method of the prediction mode in the prediction mode selection unit117is explained usingFIG.6.

In step S601, the prediction mode selection unit117determines whether the flat area flag output from the feature detection unit116is 0 and/or whether or not the quantization step size Qstep of the coding target block obtained from the quantization control unit106is smaller than the threshold Thqstep.

Thereafter, in steps S602to S608following S601, the prediction mode selection unit117puts weight on coding efficiency and always selects the prediction mode with the lowest cost value. On the other hand, in steps S609to S613, the prediction mode selection unit117puts weight on reduction of image quality degradation, and does not always select the prediction mode with the lowest cost value so that degradation of image quality due to different predicted images between blocks is less conspicuous. That is, the determination formula of step S601means that the prediction mode selection unit117selects which one of the coding efficiency and the reduction of image quality deterioration is emphasized.

In this embodiment, in a case where the region to which the coding target block belongs is not flat and/or in a case where the quantization step size is small, it is determined that the degree of image quality deterioration due to different predicted images between blocks is small, and the processes of steps S602to S608are performed with emphasis on the encoding efficiency. Conversely, in a case where the region to which the coding target block belongs is flat and the quantization step size is large, the processes of S609to S613are performed with emphasis on reducing image quality deterioration. Note that the determination method in step S601is not limited to the example shown here. For example, the prediction mode selection unit117can perform the determination using either the flat area flag or the quantization step size.

In step S601, if the flat area flag is 0 and/or the quantization step size Qstep is smaller a the threshold Thqstep, the prediction mode selection unit117advances the process to step S602.

In step S602, the prediction mode selection unit117compares the cost value IntraCost output from the intra-prediction unit112and the cost value InterCost output from the motion compensation unit114.

If InterCost is greater than IntraCost (Yes in step S602), the prediction mode selection unit117compares IntraCost with the cost value CiipCost output from the multi-reference prediction unit115(step S603). If the IntraCost is greater than the CiipCost (Yes in step S603), the prediction mode selection unit117selects multi-reference prediction as the prediction mode (step S604). If the IntraCost is less than or equal to the CiipCost (No in step S603), the prediction mode selection unit117selects intra-prediction as the prediction mode (step S605).

Also, in step S602, if InterCost is less than or equal to IntraCost (No in step S602), the prediction mode selection unit117compares InterCost with CiipCost (step S606). If InterCost is greater than CiipCost (Yes in step S606), the prediction mode selection unit117selects multi-reference prediction as the prediction mode (step S607). If InterCost is less than or equal to CiipCost (No in step S606), the prediction mode selection unit117selects inter-prediction as the prediction mode (step S608).

On the other hand, in step S601, if the flat area flag output from the feature detection unit116is 1 and the quantization step size Qstep is equal to or greater than the threshold Thqstep, the prediction mode selection unit117advances the process to step S609.

In step S609, the prediction mode selection unit117compares InterCost with the values of IntraCost+α. α is a correction value for the cost value of intra-prediction that can be arbitrarily set.

Generally, in a region where the cost values of intra-prediction and inter-prediction are about the same, image quality deterioration in intra-prediction tends to be more conspicuous than that in inter-prediction. Therefore, in this embodiment, if the coding target block belongs to a flat area, inter-prediction is preferentially selected by correcting the cost value of intra-prediction. As a result, cases in which blocks of intra-prediction and blocks of inter-prediction are adjacent to each other are reduced.

In this embodiment, the cost value is corrected by adding a as described above, but it is also possible to correct the cost value by subtracting, multiplying, or dividing the cost value by an arbitrary constant. Also, the method of correcting the cost value is not limited to the method using a as described above, and any method may be adopted as long as the method enables inter-prediction to be preferentially selected.

If InterCost is greater than IntraCost+α (Yes in step S609), the prediction mode selection unit117selects intra-prediction as the prediction mode (step S610).

If InterCost is equal to or less than IntraCost+α (No in step S609), the prediction mode selection unit117determines whether the absolute value of the difference between InterCost and IntraCost is equal to or less than a constant c (step S611). Here, the details of the determination formula used in step S611will be described.

As described above, the determination in step S609is intended to preferentially select inter-prediction. On the other hand, depending on the degree of flatness of the image, an area in which blocks of intra-prediction and blocks of inter-prediction often become adjacent to each other, and the determination result of step S609frequently switches still exists. Here, in an area where the determination result of step S609frequently changes, it is assumed that the values of InterCost and IntraCost+α are close to each other in the determination formula used in step S609. In consideration of the above, in the present embodiment, the area where the determination result in step S609frequently changes is detected using the following expression (6).

β is a constant for detecting the closeness between InterCost and the values of IntraCost+α. Here, assuming that α is a positive constant, the following expression (7) is obtained by modifying the expression (6).

That is, in the present embodiment, in a case where the absolute value of the difference between the InterCost and IntraCost of the coding target block is equal to or less than the constant c in step S611, it is determined that the block exists in an area where blocks of intra-prediction and blocks of inter-prediction block tend to be adjacent to each other.

Note that the constant c can be set arbitrarily, and may be adaptively changed using the target bit rate of the image coding apparatus, the value of the quantization parameter Qp used in the quantization unit104, or the characteristics of the image. For example, the value of the constant c can be changed for each block according to the flatness, and the like, calculated by the flatness determination unit501.

If the absolute value of the difference between InterCost and IntraCost is c or less (Yes in step S611), the prediction mode selection unit117selects multi-reference prediction as the prediction mode (step S612). That is, in the present embodiment, multi-reference prediction is applied to blocks existing in an area where blocks of intra-prediction and blocks of inter-prediction block tends to be adjacent to each other. In a case where multi-reference prediction is used, since a predicted image of intra-prediction and a predicted image of inter-prediction predicted are blended and the difference in predicted images between blocks becomes small, it is possible to reduce image quality degradation caused by predicted images being generated using different prediction modes between blocks.

In multi-reference prediction, it is necessary to encode parameters related to prediction modes for both intra-prediction and inter-prediction (intra-prediction mode, motion vector, etc.), there is a risk that a code amount related to parameters will be larger in multi-reference prediction than in inter-prediction. Therefore, the prediction mode selection unit117may further perform determination of whether or not the value of the merge flag output from the motion compensation unit114being 1 in the determination of step S611. As a result, when multi-reference prediction is selected, there is no need to encode parameters related to motion vectors, and an increase in code amount related to parameters can be suppressed.

If the absolute value of the difference between InterCost and IntraCost is not equal to or less than the constant c (No in step S611), the prediction mode selection unit117selects inter-prediction as the prediction mode (step S613).

A predicted image is generated according to the prediction mode selected as described above. Then, the difference between the image data of the coding target block and the predicted image is encoded, and the coded result is output as a coded stream (coded data).

In the example shown inFIG.6, if an area is in the flat part of the image and if the quantization step size is large, the area is detected as an area where blocks of intra-prediction and blocks of inter-prediction tend to be adjacent to each other, and the prediction mode that prioritizes reduction of image quality deterioration is applied in this area. On the other hand, there are cases where the boundary between a block of intra-prediction and a block of inter-prediction is conspicuous regardless of the degree of flatness of the image, depending on the target bit rate of the image coding apparatus. In such a case, the prediction mode selection unit117may not perform the determination of step S601, and may always detect an area where a block of intra-prediction and a block of inter-prediction tend to be adjacent to each other, and apply multi-reference prediction to the blocks existing in the detected area.

In addition, regardless of whether the boundary between a block of intra-prediction and a block of inter-prediction is conspicuous or not, the processing from step S609onward may be performed for all blocks.

As described above, according to the first embodiment, in a case where an image is encoded using different predicted images, it is possible to make the image quality deterioration caused by the difference in the predicted images less conspicuous.

In addition, in a case where the boundary between a block of intra-prediction and a block of inter-prediction is conspicuous, such as in a case where an area to which the coding target block belongs is flat and the quantization step size is large, the prediction mode selection unit117operates with an emphasis on reducing image quality deterioration. On the other hand, in a case where the boundary between a block of intra-prediction and a block of inter-prediction is inconspicuous, the prediction mode selection unit117operates with emphasis on coding efficiency. This makes it possible to reduce image quality deterioration due to differences in predicted images between blocks compared to the conventional art, without impairing coding efficiency.

Next, a method of generating coded data in the modification will be described with reference toFIG.10.

As described above with reference toFIG.3, the motion compensation unit114calculates the cost value in a case where the inter-prediction mode is applied to the coding target block (step S1001). Further, as described above with reference toFIG.2, the intra-prediction unit112calculates the cost value in a case where the intra-prediction mode is applied to the coding target block (step S1002).

Then, as in step S611, the prediction mode selection unit117determines whether or not the absolute value of the difference between the cost value of the inter-prediction mode calculated in step S1001and the cost value of the intra-prediction mode calculated in step S1002is a constant c or less (step S1003). As a result of the determination, if the absolute value of the difference is the constant c or less (Yes in step S1003), the prediction mode selection unit117selects the multi-reference prediction mode (step S1004).

On the other hand, as a result of the determination, if the absolute value of the difference is greater than the constant c (No in step S1003), the prediction mode selection unit117calculates the cost value in a case where the multi-reference prediction mode is applied to the coding target block (step S1005). The prediction mode selection unit117compares the cost values of the inter-prediction mode, the intra-prediction mode, and the multi-reference prediction mode calculated in steps S1001, S1002, and S1005, respectively, and if the cost value of the multi-reference prediction mode is the lowest (Yes in step S1006), the multi-reference prediction mode is selected (step S1004).

If the cost value of the multi-reference prediction mode is not the lowest (No in step S1006) and the cost value of the inter-prediction mode is the lowest (Yes in step S1007), the inter-prediction mode is selected (step S1008). If the cost value of the multi-reference prediction mode is not the lowest (No in step S1006) and the cost value of the inter-prediction mode is not the lowest (No in step S1007), that is, if the cost value of the intra-prediction mode is the lowest, the intra-prediction mode is selected (step S1009).

Then, if the multi-reference prediction mode is selected in step S1004, the multi-reference prediction unit115combines the predicted image data for intra-prediction output from the intra-prediction unit112and the predicted image data for inter-prediction output from the motion compensation unit114, and generates predicted image data for multi-reference prediction (step S1010). Alternatively, if the inter-prediction mode is selected in step S1008, the motion compensation unit114generates predicted image data for inter-prediction (step S1011) as in step S304. Alternatively, if the intra-prediction mode is selected in step S1009, the intra-prediction unit112generates predicted image data for intra-prediction (step S1012). The predicted image data generated in step S1010, S1011or S1012is output to the adder102by the prediction mode selection unit117.

Then, the adder102generates difference image data between the block image data to be encoded output from the frame buffer and the predicted image data, and encodes the difference image data using each unit as described above (step S1013), and the coded result is output as a coded stream (step S1014).

Before starting the processing of step S1001, similarly to step S601, the prediction mode selection unit117may determine whether or not the flat area flag output from the feature detection unit116is 0 and/or whether or not the quantization step size Qstep of the coding target block obtained from the quantization control unit106is smaller than the threshold Thqstep, and if the flat area flag is 1 and/or the quantization step size Qstep is equal to or greater than the threshold Thqstep, the processes after step S1001may be executed. Alternatively, in a case where the dynamic range of the input image is a high dynamic range equal to or greater than a predetermined value, the processing from step S1001onward may be executed.

Second Embodiment

In the latest video coding methods such as HEVC and VVC, it is possible to encode a moving image recorded with 10-bit, 12-bit, or other multi-bit HDR (High Dynamic Range) or a Log-recorded moving image. A Log-recorded moving image is a moving image shot using a gamma curve uniquely defined by the manufacturer of a digital camera or the like, so that recording can be performed in a higher dynamic range, unlike the gamma curve defined by standards such as BT.709.

HDR-recorded or Log-recorded moving images (high dynamic range images) are displayed after being converted using a set gamma curve at the time of outputting the images to a display.FIG.7Ashows the relationship between input luminance values (actual brightness at the time of shooting) and output pixel values (pixel value after coding) in HDR-recording. The horizontal axis shows the input luminance values, the vertical axis shows the output pixel values, and the relationship between the input values and the output values are defined by a gamma curve701. In a range where the input value is small, the curve is steep in the vertical direction, and many pixel values are assigned with respect to the range of the input values. On the other hand, in a range where the input value is large, the curve becomes gentle, and the number of pixel values assigned with respect to the range of the input values is small. By assigning the output pixel values to the input luminance values in this way, recording with a high dynamic range is made possible.

FIG.7Bshows the relationship between input values (pixel value after coding) and output luminance values (brightness displayed on a display) in HDR-display. The horizontal axis shows the input pixel values, and the vertical axis shows the output luminance values that are actually displayed, and the relationship between the input values and the output values is defined by a gamma curve702, which is the opposite relationship to the gamma curve701. In a range where the input values are small, the curve is gentle in the vertical direction, and the amount of change in displayed brightness with respect to the range of the input values is small. On the other hand, in a range where the input values are large, the curve becomes steep, and the amount of change in displayed brightness with respect to the range of the input values is large. By assigning output the luminance values to the input pixel values in this way, HDR-recorded moving images can be displayed with correct brightness.

As described above, in HDR-recorded or Log-recorded moving images, the amount of change in brightness that is displayed on a display is larger with respect to the amount of change in coded pixel values compared to moving images recorded with Standard Dynamic Range (SDR). In other words, deterioration in image quality after coding is amplified and displayed on the display. In addition, this tendency becomes more conspicuous in an area with high luminance values (referred to as “high-luminance area or “bright area”), and image quality degradation due to predicted images being different between blocks, which is the problem that the present invention is to solve, is also more conspicuous in this area.

In the second embodiment, such HDR-recorded or Log-recorded moving images (for convenience, referred to as “HDR images”) are coding target, and image quality deterioration that occurs in a bright area of HDR images is aimed at being reduced. Note that in the second embodiment, differences from the first embodiment will be mainly described, and descriptions of parts that are the same as those in the first embodiment will be omitted.

The overall configuration of an image encoding apparatus in the second embodiment is the same as that shown inFIG.1, but the configuration of the feature detection unit116is different from that described with reference toFIG.5in the first embodiment.

FIG.8is a block diagram showing an example of an internal configuration of the feature detection unit116according to the second embodiment. As in the first embodiment, the feature detection unit116in the second embodiment includes the flatness determination unit501, the flatness flag buffer502, and the flat area determination unit503for determining whether or not a coding target block belongs to a flat area of an image. Furthermore, the feature detection unit116has a high luminance determination unit804, a high luminance flag buffer805, and a high luminance area determination unit806for determining whether or not the coding target block belongs to a bright area of the image.

Note that the flatness determination unit501, flatness flag buffer502, and flat area determination unit503operate in the same manner as in the first embodiment, so their description is omitted here.

The high luminance determination unit804acquires an input image of the coding target block and generates a flag (high luminance flag) indicating whether or not the coding target block is of high luminance. Various methods are conceivable for determining whether or not the luminance is high, and in this embodiment, an average of luminance values of pixels in the coding target block is calculated, and if the calculated average luminance value is larger than a predetermined threshold, it is determined that the luminance is high. Further, in the present embodiment, the flag value is set to “1” if it is determined that the luminance is high, and the flag value is set to “0” if it is determined that the luminance is not high.

The high luminance flag buffer805temporarily holds the high luminance flag output from the high luminance determination unit804. The high luminance flag buffer805also holds high luminance flags of a plurality of coded blocks, and the high luminance area determination unit806, which will be described later, can appropriately refer to this information. The high luminance flag buffer805in this embodiment holds at least the high luminance flags of the coded blocks that are adjacent to the coding target block in the upper direction, the upper left direction, and the left direction.

The high luminance area determination unit806determines whether or not the coding target block belongs to the bright area of the image based on the high luminance flag of the coding target block held in the high luminance flag buffer805and the high luminance flags of a plurality of coded blocks. Various methods are conceivable for determining whether or not the coding target block belongs to the bright area, and in this embodiment, if the high luminance flag of the coding target block is “1” and if the flag values of the high luminance flags of the coded blocks adjacent to the coding target block in the upper direction, the upper left direction and the left direction are all “1”, it is determined that the coding target block belongs to the bright area. In the present embodiment, the flag value is “1” if it is determined that the coding target block belongs to the bright area, and the flag value is “0” if it is determined that the coding target block does not belong to the bright area.

Note that the method of determining whether or not the coding target block belongs to the bright area is not limited to the example described usingFIG.8. For example, the feature detection unit116may simply determine that the coding target block belongs to a bright area if the average luminance value of the coding target block is greater than a predetermined threshold. Alternatively, the feature detection unit116may determine a high luminance flag for each block of a certain frame and store the high luminance flags in the high luminance flag buffer805before starting coding processing of the frame. With this, the high luminance area determination unit806can refer not only to the high brightness flags of blocks adjacent to the coding target block, but also to the high brightness flags of blocks farther away from the coding target block, thereby it is possible to improve the accuracy of determination whether the coding target block belongs to the bright area.

In this way, the feature detection unit116generates a flat area flag and a bright area flag and outputs them to the prediction mode selection unit117.

FIG.9is a flowchart showing a prediction mode determination method in the prediction mode selection unit117in the second embodiment. The same reference numerals are given to the same processes as those shown inFIG.6, and the description thereof is omitted.

The processing in the second embodiment shown inFIG.9is different from the processing in the first embodiment shown inFIG.6in the point that a process (step S901) for determining whether or not a moving image to be encoded is a HDR-recorded or Log-recorded moving image, and whether or not a block belongs to a bright area is added. It should be noted that information on whether or not the moving image to be coded is HDR-recorded or Log-recorded is sent to the image coding apparatus in advance from a control source of the image encoding apparatus such as a CPU (not shown).

In step S901, if the moving image to be coded is an HDR-recorded or Log-recorded moving image and if the bright area flag is 1, then the prediction mode selection unit117advances the process to step S601. On the other hand, in step S901, if the image to be coded is not an HDR-recorded or Log-recorded moving image and/or if the bright area flag is not 1, then the prediction mode selection unit117skips the process of step S601, and advances the process to step S602.

That is, if the moving image to be coded is an HDR-recorded or Log-recorded moving image, if the coding target block belongs to a bright area and a flat area, and if the quantization step size is large, the prediction mode selection unit117puts a weight on reducing image quality deterioration and operates so that image quality deterioration due to a difference between predicted images of blocks becomes less conspicuous (steps S609to S613).

On the other hand, in cases other than the above, the prediction mode selection unit117puts a weight on coding efficiency and operates to always select a prediction mode with the lowest cost value (steps S602to S608).

As described above, according to the second embodiment, not only the flatness of an area to which the coding target block belongs and the quantization step size, but also whether or not the image to be coded is HDR-recorded or Log-recorded moving image and whether or not the coding target block belongs to a bright area are considered, and whether to prioritize coding efficiency or reduction of image quality deterioration is selected. As a result, it is possible to reduce image quality deterioration, which is particularly conspicuous in bright areas, compared to the conventional art, without impairing the coding efficiency when coding an HDR image.

According to this invention, it is possible to make image quality deterioration due to differences in predicted images inconspicuous in a case where one image is coded using different predicted images.

Other Embodiment