Patent ID: 12238296

DESCRIPTION OF EMBODIMENTS

An image encoding device and an image decoding device according to embodiments will be described with reference to the drawings. The image encoding device and the image decoding device according to the embodiments encode and decode a moving image typified by MPEG. The same or similar reference numerals will be assigned to the same or similar portions in the following description of the drawings.

1. First Embodiment

An image encoding device and an image decoding device according to a first embodiment will be described.

(1.1. Configuration of Image Encoding Device)

FIG.1illustrates a configuration of an image encoding device1according to the first embodiment. As illustrated inFIG.1, the image encoding device1includes a block divider100, a subtractor101, a transformer102a, a quantizer102b, an entropy encoder103, an inverse quantizer104a, an inverse transformer104b, a combiner105, an intra predictor106, a loop filter107, a frame memory108, a motion compensation predictor109, a switcher110, and an evaluator111.

The block divider100divides an input image which is in a frame (or picture) unit into blocks which are small areas, and outputs image blocks to the subtractor101(and the motion compensation predictor109). A size of the image block is, for example, 32×32 pixels, 16×16 pixels, 8×8 pixels, 4×4 pixels, or the like. The image block which is a unit of encoding to be performed by the image encoding device1and a unit of decoding to be performed by the image decoding device2, will be referred to as a target image block.

The subtractor101calculates a prediction residual indicating a difference in a pixel unit between the target image block input from the block divider100and a prediction image (prediction image block) corresponding to the target image block. Specifically, the subtractor101calculates the prediction residual by subtracting each pixel value of the prediction image from each pixel value of the encoding target block, and outputs the calculated prediction residual to the transformer102a. Note that the prediction image is input to the subtractor101from the intra predictor106or the motion compensation predictor109which will be described later via the switcher110.

The transformer102aand the quantizer102bconfigure a transformer/quantizer102which performs orthogonal transform processing and quantization processing in a block unit.

The transformer102acalculates transform coefficients by performing an orthogonal transform of the prediction residual input from the subtractor101and outputs the calculated transform coefficients to the quantizer102b. The orthogonal transform includes, for example, discrete cosine transform (DCT), discrete sine transform (DST), Karhunen-Loeve transform (KLT), and the like.

The quantizer102bquantizes the transform coefficients input from the transformer102ausing a quantization parameter (Qp) and a quantization matrix to generate quantized transform coefficients. The quantization parameter (Qp) is a parameter to be applied in common to respective transform coefficients within a block and is a parameter which defines roughness of quantization. The quantization matrix is a matrix having a quantization value upon quantization of the respective transform coefficients as an element. The quantizer102boutputs quantization control information, the generated quantized transform coefficient information, and the like, to the entropy encoder103and the inverse quantizer104a.

The entropy encoder103entropy encodes the quantized transform coefficients input from the quantizer102b, generates encoded data (bit stream) by compressing data, and outputs the encoded data to outside of the image encoding device1.

Huffman codes, context-based adaptive binary arithmetic coding (CABAC), or the like, can be used in entropy encoding. Note that the entropy encoder103receives input of information regarding prediction from the intra predictor106and the motion compensation predictor109and receives input of information regarding filter processing from the loop filter107. The entropy encoder103also entropy encodes these kinds of information.

The inverse quantizer104aand the inverse transformer104bconfigure an inverse quantizer/inverse transformer104which performs inverse quantization processing and inverse orthogonal transform processing in a block unit.

The inverse quantizer104aperforms inverse quantization processing corresponding to the quantization processing performed by the quantizer102b. Specifically, the inverse quantizer104arestores the transform coefficients by performing inverse quantization of the quantized transform coefficients input from the quantizer102busing the quantization parameter (Qp) and the quantization matrix, and outputs the restored transform coefficients to the inverse transformer104b.

The inverse transformer104bperforms inverse orthogonal transform processing corresponding to the orthogonal transform processing performed by the transformer102a. For example, in a case where the transformer102aperforms discrete cosine transform, the inverse transformer104bperforms inverse discrete cosine transform. The inverse transformer104brestores the prediction residual by performing an inverse orthogonal transform of the transform coefficients input from the inverse quantizer104aand outputs the restored prediction residual to the combiner105.

The combiner105combines the restored prediction residual input from the inverse transformer104bwith the prediction image input from the switcher110in a pixel unit. The combiner105reconstructs the target image block by adding respective pixel values of the restored prediction residual and respective pixel values of the prediction image and outputs the reconstructed image which is the reconstructed target image block to the intra predictor106and the loop filter107.

The intra predictor106generates an intra-predicted image by performing intra prediction using the reconstructed image input from the combiner105and outputs the intra-predicted image to the switcher110. Further, the intra predictor106outputs information of the selected intra prediction mode, and the like, to the entropy encoder103.

The loop filter107performs filter processing as post processing on the reconstructed image input from the combiner105and outputs the reconstructed image subjected to the filter processing to the frame memory108. Further, the loop filter107outputs information regarding the filter processing to the entropy encoder103. The filter processing includes deblocking filter processing and sample adaptive offset processing. The deblocking filter processing is processing for reducing signal degradation due to processing in a block unit and is processing of smoothing a gap of signals at a boundary portion between a block and an adjacent block. This deblocking filter processing controls strength (filter strength) of the deblocking filter processing using a signal gap at the boundary portion and a quantization parameter indicating a degree of quantization. Meanwhile, the sample adaptive offset processing is filter processing for improving image quality employed in, for example, HEVC (see Non-Patent Literature 1) and is processing of categorizing respective pixels using relative relationship between a pixel and an adjacent pixel within the block, calculating an offset value for improving image quality for each category, and uniformly providing offset values to respective pixels belonging to the same category. Since stronger filter processing is applied as the offset value is greater, the offset value can be regarded as a value which defines filter strength of the sample adaptive offset processing.

The frame memory108stores the reconstructed image input from the loop filter107in a frame unit.

The motion compensation predictor109performs inter prediction using one or more reconstructed images stored in the frame memory108as reference images.

Specifically, the motion compensation predictor109calculates a motion vector using an approach such as block matching, generates a motion compensation prediction image based on the motion vector, and outputs the motion compensation prediction image to the switcher110. Further, the motion compensation predictor109outputs information regarding the motion vector to the entropy encoder103.

The switcher110switches between the intra-predicted image input from the intra predictor106and the motion compensation prediction image input from the motion compensation predictor109, and outputs the prediction image (the intra-predicted image or the motion compensation prediction image) to the subtractor101and the combiner105.

The evaluator111calculates a degree of similarity between the plurality of reference images to be used for prediction for each image portion including one or more pixels in a case where the motion compensation predictor109performs motion compensation prediction using the plurality of reference images, evaluates prediction accuracy of the prediction image for each image portion using the degree of similarity, and outputs information of the evaluation result to the combiner105.

An example will be described where, in the present embodiment, the evaluator111calculates a degree of similarity between the plurality of reference images to be used for prediction in a unit of one pixel and evaluates prediction accuracy of the prediction image in a unit of one pixel. Note that, while not illustrated in the present embodiment, in a case of intra prediction (for example, an intra block copy mode), or the like, using a plurality of reference images, in a case where the intra predictor106performs prediction using a plurality of reference images, the evaluator111calculates a degree of similarity between the plurality of reference images, evaluates prediction accuracy of the prediction image in a pixel unit and outputs this evaluation result to the combiner105. The combiner105controls the restored prediction residual to be combined with the prediction image in a pixel unit based on the result of evaluation by the evaluator111. The evaluator111and the combiner105will be described in detail later.

(1.2. Configuration of Image Decoding Device)

FIG.2illustrates a configuration of an image decoding device2according to the first embodiment. As illustrated inFIG.2, the image decoding device2includes an entropy code decoder200, an inverse quantizer201a, an inverse transformer201b, a combiner202, an intra predictor203, a loop filter204, a frame memory205, a motion compensation predictor206, a switcher207, and an evaluator208.

The entropy code decoder200decodes the encoded data generated by the encoding device1and outputs the quantized transform coefficients to the inverse quantizer201a. Further, the entropy code decoder200decodes the encoded data to acquire information regarding prediction (intra prediction and motion compensation prediction) and information regarding the filter processing, outputs the information regarding the prediction to the intra predictor203and the motion compensation predictor206, and outputs the information regarding the filter processing to the loop filter204.

The inverse quantizer201aand the inverse transformer201bconfigure an inverse quantizer/inverse transformer201which performs inverse quantization processing and inverse orthogonal transform processing in a block unit.

The inverse quantizer201aperforms inverse quantization processing corresponding to the quantization processing performed by the quantizer102bof the image encoding device1. The inverse quantizer201arestores the transform coefficients by performing inverse quantization of the quantized transform coefficients input from the entropy code decoder200using the quantization parameter (Qp) and the quantization matrix, and outputs the restored transform coefficients to the inverse transformer201b.

The inverse transformer201bperforms inverse orthogonal transform processing corresponding to the orthogonal transform processing performed by the transformer102aof the image encoding device1. The inverse transformer201brestores the prediction residual by performing an inverse orthogonal transform of the transform coefficients input from the inverse quantizer201aand outputs the restored prediction residual to the combiner202.

The combiner202reconstructs the original target image block by combining the prediction residual input from the inverse transformer201bwith the prediction image input from the switcher207in a pixel unit, and outputs the reconstructed image to the intra predictor203and the loop filter204.

The intra predictor203generates an intra-predicted image by performing intra prediction in accordance with the intra prediction information input from the entropy code decoder200with reference to the reconstructed encoded block image input from the combiner202and outputs the intra-predicted image to the switcher207.

The loop filter204performs filter processing similar to the filter processing performed by the loop filter107of the image encoding device1on the reconstructed image input from the combiner202based on the filter processing information input from the entropy code decoder200and outputs the reconstructed image subjected to the filter processing to the frame memory205.

The frame memory205stores the reconstructed image input from the loop filter204in a frame unit. The frame memory205outputs the stored reconstructed images to outside of the image decoding device2in display order.

The motion compensation predictor206generates the motion compensation prediction image by performing motion compensation prediction (inter prediction) in accordance with the motion vector information input from the entropy code decoder200using one or more reconstructed images stored in the frame memory205as reference images, and outputs the motion compensation prediction image to the switcher207.

The switcher207switches between the intra-predicted image input from the intra predictor203and the motion compensation prediction image input from the motion compensation predictor206and outputs the prediction image (the intra-predicted image or the motion compensation prediction image) to the combiner202.

The evaluator208performs operation similar to that performed at the evaluator111of the image encoding device1. Specifically, in a case where the motion compensation predictor206performs motion compensation prediction using a plurality of reference images, the evaluator208evaluates prediction accuracy of the prediction image in a pixel unit by calculating a degree of similarity between the plurality of reference images in a pixel unit and outputs information of the evaluation result to the combiner202. The combiner202controls the restored prediction residual to be combined with the prediction image in a pixel unit based on the result of evaluation by the evaluator208.

(1.3. Motion Compensation Prediction)

FIG.3illustrates an example of motion compensation prediction.FIG.4illustrates an example of the prediction image generated through motion compensation prediction. A case will be described as a simple example of the motion compensation prediction where bi-prediction used in HEVC, particularly, forward direction and backward prediction (bidirectional prediction) are used.

As illustrated inFIG.3, the motion compensation prediction is performed with reference to temporally preceding and subsequent frames with respect to a target frame (current frame). In the example inFIG.3, motion compensation prediction of a block in an image of a t-th frame is performed with reference to a t−1-th frame and a t+1-th frame. In the motion compensation, portions (blocks) within the t−1-th reference frame and the t+1-th reference frame, which are similar to the target image block are detected from a search range set at a system.

The detected portions are reference images. Information indicating relative positions of the reference images with respect to the target image block indicated with an arrow in the drawing, will be referred to as a motion vector. Information of the motion vector is entropy encoded along with the frame information of the reference images at the image encoding device1. Meanwhile, the image decoding device2detects the reference images based on the information of the motion vector generated by the image encoding device1.

As illustrated inFIG.3andFIG.4, reference images1and2detected through motion compensation are similar partial images aligned with the target image block, within the frames to be referred to, and are thus regarded as images similar to the target image block (encoding target image). In the example inFIG.4, the target image block includes a design of a star and a design of a partial circle. The reference image1includes a design of a star and a design of the entire circle. The reference image2includes a design of a star, but does not include a design of a circle.

The prediction image is generated from such reference images1and2. Note that, in the prediction processing, the prediction image having features of the respective reference images is typically generated by averaging the reference images1and2which have different features but are partially similar to each other. However, the prediction image may be generated also using more advanced processing, for example, signal enhancement processing using a low-pass filter, a high-pass filter, or the like. Here, if the prediction image is generated by averaging the reference image1which includes a design of a circle and the reference image2which does not include a design of a circle, signals of the design of the circle in the prediction image decreases by half compared to those of the reference image1.

A difference between the prediction image obtained from the reference images1and2and the target image block (encoding target image) is the prediction residual. The prediction residual indicated inFIG.4indicates that a large difference exists only at a portion where edges of the stars in the designs are misaligned and at a portion where the circles in the designs are misaligned (shaded portions), and prediction is performed with high accuracy and has less differences at other portions (a difference does not exist in the example inFIG.4).

A difference does not exist (at a portion which does not correspond to edges of the stars in the designs and at a background portion) at portions where a degree of similarity between the reference image1and the reference image2is high, and where prediction is performed with high accuracy. Meanwhile, a large difference exists at portions unique to the respective reference images, that is, at portions where the degree of similarity between the reference image1and the reference image2is significantly low. Thus, it can be known that prediction accuracy is low and a large difference (residual) exists at portions where the degree of similarity between the reference image1and the reference image2is significantly low.

If the transform coefficients degrade due to the prediction residual including a portion with a large difference and a portion with no difference being orthogonally transformed and quantized, such degradation of the transform coefficients propagates to the whole of the image (block) through inverse quantization and inverse orthogonal transform.

Then, if the target image block is reconstructed by combining the prediction residual (restored prediction residual) restored by inverse quantization and inverse orthogonal transform with the prediction image, degradation of image quality also propagates to portions where prediction has been performed with high accuracy such as a portion which does not correspond to the edges of the stars in the designs and the background portion illustrated inFIG.4.

(1.4. Evaluator and Combiner)

The evaluator111of the image encoding device1evaluates prediction accuracy of the prediction image in a pixel unit by calculating a degree of similarity between the plurality of reference images in a pixel unit. The combiner105then controls the restored prediction residual to be combined with the prediction image in a pixel unit based on the result of evaluation by the evaluator111.

In a similar manner, the evaluator208of the image decoding device2evaluates prediction accuracy of the prediction image in a pixel unit by calculating a degree of similarity between the plurality of reference images in a pixel unit. The combiner202then controls the restored prediction residual to be combined with the prediction image in a pixel unit based on the result of evaluation by the evaluator208.

By this means, it becomes possible to suppress the restored prediction residual to be combined with the prediction image for portions where prediction is performed with high accuracy, so that it is possible to prevent degradation of image quality in the restored prediction residual from propagating to the portions where prediction is performed with high accuracy. It is therefore possible to improve image quality and improve encoding efficiency in a case where motion compensation prediction is performed using a plurality of reference images.

FIG.5illustrates an example of a configuration of the evaluator111at the image encoding device1. As illustrated inFIG.5, the evaluator Ill includes a difference calculator (subtractor)111a, a normalizer111b, and a weight adjuster111c.

The difference calculator11acalculates an absolute value of a difference value between the reference image1and the reference image2in a pixel unit and outputs the calculated absolute value of the difference value to the normalizer111b. The absolute value of the difference value is an example of a value indicating the degree of similarity. A smaller absolute value of the difference value indicates a higher degree of similarity, while a greater absolute value of the difference value indicates a lower degree of similarity. The difference calculator111amay calculate the absolute value of the difference value after performing filter processing on the respective reference images. The difference calculator111amay calculate statistics such as a square error and may use the statistics as the degree of similarity.

The normalizer111bnormalizes a difference value of each pixel input from the difference calculator111awith an absolute value of the difference value of the pixel at which the absolute value of the difference value becomes a maximum within the block (that is, a maximum value of the absolute value of the difference value within the block) and outputs the normalized difference value which is the normalized absolute value of the difference value to the weight adjuster111c. In the first embodiment, the normalized difference value is used as a weight for weighting the restored prediction residual to be combined with the prediction image at the combiner105in a pixel unit.

The weight adjuster111cadjusts the normalized difference value (weight) input from the normalizer111bbased on the quantization parameter (Qp) which defines roughness of quantization and outputs this weight. The weight adjuster111ccan weight the restored prediction residual in view of a degradation degree of the restored prediction residual, which becomes higher as quantization is rougher, by adjusting the normalized difference value (weight) based on the quantization parameter (Qp).

A weight Wij of each pixel (ij) output from the evaluator111can be expressed with, for example, the following expression (1).
Wij=(abs(Xij−Yij)/maxD×Scale(Qp))  (1)

In expression (1), Xij is a pixel value of the pixel ij of the reference image1, Yij is a pixel value of the pixel ij of the reference image2, and abs is a function for obtaining an absolute value. The difference calculator111aillustrated inFIG.5outputs abs(Xij−Yij).

Further, in expression (1), maxD is a maximum value of the difference value abs(Xij−Yij) within the block. While it is necessary to obtain difference values for all pixels within the block to obtain maxD, it is also possible to use a maximum value, or the like, of an adjacent block which has already been subjected to encoding processing as a substitute for the difference values for all the pixels within the block to skip this processing, and, for example, in a case where there is a value equal to or greater than the maximum value, it is also possible to normalize maxD by performing clipping with the maximum value which has been used. Alternatively, it is also possible to obtain maxD from the quantization parameter (Qp) using a table which defines correspondence relationship between the quantization parameter (Qp) and maxD. Alternatively, it is also possible to use a fixed value defined in specifications in advance as maxD. The normalizer111boutputs abs(Xij−Yij)/maxD.

Further, in expression (1), Scale(Qp) is a coefficient to be multiplied in accordance with the quantization parameter (Qp). Scale(Qp) is designed so as to approach 1.0 in a case where Qp is greater and approach 0 in a case where Qp is smaller, and a degree of approach is adjusted by a system. Alternatively, it is also possible to use a fixed value defined in specifications in advance as Scale(Qp). Further, to simplify the processing, it is also possible to set a fixed value such as 1.0 designed in accordance with the system as Scale(QP).

The weight adjuster111coutputs abs(Xij−Yij)/maxD×Scale(Qp) as the weight Wij. Alternatively, the weight adjuster111cmay output a weight adjusted with a sensitivity function designed in accordance with the system as this Wij. For example, sensitivity may be adjusted not only with Wij=Clip(wij, 1.0, 0.0) in abs(Xij−Yij)/maxD×Scale(Qp)=wij, but also with Wij=Clip(wij+offset, 1.0, 0.0) by adding an offset in accordance with control information such as, for example, QP. Note that Clip(x, max, min) indicates processing of performing clipping with max in a case where x exceeds max, and performing clipping with min in a case where x falls below min.

The weight Wij calculated in this manner becomes a value within a range from 0 to 1.0. Basically, the weight Wij approaches 1.0 in a case where the absolute value of the difference value of the pixel ij among the reference values is greater (that is, prediction accuracy is lower), and approaches 0 in a case where the absolute value of the difference value of the pixel ij among the reference images is smaller (that is, prediction accuracy is higher). The evaluator111outputs map information including the weights Wij of respective pixels ij within the block to the combiner105in a block unit.

Note that the evaluator111performs evaluation (calculates the weight Wij) only in a case where motion compensation prediction using a plurality of reference images is applied, and uniformly sets 1.0 as the weight Wij without performing evaluation in other modes, for example, in unidirectional prediction or in intra prediction processing in which a plurality of reference images are not used.

FIG.6illustrates an example of a configuration of the combiner105in the image encoding device1. As illustrated inFIG.6, the combiner105includes a weight applier (multiplier)105aand an adder105b.

The weight applier105aapplies a weight to the restored prediction residual input from the inverse transformer104bin a pixel unit using the map information (weight Wij) input from the evaluator111and outputs the weighted restored prediction residual to the adder105b.

The adder105bgenerates the reconstructed image by adding the weighted restored prediction residual input from the weight applier105ato the prediction image input from the motion compensation predictor109via the switcher110in a pixel unit, and outputs the generated reconstructed image.

Such processing of the combiner105can be expressed with, for example, the following expression (2).
Recij=Dij×Wij+Pij(2)

In expression (2), Recij is a pixel value of the pixel ij in the reconstructed image, Dij is a pixel value of the pixel ij in the restored prediction residual, Wij is a weight of the pixel ij in the map information, and Pij is a pixel value of the pixel ij in the prediction image.

Note that the combiner105performs weighting processing only in a case where motion compensation prediction using a plurality of reference images is applied, and does not perform weighting processing in other modes, for example, in unidirectional prediction and intra prediction processing.

Further, while the evaluator111and the combiner105in the image encoding device1have been described, the evaluator208and the combiner202in the image decoding device2are configured in a similar manner to the evaluator111and the combiner105in the image encoding device1. Specifically, the evaluator208in the image decoding device2includes a difference calculator208a, a normalizer208b, and a weight adjuster208c. The combiner202in the image decoding device2includes a weight applier (multiplier)202aand an adder202b.

(1.5. Operation of Image Encoding)

FIG.7illustrates processing flow at the image encoding device1according to the first embodiment.

As illustrated inFIG.7, in step S1101, the motion compensation predictor109predicts a target image block by performing motion compensation prediction using a plurality of reference images to generate a prediction image corresponding to the target image block. The entropy encoder103encodes the motion compensation prediction information as part of the encoded data and outputs the encoded data including the motion compensation prediction information.

In step S1102, the evaluator111evaluates prediction accuracy of the prediction image in a pixel unit by calculating a degree of similarity between the plurality of reference images in a pixel unit or evaluates the prediction accuracy in a unit of partial image by averaging prediction accuracy of a plurality of pixels, to generate map information including weights for respective pixels or partial images within the block.

In step S1103, the subtractor101calculates the prediction residual indicating a difference between the target image block and the prediction image in a pixel unit.

In step S1104, the transformer/quantizer102generates quantized transform coefficients by performing an orthogonal transform and quantization of the prediction residual calculated by the subtractor101.

In step S1105, the entropy encoder103entropy encodes the quantized transform coefficients and outputs the encoded data.

In step S1106, the inverse quantizer/inverse transformer104restores the prediction residual by performing inverse quantization and an inverse orthogonal transform the quantized transform coefficients to generate the restored prediction residual.

In step S1107, the combiner105controls the restored prediction residual to be combined with the prediction image in a pixel unit based on the result (map information) of evaluation by the evaluator111. Specifically, the combiner105performs weighting processing in a pixel unit as described above on the restored prediction residual.

In step S1108, the combiner105reconstructs the target image block by combining the weighted restored prediction residual with the prediction image in a pixel unit to generate the reconstructed image.

In step S1109, the loop filter107performs filter processing on the reconstructed image. Further, the entropy encoder103encodes information regarding the loop filter (such as an offset and category information to which the offset is to be applied) as part of the encoded data and outputs the encoded data including the information regarding the loop filter.

In step S1110, the frame memory108stores the reconstructed image subjected to the filter processing in a frame unit.

(1.6. Operation of Image Decoding)

FIG.8illustrates processing flow at the image decoding device2according to the first embodiment.

As illustrated inFIG.8, in step S1201, the entropy code decoder200decodes the encoded data to acquire the quantized transform coefficients, the motion vector information and the information regarding the loop filter.

In step S1202, the motion compensation predictor206predicts the target image block by performing motion compensation prediction using a plurality of reference images based on the motion vector information to generate the prediction image corresponding to the target image block.

In step S1203, the evaluator208evaluates prediction accuracy of the prediction image in a pixel unit by calculating a degree of similarity between the plurality of reference images in a pixel unit to generate map information including weights for respective pixels or partial images within the block.

In step S1204, the inverse quantizer/inverse transformer201restores the prediction residual by performing inverse quantization and an inverse orthogonal transform of the quantized transform coefficients to generate the restored prediction residual.

In step S1205, the combiner202controls the restored prediction residual to be combined with the prediction image in a pixel unit based on the result (map information) of evaluation by the evaluator208. Specifically, the combiner202performs weighting processing in a pixel unit as described above on the restored prediction residual.

In step S1206, the combiner202reconstructs the target image block by combining the weighted restored prediction residual with the prediction image in a pixel unit to generate the reconstructed image.

In step S1207, the loop filter204performs filter processing on the reconstructed image.

In step S1208, the frame memory205stores and outputs the reconstructed image subjected to the filter processing in a frame unit.

(1.7. Conclusion of the First Embodiment)

The evaluator111of the image encoding device1evaluates prediction accuracy of the prediction image in a pixel unit by calculating a degree of similarity between the plurality of reference images in a pixel unit. The combiner105then controls the restored prediction residual to be combined with the prediction image in a pixel unit based on the result of evaluation by the evaluator111.

The evaluator208of the image decoding device2evaluates prediction accuracy of the prediction image in a pixel unit by calculating a degree of similarity between the plurality of reference images in a pixel unit. The combiner202then controls the restored prediction residual to be combined with the prediction image in a pixel unit based on the result of evaluation by the evaluator208.

By this means, it becomes possible to suppress the restored prediction residual to be combined with the prediction image for a portion where prediction is performed with high accuracy, so that it is possible to prevent degradation of image quality in the restored prediction residual from propagating to the portion where prediction is performed with high accuracy. Consequently, it is possible to improve image quality and improve encoding efficiency in a case where motion compensation prediction is performed using a plurality of reference images.

2. Second Embodiment

An image encoding device and an image decoding device according to a second embodiment, mainly differences from the first embodiment, will be described. While, in the first embodiment, the evaluation result of the prediction accuracy is utilized in signal combining processing, in the second embodiment, the evaluation result of the prediction accuracy is utilized in filter processing.

(2.1. Image Encoding Device)

FIG.9illustrates a configuration of the image encoding device1according to the second embodiment. As illustrated inFIG.9, the evaluator111in the second embodiment outputs the evaluation result (map information) to the loop filter107. Specifically, the evaluator111evaluates prediction accuracy of the prediction image in a pixel unit by calculating a degree of similarity between the plurality of reference images in a pixel unit in a similar manner to the first embodiment.

The loop filter107controls filter strength in filter processing in a pixel unit based on the result of evaluation by the evaluator111. The loop filter107then performs filter processing (sample adaptive offset processing) by adding an offset value controlled in a pixel unit to the reconstructed image input from the combiner105in a pixel unit and outputs the reconstructed image subjected to the filter processing to the frame memory108.

(2.2. Image Decoding Device)

FIG.10illustrates a configuration of the image decoding device2according to the second embodiment. As illustrated inFIG.10, the evaluator208in the second embodiment outputs the evaluation result (map information) to the loop filter204. Specifically, the evaluator208evaluates prediction accuracy of the prediction image in a pixel unit by calculating a degree of similarity between the plurality of reference images in a pixel unit in a similar manner to the first embodiment.

The loop filter204controls filter strength in filter processing in a pixel unit based on the result of evaluation by the evaluator208. The loop filter204then performs filter processing (sample adaptive offset processing) by adding an offset value controlled in a pixel unit to the reconstructed image input from the combiner202in a pixel unit and outputs the reconstructed image subjected to the filter processing to the frame memory205.

(2.3. Loop Filter)

FIG.11illustrates an example of a configuration of the loop filter107in the image encoding device1. As illustrated inFIG.11, the loop filter107includes a weight applier (multiplier)107aand an adder107b.

The weight applier107aapplies a weight to the offset value which defines the filter strength in a pixel unit using the map information (weight Wij) input from the evaluator111. As the offset value which defines the filter strength, an offset value used in sample adaptive offset processing (see Non-Patent Literature 1) can be used. As described above, in the sample adaptive offset processing, the loop filter107categorizes respective pixels in accordance with relative relationship between a pixel and an adjacent pixel within the block and calculates an offset value so as to improve image quality for each category. The weight applier107aapplies a weight to the offset value to be used in the sample adaptive offset processing in a pixel unit and outputs the weighted offset value to the adder107b.

The adder107bperforms filter processing (sample adaptive offset processing) by adding the weighted offset value input from the weight applier107ato the reconstructed image input from the combiner105in a pixel unit and outputs the reconstructed image subjected to the filter processing.

Such processing of the loop filter107can be expressed with, for example, the following expression (3).
Recij′=Recij+dij×Wij(3)

In expression (3), Recij′ is a pixel value of the pixel ij in the reconstructed image subjected to the filter processing, Recij is a pixel value of the pixel ij in the reconstructed image before the filter processing, dij is an offset value to be added to the pixel ij, and Wij is a weight of the pixel ij in the map information.

Note that the loop filter107performs weighting processing based on the map information only in a case where motion compensation prediction using a plurality of reference images is applied, and does not perform weighting processing based on the map information in other modes, for example, in unidirectional prediction and in intra prediction processing.

Further, while the loop filter107in the image encoding device1has been described, the loop filter204in the image decoding device2is configured in a similar manner to the loop filter107in the image encoding device1. Specifically, the loop filter204in the image decoding device2includes a weight applier (multiplier204a) and an adder204b.

(2.4. Operation of Image Encoding)

FIG.12illustrates processing flow at the image encoding device1according to the second embodiment.

As illustrated inFIG.12, in step S2101, the motion compensation predictor109predicts the target image block by performing motion compensation prediction using a plurality of reference images to generate the prediction image corresponding to the target image block. The entropy encoder103encodes the motion compensation prediction information as part of the encoded data to generate the encoded data including the motion compensation prediction information and outputs the encoded data.

In step S2102, the evaluator111evaluates prediction accuracy of the prediction image in a pixel unit by calculating a degree of similarity between the plurality of reference images in a pixel unit to generate map information including weights for respective pixels within the block.

In step S2103, the subtractor101calculates the prediction residual indicating a difference in a pixel unit between the target image block and the prediction image.

In step S2104, the transformer/quantizer102generates quantized transform coefficients by performing an orthogonal transform and quantization of the prediction residual calculated by the subtractor101.

In step S2105, the entropy encoder103entropy encodes the quantized transform coefficients and outputs the encoded data.

In step S2106, the inverse quantizer/inverse transformer104restores the prediction residual by performing inverse quantization and an inverse orthogonal transform of the quantized transform coefficients to generate the restored prediction residual.

In step S2107, the combiner105reconstructs the target image block by combining the restored prediction residual with the prediction image in a pixel unit to generate the reconstructed image.

In step S2108, the loop filter107controls filter strength in filter processing in a pixel unit based on the result (map information) of evaluation by the evaluator111. Specifically, as described above, the loop filter107applies a weight to an offset value which defines the filter strength in a pixel unit. Further, the entropy encoder103encodes information regarding the loop filter (an offset, category information to which the offset is to be provided, and the like) as part of the encoded data and outputs the encoded data including the information regarding the loop filter.

In step S2109, the loop filter107performs filter processing (sample adaptive offset processing) by adding the weighted offset value to the reconstructed image in a pixel unit and outputs the reconstructed image subjected to the filter processing.

In step S2110, the frame memory108stores the reconstructed image subjected to the filter processing in a frame unit.

(2.5. Operation of Image Decoding)

FIG.13illustrates processing flow at the image decoding device2according to the second embodiment.

As illustrated inFIG.13, in step S2201, the entropy code decoder200decodes the encoded data to acquire the quantized transform coefficients, the motion vector information and the information regarding the loop filter.

In step S2202, the motion compensation predictor206predicts the target image block by performing motion compensation prediction using a plurality of reference images based on the motion vector information to generate the prediction image corresponding to the target image block.

In step S2203, the evaluator208evaluates prediction accuracy of the prediction image in a pixel unit by calculating a degree of similarity between the plurality of reference images in a pixel unit to generate map information including weights for respective pixels within the block.

In step S2204, the inverse quantizer/inverse transformer201restores the prediction residual by performing inverse quantization and an inverse orthogonal transform of the quantized transform coefficients to generate the restored prediction residual.

In step S2205, the combiner202reconstructs the target image block by combining the restored prediction residual with the prediction image in a pixel unit to generate the reconstructed image.

In step S2206, the loop filter204controls filter strength in filter processing in a pixel unit based on the result (map information) of evaluation by the evaluator208. Specifically, as described above, the loop filter204applies a weight to an offset value which defines filter strength in a pixel unit.

In step S2207, the loop filter204performs filter processing (sample adaptive offset processing) by adding the weighted offset value to the reconstructed image in a pixel unit and outputs the reconstructed image subjected to the filter processing.

In step S2208, the frame memory205stores and outputs the reconstructed image subjected to the filter processing in a frame unit.

(2.6. Conclusion of Second Embodiment)

The evaluator111of the image encoding device1evaluates prediction accuracy of the prediction image in a pixel unit by calculating a degree of similarity between the plurality of reference images in a pixel unit. The loop filter107then controls filter strength in filter processing in a pixel unit based on the result of evaluation by the evaluator111.

The evaluator208of the image decoding device2evaluates prediction accuracy of the prediction image in a pixel unit by calculating a degree of similarity between the plurality of reference images in a pixel unit. The loop filter204then controls filter strength in filter processing in a pixel unit based on the result of evaluation by the evaluator208.

By this means, it is possible to weaken filter processing by reducing filter strength at a portion where prediction is performed with high accuracy. Further, it is possible to strengthen filter processing by increasing filter strength for a portion where prediction is not performed with high accuracy. Consequently, it is possible to improve image quality and improve encoding efficiency in a case where motion compensation prediction is performed using a plurality of reference images.

3. Modifications of First and Second Embodiments

In the above-described first and second embodiments, an example has been described where the evaluator111evaluates prediction accuracy of the prediction image for each one pixel by calculating a degree of similarity between the plurality of reference images to be used for prediction for each one pixel. Further, in the above-described first embodiment, an example has been described where the combiner105controls the restored prediction residual to be combined with the prediction image for each one pixel based on the result of evaluation by the evaluator111. Further, in the above-described second embodiment, an example has been described where the loop filter107controls filter strength in filter processing for each one pixel based on the result of evaluation by the evaluator111.

However, the processing may be performed in a unit of group (sub-block) including a plurality of pixels in place of in a pixel unit. In the present modification, the target image block is divided into N sub-blocks (N is an integer of 2 or greater). Here, each sub-block includes m×n pixels, where at least one of m and n is an integer of 2 or greater. The evaluator111calculates weights Wij of respective pixels (ij) using the method in the above-described embodiments and calculates an average value Wk of the weights Wij for each k-th sub-block (where 0≤k≤N).

Then, in the above-described first embodiment, the evaluator111outputs the weight average value Wk calculated for each sub-block to the combiner105. The combiner105controls the restored prediction residual to be combined with the prediction image for each sub-block using the weight average value Wk. Specifically, the combiner105applies a weight to the restored prediction residual input from the inverse transformer104bin a sub-block unit using the weight average value Wk input from the evaluator111, and generates the reconstructed image by adding the weighted restored prediction residual to the prediction image in a pixel unit. Note that the image decoding device2also performs similar processing.

In the above-described second embodiment, the evaluator111outputs the weight average value Wk calculated for each sub-block to the loop filter107. The loop filter107controls filter strength in filter processing for each sub-block using the weight average value Wk. Specifically, the loop filter107applies a weight to an offset value to be used for sample adaptive offset processing in a sub-block unit and performs filter processing (sample adaptive offset processing) by adding the weighted offset value to the reconstructed image in a pixel unit. Note that the image decoding device2also performs similar processing.

3. Third Embodiment

(3.1. Configuration of Image Encoding Device)

FIG.16illustrates a configuration of the image encoding device1according to the third embodiment. As illustrated inFIG.16, the image encoding device1includes the block divider100, the subtractor101, the transformer102a, the quantizer102b, the entropy encoder103, the inverse quantizer104a, the inverse transformer104b, the combiner105, the intra predictor106, the loop filter107, the frame memory108, the motion compensation predictor109, the switcher110, and the evaluator111.

The block divider100divides an input image which is in a frame (or picture) unit into blocks which are small areas, and outputs image blocks to the subtractor101(and the motion compensation predictor109). A size of the image block is, for example, 32×32 pixels, 16×16 pixels, 8×8 pixels, 4×4 pixels, or the like. The image block which is a unit of encoding to be performed by the image encoding device1and a unit of decoding to be performed by the image decoding device2, will be referred to as a target image block.

The subtractor101calculates a prediction residual indicating a difference in a pixel unit between the target image block input from the block divider100and a prediction image (prediction image block) corresponding to the target image block. Specifically, the subtractor101calculates the prediction residual by subtracting each pixel value of the prediction image from each pixel value of the encoding target block, and outputs the calculated prediction residual to the transformer102a. Note that the prediction image is input to the subtractor101from the intra predictor106or the motion compensation predictor109which will be described later via the switcher110.

The transformer102aand the quantizer102bconfigure a transformer/quantizer102which performs orthogonal transform processing and quantization processing in a block unit.

The transformer102acalculates transform coefficients for each frequency component by performing an orthogonal transform of the prediction residual input from the subtractor101and outputs the calculated transform coefficients to the quantizer102b. The orthogonal transform includes, for example, discrete cosine transform (DCT), discrete sine transform (DST), Karhunen-Loeve Transform (KLT), and the like. The orthogonal transform is processing of transforming a residual signal in a pixel area into a signal in a frequency domain.

The quantizer102bquantizes the transform coefficients input from the transformer102ausing the quantization parameter (Qp) and the quantization matrix to generate the quantized transform coefficients. The quantization parameter (Qp) is a parameter to be applied in common to respective transform coefficients within a block and is a parameter which defines roughness of quantization. The quantization matrix is a matrix having a quantization value upon quantization of the respective transform coefficients as an element. The quantizer102boutputs quantization control information, the generated quantizer transform coefficient information, and the like, to the entropy encoder103and the inverse quantizer104a.

The entropy encoder103entropy encodes the quantized transform coefficients input from the quantizer102b, generates encoded data (bit stream) by compressing data, and outputs the encoded data to outside of the image encoding device1.

Huffman codes, context-based adaptive binary arithmetic coding (CABAC), or the like, can be used in entropy encoding. The entropy encoding includes processing called serialization of reading out transform coefficients arranged in two dimensions in predetermined scanning order and transforming the transform coefficients into a transform coefficient sequence in one dimension. Here, the transform coefficients are efficiently encoded up to a significant coefficient (non-zero coefficient) which is the last coefficient in the predetermined scanning order, and which is set as an end position.

Note that the entropy encoder103receives input of information regarding prediction from the intra predictor106and the motion compensation predictor109and receives input of information regarding filter processing from the loop filter107. The entropy encoder103also entropy encodes these kinds of information.

The inverse quantizer104aand the inverse transformer104bconfigure an inverse quantizer/inverse transformer104which performs inverse quantization processing and inverse orthogonal transform processing in a block unit.

The inverse quantizer104aperforms inverse quantization processing corresponding to the quantization processing performed by the quantizer102b. Specifically, the inverse quantizer104arestores the transform coefficients by performing inverse quantization of the quantized transform coefficients input from the quantizer102busing the quantization parameter (Qp) and the quantization matrix, and outputs the restored transform coefficients to the inverse transformer104b.

The inverse transformer104bperforms inverse orthogonal transform processing corresponding to the orthogonal transform processing performed by the transformer102a. For example, in a case where the transformer102aperforms discrete cosine transform, the inverse transformer104bperforms inverse discrete cosine transform. The inverse transformer104brestores the prediction residual by performing an inverse orthogonal transform of the transform coefficients input from the inverse quantizer104aand outputs the restored prediction residual to the combiner105.

The combiner105combines the restored prediction residual input from the inverse transformer104bwith the prediction image input from the switcher110in a pixel unit. The combiner105reconstructs the target image block by adding respective pixel values of the restored prediction residual and respective pixel values of the prediction image and outputs the reconstructed image which is the reconstructed target image block to the intra predictor106and the loop filter107.

The intra predictor106generates an intra-predicted image by performing intra prediction using the reconstructed image input from the combiner105and outputs the intra-predicted image to the switcher110. Further, the intra predictor106outputs information of the selected intra prediction mode, and the like, to the entropy encoder103.

The loop filter107performs filter processing as post processing on the reconstructed image input from the combiner105, and outputs the reconstructed image subjected to the filter processing to the frame memory108. Further, the loop filter107outputs information regarding the filter processing to the entropy encoder103. The filter processing includes deblocking filter processing and sample adaptive offset processing in the HEVC standards.

The frame memory108stores the reconstructed image input from the loop filter107in a frame unit.

The motion compensation predictor109performs inter prediction using one or more reconstructed images stored in the frame memory108as reference images.

Specifically, the motion compensation predictor109calculates a motion vector using am approach such as block matching, generates a motion compensation prediction image based on the motion vector, and outputs the motion compensation prediction image to the switcher110. Further, the motion compensation predictor109outputs information regarding the motion vector to the entropy encoder103.

The switcher110switches between the intra-predicted image input from the intra predictor106and the motion compensation prediction image input from the motion compensation predictor109and outputs the prediction image (the intra-predicted image or the motion compensation prediction image) to the subtractor101and the combiner105.

The evaluator111evaluates a degree of similarity between the plurality of reference images for each frequency component in a case where the motion compensation predictor109performs motion compensation prediction using the plurality of reference images, and outputs information of the evaluation result to the entropy encoder103. While not illustrated in the present embodiment, in a case of intra prediction (for example, an intra block copy mode) using a plurality of reference images, in a case where the intra predictor106performs prediction using a plurality of reference images, the evaluator111evaluates a degree of similarity between the plurality of reference images for each frequency component and outputs this evaluation result to the entropy encoder103. The entropy encoder103rearranges the transform coefficients input from the quantizer102bbased on the result of evaluation by the evaluator111and encodes the rearranged transform coefficients. The evaluator111and the entropy encoder103will be described in detail later.

(3.2. Configuration of Image Decoding Device)

FIG.17illustrates a configuration of the image decoding device2according to the third embodiment. As illustrated inFIG.17, the image decoding device2includes the entropy code decoder200, the inverse quantizer201a, the inverse transformer201b, the combiner202, the intra predictor203, the loop filter204, the frame memory205, the motion compensation predictor206, the switcher207, and the evaluator208.

The entropy code decoder200decodes the encoded data generated by the encoding device1and outputs the quantized transform coefficients to the inverse quantizer201a. Further, the entropy code decoder200decodes the encoded data to acquire the information regarding the prediction (intra prediction and motion compensation prediction) and the information regarding the filter processing, outputs the information regarding the prediction to the intra predictor203and the motion compensation predictor206, and outputs the information regarding the filter processing to the loop filter204.

The inverse quantizer201aand the inverse transformer201bconfigure an inverse quantizer/inverse transformer201which performs inverse quantization processing and inverse orthogonal transform processing in a block unit.

The inverse quantizer201aperforms inverse quantization processing corresponding to the quantization processing performed by the quantizer102bof the image encoding device1. The inverse quantizer201arestores the transform coefficients by performing inverse quantization of the quantized transform coefficients input from the entropy code decoder200using the quantization parameter (Qp) and the quantization matrix, and outputs the restored transform coefficients to the inverse transformer201b.

The inverse transformer201bperforms inverse orthogonal transform processing corresponding to the orthogonal transform processing performed by the transformer102aof the image encoding device1. The inverse transformer201brestores the prediction residual by performing an inverse orthogonal transform of the transform coefficients input from the inverse quantizer201a, and outputs the restored prediction residual to the combiner202.

The combiner202reconstructs the original target image block by combining the prediction residual input from the inverse transformer201bwith the prediction image input from the switcher207in a pixel unit, and outputs the reconstructed image to the intra predictor203and the loop filter204.

The intra predictor203generates an intra-predicted image by performing intra prediction in accordance with the intra prediction information input from the entropy code decoder200with reference to the reconstructed block image input from the combiner202, and outputs the intra-predicted image to the switcher207.

The loop filter204performs filter processing which is similar to the filter processing performed by the loop filter107of the image encoding device1, on the reconstructed image input from the combiner202based on the filter processing information input from the entropy code decoder200, and outputs the reconstructed image subjected to the filter processing to the frame memory205.

The frame memory205stores the reconstructed image input from the loop filter204in a frame unit. The frame memory205outputs the stored reconstructed images to outside of the image decoding device2in display order in a similar manner to the processing of the entropy encoder103.

The motion compensation predictor206generates a motion compensation prediction image by performing motion compensation prediction (inter prediction) in accordance with the motion vector information input from the entropy code decoder200using one or more reconstructed images stored in the frame memory205as reference images, and outputs the motion compensation prediction image to the switcher207.

The switcher207switches between the intra-predicted image input from the intra predictor203and the motion compensation prediction image input from the motion compensation predictor206and outputs the prediction image (the intra-predicted image or the motion compensation prediction image) to the combiner202.

The evaluator208performs operation similar to that performed by the evaluator111of the image encoding device1. Specifically, in a case where the motion compensation predictor206performs motion compensation prediction using a plurality of reference images, the evaluator208evaluates a degree of similarity between the plurality of reference images for each frequency component and outputs information of the evaluation result to the entropy code decoder200. The entropy code decoder200decodes the encoded data to acquire the transform coefficients for each frequency component, rearranges the transform coefficients based on the result of evaluation by the evaluator208and outputs the rearranged transform coefficients. The evaluator208and the entropy code decoder200will be described in detail later.

(3.3. Motion Compensation Prediction)

FIG.3illustrates an example of motion compensation prediction.FIG.4illustrates an example of the prediction image generated through motion compensation prediction. A case will be described as a simple example of the motion compensation prediction where bi-prediction used in HEVC, particularly, forward direction and backward prediction (bidirectional prediction) are used.

As illustrated inFIG.3, the motion compensation prediction is performed with reference to temporally preceding and subsequent frames with respect to a target frame (current frame). In the example inFIG.3, motion compensation prediction of a block in an image of a t-th frame is performed with reference to a t−1-th frame and a t+1-th frame. In the motion compensation, portions (blocks) within the t−1-th reference frame and the t+1-th reference frame, which are similar to the target image block are detected from a search range set at a system.

The detected portions are reference images. Information indicating relative positions of the reference images with respect to the target image block indicated with an arrow in the drawing, will be referred to as a motion vector. Information of the motion vector is entropy encoded along with the frame information of the reference images at the image encoding device1. Meanwhile, the image decoding device2detects the reference images based on the information of the motion vector generated by the image encoding device1.

As illustrated inFIG.3andFIG.4, reference images1and2detected through motion compensation are similar partial images aligned with the target image block, within the frames to be referred to, and are thus regarded as images similar to the target image block (encoding target image). In the example inFIG.4, the target image block includes a design of a star and a design of a partial circle. The reference image1includes a design of a star and a design of the entire circle. The reference image2includes a design of a star, but does not include a design of a circle.

The prediction image is generated from such reference images1and2. Note that, typically, the prediction processing enables generation of the image having features of the respective reference images with higher prediction accuracy by averaging the reference images1and2which have different features but are partially similar to each other. However, the prediction image may be generated also using more advanced processing, for example, signal enhancement processing using a low-pass filter, a high-pass filter, or the like. Here, if the prediction image is generated by averaging the reference image1which includes a design of a circle and the reference image2which does not include a design of a circle illustrated in the drawings, the design of the circle which cannot be predicted from the reference image2can be reflected in prediction. However, signals of the design of the circle in the prediction image decreases by half compared to those of the reference image1.

A difference between the prediction image obtained from the reference images1and2and the target image block (encoding target image) is the prediction residual. The prediction residual indicated inFIG.4indicates that a large difference exists only at a portion where edges of the stars in the designs are misaligned and at a portion where the circles in the designs are misaligned (shaded portions), and prediction is performed with high accuracy and has less differences at other portions (a difference does not exist in the example inFIG.4).

The portions where a difference does not exist (a portion which does not correspond to edges of the stars in the designs and a background portion) are portions where a degree of similarity between the reference image1and the reference image2is high, and where prediction is performed with high accuracy. Meanwhile, portions where a large difference exists are portions unique to the respective reference images, that is, portions where the degree of similarity between the reference image1and the reference image2is significantly low. Thus, it can be known that reliability of prediction is low and a large difference (residual) is likely to occur at portions where the degree of similarity between the reference image1and the reference image2is significantly low.

If the prediction residual including a portion with a large difference and a portion with no difference is orthogonally transformed, signal degradation due to quantization of the transform coefficients is uniformly multiplexed regardless of the prediction accuracy, which results in degradation of encoding quality.

The evaluator111in the third embodiment evaluates the degree of similarity between the plurality of reference images for each frequency component and outputs information of the evaluation result to the entropy encoder103. The entropy encoder103rearranges the transform coefficients input from the quantizer102bbased on the result of evaluation by the evaluator111and encodes the rearranged transform coefficients.

Here, a frequency component in which the degree of similarity between the plurality of reference images is low can be regarded as a frequency component having large energy. Meanwhile, a frequency component in which the degree of similarity between the plurality of reference images is high can be regarded as a frequency component having energy which is close to zero. Therefore, according to the rearranging order of the transform coefficients determined by the evaluator111, it is possible to efficiently encode the transform coefficients by the entropy encoder103rearranging the transform coefficients so that transform coefficients in a frequency component for which the degree of similarity is low are converged (put together).

Therefore, even in a case where degree of energy compaction of the transform coefficients is lowered due to energy being not compacted on low frequency components in the residual image after orthogonal transform, it is possible to perform efficient entropy encoding, so that it is possible to improve encoding efficiency.

(3.4. Evaluator)

FIG.18illustrates an example of a configuration of the evaluator111at the image encoding device1. As illustrated inFIG.18, the evaluator111includes a first transformer111a, a second transformer111b, a similarity degree calculator111c, and a normalizer111d. Note that, while an example where the evaluator111includes the normalizer111dwill be described, the evaluator111does not necessarily have to include the normalizer111dbecause the present invention is directed to determining encoding order of coefficients based on the degree of similarity.

The first transformer111acalculates first transform coefficients for each frequency component by performing an orthogonal transform of a reference image1(first reference image) input from the motion compensation predictor109, and outputs the calculated first transform coefficients to the similarity degree calculator111c.

The second transformer111bcalculates second transform coefficients for each frequency component by performing an orthogonal transform of a reference image2(second reference image) input from the motion compensation predictor109, and outputs the calculated second transform coefficients to the similarity degree calculator111c.

The similarity degree calculator111ccalculates a degree of similarity between the first transform coefficients input from the first transformer111aand the second transform coefficients input from the second transformer111bfor each frequency component and outputs the calculated degree of similarity to the normalizer111d. The degree of similarity includes, for example, an absolute value of a difference value. A smaller absolute value of the difference value indicates a higher degree of similarity, and a greater absolute value of the difference value indicates a lower degree of similarity. The similarity degree calculator111cmay calculate the difference value after performing filter processing on the respective reference images. The similarity degree calculator111cmay calculate statistics such as a square error and may use the statistics as the degree of similarity. An example where the absolute value of the difference value is used as the degree of similarity will be described below.

The normalizer111dnormalizes the absolute value of the difference value between the transform coefficients input from the similarity degree calculator111cwith a value in a frequency component in which the absolute value of the difference value becomes a maximum within the block (that is, a maximum value of the absolute value of the difference value within the block) and outputs the normalized absolute value of the difference value. The normalized difference value is used as a degree of importance for determining encoding order of the transform coefficients at the entropy encoder103. A degree of importance in encoding of the transform coefficients of the prediction error signal becomes lower for transform coefficients between which the absolute value is smaller because a smaller absolute value between the transform coefficients indicates a higher degree of similarity and higher prediction accuracy. Meanwhile, transform coefficients between which the absolute value is greater can be regarded as coefficients for which a degree of importance in encoding of the transform coefficients of the prediction error signal is higher because a greater absolute value for the coefficients indicates a lower degree of similarity and lower prediction accuracy. The entropy encoder103therefore preferentially encodes the transform coefficients in a frequency component with a high degree of importance.

The normalizer111dmay adjust a normalization difference value (degree of importance) input from the normalizer111dbased on at least one of the quantization parameter (Qp) which defines roughness of quantization and the quantization matrix to which different quantization values are applied for each transform coefficient, and may output the adjusted degree of importance. If a roughness degree of quantization is higher, a degradation degree of the restored prediction residual is higher. It is therefore possible to set the degree of importance in view of the degradation degree by adjusting the normalization difference value based on the quantization parameter (Qp) and the quantization value of the quantization matrix.

A degree of importance Rij of each frequency component (ij) output from the evaluator111can be expressed with, for example, the following expression (4).
Rij=(abs(Xij−Yij)/maxD×Scale(Qp))  (4)

In expression (4), Xij is a transform coefficient of the frequency component ij of the reference image1, Yij is a transform coefficient of the frequency component ij of the reference image2, and abs is a function for obtaining an absolute value. The similarity degree calculator111coutputs abs(Xij−Yij).

Further, in expression (4), maxD is a maximum value of a difference value abs(Xij−Yij) within the block. While it is necessary to obtain a difference value between the transform coefficients for all frequency components within the block to obtain maxD, it is also possible to use a maximum value, or the like, of an adjacent block which has already been subjected to encoding processing as a substitute for the difference values for all frequency components within the block to skip this processing, and, for example, in a case where there is a value equal to or greater than the maximum value, it is also possible to normalize maxD by performing clipping with the maximum value which has been used. Alternatively, it is also possible to obtain maxD from the quantization parameter (Qp) and the quantization value of the quantization matrix using a table which defines correspondence relationship between the quantization parameter (Qp) and the quantization value of the quantization matrix, and maxD. Alternatively, it is also possible to use a fixed value defined in specifications in advance as maxD. The normalizer111doutputs abs(Xij−Yij)/maxD.

Further, in expression (4), Scale(Qp) is a coefficient to be multiplied in accordance with the quantization parameter (Qp) and the quantization value of the quantization matrix. Scale(Qp) is designed so as to approach 1.0 in a case where Qp or the quantization value of the quantization matrix is greater and approach 0 in a case where Qp or the quantization value of the quantization matrix is smaller, and a degree of approach is adjusted by a system. Alternatively, it is also possible to use a fixed value defined in specifications in advance as Scale(Qp). Further, to simplify the processing, it is also possible to set a fixed value such as 1.0 designed in accordance with the system as Scale(Qp).

The normalizer111doutputs abs(Xij−Yij)/maxD×Scale(Qp) as the degree of importance Rij. Alternatively, the normalizer111dmay output a weight adjusted with a sensitivity function designed in accordance with the system as this Rij. For example, sensitivity may be adjusted not only with Rij=Clip(rij, 1.0, 0.0) in abs(Xij−Yij)/maxD×Scale(Qp)=rij, but also with Rij=Clip(rij+offset, 1.0, 0.0) by adding an offset. Note that Clip(x, max, min) indicates processing of performing clipping with max in a case where x exceeds max, and performing clipping with min in a case where x falls below min.

The degree of importance Rij calculated in this manner becomes a value within a range from 0 to 1.0. Basically, the degree of importance Rij approaches 1.0 in a case where the difference value between transform coefficients of the frequency components ij is greater (that is, prediction accuracy is lower), and approaches 0 in a case where the difference value is smaller (that is, prediction accuracy is higher). The evaluator111outputs map information (hereinafter, referred to as a “importance degree map”) including the degrees of importance Rij of the respective frequency components ij within the block to the entropy encoder103.

Note that the evaluator111performs evaluation (calculates the degree of importance Rij) only in a case where motion compensation prediction using a plurality of reference images is applied, and does not perform evaluation (does not calculate the degree of importance Rij) or uniformly sets 1.0 as the degree of importance Rij in other modes, for example, in unidirectional prediction or in intra prediction processing in which a plurality of reference images are not used.

Further, the evaluator208in the image decoding device2is configured in a similar manner to the evaluator111in the image encoding device1. Specifically, the evaluator208in the image decoding device2includes a first transformer208a, a second transformer208b, a similarity degree calculator208c, and a normalizer208d. The evaluator208in the image decoding device2outputs the importance degree map including the degrees of importance Rij of the respective frequency components ij within the block to the entropy code decoder200.

(3.5. Entropy Encoder)

FIG.19illustrates an example of a configuration of the entropy encoder103. As illustrated inFIG.19, the entropy encoder103includes a sorter103a, a serializer103b, and an encoder103c. The sorter103aand the serializer103bconfigure a rearranger.

The sorter103arearranges the degrees of importance Rij in the importance degree map input from the evaluator111in descending order. The sorter103aserializes the degrees of importance Rij arranged in two dimensions in the importance degree map, for example, in scanning order defined in advance to make an importance degree sequence R[i] and stores index labels i. The sorter103athen rearranges the index labels i in descending order of the degree of importance Rij in the importance degree sequence R[i] and outputs the index labels i rearranged in descending order of the degree of importance to the serializer103b.

The serializer103bperforms serialization processing of reading out the transform coefficients input from the quantizer102bin predetermined scanning order and outputting the transform coefficient sequence to the encoder103c. The serializer103b, for example, serializes the transform coefficients input from the quantizer102band arranged in two dimensions, in predetermined scanning order to make a transform coefficient sequence C[i]. Here, the serializer103band the sorter103ause the same scanning order. Further, the serializer103brearranges the transform coefficients in the transform coefficient sequence C[i] in descending order of the degree of importance based on the index labels i input from the sorter103aand outputs the rearranged transform coefficient sequence. In other words, the serializer103bperforms serialization processing so that the transform coefficient sequence includes transform coefficients in frequency components in ascending order of the degree of similarity between the transform coefficients based on the result of evaluation by the evaluator111. By this means, it is possible to put significant coefficients (non-zero coefficients) together.

Alternatively, the serializer103bmay determine the scanning order so that the transform coefficients are scanned in descending order of the degree of importance, and may output the transform coefficient sequence in which the transform coefficients are arranged in descending order of the degree of importance by performing scanning in the determined scanning order.

The encoder103cencodes the transform coefficients in the transform coefficient sequence input from the serializer103band outputs the encoded data. The encoder103cencodes the transform coefficients up to the last significant coefficient set as an end position, in the transform coefficient sequence input from the serializer103b. By putting the significant coefficients together as described above, it is possible to reduce the number of transform coefficients up to the end position, so that it is possible to shorten a length of the transform coefficient sequence to be encoded.

Note that the entropy encoder103may perform rearranging processing in accordance with the degree of importance only in a case where motion compensation prediction using a plurality of reference images is applied, and does not have to perform rearranging processing in accordance with the degree of importance in other modes, for example, in unidirectional prediction and intra prediction processing.

(3.6. Entropy Code Decoder)

FIG.20illustrates an example of a configuration of the entropy code decoder200. As illustrated inFIG.20, the entropy code decoder200includes a decoder200a, a sorter200b, and a deserializer200c. The sorter200band the deserializer200cconfigure a rearranger.

The decoder200adecodes the encoded data generated by the image encoding device1to acquire the transform coefficient sequence (quantized transform coefficients) and information regarding prediction (intra prediction and motion compensation prediction), outputs the transform coefficient sequence to the deserializer200cand outputs the information regarding prediction to the intra predictor203and the motion compensation predictor206.

The sorter200brearranges the degrees of importance Rij in the importance degree map input from the evaluator208in descending order. The sorter200b, for example, serializes the degrees of importance Rij arranged in two dimensions in the importance degree map in scanning order defined in advance to make an importance degree sequence R[i], and stores index labels i. The sorter200bthen rearranges the index labels i in descending order of the degree of importance Rij in the importance degree sequence R[i] and outputs the index labels i rearranged in descending order of the degree of importance and coordinate values (frequency components ij) corresponding to the index labels i to the deserializer200c.

The deserializer200cdeserializes the transform coefficient sequence input from the decoder200abased on the index labels i and the coordinate values (frequency components ij) input from the sorter200band outputs the transform coefficients arranged in two dimensions to the inverse quantizer201a.

Note that the entropy code decoder200may perform rearranging processing in accordance with the degree of importance only in a case where motion compensation prediction using a plurality of reference images is applied and does not have to perform rearranging processing in accordance with the degree of importance in other modes, for example, in unidirectional prediction and in intra prediction processing.

(3.7. Image Encoding Flow)

FIG.21illustrates processing flow at the image encoding device1according to the third embodiment. Here, operation relating to the present invention will be mainly described, and description regarding operation which is less relevant to the present invention will be omitted.

As illustrated inFIG.21, in step S3101, the motion compensation predictor109predicts the target image block by performing motion compensation prediction using a plurality of reference images to generate the prediction image corresponding to the target image block.

In step S3102, the evaluator111evaluates the degree of similarity between the plurality of reference images for each frequency component to generate an importance degree map indicating degrees of importance of respective frequency components within the block.

In step S3103, the subtractor101calculates a prediction residual indicating a difference in a pixel unit between the target image block and the prediction image.

In step S3104, the transformer/quantizer102generates quantized transform coefficients by performing an orthogonal transform and quantization of the prediction residual calculated by the subtractor101.

In step S3105, the entropy encoder103rearranges the transform coefficients input from the transformer/quantizer102(quantizer102b) in descending order of the degree of importance (that is, in ascending order of the degree of similarity between the transform coefficients) based on the result (importance degree map) of evaluation by the evaluator111and outputs the rearranged transform coefficients.

In step S3106, the entropy encoder103entropy encodes the transform coefficients rearranged in descending order of the degree of importance and outputs the encoded data.

In step S3107, the inverse quantizer/inverse transformer104restores the prediction residual by performing inverse quantization and an inverse orthogonal transform of the transform coefficients input from the transformer/quantizer102(quantizer102b) to generate the restored prediction residual.

In step S3108, the combiner105reconstructs the target image block by combining the restored prediction residual with the prediction image in a pixel unit to generate the reconstructed image.

In step S3109, the loop filter107performs filter processing on the reconstructed image.

In step S3110, the frame memory108stores the reconstructed image subjected to the filter processing in a frame unit.

(3.8. Image Decoding Flow)

FIG.22illustrates processing flow at the image decoding device2according to the third embodiment. Here, operation relating to the present invention will be mainly described, and description of operation which is less relevant to the present invention will be omitted.

As illustrated inFIG.22, in step S3201, the entropy code decoder200decodes the encoded data to acquire the motion vector information and outputs the acquired motion vector information to the motion compensation predictor206.

In step S3202, the motion compensation predictor206predicts the target image block by performing motion compensation prediction using the plurality of reference images based on the motion vector information to generate the prediction image corresponding to the target image block.

In step S3203, the evaluator208calculates the degree of similarity between the plurality of reference images for each frequency component to generate an importance degree map indicating degrees of importance of respective frequency components within the block.

In step S3204, the entropy code decoder200decodes the encoded data to acquire the transform coefficient sequence, rearranges the acquired transform coefficient sequence and outputs the transform coefficients arranged in two dimensions to the inverse quantizer/inverse transformer201.

In step S3205, the inverse quantizer/inverse transformer201restores the prediction residual by performing inverse quantization and an inverse orthogonal transform of the transform coefficients (quantized transform coefficients) to generate the restored prediction residual.

In step S3206, the combiner202reconstructs the target image block by combining the restored prediction residual with the prediction image in a pixel unit to generate the reconstructed image.

In step S3207, the loop filter204performs filter processing on the reconstructed image.

In step S3208, the frame memory205stores and outputs the reconstructed image subjected to the filter processing in a frame unit.

(3.9. Conclusion of Third Embodiment)

The evaluator111of the image encoding device1evaluates the degree of similarity between the plurality of reference images for each frequency component and outputs information of the evaluation result to the entropy encoder103. The entropy encoder103rearranges the transform coefficients input from the quantizer102bbased on the result of evaluation by the evaluator111and encodes the rearranged transform coefficients. By encoding the transform coefficients after rearranging the transform coefficients so that transform coefficients in frequency components in which the degree of similarity is low are converged (put together), it becomes possible to efficiently encode the transform coefficients and achieve efficient entropy encoding, so that it is possible to improve encoding efficiency.

The evaluator208of the image decoding device2evaluates the degree of similarity between the plurality of reference images for each frequency component and outputs information of the evaluation result to the entropy code decoder200. The entropy code decoder200decodes the encoded data to acquire transform coefficients for each frequency component, rearranges the transform coefficients based on the result of evaluation by the evaluator208and outputs the rearranged transform coefficients. By the transform coefficients being rearranged based on the result of evaluation by the evaluator208in this manner, even if information specifying details of rearranging is not transmitted from the image decoding device1, the entropy code decoder200can autonomously rearrange the transform coefficients. Therefore, the image decoding device1does not have to transmit information specifying details of rearranging, so that it is possible to avoid degradation of encoding efficiency.

(3.10. Modifications of Third Embodiment)

FIG.23illustrates a modification of a configuration of the evaluator111of the image encoding device1. As illustrated inFIG.23, an evaluator111A according to the present modification includes a similarity degree calculator111c, a transformer111e, and a normalizer111d. Note that, while an example where the evaluator111includes the normalizer111dwill be described, the evaluator111does not necessarily have to include the normalizer111d.

The similarity degree calculator111ccalculates a degree of similarity between the reference image1(first reference image) and the reference image2(second reference image) input from the motion compensation predictor109in a pixel unit, and outputs the degree of similarity calculated in a pixel unit to the transformer111e. For example, an absolute value of a difference value can be used as the degree of similarity. A smaller absolute value indicates a higher degree of similarity, and a greater absolute value indicates a lower degree of similarity. The similarity degree calculator111cmay calculate the difference value after performing filter processing on the respective reference images. The similarity degree calculator111cmay calculate statistics such as a square error and may use the statistics as the degree of similarity. An example where the absolute value of the difference value is used as the degree of similarity will be described below.

The similarity degree calculator111ccalculates the degree of similarity for each frequency component by performing an orthogonal transform of the degree of similarity (difference value) in a pixel unit input from the similarity degree calculator111c.

The normalizer111dnormalizes the difference value (transform coefficients) in a unit of frequency component input from the similarity degree calculator111cwith the difference value in a frequency component in which the difference value becomes a maximum within the block (that is, a maximum value of the absolute value of the difference value within the block) and outputs the normalized difference value.

The normalizer111dmay adjust the normalized difference value (degree of importance) input from the normalizer111dbased on at least one of the quantization parameter (Qp) which defines roughness of quantization and the quantization matrix to which quantization values different for each transform coefficient are applied, and may output the adjusted normalized difference value.

The degree of importance Rij of each frequency component (ij) output from the evaluator111A according to the modification can be expressed with, for example, the following expression (5).
Rij=abs(Dij)/maxD×Scale(Qp)  (5)

In expression (5), Dij is a transform coefficient of the frequency component ij, and abs is a function for obtaining an absolute value. The transformer111eoutputs abs(Dij).

Further, in expression (5), maxD is a maximum value of the transform coefficients within the block. While it is necessary to obtain the transform coefficients for all frequency components within the block to obtain maxD, it is also possible to use a maximum value, or the like, of an adjacent block which has already been subjected to encoding processing as a substitute for the transform coefficients for all frequency components within the block to skip this processing. Alternatively, it is also possible to obtain maxD from the quantization parameter (Qp) and the quantization value of the quantization matrix using a table which defines correspondence relationship between the quantization parameter (Qp) and the quantization value of the quantization matrix, and maxD. Alternatively, it is also possible to use a fixed value defined in specifications in advance as maxD. The normalizer111doutputs abs(Dij)/maxD. In expression (5), Scale(Qp) is similar to that in the above-described third embodiment.

In this manner, the evaluator111A according to the modification generates the importance degree map including degrees of importance Rij of respective frequency components ij within the block and outputs the generated importance degree map to the entropy encoder103.

The evaluator111A according to the modification can achieve reduction of the number of orthogonal transformers compared to the evaluator111according to the above-described third embodiment, and thus can achieve reduction of processing load. Typically, orthogonal transform used in encoding of an image is substantially normal orthogonal transform, and thus, the evaluator111A according to the modification can provide performance which is equivalent to that provided by the evaluator111according to the third embodiment.

Further, in the modification, the evaluator208A of the image decoding device2is configured in a similar manner to the evaluator111A of the image encoding device1. Specifically, the evaluator208A of the image decoding device2includes a similarity degree calculator208c, a transformer208e, and a normalizer208d. The evaluator208A of the image decoding device2outputs the importance degree map including the degrees of importance Rij of the respective frequency components ij within the block to the entropy code decoder200.

4. Other Embodiments

The above-described first embodiment and second embodiment may be combined.FIG.14illustrates a configuration of the image encoding device1in a case where the first embodiment and the second embodiment are combined. As illustrated inFIG.14, the evaluator111outputs the evaluation result (map information) to both the combiner105and the loop filter107.FIG.15illustrates a configuration of the image decoding device2in a case where the first embodiment and the second embodiment are combined. As illustrated inFIG.15, the evaluator208outputs the evaluation result (map information) to both the combiner202and the loop filter204.

An example has been described in the above-described third embodiment where the entropy encoder103reads out all the transform coefficients arranged in two dimensions in descending order of the degree of importance and performs serialization processing. However, only top several transform coefficients may be read out in descending order of the degree of importance among the transform coefficients arranged in two dimensions, and other transform coefficients may be read out in fixed order defined by the system. Alternatively, readout order of the transform coefficients arranged in two dimensions may be advanced or postponed by a predetermined number in accordance with the degree of importance.

In the above-described third embodiment, while it is possible to use zigzag scanning as employed in, for example, MPEG2 as scanning order of the transform coefficients, for example, in a case of HEVC (see Non-Patent Literature 1) which is the latest coding scheme, transform coefficients are rearranged in a unit called CG obtained by grouping the transform coefficients into 4×4 transform coefficients within the block. It is determined whether or not there is a non-zero coefficient within the CG, and in a case where there is a non-zero coefficient within the CG, transform coefficients within the CG are serialized and encoded. It is also possible to apply operation according to the above-described third embodiment to rearranging of transform coefficients upon readout of the transform coefficients within the CG. Alternatively, it is also possible to apply the operation to serialization of determining readout order of the CG and apply the operation to rearranging of readout order by calculating an average of degrees of similarity between orthogonal transform coefficients within the CG and comparing the degrees of similarity for each CG.

In the above-described respective embodiments, inter prediction has been mainly described as motion compensation prediction. In the inter prediction, reference images within a frame different from the current frame are used to predict the target image block of the current frame. However, the present invention is not limited to motion compensation prediction, and, for example, can be applied to a plurality of reference blocks in a technique called intra block copy. In the intra block copy, reference images within the same frame as the current frame are used to predict the target image block of the current frame.

The above-described specific examples of the present invention may be provided by a program which causes a computer to execute respective kinds of processing to be performed by the image encoding device1and a program which causes a computer to execute respective kinds of processing to be performed by the image decoding device2. Further, the programs may be stored in a computer readable medium. Use of the computer readable medium allows the programs to be installed onto the computer. Here, the computer readable medium in which the programs are recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, but for example, a recording medium such as a CD-ROM and a DVD-ROM. Further, circuits which execute respective kinds of processing to be performed by the image encoding device1may be integrated to configure the image encoding device1as a semiconductor integrated circuit (chip set, SoC). In a similar manner, circuits which execute respective kinds of processing to be performed by the image decoding device2may be integrated to configure the image decoding device2as a semiconductor integrated circuit (chip set, SoC).

The embodiments have been described in detail above with reference to the drawings. Specific configurations are not limited to the above-described configurations, and various design changes, and the like are possible within the scope not deviating from the gist.

Note that the present application claims the benefit of priority from Japanese Patent Application No. 2018-065895 (filed on Mar. 29, 2018), and Japanese Patent Application No. 2018-065886 (filed on Mar. 29, 2018), the entire contents of which are incorporated herein by reference.