SIGN PREDICTION IN VIDEO CODING

Methods, systems, and bitstream syntax are described for sign prediction in video coding. The method include: selection of top and left neighbors based on an image continuity check, the intra mode of the current coded unit (CU), the merge motion vector, or adaptive motion vector prediction, sign prediction based on residue domain of current CU or neighbor CUs, sign prediction based on approximated reconstruction samples, reducing the number of selected coefficients for sorting, simplifying the sequential search cost, and by combining sign prediction with sign data hiding.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority from Indian Provisional Patent Application No. 202241021948, filed on Apr. 12, 2022, which is incorporated by reference in its entirety.

TECHNOLOGY

The present document relates generally to images and video coding. More particularly, an embodiment of the present invention relates to sign prediction in video coding.

BACKGROUND

In 2020, the MPEG group in the International Standardization Organization (ISO), jointly with the International Telecommunications Union (ITU), released the first version of the Versatile Video coding Standard (VVC), also known as H.266 (Ref. [1]). More recently, the same group has been working on the development of the next generation coding standard that provides improved coding performance over existing video coding technologies. As part of this investigation, new coding techniques are also examined.

As appreciated by the inventors here, improved techniques for sign prediction in image and video coding are desired, and they are described herein.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Example embodiments that relate to sign prediction in video coding are described herein. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments of present invention. It will be apparent, however, that the various embodiments of the present invention may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail, in order to avoid unnecessarily occluding, obscuring, or obfuscating embodiments of the present invention.

SUMMARY

Example embodiments described herein relate to sign prediction of transform coefficients in image and video coding. Embodiments to improve sign prediction include: selection of top and left neighbors based on an image continuity check, the intra mode of the current coded unit (CU), the merge motion vector, or adaptive motion vector prediction, sign prediction based on residue domain of current CU or neighbor CUs, sign prediction based on approximated reconstruction samples, reducing the number of selected coefficients for sorting, simplifying the sequential search cost, and by combining sign prediction with sign data hiding.

Data Sign Prediction and Coding in Video Coding

In traditional video coding the signs of residual coefficients may be transmitted uncompressed and may account for about 10% of the bitrate in compressed streams. Sign prediction is aimed at increasing compression efficiency by reducing the bit-overhead for residue signs. A sign that has been predicted is no longer signaled uncompressed in the bitstream, but it is replaced by a coded “residual,” signaled using an associated arithmetic coding (e.g., CABAC) context, which indicates if the prediction was correct or not.

Sign Prediction Algorithm

As described in Ref. [2-3], the basic idea of the coefficient sign prediction method is to calculate reconstructed residuals for both negative and positive sign combinations for applicable transform coefficients, and then select the hypothesis that minimizes a cost function. To derive the best sign, the cost function is defined as a discontinuity measure across block boundaries, as shown in FIG. 1. The cost function is measured for all hypotheses, and the one with the smallest cost is selected as a predictor for the coefficient signs.

The cost function is defined as a sum of absolute second derivatives in the residual domain for the above rows and the left columns as follows:

where, w denotes width of prediction, h denotes height of prediction, R denotes reconstructed neighbors (105), P denotes prediction of the current block (110), and r is the residual hypothesis. The term (−R−1+2R0−P1) can be calculated only once per block and only the residual hypothesis is subtracted.

When predicting n signs in a transform unit (TU), the encoder and decoder perform n+1 partial inverse transformations and 2n border reconstructions corresponding to the 2n sign combination hypotheses, with a border-cost measure for each. These costs are examined to determine sign prediction values, and the encoder transmits a sign residual for each predicted sign indicating whether the prediction for that sign is correct or not using two additional CABAC contexts. The decoder reads these sign residuals, computes hypothesis reconstructions to compute the predictors being used, and then uses the received residuals to determine the correct signs.

Ref [4] proposed to improve the sign prediction process with the following changes:

As reported in Ref. [5], the qIdx of the level depends on the DQ state and can be computed as follows:

qIdx values represent the absolute values of the dequantized coefficients. Sorting by “levels” may not give the best results because the levels do not accurately reflect the quantization due to using two quantizers.

The proposed sign prediction is rather complex and added non-trivial complexity to the hardware decoding pipeline. As described in Ref. [4]:

Due to the dependency of the inverse transform on the reconstructed pixels of the neighbor, there are stalls in the pipeline which are illustrated in FIG. 2A for an example of a hypothetical decoder pipeline. As depicted in FIG. 2A, the pipeline includes: entropy decoding, motion vector decoding and boundary strength (for deblocking filtering) derivation, inverse quantization, sign prediction and inverse transform, Inter prediction, Intra prediction and reconstruction, and loop filtering. Regular pipeline delays are indicated by “x.” The stalls are indicated by “S.” The pipeline shows the dependencies across the several modules assuming there is one coding unit (CU) in a virtual pipeline data unit (VPDU). In the case where the number of CUs in a VPDU is more than one, the dependency will be at a micro pipeline level with the stall duration varying with the size of the CU.

Embodiments presented here aim at improving the sign prediction process from different aspects:

Intelligent Selection of the Top/Left Neighbor for Image Continuity Check

Motivation: Improve the accuracy of prediction by intelligent selection of neighboring pixels

Proposal: The current algorithm tries to minimize the cost with respect to both the top neighbor and the left neighbor; however, depending on the scene characteristics, it is possible that the image continuity is true only in one direction. There are three cases for cost calculation: 1) using only the top neighbor; 2) using only the left neighbor; 3) using both the top and left neighbors.

Cost of Top Neighbors

Cost of Left Neighbors:

Cost per pixel of top neighbors=top_cost/w

Cost per pixel of left neighbors=left_cost/h

where w and h denote respectively width and height.

Different metrics can be chosen to make the decision among the three options, such as:

Intelligent Selection of the Top/Left Neighbor Based on Intra Mode of the Current CU

Motivation: To improve the accuracy of prediction by using the intra prediction mode for intelligent selection of neighbors

Proposal: For intra blocks, the intra mode gives an indication of the prediction direction of the pixel values. For example. if the intra mode is in the vertical direction, the top pixels may be more reliable for calculating the cost. As a further improvement to this, the cost can also be calculated in the direction of the intra mode. For example, if the intra mode points to vertical 45 degrees, then the cost in equation (1) can be calculated by taking into consideration neighbor pixels at an angle, as indicated by the arrows in FIG. 3. For other intra modes, with angles which are not pointing to full pixel locations, the pixel values of the neighbor need to be interpolated to generate sub pixel positional values for cost calculation. One may apply any known pixel interpolation techniques known in the art.

Intelligent Selection of the Top/Left Neighbor Based on the Merge Motion Vector (MV) or Adaptive Motion Vector Prediction (AMVP) MV Direction

Motivation: Improve the accuracy of prediction by using the motion vector list information of the current CU for intelligent selection of neighbors. If the motion information of the current CU and the neighbor CUs are similar, then they are most likely to pass the continuity check.

Proposal: The motion vector list of the current CU consists of various spatial and temporal candidates. The current CU can prioritize the neighbor whose motion information is similar to the motion information of the current CU.

For example, in FIG. 4, the motion vectors of the current CU (405) and the left neighbor (415) are pointing to the left, and the motion vector of the top neighbor (410) is pointing in top left direction. It is more likely that left and current CU belong to similar regions and would show better image continuity. If there are multiple partitions on the left and right neighbors, certain guidelines can be followed to factor the MVs from the left and top. For example, as depicted in FIG. 5, for the current CU (510), motion vectors corresponding to pixel areas A, B, C and D can be considered for neighbor selection.

Proposals Aiming at Resolving Pipeline Issues and Complexity Reduction

Sign Prediction Based Only on Residue Domain of the Current CU

Motivation: Remove dependency on neighbor reconstructed pixels. Due to the removal of neighbor samples, locations for residual hypothesis template can be extended in many ways if required, including (but not limited to):

Proposal: Select the sign prediction hypothesis which meets one or more of the following criteria:

Removing Dependency on Neighbor Reconstructed Pixels by Using the Residue Data of Neighbor Pixels

Motivation: Remove dependency on neighbor reconstructed pixels

Proposal: Select the sign prediction hypothesis which meets one or more of the following criteria on the spatial domain residue values of the current CU and the neighbor CUs. This solution is proposed for inter CUs with at least one neighbor as inter.

Use of Constrained Top or Left Neighbor for Cost Calculation so as to Reduce the Dependency of Immediate Neighbor in Decoding Order

Motivation: The current solution for sign prediction needs the immediate top and left neighbor reconstructed pixels. This introduces a strong pipeline dependency in the decoding pipeline as the reconstructed pixels of the immediate neighbor is needed for computing the sign values of the current TU.

Proposal: The TUs are decoded in Z-scan order. The proposal is to use the neighbor based on the following criterion

Sign Prediction Using Approximated Reconstruction Samples of Top and Left Inter Prediction CUs

Motivation: Remove dependency on neighbor reconstructed pixels

Proposal: The need for reconstructed pixels of the immediate neighbors introduces a strong pipeline dependency in the decoding. Therefore, in an embodiment, one may use approximated reconstruction pixels of the neighbors for sign prediction. The approximated reconstruction samples of the top and left inter CUs can be calculated using (i) prediction samples and (ii) approximate residue samples of the 2 rows and 2 columns of the neighbor CUs using inverse transform lookup tables (The required residual samples have to be stored during the sign prediction of the respective CUs). This method will have pipeline dependency only on the prediction samples. In another embodiment, the filtered version (by linear or non-linear filtering) of prediction samples can be used to approximate neighboring reconstructed pixels.

This skips the complex serialized dependency on intraPred+Recon (last stage of reconstruction). This method cannot be applied to CUs where the neighbor is an intra and therefore this method will lose benefit for intra slices or where the neighbors for a CU are intra.

The pipeline dependency with this change is shown in FIG. 2B. Compared to FIG. 2A, the pipeline dependency on Intra prediction and reconstruction (IntraPredRec) has been removed. The delay slots are reduced from 2 to 1 as the dependency is restricted to the inter-prediction of the neighboring pixels. For intra CUs or CUs with Intra neighbors, the method suggested earlier can be applied.

Reduction of the Sorting Complexity of the Coefficients to be Selected for Sign Prediction by Reducing Max Area

Motivation: In the current solution, the area for sign prediction is defined as a square region of size 4×4, 8×8, 16×16, or 32×32. Note than an area of 32×32 would require us to sort an array of size 1024, which adds significant complexity at TU-level processing. This also increases the LUT size significantly.

Proposal: Starting at 32×32, reduce the area for sign prediction by using one or more of the following methods:

These techniques would help define a region (801-a) in the upper left corner of the transform unit (TU) (801), where the high amplitude coefficients are more likely located. FIG. 8 depicts an example of such processing for a 32×64 TU (801). In FIG. 8, A) depicts the results of the current enhanced compression model (ECM) software in JVET, and B) to D) depict the results from proposals a) to c), respectively.

Reduction of the Sorting Complexity of the Coefficients to be Selected for Sign Prediction by Selecting Coefficients Based on Threshold Values

Motivation: In the current solution, the area for sign prediction is defined as a square region of size 4×4, 8×8, 16×16, or 32×32; however, an area of 32×32 would requires us to sort an array of size 1024, which adds significant complexity at TU level processing. This also increases the LUT size significantly

Proposal: Instead of sorting all coefficients, select coefficient based on absolute levels (e.g., qIdx) in specific order (useful when the number of coded coefficients are much larger than a max number of sign prediction coefficients threshold (e.g., 8)

Simplification of Cost of the Sequential Search Process in Sign Prediction

Motivation: For simplicity, denote a dequantized transform coefficient as “coeff” or just coefficient. Sign prediction of the first dequantized transform coefficient (highest magnitude) is predicted first and then the real sign of first coefficient is used to predict the sign of subsequent coefficients. In the current process, all the hypothesis cost needs to be stored and then search the minimum cost in a range where real sign matches with the hypothesis. This proposal aims at reducing the storage cost and sequential operation of searching the minimum cost.

FIG. 7A, FIG. 7B, and FIG. 7C depict examples of the proposed dataflows for the three methods described here.

Combing Sign Prediction with Sign Data Hiding (SDH)

In Ref. [2], sign prediction is proposed to be combined with SDH. The basic idea of SDH is to omit the coding of the sign for one nonzero quantized coefficient and instead derive it from the parity of the sum of absolute values of all the quantized coefficients. SDH is applied on the basis of coefficient groups (CGs). In most cases, CG size is 4×4. If the difference between the scan indexes of the last and first nonzero level (in coding order) inside a CG is greater than 3, the sign for the last nonzero level of the CG is not coded but derived based on the sum of absolute values, where odd sums indicate negative values. In Ref. [2], when combing SDH with sign prediction, the order is to first perform the sign data hiding, then perform sign prediction on the remaining coefficients.

In an example embodiment, it is proposed to improve the coding efficiency by changing the rule which quantized coefficient should apply SDH. For example, in current implementations, since coefficients are sorted, one can apply SDH on the highest qIdx coefficient, then apply sign prediction on the remaining coefficients. In an alternative solution, based on the statistics, one can generate rules to predict the coefficients which have low accuracy when using sign prediction, then apply SDH on those coefficients.

SDH is applied on CGs (mostly 4×4) while sign prediction can be applied on variable block size, from 4×4 to 32×32. In another embodiment, one can change the sign prediction block size rule. For example, only allow 4×4 if SDH is used.

REFERENCES

Each one of the references listed herein is incorporated by reference in its entirety. The term JVET refers to the Joint Video Experts Team of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29.

Example Computer System Implementation

Embodiments of the present invention may be implemented with a computer system, systems configured in electronic circuitry and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA), or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and/or apparatus that includes one or more of such systems, devices or components. The computer and/or IC may perform, control, or execute instructions relating to sign prediction in image and video coding, such as those described herein. The computer and/or IC may compute any of a variety of parameters or values that relate to sign prediction in image and video coding described herein. The image and video embodiments may be implemented in hardware, software, firmware and various combinations thereof.

Certain implementations of the invention comprise computer processors which execute software instructions which cause the processors to perform a method of the invention. For example, one or more processors in a display, an encoder, a set top box, a transcoder, or the like may implement methods related to sign prediction in image and video coding as described above by executing software instructions in a program memory accessible to the processors. Embodiments of the invention may also be provided in the form of a program product. The program product may comprise any non-transitory and tangible medium which carries a set of computer-readable signals comprising instructions which, when executed by a data processor, cause the data processor to execute a method of the invention. Program products according to the invention may be in any of a wide variety of non-transitory and tangible forms. The program product may comprise, for example, physical media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, or the like. The computer-readable signals on the program product may optionally be compressed or encrypted.

Where a component (e.g. a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, reference to that component (including a reference to a “means”) should be interpreted as including as equivalents of that component any component which performs the function of the described component (e.g., that is functionally equivalent), including components which are not structurally equivalent to the disclosed structure which performs the function in the illustrated example embodiments of the invention.

EQUIVALENTS, EXTENSIONS, ALTERNATIVES AND MISCELLANEOUS