Removing blocking artifacts in video encoders

A method for video encoding is provided. The method comprises retrieving a first video frame comprising a plurality of pixel blocks; determining a rate distortion optimization (RDO) cost for a first prediction mode for a pixel block; determining a variance-bits ratio (VBR) of the pixel block; upon determining the VBR is greater than a predefined threshold, scaling the RDO cost for the first prediction mode based on a predefined scale factor; and selecting one of the first prediction mode and a second prediction mode for video encoding of the first video frame based on comparing the scaled RDO cost for the first prediction mode and a second RDO cost for the second prediction mode for the pixel block.

TECHNICAL FIELD

Examples of the present disclosure generally relate to video encoding.

BACKGROUND

Global video streaming accounts for around 60% of global online data traffic, and it is predicted that by 2020, video streaming will rise up to 82%. Recently, watching live streaming of video games has become very popular, with dedicated game streaming platforms having millions of subscribers. Video streaming applications are generally bandwidth hungry, and these services have flourished thanks to the improvements in existing network infrastructure, especially mobile data networks, in providing sufficient bandwidths. Recently, demand for newer formats like High Dynamic Range (HDR) and larger display sizes like Ultra High Definition (UHD) has resulted in further increase in demand for more bandwidth, hence putting pressure on existing data networks.

Efficient video compression mitigates the demand for higher bandwidths, especially compression using latest video codecs like VP9, H.264/AVC, and High Efficiency Video Coding (HEVC). With the ever increasing requirements for better quality and larger frame sizes, the video codecs are becoming increasingly complex and computing intensive. VP9 is one such video coding format that has been specially designed to compress larger frames like 1080p and beyond, more efficiently than previous codecs like H.264/AVC. VP9 focuses on meeting today's streaming video requirements and is claimed to provide 30% bitrate savings for a similar picture quality against previously popular video codec H.264/AVC. Recently, AOMedia Video 1 (AV1), a new video coding format succeeding VP9, claims further bitrate savings as compared to VP9. Given the rapid evolution of video codecs in terms of complexity and new feature adoption, implementing encoder IPs like VP9 in data centers for the streaming platforms over field programmable gate arrays (FPGAs) make a very viable use-case. Live streaming of videos requires real-time video encoding capabilities and with the given complexity of today's video codecs that's only possible with dedicated hardware acceleration IPs. However, real-time video encoding for live video streaming can impact visual presentation. That is, current real-time video encoding may introduce visual artifacts into the video image during streaming.

Currently, there is no specific prior-art known which addresses the problem for real-time video encoders using custom low-complexity methods to address visual artifacts (e.g., blocking artifacts) in real-time video streaming. Accordingly, what is needed is a solution for real-time video encoders to address blocking artifacts in video encoders.

SUMMARY

These and other aspects may be understood with reference to the following detailed description. One embodiment is a method for video encoders. The method includes retrieving a first video frame comprising a plurality of pixel blocks; determining a rate distortion optimization (RDO) cost for a first prediction mode for a pixel block; determining a variance-bits ratio (VBR) of the pixel block; upon determining the VBR is greater than a predefined threshold, scaling the RDO cost for the first prediction mode based on a predefined scale factor; and selecting one of the first prediction mode and a second prediction mode for video encoding of the first video frame based on comparing the scaled RDO cost for the first prediction mode and a second RDO cost for the second prediction mode for the pixel block.

Aspects of the present disclosure also provide apparatus, methods, processing systems, and computer readable mediums for performing the method described above.

DETAILED DESCRIPTION

Embodiments described herein describe techniques for removing blocking artifacts in video encoding. Such artifacts occur due to the choice of certain modes by the video encoder for blocks having a low bit-budget. Embodiments herein use a pixel blocks' content information, like variance and estimated encoded bits, to favor prediction modes that provide better visual appearance. Embodiments disclosed herein discuss the removal of blocking artifacts in video encoding formats such as VP9; however, the methods discussed are not limited to VP9, and include other types of video encoding formats.

In certain embodiments, the exemplary method for removing blocking artifacts in video encoding has negligible computational complexity in comparison to the complexity of overall video encoder. In certain embodiments, hardware resource-wise, a single multiplexer and a single multiplier can be used to process a complete pixel block.

FIG. 1is a flowchart100of video encoding, according to embodiments of the present disclosure. Again, while the present disclosure is not limited to VP9, VP9 is used herein as an exemplary video encoding format.

VP9 is a block based video coding format and so the video encoder using VP9 takes the raw video data (YUV) and breaks it down to pixel blocks (block102). The pixel blocks can include any number of pixels and can have any size. In certain embodiments, with a special consideration for larger frames sizes, the video encoder using VP9 supports coding of larger pixel block sizes which are 64×64 pixels (also called Coding Units (CU)). Accordingly, VP9 differs from H.264, which supports maximum block sizes of 16×16 pixel macroblocks. In some embodiments, the video encoder splits the video data into frames, slices, macroblocks, and down to pixel blocks. In certain embodiments, VP9 supports ⅛th pixel motion vector resolution and up to three reference frames. A Superblock can be sub-divided in a quad-tree structure into smaller pixel blocks of up to 8×8 pixels.

Once the video encoder has broken the raw video data into pixel blocks, the video encoder determines the rate distortion optimization (RDO) cost for each pixel block of each frame of the raw video data (block104). Video encoders use the RDO cost for selection of right combination of prediction modes for encoding the pixel blocks, because different pixel blocks can require different prediction modes. On one side, larger number of coding tools helps to improve the compression efficiency of video codecs, but at the same time it increases the complexity of implementation of encoders. Accordingly, the RDO cost helps in choosing the optimal combination of prediction modes and other coding tools to improve compression efficiency.

In certain embodiments, the RDO cost is based on a Lagrange multiplier method:
J=D+λ*R
where λ is a Lagrangian multiplier, D is distortion calculated as the mean of the squared difference between the reconstructed pixels and original pixels, R is the number of bits taken to encode residue coefficients and mode bits, and J is the RDO cost. More details about reconstructed pixels and residue coefficients are provided below.

Based on the RDO cost, the video encoder determines whether which type of prediction it will use within the pixel block (block106). In certain embodiments, the video encoder chooses the prediction mode with the lowest RDO cost. The video encoder decides between two prediction modes: intra prediction108and inter prediction110. In intra prediction108mode, the video encoder uses other pixels from other blocks within a video frame to predict the contents of the pixel block. In inter prediction110mode, the video encoder uses pixels from pixel blocks in other video frames to predict the contents of the pixel block. In certain embodiments, the video encoder can use different types of intra prediction108modes, and different types of inter prediction110modes. VP9 provides 10 intra prediction modes, 8 directional modes along with 2 non-directional prediction modes: DC and True Motion (TM) modes. A few of these intra prediction108modes are discussed with reference toFIG. 3b.

In certain embodiments, when the video encoder employs either inter prediction110or intra prediction108, the prediction operation results in residue (also called residual data). The video encoder takes the residue and transforms it into the frequency domain (block112) using 2-D DCT-like (Discrete Cosine Transform) transform. In some embodiments using VP9, the video encoder supports multiple size DCT-based transform blocks from 4×4 up to 32×32, where the largest transform for a pixel block is limited by the size of that pixel block. In some embodiments, the video encoder has an option to choose between three 8-Tap and a bilinear Motion compensation filters for sub-pixel motion estimation, where these filters are used for generating samples for motion vectors pointing to locations between pixels.

After the video encoder transforms the residue, the video encoder quantizes the coefficients that result from transforming the residue to the frequency domain (block114). In certain embodiments, lower distortion signifies lesser deviation from the reference frame (i.e., original video frame) to the current frame, thus leading to better quality. In comparison, fewer quantization coefficient bits signify better compression. Quantization of coefficients causes the difference between the reconstructed and original pixels. The video encoder determines the quantization step by the rate control algorithm of the encoder which is a key step for achieving target bitrates in video encoders.

For a pixel block with low bit budget, quantization step size is large which results in division of residue coefficients by large values thereby zeroing of most of the AC coefficients which are generally small in comparison to DC coefficient. Loss of AC coefficients mean low pass filtering of residue data, or flattening of residue data. In such cases, the video encoder predominantly determines the reconstruction of texture variation in that pixel block by the predicted pixels as the video encoder has mostly flattened the residue.

In certain embodiments, after quantization114, the video encoder entropy-codes the quantized coefficients (block116). In some embodiments, the video encoder uses VP9 and therefore, employs context-adaptive binary arithmetic coding (CABAC) as an entropy coding scheme.

In certain embodiments, the video encoder comprises an integrated video decoder. In such embodiments, the integrated video decoder decodes the quantized residue. In certain embodiments, the decoder reconstructs the pixels of a video frame by adding the decoded, inverse quantized (block118), and inverse transformed (block120) residue to the predicted pixels. Accordingly, in some embodiments, the reconstructed pixels122comprise the predicted pixel added with decoded residue (Reconstructed Pixel=Predicted Pixel+Decoded residue). In certain embodiments, the video encoder then has the same data as the decoder for video prediction.

For any real-time encoder, it is practically impossible to generate reconstructed pixels for every mode and also to do complete entropy encoding like CABAC in VP9 to generate actual bits taken. Most real-time encoders, such as a VP9 encoder, use custom RDO methods to reduce the computation cost (e.g., hardware cost). In such scenario, the video encoder measures distortion as a sum of squared differences between the actual and reconstructed transform coefficients (skipping Inverse-DCT) while estimating bits by using cost tables for coefficients and modes (Arithmetic entropy coding process is skipped). This approximation of the RDO cost is helpful in reducing the complexity though, but has an impact on Intra and Inter mode decision leading to problems with visual quality in certain cases. With computer-generated sequences like gaming videos and high motion sequences, the wrong choice of coding mode causes persistent blocking artifacts, which are discontinuities on the block boundaries. The combination of block level transforms and quantization processes causes these blocking artifacts. Further, these artifacts are more prominent when quantization step is higher, especially in low-bitrate scenarios. In certain embodiments, for streaming videos that require a constant bitrate, rate control process determines the quantization step.

FIG. 2illustrates video composition using a video encoder after the video encoder breaks the raw video data down to frames as shown inFIG. 1, according to embodiments of the present disclosure. Specifically,FIG. 2depicts a group of pictures (GOP)200forming a portion of a video. In certain embodiments, a GOP200is an arrangement of different types of pictures. In certain embodiments using VP9, a GOP200consists of two anchor frames (I-frame202, Alt-Ref frame204) at the boundary with Inter predicted frames206in between. The anchor frames can be I-frame202, which can be a past reference frame, and Alt-Ref frame204, which can be a future reference frame.

In some embodiments, the video encoder employs rate control algorithms that determine the structure of the GOP200, shown inFIG. 2. Rate control is an important block of video encoding because the rate control is responsible for achieving targeted bitrates and for maintaining video quality. Specifically, the video encoder controls how many bits to spend on a given frame.

In certain embodiments, the video encoder uses two-pass encoding (also known as multi-pass encoding) so that the video encoder maintains the quality of the video data during encoding. In the first pass of the two-pass encoding process, the video encoder generates frame-level statistics, such as the percentage of Inter coded pixel blocks, average motion vector costs, bits consumed by each frame etc., to determine the GOP200structure and frame-types and also their respective bit-budgets of the video data. These statistics represent the complexity of motion and texture in sequence. In the second pass of the two-pass encoding process, the video encoder uses the frame-level statistics to ensure the quality of the video data during encoding.

In certain embodiments, rate control also determines the quantization step114ofFIG. 1for a frame or a pixel block, depending upon the available bit budget, where the bit-budget is determined based on frame type and target bitrate. For a 1920×1080 frame size with 60 frames per second with a target bitrate of 4 Mbps, the average bit budget available per frame is: 4000000/60=66,667 bits. In certain embodiments, the video encoder modifies the budgeting depending upon the frame types. For example, the video encoder gives I-frame202and Alt-ref frame204, which are referenced by multiple frames, higher budgets (greater than 66,667 bits in this case). In comparison, the video encoder gives the frames which are less or not referenced a lower number of bits. In certain embodiments, for a video with very little or no motion, the video encoder places reference frames far apart and gives these reference frames a high bit budget, because slow motion implies very little change between the frames, so many frames can reference a single reference. A good quality reference frames provide better prediction data for rest of the frames thereby improving the overall quality of the GOP.

FIG. 3ais an illustration of inter prediction110applied to a frame of the GOP illustrated inFIG. 2, according to embodiments of the present disclosure.

As mentioned previously, when a video encoder uses inter prediction, the video encoder predicts a pixel block from pixels from other frames. In certain embodiments, the video encoder takes a block302afrom the current frame302and a block304afrom the reference frame304. The video encoder calculates the difference between these blocks302a,304aand gets the residue block306.

FIG. 3billustrates different intra prediction modes, which can be applied to a frame of the GOP illustrated inFIG. 2as compared to inter prediction ofFIG. 3a, according to embodiments of the present disclosure. As mentioned previously, when a video encoder uses intra prediction, the video encoder predicts a block of pixels from other pixels within the same frame. In some embodiments, intra prediction also uses previously decoded data in a frame.

FIG. 3aillustrates a vertical intra prediction310, horizontal intra prediction320, DC intra prediction330, and directional intra prediction340. With vertical intra prediction310, the video encoder extrapolates the contents of a pixel from the vertical neighboring pixels. With horizontal intra prediction320, the video encoder extrapolates the contents of a pixel from horizontal neighboring pixels. With DC intra prediction330, the video encoder estimates the DC coefficient of the block, and the DC coefficient is the average of all pixels in a block (e.g., the block formed from A-D by I-L). With directional intra prediction340, the video encoder performs a planar prediction by generating a linear plane (e.g., arrow from (D,I) to (A,L)) estimated from the neighboring pixels (i.e., extrapolating from neighboring pixels on the same plane).

As shown inFIG. 3b, intra prediction modes use a set of limited boundary pixels from top and left neighboring pixel blocks for prediction. The video encoder extends these boundary pixels as predictors for large numbers of pixels. For example, for a 16×16 pixel block, the video encoder uses only 16 or 17 pixels for directional prediction modes, while for DC and TM Intra modes, the video encoder uses 31 pixels as a predictor for 256 pixels. The repetition of predictors creates a smoothening effect in the direction of prediction, in cases where the prediction data predominantly determines the reconstructed pixels.

This smoothening of texture in pixel blocks along with the discontinuities at pixel block boundaries (due to transform and quantization) creates blocking artifacts as shown inFIG. 4. Whereas, in Inter prediction the predicted pixels are taken from a block in reference frames, that gives unique predictor for each pixel. For a 16×16 pixel block with 256 pixels, there are 256 different predictors. Blocking artifacts are generally not prominent in I-frames, because in a GOP structure, video encoders assign I-frames much higher bit-budgets in comparison to Alt-Ref or P-frames. When the bit-budget is sufficient, the coded residue coefficients have enough AC components, hence there is enough residual data for reconstruction.

FIG. 4is a screenshot of a video frame400illustrating blocking artifacts as a result of using intra prediction instead of inter prediction as described inFIGS. 3a(showing inter prediction) and3b(showing intra prediction), according to embodiments of the present disclosure.

As mentioned with respect toFIG. 3b, the smoothening of texture in pixel blocks and discontinuities at pixel block boundaries create blocking artifacts.FIG. 4illustrates 3 circled areas depicting blocking artifacts. In each circled area ofFIG. 4, the image appears blocky because of the discontinuities at pixel block boundaries, while inside the pixel block, the video encoder smoothed out the texture using intra prediction mode.

As described above, the video encoder selects between Inter and Intra prediction modes via the RDO process. Usage of custom RDO methods to reduce complexity impacts the choice of most appropriate mode. With computer-generated sequences, like gaming videos and high motion sequences, video frames persistently had blocking artifacts while encoding with VP9 encoder. The analysis of these video frames revealed that the video encoder used intra mode prediction with low bit budgets on most of the problematic blocks, causing the smoothening of blocks as discussed above.

Accordingly, the issues that occur downstream in the video-encoding process stem from the selection between inter prediction and intra prediction. Consequently, the method disclosed herein acts to remove and prevent blocking artifacts in video encoding.

FIG. 5illustrates example operations performed by a video encoder in order to remove the blocking artifacts shown inFIG. 4, according to embodiments of the present disclosure.

In certain embodiments, the method for removing blocking artifacts in video encoding includes retrieving a first video frame having a plurality of pixel blocks. For each pixel block of the first video frame, the video encoder determines that a RDO cost for a video prediction mode, and the video encoder also determines a variance-bits ratio (VBR). Upon determining the VBR is greater than a predefined threshold, the video encoder scales the RDO cost for the video prediction mode based on a predefined scale factor. Once the video encoder has scaled the RDO cost, the video encoder selects either the video prediction mode or another prediction mode based on comparing the scaled RDO cost for the first prediction mode and a second RDO cost for the second prediction mode for the pixel block.

Operations500begin, at502, when the video encoder retrieves a first video frame comprising a plurality of pixel blocks. The video frame can be any type of Inter coded frame like a P-frame, or an Alt-Ref frame, or a B-frame and the video frame can have any number of pixel blocks. Each pixel block can have any number of pixels, and can have any size (i.e., dimension). In certain embodiments, the video encoder is using a block-based video encoding format (e.g., VP9).

At step504, operations500continue with the video encoder determining an RDO cost for a first prediction mode for a pixel block. In certain embodiments, the first prediction mode is intra prediction.

At step506, operations500continue with the video encoder determining a variance-bits ratio (VBR) of the pixel block. In certain embodiments, the VBR is the ratio between the variance of source pixels in a pixel block and bits for coding the pixel block with intra prediction mode. In some embodiments, the video encoder estimated the bits for coding the pixel block when the video encoder calculated the RDO cost for the first prediction mode (e.g., intra prediction mode). A small VBR ratio indicates that the bits allocated for intra prediction mode are enough to efficiently code the pixel block with a given variance. A high VBR ratio indicates that the bits allocated for intra prediction mode are not enough to efficiently code the pixel block with a given variance.

In certain embodiments, blocks having texture but coded with Intra mode and lesser number of bits generally cause such visual artifacts (e.g., blocking artifacts). In such embodiments, visual artifacts occur when displaying texture information in a pixel block. Because texture information requires variance inside and between pixel blocks, the video encoder needs to calculate how much variance is needed in the pixel block to show the texture information and how many bits are required for coding the pixel block. Variance is a measurement of the differences between pixel values (e.g., YUV data) within the pixel block. In some embodiments, the video encoder calculates variance based on the source pixels of the I-frame. In some embodiments, variance represents texture. In some embodiments, bits for coding the pixel block are determined during quantization. With the variance and bits for coding the pixel block, the encoder determines the VBR of the pixel block for intra prediction mode.

At step508, operations500continues with the video encoder scaling the RDO cost for the first prediction mode based on a predefined scale factor upon determining the VBR is greater than the predefined threshold. In certain embodiments, the video encoder determines whether the VBR is greater than a predefined threshold.

In certain embodiments, the predefined threshold is a VBR threshold based on the quantization parameter (QP) for a pixel block, which can be determined by a rate control module. That is, the video encoder uses the QP of the pixel block to determine which VBR threshold to use to compare against the calculated VBR. So, the video encoder uses a pre-defined QP-to-VBR threshold conversion for the comparison against the calculated VBR. Blocks with VBR above the threshold are deemed to have blocking artifacts, and blocks with a VBR below the threshold are deemed to not have blocking artifacts. In some embodiments, the video encoder comprises a predefined look-up table of VBR thresholds based on QP of the pixel block. Table 1 is an example predefined look-up table of VBR thresholds based on QP. The method disclosed herein is not limited to Table 1 and the information included therein.

Upon the video encoder determining that the VBR is greater than the predefined threshold, the video encoder applies a predefined scale factor to the RDO cost of the first prediction. In certain embodiments, the video encoder scales D by the predefined scale factor before adding λ*R when calculating the RDO cost for applying intra prediction to the pixel block. If the VBR is less than or equal to the predefined threshold, the video encoder does not scale the RDO cost and applies the unmodified Lagrange multiplier method. In certain embodiments, this step can be expressed in the following if-else statements:
if(VBR>ThresholdVBR)
JINTRA=D*scale_factor+λ*R
else
JINTRA=D+λ*R
In certain embodiments, scale_factor is the predefined scale factor that the video encoder uses to scale the RDO cost. In some embodiments, the predefined scale factor was greater than 1. For example, using gaming sequences, a video encoder used a predefined scale factor of 1.25 to remove the blocking artifacts and without significantly impacting BD-rate (Bjontegaard Delta rate) (0.22% loss only).

At step510, operations500continues with the video encoder selecting one of the first prediction mode and a second prediction mode based on comparing the scaled RDO cost for the prediction mode and a second RDO cost for the second prediction mode for the pixel block. In some embodiments, the second prediction mode is inter prediction. In certain embodiments, the predefined scale factor increases the D, which also increases the RDO cost for the first prediction mode. In some embodiments, the predefined scale factor increases the RDO cost for intra prediction mode, and thereby affects the selection between the intra prediction mode and inter prediction mode.

In certain embodiments, the video encoder modifies the RDO cost to favor Inter prediction modes for visually impacted blocks in P-frames. In certain embodiments, for pixel blocks with VBRs above the predefined threshold, the video encoder amplifies the distortion of Intra prediction mode by artificially scaling the RDO cost for intra mode prediction before the video encoder compares the RDO costs of intra mode prediction and inter mode prediction.

FIGS. 6aand 6bare screenshots600,610of a video frame with and without blocking artifacts, according to embodiments of the present disclosure.FIG. 6ais a reproduction ofFIG. 4to provide a contrast in the video frame image toFIG. 6b, which illustrates the same video frame image without blocking artifacts. As illustrated inFIG. 6b, by employing the method disclosed herein, the video encoder using the disclosed techniques has removed and prevented blocking artifacts in the video encoding, especially in areas of the video frame where the variance is high but bit-budgets are low.

The method discussed herein removes blocking artifacts from the streams.FIG. 6ashows an impacted frame with blocking artifacts, andFIG. 6bshows the blocks using intra prediction mode that resulted in blocking artifacts getting coded as Inter prediction mode and thus appears without the blocking artifacts. Also, for Intra prediction mode blocks, which do not have blocking artifacts and did not satisfy the condition VBR>ThresholdVB, the video encoder using the disclosed techniques does not affect these blocks. Accordingly, their prediction mode selection is not impacted and they continue to be coded with Intra prediction mode.

FIG. 7A-Lillustrates different graphs700of BD-rates from different video games after removal of blocking artifacts from video frames, such as the game illustrated inFIGS. 6aand 6b, according to embodiments of the present disclosure.

In certain embodiments, computational complexity of the method disclosed herein is very minimal in comparison to the overall complexity of video encoding. In one embodiment, the impact on objective quality measure in BD-rate metric is 0.22% loss. In one embodiment, a BD-rate loss of 0.22% is acceptable given the videos are visually more appealing and do not include blocking artifacts. The BD-rate curves for 12 different gaming sequences are shown inFIG. 8.