System and method for encoding video using variable loop filter

A system and method for encoding and decoding a video image having a plurality of frames is disclosed. The method includes a loop filter, which is used to reduce blocking artifacts in reconstructed frames. A quality value is determined for each block in the reconstructed frame and each associated predictor block. A variable incremental loop filter strength is calculated for each inter-predicted block using at least one of the quality values. The variable incremental loop filter strength is used during encoding by the loop filter to filter the reconstructed frame.

TECHNICAL FIELD

The present invention relates in general to video encoding and decoding.

BACKGROUND

An increasing number of applications today make use of digital video signals for various purposes including, for example, business meetings between people in remote locations via video conferencing, high definition video entertainment, video advertisements, and sharing of user-generated videos. As technology is evolving, users have higher expectations for video quality and resolution even when video signals are transmitted over communications channels having limited bandwidth.

To realize transmission of higher quality video while limiting bandwidth consumption, a number of video compression schemes have been developed—including formats such as VPx, promulgated by Google Inc. of Mountain View, Calif., and H.264, a standard promulgated by ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG), including present and future versions thereof. H.264 is also known as MPEG-4 Part 10 or MPEG-4 AVC (formally, ISO/IEC 14496-10). These schemes may denote each image of a video as a frame, and subdivide each frame into groups of pixels called macroblocks or blocks.

These compression schemes may use prediction techniques to minimize the amount of data required to transmit video signals by using, for example, the spatial and temporal coherences in video information. Prediction techniques can allow for multiple past transmitted frames and future frames to be used as potential reference frame predictors for macroblocks in a frame. More recently, some predication techniques synthesize predictive reference frames that are not necessarily displayed during the decoding process, such as shown, for example, by U.S. Application Publication No. 2010/0061461, assigned to Google, Inc.

Many prediction techniques use block based prediction and quantized block transforms. The use of block based prediction and quantized block transforms can give rise to discontinuities along block boundaries during encoding. These discontinuities (commonly referred to as blocking artifacts) can be visually disturbing and can reduce the quality of the decoded video and the effectiveness of the reference frame used as a predictor for subsequent frames. These discontinuities can be reduced by the application of a loop filter.

A loop filter is typically applied to a reconstructed frame or a portion of a reconstructed frame at the end of the decoding process. A loop filter is typically used to reduce blocking artifacts. Once a reconstructed frame is processed by the loop filter, it may be used as a predictor for subsequent frames. Some conventional loop filters apply different filtering strengths to different block boundaries. For example, some compression systems vary the strength of the loop filter based on, for example, whether the block has been inter-coded or intra-coded. Other compression systems apply a filter strength from a set of discrete filter strengths based on, for example, motion vector strength and the type of reference frame predictor used, such as shown by U.S. Application Publication No. 2010/0061645.

SUMMARY

Systems and methods for encoding a video signal are disclosed. In accordance with one aspect of the disclosed embodiments, a method and apparatus is provided for encoding a video signal having at least one frame, each frame having at least one block. The method includes the steps of (1) identifying a current reconstructed block in a frame, the current reconstructed block having a current quality value, (2) identifying a predictor block that is associated with the current reconstructed block having a predictor quality value, (3) determining a loop filter strength for the current reconstructed block based at least on the current quality value and the predictor quality value, and (4) encoding using the loop filter strength

In another aspect of the disclosed embodiments, a method is provided for decoding frames of compressed video information, each frame including a plurality of blocks. The method includes the steps of (1) reconstructing at least two of a plurality of blocks in a frame, each reconstructed block having a base loop filter strength and a quality value, (2) determining a predictor block used to reconstruct at least one of the plurality of reconstructed blocks, each predictor block having a quality value, (3) determining a variable incremental loop filter strength for each reconstructed block encoded using one of the determined predictor blocks based on the quality value of the reconstructed block and the quality value of the reconstructed block's associated predictor block, (4) determining a loop filter strength for at least one reconstructed block by using at least one of the reconstructed block's variable incremental loop filter strength and the reconstructed block's base loop filter strength, and (5) filtering the edge of at least one reconstructed block using the at least one reconstructed block's calculated loop filter strength.

In another aspect of the disclosed embodiments, an apparatus is provided for filtering blocking artifacts between adjacent blocks reconstructed from a frame of compressed video information. The apparatus includes a memory and a processor configured to (1) identify a current block stored in the memory having a current quality value, (2) identify a predictor block that is associated with the current block, the predictor block having a predictor quality value, (3) determine a loop filter strength for the current block based at least on the current quality value and the predictor quality value, and (4) filter the current block using the loop filter strength.

DETAILED DESCRIPTION

FIG. 1is a diagram a typical video bitstream10to be encoded and decoded. Video coding formats, such as VP8 or H.264, provide a defined hierarchy of layers for video stream10. Video stream10includes a video sequence12. At the next level, video sequence12consists of a number of adjacent frames14, which can then be further subdivided into a single frame16. At the next level, frame16can be divided into a series of blocks or macroblocks18, which can contain data corresponding to, for example, a 16×16 block of displayed pixels in frame16. Each macroblock can contain luminance and chrominance data for the corresponding pixels. Macroblocks18can also be of any other suitable size such as 16×8 pixel groups or 8×16 pixel groups.

FIG. 2is a block diagram of a video compression system in accordance with one embodiment. An encoder20encodes an input video stream10. Encoder20has the following stages to perform the various functions in a forward path (shown by the solid connection lines) to produce an encoded or a compressed bitstream24: a temporal filter stage40, an intra/inter prediction stage26, a transform stage28, a quantization stage30and an entropy encoding stage32. Encoder20also includes a reconstruction path (shown by the dotted connection lines) to reconstruct a frame for encoding of further macroblocks. Encoder20has the following stages to perform the various functions in the reconstruction path: a dequantization stage34, an inverse transform stage36, a reconstruction stage37and a loop filtering stage38. Other structural variations of encoder20can be used to encode input video stream10.

Encoder20includes a temporal filter40, which as shown inFIG. 2can be included with the intra/inter prediction stage26. Temporal filter40can be used to synthesize a reference or alternative predictor frame. The parameters of the temporal filter40can be adjusted to create a reduced-noise frame for use as a predictor during the encoding process. This adjustment process can permit the filtering to take into account contextual information (such as, for example, encoding modes) and other input to establish the degree of correlation between adjacent frames to filter noise from a common underlying signal. The process can be applied at the full-frame, macro-block or any other segmentation of the frame where the degree of spatial correlation between frames can be established.

When input video stream10is presented for encoding, each frame16within input video stream10is processed in units of macroblocks. At intra/inter prediction stage26, each macroblock can be encoded using either intra-frame prediction (i.e., within a single frame) or inter-frame prediction (i.e. from frame to frame). In either case, a prediction macroblock can be formed. In the case of intra-prediction, a prediction macroblock can be formed from samples in the current frame that have been previously encoded and reconstructed. In the case of inter-prediction, a prediction macroblock can be formed from samples in one or more previously constructed reference frames as described in additional detail herein.

Next, still referring toFIG. 2, the prediction macroblock can be subtracted from the current macroblock at stage26to produce a residual macroblock (residual). Transform stage28transforms the residual into transform coefficients in, for example, the frequency domain, and quantization stage30converts the transform coefficients into discrete quantum values, which are referred to as quantized transform coefficients or quantization levels. The quantized transform coefficients are then entropy encoded by entropy encoding stage32. The entropy-encoded coefficients, together with the information required to decode the macroblock, such as the type of prediction used, motion vectors, and quantizer value, are then output to compressed bitstream24.

The reconstruction path inFIG. 2is present to ensure that both encoder20and a decoder42(described below) use the same reference frames to decode compressed bitstream24. The reconstruction path performs functions that are similar to functions that take place during the decoding process that are discussed in more detail below, including dequantizing the quantized transform coefficients at dequantization stage34and inverse transforming the dequantized transform coefficients at an inverse transform stage36in order to produce a derivative residual macroblock (derivative residual). At reconstruction stage37, the prediction macroblock that was predicted at intra/inter prediction stage26can be added to the derivative residual to create a reconstructed macroblock. A loop filter38can then be applied to the reconstructed macroblock to reduce distortion such as blocking artifacts.

Other variations of encoder20can be used to encode compressed bitstream24. For example, a non-transform based encoder can quantize the residual signal directly without transform stage28. Or the temporal filter40might not be present. In another embodiment, an encoder may have quantization stage30and dequantization stage34combined into a single stage.

The encoding process shown inFIG. 2can include two iterations or “passes” of processing the video data. The first pass can be carried out by encoder20using an encoding process that is less computationally intensive, and that gathers and stores information about input video stream10for use in the second pass. In the second pass, encoder20uses this information to optimize final encoding of compressed bitstream24. For example, encoder20may use this information to select parameters for encoding, locating key-frames and selecting coding modes used to encode macroblocks18, and allocating the number of bits to each frame. The output of the second pass can be final compressed bitstream24.

FIG. 3is a block diagram of a video decompression system or decoder42to decode compressed bitstream24. Decoder42, similar to the reconstruction path of the encoder20discussed previously, includes the following stages to perform various functions to produce an output video stream44from compressed bitstream24: an entropy decoding stage46, a dequantization stage48, an inverse transform stage50, an intra/inter prediction stage52, a reconstruction stage54, a loop filter stage56and a deblocking filtering stage58. Other structural variations of decoder42can be used to decode compressed bitstream24.

When compressed bitstream24is presented for decoding, the data elements within compressed bitstream24can be decoded by entropy decoding stage46(using, for example, Context Adaptive Binary Arithmetic Decoding) to produce a set of quantized transform coefficients. Dequantization stage48dequantizes the quantized transform coefficients, and inverse transform stage50inverse transforms the dequantized transform coefficients to produce a derivative residual that can be identical to that created by the reconstruction stage in the encoder20. Using header information decoded from the compressed bitstream24, decoder42can use intra/inter prediction stage52to create the same prediction macroblock as was created in encoder20. At the reconstruction stage54, the prediction macroblock can be added to the derivative residual to create a reconstructed macroblock. The loop filter56can be applied to the reconstructed macroblock to reduce blocking artifacts. Deblocking filter58can be applied to the reconstructed macroblock to reduce blocking distortion, and the result is output as output video stream44.

Other variations of decoder42can be used to decode compressed bitstream24. For example, a decoder may produce output video stream44without deblocking filtering stage58.

Referring again to quantization stage30, the process of quantization represents the range of transform coefficient values with a finite set of states, which as discussed previously, can be referred to as quantized transform coefficients or quantization levels. Some compression systems use, for example, a scalar quantization process, which can perform dividing operations on the transform coefficient values. The divisor used to perform the dividing operations affects the number of bits used to represent the image data as well as the quality of the resulting decoded image. Generally, when the divisor is set to a high value, the quantization process produces higher compression but also deteriorates the quality of the image. Conversely, setting the divisor to a low value results in improved image quality but less compression. During decoding, a corresponding dequantization process in. for example, dequantization stage48, can perform multiplication operations using the same divisor value to convert the quantized transform coefficients to dequantized transform coefficients.

Referring again to encoder20, video encoding methods compress video signals by using lossless or lossy compression algorithms to compress each frame or blocks of each frame of a series of frames. As can be implied from the description above, intra-frame coding refers to encoding a frame using data from that frame, while inter-frame coding refers to predictive encoding schemes such as schemes that comprise encoding a frame based on other so-called “reference” frames. For example, video signals often exhibit temporal redundancy in which frames near each other in the temporal sequence of frames have at least portions that match or at least partially match each other. Encoders can take advantage of this temporal redundancy to reduce the size of encoded data by encoding a frame in terms of the difference between the current frame and one or more reference frames.

As described briefly above, many video coding algorithms first partition each picture into macroblocks. Then, each macroblock can be coded using some form of predictive coding method. Some video coding standards use different types of predicted macroblocks in their coding. In one scenario, a macroblock may be one of three types: 1) Intra (I) macroblock that uses no information from other pictures in its coding; 2) Unidirectionally Predicted (P) macroblock that uses information from one preceding picture; and 3) Bidirectionally Predicted (B) macroblock that uses information from one preceding picture and one future picture.

To facilitate higher quality compressed video, it is helpful to have the best matching reference frame in order to have the smallest difference to encode, which generally results in a more compact encoding. Reference frames are based on past frames, future frames, or an intra-frame so that the encoder can find the best matching block to use in the predictive process as shown in, for example, U.S. Application Publication No. 2005/0286629. Reference frames can also be based on synthesized or constructed frames that are not shown to the end user after decoding, as shown in, for example, U.S. Application Publication no. 2010/0061461. Such constructed reference frames are referred to herein as “alternative reference frames.” The alternative reference frames are constructed from macroblocks, which are referred to herein as “alternative reference blocks.”

The method of constructing a reference frame can include selecting the target frame and using temporal filter40(FIG. 2) to remove video noise from several source frames centered on that target frame.

FIG. 4, which is a schematic diagram illustrating the reconstruction37and loop filter38as shown in the encoder20ofFIG. 1is now described. First shown is a current frame70which is a derivative residual, such as is output by inverse transform36of the encoder. Included in current frame70is a current block72. Current block72is encoded by, for example, inter-frame prediction or intra-frame prediction. If current block72is inter-frame predicted, it will refer to a reference block78that is included in a reference frame76as its predictor block. If current block72is intra-frame predicted, it will refer to a second block74in the current frame70as its predictor block.

Current block72and its predictor block (either second block74or predictor block78) are then combined in reconstruction stage37of the encoder to produce a reconstructed block82in a reconstructed frame80. Reconstructed block82and reconstructed frame80is then processed by loop filter38of the encoder20.

Current block72has an associated current quality value (Qc)84and reference block78also has an associated predictor quality value (Qp)86. Either quality value may be stored or obtained at a resolution of per frame, per group of blocks, or per block. Either quality value may refer to its block's quantization level, such as the divisor applied in quantization stage30as described earlier. Alternatively, the quality values may refer to any way of representing the degree of loss of data in a block in a lossy compression scheme. Other ways of representing quality include the bit rate of the compression, the difficulty of compression, or the actual loop filter strength used to previously encode and decode the predictor block.

Current block72also has an associated baseline filter strength (Sc)88. The baseline filter strength (Sc)88indicates the strength of the loop filter for that block before any incremental adjustment. The baseline filter strength (Sc)88may be defined by either the current block72or the current frame70. Alternatively, the baseline filter strength (Sc)88may be defined independently of the current block72or the current frame70by encoder20.

This embodiment of loop filter38takes as input per reconstructed block82, the current quality value (Qc)84, the predictor quality value (Qp)86, and the baseline filter strength (Sc)88. The loop filter38calculates a loop filter strength for each block by modifying the baseline filter strength (Sc)88using at least the current quality value (Qc)84and the predictor quality value (Qp)86.

Once loop filter38processes each reconstructed block82in reconstructed frame80, the loop filter38filters the entire reconstructed frame80using the calculated loop filter strengths. The resulting output from loop filter38is a decoded frame90, which includes a decoded block92that has been loop filtered using the loop filter strength calculated above. But in an alternative embodiment, it is possible for less than the entire reconstructed frame80to be filtered at a given time.

FIG. 5, which is a schematic diagram illustrating the reconstruction54and loop filter56as shown in the decoder42ofFIG. 2is now described. In this embodiment, the reconstruction and loop filter as implemented in the decoder42is very similar to that of the encoder20as shown inFIG. 4. As explained previously, loop filter38, as a part of the reconstruction path of encoder20, produces similar output as loop filter56.

First shown is a current frame100which is a derivative residual, such as is output by inverse transform50of the decoder. Included in current frame100is a current block102. Current block102was encoded by, for example, inter-frame prediction or intra-frame prediction. If current block102is inter-frame predicted, it will refer to a reference block108that is included in a reference frame106as its predictor block. If current block102is intra-frame predicted, it will refer to a second block104in the current frame100as its predictor block.

Current block102and its predictor block (either second block104or predictor block108) is then combined in reconstruction stage54of the decoder to produce a reconstructed block112in a reconstructed frame110. Reconstructed block112and reconstructed frame110is then processed by loop filter56of the decoder42.

Current block102has an associated current quality value (Qc)114and reference block108also has an associated predictor quality value (Qp)116. Either quality value may refer to its block's quantization level, such as indicated by the multiplier applied in dequantization stage48of the decoder described earlier. Alternatively, the quality values may refer to any way of representing the degree of loss of data in a block in a lossy compression scheme. Other ways of representing quality include the bit rate of the compression, the difficulty of compression, or the base loop filter strength associated with the current block.

Current block102also has an associated baseline filter strength (Sc)118. The baseline filter strength indicates the strength of the loop filter for that block before any incremental adjustment. The baseline filter strength may be defined by either the current block102or the current frame100. Alternatively, the baseline filter strength may be defined independently of the current block102or the current frame100by decoder42.

This embodiment of loop filter56takes as input per reconstructed block112, the current quality value (Qc)114, the predictor quality value (Qp)116, and the baseline filter strength (Sc)118. The loop filter56calculates the loop filter strength for each block by modifying the baseline filter strength (Sc)118using at least the current quality value (Qc)114and the predictor quality value (Qp)116.

Once loop filter56processes each reconstructed block112in reconstructed frame110, the loop filter56filters the entire reconstructed frame110using the calculated loop filter strengths. The resulting output from loop filter56is the decoded frame120, which includes the decoded block122that has been loop filtered using the loop filter strength calculated above. But in an alternative embodiment, it is possible for less than the entire reconstructed frame110to be filtered at a given time.

An embodiment of a loop filter to calculate and apply variable loop filter strengths is now disclosed. The embodiment below is described with respect to loop filter38of the encoder, but it may also be implemented in an equivalent fashion with respect to loop filter56of the decoder.

FIG. 6is a flow chart of a method of calculating and applying variable loop filter strengths in the loop filters shown inFIGS. 2-5. The first step of the method is to obtain the current frame to be decoded, such as reconstructed frame70(150). Then, the loop filter strength is calculated for each block (such as reconstructed block72) in the reconstructed frame (152). Alternatively, the loop filter strength could be calculated for some subset of blocks within the reconstructed frame. The process for calculating the loop filter strength for each block is described in more detail with respect toFIG. 7later.

Once the loop filter strength for each block or subset of blocks is calculated, those blocks are loop filtered using the calculated loop filter strengths of step152(154). Once loop filtering is completed for the entire reconstructed frame, the resulting frame (such as decoded frame90) is output and the method ends (156).

FIG. 7is a flow chart of a method160of calculating the loop filter strength of a block as referenced byFIG. 6. Method160obtains the block to be processed (such as reconstructed block82) (162). Typically, the reconstructed block would be provided by the loop filter calculation step152ofFIG. 6to the method160. Once the reconstructed block is identified, the base filter strength (Sc) (such as base filter strength (Sc)88) will be obtained for the reconstructed block (164). As described above, base filter strength (Sc) may be associated with the block or frame. Alternately, base filter strength (Sc) may be determined by the encoder20.

Then the loop filter38determines whether or not the reconstructed block was inter-predicted (166). If the reconstructed block was not inter-predicted, the filter strength of the reconstructed block will be set to Sc and the method will end for the block (168). However, if the block was inter-predicted, then quality values related to the reconstructed block will be obtained (170).

The obtained quality values include the quality of the associated current block (Qc) and the quality of the associated predictor block (Qp). These quality values are analogous to current quality value (Qc)84and predictor quality value (Qp)86. As described earlier, these quality values may indicate quantizer strength or another metric of block quality. Alternatively, these quality values may be associated the entire frame or portion of frame associated with either block.

Once the quality values are obtained, the block's loop filter strength (S) is calculated using the following formula: S=Sc+Si (172). The variable incremental loop filter strength (Si) is calculated by referencing the quality values obtained above. An exemplary formula to calculate the variable incremental loop filter strength is: Si=f(Qp−Qc). In this exemplary formula, the relative quality difference between a block and its predictor is determined. Then, the relative quality is normalized by function f to produce an appropriate variable incremental loop filter strength value. Function f typically would involve multiplying the relative quality difference by a constant value to obtain an appropriate variable incremental loop filter strength. However, it may be necessary to apply a more complex normalization function, such as a table lookup or a piecewise function.

Alternatively, the variable incremental loop filter strength may be calculated by a function of Qp and Qc, for example: Si=f(Qp, Qc). This type of function would process the quality values using some complex function, and then would also normalize the output of that complex function to produce Si. Then, once S is calculated, the loop filter strength of the reconstructed block will be set to S and the method will end for the block.

The above-described embodiments of encoding or decoding may illustrate some exemplary encoding techniques. However, in general, encoding and decoding as those terms are used in the claims are understood to mean compression, decompression, transformation or any other change to data whatsoever.

Encoder20and/or decoder42are implemented in whole or in part by one or more processors which can include computers, servers, or any other computing device or system capable of manipulating or processing information now-existing or hereafter developed including optical processors, quantum processors and/or molecular processors. Suitable processors also include, for example, general purpose processors, special purpose processors, IP cores, ASICS, programmable logic arrays, programmable logic controllers, microcode, firmware, microcontrollers, microprocessors, digital signal processors, memory, or any combination of the foregoing. In the claims, the term “processor” should be understood as including any the foregoing, either singly or in combination. The terms “signal” and “data” are used interchangeably.

Encoder20and/or decoder42also include a memory, which can be connected to the processor through, for example, a memory bus. The memory may be read only memory or random access memory (RAM) although any other type of storage device can be used. Generally, the processor receives program instructions and data from the memory, which can be used by the processor for performing the instructions. The memory can be in the same unit as the processor or located in a separate unit that is coupled to the processor.

For example, encoder20can be implemented using a general purpose processor with a computer program that, when executed, carries out any of the respective methods, algorithms and/or instructions described herein.FIG. 8illustrates one suitable implementation in which encoder20is implemented in a general purpose computer including a central processing unit (CPU)202and random access memory (RAM)204. Decoder42is implemented using a general purpose computer including a central processing unit (CPU)206and random access memory (RAM)208. In addition or alternatively, for example, a special purpose processor can be utilized which can contain specialized hardware for carrying out any of the methods, algorithms and/or instructions described herein. Portions of encoder20or decoder42do not necessarily have to be implemented in the same manner. Thus, for example, intra/inter prediction stage26can be implemented in software whereas transform stage28can be implemented in hardware. Portions of encoder20or portions of decoder42may also be distributed across multiple processors on the same machine or different machines or across a network such as a local area network, wide area network or the Internet.

Encoder20and decoder42can, for example, be implemented in a wide variety of configurations, including for example on servers in a video conference system. Alternatively, encoder20can be implemented on a server and decoder42can be implemented on a device separate from the server, such as a hand-held communications device such as a cell phone. In this instance, encoder20can compress content and transmit the compressed content to the communications device, using the Internet for example, as shown inFIG. 8. In turn, the communications device can decode the content for playback. Alternatively, the communications device can decode content stored locally on the device (i.e. no transmission is necessary). Other suitable encoders and/or decoders are available. For example, decoder42can be on a personal computer rather than a portable communications device.

The operations of encoder20or decoder42(and the algorithms, methods, instructions etc. stored thereon and/or executed thereby) can be realized in hardware, software or any combination thereof. All or a portion of embodiments of the present invention can take the form of a computer program product accessible from, for example, a computer-usable or computer-readable medium. A computer-usable or computer-readable medium can be any device that can, for example tangibly contain, store, communicate, and/or transport the program for use by or in connection with any processor. The medium can be, for example, an electronic, magnetic, optical, electromagnetic, or a semiconductor device. Other suitable mediums are also available.