Patent ID: 12262043

DETAILED DESCRIPTION

I. General Considerations

Disclosed below are representative embodiments of methods, apparatus, and systems for performing content-adaptive deblocking to improve the visual quality of video images compressed using block-based motion-predictive video coding. The disclosed methods, apparatus, and systems should not be construed as limiting in any way. Instead, the present disclosure is directed toward all novel and nonobvious features and aspects of the various disclosed embodiments, alone and in various combinations and subcombinations with one another. Furthermore, any features or aspects of the disclosed embodiments can be used in various combinations and subcombinations with one another. The disclosed methods, apparatus, and systems are not limited to any specific aspect or feature or combination thereof, nor do the disclosed embodiments require that any one or more specific advantages be present or problems be solved.

Although the operations of some of the disclosed methods are described in a particular, sequential order for convenient presentation, it should be understood that this manner of description encompasses rearrangement, unless a particular ordering is required by specific language set forth below. For example, operations described sequentially may in some cases be rearranged or performed concurrently. Moreover, for the sake of simplicity, the attached figures may not show the various ways in which the disclosed methods, apparatus, and systems can be used in conjunction with other methods, apparatus, and systems.

The disclosed methods can be implemented using computer-executable instructions stored on one or more computer-readable media (e.g., non-transitory computer-readable media, such as one or more optical media discs, volatile memory components (e.g., DRAM or SRAM), or nonvolatile memory or storage components (e.g., hard drives)) and executed on a computer e.g., any commercially available computer or a computer or image processor embedded in a device, such as a laptop computer, net book, web book, tablet computing device, smart phone, or other mobile computing device). Any of the intermediate or final data created and used during implementation of the disclosed methods or systems can also be stored on one or more computer-readable media (e.g., non-transitory computer-readable media).

For clarity, only certain selected aspects of the software-based embodiments are described, Other details that are well known in the art are omitted. For example, it should be understood that the software-based embodiments are not limited to any specific computer language or program. Likewise, embodiments of the disclosed technology are not limited to any particular computer or type of hardware. Exemplary computing environments suitable for performing any of the disclosed software-based methods are introduced below.

The disclosed methods can also be implemented using specialized computing hardware that is configured to perform any of the disclosed methods. For example, the disclosed methods can be implemented by an integrated circuit (e.g., an application specific integrated circuit (“ASIC”), a graphics processing unit (“GPU”), or programmable logic device (“PLD”), such as a field programmable gate array (“FPGA”)) specially designed to implement any of the disclosed methods (e.g., dedicated hardware configured to perform any of the disclosed image processing techniques).

A. Exemplary Computing Environments

FIG.1illustrates a generalized example of a suitable computing device environment or computing hardware environment100in which embodiments of the disclosed technology can be implemented. For example, encoders or decoders configured to perform any of the disclosed deblocking techniques can be implemented using embodiments of the computing hardware environment100. The computing hardware environment100is not intended to suggest any limitation as to the scope of use or functionality of the disclosed technology, as the technology can be implemented in diverse general-purpose or special-purpose computing environments.

With reference toFIG.1, the computing hardware environment100includes at least one processing unit110and memory120, InFIG.1, this most basic configuration130is included within a dashed line. The processing unit110executes computer-executable instructions. In a multi-processing system, multiple processing units execute computer-executable instructions to increase processing power. The memory120can be volatile memory (e.g., registers, cache, RAM, DRAM, SRAM), non-volatile memory (e.g., ROM, EEPROM, flash memory), or some combination of the two. The memory120can store software180for implementing one or more of the disclosed deblocking techniques. For example, the memory120can store software180for implementing any of the disclosed methods.

The computing hardware environment can have additional features. For example, the computing hardware environment100includes a storage device140, one or more input devices150, one or more output devices160, and one or more communication connections170. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing hardware environment100. Typically, operating system software (not shown) provides an operating environment for other software executing in the computing hardware environment100, and coordinates activities of the components of the computing hardware environment100.

The storage device140is a type of non-volatile memory and can be removable or non-removable. The storage device140includes, for instance, magnetic disks (e.g., hard drives), magnetic tapes or cassettes, optical storage media (e.g., CD-ROMs or DVDs), or any other tangible non-transitory storage medium that can be used to store information and which can be accessed within or by the computing hardware environment100. The storage device140can also store the software180for implementing any of the described techniques.

The input device(s)150can be a touch input device such as a keyboard, mouse, touch screen, pen, trackball, a voice input device, a scanning device, or another device that provides input to the computing environment100. The output device(s)160can be a display device, touch screen, printer, speaker, CD-writer, or another device that provides output from the computing environment100.

The communication connection(s)170enable communication over a communication medium to another computing entity. The communication medium conveys information such as computer-executable instructions, any of the intermediate or final messages or data used in implementing embodiments of the disclosed technology. By way of example, and not limitation, communication media include wired or wireless techniques implemented with an electrical, optical, infrared, acoustic, or other carrier.

The various methods disclosed herein (e.g., any of the disclosed deblocking techniques) can be described in the general context of computer-executable instructions stored on one or more computer-readable media (e.g., tangible non-transitory computer-readable media such as memory120and storage140). The various methods disclosed herein can also be described in the general context of computer-executable instructions, such as those included in program modules, being executed by a processor in a computing environment. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or split between program modules as desired in various embodiments. Any of the disclosed methods can also be performed using a distributed computing environment (e.g., a client-server network, cloud computing environment, wide area network, or local area network).

B. Examples of Generalized Video Encoder and Decoders

The deblocking techniques described herein can be incorporated into embodiments of a video encoder and/or decoder. For example, in certain implementations, the deblocking filter techniques are used in connection with an encoder and/or decoder implementing the H.264 AVC standard or H.264 HEVC standard. In alternative embodiments, the deblocking techniques described herein are implemented independently or in combination in the context of other digital signal compression systems, and other video codec standards.FIGS.2-5below illustrate the general operation of an example encoder and decoder (such as an H.264/AVC encoder or decoder) in which embodiments of the disclosed technology can be used. In particular,FIG.2is a block diagram of a generalized video encoder system200, andFIG.3is a block diagram of a generalized video decoder system300. The relationships shown between modules within the encoder and the decoder indicate the main flow of information in the encoder and decoder; other relationships are not shown for the sake of simplicity.FIGS.2and3generally do not show side information indicating the encoder settings, modes, tables, etc. used for a video sequence, frame, macroblock, block, etc. Such side information is sent in the output bit stream, typically after the information is entropy encoded. The format of the output bit stream can be at H.264 format or another format.

The encoder system200and decoder system300are block-based and support a 4:2:0 macroblock format with each macroblock including four 8×8 luminance blocks (at times treated as one 16×16 macroblock) and two 8×8 chrominance blocks. Alternatively, the encoder200and decoder300are object-based, use a different macroblock or block format, or perform operations on sets of pixels of different sizes or configurations than 8×8 blocks and 16×16 macroblocks.

Depending on the implementation and the type of compression desired, modules of the encoder or decoder can be added, omitted, split into multiple modules, combined with other modules, and/or replaced with like modules. In alternative embodiments, encoder or decoders with different modules and/or other configurations of modules perform one or more of the described techniques.

1. Video Encoder

FIG.2is a block diagram of a general video encoder system200. The encoder system200receives a sequence of video frames including a current frame205, and produces compressed video information295as output. Particular embodiments of video encoders typically use a variation or supplemented version of the generalized encoder system200.

The encoder system200compresses predicted frames and key frames. For the sake of presentation,FIG.2shows a path for key frames through the encoder system200and a path for predicted frames. Many of the components of the encoder system200are used for compressing both key frames and predicted frames. The exact operations performed by those components can vary depending on the type of information being compressed.

A predicted frame (also called a “P-frame,” “B-frame,” or “inter-coded frame”) is represented in terms of a prediction (or difference) from one or more reference (or anchor) frames. A prediction residual is the difference between what was predicted and the original frame. In contrast, a key frame (also called an “I-frame,” or “intra-coded frame”) is compressed without reference to other frames. Other frames also can be compressed without reference to other frames. For example, an intra B-frame (or “B/I frame”), while not a true key frame, is also compressed without reference to other frames.

If the current frame205is a forward-predicted frame, a motion estimator210estimates motion of macroblocks or other sets of pixels of the current frame205with respect to a reference frame, which is the reconstructed previous frame225buffered in a frame store (e.g., frame store220). If the current frame205is a bi-directionally-predicted frame (a B-frame), a motion estimator210estimates motion in the current frame205with respect to two reconstructed reference frames. Typically, a motion estimator estimates motion in a B-frame with respect to a temporally previous reference frame and a temporally future reference frame. Accordingly, the encoder system200can comprise separate stores220and222for backward and forward reference frames. The motion estimator210can estimate motion by pixel, ½ pixel, ¼ pixel, or other increments, and can switch the resolution of the motion estimation on a frame-by-basis or other basis. The resolution of the motion estimation can be the same or different horizontally and vertically. The motion estimator210outputs as side information motion information215, such as motion vectors. A motion compensator230applies the motion information215to the reconstructed frame(s)225to form a motion-compensated current frame235. The prediction is rarely perfect, however, and the difference between the motion-compensated current frame235and the original current frame205is the prediction residual245. Alternatively, a motion estimator and motion compensator apply another type of motion estimation/compensation. A frequency transformer260converts the spatial domain video information into frequency domain (spectral) data. For block-based video frames, the frequency transformer260applies a discrete cosine transform (“DCT”) or variant, of DCT to blocks of the pixel data or the prediction residual data, producing blocks of DCT coefficients. Alternatively, the frequency transformer260applies another conventional frequency transform such as a Fourier transform or uses wavelet or subband analysis. If the encoder uses spatial extrapolation (not shown inFIG.2) to encode blocks of key frames, the frequency transformer260can apply a re-oriented frequency transform such as a skewed DCT to blocks of prediction residuals for the key frame. In some embodiments, the frequency transformer260applies an 8×8, 8×4, 4×8, or other size frequency transform (e.g., DCT) to prediction residuals for predicted frames.

A quantized270then quantizes the blocks of spectral data coefficients. The quantizer applies uniform, scalar quantization to the spectral data with a step-size that varies on a frame-by-frame basis or other basis. Alternatively, the quantizer applies another type of quantization to the spectral data coefficients, for example, a non-uniform, vector, or non-adaptive quantization, or directly quantizes spatial domain data in an encoder system that does not use frequency transformations. In addition to adaptive quantization, the encoder system200can use frame dropping, adaptive filtering, or other techniques for rate control.

If a given macroblock in a predicted frame has no information of certain types e.g., no motion information for the macroblock and no residual information), the encoders200may encode the macroblock as a skipped macroblock. If so, the encoder signals the skipped macroblock in the output bit stream of compressed video information295.

When a reconstructed current frame is needed for subsequent motion estimation/compensation, an inverse quantizer276performs inverse quantization on the quantized spectral data coefficients. An inverse frequency transformer266then performs the inverse of the operations of the frequency transformer260, producing a reconstructed prediction residual (for a predicted frame) or a reconstructed key frame. If the current frame205was a key frame, the reconstructed key frame is taken as the reconstructed current frame (not shown). If the current frame205was a predicted frame, the reconstructed prediction residual is added to the motion-compensated current frame235to form the reconstructed current frame. A frame store e.g., frame store220) buffers the reconstructed current frame for use in predicting another frame. In some embodiments, the encoder applies a deblocking process to the reconstructed frame to adaptively smooth discontinuities in the blocks of the frame. For instance, the encoder can apply any of the deblocking techniques disclosed herein to the reconstructed frame.

The entropy coder280compresses the output of the quantizer270as well as certain side information (e.g., motion information215, spatial extrapolation modes, quantization step size). Typical entropy coding techniques include arithmetic coding, differential coding, Huffman coding, run length coding, LZ coding, dictionary coding, and combinations of the above. The entropy coder280typically uses different coding techniques for different kinds of information e.g., DC coefficients, AC coefficients, and different kinds of side information), and can choose from among multiple code tables within a particular coding technique.

The entropy coder280puts compressed video information295in the buffer290. A buffer level indicator is fed back to bit rate adaptive modules. The compressed video information295is depleted from the buffer290at a constant or relatively constant bit rate and stored for subsequent streaming at that bit rate. Therefore, the level of the buffer290is primarily a function of the entropy of the filtered, quantized video information, which affects the efficiency of the entropy coding. Alternatively, the encoder system200streams compressed video information immediately following compression, and the level of the buffer290also depends on the rate at which information is depleted from the buffer290for transmission.

Before or after the buffer290, the compressed video information295can be channel coded for transmission over the network. The channel coding can apply error detection and correction data to the compressed video information295.

2. Video Decoder

FIG.3is a block diagram of a general video decoder system300. The decoder system300receives information395for a compressed sequence of video frames and produces output including a reconstructed frame305. Particular embodiments of video decoders typically use a variation or supplemented version of the generalized decoder300.

The decoder system300decompresses predicted frames and key frames. For the sake of presentation,FIG.3shows a path for key frames through the decoder system300and a path for predicted frames. Many of the components of the decoder system300are used for decompressing both key frames and predicted frames. The exact operations performed by those components can vary depending on the type of information being decompressed.

A buffer390receives the information395for the compressed video sequence and makes the received information available to the entropy decoder380. The buffer390typically receives the information at a rate that is fairly constant over time, and includes a jitter buffer to smooth short-term variations in bandwidth or transmission. The buffer390can include a playback buffer and other buffers as well. Alternatively, the buffer390receives information at a varying rate. Before or after the buffer390, the compressed video information can be channel decoded and processed for error detection and correction.

The entropy decoder380entropy decodes entropy-coded quantized data as well as entropy-coded side information (e.g., motion information315, spatial extrapolation modes, quantization step size), typically applying the inverse of the entropy encoding performed in the encoder. Entropy decoding techniques include arithmetic decoding, differential decoding, Huffman decoding, run length decoding, LZ decoding, dictionary decoding, and combinations of the above. The entropy decoder380frequently uses different decoding techniques for different kinds of information (e.g., DC coefficients, AC coefficients, and different kinds of side information), and can choose from among multiple code tables within a particular decoding technique.

A motion compensator330applies motion information315to one or more reference frames325to form a prediction335of the frame305being reconstructed. For example, the motion compensator330uses a macroblock motion vector to find a macroblock in a reference frame325. A frame buffer (e.g., frame buffer320) stores previously reconstructed frames for use as reference frames. Typically, B-frames have more than one reference frame (e.g., a temporally previous reference frame and a temporally future reference frame). Accordingly, the decoder system300can comprise separate frame buffers320and322for backward and forward reference frames.

The motion compensator330can compensate for motion at pixel, ½ pixel, ¼ pixel, or other increments, and can switch the resolution of the motion compensation on a frame-by-frame basis or other basis. The resolution of the motion compensation can be the same or different horizontally and vertically. Alternatively, a motion compensator applies another type of motion compensation. The prediction by the motion compensator is rarely perfect, so the decoder300also reconstructs prediction residuals. When the decoder needs a reconstructed frame for subsequent motion compensation, a frame buffer (e.g., frame buffer320) buffers the reconstructed frame for use in predicting another frame. In some embodiments, the decoder applies a deblocking process to the reconstructed frame to adaptively smooth discontinuities in the blocks of the frame. For instance, the decoder can apply any of the deblocking techniques disclosed herein to the reconstructed frame.

An inverse quantizer370inverse quantizes entropy-decoded data. In general, the inverse quantizer applies uniform, scalar inverse quantization to the entropy-decoded data with a step-size that varies on a frame-by-frame basis or other basis. Alternatively, the inverse quantizer applies another type of inverse quantization to the data, for example, a non-uniform, vector, or non-adaptive quantization, or directly inverse quantizes spatial domain data in a decoder system that does not use inverse frequency transformations.

An inverse frequency transformer360converts the quantized, frequency domain data into spatial domain video information. For block-based video frames, the inverse frequency transformer360applies an inverse DCT (“IDCT”) or variant, of IDCT to blocks of the DCT coefficients, producing pixel data or prediction residual data for key frames or predicted frames, respectively. Alternatively, the frequency transformer360applies another conventional inverse frequency transform such as a Fourier transform or uses wavelet or subband synthesis. If the decoder uses spatial extrapolation (not shown inFIG.3) to decode blocks of key frames, the inverse frequency transformer360can apply a re-oriented inverse frequency transform such as a skewed IDCT to blocks of prediction residuals for the key frame. In some embodiments, the inverse frequency transformer360applies an 8×8, 8×4, 4×8, or other size inverse frequency transforms (e.g., IDCT) to prediction residuals for predicted frames.

When a skipped macroblock is signaled in the bit stream of information395for a compressed sequence of video frames, the decoder300reconstructs the skipped macroblock without using the information (e.g., motion information and/or residual information) normally included in the bit stream for non-skipped macroblocks.

C. Loop Filtering

Quantization and other lossy processing of prediction residuals can cause block artifacts (artifacts at block boundaries in reference frames that are used for motion estimation of subsequent predicted frames. Post-processing by a decoder to remove blocky artifacts after reconstruction of a video sequence improves perceptual quality. However, post-processing does not improve motion compensation using the reconstructed frames as reference frames and does not improve compression efficiency. With or without post-processing, the same amount of bits is used for compression. Moreover, the filters used for deblocking in post-processing can introduce too much smoothing in reference frames used for motion estimation/compensation.

In embodiments of the disclosed technology, a video encoder processes a reconstructed frame to reduce blocky artifacts prior to motion estimation using the reference frame. Additionally, in embodiments of the disclosed technology, a video decoder processes the reconstructed frame to reduce blocky artifacts prior to motion compensation using the reference frame. With deblocking, a reference frame becomes a better reference candidate to encode the following frame. Thus, using embodiments of the disclosed deblocking techniques can improve the quality of motion estimation/compensation, resulting in better prediction and lower bit rate for prediction residuals. The deblocking processes described herein are especially helpful in low bit rate applications.

In some embodiments, following the reconstruction of a frame in a video encoder or decoder, the encoder/decoder applies a deblocking technique to blocks in the reconstructed frame. The deblocking technique can include, for example, applying a directional deblocking or smoothing filter to blocks in the reconstructed frame. For instance, the encoder/decoder can apply the deblocking filter across boundary rows and/or columns of a selected block. The deblocking filter can remove boundary discontinuities between blocks in the reconstructed frame while preserving real edges in the video image, when appropriate, thereby improving the quality of subsequent motion estimation using the reconstructed frame as a reference frame. In certain embodiments, the encoder/decoder performs deblocking after reconstructing the frame in a motion compensation loop in order for motion compensation to work as expected. This contrasts with typical post-processing deblocking processes, which operate on the whole image outside of the motion compensation loop. Further, a decoder can apply an additional post-processing deblocking filter to further smooth a reconstructed frame for playback after applying any of the disclosed deblocking techniques to a reference frame during motion compensation.

FIG.4shows a motion estimation/compensation loop in a video encoder that includes a deblocking process.FIG.5shows a corresponding motion compensation loop in a video decoder that includes a deblocking process. The deblocking process can comprise any of the deblocking processes disclosed herein and described in more detail below (e.g., any of the content-adaptive directional deblocking techniques disclosed herein).

With reference toFIG.4, a motion estimation/compensation loop400includes motion estimation410and motion compensation420of an input frame405. The motion estimation410finds motion information for the input frame405with respect to a reference frame495, which is typically a previously reconstructed intra- or inter-coded frame. In alternative embodiments, the loop filter is applied to backward-predicted or bi-directionally-predicted frames. The motion estimation410produces motion information such as a set of motion vectors for the frame. The motion compensation420applies the motion information to the reference frame495to produce a predicted frame425.

The prediction is rarely perfect, so the encoder computes430the error/prediction residual435as the difference between the original input frame405and the predicted frame425. The frequency transformer440frequency transforms the prediction residual435, and the quantizer450quantizes the frequency coefficients for the prediction residual435before passing them to downstream components of the encoder.

In the motion estimation/compensation loop, the inverse quantizer460inverse quantizes the frequency coefficients of the prediction residual435, and the inverse frequency transformer470changes the prediction residual435back to the spatial domain, producing a reconstructed error475for the frame405.

The encoder then combines480the reconstructed error475with the predicted frame425to produce a reconstructed frame. The encoder applies a deblocking process490(e.g., a deblocking process according to any of the disclosed embodiments) to the reconstructed frame and stores the reconstructed frame in a frame buffer492for use as a reference frame495for the next input frame. Alternatively, the deblocking process490follows the frame buffer492. In alternative embodiments, the arrangement or constituents of the motion estimation/compensation loop changes, but the encoder still applies the deblocking process to reference frames.

With reference toFIG.5, a motion compensation loop500includes motion compensation520to produce a reconstructed frame585. The decoder receives motion information515from the encoder. The motion compensation520applies the motion information515to a reference frame595to produce a predicted frame525.

In a separate path, the inverse quantizer560inverse quantizes the frequency coefficients of a prediction residual, and the inverse frequency transformer570changes the prediction residual back to the spatial domain, producing a reconstructed error575for the frame585.

The decoder then combines580the reconstructed error575with the predicted frame525to produce the reconstructed frame585, which is output from the decoder. The decoder also applies a deblocking process590(e.g., a deblocking process according to any of the disclosed embodiments) to the reconstructed frame585and stores the reconstructed frame in a frame buffer592for use as the reference frame595for the next input frame. Alternatively, the loop filter590follows the frame buffer592. In alternative embodiments, the arrangement or constituents of the motion compensation loop changes, but the decoder still applies the deblocking process to reference frames.

In the video encoder200/decoder300, the compressed bitstream does not need to provide any indication whether out-of-loop deblocking should be employed. The latter is usually determined by the decoder300based on simple rules and the availability of additional computing cycles. Signals may be provided by the encoder in the bitstream indicating whether to use post-processing. On the other hand, the application of in-loop deblocking is typically indicated within the bitstream to avoid drift or mismatch. This indication may be through a sequence based flag, or possibly using frame or sub-frame based flags. A decoder that encounters a frame indicating that it has been in-loop deblocked will in turn decode and deblock that frame for bitstream compliance.

II. Exemplary Content-Adaptive Deblocking Techniques

A. A Generalized Content-Adaptive Deblocking Process

FIG.6is a block diagram showing an exemplary embodiment of a generalized deblocking process600in accordance with the disclosed technology. The illustrated deblocking process can be employed in a motion estimation/compensation loop of an encoder or decoder (e.g., as the deblocking process490shown inFIG.4, or the deblocking process590shown inFIG.5). In certain embodiments, only intra-coded frames (I-frames or key frames) are deblocked using the deblocking process600. In other embodiments, however, inter-coded frames or both intra- and inter-coded frames are deblocked using the deblocking process600.

The exemplary deblocking process600is performed on a reconstructed frame610. The reconstructed frame can be a decoded frame after inverse DCT and motion compensation, At612, the luminance components of the reconstructed frame610are processed with a deblocking filter (e.g., an H.264/AVC deblocking filter). In other embodiments, however, this initial deblocking act is absent. At614, orientation energy edge detection (“OEED”) is performed on at least a portion of the initially deblocked frame. In some embodiments, for instance, global OEED is conducted on the initially deblocked image. As more fully explained below, global OEED can be used to detect the edges in the initially deblocked frame, thereby partitioning the image into local directional features (“LDFs”). For example, OEED can also be used to identify the edge orientations of the LDFs in the image. At616, a deblocking mode is determined for each block in the frame. In particular embodiments, and as more fully explained below, the deblocking mode decision can be based at least in part on the LDF edge detection and edge orientation of the LDF in a given block. Based at least in part on the results of the OEED, different deblocking modes can be decided for blocks in the frame (e.g., each of the blocks in the frame). At618, content-adaptive deblocking is performed on both the luminance and chroma components of the decoded frame. The deblocking can be performed for the blocks using the deblocking modes determined at616. For example, in certain embodiments, the deblocking filter orientations and activation thresholds can be based at least in part on the deblocking mode decisions from616. At620, the deblocked reconstructed frame is generated. As noted above, the reconstructed frame can be used as a reference frame for further motion compensation/estimation. However, because the deblocking scheme600has smoothed or appropriately filtered the directional LDFs at the block boundaries of the reconstructed frame, undesirable and visually annoying block artifacts can be removed or substantially removed from the reconstructed frame.

B. Orientation Energy Edge Detection

In embodiments of the disclosed technology, an image is partitioned into LDFs using an orientation energy edge detection (“OEED”) technique. OEED can be used to extract edges and edge orientations. For example, edges can be extracted by orientation energy on an initially deblocked luminance image I. To implement OEED, sets of one or more Gaussian derivative filters can be used, where each set of filters is oriented along a different orientation. In particular embodiments of the disclosed technology, the OEED filters are defined as follows:
W(x,y,θ)=(Fθo*I)2+(Fθe*1)2(1)
where Fθoand Fθeare the first and second Gaussian derivative filters, respectively, at orientation θ.

In one desirable embodiment, eight sets of filters are used.FIG.8is a block diagram800showing eight sets of two filters each, where each set includes orientation energy filters having a different orientation than the filters of the other sets. In particular, each set includes a first Gaussian derivative filter (shown at row810) and a second Gaussian derivative filter (shown at row812). In the illustrated embodiment, the filters have orientations of 180° (which can also be viewed as 0°), 157.5°, 135°, 112.5°, 90°, 67.5°, 45°, and 22.5″. In particular embodiments, the first and second Gaussian derivative filters at orientation θ have the following form:

Fo(x,y,θ)=12⁢πσx⁢σy·x^2-1σx2⁢exp⁢{-12⁢(x^2σx2+y^2σy2)}⁢Fe(x,y,θ)=12⁢πσx⁢σy·-x^2σx2⁢exp⁢{-12⁢(x^2σx2+y^2σy2)}⁢x^=x⁢cos⁢θ-y⁢sin⁢θ⁢y^=x⁢sin⁢θ+y⁢cos⁢θ(2)
where σxand σyare filter parameters that can vary from implementation to implementation. In certain implementations of these embodiments, the length and taps of the filter are further defined as:

T=⌊max⁡(σx,σy)·A+0.5⌋⁢x,y∈[-T2,T2](3)
where A is a parameter that can vary from implementation to implementation. Furthermore, the angle discretization used for the filters can vary from implementation to implementation. In one implementation, however, the angle discretization is:

θ=2⁢π⁢nN,n=0,…,N/2-1(4)
The value of N can also vary from implementation to implementation. In one particular implementation, the values of σx, σy, A, and N are as follows:
σx=1.3,
σx=39,
A=4.2426,
N=16
For each pixel (x, y) in I, and with reference to OEED filters having the form shown in Expression (1), the filter output energy W will have a maximum at an orientation parallel to an edge, and an energy maximum along a line perpendicular to the edge. Therefore, and according to one embodiment, edges can be found by marking or otherwise designating pixels in the image I (e.g., each pixel throughout a plurality of blocks, such as each pixel throughout all blocks of the image) that satisfy:

∂∂74⁢W⁡(p)=0,∂∂vθW⁡(p)=0(5)
where vθis the unit vector orthogonal to θ. Edges can also be found by marking pixels where Expression (5) is near zero or approximately zero (e.g., within a deviation of zero computed to accommodate the angular discretization between the OEED filters). Furthermore, as a result of finding the edges using the OEED filters, the orientations of the edges will also be known. In particular, the orientation of the OEED filter that causes Expression (5) (or similar expression) to be satisfied corresponds to the orientation of the edge pixel being evaluated.

By using OEED, an edge map recording edge pixel locations can be created. Additionally, by using OEED, an angle map (or orientation map) recording the orientation of each pixel can also be created. For instance,FIGS.9,10,11, and12illustrate exemplary LDF partitions, edge maps, and angle maps that can be created using OEED. In particular,FIG.9shows an example of a reconstructed image900that may be input into a deblocking process, such as the deblocking process600.FIG.10shows an LDF partition1000of the image900obtained using an embodiment of the OEED described herein. In particular, the exemplary partition1000is a block-level LDF partition where blocks are shown in white if they contain an edge as detected according to a corresponding edge map obtained from an embodiment of OEED. For example,FIG.11shows an edge map1100of the image900obtained from an embodiment of OEED using Expressions (1) and (5). InFIG.11edge pixels satisfying Expression (5) are marked white.FIG.12shows an angle map1200obtained using OEED. InFIG.12, different intensities indicate different orientations.

FIG.21shows an LDF partition2100of the image900obtained from the intra prediction mode as described in J. Jeong et al., “A directional deblocking filter based on intra prediction for H.264/AVC,” IEICE Electronics Express, vol. 6, no. 12, pp. 864-869 (June 2009) (hereinafter “Jeang”). As can be seen by comparing the LDF partition1000inFIG.10to the IMF partition2100inFIG.21, OEED is more accurate than intra prediction in distinguishing LDFs, even at a low bit rate.

Compared with edge detection techniques using only image gradients, OEED can jointly consider edge location and orientation. In addition, OEED utilizes a class of nonlinear filters which are more appropriate for localizing composite edges, so it is able to provide more reliable information to distinguish IDE regions. As shown in block diagram600, an initial deblocking of the luminance components can be performed prior to DEED (e.g., using the H.264/AVC deblocking scheme). Such an initial deblocking technique can further improve the accuracy of the edge and angle maps obtained from DEED and from which certain parameters can be determined later in content-adaptive deblocking schemes.

C. Deblocking Mode Decision

In certain embodiments, one or more the LDF partition, the edge map, or the angle map resulting from DEED are used, at least in part, to select an appropriate deblocking mode for each block. Although the number of available deblocking modes can vary from implementation to implementation, one particular embodiment of the disclosed technology has eight deblocking modes: six directional deblocking modes (each having a directional deblocking filter with a different orientation), an extra smoothing mode, and a standard deblocking mode (which can comprise a deblocking technique used in a known standard, such as the H.264/AVC deblocking technique). Furthermore, in this embodiment, the six directional deblocking modes can comprise directional deblocking filters having orientations that correspond to orientations of the Gaussian derivative filters used during DEED and having modified activation thresholds. In one particular implementation, the six directional deblocking filters correspond to orientations 2, 3, 4, 6, 7, and 8 shown inFIG.8(the non-horizontal and non-vertical orientations). For example, the orientations can be 157.5°, 135°, 112.5°, 67.5°, 45°, and 22.5°.

To select which deblocking mode to use for a selected block (e.g., for each 4×4 or 8×8 transformed intra block), a variety of techniques can be used. In certain embodiments, for example, one of the directional deblocking modes is selected if the block is determined to have directional features that are consistent with one or more of its neighboring blocks. Metrics can be applied to the edge pixels of a selected block to evaluate the consistency of the edge orientations within the block. For example, an average orientation of the edge pixels in the selected block can be determined and used to evaluate edge orientation consistency. Additionally, an edge orientation deviation among the edge pixels in the block can be computed and also used to evaluate edge orientation consistency.

In certain embodiments, a deblocking mode decision is made for each transformed intra block BIwith its upper and left neighbors RUand BLavailable. In one particular implementation, if edge pixels appear in BI, an average orientation index mIis computed for the block. The average orientation index mIcan be computed by averaging the orientations of edge pixels in the block BI. The result can be rounded to be an integer value or can have some other resolution. In one exemplary implementation, the orientation values correspond to those shown inFIG.8and are integer values between 1 and 8. In certain implementations, an orientation variance value ovIis also computed for the edge pixels in the block BI. The orientation variance value can be the statistical variance computed from the observed edge orientations. In other implementations, however, other measures of the deviations between the observed edge orientations can be used (e.g., the standard deviation, absolute deviation, or other such statistical metric).

Similar average orientation indices and orientation variances values can be computed for one or more of the neighboring blocks. For example, in one particular implementation, average orientation indices and orientation variances values are computed for the upper neighboring and the leftward neighboring block, resulting in mU, ovU, mL, and ovL. If there are no edge pixels in the one or more neighboring blocks, then the corresponding average orientations and orientation variances can be set to a fixed value (e.g., 0).

A distance metric can then be used to evaluate whether the average orientation indices of the selected block and the one or more neighboring block are indicative of edges having a consistent orientation. The orientation variance values for the selected block and the one or more neighboring blocks can also be evaluated. For example, in one particular implementation, a distance metric D(ma, mb) between two different orientation indices is used. The distance metric can be the absolute value of the difference between the two average orientation indices and, in some embodiments, can be weighted. In one particular implementation, the following evaluation of the average orientation values and the orientation variance values is made:
D(mI,mU)+D(mI,mL)<T1(6)
ovI+ovU+ovL<T2(7)
where T1and T2are threshold values that can be varied from implementation to implementation depending on the desired directional filter sensitivity. Furthermore, in certain embodiments, an evaluation using Expressions (6) and (7) is only performed if the orientation variance index for the selected block Biindicates that the orientation is non-vertical or non-horizontal (e.g., in embodiments in which the orientations correspond toFIG.8, mI≠1.5). Satisfaction of the Expressions (6) and (7) indicates that the current block BIhas consistent directional features with its neighbors. Consequently, a directional deblocking mode using a directional deblocking filter having the appropriate orientation can be selected for block BI. Such a directional deblocking filter can be used to reduce the blockiness of the block boundaries (e.g., at the boundaries of mI, mU, and mL) while maintaining the existence of real edges along the detected orientation. As more fully explained below, an additional threshold evaluation can be performed before activating the deblocking filter. For example, the boundaries between the blocks can be evaluated to determine if the boundaries include “true” image edges that are desirably left, unfiltered.

In certain embodiments, if edge pixels are not present in the selected block then the block can be evaluated to determine if the block is part of a region that is free of LDFs (a consecutive non-LDF region), in which case a smoothing filter can be applied. Blocking artifacts in consecutive non-LDF regions are among the most visually displeasing blocking artifacts because the true image intends for the region to have smoothly varying color or intensity gradations, whereas the compressed version often results in obvious color and intensity contrasts at the block borders. To determine whether the selected block BIis in a consecutive non-LDF region, a variety of metrics can be used. In certain embodiments, however, intensity variances can be evaluated between the selected block BIand one or more neighboring blocks. In other embodiments, chrominance variances can additionally or alternatively be evaluated.

In one particular implementation, for each block BInot having edge pixels with its upper and left neighbors BUand BLavailable, the intensity variances ivI, ivUand ivLcan be computed for the three blocks, respectively. The intensity variance value can be the statistical variance computed from the pixel intensities in each respective block. In other implementations, however, other measures of the deviations between the pixel intensities can be used (e.g., the standard deviation, absolute deviation, or other such statistical metric), The intensity variances can then be evaluated. In one particular implementation, the following evaluation of the intensity variances is made:
ivI+ivU+ivL<T3(8)
where T3is a threshold value that can be varied from implementation to implementation depending on the desired smoothing filter sensitivity. Satisfaction of the Expression (8) indicates that the current block BIis inside of a consecutive non-LDF region. Consequently, a deblocking mode using a smoothing filter is selected for block BI. In certain implementations, the smoothing filter is applied with no further thresholds being evaluated. In other implementations, additional thresholds can be evaluated to determine whether to apply the smoothing filter.

In certain embodiments, if neither a directional deblocking or an extra smoothing mode is selected for a block BI, then a standard deblocking mode can be applied (e.g., the H.264/AVC deblocking scheme). Additionally, the standard deblocking mode can be applied to blocks having horizontally or vertically oriented edges (e.g., in embodiments in which the orientations correspond toFIG.8, mI≠1.5).

FIG.13is an exemplary image1300showing blocks in consecutive non-LDF regions detected using an evaluation according to Expression (8). In particular, the blocks having highlighted upper and left borders (several of which are shown at1310) are blocks determined to be in consecutive non-LDF regions. Accordingly, and in certain embodiments of the disclosed technology, a smoothing filter is selected for the highlighted blocks.FIG.13also illustrates that a single video frame can have a variety of content, creating different deblocking mode distributions throughout the image. The variety of image content further illustrates the desirability of a flexible deblocking scheme.

In some embodiments, the deblocking selection process can be affected by the block size being evaluated. For example, in one embodiment, only 4×4 and 8×8 blocks, which often contain image details, are evaluated to determine whether they have edge pixels present and whether a directional deblocking filter should be applied to the block. Furthermore, in certain embodiments, only 16×16 luminance blocks, which often contain smooth regions, are evaluated to determine whether they are part of a non-LDF region such that a smoothing filter should be applied. All other blocks in such embodiments can be deblocked using a standard deblocking process, such as the H.264/AVC deblocking process.

Once the deblocking mode decisions have been made, the selected deblocking filters can be applied to the video frame in a variety of manners. In one embodiment, for instance, the deblocking filters (including the directional deblocking filters and the smoothing filter) are applied in a raster-scan order at the macroblock level for both luminance and chroma components. In other embodiments, the deblocking filters are applied in different orders, such as in a box-out fashion, wipe fashion, by applying a selected filter to all of its corresponding blocks before applying a next selected filter, by applying the filters to non-LDF regions followed by LDF regions, or vice versa. In one particular implementation, once the deblocking mode decision is done, the 8 different deblocking filters are simultaneously employed in a raster-scan order at the macroblock level for both luminance and chroma components. The reconstructed frame is obtained in a single deblocking pass.

D. Exemplary Directional Deblocking Filters

In certain embodiments, when a directional deblocking mode is selected, the filtering involves pixels that are no longer in a line perpendicular to the block boundary. For example, in one particular implementation, the filtering is along six different orientations corresponding to the orientations shown inFIG.8.FIG.14is a block diagram1400illustrating examples of six filtering orientations across vertical block boundaries. Corresponding filter orientations can be used across horizontal block boundaries. In particular,FIG.14shows example filters from filter bank1410(related to orientation 7 fromFIG.8), filter bank1412(related to orientation 3 fromFIG.8), filter1414(related to orientation 8 fromFIG.8), filter bank1416(related to orientation 2 fromFIG.8), filter bank1418(related to orientation 6 fromFIG.8), and filter bank1420(related to orientation 4 fromFIG.8). InFIG.14, and for ease of presentation, only selected pixels used by the filters are shown. The other pixels used with the filters can be easily determined by those of ordinary skill in the art based on what is shown inFIG.14and this description. For example, and in the illustrated embodiment, there are four filters for each of filter bank1410(related to orientation 7 fromFIG.8), filter bank1412(related to orientation 3 fromFIG.8), filter bank1414(related to orientation 8 fromFIG.8), and filter bank1416(related to orientation 2 fromFIG.8). For these filter banks, the first and fourth filters in the respective filter banks are shown by the lines inFIG.14. In the illustrated embodiment, there are eight filters for each of filter bank1418(related to orientation 6 fromFIG.8) and filter bank1420(related to orientation 4 fromFIG.8). For these filter banks, the first and eighth filters for the respective filter banks are shown by the lines inFIG.14. As illustrated inFIG.14, the filters for filter bank1418and1420use interpolation at certain half pixel locations, even though only integer pixels are filtered. Values at these half-pixel locations can be computed, for example, using a bilinear filter. Furthermore, inFIG.14, the current block (block Br) is the 4×4 block of pixels located in the middle-right of each illustrated pixel group. Additionally, for the example filters shown in filter bank1416(related to orientation 8) and filter bank1418(related to orientation 2), some of the pixels used by the filters extend beyond the illustrated pixel group and are not shown. For example, q2and q3are not shown in the left-most filter shown in filter bank1416, whereas p2and p3are not shown in the right-most filter shown in filter bank1418. The position of these missing pixels, however, can be easily deduced as they are symmetrical with the pixel positions illustrated.

In one embodiment, if p3, . . . , p0, g0, . . . , q3correspond to the selected pixels for a ID filter, their values are modified as follows during directional deblocking:
pn=(λn,1, . . . ,λn,8)(p3, . . . ,p0,q0, . . . ,q3)+λn,0
qn=(λn,1, . . . ,λn,8)(q3, . . . ,q0,p0, . . . ,p3)+λn,0(9)
where λn,mis a filter coefficient. In a particular implementation, the filter coefficients are the same as those used H.264/AVC and have a range of (0≤n≤2, 0≤m≤8). More specifically, in one particular implementation, the filter coefficients are computed using the methodology outlined in Section 8 of the H.264/AVC standard, which is described in International Telecommunication Union-Telecommunication (ITU-T) Recommendation H.264 (2003), or in any of the subsequent amendments or corrigendums.

According to the H.264/AVC standard, there are two thresholds α and β determined by the average QP employed over the block boundary, and deblocking only takes place if:
|p0−q0|<α(QP)
|p1−p0|<β(QP)
|q1−q0|<β(QP)  (10)
As the deblocking performance depends on these thresholds, the two thresholds α and β should be carefully selected.

In embodiments of the directional deblocking filters described herein, the two thresholds α and β are also used to determine whether the directional deblocking filter should be applied, or whether the edge at the block boundary is an actual edge that should be preserved without filtering. The two thresholds are desirably selected by not only considering QP but also the directional mode, Because directional deblocking is performed along the edge orientation (or along approximately the edge orientation), the filter decision thresholds to distinguish edges from blocking degradation are relaxed in certain embodiments. For example, in one particular implementation, Expression (10) is modified with the following filter decision thresholds:
α′=A(QP)B(mI)α
β′=A(QP)B(mI)β,  (11)
where A is a magnification parameter related to QP, and B is a modulation parameter related to the directional mode mI.

The magnification parameter A can be determined by evaluating a set of images containing typical directional edges.FIG.15is a graph1500showing the relationship between A and QP according to one particular embodiment. Using the relationships illustrated inFIG.15, at high bit rate (low QP), image details are still well preserved, so directional deblocking is less encouraged to avoid smoothing anti-directional texture, while at low bit rate (high QP), directional deblocking can be much stronger, as it will not blur the remaining edges. The relationships shown inFIG.15should not be construed as limiting, however, as various other relationships can be used in embodiments of the disclosed technology. For example, other relationships can also increase the value of A as QP is correspondingly increased. Furthermore, the value of A need not, increase in a step-wise fashion, but may increase more smoothly with increased QP.

The modulation parameter B can be defined according to a variety of different methods. In certain embodiments, for instance, the modulation parameter B is based on the observation that because blocking degradation is caused by different quantization in two adjacent blocks, it is most severe across the block boundary and least severe along the boundary. Consequently, and according to certain embodiments, deblocking should be stronger if its direction is more perpendicular to the boundary. Suppose, for example, that φ is the angle between the directional mode mIand the block boundary (refer toFIG.6). According to one particular implementation, the value of B can then be computed as follows:
B(mI)=sin φ  (9)
Compared with the H.264/AVC deblocking, the use of one or more of the magnification parameter A and the modulation parameter B provides not only filter orientation flexibility but also filter threshold adaptivity.
E. Exemplary Extra Smoothing Filters

In certain embodiments, when the extra smoothing mode is selected, the filtering can involve pixels that are in a line perpendicular to the block boundary. For instance, in one particular implementation, extra smoothing is performed along lines perpendicular to the upper and left block boundaries, as shown in block diagram1600ofFIG.16. Suppose pN-1, . . . , p0, q0, . . . , qN-1are the pixels in the same row or column of two adjacent blocks. In one exemplary implementation, the pixel values are modified as follows during the extra smoothing process:

pn=(∑i=0i=n+N2pi+∑j=0j=N2-n-1qj)/(N+1)⁢qn=(∑i=0i=N2-n-pi+∑j=0j=n+N2qj)/(N+1)⁢0≤n≤N2-1,N={16for⁢luminance⁢block8for⁢chroma⁢block(13)

FIG.16shows an example of Expression (13). For example purposes, the vertical boundary of a luminance block is shown, although the smoothing filter can be adapted for a horizontal boundary. As shown inFIG.16, p7, . . . , p0, q0, . . . , q7are the pixels to be filtered.FIG.16further shows particularly the filtering of q4, Specifically, the filtered value of q4is obtained by averaging p3, . . . , p0, q0, . . . , q12(shown in brackets1610,1612). Expression (13) should not be construed as limiting, however, as there exist a variety of modifications to the pixel values that can be performed to achieve a smoothing effect. The smoothing performed by expression (13) represents a heavy and long distance smoothing across the block boundary and pixels in a consecutive non-LDF region. For example, the smoothing filter is long distance in that it is applied to more than three pixels in each of the adjacent blocks. For instance, in one exemplary implementation, the filter is applied to 8 pixels in each of the adjacent blocks when the blocks are 16×16 luminance blocks and 4 pixels in each of the adjacent blocks when the blocks are 8×8 chrominance blocks. Other numbers of pixels in adjacent blocks can also be filtered by the smoothing filter. Furthermore, in certain embodiments, no boundary strength or filter thresholds are used with the extra smoothing mode.

F. An Exemplary Deblocking Process

FIG.7is a flowchart shown an exemplary deblocking process700. The illustrated method should not be construed as limiting, however, as any one or more of the illustrated acts can be performed alone or in various combinations and subcombinations with other deblocking processes.

At710, a reconstructed frame is input (e.g., buffered or loaded into a memory or otherwise received for processing). The reconstructed frame can be input, for example, after a predicted frame is combined with the reconstructed error as part of the motion compensation/estimation loop in an encoder or decoder.

At712, an initial deblocking filter is applied to the luminance values of the video frame. In one particular implementation, the initial deblocking filter is a standard deblocking filter, such as the H.264/AVC filter.

At714, edges in the video frame are detected using orientation energy edge detection (“OEED”). For example, an LDF partition is generated using the OEED filters described above with respect toFIG.8and Expressions (1) to (5).

At716, a block of the video frame is selected. The block can be selected in a raster fashion (from left to right starting with the top row). The block can alternatively be selected in another fashion (e.g., top-to-bottom wipe or box-out fashion).

At718, the selected block is evaluated to determine whether edge pixels are present in the selected block. This determination can be made, for example, by evaluating the LDF partition, which can mark those blocks that comprise one or more pixels identified as edge pixels after OEED (e.g., one or more edge pixels that satisfy Expression (5)). If edge pixels are present in the selected block, then the process proceeds at720; otherwise, the process proceeds at726.

At720, the selected block is evaluated to determine whether the block is an 8×8 or 4×4 block and whether the one or more directional features in the selected block are consistent with one or more neighboring blocks (e.g., the upper and leftward neighboring blocks). This determination can be made, for example, by evaluating the block size and the orientation averages and orientation variances of the selected block as well as the neighboring blocks (e.g., using expression (6) and (7)). In certain implementations, the directional features of the selected can be further evaluated to determine if they are horizontal or vertical. If the directional features of the selected block are consistent with the neighboring blocks and if the selected block is an 8×8 or 4×4 block (and, in certain implementations, if the directional features are non-vertical and non-horizontal), then the process proceeds with a directional deblocking scheme at722; otherwise, the process proceeds at730with the standard deblocking mode.

At722the selected block is evaluated to determine if certain directional deblocking thresholds are met. For example, the directional deblocking filter thresholds of Expression (10) are evaluated using the modifications to α and β shown in Expressions (11) and (12) andFIG.15. If the thresholds are satisfied, then a directional deblocking filter having the appropriate orientation is applied at724(e.g., one of the directional deblocking filters shown inFIG.14). If the thresholds are not satisfied, then no deblocking filter is applied, and the process proceeds at734.

Returning to726, the selected block is evaluated to determine if the block is a 16×16 block in a consecutive non-LDF region. This determination can be made, for example, by evaluating the block size and the intensity variances of the selected block as well as the intensity variances of one or more neighboring blocks (e.g., the upper and leftward blocks). This evaluation can be made, for example, using Expression (8). If the selected block is a 16×16 block in a consecutive non-LDF region, then an extra smoothing filter is applied at728. The smoothing filter can be the smoothing filter of Expression (13). If the selected block is not a 16×16 block in a consecutive non-LDF region, then a standard deblocking filter can be applied at730.

At730, the selected block is evaluated to determine whether it meets the standard deblocking filter activation thresholds (e.g., the thresholds shown in Expression (10)). If the thresholds are met, then a standard deblocking filter is applied at732(e.g., the standard H.264/AVC filter). If the thresholds are not met, then the selected block is not filtered with any deblocking filter.

With the selected block having been filtered using either a directional deblocking filter, a smoothing filter, or the standard deblocking filter or having not been filtered because of the filter activation thresholds not being met, the process continues at734, where an evaluation is made to determine if any further blocks in the reconstructed image remain. If so, the process returns to716, where the next block is selected, Otherwise, the process terminates for the video frame.

G. Experimental Results

Experiments were performed using an embodiment of the disclosed technology. In particular, experiments were performed using an embodiment of the deblocking scheme shown inFIG.7with the 8 OEED filters shown inFIG.8and described by Expressions (1) to (5), the deblocking modes being selected by Expressions (6) to (8), the directional deblocking filters ofFIG.14and described by Expression (9) with thresholds determined by Expressions (10) to (12) andFIG.15, the smoothing filter shown byFIG.7and described by Expression (13), and with the H.264/AVC deblocking scheme as the standard deblocking mode. This exemplary embodiment was tested using a variety of sequences provided by MPEG/VCEG for HVC (High-performance Video Coding) testing. The videos were coded in an IPPP structure, with 9 or 11 P-frames between every two I-frames. The frame rate was set to 10 fps or 12 fps, and a total of 10 seconds were coded within each sequence. The three thresholds T1, T2and T3for the deblocking mode decisions were set to 2, 3, and 12, respectively.

The results were compared with results from the regular H.264/AVC deblocking scheme as well as from the intra-predictive deblocking scheme described in Jeong. The experiments were only performed on I-frames, but the quality of inter-prediction for P-fames was similarly improved,

FIG.17shows images1700,1701obtained from H.264/AVC deblocking, images1702,1703obtained from intra-predictive deblocking, and images1704,1705obtained using the exemplary method. Each image includes LDF regions in an I-frame. It can be observed that artifacts still remain around the lines in images1700,1701after H.264/AVC deblocking. Furthermore, when compared with the intra-predictive deblocking images1702,1703, the images1704,1705from the exemplary embodiment show improved removal of visual artifacts, making the lines clearer and cleaner.

FIG.18shows images1800,1802, and1804of consecutive non-LDF regions taken from an I-frame. Image1800is a block-DCT compressed video frame with no deblocking. As can be seen in image1800, the block-DCT compressed video frame is susceptible to severe blocking artifacts in large smooth regions, Image1802shows the video frame after being deblocked with the H.264/AVC deblocking scheme. As seen in image1802, traditional H.264/AVC deblocking cannot effectively eliminate these artifacts. Image1804is the video frame after being deblocked using the exemplary embodiment. As can be seen in image1804, deblocking using the exemplary embodiment results in an image that is improved over the deblocking observed in image1800and1802.

FIG.19shows comparison results for different frame types. In particular, images1900and1901show deblocking applied to an I-frame using the H.264/AVC deblocking technique (image1900) and the exemplary embodiment (image1901). Images1902and1903show deblocking applied to a second P-frame using the H.264/AVC deblocking technique. Although both images1902and1903are deblocked using the H.264/AVC deblocking technique, the P-frame of image1903is predicted from the I-frame of image1901, which was deblocked using the exemplary embodiment. Images1904and1905show deblocking applied to a fourth P-frame using the H.264/AVC deblocking technique. Although both images1904and1905are deblocked using the H.264/AVC deblocking technique, the P-frame of image1905is predicted from the I-frame of image1901, which was deblocked using the exemplary embodiment. As can be seen in images1901,1903, and1905, the blocking artifacts are more effectively reduced not only on the I-frame but also on the P-frames when using the exemplary embodiment.

The exemplary embodiments disclosed herein are capable of improving the visual quality of block-DCT compressed video at low bit-rate, which is highly demanded in the next generation of video coding standards. Because the blocking artifacts are effectively removed using the exemplary embodiment, the objective quality of the image is typically improved. Some peak signal-to-noise ratio results for the exemplary embodiment are shown in table2000ofFIG.20. Table2000also shows the average percentage of “mode B” (which corresponds to the extra smoothing mode) and “mode C” (which corresponds to directional deblocking with the directional deblocking filters ofFIG.14and described by Expression (9) and with thresholds determined by Expressions (10) to (12) andFIG.15) selected during deblocking of the tested I-frames.

Having illustrated and described the principles of the illustrated embodiments, it will be apparent to those skilled in the art that the embodiments can be modified in arrangement and detail without departing from such principles. For example, the disclosed deblocking techniques are not limited to in-loop deblocking, but can also be used to perform post-processing deblocking in certain implementations.

In view of the many possible embodiments to which the principles of the disclosed invention may be applied, it should be recognized that the illustrated embodiments are only preferred examples of the invention and should not be taken as limiting the scope of the invention. Rather, the scope of the invention is defined by the following claims and their equivalents. We therefore claim as our invention all that comes within the scope and spirit of these claims and their equivalents.