In-loop deblocking for interlaced video

An encoder/decoder obtains pixel data from one or more field lines associated with a first block in an interlaced frame coded picture comprising plural macroblocks each having an equal number of top and bottom field lines. The encoder/decoder obtains pixel data from one or more field lines associated with a second block and performs in-loop deblocking filtering across a boundary. The in-loop deblocking filtering comprises filter operations performed on pixel data from field lines of same polarity only. In another aspect, an encoder/decoder obtains transform size information for plural blocks of macroblock, obtains field/frame type information for the macroblock and selects one or more boundaries for in-loop deblocking based at least in part on the transform size information and the field/frame type information. In-loop deblocking can be performed on horizontal block boundaries prior to vertical block boundaries.

TECHNICAL FIELD

Techniques and tools for interlaced video coding and decoding are described. For example, an encoder/decoder performs in-loop deblocking filtering for interlaced frame coded pictures.

BACKGROUND

Digital video consumes large amounts of storage and transmission capacity. A typical raw digital video sequence includes 15 or 30 pictures per second. Each picture can include tens or hundreds of thousands of pixels (also called pels). Each pixel represents a tiny element of the picture. In raw form, a computer commonly represents a pixel with 24 bits or more. Thus, the number of bits per second, or bit rate, of a typical raw digital video sequence can be 5 million bits/second or more.

Most computers and computer networks lack the resources to process raw digital video. For this reason, engineers use compression (also called coding or encoding) to reduce the bit rate of digital video. Compression can be lossless, in which quality of the video does not suffer but decreases in bit rate are limited by the complexity of the video. Or, compression can be lossy, in which quality of the video suffers but decreases in bit rate are more dramatic. Decompression reverses compression.

In general, video compression techniques include “intra” compression and “inter” or predictive compression. For progressively scanned video frames, intra compression techniques compress individual pictures, typically called I-frames or key frames. Inter compression techniques compress frames with reference to preceding and/or following frames, and inter-compressed frames are typically called predicted frames, P-frames, or B-frames.

I. Inter and Intra Compression in Windows Media Video, Versions 8 and 9

Microsoft Corporation's Windows Media Video, Version 8 [“WMV8”] includes a video encoder and a video decoder. The WMV8 encoder uses intra and inter compression, and the WMV8 decoder uses intra and inter decompression. Windows Media Video, Version 9 [“WMV9”] uses a similar architecture for many operations.

FIG. 1illustrates block-based intra compression100of a block105of pixels in a key frame in the WMV8 encoder. A block is a set of pixels, for example, an 8×8 arrangement of pixels. The WMV8 encoder splits a key video frame into 8×8 blocks of pixels and applies an 8×8 Discrete Cosine Transform [“DCT”]110to individual blocks such as the block105. A DCT is a type of frequency transform that converts the 8×8 block of pixels (spatial information) into an 8×8 block of DCT coefficients115, which are frequency information. The DCT operation itself is lossless or nearly lossless. Compared to the original pixel values, however, the DCT coefficients are more efficient for the encoder to compress since most of the significant information is concentrated in low frequency coefficients (conventionally, the upper left of the block115) and many of the high frequency coefficients (conventionally, the lower right of the block115) have values of zero or close to zero.

The encoder then quantizes120the DCT coefficients, resulting in an 8×8 block of quantized DCT coefficients125. For example, the encoder applies a uniform, scalar quantization step size to each coefficient. Quantization is lossy. Since low frequency DCT coefficients tend to have higher values, quantization results in loss of precision but not complete loss of the information for the coefficients. On the other hand, since high frequency DCT coefficients tend to have values of zero or close to zero, quantization of the high frequency coefficients typically results in contiguous regions of zero values. In addition, in some cases high frequency DCT coefficients are quantized more coarsely than low frequency DCT coefficients, resulting in greater loss of precision/information for the high frequency DCT coefficients.

The encoder then prepares the 8×8 block of quantized DCT coefficients125for entropy encoding, which is a form of lossless compression. The exact type of entropy encoding can vary depending on whether a coefficient is a DC coefficient (lowest frequency), an AC coefficient (other frequencies) in the top row or left column, or another AC coefficient.

The encoder encodes the DC coefficient126as a differential from the DC coefficient136of a neighboring 8×8 block, which is a previously encoded neighbor (e.g., top or left) of the block being encoded. (FIG. 1shows a neighbor block135that is situated to the left of the block being encoded in the frame.) The encoder entropy encodes140the differential.

The entropy encoder can encode the left column or top row of AC coefficients as a differential from a corresponding column or row of the neighboring 8×8 block.FIG. 1shows the left column127of AC coefficients encoded as a differential147from the left column137of the neighboring (to the left) block135. The differential coding increases the chance that the differential coefficients have zero values. The remaining AC coefficients are from the block125of quantized DCT coefficients.

The encoder scans150the 8×8 block145of predicted, quantized AC DCT coefficients into a one-dimensional array155and then entropy encodes the scanned AC coefficients using a variation of run length coding160. The encoder selects an entropy code from one or more run/level/last tables165and outputs the entropy code.

Inter compression in the WMV8 encoder uses block-based motion compensated prediction coding followed by transform coding of the residual error.FIGS. 2 and 3illustrate the block-based inter compression for a predicted frame in the WMV8 encoder. In particular,FIG. 2illustrates motion estimation for a predicted frame210andFIG. 3illustrates compression of a prediction residual for a motion-compensated block of a predicted frame.

For example, inFIG. 2, the WMV8 encoder computes a motion vector for a macroblock215in the predicted frame210. To compute the motion vector, the encoder searches in a search area235of a reference frame230. Within the search area235, the encoder compares the macroblock215from the predicted frame210to various candidate macroblocks in order to find a candidate macroblock that is a good match. The encoder outputs information specifying the motion vector (entropy coded) for the matching macroblock.

The encoder can encode the differential between the motion vector and the motion vector predictor. After reconstructing the motion vector by adding the differential to the predictor, a decoder uses the motion vector to compute a prediction macroblock for the macroblock215using information from the reference frame230, which is a previously reconstructed frame available at the encoder and the decoder. The prediction is rarely perfect, so the encoder usually encodes blocks of pixel differences (also called the error or residual blocks) between the prediction macroblock and the macroblock215itself.

FIG. 3illustrates an example of computation and encoding of an error block335in the WMV8 encoder. The error block335is the difference between the predicted block315and the original current block325. The encoder applies a DCT340to the error block335, resulting in an 8×8 block345of coefficients. The encoder then quantizes350the DCT coefficients, resulting in an 8×8 block of quantized DCT coefficients355. The encoder scans360the 8×8 block355into a one-dimensional array365such that coefficients are generally ordered from lowest frequency to highest frequency. The encoder entropy encodes the scanned coefficients using a variation of run length coding370. The encoder selects an entropy code from one or more run/level/last tables375and outputs the entropy code.

FIG. 4shows an example of a corresponding decoding process400for an inter-coded block. In summary ofFIG. 4, a decoder decodes (410,420) entropy-coded information representing a prediction residual using variable length decoding410with one or more run/level/last tables415and run length decoding420. The decoder inverse scans430a one-dimensional array425storing the entropy-decoded information into a two-dimensional block435. The decoder inverse quantizes and inverse discrete cosine transforms (together,440) the data, resulting in a reconstructed error block445. In a separate motion compensation path, the decoder computes a predicted block465using motion vector information455for displacement from a reference frame. The decoder combines470the predicted block465with the reconstructed error block445to form the reconstructed block475.

The amount of change between the original and reconstructed frames is the distortion and the number of bits required to code the frame indicates the rate for the frame. The amount of distortion is roughly inversely proportional to the rate.

II. Interlaced Video and Progressive Video

A video frame contains lines of spatial information of a video signal. For progressive video, these lines contain samples starting from one time instant and continuing through successive lines to the bottom of the frame. A progressive I-frame is an intra-coded progressive video frame. A progressive P-frame is a progressive video frame coded using forward prediction, and a progressive B-frame is a progressive video frame coded using bi-directional prediction.

A typical interlaced video frame consists of two fields scanned starting at different times. For example, referring toFIG. 5, an interlaced video frame500includes top field510and bottom field520. Typically, the even-numbered lines (top field) are scanned starting at one time (e.g., time t) and the odd-numbered lines (bottom field) are scanned starting at a different (typically later) time (e.g., time t+1). This timing can create jagged tooth-like features in regions of an interlaced video frame where motion is present when the two fields are scanned starting at different times. For this reason, interlaced video frames can be rearranged according to a field structure, with the odd lines grouped together in one field, and the even lines grouped together in another field. This arrangement, known as field coding, is useful in high-motion pictures for reduction of such jagged edge artifacts. On the other hand, in stationary regions, image detail in the interlaced video frame may be more efficiently preserved without such a rearrangement. Accordingly, frame coding is often used in stationary or low-motion interlaced video frames, in which the original alternating field line arrangement is preserved.

A previous WMV encoder and decoder use macroblocks that are arranged according to a field structure (field-coded macroblocks) or a frame structure (frame-coded macroblocks) in interlaced video frames.FIG. 6shows how field permuting is used to produce field-coded macroblocks in the encoder and decoder. An interlaced macroblock610is permuted such that all the top field lines (e.g., even-numbered lines0,2, . . .14) are placed in the top half of the field-coded macroblock620, and all the bottom field lines (e.g., odd-numbered lines1,3, . . .15) are placed in the bottom half of the field-coded macroblock. For a frame-coded macroblock, the top field lines and bottom field lines alternate throughout the macroblock, as in interlaced macroblock610.

The encoder and decoder use a 4:1:1 macroblock format in interlaced frames. A 4:1:1 macroblock is composed of four 8×8 luminance blocks and two 4×8 blocks of each chrominance channel. In a field-coded 4:1:1 macroblock, the permuted macroblock is subdivided such that the top two 8×8 luminance blocks and the top 4×8 chrominance block in each chrominance channel contain only top field lines, while the bottom two 8×8 luminance blocks and the bottom 4×8 chrominance block in each chrominance channel contain only bottom field lines.

A typical progressive video frame consists of one frame of content with non-alternating lines. In contrast to interlaced video, progressive video does not divide video frames into separate fields, and an entire frame is scanned left to right, top to bottom starting at a single time.

II. Loop Filtering in a Previous WMV Encoder and Decoder

Quantization and other lossy processing of prediction residuals can cause blocking artifacts at block boundaries. Blocking artifacts can be especially troublesome in reference frames that are used for motion estimation and compensation of subsequent predicted frames. To reduce blocking artifacts, a previous WMV video encoder and decoder use a deblocking filter to smooth boundary discontinuities between 8×8 blocks in motion estimation/compensation loops. For example, a video encoder processes a reconstructed reference frame to reduce blocking artifacts prior to motion estimation/compensation using the reference frame, and a video decoder processes a reconstructed reference frame to reduce blocking artifacts prior to motion compensation using the reference frame. The deblocking filter improves the quality of motion estimation/compensation, resulting in better prediction and lower bitrate for prediction residuals.

A. In-loop Deblocking Filtering for Progressive Frames

The encoder and decoder perform in-loop deblocking filtering for progressive frames prior to using a reconstructed frame as a reference for motion estimation/compensation. The filtering process operates on pixels (or more precisely, on samples at pixel locations) that border neighboring blocks. The locations of block boundaries depend on the size of the inverse transform used. For progressive P-frames the block boundaries may occur at every 4th or 8th pixel row or column depending on whether an 8×8, 8×4 or 4×8 inverse transform is used. For progressive I-frames, where an 8×8 transform is used, block boundaries occur at every 8th pixel row and column.

For progressive I-frames, deblocking filtering is performed adaptively at all 8×8 block boundaries.FIGS. 7 and 8show the pixels that are filtered along the horizontal and vertical border regions in the upper left corner of a component (luma, Cbor Cr) plane.FIG. 7shows filtered vertical block boundary pixels in an I-frame.FIG. 8shows filtered horizontal block boundary pixels in an I-frame.

InFIGS. 7 and 8, crosses represent pixels (actually samples for pixels) and circled crosses represent filtered pixels. As these figures show, the top horizontal line and first vertical line in the frame are not filtered, even though they lie on a block boundary, because these lines lie on the border of the frame. Although not depicted, the bottom horizontal line and last vertical line in the frame also are not filtered for the same reason. In more formal terms, the following lines are filtered:Horizontal lines: (7, 8), (15, 16) . . . ((N−1)*8−1, (N−1)*8)Vertical lines: (7, 8), (15, 16) . . . ((M−1)*8−1, (M−1)*8)(N=number of horizontal 8×8 blocks in the plane (N*8=horizontal frame size))(M=number of vertical 8×8 blocks in the frame (M*8=vertical frame size))
For progressive I-frames, all horizontal boundary lines in the frame are filtered first, followed by the vertical boundary lines.

For progressive P-frames, blocks can be intra or inter-coded. The encoder and decoder use an 8×8 transform to transform the samples in intra-coded blocks, and the 8×8 block boundaries are always adaptively filtered. The encoder and decoder use an 8×8, 8×4, 4×8 or 4×4 transform for inter-coded blocks and a corresponding inverse transform to construct the samples that represent the residual error. Depending on the status of the neighboring blocks, the boundary between the current and neighboring blocks may or may not be adaptively filtered. The boundaries between coded (at least one non-zero coefficient) subblocks (8×4, 4×8 or 4×4) within an 8×8 block are always adaptively filtered. The boundary between a block or subblock and a neighboring block or subblock is not filtered only if both blocks are inter-coded, have the same motion vector, and have no residual error (no transform coefficients), otherwise the boundary is filtered.

FIG. 9shows examples of when filtering between neighboring blocks does and does not occur in progressive P-frames. InFIG. 9, it is assumed that the motion vectors for both blocks are the same (if the motion vectors are different, the boundary is always filtered). The shaded blocks or subblocks represent the cases where at least one nonzero coefficient is present. Clear blocks or subblocks represent cases where no transform coefficients are present. Thick lines represent the boundaries that are adaptively filtered. Thin lines represent the boundaries that are not filtered.FIG. 9illustrates only horizontal macroblock neighbors, but a previous WMV encoder and decoder applies similar rules to vertical neighbors.

FIGS. 10 and 11show an example of pixels that may be filtered in a progressive P-frame. The crosses represent pixel locations and the circled crosses represent the boundary pixels that are adaptively filtered if the conditions specified above are met.FIG. 10shows pixels filtered along horizontal boundaries. AsFIG. 10shows, the pixels on either side of the block or subblock boundary are candidates to be filtered. For the horizontal boundaries, this could be every 4thand 5th, 8thand 9th, 12thand 13th, etc., pixel row in the frame.FIG. 11shows candidate pixels to be filtered along vertical boundaries. For the vertical boundaries, every 4thand 5th, 8thand 9th, 12thand 13th, etc., pixel column in the frame may be adaptively filtered as these are the 8×8 and 4×8 vertical boundaries. The first and last row and the first and last column in the frame are not filtered.

All the 8×8 block horizontal boundary lines in the frame are adaptively filtered first, starting from the top line. Next, all 8×4 block horizontal boundary lines in the frame are adaptively filtered starting from the top line. Next, all 8×8 block vertical boundary lines are adaptively filtered starting from the leftmost line. Lastly, all 4×8 block vertical boundary lines are adaptively filtered starting with the leftmost line. The rules specified above are used to determine whether the boundary pixels are actually filtered for each block or subblock.

3. Filtering Operations

For progressive P-frames the decision criteria described above determine which vertical and horizontal boundaries are adaptively filtered. Since the minimum number of consecutive pixels that are filtered in a row or column is four and the total number of pixels in a row or column is always a multiple of four, the filtering operation is performed on segments of four pixels.

For example, if the eight pixel pairs that make up the vertical boundary between two blocks are adaptively filtered, then the eight pixels are divided into two 4-pixel segments as shown inFIG. 12. In each 4-pixel segment, the third pixel pair is adaptively filtered first as indicated by the Xs inFIG. 12. The result of this adaptive filter operation determines whether the other three pixels in the segment are also filtered.

FIG. 13shows the pixels that are used in the adaptive filtering operation performed on the 3rdpixel pair. InFIG. 13, pixels P4and P5are the pixel pair that may be changed in the filter operation.

The pseudo-code1400ofFIG. 14shows the adaptive filtering operation performed on the 3rdpixel pair in each segment. The value filter_other—3_pixels indicates whether the remaining three pixel pairs in the segment are also filtered. If filter_other—3_pixels=TRUE, then the other three pixel pairs are adaptively filtered. If filter_other—3_pixels=FALSE, then they are not filtered, and the adaptive filtering operation proceeds to the next 4-pixel segment. The pseudo-code1500ofFIG. 15shows the adaptive filtering operation that is performed on the 1st, 2ndand 4thpixel pair if filter_other—3_pixels=TRUE. In pseudo-code1400and pseudo-code1500, the variable PQUANT represents a quantization step size.

The filtering operations described above are similarly used for filtering horizontal boundary pixels.

The encoder and decoder perform in-loop deblocking filtering across vertical boundaries in interlaced frames having a 4:1:1 macroblock format. For interlaced I- and P-frames, adaptive filtering can occur for pixels located immediately on the left and right of a vertical block boundary except for those located on the picture boundaries (i.e., the first and last column of the luminance and chrominance components). InFIG. 16, pixels (more precisely, samples) that are candidates for filtering in a typical 4:1:1 macroblock in the encoder and decoder are marked M or B, where M denotes boundary pixels located across macroblock boundaries and B denotes boundary pixels located within the macroblock.

The decision on whether to filter across a vertical boundary is made on a block-by-block basis. In a 4:1:1 frame-coded macroblock, each block contains eight consecutive alternating lines of the top and bottom fields in the macroblock. In a 4:1:1 field-coded macroblock, a block contains either eight top field lines or eight bottom field lines. The filtering decision is made eight lines at a time.

The decision to filter across a vertical block boundary depends on whether the current block and the left neighboring block are frame-coded or field-coded (field/frame type), whether they are intra-coded or inter-coded, and whether they have nonzero transform coefficients. In general, the vertical block boundary pixels are adaptively filtered unless the current block's field/frame type is the same as the left neighboring block's field/frame type, both blocks are not intra-coded, and both have no nonzero transform coefficients, in which case the block boundary is not filtered. Chroma block boundaries are adaptively filtered if the corresponding luminance block boundaries are adaptively filtered. Horizontal boundaries are not filtered.

Although the encoder and decoder adaptively filter block boundaries depending in part on the field/frame type of the neighboring blocks, they do not take transform size into account when making filtering decisions in interlaced frames.

VI. Standards for Video Compression and Decompression

Several international standards relate to video compression and decompression. These standards include the Motion Picture Experts Group [“MPEG”] 1, 2, and 4 standards and the H.261, H.262 (another title for MPEG 2), H.263 and H.264 (also called JVT/AVC) standards from the International Telecommunication Union [“ITU”]. These standards specify aspects of video decoders and formats for compressed video information. Directly or by implication, they also specify certain encoder details, but other encoder details are not specified. These standards use (or support the use of) different combinations of intraframe and interframe decompression and compression.

A. Loop Filtering in the Standards

As in the previous WMV encoders and decoders, some international standards use deblocking filters to reduce the effect of blocking artifacts in reconstructed frames. The H.263 standard includes an optional deblocking filter mode in which a filter is applied across 8×8 block edge boundaries of decoded I- and P-frames (but not B-frames) to reduce blocking artifacts. Annex J of the H.263 standard describes an optional block edge filter within the coding loop in which filtering is performed on 8×8 block edges (referred to in H.263 as a deblocking edge filter). This filter affects the reconstructed pictures used for prediction of other pictures. The deblocking edge filter operates using a set of four clipped pixel values on a horizontal and/or vertical line, where two of the four values are in one block (e.g., the top block among neighboring top and bottom blocks) and the other two values are in another block (e.g., the bottom block among neighboring top and bottom blocks). Filtering across horizontal edges is performed before filtering across vertical edges to reduce rounding effects. This optional filtering mode can be signaled in the bitstream with a single bit in a field of a picture header.

According to draft JVT-d157 of the JVT/AVC video standard, deblocking filtering is performed on a macroblock basis. In interlaced frames, macroblocks are grouped into macroblock pairs (top and bottom). Macroblock pairs can be field-coded or frame-coded. In a frame-coded macroblock pair, the macroblock pair is decoded as two frame-coded macroblocks. In a field-coded macroblock pair, the top macroblock consists of the top-field lines in the macroblock pair, and the bottom macroblock consists of the bottom-field lines in the macroblock pair.

Sections 8.7 and 12.4.4 of draft JVT-d157 describe deblocking filtering. For frame-coded macroblock pairs, deblocking is performed on the frame samples, and if neighboring macroblock pair is a field macroblock pair, the neighboring field macroblock pair is converted into a frame macroblock pair before deblocking. For field-coded macroblock pairs, deblocking is performed on the field samples of the same field parity, and if a neighboring macroblock pair is a frame macroblock pairs, it is converted into a field macroblock pair before deblocking. For field-coded pictures, all decoding operations for the deblocking filter are based solely on samples within the current field. For luma filtering in a 16×16 macroblock with 16 4×4 blocks, the 16 samples of the four vertical edges of the 4×4 raster scan pattern are filtered beginning with the left edge, and the four horizontal edges are filtered beginning with the top edge. For chroma filtering, two edges of eight samples each are filtered in each direction. For additional detail, see JVT-d157.

B. Limitations of the Standards

These international standards are limited in several important ways. For example, H.263 does not describe loop filtering for interlaced video. Draft JVT-d157 of the JVT/AVC video standard describes loop filtering only for macroblock pairs in interlaced video, and does not describe, for example, loop filtering for an individual field-coded macroblock having a top field and a bottom field within the same macroblock, or loop filtering decisions for blocks or sub-blocks larger than 4×4.

Given the critical importance of video compression and decompression to digital video, it is not surprising that video compression and decompression are richly developed fields. Whatever the benefits of previous video compression and decompression techniques, however, they do not have the advantages of the following techniques and tools.

SUMMARY

In summary, the detailed description is directed to various techniques and tools for encoding and decoding interlaced video frames. Described embodiments implement one or more of the described techniques and tools including, but not limited to, the following:

In one aspect, an encoder/decoder obtains pixel data (e.g., chrominance or luminance samples) from one or more field lines (e.g., top field lines or bottom field lines) associated with a first block of a macroblock in an interlaced frame coded picture (e.g., interlaced I-frame, interlaced P-frame, interlaced B-frame, etc.) comprising plural macroblocks (e.g., 4:2:0 macroblocks). Each of the plural macroblocks has an equal number of top field lines and bottom field lines. The encoder/decoder obtains pixel data from one or more field lines associated with a second block in the picture and performs in-loop deblocking filtering across a boundary (e.g., a horizontal or vertical block boundary comprising at least one four-pixel segment) using the obtained pixel data. The in-loop deblocking filtering comprises filter operations performed on pixel data from field lines of same polarity only and can be described as field-based deblocking. Each of the plural macroblocks can be coded according to a field structure or a frame structure, which can be indicated by a transform type. The first block and the second block can each have a transform size selected from a group consisting of: 8×8, 8×4, 4×8, and 4×4.

In another aspect, an encoder/decoder obtains field/frame type information for a current macroblock in an interlaced frame coded picture. The encoder/decoder also obtains transform size information for plural blocks in the current macroblock. The encoder/decoder selects one or more boundaries for in-loop deblocking based at least in part on the transform size information and the field/frame type information. The encoder/decoder performs in-loop deblocking (e.g., field-based deblocking) on the selected boundaries. The field/frame transform type information indicates, for example, whether the current macroblock is coded according to a field structure or a frame structure. The selecting of one or more boundaries for in-loop deblocking can be further based on the picture type information (e.g., whether the interlaced frame coded picture is an interlaced I-frame, P-frame or B-frame).

In another aspect, an encoder/decoder obtains field/frame type information for a current macroblock, obtains transform size information for plural blocks in the macroblock, selects a boundary between a first block in the macroblock and a second block for in-loop deblocking based at least in part on the transform size information and the field/frame type information, obtains pixel data from one or more field lines associated with the first block and from one or more field lines associated with the second block, and performs in-loop deblocking across the boundary using the obtained pixel data. The in-loop deblocking comprises filtering operations performed on pixel data from field lines of same polarity only.

The various techniques and tools can be used in combination or independently.

Additional features and advantages will be made apparent from the following detailed description of different embodiments that proceeds with reference to the accompanying drawings.

DETAILED DESCRIPTION

The present application relates to techniques and tools for efficient compression and decompression of interlaced video. In various described embodiments, a video encoder and decoder incorporate techniques for encoding and decoding interlaced video, and corresponding signaling techniques for use with a bit stream format or syntax comprising different layers or levels (e.g., sequence level, frame level, field level, macroblock level, and/or block level).

Various alternatives to the implementations described herein are possible. For example, techniques described with reference to flowchart diagrams can be altered by changing the ordering of stages shown in the flowcharts, by repeating or omitting certain stages, etc. As another example, although some implementations are described with reference to specific macroblock formats, other formats also can be used. Further, techniques and tools described with reference to forward prediction may also be applicable to other types of prediction.

The various techniques and tools can be used in combination or independently. Different embodiments implement one or more of the described techniques and tools. Some techniques and tools described herein can be used in a video encoder or decoder, or in some other system not specifically limited to video encoding or decoding.

I. Computing Environment

FIG. 17illustrates a generalized example of a suitable computing environment1700in which several of the described embodiments may be implemented. The computing environment1700is not intended to suggest any limitation as to scope of use or functionality, as the techniques and tools may be implemented in diverse general-purpose or special-purpose computing environments.

With reference toFIG. 17, the computing environment1700includes at least one processing unit1710and memory1720. InFIG. 17, this most basic configuration1730is included within a dashed line. The processing unit1710executes computer-executable instructions and may be a real or a virtual processor. In a multi-processing system, multiple processing units execute computer-executable instructions to increase processing power. The memory1720may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two. The memory1720stores software1780implementing a video encoder or decoder with one or more of the described techniques and tools.

A computing environment may have additional features. For example, the computing environment1700includes storage1740, one or more input devices1750, one or more output devices1760, and one or more communication connections1770. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing environment1700. Typically, operating system software (not shown) provides an operating environment for other software executing in the computing environment1700, and coordinates activities of the components of the computing environment1700.

The storage1740may be removable or non-removable, and includes magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any other medium which can be used to store information and which can be accessed within the computing environment1700. The storage1740stores instructions for the software1780implementing the video encoder or decoder.

The input device(s)1750may be a touch input device such as a keyboard, mouse, pen, or trackball, a voice input device, a scanning device, or another device that provides input to the computing environment1700. For audio or video encoding, the input device(s)1750may be a sound card, video card, TV tuner card, or similar device that accepts audio or video input in analog or digital form, or a CD-ROM or CD-RW that reads audio or video samples into the computing environment1700. The output device(s)1760may be a display, printer, speaker, CD-writer, or another device that provides output from the computing environment1700.

The techniques and tools can be described in the general context of computer-readable media. Computer-readable media are any available media that can be accessed within a computing environment. By way of example, and not limitation, with the computing environment1700, computer-readable media include memory1720, storage1740, communication media, and combinations of any of the above.

For the sake of presentation, the detailed description uses terms like “determine,” “decide,” and “apply” to describe computer operations in a computing environment. These terms are high-level abstractions for operations performed by a computer, and should not be confused with acts performed by a human being. The actual computer operations corresponding to these terms vary depending on implementation.

II. Generalized Video Encoder and Decoder

FIG. 18is a block diagram of a generalized video encoder1800in conjunction with which some described embodiments may be implemented.FIG. 19is a block diagram of a generalized video decoder1900in conjunction with which some described embodiments may be implemented.

The relationships shown between modules within the encoder1800and decoder1900indicate general flows of information in the encoder and decoder; other relationships are not shown for the sake of simplicity. In particular,FIGS. 18 and 19usually do not show side information indicating the encoder settings, modes, tables, etc. used for a video sequence, picture, macroblock, block, etc. Such side information is sent in the output bitstream, typically after entropy encoding of the side information. The format of the output bitstream can be a Windows Media Video version 9 format or other format.

The encoder1800and decoder1900process video pictures, which may be video frames, video fields or combinations of frames and fields. The bitstream syntax and semantics at the picture and macroblock levels may depend on whether frames or fields are used. There may be changes to macroblock organization and overall timing as well. The encoder1800and decoder1900are block-based and use a 4:2:0 macroblock format for frames, with each macroblock including four 8×8 luminance blocks (at times treated as one 16×16 macroblock) and two 8×8 chrominance blocks. For fields, the same or a different macroblock organization and format may be used. The 8×8 blocks may be further sub-divided at different stages, e.g., at the frequency transform and entropy encoding stages. Example video frame organizations are described in more detail below. Alternatively, the encoder1800and decoder1900are object-based, use a different macroblock or block format, or perform operations on sets of pixels of different size or configuration than 8×8 blocks and 16×16 macroblocks.

Depending on implementation and the type of compression desired, modules of the encoder or decoder can be added, omitted, split into multiple modules, combined with other modules, and/or replaced with like modules. In alternative embodiments, encoders or decoders with different modules and/or other configurations of modules perform one or more of the described techniques.

A. Video Frame Organizations

In some implementations, the encoder1800and decoder1900process video frames organized as follows. A frame contains lines of spatial information of a video signal. For progressive video, these lines contain samples starting from one time instant and continuing through successive lines to the bottom of the frame. A progressive video frame is divided into macroblocks such as the macroblock2000shown inFIG. 20. The macroblock2000includes four 8×8 luminance blocks (Y0through Y3) and two 8×8 chrominance blocks that are co-located with the four luminance blocks but half resolution horizontally and vertically, following the conventional 4:2:0 macroblock format. The 8×8 blocks may be further sub-divided at different stages, e.g., at the frequency transform (e.g., 8×4, 4×8 or 4×4 DCTs) and entropy encoding stages. A progressive I-frame is an intra-coded progressive video frame. A progressive P-frame is a progressive video frame coded using forward prediction, and a progressive B-frame is a progressive video frame coded using bi-directional prediction. Progressive P- and B-frames may include intra-coded macroblocks as well as different types of predicted macroblocks.

An interlaced video frame consists of two scans of a frame—one comprising the even lines of the frame (the top field) and the other comprising the odd lines of the frame (the bottom field). The two fields may represent two different time periods or they may be from the same time period.FIG. 21Ashows part of an interlaced video frame2100, including the alternating lines of the top field and bottom field at the top left part of the interlaced video frame2100.

FIG. 21Bshows the interlaced video frame2100ofFIG. 21Aorganized for encoding/decoding as a frame2130. The interlaced video frame2100has been partitioned into macroblocks such as the macroblocks2131and2132, which use a 4:2:0 format as shown inFIG. 20. In the luminance plane, each macroblock2131,2132includes 8 lines from the top field alternating with 8 lines from the bottom field for 16 lines total, and each line is 16 pixels long. (The actual organization and placement of luminance blocks and chrominance blocks within the macroblocks2131,2132are not shown, and in fact may vary for different encoding decisions.) Within a given macroblock, the top-field information and bottom-field information may be coded jointly or separately at any of various phases. The macroblock itself may be field transform coded or frame transform coded. Field and frame transform coding for macroblocks is described in further detail below.

An interlaced I-frame is two intra-coded fields of an interlaced video frame, where a macroblock includes information for the two fields. An interlaced P-frame is two fields of an interlaced video frame coded using forward prediction, and an interlaced B-frame is two fields of an interlaced video frame coded using bi-directional prediction, where a macroblock includes information for the two fields. Interlaced P- and B-frames may include intra-coded macroblocks as well as different types of predicted macroblocks. Interlaced BI-frames are a hybrid of interlaced I-frames and interlaced B-frames; they are intra-coded, but are not used as anchors for other frames.

FIG. 21Cshows the interlaced video frame2100ofFIG. 21Aorganized for encoding/decoding as fields2160. Each of the two fields of the interlaced video frame2100is partitioned into macroblocks. The top field is partitioned into macroblocks such as the macroblock2161, and the bottom field is partitioned into macroblocks such as the macroblock2162. (Again, the macroblocks use a 4:2:0 format as shown inFIG. 20, and the organization and placement of luminance blocks and chrominance blocks within the macroblocks are not shown.) In the luminance plane, the macroblock2161includes 16 lines from the top field and the macroblock2162includes 16 lines from the bottom field, and each line is 16 pixels long. An interlaced I-field is a single, separately represented field of an interlaced video frame. An interlaced P-field is a single, separately represented field of an interlaced video frame coded using forward prediction, and an interlaced B-field is a single, separately represented field of an interlaced video frame coded using bi-directional prediction. Interlaced P- and B-fields may include intra-coded macroblocks as well as different types of predicted macroblocks. Interlaced BI-fields are a hybrid of interlaced I-fields and interlaced B-fields; they are intra-coded, but are not used as anchors for other fields.

Interlaced video frames organized for encoding/decoding as fields can include various combinations of different field types. For example, such a frame can have the same field type in both the top and bottom fields or different field types in each field. In one implementation, the possible combinations of field types include I/I, I/P, P/I, P/P, B/B, B/BI, BI/B, and BI/BI.

The term picture generally refers to source, coded or reconstructed image data. For progressive video, a picture is a progressive video frame. For interlaced video, a picture may refer to an interlaced video frame, the top field of the frame, or the bottom field of the frame, depending on the context.

Alternatively, the encoder1800and decoder1900are object-based, use a different macroblock or block format, or perform operations on sets of pixels of different size or configuration than 8×8 blocks and 16×16 macroblocks.

B. Video Encoder

FIG. 18is a block diagram of a generalized video encoder system1800. The encoder system1800receives a sequence of video pictures including a current picture1805(e.g., progressive video frame, interlaced video frame, or field of an interlaced video frame), and produces compressed video information1895as output. Particular embodiments of video encoders typically use a variation or supplemented version of the generalized encoder1800.

The encoder system1800compresses predicted pictures and key pictures. For the sake of presentation,FIG. 18shows a path for key pictures through the encoder system1800and a path for predicted pictures. Many of the components of the encoder system1800are used for compressing both key pictures and predicted pictures. The exact operations performed by those components can vary depending on the type of information being compressed.

A predicted picture (e.g., progressive P-frame or B-frame, interlaced P-field or B-field, or interlaced P-frame or B-frame) is represented in terms of prediction (or difference) from one or more other pictures (which are typically referred to as reference pictures or anchors). A prediction residual is the difference between what was predicted and the original picture. In contrast, a key picture (e.g., progressive I-frame, interlaced I-field, or interlaced I-frame) is compressed without reference to other pictures.

If the current picture1805is a forward-predicted picture, a motion estimator1810estimates motion of macroblocks or other sets of pixels of the current picture1805with respect to one or more reference pictures, for example, the reconstructed previous picture1825buffered in the picture store1820. If the current picture1805is a bi-directionally-predicted picture, a motion estimator1810estimates motion in the current picture1805with respect to up to four reconstructed reference pictures (for an interlaced B-field, for example). Typically, a motion estimator estimates motion in a B-picture with respect to one or more temporally previous reference pictures and one or more temporally future reference pictures. Accordingly, the encoder system1800can use the separate stores1820and1822for multiple reference pictures. For more information on progressive B-frames and interlaced B-frames and B-fields, see U.S. patent application Ser. No. 10/622,378, entitled, “Advanced Bi-Directional Predictive Coding of Video Frames,” filed Jul. 18, 2003, and U.S. patent application Ser. No. 10/882,135, entitled, “Advanced Bi-Directional Predictive Coding of Interlaced Video,” filed Jun. 29, 2004, which is hereby incorporated herein by reference.

The motion estimator1810can estimate motion by pixel, ½ pixel, ¼ pixel, or other increments, and can switch the precision of the motion estimation on a picture-by-picture basis or other basis. The motion estimator1810(and compensator1830) also can switch between types of reference picture pixel interpolation (e.g., between bicubic and bilinear) on a per-frame or other basis. The precision of the motion estimation can be the same or different horizontally and vertically. The motion estimator1810outputs as side information motion information1815such as differential motion vector information. The encoder1800encodes the motion information1815by, for example, computing one or more predictors for motion vectors, computing differentials between the motion vectors and predictors, and entropy coding the differentials. To reconstruct a motion vector, a motion compensator1830combines a predictor with differential motion vector information.

The motion compensator1830applies the reconstructed motion vector to the reconstructed picture(s)1825to form a motion-compensated current picture1835. The prediction is rarely perfect, however, and the difference between the motion-compensated current picture1835and the original current picture1805is the prediction residual1845. During later reconstruction of the picture, the prediction residual1845is added to the motion compensated current picture1835to obtain a reconstructed picture that is closer to the original current picture1805. In lossy compression, however, some information is still lost from the original current picture1805. Alternatively, a motion estimator and motion compensator apply another type of motion estimation/compensation.

A frequency transformer1860converts the spatial domain video information into frequency domain (i.e., spectral) data. For block-based video pictures, the frequency transformer1860applies a DCT, variant of DCT, or other block transform to blocks of the pixel data or prediction residual data, producing blocks of frequency transform coefficients. Alternatively, the frequency transformer1860applies another conventional frequency transform such as a Fourier transform or uses wavelet or sub-band analysis. The frequency transformer1860may apply an 8×8, 8×4, 4×8, 4×4 or other size frequency transform.

A quantizer1870then quantizes the blocks of spectral data coefficients. The quantizer applies uniform, scalar quantization to the spectral data with a step-size that varies on a picture-by-picture basis or other basis. Alternatively, the quantizer applies another type of quantization to the spectral data coefficients, for example, a non-uniform, vector, or non-adaptive quantization, or directly quantizes spatial domain data in an encoder system that does not use frequency transformations. In addition to adaptive quantization, the encoder1800can use frame dropping, adaptive filtering, or other techniques for rate control.

The encoder1800may use special signaling for a skipped macroblock, which is a macroblock that has no information of certain types (e.g., no motion information for the macroblock and no residual information).

When a reconstructed current picture is needed for subsequent motion estimation/compensation, an inverse quantizer1876performs inverse quantization on the quantized spectral data coefficients. An inverse frequency transformer1866then performs the inverse of the operations of the frequency transformer1860, producing a reconstructed prediction residual (for a predicted picture) or a reconstructed key picture. If the current picture1805was a key picture, the reconstructed key picture is taken as the reconstructed current picture (not shown). If the current picture1805was a predicted picture, the reconstructed prediction residual is added to the motion-compensated current picture1835to form the reconstructed current picture. One or both of the picture stores1820,1822buffers the reconstructed current picture for use in motion compensated prediction. In some embodiments, the encoder applies a de-blocking filter to the reconstructed frame to adaptively smooth discontinuities and other artifacts in the picture.

The entropy coder1880compresses the output of the quantizer1870as well as certain side information (e.g., motion information1815, quantization step size). Typical entropy coding techniques include arithmetic coding, differential coding, Huffman coding, run length coding, LZ coding, dictionary coding, and combinations of the above. The entropy coder1880typically uses different coding techniques for different kinds of information (e.g., DC coefficients, AC coefficients, different kinds of side information), and can choose from among multiple code tables within a particular coding technique.

The entropy coder1880provides compressed video information1895to the multiplexer [“MUX”]1890. The MUX1890may include a buffer, and a buffer level indicator may be fed back to bit rate adaptive modules for rate control. Before or after the MUX1890, the compressed video information1895can be channel coded for transmission over the network. The channel coding can apply error detection and correction data to the compressed video information1895.

C. Video Decoder

FIG. 19is a block diagram of a general video decoder system1900. The decoder system1900receives information1995for a compressed sequence of video pictures and produces output including a reconstructed picture1905(e.g., progressive video frame, interlaced video frame, or field of an interlaced video frame). Particular embodiments of video decoders typically use a variation or supplemented version of the generalized decoder1900.

The decoder system1900decompresses predicted pictures and key pictures. For the sake of presentation,FIG. 19shows a path for key pictures through the decoder system1900and a path for forward-predicted pictures. Many of the components of the decoder system1900are used for decompressing both key pictures and predicted pictures. The exact operations performed by those components can vary depending on the type of information being decompressed.

A DEMUX1990receives the information1995for the compressed video sequence and makes the received information available to the entropy decoder1980. The DEMUX1990may include a jitter buffer and other buffers as well. Before or after the DEMUX1990, the compressed video information can be channel decoded and processed for error detection and correction.

The entropy decoder1980entropy decodes entropy-coded quantized data as well as entropy-coded side information (e.g., motion information1915, quantization step size), typically applying the inverse of the entropy encoding performed in the encoder. Entropy decoding techniques include arithmetic decoding, differential decoding, Huffman decoding, run length decoding, LZ decoding, dictionary decoding, and combinations of the above. The entropy decoder1980typically uses different decoding techniques for different kinds of information (e.g., DC coefficients, AC coefficients, different kinds of side information), and can choose from among multiple code tables within a particular decoding technique.

The decoder1900decodes the motion information1915by, for example, computing one or more predictors for motion vectors, entropy decoding differential motion vectors, and combining decoded differential motion vectors with predictors to reconstruct motion vectors.

A motion compensator1930applies motion information1915to one or more reference pictures1925to form a prediction1935of the picture1905being reconstructed. For example, the motion compensator1930uses one or more macroblock motion vector to find macroblock(s) in the reference picture(s)1925. One or more picture stores (e.g., picture store1920,1922) store previous reconstructed pictures for use as reference pictures. Typically, B-pictures have more than one reference picture (e.g., at least one temporally previous reference picture and at least one temporally future reference picture). Accordingly, the decoder system1900can use separate picture stores1920and1922for multiple reference pictures. The motion compensator1930can compensate for motion at pixel, ½ pixel, ¼ pixel, or other increments, and can switch the precision of the motion compensation on a picture-by-picture basis or other basis. The motion compensator1930also can switch between types of reference picture pixel interpolation (e.g., between bicubic and bilinear) on a per-frame or other basis. The precision of the motion compensation can be the same or different horizontally and vertically. Alternatively, a motion compensator applies another type of motion compensation. The prediction by the motion compensator is rarely perfect, so the decoder1900also reconstructs prediction residuals.

An inverse quantizer1970inverse quantizes entropy-decoded data. In general, the inverse quantizer applies uniform, scalar inverse quantization to the entropy-decoded data with a step-size that varies on a picture-by-picture basis or other basis. Alternatively, the inverse quantizer applies another type of inverse quantization to the data, for example, to reconstruct after a non-uniform, vector, or non-adaptive quantization, or directly inverse quantizes spatial domain data in a decoder system that does not use inverse frequency transformations.

An inverse frequency transformer1960converts the quantized, frequency domain data into spatial domain video information. For block-based video pictures, the inverse frequency transformer1960applies an inverse DCT [“IDCT”], variant of IDCT, or other inverse block transform to blocks of the frequency transform coefficients, producing pixel data or prediction residual data for key pictures or predicted pictures, respectively. Alternatively, the inverse frequency transformer1960applies another conventional inverse frequency transform such as an inverse Fourier transform or uses wavelet or sub-band synthesis. The inverse frequency transformer1960may apply an 8×8, 8×4, 4×8, 4×4, or other size inverse frequency transform.

For a predicted picture, the decoder1900combines the reconstructed prediction residual1945with the motion compensated prediction1935to form the reconstructed picture1905. When the decoder needs a reconstructed picture1905for subsequent motion compensation, one or both of the picture stores (e.g., picture store1920) buffers the reconstructed picture1905for use in predicting the next picture. In some embodiments, the decoder1900applies a de-blocking filter to the reconstructed picture to adaptively smooth discontinuities and other artifacts in the picture. Various techniques for in-loop deblocking filtering are described below.

Quantization and other lossy processing of prediction residuals can cause blocking artifacts at block boundaries. Blocking artifacts can be especially troublesome in reference frames that are used for motion estimation and compensation of subsequent predicted frames. To reduce blocking artifacts, a video encoder/decoder can use a deblocking filter to perform in-loop filtering across boundary rows and/or columns in the frame. For example, a video encoder/decoder processes a reconstructed reference frame to reduce blocking artifacts prior to motion estimation/compensation using the reference frame. With in-loop deblocking, a reference frame becomes a better reference candidate to encode the following frame. The deblocking filter improves the quality of motion estimation/compensation, resulting in better prediction and lower bitrate for prediction residuals.

FIG. 22shows a motion estimation/compensation loop2200in a video encoder that includes a deblocking filter. Motion estimation/compensation loop2200includes motion estimation2210and motion compensation2220of an input picture2205. Motion estimation2210finds motion information for the input picture2205with respect to a reference picture2295(or pictures), which is typically a previously reconstructed intra- or inter-coded picture. Alternatively, the loop filter is applied to backward-predicted or bi-directionally-predicted pictures. Motion estimation2210produces motion information such as a set of one or more motion vectors for the input picture2205. Motion compensation2220applies the motion information to the reference picture(s)2295to produce a predicted picture2225. The prediction is rarely perfect, so the encoder computes2230the error or residual2235as the difference between the original input picture2205and the predicted picture2225.

Frequency transformer2240frequency transforms the prediction residual2235, and quantizer2250quantizes the frequency coefficients for the prediction residual2235before passing them to downstream components of the encoder. Inverse quantizer2260inverse quantizes the frequency coefficients of the prediction residual2235, and inverse frequency transformer2270changes the prediction residual2235back to the spatial domain, producing a reconstructed error2275for the input picture2205. The encoder combines2280the reconstructed error2275with the predicted picture2225to produce a reconstructed picture. The encoder applies the deblocking loop filter2290to the reconstructed picture and stores it in a picture buffer2292for use as a possible reference picture2295for the next input picture.

FIG. 23shows a motion compensation loop2300in a video decoder that includes a deblocking filter. Motion compensation loop2300includes motion compensation2320, which applies motion information2315received from the encoder to a reference picture2395(or pictures) to produce a predicted picture2325. In a separate path, inverse quantizer2360inverse quantizes the frequency coefficients of a prediction residual, and inverse frequency transformer2370changes the prediction residual back to the spatial domain, producing a reconstructed error2375.

The decoder combines2380the reconstructed error2375with the predicted picture2325to produce reconstructed picture2385, which is output from the decoder. The decoder applies a deblocking loop filter2390to the reconstructed picture2385and stores the reconstructed picture in a picture buffer2392for use as a possible reference picture2395for the next input picture.

Alternatively, the arrangement or constituents of motion estimation/compensation loop2200or motion compensation loop in2300can be changed, but the encoder/decoder still applies the deblocking loop filter.

IV. Innovations in In-Loop Deblocking Filtering for Interlaced Video

Described embodiments include techniques and tools for performing in-loop deblocking filtering in interlace frame coded pictures (e.g., interlaced P-frames, interlaced B-frames, interlaced I-frames, etc.) to reduce blocking artifacts. Overall use/non-use of in-loop deblocking can be signaled, for example, at entry point level or sequence level in a bitstream, so as to indicate whether or not in-loop deblocking is enabled from the entry point or in the sequence.

In some implementations, 16×16 macroblocks are subdivided into 8×8 blocks, and each inter-coded block can be transform coded using an 8×8 transform, two 4×8 transforms, two 8×4 transforms, or four 4×4 transforms. Prior to block transform coding, an encoder/decoder can permute the macroblock in such a way that all the even lines (top field lines) of the macroblock are grouped at the top of the macroblock and all the odd lines (bottom field lines) are grouped at the bottom of the macroblock. The effect of the permutation on the macroblock is to make each 8×8 block inside the macroblock contain only information from one particular field. If the macroblock is permuted in this way, the macroblock is deemed to be field coded. If the macroblock is not permuted in this way, the macroblock is deemed to be frame coded.

Field coding shifts the location of the horizontal block boundaries on the final re-interlaced macroblock/frame. For example, when a macroblock is field coded with all 8×8 blocks, the internal 8×8 block boundary of the macroblock will be shifted to the top and bottom macroblock boundaries.

Furthermore, filtering lines of different fields together can lead to blurring and distortion due to the fact that different fields are scanned at different times.

Accordingly, described embodiments implement one or more techniques and tools for performing in-loop deblocking filtering in interlaced video including, but not limited to, the following:1. Field based-in-loop deblocking filtering without filtering across field boundaries.2. Identification of correct horizontal block boundaries according to field/frame coding type and transform size to ensure filtering of correct block transform boundaries.
The described techniques and tools can be used in combination with one another or with other techniques and tools, or can be used independently.
Ordering of Filtering Operations

For both inter and intra interlaced frames, an encoder/decoder performs in-loop deblocking filtering by processing horizontal boundaries first, followed by vertical boundaries. In some implementations, the horizontal boundaries are processed one macroblock at a time in raster scan order. Similarly, the vertical edges are processed one macroblock at a time in raster scan order. Pseudo-code2400inFIG. 24describes this ordered filtering process. Other valid implementations of the filtering process are not shown for the sake of simplicity, but other valid implementations are possible.

Filter Operations

Since the minimum number of consecutive pixels that are filtered in a row or column is four and the total number of pixels in a row or column is always a multiple of four, an encoder/decoder performs adaptive filtering operations performed on segments of four pixels in some implementations.

For example, if the eight pixel pairs that make up the vertical boundary between two blocks are adaptively filtered, the eight pixel pairs are divided into two 4-pixel segments as shown inFIG. 12. In each 4-pixel segment, the third pixel pair is adaptively filtered first as indicated by the Xs inFIG. 12. The result of this filter operation determines whether the other three pixels in the segment are also adaptively filtered.

FIG. 13shows the pixels that are used in the filtering operation performed on the 3rdpixel pair. InFIG. 13, pixels P4and P5are the pixels that may be changed in the filter operation.

The pseudo-code1400ofFIG. 14shows the adaptive filtering operation performed on the 3rdpixel pair in each segment. The encoder/decoder determines whether to filter the other three pixels based on the pixel values in the line of pixels containing the 3rdpixel pair. The value filter_other—3_pixels indicates whether the remaining three pixel pairs in the segment are also filtered. If filter_other—3_pixels=TRUE, then the other three pixel pairs are adaptively filtered. If filter_other—3_pixels=FALSE, then they are not filtered, and the adaptive filtering operation proceeds to the next 4-pixel segment. The pseudo-code1500ofFIG. 15shows the adaptive filtering operation that is performed on the 1st, 2ndand 4thpixel pair if filter_other—3_pixels=TRUE. In pseudo-code1400and pseudo-code1500, the variable PQUANT represents a quantization step size.

In some implementations, an encoder/decoder performs field-based in-loop deblocking filtering. For example, an encoder/decoder filters top field lines and bottom field lines separately during in-loop deblocking filtering.

FIGS. 12,13,14and15depict the loop filtering decision process for progressive frames, which involves deciding whether to perform loop filtering for four adjacent rows (for filtering across a vertical boundary, as shown inFIG. 12) or columns (for filtering across a horizontal boundary) of samples at a time, on the four samples on each side of the vertical or horizontal boundary.

In one implementation, the filter operations described above with reference toFIGS. 12,13,14and15are modified such that the filtering is always done using the same field lines (i.e., without mixing samples of different field polarities).

FIG. 25shows a technique2500for performing field-based deblocking filtering. At2510, an encoder/decoder gets pixel data from field lines having the same polarity (e.g., top or bottom) in a current block and/or neighboring block(s). At2520, the encoder/decoder performs in-loop deblocking across a boundary within the current block or between the current block and a neighboring block.

For example, for interlaced frame coded pictures, an encoder/decoder makes a loop filtering decision for a vertical block boundary using four alternating rows of same-polarity samples instead of adjacent rows of mixed-polarity samples. The encoder/decoder makes a loop filtering decision for the two even field lines closest to the horizontal block boundary using the four even field lines on each side of the boundary. The encoder/decoder makes the decision for the two odd field lines closest to the boundary using the four odd field lines on each side of the boundary.

FIGS. 26A-26Bshow examples of field-based filtering for horizontal and vertical block boundaries, respectively. InFIG. 26A, for a horizontal block boundary between a current block2610and a neighboring block2620below the current block, the two top field lines are filtered across the block boundary using top field lines only and the two bottom field lines across the block boundary are filtered using bottom field lines only. InFIG. 26B, for a vertical block boundary between the current block2610and a neighboring block2630to the right of the current block, the top field and the bottom field are filtered separately across the block boundary. For example,FIG. 26Bshows filtering of the top field lines across the vertical block boundary.

Alternatively, an encoder/decoder performs filtering of pixels in a different way (for example, using different combinations of pixels for filtering, or by performing different filtering operations), but still filters only lines of the same fields together.

B. Determining Block Boundaries for Filtering in Interlaced Frames

FIGS. 27A-27Bshow loop filtering of luminance blocks in an interlaced field-coded macroblock in some implementations.FIG. 27Ashows field coding of luminance blocks of an interlaced macroblock. Field coding is applied to the four 8×8 luminance blocks2710of a 16×16 interlaced macroblock yielding field-coded luminance blocks2720, shown with horizontal and vertical block boundaries (in bold). Each of the four field transform coded luminance blocks2720contains only information from the top field (even numbered lines) or the bottom field (odd numbered lines).

FIG. 27Bshows reconstruction and loop filtering of the field-coded luminance blocks2720.

Field coding shifts the location of the horizontal block boundaries on the final re-interlaced macroblock/frame. As shown inFIG. 27B, if a macroblock is field coded with all 8×8 blocks, the internal 8×8 block boundary of the macroblock will be shifted to the top and bottom macroblock boundaries, since there is effectively no boundary between lines14and1, as they are from different fields. The location of block boundaries also depends on transform size.

Accordingly, in some implementations, an encoder/decoder uses field/frame type and transform size to determine block boundaries for in-loop deblocking filtering.

FIG. 28shows a technique2800for using field/frame transform type and transform size to select block boundaries for in-loop deblocking filtering. At2810, an encoder/decoder gets transform size and field/frame type information for a current macroblock. At2820, the encoder/decoder selects block boundary lines for in-loop deblocking based at least in part on the transform size and field/frame type information. At2830, the encoder/decoder performs in-loop deblocking on the selected boundary lines.

For example, after a frame has been reconstructed in a motion estimation/compensation loop, an encoder/decoder takes into account block/subblock transform size (e.g., 4×4, 4×8, 8×4, or 8×8) and field/frame transform type to determine the block boundaries to be filtered in a current macroblock. The encoder/decoder then performs in-loop deblocking on those boundaries using a field-based deblocking filter. The encoder/decoder performs an inverse permutation (re-interlacing) to form the final reconstructed frame.

FIGS. 27B and 29show examples of how the boundaries to be filtered can depend on field/frame type for macroblocks within 8×8 transform size blocks.FIG. 27Bshows loop filtering of field-coded 8×8 luminance blocks2720. In the reconstructed luminance blocks2730, there is in effect no internal horizontal boundary between blocks (no boundary between lines7and8). Instead, the block boundaries coincide with the macroblock boundaries, which are already being filtered. No internal horizontal boundary is filtered. Filtered horizontal block boundary2740is a block boundary at the bottom of the macroblock and is filtered using top field lines2750and bottom field lines2760. In field-based filtering, top field lines are filtered together and bottom field lines are filtered together without mixing fields.

FIG. 29shows loop filtering of frame-coded 8×8 luminance blocks2910. An internal horizontal block boundary lies between bottom field line7and top field line8. When the frame transform coded luminance blocks2910are reconstructed to form reconstructed luminance blocks2920, the position of the internal horizontal boundary remains the same. In the example shown inFIG. 29B, the internal block boundary (shown as filtered horizontal block boundary2930) is filtered using top field lines2940and bottom field lines2942. Filtered horizontal block boundary2932is a block boundary at the bottom of the macroblock and is filtered using top field lines2950and bottom field lines2952. Again, top field lines are filtered together and bottom field lines are filtered together without mixing fields.

The following paragraphs describe pseudo-code inFIGS. 30A-32Bfor filtering block boundaries in interlaced I-frames, P-frames and B-frames depending on field/frame type and transform size. The pseudo-code inFIGS. 30A-32Bshows examples of how an encoder/decoder determines block boundaries to be filtered in one implementation. Other implementations are possible.

In the pseudo-code inFIGS. 30A-32B, row and column numbers represent rows and columns in current macroblocks and neighboring macroblocks. Row/column numbers0-15are in a current macroblock, and row/column numbers greater than 15 are in a neighboring macroblock. Block index numbers (Y0, Y1, etc.) follow the convention shown inFIG. 20, after field/frame coding. Field/frame transform type is indicated by the variable FIELDTX. In one implementation, FIELDTX is a macroblock-level bitstream element that is explicitly signaled in intra-coded macroblocks and inferred from another macroblock-level bitstream element (MBMODE) in inter-coded macroblocks. FIELDTX and MBMODE are explained in further detail in Section V, below.

In interlaced I-frames, each macroblock is 8×8 transform coded. For each macroblock, the horizontal block boundary filtering starts by filtering the intra-macroblock horizontal boundary only if the current macroblock is frame-coded. Next, the horizontal block boundary between the current macroblock and the macroblock directly below it (if available) is filtered. The pseudo-code3000inFIG. 30Adescribes the process of horizontal filtering for a macroblock in an interlaced I-frame. Vertical block boundary filtering starts by filtering the internal vertical boundary and then filtering the boundary between the current macroblock and the right neighboring macroblock (if available). The pseudo-code3010inFIG. 30Bdescribes the process of the vertical filtering for a macroblock in an interlaced I-frame.

In interlaced P-frames and B-frames, each macroblock may be 4×4, 4×8, 8×4, or 8×8 transform coded. In one implementation, for each macroblock, the horizontal block boundary filtering occurs in the following order of blocks: Y0, Y1, Y2, Y3, Cb, Cr. The processing of the luma blocks depends on field/frame coding type. The pseudo-code3100inFIGS. 31A-Band pseudo-code3110inFIG. 31Cdescribe the process of horizontal filtering for luma and chroma blocks, respectively, for macroblocks in interlaced P-frames or B-frames.

Similarly, for each macroblock in one implementation, the vertical block boundary filtering occurs in the in the same order of blocks: Y0, Y1, Y2, Y3, Cb, Cr. As with horizontal filtering, the processing of the luma blocks depends on field/frame coding type. The pseudo-code3200inFIG. 32A-Band pseudo-code3210inFIG. 32Cdescribes the process of vertical filtering for luma and chroma blocks, respectively, for macroblocks in interlaced P-frames or B-frames.

Alternatively, an encoder/decoder uses different rules to determine which block and/or subblock boundaries are filtered or the order in which they are filtered, but still uses field/frame coding type and transform size to determine which boundaries are filtered. As another alternative, an encoder/decoder performs filtering operations in a different way (for example, using different combinations of pixels for filtering, or by performing different filtering operations).

A detailed combined implementation for a bitstream syntax, semantics, and decoder are now described, in addition to an alternative combined implementation with minor differences from the main combined implementation.

In various combined implementations, data for interlaced frame coded pictures (e.g., interlaced P-frames, interlaced B-frames, interlaced I-frames, etc.) is presented in the form of a bitstream having plural layers (e.g., sequence, entry point, frame, field, macroblock, block and/or sub-block layers).

In the syntax diagrams, arrow paths show the possible flows of syntax elements. Syntax elements shown with square-edged boundaries indicate fixed-length syntax elements; those with rounded boundaries indicate variable-length syntax elements and those with a rounded boundary within an outer rounded boundary indicate a syntax element (e.g., a bitplane) made up of simpler syntax elements. A fixed-length syntax element is defined to be a syntax element for which the length of the syntax element is not dependent on data in the syntax element itself; the length of a fixed-length syntax element is either constant or determined by prior data in the syntax flow. A lower layer in a layer diagram (e.g., a macroblock layer in a frame-layer diagram) is indicated by a rectangle within a rectangle.

Entry-point-level bitstream elements are shown inFIG. 33. In general, an entry point marks a position in a bitstream (e.g., an I-frame or other key frame) at which a decoder can begin decoding. In other words, no pictures before the entry point in the bitstream are needed to decode pictures after the entry point. An entry point header can be used to signal changes in coding control parameters (e.g., enabling or disabling compression tools, such as in-loop deblocking filtering, for frames following an entry point).

For interlaced I-frames, P-frames, and B-frames, frame-level bitstream elements are shown inFIGS. 34,35, and36, respectively. (Frame-level bitstream elements for interlaced BI-frames are identical to those for interlaced I-frames.) Data for each frame consists of a frame header followed by data for the macroblock layer (whether for intra or various inter type macroblocks).

The bitstream elements that make up the macroblock layer for interlaced P-frames (whether for intra or various inter type macroblocks) are shown inFIG. 37. Bitstream elements in the macroblock layer for interlaced P-frames (e.g., FIELDTX) may be present for macroblocks in other interlaced pictures (e.g., interlaced B-frames, interlaced I-frames etc.)

The following sections describe selected bitstream elements in the frame and macroblock layers that are related to signaling for interlaced pictures. Although the selected bitstream elements are described in the context of a particular layer, some bitstream elements can be used in more than one layer.

1. Selected Entry Point Layer Elements

LOOPFILTER is a Boolean flag that indicates whether loop filtering is enabled for the entry point segment. If LOOPFILTER=0, then loop filtering is not enabled. If LOOPFILTER=1, then loop filtering is enabled. In an alternative combined implementation, LOOPFILTER is a sequence level element.

VSTRANSFORM is a Boolean flag that indicates whether variable-sized transform coding is enabled for the sequence. If VSTRANSFORM=0, then variable-sized transform coding is not enabled. If VSTRANSFORM=1, then variable-sized transform coding is enabled.

2. Selected Frame Layer Elements

FCM is a variable length codeword [“VLC”] used to indicate the picture coding type. FCM takes on values for frame coding modes as shown in Table 1 below:

PTYPE is a variable size syntax element present in the frame header for interlaced P-frames and interlaced B-frame (or other kinds of interlaced frames such as interlaced I-frames). PTYPE takes on values for different frame types according to Table 2 below.

TABLE 2Picture Type VLCPTYPE VLCPicture Type110I0P10B1110BI1111Skipped
If PTYPE indicates that the frame is skipped then the frame is treated as a P-frame which is identical to its reference frame. The reconstruction of the skipped frame is equivalent conceptually to copying the reference frame. A skipped frame means that no further data is transmitted for this frame.
Macroblock-Level Transform Type Flag (TTMBF) (1 bit)

This syntax element is present in P-frames and B-frames if the sequence-level syntax element VSTRANSFORM=1. TTMBF is a one-bit syntax element that signals whether transform type coding is enabled at the frame or macroblock level. If TTMBF=1, the same transform type is used for all blocks in the frame. In this case, the transform type is signaled in the Frame-level Transform Type (TTFRM) syntax element that follows. If TTMBF=0, the transform type may vary throughout the frame and is signaled at the macroblock or block levels.

This syntax element is present in P-frames and B-frames if VSTRANSFORM=1 and TTMBF=1. TTFRM signals the transform type used to transform the 8×8 pixel error signal in predicted blocks. The 8×8 error blocks may be transformed using an 8×8 transform, two 8×4 transforms, two 4×8 transforms or four 4×4 transforms.

At frame level or field level, FIELDTX is a bitplane indicating whether macroblocks in an interlaced I-frame are frame-coded or field-coded. FIELDTX is explained in further detail below.

3. Selected Macroblock Layer Elements

FIG. 37is a diagram showing a macroblock-level bitstream syntax for macroblocks interlaced P-frames in the combined implementation. Specific bitstream elements are described below. Data for a macroblock consists of a macroblock header followed by block layer data. Bitstream elements in the macroblock layer for interlaced P-frames (e.g., FIELDTX) may potentially be present for macroblocks in other interlaced pictures (e.g., interlaced B-frames, etc.)

MBMODE is a variable-size syntax element that jointly specifies macroblock type (e.g., 1 MV, 2 Field MV, 4 Field MV, 4 Frame MV or Intra), transform type (e.g., field, frame, or no coded blocks), and the presence of differential motion vector data for 1 MV macroblocks. MBMODE is explained in detail below.

FIELDTX is a 1-bit syntax present in interlaced B-frame intra-coded macroblocks. This syntax element indicates whether a macroblock is frame or field coded (basically, the internal organization of the macroblock). FIELDTX=1 indicates that the macroblock is field-coded. Otherwise, the macroblock is frame-coded. In inter-coded macroblocks, this syntax element can be inferred from MBMODE as explained in detail below.

TTMB is a variable-size syntax element in P-frame and B-frame macroblocks when the picture layer syntax element TTMBF=0. TTMB specifies a transform type, transform type signal level, and subblock pattern.

If TTMB indicates the signal level is block level then the transform type is signaled at block level. At block level, for a block that contains residual information, TTBLK indicates the transform type used for the block. TTBLK is not present for the first coded block since transform type for that block is joint coded in TTMB. TTBLK is present for all the remaining coded blocks and indicates the transform type. If the transform type is 8×4 or 4×8, the subblock pattern is decoded as part of TTMB (for the first coded block) or TTBLK (for each remaining coded block after the first one). If the transform type is 4×4, the subblock pattern is encoded in SUBBLKPAT at the block level for each coded block.

If the transform type signaling level is macroblock level and the transform type is 8×4, 4×8, or 4×4, the decoder may still need information about which subblocks have non-zero coefficients. If the transform type is 8×4 or 4×8, the subblock pattern is decoded as part of TTMB (for the first coded block) or SUBBLKPAT (for each remaining coded block). If the transform type is 4×4, the subblock pattern is encoded in SUBBLKPAT at the block level for each coded block.

Finally, if the transform type signaling level is frame level and the transform type is 8×4, 4×8, or 4×4, the decoder needs information about which subblocks have non-zero coefficients. The subblock pattern is encoded in SUBBLKPAT at the block level for each coded block.

If a subblock pattern indicates no non-zero coefficients are present for a subblock, then no additional coefficient information for that subblock is present in the bitstream. For an 8×4 transform type, data for the top subblock (if present) is coded first followed by data for the bottom subblock (if present). For a 4×8 transform type, data for the left subblock (if present) is coded first followed by data for the right subblock (if present). For a 4×4 transform type, data for the upper left subblock is coded first (if present) followed, in order, by data for the upper right, lower left and lower right subblocks (if present).

B. Decoding Aspects of Interlaced P-Frames

In an interlaced P-frame, each macroblock may be motion compensated in frame mode using one or four motion vectors or in field mode using two or four motion vectors. A macroblock that is inter-coded does not contain any intra blocks. In addition, the residual after motion compensation may be coded in frame transform mode or field transform mode. More specifically, the luma component of the residual is re-arranged according to fields if it is coded in field transform mode but remains unchanged in frame transform mode, while the chroma component remains the same. A macroblock may also be coded as intra.

Motion compensation may be restricted to not include four (both field/frame) motion vectors. The type of motion compensation and residual coding is jointly indicated for each macroblock through MBMODE and a skipped macroblock signal (SKIPMB).

Macroblocks in interlaced P-frames are classified into five types: 1 MV, 2 Field MV, 4 Frame MV, 4 Field MV, and Intra. The first four types of macroblock are inter-coded while the last type indicates that the macroblock is intra-coded. The macroblock type is signaled by the MBMODE syntax element in the macroblock layer along with the skip bit. (A skip condition for the macroblock also can be signaled at frame level in a compressed bit plane.) MBMODE jointly encodes macroblock types along with various pieces of information regarding the macroblock for different types of macroblock.

Macroblock Mode Signaling

MBMODE jointly specifies the type of macroblock (1 MV, 4 Frame MV, 2 Field MV, 4 Field MV, or intra), types of transform for inter-coded macroblock (i.e. field or frame or no coded blocks), and whether there is a differential motion vector for a 1 MV macroblock. MBMODE can take one of 15 possible values:

Let <MVP> denote the signaling of whether a nonzero 1 MV differential motion vector is present or absent. Let <Field/Frame transform> denote the signaling of whether the residual of the macroblock is (1) frame transform coded; (2) field transform coded; or (3) zero coded blocks (i.e. CBP=0). MBMODE signals the following information jointly:MBMODE={<1 MV, MVP, Field/Frame transform>, <2 Field MV, Field/Frame transform>, <4 Frame MV, Field/Frame transform>, <4 Field MV, Field/Frame transform>, <INTRA>};
The case <1 MV, MVP=0, CBP=0>, is not signaled by MBMODE, but is signaled by the skip condition.

For inter-coded macroblocks, the CBPCY syntax element is not decoded when <Field/frame Transform> in MBMODE indicates no coded blocks. On the other hand, if <Field/frame Transform> in MBMODE indicates field or frame transform, then CBPCY is decoded.

The decoded <Field/frame Transform> is used to set the flag FIELDTX. If it indicates that the macroblock is field transform coded, FIELDTX is set to 1. If it indicates that the macroblock is frame transform coded, FIELDTX is set to 0. If it indicates a zero-coded block, FIELDTX is set to the same type as the motion vector, i.e., FIELDTX is set to 1 if it is a field motion vector and to 0 if it is a frame motion vector.

For non-1 MV inter-coded macroblocks, an additional field is sent to indicate which of the differential motion vectors is non-zero. In the case of 2 Field MV macroblocks, the 2 MVBP field is sent to indicate which of the two motion vectors contain nonzero differential motion vectors. Similarly, the 4 MVBP field is sent to indicate which of the four motion vectors contain nonzero differential motion vectors.

For intra-coded macroblocks, the Field/Frame transform and zero coded blocks are coded in separate fields.

C. In-Loop Deblocking Filtering for Progressive Frames

Before describing the process for in-loop deblocking filtering for interlaced frames in the combined implementation, a process for in-loop deblocking filtering for progressive frames is described. The section describing the process for in-loop deblocking filtering for interlaced frames will proceed with reference to concepts discussed in this section.

In the combined implementation, if the entry-point-layer syntax element LOOPFILTER=1, an adaptive filtering operation is performed on each reconstructed frame in the entry point segment. This filtering operation is performed prior to using the reconstructed frame as a reference for motion compensation. When there are multiple slices in a picture, the filtering for each slice is performed independently.

Since the intent of loop filtering is to smooth discontinuities at block boundaries, the filtering process operates on pixels that border neighboring blocks. The locations of block boundaries depend on the size of the inverse transform used. For P-frames the block boundaries may occur at every 4th or 8th pixel row or column depending on whether an 8×8, 8×4 or 4×8 inverse transform is used. For I-frames, where an 8×8 transform is used, block boundaries occur at every 8th pixel row and column.

For progressive I-frames, adaptive deblocking filtering is performed at all 8×8 block boundaries.FIGS. 7 and 8show the pixels that are filtered along the horizontal and vertical border regions in the upper left corner of a component (luma, Cbor Cr) plane.FIG. 7shows filtered vertical block boundary pixels in an I-frame.FIG. 8shows filtered horizontal block boundary pixels in an I-frame.

InFIGS. 7 and 8, crosses represent pixels (or, more precisely, samples) and circled crosses represent filtered pixels. As these figures show, the top horizontal line and first vertical line in the frame are not filtered, even though they lie on a block boundary, because these lines lie on the border of the frame. Although not depicted, the bottom horizontal line and last vertical line in the frame also are not filtered for the same reason. In more formal terms, the following lines are filtered:Horizontal lines: (7, 8), (15, 16) . . . ((N−1)*8−1, (N-1)*8)Vertical lines: (7, 8), (15, 16) . . . ((M−1)*8−1, (M−1)*8)(N=number of horizontal 8×8 blocks in the plane (N*8=horizontal frame size))(M=number of vertical 8×8 blocks in the frame (M*8=vertical frame size))
The order in which the pixels are filtered is important. For progressive frames in this combined implementation, all horizontal boundary lines in the frame are filtered first, followed by the vertical boundary lines.

In the combined implementation, progressive B-frame in-loop deblocking is the same as progressive I-frame deblocking. As in progressive I-frame in-loop deblocking, 8×8 block boundaries are filtered, and motion vectors and 4×8/8×4 transforms are not considered.

For progressive P-frames, blocks can be intra or inter-coded. In the combined implementation, an encoder/decoder uses an 8×8 transform to transform the samples in intra-coded blocks. When at least one of the neighboring blocks is intra-coded, the 8×8 block boundaries are always adaptively filtered. An encoder/decoder uses an 8×8, 8×4, 4×8 or 4×4 transform for inter-coded blocks and uses a corresponding inverse transform to construct the samples that represent the residual error. Depending on the status of the neighboring blocks, the boundary between the current and neighboring blocks may or may not be filtered. The decision of whether to adaptively filter a block or subblock border is as follows:1) The boundaries between coded (at least one non-zero coefficient) subblocks (8×4, 4×8 or 4×4) within an 8×8 block are always adaptively filtered.2) The boundary between a block or subblock and a neighboring block or subblock shall not be not filtered only if both blocks are inter-coded, have the same motion vector, and have no residual error (no transform coefficients). Otherwise the boundary shall be adaptively filtered.

FIG. 9shows examples of when filtering between neighboring blocks does and does not occur in progressive P-frames. In this example, it is assumed that the motion vectors for both blocks is the same (if the motion vectors are different, then the boundary is always adaptively filtered). The shaded blocks or subblocks represent the cases where at least one nonzero coefficient is present. Clear blocks or subblocks represent cases where no transform coefficients are present. Thick lines represent the boundaries that are adaptively filtered. Thin lines represent the boundaries that are not filtered. These examples illustrate only horizontal neighbors, but the same applies for vertical neighbors.

FIGS. 10 and 11show an example of pixels that may be filtered in a progressive P-frame. The crosses represent pixel locations and the circled crosses represent the boundary pixels that are filtered if the conditions specified above are met.

FIG. 10shows pixels adaptively filtered along horizontal boundaries. As the figure shows, the pixels on either side of the block or subblock boundary are candidates to be filtered. For the horizontal boundaries this could be every 4thand 5th, 8thand 9th, 12thand 13th, etc., pixel row in the frame as these are the 8×8 and 8×4 horizontal boundaries.FIG. 11shows pixels adaptively filtered along vertical boundaries. For the vertical boundaries, every 4thand 5th, 8thand 9th, 12thand 13th, etc., pixel column in the frame may be adaptively filtered as these are the 8×8 and 4×8 vertical boundaries.

In this combined implementation, the first and last row and the first and last column in the frame are not filtered. The order in which pixels are filtered is important. First, in this combined implementation, all the 8×8 block horizontal boundary lines in the frame are adaptively filtered starting from the top line. Next, all 8×4 block horizontal boundary lines in the frame are adaptively filtered starting from the top line. Next, all 8×8 block vertical boundary lines are adaptively filtered starting from the leftmost line. Lastly, all 4×8 block vertical boundary lines are adaptively filtered starting with the leftmost line. In all cases in this combined implementation, the rules specified above are used to determine whether the boundary pixels are adaptively filtered for each block or subblock.

4. Filter Operation

This section describes an adaptive filtering operation that is performed on the boundary pixels in progressive I-, B- and P-frames in the combined implementation.

For progressive P-frames the decision criteria described above determine which vertical and horizontal boundaries are adaptively filtered. For progressive I-frames, all the 8×8 vertical and horizontal boundaries are adaptively filtered. Since the minimum number of consecutive pixels that are filtered in a row or column is four and the total number of pixels in a row or column is always a multiple of four, the adaptive filtering operation is performed on segments of four pixels.

For example, if the eight pixel pairs that make up the vertical boundary between two blocks is adaptively filtered, then the eight pixels are divided into two 4-pixel segments as shown inFIG. 12. In each 4-pixel segment, the third pixel pair is adaptively filtered first as indicated by the Xs inFIG. 12. The result of this filter operation determines whether the other three pixels in the segment are also adaptively filtered, as described below.

FIG. 13shows the pixels that are used in the filtering operation performed on the 3rdpixel pair. InFIG. 13, pixels P4and P5are the pixel pairs that may be changed in the filter operation.

The pseudo-code1400ofFIG. 14shows the filtering operation performed on the 3rdpixel pair in each segment. The value filter_other—3_pixels indicates whether the remaining three pixel pairs in the segment are also adaptively filtered. If filter_other—3_pixels=TRUE, then the other three pixel pairs are adaptively filtered. If filter_other—3_pixels=FALSE, then they are not adaptively filtered, and the filtering operation proceeds to the next 4-pixel segment. The pseudo-code1500ofFIG. 15shows the filtering operation that is performed on the 1st, 2ndand 4thpixel pair if filter_other—3_pixels=TRUE.

The filtering operations described above are similarly used for adaptively filtering horizontal boundary pixels.

This section describes the process for in-loop deblocking filtering of interlaced frames in the combined implementation, with reference to concepts discussed in the previous section.

If the entry point layer syntax element LOOPFILTER=1, a filtering operation is performed on each reconstructed frame. (In an alternative combined implementation, LOOPFILTER is a sequence layer syntax element.) This filtering operation is performed prior to using the reconstructed frame as a reference for motion predictive coding.

Since the intent of loop filtering is to smooth out the discontinuities at block boundaries, the adaptive filtering process operates on the pixels that border neighboring blocks. For interlaced P-frames, the block boundaries may occur at every 4th, 8th, 12th, etc., pixel row or column, depending on whether an 8×8, 8×4, 4×8 or 4×4 inverse transform is used. For interlaced I-frames, adaptive filtering occurs at every 8th, 16th, 24th, etc., pixel row and column.

In interlace frame coded pictures, each macroblock may be frame transform coded or field transform coded according to its FIELDTX flag. The state of the FIELDTX flag along with the size of the transform used (4×4, 4×8, 8×4, 8×8) has an effect on where the in-loop deblocking takes place in the macroblock.

The adaptive filtering process is the same as described above with regard to progressive frames, with one important difference: the filtering is always done using the same field lines, never mixing different fields.FIGS. 26A-26Billustrate field-based filtering for horizontal and vertical block boundaries.

For a horizontal block boundary, the two top field lines are filtered across the block boundary using top field lines only and the two bottom field lines across the block boundary are filtered using bottom field lines only. For a vertical block boundary, the top field block boundary and the bottom field block boundary are filtered separately.

Filtering Order

For both inter (P, B) and intra (I) frame coded pictures, the in-loop deblocking process starts by processing all the horizontals edges first followed by all the vertical edges. The pseudo-code2400inFIG. 24describes this filtering process in the combined implementation one macroblock at a time for the sake of simplicity, but alternate valid implementations of the filtering process may not follow this macroblock processing order.

In interlaced I-frames, each macroblock is 8×8 transform coded. For each macroblock, the horizontal block boundary filtering starts by filtering the intra-macroblock horizontal boundary only if the current macroblock is frame transform coded. Next, the horizontal block boundary between the current macroblock and the macroblock directly below it (if available) is filtered. The pseudo-code3000in FIG.30A describes the process of horizontal filtering for a macroblock in an interlaced I-frame.

For each macroblock, the vertical block boundary filtering starts by filtering the intra-macroblock vertical boundary and then followed by the filtering of the inter-macroblock boundary between the current macroblock and the macroblock to its immediate right (if available). The pseudo-code3010inFIG. 30Bdescribes the process of the vertical filtering for a macroblock in an interlaced I-frame.

In interlaced P-frames and B-frames, each inter-coded macroblock may be 4×4, 4×8, 8×4, or 8×8 transform coded. For each macroblock, the horizontal block boundary filtering occurs in the order of block Y0, Y1, Y2, Y3, Cb, and then Cr. In this combined implementation, the luma blocks are processed differently according to field/frame coding status. The value is explicitly signaled in intra-coded macroblocks, and it is inferred from MBMODE in inter-coded macroblocks. The pseudo-code3100inFIG. 31Aand pseudo-code3110inFIG. 31Bdescribes the process of horizontal filtering for luma and chroma blocks, respectively, for a macroblock in an interlaced P-frame or B-frame.

Similarly, for each macroblock, the vertical block boundary filtering occurs in the order of block Y0, Y1, Y2, Y3, Cb, and then Cr. In this combined implementation, as with horizontal filtering, the luma blocks are processed differently according to field/frame coding status. The pseudo-code3200inFIG. 32Aand pseudo-code3210inFIG. 32Bdescribes the process of vertical filtering for luma and chroma blocks, respectively, for a macroblock in an interlaced P-frame or B-frame.