Source: http://www.google.com/patents/US8107531?dq=5579430
Timestamp: 2016-05-29 02:18:42
Document Index: 621725545

Matched Legal Cases: ['Application No. 60', 'Application No. 60', 'art 2', 'art 2', 'art 2', 'art 2']

Patent US8107531 - Signaling and repeat padding for skip frames - Google PatentsSearch Images Maps Play YouTube News Gmail Drive More »Sign inPatentsA video codec efficiently signals that a frame is identical to its reference frame, such that separate coding of its picture content is skipped. Information that a frame is skipped is represented jointly in a coding table of a frame coding type element for bit rate efficiency in signaling. Further, the...http://www.google.com/patents/US8107531?utm_source=gb-gplus-sharePatent US8107531 - Signaling and repeat padding for skip framesAdvanced Patent SearchPublication numberUS8107531 B2Publication typeGrantApplication numberUS 10/987,521Publication dateJan 31, 2012Filing dateNov 12, 2004Priority dateSep 7, 2003Fee statusPaidAlso published asUS20050152457Publication number10987521, 987521, US 8107531 B2, US 8107531B2, US-B2-8107531, US8107531 B2, US8107531B2InventorsShankar Regunathan, Chih-Lung Lin, Thomas W. Holcomb, Jie Liang, Ming-Chieh Lee, Pohsiang HsuOriginal AssigneeMicrosoft CorporationExport CitationBiBTeX, EndNote, RefManPatent Citations (112), Non-Patent Citations (49), Referenced by (1), Classifications (43), Legal Events (3) External Links: USPTO, USPTO Assignment, EspacenetSignaling and repeat padding for skip frames
US 8107531 B2Abstract
A video codec efficiently signals that a frame is identical to its reference frame, such that separate coding of its picture content is skipped. Information that a frame is skipped is represented jointly in a coding table of a frame coding type element for bit rate efficiency in signaling. Further, the video codec signals the picture type (e.g., progressive or interlaced) of skipped frames, which permits different repeat padding methods to be applied according to the picture type.
1. A method of decoding compressed video containing skip frames using a video decoder, the method comprising:
receiving, at the video decoder, encoded data in a bitstream, wherein the encoded data includes a frame-level bitstream syntax element for a current frame of a video sequence;
with the video decoder, decoding the frame-level bitstream syntax element in accordance with a variable length code table comprising:
plural codes that indicate a plurality of frame coding types, respectively; and
a code that indicates a skip frame, wherein, in the case the frame-level bitstream syntax element signals the current frame to be a skip frame, the bitstream lacks further data for decoding the current frame after the frame-level bitstream syntax element for the current frame; and
with the video decoder, in the case the frame-level bitstream syntax element signals the current frame to be a skip frame, reconstructing the current frame as a P-frame that is identical to its reference frame, including repeating active picture content from the reference frame to reconstruct the current frame, such that the current frame is identical in content to the reference frame.
2. The method of claim 1 wherein each of the plural codes in the variable length code table is a variable length code for signaling the current frame to be one of at least the frame coding types including an I-frame, a P-frame, a B-frame, and a BI-frame.
decoding a previous frame-level bitstream syntax element representing a frame mode of the current frame as being progressive or interlaced; and
applying one of at least two repeat padding methods expanding out from the active picture content of the current frame based on the frame mode of the current frame.
in the case that the frame mode of the current frame is progressive, applying a repeat padding method wherein an edge row of the active picture content is repeated to expand vertically from the active picture content of the current frame.
in the case that the frame mode of the current frame is interlaced, applying a repeat padding method wherein an edge row of each interlaced field of the active picture content is repeated to expand vertically from the active picture content of the current frame.
6. A method of decoding compressed video containing skip frames using a video decoder, the method comprising:
receiving, at the video decoder, encoded data in a bitstream, wherein the encoded data includes first and second bitstream syntax elements at frame level for a current frame of a video sequence, the first bitstream syntax element representing a coding mode of the current frame, and the second bitstream syntax element representing whether the current frame is a skip frame;
with the video decoder, when decoding the current frame, decoding the first bitstream syntax element and the second bitstream syntax element, wherein the second bitstream syntax element is coded according to a variable length coding table comprising:
plural codes that indicate a plurality of frame coding types, respectively, and
a code that indicates a skip frame,
wherein, if the current frame is represented to be a skip frame by the second bitstream syntax element, the bitstream lacks further data for decoding the current frame after the second bitstream syntax element, and otherwise the bitstream includes further data for decoding the current frame after the second bitstream syntax element; and
with the video decoder, if the current frame is represented to be a skip frame,
decoding the current skip frame as a P-frame that is identical to its reference frame, including reconstructing active picture content of the current skip frame by copying from active picture content of the reference frame, such that the current skip frame is identical in picture content to the reference frame; and
expanding out from the active picture content of the current skip frame using a form of repeat padding based on the coding mode of the current skip frame.
7. The method of claim 6, wherein the form of repeat padding based on the coding mode being a progressive mode comprises repeating an edge row of the active picture content of the current frame to expand out vertically from the active picture content.
8. The method of claim 6, wherein the form of repeat padding based on the coding mode being an interlaced mode comprises repeating two rows at an edge of the active picture content of the current frame to expand out vertically from the active picture content.
9. The method of claim 6, wherein the second bitstream syntax element is further representative of the current frame being of intra-coding type or predictive-coding type.
10. The method of claim 6, wherein the second bitstream syntax element represents whether the current frame is an I-frame, a P-frame or a skip frame.
11. The method of claim 6, wherein the second bitstream syntax element represents whether the current frame is an I-frame, a P-frame, a B-frame, a BI-frame or a skip frame.
12. A video decoder, comprising:
video bitstream parsing means for reading, from encoded data in a bitstream, a frame-level bitstream syntax element for a current frame in a video sequence;
decoding means for decoding the frame-level bitstream syntax element in accordance with a variable length code table, the variable length code table comprising:
a code that indicates a skip frame, wherein, in the case the current frame is a skip frame, the bitstream lacks further data for decoding the current frame after the frame-level bitstream syntax element for the current frame; and
frame reconstructing means for, in the case the current frame is a skip frame, reconstructing the current frame as a P-frame that is identical to its reference frame, including repeating active picture content from the reference frame, such that the current frame is identical in content to the reference frame.
13. The video decoder of claim 12 wherein each of the plural codes in the variable length code table is a variable length code for signaling the current frame to be one of at least the frame coding types including an I-frame, a P-frame, a B-frame and a BI-frame.
14. The video decoder of claim 12 further comprising:
decoding means for decoding a previous frame-level bitstream syntax element representing a frame mode of the current frame as being progressive or interlaced; and
repeat padding means for applying one of at least two repeat padding methods expanding from the active picture content of the current frame based on the frame mode of the current frame.
15. The video decoder of claim 14 wherein the repeat padding means further comprises:
means for, in the case that the frame mode of the current frame is progressive, applying a repeat padding method wherein an edge row of the active picture content is repeated to expand vertically from the active picture content of the current frame.
16. The video decoder of claim 14 wherein the repeat padding means further comprises:
means for, in the case that the frame mode of the current frame is interlaced, applying a repeat padding method wherein an edge row of each interlaced field of the active picture content is repeated to expand vertically from the active picture content of the current frame.
17. A tangible computer readable storage device having a program stored thereon that is executable on a processor to decode compressed video containing skip frames, the program comprising:
program instructions operating to receive encoded data in a bitstream, wherein the encoded data includes first and second bitstream syntax elements at frame level for a current frame of a video sequence, the first bitstream syntax element representing a coding mode of the current frame, and the second bitstream syntax element representing whether the current frame is a skip frame;
program instructions operating, when decoding the current frame, to decode the first bitstream syntax element and the second bitstream syntax element, wherein the second bitstream syntax element is coded according to a variable length coding table comprising:
program instructions operating to, if the current frame is represented to be a skip frame,
decode the current skip frame as a P-frame that is identical to its reference frame, including reconstructing active picture content of the current skip frame by copying from active picture content of the reference frame, such that the current skip frame is identical in picture content to the reference frame; and
expand out from the active picture content of the current skip frame using a form of repeat padding based on the coding mode of the current skip frame.
18. The tangible computer readable storage device of claim 17, wherein the form of repeat padding based on the coding mode being a progressive mode comprises repeating an edge row of the active picture content of the current frame to expand out vertically from the active picture content.
19. The tangible computer readable storage device of claim 17, wherein the form of repeat padding based on the coding mode being an interlaced mode comprises repeating two rows at an edge of the active picture content of the current frame to expand out vertically from the active picture content.
20. The tangible computer readable storage device of claim 17, wherein the second bitstream syntax element represents whether the current frame is an I-frame, a P-frame, a B-frame, a BI-frame or a skip frame.
21. The method of claim 1 wherein, in the case the current frame is a skip frame, the bitstream further includes one or more display syntax elements for the current frame that are not used in the decoding of the current frame, the one or more display syntax elements including at least one repeat picture syntax element and at least one pan scan display syntax element.
22. The video decoder of claim 12 wherein, in the case the current frame is a skip frame, the bitstream includes one or more display syntax elements for the current frame that are not used in the decoding of the current frame, the one or more display syntax elements including at least one repeat picture syntax element and at least one pan scan display syntax element.
This application is a continuation-in-part of U.S. patent application Ser. No. 10/934,117, entitled, “SIGNALING FOR FIELD ORDERING AND FIELD/FRAME DISPLAY REPETITION,” filed Sep. 4, 2004, which application claims the benefit of U.S. Provisional Patent Application No. 60/501,081, entitled “Video Encoding and Decoding Tools and Techniques,” filed Sep. 7, 2003, both of which are hereby incorporated by reference.
This application also is a continuation-in-part of U.S. patent application Ser. No. 10/960,384, entitled, “EFFICIENT REPEAT PADDING FOR HYBRID VIDEO SEQUENCE WITH ARBITRARY VIDEO RESOLUTION,” filed Oct. 6, 2004, which is hereby incorporated by reference.
The technical field relates to video coding and decoding, and more particularly to signaling and repeat padding of skipped frames in coding and decoding of skipped frames in a video sequence.
Some international standards describe bitstream elements for signaling field display order and for signaling whether certain fields or frames are to be repeated during display. The H.262 standard uses picture coding extension elements top_field_first and repeat_first_field to indicate field display order and field display repetition. When the sequence extension syntax element progressive_sequence is set to 1 (indicating the coded video sequence contains only progressive frames), top_field_first and repeat_first_field indicate how many times a reconstructed frame is to be output (i.e., once, twice or three times) by an H.262 decoder. When progressive13 sequence is 0 (indicating the coded video sequence many contain progressive or interlaced frames (frame-coded or field-coded)), top_field_first indicates which field of a reconstructed frame the decoder outputs first, and repeat_first_field indicates whether the first field in the frame is to be repeated in the output of the decoder.
According to draft JVT-d157of the JVT/AVC video standard, the slice header element pic_structure takes on one of five values to identify a picture as being one of five types: progressive frame, top field, bottom field, interlaced frame with top field first in time, or interlaced frame with bottom field first in time.
These international standards are limited in that they do not allow for signaling to indicate the presence or absence of bitstream elements for (1) signaling field display order and (2) signaling whether certain fields or frames are to be repeated during display. For example, although the H.262 standard uses picture coding extension elements top_field_first and repeat_first_field, the H.262 standard does not have a mechanism to “turn off” such elements when they are not needed.
IV. Repeat Padding
As previously remarked, interframe compression typically is performed by performing motion estimation and prediction for the macroblocks in a predicted frame with respect to a reference intra-coded frame. Some previously existing video systems have permitted the motion estimation to extend beyond the active picture contents of the reference intra-coded frame. In some such cases, the video systems have derived the “content” outside the picture by repeating the pixels of the picture edge to “fill” an extended region that may be used for motion estimation purposes. For example, the bottom row of the picture is repeated to vertically expand the picture downward to fill an extended motion estimation region below the picture. Likewise, the top row, left and right columns are repeated at top left and right sides to provide extended motion estimation regions at those sides of the reference picture. This process of filling areas outside the active picture content is sometimes referred to as “repeat padding.”
Various video codec tools and techniques described herein provide for efficient signaling and repeat padding of skipped frames. More particularly, the described video codec efficiently signal that a frame is identical to its reference frame, and therefore coding of its picture content is skipped.
In one aspect of the efficient skip frame signaling method described herein, information signaling that a frame is skipped is combined with information of the frame type to achieve bit rate coding efficiency. This combination of signaling skip frame and frame type information permits different repeat padding methods to be applied to the skip frame according to its frame type. In one described example implementation, two types of skipped frames can be signaled: progressive skip, and interlace frame skip. Progressive repeat padding is then applied to progressive type skip frames, while interlace repeat padding is applied to skip frames that are of the interlace type. For example, the repeat padding of progressive pictures is done by repeating the edges of the active video boundary to expand out from the active video region. Repeat padding of interlaced content pictures, on the other hand, is accomplished by repeating the last two edges rows of the active video (i.e., the last row of each interlaced field) to pad the picture vertically, whereas the edge column is repeated to pad horizontally. The different progressive and interlace repeat padding has the benefit of serving as better prediction area for the following frames.
FIG. 17 is a diagram showing an entry point-layer bitstream syntax.
FIG. 18 is a code diagram showing pseudo-code for determining a number of pan/scan windows in a combined implementation.
FIG. 19 is a flow diagram of a method utilized in the encoder of FIG. 3 for encoding a hybrid video sequence using adaptive vertical macroblock alignment.
FIG. 20 is a flow diagram of a method utilized in the decoder of FIG. 4 for decoding a hybrid video sequence using adaptive vertical macroblock alignment.
FIGS. 21A-B are diagrams of applying repeat padding and adaptive vertical macroblock alignment to progressive and interlaced pictures of a hybrid video sequence.
FIGS. 22A-B are diagrams showing repeat padding of progressive and interlaced-type pictures of a hybrid video sequence.
FIG. 23 is a pseudo-code listing of program code for calculating padded height and width of pictures in a hybrid video sequence.
The present application relates to techniques and tools for efficient compression and decompression of interlaced and progressive video. In various described embodiments, a video encoder and decoder incorporate techniques for encoding and decoding interlaced and progressive video, and corresponding signaling techniques for use with a bitstream format or syntax comprising different layers or levels (e.g., sequence level, entry point level, frame level, field level, slice level, macroblock level, and/or block level).
For the sake of presentation, the detailed description uses terms like “predict,” “compensate,” and “apply” to describe computer operations in a computing environment. These terms are high-level abstractions for operations performed by a computer, and should not be confused with acts performed by a human being. The actual computer operations corresponding to these terms vary depending on implementation.
FIG. 6C shows the interlaced video frame 600 of FIG. 6A organized for encoding/decoding as fields 660. Each of the two fields of the interlaced video frame 600 is partitioned into macroblocks. The top field is partitioned into macroblocks such as the macroblock 661, and the bottom field is partitioned into macroblocks such as the macroblock 662. (Again, the macroblocks use a 4:2:0 format as shown in FIG. 5, and the organization and placement of luminance blocks and chrominance blocks within the macroblocks are not shown.) In the luminance plane, the macroblock 661 includes 16 lines from the top field and the macroblock 662 includes 16 lines from the bottom field, and each line is 16 pixels long. An interlaced I-field is a single, separately represented field of an interlaced video frame. An interlaced P-field is a single, separately represented field of an interlaced video frame coded using forward prediction, and an interlaced B-field is a single, separately represented field of an interlaced video frame coded using bidirectional prediction. Interlaced P- and B-fields may include intra-coded macroblocks as well as different types of predicted macroblocks. Interlaced BI-fields are a hybrid of interlaced I-fields and interlaced B-fields; they are intra-coded, but are not used as anchors for other fields.
If the current picture 305 is a forward-predicted picture, a motion estimator 310 estimates motion of macroblocks or other sets of pixels of the current picture 305 with respect to one or more reference pictures, for example, the reconstructed previous picture 325 buffered in the picture store 320. If the current picture 305 is a bi-directionally-predicted picture, a motion estimator 310 estimates motion in the current picture 305 with respect to up to four reconstructed reference pictures (for an interlaced B-field, for example). Typically, a motion estimator estimates motion in a B-picture with respect to one or more temporally previous reference pictures and one or more temporally future reference pictures. Accordingly, the encoder system 300 can use the separate stores 320 and 322 for multiple reference pictures. For more information on progressive B-frames and interlaced B-frames and B-fields, see U.S. patent application Ser. No. 10/622,378, entitled, “Advanced Bi-Directional Predictive Coding of Video Frames,” filed Jul. 18, 2003, and U.S. patent application Ser. No. 10/882,135, entitled, “Advanced Bi-Directional Predictive Coding of Interlaced Video,” filed Jun. 29, 2004.
The entropy coder 380 provides compressed video information 395 to the multiplexer [“MUX”] 390. The MUX 390 may include a buffer, and a buffer level indicator may be fed back to bit rate adaptive modules for rate control. Before or after the MUX 390, the compressed video information 395 can be channel coded for transmission over the network. The channel coding can apply error detection and correction data to the compressed video information 395.
An inverse frequency transformer 460 converts the quantized, frequency domain data into spatial domain video information. For block-based video pictures, the inverse frequency transformer 460 applies an inverse DCT [“IDCT”], variant of IDCT, or other inverse block transform to blocks of the frequency transform coefficients, producing pixel data or prediction residual data for key pictures or predicted pictures, respectively. Alternatively, the inverse frequency transformer 460 applies another conventional inverse frequency transform such as an inverse Fourier transform or uses wavelet or sub-band synthesis. The inverse frequency transformer 460 may apply an 8�8, 8�4, 4�8, 4�4, or other size inverse frequency transform.
III. Signaling for Skip Frames with Repeat Padding
Described embodiments include techniques and tools for signaling skip frames. Skip frames are frames that are coded as being identical in content to a reference frame. Such skip frames can then be encoded in the compressed video 395 (FIG. 3) by signaling that the frame is a skip frame, without further encoding the picture content of the frame. At decoding, the picture content of the skip frame is recovered by repeating the picture content of the reference frame.
The encoder 300 (FIG. 3) can apply this skip frame coding when it determines that successive frames of a video sequence are identical or substantially identical, e.g., due to a lack of motion in the scene. Skip frame coding can also be used for pull-down conversions. In video pull-down conversions (e.g., from 24-frame-per-second film to 30-frame-per-second or 60-frame-per-second video), the frame/field rate is artificially increased after decoding through repeated display of the same decoded frames or fields in a video sequence. Pull-down conversions are important for interoperability of NTSC video and film footage, which have different frame rates.
Further, the described embodiments include a way to efficient signal different types of skipped frames. In one example, progressive skip frame and interlaced skip frames can be signaled. This allows the repeat padding method of a skipped frame to be varied. The repeat padding of progressive pictures is done by repeating the edges of the active video boundary to fill out the expanded region. More specifically, the edge row of the active content is repeated to pad the picture vertically, while the edge column of the active content is repeated to pad the picture horizontally. Repeat padding of interlaced content pictures, on the other hand, is accomplished by repeating the last two edges rows of the active video (i.e., the last row of each interlaced field) to pad the picture vertically, whereas the edge column is repeated to pad horizontally. This repeat padding using the active video boundary has the benefit of serving as better prediction area for the following frames. For a subsequent predictively coded frame, a macroblock's motion vector can point to the expanded region. This typically provides a better prediction of the macroblock in the predicted frame, often resulting in a zero or minimal block error signal that can be more efficiently encoded. The encoder thus can effectively “zero”-out the information that otherwise has to be transmitted for the expanded region.
The skip frame signaling in one described embodiment of the video codec signals the frame is a skip frame using frame level syntax elements. For coding economy, skip frame signaling is jointly signaled along with other frame header level information. In one implementation, variable length codes of a syntax element for the frame coding type (e.g., I-, P- or B-frame) includes an escape sequence to signal the frame is a skipped frame. Further, a frame coding mode syntax element distinguishes between the types of skipped frames, whether progressive or interlaced. This determines the repeat padding applied to the frame.
In one implementation, when a sequence has an interlaced target display type (INTERLACE=1) and pull-down is used (PULLDOWN=1), picture headers contain the one-bit repeat-picture element RFF. The time allotted for displaying a single field without repeating the field is called a field period. Thus, two field periods are required to display each field once in a frame having two fields. When the RFF flag is set for a frame, the display process displays the first field of a field pair a second time after displaying the second field of the pair—thus extending the duration of the display of the frame having the field pair to three field display periods.
The compressed video bit stream can contain one or more entry points. As discussed more fully in Holcomb et al., “Signaling Valid Entry Points In A Video Stream,” U.S. patent application Ser. No. 10/882,739, filed Jun. 30, 2004 [hereafter the “Entry-Point Patent Application”], and claiming priority to U.S. Provisional Patent Application No. 60/520,543, filed Nov. 13, 2003, the disclosures of which are hereby incorporated herein by reference, valid entry points in a bitstream are locations in an elementary bitstream from which a system (e.g., a receiver, a video splicer, a commercial insertion tool, a video editor, a summarization engine, etc.) can decode or process the bitstream without the need of any preceding information (bits) in the bitstream. Frames that can be decoded without reference to preceding frames are typically referred to as “key” frames.
For interlaced video frames with interlaced I-fields, P-fields, B-fields and/or BI-fields, frame-level bitstream elements are shown in FIG. 13. Data for each frame consists of a frame header followed by data for the field layers (shown as the repeated “FieldPicLayer” element per field) and data for the macroblock layers (whether for intra, 1 MV, or 4 MV macroblocks).
The following sections describe selected bitstream elements in the sequence, entry point and frame layers that are related to skip frame signaling and skip frame repeat padding for interlaced and progressive pictures. Although the selected bitstream elements are described in the context of a particular layer, some bitstream elements can be used in more than one layer.
INTERLACE 820 is a 1-bit syntax element. INTERLACE=0 signals that the source content is progressive. INTERLACE=1 signals that the source content is interlaced. The individual frames may still be coded using the progressive or interlace syntax when INTERLACE=1. If PULLDOWN=1, the INTERLACE syntax element specifies if it is TFF and RFF, or RPTFRM that is present in the picture headers. INTERLACE is discussed in further detail below and above in Section III.
Max Coded Width (MAX_CODED_WIDTH) (12 Bits)
The MAX_CODED_WIDTH element 821 specifies the maximum horizontal size of the coded picture within the sequence. In the illustrated implementation, this syntax element is a 12-bit binary encoding of sizes. The maximum horizontal size of the picture is equal to the value of this field multiplied by 2, plus 2. The horizontal size of the coded pictures in the video sequence may change at an entry point but is always less than, or equal to, MAX_CODED_WIDTH. Alternative implementations can utilize a maximum coded width syntax element having a different size and/or specifying the maximum horizontal size in a different way.
Max Coded Height (MAX_CODED_HEIGHT) (12 Bits)
The MAX_CODED_HEIGHT element 822 specifies the maximum vertical size of the coded picture within the video sequence. In the illustrated implementation, this syntax element is a 12-bit binary encoding of sizes. The maximum vertical size of the picture is equal to the value of this field multiplied by 2, plus 2. The vertical size of the coded pictures in the video sequence may change at an entry point but is always less than, or equal to, MAX_CODED_HEIGHT. Alternative implementations can utilize a maximum coded height syntax element having a different size and/or specifying the maximum vertical size in a different way.
FCM 920 is a variable length codeword [“VLC”] used to indicate the picture coding type. FCM takes on values for frame coding modes as shown in Table 1 below:
PTYPE 921 is a variable size syntax element present in the frame header for progressive and interlaced frames. PTYPE takes on values for different frame types according to Table 3 below.
TFF is a one-bit element that is present if the sequence header element PULLDOWN is set to ‘1’ and the sequence header element INTERLACE=1. TFF=1 implies that the top field is the first decoded field. If TFF=0, the bottom field is the first decoded field. If PULLDOWN is set to ‘0’, TFF is not present, and its value shall be assumed to be ‘1’. TFF is discussed in further detail below and above in Section III.
RFF is a one-bit element that is present if the sequence header element PULLDOWN is set to ‘1’ and the sequence header element INTERLACE=1. RFF is not part of the decoding process, but it is used during display. RFF=1 implies that the first field should be repeated during display. RFF=0 implies that no repetition is necessary. RFF is discussed in further detail below and above in Section III.
RPTFRM is a two-bit syntax element that is present if the sequence header element PULLDOWN is set to ‘1’ and the sequence header element INTERLACE=0. RPTFRM takes a value from 0 to 3 which is coded in binary using 2 bits. RPTFRM is not part of the decoding process, but it is used during display. It represents the number of times a frame is repeated during display. RPTFRM is discussed in further detail below and above in Section III.
3. Selected Entry Point Layer Elements
FIG. 17 is a syntax diagram for the entry point layer 1700. The entry point layer 1700 includes an entry point header 1710 followed by data for a group of pictures forming an entry point segment. The entry point header 1710 includes several entry point-level elements that are processed by the decoder and used to decode the following picture frames without reference to preceding picture data in the video sequence. The elements that make up the entry point header include a coded size flag (CODED_SIZE_FLAG) element 1720, a coded width (CODED_WIDTH) element 1721, and a coded height (CODED_HEIGHT) element 1722, among others.
The CODED_SIZE_FLAG signals a different coded resolution for pictures in the entry point segment. In the illustrated implementation, the CODED_SIZE_FLAG element 1720 is a 1-bit syntax element. A value of one (CODED_SIZE_FLAG=1) indicates that the CODED_WIDTH and CODED_HEIGHT syntax elements are also present in the entry header. Otherwise, a flag value of zero (CODED_SIZE_FLAG=0) indicates that the CODED_WIDTH and CODED_HEIGHT syntax elements are not present in the entry header; and the width and height of the frames within the entry point segment are specified by the MAX_CODED_WIDTH and MAX_CODED_HEIGHT syntax elements in the sequence header. Alternative implementations can utilize a different format flag or value to signal a group of pictures in the video sequence has a different coded size.
The CODED_WIDTH element 1721 specifies the coded horizontal size of pictures in the entry point segment. In the illustrated implementation, the CODED_WIDTH element 1721 is a 12 bit syntax element that is present if CODED_SIZE_FLAG=1. It specifies the coded width of the frames within the entry point segment in units of 2 pixels. The coded width of the frames within the entry point segment is equal to the value of this field multiplied by 2, plus 2. Therefore, the range is 2 to 8192. Alternative implementations can use a different syntax element format to signal the coded horizontal picture size.
Similarly, the CODED_HEIGHT element 1722 specifies the coded vertical size of pictures in the entry point segment. The CODED_HEIGHT element is a 12 bit syntax element that is present if CODED_SIZE_FLAG=1. It specifies the coded height of the frames within the entry point segment in units of 2 pixels. The coded height of the frames within the entry point segment is equal to the value of this field multiplied by 2, plus 2. Therefore, the range is 2 to 8192. Alternative implementations can use a different syntax element format to signal the coded vertical picture size.
When a sequence has an interlaced target display type (INTERLACE=1) and pull-down is used (PULLDOWN=1), picture headers contain the Boolean field RFF. When the RFF flag is set, the display process may display the first field of a field pair again after displaying the second field of the pair—thus extending the duration of the field-pair (frame) to three display field periods.
A Pan/Scan window is a portion of video displayed on a screen as a result of a view selection. Pan/Scan window information is present in picture headers if the entry point header syntax element PANSCAN_FLAG is 1. In this case, each picture header in the entry point segment has the PS_PRESENT syntax element. If PS_PRESENT is 1 then for each window in the frame there are four syntax elements—PS_HOFFSET, PS_VOFFSET, PS_WIDTH and PS_HEIGHT—that define the size and location of the window within the frame.
If PS_PRESENT is 1 then there are from one to four Pan/Scan windows in each frame. The number of Pan/Scan windows is determined by the sequence header syntax elements INTERLACE and PULLDOWN and the frame header syntax elements RFF and RPTFRM. The pseudo-code 1800 in FIG. 18 illustrates how the number of Pan/Scan windows is determined.
For each Pan/Scan window there is a set of four Pan/Scan window syntax elements in the frame header: PS_HOFFSET, PS_VOFFSET, PS_WIDTH and PS_HEIGHT. The order of the pan windows in the frame header bitstream is the same as the display order of the fields or frames—meaning that the first set of Pan/Scan window syntax elements corresponds to the first field or frame in display order.
IV. Adaptive Vertical Macroblock Alignment
The video encoder 300 (FIG. 3) and decoder 400 (FIG. 4) provide for adaptive vertical macroblock alignment of mixed mode (or hybrid) video sequences, by enforcing vertical macroblock alignment restrictions on a per frame basis rather than imposing a uniform vertical macroblock alignment restriction across the entire video sequence. FIGS. 21A-B illustrate the different macroblock alignment restrictions for progressive and interlace content pictures. For pictures 2100 (FIG. 21A) in the video sequence coded in progressive mode, the video codec enforces a vertical macroblock alignment restriction of a multiple of 16 pixels. For interlaced field and interlaced frame mode pictures 2150 (FIG. 21B), the video codec enforces a height restriction to a multiple of 32 pixels. The horizontal alignment requirement is a multiple of 16 pixels. In alternative video codec implementations the height restriction for these modes can vary, such as due to use of a different macroblock size.
In a mixed frame (or hybrid) coding sequence, each frame can be encoded as one of progressive, interlaced frame or interlaced field types. By enforcing the height alignment restriction on a frame level, the video codec can achieve significant savings on padding operations relative to a design which requires all frames in a sequence to have the same height. This is because the video codec potentially can avoid padding progressive and interlace frame type pictures to the larger alignment requirement of interlaced pictures, reducing the padding required for such frames.
With reference now to FIG. 19, the video encoder 300 (FIG. 3) performs operations for an adaptive vertical macroblock alignment encoding process 1900 when encoding a mixed video sequence. The diagram is simplified to illustrate the encoding operations relating to adaptive vertical macroblock alignment. It should be apparent to those skilled in the art that the encoding of the video sequence involves many more operations (not otherwise related to adaptive vertical macroblock alignment) as summarized above in the description of the encoder 300 in FIG. 3. Alternative implementations of the video encoder can perform the adaptive vertical macroblock alignment using fewer, more or a different arrangement of operations.
The video encoder begins this process by analyzing the video content of the sequence to determine whether any frames of the sequence have interlaced content at operation 1910. If all frame have progressive content at operations 1911-1912, the video encoder sends the sequence header for the sequence with the INTERLACE flag element 820 (FIG. 8) cleared. The video encoder then pads all pictures of the sequence to have a vertical macroblock alignment that is a multiple of 16 pixels. If any frames have interlaced content, the video encoder sends the sequence header with the interlaced content flag set, which indicates at least one frame is encoded as an interlaced field or interlaced frame type at operation 1913.
The video encoder then acquires the next picture of the video sequence from the video source at operation 1914, and determines the coding mode of the frame (whether progressive, interlaced frame or interlaced field) at operation 1915. Based on the mode, the video encoder enforces the appropriate vertical macroblock alignment restriction by padding (as necessary) to a multiple of 16 pixels for progressive and interlaced frame type pictures at operation 1917, or padding (as necessary) to a multiple of 32 pixels for interlaced field pictures. The video encoder pads by repeating the last row of the actual video content of the picture (for progressive pictures) or the last two rows (for interlaced video content) to fill the picture out vertically to the next proximate macroblock alignment. The video encoder encodes the frame as the appropriate type at operation 1919. The video encoder finally checks whether the video sequence is to end at the current frame (e.g., based on user input) at operation 1920. If so, the video encoder ends encoding of the sequence. Otherwise, the video encoder returns to acquiring a next picture of the sequence at operation 1914.
FIG. 20 illustrates operations performed by the video decoder for an adaptive vertical macroblock alignment decoding process 2000 when decoding a mixed video sequence. The diagram is simplified to illustrate the decoding operations relating to adaptive vertical macroblock alignment. It should be apparent to those skilled in the art that the decoding of the video sequence involves many more operations (not otherwise related to adaptive vertical macroblock alignment) as summarized above in the description of the decoder 400 in FIG. 4. Alternative implementations of the video decoder can perform the adaptive vertical macroblock alignment using fewer, more or a different arrangement of operations.
The video decoder begins the decoding by reading the compressed bit stream to the location of the sequence header at operation 2010. The video decoder checks the INTERLACE flag element 820 (FIG. 8) in the sequence header at operation 2011. If this flag is not set (indicating all progressive type frames), the video decoder decodes all frames, including extracting the video content of each frame's picture excluding the padding added to achieve vertical macroblock alignment at a multiple of 16 pixels.
If the interlaced content flag is set, the video decoder instead proceeds to read the picture header of the next frame at operation 2013. Depending on the picture type specified in the picture coding mode (PTYPE) element (FIG. 9-16) of the picture header, the video decoder determines the amount of padding that was used to achieve vertical macroblock alignment.
With reference to FIG. 23, the decoder calculates the padded region of the picture based on the CODED_WIDTH and CODED_HEIGHT elements 1721, 1722 (FIG. 17) specified in the picture header for the frame (or MAX_CODED_WIDTH and MAX_CODED_HEIGHT elements 821, 822 in the sequence header 810 shown in FIG. 8, if no lower coded resolution is specified for the picture), and the frame type (FCM) element 820 in the picture header. The coded resolution values indicate the active frame size 2110 (FIGS. 21A-B) of the picture. Based on the active frame size dimensions and picture type, the decoder calculates the padded frame size 2120 (FIGS. 21A-B) for the frame type (progressive or interlaced) of the picture as shown in the pseudo-code listing 2300 in FIG. 23. For progressive pictures, the horizontal and vertical padded frame dimensions are the next multiple of 16 that is larger than the active frame dimensions. For interlaced pictures, the vertical padded frame size is the next multiple of 32 larger than the active frame height, whereas the padded frame width is the next multiple of 16 larger than the active frame width.
With reference again to FIG. 20, the video decoder then extracts the video content excluding the appropriate amount of padding for the picture type at operations 2015, 2016. However, the decoder may use the decoded padding region of the picture for decoding a subsequent P-Frame or B-Frame (which may have a macroblock or macroblocks with a motion vector pointing to the padding region of a preceding I-Frame).
The video decoder then checks whether the video sequence is ended at operation 2017. If so, the video decoder ends the adaptive vertical macroblock alignment process 2000 at operation 2018. Otherwise, the video decoder returns to decode the next picture header at operation 2013.
V. Efficient Repeat Padding of Hybrid Video Sequences
With reference now to FIGS. 22A-B, the repeat padding performed by the video encoder (in operations 2017 and 2018 of the adaptive vertical macroblock alignment process 2000 in FIG. 19) also differs by frame type. For progressive type pictures 2200, the video encoder 300 (FIG. 3) repeats the last row (horizontal boundary or edge) 2210 of the active content in the frame to pad the frame out to the macroblock alignment (i.e., the next multiple of 16). For interlaced (both interlaced field- and interlaced frame-type) pictures 2250, the video encoder repeats the last row of each field of the active content (i.e., the last two rows 2260, 2261 of the active content) to pad the frame out to the macroblock alignment, which is the next multiple of 32 for interlaced field or multiple of 16 for interlaced frame mode pictures. For both progressive and interlaced pictures, the video encoder pads the picture horizontally by repeating the right vertical edge of the active content out to the padded picture size (i.e., the next multiple of 16 for both progressive and interlaced types).
In addition, each picture used as a reference frame for predictive coding is further repeat padded to expand the picture area for purposes of motion estimation and prediction for subsequent predicted (P- and B-) frames. This repeat padding for prediction also differs by frame type. For progressive type pictures 2200, the video encoder 300 (and video decoder at decoding) repeats the last row (horizontal boundary or edge) 2210 of the active content in the frame to an expanded motion estimation region. The first row also is repeated vertically upward to provide an expanded motion estimation region above the picture. For interlaced (both interlaced field- and interlaced frame-type) pictures 2250, the video encoder (and video decoder at decoding) repeats the last row of each field of the active content (i.e., the last two rows 2260, 2261 of the active content) to pad the frame out to the expanded motion estimation region. Similarly, the first two rows are repeated vertically upward to provide expanded motion estimation above the picture. The left and right edges of the coded picture are repeated to expand motion estimation respectively left and right horizontally on both progressive and interlaced pictures. In the illustrated implementation, the expanded motion estimation region for progressive pictures extends 32 pixels horizontally and vertically from the macroblock aligned frame size, and extends 38 pixels horizontally and vertically from the macroblock aligned frame size of interlaced (both frame and field-types) pictures. This expanded motion estimation region is not part of the picture encoded into the compressed bit stream, but simply used as the basis of motion estimation and prediction for coding of the subsequent predicted (P- or B-) frame.
Patent CitationsCited PatentFiling datePublication dateApplicantTitleUS4706260Nov 7, 1986Nov 10, 1987Rca CorporationDPCM system with rate-of-fill control of buffer occupancyUS4849812Feb 24, 1988Jul 18, 1989U.S. Philips CorporationTelevision system in which digitized picture signals subjected to a transform coding are transmitted from an encoding station to a decoding stationUS4954892Oct 4, 1989Sep 4, 1990Mitsubishi Denki Kabushiki KaishaBuffer controlled picture signal encoding and decoding systemUS5027206Sep 13, 1989Jun 25, 1991U.S. Philips CorporationHigh-definition television systemsUS5091782Apr 9, 1990Feb 25, 1992General Instrument CorporationApparatus and method for adaptively compressing successive blocks of digital videoUS5175618Oct 30, 1991Dec 29, 1992Victor Company Of Japan, Ltd.Compression method for interlace moving image signalsUS5191436Apr 30, 1991Mar 2, 1993Sony CorporationMethod for recording coded motion picture dataUS5212549Jun 16, 1992May 18, 1993Rca Thomson Licensing CorporationError concealment apparatus for a compressed video signal processing systemUS5223949Apr 17, 1992Jun 29, 1993Matsushita Electric Industrial Co., Ltd.Coding means for a signal processing systemUS5235618Nov 6, 1990Aug 10, 1993Fujitsu LimitedVideo signal coding apparatus, coding method used in the video signal coding apparatus and video signal coding transmission system having the video signal coding apparatusUS5291486Aug 7, 1992Mar 1, 1994Sony CorporationData multiplexing apparatus and multiplexed data demultiplexing apparatusUS5301242Aug 31, 1992Apr 5, 1994International Business Machines CorporationApparatus and method for motion video encoding employing an adaptive quantizerUS5343248Jul 16, 1992Aug 30, 1994Sony CorporationMoving image compressing and recording medium and moving image data encoder and decoderUS5365552Nov 16, 1992Nov 15, 1994Intel CorporationBuffer fullness indicatorUS5422676Oct 22, 1993Jun 6, 1995Deutsche Thomson-Brandt GmbhSystem for coding an image representative signalUS5426464Oct 18, 1994Jun 20, 1995Rca Thomson Licensing CorporationField elimination apparatus for a video compression/decompression systemUS5461420Sep 17, 1993Oct 24, 1995Sony CorporationApparatus for coding and decoding a digital video signal derived from a motion picture film sourceUS5491516Jan 14, 1993Feb 13, 1996Rca Thomson Licensing CorporationField elimination apparatus for a video compression/decompression systemUS5510840May 15, 1995Apr 23, 1996Sony CorporationMethods and devices for encoding and decoding frame signals and recording medium thereforUS5539466Sep 26, 1994Jul 23, 1996Sony CorporationEfficient coding apparatus for picture signal and decoding apparatus thereforUS5541852Apr 14, 1994Jul 30, 1996Motorola, Inc.Device, method and system for variable bit-rate packet video communicationsUS5543847Dec 13, 1993Aug 6, 1996Sony CorporationPicture coding and decoding method for random accessingUS5566208Mar 17, 1994Oct 15, 1996Philips Electronics North America Corp.Encoder buffer having an effective size which varies automatically with the channel bit-rateUS5600376Mar 20, 1995Feb 4, 1997Rca Thomson Licensing CorporationField elimination apparatus for a video compression/decompression systemUS5606539Aug 31, 1994Feb 25, 1997U.S. Philips CorporationMethod and apparatus for encoding and decoding an audio and/or video signal, and a record carrier for use with such apparatusUS5608697Mar 18, 1996Mar 4, 1997U.S. Philips CorporationRecord carrier containing an audio and/or video signal which has been encoded and includes a decoder delay time parameter indicating a time delay for one or more portions of the signalUS5691771Dec 26, 1995Nov 25, 1997Sony CorporationProcessing of redundant fields in a moving picture to achieve synchronized system operationUS5699476May 9, 1996Dec 16, 1997U.S. Philips CorporationMethod and apparatus for transmitting and/or storing a series of hierarchically encoded digital image data blocksUS5740310Jun 28, 1994Apr 14, 1998U.S. Philips CorporationMethod of maintaining display continuity from a CD with slow-motion or freeze capabilityUS5784107Jan 23, 1996Jul 21, 1998Matsushita Electric Industrial Co., Ltd.Method and apparatus for picture coding and method and apparatus for picture decodingUS5844867Sep 9, 1996Dec 1, 1998U.S. Philips CorporationMethods and apparatus for encoding and decoding an audio and/or video signal, and a record carrier used therewith or produced therefromUS5874995Sep 2, 1997Feb 23, 1999Matsuhita Electric Corporation Of AmericaMPEG video decoder having a high bandwidth memory for use in decoding interlaced and progressive signalsUS5933451Apr 22, 1994Aug 3, 1999Thomson Consumer Electronics, Inc.Complexity determining apparatusUS5946043Dec 31, 1997Aug 31, 1999Microsoft CorporationVideo coding using adaptive coding of block parameters for coded/uncoded blocksUS5952943Oct 11, 1996Sep 14, 1999Intel CorporationEncoding image data for decode rate controlUS5970173Jun 4, 1996Oct 19, 1999Microsoft CorporationImage compression and affine transformation for image motion compensationUS6060997Oct 27, 1997May 9, 2000Motorola, Inc.Selective call device and method for providing a stream of informationUS6088063Jan 29, 1997Jul 11, 2000Rohm Co., Ltd.Data encoding method and apparatus for outputting encoded data at a designated timeUS6101195May 28, 1997Aug 8, 2000Sarnoff CorporationTiming correction method and apparatusUS6141053Jan 3, 1997Oct 31, 2000Saukkonen; Jukka I.Method of optimizing bandwidth for transmitting compressed video data streamsUS6160849May 30, 1995Dec 12, 2000Sony CorporationSelectable field and frame based predictive video codingUS6198773Dec 18, 1997Mar 6, 2001Zoran CorporationVideo memory management for MPEG video decode and display systemUS6233226Jan 29, 1999May 15, 2001Verizon Laboratories Inc.System and method for analyzing and transmitting video over a switched networkUS6282245May 9, 1997Aug 28, 2001Sony CorporationProcessing of redundant fields in a moving picture to achieve synchronized system operationUS6295321 *Dec 29, 1998Sep 25, 2001Lg Electronics Inc.Video decoding method, video decoder and digital TV system using the video decoding method and video decoderUS6324216May 30, 1995Nov 27, 2001Sony CorporationVideo coding selectable between intra-frame prediction/field-based orthogonal transformation and inter-frame prediction/frame-based orthogonal transformationUS6370276Apr 3, 1998Apr 9, 2002Matsushita Electric Industrial Co., Ltd.Image predictive decoding method, image predictive decoding apparatus, image predictive coding method, image predictive coding apparatus, and data storage mediaUS6408096Oct 21, 1998Jun 18, 2002Matsushita Electric Industrial Co., Ltd.Method for computational graceful degradation in an audiovisual compression systemUS6441813May 13, 1998Aug 27, 2002Kabushiki Kaisha ToshibaComputer system, and video decoder used in the systemUS6459811Apr 1, 1999Oct 1, 2002Sarnoff CorporationBursty data transmission of compressed video dataUS6499060Mar 12, 1999Dec 24, 2002Microsoft CorporationMedia coding for loss recovery with remotely predicted data unitsUS6556627Nov 30, 2000Apr 29, 2003Sony CorporationInformation processing apparatus, information processing method and recording mediumUS6593392Jun 22, 2001Jul 15, 2003Corning IncorporatedCurable halogenated compositionsUS6654420Oct 27, 2000Nov 25, 2003Koninklijke Philips Electronics N.V.Video encoding-methodUS6671323 *Feb 4, 2000Dec 30, 2003Sony CorporationEncoding device, encoding method, decoding device, decoding method, coding system and coding methodUS6728317Apr 7, 2000Apr 27, 2004Dolby Laboratories Licensing CorporationMoving image compression quality enhancement using displacement filters with negative lobesUS6775840Dec 19, 1997Aug 10, 2004Cisco Technology, Inc.Method and apparatus for using a spectrum analyzer for locating ingress noise gapsUS6792048Oct 30, 2000Sep 14, 2004Samsung Electronics Co., Ltd.Terminal supporting signaling used in transmission and reception of MPEG-4 dataUS6842485Mar 29, 2001Jan 11, 2005Matsushita Electric Industrial Co., Ltd.Method and apparatus for reproducing compressively coded dataUS6873629Dec 20, 2000Mar 29, 2005Koninklijke Philips Electronics N.V.Method and apparatus for converting data streamsUS7010046 *May 2, 2002Mar 7, 2006Lsi Logic CorporationMethod and/or architecture for implementing MPEG frame display using four frame storesUS7064790Jun 7, 2002Jun 20, 2006Microsoft CorporationAdaptive video data frame resamplingUS7227895Dec 11, 2001Jun 5, 2007Sony CorporationSystem and method for generating decoded digital video image dataUS7298303May 18, 2006Nov 20, 2007Ntt Docomo, Inc.Signal encoding method, signal decoding method, signal encoding apparatus, signal decoding apparatus, signal encoding program, and signal decoding programUS7609762Nov 15, 2004Oct 27, 2009Microsoft CorporationSignaling for entry point frames with predicted first fieldUS20010010708Jan 29, 2001Aug 2, 2001Nec CorporationDevice for and method of converting a frame rate in a moving picture decoder, and a record medium and an integrated circuit device for implementing such a methodUS20010055469Nov 29, 2000Dec 27, 2001Tetsuro ShidaDecoder and reproducing unitUS20020001348Feb 22, 2001Jan 3, 2002Zhu YiwenApparatus and method for converting signalsUS20020012394Apr 27, 2001Jan 31, 2002Yoshiko HatanoEncoding system and encoding methodUS20020061067Jul 25, 2001May 23, 2002Lyons Paul W.Splicing compressed, local video segments into fixed time slots in a network feedUS20020061073Nov 23, 2001May 23, 2002Jun HuangApparatus and method for controlling wireless communication signalsUS20020080875Jun 1, 1999Jun 27, 2002Katsumi TaharaCoded stream splicing device and method, and coded stream generating device and methodUS20020191712Apr 9, 2002Dec 19, 2002Koninklijke Philips Electronics N.V.Packet identification mechanism at the transmitter and receiver for an enhanced ATSC 8-VSB systemUS20020196857Mar 26, 2002Dec 26, 2002Fujitsu LimitedVideo decoding device and method, and program product thereforUS20030053416Sep 19, 2001Mar 20, 2003Microsoft CorporationGeneralized reference decoder for image or video processingUS20030113026 *Dec 16, 2002Jun 19, 2003Microsoft CorporationSkip macroblock codingUS20030137600Jan 22, 2003Jul 24, 2003Chen Sherman (Xuemin)System and method of transmission and reception of progressive content with isolated fields for conversion to interlaced displayUS20030185298 *Apr 1, 2002Oct 2, 2003Alvarez Jose?Apos; R.Method of operating a video decoding systemUS20030202590 *Apr 30, 2002Oct 30, 2003Qunshan GuVideo encoding using direct mode predicted framesUS20030231593May 16, 2003Dec 18, 2003James BaumanFlexible multilevel output traffic controlUS20040042673 *Aug 22, 2003Mar 4, 2004Boon Choong SengDigital image coding/decoding method and data recording mediumUS20040130619Jan 6, 2003Jul 8, 2004Wen-Kuo LinMethod for 3:2 pull-down film source detectionUS20040142699Nov 3, 2003Jul 22, 2004Jollota James M.Method and system for indicating link quality among neighboring wireless base stationUS20040255063Oct 30, 2003Dec 16, 2004Crinon Regis J.Robust delivery of video dataUS20050074061Sep 2, 2004Apr 7, 2005Microsoft CorporationSignaling buffer fullnessUS20050083218Sep 4, 2004Apr 21, 2005Microsoft CorporationSignaling for field ordering and field/frame display repetitionUS20050100093Nov 15, 2004May 12, 2005Microsoft CorporationSignaling field type informationUS20050135484Jun 7, 2004Jun 23, 2005Daeyang Foundation (Sejong University)Method of encoding mode determination, method of motion estimation and encoding apparatusUS20050147167Dec 24, 2003Jul 7, 2005Adriana DumitrasMethod and system for video encoding using a variable number of B framesUS20050200758Jun 17, 2004Sep 15, 2005Chuck MonahanMethod and system for detecting field IDUS20060013568Jul 14, 2004Jan 19, 2006Rodriguez Arturo ASystem and method for playback of digital video pictures in compressed streamsUS20060143678Dec 10, 2004Jun 29, 2006Microsoft CorporationSystem and process for controlling the coding bit rate of streaming media data employing a linear quadratic control technique and leaky bucket modelUS20070153918Oct 13, 2006Jul 5, 2007Rodriguez Arturo ASystems and methods for processing chroma samples in videoUSRE36015Oct 2, 1995Dec 29, 1998Matsushita Electric Corporation Of AmericaApparatus and method for processing groups of fields in a video data compression systemUSRE36507Oct 21, 1997Jan 18, 2000Matsushita Electric Corporation Of AmericaApparatus and method for processing groups of fields in a video data compression system to encode a single frame as an I-field and a P-fieldUSRE37222Jul 19, 1994Jun 12, 2001Sony CorporationVideo signal transmitting systemEP0599529A2Nov 15, 1993Jun 1, 1994Matsushita Electric Industrial Co., Ltd.Method and apparatus for picture coding and method and apparatus for picture decodingEP0852445A2Oct 15, 1997Jul 8, 1998Ncr International Inc.Method of optimizing bandwidth for transmitting compressed video data streamsJP1049388A Title not availableJP4207684B2 Title not availableJP5236457B2 Title not availableJP6030394A Title not availableJP62032725A Title not availableJP2000165866A Title not availableJPH0630394A Title not availableJPH04207684A Title not availableJPH04297179A Title not availableJPH05236457A Title not availableJPH08223385A Title not availableJPH09261266A Title not availableJPS6232725A Title not availableJPS6449388A Title not available* Cited by examinerNon-Patent CitationsReference1"DivX Multi Standard Video Encoder," 2 pp. (Downloaded from the World Wide Web on Jan. 24, 2006).2AVC-362, "Picture Header of Skipped Picture," 2 pp. (1992).3Borgwardt, "Core Experiment on Interlaced Video Coding," ITU Study Group 16 Question 16, VCEG-N85, 10 pp. (Oct. 2001).4Chen et al., "Source Model for Transform Video Coder and Its Application-Part II: Variable Frame Rate Coding," IEEE Transactions on Circuits and Systems for Video Technology, vol. 7, No. 2, pp. 299-311 (Apr. 1997).5Chen et al., "Source Model for Transform Video Coder and Its Application—Part II: Variable Frame Rate Coding," IEEE Transactions on Circuits and Systems for Video Technology, vol. 7, No. 2, pp. 299-311 (Apr. 1997).6Hsu et al., "Joint Selection of Source and Channel Rate for VBR Video Transmission Under ATM Policing Constraints," IEEE Journal on Selected Areas in Communication, vol. 15, No. 6, pp. 1016-1028 (Aug. 1997).7 *International Standard ISO/IEC 14496-2 Second Edition Dec. 1, 2001. Information technology-Coding of audio-visual objects-Part 2: Visual. p. 144, 201-219.8 *International Standard ISO/IEC 14496-2 Second Edition Dec. 1, 2001. Information technology—Coding of audio-visual objects—Part 2: Visual. p. 144, 201-219.9ISO/IEC 13818-1, "MPEG-2 Systems-Coding of Moving Pictures and Associated Audio," 129 pp. (Mar. 1994).10ISO/IEC 13818-1, "MPEG-2 Systems—Coding of Moving Pictures and Associated Audio," 129 pp. (Mar. 1994).11ISO/IEC 13818-1, "MPEG-2 Systems-Coding of Moving Pictures and Associated Audio," second edition, 171 pp. (Dec. 2000).12ISO/IEC 13818-1, "MPEG-2 Systems—Coding of Moving Pictures and Associated Audio," second edition, 171 pp. (Dec. 2000).13ISO/IEC JVT-D131, "HRD and Related Issues," 5 pp. (Jul. 2002).14ISO/IEC JVT-E133, "Time-Shift Causality Constraint on the CAT-LB HRD," 8 pp. (Oct. 2002).15ISO/IEC JVT-E133, "Time—Shift Causality Constraint on the CAT-LB HRD," 8 pp. (Oct. 2002).16ISO/IEC JVT-G050r1 (Draft ITU-T Rec. H.264), "Joint Video Specification," 268 pp. (May 2003).17ISO/IEC, "ISO/IEC 11172-2, Information technology-Coding of moving pictures and associated audio for digital storage media at up to about 1,5 Mbit/s-Part 2: Video," 112 pp. (1993).18ISO/IEC, "ISO/IEC 11172-2, Information technology—Coding of moving pictures and associated audio for digital storage media at up to about 1,5 Mbit/s—Part 2: Video," 112 pp. (1993).19ISO/IEC, "JTC1/SC29/WG11 N2202, Information Technology-Coding of Audio-Visual Objects: Visual, ISO/IEC 14496-2," 329 pp. (1998).20ISO/IEC, "JTC1/SC29/WG11 N2202, Information Technology—Coding of Audio-Visual Objects: Visual, ISO/IEC 14496-2," 329 pp. (1998).21ITU-T Recommendation H.263, "Draft H.263," International Telecommunication Union, pp. 5, 46, 47 (Jan. 1998).22ITU-T, "ITU-T Recommendation H.261, Video Codec for Audiovisual Services at p x 64 kbits," 25 pp. (1993).23ITU-T, "ITU-T Recommendation H.262, Information Technology-Generic Coding of Moving Pictures and Associated Audio Information: Video," 205 pp. (1995).24ITU-T, "ITU-T Recommendation H.262, Information Technology—Generic Coding of Moving Pictures and Associated Audio Information: Video," 205 pp. (1995).25ITU-T, "ITU-T Recommendation H.263, Video coding for low bit rate communication," 162 pp. (1998).26ITU-Telecommunications Standardization Sector, VCEG-N58, "A Generalized Hypothetical Reference Decoder for H.26L," 14 pp. (Sep. 2001).27Jenkac et al., "On Video Streaming over Variable Bit-rate and Wireless Channels," presented at Packet Video 2003, 11 pp. (Apr. 28-29, 2003).28Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG, "Joint Final Committee Draft (JFCD) of Joint Video Specification," JVT-D157, 207 pp. (Aug. 2002).29Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG, ISO/IEC JVT-E075, "HRD Clean-up," 7 pp. (Oct. 2002).30Joint Video Team of ISO/IEC MPEG and ITU-T VCEG, "Final Joint Committee Draft of Joint Video Specification (ITU-T Recommendation H.264, ISO/IEC 14496-10 AVC)," 206 pp. (Aug. 2002).31Microsoft Corporation, "Microsoft Debuts New Windows Media Player 9 Series, Redefining Digital Media on the PC," 4 pp. (Sep. 4, 2002) [Downloaded from the World Wide Web on May 14, 2004].32Mook, "Next-Gen Windows Media Player Leaks to the Web," BetaNews, 17 pp. (Jul. 2002) [Downloaded from the World Wide Web on Aug. 8, 2003].33Pao et al., "Encoding Stored Video for Streaming Applications," IEEE Transactions on Circuits and Systems for Video Technology, vol. 11, No. 2, pp. 199-209 (Feb. 2001).34Printouts of FTP directories from http://ftp3.itu.ch , 8 pp. (downloaded from the World Wide Web on Sep. 20, 2005.).35Reader, "History of MPEG Video Compression-Ver. 4.0," 99 pp., document marked Dec. 16, 2003.36Reader, "History of MPEG Video Compression—Ver. 4.0," 99 pp., document marked Dec. 16, 2003.37Reed et al., "Constrained Bit-Rate Control for Very Low Bit-Rate Streaming-Video Applications," IEEE Transactions on Circuits and Systems for Video Technology, vol. 11, No. 7, pp. 882-889 (Jul. 2001).38Reibman et al., "Constraints on Variable Bit-Rate Video for ATM Networks," IEEE Transactions on Circuits and Systems for Video Technology, vol. 2, No. 4, pp. 361-372 (Dec. 1992).39Ribas-Corbera et al., "A Generalized Hypothetical Reference Decoder for H.264/AVC," IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, No. 7, 14 pp. (Jul. 2003).40Ribas-Corbera et al., A Generalized Hypothetical Reference Decoder for H.26L (proposal) ITU Study Group 16 Question 6 (Q.6/SG16), VCEG, VCEG-N58, 14th Meeting, Santa Barbara, CA, Sep. 2001.41Sheu et al., "A Buffer Allocation Mechanism for VBR Video Playback," Communication Tech. Proc. 2000, WCC-ICCT 2000, vol. 2, pp. 1641-1644 (Aug. 2000).42SMPTE, "Proposed SMPTE Standard for Television: VC-9 Compressed Video Bitstream Format and Decoding Process," Working Draft 2 (Oct. 2003).43Sullivan et al., "The H.264/AVC Advanced Video Coding Standard: Overview and Introduction to the Fidelity Range Extensions," 21 pp. (Aug. 2004).44U.S. Appl. No. 60/341,674, filed Dec. 17, 2001, Lee et al.45U.S. Appl. No. 60/488,710, filed Jul. 18, 2003, Srinivasan et al.46U.S. Appl. No. 60/501,081, filed Sep. 7, 2003, Srinivasan et al.47Walpole et al., "A Player for Adaptive MPEG Video Streaming over the Internet," Proc. SPIE, vol. 3240, pp. 270-281 (Mar. 1998).48Wang et al., "Research on HDTV decoder synchronization system," IEEE, pp. 868-870 (Dec. 2000).49Wien, "Variable Block-Size Transforms for Hybrid Video Coding," Dissertation, 182 pp. (Feb. 2004).* Cited by examinerReferenced byCiting PatentFiling datePublication dateApplicantTitleUS20110026593 *Jan 14, 2010Feb 3, 2011New Wei LeeImage processing apparatus, image processing method, program and integrated circuit* Cited by examinerClassifications U.S. Classification375/240.12, 375/240.25International ClassificationH04N7/12, H04N7/50, H04N7/36, H04N7/26Cooperative ClassificationH04N19/13, H04N19/16, H04N19/152, H04N19/46, H04N19/86, H04N19/82, H04N19/184, H04N19/70, H04N19/124, H04N19/159, H04N19/176, H04N19/523, H04N19/179, H04N19/61, H04N19/146, H04N19/51, H04N19/117, H04N19/172European ClassificationH04N7/26A4V, H04N7/50, H04N7/26M2S, H04N7/26F2, H04N7/26A8T, H04N7/26A6S2, H04N7/26A4F, H04N7/26A6E6, H04N7/26A8S, H04N7/26A8P, H04N7/26A10S, H04N7/50M, H04N7/26P4, H04N7/26A4Q, H04N7/26P6, H04N7/36C2, H04N7/26A6S4, H04N7/26A8B, H04N7/26A6ELegal EventsDateCodeEventDescriptionDec 7, 2004ASAssignmentOwner name: MICROSOFT CORPORATION, WASHINGTONFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:REGUNATHAN, SHANKAR;LIN, CHIH-LUNG;HOLCOMB, THOMAS W.;AND OTHERS;REEL/FRAME:015420/0778;SIGNING DATES FROM 20041111 TO 20041112Owner name: MICROSOFT CORPORATION, WASHINGTONFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:REGUNATHAN, SHANKAR;LIN, CHIH-LUNG;HOLCOMB, THOMAS W.;AND OTHERS;SIGNING DATES FROM 20041111 TO 20041112;REEL/FRAME:015420/0778Dec 9, 2014ASAssignmentOwner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTONFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:034543/0001Effective date: 20141014Jun 24, 2015FPAYFee paymentYear of fee payment: 4RotateOriginal ImageGoogle Home - Sitemap - USPTO Bulk Downloads - Privacy Policy - Terms of Service - About Google Patents - Send FeedbackData provided by IFI CLAIMS Patent Services