Abstract:
Presented herein are systems, methods, and apparatus for two processor architecture supporting decoupling of the outer loop and the inner loop in a video decoder. In one embodiment, there is presented a video decoder for decoding a data structure. The video decoder comprises an outer loop processor and an inner loop processor. The outer loop processor performs overhead processing for the data structure. The inner loop processor decodes the data structure.

Description:
RELATED APPLICATIONS  
     FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT  
       [0001]     [Not Applicable] 
       [MICROFICHE/COPYRIGHT REFERENCE] 
       [0002]     [Not Applicable] 
       BACKGROUND OF THE INVENTION  
       [0003]     Both MPEG-2 and H.264 use slices to group macroblocks forming a picture. The slices comprise a set of symbols. The symbols are encoded using variable length codes. In H.264, the symbols are encoded using context adaptive codes. The variable length codes of a slice can be decoded independently.  
         [0004]     Decoding slices includes overhead prior to decoding the symbols. In H.264, a large number of slices, such as two per macroblock row, are used. Additionally, the slices use reference lists that are built prior to decoding the slice. For example, the reference list can include a list of pictures upon which the macroblocks of the slice can depend. In H.264, a slice can be predicted from as many as 16 reference pictures.  
         [0005]     Further limitations and disadvantages of conventional and traditional approaches will become apparent to one of ordinary skill in the art through comparison of such systems with the present invention as set forth in the remainder of the present application with reference to the drawings.  
       BRIEF SUMMARY OF THE INVENTION  
       [0006]     Presented herein are systems, methods, and apparatus for two processor architecture supporting decoupling of the outer loop and the inner loop in a video decoder.  
         [0007]     In one embodiment, there is presented a video decoder for decoding a data structure. The video decoder comprises an outer loop processor and an inner loop processor. The outer loop processor performs overhead processing for the data structure. The inner loop processor decodes the data structure.  
         [0008]     In another embodiment, there is a method for decoding a data structure. The method comprises performing overhead processing for the data structure at a first processor; and decoding the data structure at a second processor.  
         [0009]     These and other features and advantages of the present invention may be appreciated from a review of the following detailed description of the present invention, along with the accompanying figures in which like reference numerals refer to like parts throughout.  
     
    
     BRIEF DESCRIPTION OF SEVERAL VIEWS OF THE DRAWINGS  
       [0010]      FIG. 1  is a block diagram of a frame;  
         [0011]      FIG. 2A  is a block diagram describing spatially encoded macroblocks;  
         [0012]      FIG. 2B  is a block diagram describing temporally encoded macroblocks;  
         [0013]      FIG. 2C  is a block diagram describing partitions in a block;  
         [0014]      FIG. 3  is a block diagram describing an exemplary video decoder system in accordance with an embodiment of the present invention;  
         [0015]      FIG. 4  is a block diagram describing an interface between an outer loop processor and an inner loop processor in accordance with an embodiment of the present invention;  
         [0016]      FIG. 5  is a flow diagram for decoding a data structure in accordance with an embodiment of the present invention;  
         [0017]      FIG. 6  is a block diagram describing a decoded picture buffer structure and an intermediate structure in accordance with an embodiment of the present invention; and  
         [0018]      FIG. 7  is a flow diagram for providing indicators in accordance with an embodiment of the present invention.  
     
    
     DETAILED DESCRIPTION OF THE INVENTION  
       [0019]     Referring now to  FIG. 1 , there is illustrated a block diagram of a frame  100 . A video camera captures frames  100  from a field of view during time periods known as frame durations. The successive frames  100  form a video sequence. A frame  100  comprises two-dimensional grid(s) of pixels  100 (x,y).  
         [0020]     For color video, each color component is associated with a two-dimensional grid of pixels. For example, a video can include luma, chroma red, and chroma blue components. Accordingly, the luma, chroma red, and chroma blue components are associated with a two-dimensional grid of pixels  1000 Y(x,y),  100 Cr(x,y), and  100 Cb(x,y), respectively. When the grids of two dimensional pixels  100 Y(x,y),  100 Cr(x,y), and  100 Cb(x,y) from the frame are overlayed on a display device  110 , the result is a picture of the field of view at the frame duration that the frame was captured.  
         [0021]     Generally, the human eye is more perceptive to the luma characteristics of video, compared to the chroma red and chroma blue characteristics. Accordingly, there are more pixels in the grid of luma pixels  100 Y(x,y) compared to the grids of chroma red  100 Cr(x,y) and chroma blue  100 Cb(x,y). In the MPEG 4:2:0 standard, the grids of chroma red  100 Cr(x,y) and chroma blue pixels  100 Cb(x,y) have half as many pixels as the grid of luma pixels  100 Y(x,y) in each direction.  
         [0022]     The chroma red  100 Cr(x,y) and chroma blue  100 Cb(x,y) pixels are overlayed the luma pixels in each even-numbered column  100 Y(x,2y) between each even, one-half a pixel below each even-numbered line  100 Y(2x,y). In other words, the chroma red and chroma blue pixels  100 Cr(x,y) and  100 Cb(x,y) are overlayed pixels  100 Y(2x+½, 2y).  
         [0023]     If the video camera is interlaced, the video camera captures the even-numbered lines  100 Y(2x,y),  100 Cr(2x,y), and  100 Cb(2x,y) during half of the frame duration (a field duration), and the odd-numbered lines  100 Y(2x+1,y),  100 Cr(2x+1,y), and  100 Cb(2x+1,y) during the other half of the frame duration. The even numbered lines  100 Y(2x,y),  100 Cr(2x,y), and  100 Cb(2x,y) form what is known as a top field  110 T, while odd-numbered lines  100 Y(2x+1,y),  100 Cr(2x+1,y), and  100 Cb(2x+1,y) form what is known as the bottom field  110 B. The top field  110 T and bottom field  110 T are also two dimensional grid(s) of luma  110 YT(x,y), chroma red  110 CrT(x,y), and chroma blue  110 CbT(x,y) pixels.  
         [0024]     Luma pixels of the frame  100 Y(x,y), or top/bottom fields  110 YT/B(x,y) can be divided into 16×16 pixel  100 Y(16x-&gt;16x+15, 16y-&gt;16y+15) blocks  115 Y(x,y). For each block of luma pixels  115 Y(x,y), there is a corresponding 8×8 block of chroma red pixels  115 Cr(x,y) and chroma blue pixels  115 Cb(x,y) comprising the chroma red and chroma blue pixels that are to be overlayed the block of luma pixels  115 Y(x,y). A block of luma pixels  115 Y(x,y), and the corresponding blocks of chroma red pixels  115 Cr(x,y) and chroma blue pixels  115 Cb(x,y) are collectively known as a macroblock  120 . The macroblocks  120  can be grouped into groups known as slices  122 .  
         [0025]     The ITU-H.264 Standard (H.264), also known as MPEG-4, Part 10, and Advanced Video Coding, encodes video on a frame by frame basis, and encodes frames on a macroblock by macroblock basis. H.264 specifies the use of spatial prediction, temporal prediction, DCT transformation, interlaced coding, and lossless entropy coding to compress the macroblocks  120 .  
         [0000]     Spatial Prediction  
         [0026]     Referring now to  FIG. 2A , there is illustrated a block diagram describing spatially encoded macroblocks  120 . Spatial prediction, also referred to as intraprediction, involves prediction of frame pixels from neighboring pixels. The pixels of a macroblock  120  can be predicted, either in a 16×16 mode, an 8×8 mode, or a 4×4 mode.  
         [0027]     In the 16×16 and 8×8 modes, e.g, macroblock  120   a,  and  120   b,  respectively, the pixels of the macroblock are predicted from a combination of left edge pixels  125 L, a corner pixel  125 C, and top edge pixels  125 T. The difference between the macroblock  120   a  and prediction pixels P is known as the prediction error E. The prediction error E is calculated and encoded along with an identification of the prediction pixels P and prediction mode, as will be described.  
         [0028]     In the 4×4 mode, the macroblock  120   c  is divided into 4×4 partitions  130 . The 4×4 partitions  130  of the macroblock  120   a  are predicted from a combination of left edge partitions  130 L, a corner partition  130 C, right edge partitions  130 R, and top right partitions  130 TR. The difference between the macroblock  120   a  and prediction pixels P is known as the prediction error E. The prediction error E is calculated and encoded along with an identification of the prediction pixels and prediction mode, as will be described. A macroblock  120  is encoded as the combination of the prediction errors E representing its partitions  130 .  
         [0000]     Temporal Prediction  
         [0029]     Referring now to  FIG. 2B , there is illustrated a block diagram describing temporally encoded macroblocks  120 . The temporally encoded macroblocks  120  can be divided into 16×8, 8×16, 8×8, 4×8, 8×4, and 4×4 partitions  130 . Each partition  130  of a macroblock  120 , is compared to the pixels of other frames or fields for a similar block of pixels P. A macroblock  120  is encoded as the combination of the prediction errors E representing its partitions  130 .  
         [0030]     The similar block of pixels is known as the prediction pixels P. The difference between the partition  130  and the prediction pixels P is known as the prediction error E. The prediction error E is calculated and encoded, along with an identification of the prediction pixels P. The prediction pixels P are identified by motion vectors MV. Motion vectors MV describe the spatial displacement between the partition  130  and the prediction pixels P. The motion vectors MV can, themselves, be predicted from neighboring partitions.  
         [0031]     The partition can also be predicted from blocks of pixels P in more than one field/frame. In bi-directional coding, the partition  130  can be predicted from two weighted blocks of pixels, P 0  and P 1 . According a prediction error E is calculated as the difference between the weighted average of the prediction blocks w 0 P 0 +w 1 P 1  and the partition  130 . The prediction error E, an identification of the prediction blocks P 0 , P 1  are encoded. The prediction blocks P 0  and P 1  are identified by motion vectors MV.  
         [0032]     The weights w 0 , w 1  can also be encoded explicitly, or implied from an identification of the field/frame containing the prediction blocks P 0  and P 1 . The weights w 0 , w 1  can be implied from the distance between the frames/fields containing the prediction blocks P 0  and P 1  and the frame/field containing the partition  130 . Where T 0  is the number of frame/field durations between the frame/field containing P 0  and the frame/field containing the partition, and T 1  is the number of frame/field durations for P 1 , 
 
 w 0=1− T 0/( T 1+ T 1) 
 
 w 1=1− T 1/( T 0+ T 1) 
 
         [0033]     For a high definition television picture, there are thousands of macroblocks  120  per frame  100 . The macroblocks  120 , themselves can be partitioned into potentially  16  4×4 partitions  130 , each associated with potentially different motion vector sets. Thus, coding each of the motion vectors without data compression can require a large amount of data and bandwidth.  
         [0034]     To reduce the amount of data used for coding the motion vectors, the motion vectors themselves are predicted. Referring now to  FIG. 2C , there is illustrated a block diagram describing an exemplary partition  130 . The motion vectors for the partition  130  can be predicted from the left A, top left corner D, top C, and top right corner C neighboring partitions. For example, the median of the motion vector(s) for A, B, C, and D can be calculated as the prediction value. The motion vector(s) for partition  130  can be coded as the difference (mvDelta) between itself and the prediction value. Thus the motion vector(s) for partition  130  can be represented by an indication of the prediction, median (A,B,C,D) and the difference, mvDelta. Where mvDelta is small, considerable memory and bandwidth are saved.  
         [0035]     However, where partition  130  is at the top left corner of a macroblock  120 , partition A is in the left neighboring macroblock  120 A, partition D is in the top left neighboring macroblock  120 D, while partitions B and C are in macroblock  120 B. Where partition  130  is at the top right corner of a macroblock  120 , the top left corner d and the top b neighboring partitions are in the top neighboring macroblock  120 B, while the top right corner neighboring partition c is in the top right corner neighboring macroblock  120 C.  
         [0036]     The macroblocks  120  forming a picture are grouped into what are known as slices  150 . The slices  150  comprise a set of symbols. The symbols are encoded using variable length codes. In H.264, the symbols are encoded using context adaptive codes. The variable length codes of a slice can be decoded independently.  
         [0037]     Decoding slices includes overhead prior to decoding the symbols. For example, in H.264, a large number of slices, such as two per macroblock row, are used. The macroblocks  120  from a slice can be predicted from as many as 16 reference pictures.  
         [0038]     Referring now to  FIG. 3 , there is illustrated a block diagram describing an exemplary video decoder system  300  for decoding video data in accordance with an embodiment of the present invention. The video decoder system  300  comprises an outer loop processor  305 , an inner loop processor  310 , a Context Adaptive Binary Arithmetic Code (CABAC) decoder  320 , and a symbol interpreter  325 .  
         [0039]     An encoded video bitstream is received in a code buffer  303 . The portions of the bitstream are provided to the outer loop processor  305 . Additionally, the portions of the bitstream that are CAVLC coded are also provided directly to the symbol interpreter  325 . The portions of the symbols that are CABAC coded are also provided to the CABAC decoder  320 . The CABAC decoder  320  converts the CABAC symbols to what are known as BINS and writes the BINs to a Bin Buffer that provides the BINS to the symbol interpreter  325 .  
         [0040]     The outer loop processor  305  is associated with an outer loop symbol interpreter  306  to interpret the symbols of the bitstream. Because decoding slices includes overhead prior to decoding, in H.264, a large number of slices, such as two per macroblock row, are used. For example, the macroblocks  120  from a slice can be predicted from as many as 16 reference pictures. Accordingly, the outer loop processor  305  parses the slices and performs the overhead functions. The overhead functions can include, for example but not limited to, generating and maintaining the reference lists for each slice, direct-mode table construction, implicit weighted-prediction table construction, memory management, and header parsing. According to certain embodiments of the present invention, the outer loop processor  305  prepares the slice into an internal slice structure wherein the inner loop  310  can decode the prediction errors for each of the macroblocks therein, without reference to any data outside the prepared slice structure. The slice structure can include the associated reference list, direct-mode tables, and implicit weighted prediction tables.  
         [0041]     The inner loop processor  310  manages the inverse transformer  330 , motion compensator  335 , pixel reconstructor  340 , the spatial predictor  345 , and the deblocker  350  to render pixel data from the slice structure.  
         [0042]     Referring now to  FIG. 4 , there is illustrated a block diagram describing an exemplary interface between the outer loop processor  305  and the inner loop processor  310 . The interface comprises a first queue  405  and a second queue  410 . The outer loop processor  305  places the elements onto the queue for the inner loop processor  310 . The elements can include a pointer to the slice structures in the memory. According to certain embodiment of the present invention, the elements can also include, for example, an indicator indicating, for example, whether the video data is H.264 or MPEG-2, and a channel context. Responsive to receiving the elements from the first queue  405 , the inner loop processor  305  decodes the slice structures. The inner loop processor  305  places the elements on the second queue  410 . The elements include an identifier identifying pictures, when the inner loop processor  310  has finished decoding all of the slices of the picture.  
         [0043]     Referring now to  FIG. 5 , there is illustrated a flow diagram describing decoding video data in accordance with an embodiment of the present invention. At  505 , the slice is received by the outer loop processor  305 . At  510 , the outer loop processor  305  performs the overhead processing for the slice. The overhead processing can include, for example, generating reference lists for the slice, direct-mode table construction, implicit weighted-prediction table construction, memory management, and header parsing. According to certain aspects of the present invention, the outer loop processor  305  generates a slice structure for the slice, wherein the prediction error for the slice can be generated from the slice structure without reference to additional data. At  515 , the inner loop processor  310  decodes the slice, while the outer loop processor performs the overhead processing for another slice. This,  515 , can be repeated for any number of slices.  
         [0044]     The H.264 specification provides for what is known as a decoded picture buffer structure. The decoded picture buffer structure provides a list of decoded pictures in display order. When a picture is finished decoding, the decoded picture buffer structure is updated and outputs an indicator indicating the next picture for display. According to H.264 specifications, the output indicator is removed from the decoded picture buffer list. The H.264 specification requires that this removal occur before the beginning of decoding of the next picture.  
         [0045]     To allows the outer loop processor  305  to process slices from pictures that are ahead of the pictures containing the slices processed by the inner loop processor  310 , an intermediate stage stores the outputted indicators.  
         [0046]     Referring now to  FIG. 6 , there is illustrated a block diagram describing exemplary data structures in accordance with an embodiment of the present invention. The decoded picture buffer structure  605  provides a list of decoded pictures in display order. When a picture is finished decoding, the outer loop processor  305  updates the decoded picture buffer structure. The decoded picture buffer  605  outputs an indicator  605 X indicating the next picture for display. According to H.264 specifications, the output indicator  605 X is removed from the decoded picture buffer list.  
         [0047]     The outputted indicator  605 X is stored in an intermediate structure  610 . The intermediate structure  610  stores the outputted indicators  605 X until the inner loop processor  310  finishes processing each of the slices in the picture indicated by indicator  605 X. As noted above, when the inner loop processor  310  finishes decoding all of the slices of a picture, the inner loop processor  310  notifies the outer loop processor  310 , via queue  410 . Responsive to receiving the foregoing notification, the outer loop processor  310  outputs the indicator  605 X from the intermediate structure  610 .  
         [0048]     The intermediate structure  610  stores the indicators in the order that they were released from the decoded picture buffer  605 , thus preserving the display order of the indicators.  
         [0049]     Referring now to  FIG. 7 , there is illustrated a flow diagram for providing indicators indicating pictures in a display order in accordance with an embodiment of the present invention. At  705 , the outer loop processor  305  finishes the overhead functions for each of the slices in a picture. At  710 , the outer loop processor  305  updates the decoded picture buffer structure  605 , causing the decoded picture buffer structure  605  to output an indicator  605 X indicating the next picture in the display order. At  715 , the intermediate structure  610  buffers the indicator  605 X. At  720 , the outer loop processor  305  receives a notification via queue  410  that the inner loop processor has finished processing all of the slices of the picture indicated by indicator  605 X. Responsive thereto, the outer loop processor  305  at  725  causes the intermediate structure  610  to output the indicator  605 X.  
         [0050]     While the present invention has been described with reference to certain embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the present invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the present invention without departing from its scope. Therefore, it is intended that the present invention not be limited to the particular embodiment disclosed, but that the present invention will include all embodiments falling within the scope of the appended claims.