Abstract:
Presented herein are system(s), method(s), and apparatus for effective error concealment in real-world video transmission environments. In one embodiment, there is described a method for concealing errors in video data. The method comprises detecting whether a slice group in a picture contains errors; and replacing the slice group in the picture with a slice group of a most recent intra-coded picture if the slice group contains errors.

Description:
FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT 
       [0001]    [Not Applicable] 
       MICROFICHE/COPYRIGHT REFERENCE 
       [0002]    [Not Applicable] 
       BACKGROUND OF THE INVENTION 
       [0003]    In systems such as video systems where video streams get transmitted from the encoding side to the decoding side, transmission errors such as random bit errors or packet loss, or storage errors such as disk defects, may cause damage to compressed video bit streams presented to a digital video decoder. Because of the nature of modern video compression techniques, such errors can render the decoded video output very objectionable or useless to the human observer. 
         [0004]    Further limitations and disadvantages of conventional and traditional approaches will become apparent to one of skill in the art, through comparison of such systems with some aspects of the present invention as set forth in the remainder of the present application with reference to the drawings. 
       BRIEF SUMMARY OF THE INVENTION 
       [0005]    Aspects of the present invention may be seen in system(s) and method(s) for effective error concealment in real-world video transmission environments, as illustrated or described in one or more of the following drawings and/or detailed description, as set forth more completely in the claims. 
         [0006]    These and other features and advantages of the present invention may be appreciated from a review of the following detailed description of the present invention, along with the accompanying figures in which like reference numerals refer to like parts throughout. 
     
    
     
       BRIEF DESCRIPTION OF SEVERAL VIEWS OF THE DRAWINGS 
         [0007]      FIG. 1  is a block diagram describing error concealment in accordance with an embodiment of the present invention. 
           [0008]      FIG. 2  is a flow diagram for decoding video data in accordance with an embodiment of the present invention. 
           [0009]      FIG. 3A-3C  are block diagrams describing an embodiment of the present invention in the context of H.264. 
           [0010]      FIG. 4  is a block diagram of an exemplary video decoder in accordance with an embodiment of the present invention. 
           [0011]      FIG. 5  is a flow diagram for decoding video data in accordance with an embodiment of the present invention. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0012]    Referring now to  FIG. 1 , there is illustrated a block diagram describing the error concealment of an exemplary picture  100  in accordance with an embodiment of the present invention. The picture  100  comprises a stream of symbols. The symbols together represent a compressed representation of a field/frame of a video sequence. The picture  100  can be received, either from a communication network or from a memory. 
         [0013]    A frame  105  can be compressed using a variety of techniques. For example, the frame  105  can be divided into pixel blocks  110 (x,y). Instead of representing each pixel in each pixel block  110 , the pixel block  110  can be compared to regions of other frames  105 . Since motion picture uses at least 24 frames/second, it is likely that frames that are contemporaneous with respect to each other have similar content that differs primarily by movement. Thus, for a given block  110 (x,y), a contemporaneous frame  105 ′ can be searched to find a similar appearing block  110 ′ (known as a reference block). It is noted that the reference block is not necessarily aligned with the blocks boundaries of frame  105 . The blocks  110  of the picture  100  can either not be predicted from another picture (known as an intra-picture), predicted from one picture, or predicted from two pictures. 
         [0014]    Accordingly, blocks  110 (x,y) can be represented as the pixel by prediction error E between their reference blocks  110 ′, and an identification of the similar appearing blocks  110 ′ (if they are predicted at all). The reference blocks are identified by what is known as a motion vector my that describes the spatial displacement between the block  110  and the similar appearing blocks  110 ′. The prediction error E subjected to mathematical transformations, such as a Discrete Cosine Transformation, or the Hadamard Transformation, resulting in coefficients, f 0  . . . f n  in another domain, such as frequency. The coefficients f 0  . . . f n  in the another domain can be quantized, resulting in quantized coefficients F 0  . . . F m . It is noted that quantization may result in truncating some of the coefficients to zero (particularly coefficients representing higher frequencies), resulting fewer quantized coefficients, m&lt;n. 
         [0015]    The my and the quantized coefficients F are symbols S 0  that are then losslessly encoded using a variable length code, such as a Context Adaptive Binary Arithmetic Coding (CABAC), or Context Adaptive Variable Length Coding (CAVLC). In a variable length code, fewer bits are assigned to more frequently occurring symbols while more bits are assigned to less frequency occurring symbols. However, in variable length coding, the frequency of symbol occurrence is predetermined prior to coding, and may not actually reflect the actual frequency of symbol occurrence in the present stream of data. In CABAC and CAVLC, the frequency of symbol occurrence is observed dynamically during coding, and codes can be changed based on dynamic changes in frequency of occurrence. A context model is maintained to monitor the actual frequency of symbols occurrence and change codes dynamically. 
         [0016]    As a result, in CABAC and CAVLC, decoding a code C i  is data depending on each of the preceding codes C 0 . . . C i−1  in order to regenerate the context model and code maps. However, if during transmission, bit errors occur that cannot be corrected by forward error correction, some of the symbols C 0  . . . C i−1 , e.g., C j  will be incorrect. As a result of this, the following symbols C j+1  . . . can be potentially decoded using an incorrect context model. This error can continue to propagate for all the remaining symbols C j+1  . . . . 
         [0017]    To block, for example, the foregoing error propagation, the symbols are coded in units  120 , where at the beginning of the unit, the context model is reset. The units  120  can comprise a group of blocks  110  in the picture  100 . Although the blocks  110  in the units are shown to be spatially contiguous, it is noted that the units do not necessarily have to be spatially contiguous. As a result, if bit errors corrupt the symbols of unit  120 ( i ), the decoding of the symbols of unit  120 ( i+ 1) are unaffected. 
         [0018]    However, although bit errors propagation confines errors in the decoding of the symbols to the unit  120 , prediction errors can propagate to other pictures  100 . If for example, a later picture has block that use blocks that lie at least partially in the corrupted unit, while the prediction error E and the motion vector mv of the block may be properly decoded, the reference block  105 ′ may be corrupted, or altogether unavailable. As a result, while symbols errors are confined to units  120 , prediction errors can propagate to other pictures. 
         [0019]    Accordingly, when a unit  120  of a picture  100  is detected to contain errors of be corrupted, the unit  120  is “filled in”. The unit  120  in the picture  100  is filled in with the co-located regions of the most recent intra-picture, irrespective of whether the picture  100  actually uses any blocks of the intra-picture as reference blocks. 
         [0020]    A special case occurs if the picture  100  is the first picture in a video sequence, there are no prior pictures to use. In the case where a video sequence follows a channel change, the “prior pictures” will be the last pictures of the previous channel. Using the last picture of the previous channel to fill in the corrupted unit  120 , would result in artifacts from the previous channel. This is particularly objectionable in appearance. Accordingly, in the case of a first intra-picture with a corrupted unit  120 , the corrupted unit  120  is blacked out. 
         [0021]    Referring now to  FIG. 2 , there is illustrated flow diagram describing a method for decoding video data in accordance with an embodiment of the present invention. At  200 , a picture  100  is received from either a communication channel or a memory. At  205 , symbols of the picture  100  are decoded for each unit  120 . If at  210 , any units  120  are found to be corrupted, a determination is made whether the picture  100  is a first intra-coded picture of a video sequence. If the picture  100  is a first intra-coded picture of a video sequence at  215 , the corrupted units  120  are blacked out  220 . Otherwise, the corrupted units  120  are filled in with the co-located regions in the most recent intra-picture at  225 . 
         [0022]    The present invention will now be described in the context of an exemplary video compression scheme, known as Advanced Video Coding (AVC), also known as MPEG-4, Part 10, and H.264. 
         [0023]    Referring now to  FIG. 3A , there is illustrated a block diagram describing encoding of video data in accordance with AVC. A video camera captures frames 300 from a field of view during time periods known as frame durations. The successive frames 300 form a video sequence. A frame 300 comprises two-dimensional grid(s) of pixels 300(x,y). 
         [0024]    For color video, each color component is associated with a two-dimensional grid of pixels. For example, a video can include a luma, chroma red, and chroma blue components. Accordingly, the luma, chroma red, and chroma blue components are associated with a two-dimensional grid of pixels 300Y(x,y), 300Cr(x,y), and 300Cb(x,y), respectively. When the grids of two dimensional pixels 300Y(x,y), 300Cr(x,y), and 300Cb(x,y) from the frame are overlayed on a display device 310, the result is a picture of the field of view at the frame duration that the frame was captured. 
         [0025]    The chroma red 300Cr(x,y) and chroma blue 300Cb(x,y) pixels are overlayed the luma pixels in each even-numbered column 300Y(x, 2y) between each even, one-half a pixel below each even-numbered line 300Y(2x, y). In other words, the chroma red and chroma blue pixels 300Cr(x,y) and 300Cb(x,y) are overlayed pixels 300Y(2x+½, 2y). 
         [0026]    Four 8×8 blocks of luma pixels 315Y(x,y), and co-located 8×8 blocks of chroma red pixels 315Cr(x,y) and chroma blue pixels 315Cb(x,y) are coded together forming what is collectively known as a macroblock 320. 
         [0027]    The H.264 standard encodes video on a frame by frame basis, and encodes frames on a macroblock by macroblock basis. H.264 specifies the use of temporal prediction, transformation, and lossless entropy coding to compress the macroblocks 320, among other compression techniques. 
         [0028]    Referring now to  FIG. 3B , there is illustrated a block diagram describing temporally encoded macroblocks 320. Each macroblock 320 is compared to the pixels of other frames or fields for a similar block of pixels P. A macroblock 320 is encoded as a prediction error E between the block of pixels P and the macroblock 320. 
         [0029]    The similar block of pixels is known as the prediction pixels P. The difference between the macroblock and the prediction pixels P is known as the prediction error E. The prediction error E is calculated and encoded, along with an identification of the prediction pixels P. The prediction pixels P are identified by motion vectors MV. Motion vectors MV describe the spatial displacement between the macroblock 320 and the prediction pixels P. 
         [0030]    The macroblock 320 can be predicted from blocks of pixels P in either one frame, two frames, or not predicted from blocks of pixels P from another frame. In bi-directional coding, the partition 330 can be predicted from two weighted blocks of pixels, P 0  and P 1 . Accordingly, a prediction error E is calculated as the difference between the weighted average of the prediction blocks w 0 P 0 +w 1 P 1 . The prediction error E, an identification of the prediction blocks P 0 , P 1  are encoded. The prediction blocks P 0  and P 1  are identified by motion vectors MV. 
         [0031]    Referring now to  FIG. 3C , there is illustrated a block diagram describing the encoding of the prediction error E. The prediction error E is also two-dimensional grid of pixel values for the luma Y, chroma red Cr, and chroma blue Cb components with the same dimensions as the macroblock 320. 
         [0032]    A transformation transforms the prediction error E to the frequency domain, thereby resulting in corresponding sets of frequency coefficients f 0 . . . f n . The sets of frequency coefficients are then quantized and scanned, resulting in sets of quantized frequency coefficients, F 0  . . . F m . The macroblocks 320 are represented by a prediction error E that is encoded as sets of quantized frequency coefficients F 0  . . . F n . The macroblock 320 is represented by side information, such as prediction mode indicators, and identification of prediction blocks. 
         [0033]    Context Adaptive Variable Length Coding (CAVLC) is used to encode the frequency coefficients F 0  . . . F n . Content Adaptive Binary Arithmetic Coding (CABAC) is used to encode the side information. In CABAC coding, the information is first transformed to what is known as CABAC Binary Symbols (bins). The bins are then transformed to CABAC symbols. 
         [0034]    The mv and the quantized coefficients F are symbols that are then losslessly encoded using a variable length code, such as a Context Adaptive Binary Arithmetic Coding (CABAC), or Context Adaptive Variable Length Coding (CAVLC), thereby resulting in codes C 0  . . . . In a variable length code, fewer bits are assigned to more frequently occurring symbols while more bits are assigned to less frequency occurring symbols. However, in variable length coding, the frequency of symbol occurrence is predetermined prior to coding, and may not actually reflect the actual frequency of symbol occurrence in the present stream of data. In CABAC and CAVLC, the frequency of symbol occurrence is observed dynamically during coding, and codes can be changed based on dynamic changes in frequency of occurrence. A context model is maintained to monitor the actual frequency of symbols occurrence and change codes dynamically. 
         [0035]    As a result, in CABAC and CAVLC, coding of a given symbol is data dependent on the context model and code maps resulting each of the preceding codes symbols. However, if during transmission, bit errors occur in a code, e.g., C j  that cannot be corrected by forward error correction, the context model and code maps may not be regenerated correctly. As a result of this, the following codes can be potentially decoded using an incorrect context model. This error can continue to propagate for all the remaining symbols C j+1  . . . . 
         [0036]    To block the foregoing error propagation, the macroblocks 320 are grouped into units known as slice groups 325, wherein at the beginning of the unit, the context model is reset. Although the macroblocks 320 in the slice groups 325 are shown to be spatially contiguous, it is noted that the units do not necessarily have to be spatially contiguous. If bit errors corrupt the symbols of slice group 325(i) the decoding of the symbols of unit 320(i+1) are unaffected. 
         [0037]    However, although slice groups confine bit errors in the decoding of the symbols, e.g., the frequency coefficients, motion vectors, and other side information, to the slice group 325, prediction errors can propagate to other pictures 300. If for example, a later picture has block that use blocks P that lie at least partially in the corrupted unit, while the prediction error E and the motion vector mv of the block may be properly decoded, the reference block P may be corrupted, or altogether unavailable. As a result, while symbols errors are confined to slice groups 325, prediction errors can propagate to other pictures. 
         [0038]    The video sequence is encoded as the frame forming it. The encoded video sequence is known as a video elementary stream. The video elementary stream is a bitstream that can be transmitted over a communication network to a decoder. Transmission of the bitstream instead of the video sequence consumes substantially less bandwidth. 
         [0039]    Referring now to  FIG. 4 , there is illustrated a block diagram describing an exemplary video decoder  600  in accordance with an embodiment of the present invention. The video decoder  600  includes a code buffer  605  for receiving a video elementary stream. The code buffer  605  can be a portion of a memory system, such as a dynamic random access memory (DRAM). A symbol interpreter  615  in conjunction with a context memory  610  decodes the CABAC and CAVLC symbols from the bitstream. The context memory  610  can be another portion of the same memory system as the code buffer  605 , or a portion of another memory system. 
         [0040]    The symbol interpreter  615  includes a CAVLC decoder  615 V and a CABAC decoder  615 B. The CAVLC decoder  615 V decodes CAVLC symbols, resulting in the sets  140  of quantized frequency coefficients F 0  . . . F n . The CABAC decoder  615 B decodes the CABAC symbols resulting in the side information. 
         [0041]    The symbol interpreter  615  provides the sets of scanned quantized frequency coefficients F 0  . . . F n  to an inverse scanner, quantizer, and transformer (ISQDCT)  625 . Depending on the prediction mode for the macroblock 320 associated with the scanned quantized frequency coefficients F 0  . . . F n , the symbol interpreter  615  provides the side information to either a spatial predictor  620  (if spatial prediction) or a motion compensator  630  (if temporal prediction). 
         [0042]    The ISQDCT  625  constructs the prediction error E. The spatial predictor  620  generates the prediction pixels P for spatially predicted macroblocks while the motion compensator  630  generates the prediction pixels P, or P 0 , P 1  for temporally predicted macroblocks. The motion compensator  630  retrieves the prediction pixels P, or P 0 , P 1  from picture buffers 650 that store previously frames 300. 
         [0043]    A pixel reconstructor  635  receives the prediction error E from the ISQDCT  625 , and the prediction pixels from either the motion compensator  630  or spatial predictor  620 . The pixel reconstructor  635  reconstructs the macroblock 320 from the foregoing information and provides the macroblock 320 to a deblocker  640 . The deblocker  640  smoothes pixels at the edge of the macroblock  120  to prevent the appearance of blocking. The deblocker  640  writes the decoded macroblock 320 to the picture buffer  650 . The picture buffer  650  can be a portion of DRAM. 
         [0044]    A display engine  645  provides the frames  100  from the picture buffer  650  to a display device. The symbol interpreter  615 , the ISQDCT  625 , spatial predictor  620 , motion compensatory  630 , pixel reconstructor  635 , and display engine  645  can be hardware accelerators under the control of a central processing unit (CPU). 
         [0045]    A compressed video bit stream of codes C 0  . . . may be input to the symbol interpreter  615  to extract coded information such as, for example, sequence header, picture header, macro block coding mode, motion vectors, and prediction residual coefficients. The symbol interpreter  615  determines whether a particular slice group 325 has bit errors or is otherwise corrupted. 
         [0046]    Accordingly, when the symbol interpreter  615  detects a slice group 325 contains errors or is corrupted, the unit 325 is “filled in”. The unit 325 is filled in with the co-located regions of the most recent intra-picture, irrespective of whether the picture actually uses any blocks of the intra-picture as reference blocks. 
         [0047]    A special case occurs if the picture 300 is the first picture in a video sequence, there are no prior pictures to use. In the case where a video sequence follows a channel change, the “prior pictures” will be the last pictures of the previous channel. Using the last picture of the previous channel to fill in the slice group 325, would result in artifacts from the previous channel. This is particularly objectionable in appearance. Accordingly, in the case of a first intra-picture with a slice group 325, the slice group 325 is blacked out. 
         [0048]    Referring now to  FIG. 5 , there is illustrated flow diagram describing a method for decoding video data in accordance with an embodiment of the present invention. At  200 , a frame 300 is received from either a communication channel or a memory. At  705 , symbols of the frame 300 are entropy decoded for each slice group 325. If at  710 , slice group 325 are found to be corrupted, a determination is made whether the frame 300 is a first intra-coded picture of a video sequence. If the frame 300 is a first intra-coded picture of a video sequence at  715 , the slice group 325 are blacked out at  717 . Otherwise, the slice group 325 are filled in with the co-located regions in the most recent intra-picture at  720 . 
         [0049]    The present invention may be realized in hardware, software, firmware and/or a combination thereof. The present invention may be realized in a centralized fashion in at least one computer system, or in a distributed fashion where different elements are spread across several interconnected computer systems. Any kind of computer system or other apparatus adapted for carrying out the methods described herein may be suitable. A typical combination of hardware and software may be a general-purpose computer system with a computer program that, when being loaded and executed, controls the computer system to carry out the methods described herein. 
         [0050]    The present invention may also be embedded in a computer program product comprising all of the features enabling implementation of the methods described herein which when loaded in a computer system is adapted to carry out these methods. Computer program in the present context means any expression, in any language, code or notation, of a set of instructions intended to cause a system having information processing capability to perform a particular function either directly or after either or both of the following: a) conversion to another language, code or notation; and b) reproduction in a different material form. 
         [0051]    While the present invention has been described with reference to certain embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the present invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the present invention without departing from its scope. Therefore, it is intended that the present invention not be limited to the particular embodiment disclosed, but that the present invention will include all embodiments falling within the scope of the appended claims.