Abstract:
A method of an efficient adaptive mode selection for H.264/AVC-coded video delivery in burst-packet-loss networks to generate the image data of the missing macroblocks in the decoded current frame by using the information related to the spatial redundancy in the same frame and the temporal redundancy in the inter frames is disclosed. The method first employs the Intra High-Speed Spatial Error Concealment (SEC) method for the initial frame. For the succeeding inter frames, the Temporal Error Concealment (TEC) method is used when the Adjacent External Boundary Matching Error (AEBME) of the surrounding macroblocks is not more than the dynamic threshold (DT α ). The Intra High-Speed SEC is used when AEBME is more than (DT α ) and the surrounding macroblocks are all coded intra-method, otherwise the task proceeds by employing Normal SEC.

Description:
BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The present invention relates generally to a method of an efficient adaptive mode selection for H.264/AVC-coded video delivery in burst-packet-loss networks, and a hybrid error concealment method, and more particularly, to a method of using spatial redundancy information and temporal redundancy information for concealing a spatial image error of an encoded image frame caused by missing macroblocks. 
     2. The Prior Arts 
     Image transmission service including video meeting, website browsing, image file transmission, is a very important service provided by communication and IT enterprises. Typically, an original image file desired to be transmitted is often too large to be effectively transmitted over the internet, or often occupies too much memory space. As such, an image file is often performed with a coding process with a high compression ratio and a low loss ratio before transmission for reducing the size of the image content. Such a coding process may be selected from TIFF, JPEG, MJEP, MPEG, H.264/AVC, in accordance with the demands of the system for the image compression ratio, the loss ratio, so as to obtain an optimal algorithm for static images or continuous images. 
     However, when being transmitted, the encoded image file is often interfered by the transmission interface or the electrical system of the receiving end. Such interferences usually cause damages or errors of the image file content, so that the receiving end becomes incapable of decoding and recovering the encoded image file back to original by a predetermined corresponding decoding method. 
     In accordance with psychology of vision, an individual edge would not affect visional recognition. However, such an individual edge may be mutually affected with edges adjacent thereto so as to generate a composition effect. In this concern, human vision is more sensitive to edge distortions. Generally, edge distortions include blurred edges and newly created false edges. As such, it is very important to maintain original edges and avoid the creation of false edges when executing an image process. 
     A method of recovering or obtaining image content of a missing macroblock according to image content of correct macroblocks of a same frame of the missing macroblock is generally called the SEC method. A further method proposed as another solution is called the temporal error concealment (TEC) method. According to the TEC method, a motion vector of a missing macroblock can be achieved by referring to a correct image data of a previous frame. Then, a reference frame is moved according to the motion vector, and therefore the image data of the missing macroblock can be thus obtained. In such a way, the negative affection to the entire image frame caused by the missing macroblock can be concealed. 
     The conventional SEC method is schematically introduced firstly as below. 
     There are many algorithms of conventional technologies have been proposed for spatial error concealment, such as bilinear interpolation (BI), directional interpolation (DI), and best neighborhood matching (BNM). The principles of the aforementioned algorithms can be learnt in more details by referring to related references, such as: P. Salama, N. B. Shroff, and E. J. Delp, “Error concealment in encoded video streams,” in Signal Recovery Techniques for Image and Video Compression and Transmission, A. K. Katsaggelos and N. P. Galatsanos, Eds. Norwell, Mass.: Kluwer, ch. 7, 1998; H. Sun and W. Kwok, “Concealment of damaged block transform coded images using projection onto convex set,” IEEE Trans. Image Processing, vol. 4, pp. 470-477, April 1995; and Z. Wang, Y. Yu, and D. Zhang, “Best neighborhood matching: An information loss restoration technique for block-based image coding systems,” IEEE Trans. Image Process., vol. 7, no. 7, pp. 1056-1061, July 1998. They are to be briefly illustrated herebelow for better understanding of the present invention. 
       FIG. 1  is a schematic diagram illustrating a BI method according to a conventional technology. Referring to  FIG. 1 , a missing macroblock  20  having no correct image content is surrounded by edge reference pixels  21  containing correct image content. A missing pixel P(x, y) in the missing macroblock  20  positioned at the coordinates (x, y) of BI is then interpolated by the formula (1), according to the correct image content of the four edges in both horizontal direction and vertical direction: 
                       p   ⁡     (     x   ,   y     )       =         p   ⁢           ⁢   1   ×   d   ⁢           ⁢   2     +     p   ⁢           ⁢   2   ×   d   ⁢           ⁢   1     +     p   ⁢           ⁢   3   ×   d   ⁢           ⁢   4     +     p   ⁢           ⁢   4   ×   d   ⁢           ⁢   3           d   ⁢           ⁢   1     +     d   ⁢           ⁢   2     +     d   ⁢           ⁢   3     +     d   ⁢           ⁢   4           ,           (   1   )               
where d 1 , d 2 , d 3 , d 4  are relative distances from the edge pixels P 1 , P 2 , P 3 , P 4  to the missing pixel P(x, y), respectively.
 
       FIG. 2  is a schematic diagram illustrating a DI method according to a conventional technology. As shown in  FIG. 2 , the edge pixels contain correct image content, while the macroblock surrounded by the edge pixels is a missing macroblock containing incorrect image content. The pixel P(x, y) in the missing macroblock positioned at the coordinates (x, y) is then interpolated by the formula (2), according to the correct image content of two edges along a specific direction: 
                       p   ⁢     (     x   ,   y     )       =         p   ⁢           ⁢   1   ×   d   ⁢           ⁢   1     +     p   ⁢           ⁢   2   ×   d   ⁢           ⁢   2           d   ⁢           ⁢   1     +     d   ⁢           ⁢   2           ,           (   2   )               
where d 1 , d 2  are relative distances from the edge pixels P 1 , P 2 , to the missing pixel P(x, y), respectively.
 
       FIG. 3  is a directional schematic diagram illustrating a DI method according to a conventional technology. As shown in  FIG. 3 , in the DI method, the edge direction is classified into 8 directions, in which DI(0°) represents a direction of 0°, DI(22.5°) represents a direction of 22.5°, . . . and so forth. P 1  and P 2  of  FIG. 2  are each of one of the 8 directions. 
     The conventional DI method usually employs a direction filter of an edge direction detection technique for determining direction of each pixel contained in the macroblocks surrounding the missing macroblock. Sobel operator or Prewitt operator are often used. Sobel operator, which is also known as a Sobel filter, is to be exemplified for illustration below taking a 3×3 Sobel operator as an example: 
                       S   x     =     [           -   1         0       1             -   2         0       2             -   1         0       1         ]       ⁢     
     ⁢       S   y     =     [           -   1           -   2           -   1             0       0       0           1       2       1         ]               (   3   )               
where Sx and Sy detect edge characteristics at X direction and Y direction, respectively, and obtain directional gradients at X direction and Y direction of equation (4) (G x  and G y ), respectively.
 
 G   X   =X   i+1,j−1   −X   i−1,j−1 +2 X   i+1,j −2 X   i−1,j   +X   i+1,j+1   −X   i−1,j+1  
 
 G   y   =X   i−1,j+1   −X   i−1,j−1 +2 X   i,j+1 −2 X   i,j−1   +X   i+1,j+1   −X   i−1,j−1   (4).
 
     A directional gradient (Grand) and a directional angle (θ) can be further calculated with equations (5) and (6).
 
Grand=√{square root over ( G   x   2   +G   y   2 )}  (5)
 
θ=tan −1 ( Gx/Gy )  (6)
 
where the directional gradient (Grand) is a quantity factor for evaluating the strength of the directivity. In other words, a directional angle (θ) having a maximum directional gradient (Grand) is selected serving as a direction reference for the missing macroblock.
 
     If all of the directional gradients (Grand) are too low, or even lower than a threshold, the missing macroblock is determined as having no directivity, and not adapted for SEC by DI method. As such, other methods, (e.g., BI method), are demanded for processing those missing macroblocks having no directivity. 
       FIG. 4  is a schematic diagram illustrating a BNM method according to a conventional technology. Referring to  FIG. 4 , a missing macroblock  20  of frame  40  is surrounded by a plurality of reference macroblocks  22 , and a best matching macroblock  32  is surrounded by a plurality of target macroblocks  30  in a searching area  42 . The BNM method is for finding out the best matching macroblock  32  for replacing the missing macroblock  20 . The BNM method includes the steps of: (1) taking N-pixel-wide boundary surrounding the missing macroblock as a searching image; (2) identifying best matching macroblocks nearest to the missing macroblock; and (3) replacing the corresponding missing macroblock (i.e., the macroblock surrounded by the best matching macroblocks) with the identified best matching macroblock. 
     Features of the BI method, ID method, and BNM method are to be discussed below in further details. 
     The DI method has the advantages of edge protection. For example, if there is only one strong edge near the missing macroblock, the original strong edge can be preserved and avoided from becoming blur. However, when there are several edges near the missing macroblock, the DI method may generate several false edges. Unfortunately, human vision is usually very sensitive to blur false edges. Correspondingly, the BI method is adapted to generate new blur false edges. However, if there is only one strong edge near the missing macroblock, the BI method may blur the original strong edge because the edges of the missing macroblock are interpolation processed and mixed with the adjacent pixels. Further, with respect to missing macroblocks having patterns of higher complexity, it is often difficult to obtain necessary effective reference blocks. Although the BNM method can recover the complex missing macroblock, it may cause edge discontinuity. It may also cause higher computation complexity than BI and DI methods, and thus is featured with a lower overall efficiency. 
     Further, a multi-direction interpolation (MDI) method has been proposed for preserving multi-directional edges. The MDI method is as discussed below. 
       FIG. 5  is a schematic diagram illustrating reference blocks of the MDI method. Referring to  FIG. 5 , the MDI selects a plurality of N-pixel-wide boundary reference blocks  50  as a reference block group surrounding a missing macroblock  20 . 
     First, a Sobel operator is used to calculate a directional strength (i.e., directional gradient) and a directional angle of individual pixels of the 4×4 reference block as of the DI method. And then, a directional entropy (DE) H d  is calculated according to equation (7) for representing a directional factor.
 
 H   d   =−Σp ( d   x )log 2    p ( d   x )  (7),
 
where d x  represents a directional angle of a specific pixel of the reference block, while p(dx) represents a directional probability density function related to the directional angle.
 
     A greater DE indicates a smaller directivity (i.e., having no specific edge direction), while a maximum DE value (i.e., 1) indicates absolutely evenly distributed edge directions. 
     Then, a direction threshold β is calculated according to equation (8), for obtaining an overall directional index of the 4×4 reference block. Thus the overall directional index can be accorded to determine whether to adopt the directional angle of the reference block as the directional angle of the missing macroblock group. 
                     β   =       -     ∑           ⁢       p   ⁡     (     d   x     )       ⁢     log   2     ⁢     p   ⁡     (     d   x     )               Max   ⁡     (     H   d     )           ,           (   8   )               
where Max(H d ) represents a maximum value of H d .
 
     Only when the direction threshold β is smaller than a specific threshold determining value, the directional angle is adopted, and when the direction threshold β is greater than the specific threshold determining value, the missing macroblock is determined as having no specific directivity. According to an aspect of the embodiment, the direction threshold β is preferably defined between 0.7 and 0.9. as such, the reference block group can be classified into four categories according to the direction threshold β, including no-directional background area, single-directional edge area, multi-directional edge area, and complex multi-directional edge area. With respect to reference blocks of different categories, different interpolation methods or matching methods are employed for recovering the missing macroblock. Specifically, the BI method is applied to the no-directional background area; the DI method is applied to the single-directional edge area; the BNM method is applied to the multi-directional edge area; and the MDI method is applied to the complex multi-directional edge area. 
     When the 4×4 reference block exhibits more than two strong edge directions, the missing macroblock is considered as a multi-directional edge area. In this case, the missing macroblock is recovered by calculating missing pixels of the missing macroblock by interpolation according to the directivity of the 4×4 reference block. According to an aspect of the embodiment, the MDI method is a 3-directional weighted interpolation method, compensating along a horizontal direction, a vertical direction, and a 45° direction.  FIG. 6  is a schematic diagram illustrating a 3-directional weighting interpolation of the MDI method according to a conventional technology. Referring to  FIG. 6 , the missing pixels P are recovered according to the two horizontal pixels PH 1  and PH 2  at the horizontal direction, the two vertical pixels PV 1  and PV 2  at the vertical direction, and the two diagonal pixels PD 1  and PD 2  at the diagonal direction. If the missing pixels P do not have a corresponding strong edge direction, the missing macroblock can be interpolated along the same direction corresponding to that of the nearest missing pixel. In this concern, the MDI method is similar to the DI method. 
     When most 4×4 reference blocks exhibit more than two strong edge directions, while the direction threshold β is greater than the threshold determining value, the missing macroblock is classified as a complex multi-directional edge area. If the missing macroblock includes complex content, the BNM method may be an ideal one for reconstructing the block to achieve an optimal image quality. However, the BNM method also requires a very large amount of calculation and a very long calculation time. On another hand, although requiring for a smaller calculation amount, the BI method is adapted for maintaining to display an ordinary image quality only. 
     All above-discussed methods are categorized as ordinary conventional SEC methods. Below, a fast MDI method is to be specifically introduced in more details. 
     Generally, the MDI method often spends 40% to 90% of calculation time in searching for possible directional angle and directional gradient. However, related directional data of the missing macroblock of the transmitted image would have been recorded during the original coding process. As such, the recorded data contained in the original coding can be used for reducing the calculation amount and saving the calculation time. In H.264/AVC coding, there is a correlation between intro-predication methods and edge direction. For example, an eight-directional 4×4 prediction method and a three-directional 16×16 prediction method can be used for pre-estimating for the edge direction of real images. Intra prediction information of the H.264/AVC contains data related to the original macroblock direction. As such, the fast MDI method of the present invention employs another fast estimation method, for determining whether to use the directional data of the missing macroblock contained in the original coding, so as to execute a direct interpolation. In such a way, it can reduce the complexity of the process of edge direction detection by 70% to 95% on average, and thus drastically improving the decoding efficiency. 
       FIG. 7  is a flow chart illustrating a fast classification of a fast SEC method according to a conventional technology. Referring to  FIG. 7 , missing macroblocks are classified into two categories, flat region  53  and compound region  54 , according to the intra prediction method information  51  of the original coding. Missing macroblocks in the flat region  53  are to be compensation processed by DI or BI method according to the intra prediction method information, while missing macroblocks in the compound region  54  are to be compensation processed by the aforementioned method. 
     With respect to the six neighbor blocks of each missing macroblock (i.e., top, top-right, left, right, bottom, and bottom-left), there are at least five adjacent macroblocks chosen via 16×16 prediction method. It does not need to use any Sobel operator to detect the edge direction for a smooth region of an image. On the contrary, each pixel of the missing macroblocks can be interpolated along a single direction (e.g., vertical, horizontal, or diagonal) or bi-direction (vertical-horizontal) determined by majority decision based on the intra 16×16 prediction method including method 0 (vertical), method 1 (horizontal), method 3 (plane), and method 2(DC), of corresponding surrounding macroblocks. 
     As to the compound region, in a 16×16 and 4×4 mixing prediction method of the reference macroblocks, four-pixel-wide boundary is disjoined into 4×4-pixel blocks as shown in  FIG. 8 . For the purpose of ascertaining dominant edges of each 4×4-pixel block, the intra method information and a small amount of the edge direction filters (i.e., Sobel filters) are employed). The calculation order of the Sobel filter to estimate the directional angles and directional gradients of an inner layer of pixels, p 1 , p 2 , p 3 , and p 4 , is depicted in  FIG. 8 . The calculation is stopped if a desired block is obtained. For each 4×4-pixel block, edge detection begins from p 1  and p 2 . If at least one of p 1  and p 2  exhibits a strong edge in the same direction of the intra method information, the block is determined as the desired block. If none of p 1  and p 2  exhibits a strong edge, p 3  and p 4  are further calculated. As to the intra DC prediction method, the 4×4-pixel block is directly regarded as having no strong edge, and thus does not need a Sobel filter. The final stage is similar as the foregoing embodiment of SEC method of the present invention, in which the missing macroblocks can be classified into four categories, including: no-directional background area  55 , single-directional edge area  56 , multi-directional edge area  57 , and complex multi-directional edge area  58 . Different categories are adapted for different interpolation methods. Specifically, the BI method is applied to the no-directional background area; the DI method is applied to the single-directional edge area; the BNM method is applied to the multi-directional edge area; and the MDI method is applied to the complex multi-directional edge area. 
     All of the above-discussed methods are called conventional intra high-speed SEC methods. 
     And now, the conventional TEC methods are to be discussed herebelow. 
     In conventional TEC methods, a zero motion vector, neighboring motion vectors (motion vectors), an available motion vector of a spatially corresponding macroblock in a previous frame, or a median (or an average) of spatially adjacent motion vectors can be used. A TEC method called motion field interpolation method is recently proposed. According to the motion field interpolation method, a best motion vector is determined from the above candidate motion vectors by the boundary matching algorithm (BMA), which calculates the minimum sum of absolute differences (SAD) along the one-pixel-wide boundary between the missing macroblock and its candidate replacements in the H.264 JM decoder. 
     However, the conventional technology usually calculates an optimal motion vector by complex computation, which requires a large amount of computation and consumes much time, and negatively affects the overall decoding efficiency. 
     It is assumed that all missing macroblocks have been accurately detected, and all boundary blocks surrounding the missing macroblock contain correct image data including motion vectors. Referring to  FIG. 9 , there is shown a flow chart illustrating a conventional TEC method. The flow chart includes steps S 100  through S 180 , to be discussed in more details herebelow. 
     First, at step S 100 , an optimal regression (OR) plane is set up according to motion vectors of the boundary blocks. An N-pixel-wide block group surrounding the missing macroblock includes a plurality of N×N pixel-wide blocks, in which N is a positive integer. As shown in the drawing, the embodiment is exemplified with N=4 for illustration. However, in other embodiments, N can be any other positive integers other than 4. 
       FIG. 10  is a schematic diagram illustrating boundary blocks according to a conventional technology. Referring to  FIG. 10 , it shows a missing macroblock  20  including 16 missing blocks, i.e., DB 1 , DB 2 , . . . , DB 16 . The missing blocks are surrounded by 16 boundary blocks, i.e., BB 1 , BB 2 , . . . , BB 16 . Further, each of the boundary blocks BB 1 , BB 2 , . . . , BB 16 , has a corresponding motion vectors, MV 1  through MV 16 , respectively, for representing a temporal correlation with a previous frame. 
     For convenience of illustration, a coordinate system is set up in  FIG. 10 , in which a center of the missing macroblock  20  is set as an origin of coordinate, i.e., P 0  (0, 0), and a pixel is defined as a unit of length. In this case, coordinates of the boundary blocks BB 1  through BB 16  are respectively:
 
 BB 1( x   1   , y   1 )=(−6, 10)
 
 BB 2( x   2   , y   2 )=(−2, 10)
 
 BB 3( x   3   , y   3 )=(2, 10)
 
 BB 4( x   4   , y   4 )=(6, 10)
 
 BB 5( x   5   , y   5 )=(10, 6)
 
 BB 6( x   6   , y   6 )=(10, 2)
 
 BB 7( x   7   , y   7 )=(10, −2)
 
 BB 8( x   8   , y   8 )=(10, −6)
 
 BB 9( x   9   , y   9 )=(6, −10)
 
 BB 10( x   10   , y   10 )=(2, −10)
 
 BB 11( x   11   , y   11 )=(−2, −10)
 
 BB 12( x   12   , y   12 )=(−6, −10)
 
 BB 13( x   13   , y   13 )=(−10, −6)
 
 BB 14( x   14   , y   14 )=(−10, −2)
 
 BB 15( x   15   , y   15 )=(−10, 2)
 
 BB 16( x   16   , y   16 )=(−10, 6);
 
and the motion vectors thereof are marked at centers of the corresponding boundary blocks.
 
     The OR plane is constructed by a multiple polynomial regression model of equation (9): Z(x,y)=c+a 1 x−a 2 x 2 +b 1 y+b 2 y 2 , where Z represents the motion vectors, and a 1 , a 2 , b 1 , b 2 , c are coefficients to be determined. 
     After regression analysis, an equation (10) as following can be obtained. 
     
       
         
           
             
               
                 
                   
                     
                       Z 
                       ⁡ 
                       
                         ( 
                         
                           x 
                           , 
                           y 
                         
                         ) 
                       
                     
                     = 
                     
                       [ 
                       
                         
                           1 
                           106496 
                         
                         ⁢ 
                         
                           
                             ∑ 
                             
                               i 
                               = 
                               1 
                             
                             16 
                           
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             
                               MV 
                               i 
                             
                             · 
                             
                               ( 
                               
                                 
                                   27 
                                   ⁢ 
                                   
                                       
                                   
                                   ⁢ 
                                   
                                     x 
                                     i 
                                     2 
                                   
                                 
                                 + 
                                 
                                   25 
                                   ⁢ 
                                   
                                       
                                   
                                   ⁢ 
                                   
                                     y 
                                     i 
                                     2 
                                   
                                 
                                 - 
                                 3120 
                               
                               ) 
                             
                           
                         
                       
                       ] 
                     
                   
                   ⁢ 
                   
                     
 
                   
                   ⁢ 
                   
                     
                       x 
                       2 
                     
                     + 
                     
                         
                       
                         
                           
                             [ 
                             
                               
                                 1 
                                 960 
                               
                               ⁢ 
                               
                                 
                                   ∑ 
                                   
                                     i 
                                     = 
                                     1 
                                   
                                   16 
                                 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 
                                   
                                     MV 
                                     i 
                                   
                                   · 
                                   
                                     x 
                                     i 
                                   
                                 
                               
                             
                             ] 
                           
                           ⁢ 
                           x 
                         
                         + 
                         
                           
                             [ 
                             
                               
                                 1 
                                 106496 
                               
                               ⁢ 
                               
                                 
                                   ∑ 
                                   
                                     i 
                                     = 
                                     1 
                                   
                                   16 
                                 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 
                                   
                                     MV 
                                     i 
                                   
                                   · 
                                   
                                     ( 
                                     
                                       
                                         25 
                                         ⁢ 
                                         
                                             
                                         
                                         ⁢ 
                                         
                                           x 
                                           i 
                                           2 
                                         
                                       
                                       + 
                                       
                                         27 
                                         ⁢ 
                                         
                                             
                                         
                                         ⁢ 
                                         
                                           y 
                                           i 
                                           2 
                                         
                                       
                                       - 
                                       3120 
                                     
                                     ) 
                                   
                                 
                               
                             
                             ] 
                           
                           ⁢ 
                           
                             y 
                             2 
                           
                         
                         + 
                         
                           
                             [ 
                             
                                 
                             
                             ⁢ 
                             
                               
                                 1 
                                 960 
                               
                               ⁢ 
                               
                                 
                                   ∑ 
                                   
                                     i 
                                     = 
                                     1 
                                   
                                   16 
                                 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 
                                   
                                     MV 
                                     i 
                                   
                                   · 
                                   
                                     y 
                                     i 
                                   
                                 
                               
                             
                             ] 
                           
                           ⁢ 
                           y 
                         
                         - 
                         
                           
                             1 
                             512 
                           
                           ⁢ 
                           
                             
                               ∑ 
                               
                                 i 
                                 = 
                                 1 
                               
                               16 
                             
                             ⁢ 
                             
                                 
                             
                             ⁢ 
                             
                               
                                 MV 
                                 i 
                               
                               · 
                               
                                 ( 
                                 
                                   
                                     15 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     
                                       x 
                                       i 
                                       2 
                                     
                                   
                                   + 
                                   
                                     15 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     
                                       y 
                                       i 
                                       2 
                                     
                                   
                                   - 
                                   1832 
                                 
                                 ) 
                               
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   10 
                   ) 
                 
               
             
           
         
       
     
     A curved plane defined by the equation (10) is the desired OR plane.  FIG. 11  is a schematic diagram illustrating boundary blocks according to a conventional technology. As shown in  FIG. 11 , at step S 110 , corresponding predicted spatial motion vectors of the missing blocks DB 1  through DB 16  can be obtained by the equation (10). 
     At step S 120 , temporal motion vectors are set up according to a previous image frame, i.e., taking motion vectors of the missing macroblock in the previous image frame at corresponding positions as high temporal correlated motion vectors. 
     At step S 130 , candidate motion vectors are constructed according to the boundary matching algorithm (BMA). As discussed above, the spatial motion vectors are obtained at step S 110 , the temporal VMs are obtained at step S 120 , and now at the current step, a quantitative difference between each spatial motion vector and each temporal motion vector can be estimated by the adjacent external boundary matching errors (AEBME). The motion vector having a least AEBME is selected as the candidate motion vector. 
       FIG. 12  is a schematic diagram illustrating a boundary matching algorithm (BMA) and related blocks according to a conventional technology. Referring to  FIG. 12 , the 16 missing blocks of the missing macroblock  20  are renamed with A 00 , A 01 , A 110 , A 11 , B 00 , B 01 , B 10 , B 11 , C 00 , C 01 , C 10 , C 11 , D 00 , D 01 , D 10 , D 11 , for the convenience of illustration. 
     The missing blocks A 11 , B 11 , C 11 , D 11  are located at an inside layer of the missing macroblock  20 , and are not neighboring to the boundary blocks having correct image data. The rest missing blocks locate at an outside layer of the missing macroblock  20 , and are neighboring to the boundary blocks having correct image data. In the current embodiment, only the missing blocks A 00 , A 01 , A 10 , A 11  are illustrated, and the other missing blocks can be learnt by referring to the illustration of the missing blocks A 00 , A 01 , A 10 , A 11 , according to the symmetrical relation between the missing blocks. 
     The missing block A 00  is neighboring to the correct boundary blocks BB 1  and BB 16 , and an AEBME 00  value can be obtained according to equation (11): 
                       A   ⁢           ⁢   E   ⁢           ⁢   B   ⁢           ⁢   M   ⁢           ⁢     E   00       =         ∑     i   =   1     M     ⁢           ⁢       ∑     x   =     x   0           x   0     +   N   -   1       ⁢           ⁢              A   00     ⁡     (     x   ,       y   0     +   i       )       -     R   ⁡     (       x   +   v     ,       y   0     +   i   +   u       )                  +       ∑     i   =   1     M     ⁢           ⁢       ∑     y   =     y   0           y   0     -     (     N   -   1     )         ⁢           ⁢              A   00     ⁡     (         x   0     -   i     ,   y     )       -     R   ⁡     (         x   0     -   i   +   v     ,     y   +   u       )                      ,           (   11   )               
where A 00 (x, y) represents the pixels of the missing block A 00  in the present frame, R(x, y) represents the pixels of the previous frame, (x 0 , y 0 ) is the coordinate of the missing block A 00 , (v, u) represents the spatial or temporal motion vector, N represents a quantity of the missing blocks (i.e., 4 in the current embodiment), and M represents a pixel width of the external boundary (i.e., 4 pixels in the current embodiment).
 
     The missing block A 10  is neighboring to the boundary blocks BB 15 , and an AEBME 10  value can be obtained according to equation (12): 
                       A   ⁢           ⁢   E   ⁢           ⁢   B   ⁢           ⁢   M   ⁢           ⁢     E   10       =       ∑     i   =   1     M     ⁢           ⁢       ∑     y   =     y   0           y   0     -     (     N   -   1     )         ⁢           ⁢                    A   10     ⁡     (         x   0     -   i     ,   y     )       -               R   ⁡     (         x   0     -   i   +   v     ,     y   +   u       )                        ,           (   12   )               
where A 10 (x, y) represents the pixels of the missing block A 10  in the present frame. Similarly, an AEBME 01  value of the missing block A 01  neighboring to the boundary block BB 2  can be obtained in like manner.
 
     The missing blocks A 11 , B 11 , C 11 , D 11  in the inside layer are not neighboring to any blocks having correct image data, and therefore they are determined with candidate motion vectors by majority decision according to missing blocks adjacent thereto. 
     At step S 140 , according to a complexity of each candidate motion vector of the missing macroblock, the missing blocks are divided into a plurality of sub-macroblocks having optimal sizes, e.g., sub-macroblocks having 16×16, 16×8, 8×16, or 8×8 pixels as shown in  FIG. 13 , in which T 1  represents 16×16 sub-macroblocks, T 2  represents 16×8 sub-macroblocks, T 3  represents 8×16 sub-macroblocks, and T 4  represents 8×8 sub-macroblocks. According to the Laplacian distribution of the motion field constituted by the candidate motion vectors, one having the maximum probability is selected from the T 1 , T 2 , T 3 , and T 4  sub-macroblocks by Bayes principle. If the missing macroblock has a relatively regular motion, then larger sub-macroblocks, e.g., sub-macroblocks of 16×16 pixels, are more suitable for subsequent motion refinement. On the contrary, if the missing macroblock has a relatively complex motion, the smaller sub-macroblocks, e.g., sub-macroblocks of 8×8 pixels, are more suitable for subsequent motion refinement. 
     The sub-macroblock having the maximum probability can be determined by equation (13): 
                             T   ^     =       ⁢     max   ⁢     {     P   ⁡     (       T   j     ❘   mv     )       }         ,       j   ∈     {     1   ,   2   ,   3   ,   4     }       =     max   ⁢     {       P   ⁡     (       T   j     ⋂   mv     )         P   ⁡     (   mv   )         }                       =       ⁢     max   ⁢     {         P   ⁡     (     mv   ❘     T   j       )       ⁢     P   ⁡     (     T   j     )           P   ⁡     (   mv   )         }         ,                 (   13   )               
where P(T j ) represents a probability of a sub-macroblock T j , and mv represents a motion vector motion vector.
 
     Equation (13) can be further expressed in a form of equation (14): 
                             T   ^     ′     =       ⁢     max   ⁢     {     P   ⁡     (     mv   ❘     T   j       )       }                   =       ⁢     max   ⁢     {       ∏     k   =   1       N   j       ⁢     P   ⁡     (       mv   k     ❘     B     k   ,   j         )         }                     =       ⁢     max   ⁢     {       ∏     k   =   1       N   j       ⁢           ⁢     [       ∏       mv   k     ∈     B     k   ,   j           ⁢           ⁢       P   ⁡     (       mv   k   x     ❘     B     k   ,   j         )       ·     P   ⁡     (       mv   k   y     ❘     B     k   ,   j         )           ]       }         ,                 (   14   )               
where B k,j  represents the k th  partition in the sub-macroblock T j , mv k  represents the motion vectors of the partition B k , mv k   x  represents a horizontal component of the motion vector (mv k ), mv k   y  represents a vertical component of the motion vector (mv k ), N j  represents the number of partitions in the missing macroblock, where N j =1 corresponding to T 1 , N j =2 corresponding to T 2 , N j =2 corresponding to T 3 , and N j =4 corresponding to T 4 .
 
     According to equation (14), equation (15) can be obtained based on the Laplacian distribution of the motion field: 
                             T   ^     ′     =       ⁢     max   ⁢     {       ∏     k   =   1       N   j       ⁢             ⁢             ⁢     [       ∏       mv   k     ∈     B     k   ,   j           ⁢           ⁢       P   ⁡     (       mv   k   x     ❘     B     k   ,   j         )       ·     P   ⁡     (       mv   k   y     ❘     B     k   ,   j         )           ]       }                     =       ⁢     max   ⁢     {       ∏     k   =   1       N   j       ⁢           ⁢     [             ∏       mv   k     ∈     B     k   ,   j           ⁢           ⁢       1     2   ⁢           ⁢     σ   k   x         ⁢       exp   ⁡     (       -            mv   k   x     -     μ   k   x                σ   k   x       )       ·                     1     2   ⁢           ⁢     σ   k   y         ⁢     exp   ⁡     (       -            mv   k   y     -     μ   k   y                σ   k   y       )               ]       }         ,                 (   15   )               
where μ k   x  represents the average horizontal component, μ k   y  represents the average vertical component, while σ k   x  and σ k   y  can be expressed by:
 
                 σ   k   x     =       1       2   ⁢           ⁢   a         ⁢       ∑       mv   k     ∈     B     k   ,   j           ⁢            mv   k   x     -     μ   k   x                  ;       μ   k   x     =     median   ⁢     {     mv   k   x     }                         σ   k   y     =       1       2   ⁢           ⁢   a         ⁢       ∑       mv   k     ∈     B     k   ,   j           ⁢            mv   k   y     -     μ   k   y                  ;       μ   k   y     =     median   ⁢       {     mv   k   y     }     .               
When taking a natural log to the above equation, equation (16) is obtained as following:
 
     
       
         
           
             
               
                 T 
                 ^ 
               
               ′ 
             
             ≈ 
             
               
                 max 
                 j 
               
               ⁢ 
               
                 
                   { 
                   
                     
                       ∑ 
                       
                         k 
                         = 
                         1 
                       
                       
                         N 
                         j 
                       
                     
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     
                       ( 
                       
                         
                           
                             - 
                             aln 
                           
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             σ 
                             k 
                             x 
                           
                         
                         - 
                         
                           aln 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             σ 
                             k 
                             y 
                           
                         
                       
                       ) 
                     
                   
                   } 
                 
                 . 
               
             
           
         
       
     
     At step S 150 , motion refinements are applied to the motion vectors of the sub-macroblocks of the missing macroblock. An adaptive search range (ASR) is obtained according to a first adjacent external boundary matching error (AEBME) at the starting point of a spiral scanning path, along with all of the motion vector values. If the ASR is equal to or less than a threshold value, the motion vector is taken as a motion vector of the missing macroblock, and the threshold value is determined by a proximity of the AEBME value to an original residual value of a sum of absolute differences (SAD). 
     The ASR is determined by equation (17) according to the AEBME: 
                       A   ⁢           ⁢   S   ⁢           ⁢   R     =                  A   ⁢           ⁢   E   ⁢           ⁢   B   ⁢           ⁢   M   ⁢           ⁢     E   ⁡     (     u   ,   v     )         -   μ          μ     ×     max   ⁡     (       D   x     ,     D   y       )       ×     W   λ       +   ɛ       ,           (   17   )               
where AEBME(u, v) represents the first adjacent external boundary matching error at the starting point, μ represents the original residual values of the SAD, W represents the maximum search size, and λ and ε are constant factors (λ=32 and ε=0). Further, D X  and D Y  can be expressed by equation (18):
 
 D   X =max| X ( MVs )− u| 
 
 D   Y =max| Y ( MVs )− v|   (18),
 
where X(MVs) represents the X component of the motion vector, and Y(MVs) represents the Y component of the motion vector.
 
     The dynamic threshold (DT α ) is expressed by equation (19):
 
 DT   a   =μ+N×α   (19),
 
where N is the total number of pixels, and α is a constant factor for each pixel.
 
     Generally, μ is unavailable, and therefore α is needed in addition for calculating the dynamic threshold (DT α ). 
       FIG. 14  is a schematic diagram illustrating a spiral scanning path according to a conventional technology. Referring to  FIG. 14 , the spiral scanning path starts from a starting point marked with 0 (in the current embodiment, the starting point is an average motion vector of the candidate motion vectors), in sequence along indication of the arrows, to sequentially perform refinement, so as to search for the best motion vector. For example, the motion vector shown in  FIG. 14  marked with 2 represents a desired motion vector, and therefore the motion refinement process processes only three motion vectors. The rest motion vectors are not to be further processed, and therefore the total computation for searching comparison process can be saved. The spiral scanning path saves processing time. 
     Finally, at step S 180 , the image data of the missing macroblock can be obtained by moving the candidate macroblock to the position of the missing macroblock according to the motion vector, and thus concealing the negative affection caused by the missing macroblock to the entire image frame. 
     More details of the conventional TEC methods, ordinary SEC methods and intra high-speed SEC method can be learnt by referring to the publication “Optimization of Hybridized Error Concealment for H.264”, IEEE TRANSACTIONS ON BROADCASTING, VOL. 54, NO. 3, SEPTEMBER 2008. 
     Although having an efficiency higher than the SEC method, the TEC method cannot be applied when the scenario changes, the image moves fast, the image moves irregularly, or an object appears or disappears, because there is no correct frame having temporal correlation available for reference. As such, a method of an efficient adaptive mode selection for H.264/AVC-coded video delivery in burst-packet-loss networks, and a hybridized error concealment method are highly desired, of improving the overall decoding efficiency and performance. 
     SUMMARY OF THE INVENTION 
     A primary objective of the present invention is to provide a method of an efficient adaptive mode selection for H.264/AVC-coded video delivery in burst-packet-loss networks to generate the image data of the missing macroblocks (macroblocks) in the decoded current frame by using spatial redundancy information in the same frame and the temporal redundancy information in the inter frames. The method first employs the intra high-speed spatial error concealment (SEC) method for the initial frame. For the succeeding inter frames, the temporal error concealment (TEC) method is applied when the adjacent external boundary matching error (AEBME) of the surrounding macroblocks is not more than the dynamic threshold (DT α ). The intra high-speed SEC is used when AEBME is more than (DT α ) and the surrounding macroblocks are all intra-coded, or otherwise a normal SEC method is applied. 
     According to the method of the present invention, methods for concealment can be optimally selected corresponding to damaged frames having different spatial redundancy information and temporal redundancy information. As such, the method of the present invention is adapted for drastically saving computation time, improving the overall image decoding efficiency, and improving the image decoding performance. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The present invention will be apparent to those skilled in the art by reading the following detailed description of a preferred embodiment thereof, with reference to the attached drawings, in which: 
         FIG. 1  is a schematic diagram illustrating a BI method according to a conventional technology; 
         FIG. 2  is a schematic diagram illustrating a DI method according to a conventional technology; 
         FIG. 3  is a directional schematic diagram illustrating a DI method according to a conventional technology; 
         FIG. 4  is a schematic diagram illustrating a BNM method according to a conventional technology; 
         FIG. 5  is a schematic diagram illustrating reference blocks of an MDI method according to a conventional technology; 
         FIG. 6  is a schematic diagram illustrating a 3-directional weighting interpolation of the MDI method according to a conventional technology; 
         FIG. 7  is a flow chart illustrating a fast classification of a fast SEC method according to a conventional technology; 
         FIG. 8  is a schematic diagram illustrating a fast SEC method according to a conventional technology; 
         FIG. 9  is a flow chart illustrating a conventional TEC method. 
         FIG. 10  is a solid view of boundary blocks according to a conventional technology; 
         FIG. 11  is a schematic diagram illustrating boundary blocks according to a conventional technology; 
         FIG. 12  is a schematic diagram illustrating a boundary matching algorithm (BMA) and related blocks according to a conventional technology; 
         FIG. 13  is a schematic diagram illustrating boundary blocks according to a conventional technology; 
         FIG. 14  is a schematic diagram illustrating a spiral scanning path according to a conventional technology; 
         FIG. 15  is a flow chart illustrating a first embodiment of the present invention; and 
         FIG. 16  is a flow chart illustrating a second embodiment of the present invention. 
     
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
     The accompanying drawings are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention. 
     The present invention provides a method of an efficient adaptive mode selection for H.264/AVC-coded video delivery in burst-packet-loss networks. In correspondence to damaged frames corresponding to different spatial redundancy information and temporal redundancy information, respectively, the method adaptively selects optimal error concealment methods for recovering the damaged frames, thus concealing the vision affection caused by the damaged framed to the overall image. 
     Among the intra high-speed SEC method, the normal SEC method, and the TEC method, which are employed in the present invention, the TEC method has the highest decoding efficiency. Therefore, the TEC method has the priority to be selected for recovering the damaged frames. However, regarding the first frame, i.e., the initial frame, there is not any temporal redundancy information available for reference, and therefore, when the first frame is damaged, the intra high-speed SEC method is selected for recovering the image data of the first frame according to correct blocks in the first frame and spatial redundancy information. Further, when the displayed scenario changes, an object appears or disappears, images move fast or irregularly, an object rotates or reforms, because of the lack of significant temporal correlation between successive frames, the TEC method cannot apply. Therefore, in this case, the intra high-speed SEC method or the normal SEC method is selected for recovering the damaged frame. When all adjacent macroblocks surrounding the missing blocks are intra-coded, the intra high-speed SEC method is selected, and or otherwise the normal SEC method is selected. 
       FIG. 15  is a flow chart illustrating a first embodiment of the present invention. The first embodiment of the present invention is adapted for processing erroneous image data generated when decoding an encoded frame of a video image sequence, in which the decoded frame is concealment processed with one selected from the TEC method, the normal SEC method, and the intra high-speed SEC method. 
     Referring to  FIG. 15 , the method according to the first embodiment of the present invention starts from step S 230 . When the decoded image frame includes a missing macroblock, at step S 230 , the concealment process starts, and the flow enters S 240 . At step S 240 , it is determined that whether the processed frame is the first frame, i.e., the initial frame. If the processed frame is the first frame, then the flow enters step S 270 , or otherwise if the processed frame is not the first frame, the flow enters step S 280 . At step S 270 , the intra high-speed SEC method is selected for recovering the missing macroblock, and then the flow enters step S 260  where the decoding process ends. At step S 280 , it is determined whether an adjacent external boundary matching error (AEBME) is greater than a dynamic threshold (DT α ). If it is determined that the AEBME is greater than the dynamic threshold (DT α ), then the flow enters step S 290 , or otherwise, the flow enters step S 300 . At step S 290 , it is determined whether all adjacent macroblocks surrounding the missing macroblock are all intra-coded. If it is determined that all adjacent macroblocks surrounding the missing macroblock are all intra-coded, then the flow enters step S 270 , or otherwise, the flow enters step S 310 . At step S 300 , the TEC method is selected for recovering the missing macroblock, and then the flow enters step S 260 , where the decoding process ends. At step S 310 , the normal SEC method is selected for recovering the missing macroblock, and then the flow enters step S 260 , where the flow ends. 
     The AEBME is defined by the equations (11) and (12), and the dynamic threshold (DT α ) is defined by the equation (19), and are not to be iterated hereby. 
       FIG. 16  is a flow chart illustrating a second embodiment of the present invention. The second embodiment of the present invention is adapted for processing erroneous image data generated when decoding the entirety of a video image sequence, in which all decoded frames are concealment processed with one selected from the TEC method, the normal SEC method, and the intra high-speed SEC method. 
     Referring to  FIG. 16 , the method according to the second embodiment of the present invention starts from step S 210 , where the flow starts, then the flow enters step S 220 . At step S 220 , image data of encoded frames are decoded to generate decoded video frames, then the flow enters step S 230 . At step S 230 , it is determined that whether the decoded video frames include a missing macroblock. If it is determined that the decoded video frames include a missing macroblock, then the flow enters step S 240 , or otherwise, the flow enters step S 250 . At step S 250 , it is determined whether all of the frames are decoded. If it is determined that all of the frames are decoded, then the flow enters step S 260 , where the decoding process ends, or otherwise the flow returns back to step S 220  to decoding image data of the rest frames. At step S 240 , it is determined that whether the processed frame is the first frame, i.e., the initial frame. If the processed frame is the first frame, then the flow enters step S 270 , or otherwise if the processed frame is not the first frame, the flow enters step S 280 . At step S 270 , the intra high-speed SEC method is selected for recovering the missing macroblock, and then the flow enters step S 250  for determining whether all of the frames are decoded. If it is determined that all of the frames are decoded. At step S 280 , it is determined whether the adjacent external boundary matching error (AEBME) is greater than the dynamic threshold (DT α ). If it is determined that the AEBME is greater than the dynamic threshold (DT α ), then the flow enters step S 290 , or otherwise, the flow enters step S 300 . At step S 290 , it is determined whether all adjacent macroblocks surrounding the missing macroblock are all intra-coded. If it is determined that all adjacent macroblocks surrounding the missing macroblock are all intra-coded, then the flow enters step S 270 , or otherwise, the flow enters step S 310 . At step S 300 , the TEC method is selected for recovering the missing macroblock, and then the flow enters step S 250  for determining whether all of the frames are decoded. If it is determined that all of the frames are decoded. At step S 310 , the normal SEC method is selected for recovering the missing macroblock, and then the flow enters step S 250  for determining whether all of the frames are decoded. 
     It has been experimentally proved that the method provided by the present invention can achieve a gain much higher than the decoder provided by the reference software Joint Model (JM) of the H.264 standard, for about 10.62 dB, without requiring for additional time regarding a large range basic sequence. The reference software 1M is provided by the joint video team of ISO/IEC MPEG and ITU-T VCEG (JVT association). 
     Although the present invention has been described with reference to the preferred embodiments thereof, it is apparent to those skilled in the art that a variety of modifications and changes may be made without departing from the scope of the present invention which is intended to be defined by the appended claims.