Abstract:
The invention relates to video image compression technologies, and discloses a method and system for upsampling a spatial scalable coded video image so that during upsampling computation complexity may be reduced while coding performance is substantially unchanged. In the invention, the principle that human eyes are far less sensitive to a chrominance components than to a luminance components is utilized, and a simpler filter is adopted for the chrominance components than that for the luminance components during upsampling in I_BLINTRA_Base inter-layer prediction or residual samples image inter-layer prediction, thereby reducing effectively calculation complexity while coding performance is substantially unchanged.

Description:
CROSS REFERENCE TO RELATED APPLICATION 
     The present application is a continuation application of PCT application No. PCT/CN2006/002672, filed on Oct. 11, 2006, which claims the benefit of Chinese Patent Application No. 200510030401.2, filed on Oct. 11, 2005, both of which are incorporated by reference herein in its entirety. 
    
    
     FIELD 
     Embodiments of the present invention relates to video image compression technologies and in particular to a method and system for upsampling a spatial scalable coded video image. 
     BACKGROUND 
     The H.264 video compression coding standard is established jointly by the International Telecommunication Union—Telecommunication Standardization Section (ITU-T), the International Organization for Standardization (ISO) and the Moving Picture Expert Group (MPEG) of the International Electrician Commission (IEC). 
     Currently, the H.264 has gradually become a predominating standard in multimedia communication, and numerous multimedia real time communication products using the H.264 standard, e.g., a video conferencing, a video telephony, a 3 rd  Generation (3G) mobile communication terminal, etc., and network streaming media products have emerged successively. It can be said that whether the H.264 can be supported has become a crucial factor of determining product competitiveness in the market field. Especially along with emergence of 3G mobile communication systems and rapid development of the Internet Protocol (IP), video network communication has gradually become one of dominant communication services. 
     Components and a transport mechanism of a message under the H.264 standard will be described briefly below. 
     A layered mode is adopted in the H.264 standard to define a video coding layer (VCL) and a network abstraction layer (NAL), and the NAL is designed specifically for network transmission and can be adapted to video transmission over different networks to further improve network affinity. The H.264 introduces an IP packet oriented coding mechanism, which is advantageous to packet transmission over a network, supports streaming media transmission of a video over the network and robust error resilience, especially of accommodating to requirements of wireless video transmission with a high packet loss ratio and serious interference. All of H.264 data to be transmitted, including image data and other messages, is encapsulated into packets of a uniform format for transmission, i.e., Network Abstraction Layer Units (NALU). Each NALU is a variable length character string in bytes of certain syntax elements and includes head information of one byte available for representing a data type, and payload data of several integer bytes. A NALU can carry a coded slice, various types of data segmentations or a set of sequence or image parameters. In order to enhance reliability of data, each frame of images is divided into several slices, each of which is carried in an NALU. A slice is further consisted of several smaller macroblocks and is a minimal processing unit. Generally, slices at corresponding locations in tandem frames are associated with each other, and slices at different locations are independent from each other, so that a code error can be prevented from diffusing between the slices. 
     H.264 data includes texture data of non-reference frames, sequence parameters, image parameters, Supplemental Enhancement Information (SEI), texture data of reference frames and so on. The SEI is a general designation of information playing an auxiliary role in H.264 video decoding, display and other aspects. 
       FIG. 1  illustrates a H.264 compression processing framework. A basic H.264 processing unit is a 16×16 macroblock  110 , for which advanced techniques, such as multiple frame reference, intra-frame prediction  120 , multiple macroblock type, 4×4 integer transform and quantization  130 , loop filter  140 , ¼-pel accuracy motion estimation prediction, Context-based Adaptive Variable Length Coding (CAVLC), Context Adaptive Binary Arithmetic Coding (CABAC), entropy coding  150 , etc., are adopted, therefore, H.264 compression efficiency can be improved to more than a double of that of MPEG-2, H.263 and MPEG-4 ASP. 
     During establishing the H.264 scalable coding standard, the Joint Video Team (JVT) makes a basic layer compatible with a H.264 Main Profile and uses an algorithm framework of Motion Compensated Temporal Filter (MCTF), so that functions such as spatial scalability, temporal scalability, quality or SNR scalability, complexity scalability, etc., can be implemented very well. 
     The latest reference model of the Joint Video Team Scalable Video Coding (JVT SVC) is the Joint Scalable Video Model 3 (JSVM3).  FIG. 2  illustrates a block diagram of the above SVC algorithm. Input video data ( 210 ) is received, 2-dimensional (2D) spatial sampling ( 220 ) is performed thereon, and operations such as temporal decomposition ( 230 ), motion coding ( 240 ), macroblock intra-frame prediction ( 250 ), transform/entropy coding ( 260 ), etc., are performed in a core decoder. 
     It shall be noted that a temporal decomposition process can adopt a B frame decomposition based method as illustrated in  FIG. 3  or a MCTF decomposition based method as illustrated in  FIG. 4 , in which the resolution at Layer 0 is an original frame rate, and those at Layers 3, 2, 1 are ½, ¼ and 1/12 of the original frame rate, respectively. 
     In terms of intra-frame prediction, a H.264 intra-frame prediction method is adopted for a JSVM3 basic layer. A prediction mode, I_BL, in which a macroblock at the present layer is predicated pixel by pixel from a corresponding macroblock at a lower layer, is added based upon a H.264 prediction mode for an enhanced layer. As illustrated in  FIG. 5 , macroblocks H 1 , H 2 , H 3 , etc., at Layer K+1 are predicated pixel by pixel from corresponding macroblocks H 1 , H 2 , H 3 , etc., at Layer K. 
     Furthermore, a macroblock residual image at an enhanced layer, i.e., a difference image after subtractive prediction, can be predicated from a residual image of a corresponding macroblock at a basic or lower layer in a similar way to the I_BL. 
     For spatial scalable coding, the corresponding macroblock at the basic or lower layer has to be subject to an upsampling process in I_BL prediction or residual image prediction at the enhanced layer. Upsampling is a kind of re-sampling for scaling up or down a sampled signal. Assume that original sampling points are located at integer coordinates (0, 1, 2 . . . ) and a distance between new sampling points after re-sampling is denoted by a, then it is referred to as down-sampling if a&gt;1, otherwise upsampling if a&lt;1. 
     In the prior art, an upsampling filter in I-BL prediction is a relatively complex [1 −5 20 20 −5 1]/32 filter, that in the residual image prediction is a [1 1]/2 filter, and the same filter is adopted for luminance and chrominance images. 
     In a practical application, the above solution has the following drawback: in an upsampling process in I-BL prediction, the same relatively complex 6-order filter is adopted for a chrominance component as for a luminance component, and consequently calculation complexity in the upsampling process of the chrominance component may be too high. 
     Furthermore, the [1 1]/2 filter adopted in the upsampling process in residual image predication is too simple, and consequently this may influence coding performance. 
     That is, in the prior art, when up-sampling a spatial scalable coded video image, the difference between a chrominance component and a luminance component has not been considered, and calculation complexity and coding performance have not been considered comprehensively, thus some problems, for example, high calculation complexity or poor calculation performance, arise in spatial scalable coding of a video image. 
     SUMMARY 
     Embodiments of the present invention provide a method and system for upsampling a spatial scalable coded video image so that in an upsampling process, calculation complexity and coding performance are considered comprehensively to guarantee the calculation complexity and the coding performance. 
     According to an aspect of the invention, a method for upsampling a spatial scalable coded video image includes upsampling a luminance component and a chrominance component of an image layer respectively, in which a first filter is used to upsample the luminance component of the image layer and a second filter with an order lower than that of the first filter is used to upsample the chrominance component of the image layer. 
     Optionally, the upsampling is upsampling of a macroblock at a lower layer in “I_BL” inter-layer prediction or residual image inter-layer prediction. 
     Optionally, during upsampling in the “I_BL” inter-layer prediction, the first filter is a 6-order filter and the second filter is a 4- or 2-order filter. 
     Optionally, the 6-order filter is a [1 −5 20 20 −5 1]/32 filter, the 4-order filter is a [−1 5 5 −1]/8 filter, and the 2-order filter is a [1 1]/2 filter. 
     Optionally, during upsampling in the residual image inter-layer prediction, the first filter is a [−1 5 5-1]/8 filter and the second filter is a [1 1]/2 filter. 
     According to another aspect of the invention, a system for upsampling a spatial scalable coded video image includes at least a first filter adapted to upsample a luminance component of an image layer and a second filter adapted to upsample a chrominance component of the image layer, in which the second filter has an order lower than that of the first filter. 
     Optionally, the upsampling is upsampling of a macroblock at a lower layer in “I_BL” inter-layer prediction or residual image inter-layer prediction. 
     Optionally, when applied to the upsampling in the “I_BL” inter-layer prediction, the first filter is a 6-order filter and the second filter is a 4- or 2-order filter. 
     Optionally, the 6-order filter is a [1 −5 20 20 −5 1]/32 filter, the 4-order filter is a [−1 5 5 −1]/8 filter, and the 2-order filter is a [1 1]/2 filter. 
     Optionally, when be applied to the upsampling in the residual image inter-layer prediction, the first filter is a [−1 5 5-1]/8 filter and the second filter is a [1 1]/2 filter. 
     In this embodiment of the present invention, with use of the principle that human eyes are far less sensitive to a chrominance component than to a luminance component, a simpler filter is adopted for the chrominance component than that for the luminance component during upsampling in I_BL inter-layer prediction or residual image inter-layer prediction to thereby reduce effectively calculation complexity while coding performance is substantially unchanged. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a schematic diagram of a H.264 compression processing framework in the prior art; 
         FIG. 2  is a block diagram of an SVC algorithm in the prior art; 
         FIG. 3  is a block diagram of B frame based decomposition in the prior art; 
         FIG. 4  is a block diagram of MCTF based decomposition in the prior art; 
         FIG. 5  is a schematic diagram of an I_BL prediction mode at an enhanced layer in the prior art; 
         FIG. 6  is a flow chart of an embodiment of an upsampling method according to an embodiment of the present invention; and 
         FIG. 7  is a schematic diagram of an embodiment of an upsampling system according to an embodiment of the present invention. 
     
    
    
     DETAILED DESCRIPTION 
     Embodiments of the present invention will be further described in detail hereinafter with reference to the accompanying drawings to make the objects, aspects and advantages thereof more apparent. 
     Embodiments of the present invention are based upon the natural rule that human eyes are far less sensitive to a chrominance component than to a luminance component. In an upsampling process, different filters are adopted to upsample a chrominance component and a luminance component in a way that the filter for the chrominance component is simpler than that for the luminance component. Referring to  FIG. 6 , in an upsampling process, a first filter is adopted to upsample a chrominance component of an image layer (block  610 ) and a second filter is adopted to upsample a luminance component of the image layer (block  620 ). The second filter is simpler than the first filter, which may mean that the order of the second filter is lower than that of the first filter. 
     Calculation complexity of the chrominance component will be reduced accordingly, and thus a workload of a calculation device will be reduced, so that more traffic can be borne on the same apparatus to make a processing delay smaller and bring fluent pictures. 
     In scalable coding of a video image, a coding process at an enhanced layer needs to make full use of information from a base or lower layer to remove redundancy and improve coding efficiency. For example, the SVC adopts schemes of I_BL inter-layer prediction, residual image inter-layer prediction, motion vector inter-layer prediction and so on. 
     For spatial scalable coding, a corresponding macroblock at a base or lower layer shall be subject to an upsampling process during performing I-BL prediction or residual image prediction at an enhanced layer to facilitate prediction. 
     In a first embodiment of the invention, different filters are adopted for upsampling a luminance component and a chrominance component in I_BL prediction in a way that a 6-order filter, e.g., a [1 −5 20 20 −5 1]/32 filter, is used as the filter for upsampling the luminance component, and a simpler 4-order filter, e.g., a [1 −5 −5 1]/8 filter is used as the filter for upsampling the chrominance component. During using the [1 −5 20 20 −5 1]/32 filter, 6 multiplication operations, 5 addition operations and 1 division operation are needed for calculation of each sampling point, whereas during using the [1 −5 −5 1]/8 filter, 4 multiplication operations, 3 addition operations and 1 division operation are needed for calculation of each sampling point, resulting in a reduced calculation amount approximately by one-third. Since human eyes are less sensitive to the chrominance component, large difference will not be perceived visually in practice. 
     In a second embodiment of the invention, different filters are also adopted to upsample a luminance component and a chrominance component, for example, a [−1 5 5 −1]/8 filter is adopted to upsample the luminance component and a [1 1]/2 filter is adopted to upsample the chrominance component. In this embodiment, the [−1 5 5 −1]/8 filter of a higher order is adopted for the luminance component, and this can improve coding performance while an increase in calculation complexity falls within an acceptable range. If the [−1 5 5 −1]/8 filter is also adopted for the chrominance component, then 4 multiplication operations, 3 addition operations and 1 division operation are needed for calculation of each sampling point. But now in use of the [1 1]/2 filter, only 1 addition operation and 1 division operation are needed. Considering that a division operation by an integer power of 2 can be replaced with a faster shift operation, the amount of operations may be reduced by an order of magnitude relative to the [−1 5 5 −1]/8 filter. 
     In a third embodiment of the invention, the [−1 5 5 −1]/8 filter is adopted for both a luminance component and a chrominance component in an upsampling process of residual images in order to improve coding performance. 
     Embodiments of the present invention conduct emulation tests on some typical sequences in order to demonstrate effectiveness of the filter. 
     The following table presents performance comparison between test results of quarter common intermediate format (QCIF) and common intermediate format (CIF) two-layer coding of 300 frames of images in a Harbour sequence, in which the test results of embodiments of the invention are obtained when using a [−1 5 5 −1]/8 filter for upsampling a chrominance component in I_BL prediction and using a [−1 5 5 −1]/8 filter for upsampling a luminance component and a chrominance component in residual image prediction. 
     
       
         
               
               
               
             
               
               
               
               
               
               
               
               
               
             
               
               
               
               
               
               
               
               
               
             
           
               
                   
                   
               
               
                   
                   
                 Results according to embodiments of the 
               
               
                   
                 Results in the prior art 
                 invention 
               
             
          
           
               
                 QP 
                 Code rate 
                 Y 
                 Cb 
                 Cr 
                 Code rate 
                 Y 
                 Cb 
                 Cr 
               
               
                   
               
             
          
           
               
                 8 
                 59496.902 
                 50.7719 
                 52.0737 
                 52.632 
                 59095.658 
                 50.7719 
                 52.0702 
                 52.6277 
               
               
                 12 
                 43774.38 
                 47.5296 
                 49.3939 
                 50.1502 
                 43404.455 
                 47.527 
                 49.3866 
                 50.14 
               
               
                 16 
                 30060.991 
                 44.475 
                 47.3343 
                 48.4443 
                 29752.754 
                 44.474 
                 47.3262 
                 48.4323 
               
               
                 20 
                 19368.697 
                 41.399 
                 45.8185 
                 47.3728 
                 19133.755 
                 41.3966 
                 45.8097 
                 47.3585 
               
               
                 24 
                 11262.018 
                 28.4222 
                 44.5876 
                 46.5447 
                 11124.914 
                 38.4233 
                 44.5841 
                 46.5474 
               
               
                 28 
                 5870.7624 
                 35.6978 
                 43.3869 
                 45.5161 
                 5799.3328 
                 35.6983 
                 43.378 
                 45.5082 
               
               
                 32 
                 2873.7904 
                 33.2981 
                 42.2992 
                 44.3998 
                 2847.3504 
                 33.3022 
                 42.3175 
                 44.4073 
               
               
                 36 
                 1412.7112 
                 31.1973 
                 41.2843 
                 43.2547 
                 1403.5464 
                 31.1975 
                 41.2769 
                 43.247 
               
               
                   
               
             
          
         
       
     
     The test results show that with the use of embodiments of the invention, the code rate has been reduced while the PSNR value (Y) of the luminance component signal to noise ratio has been increased and the PNSR values (Cb and Cr) of the chrominance component signal to noise ratio have been identical by and large, so that coding performance has been improved. 
     It shall be noted that the above embodiments are merely some typical applications of the solutions of the invention. In addition to the above [−1 5 5 −1]/8 filter, other filters, for example, a 2-order filter such as a [1 1]/2 filter, can also be adopted for upsampling the chrominance component in the I_BL upsampling process, and other filters can also be adopted for upsampling the luminance and chrominance components in the residual image upsampling process. 
     Referring to  FIG. 7 , it is a schematic diagram of a system for upsampling a spatial scalable coded video image according to embodiments of the invention. This upsampling system includes a first filter  710  and a second filter  720 . 
     The first filter  710  is adapted to upsample a chrominance component of an image layer and the second filter  720  is adapted to upsample a luminance component of the image layer. The second filter  720  is simpler than the first filter  710 , which may mean that the order of the second filter  720  is lower than that of the first filter  710 . 
     When applied to upsampling in “I_BL” inter-layer prediction, the first filter  710  may be a 6-order filter, e.g., a [1 −5 20 20 −5 1]/32 filter, and the second filter  720  may be a 4-order filter, e.g., a [−1 5 5 −1]/8 filter. The second filter  720  can also be a 2-order filter, e.g., a [11]/2 filter. 
     When applied to upsampling in residual image inter-layer prediction, the first filter  710  may be a [−1 5 5 −1]/8 filter, and the second filter  720  may be a [1 1]/2 filter. 
     Although embodiments of the invention has been illustrated and described with reference to some preferred embodiments of the invention, those ordinarily skilled in the art shall appreciate that various changes can be made thereto in forms and details without departing from the spirit and scope of the invention.