Abstract:
Provided are a method and apparatus for scalable video coding and decoding. The scalable video coding method performs video coding separately at each resolution, and coding results are incorporated into one resolution level for compression. The scalable video coding combines images with the respective images into a single one while providing high image quality across all resolution levels.

Description:
CROSS-REFERENCE TO RELATED APPLICATION  
       [0001]     This application claims priority from Korean Patent Application No. 10-2004-0006479 filed on Jan. 31, 2004, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety.  
       BACKGROUND OF THE INVENTION  
       [0002]     1. Field of the Invention  
         [0003]     The present invention relates to a scalable video coding and decoding method and a scalable video encoder/decoder.  
         [0004]     2. Description of the Related Art  
         [0005]     A compression coding method is requisite for transmitting multimedia data, including text, video, and audio, since the amount of multimedia data is usually large.  
         [0006]     A basic principle of data compression lies in removing data redundancy. Data can be compressed by removing spatial redundancy in which the same color or object is repeated in an image, temporal redundancy in which there is little change between adjacent frames in a moving image or the same sound is repeated in audio, or mental visual redundancy taking into account human eyesight and dull perception of high frequency information. Data compression can be classified into lossy/lossless compression depending on whether source data is lost, intraframe/interframe compression depending on whether individual frames are compressed independently, and symmetric/asymmetric compression depending on whether time required for compression is the same as the time required for recovery. In addition, data compression is defined as real-time compression when a compression/recovery time delay does not exceed 50 ms and as scalable compression when frames have different resolution levels. For text or medical data, lossless compression is usually used. For multimedia data, lossy compression is usually used. Meanwhile, intraframe compression is usually used to remove spatial redundancy, and interframe compression is usually used to remove temporal redundancy.  
         [0007]     Recently, research into wavelet-based scalable video coding, which can provide a very flexible, scalable bitstream, has been actively carried out. The scalable video coding means a video coding method having scalability. Scalability indicates the ability to partially decode a single compressed bitstream. Scalability includes spatial scalability indicating a video resolution, Signal to Noise Ratio (SNR) scalability indicating a video quality level, temporal scalability indicating a frame rate, and a combination thereof.  
         [0008]     Among many techniques used for wavelet-based scalable video coding, motion compensated temporal filtering (MCTF) that was introduced by Jens-Rainer Ohm and improved by Seung-Jong Choi and John W. Woods is an essential technique for removing temporal redundancy and for video coding having flexible temporal scalability.  
         [0009]      FIG. 1A  shows a motion compensated temporal filtering (MCTF)-based scalable video encoder.  
         [0010]     Referring to  FIG. 1A , the scalable video encoder receives a plurality of frames making up a video sequence and compresses the frames in units of group of pictures (GOP) to generate a bitstream. To achieve this function, the scalable video encoder includes a temporal transform unit  110  removing temporal redundancies from the plurality of frames, a spatial transform unit  120  removing spatial redundancies, a quantizer  130  quantizing transform coefficients created by removing the temporal and spatial redundancies, and a bitstream generator  140  combining the quantized transform coefficients and other information into a bitstream.  
         [0011]     The temporal transform unit  110  includes a motion estimator  112  and a temporal filter  114  in order to perform temporal filtering by compensating for motion between frames. The motion estimator  112  calculates a motion vector between each block in a current frame being subjected to temporal filtering and its counterpart in a reference frame. The temporal filter  114  that receives information about the motion vectors performs temporal filtering on the plurality of frames using the information.  
         [0012]     A spatial transform unit  120  uses a wavelet transform to remove spatial redundancies from the frames from which the temporal redundancies have been removed, i.e., temporally filtered frames. The spatial transform unit  120  removes spatial redundancies from the frames using a wavelet transform. In a currently known wavelet transform, a frame is decomposed into four sections (quadrants). A quarter-sized image (L image), which is substantially the same as the entire image, appears in a quadrant of the frame, and information (H image), which is needed to reconstruct the entire image from the L image, appears in the other three quadrants. In the same way, the L image may be decomposed into a quarter-sized LL image and information needed to reconstruct the L image.  
         [0013]     The temporally filtered frames are converted to transform coefficients by spatial transformation. The transform coefficients are then delivered to a quantizer  130  for quantization. The quantizer  130  quantizes the real-number transform coefficients with integer-valued coefficients. The MCTF based video encoder uses an embedded quantization technique. By performing embedded quantization on transform coefficients, it is possible to not only reduce the amount of information to be transmitted but also achieve signal-to-noise ratio (SNR) scalability. Embedded quantization algorithms currently in use are embedded zero-tree wavelet (EZW), set partitioning into hierarchical trees (SPIHT), embedded zero block coding (EZBC), and embedded block coding with optimized truncation (EBCOT).  
         [0014]     The bitstream generator  140  generates a bitstream containing coded image data, the motion vectors obtained from the motion estimator  112 , and other necessary information.  
         [0015]     The scalable video coding method includes a method of performing a spatial transform (i.e., a wavelet transform) on frames and then performing a temporal transform, which is called an in-band scalable video coding.  
         [0016]      FIG. 1B  shows an in-band scalable video encoder in which frames are subjected to spatial transform (wavelet transform) followed by temporal transform.  
         [0017]     Referring to  FIG. 1B , the in-band scalable video encoder is designed to remove temporal redundancies that exist within a plurality of frames making up a video sequence after removing spatial redundancies.  
         [0018]     A spatial transform unit  150  performs a wavelet transform on each frame in order to remove spatial redundancies among the frames.  
         [0019]     A temporal transform unit  160  includes a motion estimator  162  and a temporal filter  164  and performs temporal filtering on the frames from which the spatial redundancies have been removed in a wavelet domain in order to remove temporal redundancies.  
         [0020]     A quantizer  170  quantizes transform coefficients obtained by removing spatial and temporal redundancies from the frames.  
         [0021]     A bitstream generator  180  generates a bitstream from motion vectors and coded image subjected to quantization.  
         [0022]      FIG. 2A  is a diagram for explaining an MCTF process used in a scalable video coding algorithm to remove temporal redundancies while maintaining temporal scalability.  
         [0023]     Referring to  FIG. 2A , an L frame is a low frequency frame corresponding to an average of frames while an H frame is a high frequency frame corresponding to a difference between frames. In the illustrated coding process, pairs of frames at a low temporal level are temporally filtered and then decomposed into pairs of L frames and H frames at a higher temporal level, and the pairs of L frames are again temporally filtered and decomposed into frames at a higher temporal level.  
         [0024]     An encoder performs wavelet transformation on one L frame at the highest temporal level and the H frames and generates a bitstream. Frames indicated by shading in  FIG. 2A  are the ones that are subjected to a wavelet transform. That is to say, the coding is performed in an order from lower level frames to higher level frames.  
         [0025]     On the other hand, a decoder performs an inverse operation to the encoder on the frames indicated by shading and obtained by inverse wavelet transformation from a high level to a low level for reconstructions. L and H frames at temporal level  3  are used to reconstruct two L frames at temporal level  2 , and the two L frames and two H frames at temporal level  2  are used to reconstruct four L frames at temporal level  1 . Finally, the four L frames and four H frames at temporal level  1  are used to reconstruct eight frames. While the MCTF-based video coding scheme basically offers flexible temporal scalability, it still has several disadvantages, including unidirectional motion estimation and poor performance at a low temporal rate, which is described in several publications. One among the publications is disclosed by Woo-Jin Han (co-inventor of the present invention) in ISO/IEC JTC 1/SC 29/WG 11, entitled Successive Temporal Approximation and Referencing (STAR) for improving MCTF in Low End-to-end Delay Scalable Video Coding. The STAR will be described with reference to  FIG. 2B .  
         [0026]      FIG. 2B  is a diagram for explaining a temporal filtering process in a successive temporal approximation and referencing (STAR) algorithm. In  FIG. 2B , ‘I’ frame and ‘H’ frame denote an intracoded frame (encoded without reference to another frame) and a high frequency subband encoded with reference to one or more frames.  
         [0027]     Like a MCTF algorithm, a STAR algorithm is designed to remove temporal redundancies while maintaining temporal scalability at a decoder side. However, both coding and decoding processes in the STAR algorithm are performed in the order of highest to lowest temporal level. Referring to  FIG. 2B , coding and decoding are all performed in the order of numbers 0, 4, 2, 6, 1, 3, 5, and 7. Furthermore, unlike MCTF, STAR has a multi-reference function. The requirement for maintaining temporal scalability at encoder and decoder sides while using the multi-reference function is defined by: 
 
 R   k   ={F ( l )|( T ( l )&gt; T ( k )) or (( T ( l )= T ( k ))and ( l&lt;=k ))}
 
 where F(k) and T(k) respectively denote a frame with index k and its temporal level and k and I respectively denote indices of a frame currently being encoded and of frames being referenced. 
 
         [0028]     Referring to  FIG. 2B , frames may be encoded with reference to themselves, which is useful for rapidly varying video sequences. Encoding and decoding processes using the STAR algorithm may be performed as follows:  
         [0029]     Encoding Process 
        1. A first frame in a GOP is encoded as an I-frame.     2. Then, motion estimation is performed on frames at the next temporal level, followed by encoding using reference frames defined by Equation (1). Within the same temporal level, encoding is performed starting from the leftmost frame toward the rightmost (in order from the lowest to the highest index frame).     3. The step (2) is performed until all frames in the GOP are encoded. Subsequent encoding of frames in the next GOP continues until encoding of all GOPs is finished.        
 
         [0033]     Decoding Process 
        1. A first frame in a GOP is decoded.     2. Frames at the next temporal level are decoded with reference to previously decoded frames. Within the same temporal level, decoding is performed starting from the leftmost frame toward the rightmost (in order from the lowest to the highest index frame).     3. The step (2) is performed until all frames in the GOP are decoded. Subsequent decoding of frames in the next GOP continues until decoding of all GOPs is finished.        
 
         [0037]     MCTF and STAR algorithms are all designed to remove temporal redundancies, followed by wavelet transform to remove spatial redundancies. Removal of temporal redundancies using motion compensation will now be described with reference to  FIG. 3 .  FIG. 3  is a diagram for explaining wavelet-based video coding supporting spatial scalability.  
         [0038]     Wavelet-based video coding involves generating a residual image by subtracting referred images created using one or more referenced images from an original image and then performing wavelet transform and quantization on the generated residual image to obtain a coded image. Referring to  FIG. 3 , a wavelet-based video encoder supporting three spatial layers generates a bitstream including three layers of coded images and information (motion vectors) used to create three layers of referred images for each frame.  
         [0039]     More specifically, the encoder downsamples an original image O 1  of layer L 1  to produce an original image O 2  of layer L 2 . Similarly, the encoder downsamples the original image O 2  of layer L 2  to produce an original image O 3  of layer L 3 . The encoder uses one or more referenced images to produce a referred image R 1  of layer L 1  for temporal filtering of the original image O 1 . In the same manner, the encoder produces referred images R 2  and R 3  of layers L 1  and L 2 , respectively, using one or more referenced images for temporal filtering of the original images O 2  and O 3 . Each of the referred images R 1 , R 2 , and R 3  is generated using motion estimation between each of the original images O 1 , O 2 , and O 3  and each referenced image having temporal difference from the corresponding original image O 1 , O 2 , or O 3 . The encoder then produces residual images E 1 , E 2 , and E 3  by respectively subtracting the referred images R 1 , R 2 , and R 3  from the original images O 1 , O 2 , and O 3 . The encoder performs wavelet transform and quantization on the residual images E 1 , E 2 , and E 3  to obtain coded images with the respective layers L 1 , L 2 , and L 3 . The coded images with the respective layers L 1 , L 2 , and L 3  and information on estimated values (values of motion vectors) used to create referred images R 1 , R 2 , and R 3  are combined into a bitstream.  
         [0040]     A decoder that receives the bitstream is able to reconstruct the original video sequence composed of images having desired resolution. That is, the decoder pre-decodes a bitstream or receives a pre-decoded bitstream to reconstruct images having desired resolution among the layers L 1 , L 2 , and L 3 . However, in the wavelet-based video coding, the encoder generates the bitstream containing all coded image data and information on estimated motion vectors for the three layers L 1 , L 2 , and L 3 . That is, since the bitstream contains a great deal of redundant information on similar images, video coding efficiency is degraded.  
         [0041]     Another video encoder designed to increase the coding efficiency generates a bitstream containing information used to create the referred image R 1  having the highest resolution and coded image having the highest resolution, as opposed to a wavelet-based video encoder to generate a bitstream having information on a lower-resolution image incorporated into a high resolution image. However, actually, the values of motion vector values used to derive the referred images R 1 , R 2 , and R 3  with the respective layers L 1 , L 2 , and L 3  is actually similar but not identical. Thus, the encoder estimates the motion of a lower-resolution image with motion vectors for the highest resolution image, compared to an optimal estimation, which degrades the quality of the residual image E 2  or E 3 . In particular, this causes serious degradation of quality of the lowest resolution residual image E 3 . Allocation of more bits for the residual image E 3  during encoding may solve this problem but incurs degradation in compression efficiency.  
         [0042]     Meanwhile, the in-band scalable video encoder of  FIG. 1B  can provide high quality, low-resolution images by performing motion estimation and temporal filtering on images subjected to a wavelet transform. However, in-band video coding has a problem in that the quality of an image reconstructed at the decoder side is lower than those provided by other techniques described above because it requires temporal filtering in the wavelet domain.  
         [0043]     One of various approaches developed to solve these problems is disclosed in a paper presented by NEC Corp. [“Multi-Resolution MCTF for 3D Wavelet Transformation in Highly Scalable Video”, ISO/EEC JTC1/SC29/WG11, July 2003]. According to the paper, by replacing a high resolution low subband with a low-resolution image at the encoder side, it is possible to effectively contain information ranging from highest to lowest resolution in the highest resolution coded image. As for estimated values, the bitstream contains only motion vectors used to derive the highest resolution referred image. At the decoder side, a drift error compensation filter is used. According to this algorithm, a significant percentage of lower resolution information can be contained in a high resolution coded image by inserting a lower-resolution image into the high resolution image. However, the use of only motion vectors for the high resolution image provides lower performance than expected. Therefore, it is highly desirable to have a video coding algorithm providing high image quality at all resolution levels while reducing redundant information as much as possible.  
       SUMMARY OF THE INVENTION  
       [0044]     The present invention provides a method and apparatus for video coding and decoding designed to provide high image quality at all resolution levels while reducing redundancy in a coded image with each resolution.  
         [0045]     According to an aspect of the present invention, there is provided a scalable video coding method comprising performing low-passing filtering on each of original-resolution images in a video sequence to generate lower-resolution images corresponding to the original-resolution images and removing temporal redundancies from the original-resolution images and the lower-resolution images to generate original-resolution residual images and lower-resolution residual images, performing a wavelet transform on the original-resolution residual images and lower-resolution residual images to respectively generate an original-resolution transformed images and lower-resolution transformed images and combining the lower-resolution transformed images into the original-resolution transformed images to generate unified original-resolution transformed images, and quantizing each of the unified original-resolution transformed images to generate coded image data and generating a bitstream containing the coded image data and motion vectors obtained while removing the temporal redundancies from the original-resolution images and the lower-resolution images.  
         [0046]     Here, the low-pass filtering is preferably performed by downsampling using a wavelet 9-7 filter.  
         [0047]     The generated lower-resolution images may include first low-resolution images obtained by low-pass filtering each of the original-resolution images and second low-resolution images obtained by low-pass filtering the first low-resolution images. Here, the original-resolution images and the first and second low-resolution images are respectively converted into original-resolution transformed images and first and second low-resolution transformed images after removing the temporal redundancies therefrom, among which the first and second low-resolution transformed images are then combined together to generate unified first low-resolution transformed images, and the original-resolution transformed images and the unified first low-resolution transformed images are combined together to generate unified original-resolution transformed images.  
         [0048]     The removing of temporal redundancies may be performed by each resolution level, and may comprise performing motion estimation on each resolution image to find motion vectors to be used in removing temporal redundancies from the image by referencing one or more original images corresponding to one or more coded images, and removing temporal redundancies from the images by performing motion compensation using the motion vectors obtained by the motion estimation to generate residual images.  
         [0049]     The referenced images corresponding to the coded images may be obtained by decoding the coded images.  
         [0050]     The scalable video coding method may further comprise referencing the residual images when temporal redundancies of the residual images themselves are removed.  
         [0051]     According to another aspect of the present invention, there is provided a scalable video encoder comprising a temporal redundancy remover removing temporal redundancies from each of original-resolution images and lower-resolution images corresponding to the original-resolution image and generating original-resolution residual images and lower-resolution residual images, a spatial redundancy remover performing a wavelet transform on the original-resolution residual images and lower-resolution residual images to respectively generate original-resolution transformed images and lower-resolution transformed images and combining the lower-resolution transformed images into the original-resolution transformed image to generate unified original-resolution transformed images, and a quantizer quantizing each of the unified original-resolution transformed images to generate coded image data, and a bitstream generator generating a bitstream containing the coded image data and motion vectors obtained while removing the temporal redundancies from the original-resolution images and the lower-resolution images.  
         [0052]     The encoder may further comprise a plurality of low-pass filter performing low-pass filtering on each of the original-resolution images to generate the lower-resolution images.  
         [0053]     The generated lower-resolution images may include first low-resolution images obtained by low-pass filtering each of the original-resolution images and second low-resolution images obtained by low-pass filtering the first low-resolution images. Here, the original-resolution images and the first and second low-resolution images are respectively converted into the original-resolution transformed images and the first and second low-resolution transformed images by the spatial redundancy remover after the temporal redundancy remover removes the temporal redundancies therefrom, among which the first and second low-resolution transformed images are then combined together to generate unified first low-resolution transformed images, and the original transformed images and the unified first low-resolution transformed images are combined together to generate unified original-resolution transformed images.  
         [0054]     The temporal redundancy remover removing temporal redundancies for each resolution image may comprise one or more motion estimators finding motion vectors to be used in removing temporal redundancies from each image by referencing one or more original images corresponding to the one or more coded images, and one or more motion compensators performing motion compensation on each image using the motion vectors obtained by the motion estimation to generate residual images.  
         [0055]     The encoder may further comprise a decoding unit reconstructing original images by decoding the coded images, wherein the referenced images corresponding to the coded images are obtained by decoding the coded images by the decoding unit.  
         [0056]     The temporal redundancy remover may further comprise one or more intra-predictors removing temporal redundancies from each image with reference to the image itself.  
         [0057]     The spatial redundancy remover may comprise one or more wavelet transform units performing a wavelet transform on the original-resolution residual images and the lower-resolution residual images to respectively generate the original-resolution transformed images and the lower-resolution transformed images and a transformed image combiner that unifies the lower-resolution transformed images into the original-resolution transformed images to generate unified original-resolution transformed images.  
         [0058]     According to still another aspect of the present invention, there is provided a scalable video decoding method comprising extracting coded image data from a bitstream, and separating and inversely quantizing the coded image data to generate unified original-resolution transformed images and lower-resolution transformed images corresponding to the unified original-resolution transformed images, performing an inverse wavelet transform on each of the unified original-resolution transformed images and its lower-resolution transformed images to generate unified original-resolution residual images and lower-resolution residual images, and performing inverse motion compensation on the lower-resolution residual images using lower-resolution motion vectors extracted from the bitstream to reconstruct lower-resolution images and reconstructing original-resolution images from the unified original-resolution residual images using original-resolution motion vectors extracted from the bitstream.  
         [0059]     The generated lower-resolution transformed images may include unified first low-resolution transformed images and second low-resolution transformed images corresponding to the unified first low-resolution transformed images. Also, the unified original-resolution images, the unified first low-resolution transformed images, and the second low-resolution transformed images are subjected to the inverse wavelet transform to respectively generate unified original-resolution residual images, unified first low resolution residual images, and second low resolution residual images, and inverse motion compensation is performed on the second low resolution residual images using second low-resolution motion vectors obtained from the bitstream to reconstruct second low-resolution images and then first low-resolution images are reconstructed from the unified first low resolution residual images using first low-resolution motion vectors extracted from the bitstream.  
         [0060]     The performing of the inverse motion compensation may comprise reconstructing lower-resolution images by performing inverse motion compensation on the lower-resolution residual images using the lower-resolution motion vectors, generating original-resolution high frequency residual image from each of the unified original-resolution residual images using the lower-resolution residual images, generating each of original-resolution residual images using referred images created by the inverse motion compensation of the original resolution using the original-resolution motion vectors and the reconstructed lower-resolution images, and reconstructing original-resolution images by performing inverse motion compensation on the original-resolution residual images using the original-resolution motion vectors.  
         [0061]     According to a further aspect of the present invention, there is provided a scalable video decoding method comprising extracting coded image data from a bitstream, and separating and inversely quantizing the coded image data to generate original-resolution high-frequency transformed images and lower-resolution transformed images corresponding to the original-resolution high-frequency transformed images, performing an inverse wavelet transform on each of the original-resolution high-frequency transformed images and its lower-resolution transformed images to generate original-resolution high frequency residual images and lower-resolution residual images, and performing inverse motion compensation on the lower-resolution residual images using lower-resolution motion vectors extracted from the bitstream to reconstruct lower-resolution images, generating original-resolution residual images from the original high frequency residual images using the reconstructed lower-resolution images, and performing inverse motion compensation on the original-resolution residual images using original-resolution motion vectors extracted from the bitstream to reconstruct original-resolution images.  
         [0062]     According to another aspect of the present invention, there is provided a scalable video decoder comprising a bitstream interpreter interpreting a received bitstream and extracting coded image data and motion vectors for an original resolution and lower resolution levels from the bitstream, an inverse quantizer separating and inversely quantizing the coded image data to generate unified original-resolution transformed images and lower-resolution transformed images corresponding to the unified original-resolution transformed images, an inverse spatial redundancy remover performing an inverse wavelet transform on each of the unified original-resolution transformed images and its lower-resolution transformed images to generate unified original-resolution residual images and lower-resolution residual images, and an inverse temporal redundancy remover performing inverse motion compensation on the lower-resolution residual images using the lower-resolution motion vectors extracted from the bitstream to reconstruct lower-resolution images and reconstructing original-resolution images from the unified original-resolution residual images using the reconstructed lower-resolution images and the original-resolution motion vectors extracted from the bitstream.  
         [0063]     The inverse temporal redundancy remover may comprise one or more inverse motion compensators performing inverse motion compensation on each of the residual images using the original-resolution or lower-resolution motion vectors, one or more inverse low-pass filters increasing the resolution levels of the images, and one or more low-pass filters decreasing the resolution levels of the images. Here, the lower-resolution residual images are reconstructed into lower-resolution images while the lower-resolution residual images subjected to the inverse low-pass filtering are compared with the unified original-resolution residual images to generate original-resolution high frequency residual images, original-resolution referred images obtained by low pass filtering a referred frame created by inverse motion compensation for the original resolution are compared with the reconstructed low pass filtered images, and the images subjected to the comparing are combined with the original-resolution high frequency residual images to generate original-resolution residual images that are then subjected to inverse motion compensation and reconstructed into original-resolution images.  
         [0064]     According to another aspect of the present invention, there is provided a scalable video decoder comprising a bitstream interpreter interpreting a received bitstream and extracting coded image data and motion vectors for an original resolution and lower resolution levels from the bitstream, an inverse quantizer separating and inversely quantizing the coded image data to generate original-resolution high-frequency transformed images and lower-resolution transformed images corresponding to the original-resolution high-frequency transformed images, an inverse spatial redundancy remover performing an inverse wavelet, transform on each of the original-resolution high-frequency transformed images and its lower-resolution transformed images to generate original-resolution high frequency residual images and lower-resolution residual images, and an inverse temporal redundancy remover performing inverse motion compensation on the lower-resolution residual images using the lower-resolution motion vectors to reconstruct lower-resolution images, generating original-resolution residual images from the original-resolution high frequency residual images using the lower-resolution residual images, and performing inverse motion compensation on the original-resolution residual images using the original-resolution motion vectors to reconstruct original-resolution images. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0065]     The above and other features and advantages of the present invention will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings in which:  
         [0066]      FIG. 1A  is a schematic block diagram of a motion compensated temporal filtering (MCTF)-based scalable video encoder;  
         [0067]      FIG. 1B  is a schematic block diagram of an in-band scalable video encoder designed to perform a wavelet transform before temporal filtering;  
         [0068]      FIG. 2A  shows scalable video coding and decoding processes using a MCTF algorithm;  
         [0069]      FIG. 2B  shows scalable video coding and decoding processes using a successive temporal approximation and referencing (STAR) algorithm;  
         [0070]      FIG. 3  is a diagram for explaining wavelet-based video coding for supporting spatial scalability;  
         [0071]      FIG. 4  is a functional block diagram schematically showing the configuration of a scalable video encoder according to an embodiment of the present invention;  
         [0072]      FIG. 5  is a block diagram showing the detailed configuration of the S 1  shown in  FIG. 4 ;  
         [0073]      FIG. 6  illustrates various prediction modes for generating a referred image according to an embodiment of the present invention;  
         [0074]      FIG. 7  is a block diagram showing the detailed configuration of the spatial redundancy remover shown in  FIG. 4 ;  
         [0075]      FIG. 8  is a diagram for explaining a process for creating a unified transformed image with the original resolution;  
         [0076]      FIG. 9  is a detailed block diagram of an inverse quantizer according to a first embodiment of the present invention;  
         [0077]      FIG. 10  is a detailed block diagram of an inverse temporal redundancy remover according to a first embodiment of the present invention;  
         [0078]      FIG. 11  is a diagram showing a process of demultiplexing a coded image into images with respective resolution levels during inverse quantization according to a first embodiment of the present invention;  
         [0079]      FIG. 12  is a diagram showing a process of reconstructing an original image according to a first embodiment of the present invention;  
         [0080]      FIG. 13  is a detailed block diagram of an inverse quantizer according to a second embodiment of the present invention;  
         [0081]      FIG. 14  is a detailed block diagram of an inverse temporal redundancy remover according to a second embodiment of the present invention;  
         [0082]      FIG. 15  is a diagram showing a process of generating high frequency residual images after performing inverse quantization and inverse spatial redundancy removal according to a second embodiment of the present invention; and  
         [0083]      FIG. 16  is a functional block diagram schematically showing the configuration of a scalable video decoder according to an embodiment of the present invention. 
     
    
     DETAILED DESCRIPTION OF THE INVENTION  
       [0084]     The present invention will now be described more fully with reference to the accompanying drawings, in which exemplary embodiments of the invention are shown. While the present invention will be described with reference to a video coding scheme to generate a bitstream having three resolution levels, the invention will not be limited thereto. For the sake of convenience, the present invention describes coding and decoding of the highest-resolution image of Layer  1 , medium-resolution image of Layer  2 , and lowest-resolution image of Layer L 3 . In exemplary embodiments, coding and decoding of a frame (image) will be described.  
         [0085]      FIG. 4  is a functional block diagram schematically showing the configuration of a scalable video encoder according to an embodiment of the present invention.  
         [0086]     Referring to  FIG. 4 , a scalable video encoder according to an embodiment of the present invention obtains lower-resolution images O 2  and O 3  using a low-pass filter  402  extracting the lower-resolution image O 2  of Layer  2  from the original-resolution image O 1  and a low-pass filter  403  extracting the lower-resolution image O 3  of Layer  3  from the lower-resolution image O 2  of Layer  2 . In the illustrative embodiment, low-pass filtering is performed by downsampling using a wavelet 9-7 filter.  
         [0087]     A temporal redundancy remover removes temporal redundancies from the original-resolution image O 1 , and lower-resolution images O 2 , O 3  with the respective resolution levels in order to generate residual images E 1  through E 3  with the respective resolution levels. S 1   410 , S 2   420 , and S 3   430  in the temporal redundancy remover all have the same structure and remove temporal redundancies for the respective resolution levels. The detailed structure of the S 1   410  will be described later with reference to  FIG. 5 .  
         [0088]     Spatial redundancies are removed from the residual images E 1  through E 3  with the respective resolution levels by a spatial redundancy remover  440  and combined into a unified, transformed image W 1 . The detailed structure of the spatial redundancy remover  440  will be described later with reference to  FIG. 7 .  
         [0089]     A quantizer  450  quantizes the unified, transformed image W 1  to create a coded image Q 1 . A bitstream generator  455  generates a bitstream by combining the coded images obtained by encoding the input images with motion vectors MV 1 , MV 2 , and MV 3  for the respective resolution levels obtained by removing the temporal redundancies. The bitstream contains information about the coded images (coded image data), the motion vectors MV 1 , MV 2 , and MV 3 , and other necessary header information.  
         [0090]     Meanwhile, when a low frequency subband (L frame) is generated by updating a frame while removing temporal redundancies like in conventional motion compensated temporal filtering (MCTF)-based video coding, images referenced in removing the temporal redundancies are original images making up a video sequence. However, a video coding scheme based on unconstrained MCTF (UMCTF) or successive temporal approximation and referencing (STAR) does not include an update of A- or I-frames. In this successive coding algorithm, images referenced in removing temporal redundancies may be original images making up an input video sequence or images obtained by decoding coded images. In particular, in the latter case, coding and decoding processes form a single loop in a video encoder and are performed in an iterative fashion, which is called a “closed loop” scheme.  
         [0091]     In an open loop scheme where original images are referenced at an encoder side in removing temporal redundancies while decoded images are referenced at a decoder side in removing inverse temporal redundancies, a drift error tends to occur. In contrast to the open loop scheme, a closed loop scheme is not subjected to drift error since decoded images are referenced at both encoder and decoder sides. It should be noted that referenced images to be described below may be original images (uncoded images) or decoded images obtained by decoding coded images.  
         [0092]     A closed-loop scheme will now be described with reference to  FIG. 4 .  
         [0093]     Referring to  FIG. 4 , the coded image Q 1  is separated and inversely quantized by an inverse quantizer  460  to generate transformed images W 1  through W 3  with the respective resolution levels. The detailed structure of the inverse quantizer  460  will be described later with reference to  FIGS. 9 and 13 .  
         [0094]     The transformed images W 1  through W 3  with the respective resolution levels are then converted into residual images E 1  through E 3  with the respective resolution levels as they pass through an inverse spatial redundancy remover  470 . The residual images E 1  through E 3  with the respective resolution levels are converted into decoded images D 1  through D 3  with the respective resolution levels by an inverse temporal redundancy remover  480 . The decoded images D 1  through D 3  are stored in a buffer  490  and provided as referenced images for removing temporal redundancies from a future image. The detailed structure of the inverse temporal redundancy remover  480  will be described later with reference to  FIGS. 10 and 14 .  
         [0095]     Scalable video coding is performed in units of group of pictures (GOP) for temporal scalability. In a conventional MCTF scheme, MCTF is performed on all images in a GOP to generate one low frequency subband (L image) and a plurality of high frequency subbands (H images). In an UMCTF or STAR scheme, one image in a GOP is encoded as an A- or I-image without being subjected to MCTF while the remaining images are subjected to motion compensation with reference to one or a plurality of images to obtain residual images. The temporal redundancies are removed in blocks of predetermined size forming an image.  
         [0096]      FIG. 5  is a block diagram showing the detailed configuration of the S 1   410  shown in  FIG. 4 .  
         [0097]     Referring to  FIG. 5 , a motion estimator  512  performs motion estimation on the input image O 1  by referencing one or a plurality of images stored in a multi-image referencer  511  in order to generate motion vectors that are then provided to a motion compensator  513 . The motion compensator  513  creates a referred frame R 1  using the input image O 1  and the one or the plurality of referenced images. A comparator  515  compares the input image O 1  with the referred frame R 1  to generate a residual image E 1 . All blocks in the referred frame R 1  used for deriving the residual image E 1  from the input image O 1  may be obtained using inter-prediction in the motion compensator  513 . Alternatively, some or all of the blocks in the referred frame R 1  may be obtained by performing intra-prediction with reference to the input image O 1  in an intra-predictor  514 .  
         [0098]      FIG. 6  shows various prediction modes that can be chosen for creating a referred image according to an embodiment of the present invention.  
         [0099]     A scalable video encoder of the present invention may use only forward prediction like a conventional MCTF-based encoder, backward and bi-directional predictions like an UMCTF- or STAR-based encoder, or an intra-prediction mode like in a STAR algorithm.  
         [0100]     First, a choice of inter-prediction modes will be described.  
         [0101]     Since the present invention allows referencing of a plurality of images, it is easy to perform forward, backward, and bi-directional predictions. Inter-prediction may employ a well-known hierarchical variable size block matching (HVSBM) algorithm or fixed block size motion estimation like in the illustrative embodiment. When E(k, −1), B(k, −1), and E(k, *) respectively denote sums of absolute difference (SADs) from forward, backward, and bi-directional predictions of a k-th block, and B(k, −1), B(k, +1), and B(k, *) respectively denote a total number of bits to be allocated for quantizing forward, backward, and bi-directional motion vectors for the k-th block, costs C f , C b , and C bi  for forward, backward, and bi-directional prediction modes are defined by Equation (1): 
 
 C   f   =E ( k, − 1)+λ B ( k, − 1), 
 
 C   b   =E ( k,  1)+λ B ( k,  1), 
 
 C   bi   =E ( k,  *)+λ{ B ( k,  *)}  (1) 
 
 where λ is a Lagrange coefficient used to control balance between motion bits and texture (image) bits. Since a final bit rate is not known in a scalable video encoder, λ may be selected according to characteristics of a video sequence and a bit rate that are mainly used in a target application. An optimal inter-estimation mode can be determined for each macroblock based on minimum cost obtained using Equation (1). 
 
         [0102]     Next, a choice of an intra-prediction mode will be described.  
         [0103]     In some video sequences, scenes change very fast. In an extreme case, a frame that has no temporal redundancy compared to adjacent frames may be found. To handle such frame, a concept of a macroblock obtained through intra-estimation that is used in a standard hybrid encoder is employed. Generally, an open-loop codec cannot use adjacent macroblock information due to estimation drift. However, a hybrid codec can use an intra-estimation mode. In the present embodiment, DC prediction is used to perform intra-prediction. In the DC prediction mode, a block is intra-predicted by DC values of its Y, U, and V components. If cost for the intra-prediction mode is lower than cost for the best inter-prediction mode mentioned above, the intra-prediction mode is selected. In this case, the difference between the original pixel and DC value is then coded, and the differences between the three DC values are coded instead of motion vectors.  
         [0104]     Cost C i  for intra-prediction mode is defined by Equation (2): 
 
 C   i   =E ( k,  0)+λ B ( k,  0)   (2) 
 
 where E(k, 0) is a SAD (differences between the original luminance value and DC values) for intra-prediction of a k-th block and B(k, 0) is a total number of bits for coding the three DC values. 
 
         [0105]     If the cost C i  is lower than those defined by Equation (1), the given block is encoded using the intra-prediction mode.  
         [0106]     As described above, the spatial redundancy remover  440  removes spatial redundancies from the residual images E 1  through E 3  with the respective resolution levels from which temporal redundancies have been removed, which will be described with reference to  FIG. 7 .  
         [0107]      FIG. 7  is a detailed block diagram of the spatial redundancy remover  440 .  
         [0108]     The spatial redundancy remover  440  includes first through third wavelet transform units  741  through  743  performing an inverse wavelet transform on the residual images E 1  through E 3  with the respective resolution levels to remove spatial redundancies and a multiplexer (MUX)  745  combining transformed images W H   1 , W H   2 , and W L+H   3  with the respective resolution levels subjected to the inverse wavelet transform by the first through third wavelet transform units  741  through  743  into a single unified transformed image W L+H   1 .  
         [0109]      FIG. 8  is a diagram for explaining a process for creating a unified transformed image with the original resolution.  
         [0110]     Referring to  FIG. 8 , the residual images E 1  through E 3  with the respective resolution levels are subjected to the wavelet transform to generate transformed images. Each of the transformed images are decomposed into one low frequency transformed image L that is a reduced size image very similar to the untransformed image and three high-frequency transformed images H. The low frequency transformed image of layer L 2  is first replaced with the transformed image of layer L 3  to create a unified transformed image of L 2  (S 1 ), and then the low frequency transformed image of layer L 1  is replaced with the unified transformed image of L 2  (S 2 ) to create a unified transformed image of L 1  (S 3 ). Alternatively, instead of creating the unified transformed image of L 1 , the unified transformed image of L 2  and the transformed image of L 1  may be quantized to generate a bitstream. However, coding efficiency is degraded compared to that provided by the former method since the low frequency transformed image of L 1  having spatial redundancy needs to be encoded.  
         [0111]     The unified transformed image of L 1  is quantized to generate a coded image, and coded image data associated with coded images obtained by encoding a plurality of images in a video sequence is contained in a bitstream.  
         [0112]     A process for reconstructing a decoded image from a coded image in a decoder or closed loop encoder will now be described. A process for decoding coded images according to a first embodiment of the present invention is performed as follows:  
         [0113]     1. First, a coded low frequency image is separated from the coded image Q 1  of L 1  to obtain a coded high frequency image Q H   1  of L 1  and a coded image Q 2  of L 2 . In the same manner, the coded image Q 2  of L 2  is separated to obtain a coded high frequency image of L 2  and a coded image Q 3  of L 3 .  
         [0114]     2. A process for obtaining a decoded image D 3  of L 3  from the coded image Q 3 (=Q L+H   3 ) of L 3  is defined by Equation (3): 
 
 D   3   =DQ   —   IT[Q   L+H   3   ]+R   3   =E   L+H   3   +R   3    (3) 
 
 where DQ_IT[ ] is an inverse quantization function or inverse wavelet transform function and R 3  is a referred image of L 3  whose motion is estimated by referencing a plurality of previously decoded images. 
 
         [0115]     3. Then, to obtain a decoded image D 2  of L 2 , a low frequency residual image E L   2  of L 2  replaced by the transformed image W 3  of L 3  during encoding is reconstructed using a process defined by Equation (4): 
 
 E   L   2   =D   3 −DOWN[ R   2 ]  (4) 
 
 where DOWN[ ] and R 2  respectively represent a downsampling function and a referred image of L 2  whose motion is estimated by referencing a plurality of previously decoded images. 
 
         [0116]     The low frequency residual image E L   2  of L 2  can be reconstructed using Equation (4) since DOWN[D 2 ]−DOWN[R 2 ]=DOWN[E 2 ] where DOWN[D 2 ] is D 3  and DOWN[E 2 ] is E L   2 .  
         [0117]     Using the low frequency residual image E L   2 , a residual image E L+H   2  of L 2  is given by Equation (5): 
 
 E   L+H   2 =UP[ E   L   2   ]+E   H   2    (5) 
 
 where UP[ ] denotes an upsampling function. Finally, the decoded image D 2  of L 2  is defined by Equation (6): 
 
 D   2   =E   L+H   2   +R   2    (6) 
 
         [0118]     In the same manner, a decoded image D 1  of L 1  can be obtained using Equations (7) through (9): 
 
 E   L   1   =D   2 −DOWN[R 1 ]  (7) 
 
         [0119]     The low frequency residual image E L   1  of L 1  can be restored using Equation (7) since DOWN[D 1 ]−DOWN[R 1 ]=DOWN[E 1 ] where DOWN[D 1 ] is D 2  and DOWN[E 1 ] is E L   1 .  
         [0120]     Using the low frequency residual image E L   1 , a residual image E L+H   1  of L 1  is given by Equation (8): 
 
 E   L+H   1 =UP[ E   L   1   ]+E   H   1    (8) 
 
 Eventually, the decoded image D 1  of L 1  can be obtained using Equation (9): 
 
D 1   =E   L+H   1   +R   1    (9) 
 
         [0121]     While the resolution of an image has been described above in three resolution levels for L 1  through L 3 , the above-mentioned method can also apply to the image having three or more resolution levels.  
         [0122]     The process for decoding coded images according to the first embodiment of the present invention will now be described with reference to  FIGS. 9-12 .  FIGS. 9 and 10  are respectively detailed block diagrams of an inverse quantizer  460  and an inverse temporal redundancy remover  480  according to a first embodiment of the present invention.  
         [0123]     Referring to  FIG. 9 , the inverse quantizer  460  includes a demultiplexer (DEMUX)  964  separating a unified coded image into coded images with the respective resolution levels and first through third inverse quantizers  961  through  963  inversely quantizing the coded images with the respective resolution levels.  
         [0124]     The DEMUX  964  separates Q L+H   3  from a unified coded image Q while separating the remaining Q H   2 +Q H   1  into Q H   2  and Q H   1 . Q L+H   3  may be separated from the unified coded image Q, followed by separation of Q H   2 +Q H   1 . Otherwise, after separation Q H   1 , Q H   2 +Q L+H   3  may be separated into Q H   2  and Q L+H   3 .  
         [0125]     The separated Q L+H   3 , Q H   2 , and Q H   1  are respectively subjected to inverse quantization by the third, second, and first inverse quantizers  963 ,  962 , and  961  to generate a transformed image W L+H   3  of L 3 , a high-frequency transformed image W H   2  of L 2 , and a high-frequency transformed image W H   1  of L 1 .  
         [0126]     The transformed images W H   1 , W H   2 , and W L+H   3  with the respective resolution levels for L 1 , L 2 , and L 3  are input to the inverse spatial redundancy remover  470  to produce residual images E H   1 , E H   2 , and E L+H   3  with the respective resolution levels for L 1 , L 2 , and L 3  that is then input to the inverse temporal redundancy remover  480  to generate decoded images D 1 , D 2 , and D 3  with the respective resolution levels for L 1 , L 2 , and L 3 .  
         [0127]     More specifically, the decoded image D 3  is obtained by adding the residual image E L+H   3  to referred image R 3 . The decoded image D 3  is used to produce the decoded image D 2 . Specifically, after calculating E L   2  by subtracting the result obtained after downsampling referred image R 2  from the decoded image D 3 , the residual image E L+H   2  is calculated by adding residual image E H   2  to the result obtained by upsampling the residual image E L+H   2 . Then, the decoded image D 2  is obtained by adding the residual image E L+H   2  to referred image R 2 . Similarly, the decoded image D 2  is used to produce the decoded image D 1 . That is, after calculating E L   1  by subtracting the result obtained after downsampling referred image R 1  from the decoded image D 2 , the residual image E L+H   1  is calculated by adding residual image E H   1  to the result obtained by upsampling the residual image E L   1 . Then, the decoded image D 1  is obtained by adding the residual image E L+H   1  to referred image R 1 . The referred images R 1 , R 2 , and R 3  are respectively obtained by performing motion estimation using motion vectors for the resolution levels L 1 , L 2 , and L 3 . In this way, the present invention provides a high quality image at each resolution using the highest resolution image and motion vectors for the respective resolution levels.  
         [0128]      FIG. 11  is a diagram showing an inverse quantization process in which a unified coded image is decomposed into the lowest resolution coded image and high frequency coded image with the higher resolution levels according to a first embodiment of the present invention, and  FIG. 12  is a diagram showing a process for reconstructing an original image, i.e., decoded image D 2  using the decoded image D 3  according to a first embodiment of the present invention.  
         [0129]     While coded images with the respective images can be obtained by the inverse quantization process according to the first embodiment of the present invention, it may be actually difficult to separate Q L+H   3  from a unified coded image Q while separating the remaining Q H   2 +Q H   1  into Q H   2  and Q H   1 . In this case, coded images Q 2  and Q 3  may be obtained from the coded image Q (=Q 1 ) because a scalable video stream is inherently separated into images according to resolution. That is, while the method according to the first embodiment can apply to a bitstream generated to separate a high frequency coded image, the latter method can apply to other common bitstreams, which will be described below with reference to  FIGS. 13 and 14 .  
         [0130]      FIGS. 13 and 14  are respectively detailed block diagrams of an inverse quantizer  460  and an inverse temporal redundancy remover  470  according to a second embodiment of the present invention.  
         [0131]     While it is easy to obtain decoded image D 3  using coded image Q 3 , only images similar to decoded images D 1  and D 2  can be obtained using unified coded images Q 1  and Q 2  because low frequency components in the coded images Q 1  and Q 2  originate from coded images of L 2  and L 3 , respectively. Thus, the basic idea of the present embodiment is that the decoded images D 1 , and D 2  are obtained in the same manner as described in the first embodiment after obtaining residual images E H   1  and E H   2  from the coded images Q 1  and Q 2 .  
         [0132]     Referring to  FIG. 13 , the inverse quantizer  460  includes a DEMUX  1369  separating a unified coded image into coded images with the respective resolution levels and first through third inverse quantizers  1366  through  1368  generating unified transformed images from the unified coded images Q 1  through Q 3  with the respective resolution levels. The inverse quantizer  460  converts the unified coded images Q 1  through Q 3  into the unified transformed images W 1  through W 3 , respectively, which are then converted into unified residual image E L+H   3 +E H   2 +E H   1  of L 1 , unified residual image E L+H   3 +E H   2  of L 2 , and residual image E L+H   3  of L 3 .  
         [0133]     Referring to  FIG. 14 , a high frequency residual image E H   2  of L 2  is obtained by subtracting the result obtained after upsampling the residual image E L+H   3  of L 3  from the unified residual image E L+H   3 +E H   2  of L 2 . The upsampling operation is accomplished in order to adjust the resolution.  
         [0134]     In the same way, a high frequency residual image E H   1  of L 1  is obtained by subtracting the result obtained after upsampling the unified residual image E L+H   3 +E H   2  of L 2  from the unified residual image E L+H   3 +E H   2 +E H   1  of L 1 . Original images (decoded images) can be obtained by the process described in the first embodiment.  FIG. 15  shows a detailed process of obtaining the high frequency residual images E H   1  and E H   2 .  
         [0135]      FIG. 16  is a functional block diagram schematically showing the configuration of a scalable video decoder according to an embodiment of the present invention. Referring to  FIG. 16 , the scalable video decoder includes a bitstream interpreter  1610  receiving a bitstream and interpreting the received bitstream in order to extract unified coded image data and motion vectors for the respective resolution levels, an inverse quantizer  1620  performing inverse quantization on unified coded images contained in the unified coded image data to produce transformed images with the respective resolution levels, an inverse spatial redundancy remover  1630  producing residual images with the respective resolution levels from the transformed images with the respective resolution levels, and an inverse temporal redundancy remover  1640  reconstructing original images through inverse motion compensation using the motion vectors for the respective resolution levels.  
         [0136]     The detailed structures and operations of the inverse quantizer  1620 , the inverse spatial redundancy remover  1630 , and the inverse temporal redundancy remover  1640  are substantially the same as their counterparts in the scalable video encoder described above.  
         [0137]     In concluding the detailed description, those skilled in the art will appreciate that many variations and modifications can be made to the preferred embodiments without substantially departing from the principles of the present invention. Therefore, the disclosed preferred embodiments of the invention are used in a generic and descriptive sense only and not for purposes of limitation.  
         [0138]     According to the present invention, images with various resolution levels can be combined into a single image while providing high image quality across all resolution levels, thus enabling efficient video coding while fully taking advantage of spatial scalability.