Abstract:
A single base layer is formed by reducing information of input image data with at least a plurality of scalable factors, and a plurality of enhancement layers having information associated with a plurality of scalable factors are formed by using the base layer, thereby minimizing the amount of data to be transmitted and allowing the receiving side to arbitrarily select one of images with large information amounts based on various scalable factors. A system suited for image selection on the receiving side is also constructed.

Description:
BACKGROUND OF THE INVENTION  
         [0001]    1. Field of the Invention  
           [0002]    The present invention relates to an image processing apparatus and method and, more particularly, to an image processing apparatus capable of processing images with a plurality of different scalabilities by hierarchical coding.  
           [0003]    2. Related Background Art  
           [0004]    Conventionally, as a motion image encoding scheme, MPEG2 has been known. In this MPEG2 scheme, high image quality or a small amount of data to be transmitted can be selected with two scalabilities called spatial scalability and SNR (SN Ratio) scalability. The aim of spatial scalability in MPEG2 is to transmit one of two types of motion image information having different resolutions or image sizes. In general, only one type of image quality, i.e., one type of resolution, is selected, and data is transmitted with the selected resolution.  
           [0005]    SNR scalability in MPEG2 targets for transmission of one of two types of motion image information having the same resolution but having different code amounts, which are obtained by, for example, quantizing DCT coefficients with different quantization steps.  
           [0006]    In the following description of this specification, of two pieces of motion image information with different data rates in each scalability, information with a smaller data amount will be referred to as a base layer, and information with a larger data amount will be referred as an enhancement layer.  
           [0007]    According to another known method, two types of images (base layer and enhancement layer) having different sizes (or resolutions) are simultaneously compressed/encoded by using spatial scalability, and the decoding apparatus chooses between reconstructing an image with low spatial resolution from the base layer and reconstructing an image with high resolution from the enhancement layer, depending on the performance of a decoding circuit, image display apparatus, and the like.  
           [0008]    Spatial scalability will be described with reference to FIG. 1 by taking HDTV and NTSC image signals as examples. When an original image is an HDTV image (1440×1152 pixels), a (720×576)-pixel image obtained by thinning out the original image data by ½ in both the X and Y directions will be referred to as a base layer. Image data obtained as a prediction (comparison) image, by encoding an image (expansion base layer) obtained by up-sampling the base layer with the same size as that of the original image, in addition to forward prediction (P) and bidirectional prediction (B) for the original image, will be referred to as an enhancement layer.  
           [0009]    According to still another known method, two types of images (base layer and enhancement layer) having different code amounts (more specifically, quantization steps) are simultaneously compressed by using SNR scalability, and the decoding apparatus reconstructs an image with a low rate (low image quality) from the base layer. The decoding apparatus then reconstructs an image with a high rate (high image quality) from both the base layer and the enhancement layer.  
           [0010]    SNR scalability will be briefly described below. FIG. 2 shows a conceptual rendering of SNR scalability. Two different quantization coefficients are applied to the same image to generate different pieces of compressed image information with different compression ratios from the same image. In this case, image information with a larger compression ratio, i.e., image information with a low bit rate and low image quality, is defined as a base layer, and image information with a smaller compression ratio, i.e., image information with a high bit rate and high image quality, is defined as an enhancement layer. On the decoding apparatus side, the image information of the base layer and the image information of the enhancement layer are added together to obtain an image with a small compression ratio and high image quality.  
           [0011]    As described above, when spatial scalability or SNR scalability is to be used in MPEG2, only one of these scalabilities can be selected. A conventional encoding apparatus cannot therefore encode image data by using a plurality of scalabilities at once. In other words, only one type of scalability can be designated in one image sequence. Upon reception of image information encoded by using any one of the scalabilities, the decoding apparatus has only the following two types of image qualities as choices: a low-quality image decoded from only the base layer and a high-quality image obtained by synthesizing an image decoded from the base layer and an image decoded from the enhancement layer.  
           [0012]    In the prior art, therefore, selection of the image quality or decoding rate cannot be done in accordance with the performance of the decoding apparatus or receiver&#39;s need.  
           [0013]    Obviously, the above problem is common to so-called hierarchical encoding performed for two or more types of factors as well as to MPEG2.  
         SUMMARY OF THE INVENTION  
         [0014]    It is an object of the present invention to solve the above problem.  
           [0015]    It is another object of the present invention to provide an image processing apparatus which allows selection of an arbitrary scalable factor on the receiving end and can minimize the amount of data transmitted.  
           [0016]    In order to achieve the above objects, according to one preferred aspect of the present invention, there is provided an image processing apparatus comprising (a) means for forming a single base layer by reducing information of input image data with at least a plurality of scalable factors, and (b) means for forming a plurality of enhancement layers having information associated with a plurality of scalable factors by using the base layer.  
           [0017]    According to another aspect of the present invention, there is provided an image processing apparatus comprising (a) reconstruction means for extracting data of a base layer obtained from input data by reducing information of image data by using at least a plurality of scalable factors, and reconstructing an image signal from the extracted data of the base layer, and (b) forming means for extracting data of a plurality of enhancement layers having information associated with a plurality of scalable factors from the input data, and forming a plurality of image signals having information associated with the plurality of factors in an amount larger than that of the reconstructed image signal of the base layer from the data of the base layer and the data of the plurality of enhancement layer.  
           [0018]    The above and other objects, features, and advantages of the present invention will be apparent from the following detailed description in conjunction with the accompanying drawings and the appended claims. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0019]    [0019]FIG. 1 is a diagram for explaining spatial scalability;  
         [0020]    [0020]FIG. 2 is a diagram for explaining SNR scalability;  
         [0021]    [0021]FIG. 3 is a block diagram schematically showing an encoding apparatus according to an embodiment of the present invention;  
         [0022]    [0022]FIG. 4 is a diagram for explaining time scalability;  
         [0023]    [0023]FIG. 5 is a block diagram showing the details of the main part of the arrangement in FIG. 3;  
         [0024]    [0024]FIG. 6 is a conceptual rendering for explaining the operation of the arrangement in FIG. 5;  
         [0025]    [0025]FIG. 7 is a block diagram showing the arrangement of a decoding apparatus corresponding to the encoding apparatus in FIG. 3;  
         [0026]    [0026]FIG. 8 is a block diagram showing the details of the main part of the arrangement in FIG. 7;  
         [0027]    [0027]FIG. 9 is a view showing an example of the arrangement of a system using the apparatus in FIG. 7;  
         [0028]    [0028]FIG. 10 is a flow chart for explaining the operation of the system in FIG. 9; and  
         [0029]    [0029]FIG. 11 is a view showing how images are displayed on the system in FIG. 9. 
     
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS  
       [0030]    A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.  
         [0031]    [0031]FIG. 3 is a block diagram showing the schematic arrangement of an encoding apparatus according to an embodiment of the present invention. FIG. 4 is a block diagram showing the schematic arrangement of a decoding apparatus corresponding to this encoding apparatus. Both the apparatuses handle interlace image signals such as NTSC signals. In this embodiment, both spatial scalability and time scalability are used.  
         [0032]    Time scalability will be described below with reference to FIG. 4. Referring to FIG. 4, an image group obtained by averaging original images every two frames and performing frame thinning-out will be referred to as a base layer. An image group obtained by predicting the original images of all the frames from frames in the base layer which are adjacent to each other along the time axis and frames in all the frames which are adjacent to each other along the time axis, and encoding the images by calculating prediction differences will be referred to as an enhancement layer. In this manner, the base layer with lower resolution in the time axis direction can be obtained, together with the enhancement layer with higher resolution in the time axis direction, which is obtained by using the base layer.  
         [0033]    The arrangement shown in FIG. 3 will be described first. This arrangement includes a conversion circuit  10  for converting RGB image data to YCbCr image data with 4:2:0, a scalability selection circuit  12  for selecting use or nonuse of scalability for the output data from the conversion circuit  10  and a mode to be used, a spatial scalability enhancement layer generation circuit  14  for generating the data of an enhancement layer corresponding to spatial scalability from the output from the scalability selection circuit  12  when it selects the use of scalability, a common base layer generation circuit  16  for generating the data of a base layer common to spatial and time scalabilities from the output from the scalability selection circuit  12  when it selects the use of scalability, and a time scalability enhancement layer generation circuit  18  for generating the data of an enhancement layer corresponding to time scalability from the output from the scalability selection circuit  12  when it selects the use of scalability.  
         [0034]    This arrangement also includes a block forming processing circuit  20  for dividing the outputs from the generation circuits  14 ,  16 , and  18  and the output from the scalability selection circuit  12  upon selection of the nonuse of scalability into blocks, each consisting of n (n≧2) pixels, in unit of frames or fields in the horizontal and vertical directions, combining the blocks into macroblocks for Y, Cb, and Cr, respectively, and more specifically, a macroblock consisting of a blocks, a macroblock consisting of b blocks, and a macroblock consisting of c blocks, and outputting the respective macroblocks. The block forming processing circuit  20  includes four block forming circuits  20   a ,  20   b ,  20   c , and  20   d  corresponding to the outputs from the generation circuits  14 ,  16 , and  18  and the output from the scalability selection circuit  12  upon selection of the nonuse of scalability.  
         [0035]    This arrangement further includes an encoding circuit  22  for compressing/encoding the YCrCb block data output from the block forming processing circuit  20  in units of macroblocks by an encoding scheme known as MPEG2. More specifically, the encoding circuit  22  compresses/encodes the image data from the block forming processing circuit  20  by using a predictive encoding scheme (selection of intra-coded picture (I-picture)/inter-coded picture (P- or B-picture)), orthogonal transform (DCT), quantization, and variable-length coding.  
         [0036]    The RGB/YCbCr conversion circuit  10  converts R, G, and B image signals, each consisting of 8 bits, into Y, Cb, Cr signals, each consisting of 8 bits, with 4:2:0.  
         [0037]    When scalability is to be used in encoding, the scalability selection circuit  12  supplies the output from the conversion circuit  10  to the generation circuits  14 ,  16 , and  18 . The generation circuits  14 ,  16 , and  18  then generate a spatial scalability enhancement layer, a common base layer, and a time scalability enhancement layer, respectively, and supply them to the block forming processing circuit  20 .  
         [0038]    The block forming processing circuit  20  breaks up the input image data into n-pixel blocks in units of frames or fields in the horizontal and vertical directions, combines the blocks into macroblocks for Y, Cb, and Cr, respectively, and more specifically, a macroblock consisting of a blocks, a macroblock consisting of b blocks, and a macroblock consisting of c blocks, and outputs them. As described above, the block forming processing circuit  20  includes the block forming circuits  20   a ,  20   b ,  20   c , and  20   d , four in total, for the respective layers and the output in the case of the nonuse of scalability.  
         [0039]    The block forming circuits  20   a  to  20   c  of the block forming processing circuit  20  form the respective layer data from the generation circuits  14 ,  16 , and  18  into blocks in the manner described above, and supply them to the encoding circuit  22 . The encoding circuit  22  first performs prediction processing upon selection of a predictive coding scheme (intra-coded picture (I-picture)/inter-coded picture (P- or B-picture)), and then compresses/encodes the output data from the block forming circuits  20   a  to  20   c  by orthogonal transform (DCT), quantization, and variable-length encoding.  
         [0040]    When no scalability is to be used (normal mode), the scalability selection circuit  12  supplies the output from the conversion circuit  10  to the block forming circuit  20   d  of the block forming processing circuit  20  without any change. The encoding circuit  22  first performs prediction processing for the block data from the block forming circuit  20   d  upon selection of a predictive coding scheme ((intra-coded picture (I-picture)/inter-coded picture (P- or B-picture)), and then compresses/encodes the resultant data by orthogonal transform (DCT), quantization, and variable-length coding.  
         [0041]    A spatial scalability enhancement layer, a common base layer, a time scalability enhancement layer in this embodiment will be described further in detail below with reference to FIGS. 5 and 6.  
         [0042]    [0042]FIG. 5 is a block diagram showing the details of a main part including the circuits  14 ,  16 , and  18  in FIG. 3. Referring to FIG. 5, circuits  14   a ,  14   b ,  14   c , and  14   d  are included in the circuit  14  in FIG. 3, a circuit  16   a  is included in the circuit  16  in FIG. 3, and circuits  18   a ,  18   b , and  18   c  are included in the circuit  18  in FIG. 3.  
         [0043]    As shown in FIG. 6, when an original image group with high time resolution and high spatial resolution is input from the scalability selection circuit  12  in FIG. 3, low-pass filtering is performed first along the time axis to average two adjacent frames, and one of the two adjacent frames is thinned out, thereby reducing the time resolution of the original image group to ½. The pixel thinning-out circuit  18   a  band-limits an original image by a two-dimensional low-pass filter, and then performs pixel thinning-out by sub-sampling or the like to reduce the spatial resolution of the original image group to ½.  
         [0044]    The frame and pixel thinning-out circuit  16   a  continuously performs the above two processes to reduce both the spatial and time resolutions of the original image, thereby obtaining the common base layer shown in FIG. 6. In this case, to reduce the circuit size, the circuit  16   a  may supply the output from the circuit  14   a  to a pixel thinning-out circuit (not shown) or the output from the circuit  18   a  to a frame thinning-out circuit (not shown).  
         [0045]    The image group of the common base layer is equal in frame rate to the image group output from the frame thinning-out circuit  14   a , and is different in the number of pixels of each frame therefrom. For this reason, the image group of the common base layer is supplied to the image interpolation circuit  14   c  to become equal in the number of pixels to the output from the circuit  14   a . The image group obtaining by simply increasing the number of pixels of each image of the common base layer will be referred to as an expansion layer. The prediction circuit  14   d  predicts an output from the frame thinning-out circuit  14   a  on the basis of the image of the expansion layer. The difference circuit  14   b  calculates the difference between this prediction value and the output from the frame thinning-out circuit  14   a.    
         [0046]    In this case, prediction may be simply performed by only calculating the difference between an image of the expansion layer and an output from the frame thinning-out circuit  14   a , which temporally correspond to each other as shown in FIG. 6. However, prediction may also be performed by using a plurality of images of the expansion layer.  
         [0047]    The difference value output from the difference circuit  14   b  is supplied as the image group of the spatial enhancement layer to the block forming circuit  20   a . In addition, the image group of the common base layer is supplied to the block forming circuit  20   b.    
         [0048]    The image output from the pixel thinning-out circuit  18   a  is equal in the number of pixels to the image group of the common base layer, and hence prediction can be performed by the prediction circuit  18   b , as shown in FIG. 5. The circuits  18   a ,  18   b , and  18   c  calculate the difference between the prediction image and the image from the circuit  18   a . The output from the circuit  18   c  is supplied as a time enhancement layer to the block forming circuit  20   c.    
         [0049]    In this embodiment, in each prediction process indicated by hatching in FIG. 6, after image data is subjected to the DCT and quantization, a local decoded value is calculated from the resultant value, and a prediction difference is calculated on the basis of the decoded value. That is, this process is performed in the encoding circuit  22  in FIG. 3. As described above, in this embodiment, a time enhancement layer and a spatial enhancement layer are generated from a raster image before block formation, i.e., the DCT. However, a time enhancement layer and a spatial enhancement layer can also be generated by calculating prediction values using values obtained by temporarily and locally decoding data obtained by quantizing DCT coefficient after the DCT. In this case, each circuit for the prediction process indicated by hatching and each difference circuit may be combined into one difference circuit.  
         [0050]    In addition, in the above embodiment, only a spatial enhancement layer and a time enhancement layer are prepared. However, an SN enhancement layer may be prepared from the above common base layer. Furthermore, two arbitrary layers may be prepared from these three enhancement layers, or a plurality of enhancement layers each generated by combining two arbitrary layers may be prepared.  
         [0051]    According to the apparatus on the encoding side, which has the above arrangement, a plurality of scalabilities can be arbitrarily selected on the decoding side to meet various user&#39;s demands. In addition, since a common base layer is used to generate each enhancement layer, the circuit configuration on the transmitting side can be minimized. Furthermore, the amount of data to be transmitted can be minimized.  
         [0052]    The arrangement shown in FIG. 7 will be described next. This arrangement includes a decoding apparatus  30  for decoding MPEG2 bit stream data from the encoding apparatus in FIG. 3, a control information detection circuit  32  for reading the header information of the MPEG2 bit stream data from the encoding apparatus in FIG. 3 and controlling the overall operation of the decoding apparatus in FIG. 7, including the decoding processing in the decoding apparatus  30 , and a selection and rastering circuit  34  for rastering the image data decoded by the decoding apparatus  30 , and selecting the subsequent process on the basis of the scalability flag detected by the control information detection circuit  32 .  
         [0053]    A common base layer reconstruction circuit  36  reconstructs the data of a common base layer from the output from the selection and rastering circuit  34  which corresponds to the output from the common base layer generation circuit  16  of the encoding apparatus in FIG. 3.  
         [0054]    A spatial enhancement layer reconstruction circuit  38  refers to the image data of the expansion layer obtained by the common base layer reconstruction circuit  36  to reconstruct an image with high spatial resolution from the data of the spatial scalability enhancement layer on the basis of the output from the selection and rastering circuit  34  which corresponds to the output from the spatial scalability enhancement layer generation circuit  14  of the encoding apparatus in FIG. 3.  
         [0055]    A time enhancement layer reconstruction circuit  40  refers to the information input to the common base layer reconstruction circuit  36  which corresponds to the output from the common base layer generation circuit  16  to reconstruct an image with high time resolution from the data of the time scalability enhancement layer on the basis of the output from the selection and rastering circuit  34  which corresponds to the output from the time scalability enhancement layer generation circuit  18  of the encoding apparatus in FIG. 3.  
         [0056]    A video selection device  46  selects the video signal output from the spatial enhancement layer reconstruction circuit  38 , the video signal from the time enhancement layer reconstruction circuit  40 , the video signal output from the common base layer reconstruction circuit  36 , or the video signal output from the selection and rastering circuit  34 . As described above, when scalability is used, the video selection device  46  can select an image signal with low time/spatial resolution, an image signal with high time resolution, or an image signal with high spatial resolution.  
         [0057]    The operation of the decoding apparatus in FIG. 7 will be described. A transmitted (recorded/reproduced) bit stream is input to the decoding apparatus  30 . The header information of this bit stream is supplied to the control information detection circuit  32 . The control information detection circuit  32  decodes the encoding condition from the header information, and supplies a control signal corresponding to the decoded contents to the decoding apparatus  30  and the selection and rastering circuit  34 .  
         [0058]    The decoding apparatus  30  decodes (variable-length decoding, inverse quantization, and inverse DCT) the received data in accordance with the coding scheme (interframe/intraframe coding) designated by the control signal from the control information detection circuit  32 , and supplies the resultant data to the selection and rastering circuit  34 . The selection and rastering circuit  34  rasters the image data from the decoding apparatus  30 , and supplies the output from the decoding apparatus  30  to the reconstruction circuits  36 ,  38 , and  40  when scalability is used, and to the video selection device  46  when no scalability is used (normal mode), in accordance with the control information from the control information detection circuit  32 . In the former case, the image data of the common base layer is supplied to the common base layer reconstruction circuit  36 , the data of the spatial scalability enhancement layer is supplied to the spatial enhancement layer reconstruction circuit  38 , and the data of the time scalability enhancement layer is supplied to the time enhancement layer reconstruction circuit  40 . These data are then independently processed by the reconstruction circuits  36 ,  38 , and  40 .  
         [0059]    [0059]FIG. 8 is a block diagram for explaining the details of the common base layer reconstruction circuit  36 , the spatial enhancement layer reconstruction circuit  38 , and the time enhancement layer reconstruction circuit  40  in FIG. 7. Each layer reconstruction operation will be described in detail below with reference to FIGS. 6 and 8.  
         [0060]    Referring to FIG. 8, circuits  36   a  and  36   b  are included in the circuit  36  in FIG. 7, circuits  38   a ,  38   b , and  38   c  are included in the circuit  38  in FIG. 7, and circuits  40   a ,  40   b , and  40   c  are included in the circuit  40  in FIG. 7.  
         [0061]    The common base layer information extracted by the selection and rastering circuit  34  is supplied to the pixel interpolation circuit  36   a  to be subjected to pixel interpolation. As a result, an image having the same size as that of the original image is reconstructed. This operation is the same as that of the pixel interpolation circuit  14   c  in encoding operation. With this operation, an image of an expansion layer is obtained. Note that when the frame interpolation circuit  36   b  performs frame interpolation for the image of this expansion layer, an image signal equal in frame rate and the number of pixels to the original image can be obtained. As is obvious, both the time and spatial resolutions of the image signal output from the frame interpolation circuit  36   b  are low.  
         [0062]    In this case, the pixel interpolation circuit  36   a  serves as both a circuit for reconstructing an image (expansion layer), from this common base layer, which is to be combined with the spatial enhancement layer, and a circuit for reconstructing an image from the common base layer itself. That is, this circuit  36   a  is used for the two purposes. This can reduce the circuit size. If, however, the manner of interpolation by the circuit  36   a  is to be changed in accordance with the above two purposes, circuits for the two purposes may be independently prepared.  
         [0063]    The spatial enhancement layer information extracted by the selection and rastering circuit  34  is supplied to the addition circuit  38   b . The addition circuit  38   b  adds this information to the prediction information obtained by the prediction circuit  38   a  from the expansion base layer information obtained by the pixel interpolation circuit  36   a . Since the number of frames of the output from this addition circuit  38   b  is equal to that of the common base layer, the output from the circuit  38   b  is supplied to the frame interpolation circuit  38   c . The circuit  38   c  then performs frame interpolation for this output to obtain an image signal equal in frame rate and the number of pixels to the original image. This image information is low in time resolution but high in spatial resolution.  
         [0064]    The time enhancement layer extracted by the selection and rastering circuit  34  is supplied to the addition circuit  40   b  to be added to the prediction information obtained by the prediction circuit  40   a  from the common base layer information. Since the output from this addition circuit  40   b  is equal in the number of pixels to the common base layer, the output from the circuit  38   b  is supplied to the pixel interpolation circuit  40   c . The circuit  38   c  then performs spatial pixel interpolation for the output to obtain an image signal equal in frame rate and the number of pixels to the original image. This image information is low in spatial resolution but high in time resolution.  
         [0065]    Finally, the three types of image signals, i.e., the high-spatial-resolution image signal, low-resolution image signal, and high-time-resolution image signal, output from the circuits  36 ,  38 , and  40  are input to the video selection device  46 . The video selection device  46  is switched by a selection signal  48 , and can obtain all the three types of image signal when they are transmitted by using scalability. The video selection device  46  can selectively output these signals. When the signals are transmitted without using any scalability, only the image information decoded by the decoding apparatus  30  is output.  
         [0066]    An example of a system including the above decoding apparatus will be described next. FIG. 9 is a block diagram showing the schematic arrangement of this system. The system includes a DVD reproduction apparatus  50 , a digital VTR  52 , an authoring tool (EWS)  54  for a digital image, a server (EWS)  56  for the tool  54 , and a personal computer (terminal)  58 . Icons  60 ,  62 ,  64 , and  66  are displayed on the monitor screen of the computer  58 . The system also includes a mouse  68  connected to the computer  58 . A mouse cursor  70  corresponds to the mouse  68 .  
         [0067]    Each of the DVD reproduction apparatus  50 , the digital VTR  52 , and the tool  54  has an arrangement corresponding to the decoding apparatus in FIG. 2. That is, each of the outputs from the DVD reproduction apparatus  50 , the digital VTR  52 , and the tool  54  corresponds to the output from the video selection device  46 .  
         [0068]    The icon  60  displayed on the monitor screen of the computer  58  indicates an image associated with a digital motion image from the DVD reproduction apparatus  50 ; the icon  62 , an image associated with a digital motion image from the digital VTR  52 ; and the icons  64  and  66 , images associated with digital motion images from the tool  54  (i.e., the server  56 ). As described above, the graphic patterns of the icons  60  to  66  are formed from the start frames of the corresponding digital motion images or the common base layers of arbitrary I-frames.  
         [0069]    [0069]FIG. 10 is a flow chart for explaining the operation of the computer  58  in the system shown in FIG. 9. This operation will be described below with reference to FIGS. 10 and 11.  
         [0070]    When the power switch of the computer  58  is turned on, the computer  58  checks in steps S 1  to S 3  whether the DVD reproduction apparatus  50 , the digital VTR  52 , or the server  56  is connected. If it is determined in one of these steps that the corresponding apparatus is connected, the image reproduced or output from the apparatus is displayed as an icon, as described above (steps S 4  to S 6 ). The image signal displayed in this case is not the motion image itself but is a representative image extracted from the common base layer. This display alone does not occupy much of the transmission rate of the interface between the computer  58  and each apparatus.  
         [0071]    If the right mouse button is clicked while at least one image is displayed and the cursor  70  is moved onto the image by the mouse  68  (step S 7 ), this image is displayed as a motion image by using the common base layer (step S 8 ). In this case, since the common base layer with the minimum data amount is displayed, not much load is imposed on the interface. As is obvious from the flow chart of FIG. 10, when the cursor  70  is moved onto another image by the mouse  68 , and the right mouse button is clicked, the clicked image is displayed as a motion image. In this case, the other image is restored to the newly extracted representative image described above.  
         [0072]    If the cursor  70  is moved onto a motion image displayed by using the common base layer, and the left mouse button is clicked (step S 9 ), resolution selection display is performed, as indicated by a frame  72  in FIG. 11 (step S 10 ). If the cursor  70  is moved onto an arbitrary resolution (low resolution, high spatial resolution, or high time resolution) display, and the right mouse button of the mouse  68  is clicked, the corresponding resolution is selected. In accordance with this selection, the computer  58  supplies a selection signal to the video selection device  46  in FIG. 7 arranged in each apparatus to select an image signal with the desired resolution, receives it, and displays the corresponding motion image on the monitor (step S 11 ). At this time, since this display is a desired display, the display mode is preferably switched to the full image display mode.  
         [0073]    Detection in steps S 1  to S 3  and S 7  is always performed. If is it detected in step S 12 , during this detection, that the power supply is turned off, the processing is terminated. If the right mouse button of the mouse  68  is clicked while the cursor  70  is located in the frame  72  during resolution selection screen display, it is determined that no resolution is selected, and the flow returns to the above detection loop. Note that the processing based on the above flow chart is executed by the program stored in a ROM  90  in the computer  58 .  
         [0074]    In the system using the apparatus on the decoding side and the decoding apparatus, the user can enjoy a desired image in real time by selectively using arbitrary scalability in accordance with a user&#39;s choice. This can save the wait time for unnecessary decoding required for an image search and the like. In addition, as described above, since no motion image is basically displayed, or a motion image can be displayed by transferring minimum necessary information, the time during which the interface or the CPU in the computer is occupied can be shortened, allowing quick operation.  
         [0075]    In addition, the decoding circuit is also designed such that the circuit for the common base layer is commonly used for decoding of a plurality of enhancement layers. This arrangement is also advantageous in terms of a reduction in circuit size in each decoding apparatus.  
         [0076]    Many widely different embodiments of the present invention may be constructed without departing from the spirit and scope of the present invention. It should be understood that the present invention is not limited to the specific embodiments described in the specification, except as defined in the appended claims.