Abstract:
An image decoding apparatus which receives a compressed image stream as an input signal and decodes the input signal, includes a plurality of decoders configured to decode the input signal by predetermined data unit, and a task controller configured to divide a decoding process for the input signal into a plurality of process steps and to allocate data unit to be processed by the decoder to the decoder for each process step, each process step having dependency in an order of processing. The task controller detects processing status of the decoder, and allocates data unit and process step to be processed to the decoder, depending on the detected processing status, dependency between the process steps, and dependency between frames included in the image stream.

Description:
BACKGROUND 
       [0001]    1. Technical Field 
         [0002]    The technical field relates to an image decoding apparatus and method which can decode an image compression stream at a high speed. 
         [0003]    2. Background Art 
         [0004]    As a conventional method of decoding an image stream compressed according to MPEG2, H.264/AVC, or the like at a high speed, a method of dividing a decoding process into slices and allocating each slice to each of a plurality of processors to perform parallel processing in decoding with the plurality of processors, is generally known. However, this method cannot be applied unless one frame is divided into a plurality of slices. When one frame is not divided into a plurality of slices, a method is employed, in which a decoding process is divided in units of frames so that processors perform the divided processes in parallel. 
         [0005]    On the other hand, there may be dependence between the compressed frames. For example, in the MPEG-2 standard, I frame can be singularly decoded without depending on other frames. However, P frame is decoded with reference to I frame, and B frame is decoded with reference to I frame and P frame. That is, compressed frames can be grouped according to dependence on other frames in a decoding process. 
         [0006]    The frames can be classified into three groups including, for example, group A, group B, and group C. Group A includes a frame which does not refer to other frame, such as I frame. Group B includes a frame which refers to another frame and can be referred to by another frame, such as P frame. Group C includes a frame which refers to another frame but is not referred to by another frame, such as B frame. 
         [0007]    That is, the frame belonging to group A does not depend on a decoding result of a frame belonging to any groups. However, the frame belonging to group B depends on a decoding result of a frame belonging to group A and cannot be decoded unless a frame to be referred is decoded. The frame belonging to group C depends on a decoding result of a frame belonging to group A or group B and cannot be decoded unless a frame to be referred is decoded. 
         [0008]    Therefore, the frames belonging to group C do not depend on each other, and thus decoding processes for the frames can be performed in parallel. However, a decoding process for a frame belonging to group A and a decoding process for a frame depending on the frame belonging to group A cannot be performed in parallel. For this reason, while one processor performs the process which cannot be done in parallel, the other processor is in an idle state which is a state for waiting process. 
         [0009]    In order to decode frames which depend on each other, there is proposed an image decoding method (see JP-A-2000-295616) which determines shares of parallel processes performed by processors according to a dependence of frames. In addition, there is also proposed an image decoding method (see JP-A-2005-175997) which divides one frame into a plurality of slices and process the slices in parallel when decoding a frame which is subjected to bidirectional predictive encoding, and which performs parallel processing in units of frames when decoding a frame which is subjected to the other encoding schemes. By using these methods, idle times of the processors are reduced. 
         [0010]    Even though the above stated conventional image decoding process is used, a processor may be in an idle state for a long time. Such an example will be described below with reference to  FIG. 12 . 
         [0011]    In an example in  FIG. 12 , it is assumed that a decoding process is performed by using four processors in parallel. As frames to be decoded, a total of 6 frames including frames  0  to  5  are used with one frame composed of one slice. Frame  0  belongs to group A to which a frame which does not depend on decoding results of other frames belongs. Frame  3  belongs to group B to which a frame which refers to another frame and is referred to by the other frame, and refers to frame  0 . Frames  1 ,  2 ,  4 , and  5  belong to group C to which a frame which refers to another frame but is not referred to by the other frame, and refer to frames  0  and  3 . It is assumed that times required for decoding processes for the respective frames are equal. 
         [0012]    Since frames  1  to  5  refer to frame  0 , a decoding process for these frames cannot be started until a decoding process for frame  0  is completed. As a result, while frame  0  is being decoded, three of four processors are set in an idle state. After the decoding process for frame  0  is finished, frame  3  can be decoded. Since frames  1 ,  2 ,  4 , and  5  refer to frame  3 , decoding processes for those frames cannot be started until the decoding process for frame  3  is completed. For this reason, while frame  3  is being decoded, three of the four processors are set in an idle state. After the decoding process for frame  3  is completed, parallel processing for frames  1 ,  2 ,  4 , and  5  can be performed. In this manner, 50% of a time required for all the processes corresponds to an idle time. Specifically, only 50% of the capabilities of the processors can be disadvantageously utilized. 
       SUMMARY 
       [0013]    In order to solve the above problem, it is an object to provide, in a decoding process with parallel processing by a plurality of processors, an image decoding apparatus which can reduce a time for which the processors are in an idle state so that a compressed image can be decoded at a high speed with a low delay. 
         [0014]    In a first aspect, an image decoding apparatus is provided, which receives a compressed image stream as an input signal and decodes the input signal. The image decoding apparatus includes a plurality of decoders, each decoder configured to decode the input signal by predetermined data unit, and a task controller configured to divide a decoding process for the input signal into a plurality of process steps and to allocate data unit to be processed by the decoder to the decoder for each process step, each process step having dependency in an order of processing. The task controller detects processing status of the decoder and allocates data unit and process step to be processed to the decoder, depending on the detected processing status, dependency between the process steps, and dependency between frames included in the image stream. 
         [0015]    In a second aspect, an image decoding method is provided. The image decoding method includes receiving a compressed image stream as an input signal; dividing a decoding process for the input signal into a plurality of process steps, each process step having dependency in an order of processing; executing the decoding process of the input signal in a predetermined data unit for each process step, by a plurality of decoders; detecting processing status of the decoder; and determining data unit and process step to be processed by the decoder, depending on the detected processing status, dependency between the process steps, and dependency between frames included in the image stream. 
         [0016]    In a third aspect, a control program is provided, for an image decoding apparatus which receives a compressed image stream as an input signal and decodes the input signal. The control program makes the image decoding apparatus execute the procedures of: receiving a compressed image stream as an input signal; dividing a decoding process for the input signal into a plurality of process steps, each process step having dependency in an order of processing; executing the decoding process of the input signal in a predetermined data unit for each process step, by a plurality of decoders; detecting processing status of the decoder; and determining data unit and process step to be processed by the decoder, depending on the detected processing status, dependency between the process steps, and dependency between frames included in the image stream. 
         [0017]    According to the above aspects, an idle time required in parallel processing of image decoding processes performed by using a plurality of processors can be reduced, so that a decoding process for a compressed image can be performed in higher speed. 
     
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         [0018]      FIG. 1  is a block diagram of an image decoding apparatus according to a first embodiment. 
           [0019]      FIG. 2A  is a diagram showing a configuration of a task list, and  FIG. 2B  is a diagram showing a priority table which manages priorities of tasks for each slice type. 
           [0020]      FIG. 3  is a state transition diagram of a task controller. 
           [0021]      FIG. 4  is a flow chart showing processes in a task allocation state. 
           [0022]      FIG. 5  is a flow chart showing processes in a task addition state. 
           [0023]      FIG. 6  is a diagram showing an example of a byte stream inputted to the image decoding apparatus. 
           [0024]      FIG. 7  is a diagram showing an example of times required for the respective processes. 
           [0025]      FIGS. 8A to 8I  are diagrams for explaining changes of task lists. 
           [0026]      FIG. 9  is a diagram showing a process order of decoding parameter sets processed by decoders. 
           [0027]      FIG. 10  is a block diagram of an image decoding apparatus according to a second embodiment. 
           [0028]      FIG. 11  is a diagram for explaining frames and steps processed by decoders in the second embodiment. 
           [0029]      FIG. 12  is a diagram showing parallel processing with conventional four processors. 
       
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
       [0030]    Preferred embodiments will be described below with reference to the accompanying drawings. 
       First Embodiment 
       [0031]    An image decoding apparatus according to a first embodiment inputs a byte stream of a video compressed by an encoding method such as H.264, and decodes and outputs a digital image. 
       1. Configuration of Image Decoding Apparatus 
       [0032]      FIG. 1  shows a configuration of an image decoding apparatus according to the first embodiment. An image decoding apparatus  101  receives a byte stream through an input terminal  102  and outputs a digital image obtained by decoding the byte stream through an output terminal  103 . 
         [0033]    In the image decoding apparatus  101 , a stream buffer  104  buffers the inputted byte stream. First and second decoders  108  and  109  perform image decoding processes in units of slices. With the first and second decoders  108  and  109 , parallel processing can be achieved. A task controller  105  controls decoding processes executed by the decoders  108  and  109  based on steps (will be described later) of the decoding processes and types of slices. An intermediate buffer  106  stores coefficient data, prediction data, and a difference image which are generated in the decoding processes. A frame buffer  107  stores a decoded image which is finally obtained. Details of the task controller  105  and the decoders  108  and  109  will be described later. 
       1.1 Step and Slice Type of Image Decoding Process 
       [0034]    The image decoding apparatus according to the first embodiment divides a decoding process for one frame into three steps (stages) including an arithmetic decoding process, a difference image creating process, and a decoded image creating process. 
         [0035]    The arithmetic decoding process includes a process which decodes coefficient data required to create a difference image from the inputted byte stream and prediction data such as a prediction mode and motion vectors required to create a prediction image. This process does not depend on processes in different frames or different steps. 
         [0036]    The difference image creating process is a process which performs inverse quantization, inverse discrete cosine transform, and the like to coefficient data to create a difference image. Although this process does not depend on processes in different frames, it depends on processes in different steps because necessary coefficient data must be decoded by an arithmetic decoding process. 
         [0037]    The decoded image creating process creates a prediction image from prediction data and a decoded image required for reference, and synthesizes the prediction image with the difference image to create the decoded image. This process requires that the necessary prediction data is decoded by the arithmetic decoding process, that the necessary difference image is created by the difference image creating process, and that the necessary decoded image is created if reference is necessary. Therefore, the decoded image creating process depends on other processes in different frames and different steps. 
         [0038]    Furthermore, the slices are classified into three types based on dependence between frames. As described above, a type of slices composing a frame belonging to group A including a frame which does not refer to another frame is called “slice A”. A type of slices composing a frame belonging to group B which includes a frame which refers to another frame and is referred to by other frames is called “slice B”. A type of slices composing a frame belonging to group C which includes a frame which refers to other frames and is not referred to by any frame is called “slice C”. 
       1.2 Task Controller 
       [0039]    The task controller  105  is connected to the first and second decoders  108  and  109  (to be simply referred to as “decoders” hereinafter). Each of the decoders  108  and  109  can be in a waiting state or a processing state. The task controller  105  stores a task list  15  to manage tasks of decoding processes executed by the decoders  108  and  109  in the task controller  105 . The task list  15  manages a decoding parameter set as one element. The decoding parameter set is a pair of parameters required for a decoding process for a slice, and concretely includes the following pieces of information. 
         [0040]    slice type (information representing a type of a slice to be decoded) 
         [0041]    task (information representing a type of a step of a decoding process executed by the decoder) 
         [0042]    slice data pointer (information representing a storing position of slice data on the stream buffer  104 ) 
         [0043]    coefficient data pointer (information representing a storing position of coefficient data on the intermediate buffer  106 ) 
         [0044]    prediction data pointer (information representing a storing position of prediction data on the intermediate buffer  106 ) 
         [0045]    difference image pointer (information representing a storing position of a difference image on the intermediate buffer  106 ) 
         [0046]      FIG. 2A  shows an example of the task list  15 . The task list  15  manages, in addition to a decoding parameter set, an element number which specifies the decoding parameter set and a priority to a process for the decoding parameter set in association with the element number. 
         [0047]    The task controller  105  has a priority table  16 .  FIG. 2B  shows an example of the priority table  16 . The priority table  16  is to regulate priorities of steps in the decoding process depending on the slice types. As described above, according to the dependence between the steps, the decoded image creating process, the arithmetic decoding process, and the difference image creating process must be preferentially performed in this order. According to the dependence between the frames, slice type A, slice type B, and slice type C must be preferentially processed in this order. The priority table  16  regulates priorities of the steps depending on the slice types, in consideration of the dependences between the steps and between the frames. A task having a higher priority is preferentially processed. 
         [0048]    Furthermore, the task controller  105  stores an pending task list  17  which manages a decoding parameter set as one element to manage a task of which process is interrupted. The task controller  105  always stores priorities of the tasks processed by the first and second decoders  108  and  109 . 
         [0049]    The task controller  105  can be set in a plurality of states.  FIG. 3  shows state transition of the task controller  105 . As shown in  FIG. 3 , the task controller  105  has five states including an initial state, a stream analyzing state, a task allocation state, a task completion waiting state, and a task addition state. 
         [0050]    Initial state: Both the first and second decoders  108  and  109  are in the waiting state. 
         [0051]    Stream analyzing state: Both the decoders  108  and  109  are in the waiting state (because the stream analyzing state is shifted only from the initial state). 
         [0052]    Task allocation state: This state is shifted from another state when at least one of the first and second decoders  108  and  109  is in the waiting state. This state is shifted to the initial state or the task completion waiting state. 
         [0053]    Task completion waiting state: Both the first and second decoders  108  and  109  are in the processing state. 
         [0054]    Task addition state: This state is shifted from the task completion waiting state, and this state is shifted to the task allocation state. 
         [0055]    The states of the task controller  105  will be concretely described below. When the image decoding apparatus  101  starts a byte stream decoding process according to an external command or the like, the task controller  105  starts an operation from the initial state. In the initial state, the numbers of elements in the task list  15  and the pending task list  17  are set to 0. When a necessary byte stream can be read from the stream buffer  104 , the initial state is shifted to the stream analyzing state. 
         [0056]    In the stream analyzing state, the task controller  105  searches the byte stream read from the stream buffer  104  from the top of the byte stream to detect all slices included in the byte stream. Each time the slice is detected, a new decoding parameter set corresponding to the slice is added to the task list  15 . At this time, a new element number is generated and given to the decoding parameter set. The element number may be any number as long as it is not present in the task list  15  at the time. In the new decoding parameter set, a “slice type” is set to the type of the detected slice (any one of slice type A, slice type B, and slice type C), a “task” is set to an “arithmetic decoding process”, and a “slice data pointer” is set to a start position in the intermediate buffer  106  for data of the detected slice. At this point of time, a “coefficient data pointer”, a “prediction data pointer”, and a “difference image pointer” are not set yet. When all the read byte streams are searched and all decoding parameter sets are added to the task list  15 , the task controller  105  shifts to the task allocation state. 
         [0057]    A flow chart in the task allocation state is shown in  FIG. 4 . The task controller  105  shifts to the initial state when the numbers of elements of the task list  15  and the pending task list  17  are 0 (“YES” in  5901 ) and when the first and second decoders  108  and  109  are in the waiting state (“YES” in  5902 ) (S 910 ). When at least one of the number of elements of the task list  15  and the number of the pending task list  17  is not 0 (“NO” in S 901 ), with reference to the priority table  16  shown in  FIG. 2B , priorities of the decoding parameter sets included in the task list  15  and the pending task list  17  are determined (S 903 ). The decoder set in the waiting state is determined (S 904 ). 
         [0058]    When both the decoders  108  and  109  are in the waiting state, the task controller  105  outputs a decoding parameter set having the highest priority in the task list  15  and the pending task list  17  to the first decoder  108  (S 905 ), and outputs a decoding parameter set having the second highest priority to the second decoder  109  (S 906 ). Thereafter, the task controller  105  deletes the outputted decoding parameter sets from the task list  15  (S 907 ) and shifts to the task completion waiting state (S 911 ). 
         [0059]    When only the first decoder  108  is in the waiting state, the task controller  105  outputs a decoding parameter set having the highest priority to the first decoder  108  in the waiting state (S 908 ). Thereafter, the task controller  105  deletes the outputted decoding parameter set from the task list  15  or the pending task list  17  (S 907 ) and shifts to the task completion waiting state (S 911 ). 
         [0060]    Similarly, when only the second decoder  109  is in the waiting state, the task controller  105  outputs a decoding parameter set having the highest priority to the second decoder  109  set in the waiting state (S 909 ). Thereafter, the task controller  105  deletes the outputted decoding parameter set from the task list  15  or the pending task list  17 . (S 907 ) and shifts to the task completion waiting state (S 911 ). 
         [0061]    When a plurality of decoding parameter sets having the same priorities are present in both the task list  15  and the pending task list  17 , the decoding parameter set included in the pending task list  17  is preferentially outputted. When a plurality of decoding parameter sets having the same priorities are present in each task list, the decoding parameter sets are preferentially outputted in order of addition of the decoding parameter sets to the task list. 
         [0062]    A decoding parameter set of which “slice type” is slice B or slice C and of which “task” is “decoded image creating process” is not outputted to the first and second decoders  108  and  109  when a decoded image creating process for a reference image is not completed. In these cases, a decoding parameter set having next higher priority is outputted to the first and second decoders  108  and  109 . 
         [0063]    In the task completion waiting state, the task controller  105  waits until the decoding parameter set is inputted from the first decoder  108  or the second decoder  109 . When the decoding parameter set is inputted from the first decoder  108  or the second decoder  109 , the task controller  105  shifts to the task addition state. When the decoding parameter sets are simultaneously inputted from the plurality of decoders, the task controller  105  shifts to the task addition state to process any one of the decoding parameter sets, while the other decoding parameter sets go into a queue. A decoding parameter set for a task list which is inputted in a state other than the task completion waiting state or a state in which the other decoding parameter set is in the queue also goes into the queue. The queued decoding parameter sets are sequentially processed when the task controller  105  shifts to the task completion waiting state. 
         [0064]    A flow chart in the task addition state is shown in  FIG. 5 . The task controller  105  changes processes depending on a type of a “task” included in the inputted decoding parameter set. When the “task” included in the inputted decoding parameter set is an “arithmetic decoding process”, a new decoding parameter set is created by replacing the “task” in the inputted decoding parameter set with a “difference image creating process” (S 1002 ). The new decoding parameter set is added to the task list  15  as an element (S 1003 ), and the task controller  105  shifts to the task allocation state (S 1009 ). 
         [0065]    When the “task” included in the inputted decoding parameter set is the “difference image creating process”, a new decoding parameter set is created by replacing the “task” in the inputted decoding parameter set with the “decoded image creating process” (S 1004 ). The new decoding parameter set is added to the task list  15  as an element (S 1003 ), and the task controller  105  shifts to the task allocation state (S 1009 ). Information other than the “task” included in the new decoding parameter set is made equal to that of the inputted decoding parameter set. 
         [0066]    When the “task” included in the decoding parameter set inputted in the task controller  105  is the “decoded image creating process”, the task controller  105  searches the task list  15  for a decoding parameter set which has “task” as a “decoded image process” and refers to a decoded image to be processed by the inputted decoding parameter set. When two or more corresponding decoding parameter sets are found (“YES” in S 1005 ), a priority of a task processed by a decoder which is not in the waiting state is compared with a priority of the decoding parameter set having the lowest priority among the found decoding parameter sets. When the priority of the task processed by the decoder is lower (“YES” in S 1006 ), the task controller  105  outputs a process interrupt signal to the decoder and waits until the decoding parameter set is inputted from the decoder (S 1007 ). After the decoding parameter set is inputted from the decoder, the task controller  105  adds the inputted decoding parameter set to the pending task list  17  (S 1008 ) and shifts to the task allocation state (S 1009 ). Information other than the “task” included in the new decoding parameter set is made equal to that of the inputted decoding parameter set. When the priority of the task processed by the decoder is higher than or equal to the priority of the decoding parameter having the lowest priority (“NO” in S 1006 ), or when the number of corresponding decoding parameter sets is smaller than 2 (“NO” in S 1005 ), the task controller  105  shifts to the task allocation state without performing any operation (S 1009 ). 
       1.3 Decoder 
       [0067]    The first and second decoders  108  and  109  have two states including a waiting state and a processing state. The initial state is a waiting state which waits for input of the decoding parameter set. When the decoding parameter set is inputted, the first or second decoder  108  or  109  shifts from the waiting state to the processing state. In the processing state, the operations of the decoder  108  or  109  changes depending on a “task” included in the inputted decoding parameter set. The operations in the processing state changing depending on the “task” will be described below. 
         [0068]    When the “task” is the “arithmetic decoding process”, the decoder  108  or  109  reads a byte stream from an address on the stream buffer  104  pointed by a “slice data pointer” included in the inputted decoding parameter set, decodes coefficient data and prediction data, and writes the coefficient data and the prediction data on the intermediate buffer  106 . Subsequently, the decoder  108  or  109  outputs a new decoding parameter set, and shifts to the waiting state. In the new decoding parameter set, the “coefficient data pointer” and the “prediction data pointer” are set to pointers to the coefficient data and the prediction data which are written in the intermediate buffer  106 . Information other than the “coefficient data pointer” and the “prediction data pointer” is made equal to those in the inputted decoding parameter set. 
         [0069]    When the “task” is the “difference image creating process”, the decoder  108  or  109  reads coefficient data from an address on the intermediate buffer  106  pointed by a “coefficient data pointer” included in the inputted decoding parameter set, creates a difference image, and writes the difference image in the intermediate buffer  106 . Subsequently, the decoder  108  or  109  outputs new decoding parameter sets and then shifts to the waiting state. In the new decoding parameter set, a “difference image pointer” is set to a pointer to the difference image written in the intermediate buffer  106 , and information other than the “difference image pointer” is set to be equal to that in the inputted decoding parameter set. 
         [0070]    When the “task” is the “decoded image creating process”, the decoder  108  or  109  reads the prediction data from addresses on the intermediate buffer  106  pointed by a “prediction data pointer” included in the inputted decoding parameter set. The decoder  108  or  109  reads a decoded image which is required to create a prediction image and decoded in advance from the frame buffer  107  based on the read prediction data, and synthesizes the decoded image with the difference image read from the intermediate buffer  106  to create a decoded image. The created decoded image is written in the frame buffer  107 . Subsequently, the decoder  108  or  109  outputs the inputted decoding parameter set as it is and then shifts to the waiting state. 
         [0071]    When the decoder  108  or  109  receives a process interrupt signal in the processing state, the decoder  108  or  109  interrupts the process which is being executed at the present, sets various pointers in the decoding parameter set such that the process can be restarted later, and outputs the decoding parameter set. 
       2. Parallel Decoding Operation 
       [0072]    A decoding operation of the image decoding apparatus  101  configured as described above according to the first embodiment will be described below. In the following explanation, the following matters are assumed. The image decoding apparatus  101  receives (inputs) a byte stream composed of one GOP (Group of Pictures) including four frames, i.e., I frame, B frame, P frame, and B frame in this order as shown in  FIG. 6 , and decodes all the frames. Each frame is composed of one slice. A GOP shown in  FIG. 6  includes one slice A, one slice B, and two slices C. All referenced images are included in the same GOP. P frame uses I frame as a reference image, and B frame uses I frame and P frame as reference images. 
         [0073]    In order to simplify the explanation, times required for task addition, task deletion, task allocation, state transition, and input/output of a decoding parameter set are set to 0. Time required for each process for slice types A to C is shown in  FIG. 7 . 
         [0074]    In the image decoding apparatus  101 , a byte stream received (inputted) through the input terminal  102  is written in the stream buffer  104 . Thus, a byte stream to be decoded can be read from the stream buffer  104 , and the task controller  105  shifts from the initial state to the stream analyzing state. 
         [0075]    In the stream analyzing state, the task controller  105  reads a byte stream from the stream buffer  104  to analyze the byte stream, adds a new decoding parameter set to the task list  15 , and then shifts to the task allocation state. The state of the task list  15  obtained in this state is shown in  FIG. 8A . In  FIG. 8A , to simplify the explanation, an element number representing an order of decoding parameter sets, a “slice type” and a “task” included in each decoding parameter set, and priority of each decoding parameter set are just shown (the same manner as in  FIGS. 8B to 8I  described below). The “slice data pointer” of each decoding parameter set is set as data start position of each slice. 
         [0076]    In the task allocation state, the task controller  105  outputs a decoding parameter set (E 1  in  FIG. 8A ) having the highest priority in the task list  15  to the decoder  108  and outputs a decoding parameter set (E 3  in  FIG. 8A ) having the second highest priority to the decoder  109 . Thereafter, the task controller  105  deletes the two outputted decoding parameter sets from the task list  15  and then shifts to the task completion waiting state. Time t obtained at this time is set as 0.  FIG. 9  is a diagram showing decoding parameter sets processed by the decoders  108  and  109  along a time axis in an example (will be described below). 
         [0077]    As shown in  FIG. 9 , decoding parameter sets of element number E 1  and element number E 3  are inputted to the first decoder  108  and the second decoder  109  (Note: “decoding parameter set E 1 ”, “decoding parameter set E 3 ”, . . . may be simply abbreviated as “E 1 ”, “E 3 ”, in the following). Thereafter, the first decoder  108  and the second decoder  109  shift from the waiting state to the processing state. In the processing state, the decoders  108  and  109  perform an arithmetic decoding process with reference to the inputted decoding parameter sets. 
         [0078]    At time t=3, the second decoder  109  completes E 3 . Then the second decoder  109  outputs, to the task controller  105 , a decoding parameter set obtained by replacing a coefficient data pointer and a prediction data pointer in the decoding parameter set regarding E 3  with pointers to the coefficient data and the prediction data obtained by the arithmetic decoding process, and then shifts to the waiting state. 
         [0079]    When the task controller  105  receives the decoding parameter set from the second decoder  109 , the task controller  105  shifts to the task addition state. The “task” of the received decoding parameter set is the arithmetic decoding process. Therefore, information other than the “task” is the same as that in the received decoding parameter set, the task controller  105  adds a new decoding parameter set E 5  in which the “difference image creating process” is set as the “task” to the task list  15 , and shifts to the task allocation state. The task list  15  obtained at this time is shown in  FIG. 8B . As shown in  FIG. 8B , in the task allocation state, the decoding parameter set having the highest priority is E 5 , the task controller  105  outputs E 5  to the decoder  109 , and deletes E 5  from the task list  15 . Thereafter, the task controller  105  shifts to the task completion waiting state. 
         [0080]    When the second decoder  109  receives the decoding parameter set (E 5 ) from the task controller  105 , the decoder  109  shifts to the processing state again. The decoder  109  performs the difference image creating process with reference to the inputted decoding parameter set. 
         [0081]    The second decoder  109  completes the process for E 5  at time t=5 (see  FIG. 9 ). The second decoder  109  outputs, to the task controller  105 , a decoding parameter set obtained by replacing the pointer to the difference image in the decoding parameter set regarding E 5  with a pointer to a difference image obtained by the difference image creating process, and shifts to the waiting state. 
         [0082]    When the task controller  105  receives a decoding parameter set from the second decoder  109 , the task controller  105  shifts to the task addition state. The “task” of the inputted decoding parameter set represents the “difference image creating process”. Therefore, information other than the task is the same as that in the received decoding parameter set. The task controller  105  adds a new decoding parameter set E 6  in which a “decoded image creating process” is set as a “task” to the task list  15 , and shifts to the task allocation state. The task list  15  obtained at this time is shown in  FIG. 8C . As shown in  FIG. 8C , the decoding parameter set having the highest priority is E 6 . However, the decoded image creating process for slice B to be decoded in E 6  cannot be executed until the decoded image creating process for slice A is completed. For this reason, E 2  which is earlier one added to the task list between E 2  and E 4  having the next highest priority to E 6  in the task list  15  shown in  FIG. 8C , is outputted to the decoder  109 . E 2  is deleted from the task list. Thereafter, the task controller  105  shifts to the task completion waiting state. 
         [0083]    When the second decoder  109  receives the decoding parameter set (E 2 ) from the task controller  105 , the decoder  109  shifts to the processing state again. The second decoder  109  performs an arithmetic decoding process with reference to the received decoding parameter set. 
         [0084]    The second decoder  109  completes the process for E 2  at time t=7 (see  FIG. 9 ). The second decoder  109  outputs to the task controller  105  a decoding parameter set obtained by replacing the coefficient data pointer and the prediction data pointer in the decoding parameter set regarding E 2  with the coefficient data and the prediction data obtained by the arithmetic decoding process, and shifts to the waiting state. 
         [0085]    When the task controller  105  receives the decoding parameter set from the second decoder  109 , the task controller  105  shifts to the task addition state. The “task” of the inputted decoding parameter set represents the “arithmetic decoding process”. Therefore, information other than the task is the same as that in the received decoding parameter set. The task controller  105  adds a new decoding parameter set E 7  in which the “difference image creating process” is set as the “task” to the task list  15 , and shifts to the task allocation state. The task list  15  obtained at this time is shown in  FIG. 8D . As shown in  FIG. 8D , a decoding parameter set having the highest priority is E 6 . However, the decoded image creating process of the slice B represented by E 6  cannot be executed until the decoded image creating process for the slice A is completed. For this reason, the task controller  105  outputs E 4  having the next highest priority to E 6  to the decoder  109 , and deletes E 4  from the task list. Thereafter, the task controller  105  shifts to the task completion waiting state. As described above, when a necessary process for a decoding parameter set having the highest priority is not completed, a decoding parameter set having a priority lower than that of the decoding parameter set having the highest priority may be preferentially processed. 
         [0086]    When the second decoder  109  receives the decoding parameter set E 4  from the task controller  105 , the second decoder  109  shifts to the processing state again. The decoder  109  performs the arithmetic decoding process with reference to the received decoding parameter set. 
         [0087]    The first decoder  108  and the second decoder  109  completes processes for E 1  and E 4  at time t=9 (see  FIG. 9 ). The first decoder  108  outputs, to the task controller  105 , a decoding parameter set obtained by replacing the coefficient data pointer and the prediction data pointer in E 1  with pointers to the coefficient data and the prediction data which are obtained by the arithmetic decoding process, and shifts to the waiting state. The second decoder  109  outputs, to the task controller  105 , a decoding parameter set obtained by replacing the coefficient data pointer and the prediction data pointer in E 4  with pointers to the coefficient data and the prediction data obtained by the arithmetic decoding process, and shifts to the waiting state. When the two decoders  108  and  109  simultaneously complete the processes and shifts to the waiting state, the task controller  105  starts control of the first decoder  108 . 
         [0088]    When receiving the decoding parameter sets E 1  and E 4  from the first decoder  108  and the second decoder  109 , respectively, the task controller  105  shifts to the task addition state. 
         [0089]    The task controller  105  performs a process for E 1 . The “task” of the inputted decoding parameter set E 1  represents the “arithmetic decoding process”. Therefore, information other than the task is the same as that in the received decoding parameter set. The task controller  105  adds a new decoding parameter set E 8  in which the “difference image creating process” is set as the “task” to the task list  15  and shifts to the task allocation state. The task list  15  obtained at this time is shown in  FIG. 8E . As shown in  FIG. 8E , the decoding parameter set having the highest priority is the decoding parameter set E 6 . However, the decoded image creating process for the slice B cannot be executed until the decoded image creating process for the slice A is completed. For this reason, the task controller  105  outputs E 8  having the highest priority next to E 6  to the decoder  108  and deletes E 8  from the task list  15 . Thereafter, the task controller  105  shifts to the task completion waiting state. 
         [0090]    Subsequently, with respect to the decoding parameter set E 4  inputted from the second decoder  109 , the task controller  105  adds a new decoding parameter set E 9  to the task list  15  similarly to E 1 , and shifts to the task allocation state. The task list  15  obtained at this time is shown in  FIG. 8F . As shown in  FIG. 8F , the decoding parameter set E 6  has the highest priority. However, the decoded image creating process for the slice B cannot be executed until the decoded image creating process for the slice A is completed. For this reason, the decoding parameter set E 7  of E 7  and E 9  having the highest priority next to E 6 , which is earlier added by the task list  15 , is outputted to the second decoder  109 , and E 7  is deleted from the task list. Thereafter, the task controller  105  shifts to the task completion waiting state. 
         [0091]    When receiving the decoding parameter set E 8  from the task controller  105 , the first decoder  108  shifts to the processing state again. Similarly, when receiving the decoding parameter set E 7 , the second decoder  109  shifts to the processing state again. The first decoder  108  and the second decoder  109  perform a difference image creating process with reference to the inputted decoding parameter set. 
         [0092]    The first decoder  108  and the second decoder  109  complete processes for the decoding parameter sets E 8  and E 7  at time t=11 (see  FIG. 9 ). The decoder  108  outputs a decoding parameter set obtained by replacing the pointer to the difference image in the decoding parameter set E 8  with a pointer to the difference image obtained by the difference image creating process, and shifts to the waiting state. The decoder  109  outputs a decoding parameter set obtained by replacing a pointer to the difference image in the decoding parameter set E 7  with a pointer to the difference image obtained by the difference image creating process and shifts to the waiting state. 
         [0093]    When receiving a decoding parameter set from the decoder  108 , the task controller  105  shifts to the task addition state. A “task” of the received decoding parameter set represents the “difference image creating process”. Therefore, information other than the task is the same as that in the received decoding parameter set. The task controller  105  adds a new decoding parameter set E 10  in which the “decoded image creating process” is set as the “task” to the task list  15 , and shifts to the task allocation state. The task list  15  obtained at this time is shown in  FIG. 8G . As shown in  FIG. 8G , since the decoding parameter set having the highest priority is E 10 , the task controller  105  outputs E 10  to the first decoder  108  and deletes E 10  from the task list. Thereafter, the task controller  105  shifts to the task completion waiting state. 
         [0094]    When receiving the decoding parameter set from the second decoder  109 , the task controller  105  shifts to the task addition state. Similarly, the task controller  105  adds a new decoding parameter set E 11  to the task list  15  and shifts to the task allocation state. The task list  15  obtained at this time is shown in  FIG. 8H . As shown in  FIG. 8H , the priorities of the decoding parameter sets E 6  and E 11  are higher. However, both the decoding parameter sets E 6  and E 11  cannot be executed until the decoded image creating process for the slice A is completed. For this reason, the task controller  105  outputs E 9  having the highest priority next to E 6  and E 11  to the decoder  109 , and deletes E 9  from the task list. Thereafter, the task controller  105  shifts to the task completion waiting state. 
         [0095]    When receiving the decoding parameter set E 10  from the task controller  105 , the first decoder  108  shifts to the processing state again. The first decoder  108  performs the decoded image creating process with reference to the received decoding parameter set. Similarly, when receiving the decoding parameter set E 9 , the second decoder  109  shifts to the processing state again. The second decoder  109  performs the difference image creating process with reference to the received decoding parameter set. 
         [0096]    When the first decoder  108  completes the process for E 10  at time t=12 (see  FIG. 9 ), the first decoder  108  outputs the received decoding parameter set E 10  as it is to the task controller  105 , and shifts to the waiting state. 
         [0097]    When the task controller  105  receives the decoding parameter set from the first decoder  108 , the task controller  105  shifts to the task addition state. The task list obtained at this time is as shown in  FIG. 8I . The “task” of the received decoding parameter set represents the “decoded image creating process”. Therefore, the task controller  105  searches the task list  15  for a decoding parameter set which refers to the slice A which is a decoded image created by the task of the decoding parameter set E 10 . In the example in.  FIG. 8I , the created decoding parameter sets (“decoded image creating process”) which refer to the slice A are the two decoding parameter sets E 6  and E 11 . Therefore, a priority (“2”) of the task of E 9  being processed by the second decoder  109  which is not in the waiting state is compared with lower one (i.e., a priority “10” of E 11 ) of the priorities of the two searched decoding parameter sets (E 6  and E 11 ). In this case, the priority of E 9  being processed by the second decoder  109  is lower. Therefore, the task controller  105  interrupts the process for the task of E 9  to insert the task of E 6  into the process of the second decoder  109 . More specifically, the task controller  105  outputs a process interrupt signal to the second decoder  109 , and thereafter waits until the decoding parameter set is received (or inputted) from the second decoder  109 . 
         [0098]    As described above, the process for E 9  in the decoder is started once. However, when the processes for the decoding parameter sets E 6  and E 11  having higher priorities can be performed, the process for E 9  is interrupted, and the decoding parameter sets E 6  and E 11  having the higher priorities are preferentially processed. 
         [0099]    When the second decoder  109  receives the process interrupt signal, the decoder  109  interrupts the arithmetic decoding process for E 9  being processed at the present, outputs the decoding parameter set E 9  to the task controller  105 , and shifts to the waiting state. At this time, various pointers in the decoding parameter set E 9  are set to values obtained at an interrupt point of time so that the interrupted process can be resumed later. 
         [0100]    The task controller  105  shifts to the task allocation state after the decoding parameter set E 9  inputted from the second decoder  109  is added to the pending task list  17 . 
         [0101]    The task list  15  obtained at this time is as shown in  FIG. 8I , and two decoders  108  and  109  are in the waiting state. In the task allocation state, the decoding parameter set having the highest parameter set in the task list  15  and the pending task list  17  is E 6 , and the decoded image creating process for the slice A is already completed. Thus, the task controller  105  outputs E 6  to the decoder  108  and deletes E 6  from the task list  15 . The decoding parameter set having the second highest priority is E 11 , and the decoded image creating process for the slice A is already completed. Thus, the task controller  105  outputs E 11  to the second decoder  109 , and deletes E 11  from the task list  15 . Thereafter, the task controller  105  shifts to the task completion waiting state. 
         [0102]    When the first and second decoders  108  and  109  receive the decoding parameter set from the task controller  105 , the first and second decoders  108  and  109  shift to the processing state again. The first decoder  108  and the second decoder  109  perform the decoded image creating process with reference to the received decoding parameter set. 
         [0103]    Both the first and second decoders  108  and  109  complete the decoded image creating processes at time t=13, output the received decoding parameter sets E 6  and E 11  to the task controller  105  as they are, and shift to the waiting state. 
         [0104]    When receiving the decoding parameter set E 6  from the first decoder  108 , the task controller  105  shifts to the task addition state. Although the “task” of the received decoding parameter set E 6  is the “decoded image creating process”, a decoding parameter set which refers to the slice B serving as the created decoded image and has the “task” as the “decoded image creating process” is not present in the task list  15 . Therefore, the task controller  105  shifts to the task allocation state without performing any operation. The number of the task list  15  obtained at this time is 0. The task controller  105  refers to the pending task list  17 . Since the decoding parameter set E 9  is present in the pending task list  17 , the task controller  105  outputs E 9  to the first decoder  108  and deletes E 9  from the pending task list  17 . Thereafter, the task controller  105  shifts to the task completion waiting state. 
         [0105]    When receiving the decoding parameter set E 11  from the second decoder  109 , the task controller  105  shifts to the task addition state. The “task” of the received decoding parameter set E 11  is the “decoded image creating process”. However, a decoding parameter set which refers to the slice C as the created decoded image and has the “task” as the “decoded image creating process” is not present in the task list  15 . For this reason, the task controller  105  shifts to the task allocation state without performing any operation. The number of elements of the task list obtained at this time is 0. Although the number of elements of the pending task list  17  is also 0, since the first decoder  108  is processing, the task controller  105  shifts to the task completion waiting state without any operation. 
         [0106]    A time sequence for the decoding parameter sets processed by the first decoder  108  and the second decoder  109  as described above is shown in  FIG. 9 . It should be noted that the process for the decoding parameter set E 9  is temporarily interrupted and restarted thereafter. 
         [0107]    The decoding process for the slice C is similarly repeated to complete decoding of one GOP. 
         [0108]    According to the above method, even before a decoding process for a frame belonging to group A is completed, a decoding process for a frame belonging to group B or group C can be started, and thus an idle time of the decoder (processor) can be reduced. 
         [0109]    In the first embodiment, the number of decoders is explained as two. However, the number of decoder may be three or more. In the first embodiment, the number of slices included in one frame is explained as one. However, the number of slices may be two or more. 
       Second Embodiment 
       [0110]    An image decoding apparatus according to a second embodiment receives a byte stream of a video data compressed in an encoding method such as H.264 and outputs a decoded digital image.  FIG. 10  shows a configuration of the image decoding apparatus according to the second embodiment. Also in the second embodiment, as in the first embodiment, a decoding process for one frame is divided into three steps including an arithmetic decoding process, a difference image creating process, and a decoded image creating process. 
         [0111]    An image decoding apparatus  201  has the same functions as those of the image decoding apparatus  101  according to the first embodiment. The image decoding apparatus  201  receives (inputs) a byte stream from an input terminal  202  and outputs a digital image from an output terminal  203 . A stream buffer  204 , a task controller  205 , an intermediate buffer  206 , and a frame buffer  207  operate in the same manner as the stream buffer  104 , the task controller  105 , the intermediate buffer  106 , and the frame buffer  107  described in the first embodiment, respectively. First to fourth decoders  208  to  211  operate in the same manner as the first decoder  108  and the second decoder  109  according to the first embodiment. 
         [0112]      FIG. 11  shows frames and steps processed by the first to fourth decoders  208  to  211  in the second embodiment. Regarding the figures in parenthesis shown in  FIG. 11 , the first number in parentheses represents a frame number of a frame to be processed, and a second number in parentheses represents a step in a decoding process. Regarding the step in the decoding process, an arithmetic decoding process is defined as “0”, a difference image creating process is defined as “1”, and a decoded image creating process is defined as “2”. For example, with respect to a process which is performed for the first time by the first decoder  208 , (0,0) is shown in  FIG. 11 . This means an “arithmetic decoding process” for “frame  0 ”. In comparison with a conventional case in  FIG. 12 , it can be understood that a time for which the decoder is in an idle state is considerably reduced and that a time required for all the processes is also reduced. 
         [0113]    The second embodiment exemplifies that one GOP is entirely decoded. However, the stream buffer  104  may buffer only a necessary byte stream, or the task controller  105  may be configured not to output an unnecessary decoding parameter set. In this manner, one arbitrary frame can be decoded. A plurality of GOPs may also be decoded in block. 
         [0114]    The functions of the image decoding apparatuses explained in the first and second embodiments may be realized by hardware-such as an electron IC circuit or may be realized by a combination of a computer such as a CPU and a control program (software). 
       INDUSTRIAL APPLICABILITY  
       [0115]    According to an image decoding apparatus of the present invention, even though frames depend on each other, an idle time of a processor can be shortened, and a decoding process can be performed at a high speed. Therefore the image decoding apparatus is useful to a software decoder or the like operated on a multiprocessor. 
         [0116]    The specific embodiments are described above. However, many other modifications, corrections, and other applications are apparent to a person skilled in the art. For this reason, the present invention is not limited to the specific disclosure mentioned here, and can be limited to only the appended claims. This application is related to Japanese Patent Application No. 2008-201444 (filed on Aug. 5, 2008) and Japanese Patent Application No. 2009-099808 (filed on Apr. 16, 2009). The contents of the applications are incorporated herein by reference.