An image decoding method for decoding image data formed by a plurality of frames, the image decoding method characterized by having a current category decoding step for decoding a category number j of a current frame, the current frame being classified into one of N categories, with N ≥ 2, a reference image specifying data decoding step for decoding a reference image specifying data which specifies a reference image data, for said decoded category number, a predicted image producing step for producing a predicted image from an image data specified by said reference image specifying data, wherein the image data is selected from image data of a plurality of frames of an i-th category which have been decoded previously, for the current frame classified as j-th category, wherein for the number of the categories 1 ≤ i ≤ j is applied, a difference decoding step for decoding a difference between a decoded image of the current frame and the predicted image, a decoded image producing step for producing the decoded image of the current frame from said decoded difference data and said predicted image, and a decoded image storing step for storing said produced decoded image data of the current frame into a reference image memory, the reference image memory having memories allocated to specific categories, wherein said reference image specifying data decoding step has a tentative frame number setting step for setting a tentative frame number based on an output order of frames with respect to the image data of a frame belonging to each of an i-th category which is the j-th category of the current frame and below, among a plurality of image data stored in said reference image memory and not setting the tentative frame number with respect to the image data of a frame belonging to categories other than the i-th category, the tentative frame number being a serial number, and a tentative frame number decoding step for obtaining the tentative frame number which specifies an image data to be selected at said predicted image producing step, by decoding the reference image specifying data. a current category decoding step for decoding a category number j of a current frame, the current frame being classified into one of N categories, with N ≥ 2, a reference image specifying data decoding step for decoding a reference image specifying data which specifies a reference image data, for said decoded category number, a predicted image producing step for producing a predicted image from an image data specified by said reference image specifying data, wherein the image data is selected from image data of a plurality of frames of an i-th category which have been decoded previously, for the current frame classified as j-th category, wherein for the number of the categories 1 ≤ i ≤ j is applied, a difference decoding step for decoding a difference between a decoded image of the current frame and the predicted image, a decoded image producing step for producing the decoded image of the current frame from said decoded difference data and said predicted image, and a decoded image storing step for storing said produced decoded image data of the current frame into a reference image memory, the reference image memory having memories allocated to specific categories, wherein said reference image specifying data decoding step has a tentative frame number setting step for setting a tentative frame number based on an output order of frames with respect to the image data of a frame belonging to each of an i-th category which is the j-th category of the current frame and below, among a plurality of image data stored in said reference image memory and not setting the tentative frame number with respect to the image data of a frame belonging to categories other than the i-th category, the tentative frame number being a serial number, and a tentative frame number decoding step for obtaining the tentative frame number which specifies an image data to be selected at said predicted image producing step, by decoding the reference image specifying data.