An image encoding method for encoding into a plurality of frames three-dimensional information that is used for generating minute viewpoint movement adapted image, comprising: receiving a first image (PICm) captured by a main camera and a second image (PICs) captured simultaneously with the first image by a sub camera provided in the same plane with the main camera; for each pixel in the first image (PICm) comparing a pixel value of the pixel at first coordinates and a pixel value of the pixel at the same first coordinates in the second image (PICs) to obtain a first difference value; determining that the first coordinates are in a surface region in a case where the first difference value is not more than a first threshold value, and that the coordinates are in a back region in a case where the first difference value is more than the first threshold value, and recording information indicating whether the first coordinates are in the surface region or the back region and recording pixel values at the first coordinates being in the surface region into a first frame for output, the surface region being a region of a frame where there are objects and the only region for recording pixel data of the first image (PICm) in that frame, the back region being a region in a frame where there are objects but is a hidden region that is not seen due to an object in front, and, in the back regions, no pixel data of the first image (PICm) is recorded; pixel-shifting the second image (PICs) by a first shift amount in a direction based on a positional relationship between the main camera and the sub camera; comparing the pixel value of each pixel at the first coordinates of the first image which have been in the back region in the first frame with a pixel value of each pixel at the first coordinates of the second image after the pixel shift to obtain a second difference value, determining whether the first coordinates are in the surface region or the back region on a basis of the second difference value, recording information indicating whether those first coordinates are in the surface region or the back region into a second frame different from the first frame, recording the coordinates, whose region is recorded as the surface region in the first frame, as a transparent region into the second frame, the transparent region being a region which does not hold pixel values, and recording pixel values at the first coordinates being in the surface region in the second frame into the second frame for output; and repeating the same processing until the back regions are eliminated or a shift for a predetermined number of pixels is completed. receiving a first image (PICm) captured by a main camera and a second image (PICs) captured simultaneously with the first image by a sub camera provided in the same plane with the main camera; for each pixel in the first image (PICm) comparing a pixel value of the pixel at first coordinates and a pixel value of the pixel at the same first coordinates in the second image (PICs) to obtain a first difference value; determining that the first coordinates are in a surface region in a case where the first difference value is not more than a first threshold value, and that the coordinates are in a back region in a case where the first difference value is more than the first threshold value, and recording information indicating whether the first coordinates are in the surface region or the back region and recording pixel values at the first coordinates being in the surface region into a first frame for output, the surface region being a region of a frame where there are objects and the only region for recording pixel data of the first image (PICm) in that frame, the back region being a region in a frame where there are objects but is a hidden region that is not seen due to an object in front, and, in the back regions, no pixel data of the first image (PICm) is recorded; pixel-shifting the second image (PICs) by a first shift amount in a direction based on a positional relationship between the main camera and the sub camera; comparing the pixel value of each pixel at the first coordinates of the first image which have been in the back region in the first frame with a pixel value of each pixel at the first coordinates of the second image after the pixel shift to obtain a second difference value, determining whether the first coordinates are in the surface region or the back region on a basis of the second difference value, recording information indicating whether those first coordinates are in the surface region or the back region into a second frame different from the first frame, recording the coordinates, whose region is recorded as the surface region in the first frame, as a transparent region into the second frame, the transparent region being a region which does not hold pixel values, and recording pixel values at the first coordinates being in the surface region in the second frame into the second frame for output; and repeating the same processing until the back regions are eliminated or a shift for a predetermined number of pixels is completed.