An image generation apparatus that applies processing to an original image to generate a display image, comprising: an original image operation unit configured to calculate a displacement to be generated for each pixel of the original image for the processing and generate a vector map, in which reference vectors for referring to positions before the displacement on the original images are lined up on an image plane, after the displacement; a display image generation unit configured to determine the reference vector corresponding to each pixel of the display image by referring to the vector map and acquire, based on the determined reference vectors, pixel values at the corresponding positions on the original image to determine pixel values of the display image; and an outputting unit configured to output data of the display image, wherein the image generation unit processes a pair of the original images acquired from left and right points of view different from each other to generate the display image that allows stereoscopic viewing of an object, the original image operation unit calculates a displacement to be generated for each pixel of the original images in response to movement of a point of view of a user and generates the vector map, and the display image generation unit determines the reference vector corresponding to each pixel of the display images by sampling the reference vectors retained in the vector map, wherein the original image operation unit refers to a parallax value image, which represents values of a parallax given to the original images on an image plane, to set the reference vectors to corresponding pixels of the vector map and set, to any pixel with regard to which a value of the parallax does not exist, data that the reference vector is invalid, characterized in that the display image generation unit decides, for any pixel whose reference vector in the vector maps is invalid, to interpolate the reference vector of the pixel using surrounding reference vectors. an original image operation unit configured to calculate a displacement to be generated for each pixel of the original image for the processing and generate a vector map, in which reference vectors for referring to positions before the displacement on the original images are lined up on an image plane, after the displacement; a display image generation unit configured to determine the reference vector corresponding to each pixel of the display image by referring to the vector map and acquire, based on the determined reference vectors, pixel values at the corresponding positions on the original image to determine pixel values of the display image; and an outputting unit configured to output data of the display image, wherein the image generation unit processes a pair of the original images acquired from left and right points of view different from each other to generate the display image that allows stereoscopic viewing of an object, the original image operation unit calculates a displacement to be generated for each pixel of the original images in response to movement of a point of view of a user and generates the vector map, and the display image generation unit determines the reference vector corresponding to each pixel of the display images by sampling the reference vectors retained in the vector map, wherein the original image operation unit refers to a parallax value image, which represents values of a parallax given to the original images on an image plane, to set the reference vectors to corresponding pixels of the vector map and set, to any pixel with regard to which a value of the parallax does not exist, data that the reference vector is invalid, characterized in that the display image generation unit decides, for any pixel whose reference vector in the vector maps is invalid, to interpolate the reference vector of the pixel using surrounding reference vectors.