An image generation apparatus that applies processing to an original image to generate a display image, comprising: an original image operation unit configured to calculate a displacement to be generated for each pixel of the original image for the processing and generate a vector map in which reference vectors for referring to positions before the displacement on the original images are lined up on an image plane after the displacement, wherein a pixel center of a pixel of the original image after the displacement does not coincide with a pixel center of a pixel of the vector map; a display image generation unit configured to determine the reference vector corresponding to each pixel of the display image by referring to the vector map and acquire, based on the determined reference vectors, pixel values at the corresponding positions on the original image to determine pixel values of the display image; and an outputting unit configured to output data of the display image, wherein the image generation unit processes a pair of the original images acquired from left and right points of view different from each other to generate the display image that allows stereoscopic viewing of an object, the original image operation unit calculates a displacement to be generated for each pixel of the original images in response to movement of a point of view of a user and generates the vector map, and the display image generation unit determines the reference vector corresponding to each pixel of the display images by sampling the reference vectors retained in the vector map, characterized in that the original image operation unit refers to a parallax value image, which represents values of a parallax given to the original images on an image plane, to set the reference vectors to corresponding pixels of the vector map and set, to any pixel with regard to which a value of the parallax does not exist, data that the reference vector is invalid, and the display image generation unit decides any pixel whose reference vector in the vector maps is invalid, interpolate the reference vector of the pixel using surrounding reference vectors. an original image operation unit configured to calculate a displacement to be generated for each pixel of the original image for the processing and generate a vector map in which reference vectors for referring to positions before the displacement on the original images are lined up on an image plane after the displacement, wherein a pixel center of a pixel of the original image after the displacement does not coincide with a pixel center of a pixel of the vector map; a display image generation unit configured to determine the reference vector corresponding to each pixel of the display image by referring to the vector map and acquire, based on the determined reference vectors, pixel values at the corresponding positions on the original image to determine pixel values of the display image; and an outputting unit configured to output data of the display image, wherein the image generation unit processes a pair of the original images acquired from left and right points of view different from each other to generate the display image that allows stereoscopic viewing of an object, the original image operation unit calculates a displacement to be generated for each pixel of the original images in response to movement of a point of view of a user and generates the vector map, and the display image generation unit determines the reference vector corresponding to each pixel of the display images by sampling the reference vectors retained in the vector map, characterized in that the original image operation unit refers to a parallax value image, which represents values of a parallax given to the original images on an image plane, to set the reference vectors to corresponding pixels of the vector map and set, to any pixel with regard to which a value of the parallax does not exist, data that the reference vector is invalid, and the display image generation unit decides any pixel whose reference vector in the vector maps is invalid, interpolate the reference vector of the pixel using surrounding reference vectors.