A server apparatus arranged to execute a computer program based on instructions of a control device, the server apparatus comprising: a first image acquisition unit configured to acquire, based on a plurality of first images and a plurality of second images both captured from a plurality of directions by an image capturing means, three-dimensional shape data corresponding to an object, wherein the first images represent a structure area and an object area being distinguished from other areas and the second images represent the object area being distinguished from other areas; a second image acquisition unit (S1603, S1605) configured to acquire three-dimensional shape data corresponding to the structure; a third acquisition unit (S603, S604, S605) configured to acquire and store background image data corresponding to a background image excluding the object and the structure; a fourth acquisition unit configured to acquire information indicating a virtual viewpoint; and a generation unit (S1607, S1608) configured to generate a virtual viewpoint image based on the three-dimensional shape data corresponding to the object acquired by the first acquisition unit, the three-dimensional shape data corresponding to the structure acquired by the second image acquisition unit, the background data acquired by the third acquisition unit, and the information indicating the virtual viewpoint acquired by the fourth acquisition unit, wherein, the plurality of first images are received by the first image acquisition unit at a frequency lower than the plurality of second images; and a determination unit configured to: determine a selected image of interest as the first image or the second image; and a combining unit configured to, if the selected image of interest is the second image, combine the acquired three-dimensional shape data corresponding to the structure with the second image, wherein the second acquisition unit is configured to, if the selected image of interest is the first image, acquire the three-dimensional shape data corresponding to the structure by extracting an image area corresponding to the structure by combining a third image representing the object area distinguished from other areas with a captured image of a three dimensional space in a state where the structure does not exist corresponding to the stored background image data, and obtain a difference with the first image received by the first image acquisition unit. a first image acquisition unit configured to acquire, based on a plurality of first images and a plurality of second images both captured from a plurality of directions by an image capturing means, three-dimensional shape data corresponding to an object, wherein the first images represent a structure area and an object area being distinguished from other areas and the second images represent the object area being distinguished from other areas; a second image acquisition unit (S1603, S1605) configured to acquire three-dimensional shape data corresponding to the structure; a third acquisition unit (S603, S604, S605) configured to acquire and store background image data corresponding to a background image excluding the object and the structure; a fourth acquisition unit configured to acquire information indicating a virtual viewpoint; and a generation unit (S1607, S1608) configured to generate a virtual viewpoint image based on the three-dimensional shape data corresponding to the object acquired by the first acquisition unit, the three-dimensional shape data corresponding to the structure acquired by the second image acquisition unit, the background data acquired by the third acquisition unit, and the information indicating the virtual viewpoint acquired by the fourth acquisition unit, wherein, the plurality of first images are received by the first image acquisition unit at a frequency lower than the plurality of second images; and a determination unit configured to: determine a selected image of interest as the first image or the second image; and a combining unit configured to, if the selected image of interest is the second image, combine the acquired three-dimensional shape data corresponding to the structure with the second image, wherein the second acquisition unit is configured to, if the selected image of interest is the first image, acquire the three-dimensional shape data corresponding to the structure by extracting an image area corresponding to the structure by combining a third image representing the object area distinguished from other areas with a captured image of a three dimensional space in a state where the structure does not exist corresponding to the stored background image data, and obtain a difference with the first image received by the first image acquisition unit. determine a selected image of interest as the first image or the second image; and a combining unit configured to, if the selected image of interest is the second image, combine the acquired three-dimensional shape data corresponding to the structure with the second image, wherein the second acquisition unit is configured to, if the selected image of interest is the first image, acquire the three-dimensional shape data corresponding to the structure by extracting an image area corresponding to the structure by combining a third image representing the object area distinguished from other areas with a captured image of a three dimensional space in a state where the structure does not exist corresponding to the stored background image data, and obtain a difference with the first image received by the first image acquisition unit.