An imaging apparatus which generates, based on an image of an object, a depth map indicating a distance from said imaging apparatus to the object, said imaging apparatus comprising: an image sensor which captures light at an image capturing plane, converts the light into an electrical signal for each pixel, and outputs the electrical signal; a sensor drive unit configured to arbitrarily shift a position in an optical axis direction of said image sensor; an image capture unit configured to capture an image captured by said image sensor, and hold the captured image; a sensor drive control unit configured to control operations of said sensor drive unit and said image capture unit such that a plurality of images are captured at focal points different from each other and one of the images is a sweep image that is captured during exposure through a shift of the position in the optical axis direction of said image sensor; an all-in-focus image generation unit configured to generate, from the sweep image captured by said image capture unit, an all-in-focus image of which an entire region of the sweep image is focused; a blur amount calculation unit configured to calculate an amount of blur in each of a plurality of image regions of another image different from the sweep image; and a depth map generation unit configured to (i) calculate, from an amount of blur in each of the image regions of that other image calculated by said blur amount calculation unit and from an optical coefficient value of said imaging apparatus including a focal length of a lens, the distance between said imaging apparatus and the object in each of the image regions, and (ii) generate a depth map which indicates the calculated distance as a pixel value in each of the image regions, wherein said sensor drive control unit is configured to control said sensor drive unit and said image capture unit such that said image capture unit captures three kinds of images: (i) a near end image having a focal point at a near end of the object; (ii) a far end image having a focal point at a far end of the object; and (iii) the sweep image which is captured by exposure through a continuous shift of said image sensor from the far end to the near end of the object or from the near end to the far end of the object, and said blur amount calculation unit is configured to calculate, from (i) the near end image, (ii) the far end image, and (iii) the all-in-focus image, the amount of blur in each of the image regions of that other image. an image sensor which captures light at an image capturing plane, converts the light into an electrical signal for each pixel, and outputs the electrical signal; a sensor drive unit configured to arbitrarily shift a position in an optical axis direction of said image sensor; an image capture unit configured to capture an image captured by said image sensor, and hold the captured image; a sensor drive control unit configured to control operations of said sensor drive unit and said image capture unit such that a plurality of images are captured at focal points different from each other and one of the images is a sweep image that is captured during exposure through a shift of the position in the optical axis direction of said image sensor; an all-in-focus image generation unit configured to generate, from the sweep image captured by said image capture unit, an all-in-focus image of which an entire region of the sweep image is focused; a blur amount calculation unit configured to calculate an amount of blur in each of a plurality of image regions of another image different from the sweep image; and a depth map generation unit configured to (i) calculate, from an amount of blur in each of the image regions of that other image calculated by said blur amount calculation unit and from an optical coefficient value of said imaging apparatus including a focal length of a lens, the distance between said imaging apparatus and the object in each of the image regions, and (ii) generate a depth map which indicates the calculated distance as a pixel value in each of the image regions, wherein said sensor drive control unit is configured to control said sensor drive unit and said image capture unit such that said image capture unit captures three kinds of images: (i) a near end image having a focal point at a near end of the object; (ii) a far end image having a focal point at a far end of the object; and (iii) the sweep image which is captured by exposure through a continuous shift of said image sensor from the far end to the near end of the object or from the near end to the far end of the object, and said blur amount calculation unit is configured to calculate, from (i) the near end image, (ii) the far end image, and (iii) the all-in-focus image, the amount of blur in each of the image regions of that other image.