An image generation device for generating, from a first moving image (LH; G HL), which is a moving image of a first color component of a subject, and a second moving image (HL; R LH, B LH), which is a moving image of a second color component of the subject, an optimized high resolution moving color image (HH; R H, G H, B H) representing the subject, the image generation device comprising: an image acquisition section for acquiring the first moving image and the second moving image having different resolutions, and having different frame rates or different exposure times from each other; a light amount determination section for determining, based on a pixel value of each of pixels in each of frame images of the first moving image and the second moving image, whether or not each pixel is saturated or blocked-up-shadowed; and an image processing section for obtaining the optimized high resolution moving color image (HH; R H, G H, B H) having a frame rate which is equal to or higher than the frame rate of the first moving image and the second moving image and having a resolution of each frame image which is equal to or higher than the resolution of the first moving image and the second moving image by minimizing an evaluation function based on a current high resolution moving color image (R H, G H, B H), the first moving image (G HL) and the second moving image (R LH, B LH) including at least a first evaluation term representing an error between a pixel value of each of the acquired first moving image and second moving image and a sum of pixel values of frame images of a corresponding component of the current high resolution moving color image corresponding to the pixel value temporally and spatially, wherein said first evaluation term is suppressed for a pixel of a moving image which has been determined to be saturated or blocked-up-shadowed, the evaluation term further comprising a second evaluation term regarding spatial continuity which decreases with increasing spatial continuity of colors in the current high resolution moving color image, and the image processing section is configured to determine a weight to be applied, at a pixel position of each pixel in the current high resolution moving color image, in the second evaluation term, and to set the second evaluation term wherein the weight decreases at the pixel position where a spatial differential value of the acquired first or second moving image is large. an image acquisition section for acquiring the first moving image and the second moving image having different resolutions, and having different frame rates or different exposure times from each other; a light amount determination section for determining, based on a pixel value of each of pixels in each of frame images of the first moving image and the second moving image, whether or not each pixel is saturated or blocked-up-shadowed; and an image processing section for obtaining the optimized high resolution moving color image (HH; R H, G H, B H) having a frame rate which is equal to or higher than the frame rate of the first moving image and the second moving image and having a resolution of each frame image which is equal to or higher than the resolution of the first moving image and the second moving image by minimizing an evaluation function based on a current high resolution moving color image (R H, G H, B H), the first moving image (G HL) and the second moving image (R LH, B LH) including at least a first evaluation term representing an error between a pixel value of each of the acquired first moving image and second moving image and a sum of pixel values of frame images of a corresponding component of the current high resolution moving color image corresponding to the pixel value temporally and spatially, wherein said first evaluation term is suppressed for a pixel of a moving image which has been determined to be saturated or blocked-up-shadowed, the evaluation term further comprising a second evaluation term regarding spatial continuity which decreases with increasing spatial continuity of colors in the current high resolution moving color image, and the image processing section is configured to determine a weight to be applied, at a pixel position of each pixel in the current high resolution moving color image, in the second evaluation term, and to set the second evaluation term wherein the weight decreases at the pixel position where a spatial differential value of the acquired first or second moving image is large.