An image pickup device having a first semiconductor substrate and a second semiconductor substrate, the first semiconductor substrate and second semiconductor substrate being mutually stacked, the image pickup device comprising: an imaging unit provided on the first semiconductor substrate and having a plurality of pixels configured to receive and photoelectrically convert incident light; an AD conversion unit configured to convert an analog image signal output from the imaging unit to digital image data; a frame memory provided on the second semiconductor substrate and configured to store captured digital image data of more than one frame converted by the AD conversion unit; and a generating unit provided on the second semiconductor substrate and configured to generate HDR image data from digital image data of a plurality of images with different exposures stored in the frame memory, the digital image data of the plurality of images with different exposures being generated based on the captured digital image data of the more than one frame, characterized in that the imaging unit serially performs a plurality of capturing operations, and the generating unit synthesizes the captured digital image data of the more than one frame acquired by the plurality of capturing operations to generate the digital image data of the plurality of images with different exposures, wherein the generating unit generates the digital image data with an exposure lower than a predetermined exposure from the first captured digital image data, the digital image data with the predetermined exposure from the first and second captured digital image data, and the digital image data with an exposure higher than the predetermined exposure from the first, second, and third captured digital image data, wherein the first, second, and third captured digital image data are captured with different exposures, said exposures being controlled based on charge storage periods. an imaging unit provided on the first semiconductor substrate and having a plurality of pixels configured to receive and photoelectrically convert incident light; an AD conversion unit configured to convert an analog image signal output from the imaging unit to digital image data; a frame memory provided on the second semiconductor substrate and configured to store captured digital image data of more than one frame converted by the AD conversion unit; and a generating unit provided on the second semiconductor substrate and configured to generate HDR image data from digital image data of a plurality of images with different exposures stored in the frame memory, the digital image data of the plurality of images with different exposures being generated based on the captured digital image data of the more than one frame, characterized in that the imaging unit serially performs a plurality of capturing operations, and the generating unit synthesizes the captured digital image data of the more than one frame acquired by the plurality of capturing operations to generate the digital image data of the plurality of images with different exposures, wherein the generating unit generates the digital image data with an exposure lower than a predetermined exposure from the first captured digital image data, the digital image data with the predetermined exposure from the first and second captured digital image data, and the digital image data with an exposure higher than the predetermined exposure from the first, second, and third captured digital image data, wherein the first, second, and third captured digital image data are captured with different exposures, said exposures being controlled based on charge storage periods.