An image processing device comprising: a depth map generation unit configured to generate a depth map through a matching process using a first image generated by a first imaging unit which has a pixel configuration including a first pixel group formed by pixels with polarization characteristics and a second pixel group formed by pixels having a polarization direction different from a polarization direction of the first pixel group or pixels with no polarization characteristics and a second image generated by a second imaging unit which has a pixel configuration including a third pixel group formed by pixels having a different polarization direction from the polarization direction of the first image at a position corresponding to the position of the first pixel group and a fourth pixel group formed by pixels with the same configuration as a configuration of the second pixel group at a position corresponding to the position of the second pixel group; a normal-line map generation unit configured to generate a normal-line map based on luminance of a polarized image generated based on the first and second images generated by the first imaging unit and the second imaging unit, the polarized image having three or more polarization directions; and a map unifying unit configured to perform a process of unifying the depth map generated by the depth map generation unit and the normal-line map generated by the normal-line map generation unit, wherein the map unifying unit estimates a depth value not indicated in the depth map from a shape determined based on a depth value indicated in the depth map and the normal-line map. a depth map generation unit configured to generate a depth map through a matching process using a first image generated by a first imaging unit which has a pixel configuration including a first pixel group formed by pixels with polarization characteristics and a second pixel group formed by pixels having a polarization direction different from a polarization direction of the first pixel group or pixels with no polarization characteristics and a second image generated by a second imaging unit which has a pixel configuration including a third pixel group formed by pixels having a different polarization direction from the polarization direction of the first image at a position corresponding to the position of the first pixel group and a fourth pixel group formed by pixels with the same configuration as a configuration of the second pixel group at a position corresponding to the position of the second pixel group; a normal-line map generation unit configured to generate a normal-line map based on luminance of a polarized image generated based on the first and second images generated by the first imaging unit and the second imaging unit, the polarized image having three or more polarization directions; and a map unifying unit configured to perform a process of unifying the depth map generated by the depth map generation unit and the normal-line map generated by the normal-line map generation unit, wherein the map unifying unit estimates a depth value not indicated in the depth map from a shape determined based on a depth value indicated in the depth map and the normal-line map.