An image processing apparatus comprising at least one processor that functions as: a first calculation unit (1002, 1003) configured to calculate a first color difference value (Cr1, Cb1) between a pixel value of a first color and a pixel value of a second color of pixels in first image data (G,B,R) obtained according to an optical image formed on an image pickup device via an image pickup optical member; a restoration processing unit (1004, 1005, 1006) configured to perform restoration processing on the first image data (G,B,R), using a restoration filter operating based on an optical transfer function of the image pickup optical member, to thereby generate second image data (G,B,R); a second calculation unit (1008, 1009) configured to calculate a second color difference value (Cr2, Cb2) between a pixel value of the first color and a pixel value of the second color of pixels in the second image data; and a correction unit (1010, 1011) configured to correct pixel values of the second image data according to a result of comparison between an amount of change in the second color difference value with respect to the first color difference value, and a threshold value (Th1r, Th1b), wherein, in a case where the result of comparison is larger than the threshold value (Th1r, Th1b), said correction unit (1010, 1011) is configured to correct the pixel values of the second image data such that the result of comparison will not be larger than the threshold value, characterized in that said correction unit (1010, 1011) is configured to set the threshold value based on a gain applied when generating the first image data (G, B, R). a first calculation unit (1002, 1003) configured to calculate a first color difference value (Cr1, Cb1) between a pixel value of a first color and a pixel value of a second color of pixels in first image data (G,B,R) obtained according to an optical image formed on an image pickup device via an image pickup optical member; a restoration processing unit (1004, 1005, 1006) configured to perform restoration processing on the first image data (G,B,R), using a restoration filter operating based on an optical transfer function of the image pickup optical member, to thereby generate second image data (G,B,R); a second calculation unit (1008, 1009) configured to calculate a second color difference value (Cr2, Cb2) between a pixel value of the first color and a pixel value of the second color of pixels in the second image data; and a correction unit (1010, 1011) configured to correct pixel values of the second image data according to a result of comparison between an amount of change in the second color difference value with respect to the first color difference value, and a threshold value (Th1r, Th1b), wherein, in a case where the result of comparison is larger than the threshold value (Th1r, Th1b), said correction unit (1010, 1011) is configured to correct the pixel values of the second image data such that the result of comparison will not be larger than the threshold value, characterized in that said correction unit (1010, 1011) is configured to set the threshold value based on a gain applied when generating the first image data (G, B, R).