An image processing apparatus that restores an image quality of an image obtained based on an optical image formed on an image pickup device via an image pickup optical system, from degradation, comprising: a first color feature amount calculation unit (1001, 1002, 1003, 1004, 1005) configured to calculate, as a first color feature amount, a feature amount of a color of a pixel in the image before a predetermined restoration processing, wherein the first color feature amount indicates a color difference, which is a difference in signal value between a first color in the image before the predetermined restoration processing as a reference color, and each of a second color and a third color in the image before the predetermined restoration processing, an image restoration processing unit (1006, 1007, 1008) configured to perform the predetermined restoration processing on the image, using an image restoration filter; a second color feature amount calculation unit (1009, 1010, 1011, 1012, 1013) configured to calculate, as a second color feature amount, a feature amount of a color of a pixel in an image after the predetermined restoration processing, which is output from said image restoration processing unit, wherein the second color feature amount indicates a color difference, which is a difference in signal value between the first color in the image after the predetermined restoration processing as a reference color, and each of a second color and a third color in the image after the predetermined restoration processing; an attenuation unit (1004, 1005, 1012, 1013) configured to attenuate high-frequency components included in the first color feature amount and the second color feature amount according to a photographing condition at a time of photographing of the image, wherein said attenuation unit (1004, 1005, 1012, 1013) sets an amount of attenuation of the high-frequency components according to an aperture value of the image pickup device which is the photographing condition; and a correction unit (1014, 1015) configured to correct a pixel value in the image after the predetermined restoration processing, based on the first color feature amount and the second color feature amount of which the high-frequency components are attenuated by said attenuation unit. a first color feature amount calculation unit (1001, 1002, 1003, 1004, 1005) configured to calculate, as a first color feature amount, a feature amount of a color of a pixel in the image before a predetermined restoration processing, wherein the first color feature amount indicates a color difference, which is a difference in signal value between a first color in the image before the predetermined restoration processing as a reference color, and each of a second color and a third color in the image before the predetermined restoration processing, an image restoration processing unit (1006, 1007, 1008) configured to perform the predetermined restoration processing on the image, using an image restoration filter; a second color feature amount calculation unit (1009, 1010, 1011, 1012, 1013) configured to calculate, as a second color feature amount, a feature amount of a color of a pixel in an image after the predetermined restoration processing, which is output from said image restoration processing unit, wherein the second color feature amount indicates a color difference, which is a difference in signal value between the first color in the image after the predetermined restoration processing as a reference color, and each of a second color and a third color in the image after the predetermined restoration processing; an attenuation unit (1004, 1005, 1012, 1013) configured to attenuate high-frequency components included in the first color feature amount and the second color feature amount according to a photographing condition at a time of photographing of the image, wherein said attenuation unit (1004, 1005, 1012, 1013) sets an amount of attenuation of the high-frequency components according to an aperture value of the image pickup device which is the photographing condition; and a correction unit (1014, 1015) configured to correct a pixel value in the image after the predetermined restoration processing, based on the first color feature amount and the second color feature amount of which the high-frequency components are attenuated by said attenuation unit.