An image processing apparatus comprising: first shape obtaining means for obtaining a first surface shape of an object based on a first three-dimensional image obtained by imaging the object in a first deformation state; second shape obtaining means for obtaining a second surface shape of the object based on a second three-dimensional image obtained by imaging the object in a second deformation state; normal direction obtaining means for obtaining a normal direction on a surface concerning the first surface shape; correspondence relationship obtaining means for obtaining a correspondence relationship of a plurality of positions of surface corresponding points which are associated in a one-to-one correspondence between the first surface shape and the second surface shape; and deformation obtaining means for obtaining a deformation displacement field between the first image and the second image based on evaluation of an error of surface corresponding points of the first surface shape and the second surface shape; wherein the deformation obtaining means includes calculation means for calculating a coordinate transformation between the first deformation state and the second deformation state concerning the object using the evaluation of the error, wherein in the evaluation of the error an error of surface corresponding points in the normal direction of a curved surface of one of the first surface shape and the second surface shape is evaluated as larger than an error of surface corresponding points in a direction along the curved surface, and wherein the deformation obtaining means obtains the deformation displacement field between the first image and the second image to prevent a surface corresponding point of the first surface shape after the deformation displacement from shifting only in the normal direction from a surface corresponding point of the second surface shape based on the evaluation of the error. first shape obtaining means for obtaining a first surface shape of an object based on a first three-dimensional image obtained by imaging the object in a first deformation state; second shape obtaining means for obtaining a second surface shape of the object based on a second three-dimensional image obtained by imaging the object in a second deformation state; normal direction obtaining means for obtaining a normal direction on a surface concerning the first surface shape; correspondence relationship obtaining means for obtaining a correspondence relationship of a plurality of positions of surface corresponding points which are associated in a one-to-one correspondence between the first surface shape and the second surface shape; and deformation obtaining means for obtaining a deformation displacement field between the first image and the second image based on evaluation of an error of surface corresponding points of the first surface shape and the second surface shape; wherein the deformation obtaining means includes calculation means for calculating a coordinate transformation between the first deformation state and the second deformation state concerning the object using the evaluation of the error, wherein in the evaluation of the error an error of surface corresponding points in the normal direction of a curved surface of one of the first surface shape and the second surface shape is evaluated as larger than an error of surface corresponding points in a direction along the curved surface, and wherein the deformation obtaining means obtains the deformation displacement field between the first image and the second image to prevent a surface corresponding point of the first surface shape after the deformation displacement from shifting only in the normal direction from a surface corresponding point of the second surface shape based on the evaluation of the error.