A surface analysis device including a specimen stage having a height direction moving mechanism for moving a specimen in a direction of an optical axis of the surface analysis device and a tilting mechanism for tilting the specimen about a tilt axis, the surface analysis device comprising a control section for controlling the specimen stage, the control section being configured to execute: specimen image acquisition processing for obtaining a first specimen image captured with a first tilt angle of the specimen and a second specimen image captured with a second tilt angle of the specimen which is different from the first tilt angle; deviation amount calculation processing for calculating an amount of deviation of a specimen surface (ΔZ) from a reference height defined by the intersection between the tilt axis and the optical axis based on the first specimen image and the second specimen image; and height control processing for controlling the specimen stage based on the amount of deviation of the specimen surface (ΔZ) to align the height of the specimen surface at said intersection between the tilt axis and the optical axis, characterized in that: the control section is further configured to execute, in the deviation amount calculation processing, generating a plurality of first partial images by cutting out a plurality of areas from the first specimen image, generating a plurality of second partial images by cutting out a plurality of areas corresponding to the plurality of areas cut out from the first specimen image, from the second specimen image, enlarging the plurality of first and second partial images to make them the same size as the first specimen image and the second specimen image, respectively, calculating respective amounts of deviation of the cut-out areas from the reference height by measuring respective shift amounts between corresponding images of the first partial images and the second partial images determining a median value from the amounts of deviation of the cut-out areas from the reference height, calculated from the respective areas, and setting the median value as the amount of deviation of the specimen surface. specimen image acquisition processing for obtaining a first specimen image captured with a first tilt angle of the specimen and a second specimen image captured with a second tilt angle of the specimen which is different from the first tilt angle; deviation amount calculation processing for calculating an amount of deviation of a specimen surface (ΔZ) from a reference height defined by the intersection between the tilt axis and the optical axis based on the first specimen image and the second specimen image; and height control processing for controlling the specimen stage based on the amount of deviation of the specimen surface (ΔZ) to align the height of the specimen surface at said intersection between the tilt axis and the optical axis, the control section is further configured to execute, in the deviation amount calculation processing, generating a plurality of first partial images by cutting out a plurality of areas from the first specimen image, generating a plurality of second partial images by cutting out a plurality of areas corresponding to the plurality of areas cut out from the first specimen image, from the second specimen image, enlarging the plurality of first and second partial images to make them the same size as the first specimen image and the second specimen image, respectively, calculating respective amounts of deviation of the cut-out areas from the reference height by measuring respective shift amounts between corresponding images of the first partial images and the second partial images determining a median value from the amounts of deviation of the cut-out areas from the reference height, calculated from the respective areas, and setting the median value as the amount of deviation of the specimen surface.