An optical response measuring device for an object for measurement comprising: a light source for generating a pair of light beams having light beams of a first wavelength and a second wavelength; a first wavelength conversion element, on which the pair of light beams is made incident, for generating measurement light of a measurement wavelength whose phase is maintained relative to the pair of light beams incident thereon; a second wavelength conversion element, on which a detection light, obtained from an illuminated area on an object for measurement irradiated by the measurement light, and reference light are made incident, for generating modulated reference light, the intensity of the reference light being modulated by the intensity or phase difference of the detection light at each point, wherein the second wavelength conversion element is arranged away from the object for measurement, wherein the reference light carries the phase of the pair of light beams without being influenced by the object for measurement, wherein the detection light includes light whose phase needs to be determined, wherein the detection light has a first phase and a second phase that are influenced by responses of the object for measurement to the measurement light at a first point under illumination and a second point under illumination in the illuminated area, wherein the modulated reference light is modulated to have a first local intensity and a second local intensity in accordance respectively with the first phase and the second phase of the detection light, and wherein the light whose phase needs to be determined in the detection light and the reference light are made incident on the second wavelength conversion element while being aligned with each other, such that a collinear phase matching condition is satisfied inside the second wavelength conversion element between the light whose phase needs to be determined in the detection light and the reference light for generating the modulated reference light; at least one optical element (142, 144) arranged in an optical path between the object for measurement and the second wavelength conversion element for aligning the reference light and the detection light; and a light intensity sensor array, on which the modulated reference light is made incident, for measuring the first local intensity and the second local intensity of the modulated reference light that correspond respectively to the first phase and the second phase of the detection light. a light source for generating a pair of light beams having light beams of a first wavelength and a second wavelength; a first wavelength conversion element, on which the pair of light beams is made incident, for generating measurement light of a measurement wavelength whose phase is maintained relative to the pair of light beams incident thereon; a second wavelength conversion element, on which a detection light, obtained from an illuminated area on an object for measurement irradiated by the measurement light, and reference light are made incident, for generating modulated reference light, the intensity of the reference light being modulated by the intensity or phase difference of the detection light at each point, wherein the second wavelength conversion element is arranged away from the object for measurement, wherein the reference light carries the phase of the pair of light beams without being influenced by the object for measurement, wherein the detection light includes light whose phase needs to be determined, wherein the detection light has a first phase and a second phase that are influenced by responses of the object for measurement to the measurement light at a first point under illumination and a second point under illumination in the illuminated area, wherein the modulated reference light is modulated to have a first local intensity and a second local intensity in accordance respectively with the first phase and the second phase of the detection light, and wherein the light whose phase needs to be determined in the detection light and the reference light are made incident on the second wavelength conversion element while being aligned with each other, such that a collinear phase matching condition is satisfied inside the second wavelength conversion element between the light whose phase needs to be determined in the detection light and the reference light for generating the modulated reference light; at least one optical element (142, 144) arranged in an optical path between the object for measurement and the second wavelength conversion element for aligning the reference light and the detection light; and a light intensity sensor array, on which the modulated reference light is made incident, for measuring the first local intensity and the second local intensity of the modulated reference light that correspond respectively to the first phase and the second phase of the detection light.