A three-dimensional measurement device, comprising: a predetermined optical system configured to split a predetermined incident light into two lights, to radiate one of the two lights as a measurement light to a measurement object and the other of the two lights as a reference light to a reference surface, and to recombine the two lights to a combined light and emit the combined light; a first irradiator configured to emit a first light that includes a polarized light of a first wavelength (λ 1) and enters a first input-output element of the predetermined optical system; a second irradiator configured to emit a second light that includes a polarized light of a second wavelength (λ 2), which is different from the first wavelength (λ 1), and enters a second input-output element of the predetermined optical system; a first imaging unit configured to take an image of an output light with regard to the first light that is emitted from the second input-output element when the first light enters the first input-output element; a second imaging unit configured to take an image of an output light with regard to the second light that is emitted from the first input-output element when the second light enters the second input-output element; and an image processor configured to perform three-dimensional measurement of the measurement object, based on interference fringe images taken by the first imaging unit and the second imaging unit, characterized in that the predetermined optical system comprises a polarizing beam splitter, the first input-output element is a first surface of the polarizing beam splitter, and the second input-output element is a second surface of the polarizing beam splitter, which is different from the first surface. a predetermined optical system configured to split a predetermined incident light into two lights, to radiate one of the two lights as a measurement light to a measurement object and the other of the two lights as a reference light to a reference surface, and to recombine the two lights to a combined light and emit the combined light; a first irradiator configured to emit a first light that includes a polarized light of a first wavelength (λ 1) and enters a first input-output element of the predetermined optical system; a second irradiator configured to emit a second light that includes a polarized light of a second wavelength (λ 2), which is different from the first wavelength (λ 1), and enters a second input-output element of the predetermined optical system; a first imaging unit configured to take an image of an output light with regard to the first light that is emitted from the second input-output element when the first light enters the first input-output element; a second imaging unit configured to take an image of an output light with regard to the second light that is emitted from the first input-output element when the second light enters the second input-output element; and an image processor configured to perform three-dimensional measurement of the measurement object, based on interference fringe images taken by the first imaging unit and the second imaging unit, characterized in that the predetermined optical system comprises a polarizing beam splitter, the first input-output element is a first surface of the polarizing beam splitter, and the second input-output element is a second surface of the polarizing beam splitter, which is different from the first surface.