A phase difference control device configured to be used in a spectrometer that performs spectrometry to a sample, the phase difference control device comprising: a splitting polarizer configured to split a light incident from a light source into a measurement light and a reference light, both of which are linearly polarized; a photoelastic modulator, PEM, configured to perform a phase modulating action to impart a phase difference to the measurement light and the reference light; a PEM control circuit configured to input a voltage signal that is converted from the reference light imparted with the phase difference as a feedback signal and to output a modulation control quantity signal; a PEM driver configured to receive the modulation control quantity signal and to supply a modulation voltage in accordance with the modulation control quantity signal to the PEM for making the PEM to perform the phase modulating action; and wherein the phase difference control device further comprises a central processing unit circuit, CPU, configured to monitor a wavelength of the light in the splitting polarizer and to input a wavelength variation thereof as a wavelength signal; the central processing unit circuit being configured to convert the wavelength signal to a feedforward signal, and the feedforward signal is output to the PEM control circuit; and the PEM control circuit being configured to perform arithmetic processing by the feedback signal and the feedforward signal to output the modulation control quantity signal to the PEM driver; and characterized in that the central processing unit circuit is configured to calculate the feedforward signal by using a table prepared in advance relative to the wavelength signal, by obtaining a convergent point of circular dichroism values by raising and lowering a PEM voltage, setting the convergent point as a CD maximum value, and setting a value of the PEM voltage at the time of obtaining the CD maximum value as a voltage maximum value of the modulation voltage. a splitting polarizer configured to split a light incident from a light source into a measurement light and a reference light, both of which are linearly polarized; a photoelastic modulator, PEM, configured to perform a phase modulating action to impart a phase difference to the measurement light and the reference light; a PEM control circuit configured to input a voltage signal that is converted from the reference light imparted with the phase difference as a feedback signal and to output a modulation control quantity signal; a PEM driver configured to receive the modulation control quantity signal and to supply a modulation voltage in accordance with the modulation control quantity signal to the PEM for making the PEM to perform the phase modulating action; and wherein the phase difference control device further comprises a central processing unit circuit, CPU, configured to monitor a wavelength of the light in the splitting polarizer and to input a wavelength variation thereof as a wavelength signal; the central processing unit circuit being configured to convert the wavelength signal to a feedforward signal, and the feedforward signal is output to the PEM control circuit; and the PEM control circuit being configured to perform arithmetic processing by the feedback signal and the feedforward signal to output the modulation control quantity signal to the PEM driver; and characterized in that the central processing unit circuit is configured to calculate the feedforward signal by using a table prepared in advance relative to the wavelength signal, by obtaining a convergent point of circular dichroism values by raising and lowering a PEM voltage, setting the convergent point as a CD maximum value, and setting a value of the PEM voltage at the time of obtaining the CD maximum value as a voltage maximum value of the modulation voltage.