An optical transmission system to perform optical transmission using a spatial multiplexing optical fiber as an optical transmission line for transmission of signal light, comprising: the spatial multiplexing optical fiber; a transmitting system arranged on one end side of the spatial multiplexing optical fiber, the transmitting system having a light source to emit light, and a modulator to generate the signal light by modulating the light emitted from the light source, in accordance with a predetermined modulation format; and a receiving system arranged on the other end side of the spatial multiplexing optical fiber, the receiving system receiving the signal light from the transmitting system through the spatial multiplexing optical fiber, characterized in that when a signal light power is defined as P signal, the shortest distance between modulation symbols in a modulation diagram showing a signal arrangement of the modulation format, as S, and a signal light intensity ratio PSR between the modulation symbols as PSR = P sigal /S 2, under conditions that a transmission quality Q-factor necessary for error correction in the optical transmission is Q FEC, a margin of the transmission quality Q-factor desired for safety in the optical transmission is Q margin, a mean crosstalk to a certain propagation mode of the spatial multiplexing optical fiber is XT μ, and the Q-factor of the signal light, which propagates in only the certain propagation mode of the spatial multiplexing optical fiber after modulation in accordance with the modulation format, is Q noise, the modulator generates the signal light by modulating the light emitted from the light source, in accordance with the modulation format where said PSR falls within the range of not more than PSR max in Equation below and not less than PSR max (2 1/2 -1)/2 1/2. PSR max = 1 X T μ 1 Q FEC 2 Q margin 2 − 1 Q noise 2 the spatial multiplexing optical fiber; a transmitting system arranged on one end side of the spatial multiplexing optical fiber, the transmitting system having a light source to emit light, and a modulator to generate the signal light by modulating the light emitted from the light source, in accordance with a predetermined modulation format; and a receiving system arranged on the other end side of the spatial multiplexing optical fiber, the receiving system receiving the signal light from the transmitting system through the spatial multiplexing optical fiber, characterized in that when a signal light power is defined as P signal, the shortest distance between modulation symbols in a modulation diagram showing a signal arrangement of the modulation format, as S, and a signal light intensity ratio PSR between the modulation symbols as PSR = P sigal /S 2, under conditions that a transmission quality Q-factor necessary for error correction in the optical transmission is Q FEC, a margin of the transmission quality Q-factor desired for safety in the optical transmission is Q margin, a mean crosstalk to a certain propagation mode of the spatial multiplexing optical fiber is XT μ, and the Q-factor of the signal light, which propagates in only the certain propagation mode of the spatial multiplexing optical fiber after modulation in accordance with the modulation format, is Q noise, the modulator generates the signal light by modulating the light emitted from the light source, in accordance with the modulation format where said PSR falls within the range of not more than PSR max in Equation below and not less than PSR max (2 1/2 -1)/2 1/2. PSR max = 1 X T μ 1 Q FEC 2 Q margin 2 − 1 Q noise 2