A dispersion and nonlinearity compensation method in a coherent optical communication system, comprising: dividing a total length of an optical fiber into N spans with equal lengths, and in each span, firstly performing dispersion compensation on every data point within the span according to a dispersion compensation operator, and then performing nonlinearity compensation; wherein, the step of performing nonlinearity compensation in each span comprises: with a position of a current sampling point being a center, performing sampling on respective 2k+1 sampling points of X and Y polarization states, calculating powers of 2(2k+1) sampling points, adding powers of sampling points at a same position on X and Y polarization states to obtain 2k+1 power values, performing weighted summation on the 2k+1 power values, then multiplying the weighted sum by a preset coefficient W to obtain a nonlinear phase angle of the current sampling point, and performing compensation according to the nonlinear phase angle; wherein, N and k are positive integers; characterized in that the preset coefficient W is set according to different system configurations, the system configurations comprising an in-fiber power of every polarization state, transmission length and signal to noise ratio of the system. dividing a total length of an optical fiber into N spans with equal lengths, and in each span, firstly performing dispersion compensation on every data point within the span according to a dispersion compensation operator, and then performing nonlinearity compensation; wherein, the step of performing nonlinearity compensation in each span comprises: with a position of a current sampling point being a center, performing sampling on respective 2k+1 sampling points of X and Y polarization states, calculating powers of 2(2k+1) sampling points, adding powers of sampling points at a same position on X and Y polarization states to obtain 2k+1 power values, performing weighted summation on the 2k+1 power values, then multiplying the weighted sum by a preset coefficient W to obtain a nonlinear phase angle of the current sampling point, and performing compensation according to the nonlinear phase angle; wherein, N and k are positive integers; characterized in that the preset coefficient W is set according to different system configurations, the system configurations comprising an in-fiber power of every polarization state, transmission length and signal to noise ratio of the system. with a position of a current sampling point being a center, performing sampling on respective 2k+1 sampling points of X and Y polarization states, calculating powers of 2(2k+1) sampling points, adding powers of sampling points at a same position on X and Y polarization states to obtain 2k+1 power values, performing weighted summation on the 2k+1 power values, then multiplying the weighted sum by a preset coefficient W to obtain a nonlinear phase angle of the current sampling point, and performing compensation according to the nonlinear phase angle; wherein, N and k are positive integers; characterized in that the preset coefficient W is set according to different system configurations, the system configurations comprising an in-fiber power of every polarization state, transmission length and signal to noise ratio of the system.