# Advanced DSP for Coherent Optical Fiber Communication
## 3 Nonlinearity Compensation
The performance of DBP algorithm mainly depends on the accurate estimation of NLSE propagation parameters. NLSE is usually solved by split-step Fourier method (SSFM). The optical fiber is divided into \(N\) segments, and each section has to be small enough to guarantee the optical power not to change much within the segment, since the nonlinear effects are power dependent. Another role is to ensure that the accumulation of dispersion and nonlinear effects is not so strong so as to cause serious interactions. Then the dispersion and the nonlinear effects can be considered independently in each small section, and can be executed successively (as an approximation to the simultaneous effect). In SSFM, the linear part and the nonlinear part are solved separately. The linear part is solved in frequency domain while the nonlinear part is solved in the time domain. Using backward propagated NLSE equation, all deterministic impairments (linear and nonlinear) in optical fiber transmission can be compensated. Compensation can be performed at the transmitter before signal transmission (pre-compensation) or at the receiver after signal detection (post-compensation). In most cases, DBP is implemented in conjunction with coherent receivers as a post-compensation. In the absence of  
Figure 7: Schematic of digital back propagation (DBP) using the symmetric split-step method (SSM). noise in the transmission link, the two DBP schemes are equivalent. Since the backward propagation operator acts on the complex envelope of the electrical field, the DBP algorithm can be applied to any modulation format in principle. The performance of DBP is limited by ASE noise, because it is a stochastic noise source and cannot be compensated. DBP can only compensate for deterministic signal distortions. The team from the University of Central Florida has studied the issue of nonlinearity compensation in WDM systems [26; 27; 28; 30; 31]. Nonlinear effects in optical fibers include self-phase modulation (SPM), XPM and FWM. Three nonlinear effects in optical fibers are separated using the coupling equation, and the system performance can be optimized by neglecting the items that have little influence on equalization results, but that require significant computation load. Compared with SPM and XPM, the computation load of FWM effect is much heavier. However, FWM needs phase matching to have a significant impact and the FWM effect can be neglected in non-zero dispersion shift optical fiber transmission system because it is far from the phase matching point. In this type of system, FWM items can be omitted and only the effects of SPM and XPM need to be compensated and the computational burden is reduced [26]. The simulation results are shown in Figures 8-10. Figures 9 and 10 show that XPM and FWM compensation produces the same Q value in 100 GHz channel spacing, and the influence of FWM is negligible. Larger channel spacing leads to a higher phase mismatch, and the FWM will contribute even less. But in the low dispersion region, FWM with phase matching condition will affect the performance of the transmitted signal significantly. There is a trade-off between the computational cost and the compensation effect for the FWM mitigation. The influence of the nearest channel and the second nearest channel has to be considered, while the influence of the far channel could be neglected. Using this approximation, the Q value of the observed channel can be increased with a small amount of computation [27]. The interchannel nonlinear effect XPM can be reduced by the walk-off effect caused by dispersion. The step size of SSFM can be greatly increased and the computation cost can be reduced by separating walk-off effect [28]. As shown in Figure 11, the performance of DSP improves dramatically for the XPM compensation. The step size requirements for XPM compensation using the conventional approach, and the approach by the authors of [28], are shown in Figure 12. It indicates that the step size can be increased substantially based on the SSFM.