# Recent Progress on Novel DSP Techniques for Mode Division Multiplexing Systems: A Review
## 5 Novel DSP Techniques for MIMO Systems
### Artificial Intelligence (AI) Technology for MIMO Equalization  
Recently, AI technologies have also attracted great interest in optical fiber communication, for its high excellent ability. Deep learning, as a main branch of AI, provides many neural network models to handle various optical communication problems. For example, artificial neural network (ANN) is widely utilized in nonlinear equalization [76; 77; 78] and optical performance monitoring [79; 80], convolutional neural network (CNN) has the ability to get high accuracy of output images in MMF [81] and enabled high-spatial-density SDM framework [82]. And DNN, which is more complex than ANN, shows its excellent performance in end-to-end learning [83]. What's more, the applications of deep learning in DSP bring a revolution in MIMO transmission systems. In this part, several neural networks are introduced to achieve MIMO equalization, and better performance can be obtained compared with the traditional methods. Figure 19: Transmission results for PDM-WDM system. (**a**) Q-factors vs. launch power in different algorithm; (**b**) Optimized amplitude spectra after linear operation in different fiber segment; (**c**) Optimized nonlinear coefficients in different fiber segment; (**d**) Optimized rotation angle for PSP. As mentioned above, mode coupling is inevitable in MDM system which causes channel crosstalk and DMD. Thanks to deep learning, ANN can be used as a MIMO equalizer to mitigate these channel impairments [84]. A two mode MDM transmission system with an ANN equalizer is illustrated in Figure 20a, the ANN equalizer is placed at the receiver side and consists of three layers as depicted in Figure 20b. Meanwhile, a novel feed-forward and back propagation algorithm is employed to update the weight, in which the back propagation retrieved the feed-forward output as an input to the hidden layer. Channel impulse response is analyzed to characterize the ANN performance. Figure 21a shows the input channel impulse response for two channels, in which both channels are coupled with each other significantly. Figure 21b,c are the target and output channel impulse response, respectively. There is a high similarity between these two pictures where the impulse response of both channels is separated clearly. Hence, this ANN equalizer has a good performance in compensating mode coupling and DMD. Figure 20: (**a**) MDM transmission system with ANN equalizer; (**b**) The structure of ANN. Figure 21: (**a**) Input channel impulse response; (**b**) Target channel impulse response; (**c**) Output channel impulse response. Based on the fact that deep learning neural networks are highly nonlinear and have an ability to form arbitrarily nonlinear decision boundaries, a supervised deep learning neural network (DLNN) is employed in MIMO detection [85]. A schematic diagram is shown in Figure 22, where an MDM optical transmission system with \(N\) input modes and \(N\) output modes is adopted. At the receiver side, the received signals are fed to the DLNN to implement MIMO detection. Figure 23 shows the detailed structure of the proposed DLNN. In this scheme, \(N\) complex signals are decomposed into the real parts and imaginary parts. And then, these \(2N\) real signals are fed into a feed-forward neural network with \(M\) layers. For \(N\times N\) MIMO system, \(2N\) input neurons and \(2N\) output neurons are required in the DLNN, which means that the complexity of DLNN increases with the polynomial time complexity of the MIMO dimension. The performance of DLNN, traditional ZF detector and advanced Semi-Definite Relaxation Row-by-Row (SDR-RBR) is compared in a \(2\times 2\) MDM system. Figure 24 shows the BER curves versus channel condition number. It can be seen that DLNN has a better performance than the other two detectors with Q-factor greater than 15 dB. Figure 23: The structure of DLNN. Figure 22: MDM transmission system based on DLNN. It is generally known that the nonlinear Shannon limit has become the basic barrier of a channel maximum capacity in communication.