# Advanced DSP for Coherent Optical Fiber Communication
## 5 AI for Optical Fiber Communication  
Optical communication networks and systems have started to use artificial intelligence (AI) technologies to improve the performance, from equipment to control and management. Machine learning (ML), as a main branch of AI, provides many powerful models for optical communication. One of the main applications of ML technology is to perform optimal classification [61; 62; 63]. Demodulation of signals in point-to-point optical communication systems is crucial for achieving high-capacity transmission performance and is normally affected by nonlinear impairments and their interactions along the propagation. Classic demodulation techniques (e.g., maximum likelihood) cannot cope with high dimensional nonlinearities and channel dynamicity. Therefore, recent advances in ML and deep learning (ANNs, DNNs and CNNs) gives the potential to solve these challenges. The optimal classification can thus be directly applied for an optimal symbol detection in optical communication systems. For a memoryless nonlinearity, such as nonlinear phase noise, I/Q modulator and drive electronics nonlinearity, Euclidean distance measurement results in that the linear decision boundary is no longer optimal. For these special cases, ML techniques can be used to obtain the optimal symbol detection and decision boundary. Then DSP algorithm is used to detect the optimal symbols. ML has also been applied in many other applications such as optical performance monitoring, parameters estimation and nonlinearity mitigation [64; 65; 66; 67; 68]. A typical block diagram of ML for symbol classification in optical communication systems is illustrated in Figure 17.  
Figure 16: Comparison between conventional least mean square (LMS) and VFF-RLS/RLS: Q\({}^{2}\) vs. Distance. Figure 17: Block diagram of machine learning for symbol classification. An example is implemented here to show the performance of DNN on mitigating the inter-carrier interference (ICI) in the non-orthogonal multicarrier optical fiber transmission systems with sub-carrier spacing below the symbol rate. The spectrally efficient frequency division multiplexing (SEFDM) technique is employed in such non-orthogonal multicarrier transmission systems [69; 70]. The SEFDM transmission occupies a sub-carrier spacing below the symbol rate at the expense of a loss of orthogonality to enhance the achievable spectral efficiency. The challenge in the SEFDM system is the inter-carrier interference (ICI) during the bandwidth compression [71]. As illustrated in Figure 18, five types of neural networks are designed for suppressing the ICI of non-orthogonal signals [72], including no connection-neural network (N-NN), partial connection-neural network (P-NN), hybrid1-neural network (H1-NN), hybrid2-neural network (H2-NN), full connection-neural network (F-NN). These NN architectures represent different connectivity between the input, output and hidden cells in the neural networks. The complexity also depends on the NN architecture. The N-NN has the lowest computational complexity since all the input cells are independent and can be operated in parallel. The F-NN has the highest computational complexity since all neurons are connected and each connection link requires arithmetical computation. The employed deep neural networks follow the architectures in Figure 18, and the Sigmoid function is applied as the activation function at each layer. The neural network is trained off-line using a large amount of simulated random symbols (~40,000 QPSK) in the optical SEFDM system. The number of sub-carriers in this case is set to 4 and the bandwidth compression factor is set to \(\alpha=0.8\). Please see Ref. [72] for more detailed information regarding the transmission system. Simulation results are shown in Figure 19 to evaluate the effectiveness of NNs on SEFDM signal recovery. Their performance is compared with traditional SEFDM signal detection (hard decision). Figure 19a indicates the B2B scenario and Figure 19b indicates the scheme of 80 km transmission.