## 1 Introduction  
In the past several decades, data traffic has expanded with explosive growth due to the development of big data, cloud computing, Internet of Things, and other services. Inspired by the ever-increasing demands for transmission bandwidth and capacity, high-order modulation formats have been extensively applied in optical fiber communication systems [1]. However, high-order modulation formats are extremely susceptible to nonlinear impairments caused by devices and fiber transmission links in optical fiber communication systems. Furthermore, a higher optical signal-to-noise ratio (OSNR) is required to achieve the desired bit error ratio (BER). As an attractive solution to approach the Shannon limit in optical fiber communication systems, PS has recently become a research hot spot. PS can effectively reduce the average power of the constellation by increasing the transmission probabilities of inner constellation points while reducing the transmission probabilities of outer constellation points. Aided with PS technology, optical fiber communication systems will have a better ability to restrain nonlinearity effects and get better BER performance [2; 3; 4; 5]. In the Ref. [6], based on probabilistically shaped PDM 4096-QAM, a transmission of a 10-subcarrier superchannel with a record-high intra-channel spectral efficiency of 17.3 bit/s/Hz over 50 km fiber is demonstrated. Joint channel coding and constellation shaping based on polar-coded modulations is also proposed, which offers more than 0.6 dB gain without the need for an external distribution matcher and an increase of decoding complexity [7]. A feedback-based PS-16 QAM modulation method, which is used to adaptively modify the distribution of transmitted symbols based on errors at the receiver, is proposed in the Ref. [8]. The error feedback enables solving an optimization problem to find the distribution that maximizes the mutual information between the input and output of the channel without knowledge of the channel itself. It shows superior performance in a variety of application scenarios, and realizes ultra-high performance improvement of reducing the system bit error rate by 50%. In addition, a probabilistic shaping distribution matching algorithm employing uneven segmentation is proposed, which achieves a 0.4 dB performance improvement compared with the single forward error correction (FEC) system under the condition of maintaining low complexity of encoding and decoding [9]. Besides, with the development of coherent detection, many digital signal processing (DSP) algorithms have been proposed to compensate the nonlinear effects. Digital back-propagation (DBP), which is a traditional DSP algorithm, is effective at mitigating nonlinearity but with large complexity in real implementations [10; 11]. In recent years, machine learning (ML) techniques have attracted more and more attention to deal with nonlinear effects in optical communication systems. For example, some ML clustering algorithms are used to cluster constellation points such as k-means [12] and Gaussian mixture models (GMM) [13]. In addition, some ML classifier algorithms such as SVM [14] and K-nearest neighbor (KNN) [15] have been used for classifying constellation points. Among various ML algorithms, SVM has shown prominent proficiency for mitigating fiber nonlinearity. The M-ary SVM is introduced as a nonparameter nonlinear phase noise (NLPN) mitigation approach for the coherent optical systems [16]. A fast Newton-based SVM nonlinear equalizer (NLE) is experimentally demonstrated in 40 Gb/s 16-quadrature amplitude modulated coherent optical orthogonal frequency division multiplexing at 2000 km of transmission whose computational load is \(\sim\)6 times lower than traditional SVM [17]. In addition, a SVM-based machine learning method to mitigate modulation nonlinearity distortion for PAM-4 and PAM-8 vertical cavity surface emitter laser multi-mode fiber (VCSEL-MMF) optical link has been demonstrated in the Ref. [18].