## 1 Introduction  
The application of machine learning technique in optical communication systems has been studied in many fields in recent years [1, 2]. In the field of optical communication systems, many parts of the system, such as performance monitoring, fiber nonlinearity mitigation, carrier recovery, and equalization, have been optimized by machine learning and a neural network [3, 4, 5, 6]. In particular, as we all know, chromatic dispersion (CD) and nonlinear Kerr effects in the fiber are the main constraint in the improvement of the signal rate in the optical communication system today [7]. In order to deal with the influence of a nonlinear effect and dispersion on the signal, the traditional method is to add a backward equalizer to reduce the influence of pulse broadening on the bit error rate of the signal. In recent years, as a new method, the artificial neural network (ANN) has been of great interest on channel equalization in the field of wireless communication [8, 9, 10, 11], which shows its advantage on the better bit error rate (BER). Since Deep learning relies on features of the data and situation, it cannot be efficiently trained under the changeable situation of long-distance communication. An end-to-end communication system, which uses two neural networks (NNs) as transmitter and receiver to be trained for a specific-designed channel model, was presented in Reference [10]. The communication system is regarded as an autoencoder (AE) [12], which is composed of symmetrical NNs modeled as the transmitter and receiver, and intermediate channel modeled as a fixed layer. AE is generally optimized by gradient-based algorithms (stochastic gradient descent [12], Adam algorithm [13], etc.) in the condition of the differentiable loss function (originated from the difference between actual value and label). The AE was firstly applied in the wireless communication [11] to serve as the transmitter and receiver, and it was experimentally demonstrated that it has better performance. The concept of AE has been recently employed to other refined fields, such as orthogonal frequency-division multiplexing (OFDM) [5] and multiple-input multiple-output (MIMO) [14]. In optical fiber communication, the AE has been implemented in the dispersive linear fiber channel [15; 16], or in the nonlinear dispersive fiber channel, with nonlinear frequency division multiplexing (NFDM) [17; 18] for transmission. In other fields, the AE is popular in geometric constellation shaping (GCS) in order to obtain the theoretically achievable channel capacity, which appears in a uniformly distributed signal (such as conventional quadrature amplitude modulation (QAM)). Geometric constellation shaping is the process of redesigning the distribution of the constellation points on the I/Q plane. The main purpose of GCS is to achieve balance between Euclidean distance among different constellation points and the energy distribution of the constellation point on the given channel. In References [7; 19; 20; 21], it was demonstrated that the AE can be applied on various optical fiber channel models (such as nonlinear fiber channel [19; 20], non-dispersive channel [7], and linear link by QAM [21]) to learn and achieve better geometric constellation. In the field of AE, generative adversarial networks (GANs) are a new trend to model the non-differentiable or relatively complex channel as a simple differentiable channel on the optical fiber communication [22]. The GANs are simply used to be trained the AE for non-coherent optical fiber communication. Due to additional experiment steps caused by GANs, new data needs to be obtained from the non-differentiable channel model and used to train the AE for better discrimination between synthetic data and original data. Obviously, for ensuring the performance of the GAN, the scale of all training samples (both from non-differentiable channel model and differentiable channel) must be large enough to train GANs to reach the level of relatively accurate channel approximation.