## 2 Principles of the Proposed Method  
### OSNR Monitoring Based on the Conventional DNN Trained with AHs  
Consider a coherent optical transmission system, its coherent receiver and digital signal processing (DSP) architecture, including the proposed OSNR monitoring stage, as shown in Figure 1. The received signal in the digital coherent receiver adopts the standard signal processing algorithms in the DSP unit, including normalization, resampling and phase-orthogonal (IQ) imbalance compensation, and then followed by chromatic dispersion (CD) compensation and timing phase recovery algorithm independent of the standard modulation format. Next, we use equalization based on CMA to solve polarization demultiplexing and compensate almost all linear transmission impairments [41]. As can be seen from Figure 1, the data for OSNR monitoring is the output signal after CMA equalization. After this stage, the signal is mainly affected by the amplified spontaneous emission (ASE) noise.  
Figures 2 and 3 show the constellation diagram and the corresponding amplitude histograms of PM-RZ-QPSK and PM-NRZ-16 QAM signals after CMA equalization, as well as the constellation diagram after carrier phase estimation. AHs are generated by data samples obtained after CMA equalization module with 100 bins. We choose OSNR monitoring after CMA equalization instead of carrier phase estimation because the former only needs a few DSP units, which reduces the complexity of OPM devices, while the latter needs to process additional AHs, which increases the computational complexity and time. It can be clearly seen from the two figures that their corresponding AHs show unique and distinct signatures for different OSNRs and modulation formats. Therefore, the sensitivity of AHs to OSNR can be used as an estimated feature of OSNR monitoring. Then, the pattern recognition technology based on conventional DNN is used to estimate OSNR by using the relevant features of AHs.  
After obtaining the characteristic data for estimating OSNR, it was input into DNN structure as input data for training and test. The schematic diagram of the fully connected conventional DNN (no optimization algorithm) is shown in Figure 4. It includes an input layer, multiple hidden layers, and an output layer. The mean square error (MSE) formula of the output end, the output of the hidden layer, and the output layer neuron are shown in Figure 4. Where, \(x\) is the input vector, \(\mathrm{d}\) is the predicted value of output, \(w\) and \(\mathrm{b}\) are the weights and biases in the DNN structure, and \(G(.)\) and \(F(.)\) are the activation functions of the hidden layer and the output layer including Tanh, Sigmoid, ReLU and Linear, Sigmoid, and Softmax, respectively. The weights and biases of DNN are initialized randomly, and the whole network is trained by an iterative algorithm. The training can be stopped by adjusting the parameters according to the MSE between the current output and the label, until the error converges or reaches a certain iteration set value. In the conventional OSNR estimation algorithm, the input vector is composed of \(100\times 1\) bin counts in the AHs dataset, which is input into different DNN structures of two signals. Considering the estimated value of OSNR, the output layer neuron is 1 and the output value is scalar.  
Figure 1: Schematic of the coherent optical receiver and digital signal processing (DSP) for optical signal-to-noise ratio (OSNR) monitoring (LO: local oscillator, PBS: polarization beam splitter, and ADC: analog-to-digital converter).