## 2 System Structure  
The FSO communication system based on a rectangular QAM with probabilistic shaping is adopted in this paper, as shown in Figure 1. In the transmitter section, the uniform bitstream is reshaped into the desired distribution by the PS encoder, which will be illustrated in detail in Section 5. After that, the shaped bits encoded by the LDPC codes are mapped to QAM symbols. A Mach-Zehnder modulator (MZM) is used to modulate the laser. The in-phase and quadrature branches of the MZM are driven by the QAM electric signals. The modulated optical signal is amplified by an erbium-doped fiber amplifier (EDFA) and transmitted in the turbulence channel.  
To simulate the turbulence channel, the Gamma-Gamma turbulence model is adopted in our work for emulating the turbulence conditions from weak to strong. Its probability distribution function (PDF) is given by [23]:  
\[f(g)=\frac{2(\alpha\beta)^{(\alpha+\beta)/2}}{\Gamma(\alpha)\Gamma(\beta)}g^{( \alpha+\beta)/2-1}K_{\alpha-\beta}(2\sqrt{\alpha\beta g}), \tag{1}\]  
where \(\Gamma(\bullet)\) denotes the Gamma function and \(K_{n}(\bullet)\) is the modified Bessel function of the second kind of order \(n\). The parameters \(\alpha\) and \(\beta\) represent the effective numbers of large scale and small scale turbulence eddies, respectively, which can be expressed as a function of Rytov variance \(\sigma_{R}^{2}\). Considering plane waves, we have [23]:  
\[\alpha=\left[\exp(\frac{0.49\sigma_{R}^{2}}{\left(1+1.11\epsilon_{R}^{12/5} \right)^{7/6}})-1\right]^{-1}, \tag{2}\]  
\[\beta=\left[\exp(\frac{0.51\sigma_{R}^{2}}{\left(1+0.69\sigma_{R}^{12/5} \right)^{5/6}})-1\right]^{-1}, \tag{3}\]  
In the coherent receiver, the received optical signals are mixed with a local oscillator. The output signals are detected by two balanced photodetectors (BPD). The corresponding electrical in-phase and quadrature signals are further processed by high-speed digital signal processing (DSP). Eventually, the bit streams are obtained by an LPDC decoder and a PS decoder. The coherent detection currents can be expressed as [2]:  
\[I[t] =2\eta\sqrt{P_{L}P_{S}}\cos\{\theta_{sig}+\theta_{IF}+\theta_{h}\}+n_{I}, \tag{4}\] \[Q[t] =2\eta\sqrt{P_{L}P_{S}}\sin\{\theta_{sig}+\theta_{IF}+\theta_{h} \}+n_{Q},\]  
Figure 1: The system structure of the QAM-based PS scheme for the FSO fading channel.  
where \(\eta\) represents the responsivity of the photodiode and \(P_{L}\) is the optical power of the local oscillator beams. \(P_{S}\) stands for the received optical power in the present turbulence. \(\theta_{sig}\) denotes the modulation phase of the received signal. \(\theta_{IF}\) is the phase difference between the local oscillator and transmitter laser. \(\theta_{h}\) represents the phase fluctuation caused by atmospheric turbulence. \(n_{I}\) and \(n_{Q}\) are the compound additive white Gaussian noise.  
Perfect channel state information (CSI) is assumed at the receiver and the system is not affected by intersymbol interference. Considering the equivalent baseband model, we define the input \(X\) to be complex QAM symbols after a bit mapper block. The value of the non-equiprobable QAM symbols are taken from \(\chi=[x_{1},\ldots,x_{i},\ldots,x_{M}]\), \(i=1,\ldots,M\) according to PMF \(P_{X}=[P_{X}(x_{1}),\ldots,P_{X}(x_{i}),\ldots,P_{X}(x_{M})]\), where \(P_{X}(x_{i})\) stands for the probability when the signal \(x_{i}\) is transmitted. The corresponding electrical signal \(Y\) before demapper block is given by:  
\[Y=GX+N, \tag{5}\]  
where \(N\) is the complex zero mean Gaussian noise with variance \(\sigma^{2}\) and \(G\) is intensity attenuation related to atmospheric turbulence.