# Error Probability of a Coherent \(M\)-ary PSK FSO System Influenced by Phase Noise
## 1 Introduction
### Motivation  
Results from [23] proved that coherent FSO systems have improvements of 24-30 dB compared with SIM-based FSO systems over different turbulence conditions. Such high-performance FSO systems can be used in backhaul as an integral part of 5G beyond networks to provide a seamless connection with fiber optic counterparts [24; 25]. Recently, with the advent in coherent detection technology that allows for background noise rejection, high sensitivity, and improved spectral efficiency, there has been a resurgence of interest in coherent FSO communications with coherent homodyne/heterodyne receivers [26], and references therein. Some deep tech companies have been developing 100 Gbs coherent FSO systems capable of transmitting data over distances up to 10 km [27]. It is obvious that coherent FSO systems, regardless of the fact that they are more complex to implement compared with IM/DD PSO systems, will play a significant role in development of modern mobile networks in the near future. This research is exactly along this line and aims to provide a theoretical basis not only for evaluating the performance of these systems, but also for their design.  
At the transmitter of a coherent FSO system, an information bearing signal modulates a lightwave signal by applying amplitude, frequency or phase modulation. At the receiver, a received optical signal is combined with a signal of local oscillator (LO), and the resulting signal is converted by a photodetector to electrical signal. The difference between a received optical signal phase and an LO output signal phase is known as a phase error. This phase error is compensated by PLL in an electrical part of the receiver. In the case when the frequency of LO differs for several gigahertzs from the frequency of the received optical signal, heterodyne detection is performed [28; 29; 30]. The imperfect phase error compensation in the electrical part of the coherent receiver is a significant deleterious effect in coherent FSO systems. This imperfect phase error compensation is also known as phase noise. The error probability degradation caused by the phase noise in coherent FSO systems was examined in [28; 29]. The analysis related with the effect of phase noise in both works was carried out for binary phase-shift keying (BPSK) modulation format, and the results showed that the standard deviation of the phase noise larger than 20 degrees causes undesirable error probability degradations. The error rate analysis of FSO communication system with coherent detection in the presence of phase noise was further analyzed in [30] where pre-detection and post-detection equal-gain receivers of BPSK signals were considered.  
Higher order modulation formats are implemented in order to increase the spectral efficiency. In that case, the effect of phase noise on the error probability is expected to be stronger because the decision areas become narrower as the modulation order increases. There is a practical need to develop an efficient analytical method for estimating the effect of phase noise on the coherent FSO system performance when higher order modulated signals are transmitted over a turbulence channel.