###### Abstract  
The Gram-Schmidt orthogonalization procedure (GSOP) and Lowdin symmetric orthogonalization procedure (SYOP) are the two mainstream algorithms for the compensation of phase mismatch in an imperfect optical 90\({}^{\circ}\) hybrid. In this paper, we put forward an algorithm switching orthogonalization procedure (ASOP) according to the quality of in-phase and quadrature signals based on the Q value of the eye diagram with less computation. If the quality of the in-phase and quadrature signals has a significant difference, we use the GSOP and select the signal branch with better quality as the initial reference vector for orthogonalization. If they are of about the same quality, then we use the SYOP. We present computer simulations for a coherent free-space optical (FSO) quadrature phase-shift keying (QPSK) communication system and demonstrate the system improvement that can be achieved using the ASOP. Finally, we also show that the proposed ASOP scheme can contribute to the frequency offset and phase estimation of the FSO system in the environment of atmospheric turbulence.  
coherent communications; free-space optical; IQ imbalance; orthogonalization procedure +
Footnote †: journal: Applied Sciences  
**C.** **[**2021**]** **[  
The IQ imbalance degrades the system performance severely if left uncompensated in the DSP unit of the receiver. An imperfection in any of the 90\({}^{\circ}\) optical hybrids, balanced photodiodes, and transimpedance amplifiers (TIAs) in the front end may introduce an IQ imbalance stemming from the gain and/or the phase mismatch between the IQ branches [8]. Several methods of IQ imbalance compensation in the digital domain have been reported for the QPSK signal [9; 10; 11; 12; 13; 14; 15]. The well-known techniques for the orthogonalization of two nonorthogonal vectors are the Gram-Schmidt orthogonalization procedure (GSOP) [10] and Lowdin symmetric orthogonalization procedure (SYOP) methods [11; 12]. In [13], compensation is done by the ellipse-correction (EC) method, which is neither applicable to higher-order QAM signals nor effective when the optical signal-to-noise ratio (OSNR) is low. We will explain the above three methods in Section 3. It should be noted that the GSOP is known to depend on the ordering of the initial vector that is meant to be fixed as the reference vector, which is not necessarily a shortcoming for mathematical analysis. It is not an advantage for IQ signal orthogonalization, because if the initially selected vector is of poor quality, then the new vector orthogonal to it is of low quality too. Similarly, the SYOP and EC methods exhibit symmetry and resemblance properties, although the original reference signal is very good. As a result, the quality of the new orthogonal signal may become worse due to the symmetry property.  
In this paper, we propose a novel phase mismatch compensation scheme, where the conventional GSOP and the SYOP are modified and combined so as to achieve an optimal orthogonal state between the IQ branches. We mainly study the effects of phase mismatch, which means that the phase difference of IQ signals may be deviated from 90\({}^{\circ}\) because of the imperfect optical hybrid. This paper describes the coherent optical communication system under the condition of atmospheric turbulence with a QPSK modulation regime at the rate of 20 Gbps.