## 1 Introduction  
Free-space optical (FSO) communication has been widely researched in recent decades due to its characteristics of a high data rate, wide bandwidth, low power consumption, less mass, an unlicensed spectrum, and high security [1]. However, the system performance is impaired by the severe fluctuations in the received signal intensity caused by the atmospheric turbulence-induced beam scintillation effect, which is the major issue in FSO communication [2]. The state of polarization (SOP) maintains a stable state in the atmospheric turbulence channel using a spatially coherent plane wave [3]. Therefore, the application of polarization has been extensively studied to improve the performance of FSO communication systems.  
The polarization division multiplexing (PDM) technique has been studied to improve the capacity of FSO communication systems via the application of multiple polarizations [4, 5]. However, the performance is distorted by the scintillation-effect-induced intensity fluctuations. The polarization shift keying (PolSK) method has been researched to mitigate the scintillation effect via the estimation of SOPs using knowledge of the Stokes parameters [6, 7]. However, the performance is significantly degraded in the peak and deep-fading parts of the scintillation effect. Furthermore, PolSK is inefficient in the transmitted optical power compared to on-off keying (OOK). FSO communication is an average-power-limited system due to the average power limitation characteristics of the optical amplifiers [8]. Thus, polarization shift OOK (PS-OOK) was studied in our previous work to effectively mitigate the scintillation effect with the transmitted power efficiency [9]. However, the spectral efficiency (SE) of PS-OOK is the same as the conventional OOK due to the application of two SOPs. Higher-order modulation techniques, such as pulse-amplitude modulation 4 (PAM-4), multilevel polarization shift keying (MPoISK), and multilevel phase-shift keying (MPSK), have been researched to improve the capacity of FSO communication systems [5,7,10]. Therefore, it is preferable to improve the SE of PS-OOK via the transmission of multilevel SOPs.  
In this study, we proposed a semiconductor optical amplifier (SOA)-based multilevel polarization shift on-off keying (MPS-OOK) transmission for FSO communication. At the transmitter end, the MPS-OOK signal was modulated with a constant SOP at the high-intensity level and various SOPs at the low-intensity level in order to enhance the SE with the transmitted power efficiency. At the receiver end, a polarization-independent SOA and a linear polarizer (LP) were applied to detect the MPS-OOK signal. First, the scintillation effect was optically mitigated using the polarization-independent SOA in the gain saturation state based on nonlinear gain characteristics. Then, the intensities of the various SOPs were equalized in a much deeper gain saturation state without polarization distortion. Then, an LP with a high extinction ratio (ER) was deployed after the SOA to transform the equalized SOPs into the various intensity levels. Finally, the transformed MPS-OOK signal was detected using a single photodiode (PD) and distinguished using a multilevel fixed-threshold decision (M-FTD). The proposed technique was evaluated in experiments under the Mach-Zehnder modulator (MZM)-based fading-simulator-accommodated scintillation effect. The experimental results illustrated that the SE was enhanced by up to 2 bit/s/Hz with effective scintillation mitigation using the proposed four-level polarization shift on-off keying (4 PS-OOK) transmission.