## 1 Introduction
A 2 \(\times\) 2 aperture architecture, where each transmitter aperture transmits two multiplexed OAM beams, has been proposed to design an 80 Gbps FSO system. The result obtained shows the power penalty to be less than 3.6 dB at the BER limit of \(3.8\times 10^{-3}\). Incorporating multiple aperture sizes in MIMO based multiple OAM using spatial mode multiplexing (SMM) has been widely investigated by different researchers to enhance the channel capacity and mitigate the turbulence-induced power fluctuations. However, in a practical FSO link, the design complexity, cost, and real-time processing capability of such an adaptive optical receiver remain critical issues [12; 13]. The spiral wave-front of the OAM beams is susceptible to free space turbulence, resulting in inter-channel crosstalk (ICC) issues and the spreading of the power of the transmitted OAM states into different adjacent OAM states [14]. To maximize the capacity of the FSO link, such crosstalk has been thoroughly studied for Laguerre-Gaussian beams under various atmospheric turbulence scenarios for adaptive optic systems based on adaptive MIMO techniques [15]. In this research paper, an OAM-OFDM based spatial mode multiplexed (SMM) MIMO-FSO system design is presented, and the system performance is evaluated under different weak to strong atmospheric channel conditions. The spatial mode diversity (SMD) based MIMO signal processing scheme is proposed to improve the system's robustness against channel path loss attenuation and also to enhance the transmission capacity. By combining the benefits of OFDM and OAM multiplexing, the proposed design is not only made spectrally efficient and robust against the impact of multipath fading, but it also increases the proposed system's transmission capacity in the FSO link. This work presents 20 symmetrical hybrid spatially multiplexed (OAM-OFDM) orthogonal channels, each carrying 120 Gbps of 16-QAM modulated data at a wavelength of 1550 nm over a link distance of 2 km. When this hybrid spatially multiplexed signal is transmitted through the turbulence channel, the OAM mode state intensity profile and phase structure are distorted due to mode energy spreading to adjacent OAM states. The proposed system design BER performance is investigated under turbulent channel metrics such as the refractive index structure parameter, Rytov variance constant, and link distance. The BER performance is also compared for different transmitting OAM mode states in different turbulence conditions. The BER and the channel capacity performances of the proposed OAM-OFDM-SMM MIMO FSO system is investigated and compared by taking into consideration the MIMO and OAM-MIMO FSO systems as a function of OSNR in turbulent atmospheric conditions. Moreover, the analytical results illustrate that the BER of the OAM-OFDM-SMM MIMO FSO system decreases exponentially with an increase in OSNR values and significant improvement is observed at 20 dB OSNR. In addition, the channel capacity of the proposed system increases exponentially with an increase in OSNR and at OSNR of 18 dB, the capacity of the proposed system is improved by 3.3 bits/s/Hz compared to the other two FSO systems discussed above. The subsequent sections of this paper are organized as discussed below. In Section 2, we have described the schematic design model of the proposed OAM-OFDM-SMM multiplexed free space communication system with detailed mathematical analysis, whereas the principle of the spatial mode diversity used in the proposed system is explained in Section 3. The mathematical concepts of the generation of the 16-QAM modulated SMM multiplexed OAM beam using a spatial phase mask (SPM) technique and its electric field distribution is discussed in this section. Moreover, the channel capacity model of the proposed system under the impact of the GG atmospheric turbulence condition is reported in Section 4.