## 1 Introduction  
As a line-of-sight transmission technology, free space optical (FSO) communication technology has the advantages of no spectrum authorization, good confidentiality, and flexible networking. This technology uses a laser as a transmission carrier, and the optical signal is susceptible to atmospheric turbulence when transmitted in free space [1, 2, 3]. The irregular random movement of atmospheric turbulence will lead to random fluctuations in the optical signal, which will cause problems such as changes in the arrival angle of the beam and fluctuations in the light intensity at the receiving end, thus forming an atmospheric turbulence effect. The atmospheric turbulence effect will give rise to the bit error rate of the entire optical communication system, affecting the quality of communication [4, 5]. To suppress the effect of atmospheric turbulence on the FSO communication system, many researchers have applied the collaborative communication method to the FSO communication system. In a collaborative FSO communication system, the relay node tends to be closer to the source node than the destination node. Due to the change in communication distance, the atmospheric turbulence affected by the laser transmission in the atmosphere is also different. The longer the communication distance, the stronger the atmospheric turbulence effect. Therefore, in this case, the channel loss of the direct-pass link and the relay transmission link will be different, which will lead to asymmetric reception problems at the destination node. To solve the problem of communication system performance degradation caused by asymmetry of transmission channels, hierarchical modulation technology can be applied to collaborative communication systems. In reference [6], [PERSON] and [PERSON] proposed a collaborative communication system based on single-relay auxiliary hierarchical modulation. By using turbo code and mesh code modulation (TTCM) schemes at the source nodes, using the decode-and-forward collaboration protocol in the relay node, the power consumption of the entire system is reduced by 3.62 dB per time slot. In reference [7], [PERSON] and [PERSON] proposed a three-layer collaborative communication system assisted by dual relays; the system integrates the three schemes of TTCM three-layer HM-64 QAM and two-layer SPM-16 QAM, with a rate of 0.5 under the collaborative communication mode. Finally, the system is optimized to reliably transmit layer three HM-64 QAM signals using two-time slots, with an average signal-to-noise ratio of 6.94 dB per slot. In reference [8], [PERSON] and [PERSON] proposed a scheme that uses adaptive hierarchical modulation to simultaneously transmit voice and multiple types of data on fading channels by changing the constellation size and the priority parameters of the hierarchical signal constellation; the scheme gives the interrupt probability of speech and data transmission on the Nakagami-m fading channel, the achievable spectral efficiency, and the average bit error rate (BER) closed expression and numerical results. In reference [9], [PERSON] and [PERSON] investigated the diversity gain maximization problem in wireless relay systems and used a scheme of setting thresholds to mitigate error propagation in hierarchical modulation, and its analysis and simulation results show that the threshold must depend on the hierarchy constant. In the actual communication system, the asymmetry of the channel quality will lead to the deterioration of the performance of the communication system. Although the use of hierarchical modulation technology can effectively suppress the deterioration of asymmetric channel performance, it does not solve the problem of system throughput deterioration. Based on this, many researchers have applied network coding technology combined with hierarchical modulation technology to collaborative communication networks.