## 3 The Hybrid HM-QAM Scheme for Multi-User Scenario
\begin{split}&\text{HM\_QAM}:\max_{M_{1},M_{2},P_{1},P_{2}}B\log_{2 }M_{1}+B\log_{2}M_{2}+B\log_{2}M_{3}\\ & s.t.\\ & C1:M_{1}\leq 1+\frac{-15}{\log_{2}5P_{3,1}}\frac{P_{1}|h_{2}|^{2}}{N_{ 0}B}\\ & C2:M_{2}\leq 1+\frac{-15}{\log_{2}5P_{3,2}}\bigg{(}\frac{P_{2}|h_{ 1}|^{2}}{P_{1}|h_{1}|^{2}+N_{0}B}+\frac{P_{1}|h_{1}|^{2}}{N_{0}B}\\ &\qquad\qquad+\frac{-15}{\log_{2}5P_{3,2}}\frac{P_{1}|h_{1}|^{2}}{ N_{0}B}\frac{P_{2}|h_{1}|^{2}}{N_{1}B}\bigg{)}\\ & C3:M_{3}\leq 1+\frac{-15}{\log_{2}5P_{3,3}}\frac{P_{3}|h_{2}|^{2}}{ N_{0}B}\\ & C4:P_{1}+P_{2}+P_{3}\leq P_{total}.\end{split}  \tag{7}\]  
In the above optimization problem, constraints C1, C2, and C3 denote the bit error requirements \(P_{b,1}\), \(P_{b,2}\), and \(P_{b,3}\). According to [51], the BER of \(M_{k}\)-QAM with \(M_{k}>4\) can be obtained as \(P_{b}=\frac{a}{2}\bigg{(}1-\sqrt{\frac{3}{1+3}\frac{2}{M_{k}-1}}\bigg{)}\), where \(a=\frac{4}{\log_{2}M_{k}}\) for \(M_{k}>4\) and \(a=1\) for \(M_{k}=4\). Therefore, the modulation order can be derived as the maximum power of 2 that has the form of C1, C2, and C3 constraints. The parameters \(M_{1}\), \(M_{2}\), and \(M_{3}\) are the modulation order of the farther user, at the overlapped frequencies of the closer user and at the remaining frequencies of the closer user. By analyzing the above optimization problem, we can see that when the inequality constraints C1,C2, and C3 are met with equality, the objective function is maximized, and the maximal data rate is  
\[ \begin{split} R&=B\log_{2}\bigg{(}1+\frac{P_{2}h_{ 2}}{N_{0}B}\frac{-1.5}{\log_{2}5P_{b,2}}\bigg{)}\\ &+B\log_{2}\bigg{(}1+\frac{P_{1}h_{1}}{N_{0}B}\frac{-1.5}{\log_{ 2}5P_{b,1}}\bigg{)}\\ &+B\log_{2}\bigg{(}1+\frac{P_{2}h_{1}}{P_{1}h_{1}+N_{0}B}\frac{- 1.5}{\log_{2}5P_{b,1}}\bigg{)}\\ &+B\log_{2}\bigg{(}1+\frac{P_{3}h_{3}}{N_{0}B}\frac{-1.5}{\log_{ 2}5P_{b,3}}\bigg{)}.\end{split}  \tag{8}\]  
When \(P_{3}=P_{total}-P_{1}-P_{2}\), the above optimization problem is transformed into optimizing \(P_{1}\) and \(P_{2}\) to maximize the data rate. Due to \(\frac{d^{2}R}{dP_{2}^{2}}=0\), by setting \(\frac{dR}{dP_{2}}=0\), we can deduce the optimal \(P_{2}\) to maximize the data rate. Then, by adopting the one-dimension search, we can obtain the optimal \(P_{1}\). The detailed algorithm is shown in Algorithm 1. Before analyzing the computing complexity of Algorithm 1, we discuss the complexity of the proposed hybrid HM-QAM modulation schemes compared to HM and HBM schemes. The proposed hybrid HM-QAM scheme demonstrates better performance than both HM and HBM, as shown in Section 6. However, the computational complexity of hybrid HM-QAM is higher than that of HM and HBM. Specifically, in the hybrid HM-QAM scheme, the base station applies HM to overlapped frequencies and QAM to the remaining frequencies for closer users. In contrast, HM and HBM simply assign a higher modulation order to the closer user than to the farther user. Both hybrid HM-QAM and HBM involve dynamic bandwidth adjustment based on distance, which introduces additional computations compared to HM. Additionally, in HM, both users have the same symbol rate, which does not significantly increase complexity. Conversely, HBM and hybrid HM-QAM adjust the symbol duration, increasing the complexity in symbol rate calculations and synchronization. Finally, implementing the HM scheme is relatively straightforward with existing communication hardware and software. However, implementing HBM and hybrid HM-QAM requires more advanced hardware capable of dynamic symbol rate adjustments and sophisticated software for real-time bandwidth management. Despite the higher complexity, the proposed hybrid HM-QAM offers greater flexibility, potentially better performance, and high adaptability and efficiency. With the rapid advancement of hardware and software technologies, the proposed strategy is achievable.