## 3 The Generalized AP-HSIC Algorithm for QAM Modulations
However, a more important and clearer conclusion is the fact that the value of \(SNR\approx 0\) dB, the number of iterations is only five iterations, and, in \(SNR\approx 3\) dB and higher, the number of iterations drops to only two iterations and stabilizes permanently. There are two significant explanations for this insight. The first explanation is that, even if the theoretical design of

\[=\left\|Y\big{(}I-S^{+}S\big{)}\right\|_{F}^{2}+\left\|\big{(}YS^ {+}-(H+\Delta H)\big{)}S\right\|_{F}^{2},\]2 BER performance vs. the average SNR of three iterations of the MGSTC algorithm and systematic iterations of the AP-HSIC algorithm. (**a**) Plot graphs of BER performance vs. the average SNR that compares three iterations of the MGSTC algorithm and systematic iterations of the AP-HSIC algorithm under 32-QAM modulation in the presence of Rayleigh deep-fading and high-level AWGN. (**b**) Plot graphs of BER performance vs. the average SNR that compares three iterations of the MGSTC algorithm and systematic iterations of the AP-HSIC algorithm under 64-QAM modulation in the presence of Rayleigh deep-fading and high-level AWGN. \[=\left\|Y\big{(}I-S^{+}S\big{)}\right\|_{F}^{2}+\left\|\big{(}YS^ {+}-(H+\Delta H)\big{)}S\right\|_{F}^{2},\]3 Plot graphs of BER performance vs. the average SNR that compares three iterations of the MGSTC algorithm and systematic iteration of the AP-HSIC algorithm under 16-QAM modulation in the presence of Rayleigh deep-fading and high-level AWGN. the AP-HSIC algorithm error, \(\epsilon\), in \(SNR\approx 3\) dB, is \(\epsilon\approx 0.0775\) (e.g, \(\epsilon\geq\frac{n}{\sqrt{mSNR}}\)), practically, the actual algorithm error value drops drastically to a value of \(\epsilon\approx 10^{-9}\). This is because activating a full MIMO array causes a high diversity gain. The second explanation is that, since the AP-HSIC samples several samples before the appearance of the disturber and keeps these samples in memory, a good starting point is created, which is expressed as \(\Delta H_{0}=0\) and \(S_{0}=H^{+}Y\). This starting point makes the algorithm converge very quickly to an optimal point (see also Remark 2). In summary, Rayleigh deep-fading scenarios are extreme interference situations that cause the breakdown of the spatial SIC decoding algorithm, such as MGSTC. Increasing the transmission power does not seem to be effective in these interference scenarios; it actually amplifies the interference factors and significantly hampers the decoding processes, as indicated in (10). Our results demonstrate a substantial improvement in the performance of the AP-HSIC algorithm and its effective handling of interference scenarios such as Rayleigh deep-fading. ### LM-HSIC and AP-HSIC Simulations under Rayleigh Deep-Fading and High-Leveled AWGN

In this section, we present simulation results for the algorithms LM-HSIC and AP-HSIC in the case of Rayleigh deep-fading and high-leveled AWGN based on 16-QAM, 32-QAM, and 64-QAM modulations. Figure 7 illustrates the detailed comparison of the BER performance of the algorithms AP-HSIC and LM-HSIC in the case of Rayleigh deep-fading and high-leveled AWGN scenarios based on 16-QAM, 32-QAM, and 64-QAM modulations. In this Figure, the continuous purple, black, and green graphs represent the BER performance of the AP-HSIC algorithm in 16-, 32-, and 64-QAM constellation order, respectively, while the dotted purple, black, and green graphs represent the BER

\[=\left\|Y\big{(}I-S^{+}S\big{)}\right\|_{F}^{2}+\left\|\big{(}YS^ {+}-(H+\Delta H)\big{)}S\right\|_{F}^{2},\]4 Plot graphs of BER performance vs. the average SNR for three systematic iterations using the MGSTC algorithm and AP-HSIC algorithm. Each iteration is represented by 16-QAM, 32-QAM, and 64-QAM modulations, respectively. The performance was compared under the presence of Rayleigh deep-fading and high-level AWGN.