## 3 The Generalized AP-HSIC Algorithm for QAM Modulations
Then, the directional derivative at \((\Delta H,S,\lambda)\) in the direction \(U=(U_{1},0,0)\) with \(\left\|U\right\|_{F}=\left\|U_{1}\right\|_{F}=1\) is defined by



Finally, we calculate the last partial derivative for \(U=(0,0,U_{3})\) with \(\left\|U\right\|_{F}=\left|U_{3}\right|=1\):

\[\
abla_{U}\mathcal{L}(\Delta H,S,\lambda)=U_{3}\Big{(}\|S\|_{F}^{2}-r_{aw}^{2} \kappa k\Big{)}=0\ \ \Rightarrow\left\|S\right\|_{F}=r_{aw}\sqrt{nk} \tag{29}\]

From (24) and (28), we conclude

\[\lambda SS^{*}=0\Rightarrow|\lambda|\|S\|_{F}^{2}=0\Rightarrow\lambda=0, \tag{30}\]

since \(\left\|S\right\|_{F}^{2}=0\) would imply that \(S=0_{n,k}\), which is impossible since \(0\
ot\in QAM\) and since we assumed that \(\left\|S\right\|_{F}=r_{aw}\sqrt{nk}\). Setting \(\lambda=0\) into (28), we obtain

\[(H+\Delta H)^{*}(Y-(H+\Delta H)S)=0. \tag{31}\]

Now, from (24), we obtain

\[YS^{*}=(H+\Delta H)SS^{*}, \tag{32}\]

where, under the assumption that \(SS^{*}\) is invertible, we obtain

\[YS^{*}(SS^{*})^{-1}=H+\Delta H. \tag{33}\]

From the properties of the Moore-Penrose pseudo-inverse, if \(S\) has a full row rank, then \(S^{+}=S^{*}(SS^{*})^{-1}\) and \(SS^{+}=I_{n}\). However, even if \(SS^{*}\) is not invertible, \(YS^{+}-H\) yields the best approximation for \(\Delta H\) in terms of the Frobenius-norm. This means that \(\Delta H=YS^{+}-H\) has the minimal energy (with value \(\left\|\Delta H\right\|_{F}^{2}=\left\|YS^{+}-H\right\|_{F}^{2}\)) that explains the current corrupted signal \(Y\). There are two explanations for the assumption that \(\Delta H\) has minimal energy. The first one is related to the fact that the noise intensity depends mainly on the adequate bandwidth of the communication system and the noise density, which depends on the ambient temperature and the Boltzmann constant. Many modern wireless communication systems with wide bandwidth are susceptible to this noise, which causes a significant decrease in the energy of the transmitted symbol to a minimum. In addition, it can be assumed that \(\Delta H\) is part of the noise expression; therefore, taking the minimum energy of \(\Delta H\) is a reasonable assumption [57]. The second explanation is related to the type of interrupter. Different kinds of jammers or interference phenomena can be destructive even with minimal energy; for example, an intelligent jammer or a deep-fade. The presence of these phenomena causes a change and rotation in the entire constellation received in the receiver and results in the process of accumulating destructive errors [11]. These reasons formed the guideline for choosing \(\Delta H\) with minimal norm. We therefore set

\[\epsilon_{0}:=\sqrt{E\left[\left\|S_{0}-S_{*}\right\|_{F}^{2}+\left\|\Delta H _{0}-\Delta H_{*}\right\|_{F}^{2}\right]}\]0

as the preferred solution for \(\Delta H\) for known \(S\). From (28), we obtain

\[\epsilon_{0}:=\sqrt{E\left[\left\|S_{0}-S_{*}\right\|_{F}^{2}+\left\|\Delta H _{0}-\Delta H_{*}\right\|_{F}^{2}\right]}\]1

As explained above, we obtain the following preferred solution for \(S\) when \(\Delta H\) is known (even if \(H+\Delta H\) is not full rank):

\[\epsilon_{0}:=\sqrt{E\left[\left\|S_{0}-S_{*}\right\|_{F}^{2}+\left\|\Delta H _{0}-\Delta H_{*}\right\|_{F}^{2}\right]}\]2

as the best solution in terms of the Frobenius-norm. To conclude, from (24), (28), and (29), we need to solve

\[ \begin{cases}\Delta H=YS^{+}-H\\ S=(H+\Delta H)^{+}Y\\ \left\|S\right\|_{F}=r_{av}\sqrt{nk},\end{cases}  \tag{37}\]

to obtain a critical point of \(\mathcal{L}\) for which \(\
abla\mathcal{L}=0\). **Remark 2**. : _Note that (37) represents highly coupled non-linear non-convex equations of \(\Delta H\), \(S\).