## 3 The Generalized AP-HSIC Algorithm for QAM Modulations
Contrastingly, the function \(f(\Delta H)=\left\|Y-(H+\Delta H)S\right\|_{F}^{2}\) is a convex function of \(\Delta H\) for a given \(S\), and_

\[\left\|Y-(H+\Delta H)S\right\|_{F}^{2} =\left\|Y\big{(}I-S^{+}S\big{)}+\big{(}YS^{+}-(H+\Delta H)\big{)} S\right\|_{F}^{2}\] \[=\left\|Y\big{(}I-S^{+}S\big{)}\right\|_{F}^{2}+\left\|\big{(}YS^ {+}-(H+\Delta H)\big{)}S\right\|_{F}^{2},\]

_implying that the global minimum of \(f\) is accepted for \(\Delta H\) satisfying_

\[\big{(}YS^{+}-(H+\Delta H)\big{)}S=0,\]

_which, by the assumption that \(S\) is full rank (and therefore \(SS^{+}=I_{n}\)), is equivalent to_

\[\Delta H=YS^{+}-H.\]

_Similarly, the function \(g(S)=\left\|Y-(H+\Delta H)S\right\|_{F}^{2}\) is a convex function of \(S\) for a given \(\Delta H\), and_

\[\left\|Y-(H+\Delta H)S\right\|_{F}^{2}=\] \[=\left\|\Big{(}I-(H+\Delta H)(H+\Delta H)^{+}\Big{)}Y+\big{(}H+ \Delta H\big{)}\Big{(}(H+\Delta H)^{+}Y-S\Big{)}\right\|_{F}^{2}\] \[=\left\|\Big{(}I-(H+\Delta H)(H+\Delta H)^{+}\Big{)}Y\right\|_{F }^{2}+\left\|(H+\Delta H)\Big{(}(H+\Delta H)^{+}Y-S\Big{)}\right\|_{F}^{2},\]

_implying that the global minimum of \(g\) is accepted for \(S\) that minimizes_

\[\left\|(H+\Delta H)\Big{(}(H+\Delta H)^{+}Y-S\Big{)}\right\|_{F}^{2},\]

_that is, for_

\[S=(H+\Delta H)^{+}Y.\]

_These are exactly the equations appearing in (37). This explains the fast convergence of the algorithm since once \(\Delta H\) or \(S\) was identified correctly, the other is concluded immediately._

Algorithm 4 presented below, which we call the Lagrange Multipliers Hard Successive Interference Cancellation (LM-HSIC), solves the problem (19), after which it uses \(P_{QAM}\) to project the final \(S\) onto the closest point in the related \(QAM^{n\times k}\) modulation. In its main loop, it computes \(\Delta H_{t+1},S_{t+1}\) by calculating (37) as \(YS_{t}^{+}-H\) and \((H+\Delta H_{t})^{+}Y\), respectively, until the errors converge below some prescribed error threshold \(\epsilon>0\). The flow of computations is described in Figure 2.

The LM-HSIC assumptions are summarized as follows:

1. \(k\geq\max(m,n)=m\). 2. \(S_{t}\) is full rank for any \(t\geq 0\). 3. The exact sent signal satisfy \(\left\|S_{*}\right\|_{F}\approx\mathbb{E}[\left\|S\right\|_{F}]\). 4. \(\Delta H_{0},S_{0}\) is sufficiently close to the global minimum \(\Delta H_{*},S_{*}\) of \(\mathcal{L}\), or at least \(\Delta H_{0},S_{0}\) is sufficiently close to a local minimum of \(\mathcal{L}\). The reasoning for \(k\geq\max(m,n)\) is the same as for the AP-HSIC. The reasoning for \(S_{t}\) being full rank is to obtain an explicit solution for Equation (24) (see also Remark 2). However, note that the exact sent signal \(S_{*}\) is \(n\times k\), where \(k\) depends on the receiver sample rate that can be chosen as \(k\geq m\geq n\), thus allowing for \(S_{*}\) to be full rank [7; 15]. Moreover, the content of \(S_{*}\) can be decided by protocols that guarantee it being full rank [7; 15]. Thus, in this regard, LM-HSIC is less restrictive than the AP-HSIC. The LM-HSIC algorithm's complexity per iteration is \(O(\max(m,n,k))=O(k^{3})\) due to pseudo-inverse computations. The number of iterations needed for convergence is uncertain, but typically, it converges in just a few iterations based on our experience. A disadvantage of the algorithm is the assumption that the Frobenius-norm of the transmitted \(S_{*}\) is close to \(\mathbb{E}[\|S\|_{F}]\). The problem with this assumption might arise in cases where \(\left\|S_{*}\right\|_{F}\) is far from \(\mathbb{E}[\|S\|_{F}]\). For example, when \(S_{*}=s_{0}\mathbb{I}_{n\times k}\) (i.e., all the symbols are the same), where \(s_{0}\) is a symbol in the modulation that has the minimal radius (or the maximal radius). Now, because of the constraint, the algorithm outputs \(\widetilde{S}\), where \(\left\|\widetilde{S}\right\|_{F}\approx\mathbb{E}[\|S\|_{F}]\), and, therefore, the resulting \(\widetilde{S}\) will be inaccurate.