## 3 The Generalized AP-HSIC Algorithm for QAM Modulations
\(m\geq n\). 2. \(m\geq n\). 3. \(m\geq n\). Figure 1: AP-HSIC algorithm flowchart. \(k\geq\max(m,n)=m\). 3. \(H_{t}:=H+\Delta H_{t}\) is full rank for any \(t\geq 0\). The first point is that, assuming CSIR only, the number of receiving antennas should be at least as large as the number of transmit antennas to achieve a high diversity order. Secondly, the number of frames (denoted as \(k\)) should be greater than the number of receiving antennas in order to increase the decoding rate. The paper [56] rigorously demonstrates in Theorem F.2 on page 21 that the estimation error \(E\left[\left\|H_{\text{estimated}}-H_{\text{exact}}\right\|_{F}^{2}\right]\to 0\) as \(k\rightarrow\infty\). Furthermore, Theorem F.1 on page 21 conclusively proves that the subset of full-rank matrices in \(\mathbb{C}^{m\times n}\) is dense in \(\mathbb{C}^{m\times n}\). Therefore, for any given \(\epsilon>0\), a full-rank matrix \(H_{t,\epsilon}\) exists, such that \(\left\|H_{t,\epsilon}-H_{t}\right\|_{F}^{2}\leq\epsilon\), which allows the assumption that \(H_{t}\) itself is full rank. Regarding the complexity of the algorithm, let

\[\epsilon_{0}:=\sqrt{E\left[\left\|S_{0}-S_{*}\right\|_{F}^{2}+\left\|\Delta H _{0}-\Delta H_{*}\right\|_{F}^{2}\right]}\]

denote the starting error, where \(S_{*}\) is the exact sent signal and \(\Delta H_{*}\) is the exact interference that defines a global optimal solution. Then, in [56], it was proved that the number of AP-HSIC iterations until the current error \(\epsilon_{t}\) becomes less than or equal to the given error threshold \(\epsilon\) is \(O\left(\left|\ln\left(\frac{\epsilon}{\epsilon_{0}}\right)\right|\right)\). The complexity of each iteration is \(O\left(\max(m,n,k)^{3}\right)=O\left(k^{3}\right)\) and is dominated by the pseudo-inverses calculations. Thus, the total complexity of the AP-HSIC algorithm is \(O\left(k^{3}\cdot\left|\ln\left(\frac{\epsilon}{\epsilon_{0}}\right)\right|\right)\). In Theorem F.4, it was proven that, in the worst case, \(\epsilon\) is bounded below as \(\epsilon\geq\frac{n}{\sqrt{mSNR}}\). Thus, for a fixed number of transmit antennas \(n\) and for any given \(SNR\), one should increase the number of receive antennas \(m\) accordingly, or for fixed \(n\) and \(m\), one should increase the SNR accordingly to obtain the needed threshold error. However, our experience with the AP-HSIC algorithm shows that it converges quickly to a globally optimal solution for any practical range of \(n,m,SNR\) and \(\epsilon>0\) regardless of the choice of the initial point. This is due to the reduction of the problem (11) to a convex problem under the assumption of \(H_{t},t\geq 0\) being full rank. ## 4 The LM-HSIC Algorithm

In this section, we will define a new algorithm that finds \(S\) and \(\Delta H\) to solve the problem

\[\arg\min_{\Delta H,S}\lVert Y-(H+\Delta H)S\rVert_{F}^{2}\] (19) _such that: \[\left\|S\right\|_{F}=r_{av}\sqrt{nk},\]_

where \(r_{av}=\mathbb{E}\left[\frac{\left\|S\right\|_{F}}{\sqrt{nk}}\right]\). The new algorithm uses Lagrange multipliers optimization. A significant advantage of this algorithm is its independence on the assumption that \(H+\Delta H\) is full rank, unlike AP-HSIC, which requires this assumption. This allows for dealing with scenarios where \(\Delta H\) is a interference receiver-response matrix that attempts to eliminate \(H\), thus reducing the rank of \(H+\Delta H\). Therefore, the new algorithm is considered complementary to the AP-HSIC algorithm. Moreover, since it uses fewer Moore-Penrose pseudo-inverse computations per iteration, it performs better in terms of CPU time since it converges only locally and not globally. Let the Lagrangian be

\[\mathcal{L}(\Delta H,S,\lambda)=\left\|Y-(H+\Delta H)S\right\|_{F}^{2}+\lambda \Big{(}\left\|S\right\|_{F}^{2}-r_{av}^{2}nk\Big{)}. \tag{20}\]

Let \(U=(U_{1},U_{2},U_{3})\in\mathbb{C}^{m\times n}\times\mathbb{C}^{n\times k}\times \mathbb{R}\) be a directional matrix with \(\left\|U\right\|_{F}=1\) and let \(h>0\).