## 3 The Generalized AP-HSIC Algorithm for QAM Modulations  
In a previous study [56], and relating to (9), an algorithm was proposed for the computational self-feedback at the receiver that identifies both unknown \(\Delta H\) and an unknown symbol matrix \(S\in PSK^{n\times k}\) that solve the following problem:  
\[ \begin{split}&\arg\min_{\Delta H,S}\lVert Y-(H+\Delta H)S \rVert_{F}^{2}\\ &\text{\emph{such that:}}\ \lVert S\rVert_{F}\leq r_{max}\sqrt{nk}, \end{split}  \tag{11}\]  
where \(r_{max}=\sqrt{2}\) is the maximum absolute value of the points in the \(PSK\) modulation. Since this problem is non-convex, and since \(PSK^{n\times k}\) is a discrete finite set with at least two elements, the problem can be proven to be NP-complete. To circumvent the complexity of the problem, Ref. [56] adopted the following strategy. Assuming currently that \(\Delta H\) is known, the minimal Frobenius-norm solution of (9) for \(S\) is given by  
\[(H+\Delta H)^{+}Y=S, \tag{12}\]  
which is linear in terms of the variables \((H+\Delta H)^{+}\) and \(S\). Now, the solution space of the linear Equation (12) when \(\Delta H\) is also unknown is given by  
\[ \begin{cases}S=(W_{1}Y^{*}+W_{2})(I_{m}+YY^{*})^{-1}Y\\ (H+\Delta H)^{+}=(W_{1}Y^{*}+W_{2})(I_{m}+YY^{*})^{-1},\end{cases}  \tag{13}\]  
where \(W_{1}\in\mathbb{C}^{n\times k}\) and \(W_{2}\in\mathbb{C}^{n\times m}\) are free parameters. Finally, the following closed and convex sets in \(\mathbb{C}^{n\times m}\times\mathbb{C}^{n\times k}\) were defined as follows:  
\[\mathcal{C}_{0}=\Big{\{}\ \left(\left(H+\Delta H\right)^{+},S\right)\big{|}\ \Delta H\text{\emph{ arbitrary and}}\ S\in\mathbb{B}\Big{(}0,\sqrt{2 nk}\Big{)}\ \ \Big{\}}, \tag{14}\]  
and  
\[\mathcal{C}_{1}=\Big{\{}\Big{(}(H+\Delta H)^{+},S\Big{)}\big{|}\ \Delta H\text{ \emph{ and}}\ S\text{\emph{ are given by (\ref{eq  
We, therefore, propose an alternative for the closed-convex set \(\mathcal{C}_{0}\), so the algorithm will also work with QAM modulations. The convex sets and the projections are similar to what was defined for PSK modulations. Given a QAM modulation, let \(r_{max}\) be the maximum radius of magnitude of the modulation constellation. We define the ball of matrices \(\mathbb{B}\left(0,r_{max}\sqrt{nk}\right)\) as the set of all matrices \(S\) with the Frobenius-norm less than or equal to \(r_{max}\sqrt{nk}\). Since for each \(S\in QAM^{n\times k}\), \(\left\|S\right\|_{F}=\sqrt{\sum_{i=1}^{n}\sum_{j=1}^{k}\left|S_{ij}\right|^{2} }\leq r_{max}\sqrt{nk}\), then \(S\in\mathbb{B}\left(0,r_{max}\sqrt{nk}\right)\). Therefore, \(\mathbb{B}\left(0,r_{max}\sqrt{nk}\right)\) is a closed-convex set that contains all possible symbol matrices \(S\in QAM^{n\times k}\). We therefore define the new closed-convex set \(\mathcal{C}_{0}\) as follows:  
\[\mathcal{C}_{0}=\left\{\ \ \left(\left(H+\Delta H\right)^{+},S\right)\right|\ \Delta H\ \text{arbitrary and}\ S\in\mathbb{B}\left(0,r_{max}\sqrt{nk}\right)\ \ \right\} \tag{16}\]  
In addition, let \(P_{\mathcal{C}_{0}}:(A,B)\rightarrow(A,\beta B)\) be the (orthogonal) projection function on the closed-convex set \(\mathcal{C}_{0}\), where  
\[\beta=\left\{ \begin{array}{l}1\ \text{ if }\ \|B\|_{F}\leq r_{max}\sqrt{nk}\\ \frac{r_{max}\sqrt{nk}}{\|B\|_{F}}\ \text{otherwise}\end{array} \right.\]  
Finally, we define \(P_{QAM}\) as a function that projects the entries of \(S\) onto the closest points of the modulation to calculate the final feasible \(S\). For the definition of the orthogonal projection \(P_{\mathcal{C}_{1}}:(A,B)\rightarrow\left(\widehat{A},\widehat{B}\right) \in\mathcal{C}_{1}\), we compute (see [56] for the proof of correctness)  
\[\left[ \begin{array}{cc}W_{2}&W_{1}\end{array} \right]=\left[ \begin{array}{ cc}A(I_{m}+YY^{*})&BY^{*}(I_{m}+YY^{*})\end{array} \right]\cdot\left[ \begin{array}{ cc}I_{m}&YY^{*}\\ Y^{*}&Y^{*}YY^{*}\end{array} \right]^{+}. \tag{17}\]  
and set  
\[ \begin{cases}\widehat{A}=(W_{1}Y^{*}+W_{2})(I_{m}+YY^{*})^{-1}\\ \widehat{B}=(W_{1}Y^{*}+W_{2})(I_{m}+YY^{*})^{-1}Y,\end{cases}