## 3 Proposed Digital Canceler
### Multi-Layered Generalized HP-Based Adaptive Algorithm
We assess the performance in terms of the mean squared error (MSE), defined as follows:

\[MSE=\mathbb{E}\Big{[}(f(\mathbf{x}[n])-\hat{y}[n])^{2}\Big{]}. \tag{33}\]

The performance of the proposed algorithm and the commonly used LMS algorithms based on HPs are evaluated and the results are shown in Figure 6. In this simulation, we assume the transmitted data follow a uniform distribution in the range \([-2.5,2.5]\) and \(v[n]\) is zero-mean Gaussian noise with variance \(1\). As shown in Figure 6, compared to existing algorithms based on standard HPs, the proposed algorithm achieves faster convergence while maintaining equivalent MSE performance. In the nonlinear SIC scenario, we simulate data transmission using a uniform distribution within the range of \([-2.5,2.5]\), and the remote signal \(i[n]\in\{-1,1\}\). The nonlinear SI canceler is demonstrated in Section 3.2. Considering both computational complexity and algorithm performance, the joint SIC algorithm has a two-layer filtering structure with \(M=2\). The basis functions for all algorithms are of order up to five. Figures 7 and 8 compare the power spectrum of different signals at the receiver for \(\text{ISR}=20\) dB and \(\text{ISR}=40\) dB, respectively. As seen, in both cases, the signal after SIC more closely approximates the remote signal when using the proposed SIC algorithm compared to the method based on HP-LMS. Figure 9 shows the NMSD and bit error rate (BER) performance of the proposed algorithm and other SIC algorithms. For symbol detection at the receiver, a decision feedback equalizer (DFE) is employed, with both the feedforward and feedback filters having lengths of 15. The DFE employs a truncated version of the transmitted signal as the training signal, with tap weights being trained exclusively during the first block. In this result, as expected, the joint SIC algorithm with a generalized HP basis function with orthogonality exhibits superior performance. Figure 6: MSE performance of different SIC algorithms with the same step size, when \(P=5\). Figure 8: Power spectrum of signals at the receiver, with \(\text{ISR}=40\) dB. (**a**) Proposed algorithm; (**b**) HP-LMS algorithm. Figure 7: Power spectrum of signals at the receiver, with \(\text{ISR}=20\) dB. (**a**) Proposed algorithm; (**b**) HP-LMS algorithm. Figure 9: Performance of different algorithms, with \(\text{ISR}=40\) dB. (**a**) NMSD performance; (**b**) BER performance. ## 5 Conclusions

In this paper, we proposed two adaptive algorithms for linear and nonlinear SIC, based on a multi-layered joint channel estimator structure. In the case of linear SI, given that both the independent and dependent variables are subject to measurement errors, and considering the impact of impulsive noise present in wireless environments on the convergence process of adaptive algorithms, the m-MTLS algorithm is proposed to enhance robustness. For the nonlinear distortion generated by the PA, we combine the joint channel estimator and generalized HP basis function to propose a joint SIC method that outperforms current state-of-the-art SIC algorithms in scenarios of both high and low ISR, as well as high SNR. However, the complexity of this method is higher, mainly due to the increase in the number of basis function coefficients and the length of the filter during joint estimation, as well as the number of layers \(M\), which also contributes to the increased complexity. Conceptualization, S.S. and Y.T. ; methodology, S.S. and Y.T. ; software, S.S.; validation, S.S. and X.L. ; formal analysis, S.S. and Y.Z. ; investigation, Z.W. ; resources, X.W. and S.G.; data curation, Y.Z. ; writing--original draft preparation, S.S.; writing--review and editing, S.S., X.L. and Y.Z. ; visualization, X.L. ; supervision, S.G.; project administration, Y.T. ; funding acquisition, Y.T. All authors have read and agreed to the published version of the manuscript. This research received no external funding. Not applicable. Data Availability StatementData are contained within the article.