## 3 Proposed Digital Canceler
### Multi-Layered Generalized HP-Based Adaptive Algorithm  
Nonlinear SIC can be regarded as a problem of nonlinear system identification. Before engaging in adaptive filtering, it is necessary to linearize the SI signal using the basis functions described in (4). The nonlinear PA model first proposed in [15] employs a set of generalized HPs for its representation. The \(p\)-th order nonlinear basis function in Equation (3) is defined as  
\[\phi_{p}\big{(}x(t);\mathbf{c}_{p}\big{)}=\sum_{k=0}^{p-1}c_{p,k}|x(t)|^{2k}x(t), \tag{18}\]  
which can be rewritten in the digital domain as  
\[\phi_{p}\big{(}x[n];\mathbf{c}_{p}\big{)}=\sum_{k=0}^{p-1}c_{p,k}|x[n]|^{2k}x[n], \tag{19}\]  
where \(c_{p}=\left[c_{p,0},\ldots c_{p,p-1}\right]^{T}\) represents the polynomial coefficients of the \(p\)-th basis function. When the polynomial coefficients satisfy \(c_{p,p-1}=1\) and \(c_{p,k}=0,\forall k\
eq p-1\), the basis function simplifies to the standard HP basis function, i.e., \(\phi_{p}(x[n])=|x[n]|^{p-1}x[n]\). Then, the SI signal \(s[n]\) can be represented by the aforementioned generalized HP as  
\[ \begin{split} s[n]&=\sum_{p=1}^{p+1}\sum_{l=0}^{L- 1}w_{p}[l]\phi_{p}\big{(}x[n-l];\mathbf{c}_{p}\big{)}\\ &=\sum_{p=1}^{p+1}\sum_{l=0}^{L-1}w_{p}[l]\left(\sum_{k=0}^{p-1}c _{p,k}|x[n-l]|^{2k}x[n-l]\right).\end{split}  \tag{20}\]  
To enhance the convergence speed of the adaptive filtering algorithm, it is also necessary to determine the coefficients \(\mathbf{c}_{p}=\left[c_{p,0},\ldots c_{p,p-1}\right]^{T}\) of the generalized HP basis functions to ensure their orthogonality. A set of orthonormal basis functions is defined as \(\left\{\phi_{1}(x_{L}[n];c_{1}),\ldots,\phi_{\frac{p-1}{2}}\left(x_{L}[n];c_{\frac {p-1}{2}}\right)\right\}\), which satisfies the following orthogonality condition:  
\[ \begin{cases}E\left[\phi_{p}\left(x[n];c_{p}\right)\phi_{j}\left(x[n];c_{j} \right)\right]=1,\,p=j\\ E\left[\phi_{p}\left(x[n];c_{p}\right)\phi_{j}\left(x[n];c_{j}\right)\right]=0, \,p\
eq j\end{cases} . \tag{21}\]  
For polynomial basis functions with the highest order term of \(2p-1\)\((p>1)\), assuming orthogonality with any basis function of order less than \(2p-1\), it is possible to derive \(p-1\) orthogonal equations as  
\[E\left[\phi_{p}\left(x[n];c_{p}\right)\phi_{k}(x[n];c_{k})\right]=0,k=1,2, \ldots,p-1. \tag{22}\]  
Furthermore, we can obtain  
\[ \begin{cases}E\left[\phi_{p}\left(x[n];c_{p}\right)c_{1,0}x[n]\right]=0,\,(k= 1)\\ E\left[\phi_{p}\left(x[n];c_{p}\right)\left(c_{2,0}x[n]+c_{2,1}|x[n]|^{2}x[n] \right)\right]=0,\,(k=2)\\ \vdots\\ E\left[\phi_{p}\left(x[n];c_{p}\right)\left(c_{p-1,0}x[n]+\cdots+c_{p-1,p-2}| x[n]|^{2p-2}x[n]\right)\right]=0,\,(k=p-1).\end{cases}  \tag{23}\]  
Assume \(c_{p,p-1}=1,\forall p\). When \(k=1\), we define \(\mu_{p}=E\big{[}(x[n])^{p}\big{]}\); then, (23) can be rewritten as  
\[ \begin{array}{l}E\Big{[}\Big{(}c_{p,0}x[n]+c_{p,1}|x[n]|^{2}x[n]+\cdots+|x[n ]|^{2p-2}x[n]\Big{)}x[n]\Big{]}\\ =E\Big{[}\Big{(}c_{p,0}|x[n]|^{2}+c_{p,1}|x[n]|^{4}+\cdots+|x[n]|^{2p}\Big{)} \Big{]}\\ =c_{p,0}\mu_{2}+c_{p,1}\mu_{4}+\cdots+\mu_{2p}\\ =0\end{array} ,(k=1). \tag{24}\]  
Let \(\mathbf{\mu}_{2a}^{2b}=\{\mu_{2a},\mu_{2a+2},\ldots,\mu_{2b}\},1\leqslant a \leqslant b\). Equation (23) can be written in matrix form as  
\[\mathbf{C}_{p}\mathbf{M}_{p}\bar{\mathbf{c}}_{p}+\mathbf{C}_{p}\mathbf{\mu}_{2p}^{4p- 4}=0, \tag{25}\]  
where \(\bar{\mathbf{c}}_{p}\) is a subvector of the \(\mathbf{c}_{p}=\left[\bar{\mathbf{c}}_{p}^{T}\ 1\right]^{T}\) and \(\mathbf{C}_{p}\) is an invertible lower triangular matrix. The matrix \(\mathbf{M}_{p}\) is defined as  
\[\mathbf{M}_{p}= \begin{bmatrix}\mu_{2}&\mu_{4}&\cdots&\mu_{2p-2}\\ \mu_{4}&\mu_{6}&\cdots&\mu_{2p}\\ \vdots&\vdots&\ddots&\vdots\\ \mu_{2p-2}&\mu_{2p}&\cdots&\mu_{4p-6}\end{bmatrix}