## 3 Proposed Digital Canceler
### Multi-Layered Generalized HP-Based Adaptive Algorithm
\text{and}\ \det\!\left(\mathbf{M}_{p}\right)\
eq 0 \tag{26}\]  
Thus, we have  
\[\bar{\mathbf{c}}_{p}=-\mathbf{M}_{p}{}^{-1}\mathbf{\mu}_{2p}^{4p-4}. \tag{27}\]  
The instantaneous basis function vector is defined as  
\[\Phi[n]=\Big{[}\mathbf{\phi}_{1}(\mathbf{x}_{L}[n];c_{1}),\ldots,\mathbf{\phi}_{\frac{p+1 }{2}}\left(\mathbf{x}_{L}[n];c_{\frac{p+1}{2}}\right)\Big{]}^{T}. \tag{28}\]Integrating the nonlinear PA model with the multi-layered joint channel estimator employing the LMS algorithm results in  
\[y[n] =\mathbf{w}^{T}[n]\Phi[n]+\mathbf{h}^{T}[n]i[n]+v[n] \tag{29}\] \[=\left[\mathbf{w}^{T}[n]\ \mathbf{h}^{T}[n]\right]\ \left[\mathbf{\phi}^{T}[n]\ \mathbf{i}^{T}[n]\right]^{T}+v[n]\] \[=\mathbf{h}^{T}_{joint}[n]\mathbf{u}[n]+v[n],\]  
where \(\mathbf{h}^{T}_{joint}[n]=\left[\mathbf{w}^{T}[n]\ \mathbf{h}^{T}[n]\right]\) and \(\mathbf{u}[n]=\left[\mathbf{\phi}^{T}[n]\ \mathbf{i}^{T}[n]\right]^{T}\). Furthermore, the iterative formula can be expressed as  
\[\hat{\mathbf{h}}_{joint}[n+1]=\hat{\mathbf{h}}_{joint}[n]+\mu\mathbf{u}[n]e[n], \tag{30}\]  
where \(\mu\) is the step size. At the \(m\)-th layer, the received signal after digital SIC is  
\[y_{(m+1)}[n]=y_{(m)}[n]-\hat{\mathbf{h}}^{T}_{joint(m)}\ [n]\Phi[n]+v[n]. \tag{31}\]  
The \(M\)-layered adaptive nonlinear SIC algorithm is summarized in Algorithm 1 and Table 2.  
```
1:received signal \(y[n]\), known signals \(x[n]\) and \(i[n]\), basis order \(P\), moment collect length \(N_{m}\) and basis function generation interval \(N_{int}\)
2:signal after SIC \(\hat{r}[n]\) and RT channel estimation \(\hat{\mathbf{h}}^{T}_{(m)}[n]\)Initialization:\(n_{s}=1\) and \(\mathbf{h}^{T}_{joint(m)}[n]=\mathbf{0}\)
3:for\(n=1,2,\ldots,N\)do
4:if\(n=n_{s}+N_{m}+1\)then
5:\(\mu_{p}=\frac{1}{N}\sum_{n_{s}}^{n_{s}+N_{m}-1}\left(x[n]\right)^{p}\)
6:for\(p=2,3,\ldots,\frac{p+1}{2}\)do
7:\(\tilde{c}_{p}=-\mathbf{M}_{p}{}^{-1}\mathbf{\mu}_{2p}^{4p-4}\), basis function \(\mathbf{\phi}_{p}\big{(}x[n];\mathbf{c}_{p}\big{)}\)generation
8:endfor
9:endif
10:if\(n=n_{s}+N_{int}-1\)then
11:\(n_{s}=n\)
12:endif
13:\(y_{(1)}[n]=y[n]=\mathbf{h}^{T}_{joint}[n]\mathbf{u}[n]+v[n]\)
14:for\(m=1,2,\ldots,M\)do
15:\(e_{(m)}[n]=y_{(m)}[n]-\hat{\mathbf{h}}^{T}_{joint(m)}[n]\mathbf{u}[n]\)
16:\(\hat{\mathbf{h}}^{T}_{joint(m)}[n+1]=\hat{\mathbf{h}}^{T}_{joint(m)}[n]+\mu\mathbf{u}[n] e_{(m)}[n]\)
17:\(\hat{\mathbf{w}}^{T}_{(m)}[n+1]=\left(\hat{\mathbf{h}}^{T}_{joint(m)}[n+

## 4 Simulation Results

### Linear SI Cancel

In the scenario described by Equation (9), the unknown vector \(\mathbf{w}\), with a dimensionality of \(L=14\), conforms to the condition \(\|\mathbf{w}\|^{2}=1\). Additionally, the elements of \(\mathbf{w}\) are distributed according to a Gaussian distribution with a mean of zero. The input signal is independently generated from zero-mean Gaussian with unit variance. The input and output noises are denoted as \(v_{in}=v_{b}\) and \(v_{out}=v_{a}+v_{i}\), respectively, where \(v_{a}\) and \(v_{b}\) are Gaussian-distributed with equal variances of \(\sigma_{a}^{2}=\sigma_{b}^{2}=0.1\). The Bernoulli-Gaussian (BG) process is utilized as an impulsive noise model. The impulsive noise \(v_{i}[n]=b[n]p[n]\), where \(b[n]\) is a Bernoulli process with the probability density function \(P(b[n]=1)=P_{i},P(b[n]=0)=1-P_{i}\) and \(p[n]\sim N(0,10)\)[26]. Their performance is displayed in Figure 4, quantitatively evaluated via the normalized mean squared difference (NMSD), defined as \(=10\log\big{(}\|\mathbf{\bar{w}}_{n}-\mathbf{w}\|^{2}/\|\mathbf{w}\|^{2}\big{)}\). It can be observed that the LMS and TLS algorithms decline due to noise interference, whereas the MTLS algorithm remains largely unaffected by such disturbances. We consider a baseband STAR system characterized by a 5 MHz signal bandwidth. The system operates at a sampling rate of 10 MHz, with both the local and remote transmitters employing binary phase shift keying modulation for signal transmission. The SI and RT channels have lengths specified as \(N_{1}=4\) and \(N_{2}=10\), respectively.