## 1 Introduction
The work in [11; 12] focused on linear SIC by employing the least mean square (LMS) adaptive filter. In addition, different linear SIC methods were compared in [13]. To compensate for nonlinear PA distortion, the authors in [18] proposed novel predistorters and their parameter extraction algorithms. The proposed self-adaptive nonlinear digital cancelers in [14; 15] both utilize a novel orthogonalization procedure for nonlinear basis functions, together with low-cost LMS-based parameter learning. Both PA nonlinearity and I/Q imbalances were considered in [19; 20], and more impairments, including local oscillator leakage and baseband nonlinearity, were analyzed and measured on the Universal Software Radio Peripheral in [7]. Some studies have also proposed various frequency domain SIC methods [21; 22]. In [21], a frequency domain SI canceler based on the parallel Hammerstein (PH) model for an IBFD orthogonal frequency division multiplexing System was proposed, while Ref. [22] utilized a successive cancellation cascaded structure as a replacement for the adaptive filter, effectively reducing the computational load of SIC processing and simplifying the procedure. Most published works are based on the  
Figure 1: A block diagram illustrating the STAR system and various forms of SIC. scenario where only the own transmitter (TX) signal is known, without considering the remote transmission (RT) signal. When the RT sends a signal for channel estimation, it can be assumed to be known, and additional prior information will further enhance the performance of SIC [23]. In this paper, a robust multi-layered M-estimate total least mean square (m-MTLS) canceler is proposed to enhance the performance in scenarios where both the data matrix and the observation signal are also affected by impulsive noise. The method exhibits significant robustness against impulse noise, managing to maintain a low convergence error even in the presence of impulse noise. When both the PA nonlinearities of the local transmitter and the training signal from RT are taken into account, we introduce a novel algorithm for nonlinear digital SIC based on a set of generalized Hammerstein polynomials (HPs) [15], by estimating the SI channel and the RT channel at the same time. This method of joint estimation can more accurately estimate the remote channel while reducing the residual SI signal, compared with existing SIC methods. The simulation results show that this method exhibits a faster convergence rate and demonstrably enhances system performance as the signal-to-noise ratio (SNR) increases. The rest of this paper is organized as follows. In Section 2, the basic STAR system and linear SI signal modeling are first described. Then, considering the nonlinearity introduced by the PA, the SI model is extended to a nonlinear model, followed by the introduction of a joint estimator that can simultaneously estimate both SI and RT channels. Section 3 proposes two new algorithms for linear and nonlinear SIC based on adaptive FIR, respectively. The linear SIC also enhances the robustness of the algorithm under impulsive noise interference. The simulation results are reported in Section 4. Finally, Section 5 concludes the paper. In the following, continuous and discrete time signals are expressed in italic lowercase, while vectors are denoted by bold lowercase symbols. The acronyms used throughout the paper are also summarized in Table 1.