## 1 Introduction
Various schemes for time-varying filters have been proposed, including the explicit time-varying filter [13], the implicit time-varying filter [13], the variable bandwidth filter [14], the time-frequency projection filter [15], and the adaptive filter [16; 17]. In an adaptive filter, the received noisy signal is subjected to filtering and the filtered output is compared with a desired signal to compute the error. The task of the adaptive algorithm is to search for an optimal solution that minimizes the error. The adaptive algorithms include the gradient-descent-based search optimization algorithm, such as least mean square algorithm (LMS) [18], and the nongradient evolutionary algorithm, such as genetic algorithm (GA) [19] and particle swarm optimization algorithm(PSO) [20]. [PERSON] et al. [21; 22] compared the performances of GA, PSO and LMS algorithms, and concluded that GA and PSO performed better than LMS for the case of AWGN corrupted signal, but for non-linear random noise PSO outperformed the other two algorithms. Different filters have different advantages, disadvantages and applicable scenes. The short time Fourier transform (STFT)-based implicit time varying filter is easy to implement, but it has the problem of bad time frequency concentration and resolution [23; 24; 25]. The variable bandwidth filter-based on the extended Fourier transform (EFT) is suitable for the signal whose pass region is a band with clearly defined continuous upper bound and lower bound [26; 27]. Hence if the pass region includes an area without clear bound, the variable bandwidth filter is not applicable. The time frequency projection filter could process any shape of time frequency pass region, but there exists an obstacle to find the orthogonal subspace to construct the signal with a given TF support [28]. As for Wiener and Kalman filter, the adaptive algorithm needs multiple iterations, which affects the processing time. Moreover, it is effective to suppress the white noise rather than co-channel interference [29]. Therefore, the explicit time varying filter is adopted in this paper, and the key task is to determine the pass region and the masking threshold. The main contributions of this paper are summarized as follows. * The time frequency distributions of various communication signals are first analyzed via Choi-Williams analysis. The quadrature modulated signals' center frequency and bandwidth are not constant, but vary with time in the passband, and their distributions are irregular. Hence, it is a challenge to design the pass region for the time-varying filter. Fortunately, the distributions of the baseband and the binary modulated pulse shaping signals are regularly time-varying, which facilitates the pass region generation procedure. * The equivalence of baseband and RF power spectrum is obvious with Fourier transform. However, since the time frequency distribution is preprocessed by the analysis window, the window length will affect the equivalence of the time frequency distribution of the baseband and RF signals. Two methods are proposed to guarantee the consistence of the time frequency distribution of the baseband and RF signals by adjusting the window length and zero padding. * To suppress the non-orthogonal or co-channel interference, the explicit linear time varying filter is applied to mitigate the interference according to the energy distribution of the desired signal. Correspondingly, a masking threshold constrained time varying filter is developed with the optimal masking threshold determined by the minimum BER criterion. The remainder of this paper is organized as follows.