###### Abstract  
In this paper, a novel scheme using hybrid \(l_{1}/l_{2}\) norm minimization and the orthogonal matching pursuit (OMP) algorithm is proposed to design the sparse finite impulse response (FIR) decision feedback equalizers (DFE) in multiple input multiple output (MIMO) systems. To reduce the number of nonzero taps for the FIR DFE while ensuring its design accuracy, the problem of designing a sparse FIR DFE is transformed into an \(l_{0}\) norm minimization problem, and then the proposed scheme is used to obtain the sparse solution. In the proposed scheme, a sequence of minimum weighted \(l_{2}\) norm problems is solved using the OMP algorithm. The nonzero taps positions can be corrected with the different weights in the diagonal weighting matrix which is computed through  
complexity, it is necessary to design the FIR DFE with only a few nonzero taps (i.e., sparse FIR DFE). In [10], the number of nonzero taps is reduced by choosing only a few significant taps of the minimum mean square error (MMSE) equalizer which is not sparse in general. Nevertheless, the non-sparse MMSE solution must be calculated first which increases the design complexity. Therefore, how to determine the positions and values of nonzero taps directly is one of the main difficulties to design the sparse FIR DFE. In recent years, some new ways of designing sparse FIR DFE have arisen with the research and application of sparse representation and compressed sensing theory [11; 12; 13; 14; 15; 16]. Among them, the compressed sensing framework for sparse signal processing is proposed in [11]. A greedy search method to determine the nonzero taps positions of a single input single output (SISO) system is presented in [12]. The \(l_{1}\) norm minimization method to design sparse filters is proposed in [13], but the modeling without wireless channel is quite different from the sparse equalizer design. In [14], a sparse adaptive equalizer design approach using the stochastic gradient pursuit algorithm based on compressed sensing is proposed, but the channel impulse response in the approach is assumed to be sparse. The new design frameworks for sparse FIR MIMO equalizers based on the orthogonal matching pursuit algorithm (OMP) and sparse approximation of a vector using different dictionaries are exploited in [15; 16], respectively. They achieve better performance than the method in [10] with the same number of nonzero taps. Furthermore, for the pursuit of both sparse promotion and improved accuracy, the sparse signals reconstruction algorithms inspired by \(l_{1}\) and weighted \(l_{1}\) regularization schemes are proposed in [17; 18]; the joint smoothed \(l_{0}\) norm algorithm for direction-of-arrival estimation in MIMO radar is proposed in [19]. We aim to reduce the number of nonzero taps for the FIR DFE while ensuring its design accuracy by applying the OMP algorithm to hybrid \(l_{1}/l_{2}\) norm minimization. The OMP algorithm is a popular greedy algorithm that determines a nonzero tap position for each iteration [20]. However, there is no correction for the selected nonzero taps positions in the previous iterations, which will reduce the design accuracy for the sparse FIR DFE. The hybrid \(l_{1}/l_{2}\) norm minimization is proposed in [21] to make the solution for data fitting less sensitive to large residuals and more robust, and it has been applied in many fields such as geophysics, image processing and machine learning. Within this context, we propose a scheme using hybrid \(l_{1}/l_{2}\) norm minimization and OMP algorithm. In the proposed scheme, a minimum weighted \(l_{2}\) norm problems is solved using the OMP algorithm in each iteration, and the nonzero taps positions can be corrected with the iteratively updated weights in the diagonal weighting matrix, then the weighting matrix is updated through the hybrid \(l_{1}/l_{2}\) norm minimization.