## 2 System Model  
Consider a baseband equivalent discrete signal model in the MIMO systems with \(n_{i}\) transmit (input) antennas and \(n_{o}\) receive (output) antennas. Assuming that the channel is a linear time-invariant Rayleigh fading channel with an oversampling factor of \(l\), the output of the MIMO channel at time \(k\)i.e., the received samples from the \(n_{o}\) receive antennas which are grouped into a \(ln_{o}\times 1\) vector can be formed as:  
\[\mathbf{y}_{k}=\left[ \begin{array}{c}\mathbf{y}_{k}^{(1)}\\ \mathbf{y}_{k}^{(2)}\\ \vdots\\ \mathbf{y}_{k}^{(n_{o})}\end{array} \right]=\left[ \begin{array}{cccc}\mathbf{H}_{0}& \mathbf{H}_{1}&\cdots&\mathbf{H}_{\
u}\end{array} \right]\left[ \begin{array}{c}\mathbf{x }_{k}\\ \mathbf{x}_{k-1}\\ \vdots\\ \mathbf{x}_{k-\
u}\end{array} \right]+\left[ \begin{array}{c}\mathbf{n}_{k}^{(2)}\\ \mathbf{n}_{k}^{(2)}\\ \vdots\\ \mathbf{n}_{k}^{(n_{o})}\end{array} \right], \tag{1}\]  
where \(\mathbf{y}_{k}^{(j)}=[y_{k}^{(j)}(1),y_{k}^{(j)}(2),\cdots,y_{k}^{(j)}(l)]^{\rm T}( 1\leq j\leq n_{o})\) is the \(l\times 1\) output vector of the \(j\)-th receive antennas, \(\mathbf{H}_{m}(0\leq m\leq\
u)\) is the \(ln_{o}\times n_{i}\)\(m\)-th MIMO channel matrix, \(\mathbf{x}_{k-m}\) is the input vector of the transmit antennas at time (\(k-m\)). The parameter \(\
u\) is the maximum length of all the subchannel impulse response. \(\mathbf{n}_{k}^{(j)}\) is an additive white Gaussian noise vector of the \(j\)-th receive antennas with mean 0 and variance \(\sigma^{2}\). Considering a block of \(N_{f}\) received vector \(\mathbf{y}_{k}\), (1) can be rewritten as [9]  
\[\mathbf{y}_{k:k-N_{f}+1}=\mathbf{H}_{k:k-N_{f}-\
u+1}+\mathbf{n}_{k:k-N_{f}+1}, \tag{2}\]  
where  
\[\mathbf{H}=\left[ \begin{array}{cccccc}\mathbf{H}_{0}&\mathbf{H}_{1}&\cdots&\mathbf{H}_{\
u }&0&0&\cdots&0\\ 0&\mathbf{H}_{0}&\mathbf{H}_{1}&\cdots&\mathbf{H}_{\
u}&0&\cdots&0\\ \vdots&\ddots&\ddots&\ddots&\ddots&\ddots&\ddots&\vdots\\ 0&\cdots&0&0&\mathbf{H}_{0}&\mathbf{H}_{1}&\cdots&\mathbf{H}_{\
u}\end{array} \right]. \tag{3}\]  
Assuming that the transmit signals are independent with the average power \(\varepsilon_{x}\), the input autocorrelation matrix can be expressed as  
\[\mathbf{R}_{xx}\triangleq E\left[\mathbf{x}_{k:k-N_{f}-\
u+1}\mathbf{x}_{k:k-N_{f}-\
u+1 }^{\rm H}\right]=\varepsilon_{x}\mathbf{I}_{n_{i}(N_{f}+\
u)}, \tag{4}\]  
where \(\mathbf{I}\) is the identity matrix, and the noise autocorrelation matrix can be expressed as  
\[\mathbf{R}_{mt}\triangleq E\left[\mathbf{n}_{k:k-N_{f}+1}\mathbf{n}_{k:k-N_{f}+1}^{\rm H} \right]=\sigma^{2}\mathbf{I}_{n_{0}N_{f}}. \tag{5}\]  
Furthermore, the output-input crosscorrelation and the output autocorrelation matrices are as follows  
\[\mathbf{R}_{yx}\triangleq E\left[\mathbf{y}_{k:k-N_{f}+1}\mathbf{x}_{k:k-N_{f}-\
u+1}^{\rm H }\right]=\mathbf{H}\mathbf{R}_{xx}, \tag{6}\]  
\[\mathbf{R}_{yy}\triangleq E\left[\mathbf{y}_{k:k-N_{f}+1}\mathbf{y}_{k:k-N_{f}+1}^{\rm H} \right]=\mathbf{H}\mathbf{R}_{xx}\mathbf{H}^{\rm H}+\mathbf{R}_{nn}. \tag{7}\]  
Design of Sparse FIR DFE Using Hybrid \(\mathbf{l_{1}/l_{2}}\) Norm Minimization and OMP Algorithm  
In the FIR MIMO DFE model, in order to eliminate ISI and inter-antenna interference, then recover the input signal of the \(i\)-th transmit antenna, the feedforward filter \(\mathbf{w}_{i}\) with \(ln_{o}N_{f}\) taps processes the received signals \(\mathbf{y}_{k}\), and the feedback filter \(\mathbf{b}_{i}\) with \(n_{i}N_{b}\) taps convolves with the previous decision signals, then the result combined both together is applied to the decision device to obtain current decision signal. If the previous decision signals before time \(k\) are correct, the DFE error sample at time \(k\) for the \(i\)-th input antenna is given by  
\[\varepsilon_{k}^{(i)}=x_{k-\Delta}^{(i)}-\underbrace{\left[\mathbf{w}_{i}^{\rm H }-\mathbf{b}_{i}^{\rm H}\right]}_{\triangleq\widehat{\mathbf{w}}_{i}^{\rm H}} \underbrace{\left[ \begin{array}{c}\mathbf{y}_{k:k-N_{f}+1}\\ \mathbf{x}_{k-\Delta-1:k-\Delta-N_{b}}\end{array}