## 2 Problem Formulation  
### MIMO Radar System Model with Mutual Coupling  
Consider a narrowband monostatic MIMO radar system, shown in Figure 1, whose transmit and receive arrays are both half-wavelength \(d\) spaced uniform linear arrays (ULAs). In the transmit array, \(M\) antennas transmit \(M\) orthogonal noncircular waveforms, such as BPSK modulated signals. In the receive array, \(N\) antennas are impinged by \(P\) uncorrelated targets that can be deemed as point scatterers in the far-field. Different from the mutually-separated arrays in bistatic MIMO radar, the transmitter and the receiver are closely located in monostatic radar, as shown in Figure 2. Since the transmit array and the receive array are collocated at the same site in monostatic MIMO radar systems [17], the echo signal travels the same path as the transmitted signal. Therefore, for the \(p\)-th target, the directions of departure of the transmitted signal and arrival of the echo from the target are the same [21], i.e., \(\theta_{DOD}=\theta_{DOA}\), both of which are denoted as DOA \(\theta_{p},p=1,2,\ldots P\). When mutual coupling is taken into account in both the transmit array and the receive array, at the receiver, the output of the matched filters at a certain snapshot is given by [16]:  
\[\mathbf{x}(t)=\tilde{\mathbf{A}}\mathbf{s}(t)+\mathbf{n}(t) \tag{1}\]  
where \(\mathbf{x}(t)\in\mathbb{C}^{MN\times 1}\) is the received data vector, \(\mathbf{s}(t)=[s_{1}(t),s_{2}(t),\ldots,s_{P}(t)]^{\rm T}\in\mathbb{C}^{P \times 1}\) is the complex-valued noncircular signal vector and \(\mathbf{n}(t)\in\mathbb{C}^{MN\times 1}\) is the additional Gaussian noise vector with zero-mean and covariance matrix \(\sigma^{2}\mathbf{I}_{MN}\), in which \(\sigma^{2}\) is the noise power. Meanwhile, the detailed expression of \(\tilde{\mathbf{A}}\) is [14]:  
\[ \begin{split}\tilde{\mathbf{A}}&=[\mathbf{C}_{t} \mathbf{a}_{t}(\theta_{1})\otimes\mathbf{C}_{r}\mathbf{a}_{r}(\theta_{1}), \ldots,\mathbf{C}_{t}\mathbf{a}_{t}(\theta_{P})\otimes\mathbf{C}_{r}\mathbf{a }_{r}(\theta_{P})]\\ &=(\mathbf{C}_{t}\otimes\mathbf{C}_{r})[\mathbf{a}_{t}(\theta_{1 })\otimes\mathbf{a}_{r}(\theta_{1}),\ldots,\mathbf{a}_{t}(\theta_{P})\otimes \mathbf{a}_{r}(\theta_{P})]\end{split}  \tag{2}\]  
where \(\tilde{\mathbf{A}}\in\mathbb{C}^{MN\times P}\), \(\mathbf{a}_{t}(\theta_{p})=[1,e^{j\pi\sin(\theta_{p})},e^{j\pi 2\sin(\theta_{p})}, \ldots,e^{j\pi(M-1)\sin(\theta_{p})}]^{\rm T}\) is the transmit steering vector and \(\mathbf{a}_{r}(\theta_{p})=[1,e^{j\pi\sin(\theta_{p})},e^{j\pi 2\sin(\theta_{p})}, \ldots,e^{j\pi(N-1)\sin(\theta_{p})}]^{\rm T}\) is the receive steering vector, \(p=1,2,\ldots P\). \(\mathbf{C}_{t}\) and \(\mathbf{C}_{r}\) are mutual coupling matrices of the transmit array and the receive array, respectively. \(\mathbf{C}_{t}\) and \(\mathbf{C}_{r}\) can be modeled as banded symmetric Toeplitz matrices, which are expressed as [22; 23; 24]:\[\mathbf{C}_{i}=\left[ \begin{array}{cccccc}c_{i0}&c_{i1}&\cdots&c_{ik}&\cdots&0 \\ c_{i1}&c_{i0}&c_{i1}&\cdots&\ddots&\vdots\\ \vdots&c_{i1}&c_{i0}&\ddots&\ddots&c_{ik}\\ c_{ik}&\cdots&\ddots&\ddots&c_{i1}&\vdots\\ \vdots&\ddots&\cdots&c_{i1}&c_{i0}&c_{i1}\\ 0&\cdots&c_{ik}&\cdots&c_{i1}&c_{i0}\end{array} \right] \tag{3}\]  
where \(i=t,r\), \(c_{ij}\) is the non-zero mutual coupling coefficient, \(j=0,1,...,k\). In Figure 1, the coupling effects of the \(m\)-th and the \(n\)-th elements in the transmit and the receive arrays are described, respectively. \(\bar{\rho}_{(m,q)}\) is the coupling contribution from the \(q\)-th to the \(m\)-th elements in transmit array, and \(\bar{\beta}_{(n,q^{\prime})}\) is the contribution from the \(q^{\prime}\)-th to the \(n\)-th elements in the receive array, \(1\leq m\leq M\), \(1\leq n\leq N\), and \(|q-m|\leq k\), \(|q^{\prime}-n|\leq k\). \(\bar{\beta}_{(m,q)}\) and \(\bar{\beta}_{(n,q^{\prime})}\) contain the information of \(c_{|q-m|}\) and \(c_{r|q^{\prime}-n|}\), respectively.