## 1 Introduction  
With orthogonal transmitted waveforms, multiple-input multiple-output (MIMO) radar has drawn increasing attention in the field of wireless communications. Compared with the conventional phased-array radar, MIMO radar owns a number of advantages, such as higher resolution and better parameter identifiability [1]. In addition, MIMO radar can be classified into the following two types: statistical MIMO radar and colocated MIMO radar. Colocated MIMO radar can achieve more degrees of freedom and higher spatial resolution, because its closely-spaced antennas form a virtual array with a large aperture. Colocated MIMO radar includes the monostatic one and the bistatic one. In bistatic MIMO radar, the transmit array and the receive array are separated away, while they are close to each other in monostatic MIMO radar. In this paper, the angle estimation problem is investigated in monostatic MIMO radar. Parameter estimation is an important aspect in wireless communications and sensor array signal processing [2]. In order to estimate the direction of arrival (DOA), a large number of subspace-based algorithms have been proposed [3; 4; 5]. Recently, the emerging sparse representation (SR) and compressed sensing (CS) have attracted more and more attention in various fields [6; 7; 8], such as sensor array signal processing, imaging and image processing, and so forth. Furthermore, it has been verified that SR-based DOA estimation algorithms have remarkable advantages over other methods [9], for instance they adapt better to challenging circumstances and provide higher angle resolution. A few SR-based algorithms, such as \(l_{1}\)-SVD (singular value decomposition) [9] and \(l_{1}\)-SRACV (array covariance vectors) [10], have been proposed via sparse signal recovery. In [11], based on the array covariance vector, the revised real-valued (RV) \(l_{1}\)-SVD approach has better DOA estimation performance and lower computation burden than \(l_{1}\)-SVD [9] and RV \(l_{1}\)-SVD [12]. However, in practical situations, the sensor array manifold is often affected by the errors of unknown mutual coupling. When they are taken into account, the DOA estimation performance of the above-mentioned methods is seriously degraded. With mutual coupling, the subspace-based MUSIC-like (multiple signal classification-like) [13] algorithm is proposed. Besides, a ESPRIT-like (estimation of signal parameters via rotational invariance techniques-like) algorithm is presented in [14], which owns a low calculation burden. In [15], the SR-based revised \(l_{1}\)-SVD algorithm eliminates the errors of mutual coupling, and it achieves angle estimation by solving the problem of the \(l_{1}\)-norm constraint minimization. In [16], in the presence of mutual coupling, by introducing the refined deterministic maximum likelihood (DML) procedure, a covariance vector-based sparse representation algorithm is proposed with outstanding angle estimation performance. We call the method in [16]\(l_{1}\)-SRDML (sparse representation deterministic maximum likelihood). In practical communication systems and radar applications, the complex noncircular sources are widely used, such as binary phase shift keying (BPSK), multiple amplitude shift keying (MASK) and unbalanced quadrature phase shift keying (UQPSK) modulated signals [17]. Noncircular signals can be used to enlarge the array aperture without extra antennas, which contributes to the improvement of angle estimation performance [18]. Exploiting the signal noncircularity, some subspace-based methods have been developed for angle estimation in MIMO radar [17; 19; 20]. They have verified that compared with traditional subspace-based algorithms, the methods that are based on noncircular signals can achieve better DOA estimation performance. However, these subspace-based approaches do not take the errors of unknown mutual coupling into account. Moreover, they perform poorly when experiencing challenging circumstances, such as fewer snapshots, which can be solved by sparse representation-based methods.