## 1 Introduction  
Multiple-input multiple-output (MIMO) radar technology was formally introduced in 2004 [1]. The concept of MIMO radar is widely used in various military and civilian equipment. It has attracted much attention due to its high positioning accuracy, low-speed moving-target detection performance, and very low interception probability [2; 3; 4]. At the receiving end of MIMO radar, each receiving element receives all the transmitted signals and acquires multiple echo signals by matching the filter after signal processing of clutter filtering. MIMO radar can detect targets by transmitting multiple orthogonal signals. The mutual coupling phenomenon is obvious between the MIMO radar antenna array elements if the element distance is too close. Therefore, in order to ensure the orthogonality between MIMO radar multiple channels, it is necessary to maintain sufficient distance between the radar antenna elements, which is difficult for installation because of the bulkiness and expensiveness of a MIMO radar system. In addition, there are still many problems with civilian MIMO radar applications, such as automobile radar, which has poor robustness of output waveform and an obvious multipath effect [5]. In order to solve a series of problems caused by a large transmitting array, a sparse array method was adopted to simplify the hardware composition of the MIMO radar system [6]. However, this method will lead to an increase in the side lobe level of MIMO radar, which needs to be optimized by various algorithms to offset the side lobes without changing the size of MIMO radar. The genetic algorithm (GA), particle swarm optimization (PSO), least squares algorithm (LS), and other traditional intelligent algorithms have been previously derived to meet the requirements of antenna array distribution [7; 8; 9; 10]. As one of the important aspects of array signal processing, direction of arrival (DOA) has been widely used in many fields, such as radar, communication, sonar, seismic exploration, radio astronomy, and biomedical engineering. Multiple signal classification (MUSIC) and estimation rotational parameter via rotational invariance (ESPRIT) are two typical algorithms used for estimating the DOA. The classic process used for MUSIC performance analysis in DOA estimation is illustrated in [11], where high signal-to-noise (SNR) maximum-likelihood estimator (MLE) performance was obtained. Two methods of image generation from multistatic data were analyzed in [12], and the MUSIC method adopted the singular vectors having small or zero singular values. The performance of time-reversal (TR) MUSIC for computational TR applications was studied in [13] to give a theoretical analysis of MUSIC for DOA estimation. In [14], a combined signal parameter unitary estimation by an ESPRIT-based algorithm was used to optimize the DOA of MIMO radar. However, the genetic algorithm has limitations: it easily falls into the local extremum and cannot find the optimal solution. The DOA estimation for MIMO radar with mutual coupling is also addressed in [15; 16]. The reweighted sparse representation algorithm based on noncircular sources was suggested to provide higher resolution and better angle estimation performance than ESPRIT algorithm. A pilot pattern algorithm based on the least squares (LS) algorithm for channel estimation in MIMO-orthogonal frequency division multiplexing (OFDM) systems is discussed in [17]. The joint smoothed \(l_{0}\)-norm algorithm for fast sparse DOA estimation has been proposed for multiple measurement vectors in MIMO radar systems [18]. The results indicated that it can eliminate the white or colored Gaussian noises and achieve better DOA estimation performance. The nuclear norm minimization algorithm [19] was proposed to extend the virtual array aperture, which provides better performance compared with the conventional sparse recovery-based algorithms and can handle the case of underdetermined DOA estimation in MIMO radar systems.