# A Deep Learning Method for DOA Estimation with Covariance Matrices in Reverberant Environments
## 5 Simulations and Evaluation
### Measure  
In order to evaluate the performance, the gross error (GE) is used for the measure stage. The GE is the percentage of estimation that does not fall into an allowed threshold from the original value. Thus, for the simulated dataset, the GE is used to evaluate the performance. As for the LOCATA dataset, mean deviation and standard deviation [41] are utilized for the evaluation. The GE is defined as  
\[\mathrm{GE}_{\theta}=\frac{1}{N_{K}}\sum_{\kappa=1}^{N_{K}}\big{[}I_{e}(| \theta_{\kappa}-\hat{\theta}_{\kappa}|-\lambda)\big{]}, \tag{24}\]  
\[\mathrm{GE}_{\phi}=\frac{1}{N_{K}}\sum_{\kappa=1}^{N_{K}}\big{[}I_{e}(|\phi_{ \kappa}-\hat{\phi}_{\kappa}|-\lambda)\big{]}, \tag{25}\]  
\[\mathrm{GE}=\frac{1}{N_{K}}\sum_{\kappa=1}^{N_{K}}\big{[}I_{e}(\Delta((\theta_ {\kappa},\phi_{\kappa}),(\hat{\theta}_{\kappa},\hat{\phi}_{\kappa}))-\lambda) \big{]}, \tag{26}\]  
where \(N_{K}\) is the number of DOAs estimated, \((\theta_{\kappa},\phi_{\kappa})\) and \((\hat{\theta}_{\kappa},\hat{\phi}_{\kappa})\) are the actual and the estimated DOA of the \(\kappa\)th source, \(\lambda\) is the threshold that represents the maximum acceptable error, \(\Delta\) is the angular distance and it is defined as  
\[\Delta[(\theta,\phi),(\hat{\theta},\hat{\phi})]=\arccos[\sin\hat{\theta}\sin \theta+\cos\theta\cos\theta\cos(\hat{\phi}-\phi)], \tag{27}\]  
\(I_{e}(x)\) is  
\[I_{e}(x)=\left\{ \begin{array}{ll}1&\mathrm{for}\;x\geqslant 0\\ 0&\mathrm{for}\;x<0\end{array} \right.. \tag{28}\]  
As introduced before, for the simulated dataset, the GE is used for the evaluation. \(\lambda=10,15,20\) is considered. Table 3 demonstrates the GE for the azimuth estimation with different thresholds. Table 4 illustrates the GE for the elevation estimation with different thresholds. The symbol '-S' in the tables means that this task is finished with the segmentation strategy. The GE value shown in the two tables is an average over all the combinations of SNR and \(RT_{60}\). \begin{table}
\begin{tabular}{c c c c} \hline
**Method** & \(\lambda\) **= 20** & \(\lambda\) **= 15** & \(\lambda\) **= 10** \\ \hline MUSIC & 52.68\% & 65.02\% & 74.56\% \\ DPD-MUSIC & 38.77\% & 41.77\% & 44.34\% \\ FOA & 16.77\% & 23.96\% & 28.01\% \\ \hline TFD & 21.56\% & 26.67\% & 30.45\% \\ TFD-S & 18.23\% & 24.63\% & 27.89\% \\ \hline SHD & 16.98\% & 23.63\% & 28.77\% \\ SHD-S & 14.84\% & 19.62\% & 25.33\% \\ \hline AED & **13.05\%** & **18.08\%** & **25.08\%** \\ AED-S & **9.96\%** & **14.77\%** & **21.86\%** \\ \hline \end{tabular}
\end{table} 
Table 3: GE for the azimuth estimation. From Table 3, it can be seen that the proposed input feature, the AED matrix, is the most efficient input feature for the azimuth estimation task. Furthermore, when the input features are fixed, compared to the regular strategy, the segmentation strategy has higher accuracy in the test of the azimuth estimation. Thus, the combination of the AED matrix and the segmentation strategy is the best choice for the azimuth. It can be known from Table 4 that the proposed input feature, the AED matrix, is also the most efficient input feature for the elevation estimation task. Moreover, when the input features are fixed, compared to the regular strategy, the segmentation strategy still has higher accuracy in the test of the elevation estimation. Although the improvement is not as remarkable as that in the azimuth estimation task, the segmentation strategy does bring improvement. Therefore, the proposed method can effectively improve the accuracy for both the azimuth estimation and elevation estimation. The relationships between the results and the parameters are shown in Figures 7-9. From Table 3, Table 4, it can be known that the SH-MUSIC method and the DPD-MUSIC algorithm perform worse than the deep learning methods. Thus, these two methods are removed from the experiments for exploring the relationship between results and influencing factors. The relationship between the results and \(RT_{60}\) is given in Figure 7 with the threshold \(\lambda=20\).