# Deep Learning-Based Bird's Nest Detection on Transmission Lines Using UAV Imagery
## 5 Simulation Results and Analysis
### Comparative Experiment
#### 5.3.2 Performance Evaluation
In practical applications, some researchers uses the built-in functions of the software to automatically generate dataset labels. This greatly reduces the time for data preparation, however it affects detection accuracy of the model. We use the ground truth labeler function in MATLAB to label the training sets automatically, and compare the training result with the training result of the model using the manually labeled datasets. In this way, the recognition accuracy of the model under the automatically labeled datasets is verified. The comparison results are shown in Table 4.  
We use the enhanced dataset explained in Section 5.1 as the training set, and train the aforementioned networks. Table 5 presents the performance evaluation of the networks after testing is performed using the same test set. As presented in the results, the proposed method is able to achieve high detection accuracy for bird's nest detection. In addition, the proposed model ROI mining faster RCNN is able to outperform faster RCNN and cascade RCNN in terms of mAP, F1 score and recall. However, precision is slightly lower than cascade RCNN. It is noticeable that the proposed method of ROI mining faster RCNN improves recall rate of the object detection on the basis of classifying complex samples. Therefore, the detection accuracy enhances, while maintaining the precision. In terms of mAP, which reflects the overall performance of the detection network, the proposed method is able to cope with the inherent issues of the original model on the basis of focal loss that is added to faster RCNN. Similarly, the process of balancing the number of foregrounds and backgrounds during the training process improves mAP. The proposed method is able to achieve the mAP score of 82.51%, that proves the effectiveness of the ROI mining method. \begin{table}
\begin{tabular}{c c} \hline \hline
**Labeling Method** & **mAP** \\ \hline Automatic labeling & 0.6637 \\ Manual labeling & 0.8251 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparison of labeling methods. \begin{table}
\begin{tabular}{c c} \hline \hline
**Background** & **mAP** \\ \hline Foreign Object Detection & 0.8011 \\ Bird’s Nest Detection & 0.8251 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Comparison in different backgrounds. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Methods** & **mAP/\%** & **F1/\%** & **Recall/\%** & **Precision/\%** \\ \hline Faster RCNN & 78.29 & 48.43 & 34.52 & 75.54 \\ Faster RCNN with Focal Loss & 78.99 & 48.87 & 35.72 & 75.76 \\ Cascade RCNN & 79.19 & 53.04 & 41.20 & 79.19 \\ ROI Mining Faster RCNN & 82.51 & 54.59 & 43.75 & 76.63 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Detector performance. It is evident from the results presented in this work that accuracy and mAP of ROI mining faster RCNN outperforms the faster RCNN and cascade RCNN methods. Moreover, precision of the proposed method is also comparable. Thus, the detection method proposed in this work greatly improves the process of bird's nest recognition on transmission lines and transmission poles. In addition, the proposed method successfully copes with the class imbalance problem during training and achieves the purpose of automatically and accurately detecting birds' nests in aerial transmission line images.