# Drone vs. Bird Detection: Deep Learning Algorithms and Results from a Grand Challenge
## 7 Performance Comparison  
This section discusses the results using the final challenge test set, in order to compare the approaches from the different teams. The Drone vs. Bird Detection Challenge 2020 allowed up to three submissions for each participating team. Submission results for the top three teams are given in Table 4. The evaluation relies on the popular AP metric for object detection. Detections are counted as correct, when its intersection over union (IoU) with a ground truth box is above 0.5. The final resulting scores are averaged across all 14 sequences of the test set. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Submission** & **Alexis [\%]** & **Eagledrone [\%]** & **Gradient [\%]** \\ \hline Run 1 & 79.8 & 66.8 & **80.0** \\ Run 2 & 79.4 & ✗ & 72.9 \\ Run 3 & ✗ & ✗ & 75.2 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparison of detection results of the Drone vs. Bird Detection Challenge 2020. Each team was allowed up to three submissions. \begin{table}
\begin{tabular}{c c} \hline \hline
**Sequence** & **AP (\%)** \\ \hline Known & 93.55 \(\pm\) 0.6 \\ Unknown & 66.0 \(\pm\) 13.5 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Results (AP) of Alexis team approach when testing on known and unknown sequences sampled from the 2020 Drone vs. Birds Challenge. The mean and standard deviation for 5 different data splits (1 run per data split) was reported. Team Gradiant obtained the best overall performance of 80% with Cascade R-CNN [42] applied on 720 pixels crops using a sliding windows approach. In run 1, they personalized the anchors and trained on additional datasets. In run 2, they trained with the standard anchors while still training with additional data and in run 3, they used their personalized anchors, but only trained with the data from this challenge. Team Alexis finished in second place. They used YOLOv3 [39] applied on image crops of 480 pixels. In run 1, they used additional synthetic data during training whereas in run 2, they only used the training data of the challenge. Team Eagledrone finished in 3 rd place by using Faster-RCNN [28]. Due to limitations, they only trained their model on a subset of 3000 images. Tables 5 and 6 list the resulting scores of each teams' best submission in more detail for each of the test sequences separately. In addition, precision-recall curves given in Figure 22 are used to visualize the resulting scores of each teams' best submission for each of the test sequences. Table 5 clearly shows that there is a range in difficulty among the given test sequences. Some sequences (e.g., dji_matrice_210_midrange_cruise) can be solved almost perfectly, due to a relatively large and clearly visible drone in the sequence. Other sequences, however, are much more difficult to handle for all approaches. This is particularly the case for the latter two sequences that are either recorded by a moving camera or at twilight. But also sequences that contain segments with very distant drones (e.g., GOP847_001) can be challenging to the evaluated methods. Furthermore, the results show that the submitted approaches actually perform somewhat complementary. On several sequences the performances vary among the three approaches yet each approach achieves the best overall score on some of the sequences. A combination of the design aspects of the three methods thus looks to be a promising direction for future work. Further insight into the resulting scores can be gained from Table 6, which compares the number of present ground truth objects to the number of submitted detections, resulting recall and implicitly the number of false detections. It can be observed that the Eagledrone team generated much fewer detections, resulting in fewer false positives but also negatively impacting the overall recall and resulting AP. Conversely, the other two teams' results exhibit very large numbers of detections, and thus many false positives, for some of the test sequences. Naturally, such large numbers of detections result in better recall.