# Drone vs. Bird Detection: Deep Learning Algorithms and Results from a Grand Challenge
## 4 Gradiant Team
### Experimental Evaluation  
In order to evaluate the performance of the proposed approach, a comparison between three different configurations is provided. The first of them uses a standard Cascade R-CNN architecture, with standard anchor configuration extracted from COCO dataset and has been trained using all previously mentioned data sources. The second model uses custom anchors and has been trained using the Drone vs. Bird challenge data only. Finally, the model submitted to the grand challenge uses personalized anchors and has been trained with all the available data.  
Figure 7: Results for anchor matching.  
All experiment results have been obtained over the previously mentioned validation dataset which consist of 16 videos. Although competition results are evaluated against [EMAIL_ADDRESS], this metric is not useful for evaluating how a model would perform in real world [44]. In such situations, a confidence threshold for discarding low quality predictions and fusing its result with other sensors must be provided. All these facts are not taken into account with AP which suggests the use of other metrics that better align with real applications. In addition, most datasets for drone detection include very few categories (usually one) and often just an object per image. Under these settings, the AP ignores all false positives as long as the prediction with the higher confidence is assigned to the object of interest, as illustrated in Figure 8, where both images obtain the same AP (1.0-1.0) but a huge gap in moLRP (0.0-0.944).  
Figure 9 reports a precision versus recall at IoU score of 0.5 comparison of the three different models. It can be easily observed how default anchors model has the worst performance, which shows the importance of designing a proper anchor set for object detection problems like the one in this challenge. Differences between final submission and the model trained only with a reduced dataset are highlighted when model threshold is increased, whereas the final submission is able to maintain higher recall values.  
A similar comparison is shown in Figure 10 where different IoU values and their corresponding F1-score values are plotted for each of the trained models. Once again,  
Figure 8: Comparison between different thresholds under same image.  
Figure 9: Precision vs. Recall at IoU = 0.5.  
best results are achieved by the submitted model. For larger IoU values the three of them converge to similar scores and this is due to the difficulty of achieving large IoU values for very small ground truth objects, where small differences result in large changes in IoU.  
In Figure 11, a comparison between the number of hits (true positives) and misses (false negatives) is shown. Ground truth bounding boxes are divided in three different categories depending on their scale, so that one can visualize how models perform with respect to target sizes. It is clear that main differences occur for small drone detection, where models trained with our custom anchor set are able to achieve a higher number of hits and at the same time reduce the number of misses. This fact supports the decision made by Gradiant Team of creating specific anchors suitable to match small drones during training and increase the capabilities of the model for detecting them.