kamrulkonok/evalllm2026-french-defense huggingface dataset:
« XLAD 25 » _ Retour sur l’exercice de lutte antidrones mené sur l’aéroport du Bourget _ Ministère des Armées et des Anciens combattants.pdf — page 2

domaine de la lutte antidrones (LAD) après les Jeux olympiques et paralympiques
(JOP) de Paris 2024. En capitalisant sur les enseignements tirés lors des JOP de Paris
2024, cet exercice a permis d’intégrer de nouveaux acteurs (publics et privés) afin
d’échanger sur les évolutions et le renforcement des capacités de LAD. Le but :
monter en puissance dans le domaine, consolider la dynamique interarmées -
interministérielle et enrichir le concept d’emploi de la LAD.
Cet exercice « grandeur nature » a ainsi permis de développer un catalogue de
scénarios thématiques, allant de l’attaque d’un drone unique à l’assaut en essaim,
incluant des confrontations entre assaillants et défenseurs. Divers modes opératoires
ont été éprouvés : neutralisation par brouillage, par capture (fusil filet) ou par
percussion (drone antidrones). Ces scénarios reposaient sur une coordination entre les
différents niveaux de centres d’opérations (CO) LAD : le COLAD 1 (niveau national, en
charge de la surveillance de l’espace aérien), le COLAD 2 (niveau départemental et
préfectoral), et le COLAD 3 (niveau local, aéroport). Plus de 40 scénarios ont été joués
sur ou à proximité de la plateforme aéroportuaire civile, en tenant compte des
contraintes de la réglementation aéronautique. L’exercice a également permis de
tester la chaîne Command and Control (C2) et d’interconnecter des systèmes LAD :
PARADE et BASSALT (AAE), MILAD (armée de Terre), RADIANT (préfecture de Police).
L’exercice « XLAD » a mobilisé près de 230 participants. Parmi eux, 125 provenaient du
ministère des Armées, dont 90 de l’armée de l’Air et de l’Espace. Le ministère de
l’Intérieur était représenté par 58 personnes issues de la Gendarmerie nationale, de la
Gendarmerie des transports aériens, de la Police nationale et de la préfecture de
Police de Paris, et 47 participants étaient issus du secteur privé, notamment du groupe
Aéroports de Paris. Du côté de l’AAE, plusieurs unités étaient engagées : le Centre
d’expériences aériennes militaires (CEAM), l’escadron d’hélicoptères 3/67 « Parisis » et
les escadrons de défense sol-air (EDSA). Ils étaient accompagnés du 54  régiment
d’artillerie, de l’état-major de la zone de défense de Paris et de l’Agence de
l’innovation de défense (AID).
e
« XLAD 25 » : Retour sur l’exercice de lutte antidrones men...
https://www.defense.gouv.fr/air/actualites/xlad-25-retour-l...
2 of 6
2/4/26, 17:13
« XLAD 25 » _ Retour sur l’exercice de lutte antidrones mené sur l’aéroport du Bourget _ Ministère des Armées et des Anciens combattants.pdf — page 3

Retour sur l’exercice de lutte antidrones mené sur l’aéroport du Bourget
Partager la page
Veuillez autoriser le dépôt de cookies pour partager sur Facebook, Linkedin et Twitter-x.
A la une
« XLAD 25 » : Retour sur l’exercice de lutte antidrones men...
https://www.defense.gouv.fr/air/actualites/xlad-25-retour-l...
3 of 6
2/4/26, 17:13
« XLAD 25 » _ Retour sur l’exercice de lutte antidrones mené sur l’aéroport du Bourget _ Ministère des Armées et des Anciens combattants.pdf — page 4

Célébrer nos photographes, témoins de l’engagement de
l’AAE
Chaque 4 février, lors de la Sainte-Véronique, est un prétexte pour mettre en lumière un
métier essentiel m...
04 février 2026
Les Prix AAE et Plumes de l’air et de l’espace : quand réflexion
et plume prennent leur envol
Cette année encore, l’armée de l’Air et de l’Espace (AAE) met en lumière celles et ceux
qui font avancer la...
« XLAD 25 » : Retour sur l’exercice de lutte antidrones men...
https://www.defense.gouv.fr/air/actualites/xlad-25-retour-l...
4 of 6
2/4/26, 17:13
« XLAD 25 » _ Retour sur l’exercice de lutte antidrones mené sur l’aéroport du Bourget _ Ministère des Armées et des Anciens combattants.pdf — page 5

Haut de page
Suivez l'armée de l’Air et de l'Espace sur les réseaux sociaux
02 février 2026
Colonel Sophie Adenot : apprendre à dompter la peur avant
le grand départ
À quelques jours de son envol vers l’espace, l’Aviatrice et astronaute française Sophie
Adenot se prépare à...
30 janvier 2026
MINISTÈRE
DES ARMÉES
« XLAD 25 » : Retour sur l’exercice de lutte antidrones men...
https://www.defense.gouv.fr/air/actualites/xlad-25-retour-l...
5 of 6
2/4/26, 17:13
« XLAD 25 » _ Retour sur l’exercice de lutte antidrones mené sur l’aéroport du Bourget _ Ministère des Armées et des Anciens combattants.pdf — page 6

info.gouv.fr
service-public.gouv.fr
legifrance.gouv.fr
data.gouv.fr
Sauf mention contraire, tous les contenus de ce site sont sous licence etalab-2.0
ET DES ANCIENS
COMBATTANTS
Accessibilité : partiellement conforme
Données personnelles
Gestion des cookies
Mentions légales
« XLAD 25 » : Retour sur l’exercice de lutte antidrones men...
https://www.defense.gouv.fr/air/actualites/xlad-25-retour-l...
6 of 6
2/4/26, 17:13
MP-IST-210-1A.03.pdf — page 3

Assuring Trustworthy Computer Vision
for Rapid Counter-Drone System Testing and Deployment 
measurement framework should assess the accuracy, speed, and reliability of CV algorithms in identifying 
and tracking drones under diverse and challenging conditions, and allow for objective comparisons between 
different systems, enabling informed decisions and fostering continuous improvement in counter-drone 
technology. 
2.1 
Complexity of Performance Measurement in Networked, Inter-dependent Counter-
Drone Systems 
 
Figure 1: ASUL being presented at BAAINBw in Koblenz. Image was released by the 
 
Bundeswehr for public distribution.
Measuring the performance of CV systems for counter-drone missions can be a complex and multi-
faceted subject. Firstly, contained counter-drone systems are themselves becoming significantly more 
complex, comprising a mix of sensors, effectors, and coordination and processing software, in order 
to match the complexity and diversity of the range of threats they are required to face. For example, 
the Bundeswehr announced in Aug 2022 its deployment of the ASUL counter-drone system, highlighting 
its components [1]: “the sensor system… is composed of… three radars and a camera system with day and 
night vision. A radio frequency (RF) direction finder used to identify control signals and, if possible, also 
locate the position of the drone pilot… Once an aircraft has been detected by the radars and the 
direction finder, the system will compare its data with information available from data bases. In such a 
way, the system determines whether the aircraft could be a drone. After an alleged target has been 
detected, the system’s operator will be informed in real time and will be able to verify, with the help 
of the camera system, if the drone is to be classified as a threat.” While the exact specifications and 
workflows of the system are classified, within one counter-drone system, there is already a variety of 
visual data generated through its radar, electro-optical sensor, and thermal sensors. The system will 
likely also be engineered to handle the inherent variability in size, speed, materials, behavior, and 
payloads of various adversary drones of differing signatures, and function across a range of 
environmental conditions including electromagnetic interference and expected weather phenomena 
(sunlight, moonlight, rain, fog, or snow), and have differing levels of tolerance and 
STO-MP-IST-210
1A.03 - 3
MP-IST-210-1A.03.pdf — page 4

redundancy built into its complementary sensor suites. It is likely that all these factors affecting ASUL’s 
success in deployment would have been specified through months of painstaking procurement requirements 
development and evaluation along these dimensions to ensure its ability to meet the Bundeswehr’s counter-
drone defense missions. Table 1 highlights a comparison of the key advantages and limitations of different 
modalities for drone detection, classification, and tracking, showing the complexity of architecting multi-
modal sensors. 
 
Table 1: Key Advantages and Limitations of Different Modalities for Drone Detection, 
 
Classification, and Tracking. 
Modality 
Key Advantages 
Key Limitations 
RF- 
based 
•
High accuracy in controlled
environments
•
Effective
for 
UAV 
identification 
and 
flight 
mode classification 
•
Effective range theoretically
scales to sensor sensitivity
•
Cost-effective and can detect
multiple drones/controllers
•
Susceptible to radio congestion and
jamming
•
Completely 
ineffective 
against
autonomous drones without active RF
links, 
or 
controlled 
drones 
in
autonomous modes
•
Requires prior knowledge 
of 
RF
frequencies in use by potential UAVs,
and other unrelated urban sources of RF
clutter
Audio- 
based 
•
Detects drones based on rotor
noise and acoustic signatures
•
Effective for autonomous
drones
•
Can operate outside line-of-
sight limitations
•
Mobile and quick to deploy
•
Performance degrades significantly in
high-noise or windy environments
•
Typically very short range due to
soundwave propagation characteristics
(300-500m)
•
Short range limits coverage for large
areas
•
Some drones designed to operate
silently
Assuring Trustworthy Computer Vision 
for Rapid Counter-Drone System Testing and Deployment 
1A.03 - 4
STO-MP-IST-210
MP-IST-210-1A.03.pdf — page 5

Radar- 
based 
•
Resilient 
under
low 
visibility/weather conditions  
•
Uses
micro-Doppler 
signatures
•
Long 
range, 
constant 
tracking, accurate positioning  
•
Able to handle many objects
simultaneously
•
Able to detect any flying
object with radar signature
regardless of its operating
modality
•
Higher hardware costs and calibration
demands
•
Many systems lack resolution for very
small drones
•
Active emitting nature of sensors adds
complexity and licensing requirements
•
“Dead zones” due to curvature of the
earth and undulating terrain features
•
Clutter interference in urban/dense
environments
•
Integration 
complexity 
with 
other
sensors
Vision- 
based 
•
Targets UAV recognition via
optical images
•
Provides visuals for forensic
evidence
•
Real-time and cost-effective
solutions
•
Promising 
rapid
enhancement by advances in
AI and CV, and consumer-
grade 
image 
capturing
technologies
•
High false-alarm rates due to 2-
dimensional nature of returns
•
Environmental 
variability 
(light,
weather, obscurants) significantly affect
performance consistency
•
Challenges with small object size and
motion blur
•
High computational demands for high-
resolution streams
•
Relies on line-of-sight, vulnerable to
camouflage and obfuscation
Multi- 
Modal 
Fusion 
•
Mitigates single-sensor blind
spots
•
Significantly 
improves 
robustness and accuracy   
•
Crucial 
for 
small-object 
detection and erratic flight 
•
Adaptive sensor weighting
and 
deep 
embedding
methods
•
Increases 
complexity 
in 
real-time
deployment
•
Requires precise sensor calibration and
synchronization
•
Consumes power and bandwidth for
large-scale deployments
•
System integration complexity and cost,
including data synchronization and
fusion protocol standardization
•
Requires 
significant 
computational
resources or hardware optimisation for
real-time processing on edge devices
Second, in networked warfighting, each of these systems are likely to be embedded into larger systems-of-
systems operating across the tactical, operational and theatre level. This is because each sensor modality has 
inherent limitations as highlighted in Table 1: for example, radar may struggle with small, low-flying 
drones, optical cameras can confuse drones with birds, and RF direction finders are ineffective against 
Assuring Trustworthy Computer Vision
for Rapid Counter-Drone System Testing and Deployment 
STO-MP-IST-210
1A.03 - 5
MP-IST-210-1A.03.pdf — page 7

radio frequency (RF), audio, Light Detection and Ranging (LiDAR) data [5]: “Foundational benchmarks 
such as FL-Drones, NPS-Drones, and USC Drone provided core RGB datasets for aerial detection and 
tracking. Beyond the visual domain, DroneRF captures electromagnetic signatures for RF-based UAV 
classification, while DroneAudio and Acoustic-UAV focus on acoustic features. More recent efforts extend 
coverage to multi-modal data: Halmstad Drone fuses RGB, IR, and audio, and MMAUD integrates radar and 
LiDAR. Community-driven competitions (e.g., the 2nd and 3rd Anti-UAV Workshops and the 4th Anti-
UAV Challenge) feature adversarial or cluttered scenarios that reflect real world operational complexity, 
while UAVSwarm and mDrone emphasize large-scale swarm coordination or low-visibility tracking—both 
highlighting the need for robust multi-sensor fusion.” 
However, public datasets often suffer from several issues. One, the current state of benchmarking, while 
foundational, is characterized by critical deficiencies, particularly concerning annotation standards. Without 
discrediting the hard work and innovative approaches of their originators, many of these datasets are 
developed and maintained on a voluntary basis by researchers and graduate students, data quality can vary 
significantly. Moreover, many such large datasets were created with the use of popular foundational CV 
models such as YOLO, creating a feedback loop of poor open-source training data, leading to suboptimal 
models, leading to substandard labelling which finds its way back into poor open-source training data. 
Although no firm statistics are available on the prevalence of these data inconsistencies, Koksal et al noted in 
2020 at the 1st Anti-UAV Workshop and Challenge [9] that “such approaches might result in erroneous 
annotations”, and found in some experiments that such “annotation errors degrades the performance 
[severely]..., especially for missing boxes”, proposing to correct the benchmark challenge’s annotations 
where they found mistakes. The mislabelling that occurs often erodes trust in the dataset and contributes to 
varying levels of measurement error when models are tested on them. See Figure 2. 
 
Figure 2. Examples of mislabelled data in public UAV-related benchmarks. Top left: label applied 
 
where there is no UAV in the image, Anti-UAV300 dataset. Top right: label applied to a part of the 
 
image in which the UAV is not (actual UAV is on the left of the bounding box), Anti-UAV300 
 
dataset. Bottom two: UAVs present but unlabelled within image, DUT Anti-UAV Detection and 
 
Tracking dataset. Inconsistencies discovered by Resaro Research, 7 Jul 2025. 
Two, public datasets may not accurately reflect real-world operational environments, leading to a 
dataset bias where models perform well on the benchmark but poorly in deployed scenarios. For instance, 
Assuring Trustworthy Computer Vision
for Rapid Counter-Drone System Testing and Deployment 
STO-MP-IST-210
1A.03 - 7
MP-IST-210-1A.03.pdf — page 8

in some of the foundational public datasets mentioned above that rely on often openly available data
scraped from social media or search engines, there is a significant mismatch in image context as compared 
to common use cases found in defense and security, as these tend to highlight drone flying or 
experimentation in recreational or civilian contexts. Moreover, when the sensor type and specifications 
are considered, further issues may arise relating to domain adaptation as model performance tested on 
these benchmarks may not translate into similar performance in defense and security contexts, with 
military-grade sensors. See Figures 3 and 4 for examples of the difference in data relevance and quality 
across these benchmarks. 
Figure 3: Examples of images from the USC Drone dataset. Note the high quality, good focus 
and good contrast present in some of the optical images which suggest that they may have 
been captured either by modern smartphones or photography enthusiasts from a stabilized 
imaging platform. 
Three, these benchmarks are often static and may not evolve at the same pace as the rapid advancements 
in computer vision or the emergence of new threats. As new and customized drones and tactics emerge 
on the battlefields of Ukraine and elsewhere, there is typically a significant lag experienced before their 
inclusion into public benchmarking datasets. In 2025, this implies that datasets specifically do not 
contain sufficient 
Assuring Trustworthy Computer Vision 
for Rapid Counter-Drone System Testing and Deployment 
1A.03 - 8
STO-MP-IST-210
Figure 4: Examples of imagery from the 4th Anti-UAV Workshop dataset, which are examples of 
datasets more likely to be reflective of real-world scenarios encountered by sensors in defense 
and security. However, note also the sensor artifacts present (Chinese characters, crosshair 
boxes, date and timestamps); these artifacts may differ greatly across sensor types, especially 
those that do not operate directly on raw sensor feeds, and may corrupt measures of model 
performance in testing.
MP-IST-210-1A.03.pdf — page 13

to calculate the intersection-over-union of the areas generated by a CV model predicting where the target is, 
compared to its ground truth. Taken in aggregate, these parameters aim to provide a positive rather than 
normative perspective of the CV system’s expected performance, to inform decision-makers deciding to 
procure or operate these systems against real-world adversaries. For a more granular view, the component 
parameters of Detection Accuracy, Association Accuracy, and Localization Accuracy can also be retrieved 
to understand where each model’s strengths and weaknesses are in tracking, as the developers of these 
models would also focus on different methods of fine tuning or hyperparameter optimization focusing on 
maximizing performance in certain desired aspects. As the creator of the HOTA Thomas Luiten noted, “very 
similar overall HOTA scores… [are revealed by] this plot…. that there is a clear difference in where one is 
better than the other… if you wanted to select a tracker for a particular application, you could now decide 
whether association or detection was more important for your application and pick the most appropriate 
tracker accordingly”. [11]  
Table 2. Testing a CV system for detection, classification, and tracking of drones. 
Use Case: Automated detection, classification, and tracking of hostile drones 
Category 
Technical Indicator 
Test Protocol and Metric 
Detection 
& 
Classification 
Performance 
Tendency of CV model to produce false alarms, e.g. 
producing detection reports of drones when birds fly 
across the field of view 
False Alarm Rate 
Tendency of CV model to miss detection of actual 
drone within field of view 
Missed Detection Rate 
Extent of ability of CV model to pick up presence 
of potential hostile drone upon capture in sensor 
field of view 
Detection Speed 
Extent of ability of CV model to correctly classify 
type of potentially hostile drone upon capture in 
sensor field of view 
Classification Speed 
Extent of ability of the CV model to detect and 
classify hostile drone of interest in general, across 
different confidence levels and object classes 
Overall Accuracy and Precision 
Tracking 
Performance 
Extent of ability of CV model to ‘track’ objects 
moving across video clips in real-world use cases, 
blending aspects of detection, association and 
localization accuracy within a single view 
Overall Tracking Accuracy 
Extent of ability of CV model to find objects within 
single images, independent of their identity or 
movement across frames 
Detection Accuracy 
Extent of ability of CV model to maintain the 
correct identity of an object across a sequence of 
frames, as opposed to confusion with other objects 
Association Accuracy 
Assuring Trustworthy Computer Vision
for Rapid Counter-Drone System Testing and Deployment 
STO-MP-IST-210
1A.03 - 13
MP-IST-210-1A.03.pdf — page 14

Extent of ability of CV model to consistently 
localize and track a target across multiple frames 
through aligning a predicted bounding box or 
tracker crosshair with its actual position 
Localization Accuracy 
Against more fine-grained contextual qualities such as: 
System-to-target 
fit 
Class-based performance differentials of above 
performance metrics, across differing drone sizes, to 
ensure CV model tested will perform evenly across 
drone types.  
Drone size tolerance 
Small (l x w x h < 2m3) 
Medium drones (2m3 < l x w x h < 
10m3) 
Large drones  (l x w x h > 10m3) 
Drone type tolerance 
Fixed-wing 
Rotary-wing 
Other 
Drone quantity tolerance 
Single 
Multiple/swarm 
Robustness 
to 
environmental 
factors 
Performance differentials of above performance 
metrics, under varying environmental conditions, 
that would be expected in the deployment 
environment of interest, to stress-test CV models 
prior to actual deployment in said environments 
Indoors, enclosed 
Partial occlusion 
Weather tolerance 
Rain 
Ambient light 
Fog & Mist 
Snow 
Backdrop tolerance 
Against cloud/sky 
Against urban  
Against foliage 
Sensor Behavior 
Tolerance 
Performance differentials of above performance 
metrics, across scenarios with varying sensor 
system behaviors, characteristics, and motion, to 
stress-test CV models prior to actual deployment in 
said scenarios 
Sensor jitter 
Sensor frame drops 
Image noise 
Sensor artifacts 
Assuring Trustworthy Computer Vision 
for Rapid Counter-Drone System Testing and Deployment 
1A.03 - 14
STO-MP-IST-210
MP-IST-210-1A.03.pdf — page 17

System-to-Target Fit 
Drone Size 
Small (l x w x h < 2m3), Medium (l x w x h < 10m3) 
Drone Quantity 
Single, Multiple 
Sensor Behavior 
Image Noise 
Low, High 
Environmental 
Robustness 
Ambient Light 
Bright, Dark 
Backdrop 
Sky, Urban, Foliage 
The possible values each category can take in Table 3 are arbitrarily defined within this hypothetical scenario 
but should be adapted to the possible range of operational conditions in the desired deployment environment 
when this framework is employed in a real-world scenario. If we were to examine the robustness of the CV 
system under test individually under each parameter, and only produce results for each row, we may be 
unable to pick up possible blind spots on images containing more than one parameter of interest, e.g. images 
containing multiple small drones with high noise and dark ambient light. However, if we were to use every 
possible combination of parameters in testing, we would end up with a very large set of test cases which 
would be unlikely to be fulfilled by our collected dataset, and computationally prohibitive to generate 
otherwise. Hence, to simplify the resulting conversion of the above eleven test levels for the 5 parameters 
into test cases, we use pairwise subsets of the parameters, e.g. image noise vs. ambient light, to derive user 
friendly test tables. All possible parameter combinations would lead here to prod_{k = 1}^{d} n_d 
combinations, here 3* 2^4 = 48. 
We further assess that for some of the combinations involving medium drones, this is likely to be less of a 
concern given our scenario parameters as we are generally aware that medium-sized drones would likely 
tend to operate at a higher altitude and be used in medium-range strikes, unlike small hunter-killer FPV 
drones which might heavily utilize terrain masking to camouflage their movements. This is a frequently 
occurring problem, that not all theoretical possible combinations of the parameters are practically relevant 
and is handled by introducing constraints to the space of all parameter combinations. Hence, we drop the 
following combinations from consideration with the below considerations: 
Tuple 
Consideration for Exclusion from Test Case 
(Medium, Multiple) 
The evacuated city in our hypothetical scenario is not assessed to be 
likely to be significant enough to be targeted by large swarms of enemy 
medium-sized drones or loitering munitions. 
(Medium, Urban) 
Medium-sized drones would likely have higher operating altitudes in a 
strike mission, and the field-of-view coverage of our hypothetical 
sensor setup would not provide instances in which medium drones 
would be against urban or foliage scenery. 
(Medium, Foliage) 
Dropping combinations involving the above factors from consideration produces the following combinatorial 
coverage table of 27 combinations (Table 4). In order to provide some reasoning around reasonable sample 
Assuring Trustworthy Computer Vision
for Rapid Counter-Drone System Testing and Deployment 
STO-MP-IST-210
1A.03 - 17
MP-IST-210-1A.03.pdf — page 18

sizes, hypothetical confidence intervals are added to the combinatorial table based on the available videos. In 
our reasoning, we construct a binomial data distribution for the object detection system, i.e. it either detected 
everything in an image/video correctly or it did not and further assume a perfect result reflecting that the 
object detection system did achieve its main objective in the cases we observed. We can then calculate a 
confidence interval for the pass rate, which reflects the uncertainty for similar performance to be observed in 
new videos from the same combination of parameters, even if empirically a system did perform perfectly for 
the known examples. This uncertainty is statistically reflected by the width of the confidence interval, which 
we generate using Wilson’s method for a one-sided confidence test for a 95% threshold, as implemented in 
the popular data science library scipy through the method BinomTestResult.proportion_ci. Examining the 
confidence interval (ci_low) in each scenario gives us an indication of where the test data may need to be 
further augmented, especially if certain scenarios are of higher mission importance. 
Table 4. Combinatorial coverage showing the number of observations (n_observations) and 
resulting confidence interval (ci_low) given the data available in the visual segment. Note that IR 
data was not included in this calculation as it does not contain context on the background and
considerations involving ambient light do not apply. Nonetheless, most of the IR data contains 
single small drones against a ‘sky’ backdrop.   
drone_size drone_quantity noise 
light 
background 
n_observed 
ci_low 
small 
single 
low 
bright 
urban 
20 
0.88 
small 
single 
low 
bright 
sky 
10 
0.79 
small 
single 
low 
bright 
foliage 
18 
0.87 
small 
single 
low 
dark 
urban 
5 
0.65 
small 
single 
low 
dark 
sky 
2 
0.43 
small 
single 
low 
dark 
foliage 
2 
0.43 
small 
single 
high 
bright 
urban 
107 
0.98 
small 
single 
high 
bright 
sky 
78 
0.97 
small 
single 
high 
bright 
foliage 
103 
0.97 
small 
single 
high 
dark 
urban 
43 
0.94 
small 
single 
high 
dark 
sky 
62 
0.96 
small 
single 
high 
dark 
foliage 
0 
0 
small 
multiple 
low 
bright 
urban 
2 
0.43 
small 
multiple 
low 
bright 
sky 
3 
0.53 
small 
multiple 
low 
bright 
foliage 
3 
0.53 
small 
multiple 
low 
dark 
urban 
0 
0 
small 
multiple 
low 
dark 
sky 
2 
0.43 
small 
multiple 
low 
dark 
foliage 
1 
0.27 
small 
multiple 
high 
bright 
urban 
0 
0 
small 
multiple 
high 
bright 
sky 
2 
0.43 
Assuring Trustworthy Computer Vision 
for Rapid Counter-Drone System Testing and Deployment 
1A.03 - 18
STO-MP-IST-210
MP-IST-210-1A.03.pdf — page 20

The second pairwise combination to consider arises from the usage of the (object_size, quantity) tuple, 
which attempts to answer the potential question “will our CV system perform consistently in detecting 
different numbers and quantities of drones?” Given the number of observations present in Table 6, we see 
that we have generally sufficient coverage to answer this question, with resulting confidence intervals 
ranging from 0.85 to 0.99, except for the scenario with multiple medium drones (with only 1 observation and 
a confidence interval of 0.27). However, in this case, we have already excluded such a consideration from 
the testing scope, as “the evacuated city in our hypothetical scenario is not assessed to be likely to be 
significant enough to be targeted by large swarms of enemy medium-sized drones or loitering munitions.” 
Hence, we will focus the efforts of simulating multiple small drones in the following simulation runs. 
Table 6. Pairwise coverage for (drone size, drone quantity) tuple.
We made the conscious decision to exclude the last factor, image noise, in considering pairwise coverage as 
the nature of noise exhibits significantly more continuous variability. Nonetheless, we will also address the 
effect of noise on model performance at a later stage, where we will holistically investigate the effect of 
noise across all 3 datasets to understand how it influences performance. 
3.2.2 
Establishing Simulation Environments for Data Collection & Dynamic Testing 
While a fixed set of still images and video data may be useful to establish a baseline understanding of CV 
system performance when given these inputs, clear limitations persist. These include the inability to: (a) 
evaluate counter-drone systems on dynamic adversary drone maneuvers and scenarios (e.g. multi-drone 
attack formations), (b) account for the full range and representation of realistic environmental phenomena in 
which they are expected to operate, where data is insufficient, (c) reproduce the idiosyncratic sensor 
behaviors where real data may not exist for sensors yet to be deployed, and (d) fully appreciate the terrain 
limitations of each sensor mode, especially including line-of-sight issues for optical cameras or terrain-
masking issues for IR or LIDAR sensors. Moreover, where the dataset distribution is not sufficient to support 
a representative weightage of the data for the actual use in the operational design domain, performance 
metrics on the dataset cannot be considered transferable to the actual usage scenario. For example, a drone 
dataset consisting of 80% of bright light, clear sky, no occlusion images/videos would not provide sufficient 
assurance of consistent CV system performance under an actual usage scenario in a winter setting, with 60% 
rainy conditions. 
Assuring Trustworthy Computer Vision 
for Rapid Counter-Drone System Testing and Deployment 
1A.03 - 20
STO-MP-IST-210
Table 5. Pairwise coverage for (background, light) tuple.
MP-IST-210-1A.03.pdf — page 21

Hence, we created the third and final part of the benchmark dataset methodology, intended to generate the 
test cases deemed to be necessary but lacking in the above coverage analysis for any given curated dataset. 
This dataset will be created using a realistic simulation platform that will recreate the expected real-world 
operating environment, given a set of parameters relevant to the tester, and if necessary, with a reproduction 
of the relevant sensor in the actual simulation environment based on its technical specifications. For instance, 
as the above test coverage analysis revealed a need to focus on scenarios involving dark foliage 
environments, as well as multiple small drones, concrete simulation scenarios will be established with these 
parameters, and the ground truth and sensor outputs of these scenarios recorded into still images and videos 
as a test case. See Figure 5 for some examples. 
Another important advantage of the usage of a simulation to enhance and enlarge datasets is that  even 
after the conclusion of a test, iteratively new simulation runs can be added to search for points of 
weak performance for a system, allowing very specific criteria can be derived to identify and 
investigate edge cases, e.g. when an object detection system experiences catastrophic model failure due to 
partial occlusion of the target object by foliage. 
Assuring Trustworthy Computer Vision
for Rapid Counter-Drone System Testing and Deployment 
STO-MP-IST-210
1A.03 - 21
Figure 5. Some simulation scenarios focusing on a combination of darker 
environments, multiple small drones, and accompanying foliage as parameterized 
scenario variables. An advantage of generating data is the ability to generate 
ground-truth bounding boxes within images, economizing on labelling effort.
MP-IST-210-1A.03.pdf — page 23

structuring raw sensor streams, on-board drone information (e.g., object detection outputs, estimated 
localization), and crucial ground truth data (e.g., precise object bounding boxes, ground drone position and 
odometry data), and relating them to preset pipelines for calculating metrics, and data queries. This enables 
detailed, objective comparisons to quantify system performance. The pipeline is designed to also handle real-
world data in place of simulation data streams, helping to bridge the "simulation-to-real" gap in cases where 
real-world sensor streams are available. 
3.2.3 
Visualization and Analysis 
Visualization and analysis involve a powerful engine that evaluates various performance metrics, including 
mission-specific criteria, object detection/tracking accuracy, localization accuracy, and maneuver precision. 
It translates these technical metrics into standardized AI System Quality Indices (ASQI), providing 
comparable and interpretable grades across different capability categories.  
In order to be able to do different types of analysis on the data generated as outlined in section 3.2.1, e.g. 
coverage analysis, all relevant metadata is also collected, which can include drone positions, headings, 
movement speed, as well as data on ambient conditions such as fog or rain settings. Results are presented on 
a front-end dashboard for clear decision-making but are also available as data artefacts for individual data 
analysis projects. 
3.2.4 
System Architecture, Result Storage and Retrieval 
The simulation platform is designed with a flexible architecture supporting both air-gapped on-premises 
deployment for sensitive projects, and cloud-based parallel processing for high-throughput analysis, as well 
as single-compute-instance use cases, and parallelized use cases for high throughput. Results are stored in a 
tamper-proof environment together with extensive documentation artefacts detailing the conditions and 
parameters of test scenarios, in relation to the data generated for each case. 
4.0 CONCLUSIONS 
4.1 
Dataset Review and Analysis of Proxy Model Results 
Having created 3 segments of our aggregated dataset, we now turn to evaluating the data and re-assessing the 
combinatorial coverage to ensure that our test methodology will deliver us sufficient visibility into the 
characteristics of the hypothetical CV system being tested.  
4.1.1 
Analysis of Test Data Quality and Distribution 
First, from the real-world data collected, we find a total number of 1269 videos, of which 766 are from IR 
sensors and 503 are from visual sensors. In particular, as we were aware from previous experience that the 
available open-source datasets referenced above lack significant representation of medium and large drones, 
extra effort had been put into the data collection process to manually acquire data relating to these drones by 
scraping popular websites and Telegram channels dedicated to coverage of drone strikes in the ongoing 
Russia-Ukraine conflict and Israel-Iran conflict. 
Assuring Trustworthy Computer Vision
for Rapid Counter-Drone System Testing and Deployment 
STO-MP-IST-210
1A.03 - 23
MP-IST-210-1A.03.pdf — page 24

From the above, we find the dataset demonstrates a strong bias toward scenes involving small 
drones, particularly in the visual data segment. This overrepresentation leads to uneven coverage 
of other operationally relevant targets especially for defense use cases, where the system’s 
performance against medium-range threats such as loitering munitions and Class 2 attack UAVs may be 
relevant. The resulting imbalance may introduce dataset-driven inductive biases, potentially causing 
learned models to overfit to high-contrast or uncluttered settings, while underperforming in more 
complex real-world environments. 
Furthermore, within the data from electro-optical sensors (Figure 7), low-light (‘dark’) scenarios are 
notably underrepresented in the data and when present, are predominantly paired with small drones. 
The visual segment is overwhelmingly dominated by bright scenes, with bright images appearing across 
nearly all size and background categories. In contrast, dark scenes are both fewer in number and 
narrowly distributed, mostly involving small drones. Combinations involving medium-sized drones under 
dark lighting conditions are extremely sparse. This imbalance limits the evaluation of system robustness 
across illumination domains and may compromise performance generalization to real-world conditions 
that involve variable or degraded lighting, limiting the dataset’s ability to support evaluation of 
performance under low-light conditions. This would negatively impact confidence in the deployed CV 
system in the operational scenario discussed above if not addressed, as it opens operators up to unexpected 
risks that their CV system may malfunction and fail 
Assuring Trustworthy Computer Vision 
for Rapid Counter-Drone System Testing and Deployment 
1A.03 - 24
STO-MP-IST-210
Figure 6. High-level analysis of representations in the real-world infra-red (IR) and electro-
optical/visual (EO) Dataset. 
Figure 7. Distribution of dataset representation in light condition (left) and drone quantity (right) 
in the electro-optical visual dataset.
MP-IST-210-1A.03.pdf — page 26

4.1.2 
Performance Differences and Insights across Data Segments for Drone Detection 
Algorithm  
To highlight how such a benchmark would enable a granular and robust investigation of even out-of-the-box, 
non-fine-tuned models, we employ the methods described above to examine if we are able to find the 
potential areas of underperformance of an Ultralytics YOLOE (Real-time Seeing Anything) model 
developed by Ao Wang, Lihao Liu, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding. [18] YOLOE 
was designed for open-vocabulary detection and segmentation, and uses text, image, or internal vocabulary 
prompts, enabling real-time detection of any object class. It was selected as its developers were able to 
‘achieve state-of-the-art zero-shot performance with minimal impact on speed and accuracy’ and is likely to 
be archetypal of the architectures used in real-world counter-drone CV system development.  
We achieve the following overall results that show an overall mAP at 0.22 at the IoU threshold of 0.5 for the 
EO and augmented simulator dataset, and a score of 0.04 for the IR dataset. Further investigation into the 
metric split by bounding box areas shows that the model performs much better when the target within the 
image is large (at least 96 x 96 pixels), and the model has almost negligible performance on small targets (32 
x 32 pixels or smaller). Though unsurprising, this validates our hypothesis that there would be performance 
differences in the CV system across small and medium drones, as well as how far the drones would be 
detected away from the sensor, as further objects will appear as smaller given the same sensor field of view. 
Notably, as Wang et al. noted that YOLOE was largely trained on large COCO datasets and further grounded 
by Objects365 (V1) and GoldG datasets, we can infer that the model was not significantly exposed to infra-
red data. This hypothesis is supported by the below metrics showing negligible performance on the infra-red 
segment of our evaluation data. 
 
Table 8. Overall Accuracy & Precision Results for EO and simulator (left) dataset, and IR dataset 
 
(right). 
Moreover, observing the precision and recall statistics for each dataset reveals a difference in performance 
Classification Accuracy and Precision 
Area size (px) 
(EO) MAP@0.5 
(IR) MAP@0.5 
All 
0.22 
0.04 
Small (area < 322) 
0.01 
0.05 
Medium (322 < area < 962) 
0.28 
0.07 
Large (area < 962) 
0.55 
0.00 
Assuring Trustworthy Computer Vision 
for Rapid Counter-Drone System Testing and Deployment 
1A.03 - 26
STO-MP-IST-210
Table 7. Improved confidence in test case coverage after augmentation by simulation outputs.
MP-IST-210-1A.03.pdf — page 27

for the CV systems operating on EO and IR data (see Table 9). On the EO dataset, we observe a Missed 
Detection Rate of 63.8%, and a False Alarm Rate of 83.09%. On the IR dataset, we observe a Missed 
Detection Rate of 92.73%, and a False Alarm Rate of 65.65%. This shows that the CV system under test 
does provide a significantly lower false alarm rate on IR data, but at the cost of missing almost all the drones 
in the images. Again, this is not surprising given that open-source ‘world’ models tend to be optimized for 
full-color images, as described in earlier sections, therefore leading to significant performance degradation 
when applied to IR use cases even when fine-tuning is applied. Notwithstanding that, note we did not fine-
tune the model in this case. 
 
Table 9. Missed Detection Rate and False Alarm Rate Results for EO and IR datasets (without 
 
IoU threshold). 
GT count (# 
of drones) 
Missed (# of 
drones) 
Predicted 
(# 
of drones) 
Missed 
Detection 
Rate (%) 
False 
Alarm 
Rate (%) 
EO 
dataset 
489 
312 
988 
63.80 
83.09 
IR dataset 619 
574 
131 
92.73 
65.65 
4.1.3 
Insights into performance differences across scenarios 
Further, we also observe clear performance differences for the tested model operating under ideal conditions 
as compared to more challenging conditions, as we may observe further degradation in its drone detection 
capabilities in the presence of compression noise, differing brightness conditions, and detection of 
single/multiple drones as detailed in our test cases above. We further investigate its performance within 
different levels of these categories and arrive at the following results. 
4.1.3.1 
Performance differences between test cases of different background-light combinations 
From examining the data in Table 10, scenarios containing generally dark environments cause the CV 
system to perform poorly, with the lowest mAPs as a group ranging from 0.02 to 0.03 (sky/urban/foliage, 
dark), and at a generally high confidence level. Of the three environments, the foliage background also 
produces the worst results, with mAP ranging from 0.02 - 0.03. Notably, the CV system seemed to perform 
its best under (sky, bright), (urban, bright) scenarios, with mAP at 0.63 and 0.51 respectively with a 
confidence interval of 98% and above. This suggests that the CV system’s ability to perform under these 
conditions, even in the real-world, would be expected to be rather stable under ideal conditions, but may be 
prone to sudden and significant performance degradation should these conditions change (e.g. in an 
incoming storm, when clouds rapidly form and crowd out sunlight). A professional evaluation of such a 
system in our hypothetical scenario would consider the weather patterns, overall lighting conditions in 
various seasons within a temperature climate and design the appropriate safeguards and risk mitigation 
measures accordingly. 
Assuring Trustworthy Computer Vision
for Rapid Counter-Drone System Testing and Deployment 
STO-MP-IST-210
1A.03 - 27
MP-IST-210-1A.03.pdf — page 29

Table 12. Relative results across different combinations of drone size and drone quantity factors 
across visual dataset. 
4.1.3.3 
Impact of image noise on performance 
In measuring the impact of image noise on performance, we first discounted the IR dataset as its overall 
performance was way too low to obtain meaningful results from analyzing the sub-dataset differences in 
noise profile. For the EO dataset, we assigned each ground-truth object a noise score based on high-
frequency variance: variance(patch - GaussianBlur(patch)). This quantifies texture and graininess in the 
object region. Based on this score, objects are classified into the following categories: 
Low: Variance < 100.0 – smooth, low-noise regions. 
Medium: 100.0 ≤ Variance < 400.0 – moderate noise or texture. 
High: Variance ≥ 400.0 – strong texture or high noise presence. 
We then assess the standard detection metrics (Recall, Precision, F1, Mean IoU), along with the number of 
Low, Medium, and High noise objects per class. Classes are ranked by the number of ground-truth objects 
(NumGT), limited to the top-k if specified. For each chart, we compare performance between low-
noise/medium-noise/base-noise objects and their non-low counterparts. Metrics were computed using the 
same set of predictions. It is clear that there is significant drop-off in performance when the level of noise in 
an image increases, to the extent that in this case high noise completely leads to null scores across all 
metrics. 
Figure 9. Missed Detection Rate and False Alarm Rate Results for EO dataset (without IoU 
threshold). 
4.1.3.4 
Validation of framework and understanding follow-ups for defense and security users 
We see from the above analysis that significant performance differences can exist, and be discovered, within 
different segments of the data for an open-source, general foundation model. It is important to note that a CV 
system developer is likely to further fine-tune such foundation models with domain-specific datasets to 
overcome performance issues on each of these various edge cases. However, such fine-tuning efforts come at 
a cost and are likely to lead to trade-offs in other segments of performance, for which visibility is important 
for defense and security teams evaluating their performance, and determining if these trade-offs would be 
acceptable given certain mission parameters and requirements. In the case of the drastic performance gap in 
Assuring Trustworthy Computer Vision
for Rapid Counter-Drone System Testing and Deployment 
STO-MP-IST-210
1A.03 - 29
MP-IST-210-1A.03.pdf — page 31

the framework in finding and identifying fine-grained CV system underperformance. Future projects could 
focus on discovering and analyzing the potential underperformance of CV systems optimized for IR 
imagery, to explore the potentially interesting idiosyncrasies of processing infra-red imagery especially 
within military contexts, such as performance under cases of thermal shadow or thermal crossover. 
Furthermore, the above ASQI and test metric mapping, alongside the simulation platforms, are extensible to 
other multi-modal solutions beyond visual and IR datasets in CV systems. Potential areas of further 
extension with such a framework include extending it to thermal and radar systems (outside of where this 
framework is already potentially applicable on processed and generated images) evaluating audio AI-enabled 
sensor systems, or evaluating unmanned ground vehicle localization and mapping capabilities, as well as the 
development of a NATO-wide benchmark for certain common use cases (e.g. performance of CV systems 
classifying known adversary inventory). 
In future work, we invite those who would seek to investigate the deeper workings of their CV systems 
based on the described use cases, and others, to contact us and attempt our methods on their models, and 
further validate and develop how our framework would be useful for their needs. Beyond drone detection 
and tracking, we look forward to deepening our partnerships with the broader NATO community of like-
minded partners to discover more use cases, and contribute to the defense of our shared values and interests. 
Assuring Trustworthy Computer Vision
for Rapid Counter-Drone System Testing and Deployment 
STO-MP-IST-210
1A.03 - 31