MP-IST-210-1A.03.pdf — page 16

Lancet) and medium (HESA Shahed 136 / Geran-2), while large drones (e.g. Kronshtadt Orion) will be 
detected and tracked by other radars within a layered air defense system. This system based on a stream of 
visual imagery from electro-optical sensors positioned at various locations will need to operate in low-light 
and bright conditions but will rely on other complementary radar and acoustic systems on dark nights, when 
visibility is low. We intend for this counter-drone CV system to not be actively monitored, but to issue a 
warning to on-duty operators when potentially hostile drones are detected. In so doing, we note the need to 
minimize false alarm rates to avoid operator fatigue, finely balanced against missed detection rates to ensure 
that hostile drones are not allowed through the ‘picket line’ and threaten valuable military assets in the rear. 
Hence, we select the following aspects of the above ASQI to set up a combinatorial testing approach, with 
the intention of evaluating the counter-drone CV system on the basis of false alarm rate, missed detection 
rates, and classification accuracy and precision, and answer the finer questions asked by our leadership: “will 
our potential vendors’ models perform consistently in all-weather, all-terrain scenarios?”, and “will our CV 
system perform consistently when different numbers and quantities of drones are encountered?”  The 
mathematical formulas used in the metrics for each of the above are as follows. 
3.2.1.2 
False Alarm Rate  
Here we find the number of instances of false positives (FP) where there was a drone detection within the 
image, where there was actually no drone. This rate is expressed as a percentage of the total number of 
positive instances where drones were detected, including correctly detected instances, or true positives (TP) 
through the formula 
. As the commonly used metric for Precision is 
, the False Alarm Rate can also be directly derived from subtracting the Precision metric 
from 1, or 
. 
3.2.1.2 
Missed Detection Rate  
Here we find the number of instances of false negatives (FN) where there was no drone detection within the 
image, when there was actually a drone present. This rate is expressed as a percentage of the total number of 
drones within all images, including correctly detected instances, or true positives (TP) through the formula 
. . As the commonly used metric for Recall is 
, the 
Missed Detection Rate can also be directly derived from subtracting the Recall metric from 1, or 
. 
3.2.1.3 
Overall Accuracy and Precision 
For this aspect of quality we use the mean average precision metric, or mAP, which provides an blended 
view of average precision (AP) across multiple classes i (in this case, small and medium-sized drones) 
through the formula 
, where AP is calculated by a summation of the area under the 
precision(P)-recall(R) curve for each class through the formula 
. Specifically for 
this project, we define the 
, meaning that for predicted bounding boxes that had at least a 
50% overlap with the ground truth bounding box (often indicating that at least half of the object of interest 
was detected), it would be counted as a TP. 
Table 3. Parameters and Possible Values for Combinatorial Coverage. 
Category 
Possible Values (Arbitrarily defined by evaluator 
based on expected operating conditions) 
Assuring Trustworthy Computer Vision 
for Rapid Counter-Drone System Testing and Deployment 
1A.03 - 16
STO-MP-IST-210