SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We can evaluate the precision, recall, and mean average precision (mAP) of the baseline models on different slices of the dataset. Figure 4 provides a summary, and table 4 in appendix C contains the full results. The baseline model performs strongly, showing an mAP of 0.866 overall and a precision of over 0.990 across all categories. In comparison, we see in fig. 4 that the alternative model performs worse overall than the baseline, producing a lower mAP in every category; however, it still makes detections in conditions that were not present during training, indicating some degree of generalizability. The overall lower performance of the alternative model reinforces the need for models to be trained on comprehensive datasets, particularly when used for high-stakes tasks.
There are common patterns between the baseline and alternative models that demonstrate the potential for performance differences within categories, even when trained on a comprehensive dataset. For instance, both models perform worse as the distance between the aircraft increases. In the same way, Boeing 737-800 intruders were easiest to detect by both models even though training was spread evenly between the three aircraft. This result is likely due to the relative size of Boeing 737-800 aircraft compared to the Cessna Skyhawk and King Air C90 and highlights that performance may vary even when model training accounts for variation in conditions. One pattern shown by our test set evaluation environment that we did not intuit was both models performing worse on clear weather conditions than most of the other cloud variations. There are a few potential explanations for
Figure 4: Mean average precision (mAP) of the baseline and alternative models.
this behavior, including that the intruder tends to camouflage with the terrain more when the air is completely clear or that clouds create a more pale background against which intruder detection is made easier by a higher contrast in colors.
In addition to the similarities between the models, we see some differences that further support the argument for comprehensive training. For example, while the models result in a similar mAP on images from the Palo Alto region, there is a significant drop in mAP between the baseline and alternative models for the regions that were not used in the alternative model training. Similarly, the performance of the alternative model decreases significantly for late afternoon samples, in contrast to the baseline model performance. Some late afternoon samples show darker conditions (around dusk), so a drop in performance is not unexpected. However, we see that training the baseline on late afternoon images resulted in more consistent and higher performance on that category. Likewise, training on all locations enabled the baseline model to achieve higher performance than the alternative model. These results demonstrate the importance of comprehensive training datasets.
4.2 Downstream Task Evaluation
We evaluated the AVOIDDS baseline model using the simulator on 8640 encounters sampled from the model described in section 3.3 and appendix A. These encounters were equally distributed among each cloud type, region, time of day, and aircraft type shown in fig. 4. For each combination of conditions (of which there are 288), 30 encounters were randomly generated using the encounter model and used to evaluate the detection models. Our aim was to produce analogous results to the test set evaluation, creating the same variation in conditions and at least as many instances to evaluate (8640 encounters and 7200 validation images).
Figure 5 summarizes the safety results, and table 5 in appendix C contains the full safety and efficiency results. Overall, the results highlight the importance of both test set and downstream task evaluation for complex models. We see some alignment between both types of evaluation. In fig. 6, we see from the negative correlation between mAP and NMAC frequency that the model is producing downstream results that are consistent at a high level with the test set results. As a specific example, across all times of day, both models resulted in the highest NMAC rate and lowest mAP on late afternoon samples. However, the downstream task evaluation results in other cases that do not line up with the test set evaluation, demonstrating the need for both types of evaluation to truly capture the performance of the model. Even though the models produced higher mAP on Boeing 737-800 images
Figure 5: Rate of NMAC for the baseline and alternative models. Error bars represent standard error.
Figure 6: Correlation between downstream task evaluation and test set evaluation metrics.
than the other aircraft types, the NMAC frequency for Boeing 737-800 fell between the frequencies for the other aircraft types. The lack of precise predictability of the models' performance on the downstream task based on the test set evaluation reinforces the value of comprehensive evaluation for real-world high-stakes models.
In this work, we outlined the AVOIDDS dataset, our associated vision-based aircraft detection models, test set evaluation, and closed-loop simulation. Via evaluation of our baseline model on the test set and downstream task and comparison to the alternative model, we validated the need for sophisticated training and evaluation functionalities, particularly when it comes to real-world, high risk applications in which the margin for error is minimal. These components, while centered around aviation as an application, serve as an example of an accessible, functioning benchmark for designing and refining machine learning-based perception systems using both standard and task-specific evaluation metrics. The main potential negative impact of our work is premature deployment of models that have not been thoroughly trained and tested, resulting in unsafe conditions for the passengers or other aircraft. While we make an effort to cover as many environmental conditions as possible, we cannot guarantee that AVOIDDS covers all conditions aircraft will experience when deployed in the real world. We
also use simulated images, and while X-Plane 11 is photorealistic, the sim-to-real gap should be further investigated. Future work could also explore accounting for state uncertainty in the controller to improve overall performance. We hope AVOIDDS motivates further work on comprehensive benchmarks that can directly impact the training and deployment of complex models.
The NASA University Leadership Initiative (grant #80NSSC20M0163) provided funds to assist the authors with their research. This research was also supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE-1656518. Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of any NASA entity or the National Science Foundation.
[44] Airborne object tracking dataset, https://registry.opendraaws/airborne-object-tracking/, 2021.
[45] M. Mueller, N. Smith, and B. Ghanem, “A benchmark and simulator for UAV tracking,” in European Conference on Computer Vision (ECCV), Springer, 2016, pp. 445–461.
[46] S. Li and D.-Y. Yeung, "Visual object tracking for unmanned aerial vehicles: A benchmark and new motion models," in AAAI Conference on Artificial Intelligence (AAAI), vol. 31, 2017.
[47] Y. Wang, Z. Huang, R. Laganière, H. Zhang, and L. Ding, “A UAV to UAV tracking benchmark,” Knowledge-Based Systems, vol. 261, p. 110-197, 2023.
[48] M. J. Kochenderfer, J. E. Holland, and J. P. Chryssanthacopoulos, "Next generation airborne collision avoidance system," Lincoln Laboratory Journal, vol. 19, no. 1, pp. 17-33, 2012.
[49] S. M. Katz, L. E. Alvarez, M. Owen, S. Wu, M. W. Brittain, A. Das, and M. J. Kochenderfer, "Collision risk and operational impact of speed change advisories as aircraft collision avoidance maneuvers," in AIAA AVIATION Forum, 2022.
[50] S. M. Katz, A. Corso, S. Chinchali, A. Elhafsi, A. Sharma, M. Pavone, and M. J. Kochenderfer, "NASA ULI aircraft taxi dataset," Stanford Digital Repository, 2021.
[51] K. D. Julian and M. J. Kochenderfer, "Guaranteeing safety for neural network-based aircraft collision avoidance systems," in Digital Avionics Systems Conference (DASC), 2019.
[52] B. J. Chludzinski, “Evaluation of TCAS II version 7.1 using the FAA fast-time encounter generator model,” Massachusetts Institute of Technology, Lincoln Laboratory, Project Report ATC-346, 2009.
[53] J. E. Holland, M. J. Kochenderfer, and W. A. Olson, "Optimizing the next generation collision avoidance system for safe, suitable, and acceptable operational performance," Air Traffic Control Quarterly, vol. 21, no. 3, pp. 275-297, 2013.
[54] M. J. Kochenderfer, L. P. Espindle, J. K. Kuchar, and J. D. Griffith, "Correlated Encounter Model for Cooperative Aircraft in the National Airspace System," Massachusetts Institute of Technology, Lincoln Laboratory, Project Report ATC-344, 2008.
[55] M. J. Kochenderfer, M. W. Edwards, L. P. Espindle, J. K. Kuchar, and J. D. Griffith, "Airspace encounter models for estimating collision risk," AIAA Journal of Guidance, Control, and Dynamics, vol. 33, no. 2, pp. 487-499, 2010.
[56] A. J. Weinert, E. P. Harkleroad, J. D. Griffith, M. W. M. Edwards, and M. J. Kochenderfer, "Uncorrelated encounter model of the national airspace system version 2.0," Massachusetts Institute of Technology, Lincoln Laboratory, Project Report ATC-404, 2013.
[57] M. J. Kochenderfer, L. P. Espindle, J. K. Kuchar, and J. D. Griffith, "Correlated encounter model for cooperative aircraft in the national airspace system," Massachusetts Institute of Technology, Lincoln Laboratory, Project Report ATC-344, 2008.
[58] M. J. Kochenderfer and J. P. Chryssanthacopoulos, "Robust airborne collision avoidance through dynamic programming," Massachusetts Institute of Technology, Lincoln Laboratory, Project Report ATC-371, 2011.
A Encounter Model Details
Table 1: Parameters for data generation.
Parameter	Minimum	Maximum	Unit
Ownership Horizontal Speed	60	70	m/s
Intruder Horizontal Speed	60	70	m/s
Horizontal Miss Distance	0	100	meters
Vertical Miss Distance	-30	30	meters
Relative Heading	100	260	degrees
In order to simplify the representation of encounters in this study, we adopt a model where the ownership and intruder aircraft move along straight-line trajectories with constant horizontal speeds. To generate an encounter, we follow a two-step process. First, we randomly sample a set of encounter features from uniform distributions within the specified ranges presented in table 1. We then use these features to generate trajectories for both the ownership and intruder aircraft. The horizontal and vertical miss distance parameters indicate the range between the ownership and intruder aircraft and their relative altitude at the point of closest approach. These distances are deliberately chosen to ensure that all encounters result in a near mid-air collision (NMAC) if no collision avoidance action is taken.
Each simulated encounter has a duration of 50 seconds, with the closest point of approach occurring 40 seconds into the encounter. The range of relative headings is selected to generate encounters that are nearly head-on, with the intruder aircraft typically within the ownership's field of view. The features specified in table 1 completely determine the relative trajectories of the ownership and intruder aircraft. Once the relative trajectories are generated, we randomly position both trajectories around the origin region by applying rotations and shifts.
B Dataset Details
The AVOIDDS dataset was generated to include 72,000 images, of which 64,800 are training images and 7200 are validation images. These images are accompanied by 72,000 label files. Table 2 shows the number of images for each variable category while table 3 shows the exact distributions used to generate the images. Equal amounts of each cloud covering, region, and aircraft type are represented, and the other variables were randomized for each image.
The range between the ownership and intruder aircraft was sampled from a gamma distribution with shape and scale parameters dependent on the aircraft type. We sampled the distances for the Cessna Skyhawk and King Air C90 from a gamma distribution with shape 2 and scale 200, and the Boeing 737-800 distances were sampled from a gamma distribution with shape 3 and scale 200. The expected value of  $\Gamma(3,200)$  is about  $200\mathrm{m}$  more than that of  $\Gamma(2,200)$  which we intended to account for the larger size of the Boeing 737-800 aircraft relative to the smaller aircraft. A gamma distribution allows us to sample from a distribution that is skewed toward closer ranges, where aircraft are more likely to be visible in the image. To ensure that the aircraft were not positioned too close together, we verified that the sampled values were greater than  $20\mathrm{m}$  for the Cessna Skyhawk and King Air C90 and  $50\mathrm{m}$  for the Boeing 737-800. The position vertically and horizontally of the intruder was sampled uniformly within the ownership field of view. The time of day for each sample was randomized between 08:00 and 17:00 on January 1st in each respective location's local time. We split the day into 4 time windows for evaluation purposes: morning (08:00-10:00), midday (10:00-13:00), afternoon (13:00-15:00), and late afternoon (15:00-17:00).
Table 4 shows the specific test set evaluation values for the baseline and alternative models discussed, specifically precision, recall, and mAP. These can be compared to the downstream task metrics shown in table 5, which includes the frequencies of NMAC and advisory.
Table 2: AVOIDDS dataset overview.
Attribute	Value	Number of images
Total	Training	Validation
All	-	72,000	64,800	7200
Clouds	Clear	12,000	10,800	1200
High Cirrus	12,000	10,800	1200
Scattered	12,000	10,800	1200
Broken	12,000	10,800	1200
Overcast	12,000	10,800	1200
Stratus	12,000	10,800	1200
Region	Palo Alto, CA (PAO)	18,000	16,200	1800
Boston, MA (BOS)	18,000	16,200	1800
Oshkosh, WI (OSH)	18,000	16,200	1800
Reno, NV (RNO)	18,000	16,200	1800
Aircraft type	Cessna Skyhawk	24,000	21,600	2400
Boeing 737-800	24,000	21,600	2400
King Air C90	24,000	21,600	2400
Range	0–150 m	9124	8268	856
150–500 m	35,932	32,303	3629
>500 m	26,944	24,229	2715
Intruder rel. alt.	Below	36,048	32,482	3566
Above	35,952	32,318	3634
Time of day	Morning	15,930	14,385	1545
Midday	24,142	21,722	2420
Afternoon	15,954	14,269	1685
Late Afternoon	15,974	14,424	1550
Table 3: Parameters for the AVOIDDS Dataset.
Parameter	Minimum	Maximum	Distribution	Unit
Ownership distance east/north from origin	-5000	5000	U(-5000, 5000)	meters
Ownership distance vertically from origin	-1000	1000	U(-1000, 1000)	meters
Ownership and intruder heading	0	360	U(0, 360)	degrees
Ownership pitch	-30	30	N(0, 5)	degrees
Ownership roll	-45	45	N(0, 10)	degrees
Time of day	08:00	17:00	U(08:00, 17:00)	hours
D Experiment Reproduction
The experiments in this work can be reproduced using the AVOIDDS repository, available at this link: https://github.com/sisl/VisionBasedAircraftDAA.
D.1 Test Set Evaluation
Steps for how to reproduce the test set evaluation experiment results are as follows:
1. Download AVOIDDS dataset: Download and extract the AVOIDDS dataset (https://purl.stanford.edu/hj293cv5980) and place the folder in a convenient location.
2. Download and setup the code repository: Download the code repository (https://github.com/sisl/VisionBasedAircraftDAA). Navigate to the src/model directory and run pip3 install -r requirements.txt to install the necessary dependencies for test set evaluation.
Table 4: Test set evaluation results for baseline and alternative model
Attribute	Value	Baseline model	Alternative model
Precision	Recall	mAP	Precision	Recall	mAP
All	-	0.997	0.907	0.866	0.884	0.928	0.764
Clouds	Clear	0.991	0.905	0.856	0.818	0.918	0.687
High Cirrus	1.000	0.930	0.900	0.943	0.931	0.827
Scattered	0.997	0.923	0.887	0.952	0.936	0.841
Broken	0.997	0.921	0.888	0.934	0.936	0.826
Overcast	0.995	0.920	0.882	0.857	0.937	0.747
Stratus	0.999	0.846	0.783	0.829	0.912	0.684
Region	Palo Alto, CA (PAO)	0.999	0.930	0.900	0.984	0.934	0.865
Boston, MA (BOS)	0.996	0.922	0.885	0.819	0.938	0.722
Oshkosh, WI (OSH)	0.999	0.866	0.813	0.912	0.919	0.772
Reno, NV (RNO)	0.992	0.912	0.865	0.849	0.922	0.719
Aircraft Type	Cessna Skyhawk	0.995	0.844	0.750	0.880	0.867	0.643
Boeing 737-800	0.999	0.988	0.983	0.914	0.986	0.891
King Air C90	0.996	0.897	0.852	0.849	0.922	0.761
Range	0–150 m	0.999	0.983	0.979	0.914	0.997	0.909
150–500 m	0.997	0.960	0.942	0.879	0.979	0.843
>500 m	0.995	0.818	0.714	0.881	0.838	0.621
Intruder rel. alt.	Below	0.995	0.844	0.847	0.905	0.920	0.764
Above	0.998	0.917	0.884	0.863	0.937	0.763
Time of day	Morning	0.997	0.911	0.872	0.929	0.916	0.785
Midday	0.998	0.907	0.866	0.923	0.927	0.799
Afternoon	0.998	0.914	0.878	0.937	0.940	0.831
Late Afternoon	0.993	0.897	0.846	0.767	0.930	0.657
3. Begin evaluation: Run python3 eval.py -o baseline_results -d [PATH TO AVOIDDS DATASET]. Results for baseline model evaluation will appear in baseline_results.txt. Run python3 eval.py -o alternative_results -m ".../.../models/alternative.pt" -d [PATH TO AVOIDDS DATASET] to evaluate the alternative model with results outputting to alternative_results.txt.
D.2 Downstream Task Evaluation
Steps for how to reproduce the above downstream task experiment results are as follows:
1. Download and setup the code repository: Download the code repository (https://github.com/sisl/VisionBasedAircraftDAA). Navigate to the src/simulator directory and run pip3 install -r requirements.txt to install the necessary dependencies for downstream task evaluation.
2. Setup X-Plane and aircraft: Follow the instructions in the "Setup X-Plane" section of the benchmark repository README file (https://github.com/sisl/VisionBasedAircraftDAA/tree/main/src.simulator), setting the intruder aircraft as the one with which you would like to simulate encounters.
3. Set variables for simulation: At the bottom of the simulate.py file, uncomment and set the variables in the "BULK SIMULATION VARIABLE SETUP". Set args.craft to the aircraft you set in X-Plane in the previous step. Set args.model_path to the desired model as well.
4. Begin simulation: In the command line, run ./simulate.sh and toggle your screen such that the X-Plane window is visible and full-screen. You will need to leave this visible for the entirety of the simulation.
5. Repeat: Repeat steps 2 through 4 for each desired aircraft and model.
Table 5: Downstream task evaluation results for baseline and alternative model
Attribute	Value	Total Encs	Baseline model	Alternative model
NMAC frq	Advisory frq	NMAC frq	Advisory frq
All	-	8640	0.143	0.187	0.165	0.191
Clouds	Clear	1440	0.158	0.181	0.188	0.186
High Cirrus	1440	0.134	0.192	0.153	0.206
Scattered	1440	0.142	0.191	0.157	0.206
Broken	1440	0.146	0.184	0.156	0.194
Overcast	1440	0.138	0.188	0.163	0.183
Stratus	1440	0.142	0.185	0.174	0.172
Region	Palo Alto, CA (PAO)	2160	0.144	0.199	0.160	0.203
Boston, MA (BOS)	2160	0.147	0.185	0.163	0.198
Oshkosh, WI (OSH)	2160	0.140	0.180	0.157	0.190
Reno, NV (RNO)	2160	0.141	0.183	0.180	0.173
Aircraft Type	Cessna Skyhawk	2880	0.157	0.124	0.179	0.130
Boeing 737-800	2880	0.141	0.280	0.152	0.287
King Air C90	2880	0.132	0.156	0.164	0.156
Time of day	Morning	2160	0.140	0.186	0.178	0.193
Midday	2160	0.140	0.188	0.148	0.195
Afternoon	2160	0.144	0.188	0.153	0.194
Late Afternoon	2160	0.149	0.185	0.181	0.182