SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Elysia Q. Smyers
elysia@cs.stanford.edu
Sydney M. Katz
Department of Aeronautics and Astronautics
smkatz@stanford.edu
Anthony L. Corso
acorso@stanford.edu
Mykel J. Kochenderfer
mykel@stanford.edu
Designing robust machine learning systems remains an open problem, and there is a need for benchmark problems that cover both environmental changes and evaluation on a downstream task. In this work, we introduce AVOIDDS, a realistic object detection benchmark for the vision-based aircraft detect-and-avoid problem. We provide a labeled dataset consisting of 72,000 photorealistic images of intruder aircraft with various lighting conditions, weather conditions, relative geometries, and geographic locations. We also provide an interface that evaluates trained models on slices of this dataset to identify changes in performance with respect to changing environmental conditions. Finally, we implement a fully-integrated, closed-loop simulator of the vision-based detect-and-avoid problem to evaluate trained models with respect to the downstream collision avoidance task. This benchmark will enable further research in the design of robust machine learning systems for use in safety-critical applications. The AVOIDDS dataset and code are publicly available at https://purl.stanford.edu/hj293cv5980 and https://github.com/sisl/VisionBasedAircraftDAA, respectively.
The use of machine learning in high stakes applications will require the design of robust systems that perform well in a wide range of environmental conditions [1]–[3]. For example, a learning-based perception system designed for use in aviation will need to handle changing environmental conditions such as weather and time of day. In addition, these systems are often components of an autonomy stack, and their performance should be evaluated both in isolation and with respect to the downstream task of the system in which they operate [4], [5]. For the aviation example, perception models should not only be evaluated based on isolated metrics on a test set such as mean average precision but also on downstream tasks such as collision avoidance. Benchmark systems that allow for this comprehensive evaluation are needed to build robust machine learning systems.
A number of recent benchmark datasets such as WILDS [3] and MetaShift [6] contain distribution shifts due to environmental changes; however, these datasets only allow for evaluation on a test set and generally do not provide evaluation capabilities for downstream decision-making tasks. While recent work has highlighted the importance of task-specific design and evaluation, there is a lack of standardized and accessible benchmarks in this area [4], [5], [7], [8]. To provide a benchmark environment that covers both environmental changes and task-specific evaluation, we
Figure 1: AVOIDDS benchmark overview.
introduce AVOIDDDS (Aircraft Vision-based Intruder Detection Dataset and Simulator). AVOIDDDS is a benchmark dataset and evaluation environment for the problem of vision-based aircraft detect-and-avoid (DAA). For this task, onboard aircraft systems must detect nearby aircraft and determine proper maneuvers to avoid colliding with them. AVOIDDDS specifically focuses on vision-based models, which are trained to detect intruding aircraft from images taken by a mounted camera [9]–[13]. These models will be especially critical in the development of unmanned aircraft and future air mobility concepts.
AVOIDDS allows for training and testing of vision-based aircraft detection models. The benchmark includes three main components (see fig. 1):
- Data: We provide a dataset containing 72,000 airspace images for training detection models along with data generation code for customizable creation of new datasets. The dataset is annotated with metadata that describes the environmental conditions under which each image was captured.
- Baseline models: We provide baseline YOLOv8 aircraft detection models trained on the AVOIDDS dataset.
- Evaluation capabilities: We provide capabilities for test set evaluation and an aircraft encounter simulator to evaluate the closed-loop performance of trained detection models on the downstream task of collision avoidance.
AVOIDDS serves as a benchmark for vision-based aircraft DAA and will enable further research on the robust design and evaluation of learning-based perception systems. By defining a specific downstream task and collecting annotated data across a variety of conditions, we enable further research into solving safety-critical problems.
The AVOIDDS benchmark is related to previous work involving object detection benchmarks, task-specific evaluation, datasets with distribution shifts, and vision-based aircraft detect-and-avoid.
Object detection benchmarks Early object detection datasets include Pascal VOC [14] and COCO [15], which contain images of common objects with their corresponding labels. These datasets have
served as standard evaluation benchmarks for new object detection algorithms throughout the last decade [16]–[23]. While these benchmarks cover a wide range of common objects, they do not capture the performance of object detection algorithms on task-specific domains. Li et al. [24] created the ODinW (Object Detection in the Wild) dataset, which contains 13 task-specific datasets that were used to check zero-shot performance of language-image models. Ciaglia et al. [25] provide a more extensive version of this dataset called RF100 (Roboflow-100) that consists of 100 task-specific object detection datasets from domains such as microscopic images and video games. However, ODinW and RF100 do not provide interfaces to assess task-specific performance metrics on each dataset. In this work, we provide both a task-specific dataset for vision-based aircraft collision avoidance and a simulator to evaluate performance on the downstream collision avoidance task.
Task-specific evaluation Object detection models are often evaluated in isolation using metrics such as precision, recall, and mean average precision (mAP) [21]–[23]. However, these models are often components of a system for a high-level decision-making task. For the vision-based detect-and-avoid example, the object detection model is trained to detect intruding aircraft such that the overall system can recommend safe collision avoidance maneuvers. Furthermore, the best model according to traditional metrics does not always result in the best performance on downstream tasks [4], [5], [7], [8]. For this reason, it is important to incorporate task-aware metrics into the design and evaluation of machine learning models. Philion et al. [4] develop planner-centric metrics for a machine-learning based object detection system used in autonomous driving and show their benefit over traditional metrics. They evaluate the metrics using driving trajectories from the nuScenes dataset [26]. The nuScenes dataset provides trajectories for the autonomous driving task, but the trajectories are prerecorded and therefore do not allow for a proper simulation to assess the performance of the fully-integrated, closed-loop system [26]. The CARLA simulator enables closed-loop simulation of driving trajectories and scenarios to evaluate various models for perception and control [27]–[30]. In the aviation domain, AirSim allows for the closed-loop simulation of unmanned aerial vehicles, such as small drones [31]–[34]. Corso et al. [5] design safer perception systems by quantifying the effect of perception errors on the performance of a downstream task and apply their methods to increase the safety of a vision-based detect-and-avoid system. This work expands on this application to provide an accessible benchmark for task-specific evaluation.
Datasets with distribution shifts Previous work has shown that machine learning models often drop in performance when their test distribution differs from their training distribution [1]–[3]. This dropout is especially apparent if the model relies on spurious correlations in the training data [35]. For this reason, researchers have created benchmark datasets with distribution shifts. Early datasets containing distribution shifts focused on local perturbations such as rotation and image noise. For example, some works created distribution shifts by rotating images in standard benchmark datasets such as MNIST and CIFAR-10 [36], [37]. Hendrycks et al. [38] created the ImageNet-C dataset by applying 75 common corruptions to the ImageNet dataset and noted a drop in performance of state-of-the-art models when evaluated on the corrupted images. Datasets such as WILDS [3] and MetaShift [6] go beyond image transformations and gather data containing more general distribution shifts that machine learning models may encounter when deployed in the wild. The WILDS dataset, for example, contains 10 datasets with distribution shifts involving domain generalization and subpopulation shift [3]. However, the WILDS dataset does not have clear annotations of the semantic concepts that change in each distribution shift. Datasets such as NICO [39] and MetaShift [6] provide metadata describing the semantic concepts present in images of common objects. This metadata enables the evaluation of model performance across groupings of the dataset with similar concepts. Inspired by the metadata used in NICO and MetaShift, AVOIDDS provides annotations of the environmental conditions such as time of day, weather, and geography for each aircraft image. While previous distribution shift benchmarks evaluate changes in model performance using standard test set metrics, AVOIDDS also allows for evaluation of performance with respect to the downstream task of the broader system in which the model operates.
Vision-based aircraft detect-and-avoid Aircraft detect-and-avoid systems rely on sensor information to detect and track intruding aircraft so that they may issue proper collision avoidance advisories. Traditional sensors used for surveillance and tracking include ADS-B, onboard radar, and transponders [40], [41]; however, automated aircraft collision avoidance systems will require additional sensors both for redundancy and to replace the visual acquisition typically performed by the pilot. For this reason, vision-based traffic detection systems have been proposed, in which intruding aircraft
Figure 2: Vision-based detect-and-avoid system overview.
are detected from a camera sensor mounted on the aircraft [9]–[13], [42], [43]. Early work on vision-based aircraft detection used traditional model-based computer vision techniques [42], [43]; however, these techniques were unable to detect aircraft below the horizon in front of a background with ground clutter. Deep learning-based approaches address this limitation and are well-suited for the aircraft detection task [9]–[13]. However, since this application is safety-critical, it is important to evaluate the safety and robustness of these systems with respect to changes in the environment using realistic datasets and simulators [44]. Aerial object tracking datasets provide video sequences of aerial imagery [45]–[47]. The simulator in the AVOIDDS benchmarks produces video sequences similar to these existing tracking datasets while also allowing for the simulation of new scenarios. Of the vision-based traffic detection benchmarks and datasets that are publicly available, AVOIDDS represents the only benchmark with both an extensive dataset covering a range of environmental conditions annotated with metadata and a simulator for evaluation on the downstream collision avoidance task.
3 AVOIDDS Benchmark
An aircraft detect-and-avoid system is responsible for sensing nearby aircraft and determining the necessary maneuvers to avoid collision. Figure 2 provides an overview of this process for a vision-based system. At each time step, the equipped aircraft (referred to as the ownership) captures an image of its surroundings using a mounted camera. This image is then passed through a perception system that detects surrounding aircraft (referred to as intruders) and produces an estimate of their state. This state estimate is then passed to a controller, which selects a collision avoidance maneuver. While extensive previous work has studied aircraft collision avoidance controllers that rely on state information [40], [41], [48], [49], the image-based aircraft detection problem remains an open area of research. Therefore, the AVOIDDS benchmark focuses mainly on the detection component of the detect-and-avoid system but also provides a framework for evaluating this component within the context of the fully-integrated, closed-loop system shown in fig. 2.
Figure 1 outlines the three main components of the AVOIDDS benchmark. The first component provides capabilities for the controllable generation of labeled airspace images under a range of environmental conditions using a photo-realistic flight simulator. We used these capabilities to produce a large, public dataset of airspace images with intruder aircraft at various locations in the frame. Using this dataset, we trained baseline YOLOv8 models to detect intruder aircraft in individual frames passed to the model. The final component involves two evaluation systems: one that outputs traditional metrics such as precision and recall evaluated on a test set of individual image inputs and an aircraft encounter simulator that outputs task-specific metrics. With both of these evaluation systems, we can evaluate model performance on sample inputs and on the downstream task that the model is intended to serve.
3.1 Data Generation
A challenge for the use of computer vision in aviation is that the variability of environmental variables may lead to lower performance, posing large risks. During flight, aircraft can experience changes in terrain, lighting, and weather that will impact the performance of a vision-based model. To capture this variability, we provide an accessible interface for generating customized image datasets with
Time of Day
Midday
Geographic Region
Palo Alto
Weather
High Cirrus
Overcast
Figure 3: AVOIDDS dataset overview.
Aircraft Type
Cessna Skyhawk
King Air C90
a wide range of conditions, including intruder aircraft type, weather, geographic region, relative geometry, and time of day. We use X-Plane  $11^{\text{1}}$ , a photo-realistic flight simulator, for data generation, allowing for programmatic control of the environmental conditions and intruder location for the rendered images. X-Plane 11 also outputs ground truth position and camera data, which allows us to automatically generate bounding box labels, removing the need for manual labeling. This publicly-available commercial flight simulator has been used in prior work to generate data for an aircraft taxi scenario [50].
Using our accessible interface, we generated the AVOIDDS dataset to adequately cover the aforementioned environmental variations. The AVOIDDS dataset is a collection of 72,000 images and labels from the ownership's point of view of encounters with intruder aircraft in the airspace. Each image is randomized across a range of intruder aircraft types, locations, weather conditions, and times of day with the intruder aircraft located uniformly within the ownership's field of view. Capturing a wide variety of conditions allows us to train an associated model that accounts for these same variations in the environment, which is essential in high-stakes situations such as aircraft collision avoidance.
The 72,000 images generated for this dataset are distributed equally among 6 weather types, 3 aircraft types, and 4 regions. Figure 3 summarizes the various conditions present in the AVOIDDS dataset. The time of day for each sample was randomized between 08:00 and 17:00 on January 1st in each respective location's local time. This range includes an adequate spread of times that represents nominal lighting conditions with a small portion of samples captured around dusk or dawn. The encounter location was sampled uniformly within the surrounding region of the following four airports: Palo Alto (PAO), Reno-Tahoe (RNO), Boston Logan (BOS), and Whitman Regional (OSH).
$^{1}$  https://www.x-plane.com (Python interface at https://github.com/nasa/XPlaneConnect)
We selected these regions to create variability in the scenery determined by the geography of each region. The range between the ownership and intruder aircraft was sampled from a gamma distribution with a slight skew toward closer ranges. To account for the larger size of the Boeing 737-800 aircraft relative to the smaller aircraft, the gamma distribution was skewed toward slightly larger ranges for Boeing 737-800 images. The vertical and horizontal position of the intruder was sampled uniformly within the ownership field of view.
The AVOIDDDS dataset serves as an independent, accessible dataset that can be used for training vision-based object detection models for aircraft collision avoidance without having to interface directly with the X-Plane 11 flight simulator. The dataset abides by the YOLO format [23] with subdirectories for images and labels for both the training and validation set. We also include metadata for each image, containing positional information about the ownership and intruder as well as details about the environment (e.g. weather, time of day, region). The ability to filter the dataset using this metadata enables easy slicing of the data for model training and evaluation purposes. For instance, we can choose to evaluate a model on images from a particular location or time of day or even images with the intruder in specific orientations relative to the ownership. This capability allows for evaluation of the model not only on nominal conditions in the airspace but also on conditions that might result in unpredictable model performance.
3.2 Baseline Models
We trained a baseline YOLOv8 object detection model on the AVOIDDS dataset for 100 epochs with default hyperparameters. We used the YOLOv8s architecture, which includes 11.2 million trainable parameters. The training took  $73\mathrm{h}$  on an NVIDIA GeForce GTX 1070 Ti. We also trained an alternative model for comparison on a subset of the AVOIDDS dataset, only including samples in nominal conditions: minimal cloud cover (clear, high cirrus, or scattered clouds) between 08:00 and 15:00 in Palo Alto. The alternative model uses the same architecture and required  $6.5\mathrm{h}$  for 100 epochs with default hyperparameters on 6944 samples.
3.3 Evaluation
As shown in fig. 1, AVOIDDS provide two methods for evaluating trained models: evaluation on a test set and evaluation with respect to the downstream task. For the former, we evaluate the performance of detection models on a test set using standard metrics such as precision, recall, and mean average precision (mAP). Evaluation with respect to the downstream task, on the other hand, allows us to see how these standard metrics translate to performance in simulated aircraft encounters using X-Plane 11. This method of evaluation uses task-specific metrics to measure the model's performance.
**Simulator Overview** The AVOIDDS closed-loop simulator allows us to test vision-based aircraft detection models on the downstream task they were meant to serve: navigating encounters with other aircraft in the airspace. To evaluate closed-loop performance, we provide the three components of the problem shown in fig. 2. We define an encounter model from which the simulator can sample sets of pairwise encounters between an ownership and intruder. We also define a perception system that relies on our trained detection models to predict the intruder state. Its predictions are then passed to a controller produced in previous work [51] that determines the best course of action based on the intruder state. By simulating a full set of encounters and determining the number of encounters that resulted in a near mid-air collision (NMAC), we can evaluate the performance of the detection model with respect to the full closed-loop system. The encounter model only provides positional information and velocities for the two aircraft, allowing for custom simulation of the encounters with different regions, times of day, weather, and intruder aircraft types.
**Encounter Model** Monte Carlo analysis on airspace encounter models has been used extensively to assess the safety of aircraft collision avoidance systems [52], [53]. Encounter models are probabilistic representations of typical aircraft behavior during a close encounter with another aircraft. To analyze the safety of a particular collision avoidance system, we can simulate the system on a set of encounters and analyze the resulting trajectories. We provide a model that generates pairwise encounters in which the ownership and intruder follow straight line trajectories with various relative geometries. We sample encounters by first sampling features such as aircraft speeds, miss distances, and relative headings. We then use these features to generate trajectories for the ownership and intruder aircraft.
Appendix A provides additional details on this model. We provide this straight-line model as a baseline to demonstrate the evaluation capabilities that AVOIDDS enables, and we define a general interface between the encounter model and simulator such that more complex encounter models can easily be incorporated. For example, recent work in airspace modeling has resulted in a number of publicly available data-driven statistical encounter models that capture the full set of variations in aircraft behavior [54]–[57].
Perception System The perception system involves two steps: detecting the intruder in view and interrogating the intruder for its location relative to the ownership. The image observations for the perception system are obtained by positioning the aircraft in X-Plane 11 according to the current state. Once the aircraft are positioned, the perception system uses the vision-based detection model to detect intruding aircraft. If detected, the state of the intruder is then passed to the controller.
Controller The controller takes in the intruder state estimated by the perception system and selects an appropriate collision avoidance advisory. Example advisories include "Clear of Conflict" (COC) if no change of course is required and vertical advisories to climb or descend at different rates. We provide an interface to use the control policy defined in the VerticalCAS repository created by Julian et al. [51]. VerticalCAS contains an open-source collision avoidance logic loosely inspired by the vertical logic used in a family of collision avoidance systems called ACAS X, which model the collision avoidance problem as a Markov decision process (MDP) [40], [41], [48], [49]. The MDP is solved offline using dynamic programming, which results in a collision avoidance policy that balances between safety and efficiency [58]. The collision avoidance logic is stored as a numeric lookup table, and the ownership looks up its current state during flight based on the relative geometry of the intruder to determine optimal collision avoidance advisories. While VerticalCAS is a strong baseline inspired by real-world systems such as ACAS X, it is a notional example designed for research purposes, and it has not been put through the same rigorous testing and validation as ACAS X.
Simulator Metrics We provide capabilities to retrieve task-specific metrics from the simulation results related to safety and efficiency. We assess safety by determining the number of encounters that resulted in an NMAC, defined as a simultaneous loss of separation to less than 500 ft horizontally and 100 ft vertically. In this application, we want to minimize the number of NMACs while issuing as few alerts (advisories other than COC) as possible. To evaluate this balance, we also compute the alert frequency, defined as the fraction of time steps in which the system issues an alert.
Using our evaluation capabilities, we evaluate the trained models outlined in section 3.2 on a test set and with respect to the aircraft collision avoidance task.
4.1 Test Set Evaluation