SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Adam D. Cobb,\* Anirban Roy\* Daniel Elenius\* F. Michael Heim\* Brian Swenson
Sydney Whittington, James D. Walker; Theodore B Pty; Joseph Hite; Karthik Ramani
Christopher McComb, Susmit Jha*
* Computer Science Laboratory, SRI International
† Mechanical Engineering Division, Southwest Research Institute
Intelligent Systems Division, Southwest Research Institute
‡ Institute of Software Integrated Systems, Vanderbilt University
$^{\S}$  School of Mechanical Engineering, Purdue University
Department of Mechanical Engineering, Carnegie Mellon University
We present AircraftVerse, a publicly available aerial vehicle design dataset. Aircraft design encompasses different physics domains and, hence, multiple modalities of representation. The evaluation of these cyber-physical system (CPS) designs requires the use of scientific analytical and simulation models ranging from computer-aided design tools for structural and manufacturing analysis, computational fluid dynamics tools for drag and lift computation, battery models for energy estimation, and simulation models for flight control and dynamics. AircraftVerse contains 27,714 diverse air vehicle designs - the largest corpus of engineering designs with this level of complexity. Each design comprises the following artifacts: a symbolic design tree describing topology, propulsion subsystem, battery subsystem, and other design details; a STandard for the Exchange of Product (STEP) model data; a 3D CAD design using a stereolithography (STL) file format; a 3D point cloud for the shape of the design; and evaluation results from high fidelity state-of-the-art physics models that characterize performance metrics such as maximum flight distance and hover-time. We also present baseline surrogate models that use different modalities of design representation to predict design performance metrics, which we provide as part of our dataset release. Finally, we discuss the potential impact of this dataset on the use of learning in aircraft design and, more generally, in CPS. AircraftVerse is accompanied by a data card, and it is released under Creative Commons Attribution-ShareAlike (CC BY-SA) license. The dataset is hosted at https://zenodo.org/record/6525446; baseline models and code at https://github.com/SRI-CSL/AircraftVerse and the dataset description at https://aircraftverse.onrender.com/.
Datasets of complex cyber-physical systems (CPS) are difficult to build and large CPS datasets are not publicly available. Their availability is limited for multiple reasons, such as: proprietary restrictions; difficulty in assembling the right mix of experts; and the slow manual design process
due to their complexity. However, a huge opportunity exists to apply data-driven approaches to CPS once such a dataset becomes widely available. Electric Vertical Take-Off and Landing (eVTOL) aircraft represent an emerging class of CPS. The use of electrical propulsion and the growing energy density of available batteries have fueled rapid growth in eVTOL aircraft designs from food delivery/logistics [12], to the detection of sharks along highly populated beaches [18], and to air taxis. We expect this variation to only increase as battery technology continues to improve [10]. The diversity within the electric aerial vehicle design space is large due to the many choices available for selecting structural, mechanical, and electrical components. The multiphysics nature of CPS designs necessitate heterogeneity in representation, such as the use of stereolithographic (STL) files for computational fluid dynamics (CFD) analysis and the use of symbolic descriptions of motors and speed controllers for electrical analysis. Data-driven learning methods for CPS must be able to handle this diversity and multimodality of CPS designs. AircraftVerse provides such a dataset to enable further research into data-driven methods for characterizing or designing CPS.
(a) CAD Datasets
(b) Example of 3D CAD models of aircraft designs in AircraftVerse
Figure 1: Existing CAD datasets (SketchGraphs [28], DeepCAD [35], ABC [19]) are focused on CAD for mechanical parts. Aircraft designs in AircraftVerse include CAD models as one of the modalities. A CAD model (STEP or STL) is an assembly of several components such as propellers, wings, connectors, beams, motors, batteries, and hubs. The CAD designs in AircraftVerse are more complex compared to the existing datasets. The inset magnifications of a couple of parts of one of the designs in the figure above demonstrates this complexity. In addition to CAD models, each design also includes a symbolic design tree with additional details such as propulsion and battery subsystems that are needed for performance analysis (e.g. electrical and flight dynamics analysis). AircraftVerse also contains the result from the evaluation of each design using high-fidelity scientific and engineering tools. Thus, AircraftVerse is a CPS dataset and not just a CAD dataset.
AircraftVerse (Figure 1) contains 27,714 diverse aircraft designs where each design is represented in multiple modalities, including a computer aided design (CAD) model and symbolic description. Each design is also accompanied with detailed evaluation results. Thus, AircraftVerse is naturally amenable to emerging neurosymbolic and other deep learning methods for generative modeling, surrogate learning and sequence-to-sequence models. Overall, our paper is structured as follows. In Section 2 we provide a summary of the few existing related datasets. We then introduce AircraftVerse in Section 3 and describe the constituent parts that make up a design. We highlight the potential of AircraftVerse in Section 4 by providing an overview of the diversity of designs and by displaying experiments on surrogate modeling, while noting that this covers just a small part of what kind of experimentation is possible with this new dataset. We then conclude in Section 7
The use of machine learning in CAD has gained significant attention, and a few datasets have been proposed in recent literature to enable development and benchmarking of machine learning approaches. SketchGraphs dataset [28] is a collection of sketches extracted from parametric CAD models which begin as two-dimensional (2D) sketches consisting of geometric primitives (e.g., line
segments, arcs) and explicit constraints between them (e.g., coincidence, perpendicularity) that form the basis for three-dimensional (3D) construction operations. This dataset has been used for generative model of CAD sketches [34], and other applications of learning in physical design [29, 23]. Another example of a CAD dataset that is focused on physical structure is SimJEB [33], which is a dataset of crowdsourced mechanical brackets and accompanying structural simulations. DeepCAD [35] is a dataset of 3D shapes corresponding to objects such as flanges, pipes and screws, represented as a sequence of operations used in a CAD framework to generate these shapes. Another dataset for 3D engineering shapes is the ABC dataset [19], which comprises geometric models, each defined by parametric surfaces and associated with ground truth information on the decomposition into patches. These datasets are excellent resources for their target application domains such as extrapolating 2D sketch to CAD designs, and generating mechanical parts.
In contrast, AircraftVerse is a dataset that covers the more complex design space of electric aircraft focused on system-level CPS design and thus, complements existing CAD datasets. In Figure I, we illustrate the complexity of the aircraft designs in AircraftVerse compared to mechanical components from the existing CAD datasets. The CAD models of an aircraft design in AircraftVerse are an assembly of several mechanical components, such as propellers, motors, wings, hub and connectors. Further, the AircraftVerse dataset includes additional description of a design beyond its CAD model, such as its propulsion subsystem and its electrical subsystem. This is crucial for creating a description that can predict the performance (flight dynamics, electrical analysis) of a design as designs with similar CAD structure can have very different performances depending on the used components. We also provide the performance of each design using detailed high-fidelity physics and engineering simulation tools to enable the use of this dataset not just for generative modeling, but also for learning surrogates and design characterization and optimization.
3 AircraftVerse dataset
The key characteristics of AircraftVerse are as follows:
- The number of designs in AircraftVerse is 27,714 making it a uniquely large-scale CPS dataset with high design complexity. Design curation required finding valid aircraft configurations followed by detailed simulations to evaluate its performance objectives. We have attempted to create a balanced dataset that includes a sufficient number of aircraft designs across a range of different flight performance metrics, such as hover times and maximum flight distances. Our search for designs included over a hundred thousand candidates from which we selected these 27,714 designs to ensure diversity in design and performance. Our design process itself is diversity-preserving and ensures optimization does not lead to very similar designs [5, 6].
- AircraftVerse represents design using multiple modalities that include: a symbolic design tree describing the design topology, propulsion subsystem, battery subsystem, and other design details; a STandard for the Exchange of Product (STEP) model that is a decomposable CAD model showing each part separately; and a stereolithographic (STL) CAD model that is ideal for computational fluid dynamics analysis, and the corresponding 3D point cloud for the shape of the design.
- AircraftVerse uses results from high-fidelity physics models and an evaluation pipeline developed as a part of DARPA's Symbiotic Design of CyberPhysical System's program to evaluate different aircraft designs. These physics models characterize performance metrics such as maximum flight distance and hover-time. The evaluation pipeline uses a mixture of custom flight dynamics simulators [31] and commercial tools such as Creo [24]. The evaluation of a design to determine performances such as its drag, lift, maximum flight time and hover time requires a significant compute infrastructure and requires subject-matter expertise. We include these evaluation results as part of the dataset.
- The designs in AircraftVerse exhibit a very high degree of diversity in their topology, the choice of energy subsystem and the choice of propulsion. The designs are a mixture of rotorcrafts, winged aircrafts and hybrids with  $28\%$  capable of vertical takeoff and landing. To the best of our knowledge, there is no other available corpus of such diverse aircraft designs with the design details and the results from detailed scientific and engineering evaluation.
<sup>1</sup>https://www.darpa.mil/program/symbiotic-design-for-cyber-physical-systems
2We note that some aerial vehicles, such as many fixed wing aircraft, are launched (or catapulted) and therefore are not required to hover.
- We include multiple baseline surrogate models to predict some of the key design performance metrics. Our surrogate models include a transformer encoder model and an LSTM model that take the symbolic design configuration, as its input, as well as a graph convolution neural network and a PointNet model that use the point cloud modality. These baseline models are used to predict the mass, the maximum flight distance, the maximum hover-time and the presence of any structural interferences that needs to be avoided for fabrication and manufacturing.
Dataset Availability. The dataset is publicly available for free under Creative Commons Attribution-ShareAlike (CC BY-SA) license, which will enable future extension and adaptation of the dataset by others. With respect to its maintainability and long-term availability, the dataset is hosted at https:// zenodo.org/record/6525446 and will be maintained by SRI International<sup>3</sup>, a non-profit research institute with over 75 years of history of contributing to society and research community, with a proven track record of building, maintaining and distributing several datasets such as BioCyc [16], Voices [27], and open-source tools<sup>4</sup> - some of which have been maintained for multiple decades.
Design Structure and Representation. Each design in the dataset consists of the following:
<sup>3</sup>https://en.wikipedia.org/wiki/SRI_International
$^{4}$ https://github.com/SRI-CSL
For a complete printout see Appendix G.
- design_tree.json: The design tree describes the design topology, choice of propulsion and energy subsystems. The tree also contains continuous parameters such as wing span, wing chord, and the lengths of propeller arms and connectors. In our dataset, we also include a preorder traversal of the design tree and store this as design_seq.json to facilitate the use of sequence-friendly model architectures. In addition, we include design_low_level.json, which is a more fine-grained engineering representation of the design. This fine-grained representation includes significant repetition that is avoided in the abstract tree representation through the use of symmetry (such as specifying that the same arm structure is repeated six times around a hub). See the pictorial representation of the design tree in Figure 2 that demonstrates a simple example of such a symmetry.
- Geom.stp, cadfile.stl, and pointCloud.npy: CAD design for the aerial vehicle in compositional STEP format (ISO 10303 standard), its stereolithographic STL file, and a generated pointcloud of 10,000 points from the CAD representation.
- output.json and trims.npy: Summary files containing the vehicle's performance metrics such as maximum flight distance, maximum hover time, flight distance at maximum speed, maximum current draw, and mass. The trims.npy contains the [Distance, Flight Time, Pitch, Control Input, Thrust, Lift, Drag, Current, Power] at each evaluated trim state (velocity) of the aircraft. Designs that have more trim states and with contiguous trim states are preferable as these designs would be relatively easier to control even when the environment is noisy. Thus, both the output.json and trims.npy contain the evaluation of the performance of each design. See Figure 2 for an example
of these evaluation indicators.
design_tree.json:
trims.npy:
Figure 2: An illustration of all the key file components that make up an aircraft design. The design_tree.json captures all aspects of the design (e.g. 4-way hub) that appears in both the cadfile.stl and the Geom.stp. Additional files of design_seq.json and design_low_level.json are directly derived from the design tree. The performance of the aerial vehicle is displayed in both the performance file labeled output.json and in the trims.npy.
output.json:
{ "Interferences":2, "Mass":3.51, "Batt_amps_ratio_MFD":1.67, "Batt_amps_ratio_MxSpd":2.47, "Distance_MxSpd":482.37, "Max_Distance":505.21, "Hover_Time":31.22, "Max_Speed":31.0, "Max_uc_at_MFD":0.47, "Mot_amps_ratio_MFD":0.55, "Mot_amps_ratio_MxSpd":0.72, "Mot_power_ratio_MFD":0.34, "Mot_power_ratio_MxSpd":0.45, "Power_MFD":764.63, "Power_MxSpd":1517.65, "Speed_MFD":22.0}
Design Example. The design tree used for aircraft designs reflect the hierarchical nature of CPS designs. Figure 2 shows an example design. It has a 4-way central hub component. Each of the four arms of the hub can have a propulsion subsystem attached to it. We identify this subsystem as a MainSegment which can be implemented via different choices (subtrees expanding MainSegment) such as BendSegment, PropArm, ExtendedPropArm, WingArm, and so on. Each of these choices describe the nature of the propulsion subsystem, such as whether we are using propeller or wing, and whether the arm making connection is a simple connector or a composition of connectors. Further, some of these subsystems can be recursive, allowing fractal growth of the design. For example, an arm itself might contain another segment with a hub having a number of arms. Our symbolic representation of design also enables exploitation of symmetry to compress the design description, and inclusion of expert knowledge in the design. For example, if we want all four arms of a hub to contain the same subsystem, then we only need to specify it once (as a single child in the design tree) and use a symmetry tag over the hub, for example, ConnectedHub4_Sym denotes a hub with four connections - each having the same subsystem. In the case of Figure 2, a propulsion subsystem
Figure 3: Diverse vehicle designs in AircraftVerse with different structures and with different performances. Some designs can hover in place. Hover time is in seconds and flight distance is in meters.
comprises a propeller, motor, and a flange. Further, the hub also connects to a fuselage that contains a battery subsystem that can consist of a single or dual battery subsystem along with additional electronics and their position within the fuselage. An explicit example of this is provided in Appendix G. Additionally, the full corpus of options as included in Appendix F is also provided as a Python dictionary as part of AircraftVerse.
Design evaluation. For each design, we use state-of-the-art tools [24, 2] to compute the physical characteristics, such as the mass and the component interferences, as well as to render the 3D CAD models (STEP and STL files that are included in AircraftVerse). These details are then passed to the FDM [31] for evaluating flight dynamics (more details in Appendix C). The dynamics model considers the aircraft velocity and its roll, pitch and yaw rotations, along with the electrical state such as the power drawn from the battery and motor current, which in turn determine the torque on the propeller and its generated thrust. The dynamics model summarizes the impact of different forces such as gravity, drag, propeller lift and wing lift, and includes scientific models for conversion of electrical energy into mechanical energy in the propulsion subsystem. From these evaluations, we include Batt_amps_ratio_MFD, Batt_amps_ratio_MxSpd that correspond to the ratio of maximum current drawn by the design compared to the maximum current rating of the battery at maximum flight distance and maximum speed, respectively. These indicate the robustness of the battery subsystem of the design. We also include Mot_amps_ratio_MFD, Mot_amps_ratio_MxSpd, Mot_power_ratio_MFD, Mot_power_ratio_MxSpd that represent the ratio of the current/power in the motor divided by the maximum allowed current/power corresponding to the maximum flight distance or maximum speed of the vehicle. These indicate the robustness of the propulsion subsystem of the design, its endurance, and its agility. Finally, we include design metrics such as the flight distance at maximum speed (Distance_MxSpd), maximum flight distance (Max_Distance), and the maximum hover (Hover_Time).
4 Experiments: diversity, multimodality and baseline models
Diversity. AircraftVerse includes a diverse set of aircraft designs (samples shown in Figure 3), and the multimodal representation makes this dataset relevant to a variety of challenge problems in machine learning. The components of a design in AircraftVerse are selected from a large component corpus with hundreds of choices of propellers and motors (Appendix F). In addition, each discrete choice of component comes with its own set of attributes that fall in their own ranges (e.g.
Figure 4: The designs in AircraftVerse exhibit diverse performance with respect to characteristics such as maximum flight distance and hover-time. They also have very different physical characteristics such as mass, number of propellers, and number of wings.
motors have a large set of parameters such as motor armature-winding resistance and inductance, mass, counter-electromotive force constant, and motor torque constant). Adding to the overall complexity, there are also the continuous parameters. For example, structural components such as connectors have their radius and length as continuous parameters. In Appendix F we identify the attributes of the key components in an aerial vehicle design and whether their values are fixed or variable within a range. Some characteristics of the design diversity are captured in Figure 4. The figure highlights some of our curation choices. First, the majority of designs have an even number of propellers due to the symmetry built into our tree structure definition. Second, our dataset precludes designs without propellers and favors designs with fewer wings. Finally, while a significant number of designs can take off and achieve horizontal flight (with a large variation in ability), one of the challenges in building a diverse dataset comes at the expense of many designs failing to have the hovering capability or other eVTOL flight performance metrics. However, these designs are useful for learning potential rules as to why these designs have poor characteristics compared to others, so we also include these in the dataset.
While structural differences lead to visually striking diversity, we also highlight that design diversity has other aspects, such as the use of different components within the same design topology. Vehicle designs with the same topology can perform very differently. Figure 5 contains four symmetric quadcopters with the same topology (same design tree structure). However, their performance metrics differ significantly, with the maximum flight distance differing by a factor of over  $2\mathrm{x}$ . Additionally, the electronics of the rightmost quadcopter varies from the other three in that it has two batteries rather than one. Thus, the performance of designs with similar topologies cannot be predicted from just from the CAD design, but requires the symbolic design description in design_tree.json.
Figure 5: Diverse aerial vehicle designs in AircraftVerse with same topology structure but different motor/propeller/battery choices leading to diverse design performance.
Multimodality. The AircraftVerse uses multiple modalities to represent an aircraft design, as illustrated earlier in Figure 2. We would not expect the performance of electric aerial vehicle designs to be
accurately predicted on the basis of only one of the modalities (e.g. the 3D CAD model). Recent work on vision-language models [8,15,120,7] would be especially relevant to large CPS datasets such as AircraftVerse. However unlike in the vision-language scenario, AircraftVerse has stricter metadata that enforces physical and electrical constraints. The multi-modality of AircraftVerse makes it applicable to a wide class of machine learning models. AircraftVerse includes the following modalities:
- Structured sequential data: The success of transformer-based sequence models used in natural language [30, 9, 26, 3, 21] has driven their adoption and extension to structured data such as computer programs. The sequential data from the design trees in AircraftVerse is an interesting use case for transformer models. Further, the designs in AircraftVerse are more structured than natural language text as the design components need to satisfy physical and electrical constraints
- 3D CAD models: There is a growing interest in using machine learning for CAD design [28, 34, 23, 33, 35, 19]. The 3D CAD models (STL and STEP files) in AircraftVerse provide a large dataset with complex 3D shapes. The metadata associated with the designs can be used to train and evaluate methods that predict the physics metrics from the CAD models.
- 3D Pointcloud: The use of machine learning for learning generative models or manipulating or segmenting 3D point clouds has also received significant attention. AircraftVerse includes 3D point clouds extracted from the STL and STEP files. Thus, this dataset with the pointclouds can be used as a benchmark for 3D point cloud classification (e.g., presence or absence of interference) [36,4,13] and regression (e.g., predicting flight characteristics).