SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Baseline Models. We describe baseline models applied multiple modalities of AircraftVerse. We use a transformer encoder (T. Enc.) [30] model and a LSTM [14] model to predict a set of design performance characteristics of an aircraft from its symbolic design tree description. In our example, even using these symbolic descriptions of a design contain information from multiple modalities, as these sequences include electrical, mechanical, and topological information. We also test out the 3D modality of the design representation by applying both a graph convolutional neural network (GCNN) [17, 32] and PointNet [25] to the point clouds. For the sequences, we use the design representation in design_seq.json and use our custom-built tokenizer to convert each token of a design sequence from the symbolic representation (e.g. {'node_type': 'ConnectedHub3_2_1'}) into a tensor. For our embedding, we assign one-hot encoding for the keys and an additional one-hot encoding for the values. For keys that require floats as values (e.g. 'armLength') we ensure that there is a class that corresponds to the float in the value embedding, as well as appending the value of the float to the tensor. Each token of the sequence is a 749-element vector corresponding to the concatenation of 43 classes of keys, 673 classes of values, 32 possible attributes of electrical/mechanical components, and a final dimension for float values. We provide our tokenization code as part of the dataset release, as well as the baseline models. Further architectural details of the models are presented in Appendix D.
Results. The results of predicting the performance statistics from the design sequences are presented in Table 1. This table shows how the sequential design description, containing specific component information such as mass and electrical information, is a useful modality for certain metrics such as the ability of a design to fly. However, both the baseline GCNN model and PointNet are better at inferring structural interferences between components as it directly incorporates 3D information. Future approaches that combine the 3D structural information and symbolic design descriptions may be necessary and we have demonstrated here that the AircraftVerse dataset includes all of these informative modalities. Figure 6 shows both the receiver operator characteristic (ROC) curve and precision-recall curve for the three models. Again, highlighting the superior performance of the sequential information in estimating the high level CPS performance metrics such as ability to hover.
5 Limitations of AircraftVerse
While AircraftVerse presents itself as a unique CPS dataset with a high level of complexity, there are several limitations to using this dataset. One limitation is that aircraft designs are constrained to fit within the tree-structured representation. This tree structure currently means all designs start from a sampled single hub and are then expanded. This process is described in https://github
See Appendix F for a definition of the structure of our design trees and sequences.
Our models were trained using an NVIDIA A100 SXM4 80GB GPU.
(a) ROC curve
(b) Precision-Recall
Figure 6: ROC and precision-recall curves for predicting a design's ability to hover. These results suggest that the symbolic design description in AircraftVerse can be used to predict flight characteristics, whereas Table[1]highlights the utility of point clouds for predicting structural information.
Table 1: Specification prediction using baseline models.
APPROACH	DATA	HOVER (Acc. ↑)	HOVER TIME (R2↑)	MASS (R2↑)	FLIGHT DISTANCE (R2↑)	INTERFERENCE (F1 SCORE ↑)
T.ENC.	SEQUENCE	0.9004	0.6778	0.9964	0.6900	0.8270
LSTM	SEQUENCE	0.8838	0.6189	0.9974	0.3467	0.8184
GCNN	POINT CLOUD	0.8380	0.3936	0.8888	0.4486	0.9758
POINTNET	POINT CLOUD	0.8474	0.4571	0.9212	0.5099	0.9848
com/SRI-CSL/AircraftVerse/tree/main/prob_gen, which also includes all incorporated prior domain knowledge. Designs are also limited by the available components in the database. For example, hybrid (gas and electric) vehicles are not possible. However, all of the database components are from real-world catalogues which increases the overall value of the dataset.
As with all scientific simulators, the flight dynamics model is limited in its ability to represent all aspects of the real-world. As such, there are likely corner cases that would cause unrealistic behaviors. One weakness of the simulator is in the drag model as described in Equation (4) (Appendix C). The drag model approximates the total surface area in each axis and therefore would be superseded by higher fidelity CFD software. However, the simulator uses conservative estimates for such calculations and has been compared to higher fidelity models to ensure they capture the correct behavior. The results are therefore still useful for design prototyping purposes. A further extension to the simulator to increase fidelity would also be to model material properties such as stress and strain during flight. Further development will continue to address these simulation gaps.
Given these limitations we still highlight the value of this dataset as useful for machine learning researchers looking to explore a novel CPS dataset as well as domain experts looking to explore the potential of data-driven approaches for design. This is precisely what we found to be the case throughout the collaboration that led to this work. As a result, we believe that AircraftVerse is sufficiently complex and interesting for both sets of researchers, who would see value in the data, while ensuring that they take into account all the limitations.
6 Ethics Statement
The data included in the AircraftVerse dataset does not include any personal or sensitive information. It is a technical dataset of air vehicle designs, and thus it does not raise any direct ethical issues regarding privacy or data protection. The dataset is intended for academic and research use and is released under a Creative Commons Attribution-ShareAlike (CC BY-SA) license. This license allows for sharing, copying, distributing, and transmitting the work, as well as adapting the work and making commercial use of the work under the condition that appropriate credit is given, a link to the license is provided, and any changes made are indicated. This licensing framework is intended to promote open access to information and collaboration in the scientific community, while still recognizing and respecting the intellectual property rights of the creators.
Given the potential wide-ranging impact of this dataset on the use of learning in aircraft design and other CPS, it is crucial that future research using this dataset is conducted with careful consideration
of potential ethical implications. This includes not only the avoidance of misuse but also consideration of environmental impacts, safety, and other societal implications. We welcome community input on the dataset, its uses, and its potential ethical implications. We want to promote responsible use and want to address potential ethical concerns proactively.
We have introduced a new multi-modal CPS dataset that will enable researchers to explore a new interdisciplinary area of application for machine learning. We emphasize the richness of AircraftVerse, where we provide 27,714 aircraft designs, each with associated metadata summarizing the performance of the design. We highlight that our experiments and baseline models have only touched the surface of the different approaches and problems that can be explored with this dataset due to the availability of all the metadata associated with each design. In addition to the material presented here, we also include an extensive supplementary materials section as well as a website: https://aircraftverse.onrender.com/, a GitHub repo: https://github.com/SRI-CSL/AircraftVerse/, and the dataset release at https://zenodo.org/record/6525446
This material is based upon work supported by the United States Air Force and DARPA under Contract No. FA8750-20-C-0002. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Air Force and DARPA.
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35: 23716-23736, 2022.
[2] Theodore Bapty, Sydney Whittington, James D. Walker, Joseph Hite, Brian Swenson, Kathrine Owens, Fred Eisele, Jason Scott, and Robert Owens. Design Oracle for AI-Based CPS Design. Design Automation for CPS and IoT (DESTION), 2022.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[4] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3D model repository. arXiv preprint arXiv:1512.03012, 2015.
[5] Adam Cobb, Anirban Roy, Daniel Elenius, Kaushik Koneripalli, and Susmit Jha. On diverse system-level design using manifold learning and partial simulated annealing. Proceedings of the Design Society, 2:1541-1548, 2022.
[6] Adam D Cobb, Anirban Roy, Daniel Elenius, and Susmit Jha. Design of unmanned air vehicles using transformer surrogate models. arXiv preprint arXiv:2211.08138, 2022.
[7] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructclip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500, 2023.
[8] Samyak Datta, Karan Sikka, Anirban Roy, Karuna Ahuja, Devi Parikh, and Ajay Divakaran. Align2ground: Weakly supervised phrase grounding guided by image-caption alignment. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2601-2610, 2019.
[26] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.
[27] Colleen Richey, Maria A. Barrios, Zeb Armstrong, Chris Bartels, Horacio Franco, Martin Graciarena, Aaron Lawson, Mahesh Kumar Nandwana, Allen Stauffer, Julien van Hout, Paul Gamble, Jeff Hetherly, Cory Stephenson, and Karl Ni. Voices obscured in complex environmental settings (voices) corpus, 2018.
[28] Ari Seff, Yaniv Ovadia, Wenda Zhou, and Ryan P Adams. Sketchgraphs: A large-scale dataset for modeling relational geometry in computer-aided design. arXiv preprint arXiv:2007.08506, 2020.
[29] Ari Seff, Wenda Zhou, Nick Richardson, and Ryan P Adams. Vitruvion: A Generative Model of Parametric CAD Sketches. arXiv preprint arXiv:2109.14124, 2021.
[30] Ashish Vavwani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[31] James D. Walker, F. Michael Heim, Bapiraju Surampudi, Pablo Bueno, Alexander Carpenter, Sidney Chocron, Jon Cutshall, Richard Lammons, Theodore B Pty, Brian Swenson, and Sydney Whittington. A Flight Dynamics Model for Exploring the Distributed Electrical eVTOL Cyber Physical Design Space. Design Automation for CPS and IoT (DESTION), 2022.
[32] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog), 38(5):1-12, 2019.
[33] Eamon Whalen, Azariah Beyene, and Caitlin Mueller. Simjeb: Simulated jet engine bracket dataset. In Computer Graphics Forum, volume 40, pages 9-17. Wiley Online Library, 2021.
[34] Karl DD Willis, Pradeep Kumar Jayaraman, Joseph G Lambourne, Hang Chu, and Yewen Pu. Engineering sketch generation for computer-aided design. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2105–2114, 2021.
[35] Rundi Wu, Chang Xiao, and Changxi Zheng. DeepCAD: A deep generative network for computer-aided design models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6772–6782, 2021.
[36] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaou Tang, and Jianxiong Xiao. 3D shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015.
A Data Card
In this section we include the data card that follows the structure in Pang and Lee [22] as highlighted in Gebru et al. [11].
A.1 Motivation
For what purpose was the dataset created? The dataset was created to enable research on designing cyber physical systems. In particular, we wanted to see if it was possible to predict characteristics of aircraft designs, and be able to filter out poor performing designs before needing to use expensive scientific models. In order to achieve this, we needed to build a large representative dataset to train our models.
Who created the dataset? The dataset was created by the authors: Adam D. Cobb, Anirban Roy, Daniel Elenius, and Susmit Jha, at SRI International; Brian Swenson, Sydney Whittington, and James Walker at Southwest Research Institute; Theodore Bapty, and Joseph Hite at Vanderbilt University; Karthik Ramani at Purdue University; Christopher McComb at Carnegie Mellon University; supported by DARPA under the Symbiotic Design for Cyber-Physical Systems program with contract FA8750-20-C-0002.
Who funded the creation of the dataset? DARPA under the Symbiotic Design for Cyber-Physical Systems program with contract FA8750-20-C-0002.
A.2 Composition
What do the instances that comprise the dataset represent? The instances are aerial vehicle designs as generated by us and evaluated on scientific models [2, 31]. We also provide metadata in the form of the corpus dictionary such that the attributes of the propellers, motors, batteries, and wings are available.
How many instances are there in total There are 27,714 instances in total.
Does the dataset contain all possible instances or is it a sample of instances from a larger set?
The dataset is a sample of instances that were generated according to our grammar. As our generator includes both discrete and continuous parameters, it is not possible to capture all possible instances of the designs. Furthermore, designs are allowed to contain recursively generated components, such as arms, such that all possible topologies cannot be sampled. The plots in Figure 4 show the diversity in designs.
What does each instance consist of? Each instance consists of a design tree, an interpreted design sequence, a low level design sequence, an STL file (3D model), a npy file containing an array of trims, a npy file containing a point cloud, and a JSON file reporting flight statistics. Please refer to 2
Is there a label or target associated with each instance? The labels are given via the output.json and trims.npy files. These files contain labels such as whether a design has interferences and if they are able to fly. It is also possible to use the design tree or design sequence to label the 3D data. For example the low level design sequence can be used to label how many propellers are in the aircraft design or to predict which battery has been selected based on the performance file.
Is any information missing from individual instances? Everything is included. No data is missing.
Are relationships between individual instances made explicit? Individual instances are sampled independently from our generator and are therefore not explicitly related. The corpus dictionary can also be used to determine the attributes of the available components where appropriate, where this corpus is relevant to all designs.
Are there recommended data splits (e.g., training, development/Validation, testing)? In the notebook code we provide our training, validation, and test splits but they are not explicitly recommended as the data has multiple use-cases.
Are there any errors, sources of noise, or redundancies in the dataset? The only errors come from the available fidelity of the scientific models. Please refer to [2, 3] for the limitations of the models. As a result, some of the outputs from the metadata contain NaNs. These are simple enough to filter out from the dataset where needed.
Is the dataset self-contained, or does it link to or otherwise rely on external resources? The dataset is entirely self-contained.
Does the dataset contain data that might be considered confidential? No.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? Not that the authors are aware of.
Does the dataset identify any subpopulations (e.g., by age, gender)? Not applicable.
Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? Not applicable.
Does the dataset contain data that might be considered sensitive in any way? No.
A.3 Collection Process
How was the data associated with each instance acquired? The data was generated by sampling our probabilistic design generator and then compiling each design into a format that was suitable for the scientific models. The scientific models provided performance metrics which we then curated into the output . json files and the STL files.
What mechanisms or procedures were used to collect the data? Each design took approximately 3 minutes to evaluate. As a result we estimate that it took almost 60 days of compute power to build AircraftVerse. As part of the pipeline, we required access to a CREO license [24].
If the dataset is a sample from a larger set, what was the sampling strategy? The dataset was sampled from our probabilistic design generator and was therefore a stochastic process.
Over what timeframe was the data collected? The data was generated using four compute nodes and therefore took over 2 weeks to generate. However, this was part of a process that took around two years of development to get to. Part of that process was building domain knowledge across the collaboration and building up the right compute architecture.
Were any ethical review processes conducted? Not applicable.
Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)? Not applicable.
Were the individuals in question notified about the data collection? Not applicable.
Did the individuals in question consent to the collection and use of their data? Not applicable.
If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? Not applicable.
Has an analysis of the potential impact of the dataset and its use on data subjects been conducted? Not applicable.
A.4 Preprocessing/cleaning/labeling
Was any preprocessing/cleaning/labeling of the data done? For the machine learning models, we processed the design trees into a unique key-value sequence as described in Section 4. As part of evaluating the performance of each design, we recorded the performance metrics in the output .json file and the trims .npy file. We also used a combination of data-driven heuristic approaches and hard rules to remove certain designs from the final dataset. We limited designs to a maximum of 12 wings and 16 propellers. We further added a data-driven preprocessing step to remove designs that would definitely not hover. We were careful to select a threshold that did not sacrifice diversity.
Was the "raw" data saved in addition to the preprocessed/cleaned/ labeled data Each design contains the raw data as well as the processed sequences. We consider the design trees, the STL files and the performance files to be raw data.
Is the software that was used to preprocess/clean/label the data available? The software required to label the data requires a CREO license along with proprietary software owned by Southwest Research Institute. It is possible that the latter software will be released in the future.
A.5 Uses
Has the dataset been used for any tasks already? At the time of publication, the only use-case of our dataset can be seen in Section 4.
Is there a repository that links to any or all papers or systems that use the dataset? Not applicable at this time as this is the first instance of the data release.
What (other) tasks could the dataset be used for? The dataset could be used for anything related to modeling or understanding the behavior of aircraft. For example one could envision learning rules for good designs, inverting the design by predicting sequences from point clouds etc... . Overall the AircraftVerse dataset contains multiple modalities and there are many opportunities to test new models out. We take some time in the paper to highlight this.
Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? The fidelity of the scientific models are the main factor for users of this dataset to consider when using this data set for future applications.
Are there tasks for which the dataset should not be used? Not that the authors are aware of.
A.6 Distribution
Will the dataset be distributed to third parties outside of the entity on behalf of which the dataset was created? Yes, the dataset is publicly available.
How will the dataset will be distributed? The dataset is hosted at https://zenodo.org/record/6525446, baseline models and code are at https://github.com/SRI-CSL/AircraftVerse, and the dataset description is at https://aircraftverse.onrender.com/.
When will the dataset be distributed? The dataset was first released in 2023.
Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? See Section B.
Have any third parties imposed IP-based or other restrictions on the data associated with the instances? No.
A.7 Maintenance
Who will be supporting/hosting/maintaining the dataset? The Neuro-Symbolic Computing and Intelligence (NuSCI) Research Group at SRI International will support and maintain the dataset.
How can the owner/curator/manager of the dataset be contacted? Please see the email addresses at the start of the paper.
Will the dataset be updated? We intend to continue to add to the dataset as we evaluate further designs.
If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? Please contact the authors in regards to ways that the dataset could be extended. For example running different scientific models may be possible.
B Licensing description
Our dataset uses a Creative Commons Attribution-ShareAlike (CC BY-SA) license. This means that the data can be shared and/or adapted for any purpose under the condition that appropriate credit is given, and that a link to the license is provided, and that users indicate if any changes were made. For more details on the license, please refer to https://creativecommons.org/licenses/by/4.0/.
C Flight dynamics model
In this section, we describe the physics of the flight dynamics model (FDM). The vehicle is equipped with five major components 1) propellers, 2) motors, 3) batteries as power source, 4) wings and stabilizers, 5) body structure such as a fuselage or a board where components are connected to. The FDM considers a six-degrees of freedom where the state of the vehicle at time  $t$  is defined as follows
$$
\boldsymbol {x} ^ {t} = (U, V, W, P, Q, R, \boldsymbol {q}, x, y, z, \Omega_ {i}, Q _ {j}). \tag {1}
$$