Fusion 360 Gallery: A Dataset and Environment for Programmatic CAD Reconstruction

1 INTRODUCTION . The manufactured objects that surround us in everyday life are created in computer-aided design ( CAD ) software using common modeling operations such as sketch and extrude . With just these two modeling operations , a highly expressive range of 3D designs can be created ( Figure 1 ) . Parametric CAD files contain construction sequence information that is critical for documenting design intent , maintaining editablity , and downstream simulation and manufacturing . Despite the value of this information , it is often lost due to data translation or error and must be reverse engineered from geometry or even raw 3D scan data . The task of reconstructing CAD operations from geometry has been pursued for over 40 years ( Shah et al. , 2001 ) and is available in commercial CAD software using heuristic approaches ( Autodesk , 2012 ; Dassault , 2019 ) . Recent advances in neural networks for 3D shape generation has spurred new interest in CAD reconstruction , due to the potential to generalize better and further automate this challenging problem . However , learning-based approaches to CAD reconstruction have not yet had access to a human-designed dataset of 3D CAD construction sequences , instead relying on synthetic data for both training and testing purposes , e.g . Li et al . ( 2020 ) . The absence of real world data has limited work on CAD reconstruction using common sketch and extrude modeling operations . Instead a focus has been on reconstruction from simple geometric primitives ( Sharma et al. , 2017 ; Tian et al. , 2019 ; Ellis et al. , 2019 ) that lack the rich parametric sketches commonly used in mechanical CAD ( e.g . Figure 2 ) . As there is no existing learning-based approach to reconstruct sketch and extrude sequences , we take a first step towards this goal by introducing data , a supporting software environment , and a novel action representation for reconstructing sketch and extrude designs . In this paper we present the Fusion 360 Gallery reconstruction dataset and environment for learning CAD reconstruction . The dataset contains 8,625 designs created by users of Autodesk Fusion 360 using a simple subset of CAD modeling operations : sketch and extrude . To the best of our knowledge this dataset is the first to provide human designed 3D CAD construction sequence data for use with machine learning . To support research with the dataset we provide an environment called the Fusion 360 Gym for working with CAD reconstruction . A key motivation of this work is to provide insights into the process of how people design objects . Furthermore , our goal is to provide a universal benchmark for research and evaluation of learning-based CAD reconstruction algorithms , bridging the gap between the computer graphics and machine learning community . To this end we describe a standard CAD reconstruction task and associated evaluation metrics with respect to the ground truth construction sequence . We also introduce a novel action representation for CAD reconstruction of sketch and extrude designs using neurally guided search . This search employs a policy , trained using imitation learning , consisting of a graph neural network encoding of CAD geometry . This paper makes the following contributions : • We present the Fusion 360 Gallery reconstruction dataset containing construction sequence information for 8,625 human-designed sketch and extrude CAD models . • We introduce an environment called the Fusion 360 Gym , standardizing the CAD reconstruction task in a Markov Decision Process formulation . • We introduce a novel action representation to enable neurally guided CAD reconstruction trained on real world construction sequences for the first time . 2 RELATED WORK . CAD Datasets Existing 3D CAD datasets have largely focused on providing mesh geometry ( Chang et al. , 2015 ; Wu et al. , 2015 ; Zhou & Jacobson , 2016 ; Mo et al. , 2019b ; Kim et al. , 2020 ) . However , the de facto standard for parametric CAD is the boundary representation ( B-Rep ) format , containing valuable analytic representations of surfaces and curves suitable for high level control of 3D shapes . B-Reps are collections of trimmed parametric surfaces along with topological information which describes adjacency relationships and the ordering of elements such as faces , loops , edges , and vertices ( Weiler , 1986 ) . B-Rep datasets have recently been made available with both human-designed ( Koch et al. , 2019 ) and synthetic data ( Zhang et al. , 2018 ; Jayaraman et al. , 2020 ; Starly , 2020 ) . Missing from these datasets is construction sequence information . We believe it is critical to understand not only what is designed , but how that design came about . Parametric CAD files contain valuable information on the construction history of a design . Schulz et al . ( 2014 ) provide a standard collection of human designs with full parametric history , albeit a limited set of 67 designs in a proprietary format . SketchGraphs ( Seff et al. , 2020 ) constrains the broad area of parametric CAD by focusing on the underlying 2D engineering sketches , including sketch construction sequences . In the absence of 3D human design data , learning-based approaches have instead leveraged synthetic CAD construction sequences ( Sharma et al. , 2017 ; Li et al. , 2020 ) . The dataset presented in this paper is , to the best of our knowledge , the first to provide humandesigned 3D CAD construction sequence information suitable for use with machine learning . CAD Reconstruction The task of CAD reconstruction involves recovering the sequence of modeling operations used to construct a CAD model from geometry input , such as B-reps , triangle meshes , or point clouds . Despite extensive prior work ( Shah et al. , 2001 ) , CAD reconstruction remains a challenging problem as it requires deductions on both continuous parameters ( e.g. , ex- tracting the dimensions of primitives ) and discrete operations ( e.g. , choosing a proper operation for the next step ) , leading to a mixed combinatorial search space . To recover the sequence of operations , traditional methods typically run global search methods ( e.g. , evolutionary algorithms as in Hamza & Saitou ( 2004 ) , Weiss ( 2009 ) , Friedrich et al . ( 2019 ) , and Fayolle & Pasko ( 2016 ) ) with heuristic rules to prune the search space ( Shapiro & Vossler , 1993 ; Buchele , 2000 ; Buchele & Roles , 2001 ; Buchele & Crawford , 2003 ) . Heuristic approaches are also available in a number of commercial software tools , often as a user-guided semi-automatic system ( Autodesk , 2012 ; Dassault , 2019 ) to aid with file conversion between CAD systems . These traditional algorithms operate by removing faces from the B-rep body and reapplying them as parametric modeling operations . This strategy can recover the later modeling operations , but fail to completely rebuild the construction sequence from the first step . We instead tackle the task of recovering the entire construction sequence from the first extrusion . Another approach is using program synthesis ( Du et al. , 2018 ; Nandi et al. , 2017 ; 2018 ; 2020 ) to infer CAD programs written in domain specific languages from given shapes . CAD reconstruction is also related to the inverse procedural modeling problem ( Talton et al. , 2011 ; Stava et al. , 2014 ; Vanegas et al. , 2012 ) , which attempts to reverse-engineer procedures that can faithfully match a given target . Inverse procedural modeling methods typically work with synthetic data , while our paper tackles tasks on real CAD models and modeling operations . Compared to the rule-based or grammar-based methods above , learning-based approaches can potentially learn the types of rules that are typically hard-coded , automate scenarios that require userinput , and generalize when confronted with unfamiliar geometry . One of the earliest such works is CSGNet ( Sharma et al. , 2017 ) , which trains a neural network to infer the sequence of Constructive Solid Geometry ( CSG ) operations based on visual inputs . More recent works along this line of research include Ellis et al . ( 2019 ) , Tian et al . ( 2019 ) , and Kania et al . ( 2020 ) . Typically associated with these methods are a customized , domain specific language ( e.g. , CSG ) that parameterizes the space of geometry , some heuristic rules that limit the search space , and a neural network generative model ( Zou et al. , 2017 ; Mo et al. , 2019a ; Chen et al. , 2020 ; Jones et al. , 2020 ) . Lin et al . ( 2020 ) reconstruct input shapes with a dual action representation that first positions cuboids and then edits edge-loops for refinement . Although editing edge-loops of cuboids may be a suitable modeling operation in artistic design , it is not as expressive or precise as the sketch and extrude operations used in real mechanical components . Additionally , Lin et al . ( 2020 ) chooses to train and evaluate their network on synthetic data due to the lack of a benchmark dataset of CAD construction sequences , a space that our work aims to fill . Our approach is , to the best of our knowledge , the first to apply a learning-based method to reconstruction using common sketch and extrude CAD modeling operations from real human designs . 3 FUSION 360 GALLERY RECONSTRUCTION DATASET . The Fusion 360 Gallery reconstruction dataset1 is produced from designs submitted by users of the CAD software Autodesk Fusion 360 . The dataset contains CAD construction sequence information from a subset of sketch and extrude designs . We intentionally limit the data to the sketch and extrude modeling operations to reduce the complexity of the CAD reconstruction task . Figure 1 shows a random sampling of the designs in the dataset . Each design is provided in three different representations : B-Rep , mesh , and construction sequence JSON text format . An official 80:20 train- 1https : //github.com/xxx/xxx test split is provided with 6,900 and 1,725 designs respectively . We now briefly outline the sketch and extrude modeling operations and provide additional details in Section A.1 of the appendix . Sketch Unlike free-form sketches , sketches in CAD are composed of 2D geometric primitives ( lines , circles , splines etc . ) , associated dimensions ( distance , diameter , angle etc . ) and constraints ( symmetry , tangent , parallel etc. ) . Sketch geometry is represented by points , that create curves , that in turn form loops within profiles . The intersection of curves , as the user draws , automatically creates closed loops and profiles that are serialized as both raw curves and trimmed profiles . Sketch profiles form the basis for 3D extrusion as shown in ( Figure 2 ) . Extrude An extrude operation takes one or more sketch profiles and extrudes them from 2D into a 3D B-Rep body . A distance parameter defines how far the profile is extruded . A notable feature of extrude operations in Fusion 360 is the ability to perform Boolean operations in the same step . As a user extrudes a sketch profile , they choose to create a new body , join , cut , or intersect with other bodies in the design ( Figure 2 ) . Additional extrude options are available such as two-sided extrude , symmetrical extrude , and tapered extrude . The combination of expressive sketches and extrude operations with built in Boolean capability enables a wide variety of designs to be constructed . 4 FUSION 360 GYM . Together with the dataset we provide an open source environment , called the Fusion 360 Gym , for standardizing the CAD reconstruction task . The Fusion 360 Gym wraps the underlying Fusion 360 Python API ( Autodesk , 2014 ) and serves as the environment that interacts with an intelligent agent for the task of CAD reconstruction ( Figure 3 ) . Specifically , the Fusion 360 Gym formalizes the following Markov Decision Process : • state : Contains the current geometry , and optionally , the target geometry to be reconstructed . We use a B-Rep face-adjacency graph as our state representation . • action : A modeling operation that allows the agent to modify the current geometry . We consider two action representations : sketch extrusion and face extrusion . • transition : Fusion 360 Gym implements the transition function that applies the modeling operation to the current geometry . • reward : The user can define custom reward functions depending on the task . We provide intersection over union ( IoU ) as one measure to compare the current and target geometry .

Summary: The paper describes a new dataset of 3D geometry construction sequences based on sequential sketching (i.e. sets of two-dimensional curves) and extruding (i.e. axes, angles and distances for extrusion profiles from sketches) combined with Boolean solid geometry operations. The dataset comprises the resulting objects as well as the human designed sketch-extrude sequence that lead to the object geometry. The 8625 objects are available in three different geometry representations (boundary representation, mesh, and construction sequence in structured format). Furthermore the submission describes a new infrastructure ("gym") to train and evaluate algorithms that estimate such construction sequences, as well as evaluation metrics to gauge the efficiency and effectiveness of such estimation algorithms. The paper also provides a reference method for sequence estimation based on imitation learning as well as several baselines and their empirical evaluation with respect to the posed benchmarks.