Fusion 360 Gallery: A Dataset and Environment for Programmatic CAD Reconstruction
Authors: Karl Willis
Yewen Pu
Jieliang Luo
Hang Chu
Tao Du
Joseph Lambourne
Armando Solar-Lezama
Wojciech Matusik
Venue:  (2021)
Keywords: CAD
dataset
3D
reconstruction
environment
design
sequence

Abstract: Parametric computer-aided design (CAD) is a standard paradigm used for the design of manufactured objects. CAD designers perform modeling operations, such as sketch and extrude, to form a construction sequence that makes up a final design. Despite the pervasiveness of parametric CAD and growing interest from the research community, a dataset of human designed 3D CAD construction sequences has not been available to-date. In this paper we present the Fusion 360 Gallery reconstruction dataset and environment for learning CAD reconstruction. We provide a dataset of 8,625 designs, comprising sequential sketch and extrude modeling operations, together with a complementary environment called the Fusion 360 Gym, to assist with performing CAD reconstruction. We outline a standard CAD reconstruction task, together with evaluation metrics, and present results from a novel method using neurally guided search to recover a construction sequence from a target geometry.

Under review as a conference paper at ICLR 2021
FUSION 360 GALLERY: A DATASET AND ENVIRON- MENT FOR PROGRAMMATIC CAD RECONSTRUCTION
Anonymous authors Paper under double-blind review
ABSTRACT
Parametric computer-aided design (CAD) is standard paradigm used for the de- sign of manufactured objects. CAD designers perform modeling operations, such as sketch and extrude, to form a construction sequence that makes up a final de- sign. Despite the pervasiveness of parametric CAD and growing interest from the research community, a dataset of human designed 3D CAD construction se- quences has not been available to-date. In this paper we present the Fusion 360 Gallery reconstruction dataset and environment for learning CAD reconstruction. We provide dataset 8,625 designs, comprising sequential sketch and extrude modeling operations, together with a complementary environment called the Fu- sion 360 Gym, to assist with performing CAD reconstruction. We outline a stan- dard CAD reconstruction task, together with evaluation metrics, and present re- sults from novel method using neurally guided search to recover construction sequence from target geometry.
INTRODUCTION
The manufactured objects that surround us in everyday life are created computer-aided design (CAD) software using common modeling operations such as sketch and extrude. With just these two modeling operations, a highly expressive range 3D designs can be created (Figure [ Para- metric CAD files contain construction sequence information that is critical for documenting design intent, maintaining editablity, and downstream simulation and manufacturing. Despite the value of this information, it is often lost due to data translation error and must be reverse engineered from geometry or even raw 3D scan data. The task of reconstructing CAD operations from geometry has been pursued for over 40 years (Shah al. 2001) and is available in commercial CAD software using heuristic approaches (Autodesk) 2012
Dassault 2019). Recent advances in neural networks for 3D shape generation has spurred new interest CAD reconstruction, due to the potential to gen- eralize better and further automate this challenging problem. However, learning hamning-based approaches
Sketch 1
Extrude 1
Sketch
Extrude 2
Extrude
Figure Top: subset of designs containing 3D CAD construction sequences from the Fusion 360 Gallery reconstruction dataset. Bottom: An example construction sequence using sketch and extrude modeling operations.

---

Under review as a conference paper at ICLR 2021
to CAD reconstruction have not yet had access to human-designed dataset of 3D CAD construc- tion sequences, instead relying on synthetic data for both training and testing purposes, c.g. Li et al. 2020). The absence of real world data has limited work on CAD reconstruction using common sketch and extrude modeling operations. Instead a focus has been on reconstruction from simple geometric primitives (Sharma et al. 2017; Tian et al. 2019 Ellis et al. 2019) that lack the rich parametric sketches commonly used in mechanical CAD (c.g. Figure 2 As there is no existing learning-based approach reconstruct sketch and extrude sequences, take first step towards this goal by introducing data, a supporting software environment, and a novel action representation for reconstructing sketch and extrude designs.
In this paper we present the Fusion 360 Gallery reconstruction dataset and environment for learning CAD reconstruction. The dataset contains 8,625 designs created by users of Autodesk Fusion 360 using a simple subset of CAD modeling operations: sketch and extrude. To the best of our knowledge this dataset is the first to provide human designed 3D CAD construction sequence data for use with machine learning. To support research with the dataset we provide environment called the Fusion 360 Gym for working with CAD reconstruction. A key motivation of this work is to provide insights into the process of how people design objects. Furthermore, our goal is provide universal benchmark for research and evaluation of learning-base the gap between the computer graphies and machine learning community. To this end we describe standard CAD reconstruction task and associated evaluation metrics with respect to the ground truth construction sequence. We also introduce novel action representation for CAD reconstruction of sketch and extrude designs using neurally guided search. This search employs a policy, trained using imitation learning, consisting of graph neural network encoding of CAD geometry.
CAD reconstruction algorithms, bridging
This paper makes the following contributions:
We present the Fusion 360 Gallery reconstruction dataset containing construction sequence information for 8,625 human-designed sketch and extrude CAD models.
We introduce an environment called the Fusion 360 Gym, standardizing the CAD reconstruc- tion task in a Markov Decision Process formulation.
We introduce a novel action representation enable neurally guided CAD reconstruction trained on real world construction sequences for the first time.
2 RELATED WORK
CAD Datasets Existing 3D CAD datasets have largely focused on providing mesh geometry
Chang et al.) 2015 Wu et al., 2015 Zhou & Jacobson 2016 Mo al. [019b] Kim et al. 2020). However, the de facto standard for parametric CAD is the boundary representation (B-Rep) format, containing valuable analytic representations of surfaces and curves suitable for high level control of 3D shapes. B-Reps are collections of trimmed parametric surfaces along with topological infor- mation which describes adjacency relationships and the ordering of elements such as faces, loops, edges, and vertices (Weiler) 1986). B-Rep datasets have recently been made available with both human- designd (Koch et al. 2019) and synthetic data (Zhang et al.) 2018: Jayaraman et al. 2020) Starly 2020). Missing from these datasets is construction sequence information. We believe it is critical to understand not only what is designed, but how that design came about.
Parametric CAD files contain valuable information on the construction history of design. Schulz ett al.] (2014) provide a standard collection of human designs with full parametric history, albeit a limited of 67 designs in proprietary format. SketchGraphs (Seff al.] 2020) constrains the broad area of parametric CAD by focusing on the underlying 2D engineering sketches, including
sketch construction sequences. In the absence of 3D human design data, learning based approaches
have instead leveraged synthetic CAD construction sequences (Sharma et al. [017] Li et al. 22220).
The dataset presented in this paper is, the best of our knowledge, the first to provide human- designed 3D CAD construction sequence information suitable for use with machine learning.
CAD Reconstruction The task of CAD reconstruction involves recovering the sequence of mod- eling operations used to construct CAD model from geometry input, such as B-reps, triangle meshes, or point clouds. Despite extensive prior work (Shah et al. (001). CAD reconstruction remains challenging problem it requires deductions on both continuous parameters (e.g., ex-
2

---

Under review as a conference paper at ICLR 2021
Sketch
Extrude 1
Extrude
Sketch 2
Extrude 3
An example design sequence from the Fusion 360 Gallery reconstruction dataset. Sketch profiles are sequentially extruded to join (Extrude 1, Extrude 2) or cut (Extrude 3) geometry using Boolean operations. The colored areas show the sketch profiles that partake in each extrusion.
Figure 2:
tracting the dimensions of primitives) and discrete operations (e.g., choosing a proper operation for the next step), leading a mixed combinatorial search space. To recover the sequence of operations. traditional methods typically run global search methods (e.g., evolutioony algorithms as in Hamza
Saitou (2004), Weiss| (2009), Friedrich et al. (2019), and Fayolle Pasko (2016) with heuristic rules to prune the search space (Shapiro & Vossler, 1993] Buchele 2000 Buchele & Roles 2001]
BBuchele Crawford 2003). Heuristic approaches are also available a number of commercial software tools, often as user-guided semi-automatic system (Autodesk) 2012 Dassault, 2019) to
aid with file conversion between CAD systems. These traditional algorithms operate by removing faces from the B-rep body and reapplying them parametric modeling operations. This strategy can recover the later modeling operations, but fail to completely rebuild the construction sequence from the first step. We instead tackle the task of recovering the entire construction sequence from the first extrusion. Another approach is using program synthesis (Du et (0188) Nandi et al. 2017] 2018 2020) to infer CAD programs written in domain specific languages from given shapes. CAD reconstruction is also related to the inverse procedural modeling problem (Talton et al.) 2011 Stava al. 2014, Vanegas et al.) 2012). which attempts reverse-engineer procedures that can faithfully match a given target. Inverse procedural modeling methods typically work with synthetic data, while our paper tackles tasks on real CAD models and modeling operations.
Compared to the rule-based or grammar-based methods above, learning-basse based approaches can po- tentially learn the types of rules that are typically hard-coded, automate scenarios that require user- input, and generalize when confronted with unfamiliar geometry. One of the carliest such works is CSGNet (Sharma et al., 2017), which trains a neural network to infer the sequence of Constructive Solid Geometry (CSG) operations based on visual inputs. More recent works along this line of re- search include Ellis et al. (2019). Tian et al. (2019), and Kania et al. 2200)). Typically associated with these methods are customized, domain specific language (e.g.. CSG) that parameterizes the space of geometry, some heuristic rules that limit the search space, and a neural network generative model (Zou et al.) 2017 Mo et al., 2019a Chen al.) 2020 Jones et al., (020). Lin al. (2020) reconstruct input shapes with a dual action representation that first positions cuboids and then edits edge-loops for refinement, A Although editing edge-loops of cuboids may be suitable modeling op- eration artistic design, it is not as expressive or precise as the sketch and extrude operations used in real mechanical components. Additionally, [in et al. (2020) chooses to train and evaluate their network on synthetic data due to the lack of benchmark dataset of CAD construction sequences, a space that our work aims fill. Our approach is, to the best of our knowledge, the first to ap- ply a learning- beasning-based method to reconstruction using common sketch and extrude CAD modeling operations from real human designs.
FUSION 360 GALLERY RECONSTRUCTION DATASET
The Fusion 360 Gallery reconstruction datase is produced from designs submitted by users of the CAD software Autodesk Fusion 360. The dataset contains CAD construction sequence information from a subset of sketch and extrude designs. We intentionally limit the data to the sketch and extrude modeling operations to reduce the complexity of the CAD reconstruction task. Figure shows a random sampling of the designs in the dataset. Each design is provided in three different representations: R-Rep, mesh, and construction sequence JSON text format. An official 80:20 train-
https//ggthu..co//xs//xxx

---

Under review as a conference paper at ICLR 2021
REWARD
Facs Adjacency Grape Shate Representation
ENVIRONMENT
AGENT
Current Goometry
Target Geometry
ACTION
Modeling Operation
Figure 3: The Fusion 360 Gym interacts with agent in a sequential decision making scenario (left) with the state containing geometries represented as face adjacency graphs (right).
test split provided with 6,900 and 1,725 designs respectively. We now briefly outline the sketch and extrude modeling operations and provide additional details in Section A. of the appendix.
Sketch Unlike free-form sketches, sketches in CAD are composed of 2D geometric primitives (lines, circles, splines etc.), associated dimensions (distance, diameter, angle etc.) and constraints (symmetry, tangent, parallel etc.). Sketch geometry is represented by points, that create curves, that in turn form loops within profiles. The intersection of curves, as the user draws, automatically creates closed loops and profiles that are serialized as both raw curves and trimmed profiles. Sketch profiles form the basis for 3D extrusion as shown in (Figure
Extrude An extrude operation takes one or more sketch profiles and extrudes them from 2D into 3D B-Rep body. A distance parameter defines how far the profile is extruded. A notable feature of extrude operations Fusion 360 is the ability to perform Boolean operations in the same step. As extrudes sketch profile, they choose create bodies in the design (Figure 2 Additional extrude options are available such as two-sided extrude, symmetrical extrude, and tapered extrude. The combination of expressive sketches and extrude operations with built in Boolean capability enables wide variety of designs to be constructed.
join, cul, or intersect with other
FUSION 360 GYM
Together with the dataset we provide an open source environment, called the Fusion 360 Gym, for standardizing the CAD reconstruction task. The Fusion 360 Gym wraps the underlying Fusion 360 Python API (Autodesk] 2014) and serves as the environment that interacts with an intelligent agent for the task of CAD reconstruction (Figure 3p. Specifically, the Fusion 360 Cym formalizes the following Markov Decision Process:
state: Contains the current geometry, and optionally, the target geometry to be reconstructed. We use a B- B-Rep face-adjacency graph as our state representation.
action: A modeling operation that allows the agent to modify the current geometry. We con- sider two action representations: sketch extrusion and face extrusion.
transition: Fusion 360 Gym implements the transition function that applies the modeling op- eration to the current geometry.
reward: The user can define custom reward functions depending on the task. We provide intersection over union (loU) as measure compare the current and target geometry.
4.1 STATE REPRESENTATION
In order for an agent to successfully reconstruct the target geometry, it is important that we have suitable state representation. In Fusion 360 Gym, similar encoding scheme to Dayara- man et al. (2020) and represent the current and target geometry with a B-Rep face-adjacency graph Ansaldi et al. 1985), illustrated in Figure right. Crucial to this encoding are the geomefric

---

Under review as a conference paper at ICLR 2021
FACE EXTRUSION SKETCH EXTRUSION
Q d
Facel Extrude
1
Face Extrude
Figure 4: Action representations supported by the Fusion 360 Gym include low-level sketch extru- sion (top) and simplified face extrusion (bottom).
features of the elements, such as point-locations, and lopological features specifying how these el- ements are connected each other. Specifically, the vertices of the face-adjacency graph represent B-Rep faces (trimmed parametric surfaces) in the design, with graph vertex features representing the size, orientation, and curvature of the faces. The edges of the face-adjacency graph represents B-Rep edges in the design, that connect the adjacent B- R-Rep faces to each other.
4.2 ACTION REPRESENTATION
In the Fusion 360 Gym we support two action representations encompassing different modeling operations: sketch extrusion face extrusion.
Sketch Extrusion sketch extrusion, the agent must first select sketch plane, draw on this plane using sequence of curve primitives, such as lines and ares, form closed loop profiles. The agent then selects a profile to extrude a given distance and direction (Figure top). Using this representa- tion it is possible to construct novel geometries by generating the underlying sketch primitives and extruding them by an arbitrary amount. Although all designs in the Fusion 360 Gallery reconstruc- tion dataset be constructed using sketch extrusion, practice this challenging. Benko et al. (2002) show that to generate sketches suitable for mechanical engineering parts, the curve primitives often need be constructed alongside set of constraints which enforce regularities and symmetries in the design. Although the construction of the constraint graphs is feasible using techniques like the one shown by [iao al. 2019),, enforcing the constraints requires a complex interaction between the machine learning algorithm and suitable geometric constraint solver, greatly increasing the algorithm complexity. We alleviate this problem by introducing a simplified action representation, called face extrusion, that is well suited to learning-base based approaches.
Face Extrusion In face extrusion, face from the target design is used as the extrusion profile rather than sketch profile (Figure bottom). This is possible because the target design is known in advance during reconstruction. An action a in this scheme a triple {facestert, faceend, op} where the start and end faces parallel faces referenced from the target geometry, and the operation type one of the following: new body, join, cut, intersect. Target constrained reconstruction using face extrusion has the benefit of narrowly scoping the prediction problem with shorter action sequences and simpler actions. Conversely, not all geometries can be reconstructed with this simplified strategy due to insufficient information in the target, e.g., Extrude without leaving a start or end face. Of the design sequences in the reconstruction dataset, 59,28 59.2% can be directly converted face extrusion sequence. We estimate that approximately 80% of designs in our dataset can be reconstructed by finding alternative construction sequences.
Figure cuts across the entire design
4.3 SYNTHETIC DATA GENERATION
The Fusion 360 Gym supports generation of semi-synthetic data by taking existing designs and modifying recombining them. For instance, trusion distances, and even 'graft' sketches from one design onto another. We also support distri- bution matching of parameters, such the number of faces, ensure that synthetic designs match
randomly perturb the sketches and the

---

Under review as a conference paper at ICLR 2021
STATE
AGENT
ACTION
h
body
john
MPN
MLP
Inderannt
face nut Sace
MA
MPN
Σ
Figure 5: Given state comprising the target geometry G, and current geometry Ges the agent uses message passing network (MPN) to predict an action as a face extrusion modeling operation.
human-designed dataset distribution. Learning-based systems can leverage semi-synthetic data to expand the number of samples in the Fusion 360 Gallery reconstruction dataset. We provide examples of synthetic data and commands for the Fusion 360 Gym Section A.2 the appendix.
CAD RECONSTRUCTION
5.1
TASK
The goal of CAD reconstruction is to recover the sequence of modeling operations used to construct CAD model with only the geometry input. This task be specified using different input geometry representations, including B-Rep, mesh, or point cloud, with progressively lower fidelity. Each representation presents realistie scenario where parametric CAD information absent and needs to be recovered. Given a target geometry GG, we wish to find sequence of CAD modeling operations (actions) A {ao, ai...... } that generates an output geometry G, such that every point in space is in its interior, if and only if, it is also in the interior of Gt.
Evaluation Metrics We prescribe three evaluation metrics, IoU, exact reconstruction. and con- ciseness. IoU measures the intersection over union of G and G₁: iou(G, - G As multiple correct sequences of CAD modeling operations exist, proposed reconstruction sequence A need not match the ground truth sequence Ar provided exact reconstruction is found. To measure the quality of exact recon- structions consider the conciseness of the construction sequence. Let conciseness(A., [A], where score < indicates the agent found exact reconstruction with equal fewer steps than the ground truth, and score >> 1 indicates more inefficient exact reconstructions.
|GnGgl/IGu
Exact reconstruction measures whether iou(G, G2)
5.2 METHOD
We now present method for CAD reconstruction using neurally-guided search (Ellis et al. 2019 Kalyan et al. 2018 Tang et al. 2019 Devlin et al. 2017)) from B-Rep input using face extrusion modeling operations. Rather than discovering sequence of construction exploration, the agent is trained to match known reconstruction sequences present in the training set using imitation learning. We leverage search inference time recover the given target geometry.
Imitation Learning To perform imitation learning, we leverage the fact that we have the ground truth sequence of modeling operations (actions) Ar = {ār.o .tr.po... â., for each design G, in the dataset. We feed the ground truth action sequence Ad into the Fusion 360 Gym, starting from the empty geometry Go. and output a sequence partial constructions Gt ... Gt where Gt,z, = G,- We then collect the supervised dataset D {Goo, G,) + âg (Gr,1, Gt) a supervised agent na that takes the pair current-target constructions (G,, G,) to a modeling operation action de which would transform the current geometry closer the target. Formally,
0.,1-.. } and train

---

Under review as a conference paper at ICLR 2021
optimize the expected oggliikelhood of correct actions under the data distribution:
GiGDDG.)DD log no (āel(Ges (G)GG.GG))) G
The agent takes pair of geometries (Ge, G.) state, and outputs the corresponding face- extrusion action = {facestert: facCend: op) (Figure 5 The two geometries Ge, G2 are given using face-adjacency graph similar to Dayaraman et al. (020), where the graph vertexes represent the faces of the geometry, with vertex features calculated from each face: 10x 10 grid of 3D points, normals, and trimming mask, in addition the face surface type. The edges define connectivity of adjacent faces. Inputs are encoded using two separate message passing networks (MPN) aggregating messages along the edges of the graph. The encoded vectors representing the current geometry are summed together (he in F)g and concatenated with the encoded vertexes of the target geometry (hg- ... h.). The concatenated vectors are used output the action using multi-layer perceptron (MLP), with the end face conditioned on the vertex embedding of the predicted start face.
Agent
Given neural agent To To(a](G, G.)) capable of furthering a current geometry toward the target geometry, we amplify its performance at test time using search. Here report the result of the most basic of search strategies, random rollout, and provide results from additional search strategies in Section A.3 4.3)o of the appendix. We let the agent interact with the environment by Le. sampling a sequence of actions according to me up to a fixed rollout length of max 2; 2), where fp is the number of planar faces in G,. If the agent is successful at reconstructing the target, stop. Otherwise, we repeat the process until we exhaust a global search budget.
Search
5.3 RESULTS
We now present results on CAD reconstruction using the test set of the Fusion 360 Gallery recon- struction dataset. We seek to understand 1) how state-of-the-an baseline networks perform on the CAD reconstruction task, 2) how synthetic data performs compared human designs. In each ex- periment we use different agents to reconstruct target design, while holding the search strategy constant. For a target design G., each agent uses the random rollout search algorithm and attempts reconstruction over multiple rollouts. Each time the agent takes action during search (a search step), we track the best IoU the agent has discovered so far, and whether exact reconstruction is achieved. We cap the total search budget to 100 steps.
Baseline Comparison We evaluate five different agents to understand how state-of-the art base- line networks perform on the CAD reconstruction task. The rand agent uniformly samples from the available actions to serve as a naive baseline without any learning. mlp is simple MLP agent that does not take advantage of shape topology. gen, gin, and gat are MPN agents that use a Graph Convolution Network (Kipf & Welling 2016), Graph Isomorphism Network (Xu et al. 2018). and Graph Attention Network (Veličković et al. 2017) respectively. We use two MPN layers for all comparisons, with standard layer settings as described in the appendix. We report the evaluation
Reconstruction ToU
Cumulative Exact Reconstructions
Ground Truth
1.0
1.0
Exoet Recostructionn 36
gin
Estimated Representation Upper Lima
0.81
mdp
0.8
rind
Tat
0.8
0.5
0.5
0.4
0.4
0.3
Search Step
Search Step
Figure 6: Reconstruction results over 100 search steps using random rollouts with different agents. For exact reconstructions, 0.8 is the estimated upper limit of the face extrusion action representation.
7

---

Under review as a conference paper at ICLR 2021
Agent
IoU
Exact Reconstruction
Conciseness
Parameters
Steps
100 Steps
Steps
100 Steps
gat 0.8742
0.9128
0.6191
0.6742
1.0206
3.03M
gen 0.8644
0,9042
0.6232
0.6754
1.0168
3.02M
gin 0,8346
0.8761
0,5901
0.6301
1.0042
3.62M
mlp 0.8274
0.8596
0.5658
0.5965
0.9763
2.24M
rand
0.6840
0.8386
0.4157
0.5380
1.2824
Table 1: Reconstruction results for IoU and exact reconstruction at 20 and 100 search steps using random rollouts with different agents. Lower values are better for conciseness.
metries of each agent as a function of the number of steps in Figure We detail the exact results at step 20 and 100 in Table Step 20 represents the point where it is possible perform exact recon- structions for all designs in the test set. We also detail the conciseness of the recovered sequence for exact reconstructions. We note that all neurally guided agents outperform the random agent baseline. The topology information available with a MPN is found to improve reconstruction performance. The gat and gen agents show the best performance but fall well short of exact reconstruction all designs in the test set, demonstrating that the CAD reconstruction task is non-trivial and an open problem for future research.
Synthetic Data Performance We evaluate four gen agents trained different data sources to understand how synthetic data performs compared to human design data. real is trained on the human design training set. syn is trained synthetic data from procedurally generated sketches of rectangles and circles extruded randomly (Figure D. left). semi-syn trained on semi-synthetic designs that use existing sketches in the training set with two more extrude operations match the distribution of the number of faces in the training set (Figure 8 right). aug is trained on the human design training set mixed with additional semi-synthetic data. We hold the training data quantity constant across agents, with the exception of the aug agent that contains larger quantity from two sources. All agents are evaluated on the human design test set.
Figure 7 shows that training on human design data offers significant advantage over synthetic and semi-synthetic data. For the aug agent reconstruction performance aided early on by data augmentation. This is likely due to semi-synthetic designs with 1 or 2 extrusions appearing similar to human designs. Conversely, semi-synthetic designs with multiple randomly applied extrusions appear less and less similar to human design. This difference in distribution between human and synthetic designs becomes more prevalent as search progresses and adversely affects performance.
Qualitative Results Figure 9 shows ground truth construction sequences compared with other agents using random search. The rollout with the highest IoU is shown with the IoU score and total search steps taken. Steps that don't change the geometry or occur after the highest IoU are omitted from the visualization.
Reconstruction ToU
Cumulative Reconstructions
Ground Truth
0.9
0.9
semi-syn
Recontrucciion
Fatimated Represntation Upper Limit
syn
100
0.7
07
0.6
0.6
0.8
0.5
0.6
0.3
20
40
60
80
100
40
60
100
Search Step
Search Step
Figure 7: Reconstruction results over 100 search steps using random rollouts and gen agents trained on human-dess -dennndd data (real), mixture human- -deinned and semi- sami-synthetic data (aug), semi- synthetic data (semi-syn), and synthetic data (syn).
8

---

Under review as a conference paper at ICLR 2021
Figure 8: Example synthetic (left) and semi-synthetic data (right).
Discussion For practical application of CAD reconstruction it is necessary have an exact re- construction where all details of design are reconstructed in a concise way. It is notable that incorrect reconstructions can score well with the IoU metric, but omit important design details. We therefore suggest IoU should be secondary metric, with future work focusing on improving ex- act reconstruction performance with concise construction sequences. Conciseness should always be considered alongside exact reconstruction performance as naive approaches that only reconstruct short sequences can achieve good conciseness scores.
CONCLUSION AND FUTURE DIRECTIONS
In this paper we presented the Fusion 360 Gallery reconstruction dataset and environment for learn- ing CAD reconstruction from sequential 3D CAD data. We outlined a standard CAD reconstruction task, together with evaluation metrics, and presented results from neurally guided search approach. We envision a number of future directions that could leverage the reconstruction dataset: new repre- sentations for sequential geometry capable of performing CAD reconstruction and generation from B-Rep, mesh, point cloud, or image data; reinforcement learning approaches that mimic and im- provise sequential modeling operations; and sketch and constraint synthesis from 3D geometry or images. Finally, beyond the simplified design space of sketch and extrude lies the full breadth of rich sequential CAD modeling operations.
A
gt
Steps
Steps
aug
1.00
0.85
2
gon
1.00
0.99
8
mlp
0.93
rand
0.61
99
0.90 7
C
loU Steps
D
IeU Steps
aug
1.00
0.37 3
gon
1.00
mlp
0.93
0.31 31
rand
0.83
0.77 38
Figure 9: Qualitative construction sequence results comparing the ground truth (gt) to reconstruc- tions using different agents with random search.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Silvia Ansaldi, Leila De Floriani, and Bianca Falcidieno. Geometric modeling of solid objects by using a face adjacency graph representation. ACM SIGGRAPH Computer Graphics, 19(3):331- 139, 1985.
Autodesk. Inventor Feature Recognition, 2012. URL http::/aaps..utooeeek com/
URL Autodesk. Fusion AutodeskFFusion 7B http://2222uu1117333300-----666-
Pal Benko, Geza Kos, Pál Benkó, Laszlo Andor, Géza Kós, Tamas Varady, László Andor, and Ralph Martin. Constrained fitting in reverse engineering. 2002.
Suzanne F Buchele and Richard H Crawford. Three-dimensiona, halfspace constructive solid geom- etry tree construction from implicit boundary representations. In Proceedings of the eighth ACM symposium on Solid modeling and applications, pp. 135-144, 2003.
Suzanne F Buchele and Angela C Roles. Binary space partitioning and constructive solid ge- ometry representations for objects bounded by curved surfaces. In CCCG, pp. 49-52. Citeseer, 2001.
Suzanne Fox Buchele. Three-dimensional binary space partitioning tree and constructive solid ge- ometry tree construction from algebraic boundary representations. 2000.
Angel Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-ricl 3d model repository. arXiv preprint arXiv:1512.03012. 2015.
Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. Bsp-net: Generating compact meshes via binary space partitioning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 45-54, 2020.
Dassault. Solidworks FeatureWorkss 2019. URL htL_ittss//eell.sooocooozss htw.aa///oolllloook//ooooos///// Overview of Featureworks htmm
Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy i/o. arXiv preprint 1rXiv:1703.07469 2017.
Tao Du, Jeevana Priya Inala, Yewen Pu, Andrew Spielberg, Adriana Schulz, Daniela Rus, Armando Solar-Lezama, and Wojeiech Matusik. Inversecsg: Automatic conversion of 3d models esg trees. ACM Transactions on Graphics (TOG), 1766) 2018.
Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar-Lezama. Write, execute, assess: Program synthesis with a repl. In Advances in Neural Information Pro- cessing Systems, pp. 9169-9178, 2019.
Pierre-Alain Fayolle and Alexander Pasko. An evolutionary approach to the extraction of object construction trees from 3d point clouds. Computer-Aided Design, 74:1-17, 2016.
Markus Friedrich, Pierre-Alain Fayolle, Thomas Gabor, and Claudia Minnhoff-Popien Optimizing evolutionary csg tree extraction. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1183-1191, 2019.
Karim Hamza and Kazuhiro Saitou. Optimization of constructive solid geometry via tree-based multi-objective genetic algorithm. In Genetic and Evolutionary Computation Conference, pp- 981-992. Springer, 2004.
Pradeep Kumar Jayaraman, Aditya Sanghi, Joseph Lambourne, Thomas Davies, Hooman Shayani, and Nigel Morris. Uv-net: Learning from curve-networks and solids. arXiv preprint arXiv:2006. 102/1, 2020.
10

---

Under review as a conference paper at ICLR 2021
R. Kenny Jones, Theresa Barton, Xianghao Xu, Kai Wang, Ellen Jiang, Paul Guerrero, Niloy Mi- tra, and Daniel Ritchie. Shapeassembly: Learning to generate programs for 3d shape structure synthesis. ACM Transactions on Graphics (TOG), Siggraph Asia 2020, Artcce 234, 2020.
Ashwin Kalyan, Abhishek Mohta, Oleksandr Polozov, Dhruv Batra, Prateek Jain, and Sumit Gul- wani. Neural-guided deductive search for real-time program synthesis from examples. arXiv preprint 1804. 01 186, 2018.
Kacper Kania, Maciej Zięba, and Tomasz Kajdanowicz. Uesg nees--ne--oee net deee-neetuuuunee unsupervised discovering of constructive solid geometry tree. arXiv preprint arXiv:2006.0911 09102, 2020.
Sangpil Kim, Hyung-gun Chi, Xiao Hu, Qixing Huang, and Karthik Ramani. A large-scale anno- tated mechanical components benchmark for classification and retrieval tasks with deep neural networks. In Proceedings of 16th European Conference on Computer Vision (ECCV), 2020.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: (60K,006600000000 (29900 2016.
Sebastian Koch, Albert Matveev, Zhongshi Jiang, Francis Williams, Alexey Artemov, Evgeny Bur- naev, Mare Alexa, Denis Zorin, and Daniele Panozzo. Abc: A big cad model dataset for geometric deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pp. 9601-9611, 2019.
Changjian Li, Hao Pan, Adrien Bousseau, and Niloy J. Mitra. Sketch2cad: Sequential cad modeling by sketching in context. ACM Trans. Graph. (Proceedings of SIGGRAPH Asia 2020), 39(6): 164:1-164:14, 2020. doi: https://doi.org// 1t5p://0...1114/1444455.5551
Renjie Liao, Yujia Li, Yang Song, Shenlong Wang. Will Hamilton, David K Duvenaud, Raquel Urtasun, and Richard Zemel. Efficient graph generation with graph recurrent attention networks. In Advances in Neural Information Processing Systems, pp. 4255-4265, 2019.
Cheng Lin, Tingxiang Fan, Wenping Wang, and Matthias Nießner. Modeling 3d shapes by rein- forcement learning. ECCV, 2020.
Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy J. Mitra, and Leonidas Guibas. Structurenet: Hierarchical graph networks for 3d shape generation. ACM Trans. Graph., 38(6), November 2019a. ISSN 0730-0301. doi: 10. 14453335588933355227
Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object under- standing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 909-918, 2019b.
Chandrakana Nandi, Anat Caspi, Dan Grossman, and Zachary Tatlock. Programming language tools and techniques for 3d printing. In 2nd Summit Advances in Programming Languages (SNAPL 2017). Schloss Dagstuhl-Leibnin Zentrum fuer Informatik, 2017.
Chandrakana Nandi, James R Wilcox, Pavel Panchekha, Taylor Blau, Dan Grossman, and Zachary Tatlock. Functional programming for compiling and decompiling computer-aide design. Pro- ceedings of the ACM on Programming Languages, 2(ICFP):1-31, 2018.
Chandrakana Nandi, Max Willsey, Adam Anderson, James R. Wilcox, Eva Darulova, Dan Gross- man, and Zachary Tatlock. Synthesizing structured cad models with equality saturation and in- verse transformations. In Proceedings of the s1 ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 31 41-44, 2020.
Adriana Schulz, Ariel Shamir, David I. W. Levin, Pitchaya Sitthi-Amom, and Wojciech Matusik. Design and fabrication by example. ACM Transactions on Graphics (Proceedings SIGGRAPH 2014), 33(4), 2014.
Ari Seff, Yaniv Ovadia, Wenda Zhou, and Ryan P. Adams. Sketchgraphs: A large-scale dataset for modeling relational geometry in computer-aide aided design. In ICML 2020 Workshop on Object- Oriented Learning. 2020.
11

---

Under review as a conference paper at ICLR 2021
Jami J Shah, David Anderson, Yong Se Kim, and Sanjay Joshi. A discourse geometric feature recognition from cad models. J. Comput. Inf. Sci. Eng., 1(1):41-51, 2001.
Vadim Shapiro and Donald L Vossler. Separation for boundary to csg conversion. ACM Transactions on Graphics (TOG), 12(1):35-55, 1993.
Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos Kalogerakis, and Subhransu Maji. Csgnet: Neural shape parser for constructive solid geometry. corr abs/1712.08290 (2017). arXiv preprint urXiv:1712.0822. 08290,, 2017.
Binil Starly. fabwave
FabWave 3D Part Repository, 2020. URL httpp::/www.dde. hwtp://wwwwimmmmbbb../mm org
O. Stava, S. Pirk, J. Kratt, B. Chen, R. Mundefinedch, O. Deussen, and B. Benes. Inverse procedural modelling of trees. Comput. Graph. Forum, 33(6):118-1))) September 2014. ISSN 0167-7055. doi: 10.1 10.111
URL URL https: //o:: org/ 10 11lllcgfff..... 12282
Jerry O. Talton, Yu Lou, Steve Lesser, Jared Duke, Radomír Méch, and Vladlen Koltun. Metropolis procedural modeling. ACM Trans. Gruph., 30(2), April 2011. ISSN 0730-0301. doi: 10.1145/
1944846. 1944851. htRttttts:://dd org worgrr/// 10 11111114444441 1944851
Yunhao Tang, Shipra Agrawal, and Yuri Faenza. Reinforcement learning for integer programming: Learning cut. arXiv preprint arXiv: 19X6v::906.04880 04859, 2019.
Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T. Freeman, Joshua B. Tenen- baum, and Jiajun Wu. Learning to infer and execute 3d shape programs. In International Confer- ence on Learning Representations, 2019.
Carlos A. Vanegas, Ignacio Garcia-Dorado, Daniel G. Aliaga, Bedrich Benes, and Paul Wad- dell. Inverse design of urban procedural models. ACM Trans. Graph., 31(6), November 2012. ISSN 0730-0301. doi: 10.14452366145.236611,,,
URL
2366145. 2366187
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint 1rXiv:1710.10000 2017.
K.J. Weiler. Topological structures for geometric modeling. Technical report RPI, Center for Inter- active Computer Graphics. University Microfilms, 1986.
Daniel Weiss. Geomelry-bused structural optimization on CAD specification trees. PhD thesis, ETH Zurich, 2009.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural 2018.
networks? arXiv preprint
Zhibo Zhang, Prakhar Jaiswal, and Rahul Rai. Featurenet: machining feature recognition based on 3d convolution neural network. Computer-Aided Design, 101:12-22, 2018.
Qingnan Zhou and Alec Jacobson. Thingi10k: dataset of 10,000 3d-printing models. arXiv preprint arxi: /605:vvv...... 1605.04797, 2016.
Chuhang Zou, Ersin Yumer, Jimei Yang. Duygu Ceylan, and Derek Hoiem. 3d-prnn: Generat- ing shape primitives with recurrent neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 900-909, 2017.
12

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A. FUSION 360 GALLERY RECONSTRUCTION DATASET
In this section we provide additional details on the Fusion 360 Gallery reconstruction dataset.
A.1.1 DATA PROCESSING
We acquire the Fusion 360 designs from the Autodesk Online Gallery From the approximately 20,000 designs available we derive several datasets focused on specific areas of research. This paper introduces the reconstruction dataset as new baseline dataset for CAD reconstruction. We use the Fusion 360 Python API to parse the native .f3d files. We divide multi-componen assemblies into separate designs and suppress modeling operations other than sketch and extrude to expand the data quantity. Figure TO shows example assembly that is split up to produce multiple designs with independent construction sequences. The rounded edges are removed by suppressing fillets in the parametric CAD file.
After each construction sequence has been extracted we perform reconstruction and compare the reconstructed design to the original to ensure data validity. Failure cases and any duplicate designs, are not included in the dataset. We consider a design a duplicate when there is an exact match in all of the following: body count, face count, surface area to decimal point, volume one decimal point, and for each extrude in the construction sequence: extrude profile count, extrude body count, extrude face count, extrude side face count, extrude end face count, and extrude start face count. This process allows us to match designs that have been translated or rotated, while considering designs unique they have matching geometry but different construction sequences. Deduplication removes approximately 5,000 designs. Figure 11 shows random sampling of designs from the reconstruction dataset.
A.1.2 GEOMETRY DATA FORMAT
We provide geometry several data formats that we discuss in this section. Geometry is provided for the final design and after each extrude operation.
B-Rep consists faces, edges, loops, coedges and vertices (Weiler 1986). A face is connected region of the model's surface. An edge defines the curve where two faces meet and vertex defines the point where edges meet. Faces have an underlying parametric surface which is divided into visible and hidden regions by series of boundary loops. A set of connected faces forms a body. Designs in the dataset may contain multiple bodies.
Boundary Representation
hntps://gallery.autoode autodesk.com
Figure 10: An example multi-component assembly that is broken up into separate designs (high- lighted with color), each with independent construction sequence.
13

---

Under review as a conference paper at ICLR 2021
Figure 11: A random sampling of designs from the Fusion 360 Gallery reconstruction dataset.
14

---

Under review as a conference paper at ICLR 2021
Body Count Per Design
Design
vox
354
70%
25%
60%
Tolel 5c
50X
40%
30%
10%
208
0%
0%
8o Ris
Number Bodles
Number Puces
Figure 12: Left: The number of bodies per design shown distribution. Right: The number of B-Rep faces per design shown as distribution.
B-Rep data is provided as smmt files representing the ground truth geometry and .step as an alter- nate neutral B- B-Rep file format. The smmt file format is the native format used by Autodesk Shape Manager, the CAD kernel within Fusion 360, and has the advantage of minimizing conversion er- rors. Additionally the B- B-Rep entities, such as bodies and faces, can referenced from the construction sequence back to entities the smm file.
Mesh Mesh data is provided in .oj format representing triangulated version of the B-Rep. Each B-Rep face is triangulated separately and labeled as a group of triangles in the .obj file with the B-Rep face id the group name. This approach allows the triangles be traced back to the B-Rep face and associated extrude operation. Note that the .obj meshes provided are not manifold.
Other representations, such as point clouds voxels, can be generated using existing data conver- sion routines and are not included in the dataset. For convenience we include thumbnail .png image file together with each geometry.
Files are provided in a single directory. with a naming convention XXXXX VYYYΣYYY 2222 [ 1234] extt Here XXXXX represents the project, УУYłΚтγγ the 2
follows:
file, zzZZ the component, and 1234 the extrude index. If 1232 is absent the file represents the final design.
A.1.3 DESIGN COMPLEXITY
A key goal of the reconstruction dataset is provide a suitably scoped baseline for learning-based approaches to CAD reconstruction. Restricting the modeling operations to sketch and extrude vastly narrows the design space and enables simpler shape grammars for reconstruction. Each design represents component in Fusion 360 that can have multiple geometric bodies. Figure 12 (left) illustrates that the vast majority designs have single body. The number of B-Rep faces each design gives good indication of the complexity of the dataset. Figure 12 (right) shows the number of faces per design as a distribution, with the peak being between 5-10 faces per design. As we do not filter any of the designs based on complexity, this distribution reflects real designs where simple washers and flat plates are common components mechanical assemblies.
A.1.4 CONSTRUCTION SEQUENCE
The construction sequence is the series of sketch and extrude operations that are executed to produce the final geometry. We provide the construction sequence in a JSON format text file. Each step in the construction sequence has associated parameters that are stored in that entity. For example, sketch entities will store the curves that make up the sketch. Each construction sequence must have least one sketch and one extrude step, for minimum of two steps. The average number steps is 4.74, the median 4, the mode 2, and the maximum 61. Figure 13 illustrates the distribution of construction sequence length and the most frequent construction sequence combinations.
With access to the full parametric history, it is possible extract numerous relationships from the dataset that can be used for learning. Starting at a high level, we know the order modeling oper- ations in the construction sequence. The sketch geometry, B-Rep faces, and triangles derived from
15

---

Under review as a conference paper at ICLR 2021
Sequence Length
Sequence Prequency
400
365
595
SBSESE
S28
7oto
2056
18%
1056
Other
13
15
140
85%
15%
00.6
2255
20.%
3155
40%
Seguenoe Langth
Tolta
Figure 13: Left: The distribution construction sequence length. Right: The distribution com- mon construction sequences. indicates Sketch and E indicates Extrude operation.
them, can be traced back to a position in the construction sequence. The type of geometry created by each modeling operation is also known. For example, sketches create trimmed profiles where the curves intersect to form closed loops; extrude operations produce B-Rep faces with information such as which faces were on the side or ends of an extrusion. addition, the sequence of B-Rep models themselves contain valuable topology information that can be leveraged, such as the connectivity of B-Rep faces and edges. Finally geometric information like points and normal vectors can sam- pled from the parametric surfaces. Feature diversity enables many different learning representations and architectures to be leveraged and compared.
A.1.5 SKETCH
In this section we describe the sketch data further detail and present statistics illustrating the data distribution. Figure [4] illustrats the geometric 2D primitives, described in section 3 that make up a sketch. Sketches are represented as series of points (ptl..ptt6), that create curves (cl...c5), that turn create profiles (pr)..,rr3) illustrated with separate colors. Profiles can have inner loops to create holes, cl is the inner loop of pr2 and the outer loop of pr3. Profiles also have trimmed representation that contains only closed loops without open curves. The trimmed representation shown in the lower right of Figure 14 where the c5 is trimmed and incorporated into pr and pr
ptd O
pts
prd
pio
pto o
Points
Curves
Jr3
pr3
prz
Profiles
Trimmed Profiles
Figure 14: Sketch primitives.
16

---

Under review as a conference paper at ICLR 2021
Design
Type Combination Frequemcy
595
407
ACL
C1
30%
T
A1
2055
ACBL
1056
Other
30
400 Number of Chrros
50
70
68
5%
10%
15%
2055
25%
Total
Figure 15: Left: The number curves each design, shown as a distribution. Right: Common curve combinations in each design, shown distribution. Each curve type is abbreviated as follows: C SketchCircle, A SkketchAc, L. SketchLine, S SketcchiitedSplinz
All sketch geometry is provided in solved state, meaning sketch constraint solver is not required for standard reconstruction. The as-designed ordering of sketch operations is not stored in the na- tive design files, however consistent ordering can be derived by traversing the sketch profiles in sequence.
Points Each point is provided with universally unique identifier (UUID) key and Point.3 data structure with x, y. and z. Sketch primitives are drawn in a local 2D coordinate system and later transformed into world coordinates. As such all sketch points have value of O.
Curves Each curve has a UUID key and a below. The parameters for each curve type can be referenced via the Fusion 360 API documentation linked below.
that can represent the curve types listed
Eket.chArg
SketchCircle
SketchConicCurve
SketchEllipse
Sketcch11lipticaa..
Sket.chFitteddSpinne
SketchFixed5pl..
Sketchl.ine
Figure 15 Illuttraees the distribution of curve count per design and the frequency that different combinations are used together on lines, circles, and ares rather than spline curves.
a design. It is notable that mechanical CAD sketches rely heavily
Profiles represent a collection of curves that join together to make closed loop. In Fusion 360 profiles are automatically generated from arbitrary curves that don't 't necessarily connect the end points, In Figure 14 two profiles (pr1 and pr2) are generated when the line crosses the triangle. We provide both the original curves (Figure 14 top right) used generate the profiles (Figure 14 bottom left) and the trimmed profile information containing just the closed profile loop (Figure 14 bottom right). Loops within profiles have a flag that can be set specify holes.
Profiles
Dimensions User specified sketch dimensions are used to define set angles, diameters, distances ctc. between sketch geometry to constraint the sketch as it is edited. Each dimension has a UUID key and a SeketchDimense references one or more curves by UUID. The parameters for each dimension type can referenced via the Fusion 360 API documentation linked below.
that represent the dimension types listed below. Each dimension
SxetchAngularDDmmnn...
@ket.chConcentric rel mensi
17

---

Under review as a conference paper at ICLR 2021
Sket.ciDiimmeterrimmnsssiin
Bieechaillppoetnooorraalll...... 6xetchalneearDmenns....
Execchofffsetccrrreo]]..o...1
$ketcchoffssstt)mmensstoo
SxetchRadialDDmenssi.oo
Constraints define geometric relationships between sketch geometry. For example, symmetry constraint enables the user have geometry mirrored, a parallel constraint ensures two that can
Constraints
lines are always parallel. Each constraint has UUID key and m/Geemetricconstraaatee represent the constraint types listed below. Each constraint references one or more curves by UUID. The parameters for each constraint type can be referenced via the Fusion 360 API documentation linked below.
CiruuesPatterrCoosstaaan
ColncldentCoontzalltt
CollinnerConstrrain
Concentricconstrraitt
EqualConstraint
fiorizontaloonst.o0
MidPointConstra'nt
DffsetConstraint
ParallelConstra.nt
Perpendicularconstraa
PolygonConstraiint
Smoothhonsttraint
Byrmetryyonstraa
TangentConstraaint
VerticaJConstraint
Wort.caaPointsConsee
Figure 16 illlsstrate the distribution of dimension and constraint types in the dataset.
A.1.6 EXTRUDE
In this section we deseribe the extrude data in further detail and present statisties illustrating the data distribution. Extrude operations have number of parameters that set by the user while design- ing. Figure shows how a sketch (left) can be extruded a set distance on one side, symmetrically
Dimension Types
Constraint Types
Uiver
Coloddeor
Recional
Dimmter
Ioppandadarr
Otbet
Langert
Fanalid
Consende
Anga
Summaty
OtverOunee
Topal
Conantritinle
Colloorr
Offec
Doitts
EljeeeaaerRalus
Figure 16: The distribution of constraint (left) and dimension (right) types.
18

---

Under review as a conference paper at ICLR 2021
Sketch
One Side Extrude
Symmetric Extrude
Two Side Extrude
Taper Extrude
Figure 17: An extrude be expressed in several different ways: perpendicular from sketch for a set distance along one side, a symmetrical distance along both sides, or separate distances along two sides. Additionally the extrude can be tapered at angle.
Start Body Sketch
New Body Operation
Join Operation
Cut
Intersect Operation
Operation
Figure 18: Extrude operations include the ability to Boolean with other geometry. From the starting body shown the left, sketch is extruded to form body overlapping the starting body, joined with the starting body, cut out of the starting body, or intersected with the starting body.
sides, with different distances each side, well as tapered. Once single body has been created by an extrude operation, subsequent extrudes can interact with that body via Boolean oper- ations. Figure T8 shows starting body and sketch (left) that can be extruded to form separate bodies, single joined body, a cut through the starting body, or body at the intersection. The first extrude operation of construction sequence is always new body, with any operation possible for subsequent operations.
Figure 19 outlines the distribution different extrude types and operations. Note that tapers can be applied in addition to any extrude type, so the overall frequency of each is shown rather than relative percentage.
A.2 FUSION 360 GYM
In this section we provide additional information about the functionality available in the Fasion 360 Gym. The Fusion 360 Gym requires the Autodesk Fusion 360 desktop CAD application, available on both macOS and Windows for free to the academic community. Although Fusion 360 is a cloud
Type Distribution
Extrude Operstion Distribution
16000
40%
13000
Aroquary
20tal 56
90%
8000
6000
8000
16%
2000
OssSide
Symamatii
TwoSkder
Taper
NewBody
Joán
Cut
Intenedt
Figure 19; The distribution of extrude types (left) and operations (right).
19

---

Under review as a conference paper at ICLR 2021
connected desktop application, the Fusion 360 Cym does all processing locally. The Fusion 360 Gym consists a server that runs inside of Fusion 360 and receives commands from a client running externally. Multiple instances of the Fusion 360 Gym server can be run in parallel. The remainder of this section introduces the available commands from the client.
A.2.1 RECONSTRUCTION COMMANDS
Reconstruction commands reconstruct the existing designs different granularity levels from json files provided with the Fusion 360 Gallery reconstruction dataset.
reconstruct (file): reconstruct entire design from the provided json file.
secenstruct_sseteee (json__ds data, sketch. name, sketch. plane, scale, translate, rotate): reconstruct a sketch from the provided json data and sketch name. A skettch_plane can be cither: (1) a string value representing construction plane: XY, XZ, or Y2; (2) a B-Rep planar face id; (3) point3d on planar face of B-Rep.
reconst
(json_data, sket.ch_name, profile_id, scale, translate, rotate): reconstruct a profile from the provide json data, sketch name, and a profile id.
recenstruct_eurye (json_data, sketch_name, curve_id, scale, translate, rotate): reconstruct curve from the provide json data, sketch name, and curve id.
set_target (file): the target reconstructed with .step or .smt file. The call returns face adjacency graph representing the B-Rep geometry/topology and bounding_box the target that can be used for normalization.
revert_to_targer (): revert to the target design, removing all reconstruction geometry.
A.2.2 SKETCH EXTRUSION COMMANDS
Sketch extrusion commands allows users incrementally create new designs by generating the underlying sketch primitives and extruding them by an arbitrary amount.
(sketch ckeachh add sketch to design. A sketch plance can either: string value representing a construction plane: XY, x2, or Yz; (2) a B- B-Rep planar face id: or (3) a point3d on planar face of a B B-Rep.
pl, transform): add point to create sequential line in the given sketch. is either point the 2D sketch space point in the 3D world coordinate space if transform= "worddd"=="""""" specified.
add add_point tsketch_nane
add line (sketch. name, pl, p2, transform): add line to the given sketch. p1 and p2 are the same as defined in add_point ().
add_curve (sketch_name)
curve_data, transform); add curve the given sketch. curve_data follows the format supplied to reconstruct_cuut curve ().
close close_profile bokketch_namee close current set of lines create one or more profiles by joining the first point to the last point.
add_extrude (sketch_name profile_id, distance, operation.
export_type, is_[o add extrude the design. Four operations supported: dcinFeaturroperatiio. NewBodyF NewBodyFFetureeooee watureeeerattiioooooom vertices of the resulting hody and B-Rep face information; (2) Graph to represent a face adjacency graph representing the B- Rep geometry/lppolog. An intersection over union (IoU) value between the target and the reconstruction is calculated and returned if is IoU= "True" is specified.
CutFeature@peratioon
IntersectFeatureppe.
Two export formats are provided: (1) BRep to represent B-Rep
A.2.3 FACE EXTRUSION COMMANDS
Face extrusion commands enable a target design to be reconstructed using extrude operations from face to face.
add_ext rude_ ftw.uddeb_ygggeefacaaaeeeeeaeee (start_faco end_face, operation): add an extrude between two faces of the target. Four operations are supported: Jo werineeaurrdppraatiooo CutReature@peration IntersecttFetuuregeer tew:oodyreaturregerratii..
20

---

Under review as a conference paper at ICLR 2021
Figure 20: Example designs created using randomized reconstruction commands.
add redeesttrrs____ bb___yy______ target face (actions ons, revert): execute multiple extrude operations, between two faces of the target, in sequence.
A.2.4 RANDOMIZED RECONSTRUCTION COMMANDS
Randomized reonstruction commands allow users sample designs, sketches, and profiles from isting designs in the Fusion 360 Gallery and support distribution matching of parameters, in support of generations of semi-synthetic data. Figure 20 shows example designs created using randomized reconstruction commands.
get_di st. i ons data_dir, filter): get list of distributions from the provided dataset. The command currently supports the following distributions: the starting sketch p1 ace, the
21

---

Under review as a conference paper at ICLR 2021
number of faces, the number of extrusicns, the length of sequences, the number of curves, the number of bodies, the sketch areas, and the profile areas.
distributionn sont_oonppa 1 inn (distri ierssiuuiioonn parameters): sample distribution matching parameters for one design from the distributions.
sample_desige (data_dir): dir): randomly sample json file from the given dataset.
area distribut ion): sample one sketch from the provided design. Three sampling types are provided: (1) random, return a sketch ran- domly sampled from the provided design; (2) determin ittiiiiiiiiiiii itcc, return the largest sketch in the design; and (3) distr iibuiiie,, return sketch that its area is in the distribution of the provided dataset.
sketch (json file, samp cerppiing_tyype
sarple caappe__profiii area distributionn: sample profiles from provided sketch. Three sampling types are pro- vided: (1) random, return profiles randomly sampled from the provided sketch; (2) deterreen cnnterminnsttin return profiles that are larger than the average area of the profiles in the sketch; and (3) distributiv ive, return profiles that the areas are in the distribution of the provided dataset.
prwpillleeeeeee.ee...eoo ssketch_name max_ maa__numer caabbbe_rroffieee
sampling ng_type,
A.2.5 EXPORT COMMANDS
Export commands enable the existing designs to be exported the following formats:
mesh (file): retrieve mesh in .obj or .stl format and write it to the local file provided.
brop (file): retrieve brep in step, .mm,, or .f3d format and write it to local file provided.
sketches (dir,, r, format): retrieve each sketch in -ngg or .dxf format and write them to a local directory provided.
screenshot (f (file, width, height): retrieve screenshot of the current design png image and write it local file provided.
graph(file, (file, dir, format): retrieve face adjacency graph a given format and write it in local directory provided.
A.3 CAD RECONSTRUCTION
In this section we provide additional details of the experiments performed on the CAD reconstruction task described in Section 5
A.3.1 DATA PREPARATION
The agents trained subset of the reconstruction dataset that has been converted into face extrusion sequence. Due to the simplified face extrusion representation, not all designs from the dataset be converted to face extrusion sequence. Figure 2
| shows several common conversion limitations where necessary face information (highlighted in red) is not present in the target geome- try. The intermediate top face in Figure 2 disappears when merged with the top face of Extrude 2. In Figure 21 C a hole cut through the geometry means the intermediate top face of Extrude is absent and there no start or end face in the target geometry to perform the cut operation used in Extrude 2. Although it is possible to find alternate face extrusion sequences with heuristic rules, we instead try to maintain the user designed sequence with the exception reversing the direction of the extrusion in some scenarios, e.g. the end face becomes the start face.
A.3.2 AGENT
All MPN agents employ a network architecture able to exploit the graph structure of the data, con- sisting of two layers passing messages along the edges of the graph. The vertex features in the face-adjacency graph are as follows:
Points: A 10x 10 grid of 3D points sampled from the UV coordinate space of the B-Rep face and normalized to the bounding box of the target geometry.
Normals: A 10x 10 grid of 3D normal vectors sampled from the UV coordinate space of the B-Rep face.
Trimming Mask: A 10x 10 grid of binary values representing samples that are inside/outside the B-Rep face trimming boundary.
22

---

Under review as a conference paper at ICLR 2021
A
B
c
Sketch Profile 1
Extrude 1
Sketch Profile
Extrude
Different construction sequences (A-C) for the same geometry. During conversion to face extrusion sequence, the necessary face information (highlighted in red) does not exist in the target, meaning B and C can not be converted. Green arrows indicate new bodyfjoin extrude operations, while red arrows indicate cul extrude operations.
Figure 21:
Surface Type: A one-hot encoded flag indicating the type of surface represented by the B- Rep face: Cone, Cylinder, Elliptical, Ellisticalcylinde Nurbs, Plane, Sphere, Torus.
We denote the learned vertex embedding vectors produced by the two MPN branches as (h²) and (h'} for the current output and target graphs, respectively. We estimate the probability of the k-th operation type, and the j-th face being the start face end face
op Fop (he), he Σhi
(2)
PJ stort = sof softmax (Fstare ho)
pJ softmax Fend (br, bi ha)): s.1. j argmax pJ end
(4)
stort
where Pop, Fstart, and Fend denote linear layers that take the concatenated vectors input.
Using the face extrusion sequence data, we train the agents in an offline manner without interacting with the Fusion 360 Gym. The mlp and gen agents have hidden dimension of 256 across all layers. The gin agent has two 256-dimensional linear layers within its graph convolution layer. The gat has 8 heads of 64 hidden dimensions each. The agents trained with dropout rate of 0.1 and learning rate of 0.0001 for 100 epochs with the model saved at the lowest training loss. The learning rate decreased by a factor of 0.1 upon plateau within 10 most recent epochs. Training performed on an NVIDIA Tesla V100 with an Adam optimizer and takes approximately 6-8 hours.
A.3.3 SEARCH
In addition to the random rollout search (rand) described in Section 5.2 we implement beam search (beam) and best first search (best). Beam search explores multiple candidates in parallel, filter- ing the top-k candidates, ranked by the generation probability, until a certain length. Best search explores the search space by expanding the most-likely sequence, also ranked by the generation probability.
In all search algorithms mask out the following invalid actions they are never taken:
23

---

Under review as a conference paper at ICLR 2021
Average Reconstruction Time Design
30
Stromdds
20
10
git-hram gat-bram gin-best gen-bram mip-heat gen-best mip-bram gat-rand gat-best git-med
mipenétrao-bemmmm---eet rand-rand
Agent-Search
Figure 22: Average reconstruction time per design for combinations of agents and search strategies.
Start faces surface types that are non-planar
End faces surface types that are non-planar
Operation types other than new body when the current geometry is empty
Other invalid actions that require geometric checks, such specifying start face and end face that are co-planar, are returned as invalid from the Fusion 360 Gym and count against the search budget.
A.3.4 EVALUATION
We perform evaluation using the official test set containing 1725 designs. Evaluation is performed in an online manner using the Fusion 360 Gym. Figure [2] shows the average reconstruction time for each design with combinations of agents and search strategies. Time differences are due in part to the number of invalid actions chosen by an agent that can be quickly checked in the Fusion 360 Cym without geometry processing. The large majority evaluation time spent inside the Fusion 360 Gym executing modeling operations, graph generation, and IoU calculation. We set a hard time limit of 10 minutes per design, after which we halt search, affecting between 0-14 designs depending on the agent and search strategy. Between 0-15 designs cause software crashes. 17 designs in the test set cannot be represented as graphs due to our data pipeline not supporting edges with more than the best seen IoU, if no IoU score is available, and consider the design to fail at exact reconstruction.
two adjacent faces. In all failure cases
A.3.5 RESULTS
Table details the full set of results for agents and search strategies the extendedbaseline comparison experiment from Section 5.3 Table provides additional details of the synthetic data
compares three different search strate- gies side by side using the gen agent. We observe that all search strategies perform similarly for reconstruction IoU, while random rollout search has notable advantage in exact reconstructions. This advantage is due to the limited search budget enforce reflect real-world scenario. We expect both best and beam search to improve with larger search budgets.
performance experiment from Section 5.3
Figure[2]visuaal
A.4 TASKS
In addition to CAD reconstruction the Fusion 360 Gallery reconstruction dataset and Fusion 360 Gym can be used for range tasks such as program synthesis, sequence modeling, generative models, and geometric deep learning. Other tasks include:
Modeling operation order prediction to recover the correct order construction from raw geometry.
Sketch synthesis to recover the original sketch, including constraints and dimensions, from the 3D geometry.
Predicting next action in the design sequence for 'CAD autocomplete'.
Generative models that aware of the design sequence and constraints.
24

---

Under review as a conference paper at ICLR 2021
Agent
Search
IoU
Exact Reconstruction. %
Conciseness
Parameters.
20 Steps
100 Steps
20 Steps
100 Steps
gen
rand
0.8644
0.9042
0.6232
0.6754
1.0168
3.02M
beam
0. 0,8640
0.8982
0,5739
0,6122
0,9275
3.02M
gen
best
0.8831
0.9186
0.5971
0.6348
0.9215
3.02M
mlp
rand
0.8274
0.8596
0.5658
0.5965
0.9763
2.24M
mip
beam
0.8619
0.8995
0,5525
0.5884
0.9271
2.24M
mlp
best
0,8712
0.8991
0.5675
0.5977
0.9305
2.24M
gat
rand
0,8742
0.9128
0,6191
0,6742
1,0206
3,03M
gat
beam
0,8691
0,9016
0,5791
0,6133
0,9261
3.03M
gat
best
0.8895
0.9139
0.5994
0.6354
0.9290
3.03M
gin
rand
0.8346
0.8761
0,5901
0.6301
1.0042
3.62M
gin
beam
0.8500
0.8913
0,5594
0,5983
0,9299
3.62M
gin
best
0,8693
0.9007
0,5803
0,6122
0.9340
3.62M
rand
rand
0.6840
0.8386
0.4157
0.5380
1.2824
beam 0,4785
0.6277
0,2812
0.389
0.9118
rand
best 0.6334
0.7994
0,3693
0.4887
0.8979
Table 2: Reconstruction results for multiple agent and search combinations. IoU and exact recon- struction are shown 20 and 100 search steps. Lower values are better for conciseness.
Agent
IoU
Exact Reconstruction
Conciseness 100 Steps
Parameters
20 Steps
100 Steps
20 Steps
real
0.8644
0.9042
0.6232
0.6754
1.0168
3.02M
aug 0.8707
0.8928
0.6452
0.6701
0.9706
3.02M
semi-syn 0.8154
0.8473
0.5780
0.6104
1.0070
3.02M
syn 0.6646
0.7211
0.4383
0.4835
1.0519
3.02M
Table 3: Reconstruction results using random rollouts and gen agents trained on human-designed data (real), a mixture of human-designed and semi-synthetic data (aug), semi-synthetic data (semi- syn), and synthetic data (syn).
Reconstruction IoU
Cumulative Exact Reconstructions
Ground Truth
1.0
rand beam
Recorstruution
Estimated Representation Upper Limit
Toz
0.7
9.5
0.3
60
6o
03
Search Step
Search Step
Figure 23: Reconstruction results over 100 search steps using the gen agent with best first search (best), random rollout search (rand) and beam search (beam).
25


Reviews:
[{"review_id": "w-cQ0AdQ1hj", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I have to admit straight away that this paper is far from my field of expertise (computer vision, generative networks). I have not worked with CAD models, and I am not in expert in reinforcement/imitation learning. My review is thus written from the \"educated outsider\" viewpoint.\n\nFrom that viewpoint, the paper is very strong. It introduces a meaningful problem (CAD modeling sequence reconstruction), motivates the need for a new task and the direction of research, describes the full task and the compromises it makes (face extrudes rather than sketch extrudes are considered in the simplified task). The paper then introduces a new training/test dataset and an environment for training agents, and evaluates a reasonable set of agents rather extensively.\n\nAs a slight criticism, I found that too many details are moved from Section 5 to the supmat. E.g. what is meant by MLP with\n\"an auto-regressive connection between the two target faces\" was completely unclear before I looked into the supmat (and the phrase did not refer me to supmat either).\nOtherwise, the paper is well-written and has nice visualizations (though they crash my PDF viewer) that aid understanding.\n\nOverall, I really enjoyed reading the paper, and I am not able to identify any flaws. As far as I can judge based on my limited knowledge, the paper (together with associated dataset and environment) is likely to spur new research and to be impactful. I therefore give it a strong rating, but I cannot exclude that I have missed some flaws that would be identifiable by an expert.", "extra_scores": {}, "extra_text": {}}, {"review_id": "eaEp1oK6PiS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a new dataset of 3D geometry construction sequences based on sequential sketching (i.e. sets of two-dimensional curves) and extruding (i.e. axes, angles and distances for extrusion profiles from sketches) combined with Boolean solid geometry operations. The dataset comprises the resulting objects as well as the human designed sketch-extrude sequence that lead to the object geometry. The 8625 objects are available in three different geometry representations (boundary representation, mesh, and construction sequence in structured format). Furthermore the submission describes a new infrastructure (\"gym\") to train and evaluate algorithms that estimate such construction sequences, as well as evaluation metrics to gauge the efficiency and effectiveness of such estimation algorithms. The paper also provides a reference method for sequence estimation based on imitation learning as well as several baselines and their empirical evaluation with respect to the posed benchmarks.\n\n### Strengths\n**[S1]** The paper is written well and takes a comprehensive look on a new dataset from the perspective of the data itself, semi-synthetic generation of new data, evaluation metrics and benchmarking on the data as well as a reference method and its performance analysis.\n\n**[S2]** I particularly appreciate the synthetic data generation/augmentation capability. This may enable more structured studies in the future, for instance with respect to specifics of the constructed geometry: Does complexity of the design geometry matter, does the topology make a difference in achievable performance? \n\n**[S3]** New datasets are often valuable contributions to the community, in particular when they allow the discovery of new insight (however see W1 below)\n\n### Weaknesses\n**[W1]** The main contribution of the paper seems to me the database itself and the associated gym. While new datasets are a good contribution in general, I am not convinced that the target audience of ICLR will benefit in a substantial matter from the exposure. Specifically, I feel that the benefit of human-designed vs. synthetically generated designs is not worked out convincingly. What can I learn, what insight can I gain from the human designs that I cannot from, e.g., purely synthetical, procedural designs? Are there difference in the statistics? It would strengthen the paper to highlight evidence that the data does have benefit in the sense that I can achieve something substantially new. \n\n**[W2]** The submission claims a \"novel, neurally guided method\", but I do not see the novelty laid out clearly: outside of \"using common sketch and extrude CAD modeling operations from real human designs\" in section 2, the novelty is vague and the relation to the state of the art imprecise. Is it the use of sketch and extrude operations in an otherwise previously exposed sequence learning task? Is it using human-designed sequences (then, see W1)? It would strengthen the case for the paper if the novelty would be exposed more concisely.\n\n### Further comments:\nI do appreciate the comprehensiveness across data, benchmarks, reference solutions and evaluation of the paper and such data and gyms can be an important piece of the machine and representation learning puzzle. As it is, I am concerned that the submission is not appealing to a sufficiently large audience at ICLR.", "extra_scores": {}, "extra_text": {}}, {"review_id": "MavPf8OfhXT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary and Contributions\nThe CAD reconstruction problem is defined as the recovery of the sequence of modeling operations used to construct the CAD model, from the raw geometry input (triangle meshes, point clouds, or, B-reps). The paper proposes a novel dataset as well as a generic framework implementing an MDP (Markov Decision Process) formulation for training a neural CAD reconstruction agent.  \n\nThis the first dataset which includes the sequence of CAD modeling operations as ground truth containing 8.5K+ human-designed real-world CAD models. The CAD reconstruction task is formulated as that of training a generic neural MDP agent and provided in an programming environment to the research community to train such models. Finally, a novel algorithm is provided as a baseline to solve the CAD reconstruction problem.  \n\n### Detailed Review \nThe following is the detailed review of the paper, organized into strengths and weaknesses subsections. \n\n#### Strengths\n\n##### Relevance and Significance \nThere is a large number of man-made objects that we interact with that are created using computer-aided design. The modeling steps, which are often lost, are important in understanding, editing, simulation and manufacturing such objects, and need to be reconstructed. This topic is of considerable importance to the CAD community. In addition, approaches, that can reverse engineer the generative process should be of general interest to the wider ML and computer vision community, as well.  \n\n##### Relation to Prior Art \nThe paper does a good job of presenting the prior art, identifying the challenges and need for the presented work. The dataset including the modeling sequences for real, human-designed CAD models is the first of its kind. Since it’s a novel problem (first dataset of the kind), the proposed baseline implementing a learning-based approach to this problem is novel as well (though simplistic, see below).  \n\n##### Reproducibility \nSince the entire dataset and the baseline is released in an open-source programming environment, it should be easy to reproduce and verify the results. \n\n##### Clarity \nThe paper is written well and is easy to understand. \n\n#### Weaknesses\n \n##### Methodology \nThis is largely a dataset paper. Introduction of a new dataset to the research community needs to demonstrate that the tasks to be solved on the dataset is not trivially addressed by the known state of the art. The paper falls short of demonstrating that. \n\n1. Agent model: Two models are considered – MLP (trivial embeddings based on vertex features) and seemingly trivial embeddings of MPNs. The paper doesn't present enough details about the choices made for obtaining meaningful embeddings. This seems to be the heart of the approach and the authors don't present a compelling approach or a comparative evaluation of a set of choices based on the SOTA for graph embeddings, to model the agent. \n\n2. Search: Similarly, the search is trivially implemented using a random rollout. There are much better search strategies available in the SOTA to bring to bear on the problem. \n\n In summary. I don't think that the presented baseline properly brings the SOTA to bear on the problem and thus demonstrates a need for additional research to be spurred on by this dataset.  \n\n##### Novelty \nThe presented approach is a straight-forward application of known techniques.  \n\n##### Empirical Evaluation \nAn important question to consider when proposing a presumably difficult new problem when addressed by known art is to investigate what aspect of the new problem really taxes the state of the art and to stage the experiments and analysis carefully for the same. There are many questions to be addressed here, for e.g. (a) How is stationarity (or, invariance over the topologies) achieved across training, validation and  test sets, (b) Does the training overfit (are the architectures used of enough capacity)?, (c) Is there generalization gap? If so, why? Etc. \n\n### Assessment\nThough the problem seems relevant and of significance to the research community, the dataset of considerable value, the paper doesn't make a strong case whether and how this problem challenges the known state of the art. In its current form, I do not recommend the publication of this paper. \n\n \n\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "LV4nRcvf82", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper presents the first large scale CAD construction sequence dataset together with environment that allows us to synthesize 3D CAD from these sequences. The dataset and environment are essential building blocks for applying machine learning algorithms to CAD design process. Furthermore, the paper proposes comprehensive evaluation metrics and a baseline method for predicting the sequence of 3D CAD design from a CAD model. \n\nThe dataset and environment are highly valuable to the research community. Although the proposed baseline and task have room for improvement, I believe this work merits publication.\n\nPros:\n- The first large scale CAD dataset with the comprehensive construction sequence data. \n- The environment can construct these sequence data and allow us to synthetically generate new data. Together with the 360 Fusion Gallery, this would be a great step towards learning human design processes.\n- A good baseline method with comprehensive evaluation metrics.\n\nCons:\n- The proposed task of CAD sequence reconstruction requires a target geometry in the same CAD format. This would not be called \"raw geometry” as in the abstract. Raw geometry is often referred to as what we can derive from scan data including point clouds and meshes.\n- As the aforementioned CAD reconstruction task is not very practical, adding another practical task such as reconstructing the sequence from raw geometry data is highly recommended. \n- Regarding the evaluation metrics, it is not clear whether the conciseness is computed from only the successful reconstructions or the entire test set. If the former is the case, comparison using this metric across different approaches would not be fair. Otherwise, this does not necessarily provide a sense of how efficient the evaluated algorithm is. Clarification on this is recommended.\n\nQuestions/Remarks:\n- In terms of accuracy, the augmentation rather hurts preciseness with more steps. Why is this happening? Is this due to the limited capacity of the proposed network? Discussion on this would be helpful.\n- The following reference can be cited and discussed. \n﻿\"Modeling 3D Shapes by Reinforcement Learning” Cheng Lin, Tingxiang Fan, Wenping Wang, Matthias Nießner ECCV 2020", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received 4 reviews with mixed initial ratings: 4, 8, 5, 7. The main concerns of R1 and R2, who gave unfavorable scores, included limited methodological novelty beyond the data generation and insufficient empirical evaluation of state-of-the-art methods on the proposed dataset. The authors submitted a new revision with a summary of changes and provided detailed responses to each of the reviews separately: it addressed some of the concerns, but did not change the overall position of the reviewers.
AC agrees with R3 and R4 that the proposed dataset and the environment may have certain practical impact and enable new research in learning CAD reconstruction. However, the contributions are indeed specific to a narrow CAD community, and R1 felt that the paper needs another round of peer reviews before acceptance, as a significant number of new results have been added during the discussion stage. After discussion with PCs, the final recommendation is to reject.
Decision: Reject