SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\mathbf{D}^{2}\mathbf{C}\mathbf{S}\mathbf{G}$ : Unsupervised Learning of Compact CSG Trees with Dual Complements and Dropouts
Fenggen Yu<sup>1</sup> Qimin Chen<sup>1</sup> Maham Tanveer<sup>1</sup>
Ali Mahdavi Amiri
Hao Zhang $^{1,2}$
$^{1}$ Simon Fraser University
2Amazon
Abstract: We present  $\mathrm{D}^2\mathrm{CSG}$ , a neural model composed of two dual and complementary network branches, with dropouts, for unsupervised learning of compact constructive solid geometry (CSG) representations of 3D CAD shapes. Our network is trained to reconstruct a 3D shape by a fixed-order assembly of quadric primitives, with both branches producing a union of primitive intersections or inverses. A key difference between  $\mathrm{D}^2\mathrm{CSG}$  and all prior neural CSG models is its dedicated residual branch to assemble the potentially complex shape complement, which is subtracted from an overall shape modeled by the cover branch. With the shape complements, our network is provably general, while the weight dropout further improves compactness of the CSG tree by removing redundant primitives. We demonstrate both quantitatively and qualitatively that  $\mathrm{D}^2\mathrm{CSG}$  produces compact CSG reconstructions with superior quality and more natural primitives than all existing alternatives, especially over complex and high-genus CAD shapes.
CAD shapes have played a central role in the advancement of geometric deep learning, with most neural models to date trained on datasets such as ModelNet [51], ShapeNet [2], and PartNet [33] for classification, reconstruction, and generation tasks. These shape collections all possess well-defined category or class labels, and more often than not, the effectiveness of the data-driven methods is tied to how well the class-specific shape features can be learned. Recently, the emergence of datasets of CAD parts and assemblies such as ABC [21] and Fusion360 [49] has fueled the need for learning shape representations that are agnostic to class labels, without any reliance on class priors. Case in point, the ABC dataset does not provide any category labels, while another challenge to the ensuing representation learning problem is the rich topological varieties exhibited by the CAD shapes.
Constructive solid geometry (CSG) is a classical CAD representation; it models a 3D shape as a recursive assembly of solid primitives, e.g., cuboids, cylinders, etc., through Boolean operations including union, intersection, and difference. Of particular note is the indispensable role the difference operation plays when modeling holes and high shape genera, which are common in CAD. Recently, there have been increased interests in 3D representation learning using CSG [9, 50, 41, 40, 19, 57, 6, 3], striving for generality, compactness, and reconstruction quality of the learned models.
In terms of primitive counts, a direct indication of compactness of the CSG trees, and reconstruction quality, CAPRI-Net [57] represents the state of the art. However, it is not a general neural model, e.g., it is unable to represent CAD shapes whose assembly necessitates nested difference operations (i.e., needing to subtract a part that requires primitive differences to build itself, e.g., see the CAD model in Fig. 2). Since both operands of the (single) difference operation in CAPRI-Net can only model intersections and unions, their network cannot produce a natural and compact CSG assembly for relatively complex CAD shapes with intricate concavities and topological details; see Fig. 1.
Figure 1: Comparing CSG trees and shape reconstructions by our network,  $\mathrm{D}^2\mathrm{CSG}$ , and CAPRI-Net, the current state of the art. To reproduce the ground truth (GT) shape, a natural CSG assembly necessitates a difference operation involving a complex residual shape, which  $\mathrm{D}^2\mathrm{CSG}$  can predict with compactness (only three intermediate, general shapes) and quality. CAPRI-Net can only build it using a union of convexes, requiring unnecessarily many primitives and leading to poor reconstruction.
In this paper, we present  $\mathrm{D}^2\mathrm{CSG}$ , a novel neural network composed of two dual and complementary branches for unsupervised learning of compact CSG tree representations of 3D CAD shapes. As shown in Fig. 2, our network follows a fixed-order CSG assembly, like most previous unsupervised CSG representation learning models [57, 6, 3]. However, one key difference to all of them is the residual branch that is dedicated to the assembly of the potentially complex complement or residual shape. In turn, the shape complement is subtracted from an overall shape that is learned by the cover branch. Architecturally, the two branches are identical, both constructing a union of intersections of quadric primitives and primitive inverses, but they are modeled by independent network parameters and operate on different primitive sets. With both operands of the final difference operation capable of learning general CAD shape assemblies, our network excels at compactly representing complex and high-genus CAD shapes with higher reconstruction quality, compared to the current state of the art, as shown in Fig. 1. To improve the compactness further, we implement a dropout strategy over network weights to remove redundant primitives, based on an importance metric.
Given the challenge of unsupervised learning of CSG trees amid significant structural diversity among CAD shapes, our network is not designed to learn a unified model over a shape collection. Rather, it overfits to a given 3D CAD shape by optimizing a compact CSG assembly of quadric surface primitives to approximate the shape. The learning problem is still challenging since the number, selection, and assembly of the primitives are unknown, inducing a complex search space.
In contrast to CAPRI-Net, our method is provably general, meaning that any CSG tree can be converted into an equivalent  $\mathrm{D}^2\mathrm{CSG}$  representation. Our dual branch network is fully differentiable and can be trained end-to-end with only the conventional occupancy loss for neural implicit models [4, 3, 57]. We demonstrate both quantitatively and qualitatively that our network, when trained on ABC [21] or ShapeNet [2], produces CSG reconstructions with superior quality, more natural trees, and better quality-compactness trade-off than all existing alternatives [3, 6, 57].
In general, a 3D shape can be represented as a set of primitives or parts assembled together. Primitive fitting to point clouds has been extensively studied [29, 26, 24]. For shape abstraction, cuboid [44][54] and super quadratics [38] have been employed, while 3D Gaussians were adopted for template fitting [12]. From a single image, cuboids have also been used to estimate object parts and their relations using a convolutional-recursive auto-encoder [34]. More complex sub-shapes have been learned for shape assembly such as elementary 3D structures [8], implicit convex [7], and neural star components [20], as well as parts in the form of learnable parametric patches [41], moving or deformable primitives [30, 58, 55, 37], point clouds [28], or a part-aware latent space [10]. However, none of these techniques directly addresses reconstructing a CSG tree for a given 3D shape.
Deep CAD Models. Synthesizing and editing CAD models is challenging due to their sharp features and non-trivial topologies. Learning-based shape programs have been designed to perform these tasks by providing easy-to-use editing tools [43, 11, 17, 1]. Boundary Representations (B-Reps) are common for CAD modeling and there have been attempts to reverse engineer such representations given an input mesh or point cloud [52]. For example, BRepNet [23], UV-Net [15], and SBGCN [16]
Figure 2: For a given 3D shape  $S$  (shown at the right end), our network  $\mathrm{D}^2\mathrm{CSG}$  is trained to optimize both its network parameters and a feature code to reconstruct  $S$  by optimizing an occupancy loss. The network parameters define a CSG assembly over a set of quadric primitives and primitives inverses. The assembly is built using two identical branches: a cover branch (top), producing shape  $S_{C}$ , and a residual branch (bottom), producing shape  $S_{R}$ . After applying intersections and a union to obtain the cover shape  $S_{C}$  and residual shape  $S_{R}$ , by optimizing their respective occupancy losses, the recovered shape is obtained via a difference operation. A dropout is further applied to the parameters in the intersection and union layer for removing redundant primitives and intermediate shapes.
offer encoder-based networks to work with B-Reps and their topological information through message passing for CAD model analysis. NH-Rep [14] proposed several hand-crafted rules to infer CSG trees from B-Reps. However, the leaf nodes of their CSG trees are not parameterized primitives but implicit surfaces. Extrusion-based techniques [50, 53, 39] have also been used in deep-learning-based CAD shape generation, but extrusions do not provide general shape representations and the ensuing supervised learning methods [48, 50] are restricted to specific datasets.
Learning CSG. Learning CSG representations, e.g., primitive assembly [47] and sketch analysis [35, 25], has become an emerging topic of geometric deep learning. While most approaches are supervised, e.g., SPFN [26], ParseNet [41], DeepCAD [50], ComplexGen [13], Lambourne et al. [22] and Point2Cyl [45], there have been several recent attempts at unsupervised CSG tree reconstruction, especially under the class-agnostic setting, resorting to neural implicit representations [4, 31, 36, 3]. InverseCSG [9] is another related method, which first detects a collection of primitives over the surface of an input shape using RANSAC, then performs a search to find CSG sequences to reconstruct the desired shape. However, RANSAC requires meticulous parameter tuning and the initially detected primitives can be noisy. Additionally, the space of CSG sequences is vast, leading to a lengthy search process. CSG-Net [40] employs a GRU-based decoder to generate CSG programs and resorts to RL to find the optimal CSG sequence for a target shape. However, the large search space for their beam search limits the method's ability to reconstruct complex target shapes. PLAD [18] follows CSG-Net and improves the program training with pseudo-labels and approximate distributions, but suffers from similar limitations as CSG-Net due to the use of similar program grammars and sequential network designs. UCSG-Net [19] learns to reconstruct CSG trees with arbitrary assembly orders. Such a task is difficult due to the order flexibility, but it can be made feasible by limiting the primitives to boxes and spheres only. More success in terms of reconstruction quality and compactness of the CSG trees has been obtained by learning fixed-order assemblies, including  $\mathrm{D}^2\mathrm{CSG}$ .
BSP-Net [3] learns plane primitives whose half-spaces are assembled via intersections to obtain convexes, followed by a union operation, to reconstruct concave shapes. CAPRI-Net [57] extends BSP-Net by adding quadric surface primitives and a difference operation after primitive unions. Similar to  $\mathrm{D}^2\mathrm{CSG}$ , CAPRI-Net [57] also tries to perform a final shape difference operation by predicting a set of primitives that undergo a fixed order of intersection, union, and difference operations to obtain the target shape. However, it constrains all primitives to be convex and its sequence is not general to support all shapes. To support generality, the set of primitives in  $\mathrm{D}^2\mathrm{CSG}$  includes both convex primitives and complimentary primitives (see Section 3.1). This enables the integration of difference operations during the initial phases of the CSG sequence. See the supplementary material for a formal proof of the generality of  $\mathrm{D}^2\mathrm{CSG}$ . CSG-Stump [6] also follows a fixed CSG assembly order while including an inverse layer to model shape complements. The complement operation helps attain generality of their CSG reconstructions, in theory, but the inverse
Table 1: Comparing various aspects of CAPRI-Net [57] and  ${\mathrm{D}}^{2}\mathrm{{CSG}}$  .
CAPRI [57]	D2CSG
Primitives	Convex quadrics	Convex quadrics and their inverses
# Difference operation	One	Multiple
Generalization	Certain shapes	All shapes
CSG layer parameters	≈64K	≈32K
Training time per shape	≈3 mins	≈5 mins
layer is non-differentiable and the difference operations can only be applied to basic primitives, which can severely compromise the compactness and quality of the reconstruction. ExtrudeNet [39] follows the same CSG order as [6], but changes the modeling paradigm to extrusion to improve edibility. Later, SECAD [27] employs 2D implicit fields to produce more accurate 2D profiles for extrusion. The main issue with both methods is the lack of generality of the extrusion-based shapes.
Overfit models. Learning an overfit model to a single shape is not uncommon in applications such as compression [5], reconstruction [46], and level-of-details modeling [42] due to the high-quality results it can produce. The underlying geometry of a NeRF is essentially an overfit (i.e., fixed) to the shape/scene although the primary task of NeRF is novel view synthesis [56, 32]. Following a similar principle, to replicate fine geometric details,  $\mathrm{D}^2\mathrm{CSG}$  constructs a CSG tree for a given object via optimizing a small neural network along with a randomly initialized feature code (Fig. 2). We follow this optimization/overfitting procedure as we did not find learning a prior on CAD shapes very useful and generalizable due to the structural and topological diversity in the CAD shapes.
CSG involves performing operations on a set of primitives (leaves of a CSG tree), resulting in the creation of intermediate shapes that are then merged to form the final shape. In this work, sampled from a Gaussian distribution, the input to  $\mathrm{D}^2\mathrm{CSG}$  is a feature vector that is optimized along with the network's parameters to reconstruct a given 3D shape  $S$  and the output is a set of occupancy values that are optimized to fit  $S$  (similar to Auto-decoder in DeepSDF [36]). We avoid encoding a shape to a latent code since it does not help to converge more accurately or faster (supplementary material). Our design is efficient as  $\mathrm{D}^2\mathrm{CSG}$  uses a light network converging quickly to each shape.
As shown in Fig. 2, starting from a feature vector, we first pass it to the primitive prediction network to generate two matrices  $\mathbf{P}_C$  and  $\mathbf{P}_R$  that hold the primitives' parameters. Each matrix is used to calculate the point-to-primitive approximated signed distance (ASD) matrix  $\mathbf{D}_C$  and  $\mathbf{D}_R$  for query points sampled near the target shape. These two matrices are separately passed to the cover and residual branches, each having an intersection and a union layer. The two branches generate point occupancies  $\mathbf{a}_C$  and  $\mathbf{a}_R$  with respect to the cover and residual shapes  $S_C$  and  $S_R$ , respectively. The final shape is obtained as the difference between  $S_C$  and  $S_R$ . We adopt the multi-stage training of CAPRI-Net [57] to achieve accurate CSG reconstructions. Furthermore, since there is no explicit compactness constraint, many redundant primitives and intermediate shapes exist in the predicted CSG sequence. To this end, we introduce a novel dropout operation on the weights of each CSG layer to iteratively reduce redundancy and achieve higher level of compactness.
Comparison to CAPRI-Net. The most relevant work to our method is CAPRI-Net [57], which assembles a set of quadric primitives via a fixed order of intersection, union, and difference operations. We adopt the same equations as CAPRI-Net to define the convex quadric primitives and CSG operations. However, as highlighted in Table 1, our network,  $\mathrm{D}^2\mathrm{CSG}$ , differs from CAPRI-Net in several aspects. First,  $\mathrm{D}^2\mathrm{CSG}$  uses additional inverse primitives to support nested difference operations, which CAPRI-Net cannot handle. In general, despite of the fixed-order CSG operations, our network can reproduce the result of any CSG assembly, unlike CAPRI-Net, while using fewer network parameters. In addition, the new dropout design leads to more compact CSG assemblies.
3.1 Primitive Representation
The implicit quadric equation from CAPRI-Net [57] can cover frequently used convex primitives, e.g., planes, spheres, etc. However, its fixed CSG order (intersection, union, and difference) is not
general to combine the convexes to produce any shape; see proof in supplementary material. With  $\mathrm{D}^2\mathrm{CSG}$ , we introduce a more general primitive form to also cover inverse convex primitives, so that the difference operation can be utilized as the last, as well as the first, operation of the learned CSG sequence. This is one of the key differences between our method and CAPRI-Net [57], which only produces convex shapes from the intersection layer. Our primitive prediction network (an MLP) receives a code of size 256 and outputs two separate matrices  $\mathbf{P}_C \in \mathbb{R}^{p \times 7}$  and  $\mathbf{P}_R \in \mathbb{R}^{p \times 7}$  (for cover and residual branches), each contains parameters of  $p$  primitives (see Fig. 2). We discuss the importance of separating  $\mathbf{P}_C$  and  $\mathbf{P}_R$  in the next section. Half of the primitives in  $\mathbf{P}_C$  and  $\mathbf{P}_R$  are convex represented by a quadric equation:  $|a|x^2 + |b|y^2 + |c|z^2 + dx + ey + fz + g = 0$ .
Complementary Primitives. In  $\mathrm{D}^2\mathrm{CSG}$ , the other half of the primitives in  $\mathbf{P}_C$  and  $\mathbf{P}_R$  are inverse convex primitives with the first three coefficients to be negative:
$$
- | a | x ^ {2} - | b | y ^ {2} - | c | z ^ {2} + d x + e y + f z + g = 0. \tag {1}
$$
We avoid using a general quadric form without any non-negative constraints since it would produce complex shapes not frequently used in CAD design. For reconstruction,  $n$  points near the shape's surface are sampled and their ASD to all primitives is calculated similarly to CAPRI-Net [57]. For each point  $\mathbf{q}_j = (x_j, y_j, z_j)$ , its ASD is captured in matrix  $\mathbf{D} \in \mathbb{R}^{n \times p}$  as:  $\mathbf{D}_C(j,:) = \mathbf{Q}(j,:) \mathbf{P}_C^T$  and  $\mathbf{D}_R(j,:) = \mathbf{Q}(j,:) \mathbf{P}_R^T$ , where  $\mathbf{Q}(j,:) = (x_j^2, y_j^2, z_j^2, x_j, y_j, z_j, 1)$  is the  $j$ th row of  $\mathbf{Q}$ .
3.2 Dual CSG Branches
Previous CSG learning approaches employ a fixed CSG order and do not apply difference operation as the last operation as opposed to CAPRI-Net [57] and  $\mathrm{D}^2\mathrm{CSG}$ . Applying the difference as the last operation helps produce complex concavity for the reconstructed shape [57]. However, CAPRI-Net utilizes a shared primitive set for the two shapes that undergo the final subtraction operation. This sharing between the two operands has adverse effects on the final result. To backpropagate gradients through all the primitives, values in the selection matrix  $\mathbf{T}$  in the intersection layer are float in the early training stage (Equation 2 and Table 2). Therefore many primitives simultaneously belong to cover and residual shapes even though their associated weights might be small in  $\mathbf{T}$ . Thus, the primitives utilized by the residual shape may impact the cover shape and result in the removal of details, or the opposite may also occur. Our dual branch in  $\mathrm{D}^2\mathrm{CSG}$ , however, allows the cover and residual shapes to utilize distinct primitives, enabling them to better fit their respective target shapes.
Now, we briefly explain our dual CSG branches using similar notations of BSP-Net [3]. We input the ASD matrix  $\mathbf{D}_C$  into the cover branch and  $\mathbf{D}_R$  into the residual branch, and output vector  $\mathbf{a}_C$  and  $\mathbf{a}_R$ , indicating whether query points are inside/outside cover shape  $S_{C}$  and residual shape  $S_{R}$ . Each branch contains an intersection layer and a union layer adopted from BSP-Net [3].
During training, the inputs to intersection layers are two ASD matrices  $\mathbf{D}_R\in \mathbb{R}^{n\times p}$  and  $\mathbf{D}_C\in \mathbb{R}^{n\times p}$ . Primitives involved in forming intersected shapes are selected by two selection matrices  $\mathbf{T}_C\in \mathbb{R}^{p\times c}$  and  $\mathbf{T}_R\in \mathbb{R}^{p\times c}$ , where  $c$  is the number of intermediate shapes. The two selection matrices  $\mathbf{T}_C$  and  $\mathbf{T}_R$  are set as learnable weights in the intersection layer. We can obtain point-to-intermediate-shape indicator  $\mathbf{Con}\in \mathbb{R}^{n\times c}$  by the intersection layer and only when  $\mathbf{Con}(j,i) = 0$ , query point  $\mathbf{q}_j$  is inside the intermediate shape  $i$  ( $\mathbf{Con}_R$  is the same and only subscripts are  $R$ ):
$$
\mathbf {C o n} _ {C} = \operatorname {r e l u} \left(\mathbf {D} _ {C}\right) \mathbf {T} _ {C} \quad \left\{ \begin{array}{l l} 0 & \text {i n}, \\ > 0 & \text {o u t}. \end{array} \right. \tag {2}
$$
Then all the shapes obtained by the intersection operation are combined by two union layers to find the cover shape  $S_{C}$  and residual shape  $S_{R}$ . The inside/outside indicators of the combined shape are stored in vector  $\mathbf{a}_{R}$  and  $\mathbf{a}_{C} \in \mathbb{R}^{n \times 1}$  indicating whether a point is in/outside of the cover and residual shapes. We compute  $\mathbf{a}_{C}$  and  $\mathbf{a}_{R}$  in a multi-stage fashion ( $\mathbf{a}^{+}$  and  $\mathbf{a}^{*}$  for early and final stages) as [57]. Specifically,  $\mathbf{a}_{C}^{*}$  and  $\mathbf{a}_{R}^{*}$  are obtained by finding min of each row of  $\mathbf{Con}_{C}$  and  $\mathbf{Con}_{R}$ :
$$
\mathbf {a} _ {C} ^ {*} (j) = \min  _ {1 \leq i \leq c} \left(\mathbf {C o n} _ {C} (j, i)\right) \quad \left\{ \begin{array}{l l} 0 & \text {i n}, \\ > 0 & \text {o u t}. \end{array} \right. \tag {3}
$$
Since gradients can only be backpropagated to the minimum value in the min operation, we additionally introduce  $\mathbf{a}_C^+$  at the early training stage to facilitate learning by the following equation:
Table 2: Settings for multi-stage training.
Stage	Intersection (T)	Union (W)	Difference Op (a)	Dropout	Loss
0	float	float	a+	-	Lrec+ + LT + LW
1	float	-	a*	-	Lrec* + LT
2	binary	binary	a*	✓	Lrec*
$$
\mathbf {a} _ {C} ^ {+} (j) = \mathcal {C} \left(\sum_ {1 \leq i \leq c} \mathbf {W} _ {C} (i) \mathcal {C} \left(1 - \mathbf {C o n} _ {C} (j, i)\right)\right) \left\{ \begin{array}{l l} 1 & \approx \text {i n}, \\ <   1 & \approx \text {o u t}. \end{array} \right. \tag {4}
$$
$\mathbf{W}_C \in \mathbb{R}^c$  is a learned weighting vector and  $\mathcal{C}$  is a clip operation to  $[0,1]$ .  $\mathbf{a}_R^+$  is defined similarly.
3.3 Loss Functions and Dropout in Training
Our training process is similar to that of CAPRI-Net [57] in terms of staging. The main motivation behind this is to ensure that the values in the CSG layer weights gradually change from float values to binary values, while facilitating better gradient propagation for learning the CSG layer weights in the early stage. While the early stages (0 - 1) of our training and loss functions are the same as CAPRI-Net, stage 2, which is a crucial part of our training due to the injection of a novel dropout, is different. In Table 2, we provide an overview of each stage of training and loss functions, with more details presented below and also in the supplementary material.
Stage 0. At stage 0,  $\mathbf{T}$  in the intersection layer and  $\mathbf{W}$  in the union layer are float. The loss function is  $L^{+} = L_{rec}^{+} + L_{\mathbf{T}} + L_{\mathbf{W}}$ , where  $L_{\mathbf{T}}$  and  $L_{\mathbf{W}}$  serve to limit the entries of  $\mathbf{T}$  to be between 0 and 1, and the entries of  $\mathbf{W}$  to be almost 1. This setup would help the union layer avoid using the min operation in Equation 8 at stage 0 so that all the intermediate shapes could have gradients. Specifically,