SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
L _ {\mathbf {T}} = \sum_ {t \in T} \max  (- t, 0) + \max  (t - 1, 0), L _ {\mathbf {W}} = \sum_ {i} | \mathbf {W} _ {i} - 1 |. \tag {5}
$$
It is worth noting that we have separate network weights for the dual CSG branches:  $\mathbf{T} = [\mathbf{T}_C, \mathbf{T}_R]$  and  $\mathbf{W} = [\mathbf{W}_C, \mathbf{W}_R]$ , and they are trained separately. Our reconstruction loss  $L_{rec}^{+}$  is a combination of the cover loss  $L_C^+$  and the residual loss  $L_R^+$ , which are defined below.
$$
L _ {C} ^ {+} = \frac {1}{n} \sum_ {j = 1} ^ {n} \mathbf {M} _ {C} (j) * (\mathbf {g} (j) - \mathbf {a} _ {C} ^ {+} (j)) ^ {2}, \tag {6}
$$
$$
L _ {R} ^ {+} = \frac {1}{n} \sum_ {j = 1} ^ {n} \mathbf {M} _ {R} (j) * \left(\left(1 - \mathbf {g} (j)\right) - \mathbf {a} _ {R} ^ {+} (j)\right) ^ {2}, \tag {7}
$$
where  $\mathbf{g}$  denotes the ground truth (GT) point occupancy values.  $\mathbf{M}_C(j) = \max (\mathbf{g}(j),\mathbb{1}(\mathbf{a}_R^+ (j) < \beta))$  and  $\mathbf{M}_R(j) = \max (\mathbf{g}(j),\mathbb{1}(\mathbf{a}_C^+ (j) > \beta))$ , where  $\beta$  is set to be 0.5 to signify the shape surface at stage 0. The function  $\mathbb{1}$  maps Boolean values to float, while  $\mathbf{M}_C$  and  $\mathbf{M}_R$  adjust the losses on each branch with respect to the value of the other branch. In the supplementary material, we provide 2D visualization examples to help comprehend these various notations.
Stage 1. Here, the vector  $\mathbf{a}^{+}$  is substituted with  $\mathbf{a}^{*}$  to eliminate the use of  $\mathbf{W}$  from stage 0. This adopts min as the union operation for the cover and residual shapes and restricts the back-propagation of gradients solely to the shapes that underwent a union operation. Note that  $\mathbf{W}$  is not updated at this stage. The loss function is  $L^{*} = L_{rec}^{*} + L_{T}$ , where  $L_{T}$  is the same as in stage 0, and  $L_{rec}^{*}$  is the reconstruction loss:  $L_{rec}^{*} = L_{C}^{*} + L_{R}^{*}$ . The cover loss  $L_{C}^{*}$  and residual loss  $L_{R}^{*}$  are defined as:
$$
L _ {C} ^ {*} = \frac {1}{n} \sum_ {j = 1} ^ {n} \mathbf {M} _ {C} (j) * \left[ \left(1 - \mathbf {g} (j)\right) * \left(1 - \mathbf {a} _ {C} ^ {*} (j)\right) + w _ {C} * \mathbf {g} (j) * \mathbf {a} _ {C} ^ {*} (j) \right], \tag {8}
$$
$$
L _ {R} ^ {*} = \frac {1}{n} \sum_ {j = 1} ^ {n} \mathbf {M} _ {R} (j) * [ \mathbf {g} (j) * (1 - \mathbf {a} _ {R} ^ {*} (j)) + w _ {R} * (1 - \mathbf {g} (j)) * \mathbf {a} _ {R} ^ {*} (j) ]. \tag {9}
$$
Here,  $\mathbf{M}_C(j) = \max (\mathbf{g}(j),\mathbb{1}(\mathbf{a}_R^* (j) > \gamma))$ $\mathbf{M}_R(j) = \max (\mathbf{g}(j),\mathbb{1}(\mathbf{a}_C^* (j) < \gamma))$  ,  $\mathbf{g}$  is the GT point occupancy value operating as a mask, and  $\gamma = 0.01$  is close to the inside value O. We choose to use the  $L_{1}$  loss function in stage 1 so that the different weights can be applied to encourage utilization of the difference operation. In addition, we set  $w_{C} = 10$  and  $w_{R} = 2.5$  , as in CAPRI-Net [57], to encourage the cover and residual shapes to fulfill their roles in the reconstruction.
Stage 2. In stage 2, entries  $t$  in  $T$  are quantized into binary values by a threshold,  $t_{Binary} = (t > \eta)?1:0$ , where  $\eta = 0.01$ . This way, the network will perform interpretable intersection operations on primitives. Due to the non-differentiable nature of the  $t_{Binary}$  values, the loss term  $L_{T}$  is no longer utilized, resulting in the reduction of the loss function to only  $L_{rec}^{*}$  (see Table 2).
Dropout in Stage 2. Without an explicit compactness constraint or loss, redundancies in our CSG reconstruction are inevitable. To alleviate this, we make two crucial observations: first, the CSG sequence learned at stage 2 is entirely interpretable, allowing us to track the primitives and CSG operations involved in creating the final shape; second, altering the primitives or CSG operations will consequently modify the final shape's implicit field. Therefore, we can identify less significant primitives and intermediate shapes if their removal does not substantially impact the final result. Thus, we introduce the importance metric  $\Delta S$  to measure the impact of removing primitives or intermediate shapes on the outcome. We first define the implicit field  $s^*$  of the final shape as follows:
$$
\mathbf {s} ^ {*} (j) = \max  \left(\mathbf {a} _ {C} ^ {*} (j), \alpha - \mathbf {a} _ {R} ^ {*} (j)\right) \left\{ \begin{array}{l l} 0 & \text {i n}, \\ > 0 & \text {o u t}, \end{array} \right. \tag {10}
$$
where  $\mathbf{s}^{*}(j)$  signifies whether a query point  $\mathbf{q}_j$  is inside or outside the reconstructed shape. It is important to note that  $\alpha$  should be small and positive ( $\approx 0.2$ ), as  $\mathbf{s}^*$  approaches 0 when points are within the shape. Then the importance metric  $\Delta \mathbf{S}$  is defined as:
$$
\Delta \mathbf {S} = \sum_ {j = 1} ^ {n} \Delta \mathbb {1} \left(\mathbf {s} ^ {*} (j) <   \frac {\alpha}{2}\right). \tag {11}
$$
Here,  $\mathbf{s}^{*}(j) < \frac{\alpha}{2}$  quantizes implicit field values  $\mathbf{s}^{*}(j)$  from float to Boolean, where inside values are 1 and outside values are 0. Function  $\mathbb{1}$  maps Boolean values to integers for the sum operation,  $\Delta$  captures the difference in values in case of eliminating primitives or intermediate shapes.
Note that union layer parameters  $\mathbf{W}$  are set close to 1 by  $L_{\mathbf{W}}$  in stage 0. During stage 2, at every 4,000 iterations, if removing the intermediate shape  $i$  from the CSG sequence makes  $\Delta S$  smaller than threshold  $\sigma$ , we would discard intermediate shape  $i$  by making  $\mathbf{W}_i = 0$ . Consequently, Equation (6) will be modified to incorporate the updated weights in the union layer:
$$
\mathbf {a} _ {C} ^ {*} (j) = \min  _ {1 \leq i \leq c} \left(\mathbf {C o n} _ {C} (j, i) + (1 - \mathbf {W} _ {i}) * \theta\right) \quad \left\{ \begin{array}{l l} 0 & \text {i n}, \\ > 0 & \text {o u t}. \end{array} \right. \tag {12}
$$
This way, the removed shape  $i$  will not affect  $\mathbf{a}^*$  as long as  $\theta$  is large enough. In our experiments, we discover that  $\theta$  just needs to be larger than  $\alpha$  so that the removed shape  $i$  will not affect  $\mathbf{s}^*$ . Thus we set  $\theta$  as 100 in all experiments. Furthermore, we apply dropout to parameters in the intersection layer by setting  $T_{k,:} = 0$  if  $\Delta \mathbf{S}$  falls below  $\sigma$  after removing primitive  $k$  from the CSG sequence. We set  $\sigma$  as 3 in experiments and examine the impact of  $\sigma$  in the supplementary material. After dropouts, the network parameters will be tuned. We iteratively perform this process until the maximum number of iterations has been reached or when no primitives or intermediate shapes are dropped.
In our experiments to be presented in this section, we set the number of maximum primitives as  $p = 512$  and the number of maximum intersections as  $c = 32$  for each branch to support complex shapes. The size of our latent code is 256 and a two-layer MLP is used to predict the parameters of the primitives from the input feature code. We train  $\mathrm{D}^2\mathrm{CSG}$  per shape by optimizing the latent code, primitive prediction network, intersection layer, and union layer. To evaluate  $\mathrm{D}^2\mathrm{CSG}$  against prior methods, which all require an additional time-consuming optimization at test time to achieve satisfactory results (e.g., 30 min per shape for CSG-Stump), we have randomly selected a moderately sized subset of shapes as test set for evaluation: 500 shapes from ABC, and 50 from each of the 13 categories of ShapeNet (650 shapes in total). In addition, we ensured that 80% of the selected shapes from ABC have genus larger than two with more than 10K vertices to include complex structures. All experiments were performed using an Nvidia GeForce RTX 2080 Ti GPU.
Figure 3: Comparing CSG representation learning and reconstruction from 3D meshes in ABC dataset (columns 1-6) and ShapeNet (columns 7-12). Results are best viewed when zooming in.
4.1 Mesh to CSG Representation
Given a 3D shape, the task is to learn an accurate and compact CSG representation for it. To do so, we first sample 24,576 points around the shape's surface (i.e. with a distance up to  $1/64$ ) and 4,096 random points in 3D space. All 28,672 points are then scaled into the range  $[-0.5, 0.5]$ , these points along with their occupancy values are used to optimize the network.
We compare  $\mathrm{D}^2\mathrm{CSG}$  with InverseCSG [9] (I-CSG), BSP-Net [3], CSG-Stump [6] and CAPRI-Net [57], which output structured parametric primitives. For a fair comparison, we optimize all of these networks with the same number of iterations. BSP-Net, CSG-Stump, and CAPRI-Net are pre-trained on the training set provided in CAPRI-Net before optimization to achieve better initialization. Note that CSG-Stump uses different network settings for shapes from ABC (with shape differences) and ShapeNet (without shape difference); we therefore follow the same settings in our comparisons. For InverseCSG, we adopt the RANSAC parameters used for most of shapes in its CAD shape dataset. For each shape reconstruction, BSP-Net takes about  $15\mathrm{min}$ , I-CSG about  $17\mathrm{min}$ , CSG-Stump about  $30\mathrm{min}$ , and CAPRI-Net about  $3\mathrm{min}$  to converge. In contrast, the training process for  $\mathrm{D}^2\mathrm{CSG}$  runs 12,000 iterations for each stage, taking about 5 minutes per shape reconstruction.
Evaluation Metrics. Quantitative metrics for shape reconstruction are symmetric Chamfer Distance (CD), Normal Consistency (NC), Edge Chamfer Distance [3] (ECD). For ECD, we set the threshold for normal cross products to 0.1 for extracting points close to edges. CD and ECD are computed on 8K sample points on the surface and multiplied by 1,000. In addition, we compare the number of primitives #P to evaluate the compactness of shapes since all CSG-based modeling methods predict some primitives that are combined with intersection operations.
Evaluation and Comparison. We provide visual comparisons on representative examples from the ABC dataset and the ShapeNet dataset in Fig. 3. Since InverseCSG only accepts high-quality CAD meshes as input for its primitive detection algorithm and fails on ShapeNet, we only compare it on the ABC dataset, more visual results can be found in the supplementary material. Our method consistently reconstructs more accurate shapes with geometric details and concavities. The RANSAC based primitive detection algorithm in InverseCSG can easily produce noisy and inaccurate primitives, resulting in many degenerated planes after CSG modeling. BSP-Net simply assembles convex shapes to fit the target shape and obtain less compact results without a difference operation. CSG-Stump tends to use considerably more difference operations to reconstruct shapes. This also causes the shapes' surface to be carved by many redundant primitives (i.e., lack of compactness). In addition, since CSG-Stump does not support difference operations between complex shapes, it fails to reconstruct small holes or intricate concavities. As the intermediate shapes subtracted in CAPRI-Net's fixed CSG sequence are only unions of convex shapes, it fails to reproduce target shapes with complex concavities.  $\mathrm{D}^2\mathrm{CSG}$  achieves the best reconstruction quality and compactness in all metrics on ABC dataset and ShapeNet compared to other methods, except for NC on ShapeNet where  $\mathrm{D}^2\mathrm{CSG}$  underperformed ever so slightly ( $\Delta = 0.003$ ); see Table 3.
Table 3: Comparing CSG rep learning from 3D meshes in ABC and ShapeNet.
ABC	ShapeNet
Methods	I-CSG	BSP	STUMP	CAPRI	Ours	BSP	STUMP	CAPRI	Ours
CD ↓	0.576	0.115	0.383	0.177	0.069	0.164	2.214	0.124	0.119
NC ↑	0.877	0.921	0.850	0.903	0.928	0.882	0.794	0.890	0.887
ECD ↓	6.330	4.047	8.881	3.990	3.091	3.899	6.101	2.035	1.722
#P ↓	43.62	359.38	83.42	66.26	28.62	694.21	228.58	50.94	43.78
Table 4: Ablation study on key components of  $\mathrm{D}^2\mathrm{CSG}$ : complementary primitives (CP), dual branches (DB), and dropout (DO), on three quality metrics and three compactness metrics, the number of CSG primitives (#P), intermediate shapes (#IS), and surface segments (#Seg) resulting from a decomposition induced by the CSG tree. Winner in boldface and second place in blue.
Row ID	CP	DB	DO	CD ↓	NC ↑	ECD ↓	#P ↓	#IS ↓	#Seg ↓
1	-	-	-	0.183	0.907	3.92	77	9.2	93
2	✓	-	-	0.073	0.935	3.12	38	5.8	55
3	✓	-	✓	0.088	0.926	3.48	27	5.3	40
4	✓	✓	-	0.069	0.936	2.98	53	6.8	57
5	✓	✓	✓	0.069	0.928	3.09	29	5.7	42
Table 5: Comparing CSG rep learning from 3D point cloud in ABC and ShapeNet.
ABC	ShapeNet
Methods	BSP	STUMP	CAPRI	Ours	BSP	STUMP	CAPRI	Ours
CD ↓	0.133	0.695	0.225	0.085	0.268	1.177	0.242	0.224
NC ↑	0.919	0.841	0.894	0.924	0.889	0.840	0.888	0.886
ECD ↓	3.899	7.303	3.308	3.029	1.854	3.482	1.971	1.815
#P ↓	360.87	67.436	68.57	35.28	553.76	211.36	55.21	51.26
Ablation Study. We conducted an ablation study (see Table 4) to assess the efficacy of, and tradeoff between, the three key features of  $\mathrm{D}^2\mathrm{CSG}$ : complementary primitives, dual branch design, and dropout. As we already have three metrics for reconstruction quality, we complement #P, the only compactness measure, with two more: the number of intermediate shapes and the number of surface segments as a result of shape decomposition by the CSG tree; see more details on their definitions in the supplementary material. We observe that incorporating complementary primitives (row 2) enables better generalization to shapes with complex structures and higher accuracy than the CAPRI-Net [57] baseline (row 1). The dual branch design further enhances reconstruction accuracy (row 4 vs. 2), as primitives from the residual shape do not erase details of the cover shape, and each branch can better fit its target shape without impacting the other branch. However, the dual branch design did compromise compactness. Replacing dual branching by dropout in the final stage (row 3 vs. 4) decreases the primitive count by about  $50\%$  and improves on the other two compactness measures, while marginally compromising accuracy. The best trade-off is evidently achieved by combining all three components, as shown in row 5, where adding the dual branches has rather minimal effects on all three compactness metrics with dropouts. In the supplementary material, we provide additional ablation studies. We also examine the impact of employing basic primitives, as well as modifying the maximum primitive count and the number of sampled points.
4.2 Applications
Point Clouds to CSG. We reconstruct CAD shapes from point clouds, each containing 8,192 points. For each input point, we sample 8 points along its normal with perturbations sampled from Gaussian distribution  $(\mu = 0, \sigma = 1/64)$ . If this point is at the opposite direction of normal vectors, the occupancy value is 1, otherwise it is 0. This way, we sample 65,536 points to fit the network to each shape and other settings the same as mesh-to-CSG experiment. Quantitative comparisons in Table 5 and visual comparisons in Fig. 4 show that our network outperforms BSP-Net, CSG-Stump, and CAPRI-Net in different reconstruction similarity and compactness metrics on ABC dataset and ShapeNet. More results can be found in the supplementary material.
Figure 4: CSG reconstruction from 3D point cloud in ABC (Col. 1-6) and ShapeNet (Col. 7-12).
Figure 5:  $\mathrm{D}^2\mathrm{CSG}$  learns OpenSCAD scripts for a given shape and supports editability.
Shape Editing. Once we have acquired the primitives and CSG assembly operations, we can edit the shape by altering primitive parameters, adjusting the CSG sequence, or transforming intermediate shapes. To facilitate shape editing in popular CAD software, we further convert quadric primitives into basic primitives (e.g., cubes, spheres, cylinder, etc.) by identifying the best-fitting basic primitive for each predicted quadric primitive by our method. Subsequently, we can export the primitive parameters and CSG sequence into an OpenSCAD script. Fig. 5 shows the shape editing results.
5 Discussion, Limitation, and Future Work
We present  $\mathrm{D}^2\mathrm{CSG}$ , a simple yet effective idea, for unsupervised learning of general and compact CSG tree representations of 3D CAD objects. Extensive experiments on the ABC and ShapeNet datasets demonstrate that our network outperforms state-of-the-art methods both in reconstruction quality and compactness. We also have ample visual evidence that the CSG trees obtained by our method tend to be more compact than those produced by prior approaches.
Our network does not generalize over a shape collection; it "overfits" to a single input shape and is in essence an optimization to find a CSG assembly. While arguably limiting, this is not entirely unjustified since some of the models in the ABC dataset may not possess sufficient generalizability in their primitive assemblies. Another limitation is that our current quadric primitive representation cannot fully support complex surfaces, such as tori or NURBS. It would be an interesting future direction to change quadric primitives to NURBS or other easily editable primitives, such as those from extrusions. In addition, incorporating interpretable CSG operations into the network tends to cause gradient back-propagation issues and limits the reconstruction accuracy of small details such as decorative curves on chair legs. We would also like to extend our method to structured CAD shape reconstruction from images and free-form sketches. Another interesting direction for future work is to scale the primitive assembly optimization from CAD parts to indoor scenes.
We thank all the anonymous reviewers for their valuable comments, and KOKONI3D (https://www.kokoni3d.com/) for the helpful discussions. This work was supported in part by an NSERC Discovery Grant (No.611370) and industrial gift funds from Adobe and Autodesk.
[54] Kaizhi Yang and Xuejin Chen. Unsupervised learning for cuboid shape abstraction via joint segmentation from point clouds. ACM Transactions on Graphics (TOG), 40(4):1-11, 2021.
[55] Chun-Han Yao, Wei-Chih Hung, Varun Jampani, and Ming-Hsuan Yang. Discovering 3d parts from image collections. In ICCV, pages 12981-12990, 2021.
[56] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-4815, 2021.
[57] Fenggen Yu, Zhiqin Chen, Manyi Li, Aditya Sanghi, Hooman Shayani, Ali Mahdavi-Amiri, and Hao Zhang. Capri-net: Learning compact cad shapes with adaptive primitive assembly. In CVPR, pages 11768-11778, 2022.
[58] Chuhang Zou, Ersin Yumer, Jimei Yang, Duygu Ceylan, and Derek Hoiem. 3d-prnn: Generating shape primitives with recurrent neural networks. In ICCV, pages 900-909, 2017.