MV2Cyl: Reconstructing 3D Extrusion Cylinders from Multi-View Images

Authors: eunji  hong
minh hieu nguyen
mikaela angelina uy
minhyuk  sung
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 09:51:30

Abstract: We present MV2Cyl, a novel method for reconstructing 3D from 2D multi-view images, not merely as a field or raw geometry but as a sketch-extrude CAD. Extracting extrusion cylinders from raw 3D geometry has been extensively researched in computer vision, while the processing of 3D data through neural networks has remained a bottleneck. Since 3D scans are generally accompanied by multi-view images, leveraging 2D convolutional neural networks allows these images to be exploited as a rich source for extracting extrusion cylinder information. However, we observe that extracting only the surface information of the extrudes and utilizing it results in suboptimal outcomes due to the challenges in the occlusion and surface segmentation. By synergizing with the extracted base curve information, we achieve the optimal reconstruction result with the best accuracy in 2D sketch and extrude parameter estimation. Our experiments, comparing our method with previous work that takes a raw 3D point cloud as input, demonstrate the effectiveness of our approach by taking advantage of multi-view images.

Main paper: 
 introduction: 
 Most human-made objects in our daily lives are created using computer-aided design (CAD). Reconstructing the structural representation from raw geometry, such as 3D scans, is essential to enable the fabrication of 3D shapes and manipulate them for diverse downstream applications. Sketch-Extrude CAD model is particularly notable not only as the most common but also as a versatile CAD representation, which enables the expression of diverse shapes with 2D sketches and extruding heights.The reconstruction of a set of extrusion cylinders from raw 3D geometry has been extensively explored in previous studies [58, 65, 49, 30] , yet the suboptimal quality of the results remains a challenge. Unsupervised methods [49, 30] have demonstrated notable capabilities in parsing raw geometry but often struggle to precisely segment the shape and part-wise fit the output extrusions to the given shape. Supervised methods using a language-based architecture, such as those described in [65] , often face challenges in producing valid outputs. A notable exception is Point2Cyl [58] , which is a supervised method proposing to first segment the given raw point cloud into each extrusion region and estimate the 2D sketch and the extrusion parameters using the predicted segments and surface normal information. While it has shown successful results, the bottleneck remains in the limited performance of the 3D backbone network for segmentation.To address the limitation, we focus on the fact that 3D scans are commonly accompanied by multiview images, which are now even solely used to reconstruct a 3D shape without depth or other information thanks to the recent advancements in neural rendering techniques [40, 61, 13, 62, 5] . 2D multi-view images are a rich resource for extracting 3D extrusion information. Also, 2D convolutional neural networks have been utilized in various 3D tasks [34, 63] , owing to their superior performance compared to 3D processing networks. In light of these observations, we introduce a novel framework called MV2Cyl, which reconstructs a set of extrusion cylinders from multi-view images of an object without relying on raw geometry as input.A straightforward approach to exploiting multi-views in extrusion reconstruction is to first segment 2D regions of the extrusion in each image and reconstruct the 3D shape using the extrusion labels. However, achieving precise 3D surface reconstruction solely from the multi-view images poses challenges due to the ambiguity between the object's intrinsic color (albedo) and the effects of lighting (shading) or shadows within images (Fig. 5 ). In contrast, we observe that reconstructing only the 2D sketches of the two ends of the extrusion cylinders-the start and end planes of extrusion-provides much more accurate results while avoiding the ambiguity issue and the consequential error accumulation. However, this approach also results in failure cases when one of the bases is not properly detected in 2D image due to the sparse viewpoints. Therefore, we propose a framework that synergizes the reconstructions of labeled surfaces and labeled curves of the extrusion bases so that parameters such as extrusion center and height can be better calculated from the surface information, while the 2D sketches can be precisely recovered from the curve information. In our experiments, we demonstrate the superior performance of our method MV2Cyl on two sketch-extrude datasets: Fusion360 [64] and DeepCAD [65] .2 Related Work 3D Primitive Fitting. Primitive fitting has been extensively investigated in computer vision. One line of work involves decomposing into surface/volumetric primitives. Classical approaches include detecting simple primitives such as planes [6, 11, 42] through RANSAC [51, 31] , region growing [45] , or Hough voting [3, 8] . The emergence of neural networks gave rise to data-driven methods [29, 56, 28] learning frameworks that fit simple primitives such as planes, cylinders, cones, and spheres. Primitive fitting has also been used to approach the task of shape abstraction [75, 57] that fit cuboids given an input shape or image. However, these works all assume a fixed predefined set of primitives that may not be able to cover complex real-world objects. The focus of our work is to recover a more general primitive, i.e., extrusion cylinders, that are defined by any arbitrary closed loop and hence can explain more complex objects.Another line of work tackles fitting 3D parametric curves, where a common strategy is to first detect edges [70] and corners [35, 70, 39] from an input point cloud, and then fit or predict parametric curves from the initial proposals. A recent work, NEF [68] , takes multi-view images as input and introduces a self-supervised approach to the parametric curve fitting problem using neural fields, thus allowing them to optimize without needing clean point cloud data. Despite having multi-view images as input, NEF only tackles recovering parametric curves and not extrusion cylinders, which is a basic building block for the sketch-extrude reverse engineering task and is the focus of our work.CAD Reconstruction for Reverse Engineering. CAD reverse engineering has been a well-studied problem in computer graphics and CAD communities [4, 60, 2, 1, 38] , enabling the recovery of a structured representation of shapes for applications such as editing and manufacturing. Earlier works focused on reconstructing CAD through inverse CSG modeling [9, 55, 22, 48, 69] , which decomposes shapes into simple primitives combined with boolean operations. While achieving good reconstruction, CSG-like methods tend to combine a large number of shape primitives, making them less flexible or easily manipulated by users. Recent efforts in collating large-scale CAD datasets (ABC [25] , Fusion 360 [64] , SketchGraphs [54] ) have pushed this boundary, enabling the learning of data-driven methods for various applications such as B-Rep classification [19] and segmentation [19, 26, 10] , parametric surface inference [16, 36] , and CAD reconstruction [23, 27] and generation [66] . For the reverse engineering task, a common CAD modeling paradigm uses sketch-extrude [64, 58] as the basic building block, where a sketch [67, 47] is defined as a sequence of parametric curves forming a closed loop. This results in primitives that are more flexible and complex, not a predefined finite set, which will be the focus of this work.Previous research uses NLP-based language models conditioned on raw geometry to represent CAD models as token sequences [65, 14, 18, 72] . However these methods are not designed to understand and decompose geometry into individual primitives and may also result in invalid CAD sequences. A recent work [21] also models CAD with language but face similar issues despite allowing multi-view inputs. To address these issues, Point2Cyl [58] is the pioneering work that poses the CAD reverse engineering task as a geometry-aware decomposition problem. The idea was to first decompose an input point cloud into extrusion cylinders (sketch-extrude primitives) and introduce differentiable loss functions to recover the parameters. ExtrudeNet [49] and SECAD-Net [30] further built on this setting and extended it to the unsupervised and self-supervised settings, respectively. Our work tackles a similar setting that takes a geometry-aware approach to the reverse engineering task, except we take multi-view images as input, as opposed to existing works that only handle point cloud inputs. We propose a method that reconstructs 3D extrusion cylinders from multi-view images without relying on raw 3D geometry as input. The idea is to leverage 2D neural networks to learn 2D priors that provide 3D extrusion information (Sec. 3.2), i.e. extrusion curves and surfaces. The integration of the information into 3D is achieved through optimizing neural fields (Sec. 3.3). MV2Cyl is a combination of a curve field and a surface field that is used to recover the parameters and reconstruct the extrusion cylinders only given 2D input images (Sec. 3.4).For each extrusion cylinder, the extrusion axis n is computed through plane fitting via RANSAC [12] on the surface base points with corresponding extrusion cylinder instance label. The axis n is then set as the normal of the fitted plane. This process relies on surface information because plane fitting is more reliable using the surface points. If the base face of the cylinder fails to be reconstructed due to occlusion, we use curve points as a substitute. 
2d sketch s estimation:: 
 To obtain the sketch S for each extrusion cylinder, we first project the curve point cloud for each extrusion instance onto the plane with normal n. From the projected point cloud, we then compute for the sketch scale s. We then optimize an implicit signed distance function for each closed loop using the projected point cloud with IGR [15] . Finally, we obtain the closed loop by extracting the zero-level set of the implicit function. Implicit sketches can be converted to a parametric representation using an off-the-shelf spline fitting module. We provide example visualizations in Appendix E.7. 
problem statement and background: 
 Given a set of 2D RGB images {I i } N i=1 where I ∈ R H×W ×3 , the goal is to recover a set of extrusion cylinders {E j } K j=1 that represent the underlying 3D shape. We provide an overview here but refer the readers to Uy et al. [58] for more details.Sketch-Extrude CAD is a designing paradigm or the designed model itself in the area of computeraided design (CAD) consists of extrusion cylinders as its building blocks. An extrusion cylinder E is 3D space defined as E = (n, c, h, S, s) with the extrusion axis n ∈ S 2 , the extrusion center c ∈ R 3 , the extrusion height h ∈ R, and the normalized sketch S scaled by s ∈ R. A sketch-extrude CAD model is a 3D shape that is reconstructed from a set of extrusion cylinders, {E 1 , E 2 , ..., E K } where K is the number of extrusion cylinders. A sketch S is a set of non-self-intersecting closed loops drawn in a normalized plane.We further classify the surfaces of a sketch-extrude CAD model as base or barrel. A surface is a base if its surface normal is parallel to its extrusion axis n, and is a barrel if its surface normal is perpendicular to its extrusion axis. By this definition, base surfaces are parameterized by points on the planes and a normal (n). The base surfaces can further be distinguished into start plane and end plane where the start plane contains the point ch 2 n, while the end plane contains the point c + h 2 n. With these definitions, we will later show how our method, MV2Cyl, is able to recover the set of extrusion cylinders and their corresponding parameters given only 2D multi-view images.To reconstruct extrusion cylinders from multi-view images, we first learn 2D priors for 3D extrusions by exploiting 2D convolutional neural networks. We train two U-Net-based 2D segmentation frameworks: M curve that extracts curve information and M surface that extracts surface information. The 2D information extracted from the two frameworks is integrated into 3D using a neural field and then utilized for more robust reverse engineering. We detail each 2D segmentation frameworks below. 2D Surface Segmentation Framework M surface . The goal of the surface segmentation network is to extract the extrusion instance segmentation as well as start plane, end plane, and barrel (start-end-barrel) segmentation given an input image. That is M surface : R H×W ×3 → {P surface ∈ {0, ..., K} H×W , Q surface ∈ {0, 1, 2, 3} H×W }, where K is the number of instances, P surface represents the extrusion instance segmentation label, and Q surface denotes the start-end-barrel segmentation label. The zero index is used for background annotation. M surface is implemented as two distinct U-Nets predicting each segmentation and is trained using a multi-class cross entropy loss with ground truth labels Psurface and Qsurface . An important property is that the problem does not admit a unique solution as i) the extrusion segment can be ordered arbitrarily and ii) the start and end planes can also be arbitrarily labeled. To handle this, we use Hungarian matching to find the best one-to-one matching with the ground truth labels and reorder it as Psurface and Qsurface . This gives us the loss function of the instance segmentation:EQUATIONwhere P surface hwk is the model's predicted probability that pixel (h, w) belongs to the k-th instance in the b-th image in the batch, and Psurface hw is the pseudo GT label for that pixel, with the number of images in a batch B and the number of possible instances including the background K. The loss function of the start-end-barrel segmentation appears the similar, replacing the P as Q and the range of classes {0, ..., K} to {0, 1, 2, 3}.2D Curve Segmentation Framework M curve . We observe that detecting and extracting curves on the base planes of each extrusion cylinder segment provides additional information to recover the parameters and reconstruct the extrusion cylinders. Moreover, (feature) curves have been a longstanding and well-explored problem dating back to classical computer vision literature [46, 74, 43] thanks to their strong expressiveness. This leads us to learn a 2D curve prior that provides more discriminative and detailed outputs to make reverse engineering more robust.The goal of the curve segmentation framework is to extract both extrusion instance segmentation and start-end segmentation, where the start-end plane segmentation distinguishes the curves of the two base planes of extrusion cylinders. Concretely, M curve : R H×W ×3 → P curve ∈ {0, ..., K} H×W , Q curve ∈ {0, 1, 2} H×W , where P curve is the extrusion instance segmentation label and Q curve is the start-end segmentation label.Similar to M surface , the zero index is used for background annotation in both segmentation tasks and M curve is also implemented with two U-Nets and trained using a multi-class cross-entropy loss against the pseudo ground truth Pcurve , Qcurve that is reordered with Hungarian matching to handle order invariance and ambiguities. An additional challenge for curves compared to surfaces is the labels are a lot sparser with the majority of the images being background pixels. Hence, the model can easily fail to predict meaningful labels due to this label imbalance. To alleviate this, we additionally employ a dice loss [41] to handle the strong foreground-background imbalance and a focal loss [33] to circumvent the class imbalance between the extrusion instances. Hence the loss function used to train the curve prior is given as:EQUATION)Additional details about the segmentation framework are available in the appendix E.1. To establish correspondences and collate the information extracted from multi-view images, a natural approach to integrate 2D information into a consistent 3D representation is through learning a 3D field [40, 61, 68, 44] . For each of our 2D priors M surface and M curve , we learn a density field F and a semantic field A as detailed below, respectively.Density Field F. We learn a density field for surfaces F surface and curves F curve . Given a query point in 3D space x ∈ R 3 , the density field network F : R 3 → R outputs a scalar value that indicates how likely the 3D point x is on a surface or a curve for F surface and curves F curve , respectively. To optimize the field differentiably with only 2D images, we use volume rendering [40, 5] and a learnable transformation [68] to convert the density field F(x) to an opacity field σ(x). This transformation is in the form of a learnable sigmoid function given as:EQUATIONwhere α is a learnable parameter that adjusts the density scale, and β and ζ are constant hyperparameters. We use the same transformation in Eq. 3 for the surface field F surface and the curve field F curve , resulting in corresponding opacity fields σ surface and σ curve . We then directly render the density by using the volume rendering equation to get the expected surface and curve density for a camera ray r, which is given by the integralsEQUATIONEQUATIONwhere T surface (t) = t 0 σ surface (r(s))ds and T curve (t) = t 0 σ curve (r(s))ds are the corresponding transmittance. Inspired by NEF [68] , we train our surface field F surface and curve field F curve using an adaptive L2 loss for the reconstruction of pixel density and a sparsity loss for learning of a sharp distribution of density along the ray. The objective function is given as follows:EQUATIONwhere E(r) and Ê(r) are the 2D-observed and predicted field density, respectively, for both the surface and curve models, i indexes background pixels from input maps, j indexes the sample points along the rays, and s determines the scale of the regularizer. The 2D-observed density E(r) is derived from the output of the 2D prior being a foreground pixel, either start, end, or barrel pixel.Following Ye et al. [68] , W batch (r) is given as:EQUATIONThe weight W batch (r) addresses the imbalance between foreground and background areas in the density maps, preventing the network from converging to a local minimum that falsely classifies most pixels as background, which are more prevalent.Semantic Field A. Now with the density field, we can distill the information from the 2D priors by learning semantic fields A for both surfaces and curves, which we denote as A surface and A curve . Following Zhi et al. [73] , each semantic is modeled as an additional feature channel branching out from the density field F. For surfaces, the additional semantic field A surface = p surface , q surface is comprised of two networks p surface : R 3 → R (K+1) that predicts the extrusion instance label, and q surface : R 3 → R 4 that predicts the start-end-barrel segmentation. Similarly for the curves, the semantic field is given as A curve = {p curve , q curve }, where p curve : R 3 → R (K+1) predicts the extrusion instance label, and q curve : R 3 → R 3 predicts the start-end segmentation. The expected semantic for a ray r is then computed using the volume rendering equation which is given as:EQUATIONfor each a ∈ {p surface , q surface , p curve , q curve } that correspond to rendered predicted semantics Â ∈ { Psurface , Qsurface , Pcurve , Qcurve } from the 3D field.We train our semantic field using the predictions from our learned 2D priors F surface and F curve . For each of the 2D multi-view images, we can extract the 2D observed labels A ∈ {P surface , Q surface , P curve , Q curve } from the learned priors, which we use as supervision. As mentioned in Sec. 3.2, since the labels may not individually be consistent across the multiple views, we use Hungarian matching during training to align the labels across multiple views based on the best one-to-one matching. For each semantic, we use a multi-class cross-entropy loss to train and optimize the semantic field with the given supervision:EQUATIONwhere A l (r) and Âl (r) represent the class l semantic probabilities from the learned 2D prior and the predicted field, respectively, aligned with Hungarian matching.Training and Implementation Details. The integration into a coherent 3D field is optimized independently for each 3D shape. We use TensoRF [5] as the backbone representation for the 3D fields, where we have one model for the surface field and the other for the curve field. In both models, we train the density fields F and the semantic fields A together for 1500 iterations. Each semantic field is parameterized by a 2-layer MLP that outputs logit values. Additional implementation details are provided in the Appendix E.2.We now elaborate on how we leverage our optimized surface field F surface and curve field F curve for the reverse engineering task, specifically to recover extrusion cylinders and their defining parameters, (n, c, h, S, s). Each point within the extracted point clouds is assigned semantic information queried from the associated semantic fields, providing context for each feature in 3D space. Using these enriched point clouds, we employ a multi-step process to reconstruct each extrusion cylinder, iteratively refining the parameters to accurately fit the geometry and semantics. Further details can be found in Appendix E.3.The height h of each extrusion cylinder is computed as the distance between the two sketch centers -from the start plane and the end plane. The sketch centers are computed from the projected curve point cloud for each paired labels (extrusion instance segmentation, start-end segmentation) onto the plane with normal n. If any of the startend planes are occluded, the height is computed using the barrel points from the surface point cloud with the corresponding extrusion cylinder instance label. 4. Extrusion centroid c Estimation: The centroid c of each extrusion cylinder can be obtained by taking the average position of the 3D curve point cloud with the corresponding extrusion cylinder instance label. Similar to the above, if any of the start-end planes are occluded or missing, the centroid is computed as the average of the barrel points from the surface point cloud with the corresponding extrusion cylinder instance label. 
experiments: 
 In this section, we present our experimental evaluation to demonstrate the feasibility of our MV2Cyl in reconstructing extrusion cylinders given only multi-view images. To the best of our knowledge, our proposed pipeline is the first to tackle extrusion cylinder recovery directly from 2D input images, and we show that our method is outperforms existing methods that take clean geometry as input and their modifications to handle raw input images. Datasets. We evaluate our approach on two sketch-extrude CAD datasets Fusion360 [64] and DeepCAD [65] . We use the train and test splits as released in [58] . We enrich these datasets with multi-view image renderings and 2D curve and surface segmentation maps for the training and evaluation of our method and the baselines. More details about dataset generation are in Appendix E.4.Evaluation Metrics. Following [58] , we report the average extrusion-axis error (E.A.), extrusioncenter error (E.C.), per-extrusion cylinder fitting loss (Fit Cyl.), and global fitting loss (Fit Glob.). Additionally, we also report the extrusion-height error (E.H.) defined as the L1 distance between the GT and predicted height value:1 K K k=1 | ĥk -h k |.These metrics evaluate how well the methods reconstruct and recover the parameters of the extrusion cylinders. The final scores are the average of the metrics calculated over all test shapes.Baselines. In the absence of established methods for transforming multi-view images into sketchextrude CAD models within shape space, we benchmark our model against a point cloud to sketchextrude CAD reconstruction technique [58] and its variant to validate the effectiveness of our proposed approach.• We compare with Point2Cyl [58] , a technique for reconstructing sketch-extrude CAD given input 3D point clouds. The point clouds are sampled from clean ground truth CAD meshes and are utilized as the input for this baseline. Point2Cyl [58] initially segments the input point cloud into distinct instances and base/barrel segments, then identifies the extrusion axis as the direction that aligns with the normals of the points in the base segments and is orthogonal to the normals of the points in the barrel segments. The rest of the parameters are inferred based on this predicted axis and the segmentation labels. To evaluate the network, we trained it using the official code released by the authors. • NeuS2 [62] +Point2Cyl [58] combines NeuS2 [62] with Point2Cyl [58] . Since no prior work has reconstructed extrusion cylinders directly from multi-view images, we combine methods for i) shape reconstruction from multi-view images, with techniques for ii) extrusion cylinder reconstruction from raw geometry. This approach utilizes NeuS2 [62] , an off-the-shelf image-based surface reconstruction technique, that optimizes the photometric loss to learn underlying implicit 3D representations. For this baseline, we first reconstruct the object surface from multi-view images using NeuS2 [62] , and then sample a point cloud from the reconstructed mesh. The sampled point cloud is then directly fed into Point2Cyl [58] to predict sketches and extrusion parameters of CAD models.Tab. 1 shows the quantitative results of CAD reconstruction on the Fusion360 [64] and DeepCAD [65] datasets. We see that MV2Cyl outperforms the baselines by considerable margins across both datasets in all evaluated metrics. Qualitative comparisons are shown in Fig. 5 . NeuS2- Point2Cyl (Mesh) NeuS2- Point2Cyl (CAD) Point2Cyl (PC) Point2Cyl (CAD) (a) (b) (c) (d) (e) (f) (g)Figure 5 : Qualitative comparisons with the baselines. Each instance is identified by a different color. MV2Cyl produces high-quality geometry and even outperforms Point2Cyl [58] that directly consumes 3D point clouds. Furthermore, the comparison against a naive baseline that pipelines NeuS2 [62] , a multi-view surface reconstruction technique, to Point2Cyl demonstrates the importance of edge information when inferring 3D structures.The results of NeuS2 [62] +Point2Cyl [58] illustrate the challenges in naively using multi-view images for CAD reconstruction, where inaccurate 3D geometry reconstructed from images serve as a critical bottleneck in recovering CAD parameters. We observe that it struggles to distinguish the object's intrinsic color (albedo) from the effects of lighting or shadows (shading). This can result in discrepancies, such as omitted details or subtle smoothing at the intersection between different extrusion segments, as illustrated in (Fig. 5 ). As seen in Fig. 5 (c), a common failure arises when NeuS2 [62] fails to reconstruct fine geometric details, in this case, the boundaries between the extrusion cylinders. Such artifacts in the reconstructed mesh make the conversion from the point cloud to the CAD model unreliable.MV2Cyl also outperforms Point2Cyl [58] , which takes precise 3D geometric information as input.An advantage of taking multi-view images instead of point clouds is the images contain richer information such as edges and curves. Moreover, 2D neural networks have also shown superior performance compared to 3D processing networks leading to superior performance of MV2Cyl compared to Point2Cyl [58] . The comparison of the 2D segmentation network in MV2Cyl with the 3D segmentation network in Point2Cyl are found in Appendix F.1.Unlike the baselines that require accurate 3D surface information, MV2Cyl is able to directly handle 2D multi-view images as input by leveraging on learned 2D priors to extract 3D extrusion information. We show that extracting extrusion surfaces and curves from 2D images allows us to leverage both the curve and surface representation for better extrusion cylinder reconstruction compared to baselines that even require clean 3D geometry as input.We validate MV2Cyl on a real-world demo by 3D-printing various sketch-extrude objects from the Fusion360 test set, by capturing multi-view images with an iPhone 12 Mini and then obtaining camera poses via COLMAP. These captured images were processed through our pipeline to extract 2D information, converted to 3D using our density and semantic fields to recover extrusion cylinders and parameters. To address the domain gap between synthetic training images and real-world data, we applied preprocessing techniques, including grayscale conversion, background removal, and then fine-tuning a large vision model. Additional details are provided in Appendix B. 
conclusion: 
 We introduce MV2Cyl, a novel method for reconstructing a set of extrusion cylinders in 3D from multi-view images. Unlike previous approaches that rely only on raw 3D geometry as input, our framework utilized 2D multi-view images and demonstrates superior performance by leveraging the capabilities of 2D convolutional neural networks in image segmentation. Through the analysis of curve and surface representations, each having its own advantages and shortcomings, we propose an integration of both that takes the best of both worlds, achieving optimal performance.Limitations and Future Work. Our proposed method is not without its limitations. While MV2Cyl demonstrates state-of-the-art performance in reconstructing sketch-extrude CAD models, it shares a limitation with previous approaches, such as Point2Cyl [58] , regarding the prediction of binary operations among primitives. MV2Cyl does not explicitly predict these binary operations, which we leave as future work. Nonetheless, we propose an initial straightforward method for recovering binary operations through exhaustive search. Details are provided in the appendix E.6. Additionally, MV2Cyl does not directly accommodate textured CAD models, as the 2D segmentation models are trained on a synthetic dataset derived from rendering untextured CAD models. By utilizing generalizable large 2D models, such as Segment Anything [24] , we can preprocess images of textured CAD models for segmentation by extracting object masks and removing textures. We also leave this for future investigation.MV2Cyl is also susceptible to occlusion since it relies on 2D image inputs. Specifically, if one side of an extrusion cylinder is completely hidden by others, our 2D segmentation model cannot detect the hidden side. In such cases, the extrusion cylinder cannot be reconstructed. In Fig. 6 , the left shape represents the target CAD model, while the right one shows the reconstructed model by MV2Cyl.The target shape features an inset hexagonal cylinder with one end hidden by an outer cylinder, but MV2Cyl was unable to reconstruct the inset extrusion cylinder. Societal Impacts. This technology could make CAD more accessible to non-professionals or educational sectors, allowing more people to engage in design and engineering tasks without needing extensive training in traditional CAD software. However, such a 3D reconstruction method can also raise privacy concerns. By converting a physical object into a digital format, the shape could be reused for design purposes without the original creator's permission.reconstruction capability. Specifically, curve representation is effective against occlusion and precisely captures the closed curve of a sketch, making it suitable for reconstructing the 2D implicit sketch. However, it may not always be reconstructed successfully due to its localized presence within the scene. On the other hand, surface representation provides a more dependable basis for fitting a 3D plane to determine the extrusion axis, which is vital for inferring subsequent CAD parameters. Yet, the base faces necessary for this process might not be available. This is either because of subtractive boolean operations or occlusion from combining multiple shapes. We thus address the challenges by jointly reconstructing curves and surface representations of shape and utilizing them to recover CAD parameters. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: This research adhered to the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss societal impacts in the last section of the main paper.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This work does not pose such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All existing assets used in this work are properly cited in the text.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: In MV2Cyl, the curve and surface segmentation modules are kept separate. To evaluate how segmentation performance changes when the 2D curve and surface segmentation networks are combined, we test a shared U-Net backbone that branches into two outputs for curve and surface segmentation. The results of using merged and separate U-Nets to extract 2D segmentation maps for our reverse engineering extrusion cylinders task are shown in Tab. A3. While a slight improvement in performance is observed on some metrics (Fit Cyl. and Fit Glob.), the results from both approaches are comparable. We evaluate the impact of varying the number of instances on MV2Cyl's performance using the Fusion360 dataset. Tab. A4 shows the results breakdown of the models in the Fusion360 test set across different numbers of extrusion instances K. Note that the number of samples for K = 7 and K = 8 is very small, and these statistics should therefore be interpreted sparingly. In this section, we analyze the effect of varying the number of input images on MV2Cyl's performance using the Fusion360 dataset. Tab. A5 shows that our approach is robust to changes in the number of input images, maintaining reasonable performance even with as few as 10 input views. Although performance slightly degrades as the number of input images decreases, MV2Cyl still produces effective reconstructions, demonstrating its adaptability to limited input data. As expected, the use of 50 input images yields the best overall results due to the richer information available for reconstruction. In this section, we investigate the impact of varying the line width of curve segmentation maps on MV2Cyl's performance. Tab. A6 shows that our method is relatively robust to changes in line width, with performance remaining consistent across different widths. The results indicate that using 5 pixels (our default setting) achieves a good balance between accuracy and consistency. While slight variations in error metrics are observed when using line widths of 2.5 or 7.5 pixels, these changes are minimal, suggesting that MV2Cyl can handle a range of line width settings without significant performance loss. In this section, we showcase the feasibility of using MV2Cyl in reconstructing CAD models from real-world object captures. We provide the details of the process as follows.To reverse engineer a real-world object, we begin by capturing videos of the target object using an iPhone 12 Mini. These videos are then converted into multi-view images through frame extraction, which are subsequently input into the image segmentation models. The training dataset for our segmentation models is generated by rendering untextured models in Blender, producing RGB images with a grayscale appearance against a transparent background. To address the domain gap between the training data and real-world demo images, we preprocess the real images by converting them to grayscale and removing the background. While these preprocessing steps help align the datasets, a residual domain gap remains, which may still impact segmentation performance on real images.To address this challenge and enhance segmentation efficacy, we employ a robust 2D segmentation model known as Segment Anything (SAM) [24] . By fine-tuning the pretrained SAM model using LoRA-based finetuning code [71] , with the same training dataset applied to the U-Nets, we observe substantial improvements in segmentation performance on real images, even in the presence of distributional shifts. The finetuned SAM model leverages extensive prior knowledge gained from training on a significantly larger dataset, ensuring robust performance in 2D segmentation tasks. We acquire the camera's extrinsic and intrinsic parameters using COLMAP [52, 53] .Fig. A8 displays an example of a real image alongside the predicted segmentation maps from the finetuned SAM model. The processed multi-view images and their corresponding camera poses are then input into the reconstruction pipeline, which focuses on learning the density and semantic fields. Subsequently, CAD parameters are estimated based on this reconstruction. Tab. A7 presents results on final reconstructed CAD models in real captures, demonstrating successful output of CAD models from various real-world objects. When comparing the real and synthetic images from similar viewpoints, the segmentation outcomes are similarly effective.We quantitatively evaluate MV2Cyl on real data found in Tab. A7. Our reconstruction achieves an average Chamfer distance of 2.11 × 10 -3 from the ground truth CAD model. To compute this distance, we first sampled 8192-point point clouds from the reconstructions and then aligned the reconstructed model to the corresponding ground truth shape using ICP. After alignment, the desired metrics were computed. This process is illustrated in Fig. A9 . We further compare our MV2Cyl against an unsupervised CAD reconstruction baseline SECAD-Net [30] . Unlike our approach that takes multi-view images as input, SECAD-Net takes a 3D voxel grid as input. It trains a model that directly outputs a shape as a union of predicted extrusion cylinders.We evaluate how well their approach reconstructs individual extrusion cylinders using the same metrics as in [58] and compare them against our MV2Cyl. Results are shown in Tab. A8. We see that our approach outperforms SECAD-Net [30] for both the Fusion 360 [64] and DeepCAD [65] datasets on all metrics that quantify extrusion cylinder reconstruction.For fair comparison, we evaluated the test set as released by [58] on both datasets and used the scripts from the code release of SECAD-Net [30] to convert the test shapes to voxelized inputs and to train or fine-tune the SECAD-Net model with our data. For both datasets, we adhered to the evaluation protocol from [30] , fine-tuning SECAD-Net for 300 epochs per test shape using a pre-trained model, followed by mesh extraction at a 256 resolution with Marching Cubes [37] . We used Hungarian matching based on Chamfer distances to find the best match between ground-truth and predicted segments, which allowed us to calculate the metrics reported. The Fusion360 dataset evaluations were conducted on 798 test shapes, while the DeepCAD dataset included 1,950 test shapes. These numbers reflect instances where both MV2Cyl and SECAD-Net were able to produce valid CAD models.Table A8 : Quantitative comparison with SECAD-Net [30] using Fusion360 [64] and Deep-CAD [65] datasets. MV2Cyl consistently outperforms SECAD-Net [30] Tab. A8 demonstrates that MV2Cyl outperforms SECAD-Net [30] across all metrics. SECAD-Net lacks supervision for segmenting parts in the way CAD designers would, leading to issues such as over-segmentation. This is evident in the images of Tab. A9. For example, in the second row, SECAD-Net segments two vertically attached cylinders into several patches along their surfaces.In comparison, MV2Cyl more accurately represents these components as two distinct cylinders, showing its improved ability to align with CAD design conventions. We compare our MV2Cyl with an additional baseline Curve+Point2Cyl [58] that is a modified version of Point2Cyl that incorporates an additional "curve" feature into the input point cloud. While MV2Cyl benefits from the supervision of curve information from the CAD, we apply equivalent supervision to the baseline to maintain fairness in comparison. The evaluation was conducted using the Fusion360 [64] dataset.To implement the baseline Curve+Point2Cyl (described in the Sec. 4 in the main paper), we modified the network's front-end where it receives inputs. The original Point2Cyl takes an input tensor of size N × 3, representing the target point cloud (see Fig. A10 ). To incorporate information about specific curves within the point cloud, we concatenate the M curve points after the N input points. These curve points are assigned a label of one on an additional channel appended to the original three XYZ channels, while the original non-curve points retain a label of zero on this channel (see Fig. A11 ). The rest of the network architecture remains unchanged. N is set to 8,192 and M is set to 800. Notably, the curve points are obtained from the ground truth CAD models for both training and testing phases. Tab. A10 shows the quantitative comparison between MV2Cyl and Curve+Point2Cyl. MV2Cyl surpasses Curve+Point2Cyl across all the metrics. Despite incorporating ground truth curve features, Curve+Point2Cyl performs similarly to the original Point2Cyl. In practical applications, it might be expected to use predicted curves or edges; however, even with ground truth curves, we observed no significant improvement over the original Point2Cyl, which lacks curve data. This suggests that Point2Cyl's network may prioritize surface points over boundary points, making the additional boundary information less impactful. Input Point Cloud 𝚸 ∈ ℝ !×# PointNet++ backbone 𝓖𝜽 FC Μ & ∈ ℝ !×%& Ν & ∈ ℝ !×# W & ∈ ℝ !×& Β & ∈ 0, 1 !×%Input Point Cloud ! ∈ ℝ !×# PointNet++ backbone $$ FC Μ & ∈ ℝ !×%& Ν & ∈ ℝ !×# W & ∈ ℝ !×& Β & ∈ 0, 1 !×%In this section, we elaborate the implementation of the networks presented in the main paper. Sec. E.1 contains the details of the training 2D segmentation frameworks (Sec. 3.2 in the main paper) and Sec. E.2 includes the details of the 3D field learning from the 2D segments (Sec. 3.3 in the main paper).Each 2D segmentation framework M surface and M curve includes two U-Nets for instance segmentation P type and start-end(-barrel) segmentation Q type where the type is either surface or curve, respectively.Details of 2D Surface Segmentation Framework M surface .The 2D surface segmentation framework, M surface , maps RGB images of dimensions H × W × 3 to a set comprising the extrusion instance segmentation label, P surface ∈ {0, . . . , K} H×W , and the start-end-barrel segmentation label Q surface ∈ {0, 1, 2, 3} H×W , where K indicates the number of instances. We fix K = 8 instances following [58] and train with image resolutions of H = W = 400.For the Fusion360 dataset, 196,050 training images derived from 3,921 shapes are used, while for the DeepCAD [65] dataset, derived from a subset of the training set, 709,000 images from 14,180 shapes are utilized. The datasets are divided into training and validation sets in a 9:1 ratio for the U-Net [50] training. The training times for 3 epochs are 316 minutes for the Fusion360 [64] dataset and 381 minutes for the DeepCAD dataset, using a single NVIDIA RTX A6000 GPU.Details of 2D Curve Segmentation Framework M curve .The 2D curve segmentation framework, M curve , processes RGB images (H × W × 3) to produce two types of labels: the extrusion instance segmentation label, P curve ∈ {0, . . . , K} H×W , and the start-end segmentation label, Q curve ∈ {0, 1, 2} H×W . Similarly following [58] , we also set K = 8 with a training and validation image resolution of 400 × 400. The training objective is defined as:L curve 2D = λ CE L cross-entropy + λ focal L focal + λ dice L dice, where λ CE , λ f ocal , and λ dice are all set to 1.0, balancing the contributions of cross-entropy, focal [33] , and dice [41] losses to the overall training objective.For the Fusion360 dataset, we utilize 196,050 images from 3,921 shapes. For the DeepCAD dataset, we use 1,745,450 images from 34,909 shapes from the training set. These image datasets are split into training and validation sets at a 9:1 ratio. The Fusion360 dataset's U-Net is trained over 3 epochs, taking 307 minutes, while the DeepCAD dataset's U-Net training for a single epoch requires 20.8 hours, all on a single NVIDIA RTX A6000 GPU.The opacity field σ(x) hyper-parameters are set to ζ = 10 and β = 0.8 as in [68] . For the reconstruction loss of the density field F, we set λ sparsity = 0.5, s = 0.5, η batch = 0.1, and η image = 0.05. We use a batch size of 8,192 rays for the training of both the density field F and the semantic field A. We trained all the fields for 1500 iterations on a single NVIDIA RTX 3090 GPU. The whole training time is approximately 5 minutes per shape. Note that the segmentation process and the learning of object density and semantics are typically done as separate steps, rather than in a single, continuous process.We provide additional details on our reverse engineering process, i.e. from our density and semantic fields to the reconstruction of the extrusion cylinders, as discussed in Sec. 3.4 of the main paper. This process begins by sampling points from the density fields, and then from the extracted point clouds deriving the corresponding parameters for each extrusion cylinder. We query the learned density field value across a 400 × 400 × 400 grid. Points on this grid with an opacity σ value greater than a threshold τ ≥ 0.99 are selected and their corresponding instance segmentation and start-end(-barrel) segmentation labels are extracted. The extrusion axis (n), height (h), and center (c) are determined directly from the sampled point clouds as detailed in the main paper.The process to extract the 2D implicit sketch involves additional steps detailed as follows: After predicting the extrusion axis, the 2D sketch boundary points are created by projecting the base curve points along this axis. These 2D point clouds are then used to infer the 2D implicit sketch using IGR [15] , which takes as input the projected 2D sketch point clouds and their normals to generate the corresponding signed distance fields. To extract the 2D point normals for IGR, we modify PGR [32] to estimate 2D normals from our projected 2D sketch points. We adapt the algorithm from PGR and modified 2D basis as follows:EQUATION)where w(x) is the width function defined in PGR. For an in-depth understanding of the formula and its derivations, we direct readers to the PGR paper [32] . Fig. A12 displays the impressive results of our 2D normal estimation implementation, showcasing the accuracy and quality of our approach. These 2D point clouds and the estimated normals are fed into IGR network to output the 2D implicit sketch represented as 2D signed distance function. We detail the generation of multi-view image datasets used in our experiments. Adapted from [68] , the rendering process utilizes BlenderProc [7] and we render 50 views with resolution 400 × 400 for each shape, where the viewpoints are sampled by Fibonacci sampling [17] evenly distributed over a sphere. The surface and curve segmentation labels, we start off by coloring the ground truth mesh black, then marking each segment white. The meshes marked per instance are then rendered in diffuse mode to produce a highlight map for each segment from all viewpoints. Finally, the highlight maps are merged into a single segmentation map, creating multi-view segmentation maps consistently annotated with GT labels across views.We provide additional details on the baseline NeuS2 + Point2Cyl. There are many recent and highperforming studies on converting multi-view images to 3D geometry [40, 61, 44, 13] , and we chose NeuS2 because this work is based on InstantNGP [44] and offers fast convergence in reconstructing shapes. For instance, while NeRF [40] takes about 8 hours to reconstruct a sample from the DTU dataset [20] , NeuS2 can achieve more accurate geometry in just 5 minutes. It has been confirmed that NeuS2 achieves more accurate geometry than InstantNGP, which has converged in the same duration. We selected this study considering the efficiency and feasibility of evaluation. In the test with NeuS2+Point2Cyl, we used a set of images with 50 view points per shape, at a resolution of 400 × 400, and trained for 20,000 iterations. We then ran a marching cube at a resolution of 64 to obtain the mesh, and sampled 8,192 points from that surface to apply as input to Point2Cyl.We introduce a straightforward method for recovering binary operations. A naïve approach involves performing an exhaustive search across all possible combinations of primitives and operations, resulting in 2 K possibilities for a model with K primitives. To identify the optimal combination that aligns closely with the observed input images, we measure the average per-pixel L2 distance between the rendered images of each combination's resulting model and the observed multi-view images, selecting the combination with the lowest distance. We have implemented this approach to recover the binary operations, and examples of the results are illustrated in Fig. A13 . We also provide example showcases of converting an implicit sketch into a parametric representation. After generating the 2D implicit field for each sketch, we sample boundary points using the marching squares algorithm, which serve as knots for cubic Bézier curves. Using these points, we employ an off-the-shelf spline fitting module from the SciPy library [59] to determine the optimal B-spline representation based on the provided control and knot points. This process is illustrated in Fig. A14 .It is important to note that the resulting sketch may not comprise the minimal set of parametric curves; for example, a single line segment may be represented by multiple segments. We compute a consistency metric between the curve and surface predictions. Specifically, we compute the percentage of models where the number of predicted curve instances and surface instances are identical. Across all the test shapes in the Fusion360 dataset, 98% of the models (1,095 shapes over 1,108 shapes) have the same number of curve and surface segments. Moreover, we also measure the consistency between the extracted curve and surface point clouds. Concretely, we measure the average one-way Chamfer distance between the extracted curve point cloud and surface point cloud (0.081 × 10 -3 ). We see that this very small value suggests that the extracted curve point clouds are quite consistent with the surface point clouds.We quantitatively evaluate the 3D reconstruction quality of MV2Cyl by measuring the Chamfer distance against the ground truth CAD model. We provide additional qualitative visualizations of the reconstructed extrusion cylinders using our proposed MV2Cyl. Results are shown in Tab. A13. It showcases MV2Cyl's exceptional ability to process and accurately render shapes of diverse appearances on an instance-wise basis. This demonstration underscores the versatility and precision of our approach in capturing and reconstructing complex geometries from multi-view images. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. Justification: The computer resources required for the experiments are described in the experiment section and the appendices.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: After the rebuttal and extensive discussion, the reviewers' consensus was to accept the paper given its strengths i.e., the novelty of the proposed extrusion-based 3D reconstruction method using primitives and strong results. The AC recommends acceptance. The AC strongly encourages the authors to incorporate material from the rebuttal and discussion to improve the final version of the paper.

Reviews: review: paper_summary: paper_summary: ## Summary of the Paper:

*Problem Statement*:

     Given multi-view images (RGB) of a 3D shape, the paper aims at recovering a *set* of extrusion cylinders to represent the underlying 3D shape.


*Motivation*:
	
     Existing works that address 3D shape reconstruction problem through sketch-extrude take raw 3D geometry as input (ex: Point2Cyl) whereas this work takes multi-view images. Other works that take multi-view images as input perform parametric reconstruction (ex: NEF), which while important, does not reconstruct the 3D surface itself. And so, there exists a gap in the community in terms of either the input representation, or the output result, in the context of 3D reconstruction through sketch-extrude modeling. This forms the basis of this work, termed, MV2Cyl.

*Contributions*:

     Developing a method for sketch-extrude reconstruction of 3D shapes from multi-view images, via labeled surfaces and curves. 
	

*Input*: 

     Multi-view RGB images of a man-made 3D shape

*Output*: 

     3D reconstruction of the object based on sketch-extrude cylinders. A cylinder is 

*Dataset used*: 

     Fusion 360 and DeepCAD

*Underlying Modeling Tool*:

1)	 U-Net for processing images (segmenting curves and surfaces)

2)	 Implicit field-based volumetric rendering for integrating 2D estimated curve and surface information to corresponding 3D space


*Learning Mechanism*:  

     Strongly supervised


*Loss Functions*:

     For 2D surface segmentation, a U-Net with multi-class cross-entropy loss constrained by Hungarian matching algorithm is used. The Hungarian matching constraint is added to account for order invariance of the extrusion segments on the rendered 3D shape, as well as the to classify start-end-barrel segments.

     The 2D curve segmentation module also follows the same architecture and loss functions. In addition, due to the sparsity of labels for curves (which can cause the network to just “attending” to the background pixels), two additional loss terms are used – Dice Loss and Focal Loss

     2D to 3D integration using volumetric rendering: Here, the paper describes two fields. One is the Existence field and the other is the Attribute Field. 

     The training of the Existence field is dictated by the input 2D images, which is accomplished via differentiable rendering. A weighted combination of L2 loss (for surface) and sparsity loss (for curves) is used.

     The training of the Attribute filed is governed by a multi-class cross-entropy loss function, constrained by Hungarian matching.


*Quantitative Metric*:

     Extrusion-axis error (E.A.), extrusion-center error (E.C.), per-extrusion cylinder fitting loss (Fit Cyl.), global fitting loss (Fit Glob.)  


*Baselines*:

     Since existing works do not take multi-view images for sketch-extrude 3D reconstruction tasks, no suitable comparison exists. However, the paper compares against a closely related work (and its variant), Point2Cyl (+NeuS2), which takes a point cloud as input.
strength_weakness: strength_weakness: 1)	 I wanted to understand how essential the 2D curve segmentation network is in making the reverse engineering possible. In other words, how would the cylinder extrusion results in 3D look like if this module were not present? This is a critical piece of experiment, something that should have been shown through ablation experiments. A similar ablation experiment for 2D surface estimation is NOT necessary though, since it is straight-forward to see why this is needed in the first place, given multi-view images as inputs.
2)	 There is a lack of discussion on which kind of shapes does the proposed approach fail on. I would have liked to see a figure (with MV images, 3D GT and sketch-extrude 3D reconstructions) that shows the kind of shapes that MV2Cyl fails at, even if such results are just a tiny sample. This is consequential to the research push in this direction, and also does not do complete justice to the paper.
3)	 In Table 1, the different quantitative metrics lack corresponding units. For ex., as I understand, the extrusion axis error (E.A) is an angle measurement. The reported number must have been in degrees, yes? Likewise, it would be good to see units of measurement, if applicable, for other metrics reported therein. If you want to save space, you could just put it in the table, instead of mentioning in the caption.
4)	 The extrusion instance segmentation is performed by both 2D segmentation modules (Surface and Curve). I sense that there is a strong dependency on this task to help complete the other “auxillary” task in these modules. As this (extrusion instance segmentation) is a common task for the two, were there experiments done to assess if these two modules (Surface and Curve) could be merged? To me, there is something amiss. Would like to see an intuitive explanation justifying this two-step, common-task-occurrence justification.
questions: questions: 1)	 I do not see RGB inputs (as claimed on L 105) in the Teaser as well as Fig 1. I see just a grayscale image. Which one is it?
2)	 How do you differentiate NEF (CVPR 2023) from the 2D curve segmentation framework? The curves that are of interest can be weakly considered as edges. Can you replace 2D curve segmentation network with NEF (part/whole) to detect the extrusion curves and the base curves?
3)	 Dataset question (L259- 261)– how were these datasets enriched with 2D segmentation maps? That is, how were the ground truth 2D curve and extrusion segments obtained? Manually? Provide details.
4)	 I am curious to know why ABC dataset was not considered. It is a highly relevant dataset that would have made the comparisons exhaustive. What was the reasoning here? 
5)	 Can you talk about the extendibility of this approach? I ask this question since I find it non-trivial to train the Existence and Attribute fields, as described in the paper.
limitations: limitations: Yes, the authors have correctly addressed the limitations of the work, including any societal impact that may arise as a result of the publication.
paper_summary: paper_summary: This paper proposes a method to predict extrusion cylinders from images of a CAD part. Specifically, it takes a set of masked multi-view images as input; these are then processed independently by instance segmentors trained to find extrusion curves and surfaces. Neural fields are then fitted, that reconstruct these 2D segmentations in terms of 3D segmentations. Lastly, a set of primitives explaining the 3D segmentations is extracted using a series of heuristics (RANSAC and other hand-crafted robust estimators). Experiments are conducted on two standard benchmark datasets (Fusion 360 and Deep CAD), and show the proposed method out-performing a baseline approach (NeuS reconstruction followed by an existing 3D primitive-prediction pipeline).
strength_weakness: strength_weakness: - The work focuses on clean synthetic (rendered) inputs, with only one experiment (in the appendix) on real data. However, operating on multi-view images offers no benefit over 3D-input methods when one has access to the 3D shapes (e.g. as point-clouds). For a strongly 'applied' paper like this, the lack of focus on the practically-relevant scenario (i.e. actual photos) is problematic.
- Real-world results (though still with in-distribution CAD shapes 3D-printed from the datasets) are not very impressive, despite fine-tuning for this setting. There is also no quantitative evaluation her, which is a pity since this would be straightforward given that the ground-truth CAD models are available (modulo 3D printing imperfections)
- There is no quantitative evaluation of the quality of 3D reconstruction in the sense of distance (e.g. hausdorff / f1) from the reconstructed shape to the original CAD model. It would be valuable to add this (and compare with NeuS2), to give an idea of how well the shapes are being predicted in a more universally-meaningful metric, as opposed to the given metrics that are specific to the shape representation and difficult to interpret in a broader context.
- There are few generalisable insights present in the paper. It presents a pipeline that evidently works well for the specific task, but the technical contributions are very narrow in applicability, reducing potential impact.
- Overall the paper is a poor fit for NeurIPS – lots of engineering (obviously an achievement in itself), but minimal machine learning (just training a segmentation model on synthetic data, then fitting the single-scene neural fields), making the interest to the community relatively low.
questions: questions: How/where do you predict whether each primitive is 'added' or 'subtracted' from the model? The results in Fig. 3 indicate that the model correctly inserts holes, but it is unclear in the method where this determination is made
limitations: limitations: These are discussed briefly but adequately in Sec. 5.
paper_summary: paper_summary: MV2Cyl is a method that proposes to solve the 3D reverse engineering of CAD models. The network takes as input multi-view images and outputs extrusion cylinders. The method extends Point2Cyl [58] that proposed to predict extrusion cylinder from point clouds. It is argued that multi-view images can easily be obtained from 3D scans and that 2D CNNs have superior performance than 3D processing networks.
The network is composed of three main components. Firstly, 2D CNN U-Net-based networks learns to segment and classify the pixel image of the 2D images into surface and curve segments. Then, a NERF based approach is used to leverage the 2D segments into 3D existence and attribute fileds. Finally, 3D surface and curve point clouds are extracted from the 3D fields. The extrusion parameters (extrusion axis, height and centroid) and the sketch (implicit function) are estimated from the extracted point clouds.
The method is evaluated on the DeepCAD[64] and Fusion360[63] datasets, from which multi-view images are extracted. The results show an improvement compared to Point2Cyl[58] and a Neus[51]+Point2Cyl.
strength_weakness: strength_weakness: - While the proposed work offers improvements to Point2Cyl [58], the reviewer feels that it does not address the main limitations of Point2Cyl. Firstly, the binary operations between cylinder (how to assemble the cylinders into the GT CAD model) are not predicted. This limitation is mentioned in the paper but addressing it would have lead to a significant improvement to Point2Cyl. Secondly, the sketches are predicted as implict functions and not parametric curve primitives such as in real CAD models. This produces noisy surfaces (as shown in Figure 3 of the paper), which are quite different from the standard sharp representation of CAD models.

- The evaluation consists of a comparison with Point2Cyl using some of the metrics proposed in Point2Cyl. Nevertheless, further analysis of the results could be completed. For example, it could be meaningful to compute a consistency metric between the surface and curve predictions. One would expect the two network to predict the same number of extrusion cylinders. Also, are the extracted curve point clouds consistent with the surface point clouds? A comparison to the point cloud to CAD sequence experiment presented in DeepCAD[64] could also be included as the extrusion cylinders can easily be extracted from the CAD sequences. It would also be interesting to know how the performance of the network changes with respect to the number of extrusion cylinders (K) present in the ground truth.
questions: questions: - The representation of the output is a little unclear. Figure 3 suggests that the output of MV2Cyl is a CAD model. Can a CAD B-Rep be obtained from the output? If so, how are the sketch primitive and parameters obtained from the predicted implicit functions?
Beyond the points mentioned in the weaknesses, it would be interesting to know how the perfomance of the model changes with respect to the number of input images at test time. 

- Many qualitative results are presented in the Appendix, all of them show very impressive predictions. It would also be interesting for the reader to see examples for which the model fails to recover the correct shape.
limitations: limitations: - The authors have addressed one of the main limitations of the work that is the lack of prediction of the binary operations between the different cylinders. Potential societal impacts have also been included.
paper_summary: paper_summary: This paper proposes a new method to reconstruct extrusion cylinders from multiview images. The key idea is to train a CNN to predict the binary mask for each surface and the sketch of each surface. Then, these predictions are used in learning 3D neural fields for each surface and sketches by the volume rendering technique.
strength_weakness: strength_weakness: 1. How to associate each surface is unclear here. If I understand it correctly, the predicted labels are similar to instance segmentation labels so we may have multiple different labels for the same region. How to associate predicted surfaces with different viewpoints is not clear here.
2. The prediction of feature lines could be sensitive to the line width. If we use a large line width the localization of the feature lines could be difficult because we cannot learn a perfect 3D field for these lines. If we use a small line width, learning a feature field for signals just on very sparse pixels could be problematic and we may extract disconnected feature lines (sketches) here. The setting of the linewidth for training could be tricky here.
3. It is difficult to get some real-world images to train these CNNs, which could harm the generalization ability of the proposed method. If we have painted some textures on these CAD models, then the proposed method could fail because it is not trained on models with textures.
4. The proposed method seems to be too concentrated on a specific problem in computer graphics. Predicting extrusion cylinders is not a common task in daily life and the proposed method does not contain technically new ideas on neural networks. The paper seems to be more suitable for some graphics journals or conferences rather than NeurIPS.
questions: questions: Refer to Weakness.
limitations: limitations: Limitations are discussed in the paper, which is the binary prediction of CNNs. I think the method could be sensitive to hyperparameters, which could be a potential limitation.
paper_summary: paper_summary: The authors introduce a novel method named MV2Cyl, which reconstructs technical 3D  objects withing the sketch-extrude paradigm by utilizing a 2D prior model and a learnable radiance field derived from multi-view images. To accomplish this, the 2D prior model is trained on a labeled dataset to predict semantic labels, $K$ extrusion cylinders, and extrusion curves, which are crucial for further processing of the 3D shape. Once the model is trained, it predicts pseudo-ground-truth for the subsequent stage. In this stage, a radiance field model is trained to replicate the pseudo-ground-truth labels. As the radiance field model converges, it facilitates the extraction of surfaces and curves in 3D space. The authors then detail how to utilize this output to reconstruct the final shape. The experimental results demonstrate that each component significantly contributes to achieving 3D reconstructions that surpass the selected baselines. Given that the radiance field is trained solely from the multi-view images and the prior model, it offers improved real-world applicability compared to previous works.
strength_weakness: strength_weakness: - Point2Cyl, ExtrudeNet, and SECAD-Net encode inputs as latent vectors, which are then decoded into the target representation. An encoder in this context could be any model that processes input in a specific modality. Adapting these methods to handle multi-view images appears feasible and would support the paper's claims. Notably, an encoder understanding 3D geometry based solely on 2D images could be advantageous (for instance, utilizing part of the network from [2] as an encoder).
- While the paper is generally understandable, Section 3.4 is hard to follow. Introducing a simple visualization to illustrate the process would aid comprehension.
- In Section 3.4, points 1 and 2 prompt inquiries about the 3D model employed in this study. The authors refer to TensorRF, but it is unclear whether a standard NeRF or NeuS was used. If NeuS was utilized, the predicted normals could be enough to:
    - Determine a fitting plane by voting for the plane that most likely aligns with the normals.
    - Derive the output shape using a standard algorithm like ball-pivoting or Poisson surface reconstruction, both of which are widely accessible.
- Furthermore, a NeuS backbone would arguably be more appropriate for the application described in the paper, as it more precisely reconstructs surface fields, potentially enhancing the accuracy of the curve field.

[2] Watson, D, Chan, W, Martin-Brualla, R, Ho, J, Tagliasacchi, A, Norouzi, M. "Novel view synthesis with diffusion models". arXiv preprint arXiv:2210.04628 2022.
questions: questions: Questions:
- I will refer to:
> We first extract the surface point cloud and curve point cloud from the corresponding 3D fields by thresholding the corresponding existence fields. Each point in the point cloud would have attributes queried from the corresponding attribute fields.

How is the point cloud sampled specifically? Is it accurate the underlying radiance field uses density instead of SDF field like in NeuS?

Suggestions:
- The explanation in lines 44-47 assumes prior knowledge of the sketch-extrude principle. Without this knowledge, it's challenging to understand the source of the issue. An inset visualization could greatly aid comprehension.
- I propose renaming the "existence field" and "attribute fields". The term "existence field" implies a binary output indicating whether an object exists at a given point. "Density field" would be more appropriate, clarifying how prior values translate to opacity and aligning with the field's intended purpose.
Regarding the "attribute field", the term could be misconstrued as referring to "attributes" such as colors. "Semantic field" would be more accurate.
- It can be deducted from the context that $L_\text{existence}$ applies to both curve and attribute fields. However, stating that clearly in the text would remove potential ambiguities for other readers.
limitations: limitations: The potential negative societal impact is clearly stated and supported by the real-demo presented in the Appendix.

Regarding the limitations, it is unclear what authors mean by:
> our framework is limited in predicting binary operations across primitives. We plan to further explore predicting binary operations using multi-view image inputs.

in lines 328-331. Some examples or an visualization in the Appendix would be appreciated here. It would be worth adding a comment on the shape complexity the model cannot handle.

Comments: title: Answer to author rebuttal
Response to rebuttal
RE: Rebuttal
comment: Dear Reviewer Ajxc,

Thank you for your valuable feedback and for helping to improve our work.

We appreciate your elaboration on "extendibility" in this context. MV2Cyl could indeed be extended to other tasks requiring integrated 2D-to-3D reconstruction, such as B-Rep modeling from multi-view images. Specifically, our 2D segmentation model can decompose curve/surface patches into segments for B-spline fitting, utilizing datasets like ParSeNet (ECCV 2020). The volume rendering module can then reconstruct 3D points, which can be fitted into B-splines using tools like the NURBS-Python library. Thank you for the feedback, and we will further explore this direction.

Regarding the use of separate volume-rendering models, we did not observe any specific issues with handling the two NeRF models or porting between them. If your concern is about the consistency between the two NeRF models, we kindly ask you to refer to our rebuttal to Reviewer mRwh, where we present the consistency metrics between the curve and surface reconstruction models. If you have any other specific concerns about handling and portability, please elaborate, and we will be happy to address them.
Dear Reviewer mRwh,

We are pleased that our rebuttal has clarified the points you raised, and we greatly appreciate the insights you provided.

Regarding the link to the pre-trained models, we found that it does not include the point cloud encoder. We were unable to train the point cloud and multi-view image encoders during the short rebuttal period while allocating our time and computational resources to handle numerous requests for additional experiments from five reviewers. However, we strongly believe that our method, based on surface and curve prediction, will outperform the simple autoencoding architecture. We promise to include this additional experiment in the final version.
Dear Reviewer AtoY,

We're pleased our rebuttal addressed your questions and highlighted the benefits of our method. We greatly appreciate your detailed feedback, which has significantly improved our work.

Regarding real data results, techniques for bridging domain gaps or using more realistic textures and backgrounds in training data could improve the quality. We plan to explore these approaches in future research. Thank you for recognizing the potential and value of our work in proposing a novel method for multi-view 3D CAD reconstruction.
Dear Reviewer dRdb,

We're glad our rebuttal addressed your questions. We appreciate your detailed feedback, which greatly helped improve our work.
Authors,

Appreciate the attempt to answer all of my questions.

Re. Extendability, the term is self explanatory. But for your clarity, I will elaborate -- By extendability, I was referring to how easily this approach can modified/"rewired"? As well, how easily can this be plugged into another design/framework to tackle a similar problem? 

I am aware that the Existence and Attribute fields undergo standard NeRF-model optimization. But do you see any "handling" and "portability" issues as you have two NeRF models being used for the task? All of these questions fall under the hood of "extendability". Let me hear your thoughts.

I would stick to my original rating of BA still.
The reviewer thanks the authors for the different clarifications and additional results. Most of the concerns were properly addressed by the authors. As for the DeepCAD pretrained point-to-CAD model, the authors could have used the pretrained model (https://github.com/ChrisWu1997/DeepCAD?tab=readme-ov-file#pre-trained-models) and trained their own multi-view image encoder to learn the latent space of the pretrained CAD sequence autoencoder. Based on the elements of the rebuttal, the reviewer raises the rating from 4 to 5.
I thank the authors for their detailed rebuttal. The quantitative 3D metrics in particular are valuable and show a benefit to the proposed method. The additional results on real images are also appreciated, though I still find the results disappointing (similar to those already provided in the main paper) -- perhaps inevitable due to the reliance on synthetic data for training. As such I'm not going to increase my rating, but I'm still marginally in favor of acceptance.
I thank the authors for their response and for their effort to provide additional experiments. The attached PDF rebuttal promises a significant changes to the final version and I'll be more than happy to see that paper accepted to the conference.