Learning Manifold Patch-Based Representations of Man-Made Shapes

Choosing the right representation for geometry is crucial for making 3D models compatible with existing applications . Focusing on piecewise-smooth man-made shapes , we propose a new representation that is usable in conventional CAD modeling pipelines and can also be learned by deep neural networks . We demonstrate its benefits by applying it to the task of sketch-based modeling . Given a raster image , our system infers a set of parametric surfaces that realize the input in 3D . To capture piecewise smooth geometry , we learn a special shape representation : a deformable parametric template composed of Coons patches . Naı̈vely training such a system , however , is hampered by non-manifold artifacts in the parametric shapes and by a lack of data . To address this , we introduce loss functions that bias the network to output non-self-intersecting shapes and implement them as part of a fully self-supervised system , automatically generating both shape templates and synthetic training data . We develop a testbed for sketch-based modeling , demonstrate shape interpolation , and provide comparison to related work . 1 INTRODUCTION . While state-of-the art deep learning systems that output 3D geometry as point clouds , triangle meshes , voxel grids , and implicit surfaces yield detailed results , these representations are dense , highdimensional , and incompatible with CAD modeling pipelines . In this work , we develop a 3D representation that is parsimonious , geometrically interpretable , and easily editable with standard tools , while being compatible with deep learning . This enables a shape modeling system leveraging the ability of neural networks to process incomplete , ambiguous input and produces useful , consistent 3D output . Our primary technical contributions involve the development of machinery for learning parametric 3D surfaces in a fashion that is efficiently compatible with modern deep learning pipelines and effective for a challenging 3D modeling task . We automatically infer a template per shape category and incorporate loss functions that operate explicitly on the geometry rather than in the parametric domain or on a sampling of surrounding space . Extending learning methodologies from images and point sets to more exotic modalities like networks of surface patches is a central theme of modern graphics , vision , and learning research , and we anticipate broad application of these developments in CAD workflows . To test our system , we choose sketch-based modeling as a target application . Converting rough , incomplete 2D input into a clean , complete 3D shape is extremely ill-posed , requiring hallucination of missing parts and interpretation of noisy signal . To cope with these ambiguities , existing systems either rely on hand-designed priors , severely limiting applications , or learn the shapes from data , implicitly inferring relevant priors ( Delanoy et al. , 2018 ; Wang et al. , 2018a ; Lun et al. , 2017 ) . However , the output of the latter methods often lacks resolution and sharp features necessary for high-quality 3D modeling . In industrial design , man-made shapes are typically modeled as collections of smooth parametric patches ( e.g. , NURBS surfaces ) whose boundaries form the sharp features . To learn such shapes effectively , we use a deformable parametric template ( Jain et al. , 1998 ) —a manifold surface composed of patches , each parameterized by control points ( Fig . 3a ) . This representation enables the model to control the smoothness of each patch and introduce sharp edges between patches where necessary . Compared to traditional representations , deformable parametric templates have numerous benefits for our task . They are intuitive to edit with conventional software , are resolution-independent , and can be meshed to arbitrary accuracy . Since only boundary control points are needed , our representation has relatively few parameters . Finally , this structure admits closed-form expressions for normals and other geometric features , which can be used for loss functions that improve reconstruction quality ( §3.2 ) . Training a model for such representations faces three major challenges : detection of non-manifold surfaces , structural variation within shape categories , and lack of data . We address them as follows : • We introduce several loss functions that encourage our patch-based output to form a manifold mesh without topological artifacts or self-intersections . • Some categories of man-made shapes exhibit structural variation . To address this , for each category we algorithmically generate a varying deformable template , which allows us to support separate structural variation using a variable number of parts ( §3.1 ) . • Supervised methods mapping from sketches to 3D require a database of sketch-model pairs , and , to-date , there are no such large-scale repositories . We use a synthetic sketch augmentation pipeline inspired by artistic literature to simulate variations observed in natural drawings ( §4.1 ) . Although our model is trained on synthetic sketches , it generalizes to natural sketches . Our method is self-supervised : We predict patch parameters , but our data is not labeled with patches . 2 RELATED WORK . 2.1 DEEP LEARNING FOR SHAPE RECONSTRUCTION . Learning to reconstruct 3D geometry has recently enjoyed significant research interest . Typical forms of input are images ( Gao et al. , 2019 ; Wu et al. , 2017 ; Delanoy et al. , 2018 ; Häne et al. , 2019 ) and point clouds ( Williams et al. , 2019 ; Groueix et al. , 2018 ; Park et al. , 2019 ) . When designing a network for this task , two considerations affect the architecture : the loss function and the geometric representation . Loss Functions . One popular direction employs a differentiable renderer and measures 2D image loss between a rendering of the inferred 3D model and the input image ( Kato et al. , 2018 ; Wu et al. , 2016 ; Yan et al. , 2016 ; Rezende et al. , 2016 ; Wu et al. , 2017 ; Tulsiani et al. , 2017b ; 2018 ) . A notable example is the work by Wu et al . ( 2017 ) , which learns a mapping from a photograph to a normal map , a depth map , a silhouette , and the mapping from these outputs to a voxelization . They use a differentiable renderer and measure inconsistencies in 2D . Hand-drawn sketches , however , can not be interpreted as perfect projections of 3D objects : They are imprecise and often inconsistent ( Bessmeltsev et al. , 2016 ) . Another approach uses 3D loss functions , measuring discrepancies between the predicted and target 3D shapes directly , often via Chamfer or a regularized Wasserstein distance ( Williams et al. , 2019 ; Liu et al. , 2010 ; Mandikal et al. , 2018 ; Groueix et al. , 2018 ; Park et al. , 2019 ; Gao et al. , 2019 ; Häne et al. , 2019 ) . We build on this work , adapting Chamfer distance to patch-based geometric representations and extending the loss function with new regularizers ( §3.2 ) . Shape representation . As noted by Park et al . ( 2019 ) , geometric representations in deep learning broadly can be divided into three classes : voxel- , point- , and mesh-based . Voxel-based methods ( Delanoy et al. , 2018 ; Wu et al. , 2017 ; Zhang et al. , 2018 ; Wu et al. , 2018 ) yield dense reconstruction that are limited in resolution , offer no topological guarantees , and can not represent sharp features . Point-based approaches represent geometry as a point cloud ( Yin et al. , 2018 ; Mandikal et al. , 2018 ; Fan et al. , 2017 ; Lun et al. , 2017 ; Yang et al. , 2018 ) , sidestepping memory issues , but do not capture manifold connectivity . Some recent methods represent shapes using meshes ( Bagautdinov et al. , 2018 ; Baque et al. , 2018 ; Litany et al. , 2018 ; Kanazawa et al. , 2018 ; Wang et al. , 2019 ; Nash et al. , 2020 ) . Our parametric template representation allows us to more easily enforce piecewise smoothness and test for selfintersections ( §3.2 ) . These properties are difficult to measure on meshes in a differentiable manner . Compared to a generic template shape , such as a sphere , our category-specific templates improve reconstruction quality and enable complex reconstruction constraints , e.g. , symmetry . We compare to the deformable mesh representations in §4.5 . Most importantly , our shape representation is native to modern CAD software and can be directly edited in this software , as demonstrated in Figure 1 . The key to this flexibility is the type of the parametric patches we use , which can be trivially converted to a NURBS representation ( Piegl & Tiller , 1996 ) , the standard surface type in CAD . Other common shape representations , such as meshes or point clouds , can not be easily converted into NURBS format : algorithmically fitting NURBS surfaces is nontrivial and is an active area of research ( Yumer & Kara , 2012 ; Krishnamurthy & Levoy , 1996 ) . Finally , some recent works parameterize 3D geometry using learned deep neural networks—e.g. , they learn an implicit representation ( Mescheder et al. , 2019 ; Chen & Zhang , 2019 ; Genova et al. , 2019 ) or a mapping from parameter space to a collection of patches ( Groueix et al. , 2018 ; Deng et al. , 2020 ) . These demonstrate impressive results but are not tuned to CAD applications ; it is unclear how their output can be converted to editable CAD shape representations . 2.2 SKETCH-BASED 3D SHAPE MODELING . 3D reconstruction from sketches has a long history in graphics . A survey is beyond the scope of this paper ; see ( Delanoy et al. , 2018 ) or surveys by Ding & Liu ( 2016 ) and Olsen et al . ( 2009 ) . Unlike incremental sketch-based 3D modeling , where users progressively add new strokes ( Cherlin et al. , 2005 ; Gingold et al. , 2009 ; Chen et al. , 2013 ; Igarashi et al. , 1999 ) , our method interprets complete sketches , eliminating training for artists and enabling 3D reconstruction of legacy sketches . Some systems interpret complete sketches without extra information . This input is extremely ambiguous thanks to occlusions and inaccuracies . Hence , reconstruction algorithms rely on strong 3D shape priors . These priors are typically manually created , e.g. , for humanoids , animals , and natural shapes ( Bessmeltsev et al. , 2015 ; Entem et al. , 2015 ; Igarashi et al. , 1999 ) . Our work focuses on man-made shapes , which have characteristic sharp edges and are only piecewise smooth . Rather than relying on expert-designed priors , we automatically learn category-specific shape priors . A few deep learning approaches address sketch-based modeling . Nishida et al . ( 2016 ) and Huang et al . ( 2017 ) train networks to predict procedural model parameters that yield detailed shapes from a sketch . These methods produce complex high-resolution models but only for shapes that can be procedurally generated . Lun et al . ( 2017 ) use a CNN-based architecture to predict multi-view depth and normal maps , later converted to point clouds ; Li et al . ( 2018 ) improve on their results by first predicting a flow field from an annotated sketch . In contrast , we output a deformable parametric template , which can be converted to a manifold mesh without post-processing . Wang et al . ( 2018a ) learn from unlabeled databases of sketches and 3D models with no correspondence using an adverserial training approach . Another inspiration for our research is the work of Delanoy et al . ( 2018 ) , which reconstructs a 3D object as voxel grids ; we compare to this work in Fig . 7 .

Summary: This paper presents a method that leverages parametric surface patches as the fundamental representation in the task of shape modeling and reconstruction. This method requires a pre-generated template for each shape category. Several losses are specially designed to regularize the generation of the surface patches. Empirical results have demonstrated the performance of the proposed method in sketch-based shape reconstruction and 3D shape interpolation.