Learning Manifold Patch-Based Representations of Man-Made Shapes

Choosing the right representation for geometry is crucial for making 3D models compatible with existing applications . Focusing on piecewise-smooth man-made shapes , we propose a new representation that is usable in conventional CAD modeling pipelines and can also be learned by deep neural networks . We demonstrate its benefits by applying it to the task of sketch-based modeling . Given a raster image , our system infers a set of parametric surfaces that realize the input in 3D . To capture piecewise smooth geometry , we learn a special shape representation : a deformable parametric template composed of Coons patches . Naı̈vely training such a system , however , is hampered by non-manifold artifacts in the parametric shapes and by a lack of data . To address this , we introduce loss functions that bias the network to output non-self-intersecting shapes and implement them as part of a fully self-supervised system , automatically generating both shape templates and synthetic training data . We develop a testbed for sketch-based modeling , demonstrate shape interpolation , and provide comparison to related work . 1 INTRODUCTION . While state-of-the art deep learning systems that output 3D geometry as point clouds , triangle meshes , voxel grids , and implicit surfaces yield detailed results , these representations are dense , highdimensional , and incompatible with CAD modeling pipelines . In this work , we develop a 3D representation that is parsimonious , geometrically interpretable , and easily editable with standard tools , while being compatible with deep learning . This enables a shape modeling system leveraging the ability of neural networks to process incomplete , ambiguous input and produces useful , consistent 3D output . Our primary technical contributions involve the development of machinery for learning parametric 3D surfaces in a fashion that is efficiently compatible with modern deep learning pipelines and effective for a challenging 3D modeling task . We automatically infer a template per shape category and incorporate loss functions that operate explicitly on the geometry rather than in the parametric domain or on a sampling of surrounding space . Extending learning methodologies from images and point sets to more exotic modalities like networks of surface patches is a central theme of modern graphics , vision , and learning research , and we anticipate broad application of these developments in CAD workflows . To test our system , we choose sketch-based modeling as a target application . Converting rough , incomplete 2D input into a clean , complete 3D shape is extremely ill-posed , requiring hallucination of missing parts and interpretation of noisy signal . To cope with these ambiguities , existing systems either rely on hand-designed priors , severely limiting applications , or learn the shapes from data , implicitly inferring relevant priors ( Delanoy et al. , 2018 ; Wang et al. , 2018a ; Lun et al. , 2017 ) . However , the output of the latter methods often lacks resolution and sharp features necessary for high-quality 3D modeling . In industrial design , man-made shapes are typically modeled as collections of smooth parametric patches ( e.g. , NURBS surfaces ) whose boundaries form the sharp features . To learn such shapes effectively , we use a deformable parametric template ( Jain et al. , 1998 ) —a manifold surface composed of patches , each parameterized by control points ( Fig . 3a ) . This representation enables the model to control the smoothness of each patch and introduce sharp edges between patches where necessary . Compared to traditional representations , deformable parametric templates have numerous benefits for our task . They are intuitive to edit with conventional software , are resolution-independent , and can be meshed to arbitrary accuracy . Since only boundary control points are needed , our representation has relatively few parameters . Finally , this structure admits closed-form expressions for normals and other geometric features , which can be used for loss functions that improve reconstruction quality ( §3.2 ) . Training a model for such representations faces three major challenges : detection of non-manifold surfaces , structural variation within shape categories , and lack of data . We address them as follows : • We introduce several loss functions that encourage our patch-based output to form a manifold mesh without topological artifacts or self-intersections . • Some categories of man-made shapes exhibit structural variation . To address this , for each category we algorithmically generate a varying deformable template , which allows us to support separate structural variation using a variable number of parts ( §3.1 ) . • Supervised methods mapping from sketches to 3D require a database of sketch-model pairs , and , to-date , there are no such large-scale repositories . We use a synthetic sketch augmentation pipeline inspired by artistic literature to simulate variations observed in natural drawings ( §4.1 ) . Although our model is trained on synthetic sketches , it generalizes to natural sketches . Our method is self-supervised : We predict patch parameters , but our data is not labeled with patches . 2 RELATED WORK . 2.1 DEEP LEARNING FOR SHAPE RECONSTRUCTION . Learning to reconstruct 3D geometry has recently enjoyed significant research interest . Typical forms of input are images ( Gao et al. , 2019 ; Wu et al. , 2017 ; Delanoy et al. , 2018 ; Häne et al. , 2019 ) and point clouds ( Williams et al. , 2019 ; Groueix et al. , 2018 ; Park et al. , 2019 ) . When designing a network for this task , two considerations affect the architecture : the loss function and the geometric representation . Loss Functions . One popular direction employs a differentiable renderer and measures 2D image loss between a rendering of the inferred 3D model and the input image ( Kato et al. , 2018 ; Wu et al. , 2016 ; Yan et al. , 2016 ; Rezende et al. , 2016 ; Wu et al. , 2017 ; Tulsiani et al. , 2017b ; 2018 ) . A notable example is the work by Wu et al . ( 2017 ) , which learns a mapping from a photograph to a normal map , a depth map , a silhouette , and the mapping from these outputs to a voxelization . They use a differentiable renderer and measure inconsistencies in 2D . Hand-drawn sketches , however , can not be interpreted as perfect projections of 3D objects : They are imprecise and often inconsistent ( Bessmeltsev et al. , 2016 ) . Another approach uses 3D loss functions , measuring discrepancies between the predicted and target 3D shapes directly , often via Chamfer or a regularized Wasserstein distance ( Williams et al. , 2019 ; Liu et al. , 2010 ; Mandikal et al. , 2018 ; Groueix et al. , 2018 ; Park et al. , 2019 ; Gao et al. , 2019 ; Häne et al. , 2019 ) . We build on this work , adapting Chamfer distance to patch-based geometric representations and extending the loss function with new regularizers ( §3.2 ) . Shape representation . As noted by Park et al . ( 2019 ) , geometric representations in deep learning broadly can be divided into three classes : voxel- , point- , and mesh-based . Voxel-based methods ( Delanoy et al. , 2018 ; Wu et al. , 2017 ; Zhang et al. , 2018 ; Wu et al. , 2018 ) yield dense reconstruction that are limited in resolution , offer no topological guarantees , and can not represent sharp features . Point-based approaches represent geometry as a point cloud ( Yin et al. , 2018 ; Mandikal et al. , 2018 ; Fan et al. , 2017 ; Lun et al. , 2017 ; Yang et al. , 2018 ) , sidestepping memory issues , but do not capture manifold connectivity . Some recent methods represent shapes using meshes ( Bagautdinov et al. , 2018 ; Baque et al. , 2018 ; Litany et al. , 2018 ; Kanazawa et al. , 2018 ; Wang et al. , 2019 ; Nash et al. , 2020 ) . Our parametric template representation allows us to more easily enforce piecewise smoothness and test for selfintersections ( §3.2 ) . These properties are difficult to measure on meshes in a differentiable manner . Compared to a generic template shape , such as a sphere , our category-specific templates improve reconstruction quality and enable complex reconstruction constraints , e.g. , symmetry . We compare to the deformable mesh representations in §4.5 . Most importantly , our shape representation is native to modern CAD software and can be directly edited in this software , as demonstrated in Figure 1 . The key to this flexibility is the type of the parametric patches we use , which can be trivially converted to a NURBS representation ( Piegl & Tiller , 1996 ) , the standard surface type in CAD . Other common shape representations , such as meshes or point clouds , can not be easily converted into NURBS format : algorithmically fitting NURBS surfaces is nontrivial and is an active area of research ( Yumer & Kara , 2012 ; Krishnamurthy & Levoy , 1996 ) . Finally , some recent works parameterize 3D geometry using learned deep neural networks—e.g. , they learn an implicit representation ( Mescheder et al. , 2019 ; Chen & Zhang , 2019 ; Genova et al. , 2019 ) or a mapping from parameter space to a collection of patches ( Groueix et al. , 2018 ; Deng et al. , 2020 ) . These demonstrate impressive results but are not tuned to CAD applications ; it is unclear how their output can be converted to editable CAD shape representations . 2.2 SKETCH-BASED 3D SHAPE MODELING . 3D reconstruction from sketches has a long history in graphics . A survey is beyond the scope of this paper ; see ( Delanoy et al. , 2018 ) or surveys by Ding & Liu ( 2016 ) and Olsen et al . ( 2009 ) . Unlike incremental sketch-based 3D modeling , where users progressively add new strokes ( Cherlin et al. , 2005 ; Gingold et al. , 2009 ; Chen et al. , 2013 ; Igarashi et al. , 1999 ) , our method interprets complete sketches , eliminating training for artists and enabling 3D reconstruction of legacy sketches . Some systems interpret complete sketches without extra information . This input is extremely ambiguous thanks to occlusions and inaccuracies . Hence , reconstruction algorithms rely on strong 3D shape priors . These priors are typically manually created , e.g. , for humanoids , animals , and natural shapes ( Bessmeltsev et al. , 2015 ; Entem et al. , 2015 ; Igarashi et al. , 1999 ) . Our work focuses on man-made shapes , which have characteristic sharp edges and are only piecewise smooth . Rather than relying on expert-designed priors , we automatically learn category-specific shape priors . A few deep learning approaches address sketch-based modeling . Nishida et al . ( 2016 ) and Huang et al . ( 2017 ) train networks to predict procedural model parameters that yield detailed shapes from a sketch . These methods produce complex high-resolution models but only for shapes that can be procedurally generated . Lun et al . ( 2017 ) use a CNN-based architecture to predict multi-view depth and normal maps , later converted to point clouds ; Li et al . ( 2018 ) improve on their results by first predicting a flow field from an annotated sketch . In contrast , we output a deformable parametric template , which can be converted to a manifold mesh without post-processing . Wang et al . ( 2018a ) learn from unlabeled databases of sketches and 3D models with no correspondence using an adverserial training approach . Another inspiration for our research is the work of Delanoy et al . ( 2018 ) , which reconstructs a 3D object as voxel grids ; we compare to this work in Fig . 7 .

Summary: The paper proposes a self-supervised method to fit a template (represented as a union of Coons patches) to a certain 2D sketch. It derives a way to build a proper template, uses a network to predict the patches' parameters, and proposes a union of different losses. The qualitative results of the method are shown in several different objects.