Learning Manifold Patch-Based Representations of Man-Made Shapes
Authors: Dmitriy Smirnov
Mikhail Bessmeltsev
Justin Solomon
Venue: ICLR 2021 Poster (2021)
Keywords: 3D shape representations
CAD modeling
sketch-based modeling
computer graphics
computer vision
deep learning

Abstract: Choosing the right representation for geometry is crucial for making 3D models compatible with existing applications. Focusing on piecewise-smooth man-made shapes, we propose a new representation that is usable in conventional CAD modeling pipelines and can also be learned by deep neural networks. We demonstrate its benefits by applying it to the task of sketch-based modeling. Given a raster image, our system infers a set of parametric surfaces that realize the input in 3D. To capture piecewise smooth geometry, we learn a special shape representation: a deformable parametric template composed of Coons patches. Naively training such a system, however, is hampered by non-manifold artifacts in the parametric shapes and by a lack of data. To address this, we introduce loss functions that bias the network to output non-self-intersecting shapes and implement them as part of a fully self-supervised system, automatically generating both shape templates and synthetic training data. We develop a testbed for sketch-based modeling, demonstrate shape interpolation, and provide comparison to related work.

Published as a conference paper at ICLR 2021
LEARNING MANIFOLD PATCH-BASED REPRESENTATIONS OF MAN-MADE SHAPES
Dmitriy Smirnov
Mikhail Bessmeltsev
Justin Solomon MIT
MIT
Université de Montréal
smi miirnov@mit.edi
bmpix0irg umontreal...
iso wwwooooittcooomooomom edu
ABSTRACT
Choosing the right representation for geometry crucial for making 3D models compatible with existing applications. Focusing piecewise-smooth man-made shapes, we propose a new representation that is usable in conventional CAD mod- eling pipelines and can also be learned by deep neural networks. We demonstrate its benefits by applying it to the task of sketch-based modeling. Given a raster image, our system infers set of parametric surfaces that realize the input in 3D. To capture piecewise smooth geometry, we learn a special shape representation: deformable parametric template composed of Coons patches. Naively training such system, however, is hampered by non-manifold artifacts in the parametric shapes and by lack of data. To address this, we introduce loss functions that bias the network to output non-self-intee self-interssctingg shapes and implement them as part of fully self-supervised system, automatically generating both shape templates and synthetic training data. We develop testbed for sketch-based modeling, demonstrate shape interpolation, and provide comparison to related work.
INTRODUCTION
While state-of-the art deep learning systems that output 3D geometry as point clouds, triangle meshes, voxel grids, and implicit surfaces yield detailed results, these representations are dense, high- dimensional, and incompatible with CAD modeling pipelines. this work, we develop a 3D represen- tation that is parsimonious, geometrically interpretable, and casily editable with standard tools, while being compatible with deep learning. This enables a shape modeling system leveraging the ability of neural networks to process incomplete, ambiguous input and produces useful, consistent 3D output.
Our primary technical contributions involve the development of machinery for learning parametric permnerter 3D surfaces in fashion that is efficiently compatible with modern deep learning pipelines and effective for challenging 3D modeling task. We automatically infer a template per shape category and incor- porate loss functions that operate explicitly on the geometry rather than in the parametric domain or on a sampling of surrounding space. Extending learning methodologies from images and point sets to more exotic modalities like networks of surface patches is central theme of modern graphics, vision, and learning research, and we anticipate broad application of these developments in CAD workflows.
To test our system, we choose sketch-based modeling as target application. Converting rough, incomplete 2D input into a clean, complete 3D shape is extremely ill-posed, requiring hallucination of missing parts and interpretation of noisy signal. To cope with these ambiguities, existing systems either rely on hand-designed priors, severely limiting applications, or learn the shapes from data, implicitly inferring relevant priors (Delanoy et al.) 2018 Wang et al.) [018a] Lun al., 2017). However, the output of the latter methods often lacks resolution and sharp features necessary for high-quality 3D modeling.
In industrial design, man-made shapes are typically modeled as collections of smooth parametric patches (e.g., NURBS surfaces) whose boundaries form the sharp features. To learn such shapes ef- fectively, a deformable parametric template (Jain et al. 1998) manifold surface composed of patches, each parameterized by control points (Fig. 3a). This representation enables the model to control the smoothness of each patch and introduce sharp edges between patches where necessary.

---

Published as a conference paper at ICLR 2021
Input
Astomurically Compused Model
Filted
Figure 1: Editing 3I model produced by method. Because output.3 3D geometry as collection of consistent, well-placed NURBS patches, user edits can be made in conventional CAD software by simply moving control points. Here, we are able refine the trunk of car model with just few clicks.
Compared to traditional representations, deformable parametric templates have numerous benefits for our task. They are intuitive edit with conventional software, are poolutionnindeendeeene and can be meshed to arbitrary accuracy. Since only boundary control points are needed, our representation has relatively few parameters. Finally, this structure admits closed-fore form expressions for normals and other geometric features, which can be used for loss functions that improve reconstruction quality (33.2).
Training a model for such representations faces three major challenges: detection of non-manifold surfaces, structural variation within shape categories, and lack of data. We address them as follows:
We introduce several loss functions that encourage our patch- output to form a manifold mesh without topological artifacts or self-in ceff-intersections.
Some categories of man-made shapes exhibit structural variation. To address this, for each category algorithmically generate varying deformable template, which allows to support separate structural variation using a variable number of parts (33.I).
Supervised methods mapping from sketches 3D require a database of sketch-model pairs, and, to-date, there are no such large-scale repositories. We use synthetic sketch augmentation pipeline inspired by artistic literature to simulate variations observed in natural drawings ($4.1). Although our model is trained on synthetic sketches, it generalizes to natural sketches.
Our method self-supervised: predict patch parameters, but our data not labeled with patches.
RELATED WORK
2.1 DEEP LEARNING FOR SHAPE RECONSTRUCTION
Learning to reconstruct 3D geometry has recently enjoyed significant research interest. Typical forms of input are images (Gao et al. 2019 Wu et al. 2017 Delanoy et al. ( Häne
Häne al. 2019) and point
clouds (Williams et al. 2019 Groueix et al. 2018 Park et al. 2019). When designing a network for
this task, two considerationss affect the architecture: the loss function and the geometric representation.
Loss Functions. One popular direction employs a differentiable renderer and measures 2D image
loss between rendering of the inferred 3D model and the input image (Kato et al. 2018 Wu et al.
2016 Yan et al., 2016; Rezende et al.] [2016] Ww et al.) 2017;; Tulsiani et al.) 2017b 2018). A notable example is the work by Wu al. (2017). which learns mapping from photograph normal map, a depth map, a silhouette, and the mapping from these outputs to a voxelization. They use differentiable renderer and measure inconsistencies in 2D. Hand-drawn sketches, however, cannot be interpreted as perfect projections of 3D objects: They are imprecise and often inconsistent (Bessmeltsev et al., (2166). Another approach uses 3D loss functions, measuring discrepancies between the predicted and target 3D shapes directly, often via Chamfer or regularized Wasserstein distance (Williams et al. 2019 Liu et al. 2010, Mandikal et al. 2018 Groueix et al. 2018 Park
et al. 2019 Gao et al. 2019 Häne et al. 2019). We build on this work, adapting Chamfer distance to patch- based geometric representations and extending the loss function with new regularizers (53.2).
Shape representation. As noted by Park et al. (001)) geometric representations in deep learning broadly can divided into three classes: voxel- point- and mesh-based. Voxel-based methods Delanoy et al. 22018: Wu et al. 2017 Zhang al. 2018 Wu et al. (018)) yield dense reconstruction that are limited in resolution, offer no topological guarantees, and cannot represent sharp features. Point-based approaches represent geometry point cloud (Yin al. 2018 Mandikal al.) 2018
Fan et al.] 2017 Lun et al. 2017 Yang et al. 2018). sidestepping memory issues, but do not capture manifold connectivity.
2

---

Published as a conference paper at ICLR 2021
Some recent methods represent shapes using meshes (Bagautdinov et al., 2018 Baque et al. 2018 Litany et al. 2018 Kanazawa et al. 2018 Wang et al., 2019; Nash et al. 2020). Our parametric template representation allows us to intersections ($3.2). These properties are difficult to measure on meshes differentiable manner. Compared to generic template shape, such as sphere, category-specific templates improve reconstruction quality and enable complex reconstruction constraints, e.g., symmetry. We compare the deformable mesh representations in $4.5.
easily enforce piecewise smoothness and test for self-
Most importantly, our shape representation is native to modern CAD software and can be directly edited in this software, demonstrated in Figure 1. The key this flexibility is the type of the para- metric patches we use, which can be trivially converted to NURBS representation (Piegl & Tiller 1996), the standard surface type in CAD. Other common shape representations, such as meshes or clouds, cannot be easily converted into NURBS format: algorithmically fitting NURBS surfaces nontrivial and an active area of research Yumer Kara 2012 Krishnamurthy Levoy 1996).
Finally, some recent works parameterize 3D geometry using learned deep neural networks e.g..
they learn an implicit representation (Mescheder et al. 2019; Chen & Zhang. 2019; Genova et al.) 2019) or mapping from parameter space to a collection of patches (Groueix et al. [00] 8 Deng et al.,
2020 These demonstrate impressive results but are not tuned to CAD applications; is unclear how their output can be converted to editable CAD shape representations.
2.2 SKETCH-BASED 3D SHAPE MODELING
3D reconstruction from sketches has long history in graphics. survey beyond the scope of this paper; see (Delanoy et al. 2018 surveys by Ding & Liu (2016) alsen et al. (2009).
Unlike incremental sketch-based 3D modeling, where users progressively add new strokes (Cherlin al. 2005; Gingold et al.) 2009 Chen et al., 2013 Igarashi et al. 1199),, our method interprets complete sketches, eliminating traming for artists and enabling 3D reconstruction of legacy sketches.
Some systems interpret complete sketches without extra information. This input is extremely ambiguous thanks to occlusions and inaccuracies. Hence, reconstruction algorithms rely on strong 3D shape priors These priors are typically manually created, e.g., for humanoids, animals, and natural shapes (Bessmeltsey et (015) Entem et al., Igarashi et al. [199).. Our work focuses on man-made shapes, which have characteristic sharp edges and are only piecewise smooth. Rather than relying on -esernede priors, we automatically learn -specory-specific shape priors.
A few deep learning approaches address sketch-based modeling. Nishida al. (2016) and Huang iet al. (2017) train networks to predict procedural model parameters that yield detailed shapes from sketch. These methods produce complex high-resolution models but only for shapes that can be procedurally generated. Lun et al. (2017) use CNN-based architecture to predict multi-view depth and normal maps, later converted to point clouds; Li et al. (2018) improve on their results by first predicting flow field from annotated sketch. In contrast, we output a deformable parametric template, which can be converted to a manifold mesh without post-processing. Wang et al. (2018a) learn from unlabeled databases of sketches and 3D models with correspondence using an adverserial training approach. Another inspiration for our research is the work of Delanoy et al. 2018), which reconstruets 3D object voxel grids; we compare to this work in Fig. 7.
3 ALGORITHM
Our learning pipeline outputs a parametrically-defnne 3D surface. We describe geometric rep- resentation ($3.1) define our loss (53.2), and specify our architecture and training procedure ($3.3).
3.1
REPRESENTATION
Patches. Our surfaces collections of Coons patches (Coons) 1967), ) commonly used and rich
subset of NURBS surfaces. A patch is specified by four boundary cubic Bézier curves sharing endpoints (Fig. 3a). Each curve has control points P1 E R3, A Bézier curve e: [0,1] - R is
3

---

Published as a conference paper at ICLR 2021
60
PGS
69
P10.0
mt.a
Figure 3: Our representation is composed of Coons patches (a) organized into deformable template (b). We use the following templates (c. top to bottom, left to right): bottle, knife, guitar, car, airplane, coffee mug, gun, bathtub, 24- 24-patch sphere, 54-patch sphere.
and Coons patch 1-[0,11-++**
(1)
Templates. Templates specify patch connectivity. A template consists of the minimal number of control points necessary define the patches; shared control points are reused (Fig. 3b, c). We allow the edge of one patch to be contained within the edge of another using junction curves. A junction curve ca is constrained a lie along a parent curve and is thus parameterized by s,t E [0,1], such that c(0)= = d(s) and c(1)= =d(t).
d
cd
template provides hard topological constraints for our surfaces, an initialization of their geometry, and, optionally, a means for regularization. Templates are crucial in ensuring that the patches have consistent topology- an approach without templates would result in unstructured, non-manifold patch collections. While method works using generic sphere template, per shape category to incorporate category-specific priors. These templates capture only coarse geometric features and approximate scale. We outline an algorithm for obtaining templates below.
define templates
Algorithmic construction of templates. We design a simple system to construct a template auto- matically given input any collection of cuboids. Such collection can be computed automatically for a shape category, e.g., given a segmentation or using self-supervised methods such as (Smirnov) al. 2020 Tulsiani et al. 2017a Sun et 2019), easily produced using standard CAD software. We show templates algorithmically computed from pre-segmented shapes- we obtain a collection of cuboids by taking the bounding box around each connected component of each segmentation class.
A generic cuboid decomposition cannot be used as template, since individual cuboids may overlap. We snap cuboids to an integer lattice, split each face at grid coordi- nates, and remove overlapping and interior faces to obtain manifold quad mesh. This mesh typically consists of many faces, and so, we simplify it. We merge adjacent quads with greedy agglomerative algorithm, iterating over each quad in order of descending area and merging with an adjacent quad as long as the merge does not result in ill-defined junction curves. We show an example of this process in Fig. 2. Given decompositions of multiple shapes in category,
(b)
(
Figure 2: Summary of our template
algorithm. Given a collection of cuboids (a), form quad mesh (b), and merge faces to get template (c).
the median model w.r.t. Chamfer distance. Since models within category aligned, the median provides rough approximation of the typical geometry.
Structural variation using templates. For category-specific tem- plates, we use the fact that template patches are consistently placed on semantically meaningful parts account for structural variation. For instance, certain airplane models contain turbines, while others do not. We note which template patches come from cuboids corresponding turbines. and, during training, only use turbine patches for models that contain turbines. This allows training on the entire airplane shape category, effectively using two distinct templates. At test time, the user can toggle turbines on or off for any output model.
3.2 LOSS
We fit a collection of patches (P.) to target mesh M by optimizing a differentiable loss function. Be- low, we describe each term- reconstruction loss generalizing Chamfer distance, a normal alignment loss, self-innerreee interseccionn regularizer, patch flatness regularizer, and two template-based priors.
4

---

Published as a conference paper at ICLR 2021
Area-weighted Chamfer distance. Given two measurable shapes A.B C R and point sets X and sampled from A and B, respectively, the directed Chamfer distance between X and
Chair (X,Y)=
Σ
min d(x.w), yEY
|X
wh.re - 1|x 21122 The symmetric Chamfer distance Ch(XXY) Chair (X,Y) Chair (Y,X)
Chamfer distance is differentiable and thus a popular loss function in deep learning. However, the distribution under which X and Y are sampled has significant impact. Sampling uniformly in patch parameter space (a unit square) does not capture the surface area measure and, consequently, does yield uniform samples on the curface-regione with higher curvature end up oversampled. A closed form arc-length parameterization does not even exist for Bézier curve, and so uniform parameterization for even one-dimensional patch difficult to compute. Many works try to address this issue by designing custom sampling procedures (Wang et al., 2012 Hernández-Mederos & Estrada-Sarlabour 2003 Elkott et al. 2002). These schemes are typically computationally expensive and/or incompatible with a differentiable learning pipeline. To address this issue, following Smirnov ct al. 2020). we first define variational directed Chamfer distance, starting from equation
Chai (X,Y)
1
Σ
min (x.y)) EnoalA KEY
inf dn..)))))) DEY
1
inf d(a.y)dr del Ch (A,B),
= Arca(X) X
X
where UA is the uniform distribution on A. Ch CA,)(A is defined analogously.
While it is difficult to sample uniformly from patches, we can do so easily from their parametric domain (the unit square). We perform change of variables to get the following (derivation in $A.1):
Ch (P,M)= (ass (i
(o. [I (s.t)]
where [s,() is the Jacobian of P(s.t). We approximate this expression via Monte Carlo integration.
Since we can precompute uniformly sampled random points from the target mesh, we do not need to use area weights for Ch (M, Thus, our area-weighted Chamfer distance is
(cH(UP,, = EnEnA minge
+ min
(5)
( [((stt
where Un Un~ 1j. We compute (u.v) for patch given its control points in closed-form.
Normal alignment. The normal alignment term is computed analogously to Chair. except that instead of Euclidean distance, we use dy (2. ||n,- - -n.-nnlll 12. where n2 is the unit normal at æ. For each point y sampled from our predicted surface, we compare ny nzz where x E M is closest to y, and, symmetrically. for each x' M, we compare nz to to ny. where y' EUP; is closest to x':
Cnormal (UP,,M) E.E(m.) (NNNNN
(+))
(uD [((u,)))
[M|
where NNy (z) is the nearest neighbor to x in Y under Euclidean distance.
Intersection regularization. We introduce a collision detection loss detect patch intersections:
Leou ((on((PP))) -P.))=-csn >>xp (
$
where 7' is triangulation of patch P., and pi is set of points sampled from patch P.. To triangulate a patch, we take a fixed triangulation of the parameter space (a unit square) and compute the image of each vertex under the Coons patch map, keeping the original connectivity. With a small = 10 this expression is smooth indicator for when two patches are intersecting. For a pair of adjacent patches or those that share a junction, we truncate one patch at the adjacency before evaluating the loss.
Patch flatness regularization. To help patches to align to smooth regions and sharp creases to fall on patch boundaries, we define patch flatness regularizer, which discourages excessive curvature by regularizing each Coons patch map P: [0,1] x [0.1] + R to be close to a linear map. For each patch, sample points Up in parameter space, compute their image P(Up), and fit linear function using
5

---

Published as a conference paper at ICLR 2021
aliganent
weightdd
sround wuth
feniplaee loss
128012
ReaNes-88
filly
symmury loe)
image
conmantal
patiites
collisiou Joss
Icrepatc
Figure 4: An overview of our data generation and augmentation (a) and learning (b) pipelines.
least-squares. Thus, P(Un) = bbn for some A.b. We define patch flatness loss as EiE (u,))EUn [PP( ( ((
Laa G ((P:))
(8)
(uviee (u,,))
Template normals regularization. For categories where template is available, we not only ini- tialize the network with the template geometry but also regularize the output using template normals. This favorably positions patch seams and prevents patches from sliding over high-curvature regions: nr, (1,,))
Lremplate (eemplate((P.).(TT]))) (P.). E, (u.jje
(uEje,)) [u,,)))
where v7, is the normal vector of the ith template patch.
Global symmetry. Man-made shapes often exhibit global bilateral symmetries. Enforcing symmetry difficult in, e.g., meshes or implicit surfaces. In contrast, after computing a template's symmetry
planes, we enforce symmetric positions of the corresponding control points as additional loss term:
Laym (UP:)
Σ I(Pi-a,Pi
Pa-Pa P))
(10)
(i,))S
where S contains index pairs of symmetric control points (Pi, P)). the formula, we assume w.l.o.g. symmetry plane We use this to enforce symmetrical reconstruction of airplanes and cars.
3.3 DEEP LEARNING PIPELINE
The final loss that we optimize
Coormal (UP.M)
(11)
aana Cna ((P.))
+Cempla) Cremppatte-emplaar (1 (PP).((ii) + Caym Camm(UP)))))) (UP.).
For models scaled to fit a unit sphere, we use O'pormal = 0.008, O'aa = 2, and acall 0.00001 for all experiments, and Otemmpaate 0.0001 and Ossy = 1 for experiments that use those regularizers.
Our network inputs one or more 128 128 images and outputs patch parameters. The architecture is ResNet- 18 (He et al. 2016) followed by hidden layers with 1024, 512, and 256 units, and an output layer with size equal to the output dimension. Final layer weights are initialized to zero with bias equal to the template parameters. For multi-view input, we encode each image and do max pooling over the latent codes. We use ReLU and batch normalization after each layer except the last. We train each network for 24 hours Tesla V100 GPU, using Adam (Kingma & Bal 2014) and batch size 8 with learning rate 0.0001. At each iteration, we sample 7,000 points from the predicted and target shapes. Our pipeline illustrated in Fig. 4b.
EXPERIMENTAL RESULTS
We introduce pipeline for generating realistic sketch data and train network that coverts sketch image to a patch-based 3D representation. We show results on synthetic and human-drawn sketches, demonstrate interpolation and quad meshing, compare prior work, and do ablation study ($A.2).
4.1
DATA PREPARATION
While there exist annotated datasets of 3D models and corresponding hand-drawn sketches (Gryadit- skaya et al. 2019), ) such data are unavailable at the deep learning scale. Instead, we generate synthetic first render occluding contours and sharp edges using the
data. Guided by Cole et al. (2012), Arnold Toon Shader in Autodesk Maya for each model from representative camera views. Although the contour images capture the main features of the 3D model, they lack some of the ambiguities of

---

Published as a conference paper at ICLR 2021
Figure 5: Results on synthetic sketches. For each category, from top to bottom: input sketch, output 3D model with sphere template (54 patches), output 3D model with category-speciffe template.
rough hand-drawn sketches (Liu 2018). To this end, vectorize the images using Btss & Solomon (2019) and augment the set of contours by stochastically splitting or truncating curves. Finally, we rasterize each contour image using different stroke widths and pass through the pencil drawing generation model Simo-Serra et al. (2018) S). We illustrate this pipeline in Fig. 4a.
Thus, we obtain realistic sketch images paired with 3D models. We train on the airplane, bathtub, gui- tar, bottle, car, mug, gun, and knife categories of ShapeNet Core (v2) (Chang et et 20155. We make the meshes watertight using Huang et al. (2018) and normalize them to an origin-centered unit sphere.
4.2 REAL AND SYNTHETIC SKETCH RECONSTRUCTION
We pick random 10%-90% test-train split for each category and evaluate in Fig. 5 as well as AA.5
The templates for airplanes, guitars, guns, knives, and cars are generated automatically using segmentations of Yi et al. (2016). For mugs, start with automatic template and add a hole in the handle and a void in the interior. To demonstrate a template with distinct parts, for cars, we use the segmentation during training, computing Chamfer and normal losses for wheels and body separately. For bottle and bathtub templates, we simply place two and and five cuboids, respectively.
With generic sphere template, produce compact srrcewise-smoolte surface of comparable qual- ity to the more conventional deformable meshes. Our algorithmic construction of aategory-specific templates, however, enables higher-quality reconstruction of sharp features and details.

---

Published as a conference paper at ICLR 2021
NURBS Patches (Rhino)
NURBS Patches (ouns)
Quad Mesh (Instant Mosbes)
Quad Mesh (ours)
10000 pitches
S8 pitches
400 singularises
de singobritiss
singolariies (bady)
10000
72 pitches
singulartics
singulariiss (lurhins)
10000 putches
22 ptiches
524 singulariiis
24 singobailics
10000 patches
22 potches
218 singularitiss
24 singularties
Figure 6: We convert two airplane and two guitar models NURBS patches using Rhino 3D and compare to our Coons patches. Corresponding Coons patches across models of the same category are the same color. We also show quad meshes generated using Instant Meshes (Jakob et al.] 2015) and those from our patch decompositions. Singular points are in pink. Our representation is much more compact and hence easily editable. We produce fewer singularities and only in known places. Furthermore, unlike with other methods, our patches and quad meshes are consistent across models.
We demonstrate our method's ability to incorporate details from different views. We show output when given single view of an airplane as well as when given an additional view. The combined model incorporates elements not visible in the original view.
We also test on real sketches. Each artist was asked to sketch airplane without having seen any sketches from our dataset. The results are similar to those on synthetic sketches, demon- strating that our dataset is reflective of choices that humans make when sketching 3D objects.
4.3
3D MODEL INTERPOLATION
Our representation is naturally well-suited for interpolating between 3D models. As each model is composed of small number of consistently placed patches, we linearly interpolate the patch parameters (e.g., vertex positions) to generate models "between" our network outputs. We also perform interpolation in the latent space learned by our model (the output of the first 1024-dimensional hidden layer). While latent-space interpolation is similar to patch-space, each interpolant better resembles realistic model due the learned priors. We demonstrate putch-space (right) and latent-space (left) interpolation between two car models.
4.4 QUAD MESHING AND NURBS DECOMPOSITION
Many algorithms have been designed for converting triangle meshes into NURBS patches (Krishna- murthy & Levoy] 1996, Eck & Hoppe 1996 Bernardini et al.). By fitting a template to 3D model, our method automatically converts the model
of Coons patches. Moreover, our Coons patches are placed consistently. thus establishing correspondences between models. In Fig. 6, we compare our Coons patches NURBS patches automatically obtained using Rhino 3D, a commercial CAD program. Our decomposition is significantly more sparse and usable for further editing.
common task in computer graphies converting surfaces into quad meshes. We can easily obtain quad mesh from our patch decompasition by simply subdividing (see fAr details). We compare our quad meshes to those produced by Instant Meshes (Jakob et al., 2015). a recent quad meshing algorithm, in Fig. 4.4. It is commonly desired to minimize the number of singular points (vertices with valence not equal to four) and to consistently position them in a quad mesh. Our algorithm achieves both of these goals- thels-the number and location of singularities is determined by our template.
While the Instant Meshes results contain between 215 and 524 singular points, our quad meshes contain only 24 (guitars), 44 (airplane without turbines), 68 (airplane with turbines).
8

---

Published as a conference paper at ICLR 2021
(c)
Figure 7: Comparisons (Delanoy al., 2018 using their data (a) and our data (b), to (Lun al.
2017 (c), to AtlasNet (d), and to Pixel2Mesh (e). Our results are in blue for each comparison.
4.5 COMPARISONS
In Fig. 7. we compare to the sketch-based reconstruction methods of Delanoy et al. (2018) (a) and ct al. (2017) (c). Although we use the same species of input used to train these methods rather than attempting to re-train their models our data, the visual quality of our predictions is comparable to theirs. Moreover, our output representation sparsely captures smooth and sharp features, independent of resolution. In contrast, Delanoy et al. (2018) produce 64 voxel grid dense, fixed-resolution representation, which cannot be edited directly and offers no topological guarantees. In Fig. 7b, we evaluate their system on contours from our dataset, demonstrating that our task of reconstruction with a prior on class (airplane) rather than structure (cylinders and cuboids) is misaligned with theirs: Since our data not well-approximated by CSG models, their method cannot extract meaningful output.
Although Lun et al. (2017) ultimately produce a mesh, they perform a computationally expensive poot-psross procedure, since their forward pass returns labeled point cloud. Our method directly outputs the parameters for surface patches with no further optimization. Additionally, their final mesh contains more components (triangles) than our output (patches), making less useful for editing.
In Fig. 7d, we compare to AtlasNet (Groucix et al. 2018). Werrraiin our model with their renderings, using the 54-face sphere template. While our 3D reconstructions capture the same degree of detail, they do not suffer from topological defects. In particular, AtlasNet's surfaces contains many patch intersections and holes. Extracting a watertight mesh would require significant post-processing. Additionally, each patch in representation is parameterized sparsely by control points, in contrast to AtlasNet's patches, which must be sampled using a deep decoder network.
In PixcI2Mesh, Wang al. (2018)) input image and output triangle mesh. We train our sphere template models on their data. While their meshes have 2466 vertices (7398 degrees of freedom) we output 54 patches (816 degrees of freedom) more editable and interpretable representation. As shown in Fig. 7e, the low dimensionality of our output is not at the expense of expressiveness.
5 DISCUSSION AND CONCLUSION
As 3D data become more readily available, the need for processing, modifying, and generating 3D models in usable fashion increases. While the quality of results produced by deep learning systems improves, it is necessary to think carefully about their format, particularly with respect to existing use cases. By carefully designing representations together with compatible learning algorithms, can harness these data for the purpose of simplifying workflows in design, modeling, and manufacturing.
Our system is step toward practical 3D modeling assisted by deep learning. Our sparse patch-based representation is close to what is used in artistic and engineering practice, and we accompany it with new geometric regularizers that greatly improve the reconstruction process. Unlike meshes or voxel occupancy functions, this representation can easily be edited and tuned after 3D reconstruction, and captures a trade-off between smoothness and sharp edges reasonable for man- made shapes.
Our work suggests several avenues for future research. While we use pre-trained networks to generate sketch data, we could couple the training of these pieces to alleviate dependence on sketch-3D model pairs. We also could leverage literature in computer-aider geometric design identify other structures amenable to learning. Particularly, multiresolution representations (c.g., subdivision surfaces) might enable learning both high-level smooth structure and geometric details like filigree independently.
Perhaps the most important challenge remaining from our work- and othervsinvolves inferring shape topology. While system supports structural variability and modular parts, scaling this towards completely learned topology nontrivial. This limitation is reasonable for shape classes we consider, but generic sketch reconstruction requires automatically adding and connecting patches.

---

Published as a conference paper at ICLR 2021
6 ACKNOWLEDGEMENTS
The MIT Geometric Data Processing group acknowledges the generous support of Army Research Office grant 1911NF2010168, of Air Force Office of Scientific Research award FA9550-19-1-033 of National Science Foundation grant IIS-1838071, I, from the CSAIL Systems that Learn program, from the MIT-IBM Watson Al Laboratory, from the Toyota Joint Research Center, from gift from Adobe Systems, from an MIT.nano Immersion Lab/NCSOFT Gaming Program seed grant, and from the Skoltech-MIT Next Generation Program. This work was also supported by the National Science Foundation Graduate Research Fellowship under Grant No. 1122374. We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) grant RGPIN-201 9-05597 ("Creating Virtual Shapes via Intuitive Input") and from the Fonds de recherche du Québec Nature et technologies (FRQNT) grant 2020-NC-27008..
REFERENCES
Timur Bagautdinov, Chenglei Wu, Jason Saragih, Pascal Fua, and Yaser Sheikh. Modeling facial geometry using compositional vaes. In CVPR, June 2018.
Pierre Baque, Edoardo Remelli, François Fleuret, and Pascal Fua. Geodesic convolutional shape 2018
optimization. arXiv preprint
Fausto Bernardini, Chandrajit L Bajaj. Jindong Chen, and Daniel R Schikore. Automatic reconstruc- tion of 3d cad models from digital scans. International Journal of Computational Geometry & Applications, 9044n0)):277336)
Mikhail Bessmeltsev and Justin Solomon. Vectorizing Line Drawings via Polyvector Fields. ACM TOG, 38(1), 2019.
Mikhail Bessmeltsev, Will Chang, Nicholas Vining, Alla Sheffer, and Karan Singh. Modeling char- acter canvases from cartoon drawings. ACM Trans. Graph., (455)4622211111122: November 2015.
Mikhail Bessmeltsev, Nicholas Vining, and Alla Sheffer. Gesture3ID: Posing 3D Characters via Gesture Drawings. ACM TOG, 1566:3656:1:5:11115:1133 2016.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information- Rich 3D Model Repository. Technical report, Stanford University - Princcton University Toyota Technological Institute at Chicago, 2015.
Tao Chen, Zhe Zhu, Ariel Shamir, Shi-Min Hu, and Daniel Cohen-Or. 3-Sweep. ACM TOG, 32(6):
1-10, 2013.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In CVPR, pp- 5939-5948, 2019.
Joseph Jacob Cherlin, Faramarz Samavati, Mario Costa Sousa, and Joaquim A. Jorge. Sketch-based modeling with few strokes. SCCG, 1(212):137, 2005.
Forrester Cole, Aleksey Golovinskiy, Alex Limpaecher, Heather Stoddart Barros, Adam Finkelstein, Thomas Funkhouser, and Szymon Rusinkiewicz. Where do people draw lines? Communications of the ACM, 550) 2012.
S. A. Coons. Surfaces for computer-aided design of space forms. Technical report, 1967.
Johanna Delanoy, Mathieu Aubry, Phillip Isola, Alexei A Efros, and Adrien Bousseau. 3d sketching using multi-view deep volumetric prediction. Conference on Computer Graphics and Interactive Techniques, 1(1):1-22, 2018.
Zhantao Deng, Jan Bednaří, Mathicu Salzmann, and Pascal Fua. Better patch stitching for parametric surface reconstruction. arXiv preprint arXiv:2010.0702,, 2020.
Chao Ding and Ligang Liu. A survey of sketch based modeling systems. Front. Comput. Sci., 10(6): 985-999, December 2016.
10

---

Published as a conference paper at ICLR 2021
Matthias Eck and Hugues Hoppe. Automatic reconstruction of b-spline surfaces of arbitrary topological type. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pp. 325-334, 1996.
Diaa F Elkott, Hoda A Elmaraghy, and Waguih H Elmaraghy. Automatic sampling for cmm inspection planning of free-form surfaces. international Journal of production Research, 40(1 2653-2676, 2002. 2002.
Even Entem, Loic Barthe, Marie-Paule Cani, Frederic Cordier, and Michiel van de Panne. Modeling 3d animals from a side-view sketch. Comput. Graph., 42(C):22-133 February 2015.
Haoqiang Fan, Hao Su, and Leonidas Guibas. DeepPointStt : A Point Set Generation Network for 3D Object Reconstruction from Single Image. CVPR, 2017.
Jun Gao, Chengcheng Tang, Vignesh Ganapathi- Subramanian, Jiahui Huang, Hao Su, and Leonidas J. Guibas. DeepSpline: BntaeDriven Reconstruction of Parametric Curves and Surfaces. pp. 1-13, 2019.
Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Samna, William Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In ICCV, pp- 7154-7164, 2019.
Yotam Gingold, Takeo Igarashi, and Denis Zorin. Structured annotations for 2D-to-3D modeling. ACM TOG, 28
Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry. AtlasNet: A Papier-Mache Approach to Learning 3D Surface Generation. CVPR, pp. 216-224, 2018.
Yulia Gryaditskaya, Mark Sypesteyn, Jan Willem Hoftijzer, Sylvia Pont, Frédo Durand, and Adrien Bousseau. Opensketch: A Arichly-annotal dataset of product design sketches. ACM TOG, 38, 11 2019.
Christian Häne, Shubham Tulsiani, and Jitendra Malik. Hierarchical surface prediction. 1PAM1,2019
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016
Victoria Hernández-Mederros and Jorge Estrada-Sarlabone Sampling points on regular parametric curves with control of their distribution. Computer Aided Geometric D 20(6) 2003.
H. Huang, E. Kalogerakis, E. Yumer, and R. Mech. Shape synthesis from sketches via procedural models and convolutional networks. IEEE TVCG, 23(8):2003-2013,
2017.
Jingwei Huang, Hao Su, and Leonidas Guibas. Robust watertight manifold surface generation method for shapenet models. arXiv preprint arxiv:1802.016688 2018.
Takeo Igarashi, Satoshi Matsuoka, and Hidchiko Tanaka. Teddy: A sketching interface for 3d freeform design. In Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '99, pp. 409-416, 416, New York, NY, USA, 1999. ACM Press/Addisson- Deeess//ddisooWWeeeeyy Publishing Co. ISBN 0-201-48560-5.
Anil Jain, Yu Zhong, and Marie-Pierre Dubuisson-Joll, Deformable template models: review. Signal Process., 1192):109-122 December 1998.
Wenzel Jakob. Marco Tarini, Daniele Panozzo, and Olga Sorkine-Hornung. Instant field-aligned meshes. ACM Transactions on Graphics (Proceedings of SIGGRAPH ASIA), 34(6), November 2015. doi: 10. 145/2416795.22811111
Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, and Jitendra Malik. Learning category- specific mesh reconstruction from image collections. In ECCV, 2018.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In CVPR, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14/2 69800, 2014.
11

---

Published as a conference paper at ICLR 2021
Venkat Krishnamurthy and Mare Levoy. Fitting smooth surfaces to dense polygon meshes. In Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '96, pp. 313-324, New York, NY, USA, 1996. Association for Computing Machinery. ISBN 0897917464.
Changjian Li, Hao Pan, Yang Liu, Xin Tong, Alla Sheffer, and Wenping Wang. Robust flow-guided neural prediction for sketch-based freeform surface modeling. ACM Trans. Graph., 37(6), December 2018.
Or Litany, Alex Bronstein, Michael Bronstein, and Ameesh Makadia. Deformable shape completion with graph convolutional autoencoders. CVPR, 2018.
Chenxi Liu, Enrique Rosales, and Alla Sheffer. Strokeaggregator: Consolidating raw sketches into artist-intended curve drawings. ACM TOG, 37(4), 2018.
M. Liu, O. Tuzel, A. Veeraraghavan, and R. Chellappa. Fast directional chamfer matching, In CVPR, pp. 1696-1666000 1166-1703, June 2010.
Zhaoliang Lun, Matheus Gadelha, Evangelos Kalogerakis, Subhransu Maji, and Rui Wang. 3d shape reconstruction from sketches via multi-view convolutional networks. In 3DV, 2017.
Priyanka Mandikal, K L Navaneet, Mayank Agarwal, and R Venkatesh Babu. 3D-LMNet: Latent embedding matching for accurate and diverse 3d point cloud reconstruction from single image. In BMVC, 2018.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction function space. In CVPR, 2019.
Charlie Nash, Yaroslav Ganin, SM Eslami, and Peter W Battaglia. Polygen: An autoregressive generative model of 3d meshes. arXiv preprint arXiv:2002. 10880, 2020.
Gen Nishida, Ignacio Garcia-Dorado, Daniel G. Aliaga, Bedrich Benes, and Adrien Bousseau. Interactive sketching of urban procedural models. ACM Trans. Graph., 35(4), 2016.
Luke Olsen, Faramarz F. Samavati, Mario Costa Sousa, and Joaquim A. Jorge. Sketch-based modeling: survey. Computers Graphics, 33(1):85-103, feb 2009.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, June 2019.
Les Piegl and Wayne Tiller. The NURBS Book. Springer- Verlag, New York, NY, USA, second edition, 1996.
Danilo Jimenez Rezende, SM Ali Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess. Unsupervised learning of 3d structure from images. In Advances neural information processing systems, pp. 4996-5004, 2016.
Edgar Simo-Serra, Satoshi lizuka, and Hiroshi Ishikawa. Mastering Sketching: Adversarial Augmentation for Structured Prediction. ACM TOG, 2018.
2018
Dmitriy Smirnov, Matthew Fisher, Vladimir G. Kim, Richard Zhang, and Justin Solomon. Deep parametric shape predictions using distance fields. In CVPR, 2020.
Chunyu Sun, Qianfang Zou, Xin Tong, and Yang Liu. Learning adaptive hierarchical cuboid abstractions of 3d shape collections. ACM TOG, 38(6), 2019.
Shubham Tulsiani, Hao Su, Leonidas J. Guibas, Alexei A. Efros, and Jitendra Malik. Learning shape abstractions by assembling volumetric primitives. In CVPR, 2017a
Shubham Tulsiani, Tinghui Zhou, Alexei A. Efros, and Jitendra Malik. Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency. 2017b.
Shubham Tulsiani, Alexei Efros, and Jitendra Malik. Multi-view consistency as supervisory signal for learning shape and pose prediction. In CVPR, pp. 2897-2905, 2018.
12

---

Published as a conference paper at ICLR 2021
Jian Wang, Xiang Jiang, LA Blunt, Richard K Leach, and Paul J Scott. Intelligent sampling for the measurement of structured surfaces. Measurement Science and Technology. 23(8):085006, 2012.
Lingjing Wang, Jifei Wang, Cheng Qian, and Yi Fang, Unsupervised learning of 3d model recon- struction from hand-drawn sketches. In ACM MM, ACM MM, pp. 1820-1828, Association for Computing Machinery, Inc, 2018a. 26th ACM Multimedia conference, MM 2018; Conference date: 22-10-2018 Through 26-10-2018.
Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In ECCV, 2018b.
Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. 3dn: 3d deformation network. In CVPR, 2019.
Francis Williams, Teseo Schneider, Claudio Silva, Denis Zorin, Joan Bruna, and Daniele Panozzo. Deep geometric prior for surface reconstruction. In CVPR, pp. 10130-10139, 2019.
Jiajun Wu, Tianfan Xue, Joseph Lim, Yuandong Tian, Joshua B Tenenbaum, Antonio Torralba, and William T Freeman. Single image 3d interpreter network. In ECCV, 2016.
Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun, William T Freeman, and Joshua B Tenenbaum. MarrNet: 3D Shape Reconstruction via 2.5D Sketches. Advances In Neural Information Processing Systems, 2017.
Jiajun Wu, Chengkai Zhang, Xiuming Zhang, Zhoutong Zhang, William T Freeman, and Joshua B Tenenbaum. Learning 3D Shape Priors for Shape Completion and Reconstruction. In ECCV, 2018.
Xinchen Yan, Jimei Yang. Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. In Advances in neural information processing systems, pp. 1696-1704, 2016.
Yaoqing Yang. Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In CVPR, June 2018.
Li Yi, Vladimir G. Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation 3d shape collections. SIGGRAPH Asia, 2016.
Kangxue Yin, Hui Huang, Daniel Cohen-Or, and Hao Zhang. P2p-net: Bidirectional point displace- ment net for shape transform. ACM Transactions on Graphics( Special Issue of SIGGRAPHH) 37
(4):152::1152:13, 2018
Mehmet Ersin Yumer and Levent Burak Kara. Surface creation unstructured point sets using neural networks. Computer-Aided Design, 44(7):644- 656, 2012.
Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Joshua B Tenenbaum, William T Freeman, and Jiajun Wu. Learning to Reconstruct Shapes from Unseen Classes. NeurlPS, 2018.
13

---

Published as a conference paper at ICLR 2021
A
APPENDIX
A.1 CHANGE OF VARIABLES FOR VARIATIONAL CHAMFER DISTANCE
We derive the expression for area- area-weighted variational Chamfer distance:
Ch (P.M)) = Area(P) /
1
inf d(z.y)dr PyEM
(12)
[
inf d( ((.)sd))
(13)
Area P)
NEM
inf (ntdd((sstt)))))(sstt)ddd yEM
(14)
PP) Area (
(P) B(ae)ael
inf d ((P(ss)).)))
(15)
.VEM
B linf add ((s.))))))))
(16)
((, [s,tt]
A.2 ABLATION STUDY
We perform an ablation study our method on an airplane model, demonstrating the effect training without each term in our loss function well the difference between category- template, a 54-patch sphere template, and a lower resolution 24-patch template. The results are shown in Fig. 8. We also show an ablation results on the knife model in Fig.
(b)
(f)
g
(h)
Figure& 8: An ablation study of our model without normal alignment loss (a), without collision
detection loss (b), without patch flatness loss (c), without template normal loss (d), without symmetry loss (e), as well as using 24-patch (f) and 54-patch (g) sphere templates compared to final result (h).
14

---

Published as a conference paper at ICLR 2021
self-intersection
(a)
(b)
(c)
(d)
(e)
Figure 9: An ablation study of our model without normal alignment loss (a), without collision detection loss (b), without patch flatness loss (c), without template normal loss (d), compared the final result (c).
The ablation study demonstrates the contribution of each component of our system method to the final result. Training without collision detection loss results in patch intersections. Omitting the normal loss causes the 3D surface to suffer in smoothness. Patch flatness and and template normal losses encour- age patch seams to align to sharp features. While both sphere templates capture the geometry, using more patches captures greater details, and using non-generic template further improves the model.
We additionally provide quantitative ablation study for the knives and airplanes category in Table 1. We train model without each of the normals, flatness, template normals, and collision loss terms for 50000 steps. for each model, we report the average Chamfer loss, normals loss, flatness loss, and template normals loss (as defined 53.2) on the set, in addition to the average number of intersecting patch pairs. The results demonstrate that each loss term contributes to our full model without sacrificing reconstruction quality.
Model
Ch. Loss
Norm. Loss
Flat. Loss
Temp. Norm. Loss
# Int. Pairs
airplanes (full)
0.0118
0.725
0.0000732
0.792
0.0220
airplanes (no norm.)
0.0105
1.09
0.0000941
0,935
0.0357
airplanes (no flat.)
0.0119
0.748
0.000254
0.866
0.0302
airplanes (no temp. norm.)
0.0114
0.723
0.0000734
0.889
0.0110
airplanes (no coll.)
0.0112
0.727
0.000113
1.07
3.09
knives (full)
0.0124
0.635
0.0000950
0.669
0
knives (no norm.)
0,0124
1.12
0.0000902
1.46
3.95
knives (no flat.)
0.0122
0.648
0.00132
0.733
0.0263
knives (no temp. norm.)
0.0126
0.671
0.0000886
1.03
0.132
knives (no coll.)
0.0122
0.631
0.0000963
0.722
0,0263
Table Quantitative ablation study of our loss function.
15

---

Published as a conference paper at ICLR 2021
A.3 QUANTITATIVE COMPARISON TO PIXEL2MESH
We compare to Pixel2Mesh quantitatively in Table 2 We select 2500 random test set views and com- pute Chamfer distance using 5000 sampled points. While obtain comparable Chamfer distance values, our representation is significantly more compact, editable, and less prone to non-manifold artifacts.
CD
DOF
Category
P2M
ours
P2M
ours
airplane
0.022
0.025
7398
816
car
0.018
0.022
7398
816
Table Quantitative comparison Pixel2Mesh. We report Chamfer distance (CD) and degrees of freedom in the representation (DOF). We obtain comparable Chamfer distance using representation that is an order of magnitude more compact and without non-manifold artifacts.
A.4 QUAD MESI GENERATION
Given collection of Coons patches, obtain quad mesh by subdividing each patch into some number of quads. In order for the quad mesh to be uniform, we determine the number of subdivisions for each boundary curve of each patch solving integer linear problem (ILP). Our free variables are the number of subdivisions per curve, and the objective function encourages the number of subdivisions be proportional the are length of each curve. The constraints ensure that opposite curves for each patch are subdivided into equal number of segments, and when one curve contains multiple other curves (due edge junctions), the numbers of subdivisions are compatible. In particular, we solve the following ILP:
minimize
Σ
s.t.
24 ((()
ViEE
Σ
Jj
ViCE
(EJ(1)
24 E
ViEE,
where x; the number of subdivisions for curve s; the are length of curve & contains all the curves, O maps curve to the opposite curve in its patch, J maps curve to a list of curves contained within and a a parameter that controls the coarseness of the resulting quad mesh.
We solve the ILP using MOSEK, subdivide our Coons patches to obtain a quad mesh and then perform preseerin Laplacian smoothing in Meshl aa for 10 iterations with maximum angle displacement of 5
16

---

Published as a conference paper at ICLR 2021
A.5 ADDITIONAL QUALITATIVE RESULTS
We show models from the test set for each shape category produced by our full model trained with category-specific templates.
Figure 10: Airplanes from the test set.
17

---

Published as a conference paper at ICLR 2021
Figure 11: Bathtubs from the test set.
18

---

Published as a conference paper at ICLR 2021
Figure 12: Bottles from the test set.
19

---

Published as a conference paper at ICLR 2021
Figure 13: Cars from the test set.
20

---

Published as a conference paper at ICLR 2021
Figure 14: Guitars from the test set.
21

---

Published as a conference paper at ICLR 2021
TREFT
Figure 15: Guns from the test set.
22

---

Published as a conference paper at ICLR 2021
Figure 16: Knives from the test set.
23

---

Published as a conference paper at ICLR 2021
Figure 17: Mugs from the set.
24


Reviews:
[{"review_id": "wdO7OlqaDKE", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a self-supervised method to fit a template (represented as a union of Coons patches) to a certain 2D sketch. It derives a way to build a proper template, uses a network to predict the patches' parameters, and proposes a union of different losses. The qualitative results of the method are shown in several different objects. \n\nPROS\n- Losses: I appreciated the effort of the authors in formulating an ensemble of losses for this specific representation. I think they will be useful for future works with similar representations.\n- Versatility: this representation permits working with a large variety of different classes of objects (also disconnected ones, but not topological changes), at an arbitrary resolution and predicts a small set of values. \n\nThe paper does not clearly state if the code and the data will be made publicly available, but since the code is attached to the submission I assume this is in the will of the authors, and this is also a contribution.\n\nFinally, the paper is well organized and I like the presentations.\n\nCONS\n- Quantitative analysis: the only quantitative evaluation is reported in the supplementary material. I think this is the major weakness of the paper. I would suggest having more experiments to analyze the performance of the method. In particular, the ablation is a critical part since many losses are involved; I would suggest providing some quantitative measures, for example showing how each loss improves a particular aspect (e.g. the distance from the ground truth, the difference between normals, the intersection area, ...).\n\nI would also suggest highlighting the output patches on some qualitative results.\n\nMinor fixes:\n- Equation 4, fix the pedix of $\\mathcal{U}_{\\[0,1\\]^2}$ \n- Equation 13 (Sup.Mat), fix the Integral limits locations\n- I think \"P2P-NET: Bidirectional Point Displacement Net for Shape Transform\"(Yin et al., 2018) worth a mention since solves (among the others) a similar problem\n\n\nPRE-REBUTTAL RATING\nI am favorable to accept the paper because it provides several tools that will be useful for future works, and I think it has interest points for the Computer Graphics community. For the acceptance I would recommend adding some more quantitative analysis, both for ablation and for the shown results.\n----------\nFINAL-RATING\n\nI read the other reviews and authors' replies carefully.\n\nFirst of all, I would acknowledge the effort of the authors in replying to reviewers' concerns.\n\nAbout my points, I agree with other reviewers that the quantitative analysis is a bit limited (I think it is also the main criticism), but the introduced ablation and the new Figure 6 (numerically comparing against other SOTA methods) are convincing. The authors solved also my other concerns, and so I raise my score; I think the study of this representation is interesting, and well fit the audience of ICLR.\n\nThanks to the author for their availability, and best of luck with their work!\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "slLpPK-JeD", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I like the idea of this paper and think that finding new representations of 3D shapes in deep learning approaches is important and interesting. However I have a number of concern with respect the claims of this paper, and the lack of sufficient evidence backing them up: \n\n-> What does \"sampling uniformly from patches is difficult\" mean? Can you clarify this statement both here, and in the paper? \n\n-> You provide all these detail about how you make the chamfer distance and normal alignment work for patches, but then say you can triangulate the patches anyway. If you can triangulate the patches why can you not directly apply the chamfer distances and normal alignment to points uniformly sampled over these triangles?  See: \"GEOMetrics: Exploiting Geometric Structure for Graph-Encoded Objects\" or \"Point2Mesh: A Self-Prior for Deformable Meshes\" for detail on how this is done. Would this not be easier? This would probably also be true for the patch flatness regularizer, where instead you could condition adjacent faces to have low curvature such as in the smoothness loss of \"Neural 3D Mesh Renderer\". This is not to say that the approaches you have taken are wrong, but you are proposing alternatives to solutions that already exist, perform well, and are naturally applicable to your setting. I think is it very important that you compare your approaches to these in some way in the paper. An experiment to this effect would go a long way to increasing my score.\n\n\n-> The ablation you provide in the supplemental is not sufficient. Demonstrating the difference over a single model does not provide me a good understanding of the differences. You should provide a visual ablation over a larger number of  models, and in addition provide a quantitative ablation of your metrics( chamfer distance, normal alignment) over your whole test set. This is absolutely necessary. Again showing this here and in the paper will go a long way to increasing my score. \n\n-> You provide an explicit definition of patches, but no intuition as to what they are. A reader can go and look up coons patches independently, but it would be helpful to provide some help here. A diagram might be helpful to explain this. Patches seem to be quite similar to charts as referenced in \"AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation\n\" and \"3D Shape Reconstruction from Vision and Touch\". It might be helpful to disambiguate this in the related work. I realize you touch lightly on this in the results section, though this it too late in the paper to establish the understanding on your approach. \n\n-> There is a lack of quantitative comparison in this paper. I see that you compare visually to Pixel2Mesh and AtlasNet, but this is only over a small number of models. A reader has no way to determine if these results have been hand picked to make your approach look superior. You need to provide a comparison to contemporary methods for 3D reconstruction from images. Here Pixel2Mesh and AtlasNet are probably not sufficient either as better performing methods have since been released. The comparison in the supplemental is not sufficient, It should be over a large number of categories, and methods. \n\n-> You write: \"Unlike meshes or voxel occupancy functions, this representation can easily be edited and tuned after 3D reconstruction\". Meshes are easily edited in 3D design software and also can be converted to NURBS. In addition voxels and occupancy functions can be converted into mesh representations and so also editable in these software as well. In fact I believe 3D deep learning frameworks exists which handle this conversion natively. I think you should be clear about these claims and make them less strong. \n\nI want to be convinced that the proposed approach works better that what is already available , or at least better in some specific domain. The paper in its current form does not do this due to the concerns raised above. I am willing to revise my score if they are properly addressed. \n\n---------------------------------------------------------------------------------------------------------------------------\n\nI believe that the authors have adequately addressed my concerns, and have made a good effort to improve the quality of their work and so I am raising my score. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "S1J41CQgK35", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method that leverages parametric surface patches as the fundamental representation in the task of shape modeling and reconstruction. This method requires a pre-generated template for each shape category. Several losses are specially designed to regularize the generation of the surface patches. Empirical results have demonstrated the performance of the proposed method in sketch-based shape reconstruction and 3D shape interpolation.\n\nPros:\n- The paper shows some good results on reconstructing some simple shapes, such as bottles, mugs, bathtubs, etc.\n- The idea of using parametric surface patches may reduce the parameter space that a network has to search and could potentially lead to smoother surfaces.\n\nCons:\n- It is not clear to me why such representation is advantageous over conventional polygonal mesh-based representation. \n1) Though the parametric surface patch can typically lead to smoother reconstruction, it may also suffer from the incapability of capturing fine geometric details.  \n2) Similar to the mesh-based approach,  the patch-based method still has to fight similar challenges, e.g. avoiding the intersections, undesirable shape distortions, etc. \n3) Editing-wise, the reconstructed polygonal mesh can be easily converted to parametric surface patches to ease the following editing workflow, by registering the pre-generated template of coon patches to the reconstructed shape. \n\n- There are no quantitative evaluations in the paper.  Also, the visual comparisons with other approaches are very limited -- only one or two examples are presented. Due to the lack of evaluations, it is difficult to determine the effectiveness of the proposed approach.\n\n---- Final Rating ----\n\nThe revised version of the paper with additional experiments has addressed my concern on the limited advantage over prior deep 3D representation.  Fig.6 in the revised version has shown the potential of the proposed approach in generating directly usable representation in downstream applications, including CAD design and manufacturing. Hence, I would change my rating to positive.\n\nHowever, the paper still lacks quantitative and qualitative evaluations. I hope it could be properly addressed in the final version.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "41kuyx2AWEM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the paper:\nThis paper proposes a shape generation methods where shapes are represented by a set of parametric patches. Sets of patches from the same category will follow certain constrained defined by templates. The templates are generated from some abstract representation of the category of interested (i.e. sets of cubbies). \nThe authors evaluate the method qualitatively on sketch-to-shape task, and show that the methods allows generating meshes that’s editable, interpretable, and with certain level of details.\n\nStrength:\n1. The representation of parametric patches have several advantages compared to prior works (whose patches are represented by neural networks). It can create sparse, compact, interpretable, and editable shapes that can be directly imported into many industrial design software.\n2. The parametric patches allow computing CD without uniformly sampling from the surface.\n\nWeakness:\n\n1. The paper’s evaluation of the method is very limited. Even though the representation has many inherit advantages, I still expect the authors to demonstrate the method’s advantage in different tasks (not only sketch-modeling), different datasets (not just limited to rather small categories like Mugs), and across baselines with different representations (such as implicit representation, etc)\n2.  A major weakness of the methods comes from the template generation. It occurs to me that the paper requires human user to pre-defined templates using a set of cuboids for each set of shapes before it can start generate. This makes it a little bit hard for the method to capture all topological variation of the category (which the author also mentioned in the last section). With this regard, the method requires additional information compared many prior methods (for example, implicit methods that can handle many topology and good details, and it doesn’t take a template). \n3. There are potentially missing related works. Some of which I think the paper should compare with as baselines:\n\n[A] Nash, Charlie, et al. \"PolyGen: An autoregressive generative model of 3D meshes.\" arXiv preprint arXiv:2002.10880 (2020).\n[B] Deng, Zhantao, et al. \"Better Patch Stitching for Parametric Surface Reconstruction.\" arXiv preprint arXiv:2010.07021 (2020).\n\nWhile the paper’s idea has its value as the representation could be useful for editing in many industrial software, but it requires more sophisticated evaluation (quantitative evaluation, more baselines, more tasks) in order to show-case the effectiveness of the representation. With that, I do not recommend the paper for ICLR.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Description:
The paper presents a patch-based 3D representation of man-made shapes that can be computed with deep learning and used directly in existing CAD applications. This representation is based off a deformable parametric template with Coons patches. Results in sketch-based modelling tasks shows comparable results with STOA

Strengths:
- The patch-based representation provide several advantages: compact, sparse, interpretable, consistent and easily editable.
- Can infer the right template, and thus does not require manually created templates

Weaknesses
- Limited evaluation restrained to mostly sketch-based modelling, and missing evaluation against a few STOA methods

The paper has introduced a very impactful new representation for 3D shapes and has strong technical novelty. I recommend, as reviewers have suggested, more in-depth quantitative evaluation (against other work and ablation studies)
Decision: Accept (Poster)