Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis
Authors: Tianchang Shen
Jun Gao
Kangxue Yin
Ming-Yu Liu
Sanja Fidler
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3D Deep Learning
3D Super-Resolution
3D Content Creation
3D Shape Representation

Abstract: We introduce DMTet, a deep 3D conditional generative model that can synthesize high-resolution 3D shapes using simple user guides such as coarse voxels. It marries the merits of implicit and explicit 3D representations by leveraging a novel hybrid 3D representation. Compared to the current implicit approaches, which are trained to regress the signed distance values, DMTet directly optimizes for the reconstructed surface, which enables us to synthesize finer geometric details with fewer artifacts. Unlike deep 3D generative models that directly generate explicit representations such as meshes, our model can synthesize shapes with arbitrary topology. The core of DMTet includes a deformable tetrahedral grid that encodes a discretized signed distance function and a differentiable marching tetrahedra layer that converts the implicit signed distance representation to the explicit surface mesh representation. This combination allows joint optimization of the surface geometry and topology as well as generation of the hierarchy of subdivisions using reconstruction and adversarial losses defined explicitly on the surface mesh. Our approach significantly outperforms existing work on conditional shape synthesis from coarse voxel inputs, trained on a dataset of complex 3D animal shapes. Project page: https://nv-tlabs.github.io/DMTet/.

Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis
Tianchang Shen 1,2,3
Jun Gao 1,2,3
Kangxue Yin
Ming- Yu Liu
Sanja Fid
NVIDIA
University of Toronto
Vector Institute"
{frshon, jung, kangxoey, ningyul, sfidler)dnvidia.com
Abstract
We introduce DMTET, deep 3D conditional generative model that can synthesize high-resolution 3D shapes using simple user guides such as coarse voxels. It marries the merits of implicit and explicit 3D representations by leveraging novel hybrid 3D representation. Compared to the current implicit approaches, which are trained regress the signed distance values, DMTET directly optimizes for the reconstructed surface, which enables us to synthesize finer geometric details with fewer artifacts. Unlike deep 3D generative models that directly generate explicit representations such as meshes, our model can synthesize shapes with arbitrary topology. The core of DMTET includes a deformable tetrahedral grid that encodes a discretized signed distance function and differentiable marching tetrahedra layer that converts the implicit signed distance representation to the explicit surface mesh representation. This combination allows joint optimization of the surface geometry and topology as well as generation of the hierarchy of subdivisions using reconstruction and adversarial losses defined explicitly on the surface mesh. Our approach significantly outperforms existing work on conditional shape synthesis from coarse voxel inputs, trained dataset of complex 3D animal shapes. Project page: https: //tv///tv/tt/e// github. io/DMTet/.
Introduction
Fields such as simulation, architecture, gaming, and film rely on high-quality 3D content with rich geometric details and complex topology. However, creating such content requires tremendous expert human effort. It takes a significant amount of development time to create each individual 3D asset. In contrast, creating rough shapes with simple building blocks like voxels has been widely adopted. For example, Minecraft has been used by hundreds of millions of users for creating 3D content. Most of them are non-experts. Developing A.I. tools that enable regular people upscale coarse, voxelized objects into high resolution, beautiful 3D shapes would bring one step closer to democratizing high-quality 3D content creation. Similar tools can be envisioned for turning 3D scans of objects recorded by modern phones into high-quality forms. Our work aspires to create such capabilities.
powerful 3D representation is critical component of a learning-bate based 3D content creation framework. A good 3D representation for high-quality reconstruction and synthesis should capture local geometric details and represent objects with arbitrary topology while also being memory and computationally efficient for fast inference in interactive applications.
Recently, neural implicit representations [8, 39, 42, 51], which represent a shape via a signed distance field (SDF) or an occupancy field (OF), have emerged as an effective 3D representation. Neural implicits have the benefit of representing complex geometry
neural network implicitly
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

and topology, not limited to a predefined resolution. The success of these methods has been shown in shape compression [49, 13, 51], single-image shape generation [47, 60, 48], and point cloud reconstruction [57]. However, most of the current implicit approaches are trained by regressing to SDF or OF values and cannot utilize an explicit supervision on the target surface, which imposes useful constraints for training. To mitigate this issue, several works [45, 31] proposed to utilize iso-surfacing techniques such as the Marching Cubes (MC) algorithm to extract a surface mesh from the implicit representation, which, however, is computationally expensive.
In this work, we introduce DMTET, a deep 3D conditional generative model for regouutsoot 3D shape synthesis from user guides in the form of coarse voxels. In the heart of DMTET is new differentiable shape representation that marries implicit and explicit 3D representations. In contrast to deep implicit approaches optimized for predicting sign distance (or occupancy) values, our model employs additional supervision on the surface, which empirically renders higher quality shapes with finer geometric details. Compared to methods that learn to directly generate explicit representations, such as meshes [54], by committing to a preset topology, our DMTET can produce shapes with arbitrary topology. Specifically, DMTET predicts the underlying surface parameterized by implicit function encoded via a deformable tetrahedral grid. The underlying surface is converted into an explicit mesh with a Marching Tetrahedra (MT) algorithm, which we show is differentiable and more performant than the Marching Cubes. DMTET maintains efficiency by learning to adapt the grid resolution by deforming and selectively subdividing tetrahedra. This has the effect of spending computation only on the relevant regions in space. We achieve further gains in the overall quality of the output shape with learned surface subdivision. Our DMTET is end-to-end differentiable, allowing the network to jointly optimize the geometry and topology of the surface, as well as the hierarchy of subdivisions using loss function defined explicitly on the surface mesh.
We demonstrate our DMTET on two challenging tasks: 3D shape synthesis from coarse voxel inputs and point cloud 3D reconstruction. We outperform existing state-of-the-art methods by significant margin while being 10 times faster than alternative implicit representation-basee methods at inference time. In summary, we make the following technical contributions:
1. We show that using Marching Tetrahedra (MT) as a differentiable surfacirgggg layer allows topological change for the underlying shape represented by implicit field, in contrast to the analysis in prior works (31,45].
2. We incorporate MT in DL framework and introduce DMTET, hybrid representation that combines implicit and explicit surface representations. We demonstrate that the additional supervision (e.g. chamfer distance, adversarial loss) defined directly on the extracted surface from implicit field improves the shape synthesis quality.
3. We introduce a coarse-to- to-sse--t-fieee fine optimization strategy that scales DMTET to high resolution during training. We thus achieves better reconstruction quality than state-of-the the-art methods on challenging 3D shape synthesis tasks, while requiring a lower computation cost.
2 Related Work
We review the related work on learning-basedd 3D synthesis methods based on their 3D representations.
Voxel-based Methods Early work [59, 10, 38] represented 3D shapes as voxels, which store the coarse occupancy (inside/outside) values regular grid, which makes powerful convolutional neural networks native and renders impressive results on 3D reconstruction and synthesis [12, 11, 58, 2]. For high-resolution shape synthesis. DECOR-GAN [6] transfers geometric details from high-resolution shape represented in voxel to a low-resolution shape by utilizing a discriminator defined on 3D patches of the voxel grid. However, the computational and memory costs grow cubically as the resolution increases, prohibiting the reconstruction of fine geometric details and smooth curves. One common way to address this limitation is building hierarchical structures such as octrees [46, 52, 55, 56, 24, 52], which adapt the grid resolution locally based on the underlying shape. In this paper, we adopt hierarchical deformable tetrahedral grid to utilize the resolution better. Unlike octree-based shape synthesis, our network learns grid deformation and subdivision jointly to better represent the surface without relying on explicit supervision from a pre-computed hierarchy.
Deep Implicit Fields (DIFs) represent 3D shape as zero level set of continuous function parameterized by neural network [39, 44, 17, 40]. This formulation can represent arbitrary typol- ogy and has infinite resolution. DiFebased shape synthesis approaches have demonstrated strong

---

input
GT Shape
oUnoccucled Grdl Pokn!
Occupled
Surfaze
Loss our forfface face
output
Implicif Funciion (5DF)
Explicit Surface
(c) Refrie Boundary SDF: Deform and trm Groph
Eenohedd Crreecn
(g) Wording
toj Surfoce Suboiiiion sinn
Vaua
(0) Precicting 3DF af intool
Grrd Resolunon
Tetrchedrons.cm and inte polate SD3
(b) Selectivel oly Subdii ice
Inferpplatec SOF)
(M Mesh)
CaromehicSurfoc
Figure I: DMTET reconstructs the shape implicitly in coarse-to-fipg manner by predicting the SDF defined
a deformable tetrahedral grid. then converts the SDF to surface mesh by differentiable Marching Tetrahedra layer. DMTET is trained by optimizing the objective function defined on the final surface.
performance in many applications, including single view 3D reconstruction [60, 30, 47, 48], shape manipulation, and synthesis [26, 21, 28, 14, 1, 1.. However, as these approaches are trained by minimizing the reconstruction loss of function values at a set of sampled 3D locations (a rough proxy of the surface), they tend to render artifacts when synthesizing fine details. Furthermore, if one desires a mesh to be extracted from a DIF, an expensive iso-surfacing step based on Marching Cubes [36] Marching Tetrahedra [15] is required. Due the computational burden, iso-surfacing is often done on a smaller resolution, hence prone to quantization errors. Lei et al. [29] proposes an analytic meshing solution reduce the error, but is only applicable DIFs parametrized by MLPs with ReLU activation. Our representation scales to high resolution and does not require additional modification the backward pass for training end-to-end. DMTET can represent arbitrary typology. and is trained via direct supervision on the generated surface. Recent works [1, 9] learn to regress unsigned distance to triangle soup or point cloud. However, their iso-surfacing formulation is not differentiable in contrast to DMTET.
Surface-based Methods directly predict triangular meshes and have achieved impressive results for reconstructing and synthesizing simpler shapes [54, 23, 5, 41. 7]. Typically, they predefined the topology of the shape. e.g. equivalent to a sphere [54, 5, 25], or union of primitives [43, 53, 19] or a set of segmented parts [61, 62, 50J. As result, they can not model a distribution of shapes with complex topology variations. Recently, DefTet [18] represents a mesh with deformable tetrahedral grid where the grid vertex coordinates and the occupancy values are learned. However, similar to voxel-based methods, the computational costf increases cubically with the grid resolution. Furthermore, the occupancy loss for supervising topology learning and the surface loss for supervising geometry learning do not support joint training, it tends to generate suboptimal results. In contrast, our method is able to synthesize high-resolution 3D shapes, not shown in previous work.
3 Deep Marching Tetrahedra
We now introduce our DMTET for synthesizing high-quality 3D objects. The schematic illustration is provided in Fig. 1. Our model relies on a new, hybrid 3D representation specifically designed for high-resolutic reconstruction and synthesis, which describe in Sec. 3.1. In Sec. 3.2, we describe the neural network architecture of DMTET that predicts the shape representation from inputs such as coarse voxels. We provide the training objectives in Sec. 3.3.
3.1 3D Representation
We represent shape using sign distance field (SDF) encoded with a deformable tetrahedral grid, adopted from DefTet [18, 20]. The grid fully tetrahedralizes a unit cube, where each cell in the volume tetahedron with 4 vertices and faces. The key aspect of this representation is that the grid vertices can deform to represent the geometry of the shape more efficiently. While the original DefTet encoded occupancy defined on each tetrahedron, we here encode signed distance values defined on the vertices of the grid and represent the underlying surface implicitly (Sec. 3.1.1). The use of signed distance values, instead of occupancy values, provides more flexibility representing the underlying surface. For greater representation power while keeping memory and computation manageable, we further selectively subdivide the tetrahedra around the predicted surface (Sec. 3.2.2). We convert the signed distance- based implicit representation into triangular mesh using marching tetrahedra layer, which we discuss in See. 3.3. The final mesh is further converted into a parameterized surface with differentiable surface subdivision module, described in Sec. 3.1.4.
3

---

3.1.1 Deformable Tetrahedral Mesh as an Approximation of an Implicit Function
We adopt and extend the deformable tetrahedral grid introduced in Gao et al. [18], which we denote with (Vr,T), where Vr are the vertices in the tetrahedral grid T. Following the notation in [18], each tetrahedron T. E ' is represented with four vertices (Vay Phys Veyo Vd, J. with l E (I,...., ........KK K}. where K is the total number of tetrahedra and Vis VT-
We represent the sign distance field by interpolating SDF values defined on the vertices of the grid. Specifically, we denote the SDF value in vertex v E Vr as s(U)). SDF values for the points that lic inside the tetrahedron follow barycentric interpolation of the SDF values of the four vertices that encapsulates the point.
Va
41% +1)e) 6/c
3.1.2 Volume Subdivision
[((( +a()))
We represent shape in a coarse to fine manner for efficiency. We determine the surface tetrahedra S by checking whether tetrahedron has vertices with different SDF signs - indicating that it intersects the surface encoded by the SDF. We subdivide T sur/ as well as their immediate neighbors and increase resolution by adding the mid point to each edge. We compute SDF values of the new vertices by averaging the SDF values on the edge (Fig. 2).
Vb
Uc
Figure 2: Volume Subdivision:
Each surface tet.blue) is divided into 8 (red) by adding midpoints.
3.1.3 Marching Tetrahedra for converting between an Implicit and Explicit Representation
We use the Marching Tetrahedra [15] algo- rithm to convert the encoded SDF into an ex- plicit triangular mesh. Given the SDF values {s(wa), s(b,), s(ve), s(va)} of the vertices of a tetra- hedron, MT determines the surface typology inside the tetrahedron based on the signs of s(v). which is illustrated in Fig. 3. The total number of configu- rations is 2 = 16, which falls into 3 unique cases after considering rotation symmetry. Once the sur- face typology inside the tetrahedron is identified, the vertex location of the iso-surface is computed the zero crossings of the linear interpolation along the tetrahedron's edges, shown in Fig. 3.
Ub
pasilive SDF
V.b V. s(v)) - 10 (v
negorive SDF
x(pr)) - (Pa))
Figure 3: Threc unique surface configurations in MT. Vertex color indicates the sign of signed dis- tance value. Notice that flipping the signs of all vertices will result in the same surface configura- tion. Position of the vertex is linearly interpolated along the edges with sign change.
Prior works [45, 31] argue that the singularity in this formulation, i.c. when s(va) = s(vs), prevents the change of surface typology (sign change of s(va)) during training. However, we find that, in practise, the equation is only evaluated when sign(s(va)) * sign(s(ve)). Thus, during training, the singularity never happens and the gradient from loss defined on the extracted iso- (Sec. 3.3), can be back- proppaaae to both vertex positions and SDF values via the chain rule. A more detailed analysis is in the Appendix.
3.1.4 Surface Subdivision
Having surface mesh as output allows us to further increase the representation power and the visual quality the shapes with a differentiable surface subdivision module. We follow the scheme the Loop Subdivision method [35], but instead of using fixed set of parameters for subdivision, we make these parameters learnable DMTET. Specifically, learnable parameters include the positions of each mesh vertex v/. well as a which controls the generated surface via weighting the smoothness of neighbouring vertices. Note that different from Liu et al. [33], we only predict the per-vertex parameter at the beginning and carry it over to subsequent subdivision iterations to attain lower computational cost. We provide more details in Appendix.
3.2 DMTET: 3D Deep Conditional Generative Model
Our DMTET a neural network that utilizes our proposed 3D representation and aims to output a high resolution 3D mesh M from input x (a point cloud or a coarse voxelized shape). We describe the architecture (Fig. 4) of the generator for each module of our 3D representation See. 3.2.1, with the architecture of the discriminator presented in Sec. 3.2.2. Further details are in Appendix.

---

Initial SDF Prediction
Surface Retinement
Discriminator
Trilnnaor
(v).) x))
Sample high-curvature fromm GT
3D CNN
MLP
interp (v. P. (v.,x)
Real/fake
MLPS
GCN
Regular Tel. Grld
Sprce M/G7)
X Gradient
s(t).f())
[App,As(p))
Interp.
PCD
Encoder
Pred or GT Mesh
Build Graph from Surface Tetrahedrons(Dramge
Figure 4: Our generator and discriminator architectures. The generator is composed of two parts- one utilizes MLP to generate the initial predictions for all grid vertices and the other uses GCN to refine the surface.
3.2.1 3D Generator
Input Encoder We use PVCNN [34] from a point cloud. When the imput is a coarse voxelized shape, we sample points on its surface. We compute a feature vector Foo z) for grid vertex E I2 via trilinear interpolation.
imput encoder to extract a 3D feature volume Food (x)
Initial Prediction of SDF We predict the SDF value for each vertex in the initial deformable tetrahedral grid using fully-connected network s(v) MLP (B, x)), v). The fully-connected network additionally outputs feature vector ((v), which is used for the surface refinement in the volume subdivision stage.
Surface Refinement with Volume Subdivision After obtaining the imitial SDF, we iteratively refine the surface and subdivide the tetrahedral grid. We first identify surface tetrahedra T sur based on the current s(e) value. We then build a graph G = (Vsurf: Esurf). where Vsurf: Esurf correspond to the vertices and edges in surf.. We then predict the position offsets Au; and SDF
residual values As(e;) for each vertex i in Veurs using a Graph Convolutional Network [32] (GCN):
fe. concat(u,, (s, s(v,),
(1)
(Av, As(rs) ((:)))-..
(CN((F))t=1.--))
G).
(2)
where N is the total number of vertices in V Vaury and f(,; is the updated per-vertex feature. The vertex position and the SDF value for vertex are updated as of = vn + Au and s(v() s(v;) + (s()). This refinement step can potentially flip the sign of the SDF values refine the local typology, and also move the vertices thus improving the local geometry.
After surface refinement, we perform the volume subdivision step followed by an additional surface refinement step. In particular, we re-identify Teurf and subdivide Teury and their immediate neigh- bors. We drop the unsubdivided tetrahedra from the full tetrahedral grid in both steps, which saves memory and computation, as the size of the Teurs is proportional to the surface area of the object, and scales up quadratically rather than cubically the grid resolution increases.
Note that the SDF values and positions of the vertices are inherited from the level before subdivision, thus, the loss computed at the final surface can back prok-propagate to all vertices from all levels. Therefore, our DMTET automatically learns to subdivide the tetrahedra and does not need an additional loss term in the intermediate steps to supervise the learning of the octree hierarchy as in the prior work [52].
Learnable Surface Subdivision After extracting the surface mesh using MT, we can further apply learnalle surface subdivision. Specifically, we build new graph on the extracted mesh, and use GCN to predict the updated position of each vertex v/, and a, for Loop Subvidision. This step removes the quantization errors and mitigates the approximation errors from the classic Loop Subdivision by adjusting Os,, which are fixed in the classic method.
3.2.2 3D Discriminator
We apply a 3D discriminator D on the final surface predicted from the generator. empirically find that using a 3D CNN from DECOR -AAN [6] as the discriminator on the signed distance field that computed from the predicted mesh is effective capture the local details. Specifically, we first randomly select high-curvature vertex from the target mesh and compute the ground truth signed distance field Sreal E NNNN at a voxelized region around e. Similarly, we compute the signed distance field of the predicted surface mesh M at the same location to obtain Spred N X X Note that Spred is an analytical function of the mesh M, and thus the gradient to Spred can back- propagate the vertex positions in M. We feed Sreat or Spred into the discriminator. along with the feature vector Foor(1) x) in position v. The discriminator then predicts the probability indicating whether the input comes from the real generated shapes.
5

---

3.3 Loss Function
DMTET is end-to-end trainable. We supervise all modules to minimize the error defined on the final predicted mesh M. Our loss function contains three different terms: surface alignment loss to encourage the alignment with ground truth surface, an adversarial loss to improve realism of the generated shape, and regularizations to regularize the behavior of SDF and vertex deformations.
Surface Alignment loss We sample a set of points Pgt from the surface of the ground truth mesh Mat- Similarly, we also sample set of points from Mpred to obtain Ppreds and minimize the L2 Chamfer Distance and the normal consistency loss between Pyt and predd"
Lidd
Σ
min lpp - all2 + qEPs
Σ min llq 10-p0 pll2, Tenormal
Σ (1- ngg);
(3)
PEP
PE PEPPred
gEPae
pe P.Peeed
where ② is the point that corresponds to when computing the Chamfer Distance, and np, ng denotes the normal direction point p, 4-
Adversarial Loss
We use the adversarial loss proposed in LSGAN [37]:
Lo 2 (D(M 1)² D(Mpred))), LG = 2 (D( 1)
(4)
Regularizations The above loss functions operate on the extracted surface, thus, only the vertices that are close to the iso-surface in the tetrahedral grid receive gradients, while the other vertices do not. Moreover, the surface losses do not provide information about what is inside/outside, since flipping the SDF sign of all vertices in tetrahedron would result in the same surface being extracted by MT. This may lead to disconnected components during training. To alleviate this issuc, we add a SDF loss to regularize SDF values:
LSDF
Σ WEVV
(5)
where SDF Mgt) denotes the SDF value of point U; to the mesh Mgt. In addition, we apply the L2 regularization loss the predicted vertex deformations avoid artifacts: Tatet = Ev,ev, Au;l12
The final loss a weighted sum of all five loss terms:
Aea + Anvemal Trnommal + >aLa + ASD + A
(6)
where Ama, Anvemal, Aa, ASDE, Ader are hyperparameters (provided in the Supplement).
Experiments
We first evaluate DMTET in the challenging application of generating high high-quality animal shapes from coarse voxels. We further evaluate DMTET in reconstructing 3D shapes from noisy point clouds on ShapeNet by comparing to existing tfate-of-the-ar methods.
4.1 3D Shape Synthesis from Coarse Voxels
Experimental Settings We collected 1562 animal models from the TurboSquid website These models have wide range of diversity, ranging from cats, dogs, bears, giraffes, to rhinoceros, goats, etc. We provide visualizations in Supplement. Among 1562 shapes, we randomly select 1 1120 shapes for training, and the remaining 442 shapes for testing. We follow the pipeline in Kaolin [27] to convert shapes to watertight meshes. To prepare the input to the network, we first voxelize the mesh into the resolution of 163. and then sample 3000 points from the surface after applying marching cubes to the 16 voxel grid. Note that this preprocessing is agnostic to the representation of the input coarse shape, allowing us evaluate on different resolution voxels, or even meshes.
We compare our model with the official implementation of ConvOnet [44], which achieved SOTA performance voxel upsampling. We also compare DECOR-GAN [6], which obtained impressive results on transferring styles from a high-resolution voxel shape to a low-resolution voxel. Note that the original setting of DECOR-GAN is different from ours. For fair comparison, we use all 1120 training shapes as the high-resolution style shapes during training, and retrieve the closet training shape the test shape as the style shape during inference, which we refer DECOR-Retv. We also compare against a randomly selected style shape as reference, denoted as RECOR-RRadd
com. 'https://www.. www.wwwwww....d..commm com, we obtain consent via an agreement with TurboSquid, and following license at https://blog. https://log..urrboquud... turbosquiddddoooo http://ssuuu..ooo/uuuooooooomoe conn/utbosquiddioo ad-model-licenes

---

Imput
ConyOttt
DECOR Rand
DBCOR-Rat:
Ours Ad....
OLtS
Retv.
CT
Figure 5: Qualitative results on 3D shapes Synthesis from Coarse Voxels. Comparing with all baselines, method reconstructs shapes with much higher quality. Adding GAN further improves the realism of the generated shape. We also show the retrieved shapes from the training set in the sccond last column.
Metrics We evaluate L2 and LI Chamfer Distance, well normal consistency score to assess how well the methods reconstruct the corresponding high-resolution shape following [44]. We also report Light Field Distance [4] (LFD) which measures the visual similarity in 2D rendered views. In addition, we evaluate Cls score following [6]. Specifically, we render the predicted 3D shapes and train
image classifier distinguish whether images from the renderings of real or generated shapes. The mean classification accuracy the trained classifier is reported as Cls (lower better). More details in the Supplement.
Experimental Results We provide quantitative results in Table with qualitative examples Fig. 5. Our DMTET achieves significant improve- ments over all baselines in terms of all metrics. Compared both ConvOnet [44] and DECOR- GAN [6]. our DMTET reconstructs shapes with better quality when training without adversarial loss (5th column in Fig. 5). Further geometric de- tails, including nails, ears, eyes, mouths, etc, are captured when trained with the ad adversarial loss (6th column Fig. 5), significantly improving the realism and visual quality of the generated shape. To demonstrate the generalization ability of our DMTET, we collect cuman--reeted low-resolution ing). We provide qualitative results in Fig. 6. Despite the fact that these human-created shapes have noticeable differences with our coarse voxels used in training, e.g., different ratios of body parts compared with our training shapes (larger head, thinner legs, longer necks), our model faithfully generates high-quality 3D details conditioned each coarse voxel exciting result.
Figure 6: Qualitative Results of synthesizing high- resolution shapes from coarse voxels collected online.
voxels from Turbosquid (shapes unseen in train-
7

---

L2 Chamfer v
Li Chamfer +
Norm. Cons.
LFD
Cls
ConvOner [4
0.83
2.41
0.901
3220
0.63
DECOR Ret-.
1.32
3.81
0.876
3689
0.66
DECOR -andd
2.38
6.85
0.797
5338
0.67
DMTET wo Adv.
0.76
2.20
0.916
2846
0.58
DMTET Table 1: Super Resolution of Animal Shapes: DMTET significantly outperforms all baselines in all metrics.
0.75
2.19
0.918
2823
0.54
User Studies We conduct user studies via Amazon Machanical Turk (AMT) to further evaluate the performance of all methods. In particular, we present two shapes that are predicted from two different models to the AMT workers and ask them to evaluate which one is better looking shape and which one features more realistic details. Detailed experimental settings are provided in the Supplement. We compare DMTET against ConvONet [44], DECOR [6]-Retv, as well as DMTET without adversarial loss (w.o. Adv.). Quantitative results are reported in Table 2. Human judges agree that the shapes generated from our model have better details, compared to all baselines, in vast majority of the cases, Ablations on using adversarial loss demonstrate the effectiveness of generating higher quality geometry using a discriminator during training.
Ablation Studies To evaluate the effective- ness of our volume subdivision and surface sub- division modules, we ablate by sequentially in- troducing them to the base model (we refer as DMTET which we train on 100-resolution uniform tetrahedral grid without both volume and surface subdivision modules and adversarial loss. We conduct user studies to evaluate the improvement after each step using the protocol described in the above paragraph. We first reduce the initial resolution to 70 and employ volume subdivision to support higher output resolution (we refer this model as DMTETy) and compare with DMTET Predictions by DMTETy wins 78% of cases over DMTET B for better looking, and 61% of cases for realistic details, showing that the volume subdivision module is effective in synthesizing shape details. We then add surface subdivision on top of the DMTETy and compare with it. The new model wins 62% of cases over DMTETy for better looking, and 62% of cases for realistic details as well, demonstrating the effect of surface subdivision module in enhancing the shape details.
Ooe [44]
DECOR [6] Retv. DMTET Adv.
Haseline wins
5% 5%
25% 717%
99%%
DMTET wins
95% 795%
74% 783%
71%7775%
Table 2: User Study on 3D Shape Synthesis from Coarse voxels. each cell, we report percentages of shapes for which the users agree are hetter looking (left) or have better details (right).
4.2 Point Cloud 3D Reconstruction
Experimental Settings We follow the setting from DefTet [18], and use all categories in ShapeNet [3] core data?, which we pre-process using Kaolin [27] to watertight meshes. We sample 5000 points for each shape and add Gaussian noise with zero mean of standard deviation 0.005. For quantitative evaluation, we report the LI Chamfer Distance in the main paper, and refer readers to the Supplement for results in other metrics (3D IOU, L2 Chamfer Distance and Fl score). We additionally report average inference time on the same Nvidia V100 GPU.
We compare DMTET against state-of the-tr 3D reconstruction approaches using different represen- tations: voxels [10], deforming a mesh with a fixed template [54], deforming a mesh generated from a volumetric representation [22], DefTet [18], and implicit functions [44]. For fair comparison, we use the same point cloud encoder for all the methods, and adopt the decoders in the original papers to generate shapes in different representations. We also remove the adversarial loss in this application, since baselines also do not have it. We further compare with oracle performance of MC/MT where the ground truth SDF is utilized to extract iso-surface using MC/MT.
Experimental Results Quantitative results are summarized in Table 3, with a few qualitative examples shown in Fig. 7. Compared to DMC [31], which also predicts the SDF values and supervises with a surface loss, DMTET achieves much better reconstruction quality since training using the marching tetrahedra layer is more efficient than calculating an expectation over all possible configurations within one grid cell as done in DMC [31]. Compared to method that deforms a fixed template (sphere) [54], we reconstruct shapes with different topologies, achieving more faithful results compared to the ground truth shape. When compared with other explicit surface representations that also support different topology [18, 22], our method achieves higher quality results for local
'The ShapeNet license is explained http:///shapenet.rrggemmss
8

---

In
3DR2N2
DMC
Pix2Mesh
ConvOnat
MeshRCNN
DEFTET
Ours
GT
Figure 7: Qualitative results on 3D Reconstruction from Point Clouds: Our model reconstructs shapes with more geometric details compared to baselines.
Canserry
Airpane
Beneh
Deesser
Ca
Chair
Displa
Speaker
Rille
Sola
Tasle
Plune
Vessal
Meon
Time mal.
E.57
174
DMC
1.0
1.45
Inech [4
1.9
0.91
ComOnet (4c
0.82
0.95
0.96
1.12
1.00
0.93
1.22
1.12
0.79
0.1
0.
0.62
0.99
0.95
865
MrshRCNN (2])
6.83
1.01
1.0
133
1.10
0.99
1.30
133
0.83
0.96
1.00
0.71
1.03
1.00
228
DETTET
097
133
035
DMTET worleer Suri.)
6.8
036
000
036
0.90
0000
100
100
0.50
0.5
0.5
0.5
030
030
DMEET Worko, Stil)
0.00
032
038
0.2
0050
008
0.50
0.2
0.03
030
032
0.61
030
0381
DM we Vol.
0.65
075
884
035
0.80
0.70
0.8
0095
0.8
0.7
0.79
0.00
0.7
0.20
DMTET RS Suct.
0.00
0.7
038
0.58
0.88
0.79
033
0.94
0.50
0.75
0.79
0.59
0.76
0.78
108
Table 3: Quantitative Results on Point Cloud Reconstruction (Chamfer I.1). Note that all the networks in the baselines are not designed for this task, and thus use the same encoder and their decoder for fair comparison. We also ablate ourselves by operating fixed grid (DMTET (Def, Vol., Surf.)). removing volume subdivision (DMTET wo Vol.), or surface subdivision (DMTET wo Surf.), the both (DMTET wo (Vol., Surf.)).
geometry, benefiting from the fact that the typology is jointly optimized with the geometry, whereas is separately supervised by occupancy loss in [18, 22]. Compared to neural implicit method [44], we generate higher quality shapes with less artifacts, while running significantly faster at inference. Finally, compared to a voxel-based method [10] at the same resolution, our method recovers more geometric details, benefiting from the predicted vertex deformations as well as the surface loss.
4.2.1 Analysis
MC
We investigate how each component in our representation affects the performance and reconstruction quality.
DMTee wo Vol.Sl] (Vol.Surf)
Chamler
Comparisons with Oracle Performance of MC/MT We first demonstrate the effect of learning explicit surface via MT. We compare with the oracle performance of extracting the iso-surface with MT/MC from the ground truth signed dis- tance fields on the Chair test set ShapeNet, which contains diverse high-quality details. Specifically, for MC/MT, first compute the discretized SDF at different grid resolutions, and compare the extracted surface to the ground truth surface.
Num. of Querled 5DT (oonts(10t))
Figure 8: Comparing our DMTET with
oracle performance MC and MT.
As shown in Fig. 8, MT consistently outperforms MC when querying the same number of points. We found the staggered grids pattern in tetrahedral grid [16, 18] better captures thin structures a limited

---

resolution (Fig. 9). This makes MT a better choice for efficiency reasons. The usage of tetrahedral mesh DMTET follows this motivation. Without deforming the grid, DMTET outperforms the oracle performance of MT by a large margin when querying the same number of points, although DMTET predicts the surface from noisy point cloud. This demonstrates that directly optimizing the reconstructed surface can mitigate the discretization errors imposed by MT to a large extent.
Ablation Studies We further provide ablation studies on the entire ShapeNet test set, which is summarized in Tab. 3. We first compare the ver- sion where we only predict SDF values without learning deform the vertices and volume/surface subdivision with the version that predicts both SDF and the deformation. Predicting deformation along with SDF is significantly more performant, since vertex movements allow for better recon- struction of the underlying surface. This is espe- cially true for categories with thin structures (e.g. lamp) where the grid vertices are desired to align with them. We further ablate the use of volume subdivision and surface subdivision. We show that each component provides an improvement. In particular, volume subdivision has a significant improvement for object categories with fine-grained structural details, such as airplane and lamp, which require higher grid resolutions to model the occupancy change. Surface subdivision generates shapes with a parametric surface, avoiding the quantization errors in the planar faces and produces more visually pleasing results.
MC6x101)
MCB6x10")
(Dt..M Sui) 6x10')
(6x10%)
MT (x10') MT(6x100)
LMMlet vo (Vol.Sur)) (8x10))
GT
Figure 9: We compare trained DMTET oracle per- formance of MT and MC. Number in bracket indicates number of SDF points queried.
5 Conclusion
In this paper, introduced deep 3D conditional generative model that synthesize high- resolution 3D shapes using simple user guides such as coarse voxels. Our DMTET features a novel 3D representation that marries implicit and explicit representations by leveraging the advantages of both. We experimentally show that our approach synthesizes significantly higher quality shapes with better geometric details than existing methods, confirmed by quantitative metries and an extensive user study. By showcasing the ability to upscale coarse voxels such as Minecraft shapes, we hope that we take one step closer to democratizing 3D content creation.
6 Broad Impact
Many fields such as AR/VR, robotics, architecture, gaming and film rely on high-quality 3D content. Creating such content, however, requires human experts, i.c., experienced artists, and a significant amount of development time. contrast, platforms like Minecraft enable millions of users around the world to carve out coarse shapes with simple blocks. Our work aims at creating A.I. tools that would enable even novice users to upscale simple, low-resolution shapes into high resolution, beautiful 3D content. Our method currently focuses on 3D animal shapes. We are not currently aware of and do foresee nefarious use cases of our method.
7 Disclosure of Funding
This work was funded by NVIDIA. Tianchang Shen and Jun Gao acknowledge additional revenue in the form student scholarships from University of Toronto and the Vector Institute, which are not in direct support of this work.
References
[1] Matan Atzmon and Yaron Lipman. Sal: Sign agnostic learning of shapes from raw data, In Proceedings of the JEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2565 -274,, 2020.
[2] Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Generative and discriminative voxel modeling with convolutional neural networks. arXiv preprint 4rX1v:1608.04236, 2016.
10

---

[3] Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-riic 3d model repository. arXiv preprint 151.:0151..03012, 2015.
[4] Ding- Yun Chen, Xiao-Pot Peoo-eiiTT Tian, Yu-Te Te Shen, and Ming Ouhyoung. On visual similarity based 3d model retrieval. In Computer graphics forum, volume 22, pages 223-232. Wiley Online Library, 2003.
[5] Wenzheng Chen, Jun Gao, Huan Ling, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning to predict 3d objects with an interpolation-base -ased differentiable renderer. In Advances In Neural Information Processing Systems, 2019.
[6] Zhiqin Chen, Vladimir G. Kim, Matthew Fisher, Noam Aigerman. Hao Zhang, and Siddhartha Chaudhuri. Decor-gan: 3d shape detailization by conditional refinement. Proceedings of IEEE Conference on Computer Vision and Pottern Recognition (CVPR), 2021.
[7] Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. Bsp-net: Generating compact meshes via binary space partitioning. Proceedings of IEEE Conference on Computer Vixion and Pattern Recognition (CVPR), 2020.
[8] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[9] Julian Chibane, Aymen Mir, and Gerard Pons-Moll. Neural unsigned distance fields for implicit function learning. In Advances in Neural Information Processing Systems (NewrIPS), December 2020.
[10] Christopher B Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In European conference computer vision, pages 628-644. Springer, 2016.
[11] Angela Dai, Daniel Ritchie, Martin Bokeloh, Scott Reed, Jürgen Sturm, and Matthias NieBner. Scancom- plete: Large-scale scene completion and semantic segmentation for 3d scans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4578-4587, 2018.
[12] Angela Dai, Charles Ruizhongtai Qi, and Matthias NieBner. Shape completion using 3d-encoder-predictee cnns and shape synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1868-5877, 2017.
[13] Thomas Davies, Derek Nowrouzezahrai, and Alec Jacobson. Overfit neural networks as a compact shape representation. arXiv preprint 200.:000000098000 2020.
[14] Yu Deng, Jiaolong Yang, and Xin Tong. Deformed implicit field: Modeling 3d shapes with learned dense correspondence. In IEEE Computer Vision and Pattern Recognition, 2021.
[15] Akio Doi and Akio Koide. An efficient method of triangulating equi-valued surfaces by using tetrahedral cells. IEICE TRANSACTIONS on Information and Systems, 74(1) 74(1):214-224, 1991.
[16] Crawford Doran, Athena Chang, and Robert Bridson. Isosurface stuffing improved: acute lattices and feature matching. ACM SIGGRAPH 2013 Talks, pages 1-1, 2013.
[17] Yueqi Duan, Haidong Zhu, He Wang, Yi, Ram Nevatia. and Leonidas Guibas. Curriculum deepsdf. In European Conference on Computer Vision, pages 51-67. Springer, 2020.
[18] Jun Gao, Wenzheng Chen, Tommy Xiang, Clement Fuji Tsang, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Learning deformable tetrahedral meshes for 3d reconstruction. In Advances In Neural Information Processing Systems, 2020.
[19] Jun Gao, Chengeheng Tang, Vignesh Ganapathi-Subramaan Jiahui Huang, Hao Su, and Leonidas J Guibas. Deepspline: Data-driven reconstruction of parametric curves and surfaces. arXiv preprint arXiv:1901.03720 2019.
[20] Jun Gao, Zian Wang. Jinchen Xuan, and Sanja Fidler. Beyond fixed grid: Learning geometric image representation with deformable grid. ECCV, 2020.
[21] Lin Gao, Jie Yang. Tong Wu, Yu-Jie Yuan, Hongbo Fu, Yu-Kun Lai, and Hao Zhang. Sdm-net: Deep generative network for structured deformable mesh. ACM Transactions on Graphics (TOG), 38(6):1-15, 2019.
[22] Georgia Gkioxari, Jitendra Malik, and Justin Johnson. r-nn. In Proceedings of the IEEE International Conference on Computer Vision, pages 2019.
r-can.

---

[23] Thibault Groueix, Matthew Fisher. Vladimir G Kim. Bryan C Russell, and Mathieu Aubry. A papier-mäché approach learning 3d surface generation. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 216-224, 2018.
[24] Christian Häne, Shuhham Tulsiani, and Jitendra Malik. Hierarchical surface prediction for 3d object reconstruction. 2017 International Conference on 3D Vision (3DV), pages 412-420. IEEF, 2017.
[25] Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or Or. Point2mesh: A self celf-prior for deformable meshes. ACM Trans. Graph., (9)... 2020.
[26] Zekun Hao, Hadar Averbuch-Elor, Noah Snavely, and Serge Belongie. Dualsdf: Semantic shape manipula- tion using a two-level representation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7631-7641, 2020.
(27] Krishna Murthy J., Edward Smith, Jean-Francois Lafleche, Clement Fuji Tsang, Artem Rozantsev, Wen- zheng Chen, Tommy Xiang, Rev Lebaredian, and Sanja Fidler. Kaolin: A pytorch library for accelerating 3d deep learning research. arXiv: 1.05063, 2019.
[28] Marian Kleineberg, Matthias Fey, and Frank Weichert. Adversarial generation of continuous implicit shape representations. arXiv preprint (tti:200220004499 2020.
[29] Jiabao Lei and Kui Jia. Analytic marching: An analytic meshing solution from deep implicit surface networks. In International Conference on Machine Learning, pages 5789-5798. PMLR, 2020.
[30] Manyi Li and Hao Zhang. D2im-net: Learning detail disentangled implicit fields from single images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10246- 10255, 2021.
[31] Yiyi Liao, Simon Donné, and Andreas Geiger. Deep marching cubes: Learning explicit surface representa- tions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[32] Huan Ling, Jun Gao, Amlan Kar, Wenzheng Chen, and Sanja Fidler. Fast interactive object annotation with curve-gen. In CVPR, 2019.
[33] Hsueh-Ti Derek Liu, Vladimir G. Kim, Siddhartha Chaudhuri, Noam Aigerman, and Alec Jacobson. Neural subdivision. ACM Trans. Graph., 39(4), 2020.
[34] Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel enn for efficient 3d deep learning. In Advances in Neural Information Processing Systems, 2019.
[35] Charles Loop. Smooth subdivision surfaces based on triangles. January 1987.
[36] William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggruph computer graphics, 2144):163-169, 1987.
[37] Xudong Mao, Qing Li, Haoran Xic, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2794-2802, 2017.
[38] Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In 2015 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 922-928. IEEE, 2015.
[39] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowazin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4460-4470, 2019.
[40] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
[41] Charlie Nash, Yaroslav Ganin, S. M. Ali Eslami, and Peter W. Battaglia. Polygen: An autoregressive generative model of 3d meshes. ICML, 2020.
[42] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcomhe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2019.
[43] Despoina Paschalidou, Angelos Katharopoulos, Andreas Geiger, and Sanja Fidler. Neural parts: Learning expressive 3d shape abstractions with invertible neural networks. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
12

---

[44] Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In European Conference on Computer Vision (ECCV), 2020.
[45] Edoardo Remelli, Artem Lukoianov, Stephan Richter, Benoit Guillard, Timur Bagautdinov, Pierre Baque, and Pascal Fua. Meshsdf: Differentiable iso-surface extraction. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages Curran Associates, Inc., 2020.
22468-24 22478.
[46] Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representations at high resolutions. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3577-3586, 2017.
[47] Shunsuke Saito, Zeng Huang. Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution elothed human digitization. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
[48] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, June 2020.
[49] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Proc. NeurIPS, 2020.
[50] Minhyuk Sung, Hao Su, Vladimir G Kim, Siddhartha Chaudhuri, and Leonidas Guibas. Complementme: Weakly-supervised component suggestions for 3d modeling, ACM Transactions on Graphics (TOG), 36(6):1-12,2)) 2017.
[51] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[52] M. Tatarchenko, A. Dosovitskiy, and T. Brox. Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs. In IEEE International Conference on Computer Vision (ICCV), 2017.
[53] Shuhham Tulsiani, Hao Su, Leonidas J. Guihas, Alexei A. Efros, and Jitendra Malik. Learning shape abstractions by assembling volumetric primitives. In Computer Vision and Pattern Regognition (CVPR), 2017.
[54] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 52-67, 2018.
[55] Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Yu Sun, and Xin Tong. O-CNN: Octree-based Conva- lutional Neural Networks for 3D Shape Analysis. ACM Transactions on Gruphics (SIGGRAPH), 36(4). 2017.
[56] Peng-Shuai Wang, Yang Liu, and Xin Tong. Deep octree-based cnns with output-guided skip connections for 3d shape and scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 266 266,267,22 2020.
[57] Francis Williams, Matthew Trager, Joan Bruna, and Denis Zorin. Neural splines: Fitting 3d surfaces with infinitely-wide neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9949-9958, 2021.
[58] Jiajun Wu, Chengkai Zhang, Tianfan Xue, William T Freeman, and Joshua B Tenenbaum. Learning a probabilistic latent space of object shapes via 3d generative-ade adverssraal modeling. In Advances in Neural Information Processing Systems, pages 82 90, 2016.
[59] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015.
[60] Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and UIrich Neumann. Dism: Deep implicit surface network for high-quality single-view 3d reconstruction. In Advances in Neural Information Processing Systems, pages 490-500 500,000000000 2019.
13

---

[61] Kangxue Yin, Zhiqin Chen, Siddhartha Chaudhuri, Matthe Fisher, Vladimir Kim, and Hao Zhang. Coalesce: Component assembly by learning synthesize connections. In Proc. of 3DV, 2020.
[62] Chenyang Zhu, Kai Xu, Siddhartha Chaudhuri, Renjiao Yi, and Hao Zhang. SCORES: Shape composition with recursive substructure priors. ACM Transactions Graphics, 37(6))Artecle 211, 2018
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We provide extensive experiments in Sec.
(b) Did you describe the limitations of your work? [Yes] We provide the discussion on limitations failure cases in Supplement.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We provide the discussion in the Board Impact section with further discussions in Supplement.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as URL)? [No] The code currently quite uncleaned and requires many dependencies. We are planning release the code after cleaning.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We provide training details in both Sec. in the main paper and Supplement.
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [No] Training existing 3D models, including ours, on large-scale 3D datasets is too computation costly to repeat multiple times.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We provide in the Supplement.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We used ShapeNet [3] core dataset in Sec. 4.2. We also used official code to reproduce baselines with citations.
In particular, ConvONet [44] and DECOR-GAN -GN [6].
(b) Did you mention the license of the assets? [Yes] We provided the license of ShapeNet and Turbosquid.
(c) Did you include any new assets either in the supplemental material or URL? [No]
The Turbosquid data we are using contains proprietary information.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] We discussed in the Sec. 4 and provide further details in the Supplementary Materials.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] We provide discussion on this in Supplement.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [Yes] We provide details in the paper, with full text and screenshot in Supplement.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [No] We did not anticipate the potential participant risks, as we only conduct human studies on generated animals.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes] We provide details in Supplement
15


Reviews:
[{"review_id": "y2VAU-64mC", "reviewer": "Reviewer_f9sZ", "summary": "This paper introduces a shape refinement approach that first uses an implicit representation to obtain an initial shape. The initial shape is then converted into a mesh representation refined by loop subdivision. ", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper basically combines a couple of hand-crafted designs that are not clearly motivated. Although the experimental results are ok (not outstanding, see my comments below), the underlying principles from the machine learning and statistical aspects are rather unclear. Below are the detailed comments:\n\nFirst of all, a benefit of the implicit representation is that it is known to capture shape details. This paper uses it as initialization. The subdivision scheme produces smooth outputs that are not feature preserving under the loop subdivision scheme. \n\nIt would be good to compare the proposed approach that state-of-the-art neural implicit representations for shape synthesis. For example, the comparisons to the following paper is necessary.\nImplicit Neural Representations with Periodic Activation Functions. NeurIPS 2020. \n\nThe notion of hybrid is different from many other works that combine predictions under different representations. In this case, the errors made under one representation can be rectified under other representations or different representations help learn better features under a multi-task learning framework. Such approaches are grounded by bias and variance of optimizing over-parameterized networks, where ensembles can help reduce the variance:\nH3dnet: 3D Object Detection using Hybrid Geometric Primitives. ECCV 2020.\nCoupling Explicit and Implicit Surface Representations for Generative 3D Modeling. ECCV 2020.\nIn contrast, the proposed approach is sequential, meaning that the topology mistakes of the implicit representation cannot be rectified in the mesh representation. On the other hand, the mesh representation cannot guarantee self-intersection free during subdivision, which is an advantage of the implicit representation.\n\nThe input is an initial grid, if is rather unclear what is the definition of an initial grid? What is the suggested resolution in practice. How sensitive is the output why perturbing the initial grid? For example, when an user specifies the initial grid, which consists of many cells in my opinion, it is likely to be imprecise and inexact compare to the coarse grid of the underlying ground-truth. How robust is the approach when perturbing the input?\n\n\nThere are many inconsistencies between the main text and the supplemental material. For example, the main text states \"which controls the generated surface via weighting the smoothness of neighbouring vertices\". The supplemental material states that this is done using a graph neural network. It is unclear what is actually implemented? If the graph neural network is a major contribution, then the details should be in the main text. \n\nIn summary, the paper takes the approach of first initializing a coarse grid using an implicit representation which is followed by learnable mesh subdivision. This approach is limited in the sense that a coarse implicit representation does not capture the right topology of the underlying model, which cannot be recovered using mesh subdivision. The paper does not compare against state-of-the-art implicit based representations (performed at high resolutions). It is unclear if similar results can not be achieved by these techniques. The paper is rushed, there are many inconsistencies between the main text and the supp. material. It is recommended to revise the paper and submit to future venues in vision and graphics.\n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA.", "review_text": "", "extra_scores": {"ethical_concerns": "NA.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "fBg_UEqyoN_", "reviewer": "Reviewer_hY1b", "summary": "This paper proposes a new deep 3D shape representation that combines deformable tetrahedral grid, a learned SDF, differentiable mesh extraction, and learned subdivision. The model inputs a point cloud or coarse voxel grid and then interpolates SDF values learned on vertices of a tetrahedral grid (whose deformation is also learned), extracts mesh via deep marching tetrahedra, and applies learned a subdivision rule to obtain an output mesh. The network is trained by optimizing a loss evaluated on ground truth meshes as well as some regularization.\n\nThe authors provide comprehensive evaluation on the tasks of 3D reconstruction animals from coarse voxels as well as reconstruction from point clouds. They also include an ablation study.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a well-written paper that introduces a reasonable hybrid deep 3D shape representation. Representing the SDF on a deformable tet grid is a clever way to efficiently enable a coarse to fine representation, and ultimately extracting a mesh allows the model to be trained on mesh data with less artifacts from sampling or discretization. I also think that most of the experiments are quite comprehensive and show improvement over past work.\n\nAlthough the proposed approach largely consists of combinations and perturbations of several existing methods, I do think that there is sufficient novelty. In particular, using the tet grid to represent SDFs and optimizing gradient flow based on relevant regions are reasonable choices that seem to help the overall method.\n\nThere are a few missing citations that should be discussed and do somewhat limit the proposed method's novelty. In particular, \"Neural Subdivision\" [Liu et al., SIGGRAPH 2020] is relevant in the context of the learned mesh subdivision step. \"Neural Unsigned Distance Fields for Implicit Function Learning\" [Chibane et al., NeurIPS 2020] and \"SAL\" [Atzmon et al., CVPR 2020] learn implicit fields using explicit geometry as input, somewhat undermining the claim that implicit methods require strict supervision in the form of occupancies or distances. Also, \"Analytic Marching\" [Lei et al., ICML 2020] is relevant in the context of differentiable mesh extraction.\n\nI also have a question about Figure 6, which shows results on out-of-distribution voxels. The results are impressive, but only eight inputs are shown. How were these chosen? Are they representative, or do other specific inputs reveal failure cases?\n\nMinor issues:\n\nL44, 120: sign distance -> signed distance\n\nL75,85,148,180: typology -> topology\n\nL148: in practise -> in practice", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Sufficiently discussed in Sections 6 and in Supplementary Material.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "QMPwWWtYuG6", "reviewer": "Reviewer_Axy8", "summary": "The paper presents a method for generating high-resolution 3D shapes guided by coarse voxel-based input. The key idea is to combine tetrahedral meshes which are adaptive and more efficient, with sign-distance field, along with a differentiable marching tetrahedra layer that bridges the gap between tetrahedra and surface meshes. The method achieves improved performance compared with existing methods. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents a method for generating high-resolution 3D shapes guided by coarse voxel-based input. The key idea is to combine tetrahedral meshes which are adaptive and more efficient, with sign-distance field, along with a differentiable marching tetrahedra layer that bridges the gap between tetrahedra and surface meshes. Overall, the method is plausible and shows decent performance. I have the following concerns:\n\nFrom a technical perspective, the method is built on DefTet [16], by introducing SDF and Marching Tetrahedra. All these components are from existing publications, which somehow affects the novelty of the method. Nevertheless, the approach to combine these leads to improved performance and is itself interesting.\n\nAlthough the paper claims that the method can handle topological changes, I can't seem to find examples that clearly demonstrate this. Moreover, a large part of the assessment is based on a dataset collected by authors, which makes comparisons more challenging. Evaluation of guided synthesis on more public datasets would make it more convincing. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "9y9RJWE2bRn", "reviewer": "Reviewer_YB43", "summary": "DMTet aims to synthesize a plausible and detailed mesh from a coarse voxelized input (e.g. given by a user). It does so by regressing a low-resolution tetrahedral grid that is then refined and from which a mesh can be extracted via Marching Tetrahedras in a differentiable manner, which is the main contribution of this work. Using prior work, this mesh can be further subdivided in a differentiable manner before being used in the loss. The method is evaluated extensively on the TurboSquid dataset, including user studies. The experiments show significant improvements quantitatively and qualitatively over baselines.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method is novel and the results are impressive. The main contribution, differentiable Marching Tetrahedras, as an alternative to Deep Marching Cubes, seems like it could be useful in other contexts as well. The evaluation is thorough and convincing. I believe it exceeds the threshold for acceptance in its current form already, although I have some suggestions for minor changes.\n\nThe exact contributinos are not clear to me. Which exact parts of Sec. 3.1 are novel? What are the changes made to [16]? Switching to SDF, adding volume subdivision (and making it differentiable), and using Marching Tetrahedra for surface extraction (and making it differentiable)? These should be clearly mentioned in the introduction.\n\nThe comment regarding prior work like Deep Marching Cubes in lines 147-151 is interesting. What is missing for full differentiability though are gradients into vertices (and associated values like position and features) outside the current tetrahedra, but I do not believe that this is a problematic limitation.\n\nCurrent training time is a bit over half a week on 4 V100s. Is the implementation in any way optimized, e.g. are custom layers parallelized in CUDA? What speedups seem reasonably achievable with relatively little work? Which component is currently the major bottleneck?\n\nThe method is evaluated on only one dataset, which does contain fairly diverse animal shapes though.\n\nThe writing is fine but could use a final polishing touch.\n\nI have several minor clarification questions that should be addressed in a revised version:\n\n- What does the f(v_i) output of GCN in Eq. 2 mean? Does it overwrite the previous features f(v_i) from Eq. 1?\n- How often is the Surface Refinement with Volume Subdivision (Eq.s 1 and 2) applied per forward pass? Only once or multiple times?\n- Does the GCN for Learnable Surface Subdivision also take in the features f(v_i) from the previous step? If so, how are those features transferred from the tetrahedral structure to the extracted mesh? If not, why does Eq. 2 output f(v_i)?\n- I assume the GCN uses Eq. 1-4 from the \"GCN Model\" paragraph in [10]? If so, it would be good for completeness if those equations are included in the supplement.\n- According to lines 67-68 in the supplement, the subdivision step is applied \"iteratively\". Does that include the GCN that regresses v'_i and alpha_i, or only a repeated application of Figure B, where the Vs and alphas are regressed only once in the beginning and then the alphas are interpolated for all non-initial iterations? How many iterations are used?\n- What effect does varying alpha have? Just very simply, e.g. higher alpha leads to smoother surfaces?\n- What exactly is the small noise in line 74 in the supplement added to? The position of v? Before or after extracting the local SDF grid? And is the noisy position of v used when computing the SDF values (i.e. is the mesh actually slightly deformed) or does the noise only move the local SDF grid slightly around?\n- How is the SDF function in Eq.s 4 and 5 in the supplement computed? Determine nearest face and take point-plane distance or point-triangle? Or something else?\n- The sequence of surface refinement and subdivision steps that is mentioned in C.2 should be mentioned in the main paper already. \n- Line 116 and 118 have a typo \"et.al.\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and future work are very well explained. I agree with line 377 and do not see potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received mostly positive reviews, with the exception of reviewer f9sZ.  Taking into account feedback from the other reviewers as well as the message to the area chair, the AC tends to agree that f9sZ's concerns are not sufficient to hold back publishing this work.

The final revision of this paper should clarify issues of contribution/novelty brought up in several reviews and incorporate the additional experiments/fixes mentioned in the rebuttal posts.
Decision: Accept (Poster)