Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis
Authors: Tianchang Shen
Jun Gao
Kangxue Yin
Ming-Yu Liu
Sanja Fidler
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3D Deep Learning
3D Super-Resolution
3D Content Creation
3D Shape Representation

Abstract: We introduce DMTet, a deep 3D conditional generative model that can synthesize high-resolution 3D shapes using simple user guides such as coarse voxels. It marries the merits of implicit and explicit 3D representations by leveraging a novel hybrid 3D representation. Compared to the current implicit approaches, which are trained to regress the signed distance values, DMTet directly optimizes for the reconstructed surface, which enables us to synthesize finer geometric details with fewer artifacts. Unlike deep 3D generative models that directly generate explicit representations such as meshes, our model can synthesize shapes with arbitrary topology. The core of DMTet includes a deformable tetrahedral grid that encodes a discretized signed distance function and a differentiable marching tetrahedra layer that converts the implicit signed distance representation to the explicit surface mesh representation. This combination allows joint optimization of the surface geometry and topology as well as generation of the hierarchy of subdivisions using reconstruction and adversarial losses defined explicitly on the surface mesh. Our approach significantly outperforms existing work on conditional shape synthesis from coarse voxel inputs, trained on a dataset of complex 3D animal shapes. Project page: https://nv-tlabs.github.io/DMTet/.

Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis
Tianchang Shen 1,2,3
Jun Gao 1,2,3
Kangxue Yin
Ming- Yu Liu
Sanja Fid
NVIDIA
University of Toronto
Vector Institute"
{frshon, jung, kangxoey, ningyul, sfidler)dnvidia.com
Abstract
We introduce DMTET, deep 3D conditional generative model that can synthesize high-resolution 3D shapes using simple user guides such as coarse voxels. It marries the merits of implicit and explicit 3D representations by leveraging novel hybrid 3D representation. Compared to the current implicit approaches, which are trained regress the signed distance values, DMTET directly optimizes for the reconstructed surface, which enables us to synthesize finer geometric details with fewer artifacts. Unlike deep 3D generative models that directly generate explicit representations such as meshes, our model can synthesize shapes with arbitrary topology. The core of DMTET includes a deformable tetrahedral grid that encodes a discretized signed distance function and differentiable marching tetrahedra layer that converts the implicit signed distance representation to the explicit surface mesh representation. This combination allows joint optimization of the surface geometry and topology as well as generation of the hierarchy of subdivisions using reconstruction and adversarial losses defined explicitly on the surface mesh. Our approach significantly outperforms existing work on conditional shape synthesis from coarse voxel inputs, trained dataset of complex 3D animal shapes. Project page: https: //tv///tv/tt/e// github. io/DMTet/.
Introduction
Fields such as simulation, architecture, gaming, and film rely on high-quality 3D content with rich geometric details and complex topology. However, creating such content requires tremendous expert human effort. It takes a significant amount of development time to create each individual 3D asset. In contrast, creating rough shapes with simple building blocks like voxels has been widely adopted. For example, Minecraft has been used by hundreds of millions of users for creating 3D content. Most of them are non-experts. Developing A.I. tools that enable regular people upscale coarse, voxelized objects into high resolution, beautiful 3D shapes would bring one step closer to democratizing high-quality 3D content creation. Similar tools can be envisioned for turning 3D scans of objects recorded by modern phones into high-quality forms. Our work aspires to create such capabilities.
powerful 3D representation is critical component of a learning-bate based 3D content creation framework. A good 3D representation for high-quality reconstruction and synthesis should capture local geometric details and represent objects with arbitrary topology while also being memory and computationally efficient for fast inference in interactive applications.
Recently, neural implicit representations [8, 39, 42, 51], which represent a shape via a signed distance field (SDF) or an occupancy field (OF), have emerged as an effective 3D representation. Neural implicits have the benefit of representing complex geometry
neural network implicitly
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

and topology, not limited to a predefined resolution. The success of these methods has been shown in shape compression [49, 13, 51], single-image shape generation [47, 60, 48], and point cloud reconstruction [57]. However, most of the current implicit approaches are trained by regressing to SDF or OF values and cannot utilize an explicit supervision on the target surface, which imposes useful constraints for training. To mitigate this issue, several works [45, 31] proposed to utilize iso-surfacing techniques such as the Marching Cubes (MC) algorithm to extract a surface mesh from the implicit representation, which, however, is computationally expensive.
In this work, we introduce DMTET, a deep 3D conditional generative model for regouutsoot 3D shape synthesis from user guides in the form of coarse voxels. In the heart of DMTET is new differentiable shape representation that marries implicit and explicit 3D representations. In contrast to deep implicit approaches optimized for predicting sign distance (or occupancy) values, our model employs additional supervision on the surface, which empirically renders higher quality shapes with finer geometric details. Compared to methods that learn to directly generate explicit representations, such as meshes [54], by committing to a preset topology, our DMTET can produce shapes with arbitrary topology. Specifically, DMTET predicts the underlying surface parameterized by implicit function encoded via a deformable tetrahedral grid. The underlying surface is converted into an explicit mesh with a Marching Tetrahedra (MT) algorithm, which we show is differentiable and more performant than the Marching Cubes. DMTET maintains efficiency by learning to adapt the grid resolution by deforming and selectively subdividing tetrahedra. This has the effect of spending computation only on the relevant regions in space. We achieve further gains in the overall quality of the output shape with learned surface subdivision. Our DMTET is end-to-end differentiable, allowing the network to jointly optimize the geometry and topology of the surface, as well as the hierarchy of subdivisions using loss function defined explicitly on the surface mesh.
We demonstrate our DMTET on two challenging tasks: 3D shape synthesis from coarse voxel inputs and point cloud 3D reconstruction. We outperform existing state-of-the-art methods by significant margin while being 10 times faster than alternative implicit representation-basee methods at inference time. In summary, we make the following technical contributions:
1. We show that using Marching Tetrahedra (MT) as a differentiable surfacirgggg layer allows topological change for the underlying shape represented by implicit field, in contrast to the analysis in prior works (31,45].
2. We incorporate MT in DL framework and introduce DMTET, hybrid representation that combines implicit and explicit surface representations. We demonstrate that the additional supervision (e.g. chamfer distance, adversarial loss) defined directly on the extracted surface from implicit field improves the shape synthesis quality.
3. We introduce a coarse-to- to-sse--t-fieee fine optimization strategy that scales DMTET to high resolution during training. We thus achieves better reconstruction quality than state-of-the the-art methods on challenging 3D shape synthesis tasks, while requiring a lower computation cost.
2 Related Work
We review the related work on learning-basedd 3D synthesis methods based on their 3D representations.
Voxel-based Methods Early work [59, 10, 38] represented 3D shapes as voxels, which store the coarse occupancy (inside/outside) values regular grid, which makes powerful convolutional neural networks native and renders impressive results on 3D reconstruction and synthesis [12, 11, 58, 2]. For high-resolution shape synthesis. DECOR-GAN [6] transfers geometric details from high-resolution shape represented in voxel to a low-resolution shape by utilizing a discriminator defined on 3D patches of the voxel grid. However, the computational and memory costs grow cubically as the resolution increases, prohibiting the reconstruction of fine geometric details and smooth curves. One common way to address this limitation is building hierarchical structures such as octrees [46, 52, 55, 56, 24, 52], which adapt the grid resolution locally based on the underlying shape. In this paper, we adopt hierarchical deformable tetrahedral grid to utilize the resolution better. Unlike octree-based shape synthesis, our network learns grid deformation and subdivision jointly to better represent the surface without relying on explicit supervision from a pre-computed hierarchy.
Deep Implicit Fields (DIFs) represent 3D shape as zero level set of continuous function parameterized by neural network [39, 44, 17, 40]. This formulation can represent arbitrary typol- ogy and has infinite resolution. DiFebased shape synthesis approaches have demonstrated strong

---

input
GT Shape
oUnoccucled Grdl Pokn!
Occupled
Surfaze
Loss our forfface face
output
Implicif Funciion (5DF)
Explicit Surface
(c) Refrie Boundary SDF: Deform and trm Groph
Eenohedd Crreecn
(g) Wording
toj Surfoce Suboiiiion sinn
Vaua
(0) Precicting 3DF af intool
Grrd Resolunon
Tetrchedrons.cm and inte polate SD3
(b) Selectivel oly Subdii ice
Inferpplatec SOF)
(M Mesh)
CaromehicSurfoc
Figure I: DMTET reconstructs the shape implicitly in coarse-to-fipg manner by predicting the SDF defined
a deformable tetrahedral grid. then converts the SDF to surface mesh by differentiable Marching Tetrahedra layer. DMTET is trained by optimizing the objective function defined on the final surface.
performance in many applications, including single view 3D reconstruction [60, 30, 47, 48], shape manipulation, and synthesis [26, 21, 28, 14, 1, 1.. However, as these approaches are trained by minimizing the reconstruction loss of function values at a set of sampled 3D locations (a rough proxy of the surface), they tend to render artifacts when synthesizing fine details. Furthermore, if one desires a mesh to be extracted from a DIF, an expensive iso-surfacing step based on Marching Cubes [36] Marching Tetrahedra [15] is required. Due the computational burden, iso-surfacing is often done on a smaller resolution, hence prone to quantization errors. Lei et al. [29] proposes an analytic meshing solution reduce the error, but is only applicable DIFs parametrized by MLPs with ReLU activation. Our representation scales to high resolution and does not require additional modification the backward pass for training end-to-end. DMTET can represent arbitrary typology. and is trained via direct supervision on the generated surface. Recent works [1, 9] learn to regress unsigned distance to triangle soup or point cloud. However, their iso-surfacing formulation is not differentiable in contrast to DMTET.
Surface-based Methods directly predict triangular meshes and have achieved impressive results for reconstructing and synthesizing simpler shapes [54, 23, 5, 41. 7]. Typically, they predefined the topology of the shape. e.g. equivalent to a sphere [54, 5, 25], or union of primitives [43, 53, 19] or a set of segmented parts [61, 62, 50J. As result, they can not model a distribution of shapes with complex topology variations. Recently, DefTet [18] represents a mesh with deformable tetrahedral grid where the grid vertex coordinates and the occupancy values are learned. However, similar to voxel-based methods, the computational costf increases cubically with the grid resolution. Furthermore, the occupancy loss for supervising topology learning and the surface loss for supervising geometry learning do not support joint training, it tends to generate suboptimal results. In contrast, our method is able to synthesize high-resolution 3D shapes, not shown in previous work.
3 Deep Marching Tetrahedra
We now introduce our DMTET for synthesizing high-quality 3D objects. The schematic illustration is provided in Fig. 1. Our model relies on a new, hybrid 3D representation specifically designed for high-resolutic reconstruction and synthesis, which describe in Sec. 3.1. In Sec. 3.2, we describe the neural network architecture of DMTET that predicts the shape representation from inputs such as coarse voxels. We provide the training objectives in Sec. 3.3.
3.1 3D Representation
We represent shape using sign distance field (SDF) encoded with a deformable tetrahedral grid, adopted from DefTet [18, 20]. The grid fully tetrahedralizes a unit cube, where each cell in the volume tetahedron with 4 vertices and faces. The key aspect of this representation is that the grid vertices can deform to represent the geometry of the shape more efficiently. While the original DefTet encoded occupancy defined on each tetrahedron, we here encode signed distance values defined on the vertices of the grid and represent the underlying surface implicitly (Sec. 3.1.1). The use of signed distance values, instead of occupancy values, provides more flexibility representing the underlying surface. For greater representation power while keeping memory and computation manageable, we further selectively subdivide the tetrahedra around the predicted surface (Sec. 3.2.2). We convert the signed distance- based implicit representation into triangular mesh using marching tetrahedra layer, which we discuss in See. 3.3. The final mesh is further converted into a parameterized surface with differentiable surface subdivision module, described in Sec. 3.1.4.
3

---

3.1.1 Deformable Tetrahedral Mesh as an Approximation of an Implicit Function
We adopt and extend the deformable tetrahedral grid introduced in Gao et al. [18], which we denote with (Vr,T), where Vr are the vertices in the tetrahedral grid T. Following the notation in [18], each tetrahedron T. E ' is represented with four vertices (Vay Phys Veyo Vd, J. with l E (I,...., ........KK K}. where K is the total number of tetrahedra and Vis VT-
We represent the sign distance field by interpolating SDF values defined on the vertices of the grid. Specifically, we denote the SDF value in vertex v E Vr as s(U)). SDF values for the points that lic inside the tetrahedron follow barycentric interpolation of the SDF values of the four vertices that encapsulates the point.
Va
41% +1)e) 6/c
3.1.2 Volume Subdivision
[((( +a()))
We represent shape in a coarse to fine manner for efficiency. We determine the surface tetrahedra S by checking whether tetrahedron has vertices with different SDF signs - indicating that it intersects the surface encoded by the SDF. We subdivide T sur/ as well as their immediate neighbors and increase resolution by adding the mid point to each edge. We compute SDF values of the new vertices by averaging the SDF values on the edge (Fig. 2).
Vb
Uc
Figure 2: Volume Subdivision:
Each surface tet.blue) is divided into 8 (red) by adding midpoints.
3.1.3 Marching Tetrahedra for converting between an Implicit and Explicit Representation
We use the Marching Tetrahedra [15] algo- rithm to convert the encoded SDF into an ex- plicit triangular mesh. Given the SDF values {s(wa), s(b,), s(ve), s(va)} of the vertices of a tetra- hedron, MT determines the surface typology inside the tetrahedron based on the signs of s(v). which is illustrated in Fig. 3. The total number of configu- rations is 2 = 16, which falls into 3 unique cases after considering rotation symmetry. Once the sur- face typology inside the tetrahedron is identified, the vertex location of the iso-surface is computed the zero crossings of the linear interpolation along the tetrahedron's edges, shown in Fig. 3.
Ub
pasilive SDF
V.b V. s(v)) - 10 (v
negorive SDF
x(pr)) - (Pa))
Figure 3: Threc unique surface configurations in MT. Vertex color indicates the sign of signed dis- tance value. Notice that flipping the signs of all vertices will result in the same surface configura- tion. Position of the vertex is linearly interpolated along the edges with sign change.
Prior works [45, 31] argue that the singularity in this formulation, i.c. when s(va) = s(vs), prevents the change of surface typology (sign change of s(va)) during training. However, we find that, in practise, the equation is only evaluated when sign(s(va)) * sign(s(ve)). Thus, during training, the singularity never happens and the gradient from loss defined on the extracted iso- (Sec. 3.3), can be back- proppaaae to both vertex positions and SDF values via the chain rule. A more detailed analysis is in the Appendix.
3.1.4 Surface Subdivision
Having surface mesh as output allows us to further increase the representation power and the visual quality the shapes with a differentiable surface subdivision module. We follow the scheme the Loop Subdivision method [35], but instead of using fixed set of parameters for subdivision, we make these parameters learnable DMTET. Specifically, learnable parameters include the positions of each mesh vertex v/. well as a which controls the generated surface via weighting the smoothness of neighbouring vertices. Note that different from Liu et al. [33], we only predict the per-vertex parameter at the beginning and carry it over to subsequent subdivision iterations to attain lower computational cost. We provide more details in Appendix.
3.2 DMTET: 3D Deep Conditional Generative Model
Our DMTET a neural network that utilizes our proposed 3D representation and aims to output a high resolution 3D mesh M from input x (a point cloud or a coarse voxelized shape). We describe the architecture (Fig. 4) of the generator for each module of our 3D representation See. 3.2.1, with the architecture of the discriminator presented in Sec. 3.2.2. Further details are in Appendix.

---

Initial SDF Prediction
Surface Retinement
Discriminator
Trilnnaor
(v).) x))
Sample high-curvature fromm GT
3D CNN
MLP
interp (v. P. (v.,x)
Real/fake
MLPS
GCN
Regular Tel. Grld
Sprce M/G7)
X Gradient
s(t).f())
[App,As(p))
Interp.
PCD
Encoder
Pred or GT Mesh
Build Graph from Surface Tetrahedrons(Dramge
Figure 4: Our generator and discriminator architectures. The generator is composed of two parts- one utilizes MLP to generate the initial predictions for all grid vertices and the other uses GCN to refine the surface.
3.2.1 3D Generator
Input Encoder We use PVCNN [34] from a point cloud. When the imput is a coarse voxelized shape, we sample points on its surface. We compute a feature vector Foo z) for grid vertex E I2 via trilinear interpolation.
imput encoder to extract a 3D feature volume Food (x)
Initial Prediction of SDF We predict the SDF value for each vertex in the initial deformable tetrahedral grid using fully-connected network s(v) MLP (B, x)), v). The fully-connected network additionally outputs feature vector ((v), which is used for the surface refinement in the volume subdivision stage.
Surface Refinement with Volume Subdivision After obtaining the imitial SDF, we iteratively refine the surface and subdivide the tetrahedral grid. We first identify surface tetrahedra T sur based on the current s(e) value. We then build a graph G = (Vsurf: Esurf). where Vsurf: Esurf correspond to the vertices and edges in surf.. We then predict the position offsets Au; and SDF
residual values As(e;) for each vertex i in Veurs using a Graph Convolutional Network [32] (GCN):
fe. concat(u,, (s, s(v,),
(1)
(Av, As(rs) ((:)))-..
(CN((F))t=1.--))
G).
(2)
where N is the total number of vertices in V Vaury and f(,; is the updated per-vertex feature. The vertex position and the SDF value for vertex are updated as of = vn + Au and s(v() s(v;) + (s()). This refinement step can potentially flip the sign of the SDF values refine the local typology, and also move the vertices thus improving the local geometry.
After surface refinement, we perform the volume subdivision step followed by an additional surface refinement step. In particular, we re-identify Teurf and subdivide Teury and their immediate neigh- bors. We drop the unsubdivided tetrahedra from the full tetrahedral grid in both steps, which saves memory and computation, as the size of the Teurs is proportional to the surface area of the object, and scales up quadratically rather than cubically the grid resolution increases.
Note that the SDF values and positions of the vertices are inherited from the level before subdivision, thus, the loss computed at the final surface can back prok-propagate to all vertices from all levels. Therefore, our DMTET automatically learns to subdivide the tetrahedra and does not need an additional loss term in the intermediate steps to supervise the learning of the octree hierarchy as in the prior work [52].
Learnable Surface Subdivision After extracting the surface mesh using MT, we can further apply learnalle surface subdivision. Specifically, we build new graph on the extracted mesh, and use GCN to predict the updated position of each vertex v/, and a, for Loop Subvidision. This step removes the quantization errors and mitigates the approximation errors from the classic Loop Subdivision by adjusting Os,, which are fixed in the classic method.
3.2.2 3D Discriminator
We apply a 3D discriminator D on the final surface predicted from the generator. empirically find that using a 3D CNN from DECOR -AAN [6] as the discriminator on the signed distance field that computed from the predicted mesh is effective capture the local details. Specifically, we first randomly select high-curvature vertex from the target mesh and compute the ground truth signed distance field Sreal E NNNN at a voxelized region around e. Similarly, we compute the signed distance field of the predicted surface mesh M at the same location to obtain Spred N X X Note that Spred is an analytical function of the mesh M, and thus the gradient to Spred can back- propagate the vertex positions in M. We feed Sreat or Spred into the discriminator. along with the feature vector Foor(1) x) in position v. The discriminator then predicts the probability indicating whether the input comes from the real generated shapes.
5

---

3.3 Loss Function
DMTET is end-to-end trainable. We supervise all modules to minimize the error defined on the final predicted mesh M. Our loss function contains three different terms: surface alignment loss to encourage the alignment with ground truth surface, an adversarial loss to improve realism of the generated shape, and regularizations to regularize the behavior of SDF and vertex deformations.
Surface Alignment loss We sample a set of points Pgt from the surface of the ground truth mesh Mat- Similarly, we also sample set of points from Mpred to obtain Ppreds and minimize the L2 Chamfer Distance and the normal consistency loss between Pyt and predd"
Lidd
Σ
min lpp - all2 + qEPs
Σ min llq 10-p0 pll2, Tenormal
Σ (1- ngg);
(3)
PEP
PE PEPPred
gEPae
pe P.Peeed
where ② is the point that corresponds to when computing the Chamfer Distance, and np, ng denotes the normal direction point p, 4-
Adversarial Loss
We use the adversarial loss proposed in LSGAN [37]:
Lo 2 (D(M 1)² D(Mpred))), LG = 2 (D( 1)
(4)
Regularizations The above loss functions operate on the extracted surface, thus, only the vertices that are close to the iso-surface in the tetrahedral grid receive gradients, while the other vertices do not. Moreover, the surface losses do not provide information about what is inside/outside, since flipping the SDF sign of all vertices in tetrahedron would result in the same surface being extracted by MT. This may lead to disconnected components during training. To alleviate this issuc, we add a SDF loss to regularize SDF values:
LSDF
Σ WEVV
(5)
where SDF Mgt) denotes the SDF value of point U; to the mesh Mgt. In addition, we apply the L2 regularization loss the predicted vertex deformations avoid artifacts: Tatet = Ev,ev, Au;l12
The final loss a weighted sum of all five loss terms:
Aea + Anvemal Trnommal + >aLa + ASD + A
(6)
where Ama, Anvemal, Aa, ASDE, Ader are hyperparameters (provided in the Supplement).
Experiments
We first evaluate DMTET in the challenging application of generating high high-quality animal shapes from coarse voxels. We further evaluate DMTET in reconstructing 3D shapes from noisy point clouds on ShapeNet by comparing to existing tfate-of-the-ar methods.
4.1 3D Shape Synthesis from Coarse Voxels
Experimental Settings We collected 1562 animal models from the TurboSquid website These models have wide range of diversity, ranging from cats, dogs, bears, giraffes, to rhinoceros, goats, etc. We provide visualizations in Supplement. Among 1562 shapes, we randomly select 1 1120 shapes for training, and the remaining 442 shapes for testing. We follow the pipeline in Kaolin [27] to convert shapes to watertight meshes. To prepare the input to the network, we first voxelize the mesh into the resolution of 163. and then sample 3000 points from the surface after applying marching cubes to the 16 voxel grid. Note that this preprocessing is agnostic to the representation of the input coarse shape, allowing us evaluate on different resolution voxels, or even meshes.
We compare our model with the official implementation of ConvOnet [44], which achieved SOTA performance voxel upsampling. We also compare DECOR-GAN [6], which obtained impressive results on transferring styles from a high-resolution voxel shape to a low-resolution voxel. Note that the original setting of DECOR-GAN is different from ours. For fair comparison, we use all 1120 training shapes as the high-resolution style shapes during training, and retrieve the closet training shape the test shape as the style shape during inference, which we refer DECOR-Retv. We also compare against a randomly selected style shape as reference, denoted as RECOR-RRadd
com. 'https://www.. www.wwwwww....d..commm com, we obtain consent via an agreement with TurboSquid, and following license at https://blog. https://log..urrboquud... turbosquiddddoooo http://ssuuu..ooo/uuuooooooomoe conn/utbosquiddioo ad-model-licenes

---

Imput
ConyOttt
DECOR Rand
DBCOR-Rat:
Ours Ad....
OLtS
Retv.
CT
Figure 5: Qualitative results on 3D shapes Synthesis from Coarse Voxels. Comparing with all baselines, method reconstructs shapes with much higher quality. Adding GAN further improves the realism of the generated shape. We also show the retrieved shapes from the training set in the sccond last column.
Metrics We evaluate L2 and LI Chamfer Distance, well normal consistency score to assess how well the methods reconstruct the corresponding high-resolution shape following [44]. We also report Light Field Distance [4] (LFD) which measures the visual similarity in 2D rendered views. In addition, we evaluate Cls score following [6]. Specifically, we render the predicted 3D shapes and train
image classifier distinguish whether images from the renderings of real or generated shapes. The mean classification accuracy the trained classifier is reported as Cls (lower better). More details in the Supplement.
Experimental Results We provide quantitative results in Table with qualitative examples Fig. 5. Our DMTET achieves significant improve- ments over all baselines in terms of all metrics. Compared both ConvOnet [44] and DECOR- GAN [6]. our DMTET reconstructs shapes with better quality when training without adversarial loss (5th column in Fig. 5). Further geometric de- tails, including nails, ears, eyes, mouths, etc, are captured when trained with the ad adversarial loss (6th column Fig. 5), significantly improving the realism and visual quality of the generated shape. To demonstrate the generalization ability of our DMTET, we collect cuman--reeted low-resolution ing). We provide qualitative results in Fig. 6. Despite the fact that these human-created shapes have noticeable differences with our coarse voxels used in training, e.g., different ratios of body parts compared with our training shapes (larger head, thinner legs, longer necks), our model faithfully generates high-quality 3D details conditioned each coarse voxel exciting result.
Figure 6: Qualitative Results of synthesizing high- resolution shapes from coarse voxels collected online.
voxels from Turbosquid (shapes unseen in train-
7

---

L2 Chamfer v
Li Chamfer +
Norm. Cons.
LFD
Cls
ConvOner [4
0.83
2.41
0.901
3220
0.63
DECOR Ret-.
1.32
3.81
0.876
3689
0.66
DECOR -andd
2.38
6.85
0.797
5338
0.67
DMTET wo Adv.
0.76
2.20
0.916
2846
0.58
DMTET Table 1: Super Resolution of Animal Shapes: DMTET significantly outperforms all baselines in all metrics.
0.75
2.19
0.918
2823
0.54
User Studies We conduct user studies via Amazon Machanical Turk (AMT) to further evaluate the performance of all methods. In particular, we present two shapes that are predicted from two different models to the AMT workers and ask them to evaluate which one is better looking shape and which one features more realistic details. Detailed experimental settings are provided in the Supplement. We compare DMTET against ConvONet [44], DECOR [6]-Retv, as well as DMTET without adversarial loss (w.o. Adv.). Quantitative results are reported in Table 2. Human judges agree that the shapes generated from our model have better details, compared to all baselines, in vast majority of the cases, Ablations on using adversarial loss demonstrate the effectiveness of generating higher quality geometry using a discriminator during training.
Ablation Studies To evaluate the effective- ness of our volume subdivision and surface sub- division modules, we ablate by sequentially in- troducing them to the base model (we refer as DMTET which we train on 100-resolution uniform tetrahedral grid without both volume and surface subdivision modules and adversarial loss. We conduct user studies to evaluate the improvement after each step using the protocol described in the above paragraph. We first reduce the initial resolution to 70 and employ volume subdivision to support higher output resolution (we refer this model as DMTETy) and compare with DMTET Predictions by DMTETy wins 78% of cases over DMTET B for better looking, and 61% of cases for realistic details, showing that the volume subdivision module is effective in synthesizing shape details. We then add surface subdivision on top of the DMTETy and compare with it. The new model wins 62% of cases over DMTETy for better looking, and 62% of cases for realistic details as well, demonstrating the effect of surface subdivision module in enhancing the shape details.
Ooe [44]
DECOR [6] Retv. DMTET Adv.
Haseline wins
5% 5%
25% 717%
99%%
DMTET wins
95% 795%
74% 783%
71%7775%
Table 2: User Study on 3D Shape Synthesis from Coarse voxels. each cell, we report percentages of shapes for which the users agree are hetter looking (left) or have better details (right).
4.2 Point Cloud 3D Reconstruction
Experimental Settings We follow the setting from DefTet [18], and use all categories in ShapeNet [3] core data?, which we pre-process using Kaolin [27] to watertight meshes. We sample 5000 points for each shape and add Gaussian noise with zero mean of standard deviation 0.005. For quantitative evaluation, we report the LI Chamfer Distance in the main paper, and refer readers to the Supplement for results in other metrics (3D IOU, L2 Chamfer Distance and Fl score). We additionally report average inference time on the same Nvidia V100 GPU.
We compare DMTET against state-of the-tr 3D reconstruction approaches using different represen- tations: voxels [10], deforming a mesh with a fixed template [54], deforming a mesh generated from a volumetric representation [22], DefTet [18], and implicit functions [44]. For fair comparison, we use the same point cloud encoder for all the methods, and adopt the decoders in the original papers to generate shapes in different representations. We also remove the adversarial loss in this application, since baselines also do not have it. We further compare with oracle performance of MC/MT where the ground truth SDF is utilized to extract iso-surface using MC/MT.
Experimental Results Quantitative results are summarized in Table 3, with a few qualitative examples shown in Fig. 7. Compared to DMC [31], which also predicts the SDF values and supervises with a surface loss, DMTET achieves much better reconstruction quality since training using the marching tetrahedra layer is more efficient than calculating an expectation over all possible configurations within one grid cell as done in DMC [31]. Compared to method that deforms a fixed template (sphere) [54], we reconstruct shapes with different topologies, achieving more faithful results compared to the ground truth shape. When compared with other explicit surface representations that also support different topology [18, 22], our method achieves higher quality results for local
'The ShapeNet license is explained http:///shapenet.rrggemmss
8

---

In
3DR2N2
DMC
Pix2Mesh
ConvOnat
MeshRCNN
DEFTET
Ours
GT
Figure 7: Qualitative results on 3D Reconstruction from Point Clouds: Our model reconstructs shapes with more geometric details compared to baselines.
Canserry
Airpane
Beneh
Deesser
Ca
Chair
Displa
Speaker
Rille
Sola
Tasle
Plune
Vessal
Meon
Time mal.
E.57
174
DMC
1.0
1.45
Inech [4
1.9
0.91
ComOnet (4c
0.82
0.95
0.96
1.12
1.00
0.93
1.22
1.12
0.79
0.1
0.
0.62
0.99
0.95
865
MrshRCNN (2])
6.83
1.01
1.0
133
1.10
0.99
1.30
133
0.83
0.96
1.00
0.71
1.03
1.00
228
DETTET
097
133
035
DMTET worleer Suri.)
6.8
036
000
036
0.90
0000
100
100
0.50
0.5
0.5
0.5
030
030
DMEET Worko, Stil)
0.00
032
038
0.2
0050
008
0.50
0.2
0.03
030
032
0.61
030
0381
DM we Vol.
0.65
075
884
035
0.80
0.70
0.8
0095
0.8
0.7
0.79
0.00
0.7
0.20
DMTET RS Suct.
0.00
0.7
038
0.58
0.88
0.79
033
0.94
0.50
0.75
0.79
0.59
0.76
0.78
108
Table 3: Quantitative Results on Point Cloud Reconstruction (Chamfer I.1). Note that all the networks in the baselines are not designed for this task, and thus use the same encoder and their decoder for fair comparison. We also ablate ourselves by operating fixed grid (DMTET (Def, Vol., Surf.)). removing volume subdivision (DMTET wo Vol.), or surface subdivision (DMTET wo Surf.), the both (DMTET wo (Vol., Surf.)).
geometry, benefiting from the fact that the typology is jointly optimized with the geometry, whereas is separately supervised by occupancy loss in [18, 22]. Compared to neural implicit method [44], we generate higher quality shapes with less artifacts, while running significantly faster at inference. Finally, compared to a voxel-based method [10] at the same resolution, our method recovers more geometric details, benefiting from the predicted vertex deformations as well as the surface loss.
4.2.1 Analysis
MC
We investigate how each component in our representation affects the performance and reconstruction quality.
DMTee wo Vol.Sl] (Vol.Surf)
Chamler
Comparisons with Oracle Performance of MC/MT We first demonstrate the effect of learning explicit surface via MT. We compare with the oracle performance of extracting the iso-surface with MT/MC from the ground truth signed dis- tance fields on the Chair test set ShapeNet, which contains diverse high-quality details. Specifically, for MC/MT, first compute the discretized SDF at different grid resolutions, and compare the extracted surface to the ground truth surface.
Num. of Querled 5DT (oonts(10t))
Figure 8: Comparing our DMTET with
oracle performance MC and MT.
As shown in Fig. 8, MT consistently outperforms MC when querying the same number of points. We found the staggered grids pattern in tetrahedral grid [16, 18] better captures thin structures a limited

---

resolution (Fig. 9). This makes MT a better choice for efficiency reasons. The usage of tetrahedral mesh DMTET follows this motivation. Without deforming the grid, DMTET outperforms the oracle performance of MT by a large margin when querying the same number of points, although DMTET predicts the surface from noisy point cloud. This demonstrates that directly optimizing the reconstructed surface can mitigate the discretization errors imposed by MT to a large extent.
Ablation Studies We further provide ablation studies on the entire ShapeNet test set, which is summarized in Tab. 3. We first compare the ver- sion where we only predict SDF values without learning deform the vertices and volume/surface subdivision with the version that predicts both SDF and the deformation. Predicting deformation along with SDF is significantly more performant, since vertex movements allow for better recon- struction of the underlying surface. This is espe- cially true for categories with thin structures (e.g. lamp) where the grid vertices are desired to align with them. We further ablate the use of volume subdivision and surface subdivision. We show that each component provides an improvement. In particular, volume subdivision has a significant improvement for object categories with fine-grained structural details, such as airplane and lamp, which require higher grid resolutions to model the occupancy change. Surface subdivision generates shapes with a parametric surface, avoiding the quantization errors in the planar faces and produces more visually pleasing results.
MC6x101)
MCB6x10")
(Dt..M Sui) 6x10')
(6x10%)
MT (x10') MT(6x100)
LMMlet vo (Vol.Sur)) (8x10))
GT
Figure 9: We compare trained DMTET oracle per- formance of MT and MC. Number in bracket indicates number of SDF points queried.
5 Conclusion
In this paper, introduced deep 3D conditional generative model that synthesize high- resolution 3D shapes using simple user guides such as coarse voxels. Our DMTET features a novel 3D representation that marries implicit and explicit representations by leveraging the advantages of both. We experimentally show that our approach synthesizes significantly higher quality shapes with better geometric details than existing methods, confirmed by quantitative metries and an extensive user study. By showcasing the ability to upscale coarse voxels such as Minecraft shapes, we hope that we take one step closer to democratizing 3D content creation.
6 Broad Impact
Many fields such as AR/VR, robotics, architecture, gaming and film rely on high-quality 3D content. Creating such content, however, requires human experts, i.c., experienced artists, and a significant amount of development time. contrast, platforms like Minecraft enable millions of users around the world to carve out coarse shapes with simple blocks. Our work aims at creating A.I. tools that would enable even novice users to upscale simple, low-resolution shapes into high resolution, beautiful 3D content. Our method currently focuses on 3D animal shapes. We are not currently aware of and do foresee nefarious use cases of our method.
7 Disclosure of Funding
This work was funded by NVIDIA. Tianchang Shen and Jun Gao acknowledge additional revenue in the form student scholarships from University of Toronto and the Vector Institute, which are not in direct support of this work.
References
[1] Matan Atzmon and Yaron Lipman. Sal: Sign agnostic learning of shapes from raw data, In Proceedings of the JEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2565 -274,, 2020.
[2] Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Generative and discriminative voxel modeling with convolutional neural networks. arXiv preprint 4rX1v:1608.04236, 2016.
10

---

[3] Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-riic 3d model repository. arXiv preprint 151.:0151..03012, 2015.
[4] Ding- Yun Chen, Xiao-Pot Peoo-eiiTT Tian, Yu-Te Te Shen, and Ming Ouhyoung. On visual similarity based 3d model retrieval. In Computer graphics forum, volume 22, pages 223-232. Wiley Online Library, 2003.
[5] Wenzheng Chen, Jun Gao, Huan Ling, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning to predict 3d objects with an interpolation-base -ased differentiable renderer. In Advances In Neural Information Processing Systems, 2019.
[6] Zhiqin Chen, Vladimir G. Kim, Matthew Fisher, Noam Aigerman. Hao Zhang, and Siddhartha Chaudhuri. Decor-gan: 3d shape detailization by conditional refinement. Proceedings of IEEE Conference on Computer Vision and Pottern Recognition (CVPR), 2021.
[7] Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. Bsp-net: Generating compact meshes via binary space partitioning. Proceedings of IEEE Conference on Computer Vixion and Pattern Recognition (CVPR), 2020.
[8] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[9] Julian Chibane, Aymen Mir, and Gerard Pons-Moll. Neural unsigned distance fields for implicit function learning. In Advances in Neural Information Processing Systems (NewrIPS), December 2020.
[10] Christopher B Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In European conference computer vision, pages 628-644. Springer, 2016.
[11] Angela Dai, Daniel Ritchie, Martin Bokeloh, Scott Reed, Jürgen Sturm, and Matthias NieBner. Scancom- plete: Large-scale scene completion and semantic segmentation for 3d scans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4578-4587, 2018.
[12] Angela Dai, Charles Ruizhongtai Qi, and Matthias NieBner. Shape completion using 3d-encoder-predictee cnns and shape synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1868-5877, 2017.
[13] Thomas Davies, Derek Nowrouzezahrai, and Alec Jacobson. Overfit neural networks as a compact shape representation. arXiv preprint 200.:000000098000 2020.
[14] Yu Deng, Jiaolong Yang, and Xin Tong. Deformed implicit field: Modeling 3d shapes with learned dense correspondence. In IEEE Computer Vision and Pattern Recognition, 2021.
[15] Akio Doi and Akio Koide. An efficient method of triangulating equi-valued surfaces by using tetrahedral cells. IEICE TRANSACTIONS on Information and Systems, 74(1) 74(1):214-224, 1991.
[16] Crawford Doran, Athena Chang, and Robert Bridson. Isosurface stuffing improved: acute lattices and feature matching. ACM SIGGRAPH 2013 Talks, pages 1-1, 2013.
[17] Yueqi Duan, Haidong Zhu, He Wang, Yi, Ram Nevatia. and Leonidas Guibas. Curriculum deepsdf. In European Conference on Computer Vision, pages 51-67. Springer, 2020.
[18] Jun Gao, Wenzheng Chen, Tommy Xiang, Clement Fuji Tsang, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Learning deformable tetrahedral meshes for 3d reconstruction. In Advances In Neural Information Processing Systems, 2020.
[19] Jun Gao, Chengeheng Tang, Vignesh Ganapathi-Subramaan Jiahui Huang, Hao Su, and Leonidas J Guibas. Deepspline: Data-driven reconstruction of parametric curves and surfaces. arXiv preprint arXiv:1901.03720 2019.
[20] Jun Gao, Zian Wang. Jinchen Xuan, and Sanja Fidler. Beyond fixed grid: Learning geometric image representation with deformable grid. ECCV, 2020.
[21] Lin Gao, Jie Yang. Tong Wu, Yu-Jie Yuan, Hongbo Fu, Yu-Kun Lai, and Hao Zhang. Sdm-net: Deep generative network for structured deformable mesh. ACM Transactions on Graphics (TOG), 38(6):1-15, 2019.
[22] Georgia Gkioxari, Jitendra Malik, and Justin Johnson. r-nn. In Proceedings of the IEEE International Conference on Computer Vision, pages 2019.
r-can.

---

[23] Thibault Groueix, Matthew Fisher. Vladimir G Kim. Bryan C Russell, and Mathieu Aubry. A papier-mäché approach learning 3d surface generation. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 216-224, 2018.
[24] Christian Häne, Shuhham Tulsiani, and Jitendra Malik. Hierarchical surface prediction for 3d object reconstruction. 2017 International Conference on 3D Vision (3DV), pages 412-420. IEEF, 2017.
[25] Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or Or. Point2mesh: A self celf-prior for deformable meshes. ACM Trans. Graph., (9)... 2020.
[26] Zekun Hao, Hadar Averbuch-Elor, Noah Snavely, and Serge Belongie. Dualsdf: Semantic shape manipula- tion using a two-level representation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7631-7641, 2020.
(27] Krishna Murthy J., Edward Smith, Jean-Francois Lafleche, Clement Fuji Tsang, Artem Rozantsev, Wen- zheng Chen, Tommy Xiang, Rev Lebaredian, and Sanja Fidler. Kaolin: A pytorch library for accelerating 3d deep learning research. arXiv: 1.05063, 2019.
[28] Marian Kleineberg, Matthias Fey, and Frank Weichert. Adversarial generation of continuous implicit shape representations. arXiv preprint (tti:200220004499 2020.
[29] Jiabao Lei and Kui Jia. Analytic marching: An analytic meshing solution from deep implicit surface networks. In International Conference on Machine Learning, pages 5789-5798. PMLR, 2020.
[30] Manyi Li and Hao Zhang. D2im-net: Learning detail disentangled implicit fields from single images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10246- 10255, 2021.
[31] Yiyi Liao, Simon Donné, and Andreas Geiger. Deep marching cubes: Learning explicit surface representa- tions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[32] Huan Ling, Jun Gao, Amlan Kar, Wenzheng Chen, and Sanja Fidler. Fast interactive object annotation with curve-gen. In CVPR, 2019.
[33] Hsueh-Ti Derek Liu, Vladimir G. Kim, Siddhartha Chaudhuri, Noam Aigerman, and Alec Jacobson. Neural subdivision. ACM Trans. Graph., 39(4), 2020.
[34] Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel enn for efficient 3d deep learning. In Advances in Neural Information Processing Systems, 2019.
[35] Charles Loop. Smooth subdivision surfaces based on triangles. January 1987.
[36] William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggruph computer graphics, 2144):163-169, 1987.
[37] Xudong Mao, Qing Li, Haoran Xic, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2794-2802, 2017.
[38] Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In 2015 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 922-928. IEEE, 2015.
[39] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowazin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4460-4470, 2019.
[40] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
[41] Charlie Nash, Yaroslav Ganin, S. M. Ali Eslami, and Peter W. Battaglia. Polygen: An autoregressive generative model of 3d meshes. ICML, 2020.
[42] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcomhe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2019.
[43] Despoina Paschalidou, Angelos Katharopoulos, Andreas Geiger, and Sanja Fidler. Neural parts: Learning expressive 3d shape abstractions with invertible neural networks. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
12

---

[44] Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In European Conference on Computer Vision (ECCV), 2020.
[45] Edoardo Remelli, Artem Lukoianov, Stephan Richter, Benoit Guillard, Timur Bagautdinov, Pierre Baque, and Pascal Fua. Meshsdf: Differentiable iso-surface extraction. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages Curran Associates, Inc., 2020.
22468-24 22478.
[46] Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representations at high resolutions. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3577-3586, 2017.
[47] Shunsuke Saito, Zeng Huang. Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution elothed human digitization. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
[48] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, June 2020.
[49] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Proc. NeurIPS, 2020.
[50] Minhyuk Sung, Hao Su, Vladimir G Kim, Siddhartha Chaudhuri, and Leonidas Guibas. Complementme: Weakly-supervised component suggestions for 3d modeling, ACM Transactions on Graphics (TOG), 36(6):1-12,2)) 2017.
[51] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[52] M. Tatarchenko, A. Dosovitskiy, and T. Brox. Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs. In IEEE International Conference on Computer Vision (ICCV), 2017.
[53] Shuhham Tulsiani, Hao Su, Leonidas J. Guihas, Alexei A. Efros, and Jitendra Malik. Learning shape abstractions by assembling volumetric primitives. In Computer Vision and Pattern Regognition (CVPR), 2017.
[54] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 52-67, 2018.
[55] Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Yu Sun, and Xin Tong. O-CNN: Octree-based Conva- lutional Neural Networks for 3D Shape Analysis. ACM Transactions on Gruphics (SIGGRAPH), 36(4). 2017.
[56] Peng-Shuai Wang, Yang Liu, and Xin Tong. Deep octree-based cnns with output-guided skip connections for 3d shape and scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 266 266,267,22 2020.
[57] Francis Williams, Matthew Trager, Joan Bruna, and Denis Zorin. Neural splines: Fitting 3d surfaces with infinitely-wide neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9949-9958, 2021.
[58] Jiajun Wu, Chengkai Zhang, Tianfan Xue, William T Freeman, and Joshua B Tenenbaum. Learning a probabilistic latent space of object shapes via 3d generative-ade adverssraal modeling. In Advances in Neural Information Processing Systems, pages 82 90, 2016.
[59] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015.
[60] Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and UIrich Neumann. Dism: Deep implicit surface network for high-quality single-view 3d reconstruction. In Advances in Neural Information Processing Systems, pages 490-500 500,000000000 2019.
13

---

[61] Kangxue Yin, Zhiqin Chen, Siddhartha Chaudhuri, Matthe Fisher, Vladimir Kim, and Hao Zhang. Coalesce: Component assembly by learning synthesize connections. In Proc. of 3DV, 2020.
[62] Chenyang Zhu, Kai Xu, Siddhartha Chaudhuri, Renjiao Yi, and Hao Zhang. SCORES: Shape composition with recursive substructure priors. ACM Transactions Graphics, 37(6))Artecle 211, 2018
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We provide extensive experiments in Sec.
(b) Did you describe the limitations of your work? [Yes] We provide the discussion on limitations failure cases in Supplement.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We provide the discussion in the Board Impact section with further discussions in Supplement.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as URL)? [No] The code currently quite uncleaned and requires many dependencies. We are planning release the code after cleaning.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We provide training details in both Sec. in the main paper and Supplement.
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [No] Training existing 3D models, including ours, on large-scale 3D datasets is too computation costly to repeat multiple times.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We provide in the Supplement.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We used ShapeNet [3] core dataset in Sec. 4.2. We also used official code to reproduce baselines with citations.
In particular, ConvONet [44] and DECOR-GAN -GN [6].
(b) Did you mention the license of the assets? [Yes] We provided the license of ShapeNet and Turbosquid.
(c) Did you include any new assets either in the supplemental material or URL? [No]
The Turbosquid data we are using contains proprietary information.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] We discussed in the Sec. 4 and provide further details in the Supplementary Materials.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] We provide discussion on this in Supplement.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [Yes] We provide details in the paper, with full text and screenshot in Supplement.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [No] We did not anticipate the potential participant risks, as we only conduct human studies on generated animals.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes] We provide details in Supplement
15


Reviews:
[{"review_id": "atA49hdpDm", "reviewer": "Reviewer_ztsv", "summary": "This paper proposes deformable tetrahedral grid for 3D reconstruction. It aims to combine the advantage of both implicit surface and explicit surface by extracting the mesh using a differentiable marching tetrahedra layer and refine it using a grach convolutional neural network. The proposed method is evaluated an animal dataset collrected by the authors and ShapeNet objects, taking either a low-res voxel grid or a point cloud as input.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n1. The paper is well-written and easy to follow.\n1. It is interesting to define an adversarial loss on local regions for refining geometry.\n1. The authors motivate the choice of MT instead of MC using a toy example, which I found interesting and convincing.\n1. The proposed method demonstrates superior performance on 3D reconstruction of single objects, in particular for thin objects.\n1. User studies are conducted to evaluate the quality of the 3D reconstruction of the animal dataset.\n\n### Weaknesses\n\n1. The proposed method is fairly similar to DefTet. As I understood, the major differences compared to DefTet are: a) using SDF instead of occupancy, b) hierarchical sampling, c) using GCN to predict vertice offsets and d) adversarial loss. Despite achieving better performance, these improvements appear to be incremental conceptually. Moreover, DefTet conducts experiments on Novel View Synthesis and Multi-view 3D Reconstruction, while the proposed method is only evaluated on 3D single object reconstruction from 3D input. What is the reason for this? It would be great if the author can elaborate more about the key differences compared to DefTet in the paper.\n\n1. Considering the differentiability of MT/MC, I believe that it is also related to the loss. If there is a direct loss defined on the SDF field ($L_{sdf}$ as in this paper), I agree with the authors that the original MT/MC can be applied directly. However, it is not straightforward for me how the topology can be changed freely when only chamfer distance loss ($L_{cd}$) is considered. For example, it might be that $L_{cd}$ is not monotonically decreasing when transitioning from the initial topology to the target topology. Note that this is not a problem when using $L_{sdf}.Some more specific suggestions/questions regarding these concerns:\n\n    a. In the main experiments $L_{sdf}$ is always applied. But the experiment of Fig.A in suppmat uses $L_{cd}$ only. While it is interesting, it is not straightforward to me how the original MT changes topology. It would be great if the authors can demonstrate the topology change of one single tetrahedron when using only $L_{cd}$. For example,  one can show how the topology changes from the left topology to the middle one, and from the middle to the right in Fig.3.\n\n    b. The authors also argue that the refinement step using the GCN module is a reason that allows for topology change (L180). As I understood the topology change depends on the supervision signal (and hence the gradient) instead of the network architecture. It would be great if the authors can elaborate a bit more about this.\n\n1. The idea of using an MLP to predict the implicit surface and extracting the explicit surface such that the loss can be directly applied on the surface is conceptually similar to MeshSDF [38]. It would be good if the author can explain why a comparison to MeshSDF is not considered. \n\n1. While it is great that the authors provide ablation studies to verify some of the design choices, I found the ablation of some losses is missing (e.g. $L_{sdf}$ and $L_{def}).\n\n1. The authors only consider experiments on single objects. It is unclear whether the method can be applied to room level (like in ConvONet).\n\n1. This is not a major concern but the training scheme appears to be quite complicated with multiple steps and takes a very long time (6 days on 4 NVIDIA V100 GPUs).\n\nIn summary, I found this paper well-written and interesting but I have some major concerns regarding comparisons to existing works and the differentiability of MT/MC. I would like to increase my rating if the authors can address my concerns.\n\n### Update after rebuttal\n\nThe authors have addressed my concerns regarding the differentiability of MT/MC. Therefore I vote for accepting this paper.\n\n### Detailed comments\n\n1. It would be good if the authors can provide the motivation for choosing the LSGAN objective instead of other GAN objectives.\n\n1. Some of the citations are not up-to-date. For example, [6] and [37] are both published at conferences while they are still referred to Arxiv.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors provide failure cases in the suppmat and discuss the broader impact of their work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "RviyTzbMezr", "reviewer": "Reviewer_MFHh", "summary": "The authors propose an approach (DMTet) to obtain detailed 3D shape from coarse voxels/pointclouds.\nGiven an input DMTet predicts SDF values at discrete tetrahedral grid. Authors use this SDF to obtain a coarse surface which is further used by GCN to refine the tetrahedral grid. Surface is then recomputed. Authors use a second GCN to predict parameters for loop subdivision to further increase the mesh resolution. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The demonstrated results look good.\n+ Authors include detailed evaluations in supplementary which are appreciated.\n+ Paper is well written and easy to follow.\n\nA few comments:\n1. Supplementary, Sec E.2 Table B. Why do the numbers for point cloud reconstruction on ShapeNet for baselines look different than Table 1 in [16]? Was the experimental setup somehow different?\n2. Eq. 1,2: f(v_i) is input to the GCN, what is the advantage of reconstructing it back?\n3. L286-296: Numbers could be put in a table for easier parsing.\n4. Fig 7: I find it funny that the lamp shown in the second row to showcase superior performance of the method is also used in supplementary fig. J to show the limitations of the method.\n\nAnimals and shapenet objects are relatively smooth shapes. Can the proposed method be used to generate detailed shapes such as clothed 3D humans? This would be a challenging testbed to assess the gains from the proposed surface refinements (adversarial, volumetric and surface).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors discuss limitations of their method and broader impact in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "762C7qRitlo", "reviewer": "Reviewer_QWSU", "summary": "This paper presents a method for synthesizing 3D shapes from coarse voxels using a 3D conditional generative model. \nThe main *claimed* contribution is a differentiable marching tetrahedra layer that can convert the implicit signed distance field to the explicit surface representation. The paper also claims that with such a differentiable marching tetrahedra (MT) layer, one can jointly optimize the surface geometry and topology during training. \nDespite the fact that the presented results do look promising compared to previous works, I doubt that the improvement comes from the proposed differentiable MT layer, due to the fundamental technical flaws which I will elaborate on later.\nInstead, I believe a great part of the performance comes from the direct and full SDF supervision provided in the seemingly minor *regularization* term and the deformable tetrahedral grid, which the authors fail to provide ablation study on these two algorithmic components.  \n", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The technical contribution of the proposed work is very limited. The paper claims that the main contribution comes from the combination of the deformable tetrahedral grid and the differentiable marching tetrahedra (MT) layer. Since the deformable tetrahedral grid is an existing technique, the only new algorithmic component is the differentiable MT layer. However, in this work, they actually directly apply the traditional MT algorithm for converting the signed distance field to mesh representation, with *NO* modification. Though the work also incorporates volume and surface subdivision techniques, these are very common approaches for coarse-to-fine reconstruction. \n\n- There is a fundamental flaw in claiming the original marching tetrahedra algorithm is differentiable. As agreed by the authors, there is a singularity in the original MT algorithm -- when the signs of the edge endpoints are the same, there are no gradients flowing into this edge. In addition, in line 218 to 220, the paper also admits that only the vertices that are close to the iso-surface in the tetrahedral grid receive gradients, while the other vertices do not. Hence, I do not think it is fair to call the proposed MT layer (actually the original MT algorithm) differentiable. \n\n- The writing is very misleading, as it tries to convince the readers that the main improvement comes from the differentiable MT layers (the main contribution of the paper). However, as analyzed in the above point, due to the fundamental flaw in the differentiability of the MT layer, it is very unlikely (if not impossible) that solely using the proposed MT layer can achieve a large topology change to generate high-fidelity results. Hence, I do not believe the proposed MT layer contributes a great part in achieving good performance in the results. Instead, the performance boost should probably be credited to the deformable tetrahedral gird (an existing technique) and the full SDF supervision in the so-called *regularization* term. However, the paper fails to provide any ablation study regarding the effect of these two components. \nI would like to see the performance without the SDF regularization term, the deformable tetrahedral grid, and both. Otherwise, I do not think the evaluation is comprehensive.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "6BoLR1fBXbH", "reviewer": "Reviewer_mF8Y", "summary": "This paper presents a method to generate fine-grained shape from coarse volume or sparse point clouds. The pipeline starts from a feature extractor that produces a feature volume from the input, which is then used to predict and refine sdf values on a pre-defined  tetrahedral grid. Both the vertex location and sdf are then updated, with surface subdivision to capture more local features. At last, a differentiable marching tetrahedra is run to get the mesh. The model is supervised by both Chamfer distance on points sampled from the surface, as well as the SDF of tetrahedra vertices. The overall performance on shape synthesis with coarse volume or point cloud achieves the best quantitative performance compared to the baselines, and the advantages of MT over marching cube are analyzed.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The use of deformable tetrahedra seems to be an elegant, i.e. accurate and efficient, representation for 3D shapes.\n- The performance on both tasks clearly outperform baseline methods, and the qualitative results looks promising.\n\nWeakness:\n- The task achieved by this paper is 3D super-resolution rather than shape synthesis. The pipeline creates a one-to-one mapping between the input and output, and therefore I don't see how novel shape could be synthesized.\n- The deformable tetrahedra is a generic representation not only just valid for the proposed tasks. To fully demonstrate the effectiveness, other common tasks, e.g. shape from image, reconstruction, could be investigated.\n- It might be good to have some tight comparison to the marching cube based pipeline under two tasks. I see both of the tasks could be similarly achieved with regular cubical grid plus deep marching cube [26]. It is good that DMC [26] is compared in table 3. However, one might be interested to see how the pipeline work with exactly the same pipeline using cube instead of tetrahedra. In this way, we would be able to clearly see the advantage of using tetrahedra.\n- When compare MT and MC, run-time efficiency should be provided.\n\nOverall, this work provides an extension of the marching cube based approach with tetrahedra. The major novelty seems to be more engineering oriented (if I didn't miss anything important), and adding more experiments might be helpful to highlight the importance of the contribution.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No comment.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper initially received mixed reviews. After the rebuttal, three reviewers were positive while one remained negative. The shared concerns include the limited technical contribution, the missing ablation studies on $L_{sdf}$, and the lack of a comprehensive evaluation between MT and MC. The AC shares these concerns.

But the AC also agrees with the reviewers that all engineering efforts in making the pipeline working and achieving good results, as well as the potential impact to the community on providing an easy solution to directly adopting MT/MC in a neural network, are useful contributions. The AC, therefore, recommends acceptance of this borderline case. The authors are strongly encouraged to include the missing ablation studies and comparisons with MC in the camera ready.
Decision: Accept (Poster)